Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry

Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry

arXiv:2610.38773v1 [cs.LG] 28 Oct 2026

Igor Sokolov KAUST*

Alexander Tyurin AXXX†, Applied AI Institute‡

Peter Richtárik KAUST

Abstract The empirical success of matrix-aware optimizers, such as Muon [Jordan et al., 2024] and Shampoo [Gupta et al., 2018], highlights the potential of non-Euclidean geometry for deep learning, including large language model (LLM) training. Despite this progress, the theoretical understanding of stochastic optimization in general normed spaces remains limited, particularly in distributed settings. We analyze minibatch stochastic gradient descent (SGD) for smooth nonconvex optimization with respect to an arbitrary norm on a finite-dimensional space. Using the framework of 𝜅-regular normed spaces [Juditsky and Nemirovski, 2008], we derive high-probability oracle complexity bounds under a light-tail condition stated directly in the dual norm induced by the update norm. The bounds make the dependence on geometry explicit through the regularity constant 𝜅 of the dual space. In the Euclidean case, our bound recovers the standard minibatch SGD rate up to logarithmic factors. For some non-Euclidean geometries and noise models, it is sharper than bounds obtained by converting a Euclidean noise assumption, reducing a linear dimension dependence to a logarithmic one. Building on this framework, we introduce Rennala NSGD, a semi-asynchronous distributed algorithm, and establish wall-clock time complexity guarantees under fixed, heterogeneous worker computation times. In the Euclidean case (𝜅 = 1), our bound matches the parameter dependence of classical Rennala SGD up to logarithmic factors.

1

Introduction

Modern optimizers are non-Euclidean in practice. Modern deep-learning training increasingly uses geometry-aware primitives, including matrix preconditioning and orthogonalized updates. Matrixaware optimizers include Shampoo [Gupta et al., 2018], which uses matrix preconditioning, and Muon [Jordan et al., 2024] and Dion [Ahn et al., 2025], which use orthogonalized matrix updates. Recent studies report improved training efficiency over AdamW for Muon [Liu et al., 2025] and Dion [Ahn et al., 2025] in Transformer language-model experiments. Their empirical advantages are often interpreted through the connection between these matrix updates and steepest descent with respect to non-Euclidean norms, such as the spectral norm and related Schatten norms [Bernstein and Newhouse, 2024, Lau et al., 2025]. These norms respect the matrix structure of neural-network parameters, including the weight matrices in Transformer attention and feed-forward layers. This geometric perspective motivates a stochastic theory developed directly in general normed spaces. * Center of Excellence for Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. Correspondence to: Igor Sokolov igor.sokolov.ml@gmail.com. Website: https://igor-sokolov. com. † AXXX, Moscow, Russia. ‡ Applied AI Institute, Moscow, Russia.

40th Conference on Neural Information Processing Systems (NeurIPS 2026).

Theory gap: stochastic non-Euclidean SGD under the “right” noise model. A central modeling question in stochastic non-Euclidean optimization is how to quantify oracle noise. Most existing analyses [Kovalev, 2025, Gruntkowska et al., 2025, Li and Hong, 2025, Shen et al., 2025, Zhang and Gao, 2025, Takezawa et al., 2025] assume Euclidean (or Frobenius) moment bounds or bounded variance and then transfer results to a target non-Euclidean norm via norm equivalence. Such conversions can introduce dimension-dependent inflation and may obscure the regimes in which a geometry-aware method should provably help. We therefore seek guarantees under a noise model stated in the native norm. Our lens: dual-norm light tails and 𝜅-regularity. We measure oracle noise directly in the dual norm ‖·‖* induced by the chosen update norm ‖·‖ (Assumption 3). Using the 𝜅-regularity framework [Juditsky and Nemirovski, 2008], we obtain high-probability guarantees with the additional geometric dependence captured by the dual-space regularity constant 𝜅. Every nonzero finitedimensional normed space admits such a constant, so this approach does not restrict the choice of update norm (Section 3.3). For suitable geometries and noise models, it yields sharper bounds than Euclidean/Frobenius conversions (Section 3.6.1). Problem setup. Throughout the paper, 𝐸 denotes a finite-dimensional space with dim(𝐸) ≥ 1. We consider the distributed, non-convex stochastic optimization problem expressed as a finite sum over 𝑁 workers: {︃ }︃ 𝑁 1 ∑︁ min 𝑓 (𝑥) := 𝑓𝑖 (𝑥) , (1) 𝑥∈𝐸 𝑁 𝑖=1 where each 𝑓𝑖 is an expectation over a worker-specific data distribution 𝒟𝑖 : 𝑓𝑖 (𝑥) := E𝜉𝑖 ∼𝒟𝑖 [𝑓𝑖 (𝑥; 𝜉𝑖 )] .

(2)

The space 𝐸 may represent, for instance, vectors in R𝑑 or matrices in R𝑚×𝑛 . For each worker 𝑖, the function 𝑓𝑖 : 𝐸 × S𝑖 → R is defined over a sample space S𝑖 , and 𝜉𝑖 is a random variable drawn from 𝒟𝑖 . The problem is unconstrained: the methods we analyze contain no projection, so nothing confines the iterates to a proper subset of 𝐸, and every assumption below is stated on all of 𝐸. This formulation is particularly relevant to Federated Learning (FL) [Konečný et al., 2016, McMahan et al., 2017, Kairouz et al., 2021]. In this context, two primary forms of heterogeneity arise: device heterogeneity, which concerns variations in system capabilities, and statistical heterogeneity, where data distributions 𝒟𝑖 differ across workers. To isolate the effects of device heterogeneity, we assume that all workers share the same sample loss 𝑓 (·; ·) and sampling distribution 𝒟. Thus, 𝑓𝑖 (𝑥; 𝜉) = 𝑓 (𝑥; 𝜉) and 𝒟𝑖 = 𝒟 for every worker 𝑖, so all local objectives coincide with 𝑓 (𝑥) = E𝜉∼𝒟 [𝑓 (𝑥; 𝜉)]. In this case, (1) reduces to min {𝑓 (𝑥) := E𝜉∼𝒟 [𝑓 (𝑥; 𝜉)]} . 𝑥∈𝐸

(3)

Our algorithms access 𝑓 through a stochastic first-order oracle: given a query point 𝑥, it returns a random vector ∇𝑓 (𝑥; 𝜉) ∈ 𝐸 * that estimates ∇𝑓 (𝑥). In the sample-loss setting above, this response is the gradient of 𝑓 (𝑥; 𝜉) with respect to 𝑥. More generally, we allow stochastic estimates that need not be gradients of sample losses; the oracle assumptions are stated formally in Section 3.2. In the parallel setting, all workers use the same oracle map and sampling distribution 𝒟. Throughout the paper, we assume that 𝑓 is 𝐿-smooth (Assumption 1) and bounded from below (Assumption 2), and that the oracle responses are unbiased and satisfy a light-tail condition in the induced dual norm ‖·‖* (Assumption 3). We work in the following parallel computing setup: (i) A fixed number of 𝑁 workers are available to perform computations in parallel. (ii) Worker 𝑖 requires a fixed time 𝜏𝑖 seconds to compute one stochastic gradient after receiving a query point. These times remain fixed throughout the run, and we index the workers so that 0 < 𝜏1 ≤ 𝜏2 ≤ · · · ≤ 𝜏𝑁 . (iii) Communication in both directions between the server and the workers, and server-side processing, including evaluation of the sharp update, are treated as instantaneous, as in the computation model of Section 4. From oracle complexity to wall-clock time: heterogeneous parallelism. On the systems side, asynchronous SGD has been shown to improve wall-clock convergence guarantees over synchronous 2

minibatch SGD that waits for all workers, in settings with heterogeneous worker speeds [Mishchenko et al., 2022]. This line of work emphasizes time complexity—the wall-clock time needed to reach a prescribed accuracy—as a natural measure of parallel performance. In the Euclidean setting, Tyurin and Richtárik [2023] establish a lower bound for their zero-respecting algorithm class and a matching upper bound, up to constants, for Rennala SGD under their specified unbiased bounded-variance oracle model with fixed worker computation times. On the optimization side, non-Euclidean methods are motivated by modern matrix-aware updates, yet their interaction with computational heterogeneity remains poorly understood. Bridging these two directions requires (i) tight control of stochastic gradient noise in the induced dual norm ‖·‖* associated with the chosen update norm ‖·‖ and (ii) a wall-clock analysis that accounts for heterogeneous worker speeds under non-Euclidean updates. Together, these two directions motivate the following question: Should stochastic gradient noise be controlled in a Euclidean or Frobenius norm and then converted, via norm equivalence, to the dual norm ‖·‖* induced by the update norm ‖·‖? Or can a noise condition stated directly in ‖·‖* yield sharper oracle and wall-clock guarantees? Notation. The notation used throughout the paper is collected in Section A, which gives the main definitions followed by a consolidated reference table. Our contributions. We study non-convex stochastic optimization in a normed space (𝐸, ‖·‖) and focus on the statistically homogeneous but computationally heterogeneous parallel regime. Our goal is to characterize both (i) the number of stochastic gradients needed to reach an 𝜀-stationary point and (ii) the wall-clock time needed when workers have different speeds. Our contributions are threefold: (i) Minibatch non-Euclidean SGD under a dual-norm light-tail oracle. We establish highprobability oracle-complexity bounds for minibatch SGD under the stated smoothness, lowerboundedness, and unbiased dual-norm light-tail assumptions. The key ingredient is a minibatch concentration inequality for fresh i.i.d. samples in regular dual spaces (Lemma 1). Combining it with non-Euclidean descent, we show that an 𝜀-stationary point is reached with probability at least 1 − 𝛿 using (︂ )︂ 𝐿∆ 𝜅𝐿∆𝜎 2 ̃︀ 𝒪 + (4) 𝜀 𝜀2 stochastic gradient evaluations (Theorem 1 and Corollary 1), where ∆ = 𝑓 (𝑥0 ) − 𝑓 inf is the initial functional gap and 𝜅 is a regularity constant of the dual space (𝐸 * , ‖·‖* ). In the Euclidean case, where 𝜅 = 1, this matches the optimal oracle complexity up to logarithmic factors for general unbiased light-tail oracles, in the regime described in Section 3.6. (ii) Sharp regularity bounds and improvements over norm-equivalence conversions. We establish matching upper and lower bounds on the optimal 𝜅-regularity parameter for ℓ𝑑𝑝 and Schatten-𝑝 spaces, exact for 1 ≤ 𝑝 < 2 and up to absolute constants for 2 ≤ 𝑝 ≤ ∞ (Table 3 and Section L). For some non-Euclidean geometries and noise models, the native-norm analysis yields significantly smaller oracle-complexity bounds than those obtained by converting a Euclidean/Frobenius noise assumption. In particular, for the ℓ1 update norm under dense Rademacher perturbations, the statistical coefficient grows logarithmically rather than linearly with the dimension. We quantify these comparisons for ℓ𝑑𝑝 and Schatten-𝑝 dual geometries in Section 3.6.1 and Section J. (iii) Wall-clock guarantees for semi-asynchronous non-Euclidean SGD: Rennala NSGD. Building on the minibatch theory, we introduce Rennala NSGD, a semi-asynchronous parallel method for spaces with non-Euclidean geometry, which collects fresh minibatches at the current iterate and discards stale gradients. In a system with worker computation times 𝜏𝑖 , we prove (Theorem 2) the wall-clock time bound ⎞ ⎛ (︃ )︃−1 (︂ )︂ 𝑆 2 ∑︁ 1 𝐿∆ 𝜅𝐿∆𝜎 1 ̃︀ ⎝ min ⎠. 𝒪 + (5) 𝑆 𝑖=1 𝜏𝑖 𝜀 𝑆𝜀2 𝑆∈[𝑁 ] In the Euclidean case, where 𝜅 = 1, (5) matches the parameter dependence of the classical Rennala SGD time-complexity bound up to logarithmic factors [Tyurin and Richtárik, 2023] (Section 4.1). 3

2

Related Work

Our work connects two research threads: (i) stochastic optimization in a general normed space (𝐸, ‖·‖) with non-Euclidean norm, and (ii) time-aware parallel optimization under computational heterogeneity. Non-Euclidean Optimization. Recent work on large-scale training has explored matrix-aware optimizers, including the Muon and Shampoo families, that approximate steepest-descent directions in non-Euclidean norms [Jordan et al., 2024, Shah et al., 2025, An et al., 2025]. Several existing theoretical analyses of these methods in the non-distributed setting [Li and Hong, 2025, Shen et al., 2025, Kim and Oh, 2026, Nagashima and Iiduka, 2026, Kovalev, 2025] control stochastic gradient noise in Euclidean or Frobenius norms. Some of these analyses use norm equivalence to obtain guarantees in a target non-Euclidean norm. Such conversions can introduce dimension- or rankdependent factors. Our analysis instead imposes a light-tail condition directly in the dual norm induced by the update geometry. In distributed settings, several works have explored Muon-style methods. Zhang and Gao [2025] and Takezawa et al. [2025] analyze synchronous federated training and address the statistical bias of local updates, while Gruntkowska et al. [2025] focuses on communication efficiency via error feedback. These studies primarily target communication overhead or local-update bias within synchronized rounds (typically under Frobenius/Euclidean noise models), and do not address computational heterogeneity or wall-clock bounds. Stochastic Optimization in Regular Spaces. The 𝜅-regularity framework of Juditsky and Nemirovski [2008] provides large-deviation bounds for martingale sums in finite-dimensional normed spaces, with geometric dependence captured by the regularity constant 𝜅. Under conditional exponential-moment bounds on the squared norms of the increments, they obtain sub-Gaussian deviation bounds for the norm of the martingale sum. These tools were subsequently incorporated into the analysis of stochastic first-order methods [see, e.g., Juditsky et al., 2013]. Related martingale large-deviation bounds have also been used to derive non-asymptotic confidence bounds for assessing the quality of solutions to stochastic optimization problems [Guigues et al., 2017]. Time Complexity in Heterogeneous Parallel SGD. In the distributed regime, parallelizing SGD is a natural strategy for accelerating training. The simplest approach, minibatch SGD, synchronizes 𝑁 workers at each step to compute gradients in parallel, after which a central server performs the 𝛾 ∑︀𝑁 𝑘 𝑘 update [Gower et al., 2019, Cotter et al., 2011, Goyal et al., 2017]: 𝑥𝑘+1 = 𝑥𝑘 − 𝑁 𝑖=1 ∇𝑓 (𝑥 ; 𝜉𝑖 ). For a general minibatch size 𝐵 and a suitable stepsize, the standard Euclidean/Frobenius analysis gives (︁ )︁ 2 𝜎F 𝐿F Δ 𝐿F Δ F 4 the iteration-complexity bound 𝒯SGD (𝐵) = 𝒪 𝜀F + 𝐵𝜀2 [Ghadimi and Lan, 2013], with F 𝐵 = 𝑁 for the synchronized method above. Since each iteration waits for all workers, heterogeneous worker speeds make the slowest worker the bottleneck [Dutta et al., 2018, Chen et al., 2016]: if F worker 𝑖 takes 𝜏𝑖 seconds per gradient, the runtime scales as 𝜏max 𝒯SGD (𝑁 ), where 𝜏max := max𝑖 𝜏𝑖 . To overcome this synchronization barrier, Asynchronous SGD (ASGD) was introduced [Recht et al., 2011, Feyzmahdavian et al., 2016, Nguyen et al., 2018, Arjevani et al., 2020]. In its standard form, the server applies each arriving stochastic gradient immediately: 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 ∇𝑓 (𝑥𝑘−𝛿𝑘 ; 𝜉 𝑘−𝛿𝑘 ), where 𝛿𝑘 is the gradient delay and 𝛾𝑘 may depend on it. Recent theoretical advances [Mishchenko et al., 2022, Koloskova et al., 2022, Cohen et al., 2021] establish wall-clock bounds for suitable (︁ ∑︀ )︁−1 𝑁 F asynchronous methods with fixed worker computation times 𝜏𝑖 of order 𝑁1 𝑖=1 𝜏1𝑖 · 𝒯SGD (𝑁 ), governed by the harmonic-mean worker time rather than the slowest worker (Section C.2). However, to the best of our knowledge, the available wall-clock bounds for classical ASGD remain suboptimal in general. Tyurin and Richtárik [2023] close this gap with Rennala SGD, a semiasynchronous method that uses only gradients computed at the current iterate. Its wall-clock bound

4 The bound assumes smoothness, lower boundedness, unbiased bounded-variance noise, and fresh independent sam[︁ ]︁ ples. It ensures E ‖∇𝑓 (^ 𝑥)‖2F ≤ 𝜀F for an iterate 𝑥 ^ selected uniformly from the iterates. The subscript F denotes

Euclidean/Frobenius parameters, and Δ = 𝑓 (𝑥0 ) − 𝑓 inf .

4

is min𝑆∈[𝑁 ]

{︂(︁

1 𝑆

∑︀𝑆

1 𝑖=1 𝜏𝑖

)︁−1

}︂ F 𝒯SGD (𝑆) , where 0 < 𝜏1 ≤ · · · ≤ 𝜏𝑁 and 𝑆 counts the fastest

workers considered in the bound. This bound is minimax-optimal up to constants in their model.5 Asynchrony with non-Euclidean updates has been explored mostly in convex/online regimes [Duchi et al., 2013, Feyzmahdavian et al., 2016, Joulani et al., 2019]. These methods use mirror descent or dual averaging, with non-Euclidean geometry specified through a prox function and its associated Bregman divergence. Our approach instead uses steepest-descent steps defined by an update norm and a fixed sharp selection. These works focus on delay-robust iteration/regret guarantees, and do not provide time-optimal wall-clock bounds under computational heterogeneity, nor nonconvex stationarity guarantees in general normed spaces. We combine our stochastic analysis of SGD for arbitrary non-Euclidean norms with a Rennalastyle semi-asynchronous architecture to obtain oracle and wall-clock guarantees beyond the Euclidean/Frobenius regime. Additional literature review references are deferred to Appendix C.

3

SGD Under Non-Euclidean Norms

We derive high-probability convergence and oracle-complexity guarantees for minibatch SGD in nonconvex optimization, using steepest-descent updates in an arbitrary norm ‖·‖ and measuring oracle noise in its induced dual norm ‖·‖* . Notation is collected in Section A. 3.1

Non-Euclidean Update Rule and the Sharp Operator

Minibatch non-Euclidean SGD. At each iteration 𝑘, draw a fresh minibatch of i.i.d. samples {𝜉𝑗𝑘 }𝐵 𝑗=1 from 𝒟, independent of the past. Denote the minibatch stochastic gradient as 𝐵

𝑔 𝑘 :=

1 ∑︁ ∇𝑓 (𝑥𝑘 ; 𝜉𝑗𝑘 ) ∈ 𝐸 * . 𝐵 𝑗=1

(6)

Fix a Borel-measurable selection 𝑠 ↦→ 𝑠♯ with 𝑠♯ ∈ 𝜕𝜑* (𝑠) for every 𝑠 ∈ 𝐸 * . Given a prescribed sequence of positive stepsizes {𝛾𝑘 }, the non-Euclidean minibatch SGD update is (︀ )︀♯ (︀ )︀♯ 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 𝑔 𝑘 , where 𝑔 𝑘 ∈ 𝜕𝜑* (𝑔 𝑘 ). (7) 1 . The convergence and complexity bounds below use the constant choice 𝛾𝑘 ≡ 2𝐿

Steepest descent / LMO form. The update in (7) admits equivalent steepest-descent and Linear Minimization Oracle (LMO) characterizations in (𝐸, ‖·‖) (Section G).6 The LMO form, with the minimizer determined by the fixed sharp selection, is ⃦ ⃦ 𝑥𝑘+1 ∈ lmo (𝑔 𝑘 ), 𝑡𝑘 := 𝛾𝑘 ⃦𝑔 𝑘 ⃦ , (8) *

ℬ(𝑥𝑘 ,𝑡𝑘 )

⃦ ⃦ ⟨︀ ⟩︀ where lmoℬ(𝑥𝑘 ,𝑡𝑘 ) (𝑔 𝑘 ) := arg min𝑥∈ℬ(𝑥𝑘 ,𝑡𝑘 ) 𝑔 𝑘 , 𝑥 and ℬ(𝑥𝑘 , 𝑡𝑘 ) := {𝑥 ∈ 𝐸 : ⃦𝑥 − 𝑥𝑘 ⃦ ≤ 𝑡𝑘 }. Special cases of the update (7). Different choices of the update norm ‖·‖ recover familiar update rules (up to stepsize reparameterizations). ∙ ℓ2 norm (Euclidean case). If ‖·‖ = ‖ · ‖2 , then ‖·‖* = ‖ · ‖2 and (𝑔 𝑘 )♯ = 𝑔 𝑘 , hence (7) reduces to classical SGD [Gower et al., 2019, Gorbunov et al., 2020, Demidovich et al., 2023]. ∙ ℓ∞ norm. If ‖·‖ = ‖ · ‖∞ (so ‖·‖* = ‖ · ‖1 ), then (𝑔 𝑘 )♯ ∈ ‖𝑔 𝑘 ‖1 𝜕‖𝑔 𝑘 ‖1 and a canonical selection is (𝑔 𝑘 )♯ = ‖𝑔 𝑘 ‖1 · sign(𝑔 𝑘 ), where sign is applied componentwise with sign(0) = 0. This yields a sign-type update closely related to signSGD [Bernstein et al., 2018, Sun et al., 2023]. ∙ ℓ1 norm. If ‖·‖ = ‖ · ‖1 (so ‖·‖* = ‖ · ‖∞ ), then (𝑔 𝑘 )♯ ∈ ‖𝑔 𝑘 ‖∞ 𝜕‖𝑔 𝑘 ‖∞ , i.e., (𝑔 𝑘 )♯ is supported on (a subset of) the coordinates attaining ‖𝑔 𝑘 ‖∞ . For 𝑔 𝑘 ̸= 0, an extreme-point selection yields a 5 The guarantee concerns Euclidean/Frobenius stationarity in expectation under their unbiased bounded-variance oracle model with fixed worker computation times; optimality is relative to their class of zero-respecting algorithms. 6 The LMO view is common in analyses of LMO-based optimizers (including Muon and its federated variants); see, e.g., Takezawa et al. [2025]. More broadly, the perspective that many widely used optimizers can be interpreted as steepest-descent methods under suitable norms has been developed in detail by Bernstein and Newhouse [2024].

5

greedy single-coordinate step in (7) [Tseng and Bertsekas, 1991, Tseng and Yun, 2009, Nutini et al., 2015]. A general sharp selection may update several coordinates tied for the largest |𝑔𝑖𝑘 |. A detailed derivation of the update (7) for the ℓ1 and ℓ∞ update norms is deferred to Section F. ∙ Spectral norm and orthogonalized matrix updates. When 𝐸 = R𝑚×𝑛 and ‖·‖ = ‖·‖op is the spectral ⃦ norm, ⃦ the dual norm is the nuclear norm ‖·‖nuc . With the canonical sharp selection (G𝑘 )♯ = ⃦G𝑘 ⃦nuc orth(G𝑘 ), the update takes the following orthogonalized form (derivation in Appendix H): ⃦ ⃦ X𝑘+1 = X𝑘 − 𝛾𝑘 ⃦G𝑘 ⃦nuc · O𝑘 , O𝑘 = orth(G𝑘 ), (9) where G𝑘 is the minibatch gradient in 𝐸 * and orth(G) := (GG⊤ )†/2 G.

(10)

For a compact rank-𝑟 SVD G = U𝑟 Σ𝑟 V𝑟⊤ , where 𝑟 = rank(G) and only positive singular values are retained, one has orth(G) = U𝑟 V𝑟⊤ (the polar factor / partial isometry).7 3.2

Oracle Model

Following Tyurin and Richtárik [2023], we consider a jointly measurable stochastic gradient oracle (𝑥, 𝜉) ↦→ ∇𝑓 (𝑥; 𝜉) ∈ 𝐸 * , with a fresh independent sample 𝜉 ∼ 𝒟 at each call.8 We analyze (7) under the following assumptions. Assumption 1 (Non-Euclidean 𝐿-smoothness). The function 𝑓 is 𝐿-smooth with respect to ‖·‖ if ‖∇𝑓 (𝑥) − ∇𝑓 (𝑦)‖* ≤ 𝐿 ‖𝑥 − 𝑦‖ for all 𝑥, 𝑦 ∈ 𝐸. Assumption 2 (Lower bound). There exists 𝑓 inf ∈ R such that 𝑓 (𝑥) ≥ 𝑓 inf for all 𝑥 ∈ 𝐸. Assumption 3 (Dual-norm light-tail oracle induced by the update norm). For all 𝑥 ∈ 𝐸, the stochastic gradient ∇𝑓 (𝑥; 𝜉) is unbiased, E𝜉 [∇𝑓 (𝑥; 𝜉)] = ∇𝑓 (𝑥), and satisfies the light-tail condition in the induced dual norm )︃]︃ [︃ (︃ 2 ‖∇𝑓 (𝑥; 𝜉) − ∇𝑓 (𝑥)‖* ≤ exp(1) (11) E𝜉 exp 𝜎2 for some 𝜎 > 0. Assumptions 1 and 2 are standard in the analysis of stochastic first-order methods [Ghadimi and Lan, 2013, Gower et al., 2019]. Assumption 3 represents a non-Euclidean generalization of the standard light-tail condition—typically formulated in the Euclidean or Frobenius norm—widely used in optimization theory [Lan, 2020, Tyurin, 2024]. Our bounds use the dual-norm noise scale 𝜎 and regularity constant 𝜅 directly, without Euclidean conversion. Whether this improves the bound depends on both the geometry and the noise model (Section 3.6.1). Further discussion of Assumption 3 appears in Section D. 3.3

𝜅-Regularity and Norm-Dependent Constants

To obtain stochastic guarantees under an arbitrary update norm, we rely on the framework of 𝜅-smooth and 𝜅-regular normed spaces introduced by Juditsky and Nemirovski [2008]. We briefly recall the key definitions. Definition 1 (𝜅𝑉 -Smooth and 𝜅𝑉 -Regular Spaces). Let (𝑉, ‖ · ‖𝑉 ) be a finite-dimensional normed space and define 𝜑(𝑢) := 12 ‖𝑢‖2𝑉 . (i) We say that (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -smooth for some 𝜅𝑉 ≥ 1 if 𝜑 is continuously differentiable and satisfies 𝜑(𝑢 + 𝑣) ≤ 𝜑(𝑢) + ⟨∇𝜑(𝑢), 𝑣⟩ + 𝜅2𝑉 ‖𝑣‖2𝑉 for all 𝑢, 𝑣 ∈ 𝑉 . 7 This direction is obtained from the unregularized Shampoo update with accumulation disabled, interpreting inverse powers via pseudoinverses [Bernstein and Newhouse, 2024, Story II, Eqs. (15)–(16)]. Muon uses a numerical approximation to the same polar map, computed by Newton–Schulz iterations and applied to a momentum update [Jordan et al., 2024]. 8 The response ∇𝑓 (𝑥; 𝜉) need not be the gradient of a sample loss; sample-loss gradients are a special case of this oracle model.

6

(ii) We say that (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular if there exists an auxiliary norm ‖ · ‖𝑉,+ on 𝑉 that is 𝜅𝑉,+ -smooth for some 𝜅𝑉,+ ∈ [1, 𝜅𝑉 ] and satisfies the compatibility chain ‖𝑢‖2𝑉 ≤ ‖𝑢‖2𝑉,+ ≤

𝜅𝑉 ‖𝑢‖2𝑉 , 𝜅𝑉,+

∀𝑢 ∈ 𝑉.

(12)

The constant 𝜅𝑉 ≥ 1 for which (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular is called the constant of regularity of (𝑉, ‖ · ‖𝑉 ). Remark 1 (How we use Definition 1 in this paper). Definition 1 is stated for an arbitrary finitedimensional normed space (𝑉, ‖ · ‖𝑉 ). In the remainder of the paper, we will apply it primarily to the dual space 𝑉 = 𝐸 * equipped with the induced dual norm ‖ · ‖𝑉 = ‖·‖* , since stochastic gradients and noise live in 𝐸 * and are measured in ‖·‖* . To simplify notation, we write 𝜅 := 𝜅𝐸 * , 𝜅+ := 𝜅𝐸 * ,+ , and denote the auxiliary norm on 𝐸 * by ‖·‖*,+ := ‖ · ‖𝐸 * ,+ . With this convention, the compatibility chain (12) specializes to 2

2

‖𝑠‖* ≤ ‖𝑠‖*,+ ≤

𝜅 2 ‖𝑠‖* , 𝜅+

∀𝑠 ∈ 𝐸 * .

(13)

The condition (13) does not restrict the choice of the update norm ‖·‖ (and hence of the dual norm ‖·‖* ). Indeed, for any nonzero finite-dimensional space 𝐸, its dual (𝐸 * , ‖·‖* ) is automatically 𝜅-regular for some 𝜅 ≤ dim(𝐸) by Proposition 10, and this worst-case linear dependence on the dimension cannot be improved in general (Proposition 11). For ℓ𝑑𝑝 and Schatten-𝑝 norms, however, the regularity parameter can often be much smaller.9 Section L collects these bounds and their proofs, with a summary of the auxiliary norms in Table 3. 3.4

Regularity vs. Euclidean Norm Equivalence

A common alternative is to compare the induced dual norm ‖·‖* with a Euclidean reference norm and then apply Euclidean concentration. Regularity offers a more flexible version of this approach: it allows a smooth auxiliary norm that is comparable to ‖·‖* and tailored to the target geometry. This freedom can substantially sharpen the geometric bounds. For dual ℓ𝑑𝑝 spaces with 2 ≤ 𝑝 ≤ ∞, tuning the auxiliary norm yields the optimal order 𝜅 = Θ(min{𝑝, ln(𝑒𝑑)}), compared with 𝜅 = 𝑑1−2/𝑝 for the Euclidean choice. At 𝑝 = ∞, the gain is striking: linear dimension dependence becomes logarithmic. At 𝑝 = 2, both choices give 𝜅 = 1; for 1 ≤ 𝑝 < 2, a scaled Euclidean proxy is already optimal. These gains require no change to the algorithm: the auxiliary norm is used only in the analysis. For suitable noise models, they also translate into sharper oracle-complexity bounds (Section 3.6.1). Details and the Schatten-𝑝 counterpart appear in Section I. 3.5

Key Lemma: Minibatch Concentration in Regular Dual Spaces

The following concentration inequality is the technical core of the analysis: it controls the minibatch error directly in the induced dual norm ‖·‖* , with the geometry entering only through 𝜅. Lemma 1 (Concentration in Regular Normed Spaces). Let 𝜅 be a regularity constant of the dual [︀ space ]︀ (𝐸 * , ‖·‖* ). Assume that {𝜂 𝑗 }𝐵 of i.i.d. random vectors in 𝐸 * such that E 𝜂 𝑗 = 0 𝑗=1 is a sequence [︂ (︂ 𝑗 2 )︂]︂ ‖𝜂 ‖* and they satisfy the light-tail condition: E exp ≤ exp(1). Then for any 𝜆 ≥ 0, the aver𝜎2 }︁ {︁ √ )︀ (︀√ ∑︀ 𝐵 age 𝜂¯ = 𝐵1 𝑗=1 𝜂 𝑗 satisfies the concentration inequality: Prob ‖¯ 𝜂 ‖* ≥ 2𝜅 + 2𝜆 √𝜎𝐵 ≤ 2

exp(−𝜆 /3). Proof idea. 𝜅-regularity provides a smooth auxiliary norm on 𝐸 * , enabling a martingale concentration argument. The comparison (13) then yields concentration in ‖·‖* without a detour through Euclidean/Frobenius conversions, with geometric dependence captured by 𝜅. Further explanation and the proof are given in Section M.1. 9We focus on these families because their special cases 𝑝 ∈ {1, 2, ∞} recover norms commonly used in practice; see

Section A.

7

Space (𝐸 * , ‖·‖* ) (R𝑑 , ‖·‖1 ) (R𝑑 , ‖·‖2 ) (R𝑑 , ‖·‖∞ )

Euclidean/Frobeniusa Geometry-native, Rademacherb (︀ )︀ (︀ )︀ ̃︀ 𝑑3 𝐿Δ/𝜀2 ̃︀ 𝑑2 𝐿Δ/𝜀2 𝒪 𝒪 ̃︀ (𝑑𝐿Δ/𝜀2 ) ̃︀ (𝑑𝐿Δ/𝜀2 ) 𝒪 𝒪 2 ̃︀ ̃︀ 𝑑𝐿Δ 𝒪( /𝜀 ) 𝒪 (ln(𝑒𝑑)𝐿Δ/𝜀2 )

Geometry-native, Gaussianb (︀ )︀ ̃︀ 𝑑3 𝐿Δ/𝜀2 𝒪 ̃︀ (𝑑𝐿Δ/𝜀2 ) 𝒪 (︀ )︀ ̃︀ (ln(𝑒𝑑))2 𝐿Δ/𝜀2 𝒪

a Euclidean/Frobenius concentration followed by norm conversion; see (63). b Concentration directly in the dual norm; see (14).

Table 1: Statistical oracle-cost terms, omitting the factor 𝜈 2 and the common leading term ̃︀ Formulas hold for 𝑑 ≥ 1. Within each row, green and red mark smaller and larger 𝒪(𝐿∆/𝜀). asymptotic orders; blue marks agreement. Colours do not imply strict improvement in every finite dimension.

3.6

Main Theorem: High-Probability Oracle Complexity

We now combine the descent analysis under non-Euclidean smoothness with the minibatch concentration lemma. Theorem 1 (Minibatch non-Euclidean SGD under light tails). Let 𝜅 be a regularity constant of the dual space (𝐸 * , ‖·‖* ). Suppose Assumptions ⌈︀ 128𝐿Δ ⌉︀ 1, 2, and 3 hold. Run the update (7) with 1 the constant 𝛾𝑘 ≡ 2𝐿 for 𝑇 ⌉︃= }︃ 15𝜀 iterations, with a minibatch size chosen as {︃⌈︃stepsize (︁√ )︁2 √ 16𝜎 2 2𝜅+ 6 log(𝑇 /𝛿) 𝐵 = max , 1 . Then, with probability at least 1 − 𝛿, the iterates 𝜀 ⃦ ⃦2 satisfy min0≤𝑘≤𝑇 −1 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀. Theorem 1 sets 𝑇 using the descent argument and chooses 𝐵 to control the gradient-estimation error simultaneously over all 𝑇 iterations. Since each iteration uses 𝐵 stochastic gradients, the following corollary combines these choices to bound the total oracle cost 𝐵𝑇 . Corollary 1 (Norm-native oracle complexity under dual-norm light tails). Suppose the assumptions 1 of Theorem 1 hold. Run the update (7) with the constant stepsize 𝛾𝑘 ≡ 2𝐿 , and choose 𝑇 and 𝐵 as in that theorem. Let 𝑁* := 𝐵𝑇 denote the total number of stochastic-gradient evaluations. Then with ⃦ ⃦2 probability at least 1 − 𝛿, the method produces an iterate satisfying min0≤𝑘≤𝑇 −1 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 after at most )︂ (︂ 𝐿∆ 𝜅𝐿∆𝜎 2 ̃︀ + 𝑁* = 𝒪 (14) 𝜀 𝜀2 ̃︀ hides logarithmic factors in 𝑇 and 1/𝛿. stochastic-gradient evaluations, where 𝒪(·) The rate is optimal (︀in 𝜀 and 𝜎. Jin [2026, )︀ Theorem 4.1 and the proof of Proposition 8.9] establishes a lower bound of Ω 𝐿∆/𝜀 + 𝐿∆𝜎 2 /𝜀2 oracle calls for general unbiased first-order oracles with uniformly bounded noise, which satisfy our oracle model and light-tail condition (Section 3.2).10 In the Euclidean case, where ‖·‖* = ‖·‖2 and 𝜅 = 1, this matches (14), so our rate is optimal up to logarithmic factors over the general unbiased light-tail oracle class. For a general update norm the two are not literally comparable, since 𝜎 is then measured in ‖·‖* rather than in ‖·‖2 ; what (14) adds in that case is the single geometry factor 𝜅. (︀ )︀ ̃︀ 1 + 𝜅𝜎2/𝜀 , with 𝐵 ≥ 1. The Euclidean On the size of the minibatch. Theorem 1 uses 𝐵 = 𝒪 analyses of Tyurin and Richtárik [2023] and Tyurin [2024] have the same dependence on 𝜎 2 /𝜀, up to 1 12 logarithmic factors.11 For exact gradients, we set 𝐵 = 1 and retain 𝛾𝑘 ≡ 2𝐿 . Deterministic descent then gives 𝑁* = 𝒪(𝐿∆/𝜀), explaining the noise-independent leading term in (14). 10 The lower bound applies to our squared-gradient accuracy criterion at fixed confidence, in the worst case over dimensions, for 0 < 𝜀 ≤ 𝑐 min{𝜎 2 , 𝐿Δ} with a universal constant 𝑐 > 0. 11 The lower bound of Jin [2026] constrains the total cost 𝐵𝑇 , but not its division between iterations and batch size; it therefore does not establish that our particular batch size is necessary. 12 This case is treated separately from Assumption 3, which assumes 𝜎 > 0.

8

Algorithm 1 Rennala NSGD (server)

Algorithm 2 Rennala NSGD (worker loop) 1: Set 𝑔 = 0 and 𝑘 ′ = −1. 2: loop 3: Send (𝑔, 𝑘′ ) to the server. 4: Receive (𝑥𝑘 , 𝑘) from the server. 5: Set 𝑘′ = 𝑘. 6: Sample 𝜉 ∼ 𝒟. 7: Compute 𝑔 = ∇𝑓 (𝑥𝑘 ; 𝜉). 8: end loop

0

1: Input: initial point 𝑥 ∈ 𝐸, stepsizes {𝛾𝑘 > 0},

minibatch size 𝐵, number of iterations 𝑇 . 2: Start Algorithm 2 on all 𝑁 workers. 3: for 𝑘 = 0, 1, . . . , 𝑇 − 1 do 4: Set 𝑔 𝑘 = 0 and 𝑏 = 0. 5: while 𝑏 < 𝐵 do 6: Receive the next gradient message (𝑔, 𝑘′ ) from any worker. 7: if 𝑘′ = 𝑘 then 8: 𝑔 𝑘 ← 𝑔 𝑘 + 𝐵1 𝑔 and 𝑏 ← 𝑏 + 1. 9: end if 10: if 𝑏 < 𝐵 then 11: Send (𝑥𝑘 , 𝑘) to that worker. 12: end if 13: end while (︀ )︀♯ 14: 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 𝑔 𝑘 . 𝑘+1 15: Send (𝑥 , 𝑘 + 1) to the worker whose gradient completed the minibatch. 16: end for

3.6.1

Light-Tail Assumption in the Native Norm Can Lead to Substantial Improvement

Measuring noise directly in the induced dual norm can substantially sharpen oracle bounds. Our native guarantee (14) uses the statistical coefficient C* = 𝜅𝜎 2 . For comparison, consider a Euclidean/Frobenius light-tail oracle with scale 𝜎F and 𝛼 > 0 satisfying 𝛼 ‖𝑧‖ ⎞ ⎛ * ≤ ‖𝑧‖F . Concentrating −2 2 ⎠ ̃︀ ⎝ 𝐿Δ + 𝐿Δ the minibatch error in that norm and then converting it yields 𝑁F = 𝒪 𝜎 . 𝜀 𝜀2 𝛼 ⏟ ⏞ F CF

The two bounds have the same leading term, so we compare their statistical coefficients for two dense-noise models in dual ℓ𝑑𝑝 geometries. Both use ∇𝑓 (𝑥; 𝜉) = ∇𝑓 (𝑥) + 𝜈𝑢, where 𝜈 > 0 and each call draws a fresh independent 𝑢 with independent Rademacher or standard Gaussian entries.13 Table 1 reports the resulting statistical terms. For these models at dual 𝑝 = ∞, corresponding to the ℓ1 update norm, the native coefficient grows as ln(𝑒𝑑) for Rademacher noise and (ln(𝑒𝑑))2 for Gaussian noise, while the converted coefficient grows as 𝑑. At 𝑝 = 2 the bounds agree; at 𝑝 = 1, the native coefficient has the larger asymptotic order, 𝑑3 instead of 𝑑2 . These conclusions depend on the noise model: 𝑝 alone does not determine which analysis gives a smaller bound. Full derivations, intermediate 𝑝 regimes, and the diagonal Schatten counterparts appear in Section J.

4

Rennala NSGD: Semi-Asynchronous Parallelism Beyond Euclidean Norms

We present Rennala NSGD, a semi-asynchronous method combining the minibatch non-Euclidean update (7) with a modified collection protocol of Rennala SGD [Tyurin and Richtárik, 2023].14 A central server coordinates 𝑁 workers with heterogeneous computation speeds, using only gradients computed at the current iterate and discarding stale ones (Algorithm 1 and Algorithm 2). Computation model. We use the fixed computation model of Tyurin and Richtárik [2023]. All 𝑁 workers draw fresh independent samples from the same distribution 𝒟, and worker 𝑖 takes a fixed time 𝜏𝑖 per stochastic gradient, with 0 < 𝜏1 ≤ · · · ≤ 𝜏𝑁 . Communication and server processing are instantaneous. The server processes messages one at a time, including any update and reply, and handles simultaneous returns in a fixed worker order. 13 See Definition 9 and Definition 10. 14 The original Euclidean method is time-optimal up to constants within their zero-respecting algorithm class, under their

specified unbiased bounded-variance oracle model with fixed worker computation times.

9

Fresh gradients. Each gradient carries its query index 𝑘 ′ . At iteration 𝑘, the server accepts only fresh gradients, with 𝑘 ′ = 𝑘, and discards stale ones, with 𝑘 ′ < 𝑘. Conditional on the history, accepted gradients are i.i.d. oracle responses at 𝑥𝑘 , so the oracle model of Section 3.2 and the serial convergence analysis apply. Algorithm: Rennala NSGD. Algorithm 1 and Algorithm 2 give the pseudocode. At iteration 𝑘, the server collects the first 𝐵 fresh gradients, possibly several from the same worker, averages them as in (6), and applies (7). After each message, the server replies with the current iterate and its index. If the message completes the minibatch, it first updates and then sends (𝑥𝑘+1 , 𝑘 + 1) to that worker; otherwise, it sends (𝑥𝑘 , 𝑘).15 Other workers finish their pending computations and receive the latest iterate when their results are processed. 4.1

Main Theorem: Wall-Clock Time Complexity

The following theorem converts the high-probability oracle guarantee of Theorem 1 into a wall-clock guarantee under heterogeneous worker speeds. Theorem 2 (Wall-clock time complexity of Rennala NSGD). Suppose the assumptions of Theorem 1 hold, and use the fixed computation model of Section 4. Run Rennala NSGD with the stepsize, iteration count 𝑇 , and minibatch size 𝐵 prescribed in that theorem. For 0 < 𝜀 ≤ 𝐿∆ and 𝛿 ∈ (0, 1), ⃦ ⃦2 the iterates satisfy min0≤𝑘<𝑇 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 with probability at least 1 − 𝛿 within total wall-clock time ⎛ ⎡(︃ ⎤⎞ )︃−1 (︂ )︂ 𝑆 2 ∑︁ 𝐿∆ 𝜅𝐿∆𝜎 1 1 ̃︀ ⎝ min ⎣ ⎦⎠ . (15) + 𝑇total = 𝒪 𝑆 𝑖=1 𝜏𝑖 𝜀 𝑆𝜀2 𝑆∈[𝑁 ] The minimum in (15) balances the harmonic-mean time of the 𝑆 fastest workers against a statistical term reduced by 𝑆. Neither the worker times nor the minimizing 𝑆 are needed to run the algorithm. The update norm affects 𝐿, 𝜎, 𝜅, and the stationarity criterion, so bounds for different update norms certify different notions of accuracy. For a fixed update norm, accuracy, and worker times, the native-versus-converted comparison depends on 𝜅𝜎 2 and 𝛼−2 𝜎F2 . The smaller coefficient gives a wall-clock bound that is no larger and can be significantly smaller. In the Euclidean case, 𝜅 = 1 recovers the parameter dependence of classical Rennala SGD up to logarithmic factors [Tyurin and Richtárik, 2023]. Our guarantee holds with high probability under light tails rather than in expectation under bounded variance; this agreement does not establish optimality in our oracle model or for general update norms. See Section M.3 for the proof and Section M.4 for the full discussion.

5

Conclusion

We analyzed minibatch SGD in finite-dimensional normed spaces under dual-norm light-tailed noise. Concentration in regular dual spaces yields high-probability oracle-complexity bounds with geometric dependence captured by 𝜅 (Lemma 1 and Corollary 1). We established matching upper and lower regularity bounds for ℓ𝑝 and Schatten-𝑝 spaces (Table 3) and compared native and Euclidean/Frobenius noise analyses. For example, dense Rademacher noise with the ℓ1 update norm gives logarithmic rather than linear dimension dependence in the statistical coefficient (Proposition 5). Building on these results, Rennala NSGD combines non-Euclidean updates with fresh-minibatch collection, yielding a high-probability wall-clock bound under fixed, heterogeneous worker computation times (Theorem 2). In the Euclidean case, our oracle and time bounds match the parameter dependence of standard minibatch SGD and classical Rennala SGD, respectively, up to logarithmic factors. Together, these results connect update geometry and stochastic noise to parallel optimization time. Section B discusses the limitations of our assumptions and computation model.

15 This changes the reply-before-update order in Method 4 of Tyurin and Richtárik [2023]. Details and the proof that the

wall-clock bound is preserved appear in Section M.3.

10

Acknowledgments and Disclosure of Funding The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) CRG Grant ORFSCRG12-2024-6460, and iii) Center of Excellence for Generative AI, under award number 5940. The authors declare no competing interests. References Kwangjun Ahn, Byron Xu, Natalie Abreu, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.05295, 2025. Ahmad Ajalloeian and Sebastian U Stich. On the convergence of SGD with biased gradients. arXiv preprint arXiv:2008.00051, 2020. Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communicationefficient SGD via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017. Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. Advances in Neural Information Processing Systems, 31, 2018. Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The polar express: Optimal matrix sign methods and their application to the Muon algorithm. arXiv preprint arXiv:2505.16932, 2025. Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762, 2025. Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory, pages 111–132. PMLR, 2020. Amir Beck. First-order methods in optimization. SIAM, 2017. Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024. Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International conference on machine learning, pages 560–569. PMLR, 2018. Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. Journal of Machine Learning Research, 24(276):1–50, 2023. Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting distributed synchronous SGD. arXiv preprint arXiv:1604.00981, 2016. Alon Cohen, Amit Daniely, Yoel Drori, Tomer Koren, and Mariano Schain. Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems, 34: 9024–9035, 2021. Andrew Cotter, Ohad Shamir, Nati Srebro, and Karthik Sridharan. Better mini-batch algorithms via accelerated gradient methods. Advances in neural information processing systems, 24, 2011. Yury Demidovich, Grigory Malinovsky, Igor Sokolov, and Peter Richtárik. A guide through the zoo of biased SGD. Advances in Neural Information Processing Systems, 36:23158–23171, 2023. Yury Demidovich, Petr Ostroukhov, Grigory Malinovsky, Samuel Horváth, Martin Takáč, Peter Richtárik, and Eduard Gorbunov. Methods with local steps and random reshuffling for generally smooth non-convex federated optimization, 2024. URL https://arxiv.org/abs/2412. 02781. 11

Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1):37–75, 2014. John Duchi, Michael I Jordan, and Brendan McMahan. Estimation, optimization, and parallelism when data is sparse. Advances in Neural Information Processing Systems, 26, 2013. Lutz Dümbgen, Sara A van de Geer, Mark C Veraar, and Jon A Wellner. Nemirovski’s inequalities revisited. The American Mathematical Monthly, 117(2):138–160, 2010. Sanghamitra Dutta, Gauri Joshi, Soumyadip Ghosh, Parijat Dube, and Priya Nagpurkar. Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD. In International conference on artificial intelligence and statistics, pages 803–812. PMLR, 2018. Ilyas Fatkhullin, Igor Sokolov, Eduard Gorbunov, Zhize Li, and Peter Richtárik. EF21 with bells & whistles: Practical algorithmic extensions of modern error feedback. arXiv preprint arXiv:2110.03294, 2021. Hamid Reza Feyzmahdavian, Arda Aytekin, and Mikael Johansson. An asynchronous mini-batch algorithm for regularized stochastic optimization. IEEE Transactions on Automatic Control, 61 (12):3740–3754, 2016. Alexander Gasnikov. Universal gradient descent. arXiv preprint arXiv:1711.00394, 2017. Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368, 2013. Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. A unified theory of SGD: Variance reduction, sampling, quantization and coordinate descent. In International Conference on Artificial Intelligence and Statistics, pages 680–690. PMLR, 2020. Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General analysis and improved rates. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5200–5209. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/qian19b.html. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating Newton-Schulz iteration for orthogonalization via Chebyshev-type polynomials. arXiv preprint arXiv:2506.10935, 2025. Kaja Gruntkowska, Alexander Tyurin, and Peter Richtárik. EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning, pages 11761–11807. PMLR, 2023. Kaja Gruntkowska, Alexander Gaponov, Zhirayr Tovmasyan, and Peter Richtárik. Error feedback for Muon and friends. arXiv preprint arXiv:2510.00643, 2025. Vincent Guigues, Anatoli Juditsky, and Arkadi Nemirovski. Non-asymptotic confidence bounds for the optimal value of a stochastic program. Optimization Methods and Software, 32(5):1033–1058, 2017. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR, 2018. Samuel Horváth, Chen-Yu Ho, Ludovit Horváth, Atal Narayan Sahu, Marco Canini, and Peter Richtárik. Natural compression for distributed deep learning. In Mathematical and Scientific Machine Learning, pages 129–141. PMLR, 2022. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021. URL https: //arxiv.org/abs/2106.09685. 12

Xinmeng Huang, Yiming Chen, Wotao Yin, and Kun Yuan. Lower bounds and nearly optimal algorithms in distributed learning with communication compression. Advances in Neural Information Processing Systems, 35:18955–18969, 2022. Jikai Jin. A tight lower bound for smooth nonconvex stochastic optimization with bounded gradient noise. arXiv preprint arXiv:2608.09004, 2026. Fritz John. Extremum problems with inequalities as subsidiary conditions, studies and essays presented to r. Courant on his 60th birthday, January 8, 1948, 1948. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: //kellerjordan.github.io/posts/muon/. Pooria Joulani, András György, and Csaba Szepesvári. Think out of the" box": Genericallyconstrained asynchronous composite optimization and hedging. Advances in Neural Information Processing Systems, 32, 2019. Anatoli Juditsky and Arkadii S Nemirovski. Large deviations of vector-valued martingales in 2-smooth normed spaces. arXiv preprint arXiv:0809.0813, 2008. Anatoli Juditsky, Fatma Kılınç Karzan, and Arkadi Nemirovski. Randomized first order algorithms with applications to ℓ1 -minimization. Mathematical Programming, 142(1):269–310, 2013. Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends® in machine learning, 14(1–2):1–210, 2021. Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signSGD and other gradient compression schemes. In International Conference on Machine Learning, pages 3252–3261. PMLR, 2019. Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020. Gyu Yeol Kim and Min-hwan Oh. Convergence of Muon with Newton-Schulz. arXiv preprint arXiv:2601.19156, 2026. Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference on Machine Learning, pages 5381–5393. PMLR, 2020. Anastasiia Koloskova, Sebastian U Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. Advances in Neural Information Processing Systems, 35:17202–17215, 2022. Jakub Konečný, H. B. McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. ArXiv, abs/1610.05492, 2016. Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization. arXiv preprint arXiv:2503.12645, 2025. Dmitry Kovalev and Ekaterina Borodich. Non-Euclidean SGD for structured optimization: Unified analysis and improved rates. arXiv preprint arXiv:2511.11466, 2025. Guanghui Lan. First-order and stochastic optimization methods for machine learning, volume 1. Springer, 2020. Tim Tsz-Kit Lau, Qi Long, and Weijie Su. PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2025. 13

Jiaxiang Li and Mingyi Hong. A note on the convergence of Muon. arXiv preprint arXiv:2502.02900, 2025. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025. Grigory Malinovsky, Kai Yi, and Peter Richtárik. Variance reduced ProxSkip: Algorithm, theory and application to federated learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=edkno3SvKo. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. Pmlr, 2017. Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake E Woodworth. Asynchronous SGD beats minibatch SGD under arbitrary delays. Advances in Neural Information Processing Systems, 35:420–433, 2022. Shuntaro Nagashima and Hideaki Iiduka. Improved convergence rates of Muon optimizer for nonconvex optimization. arXiv preprint arXiv:2601.19400, 2026. Lam Nguyen, Phuong Ha Nguyen, Marten van Dijk, Peter Richtárik, Katya Scheinberg, and Martin Takác. SGD and Hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning, pages 3750–3758. PMLR, 2018. Julie Nutini, Mark Schmidt, Issam Laradji, Michael Friedlander, and Hoyt Koepke. Coordinate descent converges faster with the Gauss–Southwell rule than random selection. In International Conference on Machine Learning, pages 1632–1641. PMLR, 2015. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, 2011. Yehonathan Refael, Guy Smorodinsky, Tom Tirer, and Ofir Lindenbaum. SUMO: Subspaceaware moment-orthogonalization for accelerating memory-efficient LLM training. arXiv preprint arXiv:2505.24749, 2025. Peter Richtárik, Igor Sokolov, and Ilyas Fatkhullin. EF21: A new, simpler, theoretically better, and practically faster error feedback. Advances in Neural Information Processing Systems, 34: 4384–4396, 2021. Peter Richtárik, Igor Sokolov, Elnur Gasanov, Ilyas Fatkhullin, Zhize Li, and Eduard Gorbunov. 3PC: Three point compressors for communication-efficient distributed training and a better theory for lazy aggregation. In International Conference on Machine Learning, pages 18596–18648. PMLR, 2022. Abdurakhmon Sadiev, Grigory Malinovsky, Eduard Gorbunov, Igor Sokolov, Ahmed Khaled, Konstantin Burlachenko, and Peter Richtárik. Federated optimization algorithms with random reshuffling and gradient compression. arXiv preprint arXiv:2206.07021, 2022. Abdurakhmon Sadiev, Grigory Malinovsky, Eduard Gorbunov, Igor Sokolov, Ahmed Khaled, Konstantin Pavlovich Burlachenko, and Peter Richtárik. Don’t compress gradients in random reshuffling: Compress gradient differences. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id= CzPtBzgfae. Abdurakhmon Sadiev, Yury Demidovich, Igor Sokolov, Grigory Malinovsky, Sarit Khirirat, and Peter Richtárik. Improved convergence in parameter-agnostic error feedback through momentum. arXiv preprint arXiv:2511.14501, 2025. 14

Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin, and Peter Richtárik. Ringmaster LMO: Asynchronous linear minimization oracle momentum method. arXiv preprint arXiv:2605.18174, 2026. Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association, 2014. Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025. Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of Muon. arXiv preprint arXiv:2505.23737, 2025. Igor Sokolov. Non-convex stochastic optimization with biased gradient estimators. MS thesis, King Abdullah University of Science and Technology, 2022. URL https://doi.org/10.25781/ KAUST-5EAM7. Igor Sokolov and Peter Richtárik. MARINA-p: Superior performance in non-smooth federated optimization with adaptive stepsizes, 2024. URL https://arxiv.org/abs/2412.17082. Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich, Fawaz S Al-Qahtani, and Peter Richtárik. Bernoulli-LoRA: A theoretical framework for randomized low-rank adaptation. arXiv preprint arXiv:2508.03820, 2025. Igor Sokolov, Laurent Condat, and Peter Richtárik. SILAGE: Memory-efficient, full-gradient-free nonconvex optimization for nested finite sums. arXiv preprint arXiv:2606.15832, 2026. Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for SGD with delayed gradients and compressed communication. arXiv preprint arXiv:1909.05350, 2019. Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. Advances in neural information processing systems, 31, 2018. Fedor Stonyakin, Ilya Kuruzov, and Boris Polyak. Stopping rules for gradient methods for non-convex problems with additive noise in gradient. Journal of Optimization Theory and Applications, 198 (2):531–551, 2023. Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of signSGD under weaker assumptions. In International Conference on Machine Learning, pages 33077–33099. PMLR, 2023. Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang, and Sebastian U Stich. FedMuon: Federated learning with bias-corrected LMO-based optimization. arXiv preprint arXiv:2509.26337, 2025. Paul Tseng and Dimitri P Bertsekas. Relaxation methods for problems with strictly convex costs and linear constraints. Mathematics of operations research, 16(3):462–481, 1991. Paul Tseng and Sangwoon Yun. Block-coordinate gradient descent method for linearly constrained nonsmooth separable optimization. Journal of optimization theory and applications, 140(3): 513–535, 2009. Alexander Tyurin. Toward a unified theory of gradient descent under generalized smoothness. arXiv preprint arXiv:2412.11773, 2024. Alexander Tyurin and Peter Richtárik. Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems, 36:16515–16577, 2023. Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. PowerSGD: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019. 15

Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. ATOMO: Communication-efficient learning via atomic sparsification. Advances in neural information processing systems, 31, 2018. Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communicationefficient distributed optimization. Advances in Neural Information Processing Systems, 31, 2018. Shuo Xie, Tianhao Wang, Beining Wu, and Zhiyuan Li. A tale of two geometries: Adaptive optimizers and non-Euclidean descent. arXiv preprint arXiv:2511.20584, 2025. Kai Yi, Timur Kharisov, Igor Sokolov, and Peter Richtárik. Cohort squeeze: Beyond a single communication round per cohort in cross-device federated learning. arXiv preprint arXiv:2406.01115, 2024. Xiaotong Yuan and Ping Li. On convergence of FedProx: Local dissimilarity invariant bounds, nonsmoothness and beyond. Advances in Neural Information Processing Systems, 35:10752–10765, 2022. Xinwen Zhang and Hongchang Gao. On provable benefits of Muon in federated learning. arXiv preprint arXiv:2510.03866, 2025.

16

Contents 1

Introduction

1

2

Related Work

4

3

SGD Under Non-Euclidean Norms

5

3.1

Non-Euclidean Update Rule and the Sharp Operator . . . . . . . . . . . . . . . . .

5

3.2

Oracle Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

3.3

𝜅-Regularity and Norm-Dependent Constants . . . . . . . . . . . . . . . . . . . .

6

3.4

Regularity vs. Euclidean Norm Equivalence . . . . . . . . . . . . . . . . . . . . .

7

3.5

Key Lemma: Minibatch Concentration in Regular Dual Spaces . . . . . . . . . . .

7

3.6

Main Theorem: High-Probability Oracle Complexity . . . . . . . . . . . . . . . .

8

Light-Tail Assumption in the Native Norm Can Lead to Substantial Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

3.6.1

4

5

Rennala NSGD: Semi-Asynchronous Parallelism Beyond Euclidean Norms

9

4.1

10

Main Theorem: Wall-Clock Time Complexity . . . . . . . . . . . . . . . . . . . .

Conclusion

10

A Notation

19

B Limitations

27

C Extended Related Work

28

C.1 Non-Euclidean Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

C.2 Time Complexity in Heterogeneous Parallel SGD . . . . . . . . . . . . . . . . . .

28

C.3 Federated Learning: Challenges and Algorithmic Remedies . . . . . . . . . . . . .

28

C.4 Computational Cost of the Sharp Operator . . . . . . . . . . . . . . . . . . . . . .

29

D Additional Discussion of Assumption 3

31

D.1 Examples Satisfying the Assumption . . . . . . . . . . . . . . . . . . . . . . . . .

31

D.2 Noise in the Induced Dual Norm . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

D.3 Exponential Moments and High-Probability Guarantees . . . . . . . . . . . . . . .

31

D.4 Comparison with Bounded-Variance Oracles . . . . . . . . . . . . . . . . . . . . .

31

E Useful Auxiliary Results

32

E.1 Foundations of Convex Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

E.2 Properties of the Primal and Dual Norms . . . . . . . . . . . . . . . . . . . . . . .

33

E.3 Properties of Squared Norms and their Conjugates . . . . . . . . . . . . . . . . . .

38

E.4 Inequalities and Specific Geometries . . . . . . . . . . . . . . . . . . . . . . . . .

40

E.5 Probabilistic Preliminaries for Martingale Bounds . . . . . . . . . . . . . . . . . .

41

17

F Sharp Operator for ℓ1 and ℓ∞ Update Norms F.1

ℓ1 Update Norm: Coordinate-Type Steps . . . . . . . . . . . . . . . . . . . . . . .

44

F.2

ℓ∞ Update Norm: Sign-Type Steps . . . . . . . . . . . . . . . . . . . . . . . . . .

44

G Derivation of the Non-Euclidean SGD Update

46

G.1 Derivation from Steepest Descent . . . . . . . . . . . . . . . . . . . . . . . . . .

46

G.2 Linear Minimization Oracle (LMO) Formulation . . . . . . . . . . . . . . . . . .

46

H Derivation of the Orthogonalized Gradient Update

I

44

48

H.1 The Non-Euclidean Update with the Spectral Norm . . . . . . . . . . . . . . . . .

48

H.2 Equivalence of Orthogonalization Definitions . . . . . . . . . . . . . . . . . . . .

49

H.3 LMO and the Canonical Orthogonalized Update for the Spectral Norm . . . . . . .

50

Regularity vs. Norm Equivalence: Why Not Just Use Euclidean Smoothness?

52

I.1 I.2

ℓ𝑑𝑝 Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 𝑚×𝑛

Spaces with Schatten-𝑝 Norms on R

. . . . . . . . . . . . . . . . . . . . . . .

J A Direct Comparison of Euclidean/Frobenius and Geometry-Native Oracles

53 54 57

J.1

Common Assumptions and Complexity Bounds . . . . . . . . . . . . . . . . . . .

58

J.2

Detailed Results and Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

61

J.2.1

ℓ𝑑𝑝 Spaces

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

64

J.2.2

Spaces with Schatten-𝑝 Norms on R𝑚×𝑛 . . . . . . . . . . . . . . . . . .

67

K Fundamental Properties of Smooth and Regular Spaces

70

L Examples of 𝜅-Regular Spaces and Sharp Bounds on 𝜅

71

L.1 Universal Bound via John’s Ellipsoid . . . . . . . . . . . . . . . . . . . . . . . . . ℓ𝑑𝑝 Spaces

71

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75

L.3 Spaces with Schatten-𝑝 Norms on R𝑚×𝑛 . . . . . . . . . . . . . . . . . . . . . . .

87

L.2

M Analysis of SGD Under Dual-Norm Light-Tail Assumption M.1 Proof of the Key Concentration Inequality . . . . . . . . . . . . . . . . . . . . . .

95 95

M.2 Proof of the Convergence Rate Theorem . . . . . . . . . . . . . . . . . . . . . . . 101 M.3 Time-Complexity Analysis of Rennala NSGD Under the Light-Tailed Assumption 105 M.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N Impact Statement

109 111

18

A

Notation

We collect here the notation used throughout the paper; the objects introduced are developed further in Section 3. Spaces and norms. Our analysis takes place in a finite-dimensional normed space (𝐸, ‖·‖) with dual space (𝐸 * , ‖·‖* ). We use the canonical dual pairing ⟨𝑦, 𝑥⟩ := 𝑦(𝑥) for 𝑦 ∈ 𝐸 * and 𝑥 ∈ 𝐸, and define the dual norm by ‖𝑦‖* := sup‖𝑥‖≤1 ⟨𝑦, 𝑥⟩, 𝑦 ∈ 𝐸 * . We refer to ‖·‖ as the update norm: it determines ‖·‖* and, for a given gradient and positive stepsize, the set of admissible steepest-descent steps. The fixed sharp selection described below specifies a concrete step from this set. Gradients and stochastic gradients are elements of 𝐸 * . Our analysis uses the primal norm ‖·‖ for parameter displacements and its induced dual norm ‖·‖* for gradient smoothness, oracle noise, and stationarity. We also use auxiliary norms in the proofs and Euclidean or Frobenius norms when comparing oracle models. We write 𝜅 for a regularity constant of (𝐸 * , ‖·‖* ), and ‖·‖*,+ and 𝜅+ for a corresponding auxiliary norm and its smoothness constant (Definition 1). 2

2

The sharp operator. Let 𝜑(𝑥) := 12 ‖𝑥‖ for 𝑥 ∈ 𝐸, with Fenchel conjugate 𝜑* (𝑠) := 21 ‖𝑠‖* for 𝑠 ∈ 𝐸 * . The subdifferential 𝜕𝜑* (𝑠) ⊆ 𝐸 may contain more than one element. A sharp selection assigns to each 𝑠 ∈ 𝐸 * an element 𝑠♯ ∈ 𝜕𝜑* (𝑠). For the stochastic algorithm, we fix a Borelmeasurable selection 𝑠 ↦→ 𝑠♯ as in Section 3.1 and call this map the sharp operator. Every such ⃦ ⃦2 ⟨︀ ⟩︀ 2 2 selected vector satisfies ⃦𝑠♯ ⃦ = ‖𝑠‖* and 𝑠, 𝑠♯ = ‖𝑠‖* . In the Euclidean case 𝑠♯ = 𝑠. In the 2 quadratic case, ‖𝑥‖ = ⟨𝐴𝑥, 𝑥⟩ for a symmetric positive-definite linear map 𝐴 : 𝐸 → 𝐸 * , so ∇𝜑(𝑥) = 𝐴𝑥, ∇2 𝜑 = 𝐴, and 𝑠♯ = ∇𝜑* (𝑠) = 𝐴−1 𝑠. Here the sharp operator is the fixed linear map 𝐴−1 . For a general norm, the squared-norm functions 𝜑 and 𝜑* need not be differentiable everywhere, and the sharp operator need not admit such a linear representation. The subdifferential formulation remains valid without differentiability assumptions: 𝜕𝜑* is positively homogeneous and may contain multiple vectors at a given point. Basic properties of the sharp operator are collected in Lemma 12, and Section 3.1 derives the resulting update together with its steepest-descent and LMO forms. Vector geometries. For vector-valued parameters we write ℓ𝑑𝑝 for R𝑑 equipped with the ℓ𝑝 norm, (︀∑︀𝑑 )︀ 𝑝 1/𝑝 ‖𝑥‖𝑝 := for 1 ≤ 𝑝 < ∞ and ‖𝑥‖∞ := max1≤𝑖≤𝑑 |𝑥𝑖 |, with ℓ2 the Euclidean norm. 𝑖=1 |𝑥𝑖 | The ℓ𝑝 and ℓ𝑞 norms are dual to one another whenever 1/𝑝 + 1/𝑞 = 1, so an ℓ𝑝 update norm induces an ℓ𝑞 dual norm; in particular ℓ1 and ℓ∞ form a dual pair, which is the source of the coordinate-type and sign-type updates in Section 3.1. Matrix geometries. When the parameter space contains matrix-valued blocks, as in modern neural networks, non-Euclidean update norms are naturally expressed through Schatten norms. For X ∈ R𝑚×𝑛 with 𝑚, 𝑛 ≥ 1, set 𝑟 := min{𝑚, 𝑛} and write 𝜎(X) = (𝜎1 (X), . . . , 𝜎𝑟 (X)) ∈ R𝑟 for the full vector of singular values, ordered as 𝜎1 (X) ≥ · · · ≥ 𝜎𝑟 (X) ≥ 0. Zero singular values are included, so (︀∑︀𝑟 )︀ 𝑝 1/𝑝 𝜎(0) = 0 ∈ R𝑟 . For 1 ≤ 𝑝 < ∞, the Schatten-𝑝 norm is ‖X‖𝑆𝑝 := ‖𝜎(X)‖𝑝 = ; 𝑖=1 𝜎𝑖 (X) for 𝑝 = ∞, we define ‖X‖𝑆∞ := ‖𝜎(X)‖∞ = 𝜎1 (X). Its special cases are the nuclear norm ‖X‖nuc = ‖X‖𝑆1 , the Frobenius norm ‖X‖F = ‖X‖𝑆2 , and the spectral norm ‖X‖op = ‖X‖𝑆∞ . The spectral and nuclear norms are dual to one another. The dimension-dependent bounds use 𝑟 = min{𝑚, 𝑛}, whereas the actual rank rank(X) is the number of positive singular values. In compact-SVD arguments, 𝑟 is explicitly redefined locally as the actual matrix rank. Accuracy and complexity. We write ∆ := 𝑓 (𝑥0 ) − 𝑓 inf for the initial functional gap. A point 2 𝑥 ∈ 𝐸 is 𝜀-stationary if ‖∇𝑓 (𝑥)‖* ≤ 𝜀; stationarity is thus measured in the dual norm induced by the chosen update norm. Our light-tail guarantees are of high-probability type: for a failure probability 𝛿 ∈ (0, 1) they assert that, with probability at least 1 − 𝛿, at least one iterate among 𝑥0 , . . . , 𝑥𝑇 −1 is 𝜀-stationary. By iteration complexity we mean the number 𝑇 of algorithmic iterations required, by oracle complexity the number of stochastic gradient evaluations, and by time complexity the wall-clock time. For serial minibatch SGD, the oracle count is 𝑁* = 𝐵𝑇 , with 𝐵 evaluations per iteration; 𝑁 denotes the number of workers in the parallel setting. In wall-clock bounds, 𝑆 ∈ [𝑁 ] denotes the number of fastest workers considered in the minimization. We write [𝑁 ] := {1, . . . , 𝑁 }, ̃︀ for 𝒪(·) up to factors logarithmic in 𝑇 and 1/𝛿. and 𝒪(·) Throughout the convergence and complexity analysis, we take a positive smoothness bound 𝐿 > 0, an accuracy 𝜀 > 0, and consider the nontrivial case ∆ > 0. If ∆ = 0, then 𝑥0 is a global minimizer and 19

hence ∇𝑓 (𝑥0 ) = 0, so it can be returned without any iterations. The simplified oracle and wall-clock complexity bounds are stated in the accuracy regime 0 < 𝜀 ≤ 𝐿∆. Notation reference. The following table collects the notation used in the main text and the principal constructions in the appendices. The last column points to the definition or the relevant discussion. Symbols reused locally are listed with their scope; temporary variables in individual calculations retain their local definitions. Table 2: Notation reference. Definitions with a stated local scope do not replace the conventions used elsewhere. Symbol Spaces and norms 𝐸, 𝐸 * 𝐸 ** 𝑥, 𝑠 ⟨𝑠, 𝑥⟩ ‖·‖ ‖·‖* ‖·‖2 , ‖·‖F 𝒞, ℬ(𝑦, 𝑡) B, B* ‖ · ‖𝑎 , ‖ · ‖𝑏

Meaning

Reference / scope

Parameter space and its dual; 1 ≤ dim(𝐸) < ∞. Gradients belong to 𝐸 * . Bidual space, canonically identified with 𝐸 in finite dimensions. Generic primal vector 𝑥 ∈ 𝐸 and dual vector 𝑠 ∈ 𝐸 * . Dual pairing: the value of the linear functional 𝑠 at 𝑥. Primal norm, called the update norm. Induced dual norm, ‖𝑠‖* = sup‖𝑥‖≤1 ⟨𝑠, 𝑥⟩. Euclidean and Frobenius reference norms; in the oracle comparison, the subscript F also denotes the Euclidean norm for vectors. A constraint set and the closed update-norm ball {𝑥 ∈ 𝐸 : ‖𝑥 − 𝑦‖ ≤ 𝑡}. Primal and dual unit balls, respectively. The notation 𝐵 also denotes a convex body locally in the ellipsoid proof. Two auxiliary norms with neutral labels 𝑎, 𝑏; ‖ · ‖𝑎,* and ‖ · ‖𝑏,* are their dual norms.

Section A Theorem 5 Section A Section A Section 3.1 Lemma 3 Section J

Section G.2 Lemma 24; Lemma 8 Lemma 4

Standard notation from convex analysis dom 𝑓 , dom 𝜕𝑓 Effective domain {𝑥 : 𝑓 (𝑥) < +∞} and the set of (16) points where the subdifferential is nonempty. In convex-analysis statements, 𝑓 may be extended-valued. int(𝐶), rint(𝐶) Interior in the ambient space and relative interior in the Existence of affine hull of 𝐶. subgradients in the auxiliary results 𝜕𝑓 (𝑥) Convex subdifferential: the set of 𝑠 satisfying (16) 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨𝑠, 𝑦 − 𝑥⟩ for every 𝑦, at a finite-valued point 𝑥. 𝑓* Fenchel conjugate, 𝑓 * (𝑠) = sup𝑥 {⟨𝑠, 𝑥⟩ − 𝑓 (𝑥)}. The Theorem 6 star on a function denotes conjugacy, whereas the star on a norm denotes the dual norm. 𝜄𝐶 Extended-valued indicator: zero on 𝐶 and +∞ outside Lemma 8 𝐶. 𝜕 ‖𝑥‖, 𝜕 ‖𝑠‖* Subdifferentials of the norm functions evaluated at 𝑥 Lemma 7 and 𝑠, respectively; the short notation does not differentiate a scalar value. 2 𝜑, 𝜑* Squared-norm functions 𝜑(𝑥) = 12 ‖𝑥‖ and Lemma 9 2 1 * 𝜑 (𝑠) = 2 ‖𝑠‖* , which are Fenchel conjugates. arg min, arg max Sets of minimizers and maximizers; they need not be Section G.2 singletons. Continued on the next page

20

Notation reference (continued) Symbol

Meaning

Reference / scope

conv(𝐶), 𝜕𝐶

Convex hull and boundary of a set 𝐶. The boundary notation is distinct from the subdifferential of a function.

Theorem 7

Objective, oracle, and accuracy 𝑓 , 𝑓𝑖 Global objective and worker objectives; in the ∑︀𝑁 finite-sum formulation, 𝑓 = 𝑁 −1 𝑖=1 𝑓𝑖 . Under statistical homogeneity all 𝑓𝑖 coincide. 𝑓 (𝑥; 𝜉), 𝑓𝑖 (𝑥; 𝜉𝑖 ) Sample losses in the expectation formulation, which is a special case of the general oracle model. S𝑖 , 𝒟𝑖 , 𝒟 Worker sample space, worker sampling distribution, and common distribution used in the homogeneous setting. ∇𝑓 (𝑥) Exact derivative of 𝑓 at 𝑥, viewed as an element of 𝐸 * . ∇𝑓 (𝑥; 𝜉) Response of the jointly measurable stochastic oracle. It is an unbiased estimate of ∇𝑓 (𝑥), but need not be the gradient of a sample loss. 𝑘 𝜉, 𝜉𝑗 Fresh oracle sample; the indexed version is sample 𝑗 of iteration 𝑘. Each new minibatch is independent of the preceding history. 𝐿 Positive smoothness bound satisfying ‖∇𝑓 (𝑥) − ∇𝑓 (𝑦)‖* ≤ 𝐿 ‖𝑥 − 𝑦‖. 𝑓 inf A finite lower bound on 𝑓 ; it need not be attained. ∆ Initial functional gap, 𝑓 (𝑥0 ) − 𝑓 inf . The nontrivial complexity regime assumes ∆ > 0. 𝜎, 𝜎 2 Positive dual-norm light-tail scale and [︁ ]︁ its square: 2 2 E exp(‖∇𝑓 (𝑥; 𝜉) − ∇𝑓 (𝑥)‖* /𝜎 ) ≤ 𝑒. It also bounds the noise second moment. 𝐺 Uniform bound on the dual norm of the oracle error in the bounded-error example; distinct from a matrix gradient G. 𝜀 Squared-gradient stationarity tolerance: 2 ‖∇𝑓 (𝑥)‖* ≤ 𝜀. Simplified complexity bounds use 0 < 𝜀 ≤ 𝐿∆. 𝛿 Failure probability, 0 < 𝛿 < 1; the main guarantees hold with probability at least 1 − 𝛿. 𝑥 ˆ An iterate selected uniformly from the prescribed finite sequence, used for the companion expectation criterion. 𝐿F , 𝜀F Smoothness constant and squared-gradient accuracy measured in the Euclidean/Frobenius geometry in the literature comparison. 𝜎F2 Reference-norm noise scale: a variance bound in the bounded-variance literature discussion, and an exponential-moment scale in the light-tail oracle comparison. Updates and iteration counts 𝑥0 , 𝑥 𝑘 Initial point and iterate at index 𝑘; X𝑘 denotes the corresponding matrix iterate. ∑︀𝐵 𝑔 𝑘 , G𝑘 Minibatch gradient 𝑔 𝑘 = 𝐵 −1 𝑗=1 ∇𝑓 (𝑥𝑘 ; 𝜉𝑗𝑘 ) and its matrix counterpart.

(1)

(2) and (3) Problem setup in Section 1 Section 3.2 Section 3.2

Section 3.1

Assumption 1 Assumption 2 Section A Assumption 3

Section 3.2

Section A

Theorem 1 Section D Section 2

Section 2; Assumption 4

(7) (6)

Continued on the next page

21

Notation reference (continued) Symbol ♯

𝑠

𝛾𝑘 ℎ𝑘 , ℎ𝑘 𝑡𝑘 lmo𝒞 (𝑠) 𝐵 𝑇 𝑁* F 𝒯SGD (𝐵)

𝒪, Ω, Θ, 𝑜

̃︀ 𝒪 𝐼(𝑠), 𝑒𝑖 sign 𝐴

Meaning

Reference / scope *

Fixed Borel-measurable selection (𝑠). It ⃦ ⃦ ⟨︀ ⟩︀ from 𝜕𝜑 2 satisfies ⃦𝑠♯ ⃦ = ‖𝑠‖* and 𝑠, 𝑠♯ = ‖𝑠‖* . Positive stepsize; the main convergence and time bounds use 𝛾𝑘 ≡ 1/(2𝐿). Step displacement, selected as −𝛾𝑘 (𝑔 𝑘 )♯ to agree with the fixed sharp rule. Both index conventions occur in the derivation. ⃦ ⃦ Radius 𝛾𝑘 ⃦𝑔 𝑘 ⃦* of the equivalent LMO ball; not a wall-clock time. Set-valued linear minimization oracle, arg min𝑥∈𝒞 ⟨𝑠, 𝑥⟩. Minibatch size, a positive integer; independent of the number of workers. Prescribed iteration count; in the main bounds, 𝑇 = ⌈128𝐿∆/(15𝜀)⌉. Serial minibatch oracle count 𝐵𝑇 ; written 𝑇 𝐵* when explicitly comparing native and converted batch choices. Euclidean/Frobenius iteration-complexity bound for minibatch size 𝐵; the corresponding oracle bound is F 𝐵𝒯SGD (𝐵). Asymptotic upper, lower, matching-order, and strictly smaller-order notation, respectively. Upper-bound notation suppressing factors logarithmic in 𝑇 and 1/𝛿. Maximizing-coordinate set 𝐼(𝑠) = {𝑖 : |𝑠𝑖 | = ‖𝑠‖∞ } and coordinate unit vector. Componentwise sign function, with sign(0) = 0 for the canonical sign selection. In the quadratic-norm example, a symmetric 2 positive-definite map 𝐸 → 𝐸 * with ‖𝑥‖ = ⟨𝐴𝑥, 𝑥⟩ ♯ −1 and 𝑠 = 𝐴 𝑠.

Regularity and norm comparisons 𝑉 , ‖ · ‖𝑉 Generic finite-dimensional normed space in the regularity results; usually specialized to 𝑉 = 𝐸 * . Distinct from the potential 𝑉 (𝑠) below. 𝜅𝑉 , 𝜅𝑉,+ A regularity constant of 𝑉 and a smoothness constant of its auxiliary norm, with 1 ≤ 𝜅𝑉,+ ≤ 𝜅𝑉 . The same constant notation is used when discussing smoothness alone. ‖ · ‖𝑉,+ Auxiliary norm with ‖𝑢‖2𝑉 ≤ ‖𝑢‖2𝑉,+ ≤ (𝜅𝑉 /𝜅𝑉,+ )‖𝑢‖2𝑉 . ‖ · ‖𝑉,+,* Norm dual to ‖ · ‖𝑉,+ , used in smoothness and conjugacy characterizations. 𝜅, 𝜅+ Abbreviations for 𝜅𝐸 * and 𝜅𝐸 * ,+ in the stochastic analysis. They are admissible constants, not necessarily the smallest possible ones. ‖·‖*,+ Auxiliary norm on 𝐸 * , distinct from the update norm. The concentration proof chooses a 𝜅-smooth proxy with squared-norm distortion at most 2.

Lemma 12 Theorem 1 Section G.1

(8) Section G.2 Theorem 1 Theorem 1 Corollary 1

Section 2

Complexity and dimension comparisons Section A Section F.1 Section F.2 Section A

Definition 1

Definition 1

(12) Proposition 9 Remark 1

(13); (110)

Continued on the next page 22

Notation reference (continued) Symbol

Meaning

Reference / scope

‖ · ‖+ , 𝜑+

Local shorthand for an auxiliary norm and its half-squared function, 𝜑+ (𝑢) = 12 ‖𝑢‖2+ . In the regularity comparison, squared-norm distortion with 𝜅 = 𝜅+ 𝐶. In the two-norm equivalence lemma it instead bounds the norms themselves. Positive equivalence constants satisfying 𝛼 ‖𝑠‖* ≤ ‖𝑠‖F ≤ 𝛽 ‖𝑠‖* . Noise conversion uses 𝛼−2 ; the Euclidean proxy gives 𝜅 = (𝛽/𝛼)2 . Finite auxiliary-norm exponent in the regularity constructions; for 𝑝 ≥ 2, the optimization uses 2 ≤ 𝜌 ≤ 𝑝 and 𝜌 < ∞, also when 𝑝 = ∞. Tuned auxiliary exponent and the threshold used to express it. The vector and Schatten choices differ; their closed forms and dimension ranges are stated separately. Constructed regularity parameter for ℓ𝑑𝑝 , equal to 𝑑2/𝑝−1 for 𝑝 < 2 and obtained by minimizing (𝜌 − 1)𝑑2/𝜌−2/𝑝 for 𝑝 ≥ 2. Constructed Schatten regularity parameter using 𝑟 = min{𝑚, 𝑛}; the uniform smoothness bound used in the construction is max{2, 𝜌 − 1}. Explicit admissible parameters chosen for the oracle comparisons. They need not coincide when 𝑑 = 𝑟; the latter uses the exact value 1 at 𝑝 = 2. Conversion constant sup𝑧̸=0 ‖𝑧‖* /‖𝑧‖2 in the discussion of Ringmaster LMO; unrelated to the auxiliary exponent.

Lemma 15

𝐶

𝛼, 𝛽

𝜌

𝜌* , 𝜌ˆ

𝜅𝑝 (𝑑)

𝜅𝑝 (𝑚, 𝑛) 𝜅ℓ𝑝 (𝑑), 𝜅𝑆𝑝 (𝑟) 𝜌 (related work)

Section I; Lemma 4

(56)

Proposition 3; Proposition 4 Proposition 3; Proposition 4

Proposition 12

Proposition 14

(70) and (79)

Section C.2

Vector and matrix geometries 𝑑, 𝑚, 𝑛 Vector dimension and numbers of matrix rows and Section A; columns. In general-space statements, 𝑑 may denote Proposition 10 dim(𝑉 ). 𝑝, 𝑞 Norm exponents; when paired as dual exponents, Section A 1/𝑝 + 1/𝑞 = 1, with 1/∞ = 0. In oracle comparisons, 𝑝 indexes the dual norm. ∑︀ ℓ𝑑𝑝 , ‖ · ‖𝑝 R𝑑 equipped with the norm ( 𝑖 |𝑥𝑖 |𝑝 )1/𝑝 for finite 𝑝, Section A and ‖𝑥‖∞ = max𝑖 |𝑥𝑖 |. 𝑟 (dimension) Matrix dimension parameter min{𝑚, 𝑛} in Schatten Section A norm comparisons and regularity bounds; not the actual rank. 𝑟 (compact SVD) Locally, the actual rank rank(G), counting only Section H positive singular values. 𝜎𝑖 (X), 𝜎(X) Singular values in nonincreasing order and their full Section A vector, including zeros. This use of 𝜎 is distinct from the noise scale. ‖ · ‖𝑆𝑝 Schatten-𝑝 norm, ‖X‖𝑆𝑝 = ‖𝜎(X)‖𝑝 . Section A ‖·‖op , ‖·‖nuc Spectral norm 𝑆∞ and nuclear norm 𝑆1 , a dual pair. Section A The Frobenius norm is 𝑆2 . ⟨G, X⟩𝐹 Frobenius pairing tr(G⊤ X), identifying the matrix Lemma 18 space with its dual. Continued on the next page

23

Notation reference (continued) Symbol

Meaning

Reference / scope

U𝑟 , Σ𝑟 , V𝑟

Compact-SVD factors, G = U𝑟 Σ𝑟 V𝑟⊤ , retaining only positive singular values; the columns of U𝑟 and V𝑟 are

Section H

W 𝑀 † , 𝑀 †/2 orth(G), O𝑘 (G𝑘 )♯

𝜂𝑘 𝐼𝑑 , 𝐼𝑟 tr, det, rank 𝐸𝑖𝑖

orthonormal. Residual matrix satisfying U⊤ 𝑟 W = 0, WV𝑟 = 0, and ‖W‖op ≤ 1 in the nuclear-subgradient and spectral-LMO formulas. Moore–Penrose pseudoinverse and, for positive semidefinite 𝑀 , the positive semidefinite square root of 𝑀 †. Canonical polar map orth(G) = U𝑟 V𝑟⊤ , with orth(0) = 0; O𝑘 = orth(G𝑘 ). Canonical spectral-norm sharp selection ⃦ 𝑘⃦ ⃦G ⃦ orth(G𝑘 ); distinct from the polar factor nuc alone. ⃦ ⃦ State-dependent stepsize 𝛾𝑘 ⃦G𝑘 ⃦nuc in the orthogonalized update; distinct from a noise vector 𝜂 𝑗 . Identity matrices in the specified dimensions, distinct from the maximizing-coordinate set 𝐼(𝑠) and random coordinate index 𝐼. Matrix trace, determinant, and rank.

Lemma 18; Lemma 20 Section H.2

(10) (9)

Section H Theorem 7; Lemma 18

Theorem 7; Section H Rectangular coordinate matrix with entry 1 at (𝑖, 𝑖) and Proposition 15 zeros elsewhere.

Noise models and statistical coefficients 𝜁, 𝑍 Additive, zero-mean oracle perturbation; its distribution and norm scale depend on the stated model. 𝜈 Positive amplitude of the Rademacher or Gaussian perturbations. 𝜒𝑖 , 𝜒 Independent Rademacher signs and their vector; in the one-sparse model, 𝜒 denotes a single sign. 𝜔𝑖 , 𝜔 Independent standard Gaussian coordinates and their vector in the Gaussian oracle model. In regularity lower-bound proofs, similar symbols locally denote signs instead. Diag𝑚,𝑛 (𝑢) Rectangular diagonal embedding: entry 𝑢𝑖 at (𝑖, 𝑖) for 𝑖 ≤ min{𝑚, 𝑛} and zero elsewhere. 𝐷 Number of independent noise coordinates: 𝐷 = 𝑑 for dense vector noise and 𝐷 = 𝑟 for diagonal matrix noise. 𝜗𝑝 (𝐷) Gaussian shape factor, equal to 1 for 1 ≤ 𝑝 ≤ 2 and min{𝑝, ln(𝑒𝐷)} for 𝑝 > 2. 𝑚𝑝 , 𝑋𝑝 , 𝐻𝑝 (𝐷) In the Gaussian proof for 𝑝 > 2, 𝑚𝑝 = E [‖𝜔‖𝑝 ], 𝑋𝑝 = ‖𝜔‖𝑝 − 𝑚𝑝 , and 𝐻𝑝 (𝐷) = 𝜗𝑝 (𝐷)𝐷2/𝑝 . 𝛼𝑝 (𝑑), 𝛼𝑆𝑝 (𝑟) Sharp one-sided conversion constants for Euclidean-to-ℓ𝑝 and Frobenius-to-Schatten-𝑝 norms. 𝐵 * , 𝐵F Batch sizes prescribed by the native and converted analyses, respectively, for the same update rule and iteration count. 𝑁F Converted-analysis oracle count 𝑇 𝐵F . C* , CF Statistical coefficients 𝜅𝜎 2 and 𝛼−2 𝜎F2 . Superscripts Rad and Gau distinguish the two perturbation models.

Section 3.2; Section J Definition 9; Definition 10 Definition 9; Remark 3 Definition 10; Proposition 13

(57) Section J

(58) Lemma 22, proof (69) and (78) (60) and (59)

(62) (61)

Continued on the next page 24

Notation reference (continued) Symbol

Meaning

Reference / scope

𝐼 (one-sparse noise)

Random coordinate index in 𝜁 = 𝜈𝜒𝑒𝐼 , independent of Remark 3 the Rademacher sign. Distinct from the set 𝐼(𝑠).

Ellipsoid construction for the universal regularity bound 𝐵, ℬ2 Centrally symmetric convex body and Euclidean unit ball. Here 𝐵 is a set, not a minibatch size. ℰin Maximal-volume ellipsoid contained in 𝐵, with √ 𝐵 ⊆ 𝑑 ℰin . 𝑑 𝑑 𝑆+ , 𝑆++ Cones of real symmetric positive semidefinite and positive definite matrices, respectively. 𝑐, 𝑄, 𝑄* Center and positive-definite shape matrix in 𝑐 + 𝑄ℬ2 , and the maximizing centered shape matrix 𝑄* . 𝑑 𝒬 Feasible shape matrices {𝑄 ∈ 𝑆+ : 𝑄ℬ2 ⊆ 𝐵}. 𝑇 (ellipsoid proof) Normalizing linear map (𝑄* )−1 , distinct from the iteration count. ∘ ̃︀ ̃︀ 𝐵, 𝐵 Normalized body 𝑇 (𝐵) and its polar, ̃︀ {𝑦 : ⟨𝑦, 𝑥⟩ ≤ 1 for all 𝑥 ∈ 𝐵}. 𝑔(𝑄), 𝜇, ℒ Convex constraint function, nonnegative multiplier, and Lagrangian in the finite-dimensional shape optimization; unrelated to stochastic gradient responses. 𝑦𝑖 , 𝜆𝑖 Unit contact points and ∑︀ ∑︀positive weights with 𝐼𝑑 = 𝑖 𝜆𝑖 𝑦𝑖 𝑦𝑖⊤ and 𝑖 𝜆𝑖 = 𝑑. Probability and concentration proofs Ω, ℱ, Prob Sample space, sigma-algebra of events, and probability measure. The symbol Ω also denotes asymptotic lower bounds elsewhere. E [·], E𝜉 [·] Expectation, with a subscript specifying the sampled variable when useful; conditioning on a history is written with a vertical bar. 𝜎(𝜉) Sigma-algebra generated by a random variable; unrelated to the light-tail scale or singular values. ℱ 𝑡 , ℱ𝑘 Filtration or history available before the next sample or iteration; the Rennala proof includes pending worker states in this history. 𝜂 𝑗 , 𝜂𝑗𝑘 Centered noise vectors; at an iterate, 𝜂𝑗𝑘 = ∇𝑓 (𝑥𝑘 ; 𝜉𝑗𝑘 ) − ∇𝑓 (𝑥𝑘 ). ∑︀𝐵 ∑︀𝑗 𝜂¯, 𝑆𝑗 Average 𝐵 −1 𝑗=1 𝜂 𝑗 and partial sum 𝑡=1 𝜂 𝑡 . In ∑︀𝑡 martingale preliminaries, 𝑆𝑡 = 𝑖=1 𝜉𝑖 . 𝜉𝑡 (martingales) A vector-valued martingale difference in the auxiliary results; distinct from the oracle’s sample variable. 𝜆 Nonnegative tail parameter in the concentration bound 2 /3 𝑒−𝜆 √︀ ; for the finite-horizon union bound, 𝜆 = 3 log(𝑇 /𝛿). 2 𝜎+ Auxiliary-norm light-tail scale 2𝜎 2 in the concentration proof. 𝜑̂︀ Half-squared primal norm restricted to the unit ball, and +∞ outside it.

Theorem 7 Theorem 7 Theorem 7, proof Theorem 7, proof Theorem 7, proof Theorem 7, proof Theorem 7, proof Theorem 7, proof

(88)

Definition 4

Assumption 3

Definition 5 and the following remark Definition 7; Section M.3 Lemma 1; Section M.2 Lemma 21; Section M.1 Definition 8 Lemma 1

(111) Lemma 24

Continued on the next page

25

Notation reference (continued) Symbol

Meaning

Reference / scope

𝑉 (𝑠)

̂︀ * (𝑠): 1 ‖𝑠‖2 when ‖𝑠‖ ≤ 1, and Potential (𝜑) * * 2 ‖𝑠‖* − 12 otherwise. The concentration proof uses the

Lemma 24; Section M.1

same construction with ‖·‖*,+ . 𝛽, 𝑉𝛽 (𝑠) Positive potential-scaling parameter and 𝑉𝛽 (𝑠) = 𝛽𝑉 (𝑠/𝛽). This 𝛽 is unrelated to the norm-equivalence constant. 𝜓𝑗 , Ψ𝐵 Scalar increments 𝑉𝛽 (𝑆𝑗 ) − 𝑉𝛽 (𝑆𝑗−1 ) and their sum, which telescopes to 𝑉𝛽 (𝑆𝐵 ). 𝜇𝑗 , 𝜈𝑗 Conditional mean upper bounds and scalar exponential-moment scales. The application sets 2 𝜇𝑗 = 𝜅𝜎+ /(2𝛽) and 𝜈𝑗 = 𝜎+ . 𝜓(𝑥) Locally in the vector smoothness proof, ‖𝑥‖2𝜌 = 2𝜑+ (𝑥); distinct from the scalar random increments 𝜓𝑗 . 𝜀𝑖 , 𝑢𝑖 , 𝑈𝑖 Rademacher signs, deterministic vector directions, and matrix directions in the regularity lower-bound constructions. Indexed 𝜀𝑖 are not stationarity tolerances. 𝑛, 𝑁 (lower bounds) Local lengths of sign-sequence constructions in the vector and matrix proofs, respectively; distinct from matrix dimensions and the worker count. Parallel computation and collection-time proof 𝑁 , [𝑁 ] Number of workers and the index set {1, . . . , 𝑁 }. 𝜏𝑖 , 𝜏max Fixed time for worker 𝑖 to compute one stochastic gradient, ordered as 0 < 𝜏1 ≤ · · · ≤ 𝜏𝑁 ; 𝜏max = max𝑖 𝜏𝑖 . 𝑆 Number of fastest workers considered in the time-bound minimization, 𝑆 ∈ [𝑁 ]; not a batch size. 𝑇total Wall-clock time to complete the prescribed 𝑇 iterations under the fixed computation model. 𝑔, 𝑘 ′ Gradient response and the index of its query point in a worker message. The server accepts it only when 𝑘 ′ = 𝑘. 𝑏 Number of gradients accepted so far in the current minibatch. 𝛿𝑘 Gradient delay in the related-work description of classical asynchronous updates; distinct from failure probability 𝛿. 𝐵𝑖 Sufficient target quota for worker 𝑖 in the collection-time proof, not necessarily its actual accepted count. Quotas are relaxed to reals and then rounded up. 𝑡𝑖 Sufficient time 𝜏𝑖 (1 + 𝐵𝑖 ) when 𝐵𝑖 > 0, and zero when 𝐵𝑖 = 0. 𝑡′ , 𝑡′ (𝑆) Relaxed batch-collection ∑︀𝑆 time and the candidate 𝑡′ (𝑆) = (𝐵 + 𝑆)/( 𝑖=1 1/𝜏𝑖 ). 𝑆⋆ Smallest minimizer of 𝑡′ (𝑆) over 𝑆 ∈ [𝑁 ], used only in the runtime proof. ∑︀𝑆 𝐴𝑆 , 𝑟¯𝑆 Aggregate rate 𝐴𝑆 = 𝑖=1 1/𝜏𝑖 and average rate 𝑟¯𝑆 = 𝐴𝑆 /𝑆 of the fastest 𝑆 workers. Its reciprocal is their harmonic-mean computation time.

26

Lemma 25

Section M.1 Lemma 26

Proposition 12, proof Proposition 13; Proposition 15

Proposition 13; Proposition 15

Section 4 Section 2; Section 4

Theorem 2 (123) Algorithm 1; Algorithm 2 Algorithm 1 Section 2

Section M.3

Section M.3 Section M.3 Section M.3 Section M.3

B

Limitations

Noise assumptions. Our high-probability guarantees rely on a dual-norm light-tail condition, which is stronger than bounded variance. Under bounded variance, the counterpart discussed in Section D gives an expectation guarantee, while the high-probability analysis requires stronger control of the noise. Dependence on geometry and noise. The benefits of the native-norm formulation also depend on the problem: the regularity parameter 𝜅 can be as large as dim(𝐸), and whether our bounds improve on Euclidean/Frobenius conversions depends jointly on the geometry and the noise model. Statistical homogeneity. In the parallel setting, statistical homogeneity allows gradients from different workers to form an unbiased minibatch at the current iterate. When local distributions differ, accepting the first available fresh gradients can instead favor faster workers and bias the estimate of the global gradient. Extending our guarantees to this setting therefore requires revisiting minibatch collection and its time analysis. Computation and communication model. The current analysis also assumes fixed worker computation times and instantaneous communication and server processing, leaving arbitrary delays and communication costs outside its scope. Cost of the sharp operator. This paper does not account for the cost of computing the update direction. The computation model charges the workers only for stochastic gradients and treats the work of the server, including the evaluation of the sharp operator, as free. For a non-Euclidean update norm this cost need not be negligible: for the spectral norm, the sharp operator requires an orthogonalization of G𝑘 (Section 3.1). Accordingly, (15) should be read as the time spent on computing stochastic gradients. Our guarantees assume an exact sharp selection, so numerical approximations require additional error control. Reducing the cost of the sharp operator is an important problem in its own right, which lies outside the scope of this work; existing approaches are reviewed in Section C.4. Experimental validation. The present contribution is theoretical; assessing its practical benefits and computational tradeoffs through experiments remains future work.

27

C

Extended Related Work

C.1

Non-Euclidean Optimization

Kovalev and Borodich [2025] develop a unified analysis of non-Euclidean SGD with momentum and weight decay under structured smoothness and gradient-noise assumptions, including variants with extrapolation and momentum variance reduction. Their Assumption 2 bounds the noise second moment in a weighted Euclidean norm, whereas our light-tail assumption is formulated directly in the dual norm induced by the update geometry. In the convex setting, they compare the leading complexity terms for signSGD and Muon with those of normalized SGD. Their comparisons identify favorable regimes involving dense solutions together with strongly nonuniform coordinatewise smoothness and noise scales, or solutions of high effective rank together with approximately low-rank matrices controlling curvature and gradient noise. Although they provide explicit comparisons in terms of structural parameters, they do not construct specific objective–oracle pairs realizing the favorable regimes. Moreover, the conditions yielding an improvement may be restrictive when the smoothness and noise parameters have many components or singular values of comparable magnitude. Concurrent work by Xie et al. [2025] studies non-Euclidean noise through an adaptive variance model, whereas we use a light-tail condition in the dual norm induced by the update geometry. They prove that Normalized Steepest Descent (NSD) with momentum admits a dimension-free convergence rate for nonconvex optimization, in the sense that it avoids the norm-distortion factor that arises when controlling noise in a Euclidean reference norm and then converting to the target dual norm. C.2

Time Complexity in Heterogeneous Parallel SGD

In the asynchronous update described in Section 2, the gradient applied at iteration 𝑘 is evaluated at the earlier iterate 𝑥𝑘−𝛿𝑘 . It therefore need not be an unbiased estimator of ∇𝑓 (𝑥𝑘 ); the analysis must account for this bias, as in frameworks for biased gradient estimators [Demidovich et al., 2023]. The harmonic-mean wall-clock bounds cited in Section 2 rely on suitable stepsize or filtering rules. The results cover ASGD with delay-adaptive stepsizes and filtering variants such as Picky SGD, which skips updates when the query point is too far from the current iterate [Mishchenko et al., 2022, Koloskova et al., 2022, Cohen et al., 2021]. Sadiev et al. [2026] propose Ringmaster LMO, an asynchronous LMO-based momentum method for unconstrained nonconvex stochastic optimization, and translate its iteration complexity into wall-clock time under heterogeneous worker speeds, recovering the optimal time-complexity scaling of Tyurin and Richtárik [2023] in the standard smooth Euclidean case. This is the method closest to ours, and the two are best read as addressing different questions. The key difference lies in the noise model, which is the subject of the present paper: they assume bounded variance in the Euclidean norm and transfer it to the update geometry through the norm-equivalence constant 𝜌 := sup𝑧̸=0 ‖𝑧‖*/‖𝑧‖2 , so that the product 𝜌𝜎 enters their complexity bounds—precisely the conversion that Assumption 3 is designed to avoid by measuring noise directly in the induced dual norm ‖·‖* . C.3

Federated Learning: Challenges and Algorithmic Remedies

Problem (1) is the standard formulation of federated learning (FL), in which a server coordinates a large number of clients that keep their data local [Konečný et al., 2016, McMahan et al., 2017, Kairouz et al., 2021]. This setting presents several distinct challenges. Statistical heterogeneity means that the local distributions 𝒟𝑖 differ across clients, while device or system heterogeneity refers to differences in compute speed, availability, and reliability. With partial participation, only a sampled cohort is active in each round. Privacy motivates keeping data local, while the communication bottleneck arises because transmitting model updates is often much more expensive than computing them [Sokolov, 2022]. The same nested averaging structure also appears in centralized settings: the objective averages over groups, and each group averages over its own samples. Examples include pooled data silos, out-of-core learning, and deliberate stratification. There, the aim is to exploit this structure to reduce memory use and oracle calls, with complexity depending on similarity both across groups and within each group [Sokolov et al., 2026]. 28

The challenge closest to the agenda of the present paper is how computation is organized in a serverbased data-parallel setting, where workers repeatedly exchange gradients or model updates with a coordinating server. Two widely studied algorithmic levers aim to reduce the communication cost of these exchanges. Communication compression. The first lever is to reduce the size of each transmitted message by quantizing, sparsifying, or otherwise compressing the update while controlling the resulting error. Unbiased compressors [Alistarh et al., 2017, Wangni et al., 2018, Horváth et al., 2022] simplify the analysis: conditional on the vectors being compressed, independent compression randomness across workers ensures that the Euclidean second moment of the average compression error equals 1/𝑁 times the average of the individual error second moments. Biased compression requires additional care. Naive compression of local gradients can lead to exponential divergence in heterogeneous distributed settings, as shown by the Top-1 counterexample of Beznosikov et al. [2023, Example 1]. Error feedback is a standard way to obtain convergence guarantees with biased compressors [Seide et al., 2014, Stich et al., 2018, Alistarh et al., 2018, Karimireddy et al., 2019, Stich and Karimireddy, 2019]. For smooth nonconvex objectives, EF21 matches the 𝒪(1/𝑇 ) iteration dependence of gradient descent, with compression-dependent constants and without bounded-gradient assumptions [Richtárik et al., 2021, Theorem 2]. Subsequent developments include practical variants [Fatkhullin et al., 2021] and normalized methods that combine momentum with time-varying stepsizes requiring no tuning from problem constants [Sadiev et al., 2025]. The framework of three-point compressors connects error feedback with lazy aggregation [Richtárik et al., 2022]. Other work studies which messages to compress and how to exploit their structure. This includes server-to-worker compression [Gruntkowska et al., 2023, Sokolov and Richtárik, 2024] and lowrank or structured compression of transmitted gradients [Vogels et al., 2019, Wang et al., 2018]. Compression can also be combined with parameter-efficient fine-tuning, where the update itself is restricted to low rank [Hu et al., 2021], with quantization or error feedback applied on top [Sokolov et al., 2025]. Compression also interacts with without-replacement sampling. Its variance can mask the benefits of random reshuffling in Q-RR, whereas DIANA-RR improves the accuracy dependence in the strongly convex setting by compressing differences between gradients and learned shifts [Sadiev et al., 2022, 2024]. Lower bounds characterize the limits of communication compression [Huang et al., 2022]. Local steps. The second lever is to communicate less often, letting each client take several local update steps between synchronizations, as in FedAvg [McMahan et al., 2017]. The obstacle is client drift: local progress is made towards local optima, and the effects of the resulting bias must be controlled [Karimireddy et al., 2020, Yuan and Li, 2022, Koloskova et al., 2020]. Subsequent work combines local steps with probabilistic synchronization [Malinovsky et al., 2022], with random reshuffling [Demidovich et al., 2024], and with compression itself [Sadiev et al., 2022]. A complementary question is how much can be extracted from a single cohort: instead of the customary single communication round per sampled cohort, one may run several proximal-point rounds inside a cohort and thereby cut the total communication cost substantially [Yi et al., 2024]. Our work addresses a complementary question: how computational heterogeneity interacts with the geometry of the update. We study the wall-clock time of a fresh-minibatch protocol when workers differ in speed and updates follow steepest-descent directions in a non-Euclidean norm. Our setting is statistically homogeneous (Section 1), which removes heterogeneity-induced client drift, although local stochastic trajectories can still differ. Our time model counts stochastic-gradient computation and treats communication in both directions and server-side processing, including evaluation of the sharp update, as instantaneous. We therefore do not analyze communication compression or local-step methods here. Combining these approaches with our treatment of worker speeds and update geometry is a natural direction for future work. C.4

Computational Cost of the Sharp Operator

The two approaches discussed above reduce communication costs by changing the size or frequency of messages exchanged between workers and the server. A separate computational cost arises from evaluating the non-Euclidean update direction itself. As shown in Section 3.1, for the spectral 29

update norm the canonical sharp selection is G♯ = ‖G‖nuc orth(G), where orth(G) is the polar factor. A direct implementation uses a compact rank-𝑟 singular value decomposition (SVD), where 𝑟 = rank(G), retaining only the nonzero singular values and their corresponding singular vectors. The complete orthogonal factors of a full SVD are unnecessary. This SVD-based implementation can be computationally expensive at the scale of modern models, motivating alternative algorithms for computing the polar factor. The analysis in this paper assumes exact access to the chosen sharp selection; extending its guarantees to numerical approximations requires separate control of the approximation error. Several lines of work aim to reduce this cost. The original Muon algorithm replaces the SVD by the computationally cheaper Newton–Schulz iteration. Subsequent work has proposed efficient polardecomposition subroutines [Lau et al., 2025, Amsel et al., 2025], acceleration of the Newton–Schulz iteration with Chebyshev-type polynomials [Grishina et al., 2025], and SVD-based orthogonalization of momentum within an adaptively updated low-dimensional subspace [Refael et al., 2025]. A different approach, Dion, replaces the full-rank step by a rank-𝑟 distributed orthonormalization built on power iteration and randomized Cholesky QR [Ahn et al., 2025].

30

D

Additional Discussion of Assumption 3

This section gives examples satisfying Assumption 3, explains the choice of norm and tail condition, and compares the resulting guarantees with those under bounded variance. D.1

Examples Satisfying the Assumption

Additive perturbations. Assumption 3 is satisfied, for instance, by the classical perturbed oracle ∇𝑓[︁ (𝑥; 𝜉) )︁]︁(𝑥) + 𝑍 with 𝑍 zero-mean and sub-Gaussian in the induced dual norm, that is, (︁ = 2∇𝑓 ‖𝑍‖* ≤ exp(1). More generally, our framework applies to the additively perturbed E exp 𝜎2 and inexact-gradient models studied in the literature [Ajalloeian and Stich, 2020, Gasnikov, 2017, Stonyakin et al., 2023, Devolder et al., 2014] only when their errors have zero mean and satisfy the dual-norm light-tail condition (11) with a common 𝜎 > 0 for all 𝑥 ∈ 𝐸. Bounded errors. An unbiased oracle with almost surely bounded error, ‖∇𝑓 (𝑥; 𝜉) − ∇𝑓 (𝑥)‖* ≤ 𝐺 for every 𝑥 ∈ 𝐸 with a common 𝐺 ≥ 0, also satisfies Assumption bound [︁ 3.(︁ If 𝐺 > 0, this pointwise )︁]︁ ‖∇𝑓 (𝑥;𝜉)−∇𝑓 (𝑥)‖2* is sufficient for the light-tail condition with 𝜎 = 𝐺, since E𝜉 exp ≤ exp(1). 𝐺2 If 𝐺 = 0, then for each fixed 𝑥 ∈ 𝐸 the oracle error vanishes almost surely. Hence the expectation in (11) equals 1 for any 𝜎 > 0. D.2

Noise in the Induced Dual Norm

Stationarity is measured in ‖·‖* , and the update direction (𝑔 𝑘 )♯ is produced from 𝑔 𝑘 through the dual norm, so ‖·‖* is the norm in which the estimation error actually enters the descent inequality. A reference Euclidean/Frobenius light-tail assumption also suffices: first concentrate the minibatch error in that norm, then use ‖𝑧‖* ≤ 𝛼−1 ‖𝑧‖F to transfer the error bound. This yields the statistical coefficient CF = 𝛼−2 𝜎F2 in (63). The native analysis instead uses C* = 𝜅𝜎 2 ; which coefficient is smaller depends on the geometry and the noise model (Section J). D.3

Exponential Moments and High-Probability Guarantees ⃦ ⃦ √ For a prescribed finite horizon 𝑇 ≥ 1, our analysis requires ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀/4 to hold simultaneously for 0 ≤ 𝑘 < 𝑇 with probability at least 1 − 𝛿. We apply the concentration bound conditionally at each iterate with failure probability at most 𝛿/𝑇 , then take a union bound over the 𝑇 iterations. The resulting minibatch size contains log(𝑇 /𝛿) = log 𝑇 + log(1/𝛿): the union bound contributes the horizon dependence log 𝑇 , while the concentration tail supplies the failure-probability dependence log(1/𝛿). D.4

Comparison with Bounded-Variance Oracles

For[︁ comparison, replace the ]︁ exponential-moment condition by the dual-norm variance bound 2 E𝜉 ‖∇𝑓 (𝑥; 𝜉) − ∇𝑓 (𝑥)‖* ≤ 𝜎 2 for all 𝑥 ∈ 𝐸. Retain unbiasedness, fresh i.i.d. samples independent of the past at each iteration, and the smoothness and lower-bound assumptions Assumption 1 and Assumption 2. As above, 𝜅 denotes of the dual space (𝐸 * , ‖·‖* ). At any [︂⃦ a regularity⃦constant ]︂ 2 ∑︀ ⃦ ⃦ 2 𝐵 fixed query point, Lemma 15 gives E ⃦ 𝐵1 𝑗=1 𝜂𝑗 ⃦ ≤ 𝜅𝜎 /𝐵 for the centered minibatch noise *

vectors 𝜂𝑗 ; fresh sampling gives the same bound conditionally on the history at each iterate. 1 With [︁ 𝛾𝑘 ≡ ]︁ 2𝐿 and suitable choices of 𝑇 and 𝐵, the corresponding analysis guarantees 2 E ‖∇𝑓 (ˆ 𝑥)‖* ≤ 𝜀, where 𝑥 ˆ = 𝑥𝑅 and 𝑅 is uniform on {0, . . . , 𝑇 − 1} independently of the run. For 𝐿 > 0, ∆ > 0, and 0 < 𝜀 ≤ 𝐿∆, the resulting oracle complexity is 𝑁* = 𝐵𝑇 = 2 𝒪(𝐿Δ/𝜀 + 𝜅𝐿Δ𝜎 /𝜀2 ). This gives an expectation counterpart to our light-tail guarantees; the present paper focuses on high-probability bounds.

31

E

Useful Auxiliary Results

This section collects definitions and auxiliary results used throughout the proofs. We begin with convex analysis and properties of primal and dual norms, then describe squared norms and their conjugates. We next give a generalized Young inequality and compare Schatten norms. Finally, we introduce the probabilistic definitions needed for vector-valued martingales and prove a secondmoment bound for martingale sums in 𝜅-regular spaces. Let 𝐸 be a finite-dimensional vector space, and let 𝐸 * be its dual. For 𝑠 ∈ 𝐸 * , we write ⟨𝑠, 𝑥⟩ to denote the action of 𝑠 on 𝑥. E.1

Foundations of Convex Analysis

To analyze non-smooth optimization objectives and generalized updates, we require the standard definition of the subgradient. Definition 2 (Subgradient). Let 𝑓 : 𝐸 → R ∪ {+∞} be proper. A vector 𝑠 ∈ 𝐸 * is a subgradient of 𝑓 at a point 𝑥 ∈ 𝐸 if 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨𝑠, 𝑦 − 𝑥⟩

∀𝑦 ∈ 𝐸.

(16)

The collection of all subgradients forms the subdifferential. Definition 3 (Subdifferential). Let 𝑓 : 𝐸 → R ∪ {+∞} be proper. The set of all subgradients of 𝑓 at 𝑥 is denoted 𝜕𝑓 (𝑥), and called the subdifferential of 𝑓 at 𝑥. We can think of the subdifferential as a set-valued operator 𝑥 ↦→ 𝜕𝑓 (𝑥). If 𝜕𝑓 (𝑥) ̸= ∅, we say that 𝑓 is subdifferentiable at 𝑥. The set of all such points 𝑥 is denoted dom 𝜕𝑓 . To ensure our algorithm is well-defined, we rely on the fact that the subdifferential is non-empty in the relative interior of the domain. Theorem 3 (Existence of subgradients). Let 𝐸 be a finite-dimensional real vector space and let 𝑓 : 𝐸 → R ∪ {+∞} be a proper convex function. (i) For every 𝑥 ∈ rint(dom 𝑓 ),16 the subdifferential 𝜕𝑓 (𝑥) is non-empty, closed, and convex. (ii) If 𝑥 ∈ int(dom 𝑓 ), where the interior is taken in 𝐸, then 𝜕𝑓 (𝑥) is also compact. Note that inequality (16) can be rearranged into 𝑓 (𝑥) − ⟨𝑠, 𝑥⟩ ≤ 𝑓 (𝑦) − ⟨𝑠, 𝑦⟩

∀𝑦 ∈ 𝐸,

which can be interpreted as follows: a vector 𝑠 is a subgradient of 𝑓 at 𝑥 if and only if 𝑥 minimizes the function 𝑦 ↦→ 𝑓 (𝑦) − ⟨𝑠, 𝑦⟩. We have arrived at the following result: Lemma 2. Let 𝑓 : 𝐸 → R ∪ {+∞} be proper and consider a point 𝑥 ∈ dom 𝑓 . Then 𝑠 ∈ 𝜕𝑓 (𝑥)

⇔

𝑥 ∈ arg min{𝑓 (𝑦) − ⟨𝑠, 𝑦⟩}.

(17)

𝑦∈𝐸

We note that the set of minimizers of a function is identical to the set of maximizers of the negative of the function. Applying this simple principle, we observe that arg min{𝑓 (𝑦) − ⟨𝑠, 𝑦⟩} = arg max{⟨𝑠, 𝑦⟩ − 𝑓 (𝑦)}. 𝑦∈𝐸

(18)

𝑦∈𝐸

We also recall the fundamental relationship between a function and its convex conjugate, which allows us to invert the subdifferential mapping. Theorem 4 (Fenchel-Moreau-Rockafellar). Let 𝑓 : 𝐸 → R ∪ {+∞} be proper, closed and convex. Then 𝑠 ∈ 𝜕𝑓 (𝑥)

⇔

𝑥 ∈ 𝜕𝑓 * (𝑠).

16 The relative interior of a set 𝑆, denoted rint(𝑆), is the interior of the set when viewed within the smallest affine subspace

that contains it.

32

E.2

Properties of the Primal and Dual Norms

Our analysis depends heavily on the interplay between the primal norm ‖·‖ and the dual norm ‖·‖* . We first verify that the dual norm satisfies the standard axioms. Lemma 3 (The Dual Norm is a Norm). Let (𝐸, ‖·‖) be a finite-dimensional normed space. The dual norm ‖·‖* , defined for any 𝑠 ∈ 𝐸 * as ‖𝑠‖* := sup ⟨𝑠, 𝑥⟩ , ‖𝑥‖≤1

satisfies the three axioms of a norm: definiteness, absolute homogeneity, and the triangle inequality. Proof. We first show that ‖𝑠‖* is finite for every 𝑠 ∈ 𝐸 * . Since 𝐸 is finite-dimensional, its closed unit ball is compact, and every linear functional on 𝐸 is continuous. Therefore, the supremum defining the dual norm is attained and finite: ‖𝑠‖* = sup ⟨𝑠, 𝑥⟩ = max ⟨𝑠, 𝑥⟩ < +∞. ‖𝑥‖≤1

‖𝑥‖≤1

We now verify the three norm axioms. Proof of Property (i): Definiteness. We must show that ‖𝑠‖* ≥ 0, with equality if and only if 𝑠 = 0. • Non-negativity: The supremum is taken over the unit ball {𝑥 ∈ 𝐸 | ‖𝑥‖ ≤ 1}, which contains the origin 0𝐸 . At this point, the dual pairing evaluates to ⟨𝑠, 0𝐸 ⟩ = 0. Since the supremum over a set must be greater than or equal to the value of the objective at any point in that set, we conclude that ‖𝑠‖* ≥ 0. • Positive Definiteness: First, if 𝑠 = 0, its dual norm is clearly zero: ‖0‖* = sup‖𝑥‖≤1 ⟨0, 𝑥⟩ = 0. For the converse, assume that ‖𝑠‖* = 0. By definition of the supremum, this implies that ⟨𝑠, 𝑥⟩ ≤ 0 for all 𝑥 in the unit ball {𝑥 ∈ 𝐸 | ‖𝑥‖ ≤ 1}. We will now show that this can only be true if 𝑠 = 0. Assume for contradiction that 𝑠 ̸= 0. As a non-zero linear functional, 𝑠 cannot be zero for all inputs, so there must exist some vector 𝑦 ∈ 𝐸 such that ⟨𝑠, 𝑦⟩ ̸= 0. Let’s construct a specific vector 𝑥0 in the unit ball. If ⟨𝑠, 𝑦⟩ > 0, we can simply set 𝑥0 = 𝑦/‖𝑦‖. If ⟨𝑠, 𝑦⟩ < 0, we can set 𝑥0 = −𝑦/‖𝑦‖. In either case, we have a vector 𝑥0 such that ‖𝑥0 ‖ = 1 and, crucially, ⟨𝑠, 𝑥0 ⟩ = |⟨𝑠,𝑦⟩|/‖𝑦‖ > 0. The existence of such an 𝑥0 contradicts our initial deduction that ⟨𝑠, 𝑥⟩ ≤ 0 for all vectors in the unit ball. Therefore, the initial assumption that 𝑠 ̸= 0 must be false. We conclude that if ‖𝑠‖* = 0, then 𝑠 = 0. Proof of Property (ii): Absolute Homogeneity. We need to show that ‖𝑡𝑠‖* = |𝑡| ‖𝑠‖* for any scalar 𝑡 ∈ R. ‖𝑡𝑠‖* = sup ⟨𝑡𝑠, 𝑥⟩ = sup 𝑡 ⟨𝑠, 𝑥⟩ . ‖𝑥‖≤1

‖𝑥‖≤1

We consider three cases for the scalar 𝑡: • If 𝑡 > 0, we can pull the positive constant out of the supremum: ‖𝑡𝑠‖* = 𝑡 sup ⟨𝑠, 𝑥⟩ = 𝑡 ‖𝑠‖* = |𝑡| ‖𝑠‖* . ‖𝑥‖≤1

• If 𝑡 < 0, let 𝑡 = −|𝑡|. Then: ‖𝑡𝑠‖* = sup (−|𝑡| ⟨𝑠, 𝑥⟩) = |𝑡| sup (− ⟨𝑠, 𝑥⟩). ‖𝑥‖≤1

‖𝑥‖≤1

Let 𝑦 = −𝑥. The set {𝑦 ∈ 𝐸 | ‖𝑦‖ ≤ 1} is the same as {𝑥 ∈ 𝐸 | ‖𝑥‖ ≤ 1} because ‖−𝑥‖ = ‖𝑥‖. Therefore, we can change the variable in the supremum: sup (− ⟨𝑠, 𝑥⟩) = sup ⟨𝑠, 𝑦⟩ = ‖𝑠‖* . ‖𝑥‖≤1

‖𝑦‖≤1

Combining these gives ‖𝑡𝑠‖* = |𝑡| ‖𝑠‖* . 33

• If 𝑡 = 0, we have ‖0 · 𝑠‖* = ‖0‖* = 0, and |0| ‖𝑠‖* = 0. The identity holds for all 𝑡 ∈ R. Proof of Property (iii): Triangle Inequality. We need to show that ‖𝑠1 + 𝑠2 ‖* ≤ ‖𝑠1 ‖* + ‖𝑠2 ‖* . The proof follows directly from the definition of the dual norm and the properties of the supremum. ‖𝑠1 + 𝑠2 ‖* = sup ⟨𝑠1 + 𝑠2 , 𝑥⟩ ‖𝑥‖≤1

= sup (⟨𝑠1 , 𝑥⟩ + ⟨𝑠2 , 𝑥⟩)

(by linearity of the dual pairing)

‖𝑥‖≤1

≤ sup ⟨𝑠1 , 𝑥⟩ + sup ⟨𝑠2 , 𝑥⟩ ‖𝑥‖≤1

(by subadditivity of supremum)

‖𝑥‖≤1

= ‖𝑠1 ‖* + ‖𝑠2 ‖* . This completes the proof.

To relate geometric properties (like regularity) between primal and dual spaces, we need to understand how norm equivalence translates to the dual space. Lemma 4 (Duality of Norm Equivalence). Let ‖·‖𝑎 and ‖·‖𝑏 be two norms on a finite-dimensional real vector space 𝐸. The subscripts 𝑎 and 𝑏 are neutral labels for the two auxiliary norms considered in this lemma. Let ‖·‖𝑎,* and ‖·‖𝑏,* denote their respective dual norms on 𝐸 * . If ‖𝑥‖𝑎 ≤ ‖𝑥‖𝑏 ≤ 𝐶 ‖𝑥‖𝑎

∀𝑥 ∈ 𝐸,

for some constant 𝐶 ≥ 1, then their dual norms satisfy the reverse equivalence inequality: 1 ‖𝑠‖𝑎,* ≤ ‖𝑠‖𝑏,* ≤ ‖𝑠‖𝑎,* 𝐶

∀𝑠 ∈ 𝐸 * .

Proof. The proof relies on the definition of the dual norm as the supremum of the dual pairing over the unit ball of the primal norm. Let 𝐵𝑎 = {𝑥 ∈ 𝐸 | ‖𝑥‖𝑎 ≤ 1} and 𝐵𝑏 = {𝑥 ∈ 𝐸 | ‖𝑥‖𝑏 ≤ 1} be the unit balls of the respective primal norms. Step 1: Proving ‖𝑠‖𝑏,* ≤ ‖𝑠‖𝑎,* . The first part of the primal norm inequality is ‖𝑥‖𝑎 ≤ ‖𝑥‖𝑏 . This inequality has a direct geometric interpretation for the unit balls. If a point 𝑦 is in 𝐵𝑏 , then ‖𝑦‖𝑏 ≤ 1. From the inequality, this implies that ‖𝑦‖𝑎 ≤ 1, which means 𝑦 must also be in 𝐵𝑎 . Therefore, we have the set inclusion: 𝐵𝑏 ⊆ 𝐵𝑎 . Now we use the definition of the dual norms. For any 𝑠 ∈ 𝐸 * : ‖𝑠‖𝑏,* = sup ⟨𝑠, 𝑥⟩ . 𝑥∈𝐵𝑏

Since we are taking the supremum of the same linear function ⟨𝑠, 𝑥⟩ over a smaller set (𝐵𝑏 ) compared to the set 𝐵𝑎 , its value cannot be larger: sup ⟨𝑠, 𝑥⟩ ≤ sup ⟨𝑠, 𝑥⟩ . 𝑥∈𝐵𝑏

𝑥∈𝐵𝑎

By definition, the right-hand side is ‖𝑠‖𝑎,* . Thus, we have shown: ‖𝑠‖𝑏,* ≤ ‖𝑠‖𝑎,* . Step 2: Proving 𝐶1 ‖𝑠‖𝑎,* ≤ ‖𝑠‖𝑏,* . The second part of the primal norm inequality is ‖𝑥‖𝑏 ≤ 𝐶 ‖𝑥‖𝑎 . This can be rewritten as 𝐶1 ‖𝑥‖𝑏 ≤ ‖𝑥‖𝑎 . This implies a relationship between the unit balls in the opposite direction. If a point 𝑦 is in 𝐵𝑎 , then ‖𝑦‖𝑎 ≤ 1. From the inequality, we have ‖𝑦‖𝑏 ≤ 𝐶 · ‖𝑦‖𝑎 ≤ 𝐶 · 1 = 𝐶. This means that any point in 𝐵𝑎 is also in the ball of radius 𝐶 for the ‖·‖𝑏 norm. This gives the set inclusion: 𝐵𝑎 ⊆ 𝐶 · 𝐵𝑏 = {𝑥 ∈ 𝐸 | ‖𝑥‖𝑏 ≤ 𝐶}. 34

Now we consider the definition of the dual norm ‖𝑠‖𝑎,* : ‖𝑠‖𝑎,* = sup ⟨𝑠, 𝑥⟩ . 𝑥∈𝐵𝑎

Since 𝐵𝑎 is a subset of 𝐶 · 𝐵𝑏 , the supremum over 𝐵𝑎 cannot be larger than the supremum over 𝐶 · 𝐵𝑏 : sup ⟨𝑠, 𝑥⟩ ≤ sup ⟨𝑠, 𝑥⟩ . 𝑥∈𝐵𝑎

𝑥∈𝐶·𝐵𝑏

The supremum of a linear functional over a scaled ball is the scaling factor times the supremum over the unscaled ball: sup ⟨𝑠, 𝑥⟩ = sup ⟨𝑠, 𝐶𝑦⟩ = 𝐶 sup ⟨𝑠, 𝑦⟩ = 𝐶 ‖𝑠‖𝑏,* .

𝑥∈𝐶·𝐵𝑏

𝑦∈𝐵𝑏

𝑦∈𝐵𝑏

Combining these steps, we have shown: ‖𝑠‖𝑎,* ≤ 𝐶 ‖𝑠‖𝑏,* . Since 𝐶 ≥ 1, we can divide by 𝐶 to get the final inequality: 1 ‖𝑠‖𝑎,* ≤ ‖𝑠‖𝑏,* . 𝐶 Putting the two parts together completes the proof of the dual norm equivalence. We often require the existence of a dual vector that "certifies" the norm of a primal vector. This is guaranteed by the Hahn-Banach theorem. Lemma 5 (Consequence of Hahn-Banach theorem). Let (𝐸, ‖·‖) be a nonzero finite-dimensional real normed space. For every 𝑥 ∈ 𝐸, there exists 𝑠 ∈ 𝐸 * such that ⟨𝑠, 𝑥⟩ = ‖𝑥‖

and

‖𝑠‖* = 1.

For completeness, we also state the reflexivity property for finite-dimensional normed spaces. Theorem 5 (Bidual norm equals primal norm). Let ‖·‖** be the dual of the dual norm ‖·‖* , called the bidual norm. Then ‖·‖** = ‖·‖ . Proof. The proof consists of two parts, showing the inequality in both directions. (i) We first show that ‖𝑥‖** ≤ ‖𝑥‖. Using the definition of the bidual norm and applying the duality inequality, we get ‖𝑥‖** = sup ⟨𝑠, 𝑥⟩ ≤ sup ‖𝑠‖* ‖𝑥‖ = ‖𝑥‖ . ‖𝑠‖* ≤1

‖𝑠‖* ≤1

(ii) We will now prove that ‖𝑥‖** ≥ ‖𝑥‖. This holds trivially for 𝑥 = 0. So, let us consider 𝑥 ̸= 0. The definition of the bidual norm is ‖𝑥‖** = sup ⟨𝑠, 𝑥⟩ . ‖𝑠‖* ≤1

Using Lemma 5, we know that there exists 𝑠′ ∈ 𝐸 * such that ‖𝑠′ ‖* = 1 and ⟨𝑠′ , 𝑥⟩ = ‖𝑥‖. Since this specific 𝑠′ is in the set over which the supremum is taken, the supremum must be at least the value obtained at 𝑠′ . Therefore, ‖𝑥‖** = sup ⟨𝑠, 𝑥⟩ ≥ ⟨𝑠′ , 𝑥⟩ = ‖𝑥‖ . ‖𝑠‖* ≤1

Combining both inequalities proves the theorem. The following two lemmas characterize the subdifferentials of the norm function itself. These are building blocks for deriving the subdifferentials of squared norms. 35

Lemma 6 (Norm of the Subgradient of a Norm). Let (𝐸, ‖·‖) be a finite-dimensional real normed space, and let ‖·‖* be its dual norm. For every nonzero 𝑠 ∈ 𝐸 * and every 𝑥 ∈ 𝜕 ‖𝑠‖* ,17 we have ‖𝑥‖ = 1. Proof. The proof relies on two fundamental inequalities that characterize the subdifferential of a convex function. Let 𝑠 ∈ 𝐸 * be a non-zero vector. By definition, a vector 𝑥 ∈ 𝐸 is a subgradient of the dual norm ‖·‖* at 𝑠 if the following inequality holds for all 𝑧 ∈ 𝐸 * : ‖𝑧‖* ≥ ‖𝑠‖* + ⟨𝑥, 𝑧 − 𝑠⟩ .

(19)

Step 1: Proving ‖𝑥‖ ≤ 1. We can choose a specific value for 𝑧 in the subgradient inequality. Let us choose 𝑧 = 𝑠 + ℎ for any ℎ ∈ 𝐸 * . The inequality becomes ‖𝑠 + ℎ‖* ≥ ‖𝑠‖* + ⟨𝑥, ℎ⟩. By the triangle inequality, ‖𝑠 + ℎ‖* ≤ ‖𝑠‖* + ‖ℎ‖* . This gives: ‖𝑠‖* + ‖ℎ‖* ≥ ‖𝑠‖* + ⟨𝑥, ℎ⟩ =⇒ ⟨𝑥, ℎ⟩ ≤ ‖ℎ‖* . This inequality must hold for all ℎ ∈ 𝐸 * . By the definition of the primal norm as the dual of the dual norm, we have: ‖𝑥‖ =

sup

⟨𝑥, ℎ⟩ .

ℎ∈𝐸 * ,‖ℎ‖* ≤1

Since ⟨𝑥, ℎ⟩ ≤ ‖ℎ‖* , this implies that for any ℎ with ‖ℎ‖* ≤ 1, we have ⟨𝑥, ℎ⟩ ≤ 1. Therefore, the supremum must be less than or equal to 1, which proves ‖𝑥‖ ≤ 1. Step 2: Proving ‖𝑥‖ ≥ 1. Now we choose a different value for 𝑧 in the subgradient inequality. Let us choose 𝑧 = 0. Since ‖0‖* = 0, substituting into (19) gives: 0 ≥ ‖𝑠‖* + ⟨𝑥, 0 − 𝑠⟩ 0 ≥ ‖𝑠‖* − ⟨𝑥, 𝑠⟩ . Rearranging this gives the inequality: ⟨𝑠, 𝑥⟩ ≥ ‖𝑠‖* . Now, by the duality inequality, we know that ⟨𝑠, 𝑥⟩ ≤ ‖𝑠‖* ‖𝑥‖. Combining these two results: ‖𝑠‖* ≤ ⟨𝑠, 𝑥⟩ ≤ ‖𝑠‖* ‖𝑥‖ . This chain of inequalities implies ‖𝑠‖* ≤ ‖𝑠‖* ‖𝑥‖. Since we assumed 𝑠 is a non-zero vector, its norm ‖𝑠‖* is strictly positive. We can therefore divide both sides by ‖𝑠‖* without changing the direction of the inequality: 1 ≤ ‖𝑥‖ . Combining the results from Step 1 (‖𝑥‖ ≤ 1) and Step 2 (‖𝑥‖ ≥ 1), we conclude that we must have exactly ‖𝑥‖ = 1. Lemma 7 (Subdifferential of the norm). The subdifferentials of ‖·‖ and ‖·‖* are given by {︂ {𝑠 ∈ 𝐸 * | ‖𝑠‖* ≤ 1} if 𝑥 = 0 𝜕 ‖𝑥‖ = , {𝑠 ∈ 𝐸 * | ⟨𝑠, 𝑥⟩ = ‖𝑥‖ and ‖𝑠‖* = 1} if 𝑥 ̸= 0 {︂ {𝑥 ∈ 𝐸 | ‖𝑥‖ ≤ 1} if 𝑠 = 0 𝜕 ‖𝑠‖* = . {𝑥 ∈ 𝐸 | ⟨𝑠, 𝑥⟩ = ‖𝑠‖* and ‖𝑥‖ = 1} if 𝑠 ̸= 0

(20) (21)

Proof. We prove (20). The dual formula follows by applying the primal formula to 𝐸 * and identifying the bidual norm with the primal norm using Theorem 5. By definition, 𝑠 ∈ 𝜕 ‖𝑥‖

⇔ ⇔ ⇔

‖𝑦‖ ≥ ‖𝑥‖ + ⟨𝑠, 𝑦 − 𝑥⟩ ∀𝑦 ∈ 𝐸 ⟨𝑠, 𝑥⟩ − ‖𝑥‖ ≥ ⟨𝑠, 𝑦⟩ − ‖𝑦‖ ∀𝑦 ∈ 𝐸 𝑥 ∈ arg max{⟨𝑠, 𝑦⟩ − ‖𝑦‖}.

(22)

𝑦∈𝐸 17We write 𝜕 ‖𝑠‖

* for 𝜕(‖·‖* )(𝑠), the subdifferential of the function 𝑢 ↦→ ‖𝑢‖* evaluated at 𝑠. Similarly, 𝜕 ‖𝑥‖ denotes

𝜕(‖·‖)(𝑥).

36

We know from the proof of Lemma 8 that {︂ sup {⟨𝑠, 𝑦⟩ − ‖𝑦‖} = 𝑦∈𝐸

+∞ 0

when ‖𝑠‖* > 1 . when ‖𝑠‖* ≤ 1

Hence, for 𝑥 to be a maximizer, we must have that the maximum is finite (so ‖𝑠‖* ≤ 1) and that the value ⟨𝑠, 𝑥⟩ − ‖𝑥‖ is equal to this maximum value of 0. Therefore, 𝑥 ∈ arg max{⟨𝑠, 𝑦⟩ − ‖𝑦‖}

⇔

⟨𝑠, 𝑥⟩ = ‖𝑥‖

and

𝑦∈𝐸

‖𝑠‖* ≤ 1.

We now consider two cases for 𝑥. If 𝑥 = 0, the condition ⟨𝑠, 𝑥⟩ = ‖𝑥‖ is automatically satisfied. Thus, the characterization becomes: 0 ∈ arg max{⟨𝑠, 𝑦⟩ − ‖𝑦‖}

⇔

𝑦∈𝐸

‖𝑠‖* ≤ 1.

(23)

If 𝑥 ̸= 0, by the duality inequality, it is impossible for ⟨𝑠, 𝑥⟩ = ‖𝑥‖ to hold unless ‖𝑠‖* ≥ 1. Combining this with the condition ‖𝑠‖* ≤ 1, we must have ‖𝑠‖* = 1. Hence, 𝑥 ∈ arg max{⟨𝑠, 𝑦⟩ − ‖𝑦‖}

⇔

⟨𝑠, 𝑥⟩ = ‖𝑥‖

and

𝑦∈𝐸

‖𝑠‖* = 1.

(24)

It only remains to combine (22) with (23) and (24) to establish the result. It is also useful to identify the convex conjugate of a norm, which corresponds to the indicator function of the dual unit ball. Lemma 8 (Fenchel conjugate of the norm). The Fenchel conjugate of the norm ‖·‖ is the indicator function of the unit ball with respect to the dual norm. That is, {︂ 0 if ‖𝑠‖* ≤ 1 * (‖·‖) (𝑠) = 𝜄B* (𝑠) := , +∞ if ‖𝑠‖* > 1 where B* := {𝑠 ∈ 𝐸 * | ‖𝑠‖* ≤ 1}. Proof. We consider two cases based on the value of ‖𝑠‖* . Case 1: ‖𝑠‖* ≤ 1. By the definition of the Fenchel conjugate and the duality inequality, we have: (‖·‖)* (𝑠) = sup {⟨𝑠, 𝑦⟩ − ‖𝑦‖} 𝑦∈𝐸

≤ sup {‖𝑠‖* ‖𝑦‖ − ‖𝑦‖} 𝑦∈𝐸

≤ sup {‖𝑦‖ − ‖𝑦‖} = 0. 𝑦∈𝐸

On the other hand, by choosing 𝑦 = 0, we see that the expression ⟨𝑠, 𝑦⟩ − ‖𝑦‖ evaluates to 0. Since the supremum must be at least this value, we have (‖·‖)* (𝑠) ≥ 0. Combining these, we conclude that (‖·‖)* (𝑠) = 0. Case 2: ‖𝑠‖* > 1. Since ‖𝑠‖* = sup‖𝑥‖≤1 ⟨𝑠, 𝑥⟩, there must exist some 𝑥 ∈ 𝐸 such that ‖𝑥‖ ≤ 1 and ⟨𝑠, 𝑥⟩ > 1. For such an 𝑥, it follows that ⟨𝑠, 𝑥⟩ − ‖𝑥‖ > 0. Therefore, we can lower-bound the supremum: (‖·‖)* (𝑠) = sup {⟨𝑠, 𝑦⟩ − ‖𝑦‖} 𝑦∈𝐸

≥ sup{⟨𝑠, 𝑡𝑥⟩ − ‖𝑡𝑥‖} 𝑡>0

= sup{𝑡(⟨𝑠, 𝑥⟩ − ‖𝑥‖)}. 𝑡>0

Since the term (⟨𝑠, 𝑥⟩ − ‖𝑥‖) is a positive constant, the supremum over 𝑡 > 0 is +∞. 37

E.3

Properties of Squared Norms and their Conjugates

In this section, we establish fundamental properties relating the squared norm function 𝜑(𝑥) := 2 2 1 1 * 2 ‖𝑥‖ and its Fenchel conjugate 𝜑 (𝑠) := 2 ‖𝑠‖* to their respective subgradients. These identities are central to the analysis of non-Euclidean methods and will be used extensively in our proofs. First, we state the classic inequality bounding the dual pairing by the sum of a function and its conjugate. Theorem 6 (Fenchel-Young inequality). Let 𝑓 : 𝐸 → R ∪ {+∞} be proper, and define its Fenchel conjugate by 𝑓 * (𝑠) := sup {⟨𝑠, 𝑦⟩ − 𝑓 (𝑦)},

𝑠 ∈ 𝐸*.

𝑦∈𝐸

Then, for any 𝑥 ∈ dom 𝑓 and 𝑠 ∈ 𝐸 * , ⟨𝑠, 𝑥⟩ ≤ 𝑓 * (𝑠) + 𝑓 (𝑥).

(25)

Moreover, 𝑠 ∈ 𝜕𝑓 (𝑥)

⇔

⟨𝑠, 𝑥⟩ = 𝑓 * (𝑠) + 𝑓 (𝑥).

(26)

We next characterize the subdifferentials of the squared primal and dual norms. These are generally 2 set-valued; the sharp operator selects an element of 𝜕𝜑* (𝑠) for each 𝑠 ∈ 𝐸 * , where 𝜑* (𝑠) = 12 ‖𝑠‖* . 2

2

Lemma 9 (Subdifferential of 𝜑 and 𝜑* ). The subdifferentials of 𝜑(𝑥) := 12 ‖𝑥‖ and 𝜑* (𝑠) := 12 ‖𝑠‖* are given by: (i) 𝜕𝜑(𝑥) = {𝑠 ∈ 𝐸 * | ⟨𝑠, 𝑥⟩ = ‖𝑠‖* ‖𝑥‖ and ‖𝑠‖* = ‖𝑥‖}. (ii) 𝜕𝜑* (𝑠) = {𝑥 ∈ 𝐸 | ⟨𝑠, 𝑥⟩ = ‖𝑠‖* ‖𝑥‖ and ‖𝑥‖ = ‖𝑠‖* }. Proof. We first verify that 𝜑* is the Fenchel conjugate of 𝜑. By the duality inequality, for every 𝑠 ∈ 𝐸*, }︂ {︂ }︂ {︂ 1 2 1 1 2 2 = ‖𝑠‖* . sup ⟨𝑠, 𝑥⟩ − ‖𝑥‖ ≤ sup 𝑟 ‖𝑠‖* − 𝑟 2 2 2 𝑥∈𝐸 𝑟≥0 If 𝑠 = 0, this upper bound is attained at 𝑥 = 0. If 𝑠 ̸= 0, compactness of the unit ball and continuity of the linear functional give a maximizer 𝑢 with ⟨𝑠, 𝑢⟩ = ‖𝑠‖* > 0. This maximizer satisfies ‖𝑢‖ = 1, since otherwise scaling it to have norm one would increase ⟨𝑠, 𝑢⟩. Taking 𝑥 = ‖𝑠‖* 𝑢, we obtain ⟨𝑠, 𝑥⟩ −

1 1 1 2 2 2 2 ‖𝑥‖ = ‖𝑠‖* − ‖𝑠‖* = ‖𝑠‖* . 2 2 2 2

Thus the supremum equals 𝜑* (𝑠) = 21 ‖𝑠‖* for every 𝑠 ∈ 𝐸 * . We now prove the two subdifferential characterizations. (i) We first characterize the subdifferential 𝜕𝜑(𝑥). By the Fenchel-Young equality (Theorem 6), an element 𝑠 ∈ 𝐸 * is in 𝜕𝜑(𝑥) if and only if the equality holds in the Fenchel-Young inequality: ⟨𝑠, 𝑥⟩ = 𝜑(𝑥) + 𝜑* (𝑠). Substituting the definitions of 𝜑 and 𝜑* gives the equivalent condition: 1 1 2 2 ⟨𝑠, 𝑥⟩ = ‖𝑥‖ + ‖𝑠‖* . (27) 2 2 We now establish a general chain of inequalities. By the duality inequality, we have: ⟨𝑠, 𝑥⟩ ≤ ‖𝑠‖* ‖𝑥‖ . Furthermore, the standard Young’s inequality for scalars, 𝑎𝑏 ≤ 12 𝑎2 + 12 𝑏2 , applied with 𝑎 = ‖𝑠‖* and 𝑏 = ‖𝑥‖, yields: ‖𝑠‖* ‖𝑥‖ ≤

38

1 1 2 2 ‖𝑠‖* + ‖𝑥‖ . 2 2

Combining these two results gives the chain: ⟨𝑠, 𝑥⟩ ≤ ‖𝑠‖* ‖𝑥‖ ≤

1 1 2 2 ‖𝑠‖* + ‖𝑥‖ . 2 2

(28)

For 𝑠 to be an element of 𝜕𝜑(𝑥), the condition in (27) must hold, which means the first and last terms of the chain in (28) must be equal. This can only occur if both inequalities in the chain hold with equality. • The first inequality becomes an equality, ⟨𝑠, 𝑥⟩ = ‖𝑠‖* ‖𝑥‖, which is the equality condition in the duality inequality. 2 2 • The second inequality becomes an equality, ‖𝑠‖* ‖𝑥‖ = 12 ‖𝑠‖* + 12 ‖𝑥‖ . Rearranging 2 2 gives ‖𝑠‖* − 2 ‖𝑠‖* ‖𝑥‖ + ‖𝑥‖ = 0, which is equivalent to (‖𝑠‖* − ‖𝑥‖)2 = 0. This holds if and only if ‖𝑠‖* = ‖𝑥‖. Therefore, 𝑠 ∈ 𝜕𝜑(𝑥) if and only if both conditions are satisfied, which proves the first statement. (ii) The proof for 𝜕𝜑* (𝑠) follows from a symmetric argument. We apply the result from part (i) to the function 𝜑* (𝑠) defined on the space 𝐸 * . The dual of 𝐸 * is the bidual space 𝐸 ** , and the corresponding dual norm is the bidual norm, ‖·‖** . Applying the characterization from (i) to 𝜑* (with the roles of the spaces and norms interchanged) gives that 𝑥 ∈ 𝜕𝜑* (𝑠) if and only if: 𝑥 ∈ 𝐸 ** ,

⟨𝑥, 𝑠⟩ = ‖𝑥‖** ‖𝑠‖* ,

and

‖𝑥‖** = ‖𝑠‖* .

As stated in Theorem 5, for a finite-dimensional space, there is a canonical isomorphism between 𝐸 and 𝐸 ** , and the bidual norm is equal to the primal norm, i.e., ‖·‖** = ‖·‖. We can thus identify 𝑥 ∈ 𝐸 ** with 𝑥 ∈ 𝐸 and replace the bidual norm with the primal norm. This substitution yields the desired characterization for 𝜕𝜑* (𝑠): 𝜕𝜑* (𝑠) = {𝑥 ∈ 𝐸 | ⟨𝑠, 𝑥⟩ = ‖𝑠‖* ‖𝑥‖ and ‖𝑥‖ = ‖𝑠‖* }. This completes the proof. Nonemptiness follows directly from Lemma 5. For 𝑥 ̸= 0, choose 𝑠0 ∈ 𝐸 * with ‖𝑠0 ‖* = 1 and ⟨𝑠0 , 𝑥⟩ = ‖𝑥‖; then 𝑠 = ‖𝑥‖ 𝑠0 satisfies the conditions in part (i). For 𝑥 = 0, take 𝑠 = 0. The same argument on 𝐸 * establishes nonemptiness of 𝜕𝜑* (𝑠). For 𝑠 ̸= 0, every 𝑥 ∈ 𝜕𝜑* (𝑠) has ‖𝑥‖ = ‖𝑠‖* > 0. Writing 𝑥 = ‖𝑠‖* 𝑢 gives 2

𝜕𝜑* (𝑠) = {𝑥 ∈ 𝐸 | ⟨𝑠, 𝑥⟩ = ‖𝑠‖* , ‖𝑥‖ = ‖𝑠‖* } = ‖𝑠‖* · {𝑢 ∈ 𝐸 | ‖𝑢‖ = 1, ⟨𝑠, 𝑢⟩ = ‖𝑠‖* } = ‖𝑠‖* · 𝜕 ‖𝑠‖* ,

(29) *

where the last equality follows from Lemma 7. At 𝑠 = 0, the characterization above gives 𝜕𝜑 (0) = {0}, and 𝜕 ‖0‖* is the primal unit ball, so (29) also holds at zero. 2

Lemma 10 (Subdifferential of the Squared Dual Norm). Let 𝜑* (𝑠) := 12 ‖𝑠‖* . For any 𝑠 ∈ 𝐸 * , its subdifferential is given by 𝜕𝜑* (𝑠) = ‖𝑠‖* · 𝜕 ‖𝑠‖* , where 𝜕 ‖𝑠‖* is the subdifferential of the dual norm at 𝑠. If the dual norm is differentiable at 𝑠, then this relationship simplifies to ∇𝜑* (𝑠) = ‖𝑠‖* · ∇ ‖𝑠‖* . Proof. For 𝑠 ̸= 0, the identity follows from (29), obtained by normalizing the subgradients characterized in Lemma 9. At 𝑠 = 0, Lemma 9 gives 𝜕𝜑* (0) = {0}, while Lemma 7 gives 𝜕 ‖0‖* = {𝑥 ∈ 𝐸 : ‖𝑥‖ ≤ 1}. 39

Consequently, 𝜕𝜑* (0) = {0} = ‖0‖* · 𝜕 ‖0‖* . Thus the subdifferential identity holds for every 𝑠 ∈ 𝐸 * . 2

If the dual norm is differentiable at 𝑠, the ordinary chain rule applied to 𝜑* (𝑠) = 12 ‖𝑠‖* yields ∇𝜑* (𝑠) = ‖𝑠‖* · ∇ ‖𝑠‖* .

These subdifferential mappings satisfy useful scaling properties. Lemma 11 (Homogeneity of 𝜕𝜑 and 𝜕𝜑* ). The subdifferentials of 𝜑 and 𝜑* satisfy the relations 𝜕𝜑(𝑡𝑥) = 𝑡𝜕𝜑(𝑥), ∀𝑥 ∈ 𝐸, 𝑡 ∈ R, 𝜕𝜑* (𝑡𝑠) = 𝑡𝜕𝜑* (𝑠), ∀𝑠 ∈ 𝐸 * , 𝑡 ∈ R. In particular, 𝜕𝜑(0) = {0} and 𝜕𝜑* (0) = {0}. Finally, we derive key identities involving the sharp operator (·)♯ , which are used directly in the derivation of the descent lemma. Lemma 12 (Properties of the Sharp Operators). Let 𝑥 ∈ 𝐸 and 𝑠 ∈ 𝐸 * . The following four identities hold for any element selected from the respective subdifferentials: 2

(i) For any 𝑢 ∈ 𝜕𝜑* (𝑠), we have ⟨𝑠, 𝑢⟩ = ‖𝑠‖* . (ii) For any 𝑢 ∈ 𝜕𝜑* (𝑠), we have ‖𝑢‖ = ‖𝑠‖* . 2

(iii) For any 𝑣 ∈ 𝜕𝜑(𝑥), we have ⟨𝑣, 𝑥⟩ = ‖𝑥‖ . (iv) For any 𝑣 ∈ 𝜕𝜑(𝑥), we have ‖𝑣‖* = ‖𝑥‖. Proof. Let 𝑢 ∈ 𝜕𝜑* (𝑠) and 𝑣 ∈ 𝜕𝜑(𝑥) be arbitrary. By Lemma 9(ii), 2

‖𝑢‖ = ‖𝑠‖* ,

⟨𝑠, 𝑢⟩ = ‖𝑠‖* ‖𝑢‖ = ‖𝑠‖* ,

which proves (i) and (ii). Similarly, Lemma 9(i) gives 2

‖𝑣‖* = ‖𝑥‖ ,

⟨𝑣, 𝑥⟩ = ‖𝑣‖* ‖𝑥‖ = ‖𝑥‖ ,

which proves (iii) and (iv). E.4

Inequalities and Specific Geometries

To bound dual pairings in our convergence analysis, we require the following generalized version of Young’s inequality. Lemma 13 (Generalized Young inequality). Let ‖ · ‖ be any norm on R𝑑 with dual norm ‖𝑢‖* = sup‖𝑥‖≤1 ⟨𝑢, 𝑥⟩. Then for all 𝑢, 𝑣 ∈ R𝑑 and every 𝑠 > 0, ⟨𝑢, 𝑣⟩ ≤ ‖𝑢‖* ‖𝑣‖ ≤

1 𝑠 ‖𝑢‖2* + ‖𝑣‖2 . 2𝑠 2

Proof. (i) Duality (Hölder) inequality. If 𝑣 = 0 the claim is trivial. Otherwise, set 𝑤 := 𝑣/‖𝑣‖, so ‖𝑤‖ = 1. By the definition of the dual norm, ⟨𝑢, 𝑤⟩ ≤ sup ⟨𝑢, 𝑥⟩ = ‖𝑢‖* . ‖𝑥‖≤1

Multiplying by ‖𝑣‖ gives ⟨𝑢, 𝑣⟩ = ‖𝑣‖⟨𝑢, 𝑤⟩ ≤ ‖𝑢‖* ‖𝑣‖. (ii) Young’s inequality with a parameter. For any 𝑎, 𝑏 ∈ R and any 𝑠 > 0, √︂ )︁ (︁ 𝑎 𝑠 2 𝑎2 𝑠 0≤ √ − 𝑏 = + 𝑏2 − 𝑎𝑏, 2 2𝑠 2 2𝑠 40

2

hence 𝑎𝑏 ≤ 𝑎2𝑠 + 2𝑠 𝑏2 . Applying this with 𝑎 = ‖𝑢‖* and 𝑏 = ‖𝑣‖ yields ‖𝑢‖* ‖𝑣‖ ≤

1 𝑠 ‖𝑢‖2* + ‖𝑣‖2 . 2𝑠 2

Combining (i) and (ii) proves the statement. For our examples involving matrix optimization, we utilize the comparison inequalities between different Schatten norms. Lemma 14 (Schatten norm comparison). Let 𝑋 ∈ R𝑚×𝑛 and 𝑟 := min{𝑚, 𝑛}. For any 1 ≤ 𝜌 ≤ 𝑝 ≤ ∞, 1

1

‖𝑋‖𝑆𝑝 ≤ ‖𝑋‖𝑆𝜌 ≤ 𝑟 𝜌 − 𝑝 ‖𝑋‖𝑆𝑝 . Proof. Let 𝜎(𝑋) ∈ R𝑟+ be the vector of singular values of 𝑋. By definition, ‖𝑋‖𝑆𝑞 = ‖𝜎(𝑋)‖𝑞 for every 𝑞 ∈ [1, ∞]. Applying the standard ℓ𝑝 comparison inequality in R𝑟 to 𝜎(𝑋) yields the claim. E.5

Probabilistic Preliminaries for Martingale Bounds

To rigorously analyze the behavior of stochastic gradients in non-Euclidean norms, we need tools from the theory of vector-valued martingales. We begin by recalling the necessary probabilistic definitions. Definition 4 (Measurable space). A measurable space is a pair (𝑆, 𝒮), where 𝑆 is a set and 𝒮 is a 𝜎-algebra of subsets of 𝑆, i.e., a collection 𝒮 ⊆ 2𝑆 such that: (i) 𝑆 ∈ 𝒮; (ii) if 𝐴 ∈ 𝒮, then its complement 𝑆 ∖ 𝐴 ∈ 𝒮; (iii) if 𝐴1 , 𝐴2 , · · · ∈ 𝒮, then the countable union

⋃︀∞

𝑛=1 𝐴𝑛 ∈ 𝒮.

Elements of 𝒮 are called measurable sets. Definition 5 (ℱ-measurable random variable). Let (Ω, ℱ, P) be a probability space and let (𝑆, 𝒮) be a measurable space. A map 𝜉 : Ω → 𝑆 is called ℱ-measurable (or simply measurable) if for every measurable set 𝐴 ∈ 𝒮, the preimage 𝜉 −1 (𝐴) := {𝜔 ∈ Ω : 𝜉(𝜔) ∈ 𝐴} belongs to ℱ, i.e.,

∀𝐴 ∈ 𝒮 : 𝜉 −1 (𝐴) ∈ ℱ. In particular, when 𝑆 = R and 𝒮 = ℬ(R) is the Borel 𝜎-algebra, this becomes: ∀𝐵 ∈ ℬ(R) :

{𝜔 ∈ Ω : 𝜉(𝜔) ∈ 𝐵} ∈ ℱ.

Remark 2 (Generated 𝜎-algebra). Let 𝜉 : (Ω, ℱ) → (𝑆, 𝒮) be a measurable map. The 𝜎-algebra generated by 𝜉 is 𝜎(𝜉) := {𝜉 −1 (𝐴) : 𝐴 ∈ 𝒮}. It is the smallest 𝜎-algebra 𝒢 ⊆ ℱ such that 𝜉 is 𝒢-measurable. Definition 6 (Sub-𝜎-algebra). Let (Ω, ℱ, P) be a probability space. A collection of sets 𝒢 is called a sub-𝜎-algebra of ℱ if: (i) 𝒢 ⊆ ℱ (i.e., ∀𝐴 ⊆ Ω : 𝐴 ∈ 𝒢 ⇒ 𝐴 ∈ ℱ), ⋃︀∞ (ii) 𝒢 is a 𝜎-algebra on Ω, i.e., Ω ∈ 𝒢, 𝐴 ∈ 𝒢 ⇒ 𝐴𝑐 ∈ 𝒢, and 𝐴1 , 𝐴2 , · · · ∈ 𝒢 ⇒ 𝑛=1 𝐴𝑛 ∈ 𝒢. Definition 7 (Filtration). Let (Ω, ℱ, P) be a probability space and let 𝑇 be a totally ordered index set (e.g., 𝑇 = N0 or 𝑇 = [0, ∞)). A family of 𝜎-algebras {ℱ𝑡 }𝑡∈𝑇 is called a filtration if: (i) ℱ𝑡 ⊆ ℱ for every 𝑡 ∈ 𝑇 (each ℱ𝑡 is a sub-𝜎-algebra of ℱ); 41

(ii) for all 𝑠, 𝑡 ∈ 𝑇 with 𝑠 ≤ 𝑡, one has ℱ𝑠 ⊆ ℱ𝑡 . The inclusion ℱ𝑠 ⊆ ℱ𝑡 is an inclusion of collections of sets: it means ∀𝐴 ⊆ Ω, 𝐴 ∈ ℱ𝑠 ⇒ 𝐴 ∈ ℱ𝑡 . Generated filtrations (common construction). Given random variables {𝑋𝑢 }𝑢∈𝑇 on (Ω, ℱ, P), one can define the filtration generated by (𝑋𝑢 ) by (︀ )︀ ℱ𝑡 := 𝜎 {𝑋𝑢 : 𝑢 ≤ 𝑡} , i.e., ℱ𝑡 is the smallest 𝜎-algebra with respect to which all 𝑋𝑢 for 𝑢 ≤ 𝑡 are measurable. In particular, if 𝑠 ≤ 𝑡 then {𝑋𝑢 : 𝑢 ≤ 𝑠} ⊆ {𝑋𝑢 : 𝑢 ≤ 𝑡}, and therefore ℱ𝑠 = 𝜎({𝑋𝑢 : 𝑢 ≤ 𝑠}) ⊆ 𝜎({𝑋𝑢 : 𝑢 ≤ 𝑡}) = ℱ𝑡 . We interpret ℱ𝑡 as the information available up to time 𝑡. Definition 8 (Martingale-difference sequence). Let (Ω, ℱ, P) be a probability space, let {ℱ𝑡 }𝑡≥0 be a filtration (ℱ𝑡−1 ⊆ ℱ𝑡 ⊆ ℱ), and let (𝑉, ‖·‖𝑉 ) be a finite-dimensional normed space. A sequence of 𝑉 -valued random variables {𝜉𝑡 }𝑡≥1 is called an {ℱ𝑡 }-martingale-difference sequence if for every 𝑡 ≥ 1: (i) 𝜉𝑡 is ℱ𝑡 -measurable, (ii) E [‖𝜉𝑡 ‖𝑉 ] < ∞ (integrability), (iii) E [𝜉𝑡 | ℱ𝑡−1 ] = 0 almost surely. ∑︀𝑡 In this case, the partial sums 𝑆𝑡 := 𝑖=1 𝜉𝑖 form a 𝑉 -valued martingale with respect to {ℱ𝑡 }. The next lemma converts 𝜅-regularity into a second-moment bound for vector-valued martingale ∑︀𝑇 sums. Intuitively, 𝜅 controls how large E‖𝑆𝑇 ‖2 can be relative to the accumulated 𝑡=1 E‖𝜉𝑡 ‖2 . Lemma 15 (Second-moment bound from 𝜅𝑉 -regularity). Assume that the normed space (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular in the sense of Definition 1. Let {𝜉𝑡 }𝑇𝑡=1 be a 𝑉 -valued {ℱ𝑡 }-martingale-difference sequence such that [︀ ]︀ E ‖𝜉𝑡 ‖2𝑉 < ∞, 𝑡 = 1, . . . , 𝑇. Then, for 𝑆𝑇 :=

𝑇 ∑︁

𝜉𝑡 ,

𝑡=1

one has 𝑇 ∑︁ ]︀ [︀ ]︀ [︀ E ‖𝜉𝑡 ‖2𝑉 . E ‖𝑆𝑇 ‖2𝑉 ≤ 𝜅𝑉

(30)

𝑡=1

Proof. To simplify notation throughout this proof, we write ‖𝑥‖ := ‖𝑥‖𝑉 ,

‖𝑥‖+ := ‖𝑥‖𝑉,+ ,

𝜅 := 𝜅𝑉 ,

𝜅+ := 𝜅𝑉,+ .

By 𝜅-regularity, there exist 𝜅+ ∈ [1, 𝜅] and a 𝜅+ -smooth auxiliary norm ‖ · ‖+ on 𝑉 such that 𝜅 ‖𝑥‖2 ≤ ‖𝑥‖2+ ≤ ‖𝑥‖2 , ∀𝑥 ∈ 𝑉. (31) 𝜅+ Define the auxiliary function 𝜑+ (𝑥) :=

1 ‖𝑥‖2+ . 2

Since (𝑉, ‖ · ‖+ ) is 𝜅+ -smooth, for all 𝑥, ℎ ∈ 𝑉 we have 𝜑+ (𝑥 + ℎ) ≤ 𝜑+ (𝑥) + ⟨∇𝜑+ (𝑥), ℎ⟩ +

42

𝜅+ ‖ℎ‖2+ . 2

(32)

For each 𝑡 = 0, 1, . . . , 𝑇 , let 𝑆𝑡 :=

𝑡 ∑︁

𝜉𝑖 ,

𝑆0 := 0.

𝑖=1

Applying (32) with 𝑥 = 𝑆𝑡−1 and ℎ = 𝜉𝑡 , we get 𝜅+ ‖𝜉𝑡 ‖2+ . 2 [︀ ]︀ Before taking conditional expectations, we verify integrability. By (31), E ‖𝜉𝑖 ‖2+ < ∞ for every 𝑖. The triangle inequality and the scalar Cauchy-Schwarz inequality give 𝜑+ (𝑆𝑡 ) ≤ 𝜑+ (𝑆𝑡−1 ) + ⟨∇𝜑+ (𝑆𝑡−1 ), 𝜉𝑡 ⟩ +

𝑡 ∑︁ ]︀ [︀ ]︀ [︀ 2 E ‖𝑆𝑡 ‖+ ≤ 𝑡 E ‖𝜉𝑖 ‖2+ < ∞,

𝑡 = 1, . . . , 𝑇,

𝑖=1

and 𝑆0 = 0. Let ‖ · ‖+,* denote the dual norm of ‖ · ‖+ . By Lemma 12(iv), applied to 𝜑+ , ‖∇𝜑+ (𝑧)‖+,* = ‖𝑧‖+ ,

𝑧 ∈ 𝑉.

Consequently, the duality inequality and Cauchy-Schwarz yield E [|⟨∇𝜑+ (𝑆𝑡−1 ), 𝜉𝑡 ⟩|] ≤ E [‖𝑆𝑡−1 ‖+ ‖𝜉𝑡 ‖+ ] ]︀)︀1/2 (︀ [︀ ]︀)︀1/2 (︀ [︀ ≤ E ‖𝑆𝑡−1 ‖2+ E ‖𝜉𝑡 ‖2+ < ∞. Thus all terms in the preceding smoothness inequality are integrable. Since ∇𝜑+ is continuous and 𝑆𝑡−1 is ℱ𝑡−1 -measurable, ∇𝜑+ (𝑆𝑡−1 ) is also ℱ𝑡−1 -measurable. We may therefore take conditional expectations and use E [𝜉𝑡 | ℱ𝑡−1 ] = 0 a.s. to obtain E [⟨∇𝜑+ (𝑆𝑡−1 ), 𝜉𝑡 ⟩ | ℱ𝑡−1 ] = ⟨∇𝜑+ (𝑆𝑡−1 ), E [𝜉𝑡 | ℱ𝑡−1 ]⟩ =0 a.s. Therefore, ]︀ 𝜅+ [︀ E ‖𝜉𝑡 ‖2+ | ℱ𝑡−1 . 2 Taking full expectation and summing over 𝑡 = 1, . . . , 𝑇 yields E [𝜑+ (𝑆𝑡 ) | ℱ𝑡−1 ] ≤ 𝜑+ (𝑆𝑡−1 ) +

E [𝜑+ (𝑆𝑇 )] ≤

𝑇 ]︀ 𝜅+ ∑︁ [︀ E ‖𝜉𝑡 ‖2+ . 2 𝑡=1

(33)

Since 𝜑+ (𝑥) = 12 ‖𝑥‖2+ , (33) is equivalent to 𝑇 ∑︁ [︀ ]︀ [︀ ]︀ E ‖𝑆𝑇 ‖2+ ≤ 𝜅+ E ‖𝜉𝑡 ‖2+ . 𝑡=1

Finally, using (31) and then (34), we obtain [︀ ]︀ [︀ ]︀ E ‖𝑆𝑇 ‖2 ≤ E ‖𝑆𝑇 ‖2+ ≤ 𝜅+

𝑇 ∑︁ [︀ ]︀ E ‖𝜉𝑡 ‖2+ 𝑡=1

[︂ ]︂ 𝑇 ∑︁ 𝜅 2 ≤ 𝜅+ E ‖𝜉𝑡 ‖ 𝜅+ 𝑡=1 =𝜅

𝑇 ∑︁ ]︀ [︀ E ‖𝜉𝑡 ‖2 , 𝑡=1

which is exactly (30).

43

(34)

F

Sharp Operator for ℓ1 and ℓ∞ Update Norms

In this section, we specialize the abstract definition of the sharp operator 𝑠♯ ∈ 𝜕𝜑* (𝑠), where 2 𝜑* (𝑠) := 12 ‖𝑠‖* , to the nonsmooth update norms ‖·‖1 and ‖·‖∞ on R𝑑 . Throughout, we identify ∑︀𝑑 𝐸 = R𝑑 and 𝐸 * = R𝑑 via the standard pairing ⟨𝑠, 𝑥⟩ = 𝑖=1 𝑠𝑖 𝑥𝑖 , and we write {𝑒𝑖 }𝑑𝑖=1 for the canonical basis of R𝑑 . F.1

ℓ1 Update Norm: Coordinate-Type Steps

Assume the update norm is ‖·‖ = ‖·‖1 , so the induced dual norm is ‖·‖* = ‖·‖∞ . Then the squared 2 dual-norm potential is 𝜑* (𝑠) = 21 ‖𝑠‖∞ . By Lemma 10, for every 𝑠 ∈ R𝑑 , 𝜕𝜑* (𝑠) = ‖𝑠‖∞ · 𝜕 ‖𝑠‖∞ .

(35)

An explicit description of 𝜕 ‖𝑠‖∞ (and hence of 𝜕𝜑* (𝑠)) follows from a standard result in convex analysis; for completeness, we state it below. Proposition 1 (Subdifferential of ‖ · ‖∞ ). For 𝑠 ∈ R𝑑 , define the active set 𝐼(𝑠) := {𝑖 ∈ [𝑑] : |𝑠𝑖 | = ‖𝑠‖∞ } . If 𝑠 = 0, then 𝜕 ‖𝑠‖∞ = {𝑥 ∈ R𝑑 : ‖𝑥‖1 ≤ 1}. If 𝑠 ̸= 0, then 𝜕 ‖𝑠‖∞ = conv {sign(𝑠𝑖 ) 𝑒𝑖 : 𝑖 ∈ 𝐼(𝑠)} ⎧ ⎫ ⎨ ∑︁ ⎬ ∑︁ = 𝛼𝑖 sign(𝑠𝑖 ) 𝑒𝑖 : 𝛼𝑖 ≥ 0, 𝛼𝑖 = 1 . ⎩ ⎭ 𝑖∈𝐼(𝑠)

(36)

𝑖∈𝐼(𝑠)

Proof. See Beck [2017, Example 3.52]. Corollary 2 (Sharp operator under ‖ · ‖1 ). Assume ‖·‖ = ‖·‖1 and ‖·‖* = ‖·‖∞ . For 𝑠 ∈ R𝑑 with active set 𝐼(𝑠) as above, 𝜕𝜑* (𝑠) = ‖𝑠‖∞ 𝜕 ‖𝑠‖∞ = ‖𝑠‖∞ · conv {sign(𝑠𝑖 ) 𝑒𝑖 : 𝑖 ∈ 𝐼(𝑠)} .

(37)

In particular, if 𝑠 ̸= 0, then every selection 𝑠♯ ∈ 𝜕𝜑* (𝑠) satisfies: (i) 𝑠♯𝑖 = 0 for all 𝑖 ∈ / 𝐼(𝑠) (support contained in the maximizers of |𝑠𝑖 |), ⃦ ♯⃦ (ii) ⃦𝑠 ⃦1 = ‖𝑠‖∞ , (iii) 𝑠♯𝑖 𝑠𝑖 ≥ 0 for all 𝑖 (sign alignment on the active set). Moreover, any extreme-point selection has the coordinate form 𝑠♯ = ‖𝑠‖∞ sign(𝑠𝑖⋆ ) 𝑒𝑖⋆ ,

𝑖⋆ ∈ 𝐼(𝑠),

(38)

and yields a single-coordinate update when substituted into (7) (with 𝑠 = 𝑔 𝑘 ). Proof. The identity (37) follows from (35) and Proposition 1. Properties (i)–(iii) follow immediately from the convex-hull representation in (37). Finally, the extreme points of the simplex {𝛼 ≥ 0 : ∑︀ ⋆ 𝑖∈𝐼(𝑠) 𝛼𝑖 = 1} correspond to placing all mass on one index 𝑖 ∈ 𝐼(𝑠), which gives (38). F.2

ℓ∞ Update Norm: Sign-Type Steps

Assume the update norm is ‖·‖ = ‖·‖∞ , so the induced dual norm is ‖·‖* = ‖·‖1 . Then 𝜑* (𝑠) = 2 1 2 ‖𝑠‖1 and Lemma 10 yields 𝜕𝜑* (𝑠) = ‖𝑠‖1 · 𝜕 ‖𝑠‖1 . Similarly, one can characterize 𝜕 ‖𝑠‖1 explicitly. 44

(39)

Proposition 2 (Subdifferential of ‖·‖1 ). Let 𝑠 ∈ R𝑑 . If 𝑠 = 0, then 𝜕 ‖𝑠‖1 = {𝑥 ∈ R𝑑 : ‖𝑥‖∞ ≤ 1}. If 𝑠 ̸= 0, then {︀ }︀ 𝜕 ‖𝑠‖1 = 𝑥 ∈ R𝑑 : 𝑥𝑖 = sign(𝑠𝑖 ) for all 𝑖 with 𝑠𝑖 ̸= 0, |𝑥𝑖 | ≤ 1 for all 𝑖 with 𝑠𝑖 = 0 . (40) Equivalently, 𝜕 ‖𝑠‖1 consists of the componentwise sign vectors of 𝑠, with an arbitrary choice in [−1, 1] on coordinates where 𝑠𝑖 = 0. Proof. See Beck [2017, Example 3.41]. Corollary 3 (Sharp operator under ‖ · ‖∞ ). Assume ‖·‖ = ‖·‖∞ and ‖·‖* = ‖·‖1 . Then for any 𝑠 ∈ R𝑑 , 𝜕𝜑* (𝑠) = ‖𝑠‖1 · 𝜕 ‖𝑠‖1 .

(41)

In particular, if 𝑠 ̸= 0 then any selection 𝑠♯ ∈ 𝜕𝜑* (𝑠) satisfies 𝑠♯ = ‖𝑠‖1 · 𝑣,

𝑣𝑖 = sign(𝑠𝑖 ) for 𝑠𝑖 ̸= 0, |𝑣𝑖 | ≤ 1 for 𝑠𝑖 = 0.

(42)

A canonical selection is 𝑣 = sign(𝑠), where sign is applied componentwise with sign(0) = 0. This yields the sign-type update ⃦ ⃦ 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 ⃦𝑔 𝑘 ⃦1 sign(𝑔 𝑘 ), closely related to signSGD-style directions [Bernstein et al., 2018, Sun et al., 2023]. Proof. This follows immediately from (39) and Proposition 2.

45

G

Derivation of the Non-Euclidean SGD Update

In this section, we provide a mathematical justification for the non-Euclidean SGD update rule used in the main text: (︀ )︀♯ 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 𝑔 𝑘 . (43) We show that the steepest-descent and Linear Minimization Oracle (LMO) formulations characterize the same set of admissible steps, from which the fixed sharp selection determines the update. G.1

Derivation from Steepest Descent

The steepest descent method generates the next iterate 𝑥𝑘+1 = 𝑥𝑘 + ℎ𝑘 by minimizing a linearized model regularized by the squared norm. In minibatch SGD, the linear term uses the stochastic gradient 𝑔 𝑘 , and the stepsize 𝛾𝑘 sets the regularization weight 1/(2𝛾𝑘 ). The step ℎ𝑘 is a solution to the following optimization problem: {︂ }︂ ⟨︀ ⟩︀ 1 2 ℎ𝑘 ∈ arg min 𝑔 𝑘 , ℎ + ‖ℎ‖ , (44) 2𝛾𝑘 ℎ∈𝐸 where 𝛾𝑘 > 0 and ‖·‖ is an arbitrary norm on the space 𝐸. The objective function in (44) is convex, and its first-order optimality condition states that the zero vector must be in the subdifferential of the 2 objective at ℎ𝑘 . Letting 𝜑(ℎ) := 12 ‖ℎ‖ , this condition is: 0 ∈ 𝑔𝑘 +

1 𝜕𝜑(ℎ𝑘 ). 𝛾𝑘

Rearranging this expression, we get: −𝛾𝑘 𝑔 𝑘 ∈ 𝜕𝜑(ℎ𝑘 ). This is a condition on the subdifferential of the primal squared norm 𝜑. We can invert this relationship using the Fenchel-Moreau-Rockafellar theorem (Theorem 4), which states that 𝑠 ∈ 𝜕𝜑(𝑥) ⇔ 𝑥 ∈ 𝜕𝜑* (𝑠). Applying this gives: ℎ𝑘 ∈ 𝜕𝜑* (−𝛾𝑘 𝑔 𝑘 ). Using the homogeneity of the subdifferential of the squared norm (Lemma 11), which states that 𝜕𝜑* (𝑡𝑠) = 𝑡𝜕𝜑* (𝑠) for all 𝑡 ∈ R, we can factor out the stepsize 𝛾𝑘 : ℎ𝑘 ∈ −𝛾𝑘 𝜕𝜑* (𝑔 𝑘 ). All preceding conditions are equivalent, so the full set of minimizers is −𝛾𝑘 𝜕𝜑* (𝑔 𝑘 ). For the fixed sharp selection from the main text, choose ℎ𝑘 = −𝛾𝑘 (𝑔 𝑘 )♯ ; this is a minimizer because (𝑔 𝑘 )♯ ∈ 𝜕𝜑* (𝑔 𝑘 ). Substituting this choice into 𝑥𝑘+1 = 𝑥𝑘 + ℎ𝑘 gives the specified non-Euclidean SGD update: 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 (𝑔 𝑘 )♯ . G.2

Linear Minimization Oracle (LMO) Formulation

We now show that a Linear Minimization Oracle (LMO) gives the same set of admissible next iterates as the steepest-descent formulation. An LMO is an operator which, for a given set 𝒞 ⊂ 𝐸 and a vector 𝑠 ∈ 𝐸 * , solves the linear problem: lmo(𝑠) := arg min ⟨𝑠, 𝑥⟩ . 𝒞

𝑥∈𝒞

Our derivation will rely on two key properties, which we first establish as lemmas. Lemma 16 (Characterization of the Sharp Operator via LMO). For any non-zero 𝑠 ∈ 𝐸 * , the subdifferential 𝜕𝜑* (𝑠) can be expressed in terms of an LMO over the unit ball ℬ(0, 1) = {𝑥 ∈ 𝐸 | ‖𝑥‖ ≤ 1} as follows: 𝜕𝜑* (𝑠) = − ‖𝑠‖* · lmo (𝑠). ℬ(0,1)

46

Proof. From Equation (29) and the variational characterization of the subdifferential of the norm from Lemma 7, we have for any 𝑠 ̸= 0: 𝜕𝜑* (𝑠) = ‖𝑠‖* · 𝜕 ‖𝑠‖* = ‖𝑠‖* · arg max{⟨𝑠, 𝑥⟩ | ‖𝑥‖ ≤ 1} 𝑥∈𝐸

= − ‖𝑠‖* · arg min{⟨𝑠, 𝑥⟩ | ‖𝑥‖ ≤ 1} 𝑥∈𝐸

≡ − ‖𝑠‖* · lmo (𝑠). ℬ(0,1)

This completes the proof. Lemma 17 (Translation Property of the LMO). Let ℬ(𝑦, 𝑡) = {𝑥 ∈ 𝐸 | ‖𝑥 − 𝑦‖ ≤ 𝑡} be a ball of radius 𝑡 > 0 centered at 𝑦 ∈ 𝐸. The LMO over this ball can be expressed in terms of the LMO over the unit ball ℬ(0, 1) as: lmo (𝑠) = 𝑦 + 𝑡 · lmo (𝑠).

ℬ(𝑦,𝑡)

ℬ(0,1)

Proof. The proof follows from a change of variables. The LMO problem is to find a minimizer 𝑥* for ⟨𝑠, 𝑥⟩ over the set ℬ(𝑦, 𝑡). lmo (𝑠) =

ℬ(𝑦,𝑡)

arg min

⟨𝑠, 𝑥⟩

𝑥∈𝐸:‖𝑥−𝑦‖≤𝑡

= 𝑦 + arg min ⟨𝑠, 𝑦 + 𝑥′ ⟩

(letting 𝑥′ = 𝑥 − 𝑦)

𝑥′ ∈𝐸:‖𝑥′ ‖≤𝑡

= 𝑦 + arg min ⟨𝑠, 𝑥′ ⟩ 𝑥′ ∈𝐸:‖𝑥′ ‖≤𝑡

=𝑦+𝑡·

⟨𝑠, 𝑡𝑥′′ ⟩

arg min

(letting 𝑥′ = 𝑡𝑥′′ )

𝑥′′ ∈𝐸:‖𝑥′′ ‖≤1

=𝑦+𝑡·

⟨𝑠, 𝑥′′ ⟩

arg min 𝑥′′ ∈𝐸:‖𝑥′′ ‖≤1

= 𝑦 + 𝑡 · lmo (𝑠). ℬ(0,1)

This completes the derivation. ⃦ ⃦ For 𝑔 𝑘 ̸= 0, define 𝑡𝑘 := 𝛾𝑘 ⃦𝑔 𝑘 ⃦* > 0. Applying Lemma 17 and then Lemma 16 gives the equality of solution sets: lmo (𝑔 𝑘 ) = 𝑥𝑘 + 𝑡𝑘 · lmo (𝑔 𝑘 ) ℬ(0,1)

ℬ(𝑥𝑘 ,𝑡𝑘 )

= 𝑥 − 𝛾𝑘 𝜕𝜑* (𝑔 𝑘 ). 𝑘

The right-hand side is exactly the set of next iterates obtained from all steepest-descent minimizers. Thus the LMO formulation is ⃦ ⃦ 𝑥𝑘+1 ∈ lmo (𝑔 𝑘 ), where 𝑡𝑘 = 𝛾𝑘 ⃦𝑔 𝑘 ⃦* . (45) ℬ(𝑥𝑘 ,𝑡𝑘 )

To reproduce the algorithm with the fixed sharp selection, choose the LMO minimizer 𝑥𝑘+1 = 𝑥𝑘 − 𝛾𝑘 (𝑔 𝑘 )♯ . The equality of solution sets ensures that this choice is valid; an arbitrary different minimizer may produce a different admissible step. The radius of the ball, 𝑡𝑘 , depends on the stepsize and the dual norm of the stochastic gradient. We note that if 𝑔 𝑘 = 0, then 𝑡𝑘 = 0 and both formulations reduce to the identity update 𝑥𝑘+1 = 𝑥𝑘 : in this case the ball ℬ(𝑥𝑘 , 0) is the singleton {𝑥𝑘 }, and the LMO formulation still coincides with the steepest-descent solution.

47

H

Derivation of the Orthogonalized Gradient Update

In this section, we derive the spectral-norm update (7) for the canonical sharp selection and characterize the full family of updates allowed by the LMO formulation. H.1

The Non-Euclidean Update with the Spectral Norm

The non-Euclidean SGD update is given by X𝑘+1 = X𝑘 − 𝛾𝑘 (G𝑘 )♯ , where (G𝑘 )♯ ∈ 𝜕𝜑* (G𝑘 ). In our setting, the primal space is the space of matrices equipped with the spectral norm, ‖·‖ = ‖·‖op . The corresponding dual norm is the nuclear norm, ‖·‖* = ‖·‖nuc . The function 𝜑* is therefore the squared nuclear norm: 𝜑* (G) =

1 2 ‖G‖nuc . 2

To characterize the sharp operator (G𝑘 )♯ , we must find the subdifferential of 𝜑* . By Lemma 10, we have 𝜕𝜑* (G) = ‖G‖nuc · 𝜕 ‖G‖nuc . We therefore first characterize the subdifferential of the nuclear norm. Lemma 18 (Subdifferential of the Nuclear Norm). Let G ∈ R𝑚×𝑛 have thin singular value decomposition G = U𝑟 Σ𝑟 V𝑟⊤ , where 𝑟 = rank(G), U𝑟 ∈ R𝑚×𝑟 , V𝑟 ∈ R𝑛×𝑟 , and Σ𝑟 = diag(𝜎1 , . . . , 𝜎𝑟 ) ∈ R𝑟×𝑟 with 𝜎𝑖 > 0. The subdifferential of the nuclear norm ‖·‖nuc at G is given by {︁ }︁ 𝜕 ‖G‖nuc = U𝑟 V𝑟⊤ + W | U⊤ W = 0, WV = 0, and ‖W‖ ≤ 1 . 𝑟 𝑟 op A canonical and valid selection for an element of the subdifferential is the partial isometry U𝑟 V𝑟⊤ . Proof. We will derive this result by applying the characterization of the subdifferential of the dual norm from Lemma 7. We apply the general norm characterization with the following identification: • The primal norm is the spectral norm: ‖·‖ ≡ ‖·‖op . • The corresponding dual norm is the nuclear norm: ‖·‖* ≡ ‖·‖nuc . We consider the two cases from (21). Case 1: G = 0. According to (21), the subdifferential of the dual norm at the origin is the unit ball of the primal norm. In our case, this is the unit ball of the spectral norm: 𝜕 ‖0‖nuc = {S ∈ R𝑚×𝑛 | ‖S‖op ≤ 1}. This matches the general formula: when G = 0, we have 𝑟 = 0, and the empty factors U0 ∈ R𝑚×0 and V0 ∈ R𝑛×0 satisfy U0 V0⊤ = 0. The constraints U⊤ 0 W = 0 and WV0 = 0 are then vacuous, so the formula reduces to {W ∈ R𝑚×𝑛 | ‖W‖op ≤ 1}. Case 2: G ̸= 0. According to (21), an element S ∈ R𝑚×𝑛 belongs to the subdifferential 𝜕 ‖G‖nuc if and only if it satisfies two conditions: 1. ⟨S, G⟩ = ‖G‖nuc 2. ‖S‖op = 1 Let 𝑢𝑖 and 𝑣𝑖 be the columns of the fixed thin SVD factors U𝑟 and V𝑟 . Suppose first that S ∈ 𝜕 ‖G‖nuc . The two conditions above give 𝑟 ∑︁

𝜎𝑖 𝑢⊤ 𝑖 S𝑣𝑖 = ⟨S, G⟩F = ‖G‖nuc =

𝑖=1

𝑟 ∑︁ 𝑖=1

48

𝜎𝑖 .

⊤ ⊤ Each ⃦ ⊤ ⃦𝑢𝑖 S𝑣𝑖 ≤ ‖S𝑣𝑖 ‖2 ≤ 1, and all 𝜎𝑖 are positive, so 𝑢𝑖 S𝑣𝑖 = 1 for every 𝑖 ≤ 𝑟. Since ⃦S ⃦ = ‖S‖ = 1, equality in the Euclidean Cauchy–Schwarz inequality, applied to S𝑣𝑖 and to op op S⊤ 𝑢𝑖 , yields

S⊤ U𝑟 = V𝑟 .

SV𝑟 = U𝑟 ,

Thus W := S − U𝑟 V𝑟⊤ satisfies U⊤ 𝑟 W = 0 and WV𝑟 = 0. Moreover, ⊤ W = (I𝑚 − U𝑟 U⊤ 𝑟 )S(I𝑛 − V𝑟 V𝑟 ),

so ‖W‖op ≤ ‖S‖op = 1, because both orthogonal projections have spectral norm at most one. This proves that every subgradient has the stated form. Conversely, let W satisfy the three conditions in the lemma and set S = U𝑟 V𝑟⊤ + W. For any 𝑧 ∈ R𝑛 , write 𝑧 = V𝑟 𝑎+𝑏 with V𝑟⊤ 𝑏 = 0. Then S𝑧 = U𝑟 𝑎+W𝑏, and the two terms are orthogonal because U⊤ 𝑟 W = 0. Hence ‖S𝑧‖22 = ‖𝑎‖22 + ‖W𝑏‖22 ≤ ‖𝑎‖22 + ‖𝑏‖22 = ‖𝑧‖22 . Therefore ‖S‖op ≤ 1. Since 𝑟 ≥ 1 and S𝑣𝑖 = 𝑢𝑖 for every 𝑖 ≤ 𝑟, we also have ‖S‖op ≥ 1 and ⟨S, G⟩F =

𝑟 ∑︁

𝜎𝑖 𝑢⊤ 𝑖 S𝑣𝑖 =

𝑟 ∑︁

𝑖=1

𝜎𝑖 = ‖G‖nuc .

𝑖=1

Both conditions from (21) hold, so S ∈ 𝜕 ‖G‖nuc . This proves the reverse inclusion. Choosing W = 0 yields the canonical element S = U𝑟 V𝑟⊤ . With this result, we can characterize the sharp operator at iteration 𝑘. Let G𝑘 have thin singular value decomposition G𝑘 = U𝑘𝑟 Σ𝑘𝑟 (V𝑟𝑘 )⊤ , where 𝑟 = rank(G𝑘 ). By Lemma 18, a canonical choice of subgradient of the nuclear norm is the partial isometry ⃦ ⃦ O𝑘 := U𝑘𝑟 (V𝑟𝑘 )⊤ ∈ 𝜕 ⃦G𝑘 ⃦nuc . Using the chain rule for subdifferentials, we have ⃦ ⃦ ⃦ ⃦ (G𝑘 )♯ ∈ 𝜕𝜑* (G𝑘 ) = ⃦G𝑘 ⃦nuc · 𝜕 ⃦G𝑘 ⃦nuc , and we select ⃦ ⃦ (G𝑘 )♯ := ⃦G𝑘 ⃦nuc · O𝑘 . Substituting this back into the SGD update rule gives ⃦ ⃦ X𝑘+1 = X𝑘 − 𝛾𝑘 ⃦G𝑘 ⃦nuc O𝑘 . 𝑘 For G𝑘 ̸= ⃦ 0, ⃦the direction O has spectral norm one. If we define a state-dependent stepsize 𝜂𝑘 := 𝛾𝑘 ⃦G𝑘 ⃦nuc , the update becomes

X𝑘+1 = X𝑘 − 𝜂𝑘 O𝑘 . This is the canonical orthogonalized update used in Section 3.1. H.2

Equivalence of Orthogonalization Definitions

We now prove that the partial isometry U𝑟 V𝑟⊤ obtained from the thin SVD agrees with the explicit formula given in (10). Lemma 19. Let G ∈ R𝑚×𝑛 have thin singular value decomposition G = U𝑟 Σ𝑟 V𝑟⊤ , where 𝑟 = rank(G), U𝑟 ∈ R𝑚×𝑟 , V𝑟 ∈ R𝑛×𝑟 , and Σ𝑟 = diag(𝜎1 , . . . , 𝜎𝑟 ) ∈ R𝑟×𝑟 with 𝜎𝑖 > 0. Then the orthogonalization function yields the partial isometry: orth(G) = (GG⊤ )†/2 G = U𝑟 V𝑟⊤ . 49

Proof. If G = 0, both sides are zero, with the empty product U0 V0⊤ interpreted as the zero matrix. Assume henceforth that G ̸= 0. Using the thin SVD, we have GG⊤ = (U𝑟 Σ𝑟 V𝑟⊤ )(U𝑟 Σ𝑟 V𝑟⊤ )⊤ ⊤ 2 ⊤ = (U𝑟 Σ𝑟 V𝑟⊤ )(V𝑟 Σ⊤ 𝑟 U𝑟 ) = U𝑟 Σ𝑟 U𝑟 ,

where Σ2𝑟 = diag(𝜎12 , . . . , 𝜎𝑟2 ). The Moore–Penrose pseudoinverse is then ⊤ (GG⊤ )† = U𝑟 Σ−2 𝑟 U𝑟 ,

where Σ−2 = diag(𝜎1−2 , . . . , 𝜎𝑟−2 ). Since (GG⊤ )† is positive semidefinite, its unique positive 𝑟 semidefinite square root is ⊤ (GG⊤ )†/2 = U𝑟 Σ−1 𝑟 U𝑟 , −1 −1 with Σ−1 𝑟 = diag(𝜎1 , . . . , 𝜎𝑟 ).

Multiplying by G gives orth(G) = (GG⊤ )†/2 G ⊤ ⊤ = U𝑟 Σ−1 𝑟 U𝑟 (U𝑟 Σ𝑟 V𝑟 ) ⊤ ⊤ = U𝑟 Σ−1 𝑟 (U𝑟 U𝑟 )Σ𝑟 V𝑟 ⊤ ⊤ = U𝑟 (Σ−1 𝑟 Σ𝑟 )V𝑟 = U𝑟 V𝑟 ,

since Σ−1 𝑟 Σ𝑟 = I𝑟 . This proves the claim. H.3

LMO and the Canonical Orthogonalized Update for the Spectral Norm

We characterize all solutions of the spectral-norm LMO and recover the canonical orthogonalized update by selecting the negative partial isometry. The LMO formulation of the update is: X𝑘+1 ∈

lmo (G𝑘 ),

ℬ(X𝑘 ,𝑡𝑘 )

where the geometry is defined by the spectral norm, ‖·‖ = ‖·‖op . Consequently, the dual norm is the ⃦ ⃦ nuclear norm, ‖·‖* = ‖·‖nuc , and the radius of the ball is 𝑡𝑘 = 𝛾𝑘 ⃦G𝑘 ⃦nuc with 𝛾𝑘 > 0. We first consider G𝑘 ̸= 0, so that 𝑡𝑘 > 0. The LMO problem is to solve: {︁⟨︀ }︁ ⃦ ⃦ ⟩︀ X𝑘+1 ∈ arg min G𝑘 , X F s.t. ⃦X − X𝑘 ⃦op ≤ 𝑡𝑘 . X∈R𝑚×𝑛

Using the translation property of the LMO (Lemma 17), we can simplify this problem: X𝑘+1 ∈ X𝑘 + 𝑡𝑘 · lmo (G𝑘 ), 𝐵(0,1)

where 𝐵(0, 1) is the unit ball of the spectral norm, i.e., 𝐵(0, 1) = {X ∈ R𝑚×𝑛 | ‖X‖op ≤ 1}. The following lemma describes the full set of minimizers over this ball. Lemma 20 (LMO over the Spectral Norm Ball). Let G ∈ R𝑚×𝑛 be non-zero with thin singular value decomposition G = U𝑟 Σ𝑟 V𝑟⊤ , where 𝑟 = rank(G) and Σ𝑟 = diag(𝜎1 , . . . , 𝜎𝑟 ) has positive diagonal entries. Then lmo (G) = arg min ⟨G, X⟩F

𝐵(0,1)

‖X‖op ≤1

{︁ = −U𝑟 V𝑟⊤ + W : U⊤ 𝑟 W = 0,

WV𝑟 = 0,

}︁ ‖W‖op ≤ 1 ,

where W ∈ R𝑚×𝑛 . In particular, −U𝑟 V𝑟⊤ = − orth(G) is a canonical minimizer. The minimizer is unique if and only if 𝑟 = min{𝑚, 𝑛}. For G = 0, the LMO solution set is the entire ball 𝐵(0, 1). 50

Proof. Let 𝑢𝑖 and 𝑣𝑖 be the columns of U𝑟 and V𝑟 . For every feasible X, the bound ‖X‖op ≤ 1 gives 𝑢⊤ 𝑖 X𝑣𝑖 ≥ −1. Hence ⟨G, X⟩F =

𝑟 ∑︁

𝜎𝑖 𝑢⊤ 𝑖 X𝑣𝑖 ≥ −

𝑖=1

𝑟 ∑︁

𝜎𝑖 = − ‖G‖nuc .

𝑖=1

The matrix X0 = −U𝑟 V𝑟⊤ is feasible and attains this bound. ⊤ If X is any minimizer, positivity ⃦ ⊤ ⃦of all 𝜎𝑖 implies 𝑢𝑖 X𝑣𝑖 = −1 for every 𝑖 ≤ 𝑟. Since 𝑢𝑖 and 𝑣𝑖 are ⃦ ⃦ unit vectors and ‖X‖op = X op ≤ 1, equality in the Euclidean Cauchy–Schwarz inequality gives

X⊤ U𝑟 = −V𝑟 .

XV𝑟 = −U𝑟 ,

Therefore W := X + U𝑟 V𝑟⊤ satisfies U⊤ 𝑟 W = 0 and WV𝑟 = 0. Moreover, ⊤ W = (I𝑚 − U𝑟 U⊤ 𝑟 )X(I𝑛 − V𝑟 V𝑟 ),

so ‖W‖op ≤ ‖X‖op ≤ 1 because both orthogonal projections have spectral norm at most one. Conversely, let W satisfy the three stated conditions and set X = −U𝑟 V𝑟⊤ +W. For any 𝑧 = V𝑟 𝑎+𝑏 with V𝑟⊤ 𝑏 = 0, the vectors U𝑟 𝑎 and W𝑏 are orthogonal, so ‖X𝑧‖22 = ‖ − U𝑟 𝑎 + W𝑏‖22 = ‖𝑎‖22 + ‖W𝑏‖22 ≤ ‖𝑎‖22 + ‖𝑏‖22 = ‖𝑧‖22 . Thus ‖X‖op ≤ 1, and X𝑣𝑖 = −𝑢𝑖 implies ⟨G, X⟩F = − ‖G‖nuc . This proves the full characterization. The identity U𝑟 V𝑟⊤ = orth(G) follows from Lemma 19. If 𝑟 = min{𝑚, 𝑛}, one of the two orthogonal complements is trivial, forcing W = 0. If 𝑟 < ⊤ ⊤ min{𝑚, 𝑛}, choose unit vectors 𝑢 and 𝑣 with U⊤ ̸= 0 is 𝑟 𝑢 = 0 and V𝑟 𝑣 = 0. Then W = 𝑢𝑣 admissible, so the minimizer is not unique. Finally, if G = 0, the objective is identically zero on 𝐵(0, 1). For G𝑘 ̸= 0, Lemma 20 shows that the general LMO update has the form X𝑘+1 = X𝑘 − 𝑡𝑘 orth(G𝑘 ) + 𝑡𝑘 W𝑘 , where 𝑟 = rank(G𝑘 ) and W𝑘 satisfies (U𝑘𝑟 )⊤ W𝑘 = 0,

W𝑘 V𝑟𝑘 = 0,

⃦ 𝑘⃦ ⃦W ⃦ ≤ 1. op

By Lemma 16, this family is exactly the family of sharp updates, with corresponding selections ⃦ ⃦ (︀ )︀ (G𝑘 )♯ = ⃦G𝑘 ⃦nuc orth(G𝑘 ) − W𝑘 . Selecting W𝑘 = 0 gives the canonical update X𝑘+1 = X𝑘 − 𝑡𝑘 orth(G𝑘 ) ⃦ ⃦ = X𝑘 − 𝛾𝑘 ⃦G𝑘 ⃦ orth(G𝑘 ), nuc

which agrees with the canonical sharp selection ⃦ used ⃦ in the steepest-descent derivation and in (9). With the state-dependent stepsize 𝜂𝑘 := 𝛾𝑘 ⃦G𝑘 ⃦nuc and direction O𝑘 := orth(G𝑘 ), it takes the form X𝑘+1 = X𝑘 − 𝜂𝑘 O𝑘 . Other admissible LMO selections can produce different updates when rank(G𝑘 ) < min{𝑚, 𝑛}. When G𝑘 = 0, we have 𝑡𝑘 = 0, 𝜕𝜑* (0) = {0}, and orth(0) = 0 by (10). The translated ball is then ℬ(X𝑘 , 0) = {X𝑘 }, so both the sharp and LMO formulations reduce to the identity update X𝑘+1 = X𝑘 .

51

I

Regularity vs. Norm Equivalence: Why Not Just Use Euclidean Smoothness?

To connect non-Euclidean geometries with the standard Euclidean setting, existing analyses (e.g., [Kovalev, 2025, Gruntkowska et al., 2025]) often proceed via norm equivalence. Concretely, when the geometry is expressed in terms of a dual norm ‖·‖* , one compares it to a Euclidean reference norm (Frobenius for matrices, ℓ2 for vectors). In the matrix case, there exist constants 𝛼, 𝛽 > 0 (typically dimension- and/or rank-dependent) such that for all 𝑋 ∈ R𝑚×𝑛 , 𝛼 ‖𝑋‖* ≤ ‖𝑋‖F ≤ 𝛽 ‖𝑋‖* .

(46) 𝑑

Analogously, in the vector case there exist 𝛼, 𝛽 > 0 such that for all 𝑥 ∈ R , 𝛼 ‖𝑥‖* ≤ ‖𝑥‖2 ≤ 𝛽 ‖𝑥‖* .

(47)

These inequalities allow one to translate smoothness and noise assumptions stated in Euclidean/Frobenius norms into the target geometry. However, since 𝛼 and 𝛽 typically scale with the ambient dimension (or matrix dimension/rank), such conversions can introduce large (often polynomial) inflation factors and thus obscure the true benefits of the chosen geometry. For an 𝐸 * -valued noise second moment, only the left inequality in (47) [︁ vector ]︁ 𝜁 with finite [︁ Euclidean ]︁ 2 2 −2 is needed to obtain E ‖𝜁‖* ≤ 𝛼 E ‖𝜁‖2 . Thus the noise second-moment conversion uses 𝛼−2 ; the Frobenius comparison in (46) gives the same factor for matrices. The 𝜅-regularity framework allows an auxiliary norm that need not be Euclidean. As in Definition 1, this norm is 𝜅𝑉,+ -smooth, meaning that its squared-norm function 12 ‖ · ‖2𝑉,+ is continuously differentiable and satisfies the stated smoothness inequality. It also satisfies the comparison (12): 𝜅𝑉 ‖𝑢‖2𝑉 ≤ ‖𝑢‖2𝑉,+ ≤ 𝐶𝑉 ‖𝑢‖2𝑉 , 𝐶𝑉 = , 𝑢 ∈ 𝑉, 𝜅𝑉,+ where 1 ≤ 𝜅𝑉,+ ≤ 𝜅𝑉 . Smoothness supports the analysis in the auxiliary norm, while the comparison transfers the resulting bounds to the original norm. Choosing the auxiliary norm therefore requires balancing its smoothness parameter 𝜅𝑉,+ against the factor 𝐶𝑉 comparing the squared norms. To obtain a small regularity parameter, we minimize their product 𝜅𝑉 = 𝜅𝑉,+ 𝐶𝑉 , rather than either factor alone. The two propositions below illustrate this choice for ℓ𝑝 and Schatten-𝑝 spaces. Which space carries the norm. As in Definition 1, Proposition 10, Proposition 12 and Proposition 14, the results of this section are stated for a generic finite-dimensional normed space (𝑉, ‖ · ‖𝑉 ), with constant of regularity 𝜅𝑉 , auxiliary norm ‖ · ‖𝑉,+ and smoothness parameter 𝜅𝑉,+ . Following Remark 1, we apply them with 𝑉 = 𝐸 * and ‖ · ‖𝑉 = ‖·‖* , in which case 𝜅𝑉 , ‖ · ‖𝑉,+ and 𝜅𝑉,+ become 𝜅, ‖·‖*,+ and 𝜅+ , and the generic compatibility chain (12) specializes to (13). It is therefore the regularity constant of the dual space that enters our bounds, since that is where stochastic gradients live and where the noise is measured; an ℓ𝑝 or Schatten-𝑝 norm appearing below is a norm on 𝑉 = 𝐸 * , that is, the norm induced on 𝐸 * by the update norm on 𝐸. The Euclidean proxy and the role of the auxiliary norm. Constructing a Euclidean proxy uses 2 2 both sides of (47): the scaled norm ‖·‖*,+ := 𝛼−1 ‖·‖2 is 1-smooth and satisfies ‖𝑠‖* ≤ ‖𝑠‖*,+ ≤ 2 (𝛽/𝛼)2 ‖𝑠‖* . It therefore gives the admissible regularity parameter 𝜅 = (𝛽/𝛼)2 , distinct from the noise-conversion factor 𝛼−2 . The same construction applies to the Frobenius norm. Allowing other auxiliary norms can reduce the product 𝜅 = 𝜅+ 𝐶, where 𝐶 = 𝜅/𝜅+ is the squared-norm comparison factor in (13). The auxiliary norm is used only in the analysis and enters the guarantee through 𝜅; the update (7) is determined by the update norm ‖·‖ and the fixed sharp selection, and never evaluates ‖·‖*,+ . What this means for oracle complexity. When these constructions are applied to 𝑉 = 𝐸 * , their effect on oracle complexity depends on the full statistical coefficients for the chosen noise model. Suppressing logarithmic factors, the relevant comparison is between C* = 𝜅𝜎 2 for the native analysis and CF = 𝛼−2 𝜎F2 for the Euclidean/Frobenius conversion, as in (61). Here 𝜎 and 𝜎F measure the noise of the same oracle in the induced dual norm and the Euclidean/Frobenius norm, respectively. A 52

smaller 𝜅 alone does not guarantee a sharper oracle-complexity bound; any improvement depends on these full coefficients, as illustrated by the noise models in Section J. With this distinction in place, we compare the auxiliary norms for vectors and matrices below, before stating and proving the corresponding propositions. I.1

ℓ𝑑𝑝 Spaces

We first specialize the preceding discussion to (𝐸 * , ‖·‖* ) = (R𝑑 , ‖·‖𝑝 ). Whether a non-Euclidean auxiliary norm improves the regularity bound depends on 𝑝. The regime 1 ≤ 𝑝 < 2. A scaled Euclidean auxiliary norm is already optimal: taking ‖·‖*,+ = 1

1

2

2

2

2

𝑑 𝑝 − 2 ‖·‖2 reduces (13) to the classical ℓ𝑝 –ℓ2 equivalence, ‖𝑠‖2 ≤ ‖𝑠‖𝑝 ≤ 𝑑 𝑝 −1 ‖𝑠‖2 , and gives 2 𝑝 −1

𝜅=𝑑 for every 𝑑 ≥ 1. This parameter cannot be improved by choosing another auxiliary norm; see Remark 5 and case 1 of Proposition 3 below. The regime 2 ≤ 𝑝 ≤ ∞. Here we may use ‖·‖*,+ = ‖·‖𝜌 with 2 ≤ 𝜌 ≤ 𝑝 and 𝜌 < ∞. Its smoothness constant 𝜅+ = 𝜌 − 1 and squared-norm comparison factor 𝑑2/𝜌−2/𝑝 give the regularity bound (𝜌 − 1)𝑑2/𝜌−2/𝑝 . }︀The minimizing choice is 𝜌* := 2 for 1 ≤ 𝑑 < 8 and 𝜌* := {︀ 𝜅(𝜌) = √︀ min 𝑝, ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑 for 𝑑 ≥ 8; in particular, 𝜌* remains finite when 𝑝 = ∞. For 1 ≤ 𝑑 < 8, it gives 𝜅 = 𝑑1−2/𝑝 ≤ 7.18 For 𝑑 ≥ 8, the comparison takes the following form: Euclidean proxy: ‖·‖*,+ = ‖·‖2 ‖𝑠‖2* ≤ ‖𝑠‖22 ≤ 𝑑

2 1− 𝑝

Tuned proxy: ‖·‖*,+ = ‖·‖𝜌*

‖𝑠‖2*

‖𝑠‖2* ≤ ‖𝑠‖2𝜌* ≤ 𝒪(1) ‖𝑠‖2*

2 1− 𝑝

𝜅 = Θ (min{𝑝, ln(𝑒𝑑)})

𝜅=𝑑

Both choices of ‖·‖*,+ are 𝜅+ -smooth and satisfy the required norm-comparison inequalities, so each establishes 𝜅-regularity according to Definition 1, with its corresponding values of 𝜅+ and 𝜅. Since 𝜌 = 2 is allowed in the minimization of 𝜅(𝜌), the tuned regularity bound is no larger than the Euclidean bound 𝑑1−2/𝑝 and can be significantly smaller. √︀The constant comparison factor has a simple explanation: when 𝑝 ≤ 𝜌ˆ(𝑑), where 𝜌ˆ(𝑑) = ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑, we have 𝜌* = 𝑝 and the factor is 1; when 𝑝 > 𝜌ˆ(𝑑), it is at most 𝑒2 . Interpretation. At 𝑝 = 2, both proxies coincide with ‖·‖2 and give 𝜅 = 1. For 2 ≤ 𝑝 ≤ ∞, the tuned bound has the optimal order Θ(min{𝑝, ln(𝑒𝑑)}) in all positive dimensions: case 2 of Proposition 3 establishes the comparison for 𝑑 ≥ 8, and Remark 7 covers 1 ≤ 𝑑 < 8. Thus fixed finite 𝑝 ≥ 2 admits a bound independent of 𝑑, while at 𝑝 = ∞, the dual of the ℓ1 update norm, tuning reduces the regularity parameter from 𝑑 to Θ(ln(𝑒𝑑)). These are comparisons of regularity parameters; the oracle-complexity comparison also depends on the noise scales, as explained above. The following proposition gives the formal statements and their proof in the generic notation (𝑉, ‖ · ‖𝑉 ). Proposition 3 (Regularity vs. Euclidean equivalence in ℓ𝑑𝑝 spaces). Fix 𝑑 ∈ N and 𝑝 ∈ [1, ∞]. Let 𝑉 := R𝑑 and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑝 . Then the following hold. 1. Case 1 ≤ 𝑝 < 2: regularity recovers the classical ℓ𝑝 –ℓ2 equivalence. Take the auxiliary norm 1

1

‖𝑥‖𝑉,+ := 𝑑 𝑝 − 2 ‖𝑥‖2 . Then (𝑉, ‖ · ‖𝑉,+ ) is 1-smooth, so 𝜅𝑉,+ = 1, and the compatibility chain (12) holds with 2

𝜅𝑉 := 𝜅𝑝 (𝑑) = 𝑑 𝑝 −1 . For this choice, (12) is nothing but the standard ℓ𝑝 –ℓ2 norm equivalence: 1

1

‖𝑥‖2 ≤ ‖𝑥‖𝑝 ≤ 𝑑 𝑝 − 2 ‖𝑥‖2

⇐⇒

2

2

2

2

‖𝑥‖2 ≤ ‖𝑥‖𝑝 ≤ 𝑑 𝑝 −1 ‖𝑥‖2 .

(48)

Moreover, this value of 𝜅𝑉 cannot be improved: the matching lower bound is (106) in Proposition 13. 18 Indeed, for 𝜌 ≥ 2, we have (𝜌 − 1)/𝜌2 ≤ 1/4, so 𝜅′ (𝜌) = 𝑑2/𝜌−2/𝑝 1 − 2(ln 𝑑)(𝜌 − 1)/𝜌2

[︀

1 ln 𝑑) > 0 when 1 ≤ 𝑑 < 8. 2

53

]︀

≥ 𝑑2/𝜌−2/𝑝 (1 −

2. Case 2 ≤ 𝑝 ≤ ∞: comparison with the Euclidean proxy. For any 𝜌 ∈ [2, 𝑝] with 𝜌 < ∞, take the auxiliary norm ‖·‖𝑉,+ := ‖·‖𝜌 . Then (𝑉, ‖·‖𝜌 ) is (𝜌−1)-smooth, so 𝜅𝑉,+ = 𝜌−1, and (12) holds with compatibility factor and admissible regularity parameter 2 2 2 2 𝜅𝑉 and 𝜅𝑉 (𝜌) = (𝜌 − 1) 𝑑 𝜌 − 𝑝 , = 𝑑𝜌−𝑝 𝜅𝑉,+ respectively. Optimizing over 𝜌 ∈ [2, 𝑝] with 𝜌 < ∞ yields the best regularity parameter obtainable this way, 2

2

𝜅𝑉 := 𝜅𝑝 (𝑑) = min (𝜌 − 1) 𝑑 𝜌 − 𝑝 .

(49)

2≤𝜌≤𝑝 𝜌<∞

When 𝑝 = ∞, we use 1/𝑝 = 0 and optimize over 𝜌 ∈ [2, ∞). Since 𝜌 = 2 is admissible, 𝜅𝑝 (𝑑) ≤ 𝑑1−2/𝑝 , the parameter supplied by the Euclidean proxy. At 𝑝 = 2, both choices coincide with ‖·‖2 and give 𝜅𝑝 (𝑑) = 1. Assume in addition that ln 𝑑 ≥ 2 (e.g., 𝑑 ≥ 8) and set √︀ 𝜌ˆ(𝑑) := ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑, 𝜌* := min{𝑝, 𝜌ˆ(𝑑)}. 2

The Euclidean choice 𝜌 = 2 produces the classical compatibility factor 𝑑1− 𝑝 , whereas 2 2 the tuned choice 𝜌 = 𝜌* produces a factor 𝑑 𝜌* − 𝑝 that is never larger and is in fact dimension-free in both regimes 𝑝 ≤ 𝜌ˆ(𝑑) and 𝑝 > 𝜌ˆ(𝑑): 2

2

2

‖𝑥‖𝑝 ≤ ‖𝑥‖𝜌* ≤ 𝒪(1) ‖𝑥‖𝑝 ∀𝑥 ∈ R𝑑 . (50) (︀ )︀ Finally, the scaling 𝜅𝑝 (𝑑) = Θ min{𝑝, ln(𝑒𝑑)} is unavoidable for 𝑑 ≥ 8: the matching lower bound is (107) in Proposition 13. Proof. Both cases instantiate Definition 1 on 𝑉 = R𝑑 ; the constructions and the underlying smoothness estimates are those of Proposition 12, and the only new content is the explicit comparison with the Euclidean conversion. Case 1 ≤ 𝑝 < 2. The norm ‖ · ‖𝑉,+ is a scaled Euclidean norm and is therefore 1-smooth, so 1 1 𝜅𝑉,+ = 1. The standard inequalities ‖𝑥‖2 ≤ ‖𝑥‖𝑝 ≤ 𝑑 𝑝 − 2 ‖𝑥‖2 give ‖𝑥‖𝑝 ≤ ‖𝑥‖𝑉,+ and 2

2

2

2

2

‖𝑥‖2𝑉,+ = 𝑑 𝑝 −1 ‖𝑥‖2 ≤ 𝑑 𝑝 −1 ‖𝑥‖𝑝 , which is (12) with 𝜅𝑉 = 𝑑 𝑝 −1 and 𝜅𝑉,+ = 1; squaring the same two inequalities gives (48). The lower bound is (106). Case 2 ≤ 𝑝 ≤ ∞. Fix 2 ≤ 𝜌 ≤ 𝑝 with 𝜌 < ∞. That (R𝑑 , ‖·‖𝜌 ) is (𝜌 − 1)-smooth for 2 ≤ 𝜌 < ∞ is shown in Step 1 of the proof of Proposition 12. The standard ℓ𝑝 –ℓ𝜌 inequalities give ‖𝑥‖𝑝 ≤ 1

1

2

2

2

2

‖𝑥‖𝜌 ≤ 𝑑 𝜌 − 𝑝 ‖𝑥‖𝑝 , and squaring yields (12) with 𝜅𝑉 /𝜅𝑉,+ = 𝑑 𝜌 − 𝑝 , hence 𝜅𝑉 (𝜌) = (𝜌 − 1)𝑑 𝜌 − 𝑝 . Minimizing 𝜅𝑉 (𝜌) over 𝜌 ∈ [2, 𝑝] with 𝜌 < ∞ gives (49). When 𝑝 = ∞, this continuous function satisfies 𝜅𝑉 (𝜌) = (𝜌 − 1)𝑑2/𝜌 → ∞ as 𝜌 → ∞, so the minimum on [2, ∞) is attained at a finite 𝜌. ^ The expression for 𝜌ˆ(𝑑) and the bound 𝑑2/𝜌(𝑑) ≤ 𝑒2 (valid whenever ln 𝑑 ≥ 2) are derived in Step 3 of the proof of Proposition 12; in particular 𝜌ˆ(𝑑) ≥ ln 𝑑 ≥ 2, so 2 ≤ 𝜌* < ∞ even when 𝑝 = ∞, 2 2 2 and the tuned compatibility factor 𝑑 𝜌* − 𝑝 is never larger than the Euclidean one, 𝑑1− 𝑝 . It remains to prove (50), and we distinguish two subcases. Subcase 1: 𝑝 ≤ 𝜌ˆ(𝑑). Then 𝜌* = 𝑝, so ‖𝑥‖𝜌* = ‖𝑥‖𝑝 and (50) holds with constant 1. Subcase 2: 𝑝 > 𝜌ˆ(𝑑). Then 𝜌* = 𝜌ˆ(𝑑) and 2

2

2

2

2

2

2

^ ^ ‖𝑥‖𝜌* ≤ 𝑑 𝜌* − 𝑝 ‖𝑥‖𝑝 = 𝑑2/𝜌(𝑑) 𝑑−2/𝑝 ‖𝑥‖𝑝 ≤ 𝑑2/𝜌(𝑑) ‖𝑥‖𝑝 ≤ 𝑒2 ‖𝑥‖𝑝 ,

which yields (50) with 𝒪(1) = 𝑒2 . The lower bound is (107). I.2

Spaces with Schatten-𝑝 Norms on R𝑚×𝑛

For matrices, we compare the fixed Frobenius proxy with auxiliary Schatten norms. The two 𝑝 regimes follow the vector case, although the smoothness bounds for the auxiliary norms differ. 54

Let 𝑉 := R𝑚×𝑛 and denote 𝑟 := min{𝑚, 𝑛}. For 𝑝 ∈ [1, ∞], the Schatten-𝑝 norm is ‖𝑋‖𝑆𝑝 := ‖𝜎(𝑋)‖𝑝 , where 𝜎(𝑋) ∈ R𝑟 is the vector of singular values of 𝑋 (padded with zeros when rank(𝑋) < 𝑟). Since Schatten norms are ℓ𝑝 norms on 𝜎(𝑋), the standard comparison inequalities (see Lemma 14) give, for 1 ≤ 𝜌 ≤ 𝑝 ≤ ∞, 1

1

‖𝑋‖𝑆𝑝 ≤ ‖𝑋‖𝑆𝜌 ≤ 𝑟 𝜌 − 𝑝 ‖𝑋‖𝑆𝑝 .

(51)

(For 𝑋 ̸= 0, one may replace 𝑟 by rank(𝑋) in (51) for an instance-dependent bound, which can be tighter when rank(𝑋) ≪ 𝑟; the zero case is immediate.) 1

1

The regime 1 ≤ 𝑝 < 2. The scaled Frobenius auxiliary norm ‖ · ‖𝑉,+ = 𝑟 𝑝 − 2 ‖·‖F is 1-smooth and gives 𝜅𝑉 = 𝑟2/𝑝−1 for every 𝑟 ≥ 1. The matching lower bound shows that this parameter is optimal, so allowing non-Frobenius auxiliary norms cannot improve it. The regime 2 ≤ 𝑝 ≤ ∞. The Frobenius proxy is 1-smooth and gives the squared-norm comparison factor and regularity parameter 𝑟1−2/𝑝 . Alternatively, for 2 ≤ 𝜌 ≤ 𝑝 with 𝜌 < ∞, the auxiliary norm ‖ · ‖𝑆𝜌 has comparison factor 𝑟2/𝜌−2/𝑝 and admits the smoothness bound max{2, 𝜌 − 1}. Their product is the regularity bound optimized in the proposition below. For 𝑟 ≥√︀8, the minimizing exponent for this bound is 𝜌* = min{𝑝, max{3, 𝜌ˆ(𝑟)}}, where 𝜌ˆ(𝑟) = ln 𝑟 + (ln 𝑟)2 − 2 ln 𝑟. If 𝜌* = 𝑝, the auxiliary and target norms coincide and the comparison factor is 1; otherwise the factor is at most 𝑒2 . Thus the tuned squared-norm comparison factor is no larger than the Frobenius factor and is bounded independently of 𝑟. Interpretation. The resulting regularity bound has the optimal order Θ(min{𝑝, ln(𝑒𝑟)}) for 𝑟 ≥ 8; Remark 7 extends this order to 1 ≤ 𝑟 < 8. For fixed finite 𝑝 ≥ 2 this order is independent of 𝑟, while at 𝑝 = ∞ it is logarithmic, compared with the Frobenius proxy’s linear parameter 𝑟. At 𝑝 = 2, both auxiliary norms are Frobenius and one can take 𝜅𝑉 = 1: the uniform smoothness estimate max{2, 𝜌 − 1} is not sharp at 𝜌 = 2. As in the vector case, these geometric improvements give a smaller oracle-complexity bound only when the full statistical coefficients favor the native analysis. We now state the precise comparison and prove it. Proposition 4 (Regularity vs. Frobenius equivalence in Schatten classes). Fix 𝑚, 𝑛 ∈ N, let 𝑟 := min{𝑚, 𝑛} and let 𝑝 ∈ [1, ∞]. Let 𝑉 := R𝑚×𝑛 and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑆𝑝 . Then the following hold. 1. Case 1 ≤ 𝑝 < 2: regularity recovers the Schatten-𝑝–Frobenius equivalence. Take the scaled Frobenius auxiliary norm 1

1

‖𝑋‖𝑉,+ := 𝑟 𝑝 − 2 ‖𝑋‖F , which is 1-smooth, so 𝜅𝑉,+ = 1. Then (12) holds with 2

𝜅𝑉 := 𝜅𝑝 (𝑚, 𝑛) = 𝑟 𝑝 −1 , and it is equivalent to the standard Schatten-𝑝–Frobenius norm equivalence: 1

1

‖𝑋‖F ≤ ‖𝑋‖𝑆𝑝 ≤ 𝑟 𝑝 − 2 ‖𝑋‖F

2

2

2

2

‖𝑋‖F ≤ ‖𝑋‖𝑆𝑝 ≤ 𝑟 𝑝 −1 ‖𝑋‖F . (52)

⇐⇒

Moreover, this value of 𝜅𝑉 cannot be improved: the matching lower bound is (108) in Proposition 15. 2. Case 2 ≤ 𝑝 ≤ ∞: comparison with the Frobenius proxy. For any 𝜌 ∈ [2, 𝑝] with 𝜌 < ∞, take the auxiliary norm ‖ · ‖𝑉,+ := ‖ · ‖𝑆𝜌 . Then (51) yields the compatibility chain 2

2

2

2

2

‖𝑋‖𝑆𝑝 ≤ ‖𝑋‖𝑆𝜌 ≤ 𝑟 𝜌 − 𝑝 ‖𝑋‖𝑆𝑝 ,

(53)

and (𝑉, ‖ · ‖𝑆𝜌 ) is 𝜅𝑉,+ -smooth with 𝜅𝑉,+ = max{2, 𝜌 − 1} for 𝜌 ≥ 2 (see Proposition 14 and the references therein), so that (12) holds with 2

2

𝜅𝑉 (𝜌) = max{2, 𝜌 − 1} 𝑟 𝜌 − 𝑝 . Assume now that 𝑟 ≥ 8, so ln 𝑟 ≥ 2.(︀ Under this condition, optimizing over 𝜌 yields a )︀ regularity parameter of order 𝜅𝑉 = 𝒪 min{𝑝, ln(𝑒𝑟)} . More explicitly, set √︀ {︀ }︀ 𝜌ˆ(𝑟) := ln 𝑟 + (ln 𝑟)2 − 2 ln 𝑟, 𝜌* := min 𝑝, max{3, 𝜌ˆ(𝑟)} . 55

2

The Frobenius choice 𝜌 = 2 produces the compatibility factor 𝑟1− 𝑝 , whereas the tuned choice 𝜌 = 𝜌* produces a factor that is no larger and is bounded by an absolute constant independent of the dimension parameter 𝑟 = min{𝑚, 𝑛} in both regimes: 2

2

2

‖𝑋‖𝑆𝑝 ≤ ‖𝑋‖𝑆𝜌* ≤ 𝒪(1) ‖𝑋‖𝑆𝑝

∀𝑋 ∈ R𝑚×𝑛 .

(54)

At 𝑝 = 2, both proxies coincide with the Frobenius norm and the compatibility factor is 1. Its exact smoothness constant is also 1, so 𝜅𝑉 = 1 is admissible in this case; the uniform bound(︀max{2, 𝜌 − 1})︀used above is not sharp at 𝜌 = 2. Finally, the scaling 𝜅𝑝 (𝑚, 𝑛) = Θ min{𝑝, ln(𝑒𝑟)} is unavoidable for 𝑟 ≥ 8: the matching lower bound is (109) in Proposition 15. Proof. As in the vector case, both cases instantiate Definition 1 on 𝑉 = R𝑚×𝑛 , and the constructions and smoothness estimates are those of Proposition 14. Case 1 ≤ 𝑝 < 2. This is the Schatten analogue of case 1 of Proposition 3. Since ‖ · ‖𝑉,+ is a scaled Hilbert norm, it is 1-smooth. Applying (51) with smaller exponent 𝑝 and larger exponent 2 yields 1 1 ‖𝑋‖𝑆2 = ‖𝑋‖F ≤ ‖𝑋‖𝑆𝑝 ≤ 𝑟 𝑝 − 2 ‖𝑋‖F , i.e., (52). The compatibility chain (12) then follows by 2

construction with 𝜅𝑉 = 𝑟 𝑝 −1 and 𝜅𝑉,+ = 1. The lower bound is (108). Case 2 ≤ 𝑝 ≤ ∞. The compatibility chain (53) is exactly (51) squared. The smoothness bound for Schatten-𝜌 spaces with 𝜌 ≥ 2, and the resulting regularity parameter, are stated and proved in Proposition 14 (see also [Juditsky and Nemirovski, 2008]). For the remaining claims, assume 𝑟 ≥ 8. Under this condition, the optimization over 𝜌 is the same calculus as in the ℓ𝑝 case, with 𝑑 replaced by 𝑟, and leads to 𝜅𝑉 = 𝒪(min{𝑝, ln(𝑒𝑟)}). It remains to justify (54), and we distinguish two subcases. Subcase 1: 𝜌* = 𝑝. Then ‖𝑋‖𝑆𝜌 = ‖𝑋‖𝑆𝑝 and (54) holds with constant 1. *

Subcase 2: 𝜌* = max{3, 𝜌ˆ(𝑟)}. Since ln 𝑟 ≥ 2, we have 𝜌ˆ(𝑟) ≥ ln 𝑟 and therefore 𝜌* ≥ ln 𝑟, whence 𝑟2/𝜌* ≤ 𝑟2/ ln 𝑟 = 𝑒2 . Combining this with (53) gives 2

2

2

2

2

2

‖𝑋‖𝑆𝜌 ≤ 𝑟 𝜌* − 𝑝 ‖𝑋‖𝑆𝑝 ≤ 𝑟2/𝜌* ‖𝑋‖𝑆𝑝 ≤ 𝑒2 ‖𝑋‖𝑆𝑝 , *

which yields (54) with 𝒪(1) = 𝑒2 . The lower bound is (109).

56

J

A Direct Comparison of Euclidean/Frobenius and Geometry-Native Oracles

There are two ways to model stochastic-gradient noise in a non-Euclidean geometry. One assumes light tails in a fixed Euclidean or Frobenius norm and converts the resulting bound into the dual norm induced by the update geometry; the other states the light-tail condition directly in that dual norm. This section compares them. We keep the conversion constant 𝛼−1 , the regularity constant 𝜅 and the two noise scales separate throughout, so that one can see which part of each bound comes from the geometry and which part comes from the perturbation model. Notation. The index 𝑝 always refers to the norm on the dual space 𝐸 * , exactly as in Section 3.4 and Section I: we write (𝐸 * , ‖·‖* ) = (R𝑑 , ‖·‖𝑝 ) in the vector case and (𝐸 * , ‖·‖* ) = (R𝑚×𝑛 , ‖·‖𝑆𝑝 ) with 𝑟 := min{𝑚, 𝑛} in the matrix case, and we adopt the convention 𝑑2/∞ = 𝑟2/∞ = 1. Overview of the comparison. We compare the same oracle under the two norm-based analyses. The examples use fresh independent additive perturbations: independent Rademacher signs or standard Gaussian entries in the vector case, and their diagonal matrix counterparts (Definition 9 and Definition 10). Write 𝐷 = 𝑑 for vectors and 𝐷 = 𝑟 for these diagonal matrix models. Under the common assumptions stated below, the two oracle bounds have the same leading term; their statistical terms are governed by C* = 𝜅𝜎 2 and CF = 𝛼−2 𝜎F2 . For 1 ≤ 𝑝 < 2, the squared conversion factor 𝛼−2 and the regularity constant are both 𝐷2/𝑝−1 . Their ratio therefore cancels, leaving the comparison between the noise scales. For these dense perturbations, the ratio of squared noise scales is 𝐷2/𝑝−1 , which is also the ratio of the native and converted coefficients. At 𝑝 = 1, the converted and native coefficients consequently have orders 𝐷2 and 𝐷3 , respectively. At 𝑝 = 2, the norms coincide: choosing 𝜎F = 𝜎, 𝛼 = 1 and 𝜅 = 1 makes the bounds agree. For each fixed 2 < 𝑝 < ∞, the displayed native coefficient is bounded by a constant depending on 𝑝 times 𝐷2/𝑝 𝜈 2 , while the converted coefficient grows as 𝐷𝜈 2 . Since 𝐷2/𝑝 = 𝑜(𝐷) as 𝐷 → ∞, the native bound has smaller asymptotic dimension dependence. At 𝑝 = ∞, its coefficients have orders ln(𝑒𝐷)𝜈 2 for Rademacher perturbations and (ln(𝑒𝐷))2 𝜈 2 for the chosen Gaussian scales, compared with 𝐷𝜈 2 for conversion. Thus the ratios CF /C* have orders 𝐷/ ln(𝑒𝐷) and 𝐷/(ln(𝑒𝐷))2 , respectively. For finite dimensions, the native coefficient can still coincide with or exceed the converted one. These outcomes depend on the noise model. For example, the one-sparse perturbations in Remark 3 give equal coefficients throughout 1 ≤ 𝑝 ≤ 2, so the larger native coefficient for dense noise below 𝑝 = 2 is not universal. We therefore compare the full coefficients rather than infer an advantage from 𝑝 alone, and claim no uniform improvement over Euclidean/Frobenius conversion. In the vector case, dual 𝑝 = ∞ corresponds to an ℓ1 update norm, while dual 𝑝 = 1 corresponds to an ℓ∞ update norm. For matrices, dual 𝑝 = ∞ corresponds to the nuclear update norm, whose dual is the spectral norm. The analogous improvement here concerns diagonal perturbations, not independent noise in all matrix entries (Remark 4). We now specify the oracle models and common assumptions, derive the complexity comparison, and give the detailed vector and matrix results. Assumption 4 (Frobenius light-tail oracle). For every 𝑥 ∈ 𝐸, the stochastic gradient is unbiased, E𝜉 [∇𝑓 (𝑥; 𝜉)] = ∇𝑓 (𝑥), and its error satisfies [︃ (︃ )︃]︃ 2 ‖∇𝑓 (𝑥; 𝜉) − ∇𝑓 (𝑥)‖F E𝜉 exp ≤ exp(1) (55) 𝜎F2 for some 𝜎F > 0. Since 𝐸 * is finite-dimensional, there are constants 𝛼, 𝛽 > 0 with 𝛼 ‖𝑠‖* ≤ ‖𝑠‖F ≤ 𝛽 ‖𝑠‖* ,

∀𝑠 ∈ 𝐸 * .

(56)

Only the left inequality is used below, in the form ‖𝑠‖* ≤ 𝛼−1 ‖𝑠‖F : it is what turns a Frobenius bound into a bound in ‖·‖* , and 𝛼−1 is the entire price of doing so. 57

The converted analysis first concentrates norm, whose ⃦ the minibatch⃦ error in the ⃦ Euclidean/Frobenius ⃦ regularity constant is 1, and then uses ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦* ≤ 𝛼−1 ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦F . Consequently, its statistical coefficient is 𝛼−2 𝜎F2 , with no additional factor of 𝜅; Lemma 21 gives the concentration bound used below. Two perturbation models. For 𝑢 = (𝑢1 , . . . , 𝑢𝑟 ) ∈ R𝑟 we write Diag𝑚,𝑛 (𝑢) for the rectangular diagonal embedding, defined entrywise for 1 ≤ 𝑖 ≤ 𝑚 and 1 ≤ 𝑗 ≤ 𝑛 by {︂ [︀ ]︀ 𝑢𝑖 , 𝑖 = 𝑗 ≤ 𝑟, Diag𝑚,𝑛 (𝑢) 𝑖𝑗 := (57) 0, otherwise. The comparison is carried out on two canonical oracles. Definition 9 (Rademacher perturbed-gradient model). The oracle is ∇𝑓 (𝑥; 𝜉) = ∇𝑓 (𝑥) + 𝜁(𝜉) with E [𝜁(𝜉)] = 0. In the vector case 𝜁 := 𝜈𝜒 with 𝜒 := (𝜒1 , . . . , 𝜒𝑑 ), and in the matrix case 𝜁 := 𝜈 Diag𝑚,𝑛 ((𝜒𝑖 )𝑟𝑖=1 ), where 𝜈 > 0 and {𝜒𝑖 }𝑖≥1 are independent Rademacher variables, Prob{𝜒𝑖 = 1} = Prob{𝜒𝑖 = −1} = 12 . A fresh perturbation is drawn at every oracle call. Definition 10 (Gaussian perturbed-gradient model). The oracle is ∇𝑓 (𝑥; 𝜉) = ∇𝑓 (𝑥) + 𝜁(𝜉) with E [𝜁(𝜉)] = 0. In the vector case 𝜁 := 𝜈𝜔 with 𝜔 := (𝜔1 , . . . , 𝜔𝑑 ), and in the matrix case 𝜁 := 𝜈 Diag𝑚,𝑛 ((𝜔𝑖 )𝑟𝑖=1 ), where 𝜈 > 0 and {𝜔𝑖 }𝑖≥1 are i.i.d. standard Gaussian variables, 𝜔𝑖 ∼ 𝒩 (0, 1). A fresh perturbation is drawn at every oracle call. Both matrix models are diagonal, so the perturbation carries exactly 𝑟 independent coordinates rather than 𝑚𝑛. Write 𝐷 for that number of independent coordinates: 𝐷 = 𝑑 in the vector model and 𝐷 = 𝑟 in the matrix model. The Gaussian model additionally needs the shape factor {︃ 1, 1 ≤ 𝑝 ≤ 2, 𝜗𝑝 (𝐷) := (58) min{𝑝, ln(𝑒𝐷)}, 2 < 𝑝 ≤ ∞, again with 𝐷2/∞ := 1. With this notation the two analyses can be placed side by side before we turn to the proofs. J.1

Common Assumptions and Complexity Bounds

Throughout this comparison, suppose that Assumption 1 and Assumption 2 hold with 𝐿 > 0. Let 𝜅 denote a regularity constant of the dual space (𝐸 * , ‖·‖* ). Fix an initial point 𝑥0 with ∆ := 𝑓 (𝑥0 ) − 𝑓 inf > 0, an accuracy 0 < 𝜀 ≤ 𝐿∆, and a failure probability 𝛿 ∈ (0, 1). Both routes use the update (7) with the same update norm and fixed Borel-measurable sharp selection. Each iteration draws fresh i.i.d. oracle samples independently of the preceding history. The Frobenius route uses Assumption 4, whereas the geometry-native route uses Assumption 3. In both cases, choose ⌈︂ ⌉︂ 1 128𝐿∆ 𝛾𝑘 ≡ , 𝑇 := . 2𝐿 15𝜀 The two analyses differ only in the minibatch size they prescribe. The Frobenius route pays the conversion constant, {︂⌈︂ )︁2 ⌉︂ }︂ √︀ 16𝛼−2 𝜎F2 (︁√ 𝐵F := max 2 + 6 log(𝑇 /𝛿) ,1 , (59) 𝜀 whereas the geometry-native route pays the regularity constant instead, {︂⌈︂ )︁2 ⌉︂ }︂ √︀ 16𝜎 2 (︁√ 𝐵* := max 2𝜅 + 6 log(𝑇 /𝛿) ,1 . (60) 𝜀 To justify these choices, condition on the history ℱ𝑘 before iteration 𝑘, which fixes 𝑥𝑘 . Apply with 𝜆 = √︀ Lemma 21 for the Frobenius route and Lemma 1 for the geometry-native route, √ 3 log(𝑇 /𝛿). The corresponding batch size makes the concentration threshold at most 𝜀/4, so the conditional probability of exceeding this error is at most exp(−𝜆2 /3) = 𝛿/𝑇 . Taking expectations of these conditional probabilities and applying a union bound gives, for either run, {︂ √ }︂ ⃦ ⃦ 𝜀 Prob ∃𝑘 ∈ {0, . . . , 𝑇 − 1} : ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦* > ≤ 𝑇 exp(−𝜆2 /3) = 𝛿. 4 58

On the complementary event, the deterministic descent estimate in the proof of Theorem 1 gives ⃦ ⃦ ⃦2 15 ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦2 whenever ⃦∇𝑓 (𝑥𝑘 )⃦* > 𝜀. 𝑓 (𝑥𝑘+1 ) ≤ 𝑓 (𝑥𝑘 ) − * 128𝐿 If this strict gradient inequality held at every iteration, summing the descent inequalities along that trajectory would yield 15𝑇 𝜀 ≤ 𝑓 inf , 128𝐿 ⃦ ⃦2 contradicting Assumption 2. Thus each run satisfies min0≤𝑘<𝑇 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 with probability at least 1 − 𝛿. 𝑓 (𝑥𝑇 ) < 𝑓 (𝑥0 ) −

Suppressing logarithmic factors, each minibatch size contributes a single coefficient to the statistical term, CF := 𝛼−2 𝜎F2 ,

C* := 𝜅𝜎 2 .

(61)

̃︀ + CF /𝜀) Since 𝜀 ≤ 𝐿∆, the ceiling in 𝑇 gives 𝑇 = 𝒪(𝐿∆/𝜀). The batch sizes satisfy 𝐵F = 𝒪(1 ̃︀ and 𝐵* = 𝒪(1 + C* /𝜀), where the suppressed factors are logarithmic in 𝑇 and 1/𝛿. Multiplying by 𝑇 , the total numbers of oracle calls 𝑁F := 𝑇 𝐵F and 𝑁* := 𝑇 𝐵* therefore satisfy (︂ )︂ 𝐿∆ 𝐿∆ ̃︀ + 2 CF 𝑁F = 𝒪 (62) 𝜀 𝜀 (︂ )︂ 2 ̃︀ 𝐿∆ + 𝐿∆𝜎F =𝒪 (63) 𝜀 𝜀2 𝛼 2 and (︂ ̃︀ 𝑁* = 𝒪

)︂ 𝐿∆ 𝐿∆ + 2 C* . 𝜀 𝜀

(64)

The first term is identical in both, so the comparison reduces, up to the suppressed logarithmic factors, to CF against C* . These coefficients refer to the same oracle and update geometry, but use noise scales measured in different norms. We compare the coefficients rather than interpret their ratio as an exact ratio of oracle complexities, since both analyses provide upper bounds. Side-by-side comparison of the noise terms. The tables below place the two statistical terms next to each other, separating the regimes 1 ≤ 𝑝 < 2 and 2 < 𝑝 < ∞ and displaying the boundary cases 𝑝 = 1, 𝑝 = 2 and 𝑝 = ∞ in full, so that every dimension exponent can be read off directly rather than inferred. Four conventions apply throughout. We take 𝜈 2 = 𝒪(1) to expose the dependence on dimension; a general 𝜈 is restored by multiplying every displayed term by 𝜈 2 . A formula indexed by M ∈ {Rad, Gau} holds for both perturbation models, the numerical Gaussian factor 4 being ̃︀ the two models are written separately only where they genuinely differ. The symbols absorbed into 𝒪; 𝜅ℓ𝑝 (𝑑) and 𝜅𝑆𝑝 (𝑟) are the admissible regularity parameters established in Section J.2. Finally, for every fixed finite 𝑝 the two perturbation models carry the same dependence on dimension up to constants, and they part company only at 𝑝 = ∞, where the admissible Gaussian scale costs one additional logarithmic factor. One exponent is worth unpacking before the tables, since it is the only one that is not read off a single quantity. For 1 ≤ 𝑝 < 2 the regularity factor is 𝜅 = 𝐷2/𝑝−1 . In the Rademacher model, the native noise scale is 𝜎 2 = 𝐷2/𝑝 𝜈 2 , so the exact coefficient is CRad = 𝜅𝜎 2 * = 𝐷2/𝑝−1 · 𝐷2/𝑝 𝜈 2 = 𝐷4/𝑝−1 𝜈 2 . In the Gaussian model, the chosen admissible scale 𝜎 2 = 4𝐷2/𝑝 𝜈 2 instead gives CGau = 𝜅𝜎 2 = 𝐷2/𝑝−1 · 4𝐷2/𝑝 𝜈 2 = 4𝐷4/𝑝−1 𝜈 2 . * Thus both models have the dimension exponent 4/𝑝 − 1; only the absolute factor differs. 59

ℓ𝑑𝑝 spaces. Read the two columns against each other row by row. The left column never improves on 𝑑 and degrades to 𝑑2/𝑝 below 𝑝 = 2. The native coefficient has a larger dimension exponent for 𝑝 < 2, and the two coefficients coincide at 𝑝 = 2. For each fixed 2 < 𝑝 < ∞, the native coefficient has a smaller asymptotic order as 𝑑 → ∞. At 𝑝 = ∞, its dependence is logarithmic for Rademacher noise and squared-logarithmic for Gaussian noise, again asymptotically smaller than linear. These comparisons do not guarantee strict improvement in every finite dimension. Euclidean/Frobenius approach For 1 ≤ 𝑝 < 2, (︂ 2/𝑝 )︂ 𝑑 𝐿Δ M ̃︀ 𝑁F,noise = 𝒪 . 𝜀2

Geometry-native approach For 1 ≤ 𝑝 < 2, (︂ 4/𝑝−1 )︂ 𝑑 𝐿Δ M ̃︀ 𝑁*,noise = 𝒪 . 𝜀2

In particular, at 𝑝 = 1, (︂ 2 )︂ ⃒ 𝑑 𝐿Δ ⃒ M ̃︀ 𝑁F,noise ⃒ =𝒪 . 𝜀2 𝑝=1

In particular, at 𝑝 = 1, (︂ 3 )︂ ⃒ 𝑑 𝐿Δ ⃒ M ̃︀ 𝑁*,noise ⃒ =𝒪 . 𝜀2 𝑝=1

For every 2 ≤ 𝑝 ≤ ∞,

At 𝑝 = 2,

M ̃︀ 𝑁F,noise =𝒪

(︂

𝑑𝐿Δ 𝜀2

)︂

⃒ ⃒ M 𝑁*,noise ⃒

.

(︂ 𝑝=2

̃︀ =𝒪

𝑑𝐿Δ 𝜀2

)︂ .

For 2 < 𝑝 < ∞, (︃ Rad ̃︀ 𝑁*,noise =𝒪

(︃ Gau ̃︀ 𝑁*,noise =𝒪

𝜅ℓ𝑝 (𝑑)𝑑2/𝑝 𝐿Δ 𝜀2

)︃ ,

𝜅ℓ𝑝 (𝑑)𝜗𝑝 (𝑑)𝑑2/𝑝 𝐿Δ 𝜀2

)︃

At 𝑝 = ∞, for every 𝑑 ≥ 1, (︂ )︂ ⃒ Rad ⃒ ̃︀ ln(𝑒𝑑)𝐿Δ , =𝒪 𝑁*,noise ⃒ 𝜀2 𝑝=∞ )︂ (︂ ⃒ (ln(𝑒𝑑))2 𝐿Δ Gau ⃒ ̃︀ =𝒪 . 𝑁*,noise ⃒ 𝜀2 𝑝=∞

60

.

Spaces with Schatten-𝑝 norms. The diagonal perturbations in Definition 9 and Definition 10 allow the vector noise-scale calculations to carry over with 𝑑 replaced by 𝑟 = min{𝑚, 𝑛}. For 𝑢 ∈ R𝑟 and 𝜈 > 0, the singular values of 𝜈 Diag𝑚,𝑛 (𝑢) are the numbers 𝜈|𝑢𝑖 | arranged in nonincreasing order. Consequently, ⃦ ⃦ ⃦𝜈 Diag𝑚,𝑛 (𝑢)⃦ = 𝜈 ‖𝑢‖ , 1 ≤ 𝑝 ≤ ∞. 𝑝

𝑆𝑝

The resulting statistical coefficients have analogous asymptotic dimension dependence, but use the separately defined Schatten regularity parameter 𝜅𝑆𝑝 (𝑟). It need not equal 𝜅ℓ𝑝 (𝑑) even when 𝑑 = 𝑟. Euclidean/Frobenius approach For 1 ≤ 𝑝 < 2, )︂ (︂ 2/𝑝 M ̃︀ 𝑟 𝐿Δ . 𝑁F,noise =𝒪 𝜀2

Geometry-native approach For 1 ≤ 𝑝 < 2, (︂ 4/𝑝−1 )︂ 𝐿Δ M ̃︀ 𝑟 𝑁*,noise =𝒪 . 𝜀2

In particular, at 𝑝 = 1, (︂ 2 )︂ ⃒ 𝑟 𝐿Δ ⃒ M ̃︀ 𝑁F,noise = 𝒪 . ⃒ 𝜀2 𝑝=1

In particular, at 𝑝 = 1, (︂ 3 )︂ ⃒ 𝑟 𝐿Δ ⃒ M ̃︀ 𝑁*,noise = 𝒪 . ⃒ 𝜀2 𝑝=1

For every 2 ≤ 𝑝 ≤ ∞,

At 𝑝 = 2,

M ̃︀ 𝑁F,noise =𝒪

(︂

𝑟𝐿Δ 𝜀2

)︂

⃒ ⃒ M 𝑁*,noise ⃒

.

𝑟𝐿Δ 𝜀2

)︂

𝜅𝑆𝑝 (𝑟)𝑟2/𝑝 𝐿Δ 𝜀2

)︃

(︂ 𝑝=2

̃︀ =𝒪

.

For 2 < 𝑝 < ∞, (︃ Rad ̃︀ 𝑁*,noise =𝒪

(︃ Gau ̃︀ 𝑁*,noise =𝒪

,

𝜅𝑆𝑝 (𝑟)𝜗𝑝 (𝑟)𝑟2/𝑝 𝐿Δ 𝜀2

)︃ .

At 𝑝 = ∞, for every 𝑟 ≥ 1, (︂ )︂ ⃒ Rad ⃒ ̃︀ ln(𝑒𝑟)𝐿Δ , =𝒪 𝑁*,noise ⃒ 𝜀2 𝑝=∞ (︂ )︂ ⃒ (ln(𝑒𝑟))2 𝐿Δ Gau ⃒ ̃︀ = 𝒪 𝑁*,noise . ⃒ 𝜀2 𝑝=∞

J.2

Detailed Results and Proofs

We first establish the Euclidean/Frobenius conversion bound used in the summary. * Lemma 21 (Frobenius-to-dual concentration). Let {𝜂 𝑗 }𝐵 𝑗=1 be i.i.d. random elements of 𝐸 such that [︂ (︂ 𝑗 2 )︂]︂ [︀ 𝑗 ]︀ ∑︀𝐵 ‖𝜂 ‖F E 𝜂 = 0 and E exp ≤ exp(1). Define 𝜂¯ := 𝐵 −1 𝑗=1 𝜂 𝑗 . Then, for every 𝜆 ≥ 0, 𝜎2 F

{︂ Prob ‖¯ 𝜂 ‖* ≥

√ )︁ 𝜎F 1 (︁√ 2 + 2𝜆 √ 𝛼 𝐵

}︂

(︀ )︀ ≤ exp −𝜆2 /3 .

Proof. The Euclidean space (𝐸 * , ‖·‖F ) is 1-regular. Applying Lemma 1 in this space gives {︂ }︂ (︁√ √ )︁ 𝜎F (︀ )︀ Prob ‖¯ 𝜂 ‖F ≥ 2 + 2𝜆 √ ≤ exp −𝜆2 /3 . 𝐵 For any 𝑡 ≥ 0, the left inequality in (56) implies the event inclusion {‖¯ 𝜂 ‖* ≥ 𝑡} ⊆ {‖¯ 𝜂 ‖F ≥ 𝛼𝑡} . Taking 𝑡 :=

√ )︁ 𝜎F 1 (︁√ 2 + 2𝜆 √ 𝛼 𝐵

proves (65). 61

(65)

We next establish the Gaussian light-tail scale used for both norm families. Lemma 22 (Gaussian light tails in ℓ𝑝 ). Let 𝜔 ∼ 𝒩 (0, 𝐼𝐷 ) and 𝑝 ∈ [1, ∞]. For every 𝜈 > 0, )︃]︃ [︃ (︃ 2 ‖𝜈𝜔‖𝑝 ≤ exp(1). E exp 4𝜗𝑝 (𝐷)𝐷2/𝑝 𝜈 2

(66)

Proof. We first consider 1 ≤ 𝑝 ≤ 2. The norm comparison 2

2

2

‖𝜔‖𝑝 ≤ 𝐷 𝑝 −1 ‖𝜔‖2 gives [︃

(︃

E exp

2

‖𝜈𝜔‖𝑝 4𝐷2/𝑝 𝜈 2

)︃]︃

[︃

(︃

≤ E exp

2

‖𝜔‖2 4𝐷

)︃]︃ .

2

Since ‖𝜔‖2 has a chi-squared distribution with 𝐷 degrees of freedom, [︃ (︃ )︃]︃ (︂ )︂−𝐷/2 2 ‖𝜔‖2 1 E exp . = 1− 4𝐷 2𝐷 For 𝑥 ∈ [0, 1), the inequality − ln(1 − 𝑥) ≤ 𝑥/1−𝑥 yields (︂ )︂ 𝐷 1 1 1 1 − ln 1 − ≤ 1 ≤ 2 < 1. 2 2𝐷 4 1 − 2𝐷 Thus (66) holds for 1 ≤ 𝑝 ≤ 2. Now let 2 < 𝑝 ≤ ∞ and set [︁ ]︁ 𝑚𝑝 := E ‖𝜔‖𝑝 . For finite 𝑝, Jensen’s inequality and the Gaussian moment formula give (︁ [︁ ]︁)︁1/𝑝 𝑝 𝑚𝑝 ≤ E ‖𝜔‖𝑝 1/𝑝

= 𝐷1/𝑝 (E [|𝜔1 |𝑝 ]) (︂ 𝑝/2 )︂1/𝑝 2 Γ ((𝑝 + 1)/2) √ = 𝐷1/𝑝 𝜋 1/𝑝 √ 𝑝. ≤𝐷 1/𝑝

The last inequality is the standard Gaussian moment bound (E [|𝜔1 |𝑝 ]) follows from ‖𝜔‖𝑝 ≤ 𝐷1/𝑝 ‖𝜔‖∞ . For any 𝑡 > 0, we first have E [exp (𝑡 ‖𝜔‖∞ )] ≤

𝐷 ∑︁

E [exp (𝑡|𝜔𝑖 |)]

𝑖=1

(︀ )︀ ≤ 2𝐷 exp 𝑡2 /2 . The first inequality uses (︂ )︂ ∑︁ 𝐷 exp 𝑡 max |𝜔𝑖 | ≤ exp (𝑡|𝜔𝑖 |) , 1≤𝑖≤𝐷

𝑖=1

and the second uses E [exp (𝑡|𝜔𝑖 |)] ≤ E [exp (𝑡𝜔𝑖 )] + E [exp (−𝑡𝜔𝑖 )] (︀ )︀ = 2 exp 𝑡2 /2 . 62

≤

√

𝑝. A second bound

Consequently, 1 ln E [exp (𝑡 ‖𝜔‖∞ )] 𝑡 ln(2𝐷) 𝑡 ≤ + . 𝑡 2

E [‖𝜔‖∞ ] ≤

Taking 𝑡 =

√︀

2 ln(2𝐷) gives E [‖𝜔‖∞ ] ≤

√︀ 2 ln(2𝐷).

Combining the two estimates, including the second one at 𝑝 = ∞, gives 𝑚2𝑝 ≤ min{𝑝, 2 ln(2𝐷)}𝐷2/𝑝 .

(67)

Set 𝐻𝑝 (𝐷) := min{𝑝, ln(𝑒𝐷)}𝐷2/𝑝 . If 𝐷 = 1, then 𝐻𝑝 (1) = 1 and ‖𝜔‖𝑝 = |𝜔1 | for every 𝑝 > 2. Therefore, [︃ (︃ )︃]︃ [︂ (︂ 2 )︂]︂ 2 ‖𝜔‖𝑝 𝜔1 = E exp E exp 4𝐻𝑝 (1) 4 √ = 2 < exp(1). It remains to consider 𝐷 ≥ 2. For 𝑝 > 2, the map 𝑧 ↦→ ‖𝑧‖𝑝 is 1-Lipschitz with respect to ‖·‖2 , because |‖𝑢‖𝑝 − ‖𝑣‖𝑝 | ≤ ‖𝑢 − 𝑣‖𝑝 ≤ ‖𝑢 − 𝑣‖2 . The moment-generating-function form of Gaussian concentration, obtained from the Gaussian logSobolev inequality by the Herbst argument, therefore gives, for 𝑋𝑝 := ‖𝜔‖𝑝 − 𝑚𝑝 , the bound (︀ )︀ E [exp (𝑠𝑋𝑝 )] ≤ exp 𝑠2 /2 ,

𝑠 ∈ R.

(68)

Let 𝛾 ∼ 𝒩 (0, 1) be independent of 𝜔. Gaussian linearization and (68) give [︃ (︃ )︃]︃ )︃]︃ [︃ (︃ 2 ‖𝜔‖𝑝 𝛾 ‖𝜔‖𝑝 E exp = E𝜔,𝛾 exp √︀ 4𝐻𝑝 (𝐷) 2𝐻𝑝 (𝐷) )︃ )︃]︃]︃ [︃ (︃ [︃ (︃ 𝛾𝑚𝑝 𝛾𝑋𝑝 = E𝛾 exp √︀ E𝜔 exp √︀ 2𝐻𝑝 (𝐷) 2𝐻𝑝 (𝐷) [︃ (︃ )︃]︃ 2 𝛾𝑚𝑝 𝛾 ≤ E𝛾 exp √︀ + 2𝐻𝑝 (𝐷) 4𝐻𝑝 (𝐷) (︃ )︃ (︂ )︂−1/2 𝑚2𝑝 1 = 1− exp . 2𝐻𝑝 (𝐷) 4𝐻𝑝 (𝐷) − 2 Suppose first that 2 < 𝑝 ≤ ln(𝑒𝐷). Then 𝐻𝑝 (𝐷) = 𝑝𝐷2/𝑝 > 2, and (67) gives 𝑚2𝑝 ≤ 𝐻𝑝 (𝐷). Hence, [︃ (︃ )︃]︃ (︂ )︂ 2 ‖𝜔‖𝑝 𝑚2𝑝 1 1 ln E exp ≤ − ln 1 − + 4𝐻𝑝 (𝐷) 2 2𝐻𝑝 (𝐷) 4𝐻𝑝 (𝐷) − 2 (︂ )︂ 1 4 1 < ln + 2 3 3 < 1. 63

Suppose next that 𝑝 > ln(𝑒𝐷). Since 𝐷 ≥ 2, ln(2𝐷) ≤ ln(𝑒𝐷). Therefore, (67) gives 𝑚2𝑝 ≤ 2𝐻𝑝 (𝐷). Moreover, 𝐻𝑝 (𝐷) = ln(𝑒𝐷)𝐷2/𝑝 ≥ ln(2𝑒) 3 > . 2 It follows that [︃

(︃

ln E exp

2

)︃]︃

‖𝜔‖𝑝 4𝐻𝑝 (𝐷)

(︂ )︂ 𝑚2𝑝 1 1 ≤ − ln 1 − + 2 2𝐻𝑝 (𝐷) 4𝐻𝑝 (𝐷) − 2 (︂ )︂ 3 3 1 + < ln 2 2 4 < 1.

Rescaling 𝜔 by 𝜈 proves (66) for 𝑝 > 2. J.2.1

ℓ𝑑𝑝 Spaces

Let (𝐸 * , ‖·‖* ) = (R𝑑 , ‖·‖𝑝 ) with 𝑝 ∈ [1, ∞]. The sharp one-sided Euclidean-to-ℓ𝑝 constant is {︃ 2 𝑑 𝑝 −1 , 1 ≤ 𝑝 < 2, −2 𝛼𝑝 (𝑑) := (69) 1, 2 ≤ 𝑝 ≤ ∞. Indeed, 1

1

‖𝑧‖𝑝 ≤ 𝑑 𝑝 − 2 ‖𝑧‖2 ,

1 ≤ 𝑝 < 2,

whereas ‖𝑧‖𝑝 ≤ ‖𝑧‖2 ,

2 ≤ 𝑝 ≤ ∞.

Write 𝜅ℓ𝑝 (𝑑) for the following admissible regularity parameter of (R𝑑 , ‖·‖𝑝 ): ⎧ 2 −1 𝑝 , 1 ≤ 𝑝 < 2, ⎪ ⎨𝑑 2 2 𝜅ℓ𝑝 (𝑑) := min (𝜌 − 1)𝑑 𝜌 − 𝑝 , 2 ≤ 𝑝 ≤ ∞. ⎪ ⎩2≤𝜌≤𝑝

(70)

𝜌<∞

This is the parameter constructed in Proposition 12. It satisfies 𝜅ℓ𝑝 (𝑑) ≤ 2𝑒 min{𝑝, ln(𝑒𝑑)},

2 ≤ 𝑝 ≤ ∞.

(71)

The explicit factor 2𝑒 follows by taking 𝜌 = 𝑝 when 𝑝 ≤ 2 ln(𝑒𝑑) and 𝜌 = 2 ln(𝑒𝑑) when 𝑝 > 2 ln(𝑒𝑑). The use of ln(𝑒𝑑) keeps this exact bound valid also at 𝑑 = 1. Proposition 5 (ℓ𝑑𝑝 geometry under Rademacher perturbations). Let the oracle be as in Definition 9. Then the smallest admissible light-tail scales are 𝜎F2 = 𝑑𝜈 2 ,

𝜎 2 = 𝑑2/𝑝 𝜈 2 .

With 𝛼−2 = 𝛼𝑝 (𝑑)−2 and 𝜅 = 𝜅ℓ𝑝 (𝑑), the coefficients in (61) are {︃ 𝑑2/𝑝 𝜈 2 , 1 ≤ 𝑝 < 2, Rad CF := 𝑑𝜈 2 , 2 ≤ 𝑝 ≤ ∞, {︃ 4 −1 1 ≤ 𝑝 < 2, 𝑑 𝑝 𝜈2, CRad := * 𝜅ℓ𝑝 (𝑑)𝑑2/𝑝 𝜈 2 , 2 ≤ 𝑝 ≤ ∞. 64

(72)

(73)

(74)

The coefficients agree at 𝑝 = 2. For 𝑝 < 2, the native coefficient contains one additional factor 𝑑2/𝑝−1 . For every fixed 𝑝 > 2, CRad ≤ 2𝑒 min{𝑝, ln(𝑒𝑑)}𝑑2/𝑝 𝜈 2 , * while CRad = 𝑑𝜈 2 . At 𝑝 = ∞ and 𝑑 ≥ 1, the displayed dependences are 𝒪(ln(𝑒𝑑)𝜈 2 ) and 𝑑𝜈 2 , F respectively. Proof. Every coordinate of 𝜁 = 𝜈𝜒 has absolute value 𝜈. Therefore, 2

‖𝜁‖2 = 𝜈 2

𝑑 ∑︁

𝜒2𝑖

𝑖=1

= 𝑑𝜈 2 . For finite 𝑝, 2 ‖𝜁‖𝑝 =

)︃2/𝑝

(︃ 𝑑 ∑︁

|𝜈𝜒𝑖 |

𝑝

𝑖=1

= (𝑑𝜈 𝑝 )

2/𝑝

= 𝑑2/𝑝 𝜈 2 . At 𝑝 = ∞, 2

‖𝜁‖∞ = max |𝜈𝜒𝑖 |2 1≤𝑖≤𝑑

= 𝜈2, which agrees with 𝑑2/∞ = 1. Let 𝜎 ̃︀F > 0 and 𝜎 ̃︀ > 0 be arbitrary candidate light-tail scales for the Euclidean and ℓ𝑝 norms, respectively. Since both norms of 𝜁 are deterministic, [︃ (︃ )︃]︃ 2 ‖𝜁‖2 E exp ≤ exp(1) ⇐⇒ 𝜎 ̃︀F2 ≥ 𝑑𝜈 2 , 𝜎 ̃︀F2 )︃]︃ [︃ (︃ 2 ‖𝜁‖𝑝 ≤ exp(1) ⇐⇒ 𝜎 ̃︀2 ≥ 𝑑2/𝑝 𝜈 2 . E exp 𝜎 ̃︀2 This proves (72) and shows that both scales are minimal. For 1 ≤ 𝑝 < 2, substituting (69), (70), and (72) gives 2

CRad = 𝑑 𝑝 −1 · 𝑑𝜈 2 = 𝑑2/𝑝 𝜈 2 , F 2

4

CRad = 𝑑 𝑝 −1 · 𝑑2/𝑝 𝜈 2 = 𝑑 𝑝 −1 𝜈 2 . * For 2 ≤ 𝑝 ≤ ∞, 𝛼𝑝 (𝑑)−2 = 1, so CRad = 𝑑𝜈 2 , F CRad = 𝜅ℓ𝑝 (𝑑)𝑑2/𝑝 𝜈 2 . * The bound (71) gives the stated consequences. Proposition 6 (ℓ𝑑𝑝 geometry under Gaussian perturbations). Let the oracle be as in Definition 10. Using the explicit universal constant from Lemma 22, the scales 𝜎F2 := 4𝑑𝜈 2 ,

𝜎 2 := 4𝜗𝑝 (𝑑)𝑑2/𝑝 𝜈 2

(75)

are admissible. The Frobenius scale is also of the smallest possible order: every admissible scale 𝜎 ̃︀F satisfies 𝜎 ̃︀F2 ≥ 𝑑𝜈 2 . 65

For the choices in (75), the coefficients are {︃ 𝑑2/𝑝 𝜈 2 , 1 ≤ 𝑝 < 2, Gau CF := 4 𝑑𝜈 2 , 2 ≤ 𝑝 ≤ ∞, {︃ 4 −1 𝑑 𝑝 𝜈2, 1 ≤ 𝑝 < 2, CGau =4 * 𝜅ℓ𝑝 (𝑑)𝜗𝑝 (𝑑)𝑑2/𝑝 𝜈 2 , 2 ≤ 𝑝 ≤ ∞.

(76)

(77)

At 𝑝 = 2, both coefficients are 4𝑑𝜈 2 . For every fixed 𝑝 > 2, CGau ≤ 8𝑒 min{𝑝, ln(𝑒𝑑)}2 𝑑2/𝑝 𝜈 2 , * whereas CGau = 4𝑑𝜈 2 . At 𝑝 = ∞ and 𝑑 ≥ 1, the displayed dependences are 𝒪((ln(𝑒𝑑))2 𝜈 2 ) and F 2 𝑑𝜈 , respectively. Proof. The Frobenius claim follows from Lemma 22 with 𝐷 = 𝑑 and 𝑝 = 2. To see that its order cannot be smaller, let 𝜎 ̃︀F be any admissible Frobenius scale. Jensen’s inequality gives [︃ (︃ )︃]︃ 2 ‖𝜈𝜔‖2 exp(1) ≥ E exp 𝜎 ̃︀F2 ]︁ ⎞ [︁ ⎛ 2 E ‖𝜈𝜔‖2 ⎠. ≥ exp ⎝ 𝜎 ̃︀F2 Since ]︁ [︁ 2 E ‖𝜈𝜔‖2 = 𝑑𝜈 2 , we obtain 𝜎 ̃︀F2 ≥ 𝑑𝜈 2 . The native scale follows from the same lemma with 𝐷 = 𝑑 and the given value of 𝑝. It remains to substitute the two scales into (61). If 1 ≤ 𝑝 < 2, then 2

CGau = 𝑑 𝑝 −1 · 4𝑑𝜈 2 = 4𝑑2/𝑝 𝜈 2 , F 4

2

CGau = 𝑑 𝑝 −1 · 4𝑑2/𝑝 𝜈 2 = 4𝑑 𝑝 −1 𝜈 2 . * If 2 ≤ 𝑝 ≤ ∞, then CGau = 4𝑑𝜈 2 , F CGau = 4𝜅ℓ𝑝 (𝑑)𝜗𝑝 (𝑑)𝑑2/𝑝 𝜈 2 . * Using (71) proves the final two statements. Remark 3 (One-sparse perturbations). Fix 𝜈 > 0. Let 𝜒 be Rademacher and 𝐼 be uniform on [𝐷], independently of 𝜒, and let 𝑒1 , . . . , 𝑒𝐷 be the standard basis of R𝐷 . Define 𝜁 := 𝜈𝜒𝑒𝐼 𝜁 := 𝜈 Diag𝑚,𝑛 (𝜒𝑒𝐼 )

in the vector case, 𝐷 = 𝑑, in the matrix case, 𝐷 = 𝑟 = min{𝑚, 𝑛}.

The oracle returns ∇𝑓 (𝑥) + 𝜁 with a fresh independent pair (𝜒, 𝐼) at every call. In both cases, E [𝜁] = 0, and the Euclidean/Frobenius norm and the native ℓ𝑝 /Schatten-𝑝 (︁ norm )︁ of 𝜁 equal 𝜈 almost surely for every 1 ≤ 𝑝 ≤ ∞. Thus the light-tail condition reduces to exp

𝜈2 𝜎2

≤ 𝑒, and the minimal

squared light-tail scales are 𝜎F2 = 𝜎 2 = 𝜈 2 . For 1 ≤ 𝑝 ≤ 2, the chosen conversion and regularity 2/𝑝−1 factors both equal 𝐷

, so

CF = C* = 𝐷2/𝑝−1 𝜈 2 . Sparsity alone does not ensure that the coefficients coincide for other perturbation models. 66

Spaces with Schatten-𝑝 Norms on R𝑚×𝑛

J.2.2

Let (𝐸 * , ‖·‖* ) = (R𝑚×𝑛 , ‖·‖𝑆𝑝 ), where 𝑝 ∈ [1, ∞] and 𝑟 := min{𝑚, 𝑛}. The Frobenius norm is ‖·‖F = ‖·‖𝑆2 . Applying the vector norm comparison to the singular-value vector gives {︃ 2 𝑟 𝑝 −1 , 1 ≤ 𝑝 < 2, −2 𝛼𝑆𝑝 (𝑟) := (78) 1, 2 ≤ 𝑝 ≤ ∞. Indeed, 1

1

‖𝑍‖𝑆𝑝 ≤ 𝑟 𝑝 − 2 ‖𝑍‖F ,

1 ≤ 𝑝 < 2,

whereas ‖𝑍‖𝑆𝑝 ≤ ‖𝑍‖F ,

2 ≤ 𝑝 ≤ ∞.

Choose an admissible regularity parameter 𝜅𝑆𝑝 (𝑟) as follows: ⎧ 2 ⎪ 1 ≤ 𝑝 < 2, 𝑟 𝑝 −1 , ⎪ ⎪ ⎪ ⎨1, 𝑝 = 2, 𝜅𝑆𝑝 (𝑟) := 2 2 ⎪ ⎪ ⎪ min max{2, 𝜌 − 1}𝑟 𝜌 − 𝑝 , 2 < 𝑝 ≤ ∞. ⎪ ⎩2≤𝜌≤𝑝

(79)

𝜌<∞

The 𝑝 = 2 line uses the exact Hilbert-space constant. The remaining lines follow from Proposition 14. In particular, 𝜅𝑆𝑝 (𝑟) ≤ 2𝑒 min{𝑝, ln(𝑒𝑟)},

2 ≤ 𝑝 ≤ ∞.

(80)

As in the vector case, for 𝑝 > 2 take 𝜌 = 𝑝 when 𝑝 ≤ 2 ln(𝑒𝑟) and 𝜌 = 2 ln(𝑒𝑟) otherwise. The factor 2𝑒 is valid also at 𝑟 = 1. Proposition 7 (Schatten-𝑝 geometry under diagonal Rademacher perturbations). Let the oracle be as in Definition 9. Then the smallest admissible light-tail scales are 𝜎F2 = 𝑟𝜈 2 , With 𝛼

−2

−2

= 𝛼𝑆𝑝 (𝑟)

𝜎 2 = 𝑟2/𝑝 𝜈 2 .

and 𝜅 = 𝜅𝑆𝑝 (𝑟), the statistical coefficients are {︃ 𝑟2/𝑝 𝜈 2 , 1 ≤ 𝑝 < 2, Rad CF := 𝑟𝜈 2 , 2 ≤ 𝑝 ≤ ∞, {︃ 4 −1 𝑟 𝑝 𝜈2, 1 ≤ 𝑝 < 2, CRad := * 𝜅𝑆𝑝 (𝑟)𝑟2/𝑝 𝜈 2 , 2 ≤ 𝑝 ≤ ∞.

(81)

(82)

(83)

The coefficients agree at 𝑝 = 2. For 𝑝 < 2, the native coefficient contains one additional factor 𝑟2/𝑝−1 . For every fixed 𝑝 > 2, CRad ≤ 2𝑒 min{𝑝, ln(𝑒𝑟)}𝑟2/𝑝 𝜈 2 , * whereas CRad = 𝑟𝜈 2 . At 𝑝 = ∞ and 𝑟 ≥ 1, the displayed dependences are 𝒪(ln(𝑒𝑟)𝜈 2 ) and 𝑟𝜈 2 , F respectively. Proof. The matrix 𝜁 = 𝜈 Diag𝑚,𝑛 ((𝜒𝑖 )𝑟𝑖=1 ) has singular values 𝜎𝑖 (𝜁) = 𝜈|𝜒𝑖 | = 𝜈,

𝑖 = 1, . . . , 𝑟.

Therefore, 2

‖𝜁‖F =

𝑟 ∑︁ 𝑖=1 2

𝜎𝑖 (𝜁)2

= 𝑟𝜈 . 67

For finite 𝑝, 2 ‖𝜁‖𝑆𝑝 =

)︃2/𝑝

(︃ 𝑟 ∑︁

𝑝

𝜎𝑖 (𝜁)

𝑖=1 2/𝑝

= (𝑟𝜈 𝑝 )

= 𝑟2/𝑝 𝜈 2 . At 𝑝 = ∞, 2

‖𝜁‖𝑆∞ = max 𝜎𝑖 (𝜁)2 1≤𝑖≤𝑟

= 𝜈2. Let 𝜎 ̃︀F > 0 and 𝜎 ̃︀ > 0 be arbitrary candidate light-tail scales for the Frobenius and Schatten-𝑝 norms, respectively. Since both norms of 𝜁 are deterministic, [︃ (︃ )︃]︃ 2 ‖𝜁‖F E exp ≤ exp(1) 𝜎 ̃︀F2 (︂ 2 )︂ 𝑟𝜈 ≤ exp(1) ⇐⇒ exp 𝜎 ̃︀F2 ⇐⇒ 𝜎 ̃︀F2 ≥ 𝑟𝜈 2 , and, with the convention 𝑟2/∞ = 1, [︃ (︃ 2 )︃]︃ ‖𝜁‖𝑆𝑝 ≤ exp(1) E exp 𝜎 ̃︀2 (︂ 2/𝑝 2 )︂ 𝑟 𝜈 ⇐⇒ exp ≤ exp(1) 𝜎 ̃︀2 ⇐⇒ 𝜎 ̃︀2 ≥ 𝑟2/𝑝 𝜈 2 . Thus the choices 𝜎F2 = 𝑟𝜈 2 and 𝜎 2 = 𝑟2/𝑝 𝜈 2 in (81) are admissible and are the smallest possible scales. For 1 ≤ 𝑝 < 2, substitution gives 2

CRad = 𝑟 𝑝 −1 · 𝑟𝜈 2 = 𝑟2/𝑝 𝜈 2 , F 4

2

CRad = 𝑟 𝑝 −1 · 𝑟2/𝑝 𝜈 2 = 𝑟 𝑝 −1 𝜈 2 . * For 2 ≤ 𝑝 ≤ ∞, 𝛼𝑆𝑝 (𝑟)−2 = 1, and hence CRad = 𝑟𝜈 2 , F CRad = 𝜅𝑆𝑝 (𝑟)𝑟2/𝑝 𝜈 2 . * The bound (80) gives the stated consequences. Proposition 8 (Schatten-𝑝 geometry under diagonal Gaussian perturbations). Let the oracle be as in Definition 10. Using the explicit universal constant from Lemma 22, the scales 𝜎F2 := 4𝑟𝜈 2 ,

𝜎 2 := 4𝜗𝑝 (𝑟)𝑟2/𝑝 𝜈 2

(84)

are admissible. Every admissible Frobenius scale 𝜎 ̃︀F satisfies 𝜎 ̃︀F2 ≥ 𝑟𝜈 2 , so the displayed Frobenius scale has the smallest possible order. For the choices in (84), the coefficients are {︃ 𝑟2/𝑝 𝜈 2 , 1 ≤ 𝑝 < 2, Gau CF := 4 𝑟𝜈 2 , 2 ≤ 𝑝 ≤ ∞, {︃ 4 −1 1 ≤ 𝑝 < 2, 𝑟 𝑝 𝜈2, CGau =4 * 𝜅𝑆𝑝 (𝑟)𝜗𝑝 (𝑟)𝑟2/𝑝 𝜈 2 , 2 ≤ 𝑝 ≤ ∞. 68

(85)

(86)

At 𝑝 = 2, both coefficients are 4𝑟𝜈 2 . For every fixed 𝑝 > 2, CGau ≤ 8𝑒 min{𝑝, ln(𝑒𝑟)}2 𝑟2/𝑝 𝜈 2 , * whereas CGau = 4𝑟𝜈 2 . At 𝑝 = ∞ and 𝑟 ≥ 1, the displayed dependences are 𝒪((ln(𝑒𝑟))2 𝜈 2 ) and F 𝑟𝜈 2 , respectively. Proof. For 𝜁 = 𝜈 Diag𝑚,𝑛 ((𝜔𝑖 )𝑟𝑖=1 ) , the singular values are the numbers 𝜈|𝜔𝑖 |, 𝑖 = 1, . . . , 𝑟, arranged in nonincreasing order. Since the corresponding vector norms are invariant under permutations, we obtain ‖𝜁‖F = 𝜈 ‖𝜔‖2 , ‖𝜁‖𝑆𝑝 = 𝜈 ‖𝜔‖𝑝 , where 𝜔 = (𝜔1 , . . . , 𝜔𝑟 ) ∼ 𝒩 (0, 𝐼𝑟 ). Applying Lemma 22 with 𝐷 = 𝑟 and 𝑝 = 2 proves that 4𝑟𝜈 2 is an admissible Frobenius scale. If 𝜎 ̃︀F is any admissible Frobenius scale, then Jensen’s inequality gives [︃ (︃ )︃]︃ 2 ‖𝜁‖F exp(1) ≥ E exp 𝜎 ̃︀F2 [︁ ]︁ ⎞ ⎛ 2 E ‖𝜁‖F ⎠. ≥ exp ⎝ 𝜎 ̃︀F2 Since [︁ ]︁ 2 E ‖𝜁‖F = 𝑟𝜈 2 , we obtain 𝜎 ̃︀F2 ≥ 𝑟𝜈 2 . Applying the same lemma with 𝐷 = 𝑟 and the stated value of 𝑝 proves that the native scale in (84) is admissible. For 1 ≤ 𝑝 < 2, substitution yields 2

CGau = 𝑟 𝑝 −1 · 4𝑟𝜈 2 = 4𝑟2/𝑝 𝜈 2 , F 2

4

CGau = 𝑟 𝑝 −1 · 4𝑟2/𝑝 𝜈 2 = 4𝑟 𝑝 −1 𝜈 2 . * For 2 ≤ 𝑝 ≤ ∞, 𝛼𝑆𝑝 (𝑟)−2 = 1, and hence CGau = 4𝑟𝜈 2 , F CGau = 4𝜅𝑆𝑝 (𝑟)𝜗𝑝 (𝑟)𝑟2/𝑝 𝜈 2 . * Using (80) proves the final statements. Remark 4 (Scope of the matrix perturbation models). The two matrix perturbations are diagonal, with all 𝑟 singular-value coordinates active. They are not matrices with independent noise in all 𝑚𝑛 entries. The singular values of an entrywise random matrix have a different distribution, so neither proposition applies to that model without a new calculation of both light-tail scales.

69

K

Fundamental Properties of Smooth and Regular Spaces

This section collects general results for finite-dimensional smooth and regular normed spaces. We first state a lemma that provides a smooth auxiliary norm uniformly equivalent to a regular norm, and then recall equivalent characterizations of smoothness in terms of the squared norm and its conjugate. The ℓ𝑑𝑝 and Schatten-𝑝 examples, with matching upper and lower regularity bounds, are developed in the next section, Section L. The following lemma supplies the smooth auxiliary norm used in our proof of the concentration inequality (Section M.1). Lemma 23 (Juditsky and Nemirovski [2008, Lemma 3]). Let (𝑉, ‖ · ‖𝑉 ) be a finite-dimensional 𝜅𝑉 -regular space in the sense of Definition 1. Then there exists a 𝜅𝑉 -smooth norm ‖ · ‖𝑉,+ on 𝑉 such that ‖𝑢‖2𝑉 ≤ ‖𝑢‖2𝑉,+ ≤ 2‖𝑢‖2𝑉 ,

∀𝑢 ∈ 𝑉.

(87)

Next, we recall standard equivalent characterizations of smoothness in the primal space. For 𝜑(𝑢) := 1 2 2 ‖𝑢‖𝑉,+ , these properties relate the smoothness of the norm ‖ · ‖𝑉,+ to the strong convexity of the Fenchel conjugate 𝜑* and the Lipschitz continuity of the gradient ∇𝜑. Proposition 9 (Juditsky and Nemirovski [2008, Proposition 3.2]). Let (𝑉, ‖ · ‖𝑉,+ ) be a finitedimensional normed space with dual space 𝑉 * and induced dual norm ‖ · ‖𝑉,+,* . Define 𝜑(𝑢) := 1 1 * 2 2 2 ‖𝑢‖𝑉,+ and 𝜑 (𝑠) := 2 ‖𝑠‖𝑉,+,* . Then the following six properties are equivalent: (i) The space (𝑉, ‖ · ‖𝑉,+ ) is 𝜅𝑉,+ -smooth. (ii) The function 𝜑 is everywhere differentiable, and its gradient satisfies ⟨∇𝜑(𝑢) − ∇𝜑(𝑣), 𝑢 − 𝑣⟩ ≤ 𝜅𝑉,+ ‖𝑢 − 𝑣‖2𝑉,+ ,

∀𝑢, 𝑣 ∈ 𝑉.

(iii) The function 𝜑 is continuously differentiable, and the mapping ∇𝜑 : 𝑉 → 𝑉 * is 𝜅𝑉,+ Lipschitz continuous: ‖∇𝜑(𝑢) − ∇𝜑(𝑣)‖𝑉,+,* ≤ 𝜅𝑉,+ ‖𝑢 − 𝑣‖𝑉,+ ,

∀𝑢, 𝑣 ∈ 𝑉.

(iv) The set-valued subdifferential operator 𝜕𝜑* : 𝑉 * ⇒ 𝑉 is strongly monotone with constant 1/𝜅𝑉,+ : ⟨𝑠1 − 𝑠2 , 𝑢1 − 𝑢2 ⟩ ≥

1 ‖𝑠1 − 𝑠2 ‖2𝑉,+,* , 𝜅𝑉,+

for all 𝑠1 , 𝑠2 ∈ 𝑉 * and all 𝑢1 ∈ 𝜕𝜑* (𝑠1 ), 𝑢2 ∈ 𝜕𝜑* (𝑠2 ). (v) The inverse mapping (𝜕𝜑* )−1 = 𝜕𝜑 is single-valued and 𝜅𝑉,+ -Lipschitz continuous, which can be written as 1 ‖𝑢1 − 𝑢2 ‖𝑉,+ ≥ ‖𝑠1 − 𝑠2 ‖𝑉,+,* , 𝜅𝑉,+ for all 𝑠1 , 𝑠2 ∈ 𝑉 * and all 𝑢1 ∈ 𝜕𝜑* (𝑠1 ), 𝑢2 ∈ 𝜕𝜑* (𝑠2 ). (vi) The function 𝜑* is strongly convex with constant 1/𝜅𝑉,+ with respect to ‖ · ‖𝑉,+,* : 𝜑* (𝑠2 ) ≥ 𝜑* (𝑠1 ) + ⟨𝑢1 , 𝑠2 − 𝑠1 ⟩ + for all 𝑠1 , 𝑠2 ∈ 𝑉 * and all 𝑢1 ∈ 𝜕𝜑* (𝑠1 ).

70

1 ‖𝑠2 − 𝑠1 ‖2𝑉,+,* , 2𝜅𝑉,+

Examples of 𝜅-Regular Spaces and Sharp Bounds on 𝜅

L

This section establishes the universal regularity bound and gives explicit upper and matching lower bounds for the norm families used in the main text. For ℓ𝑑𝑝 and Schatten-𝑝 spaces, write 𝐷 = 𝑑 and 𝐷 = 𝑟 = min{𝑚, 𝑛}, respectively. The optimal parameter equals 𝐷2/𝑝−1 for 1 ≤ 𝑝 < 2 and has order min{𝑝, ln(𝑒𝐷)} for 2 ≤ 𝑝 ≤ ∞. Thus the latter regime admits a bound independent of 𝐷 for each fixed finite 𝑝, with logarithmic growth in the branch 𝑝 ≥ ln(𝑒𝐷), including 𝑝 = ∞. Table 3 summarizes the bounds and auxiliary norms using the main-text convention 𝑉 = 𝐸 * . We begin with the universal bound via John’s ellipsoid, followed by the ℓ𝑑𝑝 and Schatten-𝑝 constructions and their matching lower bounds. Space (𝐸 * , ‖·‖* )

𝜅 scalingb

Any 𝑑-dimensional normed space, 𝑑 ≥ 1a

Θ(𝑑)

(R𝑑 , ‖ · ‖𝑝 ), 1 ≤ 𝑝 < 2

(︁ 2 )︁ −1 Θ 𝑑𝑝

𝑑

(R , ‖ · ‖𝑝 ), 2 ≤ 𝑝 ≤ ∞

(︀ )︀ Θ min{𝑝, ln(𝑒𝑑)}

(︁ 2 )︁ −1 (R𝑚×𝑛 , ‖ · ‖𝑆𝑝 ), 1 ≤ 𝑝 < 2, Θ 𝑟 𝑝 𝑟 := min{𝑚, 𝑛} (︀ )︀ (R𝑚×𝑛 , ‖ · ‖𝑆𝑝 ), 2 ≤ 𝑝 ≤ ∞, Θ min{𝑝, ln(𝑒𝑟)} 𝑟 := min{𝑚, 𝑛}

Auxiliary norm ‖ · ‖*,+ Inner-product norm whose unit ball is the maximum-volume ellipsoid contained in B* = {𝑠 ∈ 𝐸 * : ‖𝑠‖* ≤ 1} (the John ellipsoid).

Upper bound

Lower bound

Prop. 10

Prop. 11

Prop. 12

Prop. 13

Prop. 14

Prop. 15

1−1

𝑑 𝑝 2 ‖ · ‖2 (scaled Euclidean norm) ‖ · ‖𝜌* , where 𝜌* = 2 for 1 ≤ 𝑑 < 8, and 𝜌* = min{𝑝, 𝜌^(𝑑)} for 𝑑 ≥ 8.c 1−1 2

𝑟𝑝

‖ · ‖𝑆2

‖ · ‖𝑆𝜌* , where 𝜌* = min{𝑝, 3} for 1 ≤ 𝑟 < 8, and 𝜌* = min{𝑝, max{3, 𝜌^(𝑟)}} for 𝑟 ≥ 8.d

a The universal bound applies to every nonzero finite-dimensional normed space, including the dual space (𝐸 * , ‖·‖

* ), and is sharp in general. The structured norms below can have substantially smaller 𝜅, although ℓ𝑑1 attains the universal bound 𝜅 = 𝑑. b For 𝑝 ≥ 2, the upper and lower bounds agree up to absolute constants, absorbed by Θ(·). Throughout the manuscript, order estimates use ln(𝑒𝐷) = 1 + ln 𝐷, with 𝐷 = 𝑑 or 𝑟, to cover all positive dimensions (Remark 7). Exact formulas and explicit-constant inequalities retain ln 𝐷 under their stated dimension conditions. √︀ c Here 𝜌 ^(𝑑) = ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑 for 𝑑 ≥ 8. This choice minimizes (𝜌 − 1)𝑑2/𝜌−2/𝑝 over 2 ≤ 𝜌 ≤ 𝑝, 𝜌 < ∞; in particular, 𝜌* remains finite √︀ when 𝑝 = ∞. d Here 𝜌 ^(𝑟) = ln 𝑟 + (ln 𝑟)2 − 2 ln 𝑟 for 𝑟 ≥ 8. This choice minimizes max{2, 𝜌 − 1}𝑟2/𝜌−2/𝑝 over 2 ≤ 𝜌 ≤ 𝑝, 𝜌 < ∞, and remains finite when 𝑝 = ∞. For 1 ≤ 𝑟 < 8, this bound is nonincreasing on [2, 3] and increasing on [3, ∞), giving 𝜌* = min{𝑝, 3}.

Table 3: Examples of 𝜅-regular spaces for 𝑉 = 𝐸 * with corresponding regularity parameters 𝜅 and auxiliary norm ‖·‖*,+ . L.1

Universal Bound via John’s Ellipsoid

The universal regularity bound follows from the symmetric form of John’s ellipsoid theorem [John, 1948]. We give a proof based on a convex optimization problem for the ellipsoid’s shape matrix. Theorem 7 (John’s Ellipsoid Theorem for Symmetric Bodies). Let 𝑉 be a 𝑑-dimensional real vector space and let 𝐵 ⊂ 𝑉 be a compact, convex body with non-empty interior containing the origin. Assume 𝐵 is centrally symmetric, i.e., if 𝑥 ∈ 𝐵 then −𝑥 ∈ 𝐵. Then: (i) There exists a unique ellipsoid ℰin of maximal volume contained in 𝐵 (the John ellipsoid). (ii) The body 𝐵 is “sandwiched” by this ellipsoid and its scaled-up version: √ ℰin ⊆ 𝐵 ⊆ 𝑑 ℰin . 71

Proof. The case 𝑑 = 0 is immediate, so assume 𝑑 ≥ 1. Fix an identification 𝑉 ≃ R𝑑 , and let ℬ2 denote the Euclidean unit ball. Volume ratios and the claimed inclusions are invariant under invertible linear changes of coordinates. Symmetry and convexity imply 0 ∈ int 𝐵, so there exist 𝜀, 𝑅 > 0 such that 𝜀ℬ2 ⊆ 𝐵 ⊆ 𝑅ℬ2 . Centering and the optimal shape matrix. Every ellipsoid of positive volume can be written 𝑑 uniquely as 𝑐 + 𝑄ℬ2 , where 𝑐 ∈ R𝑑 and 𝑄 ∈ 𝑆++ is symmetric positive definite. If 𝑐 + 𝑄ℬ2 ⊆ 𝐵, symmetry gives −𝑐 + 𝑄ℬ2 ⊆ 𝐵, and convexity then yields [︀ ]︀ 𝑄ℬ2 = 12 (𝑐 + 𝑄ℬ2 ) + (−𝑐 + 𝑄ℬ2 ) ⊆ 𝐵. Centering therefore preserves feasibility and volume, without any appeal to uniqueness. Consider the set 𝑑 𝒬 := {𝑄 ∈ 𝑆+ : 𝑄ℬ2 ⊆ 𝐵}, 𝑑 where 𝑆+ is the cone of symmetric positive semidefinite matrices. This set is closed and convex. Indeed, for 𝑄1 , 𝑄2 ∈ 𝒬, 𝑡 ∈ [0, 1], and 𝑧 ∈ ℬ2 , convexity of 𝐵 gives ((1 − 𝑡)𝑄1 + 𝑡𝑄2 )𝑧 ∈ 𝐵. Moreover, 𝒬 is bounded because every 𝑄 ∈ 𝒬 satisfies ‖𝑄‖op ≤ 𝑅. Thus 𝒬 is compact. Since 𝜀𝐼𝑑 ∈ 𝒬, the maximum of det 𝑄 over 𝒬 is positive and is attained at a positive definite matrix 𝑄* . The volume of 𝑄ℬ2 is det 𝑄 times the volume of ℬ2 , so 𝑄* ℬ2 is a maximal-volume ellipsoid among all feasible ellipsoids by the centering argument.

The centered maximizer is unique: maximizing det 𝑄 on positive definite matrices is equivalent to 𝑑 maximizing the strictly concave function log det 𝑄 over the convex set 𝒬 ∩ 𝑆++ . Strict concavity follows from 𝐷2 (log det 𝑄)[𝐻, 𝐻] = − Tr(𝑄−1 𝐻𝑄−1 𝐻) < 0 for every nonzero symmetric matrix 𝐻. We prove below that an optimal ellipsoid cannot have a nonzero center. ̃︀ := 𝑇 (𝐵). A finite contact decomposition. Normalize by the linear map 𝑇 := (𝑄* )−1 and write 𝐵 ̃︀ The body 𝐵 is centrally symmetric, contains ℬ2 , and has ℬ2 as its unique centered maximal-volume ellipsoid. Define its polar body by ̃︀ ∘ := {𝑦 ∈ R𝑑 : ⟨𝑦, 𝑥⟩ ≤ 1 for all 𝑥 ∈ 𝐵}. ̃︀ 𝐵 It is compact and contained in ℬ2 . By the separating-hyperplane characterization of a closed convex body, for a symmetric positive definite matrix 𝑄, ̃︀ 𝑄ℬ2 ⊆ 𝐵

⇐⇒

‖𝑄𝑦‖2 ≤ 1

̃︀ ∘ . for all 𝑦 ∈ 𝐵

Here we used sup𝑧∈ℬ2 ⟨𝑦, 𝑄𝑧⟩ = ‖𝑄𝑦‖2 . Consequently, 𝐼𝑑 solves the finite-dimensional convex problem min − log det 𝑄

𝑑 𝑄∈𝑆++

subject to

𝑔(𝑄) := max ‖𝑄𝑦‖22 − 1 ≤ 0. ̃︀ ∘ 𝑦∈𝐵

The function 𝑔 is finite, continuous, and convex, since it is the maximum of continuous convex quadratic functions indexed by a compact set. Slater’s condition holds: 𝑄 = 12 𝐼𝑑 is positive definite and satisfies 𝑔(𝑄) ≤ − 34 . The convex KKT theorem therefore supplies a multiplier 𝜇 ≥ 0 and a subgradient 𝐺 ∈ 𝜕𝑔(𝐼𝑑 ) such that the Lagrangian ℒ(𝑄, 𝜇) = − log det 𝑄 + 𝜇𝑔(𝑄) satisfies stationarity and complementary slackness: −𝐼𝑑 + 𝜇𝐺 = 0,

𝜇𝑔(𝐼𝑑 ) = 0.

In particular, 𝜇 > 0 and 𝑔(𝐼𝑑 ) = 0. Equip the space of symmetric matrices with the pairing ⟨𝐻, 𝐾⟩ = Tr(𝐻𝐾). The subdifferential rule for a maximum of differentiable convex functions over a compact index set gives ̃︀ ∘ , ‖𝑦‖2 = 1}, 𝜕𝑔(𝐼𝑑 ) = conv{2𝑦𝑦 ⊤ : 𝑦 ∈ 𝐵 72

because the gradient of 𝑄 ↦→ ‖𝑄𝑦‖22 at 𝐼𝑑 is 2𝑦𝑦 ⊤ . The maximum is attained, and these gradients depend continuously on 𝑦, so the compact-index maximum rule applies. By Carathéodory’s theorem in the finite-dimensional space of symmetric matrices, 𝐺 is a finite convex combination of these active gradients. Absorbing the factor 2𝜇 into the coefficients and discarding zero coefficients gives contact points 𝑦1 , . . . , 𝑦𝑘 and weights 𝜆1 , . . . , 𝜆𝑘 > 0 satisfying 𝐼𝑑 =

𝑘 ∑︁

𝜆𝑖 𝑦𝑖 𝑦𝑖⊤ .

(88)

𝑖=1

̃︀ and to 𝐵 ̃︀ ∘ , and ⟨𝑦𝑖 , 𝑦𝑖 ⟩ = 1. Thus Each 𝑦𝑖 belongs to ℬ2 ⊆ 𝐵 ̃︀ ∩ 𝜕ℬ2 , 𝑦𝑖 ∈ 𝜕 𝐵

‖𝑦𝑖 ‖2 = 1.

(89)

Taking traces in (88) yields 𝑘 ∑︁

𝜆𝑖 = 𝑑.

(90)

𝑖=1

̃︀ exists and also supports ℬ2 , because ℬ2 ⊆ 𝐵. ̃︀ At each contact point 𝑦𝑖 , a supporting hyperplane of 𝐵 The Euclidean ball has the unique supporting hyperplane {𝑥 : ⟨𝑥, 𝑦𝑖 ⟩ = 1} at 𝑦𝑖 . Hence ⟨𝑥, 𝑦𝑖 ⟩ ≤ 1,

̃︀ 𝑥 ∈ 𝐵,

𝑖 = 1, . . . , 𝑘.

(91)

By central symmetry, the same inequality applied to −𝑥 gives |⟨𝑥, 𝑦𝑖 ⟩| ≤ 1. Uniqueness among all ellipsoids. Let 𝑐 + 𝑄ℬ2 ⊆ 𝐵 be any maximal-volume ellipsoid. Centering it preserves its volume, so uniqueness of the centered maximizer gives 𝑄 = 𝑄* . After applying 𝑇 , ̃︀ ̃︀ where 𝑎 = 𝑇 𝑐. For each 𝑖, the two points 𝑎 + 𝑦𝑖 and 𝑎 − 𝑦𝑖 belong to 𝐵. this ellipsoid is 𝑎 + ℬ2 ⊆ 𝐵, Applying the two-sided supporting inequality to them gives ⟨𝑎, 𝑦𝑖 ⟩ ≤ 0 and ⟨𝑎, 𝑦𝑖 ⟩ ≥ 0, respectively. Thus ⟨𝑎, 𝑦𝑖 ⟩ = 0 for every 𝑖, and (88) implies ‖𝑎‖22 =

𝑘 ∑︁

𝜆𝑖 ⟨𝑎, 𝑦𝑖 ⟩2 = 0.

𝑖=1

Therefore 𝑐 = 0, proving uniqueness among all ellipsoids and identifying ℰin = 𝑄* ℬ2 . ̃︀ the contact decomposition and the supporting inequalThe sandwich inequality. For every 𝑥 ∈ 𝐵, ities give ‖𝑥‖22 =

𝑘 ∑︁

𝜆𝑖 ⟨𝑥, 𝑦𝑖 ⟩2 ≤

𝑖=1

̃︀ ⊆ Hence ℬ2 ⊆ 𝐵

√

𝑘 ∑︁

𝜆𝑖 = 𝑑.

𝑖=1

𝑑 ℬ2 . Applying 𝑇 −1 yields ℰin ⊆ 𝐵 ⊆

√

𝑑 ℰin .

We now proceed with a dimension-dependent regularity guarantee that holds for every 𝑑-dimensional normed space. It will serve as a baseline for comparison with the sharper, structure-dependent bounds stated next. Proposition 10 (Universal Regularity Bound). Let 𝑑 ≥ 1. Any 𝑑-dimensional normed space (𝑉, ‖·‖𝑉 ) is 𝑑-regular in the sense of Definition 1, i.e., it is 𝜅𝑉 -regular with the choice 𝜅𝑉 = 𝑑. Proof. To simplify notation throughout this proof, we omit the subscript 𝑉 and write ‖ · ‖ := ‖ · ‖𝑉 , ‖ · ‖+ := ‖ · ‖𝑉,+ , 𝜅+ := 𝜅𝑉,+ and 𝜅 := 𝜅𝑉 . The proof proceeds by explicitly constructing the required smooth norm and verifying that it satisfies the conditions of Definition 1 with 𝜅 = 𝑑. The construction relies on the geometric sandwich inequality provided by John’s Ellipsoid Theorem. Step 1: Applying John’s Theorem to the Unit Ball. Let (𝑉, ‖ · ‖) be a 𝑑-dimensional normed space. Its unit ball, 𝐵 = {𝑥 ∈ 𝑉 : ‖𝑥‖ ≤ 1}, is a centrally symmetric, compact, convex body containing 73

the origin. By John’s Ellipsoid Theorem (Theorem 7), there exists a unique ellipsoid of maximal volume ℰin contained in 𝐵, such that √ ℰin ⊆ 𝐵 ⊆ 𝑑ℰin . Step 2: From Geometric Inclusion to an Algebraic Norm Inequality. Any ellipsoid centered at the origin can be represented as the unit ball of a Euclidean norm. Let us define the Euclidean norm ‖ · ‖+ on 𝑉 as the one whose unit ball is precisely the John ellipsoid ℰin , i.e., ℰin = {𝑥 ∈ 𝑉 : ‖𝑥‖+ ≤ 1}. We can now translate the geometric inclusions into algebraic inequalities between the norms ‖ · ‖ and ‖ · ‖+ : • The inclusion ℰin ⊆ 𝐵 means that if ‖𝑥‖+ ≤ 1, then ‖𝑥‖ ≤ 1. By homogeneity of norms, this implies ‖𝑥‖ ≤ ‖𝑥‖+ for all 𝑥 ∈ 𝑉 . √ √ ⊆ 𝑑ℰin means that if ‖𝑥‖ ≤ 1, then ‖𝑥‖+ ≤ 𝑑. By homogeneity, this • The inclusion 𝐵 √ implies ‖𝑥‖+ ≤ 𝑑‖𝑥‖ for all 𝑥 ∈ 𝑉 . √ Combining these two results, we have the norm equivalence inequality ‖𝑥‖ ≤ ‖𝑥‖+ ≤ 𝑑‖𝑥‖ for all 𝑥 ∈ 𝑉 . Squaring this expression gives the form required by the definition of regularity: ‖𝑥‖2 ≤ ‖𝑥‖2+ ≤ 𝑑‖𝑥‖2 ,

∀𝑥 ∈ 𝑉.

(92)

Step 3: Verifying the Conditions for 𝑑-Regularity. To show that (𝑉, ‖ · ‖) is 𝑑-regular, we must find a 𝜅+ -smooth auxiliary norm ‖ · ‖+ and a constant 𝜅 ≥ 𝜅+ such that 𝜅 ‖𝑥‖2 ≤ ‖𝑥‖2+ ≤ ‖𝑥‖2 . 𝜅+ Let us choose the Euclidean norm ‖ · ‖+ from Step 2. As a Euclidean norm, the space (𝑉, ‖ · ‖+ ) is 1-smooth. To see this, note that for 𝜑+ (𝑥) := 21 ‖𝑥‖2+ , there exists an inner product ⟨·, ·⟩+ with ‖𝑥‖2+ = ⟨𝑥, 𝑥⟩+ , and the descent inequality holds with equality for 𝜅+ = 1: 1 1 1 ‖𝑥 + 𝑦‖2+ = ‖𝑥‖2+ + ⟨𝑥, 𝑦⟩+ + ‖𝑦‖2+ 2 2 2 1 2 = 𝜑+ (𝑥) + ⟨∇𝜑+ (𝑥), 𝑦⟩ + ‖𝑦‖+ , 2 * where ∇𝜑+ (𝑥) ∈ 𝑉 is the linear functional 𝑦 ↦→ ⟨𝑥, 𝑦⟩+ . Therefore, we can set 𝜅+ = 1. 𝜑+ (𝑥 + 𝑦) =

Now we check the compatibility inequality from Definition 1: 𝜅 ‖𝑥‖2 . ‖𝑥‖2+ ≤ 𝜅+ Substituting 𝜅+ = 1 and comparing with (92), we see a direct match if we set 𝜅 = 𝑑: ‖𝑥‖2+ ≤ 𝑑‖𝑥‖2 . The constructed norm ‖ · ‖+ is 1-smooth (so 𝜅+ = 1) and satisfies the required inequality for 𝜅 = 𝑑. Therefore, by Definition 1, the space (𝑉, ‖ · ‖𝑉 ) is 𝑑-regular. While Proposition 10 establishes that 𝜅𝑉 = 𝑑 is sufficient for any finite-dimensional space, one might ask whether a sub-linear dependence is achievable universally. The following proposition answers this in the negative by demonstrating that for the ℓ1 norm, the constant of regularity must scale linearly with the dimension. Thus, the universal bound is tight up to absolute constants. Proposition 11 (Tightness of the universal 𝑑-regularity bound). Fix 𝑑 ∈ N and consider the normed space (𝑉, ‖ · ‖𝑉 ) with 𝑉 := R𝑑 and ‖ · ‖𝑉 := ‖ · ‖1 . If (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular in the sense of Definition 1, then necessarily 𝜅𝑉 ≥ 𝑑.

(93)

In particular, the universal upper bound of Proposition 10 is tight (up to absolute constants): there exist 𝑑-dimensional normed spaces whose constant of regularity is of order 𝑑. 74

Proof. Let {𝜀𝑖 }𝑑𝑖=1 be i.i.d. Rademacher random variables, i.e. Prob{𝜀𝑖 = 1} = Prob{𝜀𝑖 = −1} = 12 , and let {𝑒𝑖 }𝑑𝑖=1 be the standard basis of R𝑑 . Define the filtration ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ) and the sequence 𝜉𝑖 := 𝜀𝑖 𝑒𝑖 ,

𝑖 = 1, . . . , 𝑑.

Then {𝜉𝑖 }𝑑𝑖=1 is an {ℱ𝑖 }-martingale-difference sequence: it is ℱ𝑖 -measurable and E [𝜉𝑖 | ℱ𝑖−1 ] = 𝑒𝑖 E [𝜀𝑖 | ℱ𝑖−1 ] = 𝑒𝑖 E [𝜀𝑖 ] = 0 [︀ ]︀ Moreover, ‖𝜉𝑖 ‖1 = ‖𝑒𝑖 ‖1 = 1, hence E ‖𝜉𝑖 ‖21 = 1 and therefore

a.s.

𝑑 ∑︁ [︀ ]︀ E ‖𝜉𝑖 ‖21 = 𝑑.

(94)

𝑖=1

Let 𝑆𝑑 :=

∑︀𝑑

𝑖=1 𝜉𝑖 = (𝜀1 , . . . , 𝜀𝑑 ). Since |𝜀𝑖 | = 1 a.s., we have deterministically

‖𝑆𝑑 ‖1 =

𝑑 ∑︁

|𝜀𝑖 | = 𝑑,

(95)

𝑖=1

and consequently [︀ ]︀ E ‖𝑆𝑑 ‖21 = 𝑑2 .

(96)

On the other hand, since (R𝑑 , ‖ · ‖1 ) is assumed 𝜅𝑉 -regular and {𝜉𝑖 } is a martingale-difference sequence, the second-moment inequality of Lemma 15 yields 𝑑 ∑︁ [︀ ]︀ [︀ ]︀ E ‖𝑆𝑑 ‖21 ≤ 𝜅𝑉 E ‖𝜉𝑖 ‖21 .

(97)

𝑖=1

Combining (97) with (94) and (96) gives 𝑑2 ≤ 𝜅𝑉 𝑑, which implies 𝜅𝑉 ≥ 𝑑. This is exactly (93). L.2

ℓ𝑑𝑝 Spaces

We now determine essentially tight regularity constants for ℓ𝑑𝑝 normed spaces (𝑉, ‖ · ‖𝑉 ) with 𝑉 = R𝑑 and ‖·‖𝑉 = ‖·‖𝑝 , distinguishing between the regime 𝑝 ≥ 2 (where 𝜅𝑉 can be as small as logarithmic in 𝑑) and the regime 𝑝 < 2 (where 𝜅𝑉 necessarily grows polynomially in 𝑑). Proposition 12 (ℓ𝑑𝑝 spaces). Fix 𝑑 ∈ N and 𝑝 ∈ [1, ∞]. Let 𝑉 := R𝑑 and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑝 , where ‖𝑥‖𝑝 :=

𝑑 (︁ ∑︁

|𝑥𝑖 |𝑝

)︁1/𝑝

(1 ≤ 𝑝 < ∞),

‖𝑥‖∞ := max |𝑥𝑖 |. 1≤𝑖≤𝑑

𝑖=1

Then the space (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular in the sense of Definition 1, with admissible regularity parameter 𝜅𝑉 := 𝜅𝑝 (𝑑) given as follows: 1. Let 2 ≤ 𝑝 ≤ ∞. Then 2

2

𝜅𝑉 := 𝜅𝑝 (𝑑) = min (𝜌 − 1) 𝑑 𝜌 − 𝑝 . 2≤𝜌≤𝑝 𝜌<∞

When 𝑝 = ∞, we use 1/𝑝 = 0 and the auxiliary exponent ranges over [2, ∞). Furthermore, if 𝑑 ≥ 8, then 𝜅𝑉 := 𝜅𝑝 (𝑑) ≤ min{𝑝 − 1, 2𝑒2 ln(𝑑)}, with the corresponding auxiliary norm ‖ · ‖𝑉,+ := ‖ · ‖𝜌* ,

{︁ }︁ √︀ 𝜌* := min 𝑝, ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑 . 75

2. Let 1 ≤ 𝑝 < 2. Then 2

𝜅𝑉 := 𝜅𝑝 (𝑑) = 𝑑 𝑝 −1 , with the corresponding auxiliary norm 1

1

‖𝑥‖𝑉,+ := 𝑑 𝑝 − 2 ‖𝑥‖2 ,

𝑥 ∈ R𝑑 .

Proof. To simplify notation throughout this proof, we write ‖𝑥‖ := ‖𝑥‖𝑉 = ‖𝑥‖𝑝 , 𝜅 := 𝜅𝑉 . Whenever an auxiliary norm is introduced, we denote it by ‖𝑥‖+ := ‖𝑥‖𝑉,+ , 𝜅+ := 𝜅𝑉,+ . The letters 𝑥, ℎ ∈ 𝑉 = R𝑑 denote generic vectors; this is only a local renaming of the dummy variables in Definition 1. 1. We start with Case 1: 2 ≤ 𝑝 ≤ +∞. The proof proceeds in three steps. First, we establish that for any 2 ≤ 𝜌 < ∞, the space (R𝑑 , ‖·‖𝜌 ) is (𝜌 − 1)-smooth. Second, we use this result with a standard norm equivalence inequality to derive the regularity constant for (R𝑑 , ‖·‖𝑝 ). Lastly, we provide a bound depending only on 𝑝 and 𝑑. Step 1: Smoothness of (R𝑑 , ‖ · ‖𝜌 ) for 2 ≤ 𝜌 < ∞. Fix 𝜌 ∈ [2, 𝑝] with 𝜌 < ∞ and use ‖𝑥‖+ := ‖𝑥‖𝜌 as the candidate auxiliary norm on 𝑉 . Define 1 1 𝜑+ (𝑥) := ‖𝑥‖2+ = ‖𝑥‖2𝜌 , 𝜓(𝑥) := 2𝜑+ (𝑥) = ‖𝑥‖2𝜌 . 2 2 To prove that (𝑉, ‖ · ‖+ ) is 𝜅+ -smooth with 𝜅+ = 𝜌 − 1, it suffices to show that ⟨︀ 2 ⟩︀ ∇ 𝜓(𝑥)ℎ, ℎ ≤ 2(𝜌 − 1)‖ℎ‖2+ = 2(𝜌 − 1)‖ℎ‖2𝜌 for all 𝑥 ̸= 0 and all ℎ ∈ 𝑉 . The case 𝜌 = 2 is Euclidean, hence 1-smooth, so we assume 𝜌 > 2. Since 𝜌 > 2, the function 𝜓 is twice continuously differentiable on R𝑑 ∖ {0}. For 𝑥 ̸= 0, its gradient is (︃ )︃ 𝜌2 −1 ∑︁ (︀ )︀⊤ 𝜌 ∇𝜓(𝑥) = 2 |𝑥𝑖 | · |𝑥1 |𝜌−1 sign(𝑥1 ), . . . , |𝑥𝑑 |𝜌−1 sign(𝑥𝑑 ) . 𝑖

At the origin, |𝜓(ℎ) − 𝜓(0)| = ‖ℎ‖𝜌 −→ 0 as ℎ → 0, ℎ ̸= 0, ‖ℎ‖𝜌 so 𝜓 is differentiable at 0 with ∇𝜓(0) = 0. Moreover, the gradient formula gives (︃ )︃ 𝜌2 −1 (︃ )︃ 𝜌−1 𝜌 ∑︁ ∑︁ 𝜌 𝜌 ‖∇𝜓(𝑥)‖𝜌/(𝜌−1) = 2 |𝑥𝑖 | |𝑥𝑖 | 𝑖

𝑖

= 2‖𝑥‖𝜌 −→ 0

as 𝑥 → 0, 𝑥 ̸= 0.

Thus the gradient is also continuous at the origin, and 𝜓 ∈ 𝐶 1 (R𝑑 ). Since 𝜓(𝑥) is homogeneous of degree 2, it suffices to verify the bound ⟨︀ for vectors ⟩︀ on the unit sphere, i.e., where ‖𝑥‖𝜌 = 1. For such an 𝑥, the Hessian action ∇2 𝜓(𝑥)ℎ, ℎ is: (︃ )︃ 𝜌2 −2 (︃ )︃2 ∑︁ ∑︁ ⟨︀ 2 ⟩︀ 𝜌 𝜌−1 ∇ 𝜓(𝑥)ℎ, ℎ = 2 (2 − 𝜌) |𝑥𝑖 | |𝑥𝑖 | sign(𝑥𝑖 )ℎ𝑖 ⏟ ⏞ 𝑖

≤0

𝑖

2

(︃ 𝜌 −1 )︃ ∑︁ ∑︁ +2 |𝑥𝑖 |𝜌 (𝜌 − 1)|𝑥𝑖 |𝜌−2 ℎ2𝑖 . 𝑖

⏟

𝑖

⏞

=1

76

Since 𝜌 > 2, the first term is non-positive. Dropping it and substituting the upper bound:

∑︀

𝑖 |𝑥𝑖 |

𝜌

= 1 gives

𝑑 ∑︁ ⟨︀ 2 ⟩︀ ∇ 𝜓(𝑥)ℎ, ℎ ≤ 2(𝜌 − 1) |𝑥𝑖 |𝜌−2 ℎ2𝑖 . 𝑖=1

We now apply Hölder’s inequality to the sum on the right-hand side. Recall that for any two vectors 𝑎, 𝑏 ∈ R𝑑 and conjugate exponents 𝑞, 𝑟 > 1 (i.e., 1𝑞 + 1𝑟 = 1), the inequality (︁∑︀ )︁1/𝑞 (︁∑︀ )︁1/𝑟 ∑︀𝑑 𝑑 𝑑 𝑞 𝑟 states: 𝑖=1 |𝑎𝑖 𝑏𝑖 | ≤ |𝑎 | |𝑏 | . In our case, we set 𝑎𝑖 = |𝑥𝑖 |𝜌−2 𝑖 𝑖 𝑖=1 𝑖=1 and 𝑏𝑖 = ℎ2𝑖 . Since both terms are non-negative, the absolute value on the left-hand side 𝜌 is not needed. The conjugate exponents are 𝑞 = 𝜌−2 and 𝑟 = 𝜌2 . Applying the inequality yields: (︃ 𝑑 )︃1/𝑞 (︃ 𝑑 )︃1/𝑟 𝑑 ∑︁ ∑︁ (︀ ∑︁ (︀ )︀𝑟 )︀ 𝜌−2 2 𝜌−2 𝑞 2 |𝑥𝑖 | ℎ𝑖 ≤ |𝑥𝑖 | ℎ𝑖 𝑖=1

𝑖=1

=

(︃ 𝑑 ∑︁

𝑖=1

|𝑥𝑖 |𝜌

)︃(𝜌−2)/𝜌 (︃ 𝑑 ∑︁

𝑖=1

=

(︁

𝜌 ‖𝑥‖𝜌

)︃2/𝜌 |ℎ𝑖 |𝜌

𝑖=1

)︁(𝜌−2)/𝜌 (︁

𝜌 ‖ℎ‖𝜌

)︁2/𝜌

𝜌−2

2

= ‖𝑥‖𝜌

‖ℎ‖𝜌 .

∑︀𝑑 2 Since we assumed ‖𝑥‖𝜌 = 1, this gives 𝑖=1 |𝑥𝑖 |𝜌−2 ℎ2𝑖 ≤ ‖ℎ‖𝜌 . Substituting back yields: ⟨︀ 2 ⟩︀ 2 ∇ 𝜓(𝑥)ℎ, ℎ ≤ 2(𝜌 − 1) ‖ℎ‖𝜌 = 2(𝜌 − 1)‖ℎ‖2+ . This confirms that the Hessian bound holds for every 𝑥 ̸= 0. To obtain the global smoothness inequality, fix 𝑥, ℎ ∈ R𝑑 . The case ℎ = 0 is immediate, so assume ℎ ̸= 0 and define 𝐹 (𝑡) := 𝜓(𝑥 + 𝑡ℎ) for 𝑡 ∈ [0, 1]. Then 𝐹 ∈ 𝐶 1 ([0, 1]), and whenever 𝑥 + 𝑡ℎ ̸= 0, ⟨︀ ⟩︀ 𝐹 ′′ (𝑡) = ∇2 𝜓(𝑥 + 𝑡ℎ)ℎ, ℎ ≤ 2(𝜌 − 1)‖ℎ‖2𝜌 . The segment can pass through the origin at at most one parameter value 𝑡0 . Integrating this bound on intervals avoiding 𝑡0 , taking limits using continuity of 𝐹 ′ , and adding the two inequalities when the interval crosses 𝑡0 gives 𝐹 ′ (𝑡) − 𝐹 ′ (0) ≤ 2(𝜌 − 1)‖ℎ‖2𝜌 𝑡,

𝑡 ∈ [0, 1].

Integrating once more yields ∫︁ 1 𝜓(𝑥 + ℎ) − 𝜓(𝑥) =

𝐹 ′ (𝑡) 𝑑𝑡

0

≤ ⟨∇𝜓(𝑥), ℎ⟩ + (𝜌 − 1)‖ℎ‖2𝜌 . Since 𝜑+ = 𝜓/2, this is equivalent to 𝜌−1 ‖ℎ‖2+ , 2 Therefore, the auxiliary norm ‖ · ‖+ = ‖ · ‖𝜌 is 𝜅+ -smooth with 𝜑+ (𝑥 + ℎ) ≤ 𝜑+ (𝑥) + ⟨∇𝜑+ (𝑥), ℎ⟩ +

𝑥, ℎ ∈ 𝑉.

𝜅+ = 𝜌 − 1. Step 2: From Smoothness to Regularity. To establish the regularity of (𝑉, ‖ · ‖) for 𝑝 ∈ [2, ∞], we use ‖ · ‖+ = ‖ · ‖𝜌 with 𝜌 ∈ [2, 𝑝] and 𝜌 < ∞ as the auxiliary smooth norm. The standard equivalence inequality for ℓ𝑝 -norms states that for any 𝑥 ∈ R𝑑 and 𝜌 ≤ 𝑝, 1

1

1

1

‖𝑥‖ = ‖𝑥‖𝑝 ≤ ‖𝑥‖𝜌 = ‖𝑥‖+ ≤ 𝑑 𝜌 − 𝑝 ‖𝑥‖𝑝 = 𝑑 𝜌 − 𝑝 ‖𝑥‖. Squaring yields 2

2

‖𝑥‖2 ≤ ‖𝑥‖2+ ≤ 𝑑 𝜌 − 𝑝 ‖𝑥‖2 , 77

which is exactly the compatibility chain required by Definition 1. From Step 1, the auxiliary norm ‖ · ‖+ is 𝜅+ -smooth with 𝜅+ = 𝜌 − 1. Therefore, Definition 1 gives 2

2

2

2

𝜅 = 𝜅+ 𝑑 𝜌 − 𝑝 = (𝜌 − 1)𝑑 𝜌 − 𝑝 . The tightest regularity constant supplied by this construction is found by minimizing this expression over all 𝜌 ∈ [2, 𝑝] with 𝜌 < ∞. This leads to the formula in the example statement. To find the tightest bound on the regularity constant, we must solve 𝜅𝑉 := 𝜅𝑝 (𝑑) = min 𝜅(𝜌), 2≤𝜌≤𝑝 𝜌<∞

2

2

𝜅(𝜌) := (𝜌 − 1)𝑑 𝜌 − 𝑝 .

(98)

The minimum is attained: for 𝑝 < ∞, the objective is continuous on [2, 𝑝]; for 𝑝 = ∞, it is continuous on [2, ∞) and tends to ∞ as 𝜌 → ∞. The term 𝑑−2/𝑝 is a constant with respect to 𝜌, so we can focus on minimizing the function 𝑔(𝜌) := (𝜌 − 1)𝑑2/𝜌 . Step 3: A 𝜌-Independent Bound on the Regularity Constant. We compute the first and second derivatives of 𝑔(𝜌) with respect to 𝜌. (︂ )︂ 2(𝜌 − 1) ln 𝑑 ′ 2/𝜌 𝑔 (𝜌) = 𝑑 1− . 𝜌2 4 ln 𝑑 𝑔 ′′ (𝜌) = 𝑑2/𝜌 4 ((𝜌 − 1) ln 𝑑 − 𝜌) . 𝜌 𝑔 ′′ (𝜌) is non-negative if and only if (ln 𝑑)(𝜌 − 1) − 𝜌 ≥ 0, which is equivalent to the 𝑑 condition 𝜌 ≥ lnln𝑑−1 . We show this holds for any 𝜌 ≥ 2 when 𝑑 ≥ 8. Since we assume 𝑑 ≥ 8, it follows that 𝑑 > 𝑒2 ≈ 7.39. This implies ln 𝑑 > 2. From ln 𝑑 > 2, we have 2(ln 𝑑 − 1) = 2 ln 𝑑 − 2 > ln 𝑑, which rearranges to: ln 𝑑 . ln 𝑑 − 1 At the same time, 𝑑 ≥ 8 > 𝑒 implies ln 𝑑 > 1, so ln 𝑑 − 1 > 0, which ensures that ln 𝑑 ln 𝑑−1 > 1. Combining these gives the sandwich inequality: 2>

2>

ln 𝑑 > 1. ln 𝑑 − 1

Therefore, for any 𝜌 ≥ 2, we have: 𝜌≥2>

ln 𝑑 . ln 𝑑 − 1

This confirms that 𝑔 ′′ (𝜌) ≥ 0, and thus 𝑔(𝜌) is convex for all 𝜌 ≥ 2 under the mild condition 𝑑 ≥ 8. Finding the Optimal 𝜌* . Since we have established that the function 𝑔(𝜌) is convex on our interval of interest for 𝑑 ≥ 8, we can find its global minimizer by setting the first derivative to zero: (︂ )︂ 2(ln 𝑑)(𝜌 − 1) ′ 2/𝜌 = 0. 𝑔 (𝜌) = 𝑑 1− 𝜌2 This requires the term in the parenthesis to be zero, which leads to a quadratic equation for 𝜌: 𝜌2 = 2(ln 𝑑)(𝜌 − 1) =⇒ 𝜌2 − (2 ln 𝑑)𝜌 + 2 ln 𝑑 = 0. 78

The solutions to this quadratic equation are: √︀ 𝜌* = ln 𝑑 ± (ln 𝑑)2 − 2 ln 𝑑. For the solutions to be real, the discriminant must be non-negative, which requires (ln 𝑑)2 − 2 ln 𝑑 ≥ 0, or ln 𝑑 ≥ 2. This condition, 𝑑 ≥ 𝑒2 ≈ 7.4, is consistent with our assumption that 𝑑 ≥ 8. The quadratic equation yields two roots. To determine which of these corresponds to a local minimum, we apply the second derivative test. Note that sign(𝑔 ′′ (𝜌)) = sign ((ln 𝑑)(𝜌 − 1) − 𝜌). At either critical point 𝜌* ∈ {𝜌− , 𝜌+ }, the first-order condition gives 𝜌*2 = 2(ln 𝑑)(𝜌* − 1). *2 We can therefore simplify the expression by substituting (ln 𝑑)(𝜌* − 1) = 𝜌2 . Plugging this into our sign-determining term: (ln 𝑑)(𝜌* − 1) − 𝜌* =

𝜌*2 𝜌* (𝜌* − 2) − 𝜌* = . 2 2

Therefore, the sign of the second derivative at a critical point 𝜌* is the same as the sign of 𝜌* − 2. Let us now analyze the location of the two roots relative to the value 2. We are working under the condition ln 𝑑 ≥ 2 (since 𝑑 ≥ 8 > 𝑒2 ). • For the smaller root, 𝜌− : We check if 𝜌− is greater or less than 2. √︀ ln 𝑑 − 2 > (ln 𝑑)2 − 2 ln 𝑑 (ln 𝑑 − 2)2 > (ln 𝑑)2 − 2 ln 𝑑 2

(squaring is valid as ln 𝑑 > 2)

2

(ln 𝑑) − 4 ln 𝑑 + 4 > (ln 𝑑) − 2 ln 𝑑 4 > 2 ln 𝑑 2 > ln 𝑑. This is a contradiction to our premise that ln 𝑑 ≥ 2. Therefore, the inequality must be reversed, which means 𝜌− ≤ 2. • For the larger root, 𝜌+ : Since ln 𝑑 ≥ 2, the term under the square root is non-negative. Therefore: √︀ 𝜌+ = ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑 ≥ ln 𝑑 ≥ 2. Now we apply our simplified second derivative test: • At 𝜌− , since 𝜌− ≤ 2, we have 𝜌− − 2 ≤ 0. This implies 𝑔 ′′ (𝜌− ) ≤ 0, so 𝜌− corresponds to a local maximum. • At 𝜌+ , since 𝜌+ ≥ 2, we have 𝜌+ −2 ≥ 0. This implies 𝑔 ′′ (𝜌+ ) ≥ 0, so 𝜌+ corresponds to a local minimum. Since 𝑔 is convex on [2, ∞), its minimizer on this interval is the larger root 𝜌+ . Analyzing the Location of the Minimum. The optimal regularity bound within this construction depends on where this unconstrained minimizer 𝜌* falls relative to the domain {𝜌 : 2 ≤ 𝜌 ≤ 𝑝, 𝜌 < ∞}. • If 𝜌* > 𝑝: This occurs when 𝑝 is small compared to ln 𝑑. Since 𝑔(𝜌) is convex, its minimum over [2, 𝑝] must occur at the boundary. As 𝑔 ′ (𝜌) < 0 for 𝜌 < 𝜌* , the function is decreasing on [2, 𝑝], so the minimum is achieved at the right endpoint, 𝜌 = 𝑝. In this case, the regularity constant is 𝜅(𝑝) = (𝑝 − 1)𝑑2/𝑝−2/𝑝 = 𝑝 − 1. • If 2 ≤ 𝜌* ≤ 𝑝: The minimum is achieved at the unconstrained minimizer 𝜌* . We now find an upper bound for the value of 𝜅(𝜌* ). 2

2

𝜅(𝜌* ) = (𝜌* − 1)𝑑 𝜌* − 𝑝 . 79

*

Since 𝜌* ≤ 𝑝 and 𝑑 ≥ 1, we have 𝑑−2/𝜌 ≤ 𝑑−2/𝑝 ≤ 1. Using the upper bound 𝑑−2/𝑝 ≤ 1 gives 2

𝜅(𝜌* ) ≤ (𝜌* − 1)𝑑 𝜌* . From the first-order optimality condition, we have the identity 2(ln 𝑑)(𝜌* − 1) = (𝜌* )2 . * 2 ) This allows us to substitute (𝜌* − 1) = (𝜌 2 ln 𝑑 . )︂ (︂ 2 2 ln 𝑑 (𝜌* )2 (𝜌* )2 . · 𝑑 𝜌* = exp 𝜅(𝜌* ) ≤ 2 ln 𝑑 2 ln 𝑑 𝜌* √︀ We know that ln 𝑑 ≤ 𝜌* = ln 𝑑 + (ln 𝑑)2 − 2 ln 𝑑 ≤ 2 ln 𝑑. Substituting these bounds: (︂ )︂ (2 ln 𝑑)2 2 ln 𝑑 𝜅(𝜌* ) ≤ exp = (2 ln 𝑑) · 𝑒2 . 2 ln 𝑑 ln 𝑑 For 𝑝 < ∞, combining the endpoint bound 𝑝 − 1 with the bound 2𝑒2 ln 𝑑 gives the estimate below. For 𝑝 = ∞, the finite minimizer 𝜌* gives the bound 2𝑒2 ln 𝑑 directly: 𝜅𝑝 (𝑑) = min 𝜅(𝜌) ≤ min{𝑝 − 1, 2𝑒2 (ln 𝑑)}. 2≤𝜌≤𝑝 𝜌<∞

2. We now proceed with Case 2: 1 ≤ 𝑝 < 2. We prove that (𝑉, ‖ · ‖) is 𝜅-regular with 2

𝜅 := 𝜅𝑉 = 𝜅𝑝 (𝑑) = 𝑑 𝑝 −1 .

(99)

In particular, the regularity constant is polynomial in 𝑑 for 𝑝 < 2. Fix 𝑝 ∈ [1, 2) and let 𝑑 ≥ 1. Define the auxiliary norm on 𝑉 by

Step 1: Construct a smooth auxiliary norm. 1

1

‖𝑥‖+ := ‖𝑥‖𝑉,+ := 𝑑 𝑝 − 2 ‖𝑥‖2 , 1

𝑥 ∈ R𝑑 .

(100)

1

Let 𝑎 := 𝑑 𝑝 − 2 > 0 and define 𝜑+ (𝑥) :=

𝑎2 1 2 2 ‖𝑥‖+ = ‖𝑥‖2 . 2 2

This is a quadratic function and is continuously differentiable on R𝑑 . Using the standard Euclidean inner product to identify 𝑉 with 𝑉 * , we obtain ∇𝜑+ (𝑥) = 𝑎2 𝑥. For any 𝑥, 𝑦 ∈ R𝑑 , expanding the square gives 𝑎2 2 ‖𝑥 + 𝑦‖2 2 )︁ 𝑎2 (︁ 2 2 = ‖𝑥‖2 + 2 ⟨𝑥, 𝑦⟩ + ‖𝑦‖2 2 ⟨︀ ⟩︀ 𝑎2 2 = 𝜑+ (𝑥) + 𝑎2 𝑥, 𝑦 + ‖𝑦‖2 2 1 2 = 𝜑+ (𝑥) + ⟨∇𝜑+ (𝑥), 𝑦⟩ + ‖𝑦‖+ . 2

𝜑+ (𝑥 + 𝑦) =

Thus the inequality in Definition 1(i) holds with equality for 𝜅+ = 1, so (𝑉, ‖·‖+ ) is 1-smooth. 80

We claim that for every 𝑥 ∈ R𝑑 ,

Step 2: Verify the compatibility inequalities.

1

1

‖𝑥‖2 ≤ ‖𝑥‖𝑝 ≤ 𝑑 𝑝 − 2 ‖𝑥‖2 .

(101)

Proof of the left inequality in (101). If 𝑥 = 0 it is trivial. Otherwise, let 𝑎𝑖 := |𝑥𝑖 |/‖𝑥‖𝑝 . Then ∑︀𝑑 𝑝 𝑝 2 𝑖=1 𝑎𝑖 = 1 and each 𝑎𝑖 ∈ [0, 1]. Since 𝑝 < 2 and 𝑎𝑖 ∈ [0, 1], we have 𝑎𝑖 ≤ 𝑎𝑖 . Hence 2

‖𝑥‖2 =

𝑑 ∑︁

|𝑥𝑖 |2

𝑖=1 2

= ‖𝑥‖𝑝

𝑑 ∑︁

𝑎2𝑖

𝑖=1 𝑑 ∑︁ 2 ≤ ‖𝑥‖𝑝 𝑎𝑝𝑖 𝑖=1 2

= ‖𝑥‖𝑝 , which yields ‖𝑥‖2 ≤ ‖𝑥‖𝑝 . Proof of the right inequality in (101). Let 𝑟 := 2/𝑝 > 1 and 𝑟′ := 2/2−𝑝 be its conjugate exponent. Using Hölder’s inequality, 𝑑 ∑︁

𝑑 ∑︁ |𝑥𝑖 | = (|𝑥𝑖 |2 )𝑝/2 · 1 𝑝

𝑖=1

𝑖=1

≤

(︃ 𝑑 ∑︁

|𝑥𝑖 |2

)︃𝑝/2 (︃ 𝑑 ∑︁

𝑖=1 𝑝 𝑝 = ‖𝑥‖2 𝑑1− 2 . 1

)︃1/𝑟′ 1𝑟

′

𝑖=1

1

Taking 𝑝-th roots gives ‖𝑥‖𝑝 ≤ 𝑑 𝑝 − 2 ‖𝑥‖2 . Combining the two inequalities proves (101). Now, by the definition (100) and (101), 1

1

‖𝑥‖𝑝 ≤ 𝑑 𝑝 − 2 ‖𝑥‖2 = ‖𝑥‖+ , 2

2

so ‖𝑥‖𝑝 ≤ ‖𝑥‖+ . Also, using ‖𝑥‖2 ≤ ‖𝑥‖𝑝 (the left inequality), 2

2

2

2

2

‖𝑥‖+ = 𝑑 𝑝 −1 ‖𝑥‖2 ≤ 𝑑 𝑝 −1 ‖𝑥‖𝑝 . Therefore, for all 𝑥 ∈ R𝑑 , 2

2

2

2

‖𝑥‖𝑝 ≤ ‖𝑥‖+ ≤ 𝑑 𝑝 −1 ‖𝑥‖𝑝 .

(102)

Step 3: Conclude 𝜅-regularity. We have shown that the auxiliary norm (𝑉, ‖ · ‖+ ) is 1-smooth, i.e., 𝜅+ = 1, and that (102) gives the compatibility chain 2

‖𝑥‖2 ≤ ‖𝑥‖2+ ≤ 𝑑 𝑝 −1 ‖𝑥‖2 . Therefore, by Definition 1, the space (𝑉, ‖ · ‖) is 𝜅-regular with 2

𝜅 = 𝑑 𝑝 −1 . Equivalently, 2

𝜅𝑉 = 𝜅𝑝 (𝑑) = 𝑑 𝑝 −1 . This establishes the upper bound in (99). 81

Remark 5 (Optimality of the Euclidean auxiliary norm for 𝑝 < 2 within the scaled ℓ𝜌 family). Fix 𝑑 ∈ N and 𝑝 ∈ [1, 2). Let 𝑉 := R𝑑 and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑝 . For every 𝜌 ∈ [2, ∞), consider the scaled auxiliary norm 1

1

‖𝑥‖𝑉,+,𝜌 := 𝑑 𝑝 − 𝜌 ‖𝑥‖𝜌 ,

𝑥 ∈ R𝑑 .

(103)

Then (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 (𝜌)-regular with respect to this auxiliary norm, where 2

2

𝜅𝑉 (𝜌) := (𝜌 − 1)𝑑 𝑝 − 𝜌 .

(104)

Moreover, the function 𝜌 ↦→ 𝜅𝑉 (𝜌) is minimized over 𝜌 ∈ [2, ∞) at 𝜌 = 2. Hence, within this family of scaled ℓ𝜌 auxiliary norms, the optimal choice is the Euclidean norm 1

1

‖𝑥‖𝑉,+ := 𝑑 𝑝 − 2 ‖𝑥‖2 . Proof. To simplify notation throughout this proof, we write ‖𝑥‖ := ‖𝑥‖𝑉 = ‖𝑥‖𝑝 ,

‖𝑥‖+,𝜌 := ‖𝑥‖𝑉,+,𝜌 .

Fix 𝜌 ∈ [2, ∞). Since 𝑝 < 𝜌, the standard norm equivalence inequalities for ℓ𝑝 spaces give 1

1

‖𝑥‖𝜌 ≤ ‖𝑥‖ ≤ 𝑑 𝑝 − 𝜌 ‖𝑥‖𝜌 ,

∀𝑥 ∈ R𝑑 .

Substituting the definition (103), we obtain 1

1

2

2

‖𝑥‖ ≤ ‖𝑥‖+,𝜌 ≤ 𝑑 𝑝 − 𝜌 ‖𝑥‖,

∀𝑥 ∈ R𝑑 .

(105)

Squaring gives the compatibility chain ‖𝑥‖2 ≤ ‖𝑥‖2+,𝜌 ≤ 𝑑 𝑝 − 𝜌 ‖𝑥‖2 ,

∀𝑥 ∈ R𝑑 .

By Step 1 in the proof of Proposition 12, the norm ‖ · ‖𝜌 is (𝜌 − 1)-smooth for every 𝜌 ∈ [2, ∞). Since scaling a norm by a positive constant does not change its smoothness parameter, the scaled norm ‖ · ‖+,𝜌 is also (𝜌 − 1)-smooth. Hence (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 (𝜌)-regular with respect to the auxiliary norm ‖ · ‖𝑉,+,𝜌 , where 2

2

𝜅𝑉 (𝜌) = (𝜌 − 1)𝑑 𝑝 − 𝜌 . To identify the best choice within this family, compare an arbitrary 𝜌 > 2 against 𝜌 = 2: 2

2

2 𝜅𝑉 (𝜌) (𝜌 − 1)𝑑 𝑝 − 𝜌 = = (𝜌 − 1)𝑑1− 𝜌 . 2 −1 𝜅𝑉 (2) 1 · 𝑑𝑝 2

Since 𝜌 > 2, we have 𝜌 − 1 > 1 and 1 − 𝜌2 > 0. Also, for every 𝑑 ≥ 1, we have 𝑑1− 𝜌 ≥ 1. Therefore, 𝜅𝑉 (𝜌) > 1, 𝜅𝑉 (2)

∀𝜌 > 2.

Thus 𝜅𝑉 (𝜌) is minimized at 𝜌 = 2, which yields the Euclidean auxiliary norm 1

1

‖𝑥‖𝑉,+ := 𝑑 𝑝 − 2 ‖𝑥‖2 .

We now complement the upper bounds from Proposition 12 with matching lower bounds on the admissible regularity parameter 𝜅𝑉 for ℓ𝑑𝑝 spaces. For 1 ≤ 𝑝 < 2, the optimal parameter is 𝑑2/𝑝−1 . For 2 ≤ 𝑝 ≤ ∞, its optimal order is min{𝑝, ln(𝑒𝑑)}, uniformly in 𝑑 ≥ 1; see Remark 7. Consequently, the optimal parameter remains bounded as 𝑑 varies with a fixed finite 𝑝 ≥ 2, and equals 1 at 𝑝 = 2. The logarithmic branch of the minimum corresponds to 𝑝 ≥ ln(𝑒𝑑), including 𝑝 = ∞, where logarithmic growth is unavoidable up to absolute constants. Proposition 13 (Lower bounds for ℓ𝑑𝑝 spaces). Fix 𝑑 ∈ N and 𝑝 ∈ [1, ∞]. Let 𝑉 := R𝑑 and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑝 . If the space (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular, then the following hold: 82

1. If 1 ≤ 𝑝 < 2, then 2

𝜅𝑉 ≥ 𝑑 𝑝 −1 .

(106)

1 min{𝑝 − 1, ln 𝑑}. 8

(107)

2. If 2 ≤ 𝑝 ≤ ∞ and 𝑑 ≥ 8, then 𝜅𝑉 ≥

Proof. To simplify notation throughout this proof, we write ‖𝑥‖ := ‖𝑥‖𝑉 = ‖𝑥‖𝑝 ,

𝜅 := 𝜅𝑉 .

1. Case 1: 1 ≤ 𝑝 < 2. Let {𝜀𝑖 }𝑑𝑖=1 be i.i.d. Rademacher random variables, and let {𝑒𝑖 }𝑑𝑖=1 be the standard basis of R𝑑 . Define ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ),

𝜉𝑖 := 𝜀𝑖 𝑒𝑖 ,

𝑖 = 1, . . . , 𝑑.

Then {𝜉𝑖 }𝑑𝑖=1 is an {ℱ𝑖 }-martingale-difference sequence: it is ℱ𝑖 -measurable and E [𝜉𝑖 | ℱ𝑖−1 ] = 𝑒𝑖 E [𝜀𝑖 | ℱ𝑖−1 ] = 0

a.s.

Moreover, ‖𝜉𝑖 ‖ = ‖𝑒𝑖 ‖𝑝 = 1,

𝑖 = 1, . . . , 𝑑.

Let 𝑆𝑑 :=

𝑑 ∑︁

𝜉𝑖 = (𝜀1 , . . . , 𝜀𝑑 ).

𝑖=1

Since |𝜀𝑖 | = 1 almost surely, we have deterministically ‖𝑆𝑑 ‖ = ‖𝑆𝑑 ‖𝑝 =

(︃ 𝑑 ∑︁

)︃1/𝑝 𝑝

|𝜀𝑖 |

= 𝑑1/𝑝 .

𝑖=1

Hence 𝑑 𝑑 ∑︁ [︀ ]︀ ∑︁ E ‖𝜉𝑖 ‖2 = 1 = 𝑑.

[︀ ]︀ E ‖𝑆𝑑 ‖2 = 𝑑2/𝑝 ,

𝑖=1

𝑖=1

Since (𝑉, ‖ · ‖𝑉 ) is 𝜅-regular and {𝜉𝑖 } is a martingale-difference sequence, Lemma 15 yields 𝑑 ∑︁ [︀ ]︀ [︀ ]︀ E ‖𝑆𝑑 ‖2 ≤ 𝜅 E ‖𝜉𝑖 ‖2 = 𝜅𝑑. 𝑖=1

[︀ ]︀ Combining with E ‖𝑆𝑑 ‖2 = 𝑑2/𝑝 gives 𝑑2/𝑝 ≤ 𝜅𝑑, and therefore 2

𝜅 ≥ 𝑑 𝑝 −1 . This proves (106). 2. Case 2: 2 ≤ 𝑝 ≤ ∞ and 𝑑 ≥ 8. We first treat the subcase 2 ≤ 𝑝 < ∞. 83

Step 1: Parameter choice and an explicit family of vectors.

Let

𝑛 := ⌊min{𝑝, log2 𝑑}⌋ . Since 𝑑 ≥ 8, we have log2 𝑑 ≥ 3 and hence 𝑛 ≥ 2. Define Ω := {±1}𝑛 ,

|Ω| = 2𝑛 ≤ 𝑑.

Fix an arbitrary enumeration 𝑛

(𝑗)

Ω = {𝜔 (1) , 𝜔 (2) , . . . , 𝜔 (2 ) },

𝜔 (𝑗) = (𝜔1 , . . . , 𝜔𝑛(𝑗) ) ∈ {±1}𝑛 .

We identify the first 2𝑛 coordinates of R𝑑 with the index set {1, . . . , 2𝑛 } and set all remaining coordinates equal to 0. For each 𝑖 ∈ {1, . . . , 𝑛}, define a vector 𝑢𝑖 ∈ R𝑑 by (𝑗)

(𝑢𝑖 )𝑗 := 2−𝑛/𝑝 𝜔𝑖

for 𝑗 = 1, . . . , 2𝑛 ,

for 𝑗 = 2𝑛 + 1, . . . , 𝑑.

(𝑢𝑖 )𝑗 := 0

Then each 𝑢𝑖 has exactly 2𝑛 nonzero entries, each equal to ±2−𝑛/𝑝 , and therefore 𝑝

‖𝑢𝑖 ‖

= ‖𝑢𝑖 ‖𝑝𝑝 =

2𝑛 ⃒ ⃒ ∑︁ ⃒ −𝑛/𝑝 (𝑗) ⃒𝑝 𝜔𝑖 ⃒ = 2𝑛 · 2−𝑛 = 1. ⃒2 𝑗=1

Hence ‖𝑢𝑖 ‖ = 1,

𝑖 = 1, . . . , 𝑛.

Step 2: A martingale-difference sequence. variables, define ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ),

Let {𝜀𝑖 }𝑛𝑖=1 be i.i.d. Rademacher random

𝜉𝑖 := 𝜀𝑖 𝑢𝑖 ,

𝑖 = 1, . . . , 𝑛.

Then {𝜉𝑖 }𝑛𝑖=1 is an {ℱ𝑖 }-martingale-difference sequence and ‖𝜉𝑖 ‖ = ‖𝑢𝑖 ‖ = 1,

𝑖 = 1, . . . , 𝑛,

so 𝑛 ∑︁ [︀ ]︀ E ‖𝜉𝑖 ‖2 = 𝑛. 𝑖=1

[︀ ]︀ Step 3: Lower bound on E ‖𝑆𝑛 ‖2 .

Let 𝑆𝑛 :=

𝑛 ∑︁

𝜉𝑖 .

𝑖=1

For each coordinate 𝑗 ∈ {1, . . . , 2𝑛 }, we have (𝑆𝑛 )𝑗 =

𝑛 ∑︁

𝜀𝑖 (𝑢𝑖 )𝑗 = 2−𝑛/𝑝

𝑖=1

𝑛 ∑︁

(𝑗)

𝜀𝑖 𝜔𝑖 .

𝑖=1

Therefore, ‖𝑆𝑛 ‖𝑝 = ‖𝑆𝑛 ‖𝑝𝑝 ⃒ ⃒𝑝 2𝑛 ⃒ 𝑛 ⃒ ∑︁ ⃒ −𝑛/𝑝 ∑︁ (𝑗) ⃒ = 𝜀𝑖 𝜔𝑖 ⃒ ⃒2 ⃒ ⃒ 𝑗=1 𝑖=1 ⃒ 𝑛 ⃒ 2 ⃒∑︁ 𝑛 ⃒𝑝 ∑︁ ⃒ (𝑗) ⃒ = 2−𝑛 𝜀𝑖 𝜔𝑖 ⃒ ⃒ ⃒ ⃒ 𝑗=1 𝑖=1 ⃒𝑝 ]︃ [︃⃒ 𝑛 ⃒∑︁ ⃒ ⃒ ⃒ = E𝜔∼Unif(Ω) ⃒ 𝜀 𝑖 𝜔𝑖 ⃒ , ⃒ ⃒ 𝑖=1

84

𝑛

where the last equality holds because {𝜔 (𝑗) }2𝑗=1 is merely an enumeration of Ω. Since 𝜔 ∼ Unif(Ω) is uniform on {±1}𝑛 , for every fixed realization of 𝜀 = (𝜀1 , . . . , 𝜀𝑛 ) ∈ {±1}𝑛 we have ⃒𝑝 ]︃ ⃒ ]︃ [︃⃒ 𝑛 [︃⃒ 𝑛 ⃒∑︁ ⃒ ⃒∑︁ ⃒𝑝 ⃒ ⃒ ⃒ ⃒ E𝜔∼Unif(Ω) ⃒ 𝜀𝑖 𝜔𝑖 ⃒ = E𝜔∼Unif(Ω) ⃒ 𝜔𝑖 ⃒ , ⃒ ⃒ ⃒ ⃒ 𝑖=1

𝑖=1

because the coordinatewise sign-flip map 𝜔 ↦→ (𝜀1 𝜔1 , . . . , 𝜀𝑛 𝜔𝑛 ) is a bijection of Ω and hence preserves the uniform law. Therefore, ⃒ ]︃)︃ (︃ [︃⃒ 𝑛 ⃒∑︁ ⃒𝑝 1/𝑝 ⃒ ⃒ ‖𝑆𝑛 ‖ = E𝜔∼Unif(Ω) ⃒ 𝜔𝑖 ⃒ . ⃒ ⃒ 𝑖=1

In particular, ‖𝑆𝑛 ‖ is deterministic, and thus ⃒ ]︃)︃ (︃ [︃⃒ 𝑛 ⃒∑︁ ⃒𝑝 2/𝑝 [︀ ]︀ ⃒ ⃒ E ‖𝑆𝑛 ‖2 = ‖𝑆𝑛 ‖2 = E𝜔∼Unif(Ω) ⃒ 𝜔𝑖 ⃒ . ⃒ ⃒ 𝑖=1

We now lower bound this quantity using a single atom of the distribution. Let 𝑛 ∑︁ 𝑍𝑛 := 𝜔𝑖 , 𝜔 ∼ Unif(Ω). 𝑖=1 −𝑛

Since Prob{𝑍𝑛 = 𝑛} = 2 , we obtain E𝜔∼Unif(Ω) [|𝑍𝑛 |𝑝 ] ≥ 𝑛𝑝 Prob{𝑍𝑛 = 𝑛} = 𝑛𝑝 2−𝑛 . Therefore, (︀ )︀1/𝑝 ‖𝑆𝑛 ‖ = E𝜔∼Unif(Ω) [|𝑍𝑛 |𝑝 ] ≥ 𝑛 · 2−𝑛/𝑝 . By construction 𝑛 ≤ 𝑝, hence 2−𝑛/𝑝 ≥ 1/2, and consequently [︀ ]︀ 𝑛2 E ‖𝑆𝑛 ‖2 = ‖𝑆𝑛 ‖2 ≥ . 4 Step 4: Conclude via the second-moment bound. Up to this point, all expectations involving 𝜔 were taken with respect to the auxiliary randomness 𝜔 ∼ Unif(Ω). We now switch back to the original probability space on which the martingale-difference sequence {𝜉𝑖 }𝑛𝑖=1 (equivalently, the Rademacher signs {𝜀𝑖 }𝑛𝑖=1 ) is defined. In particular, in the remainder of the proof, E [·] denotes expectation with respect to the randomness of (𝜀1 , . . . , 𝜀𝑛 ). Since (𝑉, ‖ · ‖𝑉 ) is 𝜅-regular and {𝜉𝑖 } is a martingale-difference sequence, Lemma 15 yields 𝑛 ∑︁ [︀ ]︀ [︀ ]︀ E ‖𝑆𝑛 ‖2 ≤ 𝜅 E ‖𝜉𝑖 ‖2 = 𝜅𝑛. 𝑖=1

[︀ ]︀ 2 Combining this with the lower bound E ‖𝑆𝑛 ‖2 ≥ 𝑛 /4 gives 𝑛 𝜅≥ . 4 Finally, since 𝑛 = ⌊min{𝑝, log2 𝑑}⌋ and 𝑑 ≥ 8, we have 𝑛 ≥ min{𝑝 − 1, log2 𝑑 − 1} 1 ≥ min{𝑝 − 1, log2 𝑑} 2 1 ≥ min{𝑝 − 1, ln 𝑑}. 2 Therefore, 𝑛 1 𝜅 ≥ ≥ min{𝑝 − 1, ln 𝑑}, 4 8 which proves (107) for 2 ≤ 𝑝 < ∞. 85

Subcase 𝑝 = ∞.

Let 𝑛 := ⌊log2 𝑑⌋, Ω := {±1}𝑛 , |Ω| = 2𝑛 ≤ 𝑑. Fix an enumeration 𝑛 (𝑗) Ω = {𝜔 (1) , . . . , 𝜔 (2 ) }, 𝜔 (𝑗) = (𝜔1 , . . . , 𝜔𝑛(𝑗) ) ∈ {±1}𝑛 .

As before, we identify the first 2𝑛 coordinates of R𝑑 with the index set {1, . . . , 2𝑛 } and set all remaining coordinates equal to 0. For each 𝑖 ∈ {1, . . . , 𝑛}, define 𝑢𝑖 ∈ R𝑑 by (𝑗)

(𝑢𝑖 )𝑗 := 𝜔𝑖 for 𝑗 = 1, . . . , 2𝑛 , (𝑢𝑖 )𝑗 := 0 for 𝑗 = 2𝑛 + 1, . . . , 𝑑. Then every nonzero entry of 𝑢𝑖 equals ±1, and hence ‖𝑢𝑖 ‖ = ‖𝑢𝑖 ‖∞ = 1, 𝑖 = 1, . . . , 𝑛. Let {𝜀𝑖 }𝑛𝑖=1 be i.i.d. Rademacher random variables and define ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ), 𝜉𝑖 := 𝜀𝑖 𝑢𝑖 , 𝑖 = 1, . . . , 𝑛. Then {𝜉𝑖 }𝑛𝑖=1 is an {ℱ𝑖 }-martingale-difference sequence and 𝑛 𝑛 ∑︁ [︀ ]︀ ∑︁ [︀ ]︀ E ‖𝜉𝑖 ‖2 = E ‖𝜉𝑖 ‖2∞ = 𝑛. 𝑖=1

𝑖=1

Now consider 𝑆𝑛 :=

𝑛 ∑︁

𝜉𝑖 .

𝑖=1

For each 𝑗 ∈ {1, . . . , 2𝑛 }, we have (𝑆𝑛 )𝑗 =

𝑛 ∑︁

𝜀𝑖 (𝑢𝑖 )𝑗 =

𝑖=1

𝑛 ∑︁

(𝑗)

𝜀𝑖 𝜔𝑖 .

𝑖=1

Hence ⃒ 𝑛 ⃒ ⃒∑︁ ⃒ ⃒ ⃒ 𝜀𝑖 𝜔𝑖 ⃒ . ‖𝑆𝑛 ‖ = ‖𝑆𝑛 ‖∞ = max 𝑛 |(𝑆𝑛 )𝑗 | = max ⃒ 𝜔∈Ω ⃒ 1≤𝑗≤2 ⃒ 𝑖=1

For any fixed realization of 𝜀 = (𝜀1 , . . . , 𝜀𝑛 ) ∈ {±1}𝑛 , we may choose 𝜔 = 𝜀. Then 𝑛 𝑛 ∑︁ ∑︁ 𝜀𝑖 𝜔𝑖 = 𝜀2𝑖 = 𝑛, 𝑖=1

𝑖=1

so the maximum is at least 𝑛. On the other hand, for every 𝜔 ∈ {±1}𝑛 , ⃒ ⃒ 𝑛 𝑛 ⃒∑︁ ⃒ ∑︁ ⃒ ⃒ 𝜀 𝜔 ≤ |𝜀𝑖 𝜔𝑖 | = 𝑛, ⃒ 𝑖 𝑖⃒ ⃒ ⃒ 𝑖=1

𝑖=1

so the maximum is also at most 𝑛. Therefore, ‖𝑆𝑛 ‖ = 𝑛 deterministically, and hence

[︀ ]︀ E ‖𝑆𝑛 ‖2 = 𝑛2 .

Finally, since (𝑉, ‖·‖𝑉 ) is 𝜅-regular and {𝜉𝑖 } is a martingale-difference sequence, Lemma 15 yields 𝑛 ∑︁ [︀ ]︀ [︀ ]︀ E ‖𝑆𝑛 ‖2 ≤ 𝜅 E ‖𝜉𝑖 ‖2 = 𝜅𝑛. 𝑖=1

[︀ ]︀ Combining with E ‖𝑆𝑛 ‖2 = 𝑛2 gives 𝜅 ≥ 𝑛. Since 𝑛 = ⌊log2 𝑑⌋ ≥ 12 ln 𝑑 for 𝑑 ≥ 8, we obtain 1 ln 𝑑, 2 which is stronger than (107) in the case 𝑝 = ∞, because min{𝑝 − 1, ln 𝑑} = ln 𝑑 when 𝑝 = ∞. 𝜅≥

86

Remark 6 (Prior work). Dümbgen et al. [2010, Example 1.2, p. 139] give a logarithmic lower bound for the second-moment constant in ℓ𝑑∞ ; their Lemma 3.6 (p. 146) gives corresponding bounds for the squared type-2 constant. These results concern moment inequalities, and imply lower bounds on our regularity parameter through Lemma 15. For 2 ≤ 𝑝 ≤ ∞, the general ℓ𝑑𝑝 order follows with an additional argument. Let 𝐾𝑝 (𝑑) be the smallest constant such that [︃ ]︃ ⃦∑︁ ⃦2 ∑︁ [︀ ]︀ ⃦ ⃦ E ⃦ 𝑋𝑖 ⃦ ≤ 𝐾𝑝 (𝑑) E ‖𝑋𝑖 ‖2𝑝 𝑝

𝑖

𝑖

for every finite sequence of independent, mean-zero random vectors in R𝑑 with finite second moments. The cited results, together with 𝐾∞ (𝑚) ≥ 1, give 𝐾∞ (𝑚) ≥ 𝑐 ln(𝑒𝑚) for all 𝑚 ≥ 1 and an absolute constant 𝑐 > 0. Embedding R𝑚 as a coordinate subspace of R𝑑 and using ‖𝑧‖∞ ≤ ‖𝑧‖𝑝 ≤ 𝑚1/𝑝 ‖𝑧‖∞ gives, for every 1 ≤ 𝑚 ≤ 𝑑, 𝜅𝑉 ≥ 𝐾𝑝 (𝑑) ≥ 𝐾𝑝 (𝑚) ≥ 𝑚−2/𝑝 𝐾∞ (𝑚) ≥ 𝑐 𝑚−2/𝑝 ln(𝑒𝑚). For finite 𝑝 ≥ 2, choose 𝑚 = min{𝑑, ⌊𝑒𝑝 ⌋}; then 𝑚2/𝑝 ≤ 𝑒2 and ln(𝑒𝑚) ≥ min{𝑝, ln(𝑒𝑑)}. For 𝑝 = ∞, take 𝑚 = 𝑑 and use 𝑚−2/𝑝 = 1. Thus the lower-bound order min{𝑝, ln(𝑒𝑑)} follows, with an unspecified absolute constant. The explicit factor 1/8 in (107) is supplied by our direct proof above. That proof was obtained independently, before consulting this reference; we include it for completeness and to keep the exposition self-contained within our framework. L.3

Spaces with Schatten-𝑝 Norms on R𝑚×𝑛

We next record the analogous regularity bounds for Schatten-𝑝 norms on matrix spaces. Since the Schatten-𝑝 norm is simply the ℓ𝑝 norm of the singular-value vector, the behavior closely parallels the vector case: for 𝑝 ≥ 2, one may take a regularity parameter that grows at most logarithmically in the effective dimension 𝑟 := min{𝑚, 𝑛}, whereas for 𝑝 < 2, the natural admissible parameter grows polynomially in 𝑟. Proposition 14 (Schatten-𝑝 norms for matrices). Fix 𝑚, 𝑛 ∈ N and let 𝑟 := min{𝑚, 𝑛}. Let 𝑉 := R𝑚×𝑛 and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑆𝑝 , where (︃ 𝑟 )︃1/𝑝 ∑︁ 𝑝 ‖𝑋‖𝑆𝑝 := ‖𝜎(𝑋)‖𝑝 = 𝜎𝑖 (𝑋) , 𝑝 ∈ [1, ∞), 𝑖=1

and ‖𝑋‖𝑆∞ := max 𝜎𝑖 (𝑋). 1≤𝑖≤𝑟

Then the space (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular in the sense of Definition 1, with admissible regularity parameter 𝜅𝑉 := 𝜅𝑝 (𝑚, 𝑛) given as follows: 1. Let 2 ≤ 𝑝 ≤ ∞. Then 𝜅𝑉 := 𝜅𝑝 (𝑚, 𝑛) = min

2≤𝜌≤𝑝 𝜌<∞

(︁

)︁ 2 2 max{2, 𝜌 − 1} 𝑟 𝜌 − 𝑝 .

For 𝑟 ≥ 8, this parameter is realized by the auxiliary norm {︁ {︁ }︁}︁ √︀ ‖𝑋‖𝑉,+ := ‖𝑋‖𝑆𝜌* , 𝜌* := min 𝑝, max 3, ln 𝑟 + (ln 𝑟)2 − 2 ln 𝑟 . Under the same condition 𝑟 ≥ 8, (︀ )︀ 𝜅𝑉 := 𝜅𝑝 (𝑚, 𝑛) = 𝒪 min{𝑝, ln(𝑒𝑟)} . 2. Let 1 ≤ 𝑝 < 2. Then 2

𝜅𝑉 := 𝜅𝑝 (𝑚, 𝑛) = 𝑟 𝑝 −1 . 87

Proof. To simplify notation throughout this proof, we write ‖𝑋‖ := ‖𝑋‖𝑉 = ‖𝑋‖𝑆𝑝 ,

𝜅 := 𝜅𝑉 .

Whenever an auxiliary norm is introduced, we denote it by ‖𝑋‖+ := ‖𝑋‖𝑉,+ ,

𝜅+ := 𝜅𝑉,+ .

We consider separately the cases 2 ≤ 𝑝 ≤ ∞ and 1 ≤ 𝑝 < 2. (i) Case 2 ≤ 𝑝 ≤ ∞. This case is established in [Juditsky and Nemirovski, 2008]; see the proof of Example 3.3 therein. In our notation, this yields that (𝑉, ‖ · ‖) is 𝜅-regular with (︁ )︁ 2 2 𝜅 := 𝜅𝑝 (𝑚, 𝑛) = min max{2, 𝜌 − 1} 𝑟 𝜌 − 𝑝 . 2≤𝜌≤𝑝 𝜌<∞

We now derive the minimizing exponent for 𝑟 ≥ 8. Set 𝑎 := ln 𝑟 > 2 and define, for finite 𝜌 ≥ 2, 2

2

𝐹 (𝜌) := max{2, 𝜌 − 1} 𝑟 𝜌 − 𝑝 , with the convention 1/𝑝 = 0 when 𝑝 = ∞. On 2 ≤ 𝜌 ≤ 3, we have 𝐹 (𝜌) = 2𝑟2/𝜌−2/𝑝 and 𝑑 2𝑎 ln 𝐹 (𝜌) = − 2 < 0, 𝑑𝜌 𝜌

2 < 𝜌 < 3.

Thus 𝐹 is strictly decreasing on [2, 3]; in particular, if 2 ≤ 𝑝 ≤ 3, its minimum over [2, 𝑝] is attained at 𝜌 = 𝑝. On 𝜌 ≥ 3, we have 𝐹 (𝜌) = (𝜌 − 1)𝑟2/𝜌−2/𝑝 , so 𝑑 1 2𝑎 𝜌2 − 2𝑎𝜌 + 2𝑎 ln 𝐹 (𝜌) = − 2 = , 𝑑𝜌 𝜌−1 𝜌 𝜌2 (𝜌 − 1)

𝜌 > 3.

The numerator has roots 𝜌− = 𝑎 −

√︀ 𝑎2 − 2𝑎,

𝜌+ = 𝑎 +

√︀

𝑎2 − 2𝑎.

Since 𝑎 > 2 and the numerator at 𝜌 = 2 equals 4 − 2𝑎 < 0, these roots satisfy 𝜌− < 2 < 𝜌+ . Consequently, if 𝜌+ > 3, then 𝐹 decreases on [3, 𝜌+ ] and increases on [𝜌+ , ∞); if 𝜌+ ≤ 3, it increases on [3, ∞). Together with continuity at 𝜌 = 3 and the decrease on [2, 3], this shows that the minimum over all finite 𝜌 ≥ 2 is attained at max{3, 𝜌+ }. Imposing 𝜌 ≤ 𝑝 therefore gives {︁ {︁ }︁}︁ √︀ 𝜌* = min 𝑝, max 3, ln 𝑟 + (ln 𝑟)2 − 2 ln 𝑟 . For 𝑝 = ∞, this yields the finite exponent 𝜌* = max{3, 𝜌+ }, so the auxiliary norm ‖ · ‖𝑆𝜌* remains within the construction above. For completeness, the order bound follows by choosing 𝜌 = 𝑝 when 2 ≤ 𝑝 < 2𝑎, which gives 𝜅 ≤ max{2, 𝑝 − 1} ≤ 𝑝 < 2𝑎. When 𝑝 ≥ 2𝑎, including 𝑝 = ∞, the admissible choice 𝜌 = 2𝑎 gives (︂ )︂ 2𝑎 𝜅 ≤ (2𝑎 − 1) exp 1 − ≤ 2𝑒𝑎. 𝑝 For every finite 𝑝 ≥ 2, the choice 𝜌 = 𝑝 also gives 𝜅 ≤ 𝑝. Hence, for 𝑟 ≥ 8, (︀ )︀ 𝜅 := 𝜅𝑝 (𝑚, 𝑛) = 𝒪 min{𝑝, ln(𝑒𝑟)} . (ii) Case 1 ≤ 𝑝 < 2. Upper bound. Let ‖ · ‖𝑆2 be the Frobenius (Schatten-2) norm and define the auxiliary norm 1

1

‖𝑋‖+ := 𝑟 𝑝 − 2 ‖𝑋‖𝑆2 . 88

This is a scaled Hilbert norm, hence 1-smooth, so we may take 𝜅+ = 1. Moreover, for every 𝑋 ∈ R𝑚×𝑛 , 1

1

1

1

‖𝑋‖ = ‖𝑋‖𝑆𝑝 = ‖𝜎(𝑋)‖𝑝 ≤ 𝑟 𝑝 − 2 ‖𝜎(𝑋)‖2 = 𝑟 𝑝 − 2 ‖𝑋‖𝑆2 = ‖𝑋‖+ , and since 𝑝 < 2 implies ‖𝑋‖𝑆2 ≤ ‖𝑋‖𝑆𝑝 , we also have 2

2

2

‖𝑋‖2+ = 𝑟 𝑝 −1 ‖𝑋‖2𝑆2 ≤ 𝑟 𝑝 −1 ‖𝑋‖2𝑆𝑝 = 𝑟 𝑝 −1 ‖𝑋‖2 . Thus 2

‖𝑋‖2 ≤ ‖𝑋‖2+ ≤ 𝑟 𝑝 −1 ‖𝑋‖2 ,

∀𝑋 ∈ 𝑉.

Therefore, by Definition 1, the space (𝑉, ‖ · ‖) is 𝜅-regular with 2

𝜅 := 𝜅𝑝 (𝑚, 𝑛) = 𝑟 𝑝 −1 .

We now complement Proposition 14 with corresponding lower bounds for the matrix setting. With 𝑟 := min{𝑚, 𝑛}, the optimal parameter is 𝑟2/𝑝−1 for 1 ≤ 𝑝 < 2, and its optimal order is min{𝑝, ln(𝑒𝑟)} for 2 ≤ 𝑝 ≤ ∞, uniformly in 𝑟 ≥ 1; see Remark 7. Consequently, the optimal parameter remains bounded as 𝑟 varies with a fixed finite 𝑝 ≥ 2, and equals 1 at 𝑝 = 2. The logarithmic branch of the minimum corresponds to 𝑝 ≥ ln(𝑒𝑟), including 𝑝 = ∞, where logarithmic growth is unavoidable up to absolute constants. Proposition 15 (Lower bounds for Schatten classes). Fix 𝑚, 𝑛 ∈ N, let 𝑟 := min{𝑚, 𝑛}, let 𝑉 := R𝑚×𝑛 , and equip it with the norm ‖ · ‖𝑉 := ‖ · ‖𝑆𝑝 . If the space (𝑉, ‖ · ‖𝑉 ) is 𝜅𝑉 -regular, then the following hold: 1. If 1 ≤ 𝑝 < 2, then 2

𝜅𝑉 ≥ 𝑟 𝑝 −1 .

(108)

1 min{𝑝 − 1, ln 𝑟}. 8

(109)

2. If 2 ≤ 𝑝 ≤ ∞ and 𝑟 ≥ 8, then 𝜅𝑉 ≥

Proof. To simplify notation throughout this proof, we write ‖𝑋‖ := ‖𝑋‖𝑉 = ‖𝑋‖𝑆𝑝 ,

𝜅 := 𝜅𝑉 .

1. Case 1: 1 ≤ 𝑝 < 2. Let {𝜀𝑖 }𝑟𝑖=1 be independent Rademacher random variables and define the filtration ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ),

𝑖 = 1, . . . , 𝑟.

For each 𝑖 = 1, . . . , 𝑟, let 𝐸𝑖𝑖 ∈ R𝑚×𝑛 denote the matrix with a 1 in position (𝑖, 𝑖) and zeros elsewhere, and set 𝜉𝑖 := 𝜀𝑖 𝐸𝑖𝑖 ,

𝑖 = 1, . . . , 𝑟.

Then {𝜉𝑖 }𝑟𝑖=1 is an {ℱ𝑖 }-martingale-difference sequence, since each 𝜉𝑖 is ℱ𝑖 -measurable and E [𝜉𝑖 | ℱ𝑖−1 ] = 𝐸𝑖𝑖 E [𝜀𝑖 | ℱ𝑖−1 ] = 0

a.s.

Next, note that 𝐸𝑖𝑖 has exactly one singular value equal to 1 and all remaining singular values equal to 0, hence [︀ ]︀ ‖𝜉𝑖 ‖ = |𝜀𝑖 |‖𝐸𝑖𝑖 ‖𝑆𝑝 = 1, E ‖𝜉𝑖 ‖2 = 1. 89

Moreover, for 𝑆𝑟 :=

𝑟 ∑︁

𝜉𝑖 ,

𝑖=1

we have 𝑆𝑟 =

𝑟 ∑︁

𝜀𝑖 𝐸𝑖𝑖 .

𝑖=1

Recall that 𝐸𝑖𝑖 ∈ R𝑚×𝑛 is the matrix with a single 1 in entry (𝑖, 𝑖) and zeros elsewhere. Therefore, 𝑆𝑟 is the rectangular diagonal matrix whose first 𝑟 diagonal entries are (𝜀1 , . . . , 𝜀𝑟 ) and whose remaining entries are zero: {︂ 𝜀 , if 𝑎 = 𝑏 ≤ 𝑟, (𝑆𝑟 )𝑎𝑏 = 𝑎 0, otherwise. For brevity, we denote this by 𝑆𝑟 = Diag𝑚×𝑛 (𝜀1 , . . . , 𝜀𝑟 ). The singular values of 𝑆𝑟 are |𝜀1 |, . . . , |𝜀𝑟 | (and zeros thereafter), so deterministically (︃ 𝑟 )︃1/𝑝 ∑︁ 𝑝 ‖𝑆𝑟 ‖ = ‖𝑆𝑟 ‖𝑆𝑝 = |𝜀𝑖 | = 𝑟1/𝑝 . 𝑖=1

Hence 𝑟 ∑︁ [︀ ]︀ E ‖𝜉𝑖 ‖2 = 𝑟.

[︀ ]︀ E ‖𝑆𝑟 ‖2 = 𝑟2/𝑝 ,

𝑖=1

Finally, since (𝑉, ‖·‖𝑉 ) is 𝜅-regular and {𝜉𝑖 } is a martingale-difference sequence, Lemma 15 gives 𝑟 ∑︁ [︀ ]︀ [︀ ]︀ E ‖𝑆𝑟 ‖2 ≤ 𝜅 E ‖𝜉𝑖 ‖2 = 𝜅𝑟. 𝑖=1

[︀ ]︀ Combining this with E ‖𝑆𝑟 ‖2 = 𝑟2/𝑝 yields 𝑟2/𝑝 ≤ 𝜅𝑟, that is, 2

𝜅 ≥ 𝑟 𝑝 −1 . This proves (108). 2. Case 2: 2 ≤ 𝑝 ≤ ∞ and 𝑟 ≥ 8. We first treat the subcase 2 ≤ 𝑝 < ∞. Step 1: Parameter choice and an explicit family of diagonal matrices.

Let

𝑁 := ⌊min{𝑝, log2 𝑟}⌋ . Since 𝑟 ≥ 8, we have log2 𝑟 ≥ 3, so in particular 𝑁 ≥ 2 and 2𝑁 ≤ 𝑟 ≤ min{𝑚, 𝑛}. Define the discrete cube Ω := {±1}𝑁 ,

|Ω| = 2𝑁 .

Fix an arbitrary enumeration 𝑁

(𝑗)

Ω = {𝜔 (1) , . . . , 𝜔 (2 ) },

(𝑗)

𝜔 (𝑗) = (𝜔1 , . . . , 𝜔𝑁 ) ∈ {±1}𝑁 . 90

For each 𝑖 ∈ {1, . . . , 𝑁 }, define a diagonal matrix 𝑈𝑖 ∈ R𝑚×𝑛 by setting for 𝑎 ̸= 𝑏,

(𝑈𝑖 )𝑎𝑏 := 0, and (𝑗)

(𝑈𝑖 )𝑗𝑗 := 2−𝑁/𝑝 𝜔𝑖 ,

𝑗 = 1, . . . , 2𝑁 ,

while 𝑗 = 2𝑁 + 1, . . . , 𝑟.

(𝑈𝑖 )𝑗𝑗 := 0,

Thus 𝑈𝑖 has exactly 2𝑁 nonzero diagonal entries, each equal to ±2−𝑁/𝑝 . Since 𝑈𝑖 is diagonal, its singular values are the absolute values of its diagonal entries (and zeros thereafter). Hence 𝑝

‖𝑈𝑖 ‖

2𝑁 ⃒ ⃒ ∑︁ ⃒ −𝑁/𝑝 (𝑗) ⃒𝑝 𝜔𝑖 ⃒ = 2𝑁 · 2−𝑁 = 1. ⃒2

= ‖𝑈𝑖 ‖𝑝𝑆𝑝 =

𝑗=1

Therefore, ‖𝑈𝑖 ‖ = 1,

𝑖 = 1, . . . , 𝑁.

Step 2: A martingale-difference sequence. variables, let ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ),

Let {𝜀𝑖 }𝑁 𝑖=1 be i.i.d. Rademacher random

𝜉𝑖 := 𝜀𝑖 𝑈𝑖 ,

𝑖 = 1, . . . , 𝑁.

Then {𝜉𝑖 }𝑁 𝑖=1 is an {ℱ𝑖 }-martingale-difference sequence and ‖𝜉𝑖 ‖ = ‖𝑈𝑖 ‖ = 1,

𝑖 = 1, . . . , 𝑁,

so 𝑁 ∑︁ [︀ ]︀ E ‖𝜉𝑖 ‖2 = 𝑁. 𝑖=1

[︀ ]︀ Step 3: Lower bound on E ‖𝑆𝑁 ‖2 .

Let

𝑆𝑁 :=

𝑁 ∑︁

𝜉𝑖 .

𝑖=1

Then 𝑆𝑁 is diagonal, and for each 𝑗 ∈ {1, . . . , 2𝑁 }, (𝑆𝑁 )𝑗𝑗 =

𝑁 ∑︁

𝜀𝑖 (𝑈𝑖 )𝑗𝑗 = 2−𝑁/𝑝

𝑖=1

𝑁 ∑︁ 𝑖=1

Therefore, ‖𝑆𝑁 ‖𝑝 = ‖𝑆𝑁 ‖𝑝𝑆𝑝 𝑁

=

2 ∑︁

|(𝑆𝑁 )𝑗𝑗 |𝑝

𝑗=1

⃒ ⃒𝑝 2𝑁 ⃒ 𝑁 ⃒ ∑︁ ⃒ −𝑁/𝑝 ∑︁ (𝑗) ⃒ = 𝜀𝑖 𝜔𝑖 ⃒ ⃒2 ⃒ ⃒ 𝑖=1 𝑗=1 ⃒𝑝 ⃒𝑁 2𝑁 ⃒∑︁ ⃒ ∑︁ ⃒ (𝑗) ⃒ −𝑁 =2 𝜀𝑖 𝜔𝑖 ⃒ ⃒ ⃒ ⃒ 𝑗=1 𝑖=1 ⃒𝑝 ]︃ [︃⃒ 𝑁 ⃒∑︁ ⃒ ⃒ ⃒ = E𝜔∼Unif(Ω) ⃒ 𝜀𝑖 𝜔𝑖 ⃒ , ⃒ ⃒ 𝑖=1

91

(𝑗)

𝜀𝑖 𝜔𝑖 .

𝑁

where the last equality holds because {𝜔 (𝑗) }2𝑗=1 is an enumeration of Ω. Since 𝜔 ∼ Unif(Ω) is uniform on {±1}𝑁 , for every fixed realization of 𝜀 = (𝜀1 , . . . , 𝜀𝑁 ) ∈ {±1}𝑁 we have ⃒𝑝 ]︃ ⃒ ]︃ [︃⃒ 𝑁 [︃⃒ 𝑁 ⃒∑︁ ⃒ ⃒∑︁ ⃒𝑝 ⃒ ⃒ ⃒ ⃒ E𝜔∼Unif(Ω) ⃒ 𝜀𝑖 𝜔𝑖 ⃒ = E𝜔∼Unif(Ω) ⃒ 𝜔𝑖 ⃒ , ⃒ ⃒ ⃒ ⃒ 𝑖=1

𝑖=1

because the coordinatewise sign-flip map 𝜔 ↦→ (𝜀1 𝜔1 , . . . , 𝜀𝑁 𝜔𝑁 ) is a bijection of Ω and therefore preserves the uniform law. Consequently, ⃒ ]︃)︃ (︃ [︃⃒ 𝑁 ⃒∑︁ ⃒𝑝 1/𝑝 ⃒ ⃒ ‖𝑆𝑁 ‖ = E𝜔∼Unif(Ω) ⃒ 𝜔𝑖 ⃒ , ⃒ ⃒ 𝑖=1

so ‖𝑆𝑁 ‖ is deterministic. In particular, ⃒ ]︃)︃ [︃⃒ 𝑁 ⃒∑︁ ⃒𝑝 2/𝑝 ⃒ ⃒ E𝜔∼Unif(Ω) ⃒ 𝜔𝑖 ⃒ . ⃒ ⃒

(︃ [︀ ]︀ E ‖𝑆𝑁 ‖2 = ‖𝑆𝑁 ‖2 =

𝑖=1

We now lower bound this quantity using a single atom. Let 𝑁 ∑︁ 𝑍𝑁 := 𝜔𝑖 , 𝜔 ∼ Unif(Ω). 𝑖=1

Since Prob{𝑍𝑁 = 𝑁 } = 2−𝑁 , we obtain E𝜔∼Unif(Ω) [|𝑍𝑁 |𝑝 ] ≥ 𝑁 𝑝 Prob{𝑍𝑁 = 𝑁 } = 𝑁 𝑝 2−𝑁 . Hence (︀ )︀1/𝑝 ‖𝑆𝑁 ‖ = E𝜔∼Unif(Ω) [|𝑍𝑁 |𝑝 ] ≥ 𝑁 · 2−𝑁/𝑝 . By construction, 𝑁 ≤ 𝑝, so 2−𝑁/𝑝 ≥ 1/2, and thus [︀ ]︀ 𝑁2 . E ‖𝑆𝑁 ‖2 = ‖𝑆𝑁 ‖2 ≥ 4 Step 4: Conclude via the second-moment bound. Up to this point, all expectations involving 𝜔 were taken with respect to the auxiliary randomness 𝜔 ∼ Unif(Ω). We now switch back to the original probability space on which the martingale-difference sequence {𝜉𝑖 }𝑁 𝑖=1 is defined. In particular, in the remainder of the proof, E [·] denotes expectation with respect to the randomness of 𝜀. Since (𝑉, ‖ · ‖𝑉 ) is 𝜅-regular and {𝜉𝑖 } is a martingale-difference sequence, Lemma 15 gives 𝑁 ∑︁ [︀ ]︀ [︀ ]︀ 2 E ‖𝑆𝑁 ‖ ≤ 𝜅 E ‖𝜉𝑖 ‖2 = 𝜅𝑁. 𝑖=1 [︀ ]︀ 2 Combining this with E ‖𝑆𝑁 ‖2 ≥ 𝑁 /4 yields 𝑁 𝜅≥ . 4

Finally, since 𝑁 = ⌊min{𝑝, log2 𝑟}⌋, we have 𝑁 ≥ min{𝑝 − 1, log2 𝑟 − 1}. Since 𝑟 ≥ 8, we have log2 𝑟 ≥ 3, and therefore 1 1 log2 𝑟 − 1 ≥ log2 𝑟 ≥ ln 𝑟. 2 2 Hence 𝑁 1 𝜅≥ ≥ min{𝑝 − 1, ln 𝑟}, 4 8 which proves (109) for 2 ≤ 𝑝 < ∞. 92

Subcase 𝑝 = ∞.

Let Ω := {±1}𝑁 ,

𝑁 := ⌊log2 𝑟⌋,

|Ω| = 2𝑁 .

Fix the same enumeration 𝑁

(𝑗)

Ω = {𝜔 (1) , . . . , 𝜔 (2 ) },

(𝑗)

𝜔 (𝑗) = (𝜔1 , . . . , 𝜔𝑁 ) ∈ {±1}𝑁 .

Define diagonal matrices 𝑈𝑖 ∈ R𝑚×𝑛 by for 𝑎 ̸= 𝑏,

(𝑈𝑖 )𝑎𝑏 := 0, and (𝑗)

𝑗 = 1, . . . , 2𝑁 ,

(𝑈𝑖 )𝑗𝑗 := 𝜔𝑖 , while

𝑗 = 2𝑁 + 1, . . . , 𝑟.

(𝑈𝑖 )𝑗𝑗 := 0,

Then every nonzero diagonal entry of 𝑈𝑖 equals ±1, hence ‖𝑈𝑖 ‖ = ‖𝑈𝑖 ‖𝑆∞ = 1,

𝑖 = 1, . . . , 𝑁.

With ℱ𝑖 := 𝜎(𝜀1 , . . . , 𝜀𝑖 ),

𝜉𝑖 := 𝜀𝑖 𝑈𝑖 ,

𝑖 = 1, . . . , 𝑁,

we again obtain an {ℱ𝑖 }-martingale-difference sequence satisfying 𝑁 𝑁 ∑︁ [︀ ]︀ ∑︁ [︀ ]︀ E ‖𝜉𝑖 ‖2 = E ‖𝜉𝑖 ‖2𝑆∞ = 𝑁. 𝑖=1

𝑖=1

For 𝑆𝑁 :=

𝑁 ∑︁

𝜉𝑖 ,

𝑖=1

we have, for each 𝑗 ∈ {1, . . . , 2𝑁 }, (𝑆𝑁 )𝑗𝑗 =

𝑁 ∑︁

𝜀𝑖 (𝑈𝑖 )𝑗𝑗 =

𝑁 ∑︁

(𝑗)

𝜀𝑖 𝜔𝑖 .

𝑖=1

𝑖=1

Since 𝑆𝑁 is diagonal, ⃒𝑁 ⃒ ⃒∑︁ ⃒ ⃒ ⃒ ‖𝑆𝑁 ‖ = ‖𝑆𝑁 ‖𝑆∞ = max |(𝑆𝑁 )𝑗𝑗 | = max ⃒ 𝜀𝑖 𝜔𝑖 ⃒ . 𝜔∈Ω ⃒ ⃒ 1≤𝑗≤2𝑁 𝑖=1

For any fixed realization of 𝜀 ∈ {±1}𝑁 , choosing 𝜔 = 𝜀 gives 𝑁 ∑︁

𝜀𝑖 𝜔𝑖 =

𝑖=1

𝑁 ∑︁

𝜀2𝑖 = 𝑁,

𝑖=1

so ‖𝑆𝑁 ‖ ≥ 𝑁 . On the other hand, for every 𝜔 ∈ Ω, ⃒𝑁 ⃒ 𝑁 ⃒∑︁ ⃒ ∑︁ ⃒ ⃒ 𝜀 𝑖 𝜔𝑖 ⃒ ≤ |𝜀𝑖 𝜔𝑖 | = 𝑁, ⃒ ⃒ ⃒ 𝑖=1

𝑖=1

so ‖𝑆𝑁 ‖ ≤ 𝑁 . Therefore, ‖𝑆𝑁 ‖ = 𝑁

deterministically, and hence 93

[︀ ]︀ E ‖𝑆𝑁 ‖2 = 𝑁 2 .

Applying Lemma 15 gives 𝑁 ∑︁ [︀ ]︀ [︀ ]︀ 𝑁 2 = E ‖𝑆𝑁 ‖2 ≤ 𝜅 E ‖𝜉𝑖 ‖2 = 𝜅𝑁, 𝑖=1

so 𝜅 ≥ 𝑁. Since 𝑁 = ⌊log2 𝑟⌋ ≥

1 ln 𝑟 2

for 𝑟 ≥ 8, this yields 1 ln 𝑟, 2 which is stronger than (109) in the case 𝑝 = ∞, since 𝜅≥

min{𝑝 − 1, ln 𝑟} = ln 𝑟

when 𝑝 = ∞.

Remark 7 (Uniform logarithmic orders). For 2 ≤ 𝑝 ≤ ∞, the optimal regularity parameters of ℓ𝑑𝑝 and Schatten-𝑝 spaces have orders (︀ )︀ (︀ )︀ Θ min{𝑝, ln(𝑒𝑑)} and Θ min{𝑝, ln(𝑒𝑟)} , respectively, with absolute constants, for every 𝑑 ≥ 1 and 𝑟 ≥ 1. Indeed, for 𝐷 ≥ 8, where 𝐷 = 𝑑 or 𝐷 = 𝑟, the preceding upper and lower bounds give these orders because 𝑝 − 1 is comparable to 𝑝 for 𝑝 ≥ 2 and ln(𝑒𝐷) = 1 + ln 𝐷 is comparable to ln 𝐷. For 1 ≤ 𝑑 < 8, taking 𝜌 = 2 in Proposition 12 gives 𝜅𝑝 (𝑑) ≤ 𝑑1−2/𝑝 ≤ 7. For 1 ≤ 𝑟 < 8, taking 𝜌 = 2 in Proposition 14 gives 𝜅𝑝 (𝑚, 𝑛) ≤ 2𝑟1−2/𝑝 ≤ 14. Together with the universal lower bound 𝜅 ≥ 1 and 1 ≤ min{𝑝, ln(𝑒𝐷)} ≤ ln(7𝑒) for 1 ≤ 𝐷 < 8, these bounds cover the remaining dimensions. The uniform notation for orders does not change the exact minimizers 𝜌* or the estimates with explicit constants stated under dimension restrictions.

94

M

Analysis of SGD Under Dual-Norm Light-Tail Assumption

This section develops the analysis under Assumption 3, from minibatch concentration to oracle and wall-clock complexity. In Section M.1, we prove Lemma 1 following the martingale large-deviation approach of Juditsky and Nemirovski [2008]. Regularity provides a smooth auxiliary norm on 𝐸 * in which we control the minibatch noise; comparison with ‖·‖* then yields concentration in the induced dual norm. In Section M.2, we apply this concentration bound conditionally at each iterate and use a union bound to control the minibatch error simultaneously over the 𝑇 iterations. Combining this error bound with descent under non-Euclidean smoothness gives the high-probability convergence guarantee and the oracle complexity. As in Section A and Corollary 1, 𝑁* = 𝐵𝑇 counts the stochastic-gradient ̃︀ with the oracle calls of serial minibatch SGD: 𝐵 calls at each of the 𝑇 iterations. We use 𝒪(·) 1 convention of Section A, suppressing logarithmic factors in both 𝑇 and /𝛿, where 𝛿 ∈ (0, 1) is the failure probability. Finally, Section M.3 combines the same iteration guarantee with a bound on minibatch collection time to establish the wall-clock complexity of Rennala NSGD under fixed worker computation times. M.1

Proof of the Key Concentration Inequality

How 𝜅-regularity enables minibatch concentration. For[︁independent centered [︁noise with finite ]︁ ⃦ 𝑗 ⃦2 ]︁ ∑︀𝐵 2 1 ⃦ ⃦ second moments, the Euclidean second-moment identity E ‖¯ 𝜂 ‖* = 𝐵 2 𝑗=1 E 𝜂 * follows by expanding the squared norm and cancelling the cross terms. This identity need not hold for a norm not induced by an inner product. Even then, norm equivalence gives 𝒪(1/𝐵) second-moment decay in any fixed finite-dimensional space, with a geometry-dependent constant; 𝜅-regularity bounds this constant by 𝜅 (Lemma 15). The high-probability argument additionally controls exponential moments of a potential built from the squared norm. Smoothness gives a quadratic upper bound on this potential (Lemma 25), and conditional centering removes the linear term associated with each martingale increment. However, the ℓ1 , ℓ∞ , nuclear, and spectral norms are generally nonsmooth away from the origin, apart from low-dimensional cases in which they are Euclidean. Their squared norms therefore need not provide the required bound. 𝜅-regularity resolves this difficulty by supplying a 𝜅+ -smooth auxiliary norm and the comparison (13). Together, these allow the martingale argument to yield concentration in the original dual norm, with the geometric dependence captured by 𝜅. A suitably scaled Euclidean norm is always an admissible 1-smooth choice, but other auxiliary norms can give significantly smaller regularity parameters. The regimes for dual ℓ𝑝 and Schatten-𝑝 norms are described in Section 3.4, with orders valid in all positive dimensions summarized in Remark 7. As discussed there, improvement is not uniform across 𝑝, and a smaller regularity parameter alone does not guarantee a smaller oracle-complexity bound. We now collect the auxiliary lemmas needed to prove Lemma 1. Lemma 24 (Fenchel Conjugate of the Restricted Squared Norm). Let 𝜑̂︀ : 𝐸 → R ∪ {+∞} be the squared norm function restricted to the primal unit ball B = {𝑥 ∈ 𝐸 | ‖𝑥‖ ≤ 1}: {︂ 1 2 ‖𝑥‖ , if ‖𝑥‖ ≤ 1 ̂︀ 𝜑(𝑥) = 2 . +∞, otherwise ̂︀ * (𝑠) = sup ̂︀ Its Fenchel conjugate, (𝜑) 𝑥∈𝐸 {⟨𝑠, 𝑥⟩ − 𝜑(𝑥)}, is precisely the potential function 𝑉 (𝑠): {︂ 1 2 if ‖𝑠‖* ≤ 1 ̂︀ * (𝑠) = 𝑉 (𝑠) := 2 ‖𝑠‖* , (𝜑) . ‖𝑠‖* − 12 , if ‖𝑠‖* > 1 Proof. Since 𝐸 is finite-dimensional, the primal unit ball B is compact. The function 𝑥 ↦→ ⟨𝑠, 𝑥⟩ − 2 1 2 ‖𝑥‖ is continuous, so {︂ }︂ ̂︀ * (𝑠) = max ⟨𝑠, 𝑥⟩ − 1 ‖𝑥‖2 . (𝜑) 𝑥∈B 2 95

If 𝑠 = 0, this maximum is zero, attained at 𝑥 = 0, and it equals max0≤𝑟≤1 {−𝑟2 /2} = 0. Now suppose 𝑠 ̸= 0. For every 𝑥 ∈ B, the duality inequality gives }︂ {︂ 1 1 𝑟2 2 2 ⟨𝑠, 𝑥⟩ − ‖𝑥‖ ≤ ‖𝑠‖* ‖𝑥‖ − ‖𝑥‖ ≤ max 𝑟 ‖𝑠‖* − . 0≤𝑟≤1 2 2 2 The continuous linear functional 𝑢 ↦→ ⟨𝑠, 𝑢⟩ attains its maximum ‖𝑠‖* > 0 on B. Any maximizer 𝑢 has ‖𝑢‖ = 1: otherwise, rescaling 𝑢 to the unit sphere would increase the pairing and contradict maximality. For every 𝑟 ∈ [0, 1], the point 𝑥 = 𝑟𝑢 therefore belongs to B and satisfies ⟨𝑠, 𝑟𝑢⟩ −

1 𝑟2 2 ‖𝑟𝑢‖ = 𝑟 ‖𝑠‖* − . 2 2

This proves the reverse bound. Together with the case 𝑠 = 0, we obtain }︂ {︂ 2 ̂︀ * (𝑠) = max 𝑟 ‖𝑠‖ − 𝑟 (𝜑) . * 0≤𝑟≤1 2 Completing the square shows that the scalar maximum is attained at 𝑟 = min{‖𝑠‖* , 1}. Hence {︂ 1 2 if ‖𝑠‖* ≤ 1, ̂︀ * (𝑠) = 2 ‖𝑠‖* , (𝜑) = 𝑉 (𝑠). ‖𝑠‖* − 12 , if ‖𝑠‖* > 1, At ‖𝑠‖* = 1, both expressions equal 12 , attained at 𝑟 = 1. Lemma 25 (Descent Property of the Potential Function 𝑉𝛽 ). Let the space (𝐸 * , ‖·‖* ) be 𝜅-smooth. Define the function 𝑉𝛽 : 𝐸 * → R as: {︂ 1 2 ‖𝑠‖* , if ‖𝑠‖* ≤ 1 𝑉𝛽 (𝑠) := 𝛽𝑉 (𝑠/𝛽), where 𝑉 (𝑠) = 2 . ‖𝑠‖* − 21 , if ‖𝑠‖* > 1 Then for any 𝛽 > 0, the potential function 𝑉𝛽 (𝑠) is continuously differentiable and satisfies the inequality: 𝑉𝛽 (𝑠 + 𝜂) ≤ 𝑉𝛽 (𝑠) + ⟨∇𝑉𝛽 (𝑠), 𝜂⟩ +

𝜅 2 ‖𝜂‖* , 2𝛽

∀𝑠, 𝜂 ∈ 𝐸 * .

Proof. The proof strategy is to first establish that the gradient of the base potential function, ∇𝑉 (𝑠), is 𝜅-Lipschitz continuous. The desired descent inequality is then a direct consequence of this property. We adapt the proof technique from the appendix of Juditsky and Nemirovski [2008]. 2

Step 1: Establishing the 𝜅-Lipschitz Continuity of ∇𝑉 (𝑠). Let 𝜑(𝑥) := 12 ‖𝑥‖ , whose Fenchel 2 conjugate is 𝜑* (𝑠) = 12 ‖𝑠‖* . Apply Proposition 9 with 𝑉 = 𝐸 * , ‖ · ‖𝑉,+ = ‖·‖* , and 𝜅𝑉,+ = 𝜅. Under the canonical identification 𝑉 * = 𝐸 ** ≃ 𝐸, the induced dual norm is ‖·‖. The proposition’s squared-norm function is therefore 𝜑* , and its Fenchel conjugate is 𝜑. By implication (i) ⇒ (vi), the assumed 𝜅-smoothness of (𝐸 * , ‖·‖* ) makes 𝜑 𝜅1 -strongly convex with respect to ‖·‖. Define {︂ 1 ̂︀ 𝜑(𝑥) =

2 ‖𝑥‖

2

,

+∞,

if ‖𝑥‖ ≤ 1, otherwise.

Since the unit ball B = {𝑥 ∈ 𝐸 : ‖𝑥‖ ≤ 1} is closed and convex, 𝜑̂︀ is proper and closed, and ̂︀ * . restricting 𝜑 to B preserves its 𝜅1 -strong convexity. By Lemma 24, its Fenchel conjugate is 𝑉 = (𝜑) The conjugate of a proper closed 𝜇-strongly convex function is differentiable, with a 𝜇1 -Lipschitz gradient in the corresponding dual norms. Applying this result with 𝜇 = 𝜅1 shows that 𝑉 is continuously differentiable and ‖∇𝑉 (𝑠1 ) − ∇𝑉 (𝑠2 )‖ ≤ 𝜅 ‖𝑠1 − 𝑠2 ‖* , 96

𝑠1 , 𝑠2 ∈ 𝐸 * .

Step 2: From Lipschitz Gradient to the Descent Inequality. The desired inequality is a standard consequence of a function having a Lipschitz continuous gradient. For any function 𝐹 with a 𝜅-Lipschitz gradient, the descent lemma states: 𝐹 (𝑠 + 𝜂) ≤ 𝐹 (𝑠) + ⟨∇𝐹 (𝑠), 𝜂⟩ +

𝜅 2 ‖𝜂‖* . 2

Applying this directly to our function 𝑉 (𝑠), which we have just shown to have a 𝜅-Lipschitz gradient, gives: 𝑉 (𝑠 + 𝜂) ≤ 𝑉 (𝑠) + ⟨∇𝑉 (𝑠), 𝜂⟩ +

𝜅 2 ‖𝜂‖* . 2

This proves the lemma for the base case where 𝛽 = 1. Step 3: Scaling the Result for 𝑉𝛽 . The final step is to extend the result to the scaled function 𝑉𝛽 (𝑠) = 𝛽𝑉 (𝑠/𝛽 ). Using the chain rule, the gradient is ∇𝑉𝛽 (𝑠) = ∇𝑉 (𝑠/𝛽 ). We start with the inequality proven in Step 2, but apply it to the points 𝑠/𝛽 and 𝜂/𝛽 : 𝑉 (𝑠/𝛽 + 𝜂/𝛽) ≤ 𝑉 (𝑠/𝛽) + ⟨∇𝑉 (𝑠/𝛽), 𝜂/𝛽⟩ +

𝜅 2 ‖𝜂/𝛽‖* . 2

Multiplying the entire inequality by the positive scalar 𝛽: (︂ )︂ 𝑠+𝜂 𝜅 2 𝛽𝑉 ≤ 𝛽𝑉 (𝑠/𝛽) + 𝛽 ⟨∇𝑉 (𝑠/𝛽), 𝜂/𝛽⟩ + 𝛽 2 ‖𝜂‖* 𝛽 2𝛽 𝜅 2 𝑉𝛽 (𝑠 + 𝜂) ≤ 𝑉𝛽 (𝑠) + ⟨∇𝑉 (𝑠/𝛽), 𝜂⟩ + ‖𝜂‖* . 2𝛽 Finally, substituting the identity for the gradient, ∇𝑉 (𝑠/𝛽 ) = ∇𝑉𝛽 (𝑠), we arrive at the desired conclusion: 𝜅 2 ‖𝜂‖* . 𝑉𝛽 (𝑠 + 𝜂) ≤ 𝑉𝛽 (𝑠) + ⟨∇𝑉𝛽 (𝑠), 𝜂⟩ + 2𝛽 This completes the proof. Lemma 26 (Scalar Concentration for Sub-Gaussian Martingales [Juditsky and Nemirovski, 2008]). 𝐵 Let {𝜓𝑗 }𝐵 𝑗=1 be a sequence of real-valued random variables adapted to a filtration {ℱ𝑗 }𝑗=0 . Assume that for constants 𝜇𝑗 ≥ 0 and 𝜈𝑗 > 0, the following conditions hold almost surely for all 𝑗 = 1, . . . , 𝐵: (i) E [𝜓𝑗 |ℱ𝑗−1 ] ≤ 𝜇𝑗 . [︁ (︁ 2 )︁ ]︁ 𝜓 (ii) E exp 𝜈 2𝑗 |ℱ𝑗−1 ≤ exp(1). 𝑗

Then for any 𝜆 ≥ 0, the sum Ψ𝐵 =

Prob

⎧ ⎨ ⎩

∑︀𝐵

Ψ𝐵 >

𝑗=1 𝜓𝑗 satisfies:

𝐵 ∑︁ 𝑗=1

⎯ ⎫ ⎸ 𝐵 ⎸∑︁ ⎬ 𝜇𝑗 + 𝜆⎷ 𝜈𝑗2 ≤ exp(−𝜆2 /3). ⎭ 𝑗=1

Lemma 27 (Lower Bound on the Potential Function). For any 𝛽 > 0, the following inequality gives a lower bound on the potential 𝑉𝛽 (𝑠): ‖𝑠‖* ≤

𝛽 + 𝑉𝛽 (𝑠), 2

∀𝑠 ∈ 𝐸 * .

Proof. The proof is most easily done in two steps. First, we prove the inequality for the base function 𝑉 (𝑠) (i.e., for 𝛽 = 1). Second, we use the scaling property of 𝑉𝛽 (𝑠) to generalize the result. 97

Step 1: Proof for the Base Case (𝛽 = 1). holds:

Our goal is to prove that for any 𝑠 ∈ 𝐸 * , the following

1 + 𝑉 (𝑠). 2 We analyze this by considering the two cases from the piecewise definition of 𝑉 (𝑠). ‖𝑠‖* ≤

• Case 1: ‖𝑠‖* ≤ 1. 2 In this case, the definition of 𝑉 (𝑠) is 𝑉 (𝑠) = 12 ‖𝑠‖* . We need to show that: 1 1 2 + ‖𝑠‖* . 2 2 Let 𝑡 = ‖𝑠‖* . The inequality becomes 𝑡 ≤ 12 + 12 𝑡2 . We can rearrange this into a simple quadratic inequality: 1 2 1 𝑡 −𝑡+ ≥0 2 2 𝑡2 − 2𝑡 + 1 ≥ 0 ‖𝑠‖* ≤

(𝑡 − 1)2 ≥ 0. A squared real number is always non-negative, so this inequality is always true. Thus, the property holds for this case. • Case 2: ‖𝑠‖* > 1. In this case, the definition of 𝑉 (𝑠) is 𝑉 (𝑠) = ‖𝑠‖* − 12 . We need to show that: (︂ )︂ 1 1 ‖𝑠‖* ≤ + ‖𝑠‖* − . 2 2 The right-hand side simplifies to ‖𝑠‖* . The inequality becomes: ‖𝑠‖* ≤ ‖𝑠‖* . This holds with equality. Thus, the property also holds for this case. Since the inequality holds in both cases, we have proven that ‖𝑠‖* ≤ 12 + 𝑉 (𝑠) for all 𝑠 ∈ 𝐸 * . Step 2: Scaling the Result to 𝑉𝛽 (𝑠). We now use the result from Step 1, which is valid for any vector in 𝐸 * . Let us apply it to the specific vector 𝑠/𝛽 : 1 ‖𝑠/𝛽‖* ≤ + 𝑉 (𝑠/𝛽). 2 By the absolute homogeneity property of norms, ‖𝑠/𝛽 ‖* = 𝛽1 ‖𝑠‖* . Substituting this in gives: 1 1 ‖𝑠‖* ≤ + 𝑉 (𝑠/𝛽). 𝛽 2 Since 𝛽 > 0, we can multiply the entire inequality by 𝛽 without changing its direction: 𝛽 ‖𝑠‖* ≤ + 𝛽𝑉 (𝑠/𝛽). 2 By definition, 𝑉𝛽 (𝑠) = 𝛽𝑉 (𝑠/𝛽 ). Substituting this into the right-hand side gives the final result: ‖𝑠‖* ≤

𝛽 + 𝑉𝛽 (𝑠). 2

This completes the proof. Lemma 1 (Concentration in Regular Normed Spaces). Let 𝜅 be a regularity constant of the dual [︀ space ]︀ (𝐸 * , ‖·‖* ). Assume that {𝜂 𝑗 }𝐵 of i.i.d. random vectors in 𝐸 * such that E 𝜂 𝑗 = 0 𝑗=1 is a sequence [︂ (︂ 𝑗 2 )︂]︂ ‖𝜂 ‖* and they satisfy the light-tail condition: E exp ≤ exp(1). Then for any 𝜆 ≥ 0, the aver𝜎2 {︁ }︁ √ )︀ (︀√ ∑︀𝐵 age 𝜂¯ = 𝐵1 𝑗=1 𝜂 𝑗 satisfies the concentration inequality: Prob ‖¯ 𝜂 ‖* ≥ 2𝜅 + 2𝜆 √𝜎𝐵 ≤ 2

exp(−𝜆 /3). 98

Proof of Lemma 1. Let 𝜅 be the regularity constant specified in the lemma. By Lemma 23, there exists a 𝜅-smooth norm ‖·‖*,+ on 𝐸 * such that 2

2

2

∀𝑠 ∈ 𝐸 * .

‖𝑠‖* ≤ ‖𝑠‖*,+ ≤ 2 ‖𝑠‖* 2 Define 𝜎+ := 2𝜎 2 . Then for every 𝑗, ⃦ 𝑗 ⃦2 ⃦𝜂 ⃦

(110)

2 ⃦𝜂 𝑗 ⃦2 2 ⃦𝜂 𝑗 ⃦* *,+ * ≤ = , 2 𝜎+ 2𝜎 2 𝜎2

⃦ ⃦

⃦ ⃦

and hence by the assumed light-tail condition, ⎡ ⎛ ⃦ ⃦2 ⎞⎤ [︃ (︃ ⃦ ⃦2 )︃]︃ ⃦𝜂 𝑗 ⃦ ⃦𝜂 𝑗 ⃦ *,+ * ⎠⎦ ≤ E exp E ⎣exp ⎝ ≤ 𝑒. 2 𝜎+ 𝜎2

(111)

Step 1: set up a martingale difference sequence. Let ℱ𝑗 := 𝜎(𝜂 1 , . . . , 𝜂 𝑗 ) and define the partial ∑︀𝑗 𝑗 𝐵 sums 𝑆0 ≡ 0, 𝑆𝑗 := 𝑡=1 𝜂 𝑡 . Then (𝑆𝑗 )𝐵 𝑗=0 is a martingale and (𝜂 )𝑗=1 is a martingale-difference sequence w.r.t. (ℱ𝑗 ). For 𝛽 > 0, define 𝑉𝛽 : 𝐸 * → R by {︃ 2 1 ‖𝑢‖*,+ , ‖𝑢‖*,+ ≤ 1, 𝑉𝛽 (𝑠) := 𝛽𝑉 (𝑠/𝛽), 𝑉 (𝑢) := 2 ‖𝑢‖*,+ − 21 , ‖𝑢‖*,+ > 1.

Step 2: the potential function.

We use the following three properties: (a) 𝑉𝛽 is 1-Lipschitz w.r.t. ‖·‖*,+ . Indeed, write {︃ 2 𝑉𝛽 (𝑠) = ℎ𝛽 (‖𝑠‖*,+ ),

𝑟 2𝛽 ,

ℎ𝛽 (𝑟) =

𝑟 − 𝛽2 ,

0 ≤ 𝑟 ≤ 𝛽, 𝑟 > 𝛽.

The two branches have matching derivatives at 𝑟 = 𝛽, and ℎ′𝛽 (𝑟) = min{𝑟/𝛽, 1} ∈ [0, 1] for 𝑟 > 0. Thus ℎ𝛽 is 1-Lipschitz on [0, ∞). Combining this with the reverse triangle inequality gives ⃒ ⃒ |𝑉𝛽 (𝑠 + 𝜂) − 𝑉𝛽 (𝑠)| ≤ ⃒‖𝑠 + 𝜂‖ − ‖𝑠‖ ⃒ ≤ ‖𝜂‖ ∀𝑠, 𝜂 ∈ 𝐸 * . (112) *,+

*,+

*,+

*

(b) By Lemma 25, applied to the 𝜅-smooth space (𝐸 , ‖·‖*,+ ), 𝑉𝛽 is continuously differentiable and satisfies the descent inequality 𝜅 2 𝑉𝛽 (𝑠 + 𝜂) ≤ 𝑉𝛽 (𝑠) + ⟨∇𝑉𝛽 (𝑠), 𝜂⟩ + ‖𝜂‖*,+ ∀𝑠, 𝜂 ∈ 𝐸 * . (113) 2𝛽 (c) By Lemma 27, applied to the same auxiliary norm, for all 𝑠 ∈ 𝐸 * , 𝑉𝛽 (𝑠) ≥ ‖𝑠‖*,+ −

𝛽 2

equivalently

Step 3: define increments and verify scalar conditions. 𝜓𝑗 := 𝑉𝛽 (𝑆𝑗 ) − 𝑉𝛽 (𝑆𝑗−1 ),

‖𝑠‖*,+ ≤ 𝑉𝛽 (𝑠) +

𝛽 . 2

(114)

Define 𝑗 = 1, . . . , 𝐵.

First, (111) and Jensen’s inequality imply the second moment bound: ⎡ ⃦ ⃦2 ⎤ ⎡ ⎛ ⃦ ⃦2 ⎞⎤ ⃦𝜂 𝑗 ⃦ ⃦𝜂 𝑗 ⃦ [︁⃦ ⃦2 ]︁ *,+ ⎦ *,+ ⎠⎦ 2 ⎣exp ⎝ E⎣ ≤ log E ≤ 1 =⇒ E ⃦𝜂 𝑗 ⃦*,+ ≤ 𝜎+ . 2 2 𝜎+ 𝜎+

(115)

Since 𝑆𝑗 is ℱ𝑗 -measurable and 𝑉𝛽 is continuous, 𝜓𝑗 is ℱ𝑗 -measurable. The continuity of ∇𝑉𝛽 also makes ∇𝑉𝛽 (𝑆𝑗−1 ) ℱ𝑗−1 -measurable. Differentiability and (112) give, for every 𝑠, 𝜂 ∈ 𝐸 * , |𝑉𝛽 (𝑠 + 𝑡𝜂) − 𝑉𝛽 (𝑠)| ≤ ‖𝜂‖*,+ . 𝑡→0 |𝑡|

|⟨∇𝑉𝛽 (𝑠), 𝜂⟩| = lim

99

Equivalently, ∇𝑉𝛽 (𝑠) has norm at most 1 in the norm dual to ‖·‖*,+ . Consequently, by Cauchy– Schwarz and (115), [︁⃦ ⃦ ]︁ √︂ [︁ ]︁ [︀⃒⟨︀ ⟩︀⃒]︀ 2 E ⃒ ∇𝑉𝛽 (𝑆𝑗−1 ), 𝜂 𝑗 ⃒ ≤ E ⃦𝜂 𝑗 ⃦*,+ ≤ E ‖𝜂 𝑗 ‖*,+ ≤ 𝜎+ < ∞. [︁⃦ ⃦ ]︁ Also, (112) implies E [|𝜓𝑗 |] ≤ E ⃦𝜂 𝑗 ⃦*,+ ≤ 𝜎+ . Thus the increment, the mixed term, and the squared-noise term are all integrable. We may therefore take conditional expectations in (113) and pull out the ℱ𝑗−1 -measurable gradient. Since 𝜂 𝑗 is independent of ℱ𝑗−1 and has mean zero, the mixed term has conditional mean zero, and we obtain [︀ ]︀ E [𝜓𝑗 | ℱ𝑗−1 ] = E 𝑉𝛽 (𝑆𝑗−1 + 𝜂 𝑗 ) − 𝑉𝛽 (𝑆𝑗−1 ) | ℱ𝑗−1 ]︂ [︂ ⃦ ⟨︀ ⟩︀ 𝜅 ⃦ ⃦𝜂 𝑗 ⃦2 | ℱ𝑗−1 ≤ E ∇𝑉𝛽 (𝑆𝑗−1 ), 𝜂 𝑗 + *,+ 2𝛽 ]︁ [︁ ⃦ ⃦2 𝜅 = E ⃦𝜂 𝑗 ⃦*,+ . 2𝛽 Therefore, E [𝜓𝑗 | ℱ𝑗−1 ] ≤ Also, by (112) and (111), [︃ E exp

(︃

𝜓𝑗2 2 𝜎+

)︃

]︃ | ℱ𝑗−1

2 𝜅𝜎+ 2𝛽

for all 𝑗.

(116)

⎛ ⃦ ⃦2 ⎞⎤ ⃦𝜂 𝑗 ⃦ *,+ ⎠⎦ ≤ E ⎣exp ⎝ ≤ 𝑒. 2 𝜎+

(117)

⎡

∑︀𝐵 Step 4: apply scalar concentration to 𝑗=1 𝜓𝑗 = 𝑉𝛽 (𝑆𝐵 ). Although Lemma 26 is stated with a strict tail event, the same bound holds with ≥. For 𝜆 > 0, apply the lemma with 0 ≤ 𝜆𝑛 < 𝜆 and 𝜆𝑛 ↑ 𝜆. The strict tail events decrease to the non-strict event at 𝜆, so continuity of probability from above and exp(−𝜆2𝑛 /3) → exp(−𝜆2 /3) yield the claim. For 𝜆 = 0, the bound is trivial because its right-hand side is 1. No continuity assumption on the distribution is needed. 2

Applying Lemma 26 (the sub-Gaussian case) with 𝜇𝑗 = 𝜅𝜎+/2𝛽 and 𝜈𝑗 = 𝜎+ yields that for all 𝜆 ≥ 0, {︂ }︂ 2 √ 𝐵𝜅𝜎+ Prob 𝑉𝛽 (𝑆𝐵 ) ≥ + 𝜆 𝐵 𝜎+ ≤ exp(−𝜆2 /3). 2𝛽 Using (114), we obtain }︂ 2 √ 𝐵𝜅𝜎+ 𝛽 + 𝜆 𝐵 𝜎+ ≤ exp(−𝜆2 /3). Prob ‖𝑆𝐵 ‖*,+ ≥ + 2 2𝛽 √ Optimizing the deterministic term in 𝛽 gives 𝛽 ⋆ = 𝐵𝜅 𝜎+ and thus {︁ }︁ √ √ Prob ‖𝑆𝐵 ‖*,+ ≥ ( 𝜅 + 𝜆) 𝐵 𝜎+ ≤ exp(−𝜆2 /3). {︂

Step 5: return to ‖·‖* and to the average. Since ‖𝑆𝐵 ‖* ≤ ‖𝑆𝐵 ‖*,+ and 𝜎+ = {︁ }︁ √ √ √ Prob ‖𝑆𝐵 ‖* ≥ ( 2𝜅 + 2𝜆) 𝐵 𝜎 ≤ exp(−𝜆2 /3).

√

2𝜎,

Finally, 𝜂¯ = 𝑆𝐵/𝐵 and norms are homogeneous, so ‖¯ 𝜂 ‖* = ‖𝑆𝐵 ‖*/𝐵 . Dividing the threshold by 𝐵 yields {︂ }︂ (︁√ √ )︁ 𝜎 Prob ‖¯ 𝜂 ‖* ≥ 2𝜅 + 2𝜆 √ ≤ exp(−𝜆2 /3). 𝐵 This completes the proof. 100

M.2

Proof of the Convergence Rate Theorem

Theorem 1 (Minibatch non-Euclidean SGD under light tails). Let 𝜅 be a regularity constant of the dual space (𝐸 * , ‖·‖* ). Suppose Assumptions ⌈︀ 128𝐿Δ ⌉︀ 1, 2, and 3 hold. Run the update (7) with 1 the constant stepsize 𝛾 ≡ for 𝑇 = iterations, with a minibatch size chosen as 𝑘 {︃⌈︃ }︃ 15𝜀 (︁√ )︁2 ⌉︃ √ 2𝐿 2 16𝜎 2𝜅+ 6 log(𝑇 /𝛿) 𝐵 = max , 1 . Then, with probability at least 1 − 𝛿, the iterates 𝜀 ⃦ ⃦2 satisfy min0≤𝑘≤𝑇 −1 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀. Proof of Theorem 1. Fix the initial point 𝑥0 and define the filtration (︀ )︀ ℱ𝑘 := 𝜎 𝑥0 , 𝜉𝑗𝑡 : 0 ≤ 𝑡 < 𝑘, 1 ≤ 𝑗 ≤ 𝐵 ,

𝑘 ≥ 0.

By induction from the update (7) and the fixed Borel-measurable sharp selection in Section 3.1, 𝑥𝑘 is ℱ𝑘 -measurable. As specified in the serial algorithm, the new batch {𝜉𝑗𝑘 }𝐵 𝑗=1 consists of i.i.d. samples from 𝒟 and is independent of ℱ𝑘 . Let 𝜂𝑗𝑘 := ∇𝑓 (𝑥𝑘 ; 𝜉𝑗𝑘 ) − ∇𝑓 (𝑥𝑘 ) be the noise of the 𝑗-th sample at iteration 𝑘. Conditionally on ℱ𝑘 , the query 𝑥𝑘 is fixed, so these noise vectors are independent and identically distributed. By Assumption 3, they satisfy, almost surely, [︃ (︃ ⃦ ⃦2 )︃ ]︃ ⃦𝜂 𝑘 ⃦ [︀ 𝑘 ]︀ 𝑗 * | ℱ𝑘 ≤ 𝑒. E 𝜂𝑗 | ℱ𝑘 = 0, E exp 𝜎2 ∑︀𝐵 The minibatch error is 𝐵1 𝑗=1 𝜂𝑗𝑘 = 𝑔 𝑘 − ∇𝑓 (𝑥𝑘 ). Thus Lemma 1 applies conditionally on ℱ𝑘 and gives, for each 𝑘 and 𝜆 ≥ 0, almost surely, {︂ }︂ √ √ ⃦ ⃦ 𝜎 Prob ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦* ≥ ( 2𝜅 + 2𝜆) √ | ℱ𝑘 ≤ exp(−𝜆2 /3). 𝐵 Let 𝐴𝑘 (𝜆) denote the event in this conditional probability. Taking expectations and using the tower property yields Prob {𝐴𝑘 (𝜆)} = E [Prob {𝐴𝑘 (𝜆) | ℱ𝑘 }] ≤ exp(−𝜆2 /3). We may now apply the unconditional union bound over the prescribed iterations 𝑘 ∈ {0, . . . , 𝑇 − 1}; independence of the events 𝐴𝑘 (𝜆) is not needed: {︃𝑇 −1 }︃ 𝑇 −1 𝑇 −1 ⋃︁ ∑︁ ∑︁ Prob 𝐴𝑘 (𝜆) ≤ Prob {𝐴𝑘 (𝜆)} ≤ exp(−𝜆2 /3) = 𝑇 exp(−𝜆2 /3). 𝑘=0

𝑘=0

𝑘=0 2

𝜆 We want √︀ this failure probability to be at most 𝛿. By setting 𝑇 exp(− /3) ≤ 𝛿, we find that choosing 𝜆 = 3 log(𝑇/𝛿) suffices. With this choice, we can conclude that with probability at least 1 − 𝛿, the following holds for all 𝑘 ∈ {0, . . . , 𝑇 − 1}: √ √ ⃦ 𝑘 ⃦ ⃦𝑔 − ∇𝑓 (𝑥𝑘 )⃦ ≤ ( 2𝜅 + 2𝜆) √𝜎 . (118) * 𝐵 √

We choose the error threshold 4𝜀 and select the batch size 𝐵 so that the right-hand side of (118) is at most this threshold. Thus, we require: √ √ √ √︀ 𝜎 𝜀 ( 2𝜅 + 2 3 log(𝑇 /𝛿)) √ ≤ . 4 𝐵 Solving for 𝐵 yields the choice: ⎧⎡ (︁√ )︁2 ⎤ ⎫ √︀ ⎪ ⎪ ⎬ ⎨ 16𝜎 2 2𝜅 + 6 log(𝑇 /𝛿) ⎥ ⎢ 𝐵 = max ⎢ ⎥,1 . ⎥ ⎪ ⎪ 𝜀 ⎩⎢ ⎥ ⎭ ⎢ 101

With these choices of 𝜆 and 𝐵, we conclude that with probability at least 1 − 𝛿, we have for all 𝑘 ∈ {0, . . . , 𝑇 − 1}: √ ⃦ 𝑘 ⃦ ⃦𝑔 − ∇𝑓 (𝑥𝑘 )⃦ ≤ 𝜀 . (119) * 4 Fix any realization on which (119) holds for every 0 ≤ 𝑘 < 𝑇 ; these realizations form an event of probability at least 1 − 𝛿. All subsequent descent inequalities are evaluated along this fixed realization. We proceed by contradiction to show that there must exist some 𝑘 ∈ {0, . . . , 𝑇 − 1} such that ⃦ ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦2 ≤ 𝜀. * Assume the contrary, i.e., that for all 𝑘 ∈ {0, . . . , 𝑇 − 1}: ⃦ ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦2 > 𝜀. * ⃦ ⃦ √ This implies ⃦∇𝑓 (𝑥𝑘 )⃦* > 𝜀. Combining this with the error bound from (119), we get: ⃦ ⃦ √ ⃦∇𝑓 (𝑥𝑘 )⃦ ⃦ ⃦ 𝑘 * ⃦𝑔 − ∇𝑓 (𝑥𝑘 )⃦ ≤ 𝜀 < . * 4 4

(120)

(121)

By the triangle inequality for the dual norm, we can bound the norm of the stochastic gradient 𝑔 𝑘 : ⃦ ⃦ ⃦ ⃦ ⃦ 3⃦ ⃦∇𝑓 (𝑥𝑘 )⃦ ≤ ⃦𝑔 𝑘 ⃦ ≤ 5 ⃦∇𝑓 (𝑥𝑘 )⃦ . * * * 4 4

(122)

We now have all the necessary components to analyze the one-step progress of the algorithm. The proof proceeds by combining the standard descent lemma for 𝐿-smooth functions with the highprobability bounds on the stochastic gradient error. Let us analyze the descent from iterate 𝑥𝑘 to 𝑥𝑘+1 . Since the function 𝑓 is 𝐿-smooth, we have: ⃦2 ⟨︀ ⟩︀ 𝐿 ⃦ 𝑓 (𝑥𝑘+1 ) ≤ 𝑓 (𝑥𝑘 ) + ∇𝑓 (𝑥𝑘 ), 𝑥𝑘+1 − 𝑥𝑘 + ⃦𝑥𝑘+1 − 𝑥𝑘 ⃦ 2 2 ⃦ ⃦ ⟨︀ ⟩︀ 𝐿𝛾 𝑘 ⃦ 𝑘 ♯ ⃦2 (𝑔 ) = 𝑓 (𝑥𝑘 ) − 𝛾𝑘 ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ + 2 ⟨︀ ⟩︀ 𝐿𝛾𝑘2 ⃦ 𝑘 ⃦2 ⃦𝑔 ⃦ . = 𝑓 (𝑥𝑘 ) − 𝛾𝑘 ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ + * 2 ⃦ ⃦2 2 In the last step, we used the fundamental property that ⃦𝑠♯ ⃦ = ‖𝑠‖* . We now rewrite the dual pairing term by adding and subtracting 𝑔 𝑘 : ⟨︀ ⟩︀ ⟨︀ ⟩︀ ⟨︀ ⟩︀ − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ = 𝑔 𝑘 − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ − 𝑔 𝑘 , (𝑔 𝑘 )♯ . ⟨︀ ⟩︀ 2 Using the identity 𝑠, 𝑠♯ = ‖𝑠‖* , this becomes: ⟨︀ ⟩︀ ⟨︀ ⟩︀ ⃦ ⃦2 − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ = 𝑔 𝑘 − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ − ⃦𝑔 𝑘 ⃦* . Substituting this back into the descent inequality, we get: ⃦ ⃦2 𝐿𝛾𝑘2 ⃦ 𝑘 ⃦2 ⟨︀ ⟩︀ ⃦𝑔 ⃦ 𝑓 (𝑥𝑘+1 ) ≤ 𝑓 (𝑥𝑘 ) + 𝛾𝑘 𝑔 𝑘 − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ − 𝛾𝑘 ⃦𝑔 𝑘 ⃦* + * 2 ⃦ ⃦ ⟨︀ ⟩︀ 2 = 𝑓 (𝑥𝑘 ) + 𝛾𝑘 𝑔 𝑘 − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ − 𝛾𝑘 (1 − 𝐿𝛾𝑘 /2) ⃦𝑔 𝑘 ⃦* . 2

2

1 We now apply the generalized Young’s inequality, ⟨𝑢, 𝑣⟩ ≤ 2𝑠 ‖𝑢‖* + 2𝑠 ‖𝑣‖ , to the dual pairing term with 𝑢 = 𝑔 𝑘 − ∇𝑓 (𝑥𝑘 ), 𝑣 = (𝑔 𝑘 )♯ , and a parameter 𝑠 > 0:

⃦ ⃦ ⃦ ⟨︀ 𝑘 ⟩︀ 1 ⃦ ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦2 + 𝑠 ⃦(𝑔 𝑘 )♯ ⃦2 𝑔 − ∇𝑓 (𝑥𝑘 ), (𝑔 𝑘 )♯ ≤ * 2𝑠 2 ⃦ 1 ⃦ 𝑠 ⃦ ⃦2 𝑘 𝑘 ⃦2 ⃦ = 𝑔 − ∇𝑓 (𝑥 ) * + ⃦𝑔 𝑘 ⃦* . 2𝑠 2 102

Plugging this bound into our main inequality gives: )︂ (︂ ⃦ ⃦2 ⃦ ⃦ 𝑠⃦ 1 ⃦ 𝑘+1 𝑘 𝑘 𝑘 ⃦2 𝑘 ⃦2 ⃦ ⃦ 𝑓 (𝑥 ) ≤ 𝑓 (𝑥 ) + 𝛾𝑘 𝑔 − ∇𝑓 (𝑥 ) * + 𝑔 * − 𝛾𝑘 (1 − 𝐿𝛾𝑘 /2) ⃦𝑔 𝑘 ⃦* 2𝑠 2 (︂ )︂ ⃦ ⃦ 𝛾𝑘 ⃦ 𝐿𝛾𝑘 𝑠 ⃦ 𝑘 𝑘 𝑘 ⃦2 ⃦ ⃦𝑔 𝑘 ⃦2 . = 𝑓 (𝑥 ) + 𝑔 − ∇𝑓 (𝑥 ) * − 𝛾𝑘 1 − − * 2𝑠 2 2 ⃦2 ⃦ 𝑘 ⃦2 ⃦ 9 ⃦ ∇𝑓 (𝑥𝑘 )⃦* into its This inequality holds for any 𝑠 > 0. To substitute the lower bound ⃦𝑔 ⃦* ≥ 16 negative term, we require 𝐿𝛾𝑘 𝑠 − ≥ 0. 2 2 1 With the prescribed stepsize 𝛾𝑘 = 2𝐿 , this condition is 34 − 2𝑠 ≥ 0. Hence we restrict the Young ⃦ ⃦2 3 parameter to 0 < 𝑠 ≤ 2 from now on. On this interval, the coefficient multiplying ⃦𝑔 𝑘 ⃦* is nonpositive, so we may use the high-probability bounds from (121) and (122) to obtain )︂ (︂ )︂ (︂ )︂ (︂ ⃦ 2 ⃦ 2 𝛾𝑘 1 ⃦ 𝐿𝛾𝑘 𝑠 3⃦ 𝑘 ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦ ⃦ 𝑓 (𝑥𝑘+1 ) ≤ 𝑓 (𝑥𝑘 ) + − 𝛾 1 − ∇𝑓 (𝑥 ) − 𝑘 * * 2𝑠 4 2 2 4 (︂ )︂ ⃦ ⃦ ⃦ 𝛾𝑘 ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦2 − 9𝛾𝑘 1 − 𝐿𝛾𝑘 − 𝑠 ⃦∇𝑓 (𝑥𝑘 )⃦2 = 𝑓 (𝑥𝑘 ) + * * 32𝑠 16 2 2 )︂ ]︂ [︂ (︂ ⃦ 𝐿𝛾𝑘 𝑠 1 ⃦ 9 ⃦∇𝑓 (𝑥𝑘 )⃦2 . 1− − − = 𝑓 (𝑥𝑘 ) − 𝛾𝑘 * 16 2 2 32𝑠 1−

To ensure descent, the step size 𝛾𝑘 and the parameter 𝑠 must make the term in the square brackets 1 positive. Using the constant step size 𝛾𝑘 ≡ 𝛾 = 2𝐿 prescribed in the theorem, the expression becomes: [︂ (︂ )︂ ]︂ ⃦ 1 9 1 𝑠 1 ⃦ 𝑘+1 𝑘 ⃦∇𝑓 (𝑥𝑘 )⃦2 . 𝑓 (𝑥 ) ≤ 𝑓 (𝑥 ) − 1− − − * 2𝐿 16 4 2 32𝑠 To obtain the strongest possible descent guarantee from this inequality, we maximize the term in the square brackets over the admissible interval 0 < 𝑠 ≤ 32 . Define (︂ )︂ 1 𝑠 1 9 1− − − . ℎ(𝑠) := 16 4 2 32𝑠 Its derivative is 9 1 + . 32 32𝑠2 1 Setting ℎ′ (𝑠) = 0 gives 9𝑠2 = 1, whose positive solution is 𝑠* = 31 . Since ℎ′′ (𝑠) = − 16𝑠 3 < 0 for 3 𝑠 > 0, this is the unique maximizer on the admissible interval 0 < 𝑠 ≤ 2 . In particular, 𝑠* = 13 is * 7 admissible and gives 1 − 𝐿𝛾2 𝑘 − 𝑠2 = 12 > 0. We now substitute this optimal value 𝑠* = 1/3 back into our one-step descent inequality: (︂ )︂ 1 1 3 15 9 ℎ(1/3) = 1− − − = . 16 4 6 32 64 ℎ′ (𝑠) = −

Substituting this optimized constant back into the full inequality gives our final one-step progress guarantee: ⃦ 1 15 ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦2 . 𝑓 (𝑥𝑘+1 ) ≤ 𝑓 (𝑥𝑘 ) − * 2𝐿 64 Summing this inequality over 𝑘 = 0, . . . , 𝑇 − 1 along the fixed realization and using (120), we obtain 𝑓 (𝑥𝑇 ) ≤ 𝑓 (𝑥0 ) −

𝑇 −1 ⃦ 15 ∑︁ ⃦ ⃦∇𝑓 (𝑥𝑘 )⃦2 * 128𝐿 𝑘=0

15𝑇 𝜀 < 𝑓 (𝑥0 ) − 128𝐿 0 ≤ 𝑓 (𝑥 ) − ∆ = 𝑓 inf . 103

⌈︀ ⌉︀ The last inequality uses the prescribed iteration count 𝑇 = 128𝐿Δ and ∆ = 𝑓 (𝑥0 ) − 𝑓 inf . This 15𝜀 𝑇 inf contradicts the lower bound 𝑓 (𝑥 ) ≥ 𝑓 from Assumption 2. Thus every realization in the good ⃦ ⃦2 event contains an iterate with ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 for some 0 ≤ 𝑘 < 𝑇 . Since this event has probability at least 1 − 𝛿, we conclude that {︂ }︂ ⃦ ⃦2 Prob min ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 ≥ 1 − 𝛿. 0≤𝑘<𝑇

This completes the proof. Corollary 1 (Norm-native oracle complexity under dual-norm light tails). Suppose the assumptions 1 of Theorem 1 hold. Run the update (7) with the constant stepsize 𝛾𝑘 ≡ 2𝐿 , and choose 𝑇 and 𝐵 as in that theorem. Let 𝑁* := 𝐵𝑇 denote the total number of stochastic-gradient evaluations. Then with ⃦ ⃦2 probability at least 1 − 𝛿, the method produces an iterate satisfying min0≤𝑘≤𝑇 −1 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 after at most )︂ (︂ 𝐿∆ 𝜅𝐿∆𝜎 2 ̃︀ (14) 𝑁* = 𝒪 + 𝜀 𝜀2 ̃︀ hides logarithmic factors in 𝑇 and 1/𝛿. stochastic-gradient evaluations, where 𝒪(·) Proof of Corollary 1. By Theorem 1, the required number of iterations is ⌈︂ ⌉︂ 128𝐿∆ 𝑇 = . 15𝜀 Since 𝜀 ≤ 𝐿∆, this implies (︂ 𝑇 =𝒪

𝐿∆ 𝜀

)︂ .

The same theorem prescribes the minibatch size ⎧⎡ (︁√ )︁2 ⎤ ⎫ √︀ ⎪ ⎪ ⎨ 16𝜎 2 ⎬ 2𝜅 + 6 log(𝑇 /𝛿) ⎢ ⎥ . 𝐵 = max ⎢ , 1 ⎥ ⎪ ⎥ ⎪ 𝜀 ⎩⎢ ⎭ ⎢ ⎥ Using (𝑎 + 𝑏)2 ≤ 2𝑎2 + 2𝑏2 , we have (︁√

2𝜅 +

√︀

6 log(𝑇 /𝛿)

)︁2

≤ 4𝜅 + 12 log(𝑇 /𝛿).

Consequently, )︂ (︂ 2 ̃︀ 1 + 𝜅𝜎 , 𝐵=𝒪 𝜀 where the suppressed factors are logarithmic in 𝑇 and 1/𝛿. Therefore, (︂ (︂ )︂)︂ 2 ̃︀ 𝐿∆ 1 + 𝜅𝜎 𝑁* = 𝐵𝑇 = 𝒪 𝜀 𝜀 )︂ (︂ 2 𝜅𝐿∆𝜎 𝐿∆ ̃︀ + , =𝒪 𝜀 𝜀2 which proves (14). The stated success probability and stationarity guarantee follow directly from Theorem 1. 104

M.3

Time-Complexity Analysis of Rennala NSGD Under the Light-Tailed Assumption

We explain the role of fresh gradients and the modified reply order in Section 4, then prove the wall-clock guarantee under the fixed computation model. Why discard stale gradients? The server may update the iterate while a worker is still computing, so a gradient can be outdated when it arrives. As discussed in Section C.2, a gradient computed at 𝑥𝑘−𝛿𝑘 need not be an unbiased estimator of ∇𝑓 (𝑥𝑘 ). Moreover, to the best of our knowledge, the available wall-clock bounds for classical ASGD, which applies every arriving gradient, are suboptimal in general relative to the lower bound of Tyurin and Richtárik [2023] for their zero-respecting algorithm class and specified unbiased bounded-variance oracle model with fixed worker computation times. Rennala SGD avoids staleness by using only gradients computed at the current iterate; in that Euclidean oracle model, its time complexity matches the lower bound up to a constant factor [Tyurin and Richtárik, 2023]. For our analysis, this restriction also connects the parallel method to the serial one: conditional on the history, the accepted gradients are i.i.d. oracle responses at 𝑥𝑘 . The first part of the proof below establishes this property and applies the serial convergence guarantee. Updating before replying. We slightly modify Method 4 of Tyurin and Richtárik [2023], whose server replies to the worker that completes the minibatch before updating the iterate. Our server updates first and then sends the new iterate and its index to that worker, allowing it to start a fresh computation immediately. Other workers finish their pending computations and receive the latest iterate when their messages are processed. Since each message is fully processed before the next is handled, including any update and reply, a simultaneous return processed after an update is checked against the new index. This change preserves the fresh-minibatch property and the wall-clock upper bound: a worker still computing at an earlier iterate needs at most 𝜏𝑖 additional time before receiving the current one, while the worker that completed the minibatch starts immediately. The timing argument below accounts for both cases. With these protocol details in place, we restate and prove the wall-clock guarantee. Theorem 2 (Wall-clock time complexity of Rennala NSGD). Suppose the assumptions of Theorem 1 hold, and use the fixed computation model of Section 4. Run Rennala NSGD with the stepsize, iteration count 𝑇 , and minibatch size 𝐵 prescribed in that theorem. For 0 < 𝜀 ≤ 𝐿∆ and 𝛿 ∈ (0, 1), ⃦ ⃦2 the iterates satisfy min0≤𝑘<𝑇 ⃦∇𝑓 (𝑥𝑘 )⃦* ≤ 𝜀 with probability at least 1 − 𝛿 within total wall-clock time ⎛ ⎡(︃ ⎤⎞ )︃−1 (︂ )︂ 𝑆 2 ∑︁ 1 1 𝜅𝐿∆𝜎 𝐿∆ ̃︀ ⎝ min ⎣ ⎦⎠ . 𝑇total = 𝒪 + (15) 𝑆 𝑖=1 𝜏𝑖 𝜀 𝑆𝜀2 𝑆∈[𝑁 ] We will also prove that, for any positive integers 𝑇 and 𝐵, the time needed to complete 𝑇 iterations satisfies ⎡(︃ ⎤ )︃−1 𝑆 ∑︁ 1 𝑇total ≤ 2𝑇 min ⎣ (𝐵 + 𝑆)⎦ . (123) 𝜏 𝑆∈[𝑁 ] 𝑖=1 𝑖 This deterministic estimate will also be used in the noise-model comparison in Section M.4. Proof of Theorem 2. The argument has two parts. We first verify that the stochastic gradients accepted during an iteration form a fresh minibatch at the current iterate, so that the serial guarantee of Theorem 1 applies verbatim; we then bound the wall-clock time required to complete 𝑇 such iterations under the fixed computation model. Step 1: the accepted gradients form a fresh minibatch. Fix an iteration 𝑘, and let ℱ𝑘 denote the complete algorithmic history immediately after 𝑥𝑘 is formed and before it is sent to any worker, including the states of workers with computations or messages pending. For 𝑘 > 0, this is just before the reply to the worker that completed the preceding minibatch; for 𝑘 = 0, it is before the initial replies. Thus 𝑥𝑘 is ℱ𝑘 -measurable, and every sample used for a gradient tagged with 𝑘 is drawn after this history is fixed. The server accepts a stochastic gradient during iteration 𝑘 only if it is tagged with the current iteration index 𝑘. Hence, every accepted gradient is computed at 𝑥𝑘 . Whenever a worker receives (𝑥𝑘 , 𝑘), 105

it draws a fresh sample from the common data distribution, independently of the past and of the samples used by other oracle calls, and computes a stochastic gradient at 𝑥𝑘 . Under the fixed computation model, the order of worker returns is determined by the constants {𝜏𝑖 }𝑁 𝑖=1 , the worker states recorded in ℱ𝑘 , and any independent tie-breaking randomness. By the tie-breaking requirement in Section 4, this order does not depend on the fresh oracle samples or their gradient values. Thus selecting the first 𝐵 returns tagged with 𝑘 preserves the sampling law: 𝑘 conditional on ℱ𝑘 , the accepted samples {𝜉𝑗𝑘 }𝐵 𝑗=1 are i.i.d. with distribution 𝒟. Since 𝑥 is ℱ𝑘 measurable, the accepted gradients are conditionally i.i.d. given ℱ𝑘 , with the conditional distribution of ∇𝑓 (𝑥𝑘 ; 𝜉) for a fresh 𝜉 ∼ 𝒟. This conditional distribution can depend on the history through 𝑥𝑘 . [︀ ⃒ ]︀ ∑︀𝐵 Writing the accepted minibatch as 𝑔 𝑘 := 𝐵1 𝑗=1 ∇𝑓 (𝑥𝑘 ; 𝜉𝑗𝑘 ), we consequently have E 𝑔 𝑘 ⃒ ℱ𝑘 = ∇𝑓 (𝑥𝑘 ), and the zero-mean noise vectors 𝜂𝑗𝑘 := ∇𝑓 (𝑥𝑘 ; 𝜉𝑗𝑘 ) − ∇𝑓 (𝑥𝑘 ) satisfy Assumption 3 conditionally on ℱ𝑘 . Applying Lemma 1 conditionally with 𝑥 = 𝑥𝑘 therefore gives, for every 𝜆 > 0, {︂ }︂ (︁√ √ )︁ 𝜎 ⃦ ⃦ Prob ⃦𝑔 𝑘 − ∇𝑓 (𝑥𝑘 )⃦* ≥ 2𝜅 + 2𝜆 √ | ℱ𝑘 ≤ exp(−𝜆2 /3). (124) 𝐵 Thus the iterates produced by Rennala NSGD satisfy the same conditional minibatch oracle properties as the serial minibatch method analyzed in Theorem 1, so that theorem applies. Step 2: iteration complexity. By Theorem 1, with the choice of 𝑇 and 𝐵 in the statement, Rennala NSGD finds an 𝜀-stationary point after at most 𝑇 iterations with probability at least 1 − 𝛿. Our subsequent wall-clock time analysis is adapted from the proof of Theorem 7.5 in [Tyurin and Richtárik, 2023] to the non-Euclidean setting. Thus, it remains to upper bound the wall-clock time required to complete 𝑇 iterations under the fixed computation model where worker 𝑖 returns one stochastic gradient every 𝜏𝑖 seconds. Without loss of generality, assume 0 < 𝜏1 ≤ · · · ≤ 𝜏𝑁 . Time to collect one minibatch. Fix an iteration index 𝑘. In iteration 𝑘, the server forms the minibatch estimator 𝑔 𝑘 by collecting 𝐵 stochastic gradients computed at the current iterate 𝑥𝑘 (i.e., gradients whose iteration index equals 𝑘), ignoring all stale gradients. Choose nonnegative integer target counts 𝐵𝑖 for the workers. These are sufficient quotas for bounding the collection time, not necessarily the numbers of gradients actually accepted from each worker before the minibatch is completed. At the instant 𝑥𝑘 is formed, each worker has at most one computation or message pending from an earlier iteration. Completing and processing it takes at most 𝜏𝑖 additional time, after which the worker receives the current iterate. The worker that completed the preceding minibatch instead receives 𝑥𝑘 immediately and starts a fresh computation; at 𝑘 = 0, all workers receive the initial iterate without first completing a stale computation. As long as iteration 𝑘 has not ended, every subsequent reply sends 𝑥𝑘 again. Consequently, unless the minibatch has already been completed, worker 𝑖 can deliver 𝐵𝑖 > 0 fresh gradients within 𝜏𝑖 (1 + 𝐵𝑖 ) time after 𝑥𝑘 is formed. The extra 𝜏𝑖 is unnecessary for a worker that starts immediately, but remains a valid upper bound. Define {︂ 𝜏 (1 + 𝐵𝑖 ), 𝐵𝑖 > 0, 𝑡𝑖 := 𝑖 0, 𝐵𝑖 = 0. Since workers run in parallel, a sufficient condition to collect a batch of size 𝐵 is the existence of 𝑡′ ≥ 0 and integers 𝐵1 , . . . , 𝐵𝑁 ∈ N0 such that 𝑁 ∑︁

𝑡′ ≥ max 𝑡𝑖 , 𝑖∈[𝑁 ]

𝐵𝑖 ≥ 𝐵.

(125)

𝑖=1

Indeed, if the iteration had not ended by time 𝑡′ , each worker would have delivered its quota, giving at least 𝐵 accepted gradients, a contradiction. Thus the modified reply order admits the same collection-time bound as the original protocol. Relaxation and rounding. We relax (125) by allowing 𝐵𝑖 ∈ R+ . If (𝑡′ , 𝐵1 , . . . , 𝐵𝑁 ) is feasible ′ for ∑︀ the relaxed ∑︀ problem, then (2𝑡 , ⌈𝐵1 ⌉, . . . , ⌈𝐵𝑁 ⌉) is feasible for the integer problem. Indeed, 𝑖 ⌈𝐵𝑖 ⌉ ≥ 𝑖 𝐵𝑖 ≥ 𝐵, and for any 𝑖 with 𝐵𝑖 > 0, 𝜏𝑖 (1 + ⌈𝐵𝑖 ⌉) ≤ 𝜏𝑖 (2 + 𝐵𝑖 ) ≤ 2𝜏𝑖 (1 + 𝐵𝑖 ) ≤ 2𝑡′ . 106

(For 𝐵𝑖 = 0, rounding leaves 𝐵𝑖 and 𝑡𝑖 equal to zero.) Hence it suffices to construct a feasible solution to the relaxed problem and then pay a factor 2. For any 𝑆 ∈ [𝑁 ], define (︃ 𝑆 )︃−1 ∑︁ 1 ′ 𝑡 (𝑆) := (𝐵 + 𝑆), 𝜏 𝑖=1 𝑖

Constructing a feasible solution.

and let 𝑆 ⋆ ∈ arg min𝑆∈[𝑁 ] 𝑡′ (𝑆) be the smallest minimizer. Consider the choice {︃ ′ ⋆ 𝑡 (𝑆 ) − 1, 𝑖 ≤ 𝑆 ⋆ , 𝜏𝑖 𝐵𝑖 := 0, 𝑖 > 𝑆⋆. Then 𝑁 ∑︁

⋆

𝑆 (︂ ′ ∑︁ 𝑡 (𝑆 ⋆ )

𝐵𝑖 =

𝑖=1

𝜏𝑖

𝑖=1

)︂ −1

⋆

′

⋆

= 𝑡 (𝑆 )

𝑆 ∑︁ 1

𝜏 𝑖=1 𝑖

− 𝑆⋆

= (𝐵 + 𝑆 ⋆ ) − 𝑆 ⋆ = 𝐵, so the batch-size constraint holds with equality. Moreover, one can verify that 𝑡′ (𝑆 ⋆ ) > 𝜏𝑆 ⋆ , hence 𝐵𝑖 > 0 for all 𝑖 ≤ 𝑆 ⋆ . Showing that 𝐵𝑖 > 0 for all 𝑖 ≤ 𝑆 ⋆ . Recall that (︃ 𝑆 )︃−1 ∑︁ 1 ′ 𝑡 (𝑆) := (𝐵 + 𝑆), 𝑆 ⋆ ∈ arg min 𝑡′ (𝑆) 𝜏 𝑖 𝑆∈[𝑁 ] 𝑖=1

is the smallest minimizer,

and we set {︃ ′

𝑡 (𝑆 ⋆ ) − 1, 𝜏𝑖

𝐵𝑖 :=

0,

𝑖 ≤ 𝑆⋆, 𝑖 > 𝑆⋆.

We prove that 𝑡′ (𝑆 ⋆ ) > 𝜏𝑆 ⋆ , which will imply 𝐵𝑖 > 0 for all 𝑖 ≤ 𝑆 ⋆ . Case 1: 𝑆 ⋆ = 1. Then 𝑡′ (1) = 𝜏1 (𝐵 + 1) > 𝜏1 = 𝜏𝑆 ⋆ . Case 2: 𝑆 ⋆ > 1. Since 𝑆 ⋆ is the smallest minimizer of 𝑡′ (·), we must have 𝑡′ (𝑆 ⋆ ) < 𝑡′ (𝑆 ⋆ − 1). Writing 𝐴𝑆 :=

∑︀𝑆

1 𝑖=1 𝜏𝑖 , this inequality is equivalent to −1 ⋆ ⋆ 𝐴−1 𝑆 ⋆ (𝐵 + 𝑆 ) < 𝐴𝑆 ⋆ −1 (𝐵 + 𝑆 − 1),

and hence (multiplying by 𝐴𝑆 ⋆ 𝐴𝑆 ⋆ −1 > 0) 𝐴𝑆 ⋆ −1 (𝐵 + 𝑆 ⋆ ) < 𝐴𝑆 ⋆ (𝐵 + 𝑆 ⋆ − 1). Using 𝐴𝑆 ⋆ = 𝐴𝑆 ⋆ −1 + 𝜏𝑆1⋆ , we obtain ⋆

(︂

𝐴𝑆 ⋆ −1 (𝐵 + 𝑆 ) <

𝐴𝑆 ⋆ −1 +

1 𝜏𝑆 ⋆

)︂

(𝐵 + 𝑆 ⋆ − 1)

= 𝐴𝑆 ⋆ −1 (𝐵 + 𝑆 ⋆ − 1) +

107

𝐵 + 𝑆⋆ − 1 . 𝜏𝑆 ⋆

Subtracting 𝐴𝑆 ⋆ −1 (𝐵 + 𝑆 ⋆ − 1) from both sides yields 𝐵 + 𝑆⋆ − 1 , 𝜏𝑆 ⋆

𝐴𝑆 ⋆ −1 < and therefore 𝐴𝑆 ⋆ = 𝐴𝑆 ⋆ −1 +

1 𝜏𝑆 ⋆

<

𝐵 + 𝑆⋆ − 1 1 𝐵 + 𝑆⋆ + = . 𝜏𝑆 ⋆ 𝜏𝑆 ⋆ 𝜏𝑆 ⋆

Finally, 𝑡′ (𝑆 ⋆ ) =

𝐵 + 𝑆⋆ > 𝜏𝑆 ⋆ . 𝐴𝑆 ⋆

Since 𝜏1 ≤ · · · ≤ 𝜏𝑆 ⋆ , for every 𝑖 ≤ 𝑆 ⋆ we have 𝐵𝑖 =

𝑡′ (𝑆 ⋆ ) 𝑡′ (𝑆 ⋆ ) −1≥ − 1 > 0. 𝜏𝑖 𝜏𝑆 ⋆

Thus indeed 𝐵𝑖 > 0 for all 𝑖 ≤ 𝑆 ⋆ . Finally, max 𝑡𝑖 = max⋆ 𝜏𝑖 (1 + 𝐵𝑖 )

𝑖∈[𝑁 ]

𝑖≤𝑆

= max⋆ 𝜏𝑖 · 𝑖≤𝑆

𝑡′ (𝑆 ⋆ ) 𝜏𝑖

= 𝑡′ (𝑆 ⋆ ), so (𝑡′ (𝑆 ⋆ ), 𝐵1 , . . . , 𝐵𝑁 ) is feasible for the relaxed problem. By the rounding argument, the server can therefore collect a batch of size 𝐵 within at most 2𝑡′ (𝑆 ⋆ ) seconds in every iteration. Total time.

Thus, the wall-clock time to execute 𝑇 iterations is at most ⎡(︃ ⎤ )︃−1 𝑆 ∑︁ 1 𝑇total ≤ 2𝑇 · min ⎣ (𝐵 + 𝑆)⎦ . 𝜏 𝑆∈[𝑁 ] 𝑖=1 𝑖

(126)

Combining this deterministic runtime bound with Theorem 1 (which guarantees success after 𝑇 iterations with probability at least 1 − 𝛿) yields the high-probability time guarantee (123). ̃︀ complexity bound. From the exact bound to a 𝒪 time (average rate) of the fastest 𝑆 workers by

Fix 𝑆 ∈ [𝑁 ] and define the average inverse

𝑆

𝑟¯𝑆 :=

1 ∑︁ 1 . 𝑆 𝑖=1 𝜏𝑖

Then we can rewrite (︃ 𝑆 )︃−1 ∑︁ 1 𝑖=1

𝜏𝑖

(︂ )︂ 𝐵 1+ 1 𝑆 𝑖=1

𝐵+𝑆 𝑆 (𝐵 + 𝑆) = ∑︀𝑆 1 = ∑︀𝑆 𝑖=1 𝜏𝑖

= 𝑟¯𝑆−1

(︂

𝐵 1+ 𝑆

𝜏𝑖

)︂ .

Plugging this into (126) gives (︂ )︂ (︂ )︂ 𝐵 𝑇𝐵 𝑇total ≤ 2𝑇 · min 𝑟¯𝑆−1 1 + = 2 min 𝑟¯𝑆−1 𝑇 + . 𝑆 𝑆 𝑆∈[𝑁 ] 𝑆∈[𝑁 ] Next, we substitute the iteration and batch-size choices from Theorem 1: ⌈︂ ⌉︂ (︂ )︂ 128𝐿∆ 𝐿∆ 𝑇 = =⇒ 𝑇 = 𝒪 , 15𝜀 𝜀 108

(127)

and ⎧⎡ )︁2 ⎤ ⎫ (︁√ √︀ ⎪ ⎪ ⎬ ⎨ 16𝜎 2 2𝜅 + 6 log(𝑇 /𝛿) ⎥ ⎢ 𝐵 = max ⎢ ⎥,1 . ⎥ ⎪ ⎪ 𝜀 ⎩⎢ ⎥ ⎭ ⎢ √︀ √ Using (𝑎 + 𝑏)2 ≤ 2𝑎2 + 2𝑏2 with 𝑎 = 2𝜅 and 𝑏 = 6 log(𝑇/𝛿), we obtain (︁√ )︁2 √︀ (︀ )︀ 2𝜅 + 6 log(𝑇 /𝛿) ≤ 2(2𝜅) + 2 6 log(𝑇 /𝛿) = 4𝜅 + 12 log(𝑇 /𝛿). Therefore, 64𝜅𝜎 2 192𝜎 2 log(𝑇 /𝛿) 16𝜎 2 (4𝜅 + 12 log(𝑇 /𝛿)) =1+ + . 𝜀 𝜀 𝜀 ̃︀ In particular, using 𝜅 ≥ 1 and hiding the logarithmic factor log(𝑇/𝛿) inside 𝒪(·), we have (︂ )︂ 2 ̃︀ 1 + 𝜅𝜎 𝐵=𝒪 . 𝜀 𝐵 ≤1+

(128)

Consequently, )︂ 𝑇 𝜅𝑇 𝜎 2 + 𝑆 𝑆𝜀 (︂ )︂ 𝐿∆ 𝜅𝐿∆𝜎 2 ̃︀ =𝒪 + . 𝑆𝜀 𝑆𝜀2

𝑇𝐵 ̃︀ =𝒪 𝑆

(︂

Since 𝑆 ≥ 1, the deterministic contribution 𝑇 /𝑆 is at most 𝑇 and can be absorbed into the leading 𝑇 term in (127). Substituting these bounds into (127) gives ⎛ ⎡(︃ ⎤⎞ )︃−1 (︂ )︂ 𝑆 2 ∑︁ 1 𝐿∆ 1 𝜅𝐿∆𝜎 ̃︀ ⎝ min ⎣ ⎦⎠ , 𝑇total = 𝒪 + 𝑆 𝑖=1 𝜏𝑖 𝜀 𝑆𝜀2 𝑆∈[𝑁 ] ̃︀ time complexity bound. which is the claimed 𝒪

M.4

Discussion

We now interpret (15), explain how the update norm and noise model affect it, and compare its Euclidean specialization with existing guarantees. A system factor and a problem factor. For each 𝑆 ∈ [𝑁 ], the bound (15) is the product of two (︀ ∑︀𝑆 )︀−1 factors. The first, 𝑆1 𝑖=1 𝜏1𝑖 , is the harmonic mean of the computation times of the 𝑆 fastest 2

𝜅𝐿Δ𝜎 workers, and it depends only on the system. The second, 𝐿Δ 𝜀 + 𝑆𝜀2 , is the oracle complexity (14) with its statistical term divided by 𝑆, and apart from 𝑆 it depends only on the problem and the update norm. Enlarging 𝑆 shrinks the statistical term but admits slower workers into the harmonic mean, and the minimum over 𝑆 selects the best balance. Running the algorithm does not require knowing either the minimizing number of workers or their computation times 𝜏𝑖 . The prescribed 𝑇 and 𝐵 do not depend on these quantities: the server simply accepts the first 𝐵 fresh gradients to arrive.

Where the update norm enters. The update norm ‖·‖ leaves the system factor untouched and acts only through the second factor, but there it affects every constant except ∆. The smoothness constant 𝐿 is defined through the pair (‖·‖ , ‖·‖* ) in Assumption 1, the noise level 𝜎 is measured in ‖·‖* in Assumption 3, the accuracy 𝜀 refers to stationarity measured in ‖·‖* , and 𝜅 is the regularity constant of (𝐸 * , ‖·‖* ). Bounds obtained for different update norms therefore certify stationarity in different dual norms, and they should not be compared as if they measured the same quantity. Native versus converted noise in wall-clock time. The comparison of Section 3.6.1 carries over from oracle complexity to wall-clock time. Under the comparison assumptions of Section J.1 and the fixed computation model of Theorem 2, apply Lemma 21 conditionally to each accepted 109

minibatch and use 𝐵F from (59). The same union bound and descent argument give the required stationarity guarantee. The deterministic minibatch-collection bound (123) applies with 𝐵 = 𝐵F . ̃︀ + CF /𝜀), substituting into that bound gives the form (15) with the full native Since 𝐵F = 𝒪(1 2 coefficient 𝜅𝜎 replaced by CF = 𝛼−2 𝜎F2 ; the system factor is unchanged. With the update norm, target accuracy, and worker times fixed, a smaller statistical coefficient reduces the noise-dependent term for every choice of 𝑆. Taking the minimum over 𝑆 therefore cannot increase the resulting expression. Thus, a smaller statistical coefficient also gives a wall-clock upper bound that is no larger and can be significantly smaller, up to the suppressed logarithmic factors. The Euclidean case. When the update norm is Euclidean, ‖·‖* = ‖·‖2 and 𝜅 = 1, and (15) coincides, up to logarithmic factors, with the time complexity of Rennala SGD [Tyurin and Richtárik, 2023].19 The assumptions differ, however: our light-tail condition is stronger than bounded variance, and our guarantee holds with high probability rather than in expectation. We therefore read this agreement as a consistency check with the Euclidean theory rather than as an optimality result, and we make no optimality claim for a general update norm.

19 Their lower bound applies to zero-respecting algorithms (Definition 6.2) under their unbiased bounded-variance oracle

model (Definition 6.3), with fixed worker computation times. They give a matching upper bound for Rennala SGD.

110

N

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

111

Related documents

• Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry#1489843
• hyperbolix: Hyperbolic Deep Learning in JAX#1064792
• GPU-Accelerated Computation of Persistent Homology for Topological Analysis of Image Data#1247886
• Learning to Optimize by Differentiable Programming#798738
• Learning to Optimize by Differentiable Programming#797324