Conceptio › Archive › arXiv CS

arXiv CSopen access

Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

arXiv:2605.06609v1 [cs.LG] 7 May 2026

Chenyang Zhang∗

Yuan Cao†

Abstract Transformers have demonstrated remarkable in-context learning (ICL) capabilities. The strong ICL performance of transformers is commonly believed to arise from their ability to implicitly execute certain algorithms on the context, thereby enhancing prediction and generation. In this work, we investigate how transformers with softmax attention perform in-context learning on linear classification data. We first construct a class of multi-layer transformers that can perform in-context logistic regression, with each layer exactly performing one step of normalized gradient descent on an in-context loss. Then, we show that our constructed transformer can be obtained through (i) training a single self-attention layer supervised by one-step gradient descent, and (ii) recurrently applying the trained layer to obtain a looped model. Training convergence guarantees of the self-attention layer and out-of-distribution generalization guarantees of the looped model are provided. Our results advance the theoretical understanding of ICL mechanism by showcasing how softmax transformers can effectively act as in-context learners.

Introduction

Transformers have achieved remarkable success in a wide range of applications, including natural language processing (Vaswani et al., 2017; Devlin, 2018; Touvron et al., 2023), computer vision (Dosovitskiy et al., 2020; Rao et al., 2021; Yuan et al., 2021; Zhang et al., 2025b), and reinforcement learning (Janner et al., 2021; Reed et al., 2022; Kim et al., 2025). One widely recognized interpretation for their empirical success is their ability to perform in-context learning (ICL): pretrained transformers are capable of performing previously unseen tasks based on demonstrations and examples in the prompt, without requiring any additional task-specific fine-tuning (Brown et al., 2020). A line of recent works interpret the in-context learning (ICL) capability of transformers from an algorithmic perspective, viewing transformers as models that can implicitly execute certain learning algorithms on the context examples. Specifically, Garg et al. (2022) proposes a theoretical framework for ICL in terms of learning a hypothesis class, and empirically shows that transformers can in-context learn the linear function class. Motivated by this empirical finding, several recent works attempt to theoretically study how transformers perform in-context learning on linear regression tasks. Akyürek et al. (2022); Von Oswald et al. (2023) construct multi-layer transformers with linear attention that can execute gradient descent on the an “in-context loss” defined on the context data, thereby enabling in-context learning of linear regression. Ahn et al. (2023); Zhang ∗ †

School of Computing & Data Science, The University of Hong Kong; [email protected] School of Computing & Data Science, The University of Hong Kong; [email protected]

et al. (2024b); Huang et al. (2024) further provide training guarantees for single-layer transformers with linear or softmax attention, showing that such models can be trained to solve in-context regression problems. Beyond standard linear regression, Guo et al. (2023); Bai et al. (2024) further demonstrate the in-context learning capacities of transformers by constructing multi-head ReLU transformers capable of performing a variety of learning algorithms, including ridge regression, Lasso regression, and generalized linear models. More recently, Frei and Vardi (2025); Shen et al. (2025) study how single-layer transformers with linear attention can be trained to solve in-context classification on Gaussian mixture data. Several recent works also investigate in-context learning and other learning tasks with looped transformers, in which the same parameter matrices are shared across different layers. Gatmiry et al. (2024); Chen et al. (2025a) theoretically show that looped transformers still implement gradient descent for in-context linear regression. Yang et al. (2024) empirically demonstrates that looped transformers can achieve performance comparable to that of standard transformers in in-context learning, while using significantly fewer parameters. Geiping et al. (2025) further shows that at inference time, transformers can effectively benefit from increased depth by recurrently applying a trained block. In this work, we study how a multi-layer (looped) transformer with softmax attention performs in context learning on classification tasks. Following the settings in Huang et al. (2025), we consider using transformers to solve an “in-context weight prediction” task, and investigate transformer’s expressive power, training guarantees, and out-of-distribution (O.O.D.) generalization performance on this task. The major contributions of this work are as follows. • We establish expressive power guarantees and demonstrate that there exists a class of multi-layer softmax transformers that can perform in-context logistic regression (defined on exponential loss function)1 via normalized gradient descent. Specifically, the hidden layers’ outputs of an L-layer transformer exactly match the first L iterates of normalized gradient descent on the in-context loss of logistic regression. Leveraging this exact equivalence and the implicit bias of normalized gradient descent, we further prove that the transformer’s output converges in direction to the maximum-margin solution of the context dataset as the depth L increases. • We also study whether our constructed models can be obtained through training. We consider the strategy to first train a single-layer transformer, and then obtain a looped transformer by recurrently applying the trained layer. For this training problem, our results precisely characterize the existence of a unique minimizer, establish a linear convergence rate, and demonstrate that the obtained model aligns well with the ones constructed in our expressive power guarantees. Interestingly, our results show that the transformer learns normalized gradient descent even though it is supervised by a “gradient descent teacher”. • We validate the capacities of the obtained looped transformers to solve in-context weight prediction by providing an O.O.D. generalization bound. Notably, this result is a point-wise high probability guarantee that holds for any input, and is stronger than the in-expectation guarantees commonly adopted in the in-context learning literature. Under mild assumptions of the num1

In the implicit-bias literature, linear classification with exponential-tailed losses is often treated under the umbrella of the term “logistic regression” and analyzed under a unified framework. In this work, following this convention, we slightly abuse the term “in-context logistic regression” to refer specifically to the in-context linear classification setting with the exponential loss.

e d , matching the PAC learning sample ber of layers L, the generalization bound is given as O n complexity lower bound (Long, 1995). • From a technical perspective, our training analysis develops several novel proof techniques, including an analysis based on an approximate training procedure, the application of the Newton–Kantorovich theorem, and the derivation of a Polyak-Lojasiewicz inequality. We believe these novel proof techniques may help the analysis of transformers with softmax attention in broad scenarios, and thus may be of independent interest. Notation. Given two sequences {xn } and {yn }, we denote xn = O(yn ) if there exist some absolute constant C1 > 0 and N > 0 such that |xn | ≤ C1 |yn | for all n ≥ N . Similarly, we denote xn = Ω(yn ) if there exist C2 > 0 and N > 0 such that |xn | ≥ C2 |yn | for all n > N . We say xn = Θ(yn ) if e e e to hide logarithmic factors in xn = O(yn ) and xn = Ω(yn ) both holds. We use O(·), Ω(·), and Θ(·) these notations respectively. Moreover, we denote xn = poly(yn ) if xn = O(ynD ) for some positive constant D, and xn = polylog(yn ) if xn = poly(log(yn )). For two scalars a and b, we denote a ∨ b = max{a, b} and a ∧ b = min{a, b}. For any n ∈ N+ , we use [n] to denote the set {1, 2, · · · , n}. We use 1n to denote a n-dimensional vector with all 1 entries. For any d1 , d2 ∈ N+ , we denote Id1 the d1 × d1 identity matrix, and 0d1 ×d2 the d1 × d2 matrix with all its entries being zero. We denote Sd−1 the d-dimensional unit sphere, and U (Sd−1 ) the uniform distribution over Sd−1 . For any a ∈ Rd and a positive definite matrix Σ ∈ Rd×d , N (a, Σ) denotes the d-dimensional Gaussian distribution with mean a and covariance matrix Σ. We use ϕ(·), and Φ(·) to denote the p.d.f. and c.d.f. of standard normal distribution, respectively. For any matrix A, we use λi (A) to denote its i-th eigenvalue. In addition, we write [A]i:j,k to denote the subvector formed by entries in rows i through j of the k-th column. Similarly, [A]i,j:k denote the subvector formed by entries in columns j through k of the i-th row.

Related works

In-context learning of transformers. In-context learning capacities of transformers are first formalized in Garg et al. (2022), with experiments on linear function hypothesis classes. Given this formulation, many recent works attempt to investigate the in-context learning capabilities under different settings. Akyürek et al. (2022); Von Oswald et al. (2023) theoretically demonstrate the expressive power of transformers, showing that they can implicitly perform algorithms on context data like gradient descent. Ahn et al. (2023) explicitly constructs a multi-layer linear transformer capable of conducting preconditioned gradient descent for linear regression, and provides the characterization of the loss landscape and training convergence. Zhang et al. (2024b) investigates the training of a single-layer linear transformer on the context data embedded from linear regression samples, and provides both in-distribution and out-of-distribution generalization guarantees. Huang et al. (2024) extends this result to the single-layer softmax transformer, while requiring a strong assumption regarding the training strategy. Also, for in-context linear regression, Chen et al. (2024a,c) investigates the mechanism of the multi-heads from the training and expressive power perspectives respectively. Zhang et al. (2024c) investigates the training of a single-layer transformer connected with an MLP block, and shows it can simulate the one-step of gradient descent with learnable initialization. Bai et al. (2024); Guo et al. (2023) further extend the scope of in-context algorithms implemented by transformers, showing that they can perform empirical risk minimization for linear regression, ridge regression, Lasso, and more general generalized linear models. Frei 3

and Vardi (2025); Shen et al. (2025) investigates the training of single-layer linear transformer to solve in-context classification with Gaussian mixture inputs. Chen and Zou (2024) study the role of depth in transformers through a set of sequence learning tasks, showing that while a single attention layer can achieve memorization, reasoning and generalization require multiple attention layers. Chen et al. (2025b) studies how test-time computation in transformers can be understood through in-context linear regression with randomness and sampling. Cao et al. (2025) investigates the expressive power of Transformers in Bayesian network sequence modeling, showing that they can implement in-context maximum likelihood estimation and autoregressive sampling. Li et al. (2025) study the context hijacking phenomenon through the lens of an optimization procedure with heterogeneous learning rates. Anwar et al. (2024) study in-context linear regression through an adversarial lens, showing that adversarial attacks transfer poorly across transformer seeds and between transformers and classical learning algorithms. Optimization of transformers. Besides the studies focusing on the ICL capacities of transformers, several recent works study the training behavior under other settings. Zhang et al. (2020); Kunstner et al. (2023); Pan and Li (2023); Li et al. (2024a); Zhang et al. (2024d) investigate how transformers behave when trained by different optimizers, covering distinct theoretical and empirical settings. Li et al. (2023); Jelassi et al. (2022) studies the training of shallow ViT-type transformers under certain specified initializations. Gao et al. (2024) addresses the global convergence of transformers given certain prerequisites. Ildiz et al. (2024); Chen et al. (2024b); Nichani et al. (2024); Shi and Cao (2025) study how transformers are trained on Markovian data and how attention mechanisms recover the underlying transition structure. Specifically, Ildiz et al. (2024) characterize self-attention as a context-conditioned Markov chain, Chen et al. (2024b) show that induction heads emerge as a copier–selector–classifier mechanism on n-gram data, Nichani et al. (2024) demonstrates that attention gradients recover latent causal graphs, and Shi and Cao (2025) shows that attention selects parent states while values implement Markov transitions in random walks. In addition, some works provide learning guarantees of transformers in certain statistical tasks, including the sparse-linear classifier (Zhang et al., 2025a), sparse token selection (Wang et al., 2024b), one-nearest neighbor selection (Li et al., 2024b), maximum hard-margin classifier (Tarzanagh et al., 2023a,b), the compositions of functions (Wang et al., 2025), implementation of spectral methods and EM updates on Gaussian mixture models (He et al., 2025a,b), and “teacher-student” distillation (Zhang et al., 2026). Implicit bias of logistic regression. A line of theoretical works studies the implicit bias of different optimizers on logistic regression (Soudry et al., 2018; Ji and Telgarsky, 2019; Nacson et al., 2019; Qian and Qian, 2019; Ji and Telgarsky, 2021; Wang et al., 2022; Zhang et al., 2024a; Wang et al., 2024a). Soudry et al. (2018) proves that the iterates of gradient descent directionally converge to the maximum ℓ2 -margin solution on separable data, and Ji and Telgarsky (2019) extends this result to the settings with non-separable data. While the previous results are established for fullbatch gradient descent, Nacson et al. (2019) studies the implicit bias of stochastic gradient descent, and demonstrates the same directional convergence results as full-batch gradient descent. Ji and Telgarsky (2021) propose a primal-dual framework, and demonstrate a fast polynomial convergence rate for implicit bias of normalized gradient descent. Wang et al. (2024a) further proposes an exponentially adaptive learning rate for gradient descent, which achieves a linear convergence rate. Wu et al. (2023) studies the implicit bias of gradient descent under the “edge of stability” setting, where the learning rate can be set as arbitrarily large. Besides gradient descent, several works have investigated adaptive gradient-based optimization methods that incorporate momentum. Qian and

Qian (2019) studies the implicit bias of AdaGrad, and shows a directional convergence toward a maximum-margin solution under certain preconditioned norm. Wang et al. (2022) studies shows that the momentum does not change the implicit bias of gradient descent. Zhang et al. (2024a) demonstrates an implicit bias towards the maximum ℓ∞ -margin of Adam.

Problem setups

In this section, we introduce the problem setting of in-context learning for weight prediction and the multi-layer softmax transformer models considered in this work. In-context learning for weight predictions. In-context learning (ICL) refers to a learning framework in which the input consists of a collection of context data pairs Dn = {(xi , yi ) : xi ∈ X , yi ∈ Y}ni=1 ∈ D, together with a query input xquery ∈ X , whose label yquery ∈ Y is unknown. An in-context learning model f (·, ·) : X × D → Y is then expected to infer the underlying featurelabel mapping from Dn , and produce a prediction for the unknown label yquery in the form of ybquery = f (xquery , Dn ). As demonstrated in recent theoretical works (Ahn et al., 2023; Bai et al., 2024; Zhang et al., 2024b), in-context learning models like transformers typically handle this type of task by implicitly performing certain learning algorithms to fit a predictor gb(·) : X → Y based on the context dataset Dn , and then generate the final prediction via ybquery = gb(xquery ). Beyond the classic in-context learning setting where the goal is to output a prediction for yquery , Huang et al. (2025) further proposes the problem of “in-context weight prediction” under b ·⟩ is a linear model, and the the linear regression setting. In this task, the predictor gb(·) = ⟨θ, b Specifically, they assume in-context learner is required to explicitly output this weight vector θ. n that each in-context set Dn = {(xi , yi )}i=1 admits a ground truth vector θ ∗ such that yi = ⟨θ ∗ , xi ⟩, b Motivated by Huang et al. (2025), in this paper, we and the objective is to estimate θ ∗ by θ. consider “in-context weight prediction” in classification, Definition 3.1. Let Dθ∗ be a distribution over the d-dimensional unit sphere Sd−1 , and Dx be a distribution over Rd . Then the context dataset Dn = {(xi , yi )}ni=1 ⊂ Rd × {±1} and its corresponding ground-truth vector θ ∗ are generated from a joint distribution D as: 1. The ground truth vector θ ∗ is generated from Dθ∗ . 2. Each feature vector xi is generated from Dx , i ∈ [n]. 3. Each label is determined as yi = sign(⟨xi , θ ∗ ⟩), i ∈ [n]. Note that the sign function is invariant to positive rescaling, rendering each label yi determined solely by the direction of θ ∗ . Consequently, we may assume without loss of generality that θ ∗ lies on the unit sphere Sd−1 . For the same reason, we only require the predicted weight θb to approximate b θ ∗ up to its direction, quantified by bθ − θ ∗ 2 . ∥θ∥2 Transformers with softmax attention. We consider solving the in-context weight prediction tasks by transformers. The embedding matrix Z0 for the context dataset Dn = {(xi , yi )}ni=1 , which serves as the input to the transformer, is defined as z 1 z 2 · · · z n 0d Z0 = ∈ R2d×(n+1) , (3.1) 0d 0d · · · 0d θ0

where zi = yi · xi for all i ∈ [n], in alignment with the common settings in linear classification. This choice does not restrict the input format: if the transformer takes the concatenated vector ⊤ [x⊤ i , yi ] as input, as commonly considered in prior works, an embedding layer can transform it into zi = yi xi , and the formal derivations are provided in Appendix A. In addition, θ0 serves as an initialization for the prediction of θ ∗ . With the input matrix in the form of (3.1), a standard self-attention layer (Vaswani et al., 2017) is defined as SA(Z; V, W) = VZsoftmax(Z⊤ WZ + M).

(3.2)

In the formulation above, V, W ∈ R2d×2d denote the value and key-query parameter matrices of the self-attention layer, respectively. Following the convention of most theoretical studies (Zhang et al., 2024b,c; Huang et al., 2024; Wang et al., 2024b; Zhang et al., 2025a), we reparameterize the original key and query matrices into a single trainable parameter matrix W. The mask matrix 0n×(n+1) M= prevents attention to the last query column. To define an L-layer transformer, −∞ · 1⊤ n+1 we denote (V0:L−1 , W0:L−1 ) = (V0 , W0 ), . . . , (VL−1 , WL−1 ) as the collection of parameter pairs across layers. Then, building upon the single self-attention layer defined in (3.2), an L-layer transformer with residual connections and parameters (V0:L−1 , W0:L−1 ) is defined recursively as TF(Z0 ; V0:L−1 , W0:L−1 ) = ZL ∈ R2d×(n+1) , Zl+1 = Zl + SA(Zl ; Vl , Wl ), l = 0, . . . , L − 1.

(3.3)

We read the entries in ZL located at the same position as θ0 in Z0 , i.e., θL = [ZL ]d+1:2d, n+1 , as the predicted weight vector corresponding to the input Z0 . This setup is consistent with the setting in Huang et al. (2025).

Main results

4.1

Overview of Theoretical Results

In this section, we present the theoretical results on how transformers can perform in-context logistic regression and solve in-context weight prediction tasks in classification. Before presenting the technical details, we first provide a high-level roadmap and summary of our these conclusions. We begin by introducing the in-context loss for linear classification, the empirical risk defined on the context dataset. Theorem 4.1 then establishes the expressive-power result that under appropriate parameterizations, an L-layer softmax transformer can exactly implement L steps of normalized gradient descent on this in-context loss. Building on this characterization, Corollary 4.3 applies the implicit bias theory of normalized gradient descent and shows that, for linearly separable context data, the transformers’ output directionally converges to the in-context maximum-margin solution. We then move from expressivity to learning guarantees. In Theorem 4.5, we consider training a single-layer softmax transformer using supervision from a one-step gradient-descent teacher, and show that the trained parameters exactly converge to an NGD-implementing structures. This demonstrates that the NGD mechanism characterized in Theorem 4.1 can be achieved through training, rather than merely existing as an explicit expressivity construction. Finally, Theorem 4.7 validates the O.O.D. generalization behavior of the looped transformer obtained by recurrently applying the trained single-layer block from Theorem 4.5. Under log-concave feature distributions,

Eq (1), (2), (3): Definition of transformers and input formats.

Thm. 3.1: Expressive power The feedforward of � layers of transformers equal to the output of � steps of NGD.

By implicit bias of NGD

Cor. 3.3: Implicit bias The output of transformers directionally converges to the in-context SVM solution.

Assumption:Linear seperable context data.

Def. 2.1: Context data distribution Binary classition context examples generated with a grounth-truth linear classfier �∗ . Lemma 3.4 and Thm. 3.5: Convergence guarantee Under supervision from one-step GD, the parameter matrices of a single-layer transformer converge linearly to the structure characterized in Thm. 3.1, showing that the transformer can indeed be trained to implement NGD.

Additional assumptions: Feature vectors follow Gaussian distribution.

Assump. 3.6: Feature vectors follow log-concave distribution.

Thm. 3.7: O.O.D. generalization For O.O.D. log-concave feature distributions, the trained looped transformer predicts �∗ with the error bounded by depth-dependent implicit bias error + finite-sample statistical error.

Figure 1: High-level roadmap of the theoretical framework, illustrating the assumptions, main results, and logical flow underlying our expressivity, implicit-bias, trainability, and O.O.D. generalization guarantees. Theorem 4.7 shows that its prediction error is controlled by the implicit-bias error from Corollary 4.3 and the finite-sample statistical error of the in-context maximum-margin solution. Figure 1 provides a schematic illustration of this roadmap, highlighting the assumptions, main results, and logical flow underlying our theoretical guarantees.

4.2

Deep transformers can perform in-context logistic regression via normalized gradient descent

To study how transformers solve in-context logistic regression, we define the empirical risk on the context dataset and refer to it as “in-context loss”. Specifically, for any θ ∈ Rd , its in-context loss on Dn = {(xi , yi )}ni=1 is given as n

LICL (θ) =

1X ℓ(⟨θ, yi · xi ⟩), n

(4.1)

i=1

where ℓ(·) : R × R → R is a commonly chosen exponential-tailed loss function, such as the exponential or logistic loss. In this work, we adopt the exponential loss, i.e. ℓ(x) = e−x , to enable cleaner mathematical results. In the following, we show that there exist a class of L-layer transformers that can exactly implement L steps of normalized gradient descent (NGD) on LICL , as formalized in Theorem 4.1. Theorem 4.1. Consider an L-layer transformer TF(·) in (3.3) with parameter matrices (V0:L−1 , W0:L−1 ) of the form 0d×d 0d×d 0d×d −Id Vl = , Wl = α el · Id 0d×d A1,l A2,l

2. Pre-softmax attention scores

1. Input matrix ��

��

... ��

��

�� − ��

��2 ��

��

. . . ��

��

�1

��

�� -��

��

�� + �� =

��

�2

��

��(�� ) −� + �� = �� (�� )

��

-��

0 +

�� + Mask

5. One-step NGD update ��(�� ) �(�0 )

��

... 0

3. Softmax weight ��

-��

...

� � �−�� =1

-��

−∞

�

�−��

...

�

�−�� 0

4. Product ��

��

�−��

��

...

��

�−��

...

�

�−�� 0

��(�� ) = �(�0 ) �(�0 ) �� 1

Figure 2: Illustration of the one-step mechanism in Theorem 4.1. Starting from the input matrix Z0 , a single softmax self-attention layer first constructs the pre-softmax attention scores, then obtains the softmax weight vector sq , and finally computes the attention output that matches one step of normalized gradient descent on the in-context loss LICL . for l = 0, 1, . . . , L − 1, where A1,l , A2,l are arbitrary d × d matrices, and α el > 0. Then for any input matrix Z0 of the form (3.1), the transformer gives hidden layer outputs Zl , l = 1, . . . , L, such that {θl = [Zl ]d+1:2d,n+1 }L l=1 are the iterates of normalized gradient descent on LICL (θ) with learning L−1 rates {e αl }l=0 : θl+1 = θl − α el

∇LICL (θl ) , LICL (θl )

l = 0, 1, . . . , L − 1.

(4.2)

Theorem 4.1 demonstrates that, under appropriate parameterizations, the outputs of softmax attention layers exactly match the iterates of normalized gradient descent applied to the in-context loss LICL . Consequently, the forward pass of an L-layer transformer can be interpreted as performing in-context logistic regression through L steps of normalized gradient descent. Specifically, Figure 2 illustrates how a single softmax self-attention layer constructs the attention weights, recovers the normalized-gradient direction, and implements one step of normalized gradient descent. As an expressive-power result, Theorem 4.1 does not impose any assumptions on the context examples. In addition, the attention-only construction should be understood as a minimal construction that isolates the role of softmax attention: adding common architectural components, such as MLP layers, gated attention, or positional encodings, does not affect the validity of the expressive-power conclusion, since these components can be parameterized so that the resulting model preserves the same input-output mapping as the attention-only transformers defined in (3.3). Moreover, we note that the update rule in (4.2) slightly differs from the standard definition of normalized gradient descent, as it normalizes the gradient by LICL (θ) instead of ∥∇LICL (θ)∥2 . 8

However, this form of normalization term is commonly adopted in theoretical studies of logistic regression (Nacson et al., 2019; Ji and Telgarsky, 2021; Wang et al., 2024a), and is also referred to as normalized gradient descent. Our use of this terminology follows this convention. We further note that if an RMSNorm-style layer normalization is incorporated into the construction, the induced update can be transformed into the standard normalized-gradient-descent form, with normalization by ∥∇LICL (θ)∥2 . Several recent works (Ahn et al., 2023; Bai et al., 2024) show that transformers with linear/ReLU attention can perform in-context linear regression with gradient descent. In comparison, our result in Theorem 4.1 shows that transformers with softmax attention can perform in-context logistic regression with normalized gradient descent. Notably, Bai et al. (2024) also covers results on in-context logistic regression, and shows that multi-head ReLU attention layers can approximate gradient descent updates. However, their results rely on universal approximation by multi-head e −2 ) heads per layer to achieve an approximation error ϵ. In ReLU attention, and require O(ϵ comparison, our result considers softmax attention, only requires a single head per layer, and the correspondence to normalized gradient descent is exact and does not suffer from any approximation error. Remark 4.2. Theorem 4.1 also enjoys an important advantage that it accommodates arbitrary parameterizations of the blocks "A1,l and A2,l within Wl . In fact, this parameterization of Wl can # e 0 −β · Id be further generalized to Wl = d×d , with βe being any positive scalar. Under this more A1,l A2,l general form, it can be shown that L-layer transformers still perform in-context logistic regression e n = {βe · xi , yi }n , and with via normalized gradient descent, but on a rescaled context dataset D i=1 L−1 e rescaled learning rates {e αl /β} l=0 . This flexibility in allowing a broad class of parameterizations for Wl plays a key role in our subsequent analysis in Subsection 4.3. The proof of Theorem 4.1 is also demonstrated for this generalized version in Appendix B Notably, regardless of the distributions Dθ∗ , Dx , any context dataset Dn following Definition 3.1 can always be linearly separated by its corresponding θ ∗ . For such linear separable datasets, a remarkable line of works (Soudry et al., 2018; Ji and Telgarsky, 2019; Nacson et al., 2019; Wang et al., 2024a) have shown that (normalized) gradient descent on logistic/exponential loss has an implicit bias towards the maximum-margin solution θSVM (Dn ) = argmax∥θ∥2 ≤1 mini∈[n] ⟨θ, yi · xi ⟩. Specifically, Theorem 4.3 in Ji and Telgarsky (2021) shows that the L-th iterate of normalized gradient descent with constant learning rate α e converges to the maximum margin solution in direction with a convergence rate O(log(n)/(e αL)). Combining this result and Theorem 4.1, we have the following corollary. Corollary 4.3. Suppose that an L-layer transformer is parameterized as in Theorem 4.1 with α el = α e ≤ O(1) for all l = 0, . . . , L − 1. Then for any context dataset Dn = {(xi , yi )}ni=1 following Definition 3.1 and any θ0 with ∥θ0 ∥2 = O(1), the predicted weight θL by this transformer directionally converges to θSVM (Dn ) as θL log n − θSVM (Dn ) ≤ O . ∥θL ∥2 α eL Corollary 4.3 indicates that the predicted weight θL produced by L-layer transformers converges in direction to its maximum margin solution θSVM (Dn ) at a rate inversely proportional to L. With 9

this result, evaluating the quality of θL as a weight predictor for θ ∗ reduces to characterizing the discrepancy between θSVM (Dn ) and θ ∗ . We elaborate this in Subsection 4.4.

4.3

Training of single softmax-attention layer

In the previous section, we have shown that under appropriate parameterizations, transformers can perform in-context logistic regression via normalized gradient descent. However, this result only demonstrates the expressive power of transformers. To give a more comprehensive analysis, in this section, we investigate whether such transformers can indeed be obtained via training. An interesting observation is that, the parameterizations in Theorem 4.1 naturally admit a looped implementation, where all layers share the same weights, i.e., V = Vl and W = Wl for all l ∈ [L]. In addition, Geiping et al. (2025) empirically demonstrates that recurrently applying the trained block enables transformers to achieve better performance at the inference stage. Motivated by these observations, we consider an effective training setup: we first train a single-layer transformer TF(·; V, W), and then obtain a multi-layer looped transformer by recurrently applying this trained layer. Notably, we adopt the one-step gradient descent, rather than normalized gradient descent, as a “teacher model” to supervise the single-layer transformer. This setup is inspired by similar settings considered in Huang et al. (2025), and allows us to test whether the model can still learn normalized gradient descent even if the teacher is a different algorithm. The one-step GD update on the context data can be expressed as θGD = θ0 − α∇LICL (θ0 ), where θ0 represents the initialization, and α denotes the learning rate for one-step GD update. We consider minimizing the discrepancy between θGD and the output of the single-layer transformer, i.e. θ1 = [TF(Z0 , V, W)]d+1;2d,n+1 . The training objective is defined as the population mean-squared error: Ltrain (V, W) = EDn ,θ0 ∥θ1 − θGD ∥22 . The expectation is taken over the context dataset Dn and the initialization θ0 , where θ0 is assumed to follow U(Sd−1 ). Moreover, we assume that Dn is generated following Definition 3.1, with the feature distribution Dx being N (0, σ 2 Id ), and the true classifier distribution Dθ∗ being U(Sd−1 ). We consider using gradient descent with zero initialization V(0) , W(0) = 02d×2d to minimize the training loss Ltrain : V(t+1) = V(t) − η∇V Ltrain (V(t) , W(t) );

(4.3)

W(t+1) = W(t) − η∇W Ltrain (V(t) , W(t) ),

(4.4)

where η denotes the learning rate. Our goal is then to theoretically study this training procedure defined above and verify whether the trained transformer learns to perform one-step normalized gradient descent. Our first observation is that the iterates V(t) , W(t) of gradient descent always preserve certain structured forms, which is summarized in the following lemma. Lemma 4.4. The iterates V(t) and W(t) of the training procedure (4.3), (4.4) always follow a

structured form as "

# " # (t) 0 0 0 −C2 ·Id d×d d×d V(t) = (t) , W(t) = d×d , C1 ·Id 0d×d 0d×d 0d×d (t)

(t)

where C1 and C2 are two scalar coefficients. Lemma 4.4 plays a key role in our training analysis: it reduces the original optimization problem concerning the evolutions of full d × d parameter matrices V, W to a much simpler one involving (t) (t) only two scalars C1 , C2 . The coefficient vector C(t) = [C1 , C2 ]⊤ equivalently follows gradient descent starting from zero initialization C(0) = 0 to minimize a proxy training loss: C(t+1) = C(t) − η∇C Letrain (C(t) ), 0d×d 0d×d 0d×d −C2 ·Id e Ltrain (C) = Ltrain , . C1 ·Id 0d×d 0d×d 0d×d The following theorem characterizes the convergence of this equivalent training procedure. Theorem 4.5. Suppose that n = Ω(d2 ), η ≤ O n1 , and α, σ ≤ O(1). Then the following results hold. 1. Invariant compact set R. For all t ≥ 0, the iterate C(t) always remains in a compact set R defined as 2

R = [0, 2αeσ /2 ] × [0, 2]. 2. Unique local minimizer in R. The loss Letrain (C) has a unique local minimizer C∗ = [C1∗ , C2∗ ]⊤ in R. In addition, this local minimizer satisfies that 2 C∗ − [αeσ /2 , 1]⊤ 2 ≤ O d1 . 3. Linear convergence of the loss and iterates. For t ≥ 0, the training loss enjoys a linear convergence rate: Letrain (C(t) ) − Ltrain (C∗ ) ηµ1,α,σ t e ≤ 1− Ltrain (C(0) ) − Letrain (C∗ ) . d Moreover, the iterates C(t) converges linearly to C∗ : (t)

ηµ1,α,σ t/2 ∗ − C 2 ≤ µ2,α,σ 1 − ∥C ∥2 . d ∗

Here, µ1,α,σ and µ2,α,σ are positive constants solely depending on α and σ. Theorem 4.5 establishes rigorous training convergence guarantees. The first and second conclusions describe the loss landscape of Letrain and proves the existence of a unique local minimizer C∗ . The third conclusion gives accurate convergence guarantees with linear rates. Importantly, by 11

(t)

the second and third conclusions, as t → ∞, one has C1 ≈ αeσ /2 and C2 ≈ 1, which implies that the trained transformer layer approximately matches the form of our constructed layers in Theorem 4.1. This demonstrates that: The trained transformer can indeed perform normalized gradient descent update, even though the model is supervised by a gradient descent teacher. This reveals a nontrivial separation between the supervising algorithm and the learned in-context algorithm, suggesting that transformers are not merely algorithm imitators, but may also discover (t) algorithmic mechanisms distinct from the supervising algorithm. Moreover, the fact that C2 is not exactly one does not affect the conclusion that the trained transformer layer can exactly perform one-step normalized gradient descent. As discussed in Remark 4.2, the coefficients C1 and C2 admit a clear algorithmic interpretation: C2 acts as the rescaling factor for feature vectors, and the ratio C1 /C2 determines the learning rate. Therefore, the trained single-layer transformer in Theorem 4.5 essentially performs one step of normalized gradient descent on the slightly rescaled 2 (t) (t) (t) dataset {(C2 · xi , yi )}ni=1 , with the learning rate C1 /C2 ≈ αeσ /2 . From a technical perspective, Theorem 4.5 introduces new theoretical tools. While a line of recent works have studied the training of softmax transformers (Jelassi et al., 2022; Wang et al., 2024b; Li et al., 2024b; Zhang et al., 2025a; Shi and Cao, 2025), we note that existing analyses are mostly under the setting where the learning tasks can be perfectly solved by having softmax attention perform certain “sparse selection”. As a result, existing convergence guarantees mostly focus on showing that certain pre-softmax attention scores diverge to infinity, and that they diverge at a faster rate compared to the rest of the scores. In comparison, the learning task we consider is fundamentally different in multiple aspects. First, since the “teacher model” is gradient descent, the learning task is “misspecified” and zero training loss cannot be perfectly achieved. In addition, as is shown in Theorem 4.5, training converges to a finite minimizer C∗ instead of giving diverging parameters in W. More importantly, the model with parameters defined by C∗ does not perform −C ∗ ⟨zi ,θ0 ⟩ “sparse selection”, as the softmax score from the last token to the i-th token is Pn e 2−C ∗ ⟨z ′ ,θ0 ⟩ , i′ =1 e

which defines a dense, weighted average over all the tokens. Finally, for our learning task, the training loss and its gradient do not admit closed-form expressions, further complicating the optimization analysis. To overcome these challenges, we develop several novel proof techniques, which are summarized in the brief proof sketch as follows. Step 1. We derive explicit non-asymptotic approximations of the gradients (Lemma C.2), and 2 show that [αeσ /2 , 1]⊤ is a fixed point of the approximated training process. Step 2. We then apply the Newton–Kantorovich theorem to show the existence of a fixed point 2 C∗ of the original training process that is close to [αeσ /2 , 1]⊤ (Lemma C.7). Step 3. We further prove a Polyak-Lojasiewicz (PL) inequality (Lemma C.8) despite the nonconvexity of the training loss, which leads to the linear convergence rate.

4.4

Multi-layer looped transformers efficiently solve in-context weight prediction

Theorems 4.1 and 4.5 together show that we can recurrently apply the trained transformer layer characterized in Theorem 4.5 to obtain a multi-layer looped transformer that solves in-context logistic regression via normalized gradient descent. In this section, we establish the final theoretical guarantee on the performance of such looped transformers in solving in-context weight prediction. In contrast to the assumption that Dx during training follows N (0, σ 2 Id ), here we study out-of12

distribution (O.O.D.) generalization performance on new “test” in-context datasets Dn for which Dx is a general log-concave distribution. Assumption 4.6 (Log-concave distribution). For the feature distribution Dx in Definition 3.1, let f (·) be its probability density function. Then it holds that 1. Log-concavity: log[f (x)] is a concave function. 2. Moment conditions: For any x ∼ Dx , it holds that E[x] = 0, and E[xx⊤ ] = Σ ≻ 0. Assumption 4.6 covers a broad class of distributions, such as centered Gaussian, uniform, and Laplace distributions. Based on it, we have the following theorem. (t) (t) Theorem parameter matrices trained in Theorem 4.5 after t = 4.7. Let V and W be(t)the d e Ω iterations. Denote TF(·, [V ]⊗L , [W(t) ]⊗L ) the L-layer looped transformer, with V(t)

ηµ1,α,σ and W(t)

being its shared weights across layers. Suppose that the feature distribution Dx in Definition 3.1 follows Assumption 4.6. Then for any input matrix Z0 of the form (3.1), with probability at least 1 − δ, the looped transformers’ prediction θL = [TF(Z0 , [V(t) ]⊗L , [W(t) ]⊗L )]d+1:2d, n+1 satisfies log n d log ρ θL + . (4.5) − θ∗ ≤ O ∥θL ∥2 αL n 2 −1 where ρ = max{n, d, λ−1 min (Σ), δ },

Theorem 4.7 provides an O.O.D. generalization guarantee for the looped transformer to solve in-context weight prediction. We note that in recent theoretical studies of ICL (Zhang et al., 2024b; Frei and Vardi, 2025; Huang et al., 2025), the generalization bounds are typically presented in expectation over the distribution of the test context dataset Dn . In comparison, Theorem 4.7 establishes a point-wise high-probability guarantee, which holds for any fixed input Z0 . In particular, by choosing δ = O nd , the conclusion of Theorem 4.7 can immediately induce an in-expectation n d log ρ bound EDn ∥θθLL∥2 − θ ∗ 2 ≤ O log , validating that our result is stronger than the clasαL + n sic in-expectation generalization bound. Moreover, this O.O.D. generalization ability stems from the good property that the output θL in direction converges to the maximum margin solution θSVM (Dn ). It is natural to decompose ∥θθLL∥2 − θ ∗ 2 ≤ ∥θθLL∥2 − θSVM (Dn ) 2 + ∥θSVM (Dn ) − θ ∗ ∥2 by triangle inequality. The first term ∥θθLL∥2 − θSVM (Dn ) 2 is controlled in Corollary 4.3, directly n ∗ yielding the term O log αL . In addition, the second term ∥θSVM (Dn ) − θ ∥2 quantifies how well the maximum-margin solution learned from the context dataset Dn approximates the true classifier θ ∗ . d log ρ The upper bound for this statistical error is demonstrated to be O n , corresponding to the second term of generalization bound. The detailed proof for Theorem 4.7 is deferred to Appendix D. Several recent works Frei and Vardi (2025); Shen et al. (2025) investigate how single-layer transformers with linear attention can be trained to solve in-context classification on Gaussianmixture data, and establish in-distribution generalization. In contrast, our Theorem 4.7 establishes O.O.D. generalization for multi-layer transformers with softmax attention. Compared with another recent work Bai et al. (2024), our work gives better bounds thanks to the fast convergence rate of normalized gradient descent. Specifically, as is discussed above, the first term in the bound of Theorem 4.7 quantifies the in-direction convergence of θL towards the maximum-margin solution, n e and the rate is given by Corollary 4.3. Consequently, as long as L = Ω αd , the generalization 13

e d , which matches the sample complexity lower bound bound can be given as ∥θθLL∥2 − θ ∗ 2 ≤ O n in classic PAC learning (Long, 1995). In contrast, Bai et al. (2024) constructs multi-head ReLU transformers that approximate standard gradient descent for in-context logistic regression. If similar analyses are applied to their setting, then by the implicit bias results of gradient descent (Soudry et al., 2018), their constructed model’s output approaches the maximum-margin solution only at the rate O logloglogL L . As a result, unless the depth of the model L is exponentially large in the e d , and therefore problem parameters, this term always dominates the statistical error term O n

fundamentally limits the performance in solving the weight prediction task.

Experiments

In this section, we present the experimental results. We consider three experimental settings: (i) training a single-layer transformer; (ii) constructing a multi-layer looped transformer from the trained layer, and evaluating its O.O.D. generalization in solving weight prediction; (iii) training a multiple-layer looped transformer from scratch, and evaluating its capacity in solving weight prediction.

5.1

Training a single-layer transformer

We first consider training a single-layer transformer to validate our Theorem 4.5. The architecture of single-layer transformer follows the definition in (3.3) with L = 1, and the training strategy aligns with the theoretical settings in Subsection 4.3. We adopt an online gradient descent algorithm to simulate training over the population loss Ltrain . Specifically, at each iteration, we generate a new batch of K = 400 context datasets {Dn,k }K k=1 , where each dataset Dn,k is generated following Definition 3.1 with Dx being N (0d , Id ) and Dθ∗ being U (Sd−1 ). For each Dn,k , we generate a corresponding θ0,k from U(Sd−1 ). Then we can obtain a batch of K input matrices {Z0,k }K k=1 embedded of the form in (3.1), and the gradient descent update in (4.3), (4.4) is conducted on this batch of inputs, with the learning rate η = 0.1. In addition, we consider two gradient descent teachers θGD with α = 0.5 and α = 1, respectively. For each case, we conduct experiments under three different configurations: (n, d) ∈ {(60, 20), (100, 25), (150, 30)}. Figure 3 reports the curves of training losses. In all settings, the training losses consistently converge near zero. Notably, configurations with a larger dimension d exhibit slower convergence. This observation aligns with our Theorem 4.5, as the linear convergence factor 1 − ηµ1,α,σ /d grows with d, thereby slowing down the optimization process. Figures 4 displays the heatmaps of the parameter matrices V(t) and W(t) obtained after training. These results demonstrate that the trained V(t) and W(t) follow the structured pattern described in Lemma 4.4: the bottom-left block of V and the top-right block of W are almost proportional to the identity matrix, with coefficients C1 > 0 and −C2 < 0, respectively, and all other blocks remain almost zero. (t) (t) Figure 5 further presents the trajectories of C(t) = [C1 , C2 ]⊤ . The trajectories exhibit clear convergence behavior, as evidenced by the dense accumulations of iterates near the end of the 2 curves. In addition, in all settings, the iterates consistently converge to points close to [αeσ /2 , 1]⊤ , aligning with the third conclusion in Theorem 4.5. The experiments in Figures 3, 4, and 5 all match our theoretical conclusions regarding the training of a single-layer transformer, validating that a single-layer transformer can be trained to

(a) Training loss, α = 0.5

(b) Training loss, α = 1

Figure 3: Training loss under two settings: α = 0.5, and α = 1. conduct a normalized gradient descent update.

5.2

O.O.D. generalization of looped transformers

Following our theoretical settings, we can obtain a multi-layer looped transformer by recurrently applying the trained single-layer transformer. In this section, we conduct experiments to validate the O.O.D. generalization of the resulting looped transformers in solving in-context weight prediction. To make sure that each O.O.D. setting covers significantly different distributions compared with the training data, we consider three different O.O.D. choices for Dx : we first marginally same from (i) standard Gaussian distribution N (0, 1); (ii) Laplace ple each entry of the random vector x distribution Laplace(0, 1); (iii) uniform distribution U([0, 1]), and then randomly generate a positive definite matrix Σ and obtain a sample x from Dx by calculating x = Σe x. For each choice of following exactly the same data generaDx , we generate a batch of K input matrices {Z0,k }K k=1 tion procedure as in training, except that the feature distribution is replaced by the corresponding O.O.D. distribution. For each setting, we consider feature dimensions d ∈ {20, 30}, and fix the in-context sample size as n = 500. We evaluate the performance of the looped transformers in in-context weight prediction. For better comparison, we also report the results from the iterates of NGD and standard GD under the same experimental settings. The results are given in Figure 6. Across all different settings, the discrepancy between the prediction produced by looped transformers and the ground-truth classifier θ ∗ consistently decays to a small value as the number of layers L increases, validating the capacities of deep transformers in solving in-context weight prediction. In addition, we observe that the hidden-layer outputs of the looped transformers remain very close to the iterates of NGD throughout the entire process, achieving nearly identical performance in in-context weight prediction and consistently outperforming standard GD. These observations further support our theoretical findings that multi-layer transformers can efficiently solve in-context weight prediction in in-context logistic regression via NGD.

(n, d) = (60, 20)

0.8

(n, d) = (60, 20)

1.00

0.6 0.4

0.50

0.2

0.25

0.5

0.25

0.0

0.00

0.0

0.00

0.2

0.25

0.4

0.50

0.6

0.75

0.8

1.00

0.8

(n, d) = (100, 25)

0.75

1.0

0.50

0.25

0.5

0.50

1.0

0.75

1.5

(b) W(t) , α = 0.5

(d) W(t) , α = 1

(n, d) = (100, 25)

1.00

(n, d) = (100, 25) 1.5

1.00

0.6

0.75

0.4

0.50

0.2

0.25

0.5

0.25

0.0

0.00

0.0

0.00

0.2

0.25

0.4

0.50

0.6

0.75

0.8

1.00

(e) V(t) , α = 0.5

0.75

1.0

0.50

0.25

0.5

0.50

1.0

0.75

1.5

(f) W(t) , α = 0.5

(n, d) = (150, 30)

(g) V(t) , α = 1

(n, d) = (150, 30)

1.00

(h) W(t) , α = 1

(n, d) = (150, 30)

0.8

(n, d) = (150, 30) 1.5

1.00

0.6

0.75

0.4

0.50

0.2

0.25

0.5

0.25

0.0

0.00

0.0

0.00

0.2

0.25

0.5

0.4

0.50

0.6

0.75

0.75 1.0

0.50

0.25 0.50

1.0

0.75

1.5

0.8

(i) V(t) , α = 0.5

(n, d) = (60, 20) 1.5

0.75

(a) V(t) , α = 0.5 (n, d) = (100, 25)

(n, d) = (60, 20)

(j) W(t) , α = 0.5

(k) V(t) , α = 1

1.00

(l) W(t) , α = 1

Figure 4: Heatmaps of the parameter matrices V(t) and W(t) when the training loss converges. The three rows correspond to (n, d) = (60, 20), (100, 25), and (150, 30), respectively. In each row, the four panels show V(t) and W(t) under α = 0.5 and α = 1.

5.3

Training of multi-layer looped transformers

In this section, we consider training a multi-layer looped transformer from scratch. We consider ∗ d−1 to supervise the training, i.e. the training loss directly using the θground ∗truth vector θ ∈ S 2 is defined as E ∥θLL∥2 − θ 2 . Similar to the previous section, we consider using online gradient descent to minimize this training loss. In addition, each batch of inputs matrices {Z0,k }K k=1 follows the same generation process with Dx being a Gaussian distribution with a randomly generated positive definite covariance matrix Σ. We set the in-context sample size and feature dimension as (n, d) = (60, 20). The experiments are conducted under three sets of the model depth L ∈ {5, 10, 20}. The results are given in Figure 7. Since the models are trained in an online fashion, the training loss itself serves as a direct measure of the generalization performance for in-context weight prediction. Figure 7a shows that deeper models achieve lower training loss and converge faster, indicating that increasing the depth significantly improves the quality of in-context weight prediction. This behavior is consistent with our theoretical analysis, as the depth L corresponds to the number of iterations of NGD. Moreover, Figures 7b, 7c display the heatmaps of the learned parameter matrices V and W of 20-layer looped transformer. The clear block-diagonal and structured patterns closely match the pattern predicted in Theorem 4.1. These results empirically demonstrate that

(a) Trajectories, α = 0.5

(b) Trajectories, α = 1 (t)

(2)

Figure 5: Trajectories of the coefficient C1 and C1 α = 1.

under two different settings that α = 0.5, and

even when trained from scratch, deep looped transformers naturally learn the parameter structures required to implement in-context logistic regression.

Conclusions and limitations

This work provides a comprehensive analysis of how transformers with softmax attention perform ICL on linear classification data. Specifically, we construct a class of softmax transformers capable of performing in-context logistic regression. We demonstrate that these transformers can be obtained by training a single-layer model and recurrently applying the trained layer. Furthermore, we establish an O.O.D. generalization bound for the trained model in in-context weight prediction. Experimental results back up our theoretical findings, highlighting the pivotal role of transformers’ depth in ICL. There are several limitations of our analysis. First, our theory focuses on in-context linear classification with exponential loss, which is a simplified setting compared with the broad range of tasks and data distributions encountered by modern transformers. Nevertheless, this setting allows us to isolate the role of softmax attention and rigorously characterize a nontrivial algorithmic mechanism, namely the implementation and learning of normalized gradient descent. Second, our expressive-power result is established for an attention-only construction. Although this does not invalidate the conclusion for richer architectures, since additional components such as MLP layers, gated attention, and positional encodings can be parameterized so as to preserve the same input-output mapping, our analysis does not fully characterize how these components interact with the NGD mechanism during training. Lastly, while we observe in experiments that direct, end-to-end training of a multi-layer looped transformer can also give a model that matches our theoretical construction well, our theoretical training analysis currently can not directly extend to this setting. Addressing these limitations, including extending the analysis to richer architectures, broader task and data settings, and end-to-end training of multi-layer transformers, is an interesting and promising direction for future work.

d=20

ngd_weight_prediction gd_weight_prediction tf_weight_prediction tf_ngd_diff

* 2

1.5

1.0

L L 2

1.0

L L 2

* 2

1.5

d=30

2.0

ngd_weight_prediction gd_weight_prediction tf_weight_prediction tf_ngd_diff

0.5

0.0

5 10 15 20 25 Number of layers / Iterations L

(a) Gaussian data, d = 20

* 2

1.0

L L 2

* 2 L L 2

d=30 ngd_weight_prediction gd_weight_prediction tf_weight_prediction tf_ngd_diff

1.5

0.5 0.0

5 10 15 20 25 Number of layers / Iterations L

1.75 1.50 1.25 1.00 0.75 0.50 0.25 0.00

ngd_weight_prediction gd_weight_prediction tf_weight_prediction tf_ngd_diff

d=30

2.0

ngd_weight_prediction gd_weight_prediction tf_weight_prediction tf_ngd_diff

* 2

1.5 1.0 L L 2

* 2 L L 2

5 10 15 20 25 Number of layers / Iterations L (d) Laplace data, d = 30

d=20 1.75 1.50 1.25 1.00 0.75 0.50 0.25 0.00

(b) Gaussian data, d = 30

d=20 2.0

5 10 15 20 25 Number of layers / Iterations L

0.5

5 10 15 20 25 Number of layers / Iterations L

0.0

(e) Uniform data, d = 20

5 10 15 20 25 Number of layers / Iterations L

(f) Uniform data, d = 30

Figure 6: Evaluation of the in-context weight-prediction produced by looped transformers, NGD iterates, and standard GD iterates on different O.O.D distributed context datasets, with d ∈ {20, 30}.

(a) Training loss for L ∈ {5, 10, 20}

(b) Heat map of V(t) , L = 20

Figure 7: Training loss curves L ∈ {5, 10, 20} and the heatmaps of parameter matrices V(t) and W(t) of 20-layer looped transformers.

Acknowledgments We would like to thank the anonymous reviewers and area chairs for their helpful comments. Yuan Cao is supported in part by NSFC 12301657, Hong Kong RGC ECS 27308624, and Hong Kong RGC GRF 17301825.

Embedding concatenated inputs into yi · xi

In this section, we show that the embedding vector zi = yi · xi used in Eq. (3.1) can be exactly ⊤ obtained from the concatenated input [x⊤ i , yi ] through a standard embedding layer. Therefore, our ⊤ analysis also applies to the setting where each in-context example is provided in the form [x⊤ i , yi ] . Lemma A.1. Suppose y ∈ {−1, 1} and ∥x∥∞ ≤ M . Let ReLU(·) denote the ReLU activation applied entrywise. Define x e + b1 ), e= f (e x; W1 , b1 , W2 ) = W2 ReLU(W1 x x ∈ Rd+1 , y where  Id M · 1d −Id M · 1d  4d×(d+1)  W1 =  ,  Id −M · 1d  ∈ R −Id −M · 1d 

b1 = −M · 14d ,

and W2 = Id −Id −Id Id ∈ Rd×4d . Then f (e x; W1 , b1 , W2 ) = y · x. Proof of Lemma A.1. We prove the claim by considering the two possible values of y. First, suppose y = 1. Then   x   x −x . W1 + b1 =   1 x − 2M · 1d  −x − 2M · 1d Since ∥x∥∞ ≤ M , we have x − 2M · 1d ≤ 0d ,

−x − 2M · 1d ≤ 0d

entrywise. Hence   ReLU(x) ReLU(−x) x . ReLU W1 + b1 =    1 0d 0d Therefore, x f ; W1 , b1 , W2 = ReLU(x) − ReLU(−x) = x. 1 Here we used the identity ReLU(a) − ReLU(−a) = a entrywise.

Next, suppose y = −1. Then  x − 2M · 1d −x − 2M · 1d  x . + b1 =    −1 x −x 

Again, since ∥x∥∞ ≤ M , the first two blocks are entrywise non-positive. Thus   0d   x 0d . + b1 =  ReLU W1  −1 ReLU(x)  ReLU(−x) Therefore, f

x ; W1 , b1 , W2 = −ReLU(x) + ReLU(−x) = −x. −1

Combining the two cases gives x f ; W1 , b1 , W2 = y · x, y for every y ∈ {−1, 1} and every x satisfying ∥x∥∞ ≤ M . ⊤ Consequently, if the input examples are given as concatenated vectors [x⊤ i , yi ] , an embedding layer of the above form maps them exactly to zi = yi ·xi before they are fed into the attention layers. Since embedding layers are standard components of transformer architectures, the use of zi = yi · xi in Eq. (3.1) does not prevent the result from applying to the concatenated-input formulation.

Proof of Theorem 4.1

In this section, we provide a detailed proof for Theorem 4.1. As we have mentioned in the discussions of Theorem " # 4.1, the parameterization of Wl can be further relaxed to the form as e 0d×d −β · Id Wl = , with βe being any positive scalar. Therefore, instead of directly proving A1,l A2,l Theorem 4.1, we prove a generalized version which allows Wl to be parameterized as above. The presentation and detailed proof is provided in the following. Theorem B.1 (Generalized version of Theorem 4.1). Consider an L-layer transformers TF(·) as defined in (3.3), whose parameter matrices (V0:L−1 , W0:L−1 ) satisfy that for l = 0, 1, . . . , L − 1, # " 0d×d 0d×d 0d×d −βe · Id , Vl = , Wl = α el · Id 0d×d A1,l A2,l where A1,l , A2,l are arbitrary d × d-dimensional real matrices, and α el , βe > 0. For each l ∈ [L], let θl = [Zl ]d+1:2d,n+1 . Then the sequence {θl }L l=1 corresponds to iterates obtained by implementing

L-steps normalized gradient descent on LeICL (θ): for l = 0, 1, . . . , L − 1, θl+1 = θl −

α el ∇LeICL (θl ) , βe LeICL (θl )

(B.1)

where α el /βel denotes the learning rate of l-th iterative step. Specifically, the LeICL denotes the in-context loss defined on the rescaled context dataset {(βe · xi , yi )}ni=1 as 1 LeICL (θ) = n

n X

e i · xi ⟩). ℓ(⟨θ, βy

i=1

It is evident that Theorem B.1 cover Theorem 4.1 when βe = 1. We first introduce a notation that Zcontext = [z1 , . . . , zn ] ∈ Rd×n , where zi = yi · xi . Now, we are ready to prove Theorem B.1. Proof of Theorem B.1. We define θel as the iterates of normalized gradient descent with 0 initialization, namely, ∇LeICL (θel ) θel+1 = θel − α e , LeICL (θel )

l = 0, 1, . . . , L − 1,

where θe0 = 0d . In simple terms, the only distinction between θel and θl is that they have different initializations. If θ0 = 0d , then θel = θl for all l ∈ [L]. In the next, we prove that Zl can be formulated as # " Zcontext 0d Zl = . (B.2) θl θel 1⊤ n Then it is evident that Theorem 4.1 follows directly from (B.2), and we proceed by induction. Notice that the initial condition Z0 satisfies (B.2). For the inductive hypothesis, we assume that Zl satisfies (B.2) and then demonstrate that this property is preserved for Zl+1 . For Zl formulated in (B.2), we can calculate that " #" #" # e⊤ 0d×d −βe · Id Zcontext 0d Z⊤ 1 θ n ⊤ l Zl WZl = context 0⊤ θl⊤ A1 A2 θel 1⊤ θl n d " #" # ⊤A e Z 0 1n θel⊤ A1 −βe · Z⊤ + 1 θ context d n l 2 context = θl⊤ A1 θl⊤ A2 θel 1⊤ θl n " # ⊤A θ e · Z⊤ el 1⊤ −βe · Z⊤ e 1n θel⊤ (A1 Zcontext + A2 θel 1⊤ ) − β θ θ + 1 θ n l 2 l n context n context l = . ⊤A θ θl⊤ A1 Zcontext + θl⊤ A2 θel 1⊤ θ 2 l n l Notice that the softmax operation in the self-attention layer (3.2) is applied column-wise. Consequently, all rank-one blocks proportional to 1n yield identical values across coordinates, and hence can be omitted as they do not affect the softmax output. Moreover, due to the presence of the mask matrix M, the attention weights corresponding to the last query column remain zero. Combining

all these observations, we can calculate that e−⟨β·z1 ,θ⟩ ⊤ Pn e e 1n  i=1 ee−⟨β·ze i ,θ⟩  e−⟨β·z2 ,θ⟩ ⊤  Pn −⟨β·z e 1n  i=1 e e i ,θ⟩



 softmax(Z⊤ l WZl + M) =  ...  −⟨β·z e n ,θ⟩ e Pe ⊤ e ,θ⟩ e 1n  n e−⟨β·z i i=1

0⊤ n

e−⟨β·z1 ,θ⟩ Pn e ,θ⟩ −⟨β·z i  i=1 e e  −⟨ β·z ,θ⟩ 2 e  Pn e ,θ⟩ −⟨β·z  i e i=1







′ e e − ℓ (⟨β·z1 ,eθ⟩) 1⊤ n  LeICL (θ)  ℓ′ (⟨β·z e 2 ,θ⟩) e ⊤ − e e 1n  L (θ)

′ e − ℓ (⟨eβ·z1 ,θ⟩) LICL (θ)   e ℓ′ (⟨β·z − e 2 ,θ⟩)  LICL (θ) 

LICL (θ) 0⊤ n

LICL (θ)

ICL = . ... ...       e n ,θ⟩ ′ (⟨β·z ′ (⟨β·z e e e −⟨ β·z ℓ , θ⟩) ℓ ,θ⟩) n n  −  e ⊤ − 1 Pn e n    e e e −⟨β·z ,θ⟩

...

i=1 e

Based on this result, it can be directly calculated that " # " # " # 0d×n 0d Zcontext 0d Zcontext 0d e e e Zl+1 = Zl + SA(Zl ; V, W) = = e + , ⊤ θ − αeel ∇eLICL (eθ) 1⊤ − αeel ∇eLICL (θ) θel 1⊤ θl θ 1 l+1 l+1 n n n (θ) β L β L (θ) ICL

ICL

e We demonstrate that (B.2) holds where the last equality holds by the iterative rules for θ and θ. for Zl+1 and hence complete the proof.

Proof of Lemma 4.4 and Theorem 4.5

In this section, we provide a comprehensive proof for Lemma 4.4 and Theorem 4.5. We first introduce a notation regarding the blocks inner parameter matrix V and W. To better align with the parameter patterns defined in Lemma 4.4, we express V and W into the form of four blocks as V11 V12 W11 W12 V= , W= , V21 V22 W21 W22 where Vk1 ,k2 , Wk1 ,k2 ∈ Rd×d for all k1 , k2 ∈ [2]. Since Lemma 4.4 and Theorem 4.5 contain too many results, we separate these contents into several lemmas and theorems and prove them respectively. Specifically, the conclusion of Lemma 4.4 is separated into Lemma C.1 and C.2. Lemma C.1 provides the gradient calculations of (4.3) and (4.3), and demonstrates that except V21 and W12 , (t) (t) other blocks of V and W always remain 0. Lemma C.2 demonstrates that V21 = C1 Id and (t) (t) (t) (t) W12 = −C2 Id , and further provides the updating rules for C1 and C2 . Theorem 4.5 is separated into Lemma C.5, C.7, and Theorem C.9, corresponding to the three conclusion respectively. Now, we start our proof. Lemma C.1 (Restatement of Lemma 4.4, part I). For the blocks V11 , V12 , V22 , W11 , W21 , W22 , (t) (t) (t) (t) (t) (t) the gradient of loss with respect to them remain zero, implying that V11 , V12 , V22 , W11 , W21 , W22 = 0d×d for all t ≥ 0. For V21 and W12 , their gradient can be expressed as # " X n n n X α X −⟨zi ,θ0 ⟩ ⊤ e zi − V21 zi si zi s i ; ∇V21 Ltrain (V, W) = −E n i=1 i=1 i=1 " X # n n X ⊤ ⊤ ⊤ α −⟨zi ,θ0 ⟩ ∇W12 Ltrain (V, W) = −E Zcontext diag(s) − ss Zcontext V21 e zi − V21 zi si θ0⊤ . n i=1

i=1

Proof of Lemma C.1. Notice that the mask matrix M prevents attending to the last query column, 23

resulting in that ⊤ softmax(Z⊤ W[0d , θ0 ]⊤ + M) = softmax [Z⊤ ,0 context , 0n×d ]W[0d , θ0 ] ⊤ n+1 = softmax Zcontext W12 θ0 , 0 ∈ R , where the last equation is directly simplified as the zero blocks quadratic form have no effect in the d , we can rewrite that on the final results. By denoting s = softmax Z⊤ W θ ∈ R 12 0 context [SA(Z0 , V, W)]d+1:2d,n+1 = V21 Zcontext s + V22 0d×n s = V21 Zcontext s. This further implies that θ1 = [TF(Z0 ; V, W}]d+1:2d,n+1 = θ0 + [SA(Z0 , V, W)]d+1:2d,n+1 = θ0 + V21 Zcontext s. Following a similar calculations in Wang et al. (2024b), we can obtain that " # ⊤ n α X −⟨zi ,θ0 ⟩ dLtrain (V, W) =E − e zi − V21 Zcontext s dV21 Zcontext s n i=1 # " ⊤ n α X −⟨zi ,θ0 ⟩ e zi − V21 Zcontext s V21 Zcontext dsoftmax Z⊤ +E − context W12 θ0 n i=1 " # ⊤ n α X −⟨zi ,θ0 ⟩ =E − e zi − V21 Zcontext s dV21 Zcontext s n i=1 # " ⊤ n α X −⟨zi ,θ0 ⟩ ⊤ ⊤ +E − e zi − V21 Zcontext s V21 Zcontext diag(s) − ss Zcontext dW12 θ0 . n i=1

From the differential calculation above, we directly conclude that only the gradients with respect to the blocks V21 and W12 are non-zero, while those of other blocks remain zero throughout the training process, as the loss is irrelevant with them. Therefore, we conclude that for all t ∈ (t) (t) (t) (t) (t) (t) N, V11 , V12 , V22 , W11 , W21 , V22 = 0d×d . For the block V21 and W12 , their gradients can be expressed as " # X n n n X α X −⟨zi ,θ0 ⟩ ∇V21 Ltrain (V, W) = −E e zi − V21 zi si z⊤ i si ; n i=1 i=1 i=1 " # X n n X α ⊤ ∇W12 Ltrain (V, W) = −E Zcontext diag(s) − ss⊤ Z⊤ e−⟨zi ,θ0 ⟩ zi − V21 zi si θ0⊤ . context V21 n i=1

i=1

This completes the proof. Lemma C.1 demonstrates that except V21 and W12 , other blocks of parameter matrices V and W remains zero throughout the training. To finish the proof for the specific patterns in Lemma 4.4, (t) (t) (t) it suffices to show that there exist two time-dependent scalars C1 and C2 , such that V21 = C1 ·Id (t) and W12 = −C2 · Id for all t ≥ 0, which are demonstrated in the following Lemma C.2. Lemma C.2 (Restatement of Lemma 4.4, part II). Under the same conditions of Theorem 4.5,

(t)

there exist time dependent scalars C1 and C2 such that for t ≥ 0, (t)

(t)

V21 = C1 · Id ; (t)

(t)

W12 = −C2 · Id .

(t)

In addition, C1 and C2 follow the iterative rules as " (t) (t) 2 σ2 ησ 2 d C2(t) σ2 F1,σ (C2 ) F2,σ (C2 ) (t+1) (t) (t) 2 C1 =C1 + αe 2 + + C2 σ + e + d π n n d # (t) (t) d [C2(t) ]2 σ2 F3,σ (C2 ) F4,σ (C2 ) (t) 2 (t) 2 2 − C1 + ; + [C2 ] σ + e + π n n d " (t) 2 (t) (t) (t) σ2 F5,σ (C2 ) F6,σ (C2 ) deC2 σ ηC1 σ 4 (t+1) (t) 2 1+ C2 =C2 + αe + + d n n d # (t) 2 2 (t) (t) F7,σ (C2 ) F8,σ (C2 ) de[C2 ] σ (t) (t) . + + − C 1 C2 1 + n n d

(C.1)

Here, for each k ∈ [8], Fk,σ (·) is continuously differentiable with respect to its argument, where σ is treated as a fixed constant. These conclusions are proved by induction. It is straightforward to verify that they hold at t = 0, since the parameters are initialized as V(0) = 02d×2d and W(0) = 02d×2d . To streamline the exposition, we reorganize the content of Lemma C.2 into two separate lemmas, namely Lemmas C.3 and C.4, which focus on the updates of V and W, respectively. This decomposition allows us to present the relevant arguments more clearly and avoids an overly lengthy proof within a single lemma. Accordingly, we establish Lemmas C.3 and C.4 independently. In the inductive step, we assume that the conclusions of both Lemma C.3 and Lemma C.4 hold at the current iteration, and then show that the conclusion of the lemma under consideration continues to hold at the next iteration. This procedure does not constitute circular reasoning. Indeed, all arguments could equivalently be organized into a single Lemma C.2. The inductive assumption simply reflects the fact that the parameter updates are coupled, and it suffices to verify that, starting from a valid initialization, the stated conclusions are preserved from one iteration to the next. Lemma C.3 (Restatement of Lemma C.2, part I). Under the same conditions of Theorem 4.5, (t) there exist a time dependent scalars C1 such that for t ≥ 0, (t)

(t)

V21 = C1 · Id . (t)

In addition, C1 follows the iterative rules as " (t) (t) 2 σ2 ησ 2 d C2(t) σ2 F1,σ (C2 ) F2,σ (C2 ) (t+1) (t) (t) 2 C1 =C1 + αe 2 + C2 σ + e + + d π n n d # (t) (t) d [C2(t) ]2 σ2 F3,σ (C2 ) F4,σ (C2 ) (t) 2 (t) 2 2 − C1 + [C2 ] σ + e + + . π n n d

Here, for each k ∈ [8], Fk,σ (·) is continuously differentiable with respect to its argument, where σ is treated as a fixed constant. (t)

(t)

Proof of Lemma C.3. Suppose that V21 = C1 · Id and W21 = −C2 · Id , then by Lemma C.1, we can calculate that " # X n n n X X α (t) (t) (t) ∇V21 Ltrain (V(t) , W(t) ) = − E e−⟨zi ,θ0 ⟩ zi − V21 zi si z⊤ i si n i=1 i=1 i=1 # # " n n " n n XX X X (t) (t) α (t) ⊤ ⊤ −⟨zi1 ,θ0 ⟩ (t) e si2 zi1 zi2 +C1 E si1 si2 zi1 zi2 . =− E n i =1 i =1 i =1 i =1 | 1 2 {z | 1 2 {z } } I1

In the next, we analyze the value of I1 and I2 respectively. For any given θ∗ and θ0 , let A be an orthogonal matrix defined as (Id − θ∗ θ∗⊤ )θ0 A = θ∗ , , ξ3 , . . . , ξd ∈ Rd , ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 where ξ3 . . . , ξd are normalized vectors orthogonal to θ∗ and θ0 (For notational consistency, we −θ∗ θ∗⊤ )θ0 will use ξ1 and ξ2 to represent θ∗ and ∥(I(Id−θ , respectively, in certain summation contexts.). ⊤ ∗ θ )θ0 ∥2 d

∗

ei = A⊤ xi , which implies that yi = sign(e In addition, We define that x xi,1 ). We further denote ei . Then by Lemma F.22, we know that all coordinates of e that e zi = y i x zi are independent with each other. The first coordinate e zi,1 follows a folded normal distribution, and other coordinates still follow normal distributions. Notice that we can re-write ⟨θ0 , zi ⟩ = ⟨θ0 , θ∗ ⟩e zi,1 +∥(Id −θ∗ θ∗⊤ )θ0 ∥2e zi,2 , (t) −⟨θ ,z ⟩ which implies that both e 0 i and si are all independent with the coordinates from e zi,3 to e zi,d . In addition, Lemma F.21 guarantees that e zi are independent with θ∗ , θ0 , and ξ3 , . . . , ξd . Based on all these preliminaries, we calculate I1 as " n n # " n n # XX X X (t) (t) ⊤ ⊤ I1 =E e−⟨zi1 ,θ0 ⟩ si2 AA⊤ zi1 z⊤ =E e−⟨zi1 ,θ0 ⟩ si2 Ae z i1 e z⊤ i2 AA i2 A i1 =1 i2 =1 n X n X

i1 =1 i2 =1

" =E

# (t)

e−⟨zi1 ,θ0 ⟩ si2 e zi1 ,1e zi2 ,1 θ∗ θ∗⊤

i1 =1 i2 =1

}

I1,1

" +E

n X n X

(Id − θ∗ θ∗⊤ )θ0 (Id − θ∗ θ∗⊤ )θ0⊤ (t) e−⟨zi1 ,θ0 ⟩ si2 e zi1 ,2e zi2 ,2 ∥(Id − θ∗ θ∗⊤ )θ0 ∥22 i1 =1 i2 =1

}

I1,2

" +E

n X n X d X

(t)

e−⟨zi1 ,θ0 ⟩ si2 e zi1 ,j e zi2 ,j ξj ξj⊤ + E

# (t)

e−⟨zi1 ,θ0 ⟩ si2 e zi1 ,j1 e zi2 ,j2 ξj1 ξj⊤2 .

i1 =1 i2 =1 j1 =1 j2 ̸=j1

i1 =1 i2 =1 j=3

n X n X d X X

I1,3

}

I1,4

(C.2)

Through the independence stated above, and the calculations demonstrated in Section F.1, we have " n n # (t) XX 1 2n 2 σ2 nf2 (C2 , σ) (t) (t) ⊤ −⟨zi1 ,θ0 ⟩ 2 I1,1 = E · Id , σ e + f1 (C2 , σ) + e si2 e zi1 ,1e zi2 ,1 E θ∗ θ∗ = d π d i1 =1 i2 =1

where the expectation of the first term is derived in Lemma F.3, with f1 , f2 being two analytic (t) functions of C2 and σ, and E[θ∗ θ∗⊤ ] = Id /d as demonstrated in Lemma F.20. Similarly, we also have " n n # XX (Id − θ∗ θ∗⊤ )θ0 (Id − θ∗ θ∗⊤ )θ0⊤ −⟨zi1 ,θ0 ⟩ (t) e e I1,2 = E e si2 zi1 ,2 zi2 ,2 E ∥(Id − θ∗ θ∗⊤ )θ0 ∥22 i1 =1 i2 =1 (t) 2 1 nf4 (C2 , σ) (t) 4 σ (t) = nC2 σ e 2 + f3 (C2 , σ) + · Id , d d where the expectation of the first term is derived in Lemma F.1. For I1,3 , we have d X n n X X d X −⟨z ,θ0 ⟩ (t) 2 ⊤ X (t) i I1,3 = E e si E[e zi,j ]E ξj ξj + E e−⟨zi1 ,θ0 ⟩ si2 E[e zi1 ,j e zi2 ,j E ξj ξj⊤ ] i1 =1 i2 ̸=i1 j=3

j=3 i=1

d X n X j=3 i=1

−⟨z ,θ0 ⟩ (t) 2 ⊤ d − 2 2 C (t) σ2 + 1 σ2 f5 (C2(t) , σ) f6 (C2(t) , σ) i 2 2 σ e + · Id , + E e si E[e zi,j ]E ξj ξj = d n d

where the second equation holds as E[e zi1 ,j e zi2 ,j ] = E[e zi1 ,j ]E[e zi2 ,j ] = 0, and the expectation of the last equation is given in Lemma F.5. Lastly, for the term I1,4 , by the independence among these random variables and the fact that each ξj is symmetric, we have I1,4 =

n X n X d X X

(t) E e−⟨zi1 ,θ0 ⟩ si2 e zi1 ,j1 e zi2 ,j2 E[ξj1 ]E[ξj⊤2 ] = 0d×d .

i1 =1 i2 =1 j1 =1 j2 ̸=j1

Combining these results into (C.2), we obtain that σ2

nσ 2 e 2 I1 = d

(t) (t) 2 d C2(t) σ2 F1,σ (C2 ) F2,σ (C2 ) (t) 2 + C2 σ + e + + Id . π n n d

(C.3)

Similarly, we can also separate I2 as " n n # " n n # X X (t) (t) X X (t) (t) ⊤ ⊤ ⊤ ⊤ ⊤ I2 =E si1 si2 AA zi1 zi2 AA =E si1 si2 Ae zi1 e zi2 A i1 =1 i2 =1 n n X X

i1 =1 i2 =1

" =E

# (t) (t) si1 si2 e zi1 ,1e zi2 ,1 θ∗ θ∗⊤

" +E

i1 =1 i2 =1

n n X X

(Id − θ∗ θ∗⊤ )θ0 (Id − θ∗ θ∗⊤ )θ0⊤ (t) (t) si1 si2 e zi1 ,2e zi2 ,2 ∥(Id − θ∗ θ∗⊤ )θ0 ∥22 i =1 i =1

}

I2,1

I2,2

# " n n d # n X n X d X X X X X (t) (t) (t) (t) ⊤ ⊤ si1 si2 e zi1 ,j e zi2 ,j ξj ξj + E +E si1 si2 e zi1 ,j1 e zi2 ,j2 ξj1 ξj2 . (C.4) "

i1 =1 i2 =1 j1 =1 j2 ̸=j1

i1 =1 i2 =1 j=3

I2,3

}

I2,4

}

We calculate each term following a similar procedure. For I2,1 , we have # " n n X X (t) (t) σ 2 2 f7 (C2(t) , σ) f8 (C2(t) , σ) ⊤ · Id , I2,1 = E zi1 ,1e zi2 ,1 E θ∗ θ∗ = + + si1 si2 e d π n d i1 =1 i2 =1

where the expectation is provided in Lemma F.4. For I2,2 , we have # " n n X X (t) (t) (Id − θ∗ θ∗⊤ )θ0 (Id − θ∗ θ∗⊤ )θ0⊤ zi2 ,2 E I2,2 =E zi1 ,2e si1 si2 e ∥(Id − θ∗ θ∗⊤ )θ0 ∥22 i1 =1 i2 =1 (t) σ2 nf10 (C2 , σ) (t) 2 2 (t) = [C2 ] σ + f9 (C2 , σ) + · Id , d d where the expectation is provided in Lemma F.2. For I2,3 , we have d X n n X X d X (t) 2 2 ⊤ X (t) (t) I2,3 = E (si ) E[e zi,j ]E ξj ξj + E si1 si2 E[e zi1 ,j e zi2 ,j E ξj ξj⊤ ] i1 =1 i2 ̸=i1 j=3

j=3 i=1

d X n X j=3 i=1

(t) 2 2 ⊤ (d − 2)σ 2 [C (t) ]2 σ2 f11 (C2(t) , σ) f12 (C2(t) , σ) 2 e + + · Id , zi,j ]E ξj ξj = E (si ) E[e d n d

where the expectation is provided in Lemma F.6. In addition, I2,4 = 0d×d by the symmetry of ξj . Substituting these results into (C.4), we obtain that σ2 I2 = d

(t) (t) 2 d [C2(t) ]2 σ2 F3,σ (C2 ) F4,σ (C2 ) (t) 2 2 + [C2 ] σ + e + + Id . π n n d (t)

(t)

(C.5) (t)

Hence, we prove the induction that by assuming at t-th iteration, V21 = C1 · Id and W12 = (t) −C2 · Id , the gradient ∇V21 Ltrain (V(t) , W(t) ) is also proportional to the identity matrix Id . In (t) addition, (C.3) and (C.5) establish the iterative rule for the coefficient C1 as " (t) (t) 2 σ2 2 ησ d C2(t) σ2 F1,σ (C2 ) F2,σ (C2 ) (t+1) (t) (t) 2 αe 2 + C2 σ + e + + C1 =C1 + d π n n d # (t) (t) d [C2(t) ]2 σ2 F3,σ (C2 ) F4,σ (C2 ) (t) 2 (t) 2 2 − C1 + [C2 ] σ + e + + . π n n d This completes the proof. Lemma C.4 (Restatement of Lemma C.2, part II). Under the same conditions of Theorem 4.5, (t) there exist a time dependent scalar C2 such that for t ≥ 0, (t)

(t)

W12 = −C2 · Id .

(t)

In addition, C2 follows the iterative rules as (t) 2 (t) (t) F5,σ (C2 ) F6,σ (C2 ) deC2 σ αe 1+ + + d n n d # (t) 2 2 (t) (t) F7,σ (C2 ) F8,σ (C2 ) de[C2 ] σ (t) (t) − C 1 C2 1 + . + + n n d

(t) ηC1 σ 4 (t+1) (t) C2 =C2 +

σ2 2

Here, for each k ∈ [8], Fk,σ (·) is continuously differentiable with respect to its argument, where σ is treated as a fixed constant. (t)

(t)

Proof of Lemma C.4. Suppose that V21 = C1 · Id and W21 = −C2 · Id , then by Lemma C.1, we can calculate that ∇W12 Ltrain (V(t) , W(t) ) " X # n n X α (t) ⊤ (t) (t) = − E Zcontext diag(s(t) )−s(t) (s(t) )⊤ Z⊤ e−⟨zi ,θ0 ⟩ zi −V21 zi si θ0 context (V21 ) n i=1 i=1 " n n n # (t) XX X αC1 (t) (t) ⊤ E =− e−⟨zi ,θ0 ⟩ si1 si2 (zi1 − zi2 )z⊤ i1 z i θ0 n i=1 i1 =1 i2 =1 {z } | I3

" n n n # X X X (t) (t) (t) ⊤ + C12 (t) E si si1 si2 (zi1 − zi2 )z⊤ i1 z i θ0 i=1 i1 =1 i2 =1

}

Utilizing the same definition of the orthogonal matrix A, e zi for all i ∈ [n], and ξj for all j ∈ [d] as in the proof of Lemma C.3, we have " n n n # XX X −⟨zi ,θ0 ⟩ (t) (t) ⊤ ⊤ ⊤ ⊤ I3 =E e si1 si2 AA (zi1 − zi2 )zi1 AA zi θ0 i=1 i1 =1 i2 =1

" n n n # XX X (t) (t) =E e−⟨zi ,θ0 ⟩ si1 si2 A(e z i1 − e zi2 )e z⊤ zi θ0⊤ i1 e i=1 i1 =1 i2 =1

" n n n d # XX XX (t) (t) =E e−⟨zi ,θ0 ⟩ si1 si2 (e zi1 ,j − e zi2 ,j )e z⊤ zi ξj θ0⊤ i1 e i=1 i1 =1 i2 =1 j=1

" =E

∥(Id − θ∗ θ∗⊤ )θ0 ∥2

n X n X n X

# (t) (t) e−⟨zi ,θ0 ⟩ si1 si2 (e zi1 ,2 − e zi2 ,2 )e z⊤ zi ξ2 ξ2⊤ i1 e

i=1 i1 =1 i2 =1

" n n n XX XX

# (t) (t)

e−⟨zi ,θ0 ⟩ si1 si2 (e zi1 ,j − e zi2 ,j )e z⊤ zi ξj θ0⊤ i1 e

i=1 i1 =1 i2 =1 j̸=2 n X n X n i X 1 h (t) (t) ⊤ e−⟨zi ,θ0 ⟩ si1 si2 (e zi1 ,2 − e zi2 ,2 )e z⊤ zi · I d . = E ∥(Id − θ∗ θ∗ )θ0 ∥2 i1 e d

(C.6)

i=1 i1 =1 i2 =1

Pn Pn Pn P −⟨zi ,θ0 ⟩ s(t) s(t) (e Here the second term E zi2 ,j )e z⊤ zi ξj θ0⊤ because for i1 e i=1 i1 =1 i2 =1 j̸=2 e i1 i2 zi1 ,j −e 29

(t) (t) (t) (t) zi1 ,j −e zi2 ,j )e z⊤ zi1 ,j −e zi2 ,j )e z⊤ zi ξj E[θ0⊤ ] = any j ̸= 2, we have E e−⟨zi ,θ0 ⟩ si1 si2 (e zi ξj θ0⊤ = E e−⟨zi ,θ0 ⟩ si1 si2 (e i1 e i1 e 0 through a similar argument in Lemma C.3. Then we get (C.6) by plugging the definition that −θ∗ θ∗⊤ )θ0 ξ2 = ∥(I(Id−θ , and the fact that E[ξj ξj⊤ ] = d1 Id . This demonstrates that I3 is always pro⊤ ∗ θ∗ )θ0 ∥2 d portional to Id . In the next, we calculate the coefficient in (C.6). We first separate the term P P P (t) (t) E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 ni=1 ni1 =1 ni2 =1 e−⟨zi ,θ0 ⟩ si1 si2 (e zi1 ,2 − e zi2 ,2 )e z⊤ zi as i1 e n X n X n i h X ⊤ ⊤ −⟨zi ,θ0 ⟩ (t) (t) e e zi1 ,2 − zi2 ,2 )e z i1 z i E ∥(Id − θ∗ θ∗ )θ0 ∥2 e si1 si2 (e i=1 i1 =1 i2 =1 n X n i h X (t) ⊤ zi,1 zi1 ,1e = E ∥(Id − θ∗ θ∗ )θ0 ∥2 zi1 ,2e e−⟨zi ,θ0 ⟩ si1 e i=1 i1 =1

h + E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2

I3,1 n n XX

} (t)

zi,2 zi1 ,2e zi1 ,2e e−⟨zi ,θ0 ⟩ si1 e

i=1 i1 =1

}

I3,2

n X n X d h i X (t) + E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 e−⟨zi ,θ0 ⟩ si1 e zi1 ,2e zi1 ,j e zi,j i=1 i1 =1 j=3

| h − E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2

I3,3 n X n X n X

} (t) (t)

e−⟨zi ,θ0 ⟩ si1 si2 e zi2 ,2e zi1 ,1e zi,1

i=1 i1 =1 i2 =1

| h − E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2

I3,4 n n n XX X

} (t) (t)

e−⟨zi ,θ0 ⟩ si1 si2 e zi2 ,2e zi1 ,2e zi,2

i=1 i1 =1 i2 =1

}

I3,5

n X n X n X d h i X (t) (t) − E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 e−⟨zi ,θ0 ⟩ si1 si2 e zi2 ,2e zi1 ,j e zi,j .

(C.7)

i=1 i1 =1 i2 =1 j=3

}

I3,6

We carefully calculate these terms trough the lemmas provided in Appendix F.1, respectively. For the term I3,1 , by Lemma F.8, we have (t)

I3,1 =

2 4 σ2 nf2 (C2 , σ) (t) (t) σ e 2 nC2 + f1 (C2 , σ) + , π d (t)

where f1 , f2 are two analytic functions of C2 and σ. For the term I3,2 , by Lemma F.7, we have 4

I3,2 = σ e

σ2 2

(t) nf4 (C2 , σ) (t) 2 (t) n σ [C2 ] + 1 + f3 (C2 , σ) + . 2

For the term I3,3 , we have I3,3 =

n X d X 2 (t) zi,2 E[e zi,j ] E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 e−⟨zi ,θ0 ⟩ si e i=1 j=3

n XX d X (t) zi,2 E[e zi,j ]E[e zi1 ,j ] E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 e−⟨zi ,θ0 ⟩ si e i=1 i1 ̸=i j=3

n X d X 2 (t) zi,2 E[e zi,j ] E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 e−⟨zi ,θ0 ⟩ si e i=1 j=3 (t) 4 (2C2(t) +1)σ 2 /2

=d 1 + C2

σ e

(t)

+ f5 (C2 , σ) +

nf6 (C2 , σ) . d

The second equation holds as by the independence between e zi1 ,j and e zi,j when i1 ̸= i, and the fact that E[e zi1 ,j ] = E[e zi,j ] = 0. The last equation is calculated based on Lemma F.9. For the term I3,4 , by Lemma F.11, we have (t)

σ2 2 nf8 (C2 , σ) (t) (t) I3,4 = σ 4 e 2 nC2 + f7 (C2 , σ) + . π d

For the term I3,5 , by Lemma F.10, we have (t)

σ2

(t)

I3,5 = σ 6 e 2 n[C2 ]2 + f9 (C2 , σ) +

nf10 (C2 , σ) . d

For the term I3,6 , similar to the procedure of calculation for I3,3 and utilizing the result of Lemma F.12, we have I3,6 =

n X n X d X 2 (t) (t) E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 e−⟨zi ,θ0 ⟩ si si2 e zi,2 E[e zi,j ] i=1 i2 =1 j=3 (t)

(t)

=dσ 4 e(2C2 +1)σ /2 C2 + f11 (C2 , σ) +

nf12 (C2 , σ) . d

Plugging all these results into (C.7), we obtain that σ2

nσ 4 e 2 I3 = d

(t) 2 (t) (t) F5,σ (C2 ) F6,σ (C2 ) deC2 σ 1+ + + Id . n n d

In the next, we consider calculating I4 through a similar procedure as " n n n # X X X (t) (t) (t) ⊤ ⊤ ⊤ ⊤ I4 =E si si1 si2 AA (zi1 − zi2 )zi1 AA zi θ0 i=1 i1 =1 i2 =1

# " n n n X X X (t) (t) (t) =E si si1 si2 A(e z i1 − e zi2 )e z⊤ zi θ0⊤ i1 e i=1 i1 =1 i2 =1

(C.8)

" n n n d # X X X X (t) (t) (t) ⊤ ⊤ =E zi1 ,j − e zi2 ,j )e z i1 e z i ξ j θ0 si si1 si2 (e i=1 i1 =1 i2 =1 j=1 n X n X n i X 1 h (t) (t) (t) e zi1 ,2 − e zi2 ,2 )e z⊤ z · Id . = E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 si si1 si2 (e i i1 d i=1 i1 =1 i2 =1

This that I is also proportional to Id . For the coefficient result demonstrates P P 4P (t) (t) (t) zi1 ,2 − e zi2 ,2 )e z⊤ zi , we separate it as E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 ni=1 ni1 =1 ni2 =1 si si1 si2 (e i1 e n X n X n i h X (t) (t) (t) e z zi1 ,2 − e zi2 ,2 )e z⊤ E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 si si1 si2 (e i1 i i=1 i1 =1 i2 =1 n X n n X n i h i h X X (t) (t) (t) (t) ⊤ ⊤ zi,1 + E ∥(Id − θ∗ θ∗ )θ0 ∥2 zi1 ,1e zi1 ,2e = E ∥(Id − θ∗ θ∗ )θ0 ∥2 si si1 e zi,2 zi1 ,2e zi1 ,2e s i s i1 e i=1 i1 =1 i=1 i1 =1

}

I4,1

I4,2

}

n X n X d h i X (t) (t) ⊤ + E ∥(Id − θ∗ θ∗ )θ0 ∥2 si si1 e zi1 ,2e zi1 ,j e zi,j i=1 i1 =1 j=3

| h − E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2

I4,3 n n X n XX

} (t) (t) (t)

si si1 si2 e zi2 ,2e zi1 ,1e zi,1

i=1 i1 =1 i2 =1

| h − E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2

I4,4 n n n XX X

} (t) (t) (t)

si si1 si2 e zi2 ,2e zi1 ,2e zi,2

i=1 i1 =1 i2 =1

}

I4,5

n X n X n X d h i X (t) (t) (t) − E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 si si1 si2 e zi2 ,2e zi1 ,j e zi,j .

(C.9)

i=1 i1 =1 i2 =1 j=3

}

I4,6

We carefully calculate these terms trough the lemmas provided in Appendix F.1, respectively. For the term I4,1 , by Lemma F.14, we have (t)

(t)

f13 (C2 , σ) f14 (C2 , σ) 2 (t) I4,1 = σ 4 C2 + + . π n d For the term I4,2 , by Lemma F.13, we have f15 (C2(t) , σ) f16 (C2(t) , σ) (t) (t) I4,2 = C2 σ 4 σ 2 [C2 ]2 + 1 + + . n d For the term I4,3 , by Lemma F.15, we have I4,3 =

n X d X 2 (t) 2 e E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 si zi,2 E[e zi,j ] i=1 j=3

(t)

(t) 2

(t)

2dC2 σ 4 eσ [C2 ] f17 (C2 , σ) f18 (C2 , σ) = + + . n n d For the term I4,4 , by Lemma F.17, we have (t)

I4,4 =

(t)

2 4 (t) f19 (C2 , σ) f20 (C2 , σ) σ C2 + + . π n d

For the term I4,5 , by Lemma F.16, we have (t) (t) f21 (C2 , σ) f22 (C2 , σ) 3 6 I4,5 = C2 (t)σ + + .

For the term I4,6 , similar to the procedure of calculation for I4,3 and utilizing the result of Lemma F.18, we have n X n X d X 2 (t) 2 (t) I4,6 = E ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 si si2 e zi,2 E[e zi,j ] i=1 i2 =1 j=3 (t)

(t) 2

(t)

f23 (C2 , σ) f24 (C2 , σ) dC σ 4 eσ [C2 ] + + = 2 n n d Plugging all these results into (C.9), we obtain that (t) 2 2 (t) (t) (t) F7,σ (C2 ) F8,σ (C2 ) de[C2 ] σ C2 σ 4 1+ + + Id . I4 = d n n d

(t)

(C.10) (t)

(t)

Therefore, we prove the induction that by assuming at t-th iteration, V21 = C1 · Id and W12 = (t) −C2 · Id , the gradient ∇W12 Ltrain (V(t) , W(t) ) is also proportional to the identity matrix Id . In (t) addition, (C.8) and (C.10) establish the iterative rule for the coefficient C1 as (t) 2 (t) (t) F5,σ (C2 ) F6,σ (C2 ) deC2 σ αe 1+ + + d n n d # (t) 2 2 (t) (t) F7,σ (C2 ) F8,σ (C2 ) de[C2 ] σ (t) (t) − C 1 C2 1 + . + + n n d

(t) ηC1 σ 4 (t+1) (t) C2 =C2 +

σ2 2

This completes the proof. Therefore, we have shown that throughout training, the parameter matrices V(t) and W(t) always remain in the structured parameterization defined in Lemma 4.4. Consequently, for our learning task, analyzing the training loss Ltrain (V, W) along the iterates V(t) , W(t) is equivalent to (t) studying a reduced two-dimensional proxy loss Letrain (C1 , C2 ) with respect to the coefficients C1 (t) and C2 . Specifically, the proxy loss Letrain (C1 , C2 ) is defined as 0d×d 0d×d 0d×d −C2 · Id e Ltrain (C1 , C2 ) := Ltrain , . C1 · Id 0d×d 0d×d 0d×d

(t)

Under this parameterization, the update rules for C1 and C2 given in (C.1) correspond exactly to gradient descent applied to Letrain (C1 , C2 ) with zero initialization. From (C.1), we further derive that Letrain (C1 , C2 ) admits the following explicit form: " 2 C2 2 d C22 σ2 F3,σ (C2 ) F4,σ (C2 ) σ 2 2 1 e + Ltrain (C1 , C2 ) = + C2 σ + e + d 2 π n n d # 2 d C2 σ2 F1,σ (C2 ) F2,σ (C2 ) σ 2 /2 2 − αe C1 + c, + C2 σ + e + + π n n d ′ (C ) = where c is a constant independent of C1 and C2 . And we can immediately conclude that F1,σ 2 ′ (C ) = F ′ (C ) = F ′ (C ) = F F5,σ (C2 ), F2,σ (C ), F (C ), F (C ). In addition, we define 2 6,σ 2 2 7,σ 2 2 8,σ 2 3,σ 4,σ that

F1,σ (C2 ) F2,σ (C2 ) 2 d 2 + C2 σ 2 + eC2 σ + + ; π n n d d 2 2 F3,σ (C2 ) F4,σ (C2 ) 2 + ; G2 (C2 ) = + C22 σ 2 + eC2 σ + π n n d 2 F5,σ (C2 ) F6,σ (C2 ) deC2 σ G3 (C2 ) =1 + + + ; n n d 2 2 F7,σ (C2 ) F8,σ (C2 ) deC2 σ G4 (C2 ) =1 + + + . n n d

G1 (C2 ) =

(C.11)

With these notations, we prove the first conclusion in Theorem 4.5, which states that there exists 2 (t) (t) an invariant compact set R = [0, 2αeσ /2 ] × [0, 2] for C(t) = [C1 , C2 ]⊤ during the training. (t)

(t)

Lemma C.5 (First conclusion in Theorem 4.5). Consider the iterative rules for C(t) = [C1 , C2 ]⊤ (0) (0) given in (C.1) with initializations C1 , C2 = 0. Suppose that η ≤ O n1 , α ≤ O(1) and n ≥ Ω(d2 ), then it holds that: (t)

(t)

0 ≤C1 ≤ 2αeσ /2 ; 0 ≤ C2 ≤ 2,

(C.12)

for all t ∈ N. Proof of Lemma C.5. We prove this lemma by induction. It’s evident that (C.12) holds at t = 0. In the next, we assume it holds at t-th iteration and attempt to prove it still holds at t + 1-th 2 (t) iteration. Since (C.12) holds, there exists K ≤ O(1) such that |Fi,σ (C2 )|, e4σ ≤ K8 for all i ∈ [8]. Consequently, combined with the condition that n ≥ Ω(d2 ), we can directly obtain that 2 K 3 K 2 K 4 K (t) (t) − ≤ G1 (C2 ) ≤ + ; − ≤ G2 (C2 ) ≤ + ; π d π d π d π d K K K K (t) (t) 1− ≤ G3 (C2 ) ≤ 1 + ; 1 − ≤ G4 (C2 ) ≤ 1 + ; d d d d (t) (t) 3 K 9 K − d 3 G1 (C2 ) 3 K G3 (C2 ) K 4π + d ≤ π4 K ≤ ≤ ≤ ; 1 − ≤ ≤1+ ; 17 K (t) (t) 5 2 d d G2 (C ) G4 (C ) π + d 8π − d 2

(t+1)

We first prove that C1

(t)

≥ 0 from two cases: (1).C1

σ 2 /2

≤ 3αe5

(t)

G3 (C2 ) (t)

≤ 2.

G1 (C2 ) (t)

; and (2).C1

σ 2 /2

> 3αe5

. If

σ 2 /2

(t)

C1 ≤ 3αe5

, we have 2

(t)

αeσ /2 G1 (C2 ) − C1 G2 (C2 ) ≥ αeσ /2 G1 (C2 ) − (t+1)

which implies that C1 σ 2 /2

αe

(t)

σ 2 /2

(t)

≥ C1 ≥ 0. If C1 > 3αe5

3αeσ /2 5 (t) G1 (C2 ) ≥ 0, 5 3

, then we have

2 (t) (t) (t) (t) (t) G1 (C2 ) − C1 G2 (C2 ) ≥ −C1 G2 (C2 ) ≥ −2αeσ /2

4 K , + π d

(t) (t) where we replace C1 and G2 (C2 ) with their upper bounds respectively. Since η ≤ O n1 , we can obtain that (t+1)

ησ 2 σ2 /2 (t) (t) (t) αe G1 (C2 ) − C1 G2 (C2 ) d 2 2 /2 σ 2ησ 2 αeσ /2 4 K 3αe − + ≥ 0. ≥ 5 d π d (t)

≥C1 +

(t+1) The last inequality holds as η ≤ O n1 is sufficiently small. This proves that C1 ≥ 0. Similarly, (t+1)

to prove C1

σ 2 /2

(t)

≤ 2αeσ /2 , we also consider two cases: (1). C1 ≥ 3αe2

σ 2 /2 (t) For the first case where C1 ≥ 3αe2 , we have

σ 2 /2

αe

σ 2 /2

(t)

; and (2). C1 < 3αe2

2 3αeσ /2 (t) (t) (t) (t) (t) (t) G1 (C2 ) − C1 G2 (C2 ) ≤ G2 (C2 ) − C1 G2 (C2 ) ≤ 0,

(t+1)

which implies that C1 calculate that σ 2 /2

αe

(t)

≤ C1

(t)

≤ 2αeσ /2 . On the other hand where C1

3 K + . π d

2 2 (t) (t) (t) (t) G1 (C2 ) − C1 G2 (C2 ) ≤ αeσ /2 G1 (C2 ) ≤ αeσ /2

σ 2 /2

< 3αe2

, we can

This can further implies that 2

(t+1) (t) C1 =C1 + η

1/2

αe

(t) (t) (t) G1 (C2 ) − C1 G2 (C2 ) (t+1)

3αeσ /2 ησ 2 αeσ /2 ≤ + 2 d

3 K + π d

≤ 2αeσ /2 . (t)

This completes the proof that C1 ≤ 2αeσ /2 . In the following, we proceed with C2 with the (t) (t+1) (t) similar techniques. We first prove that C2 ≥ 0 under (1). C2 ≤ 14 ; and (2). C2 > 14 . For the (t)

first case where C2 ≤ 14 , we have 2

(t)

αeσ /2 C1 G3 (C2 ) − C12 (t)C2 G4 (C2 ) ≥ C1 (t+1)

which implies that C2 2

(t)

1 2 2 (t) (t) αeσ /2 G3 (C2 ) − 4αeσ /2 G3 (C2 ) ≥ 0, 4 (t)

≥ C2 ≥ 0. On the other hand when C2 > 41 , we can obtain that (t)

(t)

αeσ /2 C1 G3 (C2 ) − C12 (t)C2 G4 (C2 ) ≥ − C12 (t)C2 G4 (C2 ) ≥ −8α2 eσ ,

(t)

where the last inequality holds as we substitute the upper bounds that C1 ≤ 2αeσ /2 and C2 ≤ 2. This result helps us further derive that 2

(t+1)

(t)

=C2 +

1 8ησ 4 α2 eσ σ 4 η σ2 /2 (t) (t) (t) (t) αe C1 G3 (C2 ) − C12 (t)C2 G4 (C2 ) ≥ − ≥ 0. d 4 d (t+1)

(t)

This completes the proof that C2 ≥ 0. In the next, we prove that C2 is always smaller than 2. This would be a little tricky, and we consider two different phases. We first prove that there exists σ 2 /2

(3d−πK) an iteration T1 , which serves the first time such that C1 reaches αe 4d+πK . We prove that for (t)

all t ≤ T1 , it holds that C2 ≤ 1 by induction. Notice that for any t ≤ T1 , we have (t)

(t+1)

∆C1 =C1 ≥

(t)

− C1 =

σ2 ησ 2 (t) (t) (t) αe 2 G1 (C2 ) − C1 G2 (C2 ) d

2 σ2 ησ 2 αeσ /2 (3d − πK) 3d + 3πK/4 ησ 2 σ2 (t) (t) (t) αe 2 G1 (C2 ) − G1 (C2 ) ≥ αe 2 G1 (C2 ). d 4d + πK 3d − πK 4d (t)

On the other hand, we can upper bound the increments of C2 as ησ (t) (t+1) (t) ∆C2 =C2 − C2 = d

≤

ησ 4 d

(t) (t) (t) σ 2 /2 (t) 2 αe C1 G3 (C2 ) − C1 (t)C2 G4 (C2 ) 2

(t)

αeσ /2 C1 G3 (C2 ) ≤

2 σ 2 /2

(t)

This implies that for all t ≤ T1 , 2

2 2 eσ (3d−πK)2 (t) t ≤ T1 , C2 ≤ 8α σ(4d+πK) 2

σ /2 (3d − πK) 2ησ 4 σ2 /2 (t) αe αe G1 (C2 ) . d 4d + πK

∆C2

8ασ e (3d−πK) . Consequently, we obtain that for all (t) ≤ 4d+πK

∆C1

1 ≤ 1, which holds as σ 2 ≤ 2π and α ≤ 1. Next, we prove for the (t)

σ 2 /2

(3d−πK) case when t ≥ T1 . We first prove that C1 ≥ αe 4d+πK − η for any t ≥ T1 by induction. When 2

σ /2 (3d−πK) (t) αeσ /2 (3d−πK) − η ≤ C1 ≤ αe 4d+πK , we have 4d+πK 2

(t)

αeσ /2 G1 (C2 ) − C1 G2 (C2 ) σ 2 /2

≥αe

(t+1)

This implies that C1 σ 2 /2

αe

2 αeσ /2 (3d − πK) 4d + πK (t) (t) G1 (C2 ) − G1 (C2 ) ≥ 0.

4d + πK

(t)

σ 2 /2

3d − πK (t)

σ 2 /2

(3d−πK) (3d−πK) ≥ C1 ≥ αe 4d+πK − η. When C1 > αe 4d+πK , we have

2 (t) (t) (t) (t) (t) G1 (C2 ) − C1 G2 (C2 ) ≥ −C1 G2 (C2 ) ≥ −2αeσ /2

4 K + . π d

Consequently, we have (t+1)

ησ 2 σ2 /2 (t) (t) (t) αe G1 (C2 ) − C1 G2 (C2 ) d 2 /2 2 2 σ αe (3d − πK) 2ησ 2 αeσ /2 4 K αeσ /2 (3d − πK) ≥ − + ≥ − η. 4d + πK d π d 4d + πK (t)

≥C1 +

σ 2 /2

(t)

(3d−πK) − η, we can follow the previous proof techniques Now, by establishing the fact C1 ≥ αe 4d+πK (t+1)

by induction. To prove that C2 (t)

(t)

≤ 2, we also consider two cases: (1). C2 (t)

19 ≥ 10 ; and (2).

19 C2 < 19 10 . For the first case where C2 ≥ 10 , we can obtain that 2

(t)

αeσ /2 G3 (C2 ) − C1 C2 G4 (C2 ) σ2 /2 (t) αe (3d − πK) 19 G3 (C2 ) 2 (t) ≤αeσ /2 G3 (C2 ) − −η 4d + πK 10 1 + K d (t)

(t)

≤αeσ /2 G3 (C2 ) − αeσ /2 G3 (C2 ) + 2ηG3 (C2 ) − αeσ /2 G3 (C2 )

17d 10 − (4 + 2π)K (4d + πK)(1 + K d)

2 3αeσ /2 (t) (t) ≤2ηG3 (C2 ) − G3 (C2 ) ≤ 0,

(t+1) (t) ≤ C2 ≤ where the last inequality holds as η ≤ O n1 is sufficiently small. This implies that C2 (t)

2. In addition, when C2 ≤ 19 10 , we have (t)

2 2 2 (t) (t) (t) (t) (t) (t) αeσ /2 G3 (C2 ) − C1 C2 G4 (C2 ) ≤ C1 αeσ /2 G3 (C2 ) ≤ 4α2 eσ .

Consequently, we can obtain that ησ (t+1) (t) C2 = C2 + d

(t) C1

σ 2 /2

αe

(t) (t) (t) (t) G3 (C2 ) − C1 C2 G4 (C2 )

19 4ησ 4 α2 eσ ≤ + ≤ 2, 10 d

where the last inequality holds as η ≤ O n1 is sufficiently small. This completes the proof that (t)

C2 ≤ 2 for all t ∈ N. (t)

(t)

We have demonstrated in Lemma C.5 that the iterates of C1 and C2 always stay inner the 2 region R = [0, 2αeσ /2 ] × [0, 2]. We next prove that within this region, there exists a unique local minimum (C1∗ , C2∗ ) of the loss function Letrain (C1 , C2 ). We first introduce Newton–Kantorovich Theorem, which helps demonstrate our results. Theorem C.6 (Newton–Kantorovich Theorem, cf. Theorem 5.5.1 in Kelley (1995)). Let F : Rd → Rd be a continuously differentiable function. Suppose there exists a point x0 ∈ Rd and constants β, γ, L such that: 1. F (·) is differentiable at x0 , and F ′ (x0 ) is invertible, satisfying that ∥F ′ (x0 )−1 ∥ ≤ β;

∥F ′ (x0 )−1 F (x0 )∥ ≤ γ.

2. F ′ (·) is Lipschitz continuous with constant ι in a neighborhood of x0 radius r̄ satisfying that √ 1 − 1 − 2βγι r̄ ≥ r− = βι 3. The constants β, γ, ι satisfy that βγι ≤ 12 . Then there√ exists a unique fixed point x∗ of F (·) in the neighborhood of x0 with radius equal to max{r̄, 1+ 1−2βγι }. In addition, x∗ satisfies that ∥x∗ − x0 ∥ ≤ r− . βι 37

Then the following Lemma C.7 demonstrates that there exists a unique local minimum of the proxy loss Letrain (C1 , C2 ) by utilizing the Newton-Kantorovich Theorem C.6. 2

Lemma C.7 (Second conclusion in Theorem 4.5). Let R := [0, 2αeσ /2 ] × [0, 2]. Then there exists a unique local minimum C∗ = [C1∗ , C2∗ ]⊤ of the loss function Letrain (C1 , C2 ) inner R, and this local minimum satisfies that 1 ∗ ∗ σ 2 /2 , C2 − 1 ≤ O C1 − αe . d Proof of Lemma C.7. We prove this lemma by demonstrating that ∇Letrain (C1 , C2 ) only has one unique fixed point by Newton–Kantorovich Theorem C.6. Since Newton–Kantorovich Theorem C.6 does not require a particular norm, we specify the ℓ∞ norm ∥ · ∥∞ in our proof. We first calculate the Hessian matrix of Letrain (C1 , C2 ) as ∂ 2 Letrain (C1 , C2 ) σ2 2 d C22 σ2 F3,σ (C2 ) F4,σ (C2 ) 2 2 = + C2 σ + e + + d π n n d ∂C12 " ′ ′ σ2 σ4 d C2 σ2 F1,σ (C2 ) F2,σ (C2 ) ∂ 2 Letrain (C1 , C2 ) 2 + + =− αe 1+ e ∂C1 ∂C2 d n n d # ′ (C ) ′ (C ) F4,σ d C22 σ2 F3,σ 2 2 − 2C1 C2 1 + e + + n n d " σ2 σ4 ∂ 2 Letrain (C1 , C2 ) d C2 σ2 F5,σ (C2 ) F6,σ (C2 ) =− + + αe 2 1 + e ∂C2 ∂C1 d n n d # d 2 2 F7,σ (C2 ) F8,σ (C2 ) − 2C1 C2 1 + eC2 σ + + n n d " 2 ′ (C ) ′ (C ) F5,σ F6,σ σ2 ∂ 2 Letrain (C1 , C2 ) σ 2 deC2 σ C1 σ 4 2 2 αe 2 + + =− d n n d ∂C22 # 2 2 ′ (C ) ′ (C ) F7,σ F8,σ (1 + 2C22 )deC2 σ 2 2 − C1 1 + + + . n n d 2

In addition, since we have (C1 , C2 ) ∈ R = [0, 2αeσ /2 ]×[0, 2], which is a compact set. Similar to the ′ (C )|, 9e4σ 2 } ≤ K for proof of Lemma C.5, we can find K ≤ O(1), such that max{|Fk,σ (C2 )|, |Fk,σ 2 8 e1 , C e2 ) = (αeσ2 /2 , 1). all k ∈ [8]. To check the conditions of Newton-Kantorovich Theorem, we let (C Then we can calculate that ∂ 2 Letrain (C1 , C2 ) σ2 − d ∂C12 e1 ,C2 =C e2 C1 =C

2 + σ2 π

≤

∂ 2 Letrain (C1 , C2 ) αeσ /2 σ 4 K − ≤ 2 ∂C1 ∂C2 d d e1 ,C2 =C e2 C1 =C 2

∂ 2 Letrain (C1 , C2 ) αeσ /2 σ 4 K − ≤ 2 ∂C2 ∂C1 d d e1 ,C2 =C e2 C1 =C

K d2

α2 eσ σ 4 ∂ 2 Letrain (C1 , C2 ) K − ≤ 2. 2 d d ∂C2 e1 ,C2 =C e2 C1 =C 2 e1 , C e2 ) ≥ 2α2 eσ2 σ6 − 5K3 . Consequently, we can From which we can calculate that det ∇2 Letrain (C πd d further calculate that

e1 , C e2 )]−1 ≤ [∇2 Letrain (C ∞

max

∂ 2 Letrain (Ce1 ,Ce2 ) ∂C22

(C1 ,C2 ) (C1 ,C2 ) , ∂ Ltrain + ∂ Ltrain ∂C1 ∂C2 ∂C12 e1 , C e2 ) det ∇2 Letrain (C e e

e e

(πα2 eσ σ 2 + παeσ /2 σ 2 + 2 + πσ 2 )d = β, ≤ α2 eσ2 σ 4 which completes the first condition of Newton-Kantorovich Theorem C.6. In the next, we can further calculate that ∂ Letrain (C1 , C2 ) K ≤ 2; ∂C1 d e1 ,C2 =C e2 C1 =C

∂ Letrain (C1 , C2 ) K ≤ 2. ∂C2 d e1 ,C2 =C e2 C1 =C

Consequently, we have that e1 , C e2 )]−1 ∇Letrain (C e1 , C e2 ) e1 , C e2 )]−1 e1 , C e2 ) [∇2 Letrain (C ≤ [∇2 Letrain (C ∇Letrain (C ∞ ∞ ∞ 2

(πα2 eσ σ 2 + παeσ /2 σ 2 + 2 + πσ 2 )K ≤ = γ. α2 eσ2 σ 4 d In addition, for all (C1 , C2 ) ∈ R, we can calculate that 2

∇2 Letrain (C1 , C2 ) ∞ ≤

16α2 σ 4 eσ + 9αeσ /2 σ 4 + 4σ 4 = ι, d

implying that ∇2 Letrain (C1 , C2 ) is Lipschitz continuous with the constant ι. And we can check that βγι = O( d1 ) ≤ 21 . For now, we have verified that all conditions of Theorem C.6 hold. Therefore, there exists a unique fixed point (C1∗ , C2∗ ) of ∇Letrain (C1 , C2 ) inner R, and satisfying that 1 ∗ ∗ σ 2 /2 C1 − αe , C2 − 1 ≤ r− ≤ γ ≤ O . d ∗ ∗ In the next, we prove that minimum. Since |C1∗ − this unique fixed point (C1 ,2C2 ) is a∗ local 2 /2 1 σ ∗ ∗ e1 , C e2 ) αe |, |C2 − 1| ≤ O d , we can easily obtain that ∇ Letrain (C1 , C2 ) − ∇2 Letrain (C ≤ ∞ 1 O d2 . Consequently, we can obtain that

e1 , C e2 ) − 2∥∇2 Letrain (C1∗ , C2∗ ) − ∇2 Letrain (C e1 , C e2 )∥2∞ det ∇2 Letrain (C1∗ , C2∗ ) ≥ det ∇2 Letrain (C e1 , C e2 )∥∞ ∥∇2 Letrain (C e1 , C e2 )∥∞ − 2∥∇2 Letrain (C1∗ , C2∗ ) − ∇2 Letrain (C 2 1 2α2 eσ σ 6 − O 3 > 0. ≥ 2 πd d In addition, we can also check that e1 , C e2 ) − 2∥∇2 Letrain (C1∗ , C2∗ ) − ∇2 Letrain (C e1 , C e2 )∥∞ tr ∇2 Letrain (C1∗ , C2∗ ) ≥tr ∇2 Letrain (C 39

2 α2 eσ σ 4 + 2σ 2 /π + σ 4 1 ≥ − O 2 > 0. d d These two results demonstrate that ∇2 Letrain (C1∗ , C2∗ ) is strictly positive definite. Therefore, the fixed point C∗ = [C1∗ , C2∗ ]⊤ of the gradient ∇Letrain (C1 , C2 ) is local minimum of the loss function Letrain (C1 , C2 ), which completes the proof. For now, we have completed the first and second conclusions in Theorem 4.5. In the next lemma, we demonstrate that inside the region R, the loss function Letrain (C1 , C2 ) follows a PolyakLojasiewicz (PL) condition, which serves as a critical step for final linear convergence rate. Lemma C.8 (PL condition inside R). There exists a strictly positive constant µ1,α,σ solely depending on α and σ, such that for all [C1 , C2 ]⊤ inner R, the loss function Letrain (C1 , C2 ) satisfies the following Polyak-Lojasiewicz (PL) condition 2

∇Letrain (C1 , C2 ) 2 ≥

2µα,σ e Ltrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) . d

Proof of Lemma C.8. To prove this lemma, we first introduce the nullcline of C1 in the gradient regarding C1 . By setting [∇Letrain (C1 , C2 )]1 = 0, we can derive the nullcline of C1 as σ2

αe 2 G1 (C2 ) G2 (C2 ) , and define that ∆(C1 , C2 ) = C1 − G∗ (C2 ).

G∗ (C2 ) = loss Letrain (C1 , C2 ) as

Then we can decompose the

" # 2 ∆2 (C , C )G (C ) 2 eσ 2 G2 (C ) σ α 1 2 2 2 2 1 Letrain (C1 , C2 ) = − + c, d 2 2G2 (C2 ) where c is a constants irrelevant with C1 and C2 . Since (C1∗ , C2∗ ) is the fixed point of ∇Letrain (C1 , C2 ), we can obtain that ∆(C1∗ , C2∗ ) = 0, which helps us to further derive that 2

σ ∆ (C1 , C2 )G2 (C2 ) Letrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) = + ψ(C2 ) − ψ(C2∗ ), 2d

(C.13)

where ψ(C2 ) = −

α2 eσ G21 (C2 ) 2dG2 (C2 ) . In addition, since

∂ Letrain (C1∗ , C2∗ ) σ 2 ∆(C1∗ , C2∗ ) ∂∆(C1∗ , C2∗ ) ∗ ∗ ∗ ′ ∗ = 2 G2 (C2 ) + ∆(C1 , C2 )G2 (C2 ) + ψ ′ (C2∗ ) = 0, ∂C2 2d ∂C2 we can immediately conclude that ψ ′ (C2∗ ) = 0. In addition, we can calculate that " 2 α 2 eσ ′′ ψ (C2 ) = 4G1 (C2 )G′1 (C2 )G2 (C2 )G′2 (C2 ) + G21 (C2 )G2 (C2 )G′′2 (C2 ) 2dG32 (C2 ) # 2 2 − 2G22 (C2 ) G1 (C2 )G′′1 (C2 ) + G′1 (C2 ) − 2G21 (C2 ) G′2 (C2 ) 2

α 2 eσ e 2 ). = G(C 2dG32 (C2 )

(C.14)

Similar to the previous proof of Lemma C.5 and Lemma C.7, we can find K ≤ O(1) such that 40

′ (C )|, 9e4σ } ≤ K . For the terms G (C ) and G (C ) and their derivatives, max{|Fk,σ (C2 )|, |Fk,σ 1 2 2 2 2 8 we can obtain their uniform upper and lower bounds for C2 in [0, 2] as

2 K 3 K 2 K 4 K − ≤ G1 (C2 ) ≤ + ; − ≤ G2 (C2 ) ≤ + π d π d π d π d K K K |G′1 (C2 ) − σ 2 | ≤ ; |G′′1 (C2 )| ≤ ; |G′2 (C2 ) − 2σ 2 C2 | ≤ ; d d d

|G′′2 (C2 ) − 2σ 2 | ≤

K . (C.15) d

2 2 e 2) This further implies that 21 αeσ /2 ≤ ∆(C1 , C2 ) ≤ 2αeσ /2 . Substituting these terms into the G(C defined in (C.14), we can obtain that

e 2) − G(C

24σ 4 2 12σ 6 2 8σ 6 3 16σ 2 8σ 4 48σ 4 − + C − C + C2 − C 2 π3 π2 π2 π2 2 π π 2

≤

90K . π3d

(C.16)

1 By substituting the facts that 0 ≤ C2 ≤ 2 inner R, and σ 2 ≤ 2π into (C.16), we can derive the e 2 ) as lower and upper bounds for G(C 2 8σ 2 e 2 ) ≤ 29σ . ≤ G(C 3 3 π π

e 2 ), together with the For now, we substitute the previously derived upper and lower bounds on G(C 2 K 4 K corresponding bounds on G2 (C2 ) that π − d ≤ π + d as established in the proof of Lemma C.5, into (C.14). This yields the following bounds of ψ ′′ (C2 ) uniformly for all C2 in R as 2

3α2 σ 2 eσ 4α2 σ 2 eσ ≤ ψ ′′ (C2 ) ≤ . 65d d

(C.17)

Consequently, by utilizing Taylor’s expansion, the fact ψ ′ (C2∗ ) = 0, and the uniform upper and lower bounds for ψ ′′ (C2 ), we can derive that ψ ′′ (C̄2 ) 2 3α2 σ 2 eσ2 2 ψ(C2 ) − ψ(C2∗ ) = ψ ′ (C2∗ ) C2 − C2∗ + C2 − C2∗ ≤ C2 − C2∗ , 2 2d

(C.18)

where C̄2 is an intermediate value between C2 and C2∗ . On the other hand, we can also have 2

′

ψ (C2 ) =

′

ψ (C2∗ ) + ψ ′′ (C̄2 ) C2 − C2∗

4α2 σ 2 eσ ≥ C2 − C2∗ . 65d

(C.19)

By square on both sides of (C.19) and compare it with (C.18), we obtain that ′ 2 α 2 σ 2 e σ 2 ψ (C2 ) ≥ ψ(C2 ) − ψ(C2∗ ) , 600d

(C.20)

which establishes a PL condition for ψ(C2 ). In the next, we prove that this can imply a PL e e condition for Letrain (C1 , C2 ). We first consider two components ∂ Ltrain (C1 ,C2 ) and ∂ Ltrain (C1 ,C2 ) of ∂C1

∥∇Letrain (C1 , C2 )∥2 as following: ∂ Letrain (C1 , C2 ) σ2 = G2 (C2 )∆(C1 , C2 ); ∂C1 d

∂C2

2 ∂ Letrain (C1 , C2 ) σ 2 αeσ /2 G′1 (C2 )G2 (C2 ) − G1 (C2 )G′2 (C2 ) σ2 ′ = ψ (C2 ) + ∆(C1 , C2 ) + G′2 (C2 )∆2 (C1 , C2 ). ∂C2 d G2 (C1 , C2 ) 2d By utilizing the results in (C.15), we can obtain that ∂ Letrain (C1 , C2 ) 2 17σ 4 2 ≥ 2 2 ∆ (C1 , C2 ), ∂C1 π d

(C.21)

and ∂ Letrain (C1 , C2 ) 2 ∂C2 2 2 αeσ 2 /2 G′ (C )G (C ) − G (C )G′ (C ) σ σ2 ′ 2 2 2 1 2 2 1 2 ′ 2 = ψ (C2 ) + ∆(C1 , C2 ) + G2 (C2 )∆ (C1 , C2 ) d G2 (C2 ) 2d 2 σ2 /2 ′ 2 ′ 2 G1 (C2 )G2 (C2 ) − G1 (C2 )G2 (C2 ) σ αe σ2 ′ 1 ′ 2 ∆(C1 , C2 ) + G2 (C2 )∆ (C1 , C2 ) ≥ ψ (C2 ) − 2 d G2 (C2 ) 2d σ2

2 5ασ 6 e 2 2 1 ≥ ψ ′ (C2 ) − ∆ (C1 , C2 ), 2 d2

(C.22) 2

where the first inequality holds as a2 + b2 ≥ a2 − b2 for all a, b ∈ R, and the second inequality is 17 derived by utilizing the upper and lower bounds in (C.15). We let κ = min 1, , and 2 10π 2 ασ 2 eσ /2 then can calculate that 2

∂ Letrain (C1 , C2 ) 2 ∂ Letrain (C1 , C2 ) 2 + ∂C1 ∂C2 2 ∂ Letrain (C1 , C2 ) 2 ∂ Letrain (C1 , C2 ) ≥ +κ ∂C1 ∂C2

∇Letrain (C1 , C2 ) 2 =

σ2

2 κ 5ασ 6 e 2 2 17σ 4 ∆ (C1 , C2 ) ≥ 2 2 ∆2 (C1 , C2 ) + ψ ′ (C2 ) − κ π d 2 d2 2 17σ 4 κ ≥ 2 2 ∆2 (C1 , C2 ) + ψ ′ (C2 ) 2π d 2 2 2 2 2σ σ α2 σ 2 eσ κ 2 ≥ G2 (C2 )∆ (C1 , C2 ) + ψ(C2 ) − ψ(C2∗ ) 5πd 2d 1200 2 2 2 2 2 σ α2 σ 2 eσ αeσ /2 σ 2 ∗ ≥ min , , G2 (C2 )∆ (C1 , C2 ) + ψ(C2 ) − ψ(C2 ) d 5πd 600d 375π 2 2d 2µ1,α,σ e ≥ Ltrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) , d σ2 α2 σ2 eσ2 αeσ2 /2 where µ1,α,σ = min 5πd , 600d , 375π2 is a constant solely depending on α and σ. Here, the first inequality holds as κ ≤ 1. The second inequality is derived by applying the results of (C.21) 6

σ2

17σ and (C.22). The third inequality holds as κ 5ασd2e 2 ≤ 2π 2 d2 by definition of κ. The forth inequality is established by substituting the upper bound for G2 (C2 ) and applying the result of (C.20). Finally, we obtain the last inequality by applying the result of (C.13). This completes the proof.

Now, we are ready to prove the last conclusion of Theorem 4.5, i.e. the linear convergence rate

for both training loss and parameters, which are presented in the following Theorem C.9. Theorem C.9 (Third conclusion in Theorem 4.5). Under the same conditions as Theorem 4.5, the (t) (t) iterates C1 , C2 of gradient descent updates with respect to the training loss Letrain (C1 , C2 ) given in Lemma C.2, converge linearly to the unique local minimizer C∗ = [C1∗ , C2∗ ]⊤ of the training loss Letrain (C1 , C2 ). In particular, for all t ≥ 0, the training loss decays as (t) (t) Letrain (C1 , C2 ) − Letrain (C1∗ , C2∗ )

≤

ηµ1,α,σ 1− d

(0) (0) Letrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) .

(t) (t) ⊤ Moreover, the coefficient vector C(t) = C1 , C2 converges linearly to C∗ as (t)

ηµ1,α,σ t/2 ∗ ∥C ∥2 . − C 2 ≤ µ2,α,σ 1 − d ∗

Here, µ1,α,σ and µ2,α,σ are both positive constants solely depending on α and σ. Proof of Theorem C.9. Notice that we have demonstrated in the proof of Lemma C.7 that 2 4 σ2 σ 2 /2 σ 4 +4σ 4 ∥∇2 Letrain (C1 , C2 )∥∞ ≤ 16α σ e +9αe = ι for all [C1 , C2 ]⊤ ∈ R, which can implies that d √ ∥∇2 Letrain (C1 , C2 )∥2 ≤ 2ι. Then we can obtain that √ 2 2η ι (t+1) (t+1) (t) (t) (t) (t) 2 (t) (t) 2 e e e Ltrain (C1 , C2 ) ≤Ltrain (C1 , C2 ) − η ∇Ltrain (C1 , C2 ) 2 + ∇Letrain (C1 , C2 ) 2 2 √ 2ηι (t) (t) 2 (t) (t) ∇Letrain (C1 , C2 ) 2 ≤Letrain (C1 , C2 ) − η 1 − 2 ηµ1,α,σ e (t) (t) (t) (t) Ltrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) , ≤Letrain (C1 , C2 ) − d where the first inequality is established by second-order Taylor’s expansion and the fact that (t+1) (t+1) ⊤ (t) (t) (t) (t) e [C1 , C2 ] − [C1 , C2 ]⊤ = η∇L is obtained by PL condi√train (C1 , C2 ). The last inequality ∗ e tion established in Lemma C.8, and 2ηι ≤ 1. Then by minus Ltrain (C1 , C2∗ ) on both sides of the inequality above, we can obtain that ηµ 1,α,σ (t+1) (t+1) (t) (t) ∗ ∗ Letrain (C1 , C2 ) − Letrain (C1 , C2 ) ≤ 1 − Letrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) d ≤... ηµ1,α,σ t+1 e (0) (0) ≤ 1− Ltrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) . d (C.23) This completes the proof for the linear convergence rate of loss decay. In the next, we prove the parameter convergence. Notice that for any [C1 , C2 ]⊤ ∈ R, utilizing the definitions of G∗ (C2 ) and ∆(C1 , C2 ) in Lemma C.8, we have 2

C1 − C1∗ = ∆(C1 , C2 ) + G∗ (C2 ) − G∗ (C2∗ ) ≤ ∆(C1 , C2 ) + 2αeσ /2 C2 − C2∗ where the equality holds by the definition of ∆(C1 , C2 ), and the fact that G∗1 = G∗ (C2∗ ). The 2 second inequality is derived by triangle inequality and |G′∗ (C2 )| ≤ 2αeσ /2 . Therefore, by (a + b)2 ≤ 43

2a2 + 2b2 , and the inequality established above, we can further obtain that C1 − C1∗

+ C2 − C2∗

≤ 2∆2 (C1 , C2 ) + 1 + 4αeσ

C2 − C2∗ .

(C.24)

On the other hand, we can calculate that 2

σ ∆ (C1 , C2 )G2 (C2 ) Letrain (C1 , C2 ) − Ltrain (C1∗ , C2∗ ) = + ψ(C2 ) − ψ(C2∗ ) 2d 2 σ2 2 2α2 σ 2 eσ 2 C2 − C2∗ , ≥ ∆ (C1 , C2 ) + 2πd 65d

(C.25) 2 2 σ2

σ e where the second inequality holds as G2 (C2 ) ≥ π1 as demonstrated in (C.15), and ψ ′′ (C2 ) ≥ 4α 65d ⊤ as demonstrated in (C.17). Compare (C.24) and (C.25), we can obtain that for all [C1 , C2 ] ∈ R, it holds

C1 − C1∗

+ C2 − C2∗

≤

2 130πd max{2, 1 + 4α2 σ 2 eσ } e ∗ ∗ e L (C , C ) − L (C , C ) . (C.26) train 1 2 train 1 2 min{65σ 2 , 4πα2 σ 2 eσ2 }

(0)

On the other hand, for the initialization C1 = C2 = 0, we have G∗ (C2 ) − G∗ (C2∗ ) ≤ O d1 2 (0) (0) and ∆(C1 , C2 ) + αeσ /2 ≤ O d1 . This implies that

1 (0) (0) (0) (0) (0) (0) C1 − C1∗ = ∆(C1 , C2 ) + G∗ (C2 ) − G∗ (C2∗ ) ≥ √ ∆(C1 , C2 ) . 2 Consequently, 2

(0)

C∗ 2 = C(0) − C∗ 2 = C1 − C1∗

(0)

+ C2 − C2∗

1 2 (0) (0) (0) ≥ ∆2 (C1 , C2 ) + C2 − C2∗ . (C.27) 2

Similar to the calculations in (C.25), we can obtain an upper bound for Letrain (C1 , C2 )−Letrain (C1∗ , C2∗ ) as 2

σ ∆ (C1 , C2 )G2 (C2 ) Letrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) = + ψ(C2 ) − ψ(C2∗ ) 2d 2 5σ 2 2 3α2 σ 2 eσ 2 ≤ ∆ (C1 , C2 ) + C2 − C2∗ , 2πd 2d

(C.28)

where the second inequality holds as G2 (C2 ) ≤ π5 as demonstrated in (C.15), and ψ ′′ (C2 ) ≥ 2 3α2 σ 2 eσ as demonstrated in (C.17). Compare (C.27) and (C.28), we can conclude that 2

max{2σ 2 , 3α2 σ 2 eσ } ∗ 2 (0) (0) Letrain (C1 , C2 ) − Letrain (C1∗ , C2∗ ) ≤ C 2. d For now, we have finished all the required inequalities, and we can finally derive that 2

(t)

C(t) − C∗ 2 = C1 − C1∗

(t)

+ C2 − C2∗

2 2

130πd max{2, 1 + 4α2 σ 2 eσ } e (t) (t) ∗ ∗ ≤ L (C , C ) − L (C , C ) train train 1 2 1 2 min{65σ 2 , 4πα2 σ 2 eσ2 }

(C.29)

2 ηµ1,α,σ t e 130πd max{2, 1 + 4α2 σ 2 eσ } (0) (0) ∗ ∗ 1 − L (C , C ) − L (C , C ) ≤ train train 1 2 1 2 d min{65σ 2 , 4πα2 σ 2 eσ2 } 2 2 ηµ1,α,σ t ∗ 2 130π max{2, 1 + 4α2 σ 2 eσ } · max{2σ 2 , 3α2 σ 2 eσ } ≤ 1 − C 2 d min{65σ 2 , 4πα2 σ 2 eσ2 } ηµ1,α,σ t ∗ 2 2 =µ2,α,σ 1 − C 2. d Here, the inequality is established by (C.26). The second inequality is derived by (C.23). And the last inequality is obtained by implying the results of (C.29). This completes the proof.

Proof of Theorem 4.7

In this section, we present the proof of Theorem 4.7. We begin by outlining the key insights underlying the argument. Corollary 4.3 establishes that the weight prediction produced by an L-layer transformer, when parameterized as in Theorem 4.1, converges to the maximum-margin solution on the context data. Moreover, Theorem 4.5 shows that, under supervision from a onestep GD, the parameter matrices V(t) and W(t) of the self-attention layer converge to the specific parameterization described in Theorem 4.1. Taken together, these results imply that it suffices to derive an upper bound on the Euclidean distance between the maximum-margin solution θSVM (Z0 ) and the ground truth θ ∗ . We first introduce several lemmas which will be utilized in proof. Lemma D.1. Suppose that xi are i.i.d. samples from a d-dimensional log-concave distribution with a positive definite covariance matrix Σ for all i ∈ [n]. Then for any δ > 0 and any θ ∈ Sd−1 , the following conclusion holds with probability at least 1 − δ: p λmin (Σ)δ |⟨θ, xi ⟩| ≥ , for all i ∈ [n]. 2n Proof of Lemma D.1. For all i ∈ [n], we know that ⟨θ, xi ⟩ follows a one-dimensional log-concave l distribution with mean µl = ⟨θ, E[xi ]⟩ and variance σl2 = θ ⊤ Σθ. This implies that ⟨θ,xσi ⟩−µ follows l one-dimensional isotropic log-concave distribution. Consequently, for any s > 0, we have s ⟨θ, xi ⟩ − µl µl P |⟨θ, xi ⟩| < s =P + < σl σl σl s µl ⟨θ, xi ⟩ − µl s µl 2s =P − − ≤ < − ≤ , σl σl σl σl σl σl where the last inequality holds as the probability density function of isotropic log concave distribution is always smaller than 1 as demonstrated in Lemma F.23. Taking a union bound for all i ∈ [n], we have X n 2ns P min |⟨θ, xi ⟩| < s ≤ P |⟨θ, xi ⟩| < s ≤ . σl i∈[n]

i=1

√ lδ Let the right hand side of the above inequality smaller than δ, we derive that s ≥ σ2n ≥

which completes the proof. 45

λmin (Σ)δ , 2n

Lemma D.2. Suppose that xi are i.i.d. samples from a d-dimensional log-concave distribution with mean vector µ and a positive definite covariance matrix Σ for all i ∈ [n]. Then for any δ > 0, the following conclusion holds with probability at least 1 − δ: n √ ∥xi ∥2 ≤ O d + log , for all i ∈ [n]. δ 1

ei = Σ− 2 (xi − µ) follows a d-dimensional isotropic log-concave distribution. Proof. Notice that x Then by Lemma F.25, for any s > 1 and i ∈ [n], it holds √ √ s d P ∥e xi ∥2 ≥ s d ≤ e− c ,

where c is an absolute positive constant. In addition, we have 1

ei ∥xi ∥2 ≤ Σ 2 x

+ ∥µ∥2 ≤

p λmax (Σ)∥e xi ∥2 + ∥µ∥2 .

Combine these two results, we can obtain that for any s > 1, P ∥xi ∥2 ≥

√ √ √ s d λmax (Σ)s d + ∥µ∥2 ≤ P ∥e xi ∥2 ≥ s d ≤ e− c .

Taking a union bound for all i ∈ [n], we have X n √ p p √ √ s d P max ∥xi ∥2 ≥ λmax (Σ)s d + ∥µ∥2 ≤ P ∥xi ∥2 ≥ λmax (Σ)s d + ∥µ∥2 ≤ ne− c . i∈[n]

i=1

Let the right hand side of the above inequality smaller than δ, we derive that s ≥ √cd log Therefore, by setting s = 1 + √cd log nδ , we complete the proof.

n δ

Now, we are ready to prove Theorem 4.7 Proof of Theorem 4.7. Combining the results of Theorem B.1, and Theorem 4.5, we know that the weight prediction θL = [TF(Z0 ; [V(t) ]⊗L , [W(t) ]⊗L )]d+1:2d,n+1 of the looped transformers equals to (t) that obtained by applying normalized gradient descent on rescaled dataset {(C2 xi , yi )}ni=1 with a (t) (t) (t) learning rate C1 /C2 . In addition, the rescaled dataset {(C2 xi , yi )}ni=1 shares the same maximum ∗

σ 2 /2 )]

2 )−log(1∧αe margin solution with the original context data, i.e. θSVM (Z0 ). When t ≥ Ω d[log(3∥C ∥ηµ 1,α,σ the convergence results in Theorem 4.5 implies that 2

αeσ /2 2 (t) ≤ C1 ≤ 2αeσ /2 ; 2

1 (t) ≤ C2 ≤ 2. 2

(t)

Consequently, we have

(t)

= Θ(α) ≤ O(1), and Theorem 4.3 guarantees that θL log n − θSVM (Z0 ) ≤ O . ∥θL ∥2 αL 2

(D.1)

This result demonstrates that it suffices to provide an upper bound for ∥θSVM (Z0 ) − θ ∗ ∥2 , which

can be converted to the test error of θSVM (Z0 ) as ∠(θSVM (Z0 ), θ ∗ ) ∗ θSVM (Z0 ) − θ 2 =2 sin ≤ ∠(θSVM (Z0 ), θ ∗ ) 2 −1 ∗ ≤c−1 − Px∼Dx sign(⟨θSVM (Z0 ), x⟩) ̸= sign(⟨θ , x⟩) = c− p θSVM (Z0 ) ,

(D.2) (D.3)

where the last inequality holds by Lemma F.24, and c− is an absolute positive constant. In addition, p(θ) is defined as: p(θ) = Px∼Dx sign(⟨θ, x⟩) ̸= sign(⟨θ ∗ , x⟩) for all θ ∈ Sd−1 , exactly representing the test error of θ. In the following, we focus on providing the upper bound for this term. By Lemma D.1 and Lemma D.2, with probability at least 1 − 23 δ, it holds that p n √ λmin (Σ)δ ∗ min yi ⟨θ , xi ⟩ ≥ d + log ; max ∥xi ∥2 ≤ c1 , (D.4) 6n δ i∈[n] i∈[n] √ where c1 is an absolute positive constant. We set ϵ =

12c1 n

λmin (Σ)δ , and denote N (Sd−1 , ϵ) √ d+log(nδ −1 )

as the ϵ-net on the d-dimensional unit sphere Sd−1 . Then by the definition of ϵ-net, there exist e 2 ≤ ϵ. Then for θ, e it can still achieve zero classification θe ∈ N (Sd−1 , ϵ) such that ∥θSVM (Z0 ) − θ∥ error on context dataset {(xi , yi )}ni=1 , as for any i ∈ [n], e ≥ min yi ⟨xi , θSVM (Z0 )⟩ + yi ⟨xi , θe − θSVM (Z0 )⟩ yi ⟨xi , θ⟩ i∈[n]

≥ min yi ⟨xi , θ ∗ ⟩ − ∥xi ∥2 ∥θ − θSVM (Z0 )∥2 i∈[n] p p n √ λmin (Σ)δ λmin (Σ)δ ≥ − c1 d + log ϵ≥ . 6n δ 12n Here, the second inequality holds as mini∈[n] yi ⟨xi , θSVM (Z0 )⟩ ≥ mini∈[n] yi ⟨xi , θ ∗ ⟩ by the definition e SVM (Z0 )⟩ ≥ −∥xi ∥2 ∥θ−θSVM (Z0 )∥2 by Cauchy-Schwarz inequality. of SVM solution, and yi ⟨xi , θ−θ The third inequality is derived by (D.4), and the last inequality holds by our definition of ϵ. Then for any ε > 0, we can derive that e ≥ ε ≤P ∃ θ ∈ N (Sd−1 , ϵ) : p(θ) ≥ ε, and min yi ⟨θ, xi ⟩ > 0 P p(θ) i∈[n]

d 3 d−1 −nε ≤ N (S , ϵ) e ≤ e−nε . ϵ Here, the second inequality holds as for any θ ∈ Sd−1 , P(mini∈[n] yi ⟨θ, xi ⟩ > 0) ≤ (1 − p(θ))n ≤ e−np(θ) , and we take an union bound for all θ ∈ N (Sd−1 , ϵ). The last inequality holds as Lemma F.26 d d guarantees that N (Sd−1 , ϵ) ≤ 3ϵ . Bysetting 3ϵ e−nε ≤ 3δ and replacing the definition of ϵ, √ d log √ n d 2 λmin (Σ)δ we can derive that ε = Θ . Therefore, combined with fact that (D.4) holds with n

probability at least 1 − 2δ 3 , we can conclude that p √ d log(n d) − log( λmin (Σ)δ 2 ) e p(θ) ≤ O n holds with probability at least 1 − δ. On the other hand, we can further calculate that e + Px∼D sign(⟨θSVM (Z0 ), x⟩) ̸= sign(⟨θ, e x⟩) p θeSVM (Z0 ) ≤p(θ) x e + c+ πϵ e + c+ π θSVM (Z0 ) − θe ≤ p(θ) ≤p(θ) p 2 √ d log(n d) − log( λmin (Σ)δ 2 ) ≤O . n Here, the second inequality holds by Lemma F.24, where c+ is a positive absolute constant. The third inequality is derived as θSVM (Z0 ) − θe 2 ≤ ϵ by the choice of θe in N (Sd−1 , ϵ). And the last √ λmin (Σ)δ ≤ O nd . Combined this result with (D.1) and (D.2), we inequality holds as ϵ = √ −1 12c1 n

d+log(nδ

)

finally conclude that θL θL − θ∗ ≤ − θSVM (Z0 ) + θSVM (Z0 ) − θ ∗ 2 ∥θL ∥2 ∥θ ∥ 2 L 2 2 log n d log max{n, d, λmin (Σ)−1 , δ −1 } ≤O + . αL n This completes the proof.

Additional experimental results

In this section, we further validate the equivalence between softmax transformers and normalized gradient descent in Theorem 4.1 on real-world datasets. We consider two datasets from different modalities: MNIST for image classification and SST-2 for sentiment classification. For MNIST, we use the binary classification task between digits 1 and 2, and convert each image into a d = 20 dimensional vector representation using a trained CNN encoder. For SST-2, we use a pretrained BERT encoder followed by a projection layer to obtain d = 20 dimensional sentence representations. In both cases, given the encoded feature vector xi and binary label yi ∈ {−1, 1}, we form the signed feature vector zi = yi · xi and construct the input matrix Z0 in the same form as (3.1). We then compare the hidden-layer outputs of the manually constructed looped softmax transformer in Theorem 4.1 with the corresponding normalized-gradient-descent iterates over L = 30 layers/iterations. θNGD TF Figure 8 reports the discrepancy ∥θθTF ∥2 − ∥θNGD ∥2 2 between the transformer output and the NGD iterate across layers. The discrepancy remains negligible on both MNIST and SST-2 throughout the entire trajectory. This shows that, after raw inputs are converted into vector representations by standard encoders, the constructed softmax transformer continues to closely match the NGD dynamics. These results provide empirical evidence that the equivalence in Theorem 4.1 is robust across different input modalities, including image and text data.

Figure 8: Difference between the normalized transformer output and the normalized NGD iterate on MNIST and SST-2.

Technical lemmas

F.1

Expectation calculations

Lemma F.1. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, then we have Pn Pn xi2 |)−∥(Id −θ∗ θ∗⊤ )θ0 ∥2 (xi1 +axi2 ) −⟨θ0 ,θ∗ ⟩(|e xi1 |+a|e xi1 xi2 σ2 i1 =1 i2 =1 e E − 2naσ 4 e 2 B ≤ f1 (a, σ). Pn −a⟨θ ,θ ⟩|ex |−a∥(I −θ θ⊤ )θ ∥ x ∗ ∗ 0 ∗ 0 2 i i d i=1 e Here, B = E[∥(Id − θ∗ θ∗⊤ )θ0 ∥22 Φ(−⟨θ0 , θ∗ ⟩σ)] is an absolute constant independent of a, n satisfying that B − 21 ≤ c1 (a,σ) , and f1 (a, σ) is an analytic function of a and σ, and irrelevant with n, d. d Proof of Lemma F.1. We denote K1 = ⟨θ0 , θ∗ ⟩, and K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 , then we have K12 + K22 = 1, and the p.d.f.R of K1 is given in Lemma F.20. In the next, by utilizing the Laplace ∞ transform identity S1 = 0 e−sS ds, we have 1 Pn

i=1 e

−aK1 |e xi |−aK2 xi

Z ∞

exp − s

= 0

n X

−aK1 |e xi |−aK2 xi

ds.

i=1

Substituting this into the expectation and utilizing Fubini’s Theorem to exchange the order of integral calculations, we obtain Pn Pn −K1 (|e xi1 |+a|e xi2 |)−K2 (xi1 +axi2 ) x i1 x i2 i1 =1 i2 =1 e Pn −aK |ex |−aK x E 1 i 2 i i=1 e Z ∞ X n n X −K1 (|e xi1 |+a|e xi2 |)−K2 (xi1 +axi2 ) ′ −aK1 |e xi |−aK2 xi = E e xi1 xi2 exp −s e ds′ 0

i1 ,i2 =1

i=1

Z ∞

′ −a(K1 |ex1 |+K2 x1 ) n−1 ′ ′ −a(K1 |e x1 |+K2 x1 ) ds E e−K1 (1+a)|ex1 |−K2 (1+a)x1 x21 e−s e |K1 E e−s e |K1 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) + n(n − 1)E E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) x1 x2 e−s e |K1 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e |K1 ds Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 2 − s e−a(K1 |ex1 |+K2 x1 ) − s e−a(K1 |ex1 |+K2 x1 ) n−1 =E E e x1 e n ds |K1 E e n |K1 0 Z ∞ s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + (n − 1)E E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) x1 x2 e− n e |K1 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne ds |K1 · E e Z ∞ n−1 =E A1 (s/n, 1 + a, a, K1 )[M (s/n, a, K1 )] ds |0 {z }

=nE

(I)

Z ∞ n−2 + (n − 1)E A2 (s/n, 1, a, K1 )A2 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds , {z } |0

(F.1)

(II)

where the second equation holds by the symmetries among x1 , . . . , xn and x e1 , . . . , x en , and the third equation holds by replacing s′ with s/n in the integral. In addition, the terms A1 (λ, α, a, K1 ), A2 (λ, α, a, K1 ) and M (λ, a, K1 ) are defined as: −a(K1 |e x1 |+K2 x1 ) A1 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 x21 e−λe |K1 ; −a(K1 |e x1 |+K2 x1 ) A2 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 x1 e−λe |K1 ; −a(K1 |e x1 |+K2 x1 ) M (λ, a, K1 ) = E e−λe |K1 . For the term A1 (λ, α, a, K1 ), by the fact that 1 − z ≤ e−z ≤ 1 for all z > 0, we have that A1 (λ, α, a, K1 ) ≥ E e−αK1 |ex1 |−αK2 x1 x21 |K1 − λE e−(a+α)K1 |ex1 |−(a+α)K2 x1 x21 |K1 ; A1 (λ, α, a, K1 ) ≤ E e−αK1 |ex1 |−αK2 x1 x21 |K1 .

(F.2)

(a+1)2 σ2 =2σ 2 K22 (1 + a)2 σ 2 + 1 e 2 Φ(−K1 (a + 1)σ)E

Z ∞

s Pn −a(K1 |e xi |+K2 xi ) i=2 e

e− n

ds K1

Z ∞ s Pn K22 (1 + 2a)2 σ 2 + 1 (2a+1)2 σ2 −a(K1 |e xi |+K2 xi ) e − 2 − ds K1 e 2Φ(−K1 (2a + 1)σ)E se n i=2 n 0 (a+1)2 σ2 n 2 2 2 2 2 =2σ K2 (1 + a) σ + 1 e Φ(−K1 (a + 1)σ)E Pn −a(K |ex |+K x ) K1 1 i 2 i i=2 e K22 (1 + 2a)2 σ 2 + 1 (2a+1)2 σ2 n2 2 − e 2Φ(−K1 (2a + 1)σ)E Pn 2 K1 n e−a(K1 |exi |+K2 xi ) i=2

(2a+1)σ2 Φ(−K1 (a + 1)σ) c1 (aσ) (2a+1)σ 2 ≥ σ 2 K22 (1 + a)2 σ 2 + 1 e 2 − ≥ σ 2 e 2 Φ(−(a + 1)σ) − 1 (F.3) Φ(−K1 aσ) n The first equality is true because for a normal variable z ∼ N (0, σ 2 ) and any scalar c, we have 2 2 2 2 E[z 2 e−cz ] = σ 2 (c2 σ 2 + 1)ec σ /2 and E[e−c|z| ] = 2ec σ /2 Φ(−cσ). The second equality is obtained by applying Fubini’s theorem to exchange the order of integration, and the third follows from direct calculation. The penultimate inequality is derived by applying Lemma F.19, where c1 (a) is a constant solely depending on a. Following a similar (but simpler) calculation, we can also get the upper bound for (I) as Z ∞ (I) ≤ E e−K1 (1+a)|ex1 |−K2 (1+a)x1 x21 |K1 [M (s/n, a, K1 )]n−1 ds 0 (a+1)2 σ2 n 2 2 2 2 2 =2σ K2 (1 + a) σ + 1 e Φ(−K1 (a + 1)σ)E Pn −a(K |ex |+K x ) K1 1 i 2 i i=2 e (2a+1)σ2 Φ(−K1 (a + 1)σ) c1 (aσ) σ 2 ((1 + a)2 σ 2 + 1) (2a+1)σ2 ≤σ 2 K22 (1 + a)2 σ 2 + 1 e 2 − ≤ e 2 + 1. Φ(−K1 aσ) n Φ(−aσ) (F.4) To calculate the upper and lower bounds for the term (II) is a little tricky, and we first calculate the derivatives of A2 (λ, α, a, K1 ) w.r.t. λ as dA2 (λ, α, a, K1 ) −a(K1 |e x1 |+K2 x1 ) = − E e−(α+a)(K1 |ex1 |+K2 x1 ) x1 e−λe |K1 dλ 2 2 ≤ E |x1 |e−(α+a)(K1 |ex1 |+K2 x1 ) ≤ 2σe(a+α) σ , which implies that both A2 (λ, α, a, K1 ) is Lipschitz continuous w.r.t. λ . Therefore, we can further derive that |A2 (λ, 1, a, K1 )A2 (λ, a, a, K1 ) − A2 (0, 1, a, K1 )A2 (0, a, a, K1 )| =|(A2 (λ, 1, a, K1 ) − A2 (0, 1, a, K1 ))A2 (λ, a, a, K1 ) + A2 (0, 1, a, K1 )(A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 ))| ≤|A2 (λ, 1, a, K1 ) − A2 (0, 1, a, K1 )||A2 (λ, a, a, K1 )| + |A2 (0, 1, a, K1 )||A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 )| ≤|A2 (λ, 1, a, K1 ) − A2 (0, 1, a, K1 )|(|A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 )| + |A2 (0, a, a, K1 )|) 2 2

2 2

+ |A2 (0, 1, a, K1 )||A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 )| ≤ 8λσ 3 ae5(a∨1) σ + 4λ2 σ 2 e8(a∨1) σ

where the last inequality holds by using the Lipschitz continuous properties of A2 (λ, α, a, K1 ), and 2 2 2 the facts that A2 (0, 1, a, K1 ) = −2σ 2 K2 eσ /2 Φ(−K1 σ), A2 (0, a, a, K1 ) = −2aσ 2 K2 ea σ /2 Φ(−aK1 σ)

and |K1 |, |K2 | ≤ 1. With the inequality established above and the triangle inequality, we have A2 (λ, 1, a, K1 )A2 (λ, a, a, K1 ) ≤ 4aσ 4 e A2 (λ, 1, a, K1 )A2 (λ, a, a, K1 ) ≥ 4aσ 4 e

(a2 +1)σ 2 2 (a2 +1)σ 2 2

2 2

K22 Φ(−K1 σ)Φ(−aK1 σ) + 8λσ 3 ae5(a∨1) σ + 4λ2 σ 4 e8(a∨1) σ ; K22 Φ(−K1 σ)Φ(−aK1 σ) − 8λσ 3 ae5(a∨1) σ − 4λ2 σ 2 e8(a∨1) σ . (F.5)

Now, we are ready to derive the lower and upper bounds for the term (II) based on (F.5). We first derive the lower bound as Z ∞ A2 (s/n, 1, a, K1 )A2 (s/n, a, a, K1 )[M (s/n, a, K1 )]n−2 ds (II) = 0 Z ∞ (a2 +1)σ 2 4 2 [M (s/n, a, K1 )]n−2 ds ≥4aσ e 2 K2 Φ(−K1 σ)Φ(−aK1 σ) 0 2 2

2 2 Z ∞ 4σ 2 e8(a∨1) σ s[M (s/n, a, K1 )] ds − s2 [M (s/n, a, K1 )]n−2 ds 2 n 0 0 (a2 +1)σ 2 n 4 2 =4aσ e 2 K2 Φ(−K1 σ)Φ(−aK1 σ)E Pn −a(K |ex |+K x ) K1 1 i 2 i i=3 e 2 2 2 2 n2 n3 8σ 3 ae5(a∨1) σ 8σ 2 e8(a∨1) σ E Pn E Pn − − −a(K1 |e xi |+K2 xi ) 2 −a(K1 |e xi |+K2 xi ) 3 n n2 i=3 e i=3 e σ2 c3 (a, σ) , (F.6) ≥2aσ 4 e 2 K22 Φ(−K1 σ) − n

8σ 3 ae5(a∨1) σ − n

Z ∞

n−2

3 4(a∨1)2 4σ 2

where c3 (a, σ) = 2aσ Φe2 (−aσ) , a continuous function of a and σ. Here the last equation is derived by Fubini’s theorem to exchange the order of the integral, and the last inequality is established by Lemma F.19. Similarly, we can also obtain that Z ∞ (a2 +1)σ 2 4 2 [M (s/n, a, K1 )]n−2 ds (II) ≤4aσ e 2 K2 Φ(−K1 σ)Φ(−aK1 σ) 0

2 2 Z ∞ 8aσ 3 e5(a∨1) σ

n σ2

2 2

n−2

s[M (s/n, a, K1 )]

≤2aσ 4 e 2 K22 Φ(−K1 σ) +

4σ 2 e8(a∨1) σ ds + n2

Z ∞

s2 [M (s/n, a, K1 )]n−2 ds

c3 (a, σ) . n

(F.7)

Substituting the results of (F.3), (F.4), (F.6), and (F.7) into (F.1), we complete the proof. Lemma F.2. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d standard Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, then we have Pn Pn −a⟨θ0 ,θ∗ ⟩(|e xi1 |+|e xi2 |)−a∥(Id −θ∗ θ∗⊤ )θ0 ∥2 (xi1 +xi2 ) x i1 x i2 f2 (a, σ) 1 i1 =1 i2 =1 e 2 4 E ≤ − a σ 1 − , Pn −a⟨θ ,θ ⟩|ex |−a∥(I −θ θ⊤ )θ ∥ x 2 d n ∗ ∗ 0 0 2 i i d ∗ e i=1 where f2 (a, σ) is an analytic function of a and σ, and irrelevant with n, d. Proof of Lemma F.2. The proof of this lemma is quite similar to that of Lemma F.1. We repeatedly 52

use the previous notations thatR K1 = ⟨θ0 , θ∗ ⟩, and K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . For this lemma, we ∞ leverage another identity S12 = 0 se−sS ds to obtain that Z ∞

1 Pn

i=1 e

−aK1 |e xi |−aK2 xi

2 =

X n s exp −s e−aK1 |exi |−aK2 xi ds.

i=1

Following a similar procedure in the proof of Lemma F.1, we substitute the identity above into the expectation and utilize Fubini’s Theorem to exchange the order of integral calculations to obtain, Pn Pn −aK1 (|e xi1 |+|e xi2 |)−aK2 (xi1 +xi2 ) xi1 xi2 i2 =1 e i1 =1 E Pn −aK |ex |−aK x 2 1 i 2 i i=1 e X Z ∞ n n X ′ ′ −aK1 |e xi |−aK2 xi −aK1 (|e xi1 |+|e xi2 |)−aK2 (xi1 +xi2 ) ds′ sE e = e xi1 xi2 exp −s 0

i=1

i1 ,i2 =1

Z ∞ −2aK1 |ex1 |−2aK2 x1 2 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ ′ ds sE e x1 e |K1 E e |K1 =nE 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) s′ E e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) x1 x2 e−s e |K1 + n(n − 1)E 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e ds · E e |K1 Z ∞ s −a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) = E ds |K1 E e− n e |K1 sE e−2aK1 |ex1 |−2aK2 x1 x21 e− n e n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + E sE e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) x1 x2 e− n e |K1 n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne |K1 ds · E e Z ∞ 1 n−1 = E sA1 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds n 0 | {z } (I)

Z ∞ n−1 2 n−2 s[A2 (s/n, a, a, K1 )] [M (s/n, a, K1 )] ds . + E n 0 | {z }

(F.8)

(II)

Here, A1 (λ, α, a, K1 ), A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1. We can use similar procedures in (F.3) and (F.4) from the proof of Lemma F.1 to calculate the upper and lower bounds for the term (I) as Z ∞ (I) ≥ sE e−2aK1 |ex1 |−2aK2 x1 x21 |K1 [M (s/n, a, K1 )]n−1 ds 0 Z 1 ∞ 2 −3aK1 |ex1 |−3aK2 x1 2 − s E e x1 |K1 [M (s/n, a, K1 )]n−1 ds n 0 Z ∞ 2a2 σ2 s Pn xi |+K2 xi ) 2 2 2 2 e−a(K1 |e −n i=2 =2σ 4a K2 σ + 1 e Φ(−2aσK1 )E se ds K1 0 Z ∞ s Pn 9a2 σ 2 K22 + 1 9a2 σ2 −a(K1 |e xi |+K2 xi ) 2 − e ds K1 − e 2 2Φ(−3aσK1 )E s e n i=2 n 0 53

=2σ

2 2

2 2 4a σ K22 + 1 e2a σ Φ(−2aσK1 )E

i=2 e

−a(K1 |e xi |+K2 xi ) 2

n3 9a2 σ 2 K22 + 1 9a2 σ2 e 2 4Φ(−3aσK1 )E Pn − K1 −a(K1 |e xi |+K2 xi ) 3 n e i=2 2 2 3a2 σ2 Φ(−2aK1 σ) c1 (a, σ) σ 2 e3a σ /2 Φ(−2aσ) ≥σ 2 4a2 σ 2 K22 + 1 e 2 − ≥ −1 2[Φ(−K1 aσ)]2 n 2

(F.9)

The first equality is true because for a normal random variable z ∼ N (0, σ 2 ) and any scalar c, 2 2 2 2 we have E[z 2 e−cz ] = σ 2 (c2 σ 2 + 1)ec σ /2 and E[e−c|z| ] = 2ec σ /2 Φ(−cσ). The second equality is obtained by applying Fubini’s theorem to exchange the order of integration, and the third follows from direct calculation. The penultimate inequality is derived by applying Lemma F.19, where c1 (a) is a constant solely depending on a. Following a similar (but simpler) calculation, we can also get the upper bound for (I) as Z ∞ (I) ≤ sE e−2aK1 |ex1 |−2aK2 x1 x21 |K1 [M (s/n, a, K1 )]n−1 ds 0 2a2 σ2 n2 2 2 2 2 =2σ 4a σ K2 + 1 e Φ(−2aσK1 )E Pn K1 −a(K1 |e xi |+K2 xi ) 2 i=2 e 3a2 σ2 Φ(−2aσK1 ) c1 (a, σ) σ 2 (4a2 σ 2 + 1) 3a2 σ2 ≤σ 2 4a2 σ 2 K22 + 1 e 2 + ≤ e 2 + 1, (F.10) 2[Φ(−K1 aσ)]2 n 2Φ2 (aσ) With the Lipschitz continuity of A2 (λ, α, a, K1 ) derived in the proof of Lemma F.1, we can also demonstrate that |[A2 (λ, a, a, K1 )]2 − [A2 (0, a, a, K1 )]2 | =|A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 )||A2 (λ, a, a, K1 ) + A2 (0, a, a, K1 )| ≤2|A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 )||A2 (0, a, a, K1 )| + |A2 (λ, a, a, K1 ) − A2 (0, a, a, K1 )|2 2 2

2 2

≤8λaσ 3 e9a σ /2 + 4λ2 σ 2 e8a σ . Now, by using the triangle inequality to establish upper and lower bounds for [A2 (λ, a, a, K1 )]2 from the inequalities above, we are ready to derive the lower and upper bounds for the term (II). We first derive the lower bound as Z ∞ (II) = s[A2 (s/n, a, a, K1 )]2 [M (s/n, a, K1 )]n−2 ds 0 2 2

≥4a2 σ 4 ea σ K22 [Φ(−aσK1 )]2

Z ∞

s[M (s/n, a, K1 )]n−2 ds −

0 2 2

4σ 2 e8a σ − n2

Z ∞

8aσ 3 e9a /2 n

Z ∞

s2 [M (s/n, a, K1 )]n−2 ds

s3 [M (s/n, a, K1 )]n−2 ds 0 n2 2 2 2 4 a σ 2 2 =4a σ e K2 [Φ(−aσK1 )] E Pn K1 −a(K1 |e xi |+K2 xi ) 2 i=3 e 2 2 2 2 16aσ 3 e9a σ /2 n3 24e8a σ n4 − E Pn E Pn − −a(K1 |e xi |+K2 xi ) 3 −a(K1 |e xi |+K2 xi ) 4 n n2 i=3 e i=3 e

≥a2 σ 4 K22 −

c3 (a, σ) , n

(F.11)

3a2 σ 3

where c3 (a, σ) = Φ3e3 (−aσ) is a positive continuous function of a and σ. Here the last equation is derived by Fubini’s theorem to exchange the order of the integral, and the last inequality is established by Lemma F.19. Similarly, we can also obtain that 2 4 a2 σ 2

(II) ≤4a σ e

K22 [Φ(−aσK1 )]2

Z ∞

n−2

s[M (s/n, a, K1 )] 0

4σ 2 e8a + n2

≤a2 σ 4 K22 +

Z ∞

8aσ 3 e9a ,σ /2 ds + n

Z ∞

s2 [M (s/n, a, K1 )]n−2 ds

s3 [M (s/n, a, K1 )]n−2 ds

c3 (a, σ) . n

(F.12)

Substituting the results of (F.9), (F.10), (F.11), and (F.12) into (F.8), and utilizing the fact that E[K22 ] = 1 − d1 , we complete the proof. Lemma F.3. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, then we have Pn Pn −⟨θ0 ,θ∗ ⟩(|e xi1 |+a|e xi2 |)−∥(Id −θ∗ θ∗⊤ )θ0 ∥2 (xi1 +axi2 ) |e xi1 ||e x i2 | nf4 (a, σ) i1 =1 i2 =1 e − nBa,σ,1 ≤ f3 (a, σ) + E , Pn −a⟨θ ,θ ⟩|ex |−a∥(I −θ θ⊤ )θ ∥ x ∗ ∗ 0 ∗ 0 2 i i d d i=1 e 2

σ2

where Ba,σ,1 = 2σ πe 2 , and f3 (a, σ), f4 (a, σ) are both smooth functions of a and σ, and irrelevant with n, d. Proof of Lemma F.3. We repeatedly R ∞ −sSuse the previous notations that K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − 1 ⊤ θ∗ θ∗ )θ0 ∥2 , and leverage S = 0 e ds to obtain that 1 Pn

i=1 e

−aK1 |e xi |−aK2 xi

Z ∞ =

X n −aK1 |e xi |−aK2 xi ds. exp −s e

i=1

Following a similar procedure in the proof of Lemma F.1, we substitute the identity above into the expectation and utilize Fubini’s Theorem to exchange the order of integral calculations to obtain, Pn Pn −K1 (|e xi1 |+a|e xi2 |)−K2 (xi1 +axi2 ) |e xi1 ||e x i2 | i1 =1 i2 =1 e Pn −aK |ex |−aK x E 1 i 2 i i=1 e Z ∞ X n n X −K1 (|e xi1 |+a|e xi2 |)−K2 (xi1 +axi2 ) ′ −aK1 |e xi |−aK2 xi = E e |e xi1 ||e xi2 | exp −s e ds′ 0

i1 ,i2 =1

i=1

Z ∞

−K1 (1+a)|ex1 |−K2 (1+a)x1 −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ x1 |+K2 x1 ) 2 −s′ e−a(K1 |e =nE E e |e x1 | e |K1 E e |K1 ds 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) + n(n − 1)E E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) |e x1 ||e x2 |e−s e |K1 0

′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 · E e−s e |K1 ds Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 − s e−a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e x1 |+K2 x1 ) 2 −n e =E E e |e x1 | e |K1 E e n |K1 ds 0 Z ∞ s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n x1 ||e x2 |e− n e + (n − 1)E E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) |e |K1 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne ds · E e |K1 Z ∞ n−1 =E A3 (s/n, 1 + a, a, K1 )[M (s/n, a, K1 )] ds |0 {z } (I)

Z ∞ n−2 + (n − 1)E A4 (s/n, 1, a, K1 )A4 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds , |0 {z }

(F.13)

(II)

Here, M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, and the terms A3 (λ, α, a, K1 ), A4 (λ, α, a, K1 ) are defined as: −a(K1 |e x1 |+K2 x1 ) A3 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 |e x1 |2 e−λe |K1 ; −a(K1 |e x1 |+K2 x1 ) A4 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 |e x1 |e−λe |K1 . We use the fact 1 − z ≤ e−z ≤ 1 to derive the upper and lower bounds for A3 (λ, α, a, K1 ) as A3 (λ, α, a, K1 ) ≥ E e−αK1 |ex1 |−αK2 x1 |e x1 |2 |K1 − λE e−K1 (α+a)|ex1 |−K2 (α+a)x1 |e x1 |2 |K1 ; A3 (λ, α, a, K1 ) ≤ E e−αK1 |ex1 |−αK2 x1 |e x1 |2 |K1 . We can establish the upper and lower bounds for the term (I) based on the inequalities above as, Z ∞ (I) ≥ E e−K1 (1+a)|ex1 |−K2 (1+a)x1 |e x1 |2 |K1 [M (s/n, a, K1 )]n−1 ds 0 Z 1 ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 sE e |e x1 |2 |K1 [M (s/n, a, K1 )]n−1 ds − n 0 r K12 (a+1)2 σ2 2 2 2 2 2 2 Φ(−K1 (a + 1)σ) − K1 (1 + a)σ =σ 2 1 + K1 (1 + a) σ e π 2 (a+1)2 σ 2 Z ∞ K2 2 ·e [M (s/n, a, K1 )]n−1 ds 0

−

q K12 (2a+1)2 σ2 2 σ 2 2 1 + K12 (1 + 2a)2 σ 2 e Φ(−K1 (2a + 1)σ) − π2 K1 (1 + 2a)σ n 2 (2a+1)2 σ 2 K2 2

Z ∞

s[M (s/n, a, K1 )]n−1 ds 0 (a+1)2 σ 2 n 2 ≥ − σ(a + 1)e E Pn −a(K |ex |+K x ) K1 1 i 2 i i=2 e ·e

σ 2 (8a2 + 10a + 5) (2a+1)2 σ2 n2 2 − E Pn e K1 −a(K1 |e xi |+K2 xi ) 2 n i=2 e σ(a + 1) (2a+1)σ2 −1 ≥− e 2 Φ(−aσ)

(F.14)

The first equality is true because for a standard normalq variable z ∼ N (0, σ 2 ) and any scalar c, 2 2 we have E[|z|2 e−c|z| ] = σ 2 2(1 + c2 σ 2 )ec σ /2 Φ(−cσ) − cσ π2 . The second equality is obtained by applying Fubini’s theorem to exchange the order of integration, and the fact that |K1 |, |K2 | ≤ 1. The last inequality is derived by applying Lemma F.19, where c1 (a) is a constant solely depending on a. Following a similar (but simpler) calculation, we can also get the upper bound for (I) as Z ∞ E e−K1 (1+a)|ex1 |−K2 (1+a)x1 |e x1 |2 |K1 [M (s/n, a, K1 )]n−1 ds (I) ≤ 0 (a+1)2 σ 2 n 2 2 ≤σ (2a + 5a + 5)e 2 E Pn −a(K |ex |+K x ) K1 1 i 2 i i=2 e 2 2 2 σ (2a + 5a + 5) (2a+1)σ ≤ e 2 +1 (F.15) Φ(−aσ) Following a similar procedure in the proof of Lemma F.1, we first calculate the derivative of A4 (λ, a, K1 ) w.r.t. λ as dA4 (λ, α, a, K1 ) −a(K1 |e x1 |+K2 x1 ) = − E e−(a+α)(K1 |ex1 |+K2 x1 ) |e x1 |e−λe |K1 dλ 2 2 ≤ E |e x1 |e−(a+α)(K1 |ex1 |+K2 x1 ) |K1 ≤ σ(2|a + α|σ + 1)e(a+α) σ , which implies that both A4 (λ, α, a, K1 ) and is Lipschitz continuous w.r.t. λ . Therefore, we can further derive that |A4 (λ, 1, a, K1 )A4 (λ, a, a, K1 ) − A4 (0, 1, a, K1 )A4 (0, a, a, K1 )| =|(A4 (λ, 1, a, K1 ) − A4 (0, 1, a, K1 ))A4 (λ, a, a, K1 ) + A4 (0, 1, a, K1 )(A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 ))| ≤|A4 (λ, 1, a, K1 ) − A4 (0, 1, a, K1 )||A4 (λ, a, a, K1 )| + |A4 (0, 1, a, K1 )||A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 )| ≤|A4 (λ, 1, a, K1 ) − A4 (0, 1, a, K1 )|(|A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 ))| + |A4 (0, a, a, K1 )|) + |A4 (0, 1, a, K1 )||A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 )| 2 2

2 2

≤30λ max{a, 1, σ}3 e9(|a|∨1) σ /2 + 25λ2 max{a, 1, σ}4 e8(|a|∨1) σ , where the last inequality holds (λ, α, a, K1 ), the q by using the Lipschitz continuous properties of A4q facts that A4 (0, 1, a, K1 ) =

2 K22 σ 2 /2 − 2σ 2 K Φ(−K σ)eσ 2 /2 , A (0, a, a, K ) = 1 1 4 1 π σe

2 2

2 a2 σ 2 K22 /2 − π σe

2aK1 σ 2 Φ(−aK1 σ)ea σ /2 and |K1 |, |K2 | ≤ 1. With the inequality established above and the triangle inequality, we calculate the lower and upper bounds for the term (II) as Z ∞ (II) = A4 (s/n, 1, a, K1 )A4 (s/n, a, a, K1 )[M (s/n, a, K1 )]n−2 ds 0 Z ∞ ≥A4 (0, 1, a, K1 )A4 (0, a, a, K1 ) [M (s/n, a, K1 )]n−2 ds− 0

2 2Z ∞ 30 max{a, 1, σ}3 e5(|a|∨1) σ s[M (s/n, a, K1 )]n−2 ds − n 0 2 Z ∞ 25 max{a, 1, σ}4 e8(|a|∨1) − s2 [M (s/n, a, K1 )]n−2 ds n2 0 n =A4 (0, 1, a, K1 )A4 (0, a, a, K1 )E Pn −a(K |ex |+K x ) K1 1 i 2 i i=3 e 2 2 30 max{a, 1, σ}3 e5(|a|∨1) σ n2 − E Pn −a(K1 |e xi |+K2 xi ) 2 n i=3 e 2 n3 50 max{a, 1, σ}4 e8(|a|∨1) E Pn − −a(K1 |e xi |+K2 xi ) 3 n2 i=3 e A4 (0, 1, a, K1 )A4 (0, a, a, K1 ) c3 (a, σ) ≥ − , n 2ea2 σ2 /2 Φ(−aK1 σ)

(F.16)

3 7(|a|∨1)2 σ 2

e , a positive continuous function of a and σ. Here the last where c3 (a, σ) = 8 max{a,1,σ} Φ2 (−aσ) equation is derived by Fubini’s theorem to exchange the order of the integral, and the last inequality is established by Lemma F.19. Similarly, we can also obtain that Z ∞ (II) ≤A4 (0, 1, a, K1 )A4 (0, a, a, K1 ) [M (s/n, a, K1 )]n−2 ds

0 2 σ2 Z ∞ 3 5(|a|∨1) 30 max{a, 1, σ} e

s[M (s/n, a, K1 )]n−2 ds

0 4 8(|a|∨1)2 Z ∞

25 max{a, 1, σ} e s2 [M (s/n, a, K1 )]n−2 ds n2 0 A4 (0, 1, a, K1 )A4 (0, a, a, K1 ) c3 (a, σ) ≤ + . n 2ea2 σ2 /2 Φ(−aK1 σ) +

(F.17)

1 )A4 (0,a,a,K1 ) , we To further derive a concrete bounds for the leading term F (a, K1 ) = A4 (0,1,a,K a2 σ 2 /2

Φ(−aK1 )

consider the second order Taylor’s expansion of F (a, K1 ) at K1 = 0. By utilizing the conclusion that K1 has a symmetric distribution, we have ′′ F (a, ξ)K12 E[F (a, K1 )] = F (a, 0) + E , 2 where ξ is a random variable between 0 and K1 . Since the function F (a, k) is analytic w.r.t. k on [−1, 1], and |K1 | ≤ 1 (hence |ξ| ≤ 1), its second-order derivative F ′′ (a, k) is continuous and bounded on this compact interval. Therefore, there exists a constant c4 (a, σ) such that |F ′′ (a, ξ)| ≤ c4 (a, σ). 2 σ 2 /2

By E[K12 ] = 1/d and F (a, 0) = 2σ eπ

, we can further derive that σ2

2σ 2 e 2 E[F (a, K1 )] − π

c4 (a, σ) ≤E . d

(F.18)

Substituting these results of (F.14), (F.15), (F.16), (F.17), and (F.18) into (F.13), we complete the proof.

Lemma F.4. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d standard Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, then we have Pn Pn −a⟨θ0 ,θ∗ ⟩(|e xi1 |+|e xi2 |)−a∥(Id −θ∗ θ∗⊤ )θ0 ∥2 (xi1 +xi2 ) x i2 | |e xi1 ||e 2σ 2 f5 (a, σ) f6 (a, σ) i2 =1 e i1 =1 − E ≤ + , Pn −a⟨θ ,θ ⟩|ex |−a∥(I −θ θ⊤ )θ ∥ x 2 π n d ∗ ∗ 0 2 0 i i d ∗ e i=1 where f5 (a, σ) and f6 (a, σ) are both analytic functions of a and σ, and irrelevant with n, d. Proof of Lemma F.4. We repeatedly use the previous notations that K1 = ⟨θ0 , θ∗ ⟩, and K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Following a similar procedure in the proof of Lemma F.2, we can obtain that Pn Pn −aK1 (|e xi1 |+|e xi2 |)−aK2 (xi1 +xi2 ) x i2 | |e xi1 ||e i1 =1 i2 =1 e E Pn −aK |ex |−aK x 2 1 i 2 i i=1 e X Z ∞ n n X ′ −aK1 (|e xi1 |+|e xi2 |)−aK2 (xi1 +xi2 ) ′ −aK1 |e xi |−aK2 xi = sE e |e xi1 ||e xi2 | exp −s ds′ e 0

i1 ,i2 =1

i=1

Z ∞

−2aK1 |ex1 |−2aK2 x1 −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ x1 |+K2 x1 ) ′ 2 −s′ e−a(K1 |e |K1 sE e |e x1 | e |K1 E e ds =nE 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) s′ E e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) |e x1 ||e x2 |e−s e |K1 + n(n − 1)E 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e ds · E e |K1 Z ∞ −2aK1 |ex1 |−2aK2 x1 − s e−a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) 2 −n e n = E sE e |e x1 | e ds |K1 E e |K1 n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n E sE e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) |e x1 ||e x2 |e− n e + |K1 n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne |K1 ds · E e Z ∞ 1 n−1 = E sA3 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds n 0 | {z } (I)

Z ∞ n−1 2 n−2 + E s[A4 (s/n, a, a, K1 )] [M (s/n, a, K1 )] ds . n 0 | {z }

(F.19)

(II)

Here, A3 (λ, α, a, K1 ), A4 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.3. Through similar calculation procedures in (F.14) and (F.15) from the proof of Lemma F.3, we can calculate the upper and lower bounds for the term (I) as Z ∞ (I) ≥ sE e−2aK1 |ex1 |−2aK2 x1 |e x1 |2 |K1 [M (s/n, a, K1 )]n−1 ds 0 Z 1 ∞ 2 −3aK1 |ex1 |−3aK2 x1 − s E e |e x1 |2 |K1 [M (s/n, a, K1 )]n−1 ds n 0

r Z ∞ 2 2 2 2 2K12 a2 σ 2 2K22 a2 σ 2 =σ 2 1 + 4K1 a σ e Φ(−2K1 aσ) − 2 K1 aσ e s[M (s/n, a, K1 )]n−1 ds π 0 q 9K12 a2 σ2 2 2 2 2 a2 σ 2 Z ∞ 2 Φ(−3K1 aσ) − π2 3K1 aσ 2 1 + 9K1 a σ e 9K2 2 σ e 2 s2 [M (s/n, a, K1 )]n−1 ds − n 0 2 n 2 2 ≥ − 2aσe2a σ E Pn K1 −a(K1 |e xi |+K2 xi ) 2 i=2 e 2σ 2 (18a2 + 3a + 2) 9a2 σ2 n3 − K e 2 E Pn 1 −a(K1 |e xi |+K2 xi ) 3 n i=2 e aσ 2 2 ≥− ea σ − 1. (F.20) 2 2Φ (−aσ) 2

The first equality is true because for a normal randomq variable z ∼ N (0, σ 2 ) and any scalar c, we 2 2 2 2 have E[|z|2 e−c|z| ] = σ 2 2(1 + c2 σ 2 )ec σ /2 Φ(−cσ) − cσ π2 and E[e−c|z| ] = 2ec σ /2 Φ(−cσ). The second equality is obtained by applying Fubini’s theorem to exchange the order of integration, and the third follows from direct calculation. The penultimate inequality is derived by applying Lemma F.19 Following a similar (but simpler) calculation, we can also get the upper bound for (I) as Z ∞ (I) ≤ sE e−2aK1 |ex1 |−2aK2 x1 |e x1 |2 |K1 [M (s/n, a, K1 )]n−1 ds 0 9a2 σ2 /2 n2 2 2 2 ≤2σ 4a σ + 1 e E Pn K1 −a(K1 |e xi |+K2 xi ) 2 i=2 e 2 2

(4a2 σ 2 + 1)e9a σ /2 ≤ + 1. 2Φ2 (−aσ)

(F.21)

With the Lipschitz continuity of A4 (λ, α, a, K1 ) derived in the proof of Lemma F.3, we can also demonstrate that |[A4 (λ, a, a, K1 )]2 − [A4 (0, a, a, K1 )]2 | =|A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 )||A4 (λ, a, a, K1 ) + A4 (0, a, a, K1 )| ≤2|A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 )||A4 (0, a, a, K1 )| + |A4 (λ, a, a, K1 ) − A4 (0, a, a, K1 )|2 2 2

2 2

≤10λ max{a, 1, σ}3 e9a σ /2 + 25λ2 max{a, 1, σ}4 e8a σ , Now, by using the triangle inequality to establish upper and lower bounds for [A4 (λ, a, a, K1 )]2 from the inequalities above, we are ready to derive the lower and upper bounds for the term (II). We first derive the lower bound as Z ∞ (II) = s[A4 (s/n, a, a, K1 )]2 [M (s/n, a, K1 )]n−2 ds 0

≥[A4 (0, a, a, K1 )]2 −

Z ∞

2 2

s[M (s/n, a, K1 )]n−2 ds −

0 2 2 Z ∞ 25 max{a, 1, σ}4 e8a σ

10 max{a, 1, σ}3 e9a σ /2 n

s3 [M (s/n, a, K1 )]n−2 ds

Z ∞ 0

s2 [M (s/n, a, K1 )]n−2 ds

=[A4 (0, a, a, K1 )] E Pn 2

−

−a(K1 |e xi |+K2 xi ) 2 i=3 e 2 2 n3 20 max{a, 1, σ}3 e9a σ /2

−a(K1 |e xi |+K2 xi ) 3 i=3 e n4

2 2 150 max{a, 1, σ}4 e8a σ − E Pn −a(K1 |e xi |+K2 xi ) 4 n2 i=3 e ≥

[A4 (0, a, a, K1 )]2 c3 (a, σ) − , 2 σ2 2 a n 4e Φ (−aK1 σ) 4 3a2

(F.22)

e σ where c3 (a, σ) = 2 max{a,1,σ} , a positive continuous function of a and σ. Here the last equation Φ3 (−aσ) is derived by Fubini’s theorem to exchange the order of the integral, and the last inequality is established by Lemma F.19. Similarly, we can also obtain that

(II) ≤[A4 (0, a, a, K1 )] +

2 2

Z ∞

n−2

s[M (s/n, a, K1 )]

0 2 2 Z ∞ 25 max{a, 1, σ}4 e8a σ

n2 0 2 [A4 (0, a, a, K1 )] c3 (a, σ) ≤ a2 σ2 2 . + n 4e Φ (−aK1 σ)

10 max{a, 1, σ}3 e9a σ /2 ds + n

Z ∞

s2 [M (s/n, a, K1 )]n−2 ds

s3 [M (s/n, a, K1 )]n−2 ds (F.23)

To further derive a concrete bounds for the leading term F (a, K1 ) =

[A4 (0,a,a,K1 )]2 , we consider 4ea2 σ2 Φ2 (−aK1 σ)

the forth order Taylor’s expansion of F (a, K1 ) at K1 = 0. By utilizing the conclusion that K1 has a symmetric distribution, we have ′′ F (a, ξ)K12 E[F (a, K1 )] = F (a, 0) + E , 2 where ξ is a random variable between 0 and K1 . Since the function F (a, k) is analytic w.r.t. k on [−1, 1], and |K1 | ≤ 1 (hence |ξ| ≤ 1), its fourth derivative F (2) (a, k) is continuous and bounded on this compact interval. Therefore, there exists a constant c4 (a, σ) such that |F (4) (a, ξ)| ≤ c4 (a, σ). By the fact that F (a, 0) = π2 σ 2 and E[K12 ] = 1/d, we finally derive that E[F (a, K1 )] −

2σ 2 c4 (a, σ) ≤ . π d

(F.24)

Substituting these results of (F.20), (F.21), (F.22), (F.23), and (F.24) into (F.19), we complete the proof. Lemma F.5. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have Pn −⟨θ0 ,θ∗ ⟩(1+a)|exi |−∥(Id −θ∗ θ⊤ )θ0 ∥2 (xi +axi ) ∗ 1 f7 (a, σ) 2 i=1 e E − Ba,2 e(a+ 2 )σ ≤ , Pn −a⟨θ ,θ ⟩|ex |−a∥(I −θ θ⊤ )θ ∥ x ∗ ∗ 0 0 2 i i d ∗ n i=1 e 61

1 (a+1)σ) 1 (a+1)σ) where Ba,2 = E Φ(−K . It satisfies that E Φ(−K −1 ≤ f8 (a,σ) , where f7 (a, σ), f8 (a, σ) d Φ(−K1 aσ) Φ(−K1 aσ) are both continuous functions of a and σ, and irrelevant with n, d. Proof of Lemma F.5. Following a similar procedure in the proof of Lemma F.1 to obtain that, Pn −K1 (1+a)|exi |−K2 (1+a)xi Z ∞ X n n X −K1 (1+a)|e xi |−K2 (1+a)xi ′ −aK1 |e xi |−aK2 xi i=1 e Pn −aK |ex |−aK x E ds′ = e exp −s e E 1 i 2 i e 0 i=1 i=1 i=1 Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ =nE E e e |K1 E e |K1 ds 0 Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 − s e−a(K1 |ex1 |+K2 x1 ) − s e−a(K1 |ex1 |+K2 x1 ) n−1 =E E e e n |K1 E e n |K1 ds 0 Z ∞ =E A5 (s/n, 1 + a, a, K1 )[M (s/n, a, K1 )]n−1 ds , (F.25) 0 | {z } (I)

Here, M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, and the terms A5 (λ, α, a, K1 ) is defined as: −a(K1 |e x1 |+K2 x1 ) A5 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 e−λe |K1 . We use the fact 1 − z ≤ e−z ≤ 1 to derive the upper and lower bounds for A5 (λ, α, a, K1 ) as A5 (λ, α, a, K1 ) ≥ E e−αK1 |ex1 |−αK2 x1 |K1 − λE e−K1 (α+a)|ex1 |−K2 (α+a)x1 |K1 ; A5 (λ, α, a, K1 ) ≤ E e−αK1 |ex1 |−αK2 x1 |K1 . We can establish the upper and lower bounds for the term (I) based on the inequalities above as, Z ∞ (I) ≥ E e−K1 (1+a)|ex1 |−K2 (1+a)x1 |K1 [M (s/n, a, K1 )]n−1 ds 0 Z 1 ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 sE e |K1 [M (s/n, a, K1 )]n−1 ds − n 0 Z ∞ (a+1)2 σ 2 2 =2Φ(−K1 (a + 1)σ)e [M (s/n, a, K1 )]n−1 ds 0 Z 2Φ(−K1 (2a + 1)σ) (2a+1)2 σ2 ∞ 2 − e s[M (s/n, a, K1 )]n−1 ds n 0 (2a+1)2 σ 2 2 2 2 (a+1) σ 2e n n2 ≥2Φ(−(a + 1)σ)e 2 E Pn −a(K |ex |+K x ) K1 − E Pn K1 1 i 2 i −a(K1 |e xi |+K2 xi ) 2 n i=2 e i=2 e Φ(−K1 (a + 1)σ) 2a+1 σ2 c1 (a, σ) ≥ e 2 − . (F.26) Φ(−K1 aσ) n The first equality is true because for a normal random variable z ∼ N (0, σ 2 ) and any scalar c, we 2 2 have E[e−c|z| ] = 2ec σ /2 Φ(−cσ). The second equality is obtained by applying Fubini’s theorem to exchange the order of integration, and the fact that |K1 |, |K2 | ≤ 1. The last inequality is derived by applying Lemma F.19. Following a similar (but simpler) calculation, we can also get the upper

bound for (I) as Z ∞ E e−K1 (1+a)|ex1 |−K2 (1+a)x1 |K1 [M (s/n, a, K1 )]n−1 ds (I) ≤ 0 (a+1)2 σ 2 Φ(−K1 (a + 1)σ) 2a+1 σ2 c1 (a, σ) n =2Φ(−K1 (a + 1)σ)e 2 E Pn −a(K |ex |+K x ) K1 ≤ e 2 + . 1 i 2 i Φ(−K1 aσ) n i=2 e (F.27) 1 (a+1)σ) In addition, by utilizing the Taylor’s expansion regarding the function E Φ(−K w.r.t. K1 Φ(−K1 aσ) Φ(−K1 (a+1)σ) c2 (a,σ) −1 ≤ d . at K1 = 0, we can derive that E Φ(−K1 aσ) Lemma F.6. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have Pn −2a⟨θ0 ,θ∗ ⟩|exi |−2a∥(Id −θ∗ θ⊤ )θ0 ∥2 xi 2 2 ∗ Ba,3 ea σ f9 (a, σ) i=1 e ≤ . E Pn 2 − ⊤ −a⟨θ0 ,θ∗ ⟩|e xi |−a∥(Id −θ∗ θ∗ )θ0 ∥2 xi n n2 i=1 e Φ(−2aK1 σ) Φ(−2aK1 σ) f10 (a,σ) , where f9 (a, σ), f10 (a, σ) where Ba,3 = E 2Φ 2 (−aK σ) . It satisfies that E 2Φ2 (−aK σ) − 1 ≤ d 1 1 are both continuous functions of a and σ, and irrelevant with n, d. Proof of Lemma F.6. Following a similar procedure in the proof of Lemma F.2, we can obtain that X Pn −2aK1 |exi |−2aK2 xi Z ∞ n n X e ′ −aK1 |e xi |−aK2 xi ′ −2aK1 |e xi |−2aK2 xi ds′ exp −s e = s E e E Pi=1 2 n −aK |e x |−aK x 1 2 i i 0 i=1 i=1 i=1 e Z ∞ ′ −a(K |ex |+K x ) n−1 ′ ′ −a(K |e x |+K x ) 1 1 2 1 1 1 2 1 ′ −2aK1 |e x1 |−2aK2 x1 −s e −s e =nE sE e e |K1 E e |K1 ds 0 Z ∞ s −a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) = E sE e−2aK1 |ex1 |−2aK2 x1 e− n e |K1 E e− n e |K1 ds n Z0 ∞ 1 n−1 = E sA5 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds . (F.28) n |0 {z } (I)

Here, A5 (λ, α, a, K1 ) share the same definitions as in the proof in Lemma F.5. Through similar calculation procedures in the proof of Lemma F.5, we can calculate the upper and lower bounds for the term (I) as Z ∞ (I) ≥ sE e−2aK1 |ex1 |−2aK2 x1 |K1 [M (s/n, a, K1 )]n−1 ds 0 Z 1 ∞ 2 −3aK1 |ex1 |−3aK2 x1 − s E e |K1 [M (s/n, a, K1 )]n−1 ds n 0 Z ∞ Z 2Φ(−3aK1 σ) 2a2 σ2 ∞ 2 2 2 =2Φ(−2aK1 σ)e2a σ s[M (s/n, a, K1 )]n−1 ds − e s [M (s/n, a, K1 )]n−1 ds n 0 0 2 σ2 2 2a n 4e n3 2 2 ≥2Φ(−2aK1 σ)e2a σ E Pn K − E K 1 Pn −a(K |ex |+K x ) 3 1 −a(K1 |e xi |+K2 xi ) 2 n 1 i 2 i i=2 e i=2 e 63

≥

Φ(−2aK1 σ) a2 σ2 c1 (a, σ) e − . 2Φ2 (−aK1 σ) n

(F.29)

The first equality is true because for a normal random variable z ∼ N (0, σ 2 ) and any scalar c, we 2 2 have E[e−c|z| ] = 2ec σ /2 Φ(−cσ). The second equality is obtained by applying Fubini’s theorem to exchange the order of integration, and the fact that |K1 |, |K2 | ≤ 1. The last inequality is derived by applying Lemma F.19. Following a similar (but simpler) calculation, we can also get the upper bound for (I) as Z ∞ sE e−2aK1 |ex1 |−2aK2 x1 |K1 [M (s/n, a, K1 )]n−1 ds (I) ≤ 0 n2 Φ(−2aK1 σ) a2 σ2 c1 (a, σ) 2a2 σ 2 E Pn e + . (F.30) =2Φ(−2aK1 σ)e 2 K1 ≤ −a(K |e x |+K x ) 2Φ2 (−aK1 σ) n 1 i 2 i e i=2

Φ(−2aK1 σ) In addition, by utilizing the Taylor’s expansion regarding the function E 2Φ w.r.t. K1 at 2 (−aK σ) 1 Φ(−2aK1 σ) c2 (a,σ) K1 = 0, we can derive that E 2Φ2 (−aK1 σ) − 1 ≤ d . Lemma F.7. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have Pn Pn ⊤ e−⟨θ0 ,θ∗ ⟩(|exi1 |+a|exi2 |)−∥(Id −θ∗ θ∗ )θ0 ∥2 (xi1 +axi2 ) xi1 x2i2 ⊤ − nBa,4 ≤ f11 (a, σ). E ∥(Id − θ∗ θ∗ )θ0 ∥2 i1 =1 i2 =1 Pn −a⟨θ ,θ ⟩|ex |−a∥(I −θ θ⊤ )θ ∥ x ∗ ∗ 0 ∗ 0 2 i i d i=1 e σ2

σ2

Here, Ba,4 = E[−2σ 4 e 2 K22 (a2 σ 2 K22 +1)Φ(−K1 σ)], satisfying that |Ba,4 +σ 4 e 2 (a2 σ 2 +1)| ≤ f12 (a,σ) , d and f11 (a, σ), f12 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.7. We repeatedly use the previous notations that K1 = ⟨θ0 , θ∗ ⟩, and K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Following a similar procedure in the proof of Lemma F.1, we can obtain that Pn Pn xi2 |)−K2 (xi1 +axi2 ) −K1 (|e xi1 |+a|e xi1 x2i2 i1 =1 i2 =1 e Pn −aK |ex |−aK x E K2 1 i 2 i i=1 e Z ∞ n n X X −K1 (|e xi1 |+a|e xi2 |)−K2 (xi1 +axi2 ) 2 ′ −aK1 |e xi |−aK2 xi = E K2 e xi1 xi2 exp −s e ds′ 0

i1 ,i2 =1

i=1

Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 3 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ =nE K2 E e x1 e |K1 E e |K1 ds 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) + n(n − 1)E K2 E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) x1 x22 e−s e |K1 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e |K1 ds Z ∞ − s e−a(K1 |ex1 |+K2 x1 ) n−1 −K1 (1+a)|ex1 |−K2 (1+a)x1 3 − s e−a(K1 |ex1 |+K2 x1 ) n n |K1 E e |K1 ds =E K2 E e x1 e 0 Z ∞ s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + (n − 1)E K2 E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) x1 x22 e− n e |K1 0

s −a(K |ex |+K x ) n−2 1 1 2 1 · E e− n e |K1 ds Z ∞ n−1 A6 (s/n, 1 + a, a, K1 )[M (s/n, a, K1 )] ds =E K2 0 | {z } (I)

Z ∞ n−2 A2 (s/n, 1, a, K1 )A1 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds . + (n − 1)E K2 0 | {z }

(F.31)

(II)

Here, A1 (λ, α, a, K1 ), A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1. In addition, the term A6 (λ, α, a, K1 ) is defined as: −a(K1 |e x1 |+K2 x1 ) A6 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 x31 e−λe |K1 . And we can further calculate the derivative of A6 (λ, α, a, K1 ) w.r.t. λ as dA6 (λ, α, a, K1 ) −a(K1 |e x1 |+K2 x1 ) = − E e−(α+a)(K1 |ex1 |+K2 x1 ) x31 e−λe |K1 dλ √ 2 2 ≤ E |x1 |3 e−(α+a)(K1 |ex1 |+K2 x1 ) ≤ 2 15σ 3 e(a+α) σ , which implies that A6 (λ, 1 ) is Lipschitz continuous w.r.t. λ, and |A6 (s/n, 1 + a, a, K1 ) − √α, a,3K(a+α) 2 A6 (0, 1 + a, a, K1 )| ≤ 2 15σ e s/n. Consequently, we can establish the upper and lower bounds for the term (I) as √ 2 2 Z ∞ 2 15σ 3 e(2a+1) σ (I) ≥A6 (0, 1 + a, a, K1 ) [M (s/n, a, K1 )] ds − s[M (s/n, a, K1 )]n−1 ds n 0 0 (a+1)2 σ2 n 4 2 2 2 = − 2K2 (1 + a)σ K2 (1 + a) σ + 3 e 2 Φ(−K1 (a + 1)σ)E Pn −a(K |ex |+K x ) K1 1 i 2 i i=2 e √ 2 2 n2 2 15σ 3 e(2a+1) σ E Pn − K1 −a(K1 |e xi |+K2 xi ) 2 n i=2 e 2a+1 2 Φ(−K1 (a + 1)σ) − 1. (F.32) ≥ − K2 (1 + a)σ 4 K22 (1 + a)2 σ 2 + 3 e 2 σ Φ(−K1 aσ) Z ∞

n−1

The first equality is true because for a normal random variable z ∼ N (0, σ 2 ) and any scalar c, 2 2 2 2 we have E[z 3 e−cz ] = σ 3 (c3 σ 3 + 3cσ)ec σ /2 , and E[e−c|z| ] = 2ec σ /2 Φ(−cσ), and we apply Fubini’s theorem to exchange the order of integration. The penultimate inequality is derived by applying Lemma F.19. Following a similar calculation, we can also get the upper bound for (I) as √ 2 2 Z ∞ 2 15σ 3 e(2a+1) σ s[M (s/n, a, K1 )]n−1 ds (I) ≤A6 (0, 1 + a, a, K1 ) [M (s/n, a, K1 )] ds + n 0 0 2a+1 2 Φ(−K1 (a + 1)σ) ≤ − K2 (1 + a)σ 4 K22 (1 + a)2 σ 2 + 3 e 2 σ + 1. (F.33) Φ(−K1 aσ) Z ∞

n−1

To calculate the upper and lower bounds for the term (II), we also first calculate the derivatives

of A1 (λ, α, a, K1 ) w.r.t. λ as dA1 (λ, α, a, K1 ) −a(K1 |e x1 |+K2 x1 ) = − E e−(α+a)(K1 |ex1 |+K2 x1 ) x21 e−λe |K1 dλ √ 2 2 ≤ E |x1 |2 e−(α+a)(K1 |ex1 |+K2 x1 ) ≤ 2 3σ 2 e(a+α) σ , which implies that A1 (λ, α, a, K1 ) is Lipschitz continuous w.r.t. λ. Combined with the Lipschitz continuity of A2 (λ, α, a, K1 ) established in the proof of Lemma F.1, we have |A2 (λ, 1, a, K1 )A1 (λ, a, a, K1 ) − A2 (0, 1, a, K1 )A1 (0, a, a, K1 )| ≤|A2 (λ, 1, a, K1 ) − A2 (0, 1, a, K1 )|(|A1 (λ, a, a, K1 ) − A1 (0, a, a, K1 )| + |A1 (0, a, a, K1 )|) + |A2 (0, 1, a, K1 )||A1 (λ, a, a, K1 ) − A1 (0, a, a, K1 )| 2

≤8λ(a2 σ 2 +1)e9(a∨1) /2 + 12λ2 σ 4 e9(a∨1) /2 , where the last inequality holds by using the Lipschitz continuous properties of A2 (λ, α, a, K1 ), and 2 2 2 the facts that A2 (0, 1, a, K1 ) = −2σ 2 K2 eσ /2 Φ(−K1 σ), A1 (0, a, a, K1 ) = 2σ 2 (a2 K22 σ 2 +1)ea σ /2 Φ(−aσK1 ) and |K1 |, |K2 | ≤ 1. With the inequality established above and the triangle inequality, we have A2 (λ, 1, a, K1 )A1 (λ, a, a, K1 ) ≤ −4σ 4 K2 (a2 K22 σ 2 + 1)e A2 (λ, 1, a, K1 )A2 (λ, a, a, K1 ) ≥ −4σ 4 K2 (a2 K22 σ 2 + 1)e

a2 +1 2 σ 2 a2 +1 2 σ 2

Φ(−K1 σ)Φ(−aK1 σ) + λc2 (a, σ) + λ2 c3 (a, σ); Φ(−K1 σ)Φ(−aK1 σ) − λc2 (a, σ) − λ2 c3 (a, σ). (F.34)

Now, we are ready to derive the lower and upper bounds for the term (II) based on (F.34). We first derive the lower bound as Z ∞ (II) = A2 (s/n, 1, a, K1 )A1 (s/n, a, a, K1 )[M (s/n, a, K1 )]n−2 ds 0 Z ∞ a2 +1 2 ≥ − 4σ 4 K2 (a2 K22 σ 2 + 1)e 2 σ Φ(−K1 σ)Φ(−aK1 σ) [M (s/n, a, K1 )]n−2 ds Z Z 0∞ c2 (a, σ) ∞ c (a, σ) 3 − s[M (s/n, a, K1 )]n−2 ds − s2 [M (s/n, a, K1 )]n−2 ds n n2 0 0 a2 +1 2 n 4 2 2 2 σ = − 4σ K2 (a K2 σ + 1)e 2 Φ(−K1 σ)Φ(−aK1 σ)E Pn −a(K |ex |+K x ) K1 1 i 2 i e i=3 2 3 c2 (a, σ) n 2c3 (a, σ) n E Pn − E Pn − −a(K1 |e xi |+K2 xi ) 2 −a(K1 |e xi |+K2 xi ) 3 n n2 i=3 e i=3 e σ2 c4 (a, σ) ≥ − 2σ 4 e 2 K2 (a2 σ 2 K22 + 1)Φ(−K1 σ) − , (F.35) n where c4 (a, σ) is a positive constant solely depending on a and σ. Here the last equation is derived by Fubini’s theorem to exchange the order of the integral, and the last inequality is established by Lemma F.19. Similarly, we can also obtain that Z ∞ a2 +1 2 4 2 2 2 σ (II) ≤ − 4σ K2 (a K2 σ + 1)e 2 Φ(−K1 σ)Φ(−aK1 σ) [M (s/n, a, K1 )]n−2 ds 0 Z Z c2 (a, σ) ∞ c3 (a, σ) ∞ 2 + s[M (s/n, a, K1 )]n−2 ds + s [M (s/n, a, K1 )]n−2 ds 2 n n 0 0 66

σ2

≤ − 2σ 4 e 2 K2 (a2 σ 2 K22 + 1)Φ(−K1 σ) +

c4 (a, σ) . n

(F.36)

2 4 e σ2 K 2 (a2 σ 2 K 2 + Lastly, by utilizing Taylor’s expansion regarding the function E[K (II)] = E −2σ 2 2 2 1)Φ(−K1 σ) w.r.t. K1 at K1 = 0, and the facts that E[K1 ] = 0 and E[K12 ] = 1/d, we can derive that

σ2 c5 (a, σ) E K2 (II) + σ 4 e 2 (a2 σ 2 + 1) ≤ . d

(F.37)

Substituting the results of (F.32), (F.33), (F.35), (F.36), and (F.37) into (F.31), we complete the proof. Lemma F.8. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P xi1 ||e x i2 | K2 ni1 =1 ni2 =1 e−K1 (|exi1 |+a|exi2 |)−K2 (xi1 +axi2 ) xi2 |e σ2 2 nf14 (a, σ) Pn −aK |ex |−aK x + anσ 4 e 2 ≤ f13 (a, σ) + E . 1 2 i i π d i=1 e Here, f13 (a, σ) and f14 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.8. Following a similar procedure in the proof of Lemma F.3, we can obtain that P P xi1 ||e x i2 | K2 ni1 =1 ni2 =1 e−K1 (|exi1 |+a|exi2 |)−K2 (xi1 +axi2 ) xi1 |e Pn −aK |ex |−aK x E 1 i 2 i i=1 e Z ∞ n n X X ′ −aK1 |e xi |−aK2 xi xi2 |)−K2 (xi1 +axi2 ) −K1 (|e xi1 |+a|e ds′ xi1 |e xi1 ||e xi2 | exp −s e E K2 e = 0

i=1

i1 ,i2 =1

Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 n−1 ′ −s′ e−a(K1 |ex1 |+K2 x1 ) x1 |+K2 x1 ) 2 −s′ e−a(K1 |e ds |K1 |K1 E e =nE K2 E e x1 x e1 e 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) |K1 + n(n − 1)E K2 E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) x1 |e x1 ||e x2 |e−s e 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e |K1 ds Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 − s e−a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e x1 |+K2 x1 ) 2 −n e =E K2 E e x1 x e1 e |K1 E e n |K1 ds 0 Z ∞ s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + (n − 1)E K2 E e−K1 (|ex1 |+a|ex2 |)−K2 (x1 +ax2 ) x1 |e x1 ||e x2 |e− n e |K1 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne · E e |K1 ds Z ∞ n−1 =E K2 A8 (s/n, 1 + a, a, K1 )[M (s/n, a, K1 )] ds |0 {z } (I)

Z ∞ A7 (s/n, a, a, K1 )A4 (s/n, 1, a, K1 )[M (s/n, a, K1 )]n−2 ds . + (n − 1)E K2 |0 {z }

(F.38)

(II)

Here, A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, A3 (λ, α, a, K1 ) and A4 (λ, α, a, K1 ) shares the same definitions as in the proof in Lemma F.3. In addition, the terms A7 (λ, α, a, K1 ) and A8 (λ, α, a, K1 ) are defined as: −a(K1 |e x1 |+K2 x1 ) A7 (λ, α, a, K1 ) = E e−αK1 |ex1 |−αK2 x1 |e x1 |x1 e−λe |K1 ; −a(K1 |e x1 |+K2 x1 ) A8 (λ, α, a, K1 ) = E e−K1 α|ex1 |−K2 αx1 x1 |e x1 |2 e−λe |K1 . To calculate the upper and lower bounds for the terms (I) and (II), we first calculate the derivatives of A7 (λ, α, a, K1 ) and A8 (λ, α, a, K1 ) w.r.t. λ as dA7 (λ, α, a, K1 ) −a(K1 |e x1 |+K2 x1 ) = − E e−(α+a)K1 |ex1 |−(α+a)K2 x1 |e x1 |x1 e−λe |K1 dλ 2 2 ≤E |x1 ||e x1 |e−(α+a)K1 |ex1 |−(α+a)K2 x1 |K1 ≤ 2σ 2 e(a+α) σ ; dA8 (λ, α, a, K1 ) −a(K1 |e x1 |+K2 x1 ) = − E e−(α+a)K1 |ex1 |−(α+a)K2 x1 |e x1 |2 x1 e−λe |K1 dλ 2 2 ≤E |x1 ||e x1 |2 e−(α+a)K1 |ex1 |−(α+a)K2 x1 |K1 ≤ 6σ 2 e(a+α) σ , which implies that A7 (λ, α, a, K1 ) and A8 (λ, α, a, K1 ) is Lipschitz continuous w.r.t. λ, and |A7 (s/n, a, a, K1 )− 2 2 2 2 A7 (0, a, a, K1 )| ≤ 2σ 2 e4a σ s/n, |A8 (s/n, 1+a, a, K1 )−A8 (0, 1+a, a, K1 )| ≤ 6σ 2 e(2a+1) σ s/n. Consequently, we can establish the upper and lower bounds for the term (I) as Z ∞ Z c1 (a, σ) ∞ s[M (s/n, a, K1 )]n−1 ds (I) ≥A8 (0, 1 + a, a, K1 ) [M (s/n, a, K1 )]n−1 ds − n 0 0 n n2 c1 (a, σ) =A8 (0, 1 + a, a, K1 )E Pn −a(K |ex |+K x ) K1 − E Pn K1 1 i 2 i −a(K1 |e xi |+K2 xi ) 2 n i=2 e i=2 e (2a+1)σ 2 (1 + (1 + a)2 K12 σ 2 )Φ(−(1 + a)K1 σ) − (1 + a)K1 σϕ((1 + a)K1 σ) (1 + a)σ 4 K2 e 2 ≥− − 1. 2Φ2 (−aσK1 ) (F.39) The first equality is true by applying Fubini’s theorem to exchange the order of integration. The penultimate inequality isderived by applying Lemma F.19, and the fact that A8 (0, 1 + a, a, K1 ) = 2 2 −2(1+a)σ 4 K2 e(1+a) σ /2 (1+(1+a)2 K12 σ 2 )Φ(−(1+a)K1 σ)−(1+a)K1 σ ϕ((1+a)K1 σ) . Following a similar calculation, we can also get the upper bound for (I) as Z ∞ Z c1 (a, σ) ∞ n−1 (I) ≤A8 (0, 1 + a, a, K1 ) [M (s/n, a, K1 )] ds + s[M (s/n, a, K1 )]n−1 ds n 0 0 (2a+1)σ 2 (1 + a)σ 4 K2 e 2 (1 + (1 + a)2 K12 σ 2 )Φ(−(1 + a)K1 σ) − (1 + a)K1 σϕ((1 + a)K1 σ) ≤− + 1. 2Φ2 (−aσK1 ) (F.40) Combined with the Lipschitz continuity of A4 (λ, α, a, K1 ) established in the proof of Lemma F.3, 68

we have |A7 (s/n, a, a, K1 )A4 (s/n, 1, a, K1 ) − A7 (0, a, a, K1 )A4 (0, 1, a, K1 )| ≤

c3 (a, σ)s c4 (a, σ)s2 + . n n2

Now, we are ready to derive the lower and upper bounds for the term (II) as Z ∞ A7 (s/n, a, a, K1 )A4 (s/n, 1, a, K1 )[M (s/n, a, K1 )]n−2 ds (II) = 0 Z ∞ [M (s/n, a, K1 )]n−2 ds ≥A7 (0, a, a, K1 )A4 (0, 1, a, K1 ) 0 Z Z c3 (a, σ) ∞ c4 (a, σ) ∞ 2 n−2 − s[M (s/n, a, K1 )] ds − s [M (s/n, a, K1 )]n−2 ds n n2 0 0 n =A7 (0, a, a, K1 )A4 (0, 1, a, K1 )E Pn −a(K |ex |+K x ) K1 1 i 2 i i=3 e 2 c3 (a, σ) 2c4 (a, σ) n n3 − E Pn E Pn − −a(K1 |e xi |+K2 xi ) 2 −a(K1 |e xi |+K2 xi ) 3 n n2 i=3 e i=3 e A7 (0, a, a, K1 )A4 (0, 1, a, K1 ) c5 (a, σ) ≥ − , (F.41) n 2ea2 σ2 /2 Φ(−aσK1 ) where c5 (a, σ) is a positive constant solely depending on a and σ. Here the last equation is derived by Fubini’s theorem to exchange the order of the integral, and the last inequality is established by Lemma F.19. Similarly, we can also obtain that Z ∞ [M (s/n, a, K1 )]n−2 ds (II) ≤A7 (0, a, a, K1 )A4 (0, 1, a, K1 ) 0 Z Z c3 (a, σ) ∞ c4 (a, σ) ∞ 2 n−2 + s[M (s/n, a, K1 )] ds + s [M (s/n, a, K1 )]n−2 ds 2 n n 0 0 A7 (0, a, a, K1 )A4 (0, 1, a, K1 ) c5 (a, σ) + . (F.42) ≤ n 2ea2 σ2 /2 Φ(−aσK1 ) 1 )A4 (0,1,a,K1 ) , we consider its Taylor’s To provide an exact lower and upper bound for E K2 A7 (0,a,a,K a2 σ 2 /2 2e

Φ(−aσK1 )

expansion at K1 = 0, then we have c6 (a, σ) K2 A7 (0, a, a, K1 )A4 (0, 1, a, K1 ) 2 4 σ2 E + aσ e 2 ≤ 2 2 a σ /2 π d 2e Φ(−aσK1 )

(F.43)

Substituting the results of (F.39), (F.40), (F.41), (F.42), and (F.43) into (F.38), we complete the proof. Lemma F.9. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P K2 ni=1 e−(1+a)K1 |exi |−(1+a)K2 xi xi f15 (a, σ) f16 (a, σ) 2 Pn −aK |ex |−aK x E + (1 + a)σ 2 e(2a+1)σ /2 ≤ + . 1 2 i i n d i=1 e 69

Here, f15 (a, σ) and f16 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.9. Following a similar procedure in the proof of Lemma F.5, we can obtain that Pn −K1 (1+a)|exi |−K2 (1+a)xi e xi E K2 i=1Pn −aK |ex |−aK x 1 i 2 i i=1 e Z ∞ X n n X = E K2 e−K1 (1+a)|exi |−K2 (1+a)xi xi exp −s′ e−aK1 |exi |−aK2 xi ds′ 0

i=1

Z ∞ −K1 (1+a)|ex1 |−K2 (1+a)x1 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ ds E e e x1 |K1 E e |K1 =nE K2 0 Z ∞ s −a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e x1 |+K2 x1 ) E e−K1 (1+a)|ex1 |−K2 (1+a)x1 e− n e =E K2 ds x1 |K1 E e− n e |K1 Z0 ∞ n−1 A2 (s/n, 1 + a, a, K1 )[M (s/n, a, K1 )] ds , =E K2 (F.44) 0 | {z } (I)

Here, A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1. By using the Lipschitz continuity of A2 (λ, α, a, K1 ) established in the proof of Lemma F.1, we can establish the upper and lower bounds for the term (I) as Z ∞ Z c1 (a, σ) ∞ n−1 (I) ≥A2 (0, 1 + a, a, K1 ) [M (s/n, a, K1 )] ds − s[M (s/n, a, K1 )]n−1 ds n 0 0 n2 n c1 (a, σ) P E Pn =A2 (0, 1 + a, a, K1 )E K1 − K1 n −a(K1 |e xi |+K2 xi ) −a(K1 |e xi |+K2 xi ) 2 n i=2 e i=2 e 2

(1 + a)σ 2 K2 e(2a+1)σ /2 Φ(−(a + 1)K1 σ) c2 (a, σ) ≥− − . Φ(−aK1 σ) n

(F.45)

The first equality is true by applying Fubini’s theorem to exchange the order of integration. The penultimate inequality is derived by applying Lemma F.19, and the fact that A2 (0, 1 + a, a, K1 ) = −2(1 + a)σ 2 K2 e

(1+a)2 σ 2 2

Φ(−(1 + a)K1 σ). Similarly, we also have n2 n c1 (a, σ) (I) ≤A2 (0, 1 + a, a, K1 )E Pn −a(K |ex |+K x ) K1 + E Pn K1 1 i 2 i −a(K1 |e xi |+K2 xi ) 2 n i=2 e i=2 e 2

≤−

(1 + a)σ 2 K2 e(2a+1)σ /2 Φ(−(a + 1)K1 σ) c2 (a, σ) + . Φ(−aK1 σ) n

(F.46)

(1+a)σ2 K22 e(2a+1)σ2 /2 Φ(−(a+1)K1 σ) To provide an exact lower and upper bound for E − , we consider Φ(−aK1 σ) its Taylor’s expansion at K1 = 0, then we have 2 (1 + a)σ 2 K22 eσ /2 Φ(−(a + 1)K1 σ) c3 (a, σ) 2 E − + (1 + a)σ 2 e(2a+1)σ /2 ≤ Φ(−aK1 σ) d

(F.47)

Substituting the results of (F.45), (F.46), and (F.47) into (F.44), we complete the proof. Lemma F.10. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 70

N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P P K2 ni1 =1 ni2 =1 ni3 =1 e−K1 (|exi1 |+a(|exi2 |+|exi3 |))−K2 (xi1 +axi2 +axi3 ) xi1 xi2 xi3 σ2 Pn −aK |ex |−aK x 2 E + na2 σ 6 e 2 1 i 2 i) ( i=1 e ≤f17 (a, σ) +

nf18 (a, σ) . d

Here,f17 (a, σ), f18 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.10. Following a similar procedure in the proof of Lemma F.2, we can obtain that P P P K2 ni1 =1 ni2 =1 ni3 =1 e−K1 (|exi1 |+a(|exi2 |+|exi3 |))−K2 (xi1 +axi2 +axi3 ) xi1 xi2 xi3 Pn −aK |ex |−aK x 2 E 1 i 2 i) ( i=1 e Z ∞ n n X X ′ −K1 (|e xi1 |+a(|e xi2 |+|e xi3 |))−K2 (xi1 +axi2 +axi3 ) ′ −aK1 |e xi |−aK2 xi = s E K2 e xi1 xi2 xi3 exp −s ds′ e 0

i1 ,i2 ,i3 =1

i=1

Z ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 3 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ ′ |K1 =nE K2 sE e x1 e |K1 E e ds 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) s′ E e−K1 (|ex1 |+2a|ex2 |)−K2 (x1 +2ax2 ) x1 x22 e−s e |K1 + n(n − 1)E K2 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e ds · E e |K1 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) + 2n(n − 1)E K2 s′ E e−K1 (a|ex1 |+(a+1)|ex2 |)−K2 (ax1 +(a+1)x2 ) x1 x22 e−s e |K1 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e ds · E e |K1 Z ∞ + n(n − 1)(n − 2)E K2 s′ E e−K1 (|ex1 |+a|ex2 |+a|ex3 |)−K2 (x1 +ax2 +ax3 ) x1 x2 x3 0 −s′ e−a(K1 |ex1 |+K2 x1 ) n−3 ′ ′ −a(K |e x |+K x ) ′ −a(K x2 |+K2 x2 ) −s′ e−a(K1 |e x3 |+K2 x3 ) 1 1 2 1 −s e 1 |e −s e ds ·e |K1 E e |K1 Z ∞ s −a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) = E K2 sE e−K1 (1+2a)|ex1 |−K2 (1+2a)x1 x31 e− n e |K1 E e− n e |K1 ds n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + E K2 sE e−K1 (|ex1 |+2a|ex2 |)−K2 (x1 +2ax2 ) x1 x22 e− n e |K1 n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne · E e |K1 ds Z ∞ s −a(K1 |e 2(n − 1) x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n E K2 |K1 + sE e−K1 (a|ex1 |+(a+1)|ex2 |)−K2 (ax1 +(a+1)x2 ) x1 x22 e− n e n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne · E e |K1 ds Z ∞ (n − 1)(n − 2) + E K2 sE e−K1 (|ex1 |+a|ex2 |+a|ex3 |)−K2 (x1 +ax2 +ax3 ) x1 x2 x3 n 0 71

s −a(K1 |ex1 |+K2 x1 ) n−3 s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) − s e−a(K1 |e x3 |+K2 x3 ) n n · e− n e |K1 E e− n e |K1 ds Z ∞ 1 n−1 sA6 (s/n, 1 + 2a, a, K1 )[M (s/n, a, K1 )] ds = E K2 n 0 | {z } (I)

Z ∞ n−1 n−2 sA2 (s/n, 1, a, K1 )A1 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds E K2 + n 0 | {z } (II)

Z ∞ 2(n − 1) n−2 sA2 (s/n, a, a, K1 )A1 (s/n, a + 1, a, K1 )[M (s/n, a, K1 )] ds + E K2 n |0 {z } (III)

Z ∞ (n − 1)(n − 2) 2 n−3 + sA2 (s/n, a, a, K1 )A2 (s/n, 1, a, K1 )[M (s/n, a, K1 )] ds . E K2 n {z } |0

(F.48)

(IV )

A6 (0, 1 + 2a, a, K1 ) 4ea2 σ2 Φ2 (−aσK1 )

2a2 +4a+1 σ2 Φ(−K1 (2a + 1)σ) c1 (a, σ) 2 = (I) + K2 (1 + 2a)σ 4 K22 (1 + 2a)2 σ 2 + 3 e ; ≤ Φ(−K1 aσ) n A1 (0, 2a, a, K1 )A2 (0, 1, a, K1 ) (II) + 4ea2 σ2 Φ2 (−aσK1 ) (2a2 +1)σ2 Φ(−K σ)Φ(−2aK σ) c2 (a, σ) 1 1 2 = (II) + σ 4 K2 1 + 4a2 K22 σ 2 e ≤ ; Φ2 (−aK1 σ) n A1 (0, a + 1, a, K1 )A2 (0, a, a, K1 ) (III) + 4ea2 σ2 Φ2 (−aσK1 ) 2a+1 2 Φ(−(a + 1)K σ) c3 (a, σ) 1 = (III) + aσ 4 K2 1 + (a + 1)2 K22 σ 2 e 2 σ ≤ . (F.49) Φ(−aK1 σ) n Specifically, for the term (IV ), we have (IV ) +

A2 (0, 1, a, K1 )A22 (0, a, a, K1 ) Φ(−K1 σ) c4 (a, σ) 2 = (IV ) + a2 σ 6 K23 eσ /2 ≤ . 2 σ2 2 a Φ(−aK1 σ) n 4e Φ (−aσK1 )

(F.50)

2 Φ(−K1 σ) Lastly, by utilizing Taylor’s expansion regarding the function E a2 σ 6 K24 eσ /2 Φ(−aK w.r.t. K1 1 σ) 2 at K1 = 0, and the facts that E[K1 ] = 0 and E[K1 ] = 1/d, we can derive that σ2 c4 (a, σ) c5 (a, σ) + . E K2 (IV ) + a2 σ 6 e 2 ≤ n d Substituting the results of (F.49), (F.50), and (F.51) into (F.48), we complete the proof.

(F.51)

Lemma F.11. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P P K2 ni1 =1 ni2 =1 ni3 =1 e−K1 (|exi1 |+a(|exi2 |+|exi3 |))−K2 (xi1 +axi2 +axi3 ) |e xi2 |xi3 xi1 ||e σ2 2 Pn −aK |ex |−aK x 2 + naσ 4 e 2 E 1 i 2 i) π ( i=1 e ≤f19 (a, σ) +

nf20 (a, σ) . d

Here, f19 (a, σ), f20 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.11. Following a similar procedure in the proof of Lemma F.10, we can obtain that P P P xi1 ||e K2 ni1 =1 ni2 =1 ni3 =1 e−K1 (|exi1 |+a(|exi2 |+|exi3 |))−K2 (xi1 +axi2 +axi3 ) |e xi2 |xi3 P E ( ni=1 e−aK1 |exi |−aK2 xi )2 Z ∞ n n X X ′ xi3 |))−K2 (xi1 +axi2 +axi3 ) −aK1 |e xi |−aK2 xi ′ xi2 |+|e −K1 (|e xi1 |+a(|e |e xi1 ||e xi2 |xi3 exp −s e ds′ = s E K2 e 0

i=1

i1 ,i2 ,i3 =1

Z ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ x1 |+K2 x1 ) ′ 2 −s′ e−a(K1 |e =nE K2 sE e x1 |e x1 | e |K1 E e |K1 ds 0 Z ∞ + n(n − 1)E K2 s′ E e−K1 (|ex1 |+2a|ex2 |)−K2 (x1 +2ax2 ) |e x1 ||e x2 |x2 0 n−2 ′ −s′ e−a(K1 |ex1 |+K2 x1 ) x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) −s′ e−a(K1 |e ds |K1 ·e |K1 E e Z ∞ s′ E e−K1 (a|ex1 |+(a+1)|ex2 |)−K2 (ax1 +(a+1)x2 ) x1 |e x2 |2 + n(n − 1)E K2 0 n−2 ′ ′ −a(K1 |ex1 |+K2 x1 ) ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) ds |K1 · e−s e |K1 E e−s e Z ∞ + n(n − 1)E K2 s′ E e−K1 (a|ex1 |+(a+1)|ex2 |)−K2 (ax1 +(a+1)x2 ) |e x1 ||e x2 |x2 0 ′ −a(K1 |ex1 |+K2 x1 ) n−2 ′ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) · e−s e |K1 E e−s e |K1 ds Z ∞ + n(n − 1)(n − 2)E K2 x1 ||e x2 |x3 s′ E e−K1 (|ex1 |+a|ex2 |+a|ex3 |)−K2 (x1 +ax2 +ax3 ) |e 0 −s′ e−a(K1 |ex1 |+K2 x1 ) n−3 ′ x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) −s′ e−a(K1 |e x3 |+K2 x3 ) −s′ e−a(K1 |e |K1 E e |K1 ds ·e Z ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 − s e−a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) 2 −n e = E K2 sE e x1 |e x1 | e |K1 E e n |K1 ds n 0 Z ∞ n−1 + E K2 sE e−K1 (|ex1 |+2a|ex2 |)−K2 (x1 +2ax2 ) |e x1 ||e x2 |x2 n 0 − s e−a(K1 |ex1 |+K2 x1 ) n−2 s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) −n e n ·e |K1 E e n |K1 ds

Z ∞ n−1 sE e−K1 (a|ex1 |+(a+1)|ex2 |)−K2 (ax1 +(a+1)x2 ) x1 |e x 2 |2 E K2 n 0 − s e−a(K1 |ex1 |+K2 x1 ) n−2 s −a(K1 |e x |+K x2 |+K2 x2 ) 1 2 x1 ) − s e−a(K1 |e −ne n ·e |K1 E e n |K1 ds Z ∞ n−1 + sE e−K1 (a|ex1 |+(a+1)|ex2 |)−K2 (ax1 +(a+1)x2 ) |e x1 ||e x2 |x2 E K2 n 0 − s e−a(K1 |ex1 |+K2 x1 ) n−2 s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) −n e n ·e ds |K1 E e n |K1 Z ∞ sE e−K1 (|ex1 |+a|ex2 |+a|ex3 |)−K2 (x1 +ax2 +ax3 ) |e x1 ||e x2 |x3 + n(n − 1)(n − 2)E K2

− s s e−a(K1 |ex1 |+K2 x1 ) n−3 ·e ds |K1 E e n n |K1 Z ∞ 1 n−1 = E K2 sA8 (s/n, 1 + 2a, a, K1 )[M (s/n, a, K1 )] ds n |0 {z } s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) − s e−a(K1 |e x3 |+K2 x3 ) −n e n n

(I)

Z ∞ n−1 n−2 E K2 + sA4 (s/n, 1, a, K1 )A7 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds n |0 {z } (II)

Z ∞ n−1 n−2 + E K2 sA2 (s/n, a, a, K1 )A3 (s/n, a + 1, a, K1 )[M (s/n, a, K1 )] ds n |0 {z } (III)

Z ∞ n−1 n−2 sA4 (s/n, a, a, K1 )A7 (s/n, a + 1, a, K1 )[M (s/n, a, K1 )] ds E K2 + n |0 {z } (IV )

Z ∞ (n − 1)(n − 2) n−3 E K2 sA2 (s/n, a, a, K1 )A4 (s/n, a, a, K1 )A4 (s/n, 1, a, K1 )[M (s/n, a, K1 )] ds . + n |0 {z } (V )

(F.52) Here, A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, A3 (λ, α, a, K1 ) and A4 (λ, α, a, K1 ) share the same definitions as in the proof in Lemma F.3, and A7 (λ, α, a, K1 ) and A8 (λ, α, a, K1 ) shares the same definition as in the proof in Lemma F.8. Then, through similar procedures in the proof of Lemma F.7 and F.8 (by utilizing the Lipschitz continuities of these functions), we can derive that: (I) +

A8 (0, 2a + 1, a, K1 ) 4ea2 σ2 Φ2 (−aσK1 ) (1+2a)σ 4 K2 e

2a2 +4a+1 2 σ 2

(1+(1+2a)2 K12 σ 2 )Φ(−(1+2a)K1 σ)−(1+2a)K1 σϕ((1+2a)K1 σ) c1 (a, σ) = (I) + ≤ ; 2 2Φ (−aσK1 ) n A4 (0, 1, a, K1 )A7 (0, 2a, a, K1 ) (II) + 4ea2 σ2 Φ2 (−aσK1 )

2 2

= (II) +

aσ 2 K2 ea σ

2 π σe

2 σ 2 K2 2

σ2 − 2K1 σ 2 e 2 Φ(−K1 σ) σϕ(2aK1 σ) − 2aK1 σ 2 Φ(−2aK1 σ) Φ2 (−aσK1 )

(III) +

≤

c2 (a, σ) ; n

A2 (0, a, a, K1 )A3 (0, a + 1, a, K1 ) 4ea2 σ2 Φ2 (−aσK1 ) 2

(2a+1)σ c3 (a, σ) aσ 2 K2 e 2 σ 2 (1 + (a + 1)2 K12 σ 2 )Φ(−(a + 1)K1 σ) − (a + 1)K1 σ 3 ϕ((a + 1)K1 σ) ≤ ; = (III) + Φ(−aK1 σ) n A4 (0, a, a, K1 )A7 (0, a + 1, a, K1 ) (IV ) + 4ea2 σ2 Φ2 (−aσK1 ) q 2 a2 σ 2 K2 (2a+1)σ 2 a2 σ 2 2 2 (a+1)σ 2 K2 e 2 σe −2aK1 σ 2 e 2 Φ(−aK1 σ) π = (IV ) + 2Φ2 (−aσK1 ) c4 (a, σ) . (F.53) · σϕ((a+1)K1 σ)−(a+1)K1 σ 2 Φ(−(a+1)K1 σ) ≤ n

Specifically, for the term (V ), we have (V ) +

= (V ) +

A2 (0, a, a, K1 )A4 (0, a, a, K1 )A4 (0, 1, a, K1 ) 4ea2 σ2 Φ2 (−aσK1 ) q a2 σ 2 K12 − 2 2 Φ(−aK σ) q 2 − 2aK σ σe aσ 2 K2 1 1 π 2Φ(−aσK1 )

2 π σe

2 σ 2 K2 2

σ2 c5 (a, σ) . − 2K1 σ 2 e 2 Φ(−K1 σ) ≤ n (F.54)

Lastly, by utilizing Taylor’s expansion regarding the function above w.r.t. K1 at K1 = 0, and the facts that E[K1 ] = 0 and E[K12 ] = 1/d, we can derive that 2 σ2 c5 (a, σ) c6 (a, σ) E K2 (V ) + aσ 4 e 2 ≤ + . π n d

(F.55)

Substituting the results of (F.53), (F.54), and (F.55) into (F.52), we complete the proof. Lemma F.12. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P K2 ni1 =1 ni2 =1 e−K1 ((1+a)|exi1 |+a|exi2 |)−K2 ((1+a)xi1 +axi2 ) xi2 f21 (a, σ) f22 (a, σ) 2 Pn −aK |ex |−aK x 2 E + . + aσ 2 e(2a+1)σ /2 ≤ 1 2 i i n d ( i=1 e ) Here, f21 (a, σ) and f22 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.12. Following a similar procedure in the proof of Lemma F.9, we can obtain that P P K2 ni1 =1 ni2 =1 e−K1 ((1+a)|exi1 |+a|exi2 |)−K2 ((1+a)xi1 +axi2 ) xi2 P E ( ni=1 e−aK1 |exi |−aK2 xi )2 75

Z ∞ = 0

n n X X ′ −K1 ((1+a)|e xi1 |+a|e xi2 |)−K2 ((1+a)xi1 +axi2 ) −aK1 |e xi |−aK2 xi ds′ s E K2 e xi2 exp −s e ′

i1 ,i2 =1

i=1

Z ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ ′ ds sE e e x1 |K1 E e |K1 =nE K2 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) s′ E e−K1 ((1+a)|ex1 |+a|ex2 |)−K2 ((1+a)x1 +ax2 ) x2 e−s e |K1 + n(n − 1)E K2 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e |K1 ds Z ∞ −K1 (1+2a)|ex1 |−K2 (1+2a)x1 − s e−a(K1 |ex1 |+K2 x1 ) − s e−a(K1 |ex1 |+K2 x1 ) n−1 1 n n = E K2 e x1 |K1 E e |K1 ds sE e n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n sE e−K1 ((1+a)|ex1 |+a|ex2 |)−K2 ((1+a)x1 +ax2 ) x2 e− n e + E K2 |K1 n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne ds · E e |K1 Z ∞ 1 n−1 = E K2 sA2 (s/n, 1 + 2a, a, K1 )[M (s/n, a, K1 )] ds n 0 | {z } (I)

Z ∞ n−1 n−2 + E K2 sA5 (s/n, 1 + a, a, K1 )A2 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds , n 0 | {z }

(F.56)

(II)

Here, A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, and A5 (λ, α, a, K1 ) shares the same definitions as in the proof in Lemma F.5. Then, through similar procedures in the proof of previous lemmas (by utilizing the Lipschitz continuities of these functions), we can derive that: (2a2 +4a+1)σ 2

2 (1 + 2a)σ 2 K2 e Φ(−(1 + 2a)K1 σ) c1 (a, σ) A2 (0, 2a + 1, a, K1 ) = , (I) + ≤ (I) + 2 2 2Φ(−aσK1 ) n 4ea σ Φ2 (−aσK1 ) (F.57)

and A5 (0, a + 1, a, K1 )A2 (0, a, a, K1 ) aσ 2 K2 e (II) + = (II) + 2 2 4ea σ Φ2 (−aσK1 )

(2a+1)σ 2 2

c2 (a, σ) Φ(−(1 + a)K1 σ) ≤ . Φ(−aK1 σ) n (F.58)

Specifically, by considering Taylor’s expansion at K1 = 0 of the function above, we can obtain c2 (a, σ) c3 (a, σ) 2 E K2 (II) + aσ 2 e(2a+1)σ /2 ≤ + n d

(F.59)

Substituting the results of (F.57), (F.58), and (F.59) into (F.56), we complete the proof. Lemma F.13. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and

θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P K2 ni1 =1 ni2 =1 e−aK1 (|exi1 |+|exi2 |)−aK2 (xi1 +xi2 ) xi1 x2i2 f23 (a, σ) f24 (a, σ) + aσ 4 (1 + a2 σ 2 ) ≤ E + . Pn −aK |ex |−aK x 2 n d 1 2 i i e i=1 Here, f23 (a, σ), f24 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof Rof Lemma F.13. Following a similar technique utilized in the proof of previous lemmas, that ∞ 1 = 0 se−sS ds, we can obtain that S2 P P K2 ni1 =1 ni2 =1 e−aK1 (|exi1 |+|exi2 |)−aK2 (xi1 +xi2 ) xi1 x2i2 E Pn −aK |ex |−aK x 2 1 i 2 i i=1 e Z ∞ n n X X 2 ′ ′ −aK1 (|e xi1 |+|e xi2 |)−aK2 (xi1 +xi2 ) −aK1 |e xi |−aK2 xi ds′ s E K2 e xi1 xi2 exp −s e = 0

i1 ,i2 =1

i=1

Z ∞ −2aK1 |ex1 |−2aK2 x1 3 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ ′ =nE K2 sE e x1 e |K1 E e |K1 ds 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) + n(n − 1)E K2 s′ E e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) x1 x22 e−s e |K1 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e |K1 ds Z ∞ −2aK1 |ex1 |−2aK2 x1 3 − s e−a(K1 |ex1 |+K2 x1 ) − s e−a(K1 |ex1 |+K2 x1 ) n−1 1 n n = E K2 sE e x1 e |K1 E e |K1 ds n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n E K2 E e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) x1 x22 e− n e |K1 + n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne · E e ds |K1 Z ∞ 1 n−1 = E K2 sA6 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds n 0 | {z } (I)

Z ∞ n−1 n−2 E K2 sA2 (s/n, a, a, K1 )A1 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds . + n 0 | {z }

(F.60)

(II)

Here, A1 (λ, α, a, K1 ), A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, and A6 (λ, α, a, K1 ) shares the same definitions as in the proof in Lemma F.7. By utilizing the Lipschitz continuity of these terms established previously, we can obtain that 2 2

aσ 4 K2 ea σ (3 + 4a2 σ 2 K22 )Φ(−2aK1 σ) A6 (0, 2a, a, K1 ) c1 (a, σ) (I) + a2 σ2 2 = (I) + ≤ , 2 Φ (−aσK1 ) n 4e Φ (−aσK1 )

(F.61)

and (II) +

A2 (0, a, a, K1 )A1 (0, a, a, K1 ) c2 (a, σ) = (II) + σ 4 aK2 (1 + a2 K22 σ 2 ) ≤ . 2 σ2 2 a n 4e Φ (−aσK1 ) 77

(F.62)

Specifically, by considering Taylor’s expansion of the function above at K1 = 0, we can further derive that E[K2 (II)] + aσ 4 (1 + a2 σ 2 ) ≤

c2 (a, σ) c3 (a, σ) + . n d

(F.63)

Substituting the results of (F.61), (F.62), and (F.63) into (F.60), we complete the proof. Lemma F.14. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P x i2 | xi1 ||e K2 ni1 =1 ni2 =1 e−aK1 (|exi1 |+|exi2 |)−aK2 (xi1 +xi2 ) xi2 |e 2aσ 4 f25 (a, σ) f26 (a, σ) E + ≤ + . Pn −aK |ex |−aK x 2 π n d 1 i 2 i i=1 e Here, f25 (a, σ) and f26 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.14. Following a similar procedure in the proof of Lemma F.13, we can obtain that P P K2 ni1 =1 ni2 =1 e−aK1 (|exi1 |+|exi2 |)−aK2 (xi1 +xi2 ) xi2 |e xi1 ||e x i2 | E Pn −aK |ex |−aK x 2 1 i 2 i i=1 e Z ∞ n n X X ′ −aK1 (|e xi1 |+|e xi2 |)−aK2 (xi1 +xi2 ) ′ −aK1 |e xi |−aK2 xi = s E K2 e xi1 |e xi1 ||e xi2 | exp −s ds′ e 0

i1 ,i2 =1

i=1

Z ∞ −2aK1 |ex1 |−2aK2 x1 −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ x1 |+K2 x1 ) ′ 2 −s′ e−a(K1 |e |K1 sE e x1 x e1 e |K1 E e ds =nE K2 0 Z ∞ ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) s′ E e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) x1 |e x1 ||e x2 |e−s e |K1 + n(n − 1)E K2 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e ds · E e |K1 Z ∞ −2aK1 |ex1 |−2aK2 x1 − s e−a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) 2 −n e = E K2 sE e x1 x e1 e |K1 E e n |K1 ds n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + E K2 sE e−aK1 (|ex1 |+|ex2 |)−aK2 (x1 +x2 ) x1 |e x1 ||e x2 |e− n e |K1 n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne · E e |K1 ds Z ∞ 1 n−1 = E K2 sA8 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds n |0 {z } (I)

Z ∞ n−1 n−2 + E K2 sA7 (s/n, a, a, K1 )A4 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds . n |0 {z }

(F.64)

(II)

Here, M (λ, a, K1 ) shares the same definitions as in the proof in Lemma F.1, A4 (λ, α, a, K1 ) shares the same definitions as in the proof in Lemma F.3, and A7 (λ, α, a, K1 ) and A8 (λ, α, a, K1 ) shares 78

the same definitions as in the proof in Lemma F.8. Through a similar technique to utilize the Lipschitz continuity of these terms, we can obtain that (I) +

A8 (0, 2a, a, K1 ) 4ea2 σ2 Φ2 (−aσK1 )

2 2 aσ 4 K2 ea σ (1 + 4a2 K12 σ 2 )Φ(−2aK1 σ) − 2aK1 σϕ(2aK1 σ) c1 (a, σ) = (I) + ≤ , 2 Φ (−aσK1 ) n

(F.65)

and A7 (0, a, a, K1 )A4 (0, a, a, K1 ) 4ea2 σ2 Φ2 (−aσK1 ) q c2 (a, σ) aσ 2 K2 [σϕ(aK1 σ) − aK1 σ 2 Φ(−aK1 σ)] 2 − a2 σ2 K12 2 2 − 2aK σ Φ(−aK σ) ≤ . = (II) + σe 1 1 π 2 2Φ (−aσK1 ) n (F.66) (II) +

Specifically, by considering Taylor’s expansion of the function above at K1 = 0, we can further derive that E[K2 (II)] +

c2 (a, σ) c3 (a, σ) 2aσ 4 ≤ + . π n d

(F.67)

Substituting the results of (F.65), (F.66), and (F.67) into (F.64), we complete the proof. Lemma F.15. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P 2 2 K2 ni=1 e−2aK1 |exi |−2aK2 xi xi 2aσ 2 ea σ f27 (a, σ) f28 (a, σ) E + ≤ + . Pn −aK |ex |−aK x 2 n n2 nd 1 i 2 i e i=1 Here, f27 (a, σ) and f28 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.15. Following a similar procedure in the proof of Lemma F.9, we can obtain that P K2 ni=1 e−2aK1 |exi |−2aK2 xi xi E Pn −aK |ex |−aK x 2 1 i 2 i i=1 e X Z ∞ n n X ′ −2aK1 |e xi |−2aK2 xi ′ −aK1 |e xi |−aK2 xi = s E K2 e xi exp −s e ds′ 0

i=1

Z ∞ −2aK1 |ex1 |−2aK2 x1 −s′ e−a(K1 |ex1 |+K2 x1 ) −s′ e−a(K1 |ex1 |+K2 x1 ) n−1 ′ ′ =nE K2 sE e e x1 |K1 E e |K1 ds 0 Z ∞ − s e−a(K1 |ex1 |+K2 x1 ) n−1 −2aK1 |ex1 |−2aK2 x1 − s e−a(K1 |ex1 |+K2 x1 ) 1 n n sE e e x1 |K1 E e |K1 ds = E K2 n 0

Z ∞ 1 sA2 (s/n, 2a, a, K1 )[M (s/n, a, K1 )]n−1 ds . = E K2 n |0 {z }

(F.68)

(I)

(I) +

A2 (0, 2a, a, K1 ) c1 (a, σ) aσ 2 K2 ea σ Φ(−2aK1 σ) = ≤ . (I) + 2 2 Φ2 (−aσK1 ) n 4ea σ Φ2 (−aσK1 )

(F.69)

Specifically, by considering Taylor’s expansion of the function above at K1 = 0, we can further derive that 2 2

E[K2 (I)] + 2aσ 2 ea σ

≤

c1 (a, σ) c2 (a, σ) + . n d

(F.70)

Substituting the results of (F.69), and (F.70) into (F.68), we complete the proof. Lemma F.16. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ N (0, σ 2 ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P P K2 ni1 =1 ni2 =1 ni3 =1 e−aK1 (|exi1 |+|exi2 |+|exi3 |)−aK2 (xi1 +xi2 +xi3 ) xi1 xi2 xi3 f29 (a, σ) f30 (a, σ) Pn −aK |ex |−aK x 3 + a3 σ 6 ≤ E + . 1 2 i i n d ) ( i=1 e Here,f29 (a, σ), f30 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. R ∞ n−1 −sS 1 Proof of Lemma F.16. Following the identity that S1n = (n−1)! e ds, we can obtain that 0 s P P P K2 ni1 =1 ni2 =1 ni3 =1 e−aK1 (|exi1 |+|exi2 |+|exi3 |)−aK2 (xi1 +xi2 +xi3 ) xi1 xi2 xi3 Pn −aK |ex |−aK x 3 E 1 i 2 i) ( i=1 e Z ∞ n n X X 1 ′ 2 −aK1 (|e xi1 |+|e xi2 |+|e xi3 |)−aK2 (xi1 +xi2 +xi3 ) −aK1 |e xi |−aK2 xi ′ = (s ) E K2 e xi1 xi2 xi3 exp −s e ds′ 2 0 i1 ,i2 ,i3 =1 i=1 Z ∞ n−1 ′ n x1 |+K2 x1 ) x1 |+K2 x1 ) ′ 2 −3aK1 |e x1 |−3aK2 x1 3 −s′ e−a(K1 |e −s′ e−a(K1 |e (s ) E e x1 e |K1 E e ds = E K2 |K1 2 0 Z ∞ 3n(n − 1) ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) + E K2 (s′ )2 E e−aK1 (|ex1 |+2|ex2 |)−aK2 (x1 +2x2 ) x1 x22 e−s e |K1 2 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e ds |K1 Z ∞ n(n − 1)(n − 2) + E K2 (s′ )2 E e−aK1 (|ex1 |+|ex2 |+|ex3 |)−aK2 (x1 +x2 +x3 ) x1 x2 x3 2 0 −s′ e−a(K1 |ex1 |+K2 x1 ) n−3 ′ ′ −a(K |e x |+K x ) ′ −a(K x2 |+K2 x2 ) −s′ e−a(K1 |e x3 |+K2 x3 ) 1 1 2 1 1 |e −s e −s e ·e |K1 E e |K1 ds Z ∞ s −a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) = 2 E K2 s2 E e−3aK1 |ex1 |−3aK2 x1 x31 e− n e |K1 E e− n e |K1 ds 2n 0 80

Z ∞ −aK1 (|ex1 |+2|ex2 |)−aK2 (x1 +2x2 ) s −a(K1 |e 3(n − 1) x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) 2 2 −n e n s E e x x e E K |K 1 2 1 2 2n2 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne |K1 ds · E e Z ∞ −aK1 (|ex1 |+|ex2 |+|ex3 |)−aK2 (x1 +x2 +x3 ) (n − 1)(n − 2) x1 x2 x3 sE e + E K 2 2n2 0 − s e−a(K1 |ex1 |+K2 x1 ) n−3 s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) − s e−a(K1 |e x3 |+K2 x3 ) −n e n n ·e ds |K1 E e n |K1 Z ∞ 1 2 n−1 s A6 (s/n, 3a, a, K1 )[M (s/n, a, K1 )] ds = 2 E K2 2n |0 {z } +

(I)

Z ∞ 3(n − 1) 2 n−2 s A2 (s/n, a, a, K1 )A1 (s/n, 2a, a, K1 )[M (s/n, a, K1 )] ds + E K2 n2 |0 {z } (II)

Z ∞ (n − 1)(n − 2) 2 3 n−3 E K2 s A2 (s/n, a, a, K1 )[M (s/n, a, K1 )] ds . + 2n2 |0 {z }

(F.71)

(III)

Here, A1 (λ, α, a, K1 ), A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1 and A6 (λ, α, a, K1 ) shares the same definition as in the proof in Lemma F.7. Through similar procedures in the proof of Lemma F.10 (by utilizing the Lipschitz continuities of these functions), we can derive that: 2 2 3aσ 4 K2 e3a σ 3 + 9a2 σ 2 K22 Φ(−3aK1 σ) A6 (0, 3a, a, K1 ) c1 (a, σ) (I) + 3a2 σ2 /2 3 = (I) + ≤ ; 3 2Φ (−aσK1 ) n 4e Φ (−aσK1 ) 2 2

(II) +

A1 (0, 2a, a, K1 )A2 (0, a, a, K1 ) c2 (a, σ) aσ 4 K2 (1 + 4a2 K22 σ 2 )ea σ Φ(−2aK1 σ) . = ≤ (II) + 2 σ 2 /2 3 2 3a Φ (−aK1 σ) n 4e Φ (−aσK1 ) (F.72)

Specifically, for the term (IV ), we have A32 (0, a, a, K1 ) c3 (a, σ) = (IV ) + 2a3 σ 6 K23 ≤ . (F.73) 2 σ 2 /2 3 3a n 4e Φ (−aσK1 ) Lastly, by utilizing Taylor’s expansion regarding the function E 2a3 σ 6 K24 w.r.t. K1 at K1 = 0, and the facts that E[K1 ] = 0 and E[K12 ] = 1/d, we can derive that (III) +

c3 (a, σ) c4 (a, σ) + . E K2 (III) + 2a3 σ 6 ≤ n d

(F.74)

Substituting the results of (F.72), (F.73), and (F.74) into (F.71), we complete the proof. Lemma F.17. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribu-

tion, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P K2 ni1 ,i2 ,i3 =1 e−aK1 (|exi1 |+|exi2 |+|exi3 |)−aK2 (xi1 +xi2 +xi3 ) |e xi2 |xi3 xi1 ||e 2 f31 (a, σ) f32 (a, σ) Pn −aK |ex |−aK x 3 + aσ 4 ≤ E + . 1 2 i i π n d ) ( i=1 e Here, f30 (a, σ), f31 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.17. Following a similar procedure in the proof of Lemma F.16, we can obtain that P P P xi2 |xi3 xi1 ||e K2 ni1 =1 ni2 =1 ni3 =1 e−aK1 (|exi1 |+|exi2 |+|exi3 |)−aK2 (xi1 +xi2 +xi3 ) |e P E ( ni=1 e−aK1 |exi |−aK2 xi )3 Z n n X X 1 ∞ ′ 2 ′ −aK1 |e xi |−aK2 xi −aK1 (|e xi1 |+|e xi2 |+|e xi3 |)−aK2 (xi1 +xi2 +xi3 ) ds′ = e xi2 |xi3 exp −s (s ) E K2 e |e xi1 ||e 2 0 i=1 i1 ,i2 ,i3 =1 Z ∞ −3aK1 |ex1 |−3aK2 x1 n−1 ′ −s′ e−a(K1 |ex1 |+K2 x1 ) n x1 |+K2 x1 ) ′ 2 2 −s′ e−a(K1 |e = E K2 (s ) E e |K1 ds x1 |e x1 | e |K1 E e 2 0 Z ∞ + n(n − 1)E K2 (s′ )2 E e−aK1 (|ex1 |+2|ex2 |)−aK2 (x1 +2x2 ) |e x1 ||e x2 |x2 0 −s′ e−a(K1 |ex1 |+K2 x1 ) n−2 ′ x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) −s′ e−a(K1 |e ·e |K1 E e |K1 ds Z ∞ n(n − 1) E K2 (s′ )2 E e−aK1 (|ex1 |+2|ex2 |)−aK2 (x1 +2x2 ) x1 |e x2 |2 + 2 0 −s′ e−a(K1 |ex1 |+K2 x1 ) n−2 ′ ′ −a(K |e x |+K x2 |+K2 x2 ) 1 1 2 x1 ) −s′ e−a(K1 |e −s e ·e |K1 E e |K1 ds Z ∞ n(n − 1)(n − 2) + (s′ )2 E e−aK1 (|ex1 |+|ex2 |+|ex3 |)−aK2 (x1 +x2 +x3 ) |e x1 ||e x2 |x3 E K2 2 0 n−3 ′ −s′ e−a(K1 |ex1 |+K2 x1 ) x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) −s′ e−a(K1 |e x3 |+K2 x3 ) −s′ e−a(K1 |e ds |K1 ·e |K1 E e Z ∞ −3aK1 |ex1 |−3aK2 x1 − s e−a(K1 |ex1 |+K2 x1 ) n−1 s −a(K1 |e 1 x1 |+K2 x1 ) 2 2 −n e n = 2 E K2 s E e x1 |e x1 | e |K1 E e |K1 ds 2n 0 Z ∞ n−1 E K2 s2 E e−aK1 (|ex1 |+2|ex2 |)−aK2 (x1 +2x2 ) |e x1 ||e x2 |x2 + 2 n 0 − s e−a(K1 |ex1 |+K2 x1 ) n−2 s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) −n e n n ·e |K1 E e |K1 ds Z ∞ n−1 + E K2 s2 E e−aK1 (|ex1 |+2|ex2 |)−aK2 (x1 +2x2 ) x1 |e x 2 |2 2 2n 0 − s e−a(K1 |ex1 |+K2 x1 ) n−2 s −a(K1 |e x |+K x2 |+K2 x2 ) 1 2 x1 ) − s e−a(K1 |e −ne n n ·e |K1 E e |K1 ds Z ∞ (n − 1)(n − 2) + E K2 s2 E e−aK1 (|ex1 |+|ex2 |+|ex3 |)−aK2 (x1 +x2 +x3 ) |e x1 ||e x2 |x3 2 2n 0 − s s e−a(K1 |ex1 |+K2 x1 ) n−3 s −a(K1 |e x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) − s e−a(K1 |e x3 |+K2 x3 ) −n e n n |K1 E e n n |K1 ds ·e

Z ∞ 1 2 n−1 s A (s/n, 3a, a, K )[M (s/n, a, K )] ds E K 8 1 1 2 2n2 |0 {z } (I)

Z ∞ n−1 2 n−2 s A (s/n, a, a, K )A (s/n, 2a, a, K )[M (s/n, a, K )] ds + E K 4 1 7 1 1 2 n2 |0 {z } (II)

Z ∞ n−1 2 n−2 s A (s/n, a, a, K )A (s/n, 2a, a, K )[M (s/n, a, K )] ds + E K 2 1 3 1 1 2 2n2 |0 {z } (III)

Z ∞ (n − 1)(n − 2) 2 2 n−3 E K + . (F.75) s A (s/n, a, a, K )A (s/n, a, a, K )[M (s/n, a, K )] ds 2 1 1 1 2 4 2n2 0 | {z } (IV )

Here, A2 (λ, α, a, K1 ) and M (λ, a, K1 ) share the same definitions as in the proof in Lemma F.1, A3 (λ, α, a, K1 ) and A4 (λ, α, a, K1 ) share the same definitions as in the proof in Lemma F.3, and A7 (λ, α, a, K1 ) and A8 (λ, α, a, K1 ) shares the same definition as in the proof in Lemma F.8. Then, through similar procedures in the proof of Lemma F.11 (by utilizing the Lipschitz continuities of these functions), we can derive that: A8 (0, 3a, a, K1 ) 3a 4e 2 σ2 /2 Φ3 (−aσK1 ) 2 2 3aσ 4 K2 e3a σ (1 + 9a2 K12 σ 2 )Φ(−3aK1 σ) − 3aK1 σϕ(3aK1 σ) = (I) + 2Φ3 (−aσK1 ) (I) +

≤

c1 (a, σ) ; n

A4 (0, a, a, K1 )A7 (0, 2a, a, K1 ) 4e3a2 σ2 /2 Φ3 (−aσK1 ) 2 2 q 2 a2 σ 2 K1 aσ 4 K2 ea σ c2 (a, σ) 2 − 2 e = (II) + 3 ϕ(2aK1 σ) − 2aK1 σΦ(−2aK1 σ) + 2aK σΦ(−aK σ) ≤ ; 1 1 π Φ (−aK1 σ) n A2 (0, a, a, K1 )A3 (0, 2a, a, K1 ) (III) + 4e3a2 σ2 /2 Φ3 (−aσK1 ) 2 2 2 c3 (a, σ) 2 2 (1 + 4a K1 σ )Φ(−2aK1 σ) − 2aK1 σϕ(2aK1 σ) = (III) + aσ 4 K2 ea σ ≤ . (F.76) 2 Φ (−aK1 σ) n (II) +

Specifically, for the term (IV ), we have A2 (0, a, a, K1 )A24 (0, a, a, K1 ) 4e3a2 σ2 /2 Φ3 (−aσK1 ) q 2 2 a2 σ 2 K1 aσ 4 K2 c4 (a, σ) 2 − 2 ≤ = (IV ) + − 2aK1 σΦ(−aK1 σ) . πe 2Φ2 (−aK1 σ) n (IV ) +

(F.77)

Lastly, by utilizing Taylor’s expansion regarding the function above w.r.t. K1 at K1 = 0, and the facts that E[K1 ] = 0 and E[K12 ] = 1/d, we can derive that 4 c4 (a, σ) c5 (a, σ) E K2 (IV ) + aσ 4 ≤ + . π n d 83

(F.78)

Substituting the results of (F.76), (F.77), and (F.78) into (F.75), we complete the proof. Lemma F.18. Let x1 , x2 , . . . , xn ∼ N (0, σ 2 ) be n i.i.d Gaussian random variables, and x e1 , x e2 , . . . , x en ∼ 2 N (0, σ ) be another n i.i.d Gaussian random variables. In addition, a is a positive scalar, and θ∗ , θ0 ∈ Rd are two independent random vectors following uniform d−dimensional sphere distribution, and we denote K1 = ⟨θ0 , θ∗ ⟩, K2 = ∥(Id − θ∗ θ∗⊤ )θ0 ∥2 . Then we have P P 2 2 K2 ni1 =1 ni2 =1 e−aK1 (2|exi1 |+|exi2 |)−aK2 (2xi1 +xi2 ) xi2 aσ 2 ea σ f33 (a, σ) f34 (a, σ) Pn −aK |ex |−aK x 3 + E + . ≤ 1 i 2 i) n n2 nd ( i=1 e Here, f33 (a, σ) and f34 (a, σ) are both analytic functions of a and σ while irrelevant with n and d. Proof of Lemma F.18. Following a similar procedure in the proof of Lemma F.12, we can obtain that P P K2 ni1 =1 ni2 =1 e−aK1 (2|exi1 |+|exi2 |)−aK2 (2xi1 +xi2 ) xi2 P E ( ni=1 e−aK1 |exi |−aK2 xi )3 Z n n X X 1 ∞ ′ 2 (s ) E K2 = e−aK1 (2|exi1 |+|exi2 |)−aK2 (2xi1 +xi2 ) xi2 exp −s′ e−aK1 |exi |−aK2 xi ds′ 2 0 i1 ,i2 =1 i=1 Z ∞ ′ −a(K |ex |+K x ) n−1 ′ n ′ −a(K |e x |+K x ) 1 1 2 1 1 1 2 1 ′ 2 −3aK1 |e x1 |−3aK2 x1 −s e −s e (s ) E e e x1 |K1 E e |K1 ds = E K2 2 0 Z ∞ n(n − 1) ′ −a(K1 |e x1 |+K2 x1 ) −s′ e−a(K1 |e x2 |+K2 x2 ) E K2 (s′ )2 E e−aK1 (2|ex1 |+|ex2 |)−aK2 (2x1 +x2 ) x2 e−s e |K1 + 2 0 ′ −a(K |ex |+K x ) n−2 ′ 1 1 2 1 −s e · E e |K1 ds Z ∞ −3aK1 |ex1 |−3aK2 x1 − s e−a(K1 |ex1 |+K2 x1 ) − s e−a(K1 |ex1 |+K2 x1 ) n−1 1 2 s E e e n ds = 2 E K2 x1 |K1 E e n |K1 2n 0 Z ∞ s −a(K1 |e n−1 x1 |+K2 x1 ) − s e−a(K1 |e x2 |+K2 x2 ) n + E K2 s2 E e−aK1 (2|ex1 |+|ex2 |)−aK2 (2x1 +x2 ) x2 e− n e |K1 2 2n 0 s −a(K |ex |+K x ) n−2 1 1 2 1 −ne |K1 ds · E e Z ∞ 1 = 2 E K2 s2 A2 (s/n, 3a, a, K1 )[M (s/n, a, K1 )]n−1 ds 2n |0 {z } (I)

Z ∞ n−1 n−2 + E K sA (s/n, 2a, a, K )A (s/n, a, a, K )[M (s/n, a, K )] ds , 2 5 1 2 1 1 2n2 0 {z } |

(F.79)

(II)

A2 (0, 3a, a, K1 ) 3aσ 2 K2 e3a σ Φ(−3aK1 σ) c1 (a, σ) (I) + 3a2 σ2 /2 3 ≤ , = (I) + 3 2Φ (−aσK ) n 4e Φ (−aσK1 ) 1 84

(F.80)

and 2 2

A5 (0, 2a, a, K1 )A2 (0, a, a, K1 ) c2 (a, σ) aσ 2 K2 ea σ Φ(−2aK1 σ) (II) + ≤ = (II) + . 2 σ 2 /2 3 2 3a Φ (−aK1 σ) n 4e Φ (−aσK1 )

(F.81)

Specifically, by considering Taylor’s expansion at K1 = 0 of the function above, we can obtain c2 (a, σ) c3 (a, σ) 2 2 + E K2 (II) + 2aσ 2 ea σ ≤ n d

(F.82)

Substituting the results of (F.80), (F.81), and (F.82) into (F.79), we complete the proof. Lemma F.19. Let x1 , x2 . . . , xn ∼ N (0, σ 2 ) be n Gaussian random variables and x e1 , x e2 . . . , x en ∼ N (0, σ 2 ) be another n Gaussian random variables. In addition, let a, b1 , b2 , k be any absolute constants satisfying that k is an integer and b21 + b22 = 1. Then for sufficiently large n, it holds that 1 nk e

ka2 σ 2 2

where ca,σ,k =

[2Φ(−aσb1 )]k

X n −k −a(b1 |e xi |+b2 xi ) ≤E e ≤ i=1

1 nk e

ka2 σ 2 2

[2Φ(−aσb1 )]k

ca,σ,k , nk+1

2Φ(2|a|σ) + 1 is a constant solely depending on a, σ and k, and Φ(·) 2 2 nk+1 e(k−2)a σ /2 Φ(−|a|σ)k+2

denotes the c.d.f. of standard Gaussian random variable. Proof of Lemma F.19. We first denote that yi = e−a(b1 |exi |+b2 xi ) , and Sn =

i=1 yi . In addition, m2 a2 σ 2 m we can calculate that for any scalar m, E[yi ] = 2e 2 Φ(−maσb1 ). For the lower bound, since the function f (z) = z −k is convex for any z ∈ (0, ∞), we utilize the Jensen’s inequality to directly

derive that −k −k E Sn−k ≥ E[Sn ] = nE[y1 ] =

1 nk e

ka2 σ 2 2

(F.83)

[2Φ(−aσb1 )]k

To provide the upper bound, we leverage the Chernoff bound to obtain that 2 2 a2 σ 2 /2 Φ(−aσb ) 1 P Sn < nea σ /2 Φ(−aσb1 ) = P e−λSn ≥ e−λne ≤ exp nφ(λ) , 2 2 where λ is any positive scalar and φ(λ) = λea σ /2 Φ(−aσb1 ) + log E e−λyi . By the fact that 2 log(z) ≤ z − 1 and e−z ≤ 1 − z + z2 for any z ≥ 0, we can derive that a2 σ 2 2 2 2 log E e−λyi ≤ E e−λyi − 1 ≤ −λE[yi ] + λ2 E[yi2 ] = −2λe 2 Φ(−aσb1 ) + λ2 e2a σ Φ(−2aσb1 ), which implies that φ(λ) ≤ −λe

a2 σ 2 2

2 2

−3a2 σ 2 /2 Φ(−aσb ) 1

Φ(−aσb1 )+λ2 e2a σ Φ(−2aσb1 ). By choosing λ0 = e

2Φ(−2aσb1 ) Φ(−aσb1 )2 −a2 σ 2 Φ(−|a|σ)2 −a2 σ 2 we have φ(λ0 ) ≤ − 4Φ(−2aσb1 ) e ≤ − 4Φ(2|a|σ) e = −f (a, σ), and further result that

2 2 P Sn < nea σ /2 Φ(−aσb1 ) ≤ e−f (a,σ)n .

(F.84)

Now, we can start to derive the upper bound based on the established concentration results above.

With Taylor’s expansion of g(z) = z −k , we have Sn−k =

2 a2 σ 2 a2 σ 2 k(k + 1) Sn − 2ne 2 Φ(−aσb1 ) k Sn − 2ne 2 Φ(−aσb1 ) + , − k+1 (k+1)a2 σ2 /2 ka2 σ 2 n e [2Φ(−aσb1 )]k+1 ξnk+2 nk e 2 [2Φ(−aσb )]k 1

where ξn is a random variable between Sn and 2ne sides and derive that E Sn−k =

a2 σ 2 2

Φ(−aσb1 ). We take the expectation on both

2 a2 σ 2 Sn − 2ne 2 Φ(−aσb1 ) . + k(k + 1)E ka2 σ 2 ξnk+2 nk e 2 [2Φ(−aσb1 )]k 1

h i a2 σ 2 2 a2 σ 2 2 In the next, we separate the term E (Sn −2ne k+2Φ(−aσb1 )) with the event {Sn ≥ ne 2 Φ(−aσb1 )} ξn as 2 2 a2 σ 2 a2 σ 2 Sn − 2ne 2 Φ(−aσb1 ) Sn − 2ne 2 Φ(−aσb1 ) E =E 1 2 2 {Sn ≥nea σ /2 Φ(−aσb1 )} ξnk+2 ξnk+2 2 a2 σ 2 Sn − 2ne 2 Φ(−aσb1 ) +E 1{Sn <nea2 σ2 /2 Φ(−aσb1 )} . ξnk+2 We also consider providing the upper bounds for these two components, respectively. For the first term, we have 2 2 a2 σ 2 a2 σ 2 E[ Sn − 2ne 2 Φ(−aσb1 ) ] Sn − 2ne 2 Φ(−aσb1 ) 1{Sn ≥nea2 σ2 /2 Φ(−aσb1 )} ≤ k+2 (k+2)a2 σ2 /2 E n e Φ(−|a|σ)k+2 ξnk+2 2Φ(2|a|σ) ≤ k+1 (k−2)a2 σ2 /2 , n e Φ(−|a|σ)k+2 where the first inequality holds if 1{Sn ≥nea2 σ2 /2 Φ(−aσb1 )} = 1, then ξn is also larger than ne and the second inequality holds as E[(Sn −2ne For the second term, we can obtain that

a2 σ 2 /2

a2 σ 2 2

(F.85)

Φ(−aσb1 ), 2 2

Φ(−aσb1 ))2 ] = nVar(yi ) ≤ nE[y12 ] ≤ 2ne2a σ Φ(2|a|σ).

2 a2 σ 2 Sn − 2ne 2 Φ(−aσb1 ) E 1{Sn <nea2 σ2 /2 Φ(−aσb1 )} ξnk+2 2 a2 σ 2 Sn − 2ne 2 Φ(−aσb1 ) ≤E 1{Sn <nea2 σ2 /2 Φ(−aσb1 )} Snk+2 4 1/2 q a2 σ 2 Sn − 2ne 2 Φ(−aσb1 ) a2 σ 2 /2 Φ(−aσb ) ≤E P S < ne n 1 Sn2k+4 2 2

≤E[(Sn − 2nea σ /2 Φ(−aσb1 ))8 ]1/4 E[yi−4k−8 ]1/4 e−f (a,σ)n ≤ 2 2

1 . k(k + 1)nk+1

(F.86)

Here, the first inequality holds as ξn > Sn if Sn < nea σ /2 Φ(−aσb1 ). The second and third inequalities are both derived by Cauchy-Schwarz’s inequality, the facts that Sn−k ≤ y1−k , and

2 2 P Sn < nea σ /2 Φ(−aσb1 ) ≤ e−f (a,σ)n derived in (F.84). The last inequality holds as the E[(Sn − 2 2 2nea σ /2 Φ(−aσb1 ))8 ] ≤ c′a,σ,k n4 by Rosenthal’s inequality (Rosenthal, 1970), and E[y1−4k−8 ]1/4 ≤ 2 2 2

2e2(k+2) a σ . Combining the results of (F.85) and (F.86), we finish the proof for the upper bound.

F.2

Properties of Gaussian random variables, uniform sphere random variables, log-concave random variables, and covering number on unit sphere

Lemma F.20. Let a, b be two independent random vectors, each distributed uniformly on the unit sphere Sd−1 ⊂ Rd (that is, each is a unit random vector with the rotation-invariant probability measure). Let K = ⟨a, b⟩. Then its probability density function is Γ d2 d−3 (1 − k 2 ) 2 1{−1≤k≤1} . P(K ≤ k) := fK (k) = √ d−1 πΓ 2 In addition, K is independent with a, and b respectively. Moreover E[K] = 0, Var(K) = 1/d, and Γ2 ( d )

2 E[∥(Id − aa⊤ )b∥2 ] = Γ( d−1 )Γ( d+1 . ) 2

Proof of Lemma F.20. Since a and b are rotation invariant, implying that for any orthogonal matrix R, we have Ra and Rb still following the unit sphere distribution. Consequently, we can always find a specific R such that Ra = e1 . Then we have K = ⟨a, b⟩ = ⟨Ra, Rb⟩ = (Rb)1 , the first coordinate of a unit sphere random vector. And it is evident that this random variable would be independent with a, and similarly also independent with b. In the next we derive the p.d.f. of K. We have shown that K has the same distribution as a1 . In addition, the differential w.r.t. the polar coordinate system indicates that dS = sind−2 (α1 ) sind−3 (α2 ) · · · sin(ϕd−2 )dα1 · · · dαd−2 , where S is the area of unit sphere Sd−1 ⊂ Rd , and α1 , . . . , αd−1 are the angles of the polar coordinate system. Since a1 = cos α1 , and the marginal density function of α1 is proportional to sind−2 (α1 ), we have fK (k) ∝ sind−2 (α1 )

d−3 1 = (1 − k 2 ) 2 1{−1≤k≤1} . sin(α1 )

R1 R1 d−3 d−3 In addition, we have −1 (1−t2 ) 2 dt = 2 0 (1−t2 ) 2 dt = Beta 21 , d−1 2 , which proves the p.d.f. of K. By symmetry, fK is an even function, hence E[K] = 0. To compute E[K 2 ], one can use either the density or a coordinate argument. Using coordinates, write a = (a1 , . . . , ad ) and b = (b1 , . . . , bd ). By independence and symmetry of the uniform spherical law, hX i X E[K 2 ] = E ⟨a, b⟩2 = E ai aj bi bj = E[ai aj ] E[bi bj ]. i,j

i,j

For a uniform unit vector on Sd−1 we have E[ai aj ] = d1 δij . Hence E[K 2 ] =

X1 i,j

X 1 1 1 δij · δij = = . 2 d d d d i

Thus Var(K) = E[K 2 ] − (E[K])2 = 1/d. In addition, we can calculate that ∥(Id − aa⊤ )b∥22 = ∥b∥22 − 2⟨a, b⟩2 + ⟨a, b⟩2 ∥a∥22 = 1 − ⟨a, b⟩2 . 87

Then by the density function of K = ⟨a, b⟩ demonstrated previously, we can easily derive that Γ2 ( d2 )

⊤

p E[∥(Id − aa )b∥2 ] = E[ 1 − ⟨a, b⟩2 ] =

d+1 Γ( d−1 2 )Γ( 2 )

In addition, by Gautschi’s inequality, we have Γ2 ( d2 )

d−1 Γ2 ( d2 ) 1 2 = ≥ =1− . d−1 2 d−1 d−1 d+1 d/2 d Γ( 2 )Γ( 2 ) 2 Γ ( 2 )

This completes the proof of the lemma. Lemma F.21. Let a be a random vector distributed uniformly on the unit sphere Sd−1 ⊂ Rd , and x ∈ Rd be a standard Gaussian random vector. Then it holds that ⟨a, x⟩ is independent with a. Proof of Lemma F.21. It is evident that for any fixed a ∈ Sd−1 , the conditional distribution ⟨a, x⟩|a ∼ N (0, 1). Since this conditional distribution does not depend on the specific choice of a, the random variable ⟨a, x⟩ is independent of the random vector a. This completes the proof Lemma F.22. Let x1 , x2 be two Gaussian random variables with zero mean, and y = sign(x1 ). Then y is independent with y · x1 and y · x2 . Moreover, y · x2 also follows the normal distribution, which has zero mean and the same variance with x2 . Proof of Lemma F.22. W.L.O.G., we assume that x1 , x2 are standard Gaussian random variables. We first prove that y is independent with y·x1 . It is clear that y·x1 = |x1 |, which is independent with y. Next, we prove that y is independent with y·x2 and y·x2 follows the standard normal distribution. If y = 1 then y · x2 = x2 ∼ N (0, 1). On the other hand when y = −1, y · x2 = −x2 ∼ N (0, 1). Therefore, y · x2 |y ∼ N (0, 1). Since this conditional distribution does not depend on y, we have y · x2 is independent with y, and follows the standard Gaussian distribution. This completes the proof. Lemma F.23 (Lemma 2 in Balcan and Long (2013)). Suppose that a is a one-dimensional isotropic log-concave random variable, with fa (x) denoting its probability density function, then fa (x) ≤ 1 for all x ∈ R. Lemma F.24 (Lemma 3 and Theorem 4 in Balcan and Long (2013)). Suppose that u, v ∈ Sd−1 are two d-dimensional unit vectors. In addition, let x ∈ Rd be a random vector generated from an isotropic log-concave distribution, then there exist two absolute positive constants c− ≤ c+ such that c− · ∠(u, v) ≤ P sign(⟨u, x⟩) ̸= sign(⟨v, x⟩) ≤ c+ · ∠(u, v), where ∠(u, v) denotes the angle between unit vectors u and v. Lemma F.25 (Paouris’ inequality). Suppose that x ∈ Rd follows d-dimensional isotropic logconcave distribution, then for any s > 0, √ √ P ∥x∥2 ≥ cs d ≤ e−s d , where c is an absolute positive constant. 88

Lemma F.26 (Lemma 5.2 in Vershynin (2010)). Let N (Sd−1 , ϵ) denotes the ϵ-net on unit Euclidean sphere Sd−1 equipped with the Euclidean metric. Then it holds that for any ϵ > 0: d−1

N (S

, ϵ) ≤

2 1+ ϵ

d .

The proofs of Lemmas F.23 and F.24 can be found in Balcan and Long (2013), the proof of Lemma F.25 can be found in Adamczak et al. (2012), and the proof of Lemma F.26 can be found in Vershynin (2010).

References Adamczak, R., Latala, R., Litvak, A. E., Oleszkiewicz, K., Pajor, A. and TomczakJaegermann, N. (2012). A short proof of paouris’ inequality. arXiv preprint arXiv:1205.2515 . Ahn, K., Cheng, X., Daneshmand, H. and Sra, S. (2023). Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems 36 45614–45650. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T. and Zhou, D. (2022). What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 . Anwar, U., Von Oswald, J., Kirsch, L., Krueger, D. and Frei, S. (2024). Understanding in-context learning of linear models in transformers through an adversarial lens. arXiv preprint arXiv:2411.05189 . Bai, Y., Chen, F., Wang, H., Xiong, C. and Mei, S. (2024). Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems 36. Balcan, M.-F. and Long, P. (2013). Active and passive learning of linear separators under log-concave distributions. In Conference on Learning Theory. PMLR. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020). Language models are few-shot learners. Advances in neural information processing systems 33 1877–1901. Cao, Y., He, Y., Wu, D., Chen, H.-Y., Fan, J. and Liu, H. (2025). Transformers simulate mle for sequence generation in bayesian networks. arXiv preprint arXiv:2501.02547 . Chen, B., Li, X., Liang, Y., Shi, Z. and Song, Z. (2025a). Bypassing the exponential dependency: Looped transformers efficiently learn in-context by multi-step gradient descent. In International Conference on Artificial Intelligence and Statistics. PMLR. Chen, S., Sheen, H., Wang, T. and Yang, Z. (2024a). Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. In The Thirty Seventh Annual Conference on Learning Theory. PMLR. 89

Chen, S., Sheen, H., Wang, T. and Yang, Z. (2024b). Unveiling induction heads: Provable training dynamics and feature learning in transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Chen, X., Lu, M., Wu, B. and Zou, D. (2025b). Towards theoretical understanding of transformer test-time computing: Investigation on in-context linear regression. arXiv preprint arXiv:2508.07571 . Chen, X., Zhao, L. and Zou, D. (2024c). How transformers utilize multi-head attention in in-context learning? a case study on sparse linear regression. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models. Chen, X. and Zou, D. (2024). What can transformer learn with varying depth? case studies on sequence learning tasks. In Forty-first International Conference on Machine Learning. Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 . Frei, S. and Vardi, G. (2025). Trained transformer classifiers generalize and exhibit benign overfitting in-context. In The Thirteenth International Conference on Learning Representations. Gao, C., Cao, Y., Li, Z., He, Y., Wang, M., Liu, H., Klusowski, J. and Fan, J. (2024). Global convergence in training large-scale transformers. Advances in Neural Information Processing Systems 37 29213–29284. Garg, S., Tsipras, D., Liang, P. S. and Valiant, G. (2022). What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems 35 30583–30598. Gatmiry, K., Saunshi, N., Reddi, S. J., Jegelka, S. and Kumar, S. (2024). Can looped transformers learn to implement multi-step gradient descent for in-context learning? arXiv preprint arXiv:2410.08292 . Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A. and Goldstein, T. (2025). Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171 . Guo, T., Hu, W., Mei, S., Wang, H., Xiong, C., Savarese, S. and Bai, Y. (2023). How do transformers learn in-context beyond simple functions? a case study on learning with representations. arXiv preprint arXiv:2310.10616 . He, Y., Cao, Y., Chen, H.-Y., Wu, D., Fan, J. and Liu, H. (2025a). Learning spectral methods by transformers. arXiv preprint arXiv:2501.01312 . He, Y., Chen, H.-Y., Cao, Y., Fan, J. and Liu, H. (2025b). Transformers versus the em algorithm in multi-class clustering. arXiv preprint arXiv:2502.06007 .

Huang, J., Wang, Z. and Lee, J. D. (2025). Transformers learn to implement multi-step gradient descent with chain of thought. In The Thirteenth International Conference on Learning Representations. Huang, Y., Cheng, Y. and Liang, Y. (2024). In-context convergence of transformers. In Fortyfirst International Conference on Machine Learning. Ildiz, M. E., HUANG, Y., Li, Y., Rawat, A. S. and Oymak, S. (2024). From self-attention to markov models: Unveiling the dynamics of generative transformers. In Forty-first International Conference on Machine Learning. Janner, M., Li, Q. and Levine, S. (2021). Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems 34 1273–1286. Jelassi, S., Sander, M. and Li, Y. (2022). Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems 35 37822–37836. Ji, Z. and Telgarsky, M. (2019). The implicit bias of gradient descent on nonseparable data. In Conference on learning theory. PMLR. Ji, Z. and Telgarsky, M. (2021). Characterizing the implicit bias via a primal-dual analysis. In Algorithmic Learning Theory. PMLR. Kelley, C. T. (1995). Iterative methods for linear and nonlinear equations. SIAM. Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E. P., Sanketi, P. R., Vuong, Q. et al. (2025). Openvla: An open-source vision-language-action model. In 8th Annual Conference on Robot Learning. Kunstner, F., Chen, J., Lavington, J. W. and Schmidt, M. (2023). Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations. Li, B., Huang, W., Han, A., Zhou, Z., Suzuki, T., Zhu, J. and Chen, J. (2024a). On the optimization and generalization of two-layer transformers with sign gradient descent. arXiv preprint arXiv:2410.04870 . Li, H., Wang, M., Liu, S. and Chen, P.-Y. (2023). A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. In The Eleventh International Conference on Learning Representations. Li, T., Zhang, C., Chen, X., Cao, Y. and Zou, D. (2025). On the robustness of transformers against context hijacking for linear classification. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Li, Z., Cao, Y., Gao, C., He, Y., Liu, H., Klusowski, J., Fan, J. and Wang, M. (2024b). One-layer transformer provably learns one-nearest neighbor in context. Advances in Neural Information Processing Systems 37 82166–82204. Long, P. M. (1995). On the sample complexity of pac learning half-spaces against the uniform distribution. IEEE Transactions on Neural Networks 6 1556–1559. 91

Nacson, M. S., Lee, J., Gunasekar, S., Savarese, P. H. P., Srebro, N. and Soudry, D. (2019). Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR. Nichani, E., Damian, A. and Lee, J. D. (2024). How transformers learn causal structure with gradient descent. In Forty-first International Conference on Machine Learning. Pan, Y. and Li, Y. (2023). Toward understanding why adam converges faster than sgd for transformers. arXiv preprint arXiv:2306.00204 . Qian, Q. and Qian, X. (2019). The implicit bias of adagrad on separable data. Advances in Neural Information Processing Systems 32. Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J. and Hsieh, C.-J. (2021). Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems 34 13937–13949. Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T. et al. (2022). A generalist agent. arXiv preprint arXiv:2205.06175 . Rosenthal, H. P. (1970). On the subspaces of Lp (p > 2) spanned by sequences of independent random variables. Israel Journal of Mathematics 8 273–303. Shen, W., Zhou, R., Yang, J. and Shen, C. (2025). On the training convergence of transformers for in-context classification of gaussian mixtures. In Forty-second International Conference on Machine Learning. Shi, W. and Cao, Y. (2025). Towards understanding transformers in learning random walks. arXiv preprint arXiv:2511.23239 . Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S. and Srebro, N. (2018). The implicit bias of gradient descent on separable data. Journal of Machine Learning Research 19 1–57. Tarzanagh, D. A., Li, Y., Thrampoulidis, C. and Oymak, S. (2023a). Transformers as support vector machines. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning. Tarzanagh, D. A., Li, Y., Zhang, X. and Oymak, S. (2023b). Max-margin token selection in attention mechanism. Advances in Neural Information Processing Systems 36 48314–48362. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F. et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 . Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems 30. Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 .

Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A. and Vladymyrov, M. (2023). Transformers learn in-context by gradient descent. In International Conference on Machine Learning. PMLR. Wang, B., Meng, Q., Zhang, H., Sun, R., Chen, W., Ma, Z.-M. and Liu, T.-Y. (2022). Does momentum change the implicit regularization on separable data? Advances in Neural Information Processing Systems 35 26764–26776. Wang, M., Min, Z. and Wu, L. (2024a). Achieving margin maximization exponentially fast via progressive norm rescaling. In International Conference on Machine Learning. PMLR. Wang, Z., Nichani, E., Bietti, A., Damian, A., Hsu, D., Lee, J. D. and Wu, D. (2025). Learning compositional functions with transformers from easy-to-hard data. arXiv preprint arXiv:2505.23683 . Wang, Z., Wei, S., Hsu, D. and Lee, J. D. (2024b). Transformers provably learn sparse token selection while fully-connected nets cannot. In Forty-first International Conference on Machine Learning. Wu, J., Braverman, V. and Lee, J. D. (2023). Implicit bias of gradient descent for logistic regression at the edge of stability. Advances in Neural Information Processing Systems 36 74229– 74256. Yang, L., Lee, K., Nowak, R. D. and Papailiopoulos, D. (2024). Looped transformers are better at learning learning algorithms. In The Twelfth International Conference on Learning Representations. Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.-H., Tay, F. E., Feng, J. and Yan, S. (2021). Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision. Zhang, C., Meng, X. and Cao, Y. (2025a). Transformer learns optimal variable selection in group-sparse classification. In The Thirteenth International Conference on Learning Representations. Zhang, C., Zhao, Q., Gu, Q. and Cao, Y. (2026). Transformers trained via gradient descent can provably learn a class of teacher models. In The Fourteenth International Conference on Learning Representations. Zhang, C., Zou, D. and Cao, Y. (2024a). The implicit bias of adam on separable data. Advances in Neural Information Processing Systems 37 23988–24021. Zhang, C.-B., Zhong, Y. and Han, K. (2025b). Mr. detr: Instructive multi-route training for detection transformers. In Proceedings of the Computer Vision and Pattern Recognition Conference. Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S. and Sra, S. (2020). Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems 33 15383–15393.

Zhang, R., Frei, S. and Bartlett, P. L. (2024b). Trained transformers learn linear models in-context. Journal of Machine Learning Research 25 1–55. Zhang, R., Wu, J. and Bartlett, P. L. (2024c). In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization. arXiv preprint arXiv:2402.14951 . Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R. and Luo, Z. (2024d). Why transformers need adam: A hessian perspective. Advances in neural information processing systems 37 131786– 131823.

Record · ID 168317 · SHA-256 bf3a0390b61ae97e
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.