SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model Tyler LaBonte1

arXiv:2606.30444v1 [stat.ML] 29 Jun 2026

Vidya Muthukumar2,1

H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology 2 School of Electrical and Computer Engineering, Georgia Institute of Technology {tlabonte, vmuthukumar8}@gatech.edu

Abstract Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations. In this work, we provide the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss. We consider data drawn from the high-dimensional Boolean hypercube with a quadratic signal function (namely XOR) and a linear spurious correlation. We show that SGD learns the spurious feature first, and exponentially fast. Moreover, the optimization dynamics couple the spurious and signal features, with a stronger spurious component inhibiting signal feature learning. Our analysis reveals precise phase transitions in the learning dynamics. In the first phase, alignment between the signs of the spurious feature and second-layer weight drives rapid growth of the spurious feature. In the second phase, large majority group margin slows learning and the signal feature remains suppressed. When the spurious correlation is maximally strong, we show theoretically that the spurious feature dominates even at the sample complexity threshold where XOR would be learned in isolation (i.e., if the spurious feature was absent). In contrast, when the correlation strength is constant, we provide preliminary empirical evidence that the model can eventually learn the XOR signal, although the spurious feature is not forgotten.

Introduction

Classification tasks in machine learning are sensitive to spurious correlations: facile patterns which are predictive of the class label in the training dataset but irrelevant to the target function (Sagawa et al., 2020a). These “shortcuts” can be as simple as using the background of an image to predict its content (Beery et al., 2018; Xiao et al., 2021), and have been observed in applications ranging from medicine (Zech et al., 2018) to justice (Chouldechova, 2016) and facial recognition (Z. Liu et al., 2015). Models which over-rely on these shortcuts can be accurate on average, but perform no better than random guessing on data where the spurious correlation is absent or opposite (Shah et al., 2020). Understanding how neural networks learn features is increasingly viewed as essential to explaining their reliance on spurious correlations (Izmailov et al., 2022). In particular, it is unclear how quickly signal and spurious features are learned, the impact of their relative complexity and strength of the spurious correlation, and how these features are represented in parameter space. Insights would have significant implications for the design of debiasing algorithms. Among the most popular such methods, Just Train Twice (E. Z. Liu et al., 2021) is predicated on faster learning of spurious features than signal features, while Deep Feature Reweighting (Kirichenko et al., 2023) assumes the model learns both signal and spurious features even if it primarily utilizes the latter — these are highly nontrivial conditions on the learned representations which currently lack theoretical justification. Despite recent advances in linearized settings or with modified training algorithms (Bombari and Mondelli, 2024; Hermann et al., 2024; Yang et al., 2024), we do not understand the mechanism underlying spurious feature learning in nonlinear models, even for basic target functions. In this paper, we theoretically characterize the feature learning process of stochastic gradient descent (SGD) on a two-layer ReLU neural network where the ground-truth (signal) model is the Boolean exclusive-OR (XOR) problem, but the data contains a linear spurious correlation. As the spurious feature is quantitatively 1

“simpler” than the quadratic XOR signal, our setting enables rigorous study of how neural networks exploit low-complexity shortcuts (G. Qiu et al., 2024). Compared to learning the XOR in isolation (Glasgow, 2024), we find that the signal and spurious features exhibit nonlinear interdependencies which complicate the learning dynamics — we show that the spurious feature competes with, and eventually suppresses, the signal feature. Our main theoretical result (Theorem 3.2) states that the spurious feature inevitably dominates when the spurious correlation is maximally strong. Our theory identifies several interesting phase transitions in the learning dynamics, illustrated in Figure 1. In more detail, our main contributions include the below: • We provide the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss (i.e., no model linearization or layer-wise training). • If the spurious correlation is maximally strong, we show theoretically that the spurious feature suppresses signal learning, inducing shortcut reliance even at the sample complexity threshold for learning XOR in isolation. Our analysis reveals precise phase transitions in the learning dynamics which align with simulations. • If the correlation strength is constant, we provide empirical evidence that the model eventually learns the XOR signal. Moreover, the spurious feature is not forgotten, and the network decomposes into disjoint signal and spurious subnetworks. Novelty of techniques. Our proofs build on the XOR feature learning analysis done by Glasgow, 2024 without a spurious correlation. Our framework similarly involves two phases, though our phase transitions are different and governed by the spurious feature. Despite sharing the basic setting of Glasgow, 2024 and some technical tools (e.g., Theorem F.9), our analysis diverges almost immediately due to nonlinear interactions between the signal and spurious features that fundamentally alter the learning dynamics. The starkest difference is the exponential suppression of the signal in Phase II, which necessitates a disparate approach focused on the data margins. Additionally, the spurious feature grows so rapidly that a useful Taylor approximation to the population gradients also used by Glasgow, 2024 becomes vacuous after O(log log(d)η −1 ) iterations rather than the O(log(d)η −1 ) of Glasgow, 2024, where η is the learning rate. This is a significant compression since the full training run lasts O(log(d)η −1 ) iterations, and it induces substantial differences in Phase I analysis. The signal-spurious interdependence also introduces new technical challenges, the greatest being the analysis of the orthogonal component w⊥ in Phase I. We elaborate on these challenges and our approach in Section 4. Notation. We provide a full table of notation in Section A, and describe here only what is necessary to read the main paper. We use uppercase bold symbols to denote matrices (e.g., X), lowercase bold symbols to denote vectors (e.g., x), and italicized symbols to denote scalars (e.g., x). We write ek for the k-th standard basis vector and ∥ · ∥ for the vector ℓ2 -norm. Let Px and Ex denote a probability and expectation with respect to a random vector x, respectively. Let Unif(Sd−1 (θ)) denote the uniform distribution on the ℓ2 -sphere in Rd of radius θ. We define the following asymptotic notation with respect to growing data dimension d: x ≪ y ⇐⇒ x = o(y), x ≲ y ⇐⇒ x = O(y), x ≫ y ⇐⇒ x = ω(y), x ≳ y ⇐⇒ x = Ω(y), and x ≍ y ⇐⇒ x = Θ(y). Finally, we write x = (1 ± o(1)) · y as shorthand for x ∈ [(1 − o(1)) · y, (1 + o(1)) · y].

1.1

Related Work

Here we provide a brief summary of related work across three axes.

Spurious correlations. The proclivity of neural networks to over-reliance on spurious features has been widely observed (Geirhos et al., 2020; Singla and Feizi, 2022). These features often manifest as simple “shortcuts” — including image backgrounds (Beery et al., 2018; Xiao et al., 2021) and secondary objects (Rosenfeld et al., 2018; Shetty et al., 2019) in computer vision, and syntactical or statistical heuristics in NLP (Gururangan et al., 2018; McCoy et al., 2019; Niven and Kao, 2019). In applications, such shortcuts are known to exacerbate biases (Hovy and Søgaard, 2015; Blodgett et al., 2016; Tatman, 2017; Hashimoto et al., 2018) and cause failure in high-stakes scenarios (Z. Liu et al., 2015; Chouldechova, 2016; Zech et al., 2018; Oakden-Rayner et al., 2019). Substantial research in group robustness and out-of-distribution generalization has investigated algorithms to mitigate these issues (Arjovsky et al., 2019; Sagawa et al., 2020a; Nam et al., 2020; E. Z. Liu et al., 2021; Idrissi et al., 2022; Pagliardini et al., 2023; Kirichenko et al., 2023; S. Qiu et al., 2023; LaBonte et al., 2023; Vasudeva et al., 2024; Noohdani et al., 2024; Tifrea et al., 2024). These methods are often predicated on tenuous assumptions about the feature learning process, e.g., that the spurious features are learned faster (E. Z. Liu et al., 2021) or have simpler representations (Vasudeva et al., 2024), or that the signal and spurious features are jointly learned but improperly weighted in the last layer (Kirichenko et al., 2023; S. Qiu et al., 2023; LaBonte et al., 2023). Theoretical analysis such as ours would (in the long term) clarify these assumptions and possibly be prescriptive of novel methodology. Feature learning theory without spurious correlations. A profound advantage of neural networks over classical kernel methods (including the neural tangent kernel (NTK) (Jacot et al., 2018)) is their ability to learn features from data (Karp et al., 2021; Telgarsky, 2023; Vyas et al., 2023; Radhakrishnan et al., 2024). Showing how feature learning reduces the sample complexity of learning implicitly low-dimensional functions is an active area of research (Mei et al., 2018; Y. Li et al., 2020; Abbe et al., 2021; Bietti et al., 2022; Abbe et al., 2022; Damian et al., 2022; Tan and Vershynin, 2023; Mousavi-Hosseini et al., 2023; Wu et al., 2023; Abbe et al., 2023; J. D. Lee et al., 2024). Many such analyses are limited by their modification of the training algorithm, e.g., via layer-wise training or gradient clipping. Our work is particularly inspired by Glasgow, 2024, which showed that two-layer ReLU neural networks trained by standard SGD (with no such modifications) can learn the Boolean XOR in d dimensions with d · polylog(d) sample complexity. Ultimately, we show that this sample complexity is insufficient to learn the XOR in the presence of the linear spurious correlation. Spurious feature learning. Theoretical insight into spurious correlations is useful even in linear models (Sagawa et al., 2020b; Nagarajan et al., 2021; Ye et al., 2023; Puli et al., 2023), but recent work suggests that feature learning underlies the design of robustness algorithms (Izmailov et al., 2022). Several works have studied shortcut learning in the random features and NTK regimes (Bombari and Mondelli, 2024; Hermann et al., 2024; Roy et al., 2025). Most closely related is Yang et al., 2024, which showed that spurious features are learned early in SGD training and characterized separability of majority and minority groups; their analysis uses square loss and a linearization similar to the NTK (Hu et al., 2020). Another related work is G. Qiu et al., 2024, which characterized the gradients with and without spurious correlations (under layer-wise training), but did not provide an end-to-end analysis of test error. In contrast, we directly analyze nonlinear neural networks trained by online minibatch SGD and explicitly characterize the majority/minority group error. We contextualize our results with the empirical findings of G. Qiu et al., 2024 in Section 6.

Setting

We now describe our Boolean XOR setting with a linear spurious correlation, as well as our two-layer neural network model and training procedure.

2.1

Data

Define µ1 := e1 − e2 and µ2 := e1 + e2 . Let λ ∈ (0, 12 ) denote the strength of the spurious correlation, where lower λ indicates stronger correlation. We model the data distribution Pd (λ) by:1  µ1 + e3 + ξ w.p. 1/4 − λ/4 µ2 − e3 + ξ w.p. 1/4 − λ/4    µ1 − e3 + ξ w.p. λ/4 µ2 + e3 + ξ w.p. λ/4 x= , (1) −µ1 + e3 + ξ w.p. 1/4 − λ/4 −µ2 − e3 + ξ w.p. 1/4 − λ/4    λ λ −µ1 − e3 + ξ w.p. /4 −µ2 + e3 + ξ w.p. /4 where ξ ∼ Unif(03 × {±1}d−3 ) so that ξ ⊥ {µ1 , µ2 , e3 }. For the remainder of the paper, we write Px := Px∼Pd (λ) and Ex := Ex∼Pd (λ) as shorthand. We will also write x := z + s + ξ as shorthand, where z := x1 e1 + x2 e2 and s := x3 e3 and the distribution of both z and s can be derived from Equation (1) above. Note that for λ = 12 we would have x ∼ Unif({±1}d ), recovering the data distribution of Glasgow, 2024. The target function is the XOR formula on the first two dimensions, i.e., y(x) := y(z) := −x1 x2 with no label noise. We denote the majority group by Xmaj := {x ∈ {±1}d : y(x) = x3 } and the minority group by Xmin := {x ∈ {±1}d : y(x) = −x3 }. It can be verified directly from Equation (1) that Px (x ∈ Xmaj ) = 1 − λ while Px (x ∈ Xmin ) = λ. We denote the accuracy of a real-valued predictor f : {±1}d → R on data uniformly drawn from a set X ⊆ {±1}d by AccX (f ) := sgn(f (x)) = y(x) . P x∼Unif(X )

Note that the fully spurious predictor f sp (x) = x3 has AccXmaj (f sp ) = 1 and AccXmin (f sp ) = 0.

2.2

Model and Training

We simultaneously train both layers of a two-layer ReLU neural network with p neurons {(aj , wj )}pj=1 . For initialization parameter θ ∈ R, we initialize each neuron i.i.d. with wj ∼ Unif(Sd−1 (θ)) so that ∥wj ∥ = θ and aj = rj θ where rj ∼ Unif({±1}). We represent the prediction as an expectation over the empirical distribution ρ of the neurons, i.e., p

fρ (x) :=

[aσ(w⊤ x)] =

(a,w)∼ρ

1X aj σ(wj⊤ x), p j=1

where σ(α) := max(0, α) is the ReLU. We often drop the j and simply refer to a neuron as (a, w). We train via online minibatch SGD with learning rate η > 0 on the logistic loss. Let γ(x) := y(x)fρ (x) denote the (unnormalized) margin of datum x and ψ(u) := 1/(1 + e−u ) denote the sigmoid. Then, we write the logistic loss by the composite notation ℓρ (x) := h(γ(x)) where h(γ) := −2 log(ψ(γ)). We define the population loss as Lρ := Ex [ℓρ (x)] and the empirical loss on the t-th minibatch of size m, denoted ∂L b ρ(t) := 1 P M (t) ∼ Pdm (λ), as L x∈M (t) ℓρ(t) (x). We write ∇w L := p · ∂w for the p-scaled gradient of L with m 2 respect to w and ∂u L := p · ∂L ∂u for the p-scaled partial derivative of L with respect to u ∈ (a, wi ). We denote the minibatch SGD update by b ρ(t) a(t+1) = a(t) − η∂a(t) L

b ρ(t) . w(t+1) = w(t) − η∇w(t) L

1 For simplicity, we detail our setting for an axis-aligned Boolean hypercube where the first three coordinates comprise the ground truth and spurious correlation. Nevertheless, all aspects of our analysis are rotationally invariant, and thus our results hold for Boolean hypercubes in any basis. 2 We scale by p following Glasgow, 2024 to match the conventional mean-field scaling and avoid an excessive amount of 1 p factors.

(a) Phase Ia

(b) Phase Ib

Figure 1: Phase transitions in spurious feature learning. We display the results of a training run with dimension d = 100, spurious correlation strength λ = 0.1, learning rate η = 0.05, width p = 10, initialization scale θ = 0.01, and batch size m = 5000. For each of the p = 10 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3). Phase Ia is a very short initial phase which induces alignment between sgn(a) and sgn(wsp ); we have omitted wsig and wopp to emphasize how some wsp lines “bounce off” zero. In Phase Ib, the spurious feature wsp grows exponentially fast and dominates all other features. Finally, in Phase II, the growth of wsp slows in a sigmoidal manner, yet continues monotonically increasing, while the signal feature wsig is greatly suppressed.

2.3

Feature Learning Analysis

For a neuron (a, w), let us write w := w1:2 + wsp + w⊥ and w1:2 := wsig + wopp , where ( ( 1 1 µ1 µ⊤ µ2 µ⊤ 1w a≥0 2w a≥0 2 wsig := 1 wopp := 21 wsp := w3 e3 . ⊤ ⊤ µ µ w a < 0 µ µ 2 2 2 2 1 1w a<0 We will often write wsp := w3 as shorthand for the scalar value of wsp . Note that wsig and wopp are defined with respect to sgn(a); our analysis shows sgn(a) does not change. We also define the “positive” neurons by S + := {(a, w) : sgn(a) = 1} and the “negative” neurons by S − := {(a, w) : sgn(a) = −1}, where importantly µ1 ∥ wsig for neurons in S + while µ2 ∥ wsig for neurons in S − . Intuitively, wsig is the feature in the “signal” direction — towards the correct classification of the XOR label — while wopp is the feature in the “opposing”, or negative, direction. Likewise, wsp denotes the “spurious” feature, while w⊥ comprises “orthogonal” features. The majority of our technical analysis will be towards providing a high-probability quantification of the growth rates of ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ for each neuron (where probability is over all the minibatched data and the random initialization). A model which isolates the ground-truth XOR signal will have w = ±wsig for all neurons (a, w). For example, if w = wsig with a ≥ 0 then w = 12 µ1 µ⊤ 1 w, so w = αµ1 for α ∈ R. In particular, the smallest optimal (normalized) neural network classifier has four neurons as follows: 1 ⊤ ⊤ ⊤ f ⋆ (x) = σ(µ⊤ 1 x) + σ(−µ1 x) − σ(µ2 x) − σ(−µ2 x) . 2

Main Result

In this section, we introduce our main theoretical result. It states that the spurious feature inevitably dominates even at the sample complexity threshold for learning XOR in isolation, i.e., O(d · polylog(d)) samples (Glasgow, 2024). We will need the following scalings on our model parameters. Assumption 3.1. We require the following for large enough constant C > 0: the learning rate log(d)d−C ≪ η ≪ log−3 (d), the width log5 (d) ≪ p ≪ dC , the initialization scale d−C/2 ≪ θ ≪ log−5C (d), the batch size m ≫ d log6 (d)θ−2 , and the spurious correlation strength λ ≪ log−1 (d). 5

The following theorem is our main result. Theorem 3.2. There exists a large enough constant C > 0 such that the following holds. If Theorem 3.1 is satisfied, then with probability at least 1 − d−C , upon ( log(d)(log log(d))−1 η −1 if θ ≍ polylog−1 (d) T ≍ log(d)η −1 if θ ≍ poly−1 (d) iterations of online minibatch SGD under the ℓρ loss, we have AccXmaj (fρ(T ) ) ≥ 1 − d−C

AccXmin (fρ(T ) ) ≤ d−C .

Theorem 3.2 implies that the majority group accuracy tends to 1 while the minority group accuracy tends to 0 as d → ∞, and it is proved at the end of Section E.2. Remark 3.3. Recall that since we are running online minibatch SGD, the sample complexity is m · T for batch size m and T iterations. Theorem 3.2 holds in two regimes demarcated by the initialization scale θ. If θ ≍ polylog−1 (d), we show that the spurious feature dominates at the sample complexity for learning XOR, up to log log factors, with standard d · polylog(d) batch size. On the other hand, if θ ≍ poly−1 (d) we precisely match the O(log(d)η −1 ) iterations of Glasgow, 2024, though we require a large poly(d) batch size.3 Remark 3.4. Our theoretical analysis requires the strength of the spurious correlation to decay sufficiently fast, i.e., λ ≪ log−1 (d). This corresponds to an “extreme” correlation case where the proportion of minority group data decays to zero. This setting is well-studied in both the empirical (Y. Lee et al., 2023; Pagliardini et al., 2023) and theoretical (Lai and Muthukumar, 2024; Park et al., 2026) literature, and it is closely related to out-of-domain generalization (Koh et al., 2021). Notably, the condition λ ≪ log−1 (d) is only required for our Phase II analysis, while our Phase I analysis holds with constant correlation strength. Our analysis involves a precise characterization of the feature learning dynamics in three distinct phases, illustrated in Figure 1. This characterization may be of independent interest. We briefly summarize each phase here and provide details in Section 4. • Phase Ia (Figure 1a) is a very short initial phase lasting only TIa ≍ log1/2 (d)d−1/2 η −1 iterations. By the end of Phase Ia, we have sign alignment between the spurious features and second-layer weights, i.e., sgn(a) = sgn(wsp ) for all neurons (a, w). • Phase Ib (Figure 1b) lasts TIb ≍ log log(d)η −1 iterations, wherein sign alignment implies a geometric lower bound on spurious feature growth. By the end of Phase Ib, the margins on majority and minority group data concentrate in the following (informally stated) way: with high probability over Unif(Xmaj ), 2 a datum xmaj has positive margin proportional to wsp ; similarly, over Unif(Xmin ), a datum xmin has 2 negative margin proportional to −wsp . On this event, we already have sgn(fρ (x)) = x3 , i.e., total dependence on the spurious correlation. 1 • Phase II (Figure 1c) lasts TII ≍ η1 loglog(d) iterations, i.e., most of the training period. log(d) + log θ Two key phenomena occur during this phase. First, large majority group margin leads to a sigmoidal slowdown of the growth of the spurious feature, while the condition λ ≪ log−1 (d) ensures that it remains monotonically increasing. Second, the spurious feature becomes massive enough to exponentially suppress the growth of ∥wsig ∥, inhibiting acquisition of the signal feature.

Proof Sketch

We now sketch the proof of Theorem 3.2, treating Phase I and Phase II separately. In this section, we let C > 0 denote a sufficiently large constant which does not vary from line to line. 3 This is a strong result from a sample complexity perspective, as one observes poly(d) more data than Glasgow, 2024, but the XOR signal is still not learned. Our perspective is that the small initialization scale slows down XOR feature learning even more.

4.1

Phase I Proof Sketch

In Phase I, the neural network is small, and thus the loss ℓρ (x) is well-approximated by a first-order Taylor expansion about fρ = 0, i.e., ℓ0 (x) := −2 log( 12 ) − y(x)fρ (x). Let us define the L0 population loss by L0 := Ex [ℓ0 (x)]. Importantly, this first-order approximation is only an analysis technique, and does not represent a linearization of the model. That is, we still train a nonlinear model with the standard ℓρ loss, but we bound feature norms by their growth under L0 and the deviation between the L0 and Lρ gradients. The goal of Phase I is to verify the conditions of Theorem D.1 for all neurons (a, w): that sgn(a) = sgn(wsp ) and the spurious feature dominates in the sense that C1 ∥wsp ∥ ≥ ∥wsig ∥ + ∥wopp ∥ + ∥w⊥ ∥ log1/2 (d). Once both conditions hold, each active neuron contributes a term matching sgn(x3 ) to fρ (x) such that sgn(fρ (x)) = x3 with high probability over x ∼ Pd (λ) — that is, the network predicts entirely via the spurious feature on typical test data. Why the spurious feature grows exponentially fast. Under the L0 approximation, Theorem B.2 ⊤ yields the population gradient −wsp ∇w L0 = awsp ( 12 + ε4w − λ), where εw is an asymmetry term that is exponentially small in ∥wsp ∥ (Theorem B.7) and hence negligible. Since λ < 12 , once sgn(a) = sgn(wsp ) occurs, Theorem D.10 gives the clean geometric recurrence (t+1) (t) (t) wsp − wsp ≍ ηsgn(a(0) )(|wsp | + θ),

where we used that sgn(a(t) ) = sgn(a(0) ) (i.e., the sign of a does not change) by Lemma D.7(ii). While (0) sgn(a(0) ) and sgn(wsp ) may be initially misaligned, we show that they align (and remain aligned) after at most TIa ≍ log1/2 (d)d−1/2 η −1 iterations. Once aligned, ∥wsp ∥ grows geometrically, reaching ∥wsp ∥ ≍ θ logC (d) after at most an additional TIb ≍ log log(d)η −1 iterations. Why the signal feature grows slowly, but is not yet suppressed. Under the L0 approximation, √ √ 2 ⊤ Theorem B.1 yields the population gradient −wsig ∇w L0 = 4 |a|∥wsig ∥ Pξ (|w⊤ ξ + w⊤ e3 | ≤ 2∥wsig ∥). Unraveling the probability term requires a delicate “perturbed Berry-Esseen” lemma analogous to Glasgow, 2024, Lemma B.4, which we present a modified version of in Theorem F.9. Ultimately, we get the recurrence that is stated in Theorem D.11, that is, (t+1)

∥wsig

(t)

− wsig ∥ ≲ η∥wsig ∥ + ηθ logC (d)d−1/2 .

This has the same geometric structure as the wsp recurrence — both grow with multiplicative rate 1 + Θ(η) — but the additive term for wsig is smaller by a factor of d−1/2 . There is no suppression of signal growth at this stage: both the spurious and signal features grow geometrically, with the spurious feature simply having a much larger additive “head start”. Summing this recursion over TI ≍ log log(d)η −1 iterations yields ∥wsig ∥ ≲ θ log2C (d)d−1/2 , which is only a polylogarithmic factor larger than the initialization upper bound. The opposing component wopp follows is similar (Theorem D.12). Why the orthogonal components stay controlled. It remains to track ∥w⊥ ∥ and ∥w⊥ ∥∞ . While this may seem like a formality, it is one of the more mathematically involved parts of our analysis. For ∥w⊥ ∥, the key observation is that the L0 gradient ∇w⊥ L0 is nearly parallel to w⊥ . The gradient takes the ⊤ ⊤ form Eξ [ξ · h(w⊥ ξ)] for a certain function h; for Gaussian inputs g, Stein’s lemma gives Eg [g · h(w⊥ g)] = ′ ⊤ Eg [h (w⊥ g)]·w⊥ which is exactly parallel to w⊥ . The deviation from Gaussianity is controlled by a Lindeberg exchange argument (Theorem F.11). For the component parallel to w⊥ , the gradient involves truncated moments of the form Eξ [|w⊤ ξ| · 1(|w⊤ ξ| ≥ k)]. Theorem F.10 approximates these by their Gaussian counterparts uniformly over k, with error proportional to ∥w⊥ ∥∞ . The contributions from the signal and (t+1) (t) opposing directions nearly cancel in the Gaussian approximation, giving ∥w⊥ − w⊥ ∥ ≲ ηθ log−2C (d) (Theorem D.13). Controlling ∥w⊥ ∥∞ requires a separate coordinate-wise analysis (Theorem D.14), which (t+1) (t) shows ∥w⊥ − w⊥ ∥∞ ≲ ηθ log3C (d)d−1/2 and keeps the Theorem F.10 approximation sharp throughout Phase I. 7

Controlling approximation errors and closing the induction. The L0 approximations above are b ρ − ∇w L0 ∥ remains negligible. While Glasgow, 2024 handle this using a leave-one-out useful only if ∥∇w L symmetrization, the presence of the spurious correlation requires us to take a more complex leave-two-out symmetrization instead (Theorem C.3). Moreover, the new approximation error for wsp is much larger, but taking the initialization scale θ ≪ log−5C (d) turns out to be sufficient to analyze the Phase I dynamics b ρ − ∇w Lρ ∥ are then controlled by Hoeffding’s inequality (Theorem D.8). The minibatch fluctuations ∥∇w L 7C (Lemma F.3(ii)) with batch size m ≫ d log (d). All five component bounds — on ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, ∥w⊥ ∥, and ∥w⊥ ∥∞ — are maintained jointly on a single inductive hypothesis (Theorem D.4) over TI = TIa +TIb iterations.

4.2

Phase II Proof Sketch

By the end of Phase I, we have sgn(fρ (x)) = x3 with high probability — meaning that the network predicts entirely via the spurious feature. The goal of Phase II is to show this remains true for a further O(log(d)η −1 ) iterations, matching the sample complexity needed to learn the XOR signal in isolation (Glasgow, 2024). This is nontrivial because spurious feature growth slows down in Phase II, and we must rule out the possibility that the signal feature eventually overtakes it. The key technical challenge of Phase II is that the neural network is too large for the L0 approximation of Phase I to be valid, so we must bound the original Lρ gradient by characterizing the margin on typical data. Recall that ψ(x) := 1/(1 + e−x ) denotes the sigmoid and γ(x) := y(x)fρ (x) denotes the margin. Then, we have the following decomposition, which enables us to analyze the Lρ gradient: h i ∇w Lρ = p E ℓ(1) ρ (x)∇w fρ (x) = 2p E ψ(−γ(x)) · −y(x)∇w fρ (x) = 2 E ψ(−γ(x)) · ∇w pℓ0 (x) . x

Why the margins are “equal and opposite”. The central structural fact of Phase II is that the 2 classification margins concentrate tightly about certain average values of ±wsp , where the sign is positive for the majority group and negative for the minority group (see Figure 3a). Define γ+ :=

1 p

(wsp )2

γ− :=

(a,w)∈S +

1 p

(wsp )2 .

(a,w)∈S −

Then, e.g., in the case where y(x) = 1, Theorem E.8 states that γ(xmaj ) ≈ γ+ and γ(xmin ) ≈ −γ− with high probability over xmaj ∼ Unif(Xmaj ) and xmin ∼ Unif(Xmin ). This holds because the active ReLUs are the same for all majority group and minority group points, respectively. The key object we track in Phase II is the average of the positive and negative margins, defined by γ̄ := 21 (γ+ + γ− ). Why the spurious feature continues to grow, despite slowing down. With this margin behavior in hand, we can compute ∇wsp Lρ directly using the decomposition above (Theorem C.5). After controlling error terms in Theorem E.9, we obtain the recurrence (t+1) (t) (t) wsp − wsp ≍ ηwsp 1 − λ − ψ(γ̄ (t) ) . (2) This recurrence induces a sigmoidal structure: when γ̄ ≪ 1, we have ψ(γ̄) ≈ 21 , so ∥wsp ∥ continues to increase exponentially. Once γ̄ ≫ 1, we have 1 − ψ(γ̄) ≈ e−γ̄ , so growth decelerates dramatically as the margins 1 become large. We show in Theorem E.6 that Phase II lasts TII ≍ η1 loglog(d) log(d) + log( θ ) iterations until γ̄ ≍ log log(d). Crucially, monotone growth of γ̄ throughout Phase II requires e−γ̄ − λ ≥ 0, i.e., λ ≪ log−1 (d). This is the only place where we require the “extreme” correlation condition.

(a) No spurious feature

(b) λ = 0.1

(d) λ = 0.2

Figure 2: XOR signal can be learned for large enough λ. We display the results of training runs with dimension d = 100, learning rate η = 0.05, width p = 10, initialization scale θ = 0.01, and batch size m = 5000. For each of the p = 10 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3). We find that for large enough λ — corresponding to λ ≫ log−1 (d) according to our theory — the XOR signal can overtake the spurious feature. The network decomposes into disjoint signal and spurious subnetworks, with the size of the signal subnetwork increasing in λ. Why the signal feature is suppressed. The key difference from Phase I is that signal growth is no longer merely slow — it is actively suppressed by the growing spurious feature. Specifically, we show that 2 both terms in the signal gradient bound of Theorem C.4 become exponentially small in wsp : the first term via our margin-based analysis, and the second term via Hoeffding’s inequality. Combining the terms in Theorem E.10 yields (t) (t+1) (t) (t) (t) 2 ∥wsig − wsig ∥ ≲ ηe−γ̄ max (wsp ) ∥wsig ∥ + ηθ log−1 (d)d−1/2 . (a(t) ,w(t) )

As γ̄ grows monotonically throughout Phase II, the exponential suppression only strengthens over time, guaranteeing that ∥wsig ∥ grows by only a polylogarithmic factor (Lemma E.6(iii)). The same decay governs the ∥w⊥ ∥ and ∥w⊥ ∥∞ inductions (Theorem E.11 and Theorem E.12 respectively). Closing the induction. As in Phase I, all five component bounds are maintained jointly as an inductive hypothesis (Theorem E.3) across TII iterations. Minibatch concentration requires a larger batch size m ≫ d log6 (d)θ−2 to keep the component norms proportional to the initialization scale θ; note that this could be poly(d) if θ ≍ poly−1 (d). The Phase II induction concludes that sgn(fρ (x)) = x3 with high probability, such that AccXmaj (fρ ) ≥ 1 − d−C and AccXmin (fρ ) ≤ d−C .

The Constant λ Case: When can the XOR Signal be Learned?

A remaining question is what happens when the “extreme” correlation condition λ ≪ log−1 (d) is not satisfied, e.g., the spurious correlation strength λ is a constant. In particular, Phase II would not satisfy the requisite properties for monotonic growth of ∥wsp ∥. Observe that the Phase II spurious feature recurrence (Equation (2)) has a fixed point ψ(γ̄) = 1 − λ. Since γ̄ initially grows monotonically, if λ were constant we would eventually reach a value which overshoots this fixed point. Then, the right-hand side of the recurrence would become (t+1) (t) (t) negative, i.e., wsp − wsp ≈ −ηwsp , such that ∥wsp ∥ would begin to decrease geometrically! If it decreased enough, the exponential suppression of the signal feature would weaken, which may provide an opportunity for the XOR signal to be learned later in training. In Figure 2, we validate this hypothesis through simulation. We contrast signal learning in the standard XOR model without a spurious feature (Glasgow, 2024) (Figure 2a) with our setting, varying the strength of the spurious correlation. We find that the XOR signal can be learned for large enough λ, as the nonmonotonic evolution of wsp reduces signal suppression, enabling several neurons to pick up wsig around step 500. Figure 2c shows that ∥wsp ∥ increases geometrically — in line with our theory — then decreases 9

(a) λ = 0.1

(b) λ = 0.15

Figure 3: “Equal and opposite” margins only hold for small λ. We display the results of training runs with dimension d = 100, learning rate η = 0.05, width p = 50, initialization scale θ = 0.01, and batch size m = 5000. We plot the average per-group margin in each minibatch. We find that for small enough λ — corresponding to λ ≪ log−1 (d) according to our theory — the margins concentrate about certain averages of 2 ±wsp , where the sign is positive for the majority group and negative for the minority group. For larger λ, the minority group margin exhibits improved behavior, eventually leading to perfect classification on both the majority and minority groups. geometrically. An important observation about Figure 2c and Figure 2d is that the neurons decompose into disjoint subnetworks which emphasize only one of the signal or spurious feature — that is, the large ∥wsig ∥ lines in Figure 2c correspond precisely to neurons with small ∥wsp ∥, and vice versa. Moreover, the spurious feature is not fully “forgotten” even for large λ. The dynamics of the constant correlation case have a substantial impact on margin behavior and worstgroup accuracy. In Figure 3, we plot the average margin over the majority and minority groups in each minibatch: while the minority group has highly negative margin at small values of λ (implying low minority group accuracy), the picture is more benign at large values of λ (implying high minority group accuracy). Figure 3 also suggests that a theoretical analysis of this case will require a different approach — margin concentration is the critical structural fact of our Phase II analysis, and it clearly does not hold for large λ. Nevertheless, the ideas in our Phase II analysis could be adapted in future work together with a more fine-grained characterization of the margin.

Discussion

In this paper, we presented the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss. Our results provide a foundation for rigorous understanding of spurious feature learning. Of particular note, our insights formalize and/or theoretically corroborate the four main findings of G. Qiu et al., 2024 as follows: • Easier spurious features lead to slower signal feature emergence: We formally prove this statement, with precise rates, for a linear spurious feature and quadratic signal feature. • Learning may not separate into “spurious” and “signal” phases: Our results imply that spurious and signal learning are entangled in Phase Ia (and in the constant λ case). • Spurious features are retained after the signal feature is learned : We show empirically that the spurious feature is preserved even when λ is large and constant. • Spurious and signal features are learned by disjoint subnetworks: We show empirically that the model decomposes into signal and spurious subnetworks in the constant λ case.

Overall, we believe our theory could eventually inform the design of robustness-advancing interventions. For example, the length of our Phase I constitutes a lower bound on the number of iterations necessary before the early-stopped checkpoint of Just Train Twice (E. Z. Liu et al., 2021) is saved — otherwise, the misclassifications may not correlate well with the minority group. Likewise, our observation that disjoint signal and spurious subnetworks emerge in the constant-correlation case justifies the assumption of Deep Feature Reweighting (Kirichenko et al., 2023), and suggests that it works by identifying and upweighting the spurious subnetwork. Finally, group/class-balancing, or throwing away majority group/class data, has also been observed to improve minority group accuracy (Idrissi et al., 2022; Chaudhuri et al., 2023; LaBonte et al., 2024). This mechanism increases the effective value of λ, which our Phase I theory predicts would lead to slower initial acquisition of the spurious feature.

Acknowledgements We thank Jacob Abernethy for compute assistance, and Surbhi Goel and Steve Mussmann for insightful conversations. VM gratefully acknowledges the support of the NSF (through award CCF-2239151 and award IIS-2212182), an Adobe Data Science Research Award, and an Amazon Research Award.

References Abbe, Emmanuel, Enric Boix-Adserà, Matthew Brennan, Guy Bresler, and Dheeraj Nagaraj (2021). “The staircase property: How hierarchical structure can guide deep learning”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Abbe, Emmanuel, Enric Boix-Adserà, and Theodor Misiakiewicz (2022). “The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks.” In: Conference on Learning Theory (COLT) (cit. on p. 3). — (2023). “SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics”. In: Conference on Learning Theory (COLT) (cit. on p. 3). Arjovsky, Martin, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz (2019). “Invariant Risk Minimization”. In: arXiv preprint 1907.02893 (cit. on p. 3). Beery, Sara, Grant van Horn, and Pietro Perona (2018). “Recognition in Terra Incognita”. In: European Conference on Computer Vision (ECCV) (cit. on pp. 1, 3). Bietti, Alberto, Joan Bruna, Clayton Sanford, and Min Jae Song (2022). “Learning single-index models with shallow neural networks”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Blodgett, Su Lin, Lisa Green, and Brendan O’Connor (2016). “Demographic Dialectal Variation in Social Media: A Case Study of African-American English”. In: Empirical Methods in Natural Language Processing (EMNLP) (cit. on p. 3). Bombari, Simone and Marco Mondelli (2024). “How Spurious Features are Memorized: Precise Analysis for Random and NTK Features”. In: International Conference on Machine Learning (ICML) (cit. on pp. 1, 3). Chaudhuri, Kamalika, Kartik Ahuja, Martin Arjovsky, and David Lopez-Paz (2023). “Why does throwing away data improve worst-group error?” In: International Conference on Machine Learning (ICML) (cit. on p. 11). Chouldechova, Alexandra (2016). “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments”. In: Conference on Fairness, Accountability, and Transparency in Machine Learning (FATML) (cit. on pp. 1, 3). Damian, Alex, Jason D. Lee, and Mahdi Soltanolkotabi (2022). “Neural networks can learn representations with gradient descent”. In: Conference on Learning Theory (COLT) (cit. on p. 3). Temme, N. M. (2026). NIST Digital Library of Mathematical Functions. Release 1.2.6 of 2026-03-15. Chapter 6 (cit. on p. 71). Geirhos, Robert, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann (2020). “Shortcut learning in deep neural networks”. In: Nature Machine Intelligence 2 (1), pp. 665–673 (cit. on p. 3). Glasgow, Margalit (2024). “SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem”. In: International Conference on Learning Representations (ICLR) (cit. on pp. 2–9, 21, 23, 30–32, 46, 58, 73, 78, 84, 87, 91). Gururangan, Suchin, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith (2018). “Annotation Artifacts in Natural Language Inference Data”. In: North American Association for Computational Linguistics (NAACL) (cit. on p. 3). Hashimoto, Tatsunori B., Megha Srivastava, Hongseok Namkoong, and Percy Liang (2018). “Fairness Without Demographics in Repeated Loss Minimization”. In: International Conference on Machine Learning (ICML) (cit. on p. 3). Hermann, Katherine L., Hossein Mobahi, Thomas Fel, and Michael C. Mozer (2024). “On the Foundations of Shortcut Learning”. In: International Conference on Learning Representations (ICLR) (cit. on pp. 1, 3). Hovy, Dirk and Anders Søgaard (2015). “Tagging Performance Correlates with Author Age”. In: Association for Computational Linguistics (ACL) (cit. on p. 3). Hu, Wei, Lechao Xiao, Ben Adlam, and Jeffrey Pennington (2020). “The Surprising Simplicity of the EarlyTime Learning Dynamics of Neural Networks”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). 12

Idrissi, Badr Youbi, Martín Arjovsky, Mohammad Pezeshki, and David Lopez-Paz (2022). “Simple data balancing achieves competitive worst-group-accuracy”. In: Conference on Causal Learning and Reasoning (CLeaR) (cit. on pp. 3, 11). Izmailov, Pavel, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson (2022). “On Feature Learning in the Presence of Spurious Correlations”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on pp. 1, 3). Jacot, Arthur, Franck Gabriel, and Clément Hongler (2018). “Neural tangent kernel: convergence and generalization in neural networks”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Karp, Stefani, Ezra Winston, Yuanzhi Li, and Aarti Singh (2021). “Local Signal Adaptivity: Provable Feature Learning in Neural Networks Beyond Kernels”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Kim, Nayeong, Juwon Kang, Sungsoo Ahn, Jungseul Ok, and Suha Kwak (2024). “Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization”. In: International Conference on Machine Learning (ICML) (cit. on p. 91). Kirichenko, Polina, Pavel Izmailov, and Andrew Gordon Wilson (2023). “Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations”. In: International Conference on Learning Representations (ICLR) (cit. on pp. 1, 3, 11). Koh, Pang Wei, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang (2021). “WILDS: A Benchmark of in-the-Wild Distribution Shifts”. In: International Conference on Machine Learning (ICML) (cit. on p. 6). LaBonte, Tyler, John C. Hill, Xinchen Zhang, Vidya Muthukumar, and Abhishek Kumar (2024). “The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 11). LaBonte, Tyler, Vidya Muthukumar, and Abhishek Kumar (2023). “Towards Last-layer Retraining for Group Robustness with Fewer Annotations”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Lai, Kuo-Wei and Vidya Muthukumar (2024). “Sharp Analysis of Out-of-Distribution Error for “ImportanceWeighted” Estimators in the Overparameterized Regime”. In: International Symposium on Information Theory (ISIT) (cit. on p. 6). Laurent, B. and P. Massart (2000). “Adaptive Estimation of a Quadratic Functional by Model Selection”. In: Annals of Statistics 28.5, pp. 1302–1338 (cit. on p. 73). Lee, Jason D., Kazusato Oko, Taiji Suzuki, and Denny Wu (2024). “Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Lee, Yoonho, Huaxiu Yao, and Chelsea Finn (2023). “Diversify and Disambiguate: Learning From Underspecified Data”. In: International Conference on Learning Representations (ICLR) (cit. on p. 6). Li, Yuanzhi, Tengyu Ma, and Hongyang R. Zhang (2020). “Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK”. In: Conference on Learning Theory (COLT) (cit. on p. 3). Li, Zhiheng, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim (2023). “A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others”. In: Conference on Computer Vision and Pattern Recognition (CVPR) (cit. on p. 91). Liu, Evan Zheran, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn (2021). “Just Train Twice: Improving Group Robustness without Training Group Information”. In: International Conference on Machine Learning (ICML) (cit. on pp. 1, 3, 11). Liu, Ziwei, Ping Luo, Xiaogang Wang, and Xiaoou Tang (2015). “Deep Learning Face Attributes in the Wild”. In: International Conference on Computer Vision (ICCV) (cit. on pp. 1, 3).

McCoy, Tom, Ellie Pavlick, and Tal Linzen (2019). “Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference”. In: Association for Computational Linguistics (ACL) (cit. on p. 3). Mei, Song, Andrea Montanari, and Phan-Minh Nguyen (2018). “A mean field view of the landscape of two-layer neural networks”. In: Proceedings of the National Academy of Sciences (PNAS) 115.33, E7665–E7671 (cit. on p. 3). Mousavi-Hosseini, Alireza, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A. Erdogdu (2023). “Neural Networks Efficiently Learn Low-Dimensional Representations with SGD”. In: International Conference on Learning Representations (ICLR) (cit. on p. 3). Nagarajan, Vaishnavh, Anders Andreassen, and Behnam Neyshabur (2021). “Understanding the failure modes of out-of-distribution generalization”. In: International Conference on Learning Representations (ICLR) (cit. on p. 3). Nam, Junhyun, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin (2020). “Learning from Failure: Training Debiased Classifier from Biased Classifier”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Nichani, Eshaan, Alex Damian, and Jason Lee (2023). “Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 91). Niven, Timothy and Hung-Yu Kao (2019). “Probing Neural Network Comprehension of Natural Language Arguments”. In: Association for Computational Linguistics (ACL) (cit. on p. 3). Noohdani, Fahimeh Hosseini, Parsa Hosseini, Aryan Yazdan Parast, Hamidreza Yaghoubi Araghi, and Mahdieh Soleymani Baghshah (2024). “Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation”. In: Conference on Computer Vision and Pattern Recognition (CVPR) (cit. on p. 3). Oakden-Rayner, Luke, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré (2019). “Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging”. In: Conference on Neural Information Processing Systems (NeurIPS) Workshop on Machine Learning for Health (cit. on p. 3). Pagliardini, Matteo, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy (2023). “Agree to Disagree: Diversity through Disagreement for Better Transferability”. In: International Conference on Learning Representations (ICLR) (cit. on pp. 3, 6). Park, Subeen, Joowang Kim, Hakyung Lee, Sunjae Yoo, and Kyungwoo Song (2026). “Spurious CorrelationAware Embedding Regularization for Worst-Group Robustness”. In: International Conference on Learning Representations (ICLR) (cit. on p. 6). Puli, Aahlad, Lily Zhang, Yoav Wald, and Rajesh Ranganath (2023). “Don’t blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 3). Qiu, GuanWen, Da Kuang, and Surbhi Goel (2024). “Complexity Matters: Feature Learning in the Presence of Spurious Correlations”. In: International Conference on Machine Learning (ICML) (cit. on pp. 2, 3, 10, 91). Qiu, Shikai, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson (2023). “Simple and Fast Group Robustness by Automatic Feature Reweighting”. In: International Conference on Machine Learning (ICML) (cit. on p. 3). Radhakrishnan, Adityanarayanan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin (2024). “Mechanism for feature learning in neural networks and backpropagation-free machine learning models”. In: Science 383.6690, pp. 1461–1467 (cit. on p. 3). Rosenfeld, Amir, Richard Zemel, and John K. Tsotsos (2018). “The Elephant in the Room”. In: arXiv preprint 1808.03305 (cit. on p. 3). Roy, Saptarshi, Subha Maity, Songkai Xue, Mikhail Yurochkin, and Yuekai Sun (2025). “How does overparametrization affect performance on minority groups?” In: Transactions on Machine Learning Research (TMLR) (cit. on p. 3).

Sagawa, Shiori, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang (2020a). “Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization”. In: International Conference on Learning Representations (ICLR) (cit. on pp. 1, 3). Sagawa, Shiori, Aditi Raghunathan, Pang Wei Koh, and Percy Liang (2020b). “An investigation of why overparameterization exacerbates spurious correlations”. In: International Conference on Machine Learning (ICML) (cit. on p. 3). Shah, Harshay, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli (2020). “The Pitfalls of Simplicity Bias in Neural Networks”. In: Conference on Neural Information Processing Systems (NeurIPS) (cit. on p. 1). Shetty, Rakshith, Bernt Schiele, and Mario Fritz (2019). “Not Using the Car to See the Sidewalk: Quantifying and Controlling the Effects of Context in Classification and Segmentation”. In: Conference on Computer Vision and Pattern Recognition (CVPR) (cit. on p. 3). Singla, Sahil and Soheil Feizi (2022). “Salient ImageNet: How to discover spurious features in Deep Learning?” In: International Conference on Learning Representations (ICLR) (cit. on p. 3). Tan, Yan Shuo and Roman Vershynin (2023). “Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrieval”. In: Journal of Machine Learning Research (JMLR) 24 (48), pp. 1–47 (cit. on p. 3). Tao, Terence (2015). 275A, Notes 4: The central limit theorem (cit. on p. 81). Tatman, Rachael (2017). “Gender and Dialect Bias in YouTube’s Automatic Captions”. In: Association for Computational Linguistics (ACL) Workshop on Ethics in Natural Language Processing (cit. on p. 3). Telgarsky, Matus (2023). “Feature selection and low test error in shallow low-rotation ReLU networks”. In: International Conference on Learning Representations (ICLR) (cit. on p. 3). Tifrea, Alexandru, Preethi Lahoti, Ben Packer, Yoni Halpern, Ahmad Beirami, and Flavien Prost (2024). “FRAPPÉ: A Group Fairness Framework for Post-Processing Everything”. In: International Conference on Machine Learning (ICML) (cit. on p. 3). Vasudeva, Bhavya, Kameron Shahabi, and Vatsal Sharan (2024). “Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness”. In: Transactions on Machine Learning Research (TMLR) (cit. on p. 3). Vyas, Nikhil, Yamini Bansal, and Preetum Nakkiran (2023). “Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning”. In: Transactions on Machine Learning Research (TMLR) (cit. on p. 3). Wu, Jingfeng, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade (2023). “Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuron”. In: International Conference on Machine Learning (ICML) (cit. on p. 3). Xiao, Kai Yuanqing, Logan Engstrom, Andrew Ilyas, and Aleksander Mądry (2021). “Noise or Signal: The Role of Image Backgrounds in Object Recognition”. In: International Conference on Learning Representations (ICLR) (cit. on pp. 1, 3). Yang, Yu, Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman (2024). “Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias”. In: International Conference on Artificial Intelligence and Statistics (AISTATS) (cit. on pp. 1, 3). Ye, Haotian, James Zou, and Linjun Zhang (2023). “Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noise”. In: International Conference on Artificial Intelligence and Statistics (AISTATS) (cit. on p. 3). Zech, John R., Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann (2018). “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study”. In: PLoS Medicine 15 (11), e1002683 (cit. on pp. 1, 3).

Appendix Table of Contents A Notation

B L0 Analysis 21 B.1 ∇L0 Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 B.2 εw Asymmetry Term Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 C Lρ Analysis 30 C.1 ∇Lρ − ∇L0 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 C.2 ∇Lρ Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 D Phase I Induction 42 D.1 Technical Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 b ρ − ∇L0 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 D.2 Phase I ∇L D.3 Phase I Inductive Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 E Phase II Induction 58 E.1 Technical Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 E.2 Phase II Inductive Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 F Technical Lemmas 72 F.1 High-probability Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 F.2 From Booleans to Gaussians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 F.3 Simultaneous Training of Two-Layer ReLU Neural Networks . . . . . . . . . . . . . . . . . . . 84 F.4 Miscellaneous Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 G Additional Simulations

H Limitations, Broader Impacts, and LLM Usage

Notation

We summarize the notation used throughout this paper. Unless otherwise stated, all asymptotic notation is with respect to d → ∞.

General Mathematical Notation Symbol

Description

X, x, x

Uppercase bold: matrix; lowercase bold: vector; italic: scalar

k-th standard basis vector (dimension clear from context)

All-zeros vector (dimension clear from context)

∥·∥

Euclidean (ℓ2 ) norm of a vector

∥ · ∥∞

ℓ∞ norm of a vector

∥ · ∥p

ℓp norm of a vector for 1 ≤ p < ∞

Px , Ex

Probability and expectation with respect to random vector x

1(·)

Indicator function of an event

sgn(·)

Sign function

[t]

{1, 2, . . . , t} for t ∈ N

A∆B

Symmetric difference of sets: A∆B := (A \ B) ∪ (B \ A)

Projy (x)

y x Projection of x onto y: Projy (x) := ∥y∥ 2 y

⊤

Asymptotic notation (all with respect to d → ∞) x≪y

x = o(y)

x≲y

x = O(y)

x≫y

x = ω(y)

x≳y

x = Ω(y)

x≍y

x = Θ(y)

Probability distributions Unif({±1})

Univariate Rademacher distribution

Unif({±1} )

Multivariate Rademacher distribution

(θ))

Uniform distribution on the ℓ2 -sphere in Rd of radius θ

d−1

Unif(S

N (µ, Σ)

Multivariate Gaussian with mean µ and covariance Σ

N (µ, σ 2 )

Univariate Gaussian with mean µ and variance σ 2

Φ(·), ϕ(·)

Standard Gaussian CDF and PDF, respectively

Problem Setting Symbol

Description

Data dimension

µ1 = e1 − e2

Positive signal direction for class +1

Symbol

Description

µ2 = e1 + e2

Positive signal direction for class −1

λ ∈ (0, Λ], Λ < 21

Spurious correlation strength; smaller λ indicates stronger spurious correlation; λ → 12 not allowed Data distribution parameterized by λ (Equation (1))

Pd (λ) ξ ∼ Unif(03 × {±1}d−3 ) z := x1 e1 + x2 e2

Noise vector, independent of µ1 , µ2 , e3 Signal component of x

s := x3 e3

Spurious component of x

x := z + s + ξ y(x) := −x1 x2

Shorthand decomposition of a data point

Xmaj

Majority group: {x ∈ {±1}d : y(x) = x3 }

Xmin

Minority group: {x ∈ {±1}d : y(x) = −x3 }

f ⋆ (x)

Optimal (max-margin) neural network classifier

f (x) := x3

XOR target function (depends only on z)

Spurious predictor; achieves 1 − λ average accuracy and 0 worst-group accuracy

Neural Network and Training Symbol

Description

Number of neurons (network width)

(aj , wj )

j-th neuron: scalar output weight aj ∈ R, vector input weight wj ∈ Rd ; we often drop the j indexing Empirical distribution over neurons {(aj , wj )}pj=1 ; written ρ(t) at iteration t Pp Network output: E(a,w)∼ρ [a σ(w⊤ x)] = p1 j=1 aj σ(wj⊤ x)

ρ fρ (x) σ(α) := max(0, α)

ReLU activation function

SGD step size/learning rate

Minibatch size

r ∼ Unif({±1})

Rademacher variable used only for initializations

Initialization scale: ∥w(0) ∥ = θ and |a(0) | = θ

M(t) ∼ Pdm (λ) ψ(u) := 1/(1 + e−u )

t-th minibatch of size m

γ(x) := y(x) fρ (x)

Classification margin of datum x

(t) γ+ (t) γ−

Margin on the positive neurons at iteration t

γ̄

(t)

Sigmoid function

Margin on the negative neurons at iteration t (t)

(t)

Average margin: 12 (γ+ + γ− )

h(γ) := −2 log(ψ(γ)) ℓρ (x) := h(γ(x))

Logistic loss as a function of margin

(1) ℓρ (x) := h′ (γ(x))

First derivative of the per-sample loss

(2) ℓρ (x) := h′′ (γ(x))

Second derivative of the per-sample loss

Per-sample logistic loss

Symbol

Description

Lρ := Ex [ℓρ (x)] b ρ(t) L

Population logistic loss

∇w L

∂L p-scaled gradient of L with respect to w: ∇w L := p · ∂w

∂u L

p-scaled partial derivative of L with respect to scalar u ∈ {a, wi }: ∂u L := p · ∂L ∂u

Empirical loss on the t-th minibatch M(t)

Weight Decomposition Each neuron weight vector w is decomposed into four orthogonal components: w = wsig + wopp + wsp + w⊥ , where w1:2 := wsig + wopp . Symbol

Description

wsig

Core (signal) feature component, aligned with the XOR-correct classification direction; µ1 ∥ wsig for S + neurons and µ2 ∥ wsig for S − neurons Opposing feature component, aligned opposite to the signal direction

wopp wsp := w3 e3

Spurious feature component (vector)

wsp := w3

Spurious feature component (scalar); projection onto e3

w⊥

Orthogonal noise component; perpendicular to span{µ1 , µ2 , e3 }

w1:2 := wsig + wopp

Combined projection onto span{µ1 , µ2 } Positive neurons: S + := {(a, w) : sgn(a) = +1}

S−

Negative neurons: S − := {(a, w) : sgn(a) = −1}

smaj

Spurious direction aligned with the majority group: smaj = e3 if (a, w) ∈ S + , and smaj = −e3 if (a, w) ∈ S −

Analysis-Specific Notation Symbol

Description

C>0

Positive constant chosen large enough, which do not vary from line to line Other constants, which do not vary from line to line

c < 1, C ′ < C − 3 ℓ0 (x) L0 := Ex [ℓ0 (x)]

First-order Taylor approximation of ℓρ (x) about fρ = 0; ℓ0 (x) = −2 log( 12 ) − y(x)fρ (x) Population loss under the ℓ0 approximation

ℓ0 (x) = −1

(1)

Shorthand for the (constant) first derivative of ℓ0

εw

Asymmetry term in the spurious feature gradient (Theorem B.2); decays exponentially in ∥wsp ∥ Error tolerance on the margin

δ High-probability events (j)

Einit

The initialization event for the j-th neuron

Symbol Tp (j) Einit := j=1 Einit

Description The initialization event over all neurons (p clear from context)

(t) Ebatch

The concentration event for the t-th minibatch (t) t=1 Ebatch

Ebatch := Etrain := Einit ∩ Ebatch

The concentration event over all minibatches (T clear from context)

(j) Etest

The concentration event for the j-th neuron

Etest :=

(j) j=1 Etest

The train event over all neuron initializations and minibatches The test event over all neurons (p clear from context)

Training phases Total number of iterations in Phase I

TI = TIa + TIb TIa ≍ log

1/2

(d)d

−1/2 −1

TII ≍(log(d)(log log(d))

−1

Duration of Phase Ia (sign alignment of wsp and a) Duration of Phase Ib (growth until wsp dominates)

TIb ≍ log log(d) η −1 +

Total number of iterations in Phase II (until γ ≍ log log(d))

log(θ −1 ))η −1

Gt1 →t2

Growth factor from iteration t1 to t2

Technical lemma notation B := {i : |∆i | ≥ θd−1/2 }

“Bad” index set for Berry-Esseen approximation arguments

vB , v\B

Restrictions of vector v to indices in B and not in B, respectively

∆(v, k)

Truncated moment approximation error (Theorem F.10)

Ψ(k) x\i

EG∼N (0,∥w⊥ ∥2 ) [|G| 1(|G| ≥ k)] x − xi ei (vector with i-th coordinate zeroed)

x1:2 , x3:

Shorthands: x1:2 = (x1 , x2 , 0) and x3: = (0, 0, x3 , ξ)

L0 Analysis

In this section, we provide analysis when the neural network is small and hence ℓρ is well-approximated by a first-order Taylor expansion about fρ = 0, i.e., ℓ0 (x) := −2 log( 12 ) − y(x)fρ (x). In Section B.1, we derive the L0 population partial derivatives for any neuron (a, w). Then, in Section B.2, we provide a bound for a certain asymmetry term in the wsp partial derivative.

B.1

∇L0 Computation

Let us begin with computation of the L0 population partial derivatives (c.f. Lemma C.4 of Glasgow, 2024). Recall we define the positive neurons by S + := {(a, w) : sgn(a) = 1} and the “negative neurons” by S − := {(a, w) : sgn(a) = −1}. Moreover, for any of the p neurons (a, w), we have ∂ ∇w L0 := E pℓ0 (x) , x ∂w and that µ1 ∥ wsig if (a, w) ∈ S + while µ2 ∥ wsig if (a, w) ∈ S − . Lemma B.1. For any neuron (a, w), we have √ √ 2 ⊤ |a|∥wsig ∥ P |w⊤ ξ + w⊤ e3 | ≤ 2∥wsig ∥ −wsig ∇w L0 = 4 ξ √ √ 2 ⊤ −wopp ∇w L0 = − |a|∥wopp ∥ P |w⊤ ξ + w⊤ e3 | ≤ 2∥wopp ∥ . 4 ξ Proof. For the wsig component, we have by definition of L0 and ℓ0 that ∂ ⊤ ⊤ −wsig ∇w L0 = −wsig E pℓ0 (x) x ∂w ⊤ ′ ⊤ = wsig E ay(x)σ (w x)x . x

where y(x) := −x1 x2 is the XOR function. Recall that z := x1 e1 + x2 e2 ∈ {±µ1 , ±µ2 }. Then, since y(x) = y(z) for any z ∈ {±µ1 , ±µ2 } and wsig ⊥ x − z, we have ⊤ ⊤ −wsig ∇w L0 = E ay(z)σ ′ (w⊤ x)wsig z . (3) x,z

Depending on sgn(a), we either have (µ1 ∥ wsig and µ2 ∥ wopp ) or (µ1 ∥ wopp and µ2 ∥ wsig ). Moreover, by ⊤ definition µ1 ⊥ µ2 . Hence, wsig z = 0 unless z = ±µ1 and µ1 ∥ wsig or z = ±µ2 and µ2 ∥ wsig — in either ⊤ case, wsig z ̸= 0 only if z ∥ wsig . This gives us ⊤ −wsig ∇w L0 =

1 2

x,z z∥wsig

⊤ ay(z)σ ′ (w⊤ x)wsig z .

Now, recall that we defined x := z + s + ξ and that ξ is independent of z and s. Therefore, we can take an iterated expectation over the pair (z, s) and then over ξ. Since we are conditioning on the case where z ∥ wsig , the only cases we have to consider are z = ±µ1 for a neuron where wsig ∥ µ1 , and z = ±µ2 for a neuron where wsig ∥ µ2 . Define smaj := e3 if (a, w) ∈ S + and smaj := −e3 if (a, w) ∈ S − . Then, by Equation (1), it is easy to see that, conditioned on z ∥ wsig ,  (z0 , smaj ) w.p. (1 − λ)/2    (z , −s ) w.p. λ/2 0 maj (z, s) =  (−z0 , smaj ) w.p. (1 − λ)/2    (−z0 , −smaj ) w.p. λ/2. 21

In the above, we defined z0 ∈ {µ1 , µ2 } depending on which case the neuron lies in. Substituting this into our expression gives us 1 ⊤ −wsig ∇w L0 = a 2 1 λ ⊤ ⊤ − · E y(z0 ) · wsig z0 · σ ′ wsig z0 + w⊤ smaj + w⊤ ξ 2 2 ξ λ ⊤ ⊤ + · E y(z0 ) · wsig z0 · σ ′ wsig z0 + w⊤ (−smaj ) + w⊤ ξ 2 ξ 1 λ ⊤ ⊤ + − · E y(−z0 ) · wsig (−z0 ) · σ ′ wsig (−z0 ) + w⊤ smaj + w⊤ ξ 2 2 ξ λ ⊤ ′ ⊤ ⊤ ⊤ . + · E y(−z0 ) · wsig (−z0 ) · σ wsig (−z0 ) + w (−smaj ) + w ξ 2 ξ Since y(z0 ) = y(−z0 ), we can take y(z0 ) out as a common factor. Moreover, we have ay(z0 ) > 0 as z0 ∥ wsig . This yields 1 ⊤ ⊤ z0 · −wsig ∇w L0 = |a| · wsig 4 ⊤ ⊤ (1 − λ) · E σ ′ wsig z0 + w⊤ smaj + w⊤ ξ + λ · E σ ′ wsig z0 − w⊤ smaj + w⊤ ξ ξ ξ ′ ⊤ ⊤ ⊤ ⊤ z0 − w⊤ smaj + w⊤ ξ . − (1 − λ) · E σ −wsig z0 + w smaj + w ξ − λ · E σ ′ −wsig ξ

Collecting the terms that are multiplied by λ, we have 1 ⊤ ⊤ −wsig ∇w L0 = |a| · wsig z0 · 4 ′ ⊤ ⊤ ⊤ ′ ⊤ ⊤ ⊤ E σ wsig z0 + w smaj + w ξ − σ −wsig z0 + w smaj + w ξ ξ ⊤ ⊤ z0 + w⊤ smaj + w⊤ ξ − σ ′ wsig z0 + w⊤ smaj + w⊤ ξ + λ · E σ ′ −wsig ξ

⊤ ⊤ z0 − w⊤ smaj + w⊤ ξ − σ ′ −wsig z0 − w⊤ smaj + w⊤ ξ + λ · E σ ′ wsig . ξ

Performing casework on σ ′ gives ⊤ −wsig ∇w L0 =

√

2∥wsig ∥ for z0 ∥ wsig and z0 ∈ {µ1 , µ2 },

√ (1 − λ) · P |w⊤ ξ + w⊤ smaj | ≤ 2∥wsig ∥ ξ √ + λ · P |w⊤ ξ − w⊤ smaj | ≤ 2∥wsig ∥ . ξ

Therefore,

√

√ 2 |a|∥wsig ∥ P |w⊤ ξ + w⊤ e3 | ≤ 2∥wsig ∥ , 4 ξ

⊤ −wsig ∇w L0 =

which proves the first equation. For the wopp component, the same calculations with ay(z) < 0 for z ∥ wopp give √ √ 2 ⊤ −wopp ∇w L0 = − |a|∥wopp ∥ P |w⊤ ξ + w⊤ e3 | ≤ 2∥wopp ∥ , 4 ξ which proves the second equation. This completes the proof of the lemma. Now we will show the lemma for the wsp component. Lemma B.2. For any neuron (a, w), define A = min({±w⊤ µ1 , ±w⊤ µ2 }) B = min({±w⊤ µ1 , ±w⊤ µ2 } \ {A}) and

( εw =

Pξ w⊤ ξ − wsp ∈ [A, B] − Pξ w⊤ ξ + wsp ∈ [A, B] Pξ w⊤ ξ + wsp ∈ [A, B] − Pξ w⊤ ξ − wsp ∈ [A, B]

We then have ⊤ −wsp ∇w L0 = awsp

if sgn(w1 ) = sgn(w2 ) . if sgn(w1 ) ̸= sgn(w2 )

1 εw + −λ . 2 4

Remark B.3. The term εw is an intermediate term resulting from a slight asymmetry in the spurious feature gradient. In Section B.2, we show that it is exponentially decreasing in ∥wsp ∥. Proof. We have (c.f. Equation (3)) ⊤ −wsp ∇w L0 = E

x,z,s

⊤ ay(z)σ ′ (w⊤ x)wsp s .

(4)

Observe from Equation (1) that s = y(z)e3 with probability at least 1 − λ and s = −y(z)e3 with probability at least λ. Recall our definitions of the majority group as Xmaj := {x ∈ {±1}d : y(x) = x3 } and the minority group as Xmin := {x ∈ {±1}d : y(x) = −x3 }. Equivalently, Xmaj := {x ∈ {±1}d : s = y(z)e3 } and Xmin := {x ∈ {±1}d : s = −y(z)e3 }. Taking expectations only over s, we have ⊤ −wsp ∇w L0 = a ⊤ (1 − λ) · y(z) · σ ′ w⊤ z + w⊤ (y(z)e3 ) + w⊤ ξ · wsp (y(z)e3 ) E z,ξ|x∈Xmaj ⊤ +λ· y(z) · σ ′ w⊤ z + w⊤ (−y(z)e3 ) + w⊤ ξ · wsp (−y(z)e3 ) . E z,ξ|x∈Xmin

Since y(z)2 = 1, we have ⊤ −wsp ∇w L0 = awsp (

(1 − λ)

[σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)]

z,ξ|x∈Xmaj

−λ

[σ ′ (w⊤ z − y(z)wsp + w⊤ ξ)]).

z,ξ|x∈Xmin

Using that the conditional distribution of z, ξ is the same for the majority and minority groups, and noting that σ ′ (u) = 1(u ≥ 0), we have ⊤ −wsp ∇w L0 = awsp (1 − λ) · P w⊤ z + w⊤ ξ > −y(z)wsp z,ξ − λ · P w⊤ z + w⊤ ξ > y(z)wsp . z,ξ

Recall that ξ is a symmetric random variable and independent of z. Therefore, for a fixed z we have ⊤ ⊤ ⊤ ⊤ P w z + w ξ > − y(z)wsp + P w (−z) + w ξ > y(z)wsp ξ ξ ⊤ = P w z + w⊤ ξ > −y(z)wsp + P w⊤ (−z) + w⊤ (−ξ) > y(z)wsp ξ ξ ⊤ ⊤ = P w z + w ξ > −y(z)wsp + P w⊤ z + w⊤ ξ ≤ −y(z)wsp ξ

(5)

= 1.

Moreover, it is immediate from Equation (1) that z is also a symmetric random variable. Using this along with the fact that y(z) = y(−z), the above gives us ⊤ ⊤ ⊤ ⊤ P w z + w ξ > −y(z)wsp + P w z + w ξ > y(z)wsp = 1. ξ

This implies that ⊤ −wsp ∇w L0 = awsp

It remains to show that

⊤

P w z + w ξ > −y(z)wsp − λ .

z,ξ

1 εw ⊤ ⊤ . P w z + w ξ > −y(z)wsp = + 2 4 z,ξ

(6)

We begin by writing out the probabilities over z. It is immediate from Equation (1) that z is uniformly distributed over {±µ1 , ±µ2 }. Therefore, we have 1 ⊤ ⊤ P w z + w ξ > −y(z)wsp = 4 z,ξ ⊤ ⊤ ⊤ ⊤ P w ξ + wsp > −w µ1 + P w ξ + wsp > w µ1 ξ ξ ⊤ ⊤ + P w ξ − wsp > −w µ2 + P w⊤ ξ − wsp > w⊤ µ2 . ξ

Taking complements and using that ξ is symmetric about 0, we have 1 1 ⊤ ⊤ P w z + w ξ > −y(z)wsp = + 2 4 z,ξ − P w⊤ ξ + wsp ≤ −w⊤ µ1 − P w⊤ ξ + wsp ≤ w⊤ µ1 ξ

⊤

+ P w ξ + wsp ≤ −w µ2 + P w⊤ ξ + wsp ≤ w⊤ µ2 ξ

For notational convenience, let us define εw := − P w⊤ ξ + wsp ≤ −w⊤ µ1 − P w⊤ ξ + wsp ≤ w⊤ µ1 ξ ξ ⊤ ⊤ + P w ξ + wsp ≤ −w µ2 + P w⊤ ξ + wsp ≤ w⊤ µ2 . ξ

We will show that εw defined here is equal to the formula in the statement of Theorem B.2. Let us write εw in interval notation to find that certain segments cancel. Let A ≤ B ≤ C ≤ D be the ordering on {±w⊤ µ1 , ±w⊤ µ2 } that is a function of w (i.e., A is the smallest element, B is the second smallest, and so on). Note that A = −D and B = −C. Moreover, if sgn(w1 ) = sgn(w2 ), then A ∈ {±w⊤ µ2 }, whereas if sgn(w1 ) ̸= sgn(w2 ), then A ∈ {±w⊤ µ1 }. We now handle these cases separately. If sgn(w1 ) = sgn(w2 ), we have εw = 2 P w⊤ ξ + wsp ≤ A + P A ≤ w⊤ ξ + wsp ≤ D − 2 P w⊤ ξ + wsp ≤ A ξ ξ ξ ⊤ ⊤ − P A ≤ w ξ + wsp ≤ B − P A ≤ w ξ + wsp ≤ C ξ ξ = P C ≤ w⊤ ξ + wsp ≤ D − P A ≤ w⊤ ξ + wsp ≤ B . ξ

Finally, since A = −D and B = −C and ξ is symmetric around 0, we have εw = P A ≤ w⊤ ξ − wsp ≤ B − P A ≤ w⊤ ξ + wsp ≤ B . ξ

Likewise, if sgn(w1 ) ̸= sgn(w2 ), we have εw = 2 P w⊤ ξ + wsp ≤ A + P A ≤ w⊤ ξ + wsp ≤ B + P A ≤ w⊤ ξ + wsp ≤ C ξ ξ ξ ⊤ ⊤ − 2 P w ξ + wsp ≤ A − P A ≤ w ξ + wsp ≤ D ξ ξ = P A ≤ w⊤ ξ + wsp ≤ B − P C ≤ w⊤ ξ + wsp ≤ D . ξ

Finally, since ξ is symmetric about 0 with A = −D and B = −C, we have εw = P A ≤ w⊤ ξ + wsp ≤ B − P A ≤ w⊤ ξ − wsp ≤ B . ξ

This completes the proof of the lemma. Next, we will show the lemma for the w⊥ component. Lemma B.4. For any neuron (a, w), we have h 1 ⊤ −w⊥ ∇w L0 = |a| E |w⊤ ξ| 8 ξ √ √ 1 |w⊤ ξ| ≥ | 2∥wsig ∥ + wsp | + 1 |w⊤ ξ| ≥ | 2∥wsig ∥ − wsp | i √ √ − 1 |w⊤ ξ| ≥ | 2∥wopp ∥ + wsp | − 1 |w⊤ ξ| ≥ | 2∥wopp ∥ − wsp | . Proof. We have (c.f. Equation (3)) ⊤ −w⊥ ∇w L0 = E

x,z,ξ

⊤ ay(z)σ ′ (w⊤ x)w⊥ ξ .

(7)

Recall that x = z + s + ξ. Noting that ξ is symmetric around 0, we perform a symmetrization argument to get 1 ⊤ a E y(z) · σ ′ w⊤ z + w⊤ s + w⊤ ξ · w⊥ ξ 2 z,s,ξ 1 ⊤ + a E y(z) · σ ′ w⊤ z + w⊤ s + w⊤ (−ξ) · w⊥ (−ξ) 2 z,s,ξ 1 ⊤ = a E y(z) · w⊥ ξ · σ ′ w⊤ z + w⊤ s + w⊤ ξ − σ ′ w⊤ z + w⊤ s − w⊤ ξ . 2 z,s,ξ

⊤ −w⊥ ∇w L0 =

Noting that σ ′ (u) = 1(u ≥ 0) and performing casework gives on σ ′ (·) gives ⊤ −w⊥ ∇w L0 =

1 ⊤ ⊤ a E y(z) · w⊥ ξ · 1 |w⊤ ξ| ≥ |w⊤ z + w⊤ s| · sgn(w⊥ ξ) . 2 z,s,ξ

⊤ ⊤ ⊤ Since sgn(w⊥ ξ)w⊥ ξ = |w⊥ ξ|, we obtain ⊤ −w⊥ ∇w L0 =

1 ⊤ a E y(z) · |w⊥ ξ| · 1 |w⊤ ξ| ≥ |w⊤ z + w⊤ s| . 2 z,s,ξ

Since ξ is independent of (z, s), we can expand the expectation over (z, s) using the joint distribution defined in Equation (1) to obtain h 1 ⊤ −w⊥ ∇w L0 = a E |w⊤ ξ| 8 ξ 1 |w⊤ ξ| ≥ |w⊤ µ1 + wsp | + 1 |w⊤ ξ| ≥ |w⊤ µ1 − wsp | i . − 1 |w⊤ ξ| ≥ |w⊤ µ2 + wsp | − 1 |w⊤ ξ| ≥ |w⊤ µ2 − wsp | √ ⊤ If (a, w) ∈ S + we have µ1 ∥ wsig , µ2 ∥ wopp , and a > 0. Since w⊤ µ1 = wsig µ1 = 2∥wsig ∥ and √ ⊤ w⊤ µ2 = wopp µ2 = 2∥wopp ∥ in this case, we have h 1 ⊤ −w⊥ ∇w L0 = |a| E |w⊤ ξ| 8 ξ √ √ 1 |w⊤ ξ| ≥ | 2∥wsig ∥ + wsp | + 1 |w⊤ ξ| ≥ | 2∥wsig ∥ − wsp | i √ √ . (8) − 1 |w⊤ ξ| ≥ | 2∥wopp ∥ + wsp | − 1 |w⊤ ξ| ≥ | 2∥wopp ∥ − wsp | ⊤ On the other hand, if (a, w) ∈ S − we have µ2 ∥ wsig , µ1 ∥ wopp , and a < 0. Since w⊤ µ2 = wsig µ2 = √ √ ⊤ ⊤ 2∥wsig ∥ and w µ1 = wopp µ1 = 2∥wopp ∥ in this case, simplification results in the same expression as Equation (8). This completes the proof.

Finally, we repeat the analysis for any individual coordinate of w⊥ . Lemma B.5. For any neuron (a, w) and i > 3, we have 1 −wi ∂wi L0 = |a||wi | 8 √ √ ⊤ ⊤ P |wi | ≥ | 2∥wsig ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wsig ∥ − wsp + w ξ\i | ξ ξ √ √ − P |wi | ≥ | 2∥wopp ∥ + wsp + w⊤ ξ\i | − P |wi | ≥ | 2∥wopp ∥ − wsp + w⊤ ξ\i | . ξ

Proof. We have (c.f. Equation (3)) −wi ∂wi L0 = E ay(x)σ ′ (w⊤ x)wi xi . x

(9)

Noting that xi is symmetric around 0 (it is a single coordinate of ξ), we perform a similar symmetrization to Theorem B.4 to get −wi ∂wi L0 =

1 a E y(x) · wi xi · σ ′ w⊤ x\i + wi xi − σ ′ w⊤ x\i − wi xi 2 x

Noting that σ ′ (u) = 1(u ≥ 0) and performing casework gives on σ ′ (·) gives us −wi ∂wi L0 =

1 a E y(x) · wi xi · 1 |wi | ≥ |w⊤ x\i | · sgn(wi xi ) . 2 x 26

Since sgn(wi xi )wi xi = |wi xi | = |wi |, we obtain −wi ∂wi L0 =

1 a|wi | E y(x) · 1 |wi | ≥ |w⊤ x\i | . 2 x

Since ξ is independent of (z, s) and symmetric, we can expand the expectation over (z, s) using the joint distribution defined in Equation (1) to obtain −wi ∂wi L0 =

1 a|wi | 8 ⊤ ⊤ ⊤ ⊤ P |wi | ≥ |w µ1 + wsp + w ξ\i | + P |wi | ≥ |w µ1 − wsp + w ξ\i | ξ

− P |wi | ≥ |w µ2 + wsp + w ξ\i | − P |wi | ≥ |w⊤ µ2 − wsp + w⊤ ξ\i | . ⊤

⊤

⊤ If (a, w) ∈ S + we have µ1 ∥ wsig , µ2 ∥ wopp , and a > 0. Since w⊤ µ1 = wsig µ1 = √ ⊤ ⊤ w µ2 = wopp µ2 = 2∥wopp ∥ in this case, we have

−wi ∂wi L0 =

√

2∥wsig ∥ and

1 |a||wi | 8

√ √ ⊤ ⊤ P |wi | ≥ | 2∥wsig ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wsig ∥ − wsp + w ξ\i | ξ ξ √ √ − P(|wi | ≥ | 2∥wopp ∥ + wsp + w⊤ ξ\i |) − P |wi | ≥ | 2∥wopp ∥ − wsp + w⊤ ξ\i | . ξ

(10)

⊤ On the other hand, if (a, w) ∈ S − we have µ2 ∥ wsig , µ1 ∥ wopp , and a < 0. Since w⊤ µ2 = wsig µ2 = √ √ ⊤ ⊤ 2∥wsig ∥ and w µ1 = wopp µ1 = 2∥wopp ∥ in this case, simplification results in the same expression as Equation (10). This completes the proof of the lemma.

B.2

εw Asymmetry Term Bound

In this section, we will show that the asymmetry term εw from Theorem B.2 decays exponentially in ∥wsp ∥. Recall that A = min({±w⊤ µ1 , ±w⊤ µ2 }) B = min({±w⊤ µ1 , ±w⊤ µ2 } \ {A}) and

( εw :=

Pξ w⊤ ξ − wsp ∈ [A, B] − Pξ w⊤ ξ + wsp ∈ [A, B] if sgn(w1 ) = sgn(w2 ) . Pξ w⊤ ξ + wsp ∈ [A, B] − Pξ w⊤ ξ − wsp ∈ [A, B] if sgn(w1 ) ̸= sgn(w2 )

We begin with a tight bound on |εw | using the Berry-Esseen theorem. −x2 Lemma B.6. For any neuron (a, w), write ϕ(x) := ∥w 1∥√2π exp 2∥w for the probability density function 2 ∥ ⊥ ⊥

of the Gaussian distribution N (0, ∥w⊥ ∥2 ). Then, we have Z B |εw | ≲ ϕ(wsp )

exp

−u2 2∥w⊥ ∥2

sinh

uwsp ∥w⊥ ∥2

du +

∥w⊥ ∥33 . ∥w⊥ ∥32

Proof. Without loss of generality let sgn(w1 ) = sgn(w2 ). (Note that the case sgn(w1 ) ̸= sgn(w2 ) proceeds identically because the expression for |εw | is the same in both cases and w⊤ ξ does not depend on w1 and

w2 .) Write Φ(x) := PG∼N (0,∥w⊥ ∥2 ) (G ≤ x) for the cumulative distribution function of the centered Gaussian with variance ∥w⊥ ∥2 . We apply Theorem F.5 with v := w⊥ to obtain εw := P w⊤ ξ − wsp ∈ [A, B] − P w⊤ ξ + wsp ∈ [A, B] ξ ξ ⊤ ⊤ = P w⊥ ξ ≤ B + wsp − P w⊥ ξ ≤ A + wsp ξ ξ ⊤ ⊤ − P w⊥ ξ ≤ B − wsp + P w⊥ ξ ≤ A − wsp ξ

≲ Φ(B + wsp ) − Φ(A + wsp ) − Φ(B − wsp ) + Φ(A − wsp ) +

∥w⊥ ∥33 . ∥w⊥ ∥32

In particular, Z B+wsp

Z B−wsp ϕ(u)du −

εw ≲ A+wsp

ϕ(u)du + A−wsp

Z B ϕ(u + wsp ) − ϕ(u − wsp )du +

= A

∥w⊥ ∥33 ∥w⊥ ∥32

∥w⊥ ∥33 , ∥w⊥ ∥32

where in the last step we changed variables. This expression is convenient to see that εw ≈ 0 when A, B ≈ 0 or wsp ≈ 0. To formalize this intuition, we expand and factorize to obtain Z B Z B −(u+wsp )2 −(u−wsp )2 1 1 2∥w⊥ ∥2 2∥w⊥ ∥2 √ √ ϕ(u + wsp ) − ϕ(u − wsp )du = − du e e ∥w⊥ ∥ 2π ∥w⊥ ∥ 2π A A −uwsp Z B 2 −wsp uwsp −u2 1 √ e 2∥w⊥ ∥2 e 2∥w⊥ ∥2 e ∥w⊥ ∥2 − e ∥w⊥ ∥2 du = A ∥w⊥ ∥ 2π Z B −u2 uwsp 2∥w⊥ ∥2 = −2ϕ(wsp ) sinh dt. e ∥w⊥ ∥2 A Taking the absolute value and using the triangle inequality, we have Z B uwsp ∥w⊥ ∥33 −u2 sinh du + |εw | ≲ ϕ(wsp ) exp , 2∥w⊥ ∥2 ∥w⊥ ∥2 ∥w⊥ ∥32 A as desired. This completes the proof of the lemma. Now, let us show a more sophisticated upper bound on |εw | which implies exponential decay in ∥wsp ∥. Lemma B.7. For any neuron (a, w) we have ∥w1:2 ∥ |εw | ≲ exp ∥w⊥ ∥

! √ −∥wsp ∥2 + 2 2∥w1:2 ∥∥wsp ∥ ∥w⊥ ∥33 + . 2∥w⊥ ∥2 ∥w⊥ ∥32

Since exp

−u2 2∥w⊥ ∥2

≤ 1 and supu∈[A,B] exp

|uwsp | ∥w⊥ ∥2

= exp

|A|∥wsp ∥ ∥w⊥ ∥2

, we have

|A|∥wsp ∥ ∥w⊥ ∥33 |εw | ≲ ϕ(wsp )(B − A) exp + 2 ∥w⊥ ∥ ∥w⊥ ∥32 ∥w⊥ ∥33 B−A −∥wsp ∥2 + 2|A|∥wsp ∥ + . ≲ exp ∥w⊥ ∥ 2∥w⊥ ∥2 ∥w⊥ ∥32 √ Finally since B − A ≤ |A| ≤ 2∥w1:2 ∥ we obtain ! √ ∥w⊥ ∥33 ∥w1:2 ∥ −∥wsp ∥2 + 2 2∥w1:2 ∥∥wsp ∥ + , exp |εw | ≲ ∥w⊥ ∥ 2∥w⊥ ∥2 ∥w⊥ ∥32 which completes the proof of the lemma.

Lρ Analysis

In this section, we directly analyze the dynamics of the neural network trained under the standard logistic loss ℓρ , i.e., 1 ℓρ (x) = −2 log . 1 + exp(−y(x)fρ (x)) In Section C.1, we bound the deviation between the L0 and Lρ population gradients. In Section C.2, we directly characterize the Lρ population gradients assuming conditions on the empirical margin of the data that will hold later in our analysis.

C.1

∇Lρ − ∇L0 Error Analysis

In this section, we will bound the absolute values of the coordinates of ∇Lρ − ∇L0 . We will find this difference to be small throughout Phase I of our analysis. However, in Phase II the neural network is too large for this L0 -approximation to be useful, and we must directly analyze the Lρ gradient (see Section C.2). We define γ(x) := y(x)fρ (x) as the margin of data point x, and the composite notation ℓρ (x) := h(γ(x)) where h(γ) := −2 log(ψ(γ)) (where ψ(u) := 1/(1 + e−u ) denotes the sigmoid). Accordingly, we define (1) (2) ℓρ (x) := h′ (γ(x)) and ℓρ (x) := h′′ (γ(x)). Similarly, note that ℓ0 (x) = −2 log( 12 ) − γ(x) and so we can (1) define ℓ0 (x) := −1 as shorthand. First, we require the following generalization of Glasgow, 2024, Lemma C.5. Lemma C.1. Define gij : {±1}d → {±1}d to flip the ith and j th bits of the input x and indices (i, j) are chosen such that y(g(x)) = y(x).4 For any x ∈ {±1}d , we have (1) |ℓ(1) ρ (x) − ℓρ (gij (x))| ≤

[|awi | + |awj |].

(a,w)∼ρ

(1)

Proof. Recall that ℓρ (x) := h′ (γ(x)), and note that h′ (γ) is 12 -Lipschitz with respect to γ.5 Since we have assumed that y(x) = y(gij (x)), this gives us (1) |ℓ(1) ρ (x) − ℓρ (gij (x))| ≤

1 |fρ (x) − fρ (gij (x))|. 2

(11)

By definition of gij we have |w⊤ (x − gij (x))| = |2wi xi + 2wj xj |, and using the triangle inequality once again gives |a| · |w⊤ (x − gij (x))| ≤ 2 E [|awi | + |awj |]. E (a,w)∼ρ

(a,w)∼ρ

Substituting this into Equation (11) completes the proof of the lemma. We will first show a bound on all components of ∇Lρ − ∇L0 which is effective when the network is small. Lemma C.2. Suppose E(a,w)∼ρ [∥aw∥] ≤ dC for a constant C > 0. Then, for any neuron (a, w) and any i ∈ [d], we have 2 |∂wi Lρ − ∂wi L0 | ≤ log(d) · E [∥aw∥] + d−C , |a| (a,w)∼ρ where ∂wi L denotes the p-scaled partial derivative of L with respect to the i-th coordinate of w. 4 Note that this is the case as long as i, j ∈ {1, 2} or both i, j ∈ / {1, 2}. 2e−γ 5 Glasgow, 2024 has 2-Lipschitz but 1 is actually the tight constant; it follows by noting that h′′ (γ) = ≤ 12 . 2 (1+e−γ )2

(1)

Proof. Define ∆x := (ℓρ (x) − ℓ0 (x)) · σ ′ (w⊤ x) as shorthand. We have by the triangle inequality and |xi | = 1 that 1 |∂wi Lρ − ∂wi L0 | = | E[∆x xi ]| ≤ E[|∆x |]. |a| x x (1)

(1)

Now, note that ℓρ (x) = h′ (γ(x)) and ℓ0 (x) = −1 = h′ (0). Therefore, again using the 12 -Lipschitzness of h′ (γ) and σ ′ (u) ≤ 1 for the ReLU activation, we have E[|∆x |] ≤ x

1 1 E[|γ(x)|] ≤ E[|fρ (x)|]. 2x 2x

Splitting on the event that |fρ (x)| ≤ log(d) E(a,w)∼ρ [∥aw∥] and using Lemma F.14(iii), we have for a fixed x that |fρ (x)| ≤ log(d) · E [∥aw∥] · 1 |fρ (x)| ≤ log(d) E [∥aw∥] (a,w)∼ρ

√ +

(a,w)∼ρ

d·

[∥aw∥] · 1 |fρ (x)| ≥ log(d)

[∥aw∥] +

(a,w)∼ρ

≤ log(d) ·

√

d·

(a,w)∼ρ

[∥aw∥]

E [∥aw∥] · 1 |fρ (x)| ≥ log(d) (a,w)∼ρ

(a,w)∼ρ

[∥aw∥] .

(a,w)∼ρ

Bringing back the expectation over x, we have 1 1√ 1 d · E [∥aw∥] · P |fρ (x)| ≥ log(d) E [∥aw∥] E[|fρ (x)|] ≤ log(d) · E [∥aw∥] + 2x 2 2 x (a,w)∼ρ (a,w)∼ρ (a,w)∼ρ 1 −C ≤ log(d) · E [∥aw∥] + d , 2 (a,w)∼ρ where the last step follows from E(a,w)∼ρ [∥aw∥] ≤ dC and Lemma F.14(iv) (choosing the constant in Lemma F.14(iv) greater than 2C + 1). This completes the proof of the lemma. Let us show the key lemma of this section. It is a modified version of Glasgow, 2024, Lemma C.3 which accounts for the spurious correlation. In contrast to Glasgow, 2024, for which a leave-one-out analysis is sufficient, we must proceed with a slightly more complicated leave-two-out technique. Lemma C.3. Recall that we write x := z + s + ξ. Define x\i := x − xi ei and suppose E(a,w)∼ρ [∥aw∥] ≤ dC for a constant C > 0. Then, for any neuron (a, w), we have 2 |∂wi Lρ − ∂wi L0 | |a| ( E(a,w)∼ρ [|aw1 | + |aw2 |] + log(d) · Px |w⊤ (s + ξ)| < |w⊤ z| · E(a,w)∼ρ [∥aw∥] + d−C ≤ E(a,w)∼ρ [|awi |] + log(d) · Px |w⊤ x\i | < |wi | · E(a,w)∼ρ [∥aw∥] + d−C

i ∈ {1, 2} i > 3.

Proof. Let us begin with the case i ∈ {1, 2}. The symmetry required by Glasgow, 2024 is that flipping one bit does not change the marginal, i.e., P(x\i + xi ei ) = P(x\i − xi ei ). While this no longer holds due to the spurious correlation, it is in fact the case that flipping two specific bits does not change the marginal. Define g := g12 to flip the first two bits of x, e.g., g(z + s + ξ) = −z + s + ξ. It is immediate from Equation (1) (1) (1) that P(x) = P(g(x)). Once again, define ∆x := (ℓρ (x) − ℓ0 (x)) · σ ′ (w⊤ x) as shorthand. We have 1 |∂wi Lρ − ∂wi L0 | = | E[∆x xi ]| |a| x 1 = | E[∆x xi ] + E[∆g(x) g(x)i ]| 2 x x 1 = | E[(∆x − ∆g(x) )xi ]|, 2 x 31

(12)

where we used g(x)i = −xi for i ∈ {1, 2} by definition. Notice that if |w⊤ (s + ξ))| ≥ |w⊤ z| then σ ′ (w⊤ x) = σ ′ (w⊤ g(x)) as the sign of w⊤ x is specified by the sign of w⊤ (s + ξ)). Under this assumption, we have (1) ′ ′ (1) |∆x − ∆g(x) | = ℓ(1) = |ℓ(1) ρ (x) − ℓρ (g(x)) − ℓ0 (x) − ℓ0 (g(x)) ρ (x) − ℓρ (g(x))|, where in the last step we used ℓ′0 (x) = −y(x) and y(x) = y(g(x)) by definition. On the other hand, we have a coarser bound using the triangle inequality and |xi | = 1 as follows: 1 1 | E[(∆x − ∆g(x) )xi ]| ≤ E[|∆x | + |∆g(x) |] = E[|∆x |], 2 x 2x x where the last step holds since each x is counted exactly twice. Returning to Equation (12) and using the triangle inequality once again, we have 1 1 ⊤ ⊤ | E[(∆x − ∆g(x) )xi ]| ≤ E 1 |w (s + ξ)| ≥ |w z| · (∆x − ∆g(x) )xi 2 x 2 x 1 + E 1 |w⊤ (s + ξ)| < |w⊤ z| · (∆x − ∆g(x) )xi 2 x 1 ≤ sup |ℓ(1) (x) − ℓ(1) ρ (g(x))| 2 x∈{±1}d ρ + E 1 |w⊤ (s + ξ)| < |w⊤ z| · |∆x | . x

(1) (1) By Theorem C.1 we have supx∈{±1}d |ℓρ (x) − ℓρ (g(x))| ≤ E(a,w)∼ρ [|aw1 | + |aw2 |].

The remainder of the proof follows in the same way as Theorem C.2. For the i > 3 case we can use the same leave-one-out symmetrization as Glasgow, 2024 instead of the gi symmetrization. This completes the proof of the lemma.

C.2

∇Lρ Computation

In this section, we will derive some Lρ population gradients assuming conditions on the empirical margin of the data which will hold during our Phase II. Let us first recall some notation. We defined the p-scaled Lρ gradient as ∂ ℓρ (x) . ∇w Lρ := p E x ∂w Recall that we denote the majority group as Xmaj := {x ∈ {±1}d : y(x) = x3 } and the minority group as Xmin := {x ∈ {±1}d : y(x) = −x3 }. Additionally, ψ(u) := 1/(1 + e−u ) denotes the sigmoid and γ(x) := y(x)fρ (x) denotes the margin. Moreover, we let C > 0 denote a sufficiently large constant which does not change from line to line. In the remainder of this section, we will often reference the event Etest defined in Definition F.1(vii); for a point x = (z, s, ξ), we will equivalently write that x satisfies Etest and ξ satisfies Etest , as the event is solely a property of ξ. Our key decomposition of the Lρ gradient is as follows: h i ∇w Lρ = p E ℓ(1) (x)∇ f (x) w ρ ρ x −2y(x) exp(−γ(x)) = pE ∇w fρ (x) 1 + exp(−γ(x)) x = 2p E [ψ(−γ(x)) · (−y(x)∇w fρ (x))] x

= 2 E [ψ(−γ(x)) · ∇w pℓ0 (x)] . x

(13)

Hence, the Lρ gradient may be analyzed as a product of a margin term and an ∇L0 term. We first show an upper bound on the Lρ gradient for the wsig and wopp components for neural network parameters under the condition that the margins for the majority and minority groups are approximately “equal and opposite”. 32

Lemma C.4. Assume there exists γ > 0 such that ( (1 ± o(1)) · γ ∀x ∈ Xmaj satisfying Etest γ(x) = (−1 ± o(1)) · γ ∀x ∈ Xmin satisfying Etest ,

(14)

Proof. Combining Equation (3) and Equation (13), we have ⊤ ⊤ −wsig ∇w Lρ = 2 E ψ(−γ(x)) · ay(z)σ ′ (w⊤ x)wsig z x,z

⊤ Following Theorem B.1, we have wsig z ̸= 0 only if z ∥ wsig , which yields ⊤ ⊤ z . −wsig ∇w Lρ = E ψ(−γ(x)) · ay(z)σ ′ (w⊤ x)wsig x,z z∥wsig

Following Theorem B.1 again, define z0 ∈ {µ1 , µ2 } depending on whether wsig ∥ µ1 or wsig ∥ µ2 . Using the distribution of (z, s), we have y(z0 ) = y(−z0 ) and ay(z0 ) > 0 as z0 ∥ wsig . Recall that we define smaj := e3 if (a, w) ∈ S + and smaj := −e3 if (a, w) ∈ S − (all as in Theorem B.1). Therefore, we have 1 ⊤ ⊤ −wsig ∇w Lρ = |a| · wsig z0 · 2 ⊤ (1 − λ) E ψ(−γ(z0 + smaj + ξ)) · σ ′ (wsig z0 + w⊤ smaj + w⊤ ξ) ξ ⊤ + λ E ψ(−γ(z0 − smaj + ξ)) · σ ′ (wsig z0 − w⊤ smaj + w⊤ ξ) ξ ⊤ − (1 − λ) E ψ(−γ(−z0 + smaj + ξ)) · σ ′ (−wsig z0 + w⊤ smaj + w⊤ ξ) ξ ⊤ − λ E ψ(−γ(−z0 − smaj + ξ)) · σ ′ (−wsig z0 − w⊤ smaj + w⊤ ξ) . (15) ξ

for any c < 1 and ξ satisfying the event Etest . Under this same event, using the norm bound in the statement of the lemma, we have sgn(w⊤ x) = sgn(wsp x3 ) for all neurons (a, w) following Theorem D.1. Hence, the set S of active ReLUs are the same for any two points in Xmaj and any two points in Xmin that satisfy the event Etest . Therefore, for any ξ satisfying the event Etest , we have |f (z0 + smaj + ξ) − f (−z0 + smaj + ξ)| 1 X 1 X = aσ(w⊤ (z0 + smaj + ξ)) − aσ(w⊤ (−z0 + smaj + ξ)) p p (a,w)

= =

1 p 2 p

(a,w)

aw⊤ (z0 + smaj + ξ) −

(a,w)∈S

1 p

aw⊤ (−z0 + smaj + ξ)

(a,w)∈S

aw⊤ z0

(a,w)∈S

≲ max |a|∥wsig ∥. (a,w)

Using |ac − bd| ≤ |ac − bc| + |bc − bd| ≤ |a − b| + |c − d| for 0 ≤ a, b, c, d ≤ 1, we then have h ′ ⊤ ⊤ ⊤ E ψ(−γ(z0 + smaj + ξ)) · σ (wsig z0 + w smaj + w ξ) ξ i ⊤ − ψ(−γ(−z0 + smaj + ξ)) · σ ′ (−wsig z0 + w⊤ smaj + w⊤ ξ) √ ≲ e−cγ max |a|∥wsig ∥ + P |w⊤ ξ + w⊤ e3 | ≤ 2∥wsig ∥ + d−C , (a,w)

C where the final d−C follows in the case that ξ does not satisfy the event Etest , using Pξ [Etest ] ≤ d−C by Theorem F.4. An identical argument and inequality results for the minority group terms in Equation (15). √ ⊤ Combining the majority group and minority group terms in Equation (15) with |wsig z0 | = 2∥wsig ∥, we obtain √ ⊤ −wsig ∇w Lρ ≲ |a|∥wsig ∥ e−cγ max |a|∥wsig ∥ + P |w⊤ ξ + w⊤ e3 | ≤ 2∥wsig ∥ + d−C , (a,w)

This completes the proof of the lemma. Next, we provide the lemma for the wsp component under the same margin concentration condition. This lemma recovers Theorem B.2 when ψ(γ) = 12 , and can be thought of as its generalization to the Lρ gradient. Lemma C.5. Assume there exists γ, ϵ > 0 such that ( (1 ± ϵ) · γ ∀x ∈ Xmaj satisfying Etest γ(x) = (−1 ± ϵ) · γ ∀x ∈ Xmin satisfying Etest , where the event Etest is defined in Definition F.1(vii). We then have εw ⊤ − wsp ∇w Lρ − awsp (1 − λ − ψ(γ)) · 1 + + λψ(γ)εw ≲ |a||wsp |(ϵγ + d−C ). 2

(16)

Proof. Combining Equation (4) and Equation (13), we have ⊤ ⊤ −wsp ∇w Lρ = 2 E ψ(−γ(x)) · ay(z)σ ′ (w⊤ x)wsp s . x,z,s

By definition of s, and noting that x ∈ Xmaj with probability 1 − λ and x ∈ Xmin with probability λ, we have ⊤ −wsp ∇w Lρ = 2a ⊤ (1 − λ) ψ(−γ(x)) · y(z) · σ ′ (w⊤ z + w⊤ (y(z)e3 ) + w⊤ ξ) · wsp (y(z)e3 ) E z,ξ|x∈Xmaj ⊤ +λ ψ(−γ(x)) · y(z) · σ ′ (w⊤ z + w⊤ (−y(z)e3 ) + w⊤ ξ) · wsp (−y(z)e3 ) . E z,ξ|x∈Xmin

Above, we used that the conditional distribution of (z, ξ) is the same for majority and minority groups (already noted in the proof of Theorem B.2). Since y(z)2 = 1, we obtain ⊤ −wsp ∇w Lρ = 2awsp (1 − λ) ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj −λ ψ(−γ(x)) · σ ′ (w⊤ z − y(z)wsp + w⊤ ξ) . (17) E z,ξ|x∈Xmin

Notation. Let us use the shorthand ξ ∈ Etest to denote that ξ satisfies the event Etest . (Note that Etest is used to denote an event and a set interchangeably; this is an overloading of notation which is valid since ξ is uniformly distributed). Recall that we defined x := z + s + ξ, and that the event x ∈ Xmaj is defined only on z, s (in particular, x ∈ Xmaj ⇐⇒ x3 = −x1 x2 , i.e., s3 = −z1 z2 ). We write (z, s) ∈ Xmaj as shorthand. Recall that ξ ∼ Unif(03 × {±1}d−3 ) and ξ is independent of (z, s). Considering the majority group conditional expectation term in Equation (17), we can therefore write ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj





1  ψ(−γ(z + s + ξ)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)  E 2d−3 (z,s)∈X maj ξ∈03 ×{±1}d−3   X 1 = d−3  ψ(−γ(z + s + ξ)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)  E 2 (z,s)∈Xmaj ξ∈Etest   X 1 + d−3  ψ(−γ(z + s + ξ)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)  . E 2 (z,s)∈Xmaj

(18)

ξ∈E / test

Now, since ξ ∼ Unif(03 × {±1}d−3 ) and ξ is independent of (z, s), the conditional distribution of ξ | ξ ∈ Etest test | is uniform over the set Etest ; moreover, Pξ (ξ ∈ Etest ) = |E2d−3 . This means that P(ξ ∈ Etest ) · ξ

ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)

z,ξ|x∈Xmaj ,ξ∈Etest

 X |Etest | 1 = d−3 · · 2 |Etest |

ξ∈Etest

(z,s)∈Xmaj

 ψ(−γ(z + s + ξ)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)  ,

and similarly, / Etest ) · P(ξ ∈ ξ

ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)

z,ξ|x∈Xmaj ,ξ∈Etest

 C X 1 |Etest | = d−3 · C ·  2 |Etest |

ξ∈E / test

(z,s)∈Xmaj

 ψ(−γ(z + s + ξ)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)  .

Substituting these into the display in Equation (18) yields ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj

= P(ξ ∈ Etest ) · ξ

ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)

z,ξ|x∈Xmaj ,ξ∈Etest

+ P(ξ ∈ / Etest ) · ξ

z,ξ|x∈Xmaj ,ξ∈E / test

ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ) .

(19)

For the first term in Equation (19), we use the condition in Equation (16) and the fact that σ ′ (·) ≥ 0 to obtain the sandwich inequality ′ ⊤ ψ(−(1 + ϵ)γ) · σ (w z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj ,ξ∈Etest

≤

ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)

z,ξ|x∈Xmaj ,ξ∈Etest

≤ ψ(−(1 − ϵ)γ) ·

′ ⊤ σ (w z + y(z)wsp + w⊤ ξ) .

z,ξ|x∈Xmaj ,ξ∈Etest

By 14 -Lipschitzness of ψ, we then have |ψ(−γ) − ψ(−(1 ± ϵ)γ))| ≲ ϵγ. Combined with σ ′ (·) ≤ 1 and the above display, we obtain the two-sided inequality ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj ,ξ∈Etest

− ψ(−γ) ·

z,ξ|x∈Xmaj ,ξ∈Etest

′ ⊤ σ (w z + y(z)wsp + w⊤ ξ) ≲ ϵγ.

For the second term in Equation (19), we use ψ(·) ≤ 1 and σ ′ (·) ≤ 1 to obtain ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ) ≤ 1. E z,ξ|x∈Xmaj ,ξ∈E / test

(20)

(21)

We will now substitute Equation (20) and Equation (21) into Equation (19). Using the triangle inequality and Pξ [ξ ∈ / Etest ] ≤ d−C by Theorem F.4, we obtain E

ψ(−γ(x)) · σ ′ (w⊤ z + y(z)wsp + w⊤ ξ)

z,ξ|x∈Xmaj

− ψ(−γ) ·

z,ξ|x∈Xmaj ,ξ∈Etest

′ ⊤ σ (w z + y(z)wsp + w⊤ ξ) ≲ ϵγ + d−C .

(22)

An identical series of steps to the above follows for the minority group. Substituting the result into Equation (17) yields ⊤ − wsp ∇w Lρ − 2awsp ′ ⊤ (1 − λ) · ψ(−γ) · σ (w z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj ,ξ∈Etest

− λ · ψ(γ) ·

z,ξ|x∈Xmin ,ξ∈Etest

′ ⊤ σ (w z − y(z)wsp + w⊤ ξ) ≲ |a||wsp |(ϵγ + d−C ).

(23)

The remainder of the proof will simplify the expression on the left-hand side of Equation (23) to obtain the desired statement of the lemma. These steps will resemble corresponding steps in the proof of Theorem B.2. Using ψ(−γ) = 1 − ψ(γ) and the definition of σ, we obtain ′ ⊤ (1 − λ) · ψ(−γ) · σ (w z + y(z)wsp + w⊤ ξ) E z,ξ|x∈Xmaj ,ξ∈Etest

− λ · ψ(γ) ·

′ ⊤ σ (w z − y(z)wsp + w⊤ ξ)

z,ξ|x∈Xmin ,ξ∈Etest

= (1 − λ) · (1 − ψ(γ)) ·

w⊤ z + w⊤ ξ > −y(z)wsp

z,ξ|x∈Xmaj ,ξ∈Etest

− λ · ψ(γ) ·

w⊤ z + w⊤ ξ > y(z)wsp .

z,ξ|x∈Xmin ,ξ∈Etest

Note that ξ | ξ ∈ Etest is symmetric about 0 since ξ is symmetric and enters Etest only through the term |w⊤ ξ|, which is invariant under the mapping ξ 7→ −ξ. Then, following Equation (5) and using that both z and ξ | ξ ∈ Etest are symmetric about 0, we have w⊤ z + w⊤ ξ > −y(z)wsp − w⊤ z + w⊤ ξ > y(z)wsp P P z,ξ|ξ∈Etest

z,ξ|ξ∈Etest

⊤

w z + w ξ > −y(z)wsp − 1.

z,ξ|ξ∈Etest

Using that the conditional and unconditional distributions of (z, ξ) coincide according to Equation (1), the left-hand side of Equation (23) simplifies to 2awsp (1 − λ − ψ(γ)) · w⊤ z + w⊤ ξ > −y(z)wsp P z,ξ|ξ∈Etest + λ · ψ(γ) · (2 w⊤ z + w⊤ ξ > −y(z)wsp − 1) P z,ξ|ξ∈Etest

= 2awsp (1 − λ − ψ(γ) + 2λψ(γ)) ·

w⊤ z + w⊤ ξ > −y(z)wsp − λψ(γ)

z,ξ|ξ∈Etest

(24)

By the law of total probability, we have for any event A that / Etest ) · P(A | ξ ∈ / Etest ) P(A) = P(ξ ∈ Etest ) · P(A | ξ ∈ Etest ) + P(ξ ∈ ξ

= P(A | ξ ∈ Etest ) − P(ξ ∈ / Etest ) · P(A|ξ ∈ Etest ) + P(ξ ∈ / Etest ) · P(A | ξ ∈ / Etest ). ξ

Rearranging and taking absolute values on both sides then gives us / Etest ) ≤ d−C . P(A) − P(A | ξ ∈ Etest ) ≤ P(ξ ∈ ξ

(25)

By Equation (6), we have Pz,ξ w⊤ z + w⊤ ξ > −y(z)wsp = 21 + ε4w . Hence, by Equation (25) we have 1 εw ⊤ ⊤ w z + w ξ > −y(z)w − + ≤ d−C . P sp 2 4 z,ξ|ξ∈Etest Applying this to Equation (24), we finally have 2awsp (1 − λ − ψ(γ) + 2λψ(γ)) ·

w⊤ z + w⊤ ξ > −y(z)wsp − λψ(γ)

z,ξ|ξ∈Etest

− awsp (1 − λ − ψ(γ)) · 1 +

εw + λψ(γ)εw ≲ |a||wsp |d−C , 2

where the error term can be combined with the right-hand side of Equation (23). This completes the proof of the lemma. 37

Next, we will show the lemma for the w⊥ component, again under the margin concentration condition. Lemma C.6. Assume there exists γ > 0 such that ( (1 ± o(1)) · γ ∀x ∈ Xmaj satisfying Etest γ(x) = (−1 ± o(1)) · γ ∀x ∈ Xmin satisfying Etest ,

(26)

where the event Etest is defined in Definition F.1(vii). If all neurons (a, w) satisfy C1 ∥wsp ∥ ≥ ∥wsig ∥ + ∥wopp ∥ + ∥w⊥ ∥ log1/2 (d), then for any neuron (a, w) and any c < 1 we have r 1 ∥∇w⊥ Lρ ∥ ≲ e−cγ |a|∥w⊥ ∥ + P (|w⊤ ξ| ≥ |w⊤ z + w⊤ s|) + d1/2−C . |a| ξ Proof. By the dual norm characterization we have ∥∇w⊥ Lρ ∥ = sup |v ⊤ ∇w⊥ Lρ |. ∥v∥=1

Combining Equation (7) and Equation (13), we have ′ ⊤ ⊤ E ψ(−γ(x) · ay(x)σ (w x)v ξ .

∥∇w⊥ Lρ ∥ = 2 sup

∥v∥=1 x,ξ

Using that ξ is symmetric about zero, we have ∥∇w⊥ Lρ ∥ = sup

h ay(z)v ⊤ ξ

∥v∥=1 z,s,ξ

ψ(−γ(z + s + ξ)) · σ ′ (w⊤ (z + s + ξ)) − ψ(−γ(z + s − ξ)) · σ ′ (w⊤ (z + s − ξ))

Bringing the absolute value inside by Jensen’s inequality, and using |y(z)| = 1 and |ac − bd| ≤ |a − b| + |c − d| for 0 ≤ a, b, c, d ≤ 1, we have h ∥∇w⊥ Lρ ∥ ≤ |a| sup E |v ⊤ ξ| · ∥v∥=1 z,s,ξ

|ψ(−γ(z + s + ξ)) − ψ(−γ(z + s − ξ))| + |σ ′ (w⊤ (z + s + ξ)) − σ ′ (w⊤ (z + s − ξ))|

i .

S of active ReLUs are the same for any two points in Xmaj and any two points in Xmin that satisfy the event Etest . Therefore, for any ξ satisfying the event Etest , we have |f (z + s + ξ) − f (z + s − ξ)| 1 X 1 X = aσ(w⊤ (z0 + smaj + ξ)) − aσ(w⊤ (z0 + smaj − ξ)) p p (a,w)

= =

1 p 2 p

(a,w)

aw⊤ (z0 + smaj + ξ) −

(a,w)∈S

1 p

aw⊤ (z0 + smaj − ξ)

(a,w)∈S

|a||w⊤ ξ|.

(a,w)∈S

√ On the other hand, if ξ does not satisfy the event Etest then we can use |v ⊤ ξ| ≤ d and 0 ≤ ψ(·) ≤ 1 to find √ |v ⊤ ξ| · |ψ(−γ(z + s + ξ)) − ψ(−γ(z + s − ξ))| ≤ d. As in Theorem C.5, let us use the shorthand ξ ∈ Etest to denote that ξ satisfies the event Etest . Then we have ⊤ 1 2e−cγ X ∥∇w⊥ Lρ ∥ ≲ sup |a| · E |v ξ| · |w⊤ ξ| |a| p ξ|ξ∈Etest ∥v∥=1 (a,w)∈S + E v ⊤ ξ · 1 |w⊤ ξ| ≥ |w⊤ z + w⊤ s| + d1/2−C , ξ

(27)

C where the final d1/2−C follows in the case that ξ does not satisfy the event Etest , using Pξ [Etest ] ≤ d−C by d−3 ⊤ 2 2 Theorem F.4. For any q ∈ R we have Eξ [(q ξ) ] = ∥q∥ by isotropy of ξ. When we condition on Etest , this equality may not hold precisely, but we can use the following fact (similarly to Theorem C.5). Because Pξ (Etest ) ≥ 1 − d−C by Theorem F.4 and the distribution of ξ is uniform, we have ⊤ 2 E[(q ξ) ] = ξ

1 2d−3

(q ⊤ ξ)2

ξ∈{0,1}d−3

(q ⊤ ξ)2 +

ξ∈Etest

1 2d−3

⊤

(q ⊤ ξ)2

ξ∈E / test

≥ P(ξ ∈ Etest ) E [(q ξ) ] ξ|Etest

≥ (1 − d

−C

) E [(q ⊤ ξ)2 ], ξ|Etest

which then gives us ⊤ 2 E [(q ξ) ] ≤

ξ|Etest

Eξ [(q ⊤ ξ)2 ] ≲ E[(q ⊤ ξ)2 ] = ∥q∥2 . 1 − d−C ξ

For the first term in Equation (27), we have by the Cauchy-Schwarz inequality that r ⊤ r |v ξ| · |w⊤ ξ| ≤ E E [(v ⊤ ξ)2 ] · E [(w⊤ ξ)2 ] ≲ ∥v∥∥w⊥ ∥. ξ|ξ∈Etest

ξ|ξ∈Etest

Therefore, using |S| ≤ p we have 2e−cγ p ∥v∥=1 sup

X (a,w)∈S

|a| ·

⊤ 2e−cγ |v ξ| · |w⊤ ξ| ≲ sup p ξ|ξ∈Etest ∥v∥=1 E

X (a,w)∈S

|a|∥v∥∥w⊥ ∥ ≲ e−cγ |a|∥w⊥ ∥.

Then, for the second term in Equation (27), we have by the Cauchy-Schwarz inequality that r r sup E |v ⊤ ξ| · 1 |w⊤ ξ| ≥ |w⊤ z + w⊤ s| ≤ sup E [(v ⊤ ξ)2 ] · E [1 (|w⊤ ξ| ≥ |w⊤ z + w⊤ s|)] ∥v∥=1 ξ

∥v∥=1

= sup ∥v∥ ·

P (|w⊤ ξ| ≥ |w⊤ z + w⊤ s|) ξ

∥v∥=1

P (|w⊤ ξ| ≥ |w⊤ z + w⊤ s|). ξ

Putting it together with subadditivity of the supremum we have r 1 ∥∇w⊥ Lρ ∥ ≲ e−cγ |a|∥w⊥ ∥ + P (|w⊤ ξ| ≥ |w⊤ z + w⊤ s|) + d1/2−C . |a| ξ This completes the proof of the lemma. Finally, we repeat the analysis for any individual coordinate of w⊥ under the same margin concentration condition. Lemma C.7. Assume there exists γ > 0 such that ( (1 ± o(1)) · γ ∀x ∈ Xmaj satisfying Etest γ(x) = (−1 ± o(1)) · γ ∀x ∈ Xmin satisfying Etest ,

(28)

where the event Etest is defined in Definition F.1(vii). If all neurons (a, w) satisfy C1 ∥wsp ∥ ≥ ∥wsig ∥ + ∥wopp ∥ + ∥w⊥ ∥ log1/2 (d), then for any neuron (a, w), any c < 1, and any i > 3 we have −wi ∂wi Lρ ≲ |a||wi | e−cγ max |a||wi | + d−C + (a,w) √ √ ⊤ ⊤ P |wi | ≥ | 2∥wsig ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wsig ∥ − wsp + w ξ\i | ξ ξ √ √ ⊤ . + P |wi | ≥ | 2∥wopp ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wopp ∥ − wsp + w⊤ ξ\i | ξ

Proof. Combining Equation (9) and Equation (13), we have −wi ∂wi Lρ = 2 E ψ(−γ(x)) · ay(x)σ ′ (w⊤ x)wi xi = 2awi E ψ(−γ(x)) · y(x)σ ′ (w⊤ x)xi . x

Using that xi is symmetric about zero (it is a single coordinate of ξ), we have −wi ∂wi Lρ = awi ′ ⊤ E ψ(−γ(x\i + xi )) · y(x) · σ (w x\i + wi xi ) x

− E ψ(−γ(x\i − xi )) · y(x) · σ ′ (w⊤ x\i − wi xi ) . x

Recall from Theorem B.5 that by casework on σ ′ we have ′ ⊤ ′ ⊤ ⊤ E |σ (w x\i + wi xi ) − σ (w x\i − wi xi )| = P |wi | ≥ |w x\i | . x

for any c < 1 and ξ satisfying the event Etest (by definition, if ξ satisfies Etest then any one-bit-flip of ξ also satisfies Etest ). Under this same event, using the norm bound in the statement of the lemma, we have sgn(w⊤ x) = sgn(wsp x3 ) for all neurons (a, w) following Theorem D.1. Hence, the set S of active ReLUs are the same for any two points in Xmaj and any two points in Xmin that satisfy the event Etest . Therefore, for any ξ satisfying the event Etest , we have |f (x\i + xi ) − f (x\i − xi )| 1 X 1 X = aσ(w⊤ (x\i + xi )) − aσ(w⊤ (x\i − xi )) p p (a,w)

= =

1 p 2 p

(a,w)

aw⊤ (x\i + xi ) −

(a,w)∈S

1 p

aw⊤ (x\i − xi )

(a,w)∈S

awi xi

(a,w)∈S

≲ max |a||wi |. (a,w)

Using |ac − bd| ≤ |a − b| + |c − d| for 0 ≤ a, b, c, d ≤ 1, we then have ′ ⊤ ′ ⊤ E ψ(−γ(x\i + xi )) · y(x)σ (w x\i + wi xi ) − ψ(−γ(x\i − xi )) · y(x)σ (w x\i − wi xi ) x ≲ e−cγ max |a||wi | + P |wi | ≥ |w⊤ x\i | + d−C , (a,w)

C where the final d−C follows in the case that ξ does not satisfy the event Etest , using Pξ [Etest ] ≤ d−C by Theorem F.4. In particular, −wi ∂wi Lρ ≲ |a||wi | e−cγ max |a||wi | + P |wi | ≥ |w⊤ x\i | + d−C . (a,w)

We already simplified Pξ |wi | ≥ |w⊤ x\i | in Theorem B.5; substituting the result (but with all terms positive since there is no y(x) term), we obtain −wi ∂wi Lρ ≲ |a||wi | e−cγ max |a||wi | + d−C + (a,w) √ √ ⊤ ⊤ P |wi | ≥ | 2∥wsig ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wsig ∥ − wsp + w ξ\i | ξ ξ √ √ ⊤ + P |wi | ≥ | 2∥wopp ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wopp ∥ − wsp + w⊤ ξ\i | . ξ

This completes the proof of the lemma.

Phase I Induction

Recall that in Phase I, the neural network is small and hence the logistic loss function ℓρ is well-approximated by a first-order Taylor expansion about fρ = 0, i.e., ℓ0 (x) := −2 log( 12 ) − y(x)fρ (x). In this section, we perform an induction that holds throughout Phase I to characterize SGD feature learning in the first TI ≍ log log(d)η −1 iterations, where η ≪ 1 is the learning rate. We prove some preliminary lemmas, including our definition of the inductive hypothesis, in Section D.1. b ρ ) and the gradient We use these properties to bound deviation between the gradient of the minibatch loss (L of the population first-order approximation of the loss (L0 ) in Section D.2. Then, we show the inductive step in Section D.3. Throughout this section, we write ρ := ρ(t) as shorthand where appropriate, and we let C > 0 denote a sufficiently large constant which does not change from line to line. We will work in high probability under the events Einit , Etrain , and Etest established in Theorem F.1 and Theorem F.4.

D.1

Technical Preliminaries

In Phase I, we will show for all neurons (a, w) that the norm of the spurious component ∥wsp ∥ grows exponentially fast, while the norm of the signal component ∥wsig ∥, opposing component ∥wopp ∥, and orthogonal component ∥w⊥ ∥ remain close to their initialization. In order to lower bound the growth of ∥wsp ∥ we require sign alignment, i.e., sgn(a) = sgn(wsp ). To achieve this condition, we divide Phase I into two sub-phases (we omit the superscripts (t) for brevity, but include them in formal lemmas and proofs). During Phase Ia, sign alignment may or may not be satisfied, but by the end of Phase Ia we can guarantee that sgn(a) = sgn(wsp ) for all neurons (a, w) with high probability. We show that Phase Ia lasts TIa ≍ log1/2 (d)d−1/2 η −1 iterations. Then, Phase Ib lasts TIb ≍ log log(d)η −1 iterations, during which the spurious component grows very fast. By the end of Phase Ib, dominance of the spurious component wsp of every neuron is guaranteed (in the sense described informally above, and made formal by Theorem D.1). The ultimate goal of Phase I is to show that sgn(fρ (x)) = x3 with high probability; that is, the prediction is uniformly dominated by the shortcut spurious correlation. Our first lemma establishes a sufficient condition for this phenomenon. Recall that we define the “positive” neurons by S + := {(a, w) : sgn(a) = 1} and the “negative” neurons by S − := {(a, w) : sgn(a) = −1}. Lemma D.1. Suppose |S + |, |S − | > 0. Under the event Etest (Definition F.1(vii)), if all neurons (a, w) satisfy sgn(wsp ) = sgn(a) and C1 ∥wsp ∥ ≥ ∥wsig ∥ + ∥wopp ∥ + ∥w⊥ ∥ log1/2 (d), then sgn(fρ (x)) = x3 . ⊤ x| < C∥w⊥ ∥ log1/2 (d) for all neurons (a, w). Under this Proof. The definition of the event Etest is that |w⊥ event, the norm bound in the statement of the lemma gives

C ⊤ |wsp x3 | = ∥wsp ∥ ≥ C∥wsig ∥ + C∥wopp ∥ + C∥w⊥ ∥ log1/2 (d) > √ |w1 x1 + w2 x2 | + |w⊥ x|, 2 √ wherein choosing C > 2 implies sgn(w⊤ x) = sgn(wsp x3 ). Moreover, the sign alignment condition sgn(wsp ) = sgn(a) gives ( x3 sgn(wsp ) = x3 ⊤ sgn(aσ(w x)) = (29) 0 sgn(wsp ) ̸= x3 . Since |S + |, |S − | > 0 we may write pfρ (x) =

aσ(w⊤ x) +

X (a,w)∈S −

(a,w)∈S +

aσ(w⊤ x).

By Equation (29), if x3 = 1 then each term in the S + summation is positive while the entire S − summation is zero, telling us that sgn(fρ (x)) = 1 in this case. On the other hand if x3 = −1 then each term in the S − summation is negative while the entire S + summation is zero, telling us that sgn(fρ (x)) = −1 in this case. This completes the proof of the lemma. Throughout Phase I, we will require certain scaling factors to be satisfied. Definition D.2. We say the Phase I scalings are satisfied if the following conditions are met: (i) The learning rate log log(d)d−C ≪ η ≪ log−3 (d). (ii) The iteration t ≲ log log(d)η −1 . (iii) The width log(d) ≪ p ≪ dC . (iv) The initialization scale θ ≪ log−5C (d). (v) The batch size m ≫ d log7C (d). Remark D.3. For clarity, we list the limiting usage of each scaling (i.e., the result which requires the tightest application of each item in Theorem D.2): • Definition D.2(i) is limited by Lemma D.7(i). The upper bound helps with the balancedness condition |a| ≈ ∥w∥, while the lower bound ensures t ≲ dC . • Definition D.2(ii) is limited by Theorem D.6 and Theorem D.7. It is essentially the duration of Phase I. • Definition D.2(iii) is limited by Theorem F.2 — it ensures the event Einit occurs with high probability. • Definition D.2(iv) is limited by Theorem D.13. Throughout this section, the initialization scale θ acts as a free parameter which we can make sufficiently small. • Definition D.2(v) is limited by Theorem D.9, where it is used in Hoeffding’s inequality to concentrate the empirical gradient. We can now define the Phase I inductive hypothesis. Note that the Phase I “base case” is essentially given by the event Einit (Definition F.1(ii)). Definition D.4. A neural network fρ is said to obey the Phase I inductive hypothesis at iteration t ≥ 1 if the following conditions are met for all neurons (a(t) , w(t) ): (t) (t−1) (t−1) (i) wsp − wsp = η 1 ± o(log−2 (d)) · 12 − λ · sgn(a(0) )(|wsp | + θ). (t−1)

(t)

(t−1)

(t)

(t−1)

(t)

(t−1)

∥ ≲ ηθ log−2C (d).

(t)

(t−1)

∥∞ ≲ ηθ log3C (d)d−1/2 .

(ii) ∥wsig − wsig

∥ ≲ η(∥wsig

∥ + θ logC (d)d−1/2 ).

(t−1)

(iii) ∥wopp − wopp ∥ ≲ η(∥wopp ∥ + θ logC (d)d−1/2 ). (iv) ∥w⊥ − w⊥

(v) ∥w⊥ − w⊥

(vi) |a(t) | ≤ ∥w(t) ∥. Remark D.5. The inductive hypothesis defined above implies that each of ∥wsp ∥, ∥wsig ∥, and ∥wopp ∥ grow (at most) geometrically with additive factor, while ∥w⊥ ∥ only grows linearly. The additive factor for ∥wsp ∥, which is on the order θ, is extremely large — polynomially larger in d than the additive factor of ∥wsig ∥, which is on the order θd−1/2 . Moreover, the sign of wsp growth is modulated by sgn(a(0) ). Given these recurrences, we can sketch the proof that this inductive hypothesis holds throughout Phase I (Theorem D.15): Phase Ia lasts TIa ≍ log1/2 (d)d−1/2 η −1 iterations, after which sgn(a(0) ) = sgn(wsp ) for all neurons (a, w). Phase Ib lasts TIb ≍ log log(d)η −1 iterations, wherein the spurious feature grows quickly and ultimately ensures ∥wsp ∥ ≍ θ logC (d). The non-spurious weights grow by only a polylogarithmic factor during this time. 43

The next lemma controls the norms of all the neuron components during Phase I relative to the initialization scale θ assuming that the inductive hypothesis holds up until that point. Notice that the bounds hold for (0) (T ) any t ≤ TI ; for example, we have ∥w(0) ∥ ≍ ∥w⊥ ∥ but ∥w(TI ) ∥ ≍ ∥wsp I ∥, and Lemma D.6(v) holds in either (t) case. A second remark is that Lemma D.6(i) will eventually become tight such that ∥wsp ∥ ≍ θ logC (d), as we show in Theorem D.15. Lemma D.6. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Einit (Definition F.1(ii)), if the neural network fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then the following conditions hold on all neurons (a(t) , w(t) ): (t)

(i) ∥wsp ∥ ≲ θ logC (d). (t)

(t)

(ii) ∥wsig ∥, ∥wopp ∥ ≲ θ log2C (d)d−1/2 . (t)

(t)

(0)

(iii) ∥w⊥ ∥ ≍ θ with ∥w⊥ − w⊥ ∥ ≲ θ log−C (d).6 (t)

(iv) ∥w⊥ ∥∞ ≲ θ log4C (d)d−1/2 . (t)

(t)

(v) ∥w(t) ∥ ≍ ∥wsp ∥ + ∥w⊥ ∥ ≲ θ logC (d). (k)

Proof. For Lemma D.6(i), suppose sgn(a(0) ) = sgn(wsp ) for all k ≤ t to obtain an upper bound, as otherwise (t) ∥wsp ∥ will be strictly smaller.7 In this case, since Definition D.4(i) holds for all k ≤ t, we can apply Theorem F.15 with w = wsp , z = θ, µ = 21 − λ, and δ = log−2 (d). This gives t 1 −2 (0) (t) −λ · (∥wsp ∥wsp ∥ = 1 ± 3ηt log (d) · 1 + η ∥ + θ) − θ. 2 t Using t ≲ log log(d)η −1 we have ηt log−2 (d)) ≪ 1. Moreover, 1 + η 12 − λ ≤ logC (d) by the inequality (0) ez ≥ 1 + z. Under the event Einit (see Definition F.1) we have ∥wsp ∥ ≤ ∥w(0) ∥∞ ≲ θ log1/2 (d)d−1/2 . Hence, (t) ∥wsp ∥ ≲ θ(logC (d) − 1) ≲ θ logC (d). For Lemma D.6(ii), since Definition D.4(ii) holds for all k ≤ t, we can apply the upper bound of Theorem F.15 with w = wsig , z = θ logC (d)d−1/2 , δ = 0, and µ = c0 for a constant c0 > 0 resulting from the ≲ in Definition D.4(ii). This gives (t)

(0)

∥wsig ∥ ≤ (1 + c0 η)t (∥wsig ∥ + θ logC (d)d−1/2 ) − θ logC (d)d−1/2 . (0)

Using t ≲ log log(d)η −1 , we have (1+c0 η)t ≤ logC (d). Under the event Einit we have ∥wsig ∥ ≲ θ log1/2 (d)d−1/2 . (t)

Hence, ∥wsig ∥ ≲ θ logC (d)d−1/2 (logC (d) − 1) ≲ θ log2C (d)d−1/2 . The result for wopp follows in the exact same way. (t) (0) For Lemma D.6(iii), since Definition D.4(iv) holds for all k ≤ t, we have both ∥w⊥ ∥ ≲ ∥w⊥ ∥ + (t) (0) tηθ log−2C (d) and ∥w⊥ ∥ ≳ ∥w⊥ ∥ − tηθ log−2C (d). Using t ≲ log log(d)η −1 we find tηθ log−2C (d) ≲ (0) (t) θ log−C (d), and under the event Einit we have ∥w⊥ ∥ ≍ θ. Hence, we have both ∥w⊥ ∥ ≲ θ + θ log−C (d) and (t) (t) (t) (0) ∥w⊥ ∥ ≳ θ − θ log−C (d); it follows that ∥w⊥ ∥ ≍ θ with ∥w⊥ − w⊥ ∥ ≲ θ log−C (d). For Lemma D.6(iv), since Definition D.4(v) holds for all k ≤ t, we can apply the triangle inequality t (t) (0) times under the event Einit to obtain ∥w⊥ ∥∞ ≲ ∥w⊥ ∥∞ + tηθ log3C (d)d−1/2 ≲ θ log4C (d)d−1/2 . (t) (t) (t) (t) For Lemma D.6(v), by the triangle inequality we have ∥w(t) ∥ ≤ ∥wsig ∥ + ∥wopp ∥ + ∥wsp ∥ + ∥w⊥ ∥. (t)

(t)

By Lemma D.6(ii) and Lemma D.6(iii) we have ∥wsig ∥, ∥wopp ∥ ≪ ∥w⊥ ∥, which implies the upper bound (t)

(t)

∥w(t) ∥ ≲ ∥wsp ∥ + ∥w⊥ ∥. We then have the lower bound by ∥w(t) ∥ ≥ ∥wsp + w⊥ ∥ ≥ √12 (∥wsp ∥ + ∥w⊥ ∥). Thus, we have proved all the parts of the lemma. 6 The latter condition is important: it corresponds to the requirement on ∆ in Theorem F.9. 7 In Theorem D.15 we will show that this alignment occurs naturally by the end of Phase Ia.

Next, we present the partner lemma to Theorem D.6 which controls behavior of the scalar weight a for any neuron (a, w). It requires minibatch concentration and thus holds over the event Etrain . Lemma D.7. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain (Definition F.1(v)), if the neural network fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then the following conditions hold on all neurons (a(t) , w(t) ): (t) (t) (i) |a(t) | = 1 ± o(log−2 (d)) · ∥w(t) ∥ = 1 ± o(log−2 (d)) · (∥wsp ∥ + ∥w⊥ ∥) ≲ θ logC (d). (ii) sgn(a(t) ) = sgn(a(0) ). (iii) E(a,w)∼ρ [∥a(t) w(t) ∥] ≲ θ2 log2C (d). Proof. For Lemma D.7(i), for the upper bound we have |a(t) | ≤ ∥w(t) ∥ by Definition D.4(vi). For the lower bound, we denote as shorthand E(t) := ∥w(t) ∥2 − (a(t) )2 and note that E(0) = 0 by definition. Under the event Etrain , by Lemma F.13(iv) we have E(t) ≤ 10η 2 (a(t) )2 + E(t − 1) for all t ≪ dC iterations. Using (k) Definition D.4(vi), Lemma D.6(iii), and Lemma D.6(v), we have by strong induction that (a(k) )2 ≲ ∥wsp ∥2 +θ2 for all k ≤ t. In particular, t−1 X

E(t) ≤

(k) 2 10η 2 (a(k) )2 ≲ tη 2 · (max ∥wsp ∥ + θ2 ). k<t

k=1

(0)

where the last inequality uses the inductive step. Under the event Etrain , we have ∥wsp ∥ ≲ θ log1/2 (d)d−1/2 . (k) (k) Moreover, by Definition D.4(i), we have that ∥wsp ∥ is monotonically increasing if sgn(wsp ) = sgn(a(0) ) and monotonically decreasing otherwise (recall that sgn(wsp ) may flip). Thus, we can write (30)

(k) 2 (t) 2 max ∥wsp ∥ ≲ max(θ2 log(d)d−1 , ∥wsp ∥ ). k<t

If the maximum in Equation (30) is achieved by θ2 log(d)d−1 we have E(t) ≪ tη 2 θ2 ≲ tη 2 ∥w(t) ∥2 by (t) Lemma D.6(iii). On the other hand, if the maximum in Equation (30) is achieved by ∥wsp ∥2 we have (t) E(t) ≪ tη 2 (∥wsp ∥2 + θ2 ) ≲ tη 2 ∥w(t) ∥2 by Lemma D.6(iii) and orthogonality of wsp and w⊥ . In either case, applying t ≲ log log(d)η −1 and η ≪ log−3 (d) we have (a(t) )2 = ∥w(t) ∥2 − E(t) = 1 − O(tη 2 ) · ∥w(t) ∥2 = 1 − o(log−2 (d)) · ∥w(t) ∥2 , so we have ultimately shown the desired lower bound |a(t) | = 1 − o(log−2 (d)) · ∥w(t) ∥.8 The remainder of the result follows from Lemma D.6(v). For Lemma D.7(ii), assume for a contradiction that sgn(a(t) ) ̸= sgn(a(0) ). Then there exists some iteration k < t where its sign flips, i.e., |a(k) − a(k+1) | = |a(k) | + |a(k+1) | ≳ ∥w(k) ∥ + ∥w(k+1) ∥ ≥ ∥w(k) ∥,

(31)

|a(k) − a(k+1) | ≲ η∥w(k) ∥ ≪ ∥w(k) ∥. 8 Here we used the Taylor expansion

√

1 − x = 1 − x/2 − O(x2 ) for x ≪ 1 such that

1 − o(log−2 (d)) = 1 − o(log−2 (d)).

Therefore, Equation (31) and Equation (32) are contradictory. This implies sgn(a(t) ) = sgn(a(0) ). For Lemma D.7(iii) we have by Lemma D.6(v) and Lemma D.7(i) that p

[∥a(t) w(t) ∥] =

(a,w)∼ρ

1X ∥aj wj ∥ ≲ θ2 log2C (d). p j=1

Thus, we have proved all parts of the lemma.

D.2

bρ − ∇L0 Error Analysis Phase I ∇L

b ρ − ∇L0 ∥ is sufficiently small, justifying In this section, we will leverage Phase I properties to show that ∥∇L the L0 approximation. First, we will show an extension of Theorem C.3 that uses the central limit theorem and the Phase I inductive hypothesis to control the approximation error of the population gradient, ∥∇Lρ − ∇L0 ∥. This result is related to Glasgow, 2024, Lemma C.17, though it depends on the leave-two-out analysis we developed in Theorem C.3. Lemma D.8. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have  3 θ log6C (d)d−1/2 + θ logC (d)d−C i < 3   |∂w(t) Lρ − ∂w(t) L0 | ≲ θ3 log4C (d) + θ logC (d)d−C i=3 i i   3 8C C −1/2 −C θ log (d)d + θ log (d)d i>3 for all neurons (a(t) , w(t) ). Proof. Let us omit (t) superscripts for clarity and begin with the case i < 3. Recall that we √write x := z +s+ξ and w1:2 := wsig + wopp . Using the law of total probability and |w⊤ z| = ∥z∥∥wsig ∥ = 2∥w1:2 ∥, we have √ ⊤ ⊤ ⊤ ⊤ ⊤ P |w ξ + w s| < |w z| ≤ E P |w ξ + w s| < 2∥w1:2 ∥ | s . s

(0)

(t)

(0)

Let √ us apply Theorem F.9 to the right-hand side with v (0) = w⊥ , ∆ = w⊥ − w⊥ , µ = −w⊤ s, and 1 k = 2∥w1:2 ∥. Under the event Etrain , the condition on v (0) is satisfied; we use LemmaD.6(iii) and C > 2 for the condition on ∆, and Lemma D.6(ii) for the condition on k. Upper bounding exp

−µ2 Cθ 2

≤ 1, we have

∥w1:2 ∥ 1 ⊤ ⊤ ⊤ +√ . P |w ξ + w s| < |w z| ≲ θ x d By Lemma D.6(ii), we have ∥w1:2 ∥ ≲ θ log2C (d)d−1/2 . Hence, 2C ⊤ ⊤ ⊤ −1/2 + d−1/2 ≲ log2C (d)d−1/2 . P |w ξ + w s| < |w z| ≲ log (d)d

(33)

By Lemma D.7(i) and Lemma D.7(iii) we have |a| ≲ θ logC (d) and E(a,w)∼ρ [∥aw∥] ≲ θ2 log2C (d). Together with Equation (33), we can apply Theorem C.3 as follows: |∂wi Lρ − ∂wi L0 | ⊤ ⊤ ⊤ ≲ |a| E [|aw1 | + |aw2 |] + log(d) · P |w ξ + w s| < |w z| · (a,w)∼ρ

≲ θ logC (d) θ2 log3C (d)d−1/2 + θ2 log4C+1 (d)d−1/2 + d−C ≲ θ3 log6C (d)d−1/2 + θ logC (d)d−C , 46

[∥aw∥] + d

(a,w)∼ρ

−C

where we used C > 1. This completes the proof for the case i < 3. Now, consider the case i = 3. By Lemma D.7(i) and Lemma D.7(iii) we have |a| ≲ θ logC (d) and 2C E(a,w)∼ρ [∥aw∥] ≲ θ2 log (d). Hence, we can apply Theorem C.2 as follows: |∂wi Lρ − ∂wi L0 | ≲ |a| log(d) · E [∥aw∥] + d−C (a,w)∼ρ

≲ θ logC (d) θ2 log2C+1 (d) + d−C

≲ θ3 log4C (d) + θ logC (d)d−C , where we used C > 1. This completes the proof for i = 3. Finally, consider the case i > 3 and recall x\i := x − xi ei . Using the law of total probability, we have ⊤ ⊤ ⊤ ⊤ |w x | < |w | = |w ξ + w z + w s| < |w | | z, s . P E P i i \i \i x

z,s

(0)

(t)

(0)

Let us apply Theorem F.9 to the right-hand side with v (0) = w⊥\i , ∆ = w⊥\i − w⊥\i , µ = −w⊤ z − w⊤ s, 1 and k = |wi |. Under the event Etrain , the condition on v (0) is satisfied; we use Lemma D.6(iii) and C > 2 for

the condition on ∆, and Lemma D.6(iv) for the condition on k.9 Upper bounding exp

−µ2 Cθ 2

≤ 1, we have

|wi | 1 ⊤ +√ . P |w x\i | < |wi | ≲ θ x d By Lemma D.6(iv), we have ∥w⊥ ∥∞ ≲ θ log4C (d)d−1/2 . Hence, 4C ⊤ −1/2 + d−1/2 ≲ log4C (d)d−1/2 . P |w x\i | < |wi | ≲ log (d)d x

(34)

By Lemma D.7(i) and Lemma D.7(iii) we have |a| ≲ θ logC (d) and E(a,w)∼ρ [∥aw∥] ≲ θ2 log2C (d). Together with Equation (34), we can apply Theorem C.3 as follows: ⊤ −C |∂wi Lρ − ∂wi L0 | ≲ |a| E [|awi |] + log(d) · P |w x\i | < |wi | · E [∥aw∥] + d x

(a,w)∼ρ

≲ θ log (d) θ log

−1/2

(d)d

(a,w)∼ρ

+ θ log

6C+1

−1/2

(d)d

+ d−C

≲ θ3 log8C (d)d−1/2 + θ logC (d)d−C , where we used C > 1. This completes the proof of the lemma. b ρ − ∇w L0 , which is what We can now obtain general bounds on the magnitude of each component of ∇w L we will ultimately need. Lemma D.9. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have  3 6C −1/2 + θ log−2C (d)d−1/2 i < 3 θ log (d)d  b ρ − ∂ (t) L0 | ≲ θ3 log4C (d) + θ log−2C (d)d−1/2 |∂w(t) L i=3 wi i   3 8C −2C −1/2 −1/2 θ log (d)d + θ log (d)d i > 3. for all neurons (a(t) , w(t) ). 9 While Theorem F.2 and Lemma D.6(iii) are stated for w , it is straightforward to see they hold for w ⊥ ⊥\i .

Proof. Let us omit (t) superscripts for clarity. By the triangle inequality we have for any i ∈ [d] that b ρ − ∇w L0 | ≤ |∂w L b ρ − ∂w Lρ | + |∂w Lρ − ∂w L0 |. |∂wi L i i i i i By Lemma D.7(i) we have |a| ≲ θ logC (d). Under the event Etrain , by Lemma F.3(i) with m ≫ d log7C (d) and C > 2, we have b ρ − ∂w Lρ | ≪ |a(t) | log−3C d−1/2 ≲ θ log−2C (d)d−1/2 . |∂wi L i The result then follows directly from Theorem D.8.

D.3

Phase I Inductive Step

In this section, we show the inductive steps for Theorem D.4 and ultimately obtain the Phase I result in Theorem D.15. We first characterize the growth of wsp , constituting the inductive step for Definition D.4(i). Lemma D.10. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain , if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have 1 (t+1) (t) (t) wsp − wsp = η 1 ± o(log−2 (d)) · − λ · sgn(a(0) )(|wsp | + θ). 2 for all neurons (a(t+1) , w(t+1) ). Proof. We first compute the order of the intermediate term εw(t) defined in Theorem B.2. By Theorem B.7, we have ! √ (t) (t) (t) (t) ∥w1:2 ∥ −∥wsp ∥2 + 2 2∥w1:2 ∥∥wsp ∥ ∥w⊥ ∥33 |εw(t) | ≲ exp + (t) (t) ∥w⊥ ∥3 ∥w⊥ ∥ 2∥w⊥ ∥2 ! √ (t) (t) (t) ∥w1:2 ∥ ∥w⊥ ∥∞ 2 2∥w1:2 ∥∥wsp ∥ ≤ exp + , (t) (t) 2 ∥w⊥ ∥ ∥w ∥ 2∥w ∥ ⊥

⊥

(t) where we used the basic inequalities −∥wsp ∥2

(t)

≤ 0 and ∥w⊥ ∥33 ≤ ∥w⊥ ∥∞ ∥w⊥ ∥2 . By Lemma D.6(i), (t) (t) Lemma D.6(ii), Lemma D.6(iii), and Lemma D.6(iv) we have ∥wsp ∥ ≲ θ logC (d), ∥w1:2 ∥ ≲ θ log2C (d)d−1/2 , (t) (t) ∥w⊥ ∥ ≍ θ, and ∥w⊥ ∥∞ ≲ θ log4C (d)d−1/2 . Using eu ≲ 1 + u for 0 < u ≪ 1,10 we have |εw(t) | ≲ log2C (d)d−1/2 · exp(C log3C (d)d−1/2 ) + log4C (d)d−1/2 ≲ log2C (d)d−1/2 · (1 + log3C (d)d−1/2 ) + log4C (d)d−1/2 (35)

≲ log5C (d)d−1/2 . By the definition of gradient descent, we have (t+1) (t) bρ wsp = wsp − η∂w(t) L sp

(t)⊤

(t) = wsp −η

bρ wsp ∇w(t) L (t)

wsp

(t)⊤ (t)⊤ b ρ − ∇w(t) L0 ) wsp ∇w(t) L0 + wsp (∇w(t) L

By Theorem B.2 we have (t)⊤ −wsp ∇w(t) L0 =

(t)

wsp

1 (t) (t) ε (t) a wsp · 1 + w − 2λ . 2 2

10 This can be seen via the following Taylor expansion: eu ≤ 1 + u + Cu2 ≲ 1 + u for 0 < u ≪ 1.

(36)

Using |εw(t) | ≲ log5C (d)d−1/2 by Equation (35) and recalling that we assumed 1/2 − λ > 0, we obtain11 1 −2 (t)⊤ (t) −wsp ∇w(t) L0 = 1 ± o(log (d)) · − λ · a(t) wsp . 2 (t) Moreover, by Lemma D.6(iii) and Lemma D.7(i) we have |a(t) | = 1 ± o(log−2 (d)) · (|wsp | + θ), implying that 1 −2 (t)⊤ (t) (t) (t) −wsp ∇w(t) L0 = 1 ± o(log (d)) · − λ · sgn(a(t) )sgn(wsp ) · (|wsp | + θ)|wsp |. (37) 2 Next, by the Cauchy-Schwarz inequality and Theorem D.9 with i = 3, we have (t)⊤ (t) b ρ − ∇w(t) L0 )| ≤ |wsp b ρ − ∇ (t) L0 ∥ |wsp (∇w(t) L |∥∇w(t) L w sp

(t) ≲ |wsp |(θ3 log4C (d) + θ log−2C (d)d−1/2 ).

Using θ ≪ log−3C (d) we have θ3 log4C (d) ≪ θ log−2 (d). We also have θ log−2C (d)d−1/2 ≪ θ log−2 (d). Hence, (t) (t)⊤ b ρ − ∇w(t) L0 )| ≪ θ|wsp |wsp (∇w(t) L |,

(38)

in particular, Equation (38) is dominated by Equation (37). Returning to Equation (36), we then have η 1 −2 (t+1) (t) (t) (t) (t) wsp − wsp = (t) 1 ± o(log (d)) · − λ · sgn(a(t) )sgn(wsp ) · (|wsp | + θ)|wsp | 2 wsp 1 (t) = η 1 ± o(log−2 (d)) · − λ · sgn(a(t) )(|wsp | + θ) 2 Finally, by Lemma D.7(ii) we have sgn(a(t) ) = sgn(a(0) ). Hence, 1 −2 (t) (t+1) (t) − λ · sgn(a(0) )(|wsp | + θ). wsp − wsp = η 1 ± o(log (d)) · 2 This completes the proof of the lemma. We will now upper bound the growth of wsig , constituting the inductive step for Definition D.4(ii). Lemma D.11. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have (t+1) (t) (t) ∥wsig − wsig ∥ ≲ η∥wsig ∥ + ηθ logC (d)d−1/2 for all neurons (a(t+1) , w(t+1) ). Proof. By the definition of gradient descent, we have (t+1)

wsig

(t)

bρ = wsig − η∇w(t) L sig

(t)⊤ bρ wsig ∇w(t) L (t) (t) wsig = wsig − η (t) 2 ∥wsig ∥ (t)⊤ (t)⊤ b ρ − ∇w(t) L0 ) wsig ∇w(t) L0 + wsig (∇w(t) L (t) (t) = wsig − η wsig . (t) 2 ∥wsig ∥ 11 This is also where we use the upper bound λ ∈ (0, Λ] for a constant Λ < 1 , i.e., we cannot have λ → 1 . 2 2

(39)

By Theorem B.1, we have √ (t)⊤ |wsig ∇w(t) L0 | =

√ 2 (t) (t) (t) |a |∥wsig ∥ · P |w(t)⊤ ξ + w(t)⊤ e3 | ≤ 2∥wsig ∥ . 4 ξ

√ (0) (t) (0) (t) Let us apply Theorem F.9 with v (0) = w⊥ , ∆ = w⊥ − w⊥ , µ = −w(t)⊤ e3 , and k = 2∥wsig ∥. Under the event Etrain , the condition on v (0) is satisfied; we use Lemma D.6(iii) and C > 12 for the condition on ∆, and (t) Lemma D.6(ii) for the condition on k. Using |w(t)⊤ e3 | = ∥wsp ∥, we have ! ! (t) (t) ∥wsig ∥ −∥wsp ∥2 1 (t)⊤ (t) (t) |wsig ∇w(t) L0 | ≲ |a |∥wsig ∥ exp +√ . (40) θ Cθ2 d 2 q C = 2e which is a constant. In particular, Now, note that maxy,z∈R yz exp −y Cz 2 (t)

Moreover,

(t)

∥wsig ∥ (t) ∥wsp ∥ exp θ

−∥wsp ∥2 Cθ2

(t) (t) ∥wsig ∥ exp ∥w⊥ ∥

−∥wsp ∥2 Cθ2

(t)

! (t)

(41)

(t)

(42)

≲ ∥wsig ∥.

! ≲ ∥wsig ∥,

(t) 2 −∥wsp ∥ (t) (t) where we used exp ≤ 1. By Lemma D.7(i), we have |a(t) | ≍ ∥wsp ∥ + ∥w⊥ ∥ ≲ θ logC (d). Cθ 2 Combining Equation (40), Equation (41), and Equation (42) we have (t)⊤

(t)

(43)

|wsig ∇w(t) L0 | ≲ ∥wsig ∥2 + ∥wsig ∥θ logC (d)d−1/2 . Next, by the Cauchy-Schwarz inequality and Theorem D.9 with i < 3, we have (t)⊤

(t)

b ρ − ∇w(t) L0 )| ≤ ∥w ∥ · ∥∇ (t) L b ρ − ∇ (t) L0 ∥ |wsig (∇w(t) L sig w w sig

sig

(t) ≲ ∥wsig ∥ · (θ3 log6C (d)d−1/2 + θ log−2C (d)d−1/2 ) (t)

(44)

≲ ∥wsig ∥ · θd−1/2 ,

where we used θ ≪ log−3C (d). Returning to Equation (39) and combining Equation (43) and Equation (44), we have (t)

(t+1)

∥wsig

(t)

− wsig ∥ ≲ η

∥wsig ∥ + θ logC (d)d−1/2 + θd−1/2 (t) ∥wsig ∥

(t)

∥wsig ∥

(t)

≲ η∥wsig ∥ + ηθ logC (d)d−1/2 . This completes the proof of the lemma. Similarly, we can upper bound the growth of wopp , constituting the inductive step for Definition D.4(iii). Lemma D.12. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have (t+1) (t) (t) ∥wopp − wopp ∥ ≲ η∥wopp ∥ + ηθ logC (d)d−1/2

for all neurons (a(t+1) , w(t+1) ). 50

Proof. By the definition of gradient descent, we have (t+1) (t) bρ wopp = wopp − η∇w(t) L opp

(t)⊤ bρ wopp ∇w(t) L (t) (t) wopp = wopp −η (t) 2 ∥wopp ∥ (t) = wopp −η

(t)⊤ (t)⊤ b ρ − ∇w(t) L0 ) wopp ∇w(t) L0 + wopp (∇w(t) L (t) ∥wopp ∥2

(t) wopp .

(45)

By Theorem B.1, we have √ (t)⊤ |wopp ∇w(t) L0 | =

√ 2 (t) (t) (t) ∥ . |a |∥wopp ∥ · P |w(t)⊤ ξ + w(t)⊤ e3 | ≤ 2∥wopp 4 ξ

Following the same analysis of Theorem D.11, we obtain (t)⊤ (t) 2 (t) |wopp ∇w(t) L0 | ≲ ∥wopp ∥ + ∥wopp ∥θ logC (d)d−1/2 .

(46)

Next, by the Cauchy-Schwarz inequality and Theorem D.9 with i < 3, we have (t)⊤ (t) b ρ − ∇ (t) L0 ∥ b ρ − ∇w(t) L0 )| ≤ ∥wopp |wopp (∇w(t) L ∥ · ∥∇w(t) L w opp

opp

(t) ≲ ∥wopp ∥ · (θ3 log6C (d)d−1/2 + θ log−2C (d)d−1/2 )

(47)

(t) ≲ ∥wopp ∥ · θd−1/2 ,

where we used θ ≪ log−3C (d). Returning to Equation (45) and combining Equation (46) and Equation (47), we have (t)

(t+1) (t) ∥wopp − wopp ∥≲η

∥wopp ∥ + θ logC (d)d−1/2 + θd−1/2 (t)

∥wopp ∥

(t) ∥wopp ∥

(t) ≲ η∥wopp ∥ + ηθ logC (d)d−1/2 .

This completes the proof of the lemma. Next, we can upper bound the growth of w⊥ , constituting the inductive step for Definition D.4(iv). Lemma D.13. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain , if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have (t+1)

∥w⊥

(t)

− w⊥ ∥ ≲ ηθ log−2C (d).

for all neurons (a(t+1) , w(t+1) ). Proof. By the definition of gradient descent, we have (t+1)

w⊥

(t) bρ = w⊥ − η∇w(t) L ⊥

(t) b ρ − ∇ (t) L0 ). = w⊥ − η∇w(t) L0 − η(∇w(t) L w⊥ ⊥ ⊥

Note that

∇w(t) L0 = Projw(t) ∇L0 + ProjV (t) ∇L0 , ⊥

⊥

(48)

(t)

where V (t) := {v ∈ Rd : v ⊤ w⊥ = 0} is the orthogonal complement of w⊥ . By Theorem B.4 we have12 h 1 ∇w(t) L0 = − |a(t) | E ξ · sgn(w(t)⊤ ξ) ⊥ 8 ξ √ √ (t) (t) (t) (t) ∥ + wsp 1 |w(t)⊤ ξ| ≥ | 2∥wsig | + 1 |w(t)⊤ ξ| ≥ | 2∥wsig ∥ − wsp | i √ √ (t) (t) (t) (t) − 1 |w(t)⊤ ξ| ≥ | 2∥wopp ∥ + wsp | − 1 |w(t)⊤ ξ| ≥ | 2∥wopp ∥ − wsp | . (49) Consider the term

i h √ (t) (t) | . q := E ξ · sgn(w(t)⊤ ξ) · 1 |w(t)⊤ ξ| ≥ | 2∥wsig ∥ + wsp ξ

√ (t) (t) Let us apply Theorem F.11 to q with κ = | 2∥wsig ∥ + wsp | to find that ∥ProjV (t) q∥ ≲

(t)

∥w⊥ ∥∞ (t)

∥w⊥ ∥

1/7

Repeating this procedure for each term in Equation (49) and combining via the triangle inequality, we have (t)

∥w⊥ ∥∞

(t)

∥ProjV (t) ∇L0 ∥ ≲ |a |

!1/7 .

(t)

∥w⊥ ∥

(t)

By Lemma D.6(iii), Lemma D.6(iv), and Lemma D.7(i), we have ∥w⊥ ∥ ≍ θ, ∥w⊥ ∥∞ ≲ θ log4C (d)d−1/2 , and |a(t) | ≲ θ logC (d). Thus, we obtain ∥ProjV (t) ∇L0 ∥ ≲ θ log11C/7 (d)d−1/14 . Now let us analyze the term

(t)⊤

Projw(t) ∇L0 =

w⊥

∇w(t) L0 (t)

∥w⊥ ∥2

⊥

(50)

(t)

w⊥ .

By Theorem B.4 we have (t)⊤

−w⊥

1 (t) h (t)⊤ |a | E |w ξ| 8 ξ √ √ (t) (t) (t) (t) ∥ + wsp | + 1 |w(t)⊤ ξ| ≥ | 2∥wsig ∥ − wsp | 1 |w(t)⊤ ξ| ≥ | 2∥wsig i √ √ (t) (t) (t) (t) − 1 |w(t)⊤ ξ| ≥ | 2∥wopp ∥ + wsp | − 1 |w(t)⊤ ξ| ≥ | 2∥wopp ∥ − wsp | .

∇w(t) L0 =

(51)

Recalling the application of Lemma D.6(iii) and Lemma D.6(iv) above, we have ! (t) ∥w⊥ ∥ d1/2 (t) 4C 1/2 1/2 −1/2 ≲ θ log (d)d · log ∥w⊥ ∥∞ log (t) log4C (d) ∥w⊥ ∥∞ ≲ θ log5C (d)d−1/2 ,

(52)

where we used that the function x log1/2 ( xy ) is increasing in x for all x ≲ y. Define Ψ(k) :=

(t)

[|G|1(|G| ≥ k)]

G∼N (0,∥w⊥ ∥2 )

for k ≥ 0, and consider the term h i √ (t) (t) r := E |w(t)⊤ ξ| · 1 |w(t)⊤ ξ| ≥ | 2∥wsig ∥ + wsp | . ξ

12 This is not the exact form of Theorem B.4, but is straightforward to see by following the proof without the w ⊥ multiplied on the outside.

√ (t) (t) Let us apply Theorem F.10 to r with v = w(t) and k = | 2∥wsig ∥ + wsp | to find that √ (t) (t) r − Ψ | 2∥wsig ∥ + wsp | ≲ θ log5C (d)d−1/2 ,

(53)

where we used Equation (52). Applying this procedure to each term in Equation (51) gives us Ψ-approximations for each truncated moment. Now, note that ! ! r (t) 2 2 2 k − ∥w ∥ 2 −k ⊥ Ψ′′ (k) = exp , (t) (t) π ∥w ∥3 2∥w ∥2 ⊥

⊥

such that sup |Ψ′′ (k)| ≲ k≥0

1 (t) ∥w⊥ ∥

≲

1 , θ

where the last inequality follows by Lemma D.6(iii) again. Using Taylor’s theorem to linearize the Ψapproximations of each term in Equation (51), we obtain √ √ (t) (t) (t) (t) (t) (t) (t) | − Ψ(|wsp ) · Ψ′ (|wsp |) ≲ ∥wsig ∥2 θ−1 Ψ | 2∥wsig ∥ + wsp |) + 2∥wsig ∥ · sgn(wsp √ √ (t) (t) (t) (t) (t) (t) (t) Ψ | 2∥wsig ∥ − wsp | − Ψ(|wsp |) − 2∥wsig ∥ · sgn(wsp ) · Ψ′ (|wsp |) ≲ ∥wsig ∥2 θ−1 √ √ (t) (t) (t) (t) (t) (t) (t) 2 −1 Ψ | 2∥wopp ∥ + wsp | − Ψ(|wsp |) + 2∥wopp ∥ · sgn(wsp ) · Ψ′ (|wsp |) ≲ ∥wopp ∥ θ √ √ (t) (t) (t) (t) (t) (t) (t) 2 −1 Ψ | 2∥wopp ∥ − wsp | − Ψ(|wsp |) − 2∥wopp ∥ · sgn(wsp ) · Ψ′ (|wsp |) ≲ ∥wopp ∥ θ . We will use these inequalities to show that the sum of the Ψ-approximations of each term in Equation (51) is extremely small. We have √ √ √ √ (t) (t) (t) (t) (t) (t) (t) (t) | + Ψ | 2∥wsig ∥ − wsp | −Ψ Ψ | 2∥wsig ∥ + wsp 2∥wopp ∥ + wsp | − Ψ | 2∥wopp ∥ − wsp | √ i h √ (t) (t) (t) (t) (t) | + Ψ | 2∥wsig ∥ − wsp | − 2Ψ(|wsp |) = Ψ | 2∥wsig ∥ + wsp h √ √ i (t) (t) (t) (t) (t) − Ψ | 2∥wopp ∥ + wsp | + Ψ | 2∥wopp ∥ − wsp | − 2Ψ(|wsp |) √ √ (t) (t) (t) (t) (t) ≤ Ψ | 2∥wsig ∥ + wsp | + Ψ | 2∥wsig ∥ − wsp | − 2Ψ(|wsp |) √ √ (t) (t) (t) (t) (t) + Ψ | 2∥wopp ∥ + wsp | + Ψ | 2∥wopp ∥ − wsp | − 2Ψ(|wsp |) . We then have by the linearization inequalities above that √ √ (t) (t) (t) (t) (t) (t) Ψ | 2∥wsig ∥ + wsp | + Ψ | 2∥wsig ∥ − wsp | − 2Ψ(|wsp |) ≲ ∥wsig ∥2 θ−1 ≲ θ log4C (d)d−1 √ √ (t) (t) (t) (t) (t) (t) 2 −1 Ψ | 2∥wopp ∥ + wsp | + Ψ | 2∥wopp ∥ − wsp | − 2Ψ(|wsp |) ≲ ∥wopp ∥ θ ≲ θ log4C (d)d−1 , (t)

(t)

where the first-order terms precisely cancel due to opposing signs, and we used ∥wsig ∥, ∥wopp ∥ ≲ θ log2C (d)d−1/2 by Lemma D.6(ii). Combining this result with Equation (51) and Equation (53) we finally obtain (t)⊤

−w⊥

∇w(t) L0 ≲ |a(t) | · (θ log4C (d)d−1 + θ log5C (d)d−1/2 ) ≲ θ2 log6C (d)d−1/2 ,

(54)

where the last inequality follows by Lemma D.7(i) again. Next, by the Cauchy-Schwarz inequality and

Theorem D.9 with i > 3, we have (t)⊤

|w⊥

b ρ − ∇w(t) L0 )| ≤ ∥w(t) ∥ · ∥∇ (t) L b ρ − ∇ (t) L0 ∥ (∇w(t) L ⊥ w⊥ w⊥ √ (t) b ρ − ∂ (t) L0 | ≲ ∥w⊥ ∥ · d · max |∂w(t) L w i>3

(t) ≲ ∥w⊥ ∥ · (θ3 log8C (d) + θ log−2C (d)) (t) ≲ ∥w⊥ ∥ · θ log−2C (d),

(55)

√

where we used θ ≪ log−5C (d) and the basic inequality ∥ · ∥ ≤ d∥ · ∥∞ . Returning to Equation (48) and combining Equation (50), Equation (54), and Equation (55), we have (t)

(t+1)

∥w⊥

(t)

− w⊥ ∥ ≲ η

θ2 log6C (d)d−1/2 + ∥w⊥ ∥θ log−2C (d) (t) ∥w⊥ ∥2

(t)

∥w⊥ ∥ + ηθ log11C/7 (d)d−1/14

≲ ηθ log−2C (d), where the last inequality follows by Lemma D.6(iii) again. This completes the proof of the lemma. Finally, we can upper bound the growth of ∥w⊥ ∥∞ , constituting the inductive step for Definition D.4(v). Lemma D.14. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain , if fρ obeys the Phase I inductive hypothesis for all iterations k ≤ t (Theorem D.4), then we have (t+1)

∥w⊥

(t)

− w⊥ ∥∞ ≲ ηθ log3C (d)d−1/2

for all neurons (a(t+1) , w(t+1) ). Proof. Let i > 3; we will examine the growth of |wi | to obtain the desired bound. By the definition of gradient descent, we have (t+1)

(t) bρ = wi − η∂w(t) L i

(t)

= wi − η (t)

= wi − η

(t) bρ wi ∂w(t) L i (t)

(t) (t) b ρ − ∂ (t) L0 ) wi ∂w(t) L0 + wi (∂w(t) L w i

(t) wi

(56)

Recall we denote ξ\i := ξ − xi ei . By Theorem B.5 we have 1 (t) (t) −wi ∂w(t) L0 = |a(t) ||wi | i 8 √ √ (t) (t) (t) (t) (t) (t)⊤ (t) ξ\i | + P |wi | ≥ | 2∥wsig ∥ − wsp + w(t)⊤ ξ\i | P |wi | ≥ | 2∥wsig ∥ + wsp + w ξ ξ √ √ (t) (t) (t) (t) (t) (t) − P |wi | ≥ | 2∥wopp ∥ + wsp + w(t)⊤ ξ\i | − P |wi | ≥ | 2∥wopp ∥ − wsp + w(t)⊤ ξ\i | . ξ

We will rewrite the sum of four probabilities in interval notation to find that some segments cancel. Define X := w(t)⊤ ξ\i and h i √ √ (t) (t) (t) (t) (t) (t) I1 := −|wi | − 2∥wsig ∥ − wsp , |wi | − 2∥wsig ∥ − wsp h i √ √ (t) (t) (t) (t) (t) (t) I2 := −|wi | − 2∥wsig ∥ + wsp , |wi | − 2∥wsig ∥ + wsp i h √ √ (t) (t) (t) (t) (t) (t) ∥ − wsp , |wi | − 2∥wopp ∥ − wsp I3 := −|wi | − 2∥wopp h i √ √ (t) (t) (t) (t) (t) (t) I4 := −|wi | − 2∥wopp ∥ + wsp , |wi | − 2∥wopp ∥ + wsp . 54

Then, we can write 1 (t) (t) −wi ∂w(t) L0 = |a(t) ||wi | i

P(X ∈ I1 ) + P(X ∈ I2 ) − P(X ∈ I3 ) − P(X ∈ I4 )

≲ |a | P(X ∈ I1 ) − P(X ∈ I3 )| + | P(X ∈ I2 ) − P(X ∈ I4 )| ξ ξ ξ ξ (t) ≤ |a(t) ||wi | P(X ∈ I1 ∆I3 ) + P(X ∈ I2 ∆I4 ) , (t)

(t) ||wi |

where we used the triangle inequality, and I∆J := (I \ J) ∪ (J \ I) denotes the symmetric difference of (t) intervals√I and J. Note I1 and I3 (respectively I2 and I4 ) are intervals of width 2|wi | whose centers differ by only 2(∥wsig ∥ − ∥wopp ∥) ≲ θ log2C (d)d−1/2 , where the inequality follows by Lemma D.6(ii). So, I1 ∆I3 (respectively I2 ∆I4 ) comprises two intervals of width at most θ log2C (d)d−1/2 . Let us apply Theorem F.9 to (0) (0) each of the four intervals, with v (0) = w⊥\i , ∆ = w⊥\i − w⊥\i , and k ≲ θ log2C (d)d−1/2 .13 Under the event Etrain , the condition on v (0) is satisfied; we use Lemma D.6(iii) for the condition on ∆.14 Hence, we have P(X ∈ I1 ∆I3 ) ≲

θ log2C (d)d−1/2 1 +√ θ d

P(X ∈ I2 ∆I4 ) ≲

1 θ log2C (d)d−1/2 +√ , θ d

and therefore (t)

(t)

−wi ∂w(t) L0 ≲ |a(t) ||wi | · (log2C (d)d−1/2 + d−1/2 ) ≲ |wi | · θ log3C (d)d−1/2 , i

(57)

where we used |a(t) | ≲ θ logC (d) by Lemma D.7(i). Next, by the Cauchy-Schwarz inequality and Theorem D.9 with i > 3, we have (t) b ρ − ∂ (t) L0 )| ≤ |w(t) | · |∂ (t) L b ρ − ∂ (t) L0 | |wi (∂w(t) L i w w w i

(t) ≲ |wi | · (θ3 log8C (d)d−1/2 + θ logC (d)d−1/2 ) (t) ≲ |wi | · θ log2C (d)d−1/2 ,

(58)

where we used θ ≪ log−3C (d). Returning to Equation (56) and combining Equation (57) and Equation (58), we have (t+1)

|wi

(t)

− wi | ≲ η

θ log3C (d)d−1/2 + θ log2C (d)d−1/2 (t)

|wi |

(t)

|wi |

≲ ηθ log3C (d)d−1/2 . Since this holds for any i > 3, we have (t+1)

∥w⊥

(t)

− w⊥ ∥∞ ≲ ηθ log3C (d)d−1/2 .

This completes the proof of the lemma. We are now ready to show the Phase I result. 13 Here, µ is the center of each interval, but its value doesn’t matter as we upper bound exp

−µ2 Cθ 2

≤ 1 anyway.

14 While Theorem F.2 and Lemma D.6(iii) are stated for w , it is straightforward to see they hold for w ⊥ ⊥\i .

Proposition D.15. Suppose the Phase I scalings are satisfied (Theorem D.2). Under the event Etrain (Definition F.1(v)), upon TI ≍ log log(d)η −1 iterations of online minibatch SGD under the ℓρ loss, all neurons (T ) (a(TI ) , w(TI ) ) obey the Phase I inductive hypothesis (Theorem D.4) with sgn(wsp I ) = sgn(a(0) ). Moreover, for any test point x satisfying the event Etest (Definition F.1(vii)), we have sgn(fρ(TI ) (x)) = x3 . Proof. Under the event Etrain , the inductive steps for Theorem D.4 are given as follows: • The Definition D.4(i) inductive step is given by Theorem D.10. • The Definition D.4(ii) inductive step is given by Theorem D.11. • The Definition D.4(iii) inductive step is given by Theorem D.12. • The Definition D.4(iv) inductive step is given by Theorem D.13. • The Definition D.4(v) inductive step is given by Theorem D.14. • The Definition D.4(vi) inductive step is given by Lemma F.13(iii). By Lemma D.7(ii), we have for all neurons (a(TI ) , w(TI ) ) that sgn(a(TI ) ) = sgn(a(0) ), and by definition of the event Etrain we have |S + |, |S − | > 0. Hence, by Theorem D.1, we have sgn(fρ (x)) = x3 for any test (T ) point x that satisfies the event Etest if for all neurons (a(TI ) , w(TI ) ) we have sgn(wsp I ) = sgn(a(TI ) ) and (T ) (T ) (T ) (T ) 1/2 I I I I 1 (d). We will now show these conditions hold. C ∥wsp ∥ ≥ ∥wsig ∥ + ∥wopp ∥ + ∥w⊥ ∥ log (T )

The condition sgn(wsp I ) = sgn(a(TI ) ) is satisfied after at most TIa ≍ log1/2 (d)d−1/2 η −1 iterations; this constitutes Phase Ia. The intuition is that wsp → 0 monotonically if sgn(wsp ) ̸= sgn(a(0) ) and vice versa; in the former case wsp takes at most TIa iterations to flip signs. We now formalize this intuition. If (0) sgn(a(0) ) ̸= sgn(wsp ), we have by Definition D.4(i) that 1 −2 (t) (t−1) (t−1) (t−1) wsp − wsp = −η 1 ± o(log (d)) · − λ · sgn(wsp )(|wsp | + θ). 2 Following Theorem F.15 with w = wsp , z = θ, µ = 12 − λ, and δ = log−2 (d), we find (TIa ) |wsp |=

1 ± 3ηTIa log

−2

TIa 1 (0) (d) · 1 − η −λ · (|wsp | + θ) − θ. 2

Using TIa ≲ log1/2 (d)d−1/2 η −1 we have ηTIa log−2 (d) ≪ 1; hence for a constant c1 > 0 we have (TIa ) (0) |wsp | ≤ (1 − c1 η)TIa · (|wsp | + θ) − θ.

Setting the upper bound to zero to find the transition point, we find (0) |wsp | , TIa log(1 − c1 η) = − log 1 + θ (0)

wherein applying log(1 + x) ≤ x with η ≪ 1 and |wsp | ≲ θ log1/2 (d)d−1/2 by Theorem F.2 gives TIa · c1 η ≲ log1/2 (d)d−1/2 , that is TIa ≲ log1/2 (d)d−1/2 η −1 . Moreover, by observation of Definition D.4(i), sgn(wsp ) does not change once sgn(wsp ) = sgn(a(0) ). (T ) (T ) (T ) (T ) The condition C1 ∥wsp I ∥ ≥ ∥wsigI ∥ + ∥woppI ∥ + ∥w⊥ I ∥ log1/2 (d) is satisfied after at most TIb ≍ log log(d)η −1 additional iterations; this constitutes Phase Ib. Specifically, by Lemma D.6(ii) and Lemma D.6(iii) (T ) (T ) (T ) we have ∥wsigI ∥, ∥woppI ∥ ≲ θ log2C (d)d−1/2 and ∥w⊥ I ∥ ≲ θ. On the other hand, since sgn(wsp ) = sgn(a(0) ), by Theorem F.15 with w = wsp , z = θ, µ = 21 − λ, and δ = log−2 (d), we find (TI ) |wsp |≥

1 − 3ηTIb log

−2

TIb 1 (0) (d) · 1 + η −λ (|wsp | + θ) − θ. 2 56

Using TIb ≲ log log(d)η −1 we have ηTIb log−2 (d) ≪ 1; hence for a constant c2 > 0 (that depends on λ ∈ (0, 21 )), we have (TI ) (0) |wsp | ≥ (1 + c2 η)TIb (|wsp | + θ) − θ. Choose TIb = c3 log log(d)η −1 for some large enough constant c3 > 0. We then have c3 log log(d) (1 + c2 η)TIb = exp log(1 + c2 η) , η wherein using log(1 + c2 η) ≥ c22η for η ≪ 1 gives c3 log log(d) c2 c3 log log(d) log(1 + c2 η) ≥ ≥ C log log(d). η 2 (0)

(T )

Therefore, we have (1 + c2 η)TIb ≥ exp(C log log(d)) = logC (d). Since |wsp | ≥ 0, this gives |wsp I | ≥ θ(logC (d) − 1) ≳ θ logC (d). Choosing c3 to be large enough to ensure that C > 12 yields the result.15 Finally, we have TI = TIa + TIb ≍ log1/2 (d)d−1/2 η −1 + log log(d)η −1 ≍ log log(d)η −1 , i.e., the length of Phase I is dominated by the length of Phase Ib. This completes the proof of Phase I.

15 To make the key point explicit: combined with Lemma D.6(i) we have shown ∥w (TI ) ∥ ≍ θ logC (d) for a constant C > 1 . sp 2

Phase II Induction

Recall from Section D that upon TI ≍ log log(d)η −1 iterations (i.e., the end of Phase I), we have sgn(fρ(TI ) (x)) = x3 when x satisfies the event Etest (Definition F.1(vii)). (Moreover, we know that x satisfies the event Etest with probability at least 1 − d−C ; therefore, with high probability the neural network’s prediction entirely relies on the spurious feature at test time.) In Phase II, we will show that this condition continues to hold for roughly log(d)η −1 iterations, which is the sample complexity to learn the quadratic feature in the counterfactual situation where the linear spurious correlation did not exist (Glasgow, 2024). The key characteristic of the Phase II analysis is that the L0 approximation of the population gradients is no longer sufficient as the network weights have become too large by this point. Hence, we must analyze ∇Lρ directly via characterizing the training data margin induced by the network. Despite this additional complexity, we show that the neurons continue to grow identically under the “extreme correlation” condition λ ≪ log−1 (d). We prove some preliminary lemmas, including our definition of the inductive hypothesis, in Section E.1. We then show the inductive step in Section E.2. Throughout this section, we write ρ := ρ(t) as shorthand where appropriate, and we let C > 0 denote a sufficiently large constant which does not change from line to line. We will also use constants c < 1 and C ′ < C − 3 which do not change from line to line. We will work in high probability under the events Einit , Etrain , and Etest established in Theorem F.1 and Theorem F.4.

E.1

Technical Preliminaries

Our Phase II analysis requires the “extreme” correlation condition λ ≪ log−1 (d). In Phase II, the spurious feature grows very slowly, but the extreme correlation ensures it is still monotonically increasing. The key object we track in Phase II is the margin γ(x) := y(x)fρ (x). We define Phase II to last until |γ(x)| ≈ log log(d) (see Equation (59)) for the formal statement), which we show takes TII ≍ (log(θ−1 ) + log(d) log log(d))−1 )η −1 iterations. Throughout Phase II, we will require certain scaling factors to be satisfied. Note that the Phase II scalings are stricter than the Phase I scalings (Theorem D.2). In particular, note that if θ ≍ poly−1 (d) we require a large batch size m ≍ poly(d). Definition E.1. We say the Phase II scalings are satisfied if the following conditions are met: (i) The learning rate log(d)d−C ≪ η ≪ log−3 (d). (ii) The iteration t ≤ TII ≲ log(d)η −1 . (iii) The width log5 (d) ≪ p ≪ dC . (iv) The initialization scale d−C/2 ≪ θ ≪ log−5C (d). (v) The batch size m ≫ d log6 (d)θ−2 . (vi) The spurious correlation strength λ ≪ log−1 (d). Remark E.2. For clarity, we list the limiting usage of each scaling (i.e., the result which requires the tightest application of each item in Theorem E.1): • Definition E.1(i) is limited by Lemma E.7(i). The upper bound helps with the balancedness condition |a| ≈ ∥w∥, while the lower bound ensures that t ≪ dC throughout Phase II (in particular, TII ≪ dC , which is sufficient for this). • Definition E.1(ii) is limited by Theorem E.6 and Theorem E.7. It is essentially the duration of Phase II. • Definition E.1(iii) is limited by Theorem E.8 — it ensures the event Einit occurs with high probability, and the exponent on the lower bound ensures |S + |, |S − | are sufficiently close to half the number of neurons p2 .

• Definition E.1(iv) is limited by Theorem E.10, Theorem E.11, and Theorem E.12. The lower bound is loose but suffices for analysis, and the upper bound follows by the Phase I scalings. Similarly to Phase I, the initialization scale θ acts as a free parameter which we can make sufficiently small. • Definition E.1(v) is limited by Theorem E.10, Theorem E.11, and Theorem E.12, where it is used in Hoeffding’s inequality to concentrate the empirical gradient. (t)

• Definition E.1(vi) is limited by the stopping condition γ̄ (t) ≍ log log(d) such that e−γ̄ the “average margin” over neurons (which we will define shortly) grows monotonically.

− λ ≥ 0, i.e.,

Recall that TI = TIa + TIb ≍ log log(d)η −1 and we define ψ(x) := 1/(1 + e−x ) to be the sigmoid. Recall also that we define the “positive” neurons by S + := {(a(t) , w(t) ) : sgn(a(t) ) = 1} and the “negative” neurons by S − := {(a(t) , w(t) ) : sgn(a(t) ) = −1}. In Phase II, we will primarily be interested in the unnormalized margins induced by the positive and negative neurons on a test sample, which we will show concentrate (with high probability over the test data) about (t)

γ+ :=

1 p

(t)

(t) 2 (wsp )

γ− :=

(a(t) ,w(t) )∈S +

1 p

(t) 2 (wsp ) .

(59)

(a(t) ,w(t) )∈S −

(t) (t) We track the average margin γ̄ (t) := 12 (γ+ + γ− ) and define Phase II to end when γ̄ (t) ≍ log log(d). We can now define our Phase II inductive hypothesis.

Definition E.3. A neural network fρ is said to obey the Phase II inductive hypothesis at iteration t ≥ TI if there exists c < 1 such that the following conditions are met for all neurons (a(t) , w(t) ): (t) (t−1) (t−1) (i) wsp − wsp = η(1 ± o(1)) · wsp 1 − λ − ψ(γ̄ (t−1) ) . (t−1) (t) (t−1) (t−1) (t−1) (ii) ∥wsig − wsig ∥ ≲ ηe−cγ̄ max(a,w) (wsp )2 ∥wsig ∥ + ηθ log−1 (d)d−1/2 . (t)

(t−1)

(t)

(t−1)

(iii) ∥wopp − wopp ∥ ≲ ηe−cγ̄

(t−1) ) ∥wopp ∥ + ηθ log−1 (d)d−1/2 .

(t−1) 2

max(a,w) (wsp (t−1)

(t−1)

(wsp )2 ∥w⊥ ∥ + ηθ log−1 (d). (t−1) (t) (t−1) (t−1) (t−1) (v) ∥w⊥ − w⊥ ∥∞ ≲ ηe−cγ̄ max(a,w) (wsp )2 ∥w⊥ ∥∞ + ηθ log−1 (d)d−1/2 .

(iv) ∥w⊥ − w⊥

∥ ≲ ηe−cγ̄

(vi) |a(t) | ≤ ∥w(t) ∥. Remark E.4. The inductive hypothesis defined above implies that the growth of wsp decelerates exponentially. (t) Specifically, since 1 − λ − ψ(γ̄ (t) ) ≍ e−γ̄ when λ ≪ log−1 (d), the first recurrence (Definition E.3(i)) implies that (t) (t−1) (t) −γ̄ (t) wsp − wsp ≍ ηwsp e . (60) Yet, the condition λ ≪ log−1 (d) is sufficient to show that wsp growth remains monotonic. The geometric growth factor of wsig , wopp , and w⊥ also decelerates exponentially, leading to domination by small additive factors (the second terms in Definitions E.3(ii), E.3(iii), E.3(iv) and E.3(v)). In Theorem E.13, we use a continuous approximation of Equation (60) to show that Phase II lasts TII ≍ (log(θ−1 ) + log(d) log log(d))−1 )η −1 iterations, whereupon ∥wsp ∥ ≍ (log log(d))1/2 . The non-spurious weights grow by only a polylogarithmic factor during this time, and remain an order of magnitude smaller. Let us recall some properties of the neural network at the end of Phase I, which will effectively function as our Phase II “base case”. Remark E.5. Recall that, under the event Etrain (Definition F.1(v)) and with the Phase I scalings satisfied (Theorem D.2), the neural network fρ satisfies the following properties at iteration TI for all neurons (a(TI ) , w(TI ) ): 59

(T )

(i) ∥wsp I ∥ ≍ θ logC (d). (T )

(T )

(ii) ∥wsigI ∥, ∥woppI ∥ ≲ θ log2C (d)d−1/2 . (T )

(iii) ∥w⊥ I ∥ ≍ θ. (T )

(iv) ∥w⊥ I ∥∞ ≲ θ log4C (d)d−1/2 . (T ) (v) |a(TI ) | = 1 ± o(log−2 (d)) · ∥wsp I ∥. (T )

(vi) sgn(a(TI ) ) = sgn(a(0) ) = sgn(wsp I ). The next lemma controls the norms of all the neuron components during Phase II relative to the initialization scale θ assuming that the Phase II inductive hypothesis holds up until that point. Notice that (T ) (T ) the bounds hold for any TI ≤ t ≤ TII ; two special cases are ∥wsp I ∥ ≍ θ logC (d) but ∥wsp II ∥ ≍ (log log(d))1/2 . A second remark is that the upper bounds of Lemma E.6(i) and Lemma E.6(ii) will eventually become tight, (t) such that γ̄ (t) ≍ log log(d) and ∥wsp ∥ ≍ (log log(d))1/2 , as we show in Theorem E.13. Lemma E.6. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then the following conditions hold on all neurons (a(t) , w(t) ): (i) θ2 log2C (d) ≲ γ̄ (t) ≲ log log(d). (t)

(ii) θ logC (d) ≲ ∥wsp ∥ ≲ (log log(d))1/2 . (t)

(t)

(iii) ∥wsig ∥, ∥wopp ∥ ≲ θ log3C (d)d−1/2 . ′

′

(t)

(iv) θ log−C (d) ≲ ∥w⊥ ∥ ≲ θ logC (d) where C ′ < C − 3. (t)

(v) ∥w⊥ ∥∞ ≲ θ log5C (d)d−1/2 . (t) (vi) ∥w(t) ∥ = 1 ± o(log−2 (d)) · ∥wsp ∥ ≲ (log log(d))1/2 . Proof. For Lemma E.6(i), the upper bound holds by definition: Phase II lasts until γ̄ (t) ≍ log log(d). For (t) the Lemma E.6(i) lower bound, note that the growth of wsp in Definition E.3(i) is monotonic (increasing if (t) (t) (t) wsp > 0, decreasing if wsp < 0), hence so is the growth of γ̄ (t) . The monotonicity of wsp is formally shown below: using 1 − ψ(z) ≍ e−z for z ≥ 0 with γ̄ (t) ≲ log log(d) and λ ≪ log−1 (d), we have 1 − λ − ψ(γ̄ (t) ) ≍ e−γ̄

(t)

− λ ≳ log−1 (d) ≥ 0.

(t)

Regardless of the sign of wsp , the above argument implies that ∥wsp ∥ is monotonically increasing. Hence the (T ) lower bound holds by ∥wsp I ∥ ≳ θ logC (d) from Theorem E.5, monotonicity, and concentration of |S + |, |S − | under Etrain . (t) For the Lemma E.6(ii) upper bound, if (wsp )2 ≫ log log(d), then we must have γ̄ (t) ≫ log log(d) which is a contradiction with the stopping condition of Phase II. We already showed the lower bound in Lemma E.6(i). (T ) (T ) For Lemma E.6(iii), we may study the neuron that maximizes (wsp I )2 ∥wsigI ∥ to obtain an upper bound. In particular, the maximizing neuron at time t is at most as large as the maximizing neuron at time t − 1 plus the maximal allowed growth from t − 1 to t. Moreover, this maximal growth is the same for all neurons, as it depends only on γ̄ (t−1) which is a global property of the network. So, treating the recurrence on the maximizing neuron at time TI would give an upper bound over all neurons. Therefore, we consider the simpler form of Definition E.3(ii) as (t)

(t−1)

∥wsig − wsig

∥ ≲ ηe−cγ̄

(t−1)

(t−1) 2 (wsp ) ∥wsig

∥ + ηθ log−1 (d)d−1/2 .

For TI ≤ t1 < t2 ≤ TII define the growth factor Gt1 →t2 := log Gt1 →t2 =

tX 2 −1

Qt2 −1

(t) −cγ̄ (t) (wsp )2 ). Taking logs, we obtain t=t1 (1 + ηe

tX 2 −1 (t) (t) (t) 2 (t) 2 log 1 + ηe−cγ̄ (wsp ) ≤η e−cγ̄ (wsp ) ,

t=t1

(61)

t=t1

where we used log(1 + x) ≤ x. We will now upper bound the last term in Equation (61). Recall that 1 −z ≤ 1 − ψ(z) ≤ e−z for z ≥ 0. Using γ̄ (t) ≲ log log(d) and λ ≪ log−1 (d), we have 2e (t) 1 −cγ̄ (t) e ≤ 1 − λ − ψ(γ̄ (t) ) ≤ e−cγ̄ . 2

From Definition E.3(i) we then obtain η (t) −cγ̄ (t) (t+1) (t) (t) −cγ̄ (t) (1 − o(1)) · wsp e ≤ wsp − wsp ≤ η(1 + o(1)) · wsp e . 2 Using the difference of squares formula, (t+1) 2 (t) 2 (t+1) (t) (t+1) (t) (t) (t+1) (t) (t+1) (t) (wsp ) − (wsp ) = (wsp + wsp )(wsp − wsp ) = 2wsp + (wsp − wsp ) · (wsp − wsp ). For the lower bound, (t)

(62)

(t+1) 2 (t) 2 (t) (t+1) (t) (t) 2 −cγ̄ (wsp ) − (wsp ) ≥ 2wsp (wsp − wsp ) ≥ η(1 − o(1)) · (wsp ) e .

For the upper bound, (t+1) 2 (t) 2 (t) (t+1) (t) (t+1) (t) 2 (wsp ) − (wsp ) = 2wsp (wsp − wsp ) + (wsp − wsp ) (t) 2 −cγ̄ ≤ 2η(1 + o(1)) · (wsp ) e

wherein the latter term is lower-order by η, e−γ̄

(t)

(t) 2 −2cγ̄ + η 2 (1 + o(1))2 · (wsp ) e ,

≪ 1 to obtain (t)

(63)

(t+1) 2 (t) 2 (t) 2 −cγ̄ (wsp ) − (wsp ) ≤ 2η(1 + o(1)) · (wsp ) e .

Combining Equation (62) and Equation (63) gives (t) 2 −cγ̄ η(1 − o(1)) · (wsp ) e (t)

In particular, η(wsp )2 e−cγ̄ sum η

(t)

(t+1) 2 (t) 2 (t) 2 −cγ̄ ≤ (wsp ) − (wsp ) ≤ 2η(1 + o(1)) · (wsp ) e .

(t+1) 2

(t) 2 −cγ̄ (wsp ) e

(t)

) −(wsp )2 ). Returning now to Equation (61), we have the telescoping

≤ 2((wsp

TX II −1

(64)

≤2

t=t1

TX II −1

(t+1) 2 (t) 2 (TII ) 2 (t1 ) 2 (wsp ) − (wsp ) = 2((wsp ) − (wsp ) ),

t=t1 (T )

(t )

(T )

in other words log Gt1 →TII ≤ 2((wsp II )2 − (wsp1 )2 ). Using (wsp II )2 ≲ log log(d) we have Gt1 →TII ≲ ′ (T ) (t ) exp C ′ ((wsp II )2 − (wsp1 )2 ) = logC (d) (recalling that C > 0 is chosen sufficiently large). Therefore, (T )

(T )

∥wsigII ∥ ≲ ∥wsigI ∥GTI →TII +

TX II −1

ηθ log−1 (d)d−1/2 Gt→TII

t=TI

≲ log

C′ ′

′ (T ) (d)∥wsigI ∥ + TII ηθ logC −1 (d)d−1/2 ′

(T )

≲ logC (d)∥wsigI ∥ + logC (d)θd−1/2 .

(T )

Substituting ∥wsigI ∥ ≲ θ log2C (d)d−1/2 from Theorem E.5, we find ∥wsigII ∥ ≲ θ log3C (d)d−1/2 , as desired. The wopp result follows by a similar analysis. Lemma E.6(iv) and Lemma E.6(v) follow in the same way as Lemma E.6(iii), except with initial conditions (TI ) (T ) ∥w⊥ ∥ ≲ θ and ∥w⊥ I ∥∞ ≲ θ log4C (d)d−1/2 , respectively, from Theorem E.5. We obtain the lower bound (T ) in Lemma E.6(iv) using Definition E.3(iv) and the initial condition ∥w⊥ I ∥ ≳ θ from Theorem E.5. (t) (t) (t) For Lemma E.6(vi), we infer from Lemma E.6(iii) and Lemma E.6(iv) that ∥wsig ∥, ∥wopp ∥ ≪ ∥wsp ∥. Therefore, we have (t)

′

(t) 2 |∥w(t) ∥2 − ∥wsp ∥ | ≲ ∥w⊥ ∥2 ≲ θ2 log2C (d). (t)

But ∥wsp ∥2 ≳ θ2 log2C (d) with C ′ < C − 3 such that ′

θ2 log2C (d) (t) ∥wsp ∥2

′

≲ log2(C −C) (d) ≪ log−6 (d).

(t) In particular, ∥w(t) ∥ = 1 ± o(log−2 (d)) · ∥wsp ∥ as desired. Thus, we have proved all parts of the lemma. Next, we present the partner lemma to Theorem E.6 which controls behavior of the scalar weight a for any neuron (a, w). Lemma E.7. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then the following conditions hold on all neurons (a(t) , w(t) ): (t) (i) |a(t) | = 1 ± o(log−2 (d)) · ∥wsp ∥ ≲ (log log(d))1/2 . (t)

(ii) sgn(a(t) ) = sgn(a(0) ) = sgn(wsp ). (t)

Proof. For Lemma E.7(i), for the upper bound we have |a(t) | ≤ ∥w(t) ∥ = (1 ± o(log−2 (d)) · ∥wsp ∥ by Definition E.3(vi) and Lemma E.6(vi). For the lower bound, we denote as shorthand E(t) := ∥w(t) ∥2 − (a(t) )2 and note that E(TI ) ≪ log−2 (d) · ∥w(TI ) ∥2 by Theorem E.5 (see Lemma D.7(i)). Under the event Etrain , by Lemma F.13(iv) we have E(t) ≤ 10η 2 (a(t) )2 +E(t−1) for all iterations t ≪ dC . Moreover, by Definition E.3(vi) we have (a(k) )2 ≤ ∥w(k) ∥2 for all k ≤ t. In particular, E(t) ≤ E(TI ) +

t−1 X

10η 2 (a(k) )2 ≲ log−2 (d) · ∥w(TI ) ∥2 + (t − TI )η 2 ∥w(t) ∥2 .

(65)

k=TI

where the last inequality uses the inductive step. By t ≲ log(d)η −1 and η ≪ log−3 (d) we have tη 2 ≪ log−2 (d). (t) Using Lemma E.6(vi) again with the property that ∥wsp ∥ is increasing during Phase II (established in the (T ) (t) proof of Theorem E.6), we have ∥w(TI ) ∥ ≍ ∥wsp I ∥ ≤ ∥wsp ∥ ≍ ∥w(t) ∥ such that E(t) ≪ log−2 (d) · ∥w(t) ∥2 . Hence, (a(t) )2 = ∥w(t) ∥2 − E(t) = (1 − o(log−2 (d)) · ∥w(t) ∥2 , so we have shown |a(t) | = 1 − o(log−2 (d)) · ∥w(t) ∥,16 and applying Lemma E.6(vi) we obtain the desired (t) bound |a(t) | = 1 − o(log−2 (d)) · ∥wsp ∥. The remainder of the result follows from Lemma E.6(vi). For Lemma E.7(ii), the condition sgn(a(t) ) = sgn(a(0) ) follows identically to Lemma D.7(ii) (using only (T ) m ≫ d log2 (d) and η ≪ 1). Moreover, we have sgn(a(TI ) ) = sgn(wsp I ) by Theorem E.5 and monotonic growth of wsp in Phase II — increasing if wsp > 0 and decreasing if wsp < 0 — implies that sgn(wsp ) does not change, giving the second part of Lemma E.7(ii). Thus, we have proved all the parts of the lemma. 16 Here we used the Taylor expansion

√

1 − x = 1 − x/2 − O(x2 ) for x ≪ 1 such that

1 − o(log−2 (d)) = 1 − o(log−2 (d)).

Recall the majority group is denoted Xmaj := {x ∈ {±1}d : y(x) = x3 } and the minority group is denoted Xmin := {x ∈ {±1}d : y(x) = −x3 }. We will now show that the unnormalized margin γ(x) := y(x)fρ (x) 2 concentrates tightly about certain average values of ±wsp , where the sign is positive for the majority group (t)

(t)

and negative for the minority group. Recall the margins γ+ , γ− , and γ̄ (t) are defined in Equation (59). Lemma E.8. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the events Etrain and Etest (Definition F.1(v) and Definition F.1(vii)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then ( 1 ± o(log−2 (d)) · γ̄ (t) if x ∈ Xmaj and satisfies Etest γ(x) = −1 ± o(log−2 (d)) · γ̄ (t) if x ∈ Xmin and satisfies Etest . Proof. Let us omit (t) superscripts for clarity. By Theorem E.6 we have ∥wsp ∥ ≳ θ logC (d), while ′ ∥wsig ∥, ∥wopp ∥, ∥w⊥ ∥ ≲ θ logC (d) where C ′ < C − 3. Hence, following Theorem D.1 we have sgn(w⊤ x) = sgn(wsp x3 ) when x satisfies Etest . Moreover, by definition of the event Etest , we have ⊤ |w⊤ x − wsp x3 | ≲ |w⊥ x| < C∥w⊥ ∥ log1/2 (d) ≲ θ logC−5/2 (d),

for all neurons (a, w), where we used ∥w1:2 ∥ ≪ ∥w⊥ ∥ by Theorem E.6. In particular, for r ∈ {±1}, applying the triangle inequality gives us 1 p

wsp (w⊤ x − wsp x3 ) ≲

(a,w)∈S r

1 p

|wsp | · θ logC−5/2 (d).

(a,w)∈S r

Using again the fact that |wsp | ≳ θ logC (d), we have |wsp | · θ logC−5/2 (d) ≪ (wsp )2 · log−2 (d). Hence 1 p

wsp (w⊤ x − wsp x3 ) ≪ γr · log−2 (d).

(66)

(a,w)∈S r (0)

Next, since the Phase II dynamics of |wsp | depend only on |wsp | and the shared margins {γ̄ (s) }s≤t , they are independent of sgn(a(0) ). Treating the {wsp } over 1 ≤ j ≤ p as fixed, the only randomness is over the partition {S + , S − }; γ̄ is deterministic and γ+ , γ− are sums of independent bounded random variables with (t) E[γ+ ], E[γ− ] = γ̄. Since all wsp are within a constant factor of each other (follows by Theorem E.5 and Definition E.3(i)), by Hoeffding’s inequality we have s log(d) ≪ γ̄ · log−2 (d), |γ+ − γ̄| ≲ γ̄ p where we used p ≫ log5 (d). The same holds for γ− such that γ+ , γ− = 1 ± o(log−2 (d)) · γ̄.

(67)

Let us now consider each possible pairing of the label and spurious feature. 1. In the first case, we have y(x) = x3 = 1. Since Lemma E.7(ii) implies that sgn(a) = sgn(wsp ), we have σ(w⊤ x) = 0 if and only if (a, w) ∈ S − . In particular, 1 X γ(x) = aw⊤ x. p + (a,w)∈S

By Lemma E.7(i) we have a = 1 ± o(log−2 (d)) · wsp such that 1 γ(x) = 1 ± o(log−2 (d)) · p 63

X (a,w)∈S +

(t) ⊤ wsp w x.

Using x3 = 1 and Equation (66), we have  1 γ(x) = 1 ± o(log−2 (d)) · γ+ + p

 X

wsp (w⊤ x − wsp x3 ) = 1 ± o(log−2 (d)) · γ+ .

(a,w)∈S +

Finally using Equation (67) we have γ(x) = 1 ± o(log−2 (d)) · γ̄ as desired. 2. In the second case, we have y(x) = x3 = −1. Since Lemma E.7(ii) implies that sgn(a) = sgn(wsp ), we have σ(w⊤ x) = 0 if and only if (a(t) , w(t) ) ∈ S + . In particular, 1 X γ(x) = − aw⊤ x. p − (a,w)∈S

By Lemma E.7(i) we have a = 1 ± o(log−2 (d)) · wsp such that

1 γ(x) = − 1 ± o(log−2 (d)) · p

wsp w⊤ x.

(a,w)∈S −

Using x3 = −1 and Equation (66), we have   X 1 γ(x) = 1 ± o(log−2 (d)) · γ− + wsp (w⊤ x − wsp x3 ) = 1 ± o(log−2 (d)) · γ− . p − (a,w)∈S

Finally using Equation (67) we have γ(x) = 1 ± o(log−2 (d)) · γ̄ as desired. 3. In the third case, we have y(x) = −x3 = 1. Since Lemma E.7(ii) implies that sgn(a) = sgn(wsp ), we have σ(w⊤ x) = 0 if and only if (a, w) ∈ S + . In particular, 1 X γ(x) = aw⊤ x. p − (a,w)∈S

By Lemma E.7(i) we have a = 1 ± o(log−2 (d)) · wsp such that

1 γ(x) = 1 ± o(log−2 (d)) · p

wsp w⊤ x.

(a,w)∈S −

Using x3 = −1 and Equation (66), we have   X 1 γ(x) = 1 ± o(log−2 (d)) · −γ− + wsp (w⊤ x − wsp x3 ) = −1 ± o(log−2 (d)) · γ− . p − (a,w)∈S

Finally using Equation (67) we have γ(x) = −1 ± o(log−2 (d)) · γ̄ as desired. 4. In the fourth case, we have y(x) = −x3 = −1. Since Lemma E.7(ii) implies that sgn(a) = sgn(wsp ), we have σ(w⊤ x) = 0 if and only if (a, w) ∈ S − . In particular, 1 X γ(x) = − aw⊤ x. p + (a,w)∈S

By Lemma E.7(i) we have a = 1 ± o(log−2 (d)) · wsp such that

1 γ(x) = − 1 ± o(log−2 (d)) · p 64

X (a,w)∈S +

wsp w⊤ x.

Using x3 = 1 and Equation (66), we have  1 γ(x) = − 1 ± o(log−2 (d)) · γ+ + p

 X

wsp (w⊤ x − wsp x3 ) = −1 ± o(log−2 (d)) · γ+ .

(a,w)∈S +

Finally using Equation (67) we have γ(x) = −1 ± o(log−2 (d)) · γ̄ as desired. This completes the proof of the lemma.

E.2

Phase II Inductive Step

In this section, we show the Phase II inductive step and ultimately obtain the Phase II result in Theorem E.13. We first characterize the growth of wsp , constituting the inductive step for Definition E.3(i). Lemma E.9. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then we have (t+1) (t) (t) wsp − wsp = η(1 ± o(1)) · wsp 1 − λ − ψ(γ̄ (t) ) for all neurons (a(t+1) , w(t+1) ). Proof. We first compute the order of the intermediate term εw(t) defined in Theorem B.2. By Theorem B.7, we have ! √ (t) (t) (t) (t) ∥w1:2 ∥ ∥w⊥ ∥33 −∥wsp ∥2 + 2 2∥w1:2 ∥∥wsp ∥ |εw(t) | ≲ exp + (t) (t) 2 ∥w⊥ ∥3 ∥w⊥ ∥ 2∥w⊥ ∥ ! √ (t) (t) (t) 2 2∥w1:2 ∥∥wsp ∥ ∥w⊥ ∥∞ ∥w1:2 ∥ exp + , ≤ (t) (t) 2 ∥w⊥ ∥ ∥w ∥ 2∥w ∥ ⊥

⊥

(t)

where we used the basic inequalities −∥wsp ∥2 ≤ 0 and ∥w⊥ ∥33 ≤ ∥w⊥ ∥∞ ∥w⊥ ∥2 . By Lemma E.6(ii), (t) (t) Lemma E.6(iii), Lemma E.6(iv), and Lemma E.6(v) we have ∥wsp ∥ ≲ (log log(d))1/2 , ∥w1:2 ∥ ≲ θ log3C (d)d−1/2 , (t) (t) 5C −C ′ C′ θ log (d) ≲ ∥w⊥ ∥ ≲ θ log (d) for C ′ < C − 3, and ∥w⊥ ∥∞ ≲ θ log (d)d−1/2 . Using eu ≲ 1 + u for 0 < u ≪ 1,17 we have |εw(t) | ≲ log4C (d)d−1/2 · exp(C log6C (d)d−1/2 ) + log6C (d)d−1/2 ≲ log4C (d)d−1/2 · (1 + log6C (d)d−1/2 ) + log6C (d)d−1/2 ≲ log10C (d)d−1/2 (68)

= o(1). By the definition of gradient descent, we have (t+1) (t) bρ wsp = wsp − η∂w(t) L sp

(t)⊤ bρ wsp ∇w(t) L (t) = wsp −η (t) wsp (t)⊤

(t) = wsp −η

(t)⊤

b ρ − ∇w(t) Lρ ) wsp ∇w(t) Lρ + wsp (∇w(t) L (t)

wsp

17 This can be seen via the following Taylor expansion: eu ≤ 1 + u + Cu2 ≲ 1 + u for 0 < u ≪ 1.

(69)

By Theorem E.8, the assumption of Theorem C.5 is satisfied with ϵ = log−2 (d) and γ = γ̄ (t) . Therefore, we have ε (t) (t)⊤ (t) + λψ(γ̄ (t) )εw(t) − wsp ∇w(t) Lρ − a(t) wsp (1 − λ − ψ(γ̄ (t) )) · 1 + w 2 (t) ≲ |a(t) ||wsp |(log−2 (d)γ̄ (t) + d−C ). We will now use the above to show that (t)⊤ (t) −wsp ∇w(t) Lρ = a(t) wsp (1 − λ − ψ(γ̄ (t) ))(1 ± o(1)).

(70)

To do this, it suffices to show that three terms are lower-order in the following sense: |εw(t) | = o(1) λψ(γ̄ (t) )|εw(t) | ≪ 1 − λ − ψ(γ̄ (t) ) log−2 (d)γ̄ (t) + d−C ≪ 1 − λ − ψ(γ̄ (t) ). The first relation above follows directly from Equation (68). To show the second relation, we use the fact that 1 − ψ(z) ≍ e−z for z ≥ 0 with γ̄ (t) ≲ log log(d) by Lemma E.6(i) and λ ≪ log−1 (d). From this, we have 1 − λ − ψ(γ̄ (t) ) ≍ e−γ̄

(t)

− λ ≳ log−1 (d).

On the other hand, λψ(γ̄ (t) )|εw(t) | ≪ log−1 (d) log10C (d)d−1/2 ≪ log−1 (d); therefore, the second relation holds. Finally, to show the third relation, note that log−2 (d)γ̄ (t) + d−C ≪ log−1 (d). Putting these together directly yields Equation (70). Further, plugging in the expression in Lemma E.7(i) for a(t) yields (t)⊤ (t) (t) 2 −wsp ∇w(t) Lρ = a(t) wsp · (1 ± o(1)) · (1 − λ − ψ(γ̄ (t) )) = (wsp ) · (1 ± o(1)) · (1 − λ − ψ(γ̄ (t) )) (t)

where we also used sgn(a(t) ) = sgn(wsp ) by Lemma E.7(ii). Next, applying Lemma F.3(i) with m ≫ d log6 (d)θ−2 and i = 3 we have b ρ − ∂ (t) Lρ | ≪ |a(t) |θ log−2 (d)d−1/2 ≲ θ log−1 (d)d−1/2 , |∂w(t) L w sp

where we used |a | ≲ (log log(d)) (t)

1/2

by Lemma E.7(i). Hence, we have

(t)⊤ b ρ − ∇w(t) Lρ )| ≲ |w(t) |θ log−1 (d)d−1/2 . |wsp (∇w(t) L sp (t)

We now justify that this term is lower-order compared to (wsp )2 (1 − λ − ψ(γ̄ (t) )). By Lemma E.6(ii) we have (t) that |wsp | ≳ θ logC (d). Using (1 − λ − ψ(γ̄ (t) )) ≳ log−1 (d) again, we compare θ logC−1 (d) to θ log−1 (d)d−1/2 , and the latter is clearly lower-order. Returning to Equation (69), we have (t)

(t+1) (t) wsp − wsp =η

(wsp )2 · (1 ± o(1)) · (1 − λ − ψ(γ̄ (t) )) (t)

wsp

(t) = η(1 ± o(1)) · wsp (1 − λ − ψ(γ̄ (t) )).

This completes the proof of the lemma. We will now upper bound the growth of wsig and wopp , constituting the inductive step for Definition E.3(ii) and Definition E.3(iii). Lemma E.10. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then for any constant c < 1 we have (t) (t+1) (t) (t) (t) 2 ∥wsig − wsig ∥ ≲ ηe−cγ̄ max (wsp ) ∥wsig ∥ + ηθ log−1 (d)d−1/2 (a,w) (t) (t+1) (t) (t) 2 (t) ∥wopp − wopp ∥ ≲ ηe−cγ̄ max (wsp ) ∥wopp ∥ + ηθ log−1 (d)d−1/2 (a,w)

for all neurons (a(t+1) , w(t+1) ). 66

Proof. By the definition of gradient descent, we have (t+1)

wsig

(t) bρ = wsig − η∇w(t) L sig

(t)⊤ bρ wsig ∇w(t) L (t) (t) wsig = wsig − η (t) ∥wsig ∥2 (t)

= wsig − η

(t)⊤ (t)⊤ b ρ − ∇w(t) Lρ ) wsig ∇w(t) Lρ + wsig (∇w(t) L (t)

∥wsig ∥2

(t)

(71)

wsig .

For the second term, by Hoeffding’s inequality we have

P |w

(t)⊤

ξ+w

(t)⊤

e3 | ≤

√

(t) 2∥wsig ∥

(t)

−(wsp )2

≲ exp

(t)

≲ exp

(t)

C∥w⊥ ∥2

− log6 (d) C

′

(t)

where we used |wsp | ≳ θ logC (d) by Lemma E.6(ii) and ∥w⊥ ∥ ≲ θ logC (d) for C ′ < C − 3 by Lemma E.6(iv). Moreover, applying Lemma F.3(i) with m ≫ d log6 (d)θ−2 we have that for any i, b ρ − ∂ (t) Lρ | ≪ |a(t) |θ log−2 (d)d−1/2 ≲ θ log−1 (d)d−1/2 , |∂w(t) L w i

where we used |a(t) | ≲ (log log(d))1/2 by Lemma E.7(i). By the Cauchy-Schwarz inequality, we obtain (t)⊤ b ρ − ∇w(t) Lρ )| ≲ ∥w(t) ∥θ log−1 (d)d−1/2 . |wsig (∇w(t) L sig

The exp

− log6 (d) C

and d−C terms are both lower-order compared to θ log−1 (d)d−1/2 as θ ≫ d−C/2 . Returning (t)

to Equation (71), and using |a(t) | ≍ |wsp | by Lemma E.7(i), we have (t) (t) (t) (t) ∥wsig ∥(e−cγ̄ max(a,w) (wsp )2 ∥wsig ∥ + θ log−1 (d)d−1/2 ) (t+1) (t) ∥wsig − wsig ∥ ≲ η (t) ∥wsig ∥ (t) (t) (t) 2 ≲ ηe−cγ̄ max (wsp ) ∥wsig ∥ + ηθ log−1 (d)d−1/2 . (a,w)

The result for wopp follows in a similar way. This completes the proof of the lemma. Next, we can upper bound the growth of ∥w⊥ ∥, constituting the inductive step for Definition E.3(iv). Lemma E.11. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then for any constant c < 1 we have (t+1)

∥w⊥

(t)

(t) 2 − w⊥ ∥ ≲ ηe−cγ̄ (wsp ) ∥w⊥ ∥ + ηθ log−1 (d)

for all neurons (a(t+1) , w(t+1) ). Proof. By the definition of gradient descent, we have (t+1)

w⊥

(t) bρ , = w⊥ − η∇w(t) L ⊥

hence by the triangle inequality, (t+1)

∥w⊥

(t) b ρ − ∇ (t) Lρ ∥. − w⊥ ∥ ≤ η∥∇w(t) Lρ ∥ + η∥∇w(t) L w

(72)

⊥

By Theorem E.8, the assumption of Theorem C.6 is satisfied with γ = γ̄ (t) . Hence, r (t) 1 (t) ∥∇w⊥ Lρ ∥ ≲ e−cγ̄ |a(t) |∥w⊥ ∥ + P (|w⊤ ξ| ≥ |w⊤ z + w⊤ s|) + d1/2−C . (t) |a | ξ For the second term, by Hoeffding’s inequality we have (t)

⊤

P |w ξ| ≥ |w z + w s| ≲ exp ξ

(t)

−(wsp )2

≲ exp

(t)

C∥w⊥ ∥2

− log6 (d) C

′

(t)

where we used |wsp | ≳ θ logC (d) by Lemma E.6(ii) and ∥w⊥ ∥ ≲ θ logC (d) for C ′ < C − 3 by Lemma E.6(iv). Moreover, applying Lemma F.3(ii) with m ≫ d log6 (d)θ−2 we have b ρ − ∇ (t) Lρ ∥ ≲ |a(t) |θ log−2 (d) ≲ θ log−1 (d), ∥∇w(t) L w⊥ ⊥ 6 where we used |a(t) | ≲ (log log(d))1/2 by Lemma E.7(i). The exp − logC (d) and d1/2−C terms are both (t)

lower-order compared to θ log−1 (d) as θ ≫ d−C/2 . Returning to Equation (72), and using |a(t) | ≍ |wsp | by Lemma E.7(i), we have (t+1)

∥w⊥

(t)

(t) 2 − w⊥ ∥ ≲ ηe−cγ̄ (wsp ) ∥w⊥ ∥ + ηθ log−1 (d).

This completes the proof of the lemma. Finally, we can upper bound the growth of ∥w⊥ ∥∞ , constituting the inductive step for Definition E.3(v). Lemma E.12. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), if fρ obeys the Phase II inductive hypothesis for all iterations TI ≤ k ≤ t (Theorem E.3), then for a constant c < 1 we have (t) (t+1) (t) (t) (t) 2 ∥w⊥ − w⊥ ∥∞ ≲ ηe−cγ̄ max (wsp ) ∥w⊥ ∥∞ + ηθ log−1 (d)d−1/2 (a,w)

for all neurons (a(t+1) , w(t+1) ). Proof. Let i > 3; we will examine the growth of |wi | to obtain the desired bound. By the definition of gradient descent, we have (t+1)

(t) bρ = wi − η∂w(t) L i

(t)

= wi − η (t)

= wi − η

(t) bρ wi ∂w(t) L i (t)

(t) (t) b ρ − ∂ (t) Lρ ) wi ∂w(t) Lρ + wi (∂w(t) L w i

(t) wi

(73)

By Theorem E.8, the assumption of Theorem C.7 is satisfied with γ = γ̄ (t) . Hence we have (t) (t) (t) −wi ∂wi Lρ ≲ |a(t) ||wi | e−cγ̄ max |a(t) ||wi | + d−C + (a,w) √ √ (t) (t) (t) (t) (t) (t)⊤ (t) ξ\i | + P |wi | ≥ | 2∥wsig ∥ − wsp + w(t)⊤ ξ\i | P |wi | ≥ | 2∥wsig ∥ + wsp + w ξ ξ √ √ (t) (t) (t) (t) (t)⊤ (t) (t) + P |wi | ≥ | 2∥wopp ∥ + wsp + w ξ\i | + P |wi | ≥ | 2∥wopp ∥ − wsp + w(t)⊤ ξ\i | . ξ

For the last term, let us rewrite the probabilities in interval notation. Define X := w(t)⊤ ξ\i and i h √ √ (t) (t) (t) (t) (t) (t) I1 := −|wi | − 2∥wsig ∥ − wsp , |wi | − 2∥wsig ∥ − wsp h i √ √ (t) (t) (t) (t) (t) (t) I2 := −|wi | − 2∥wsig ∥ + wsp , |wi | − 2∥wsig ∥ + wsp i h √ √ (t) (t) (t) (t) (t) (t) I3 := −|wi | − 2∥wopp ∥ − wsp , |wi | − 2∥wopp ∥ − wsp i h √ √ (t) (t) (t) (t) (t) (t) . I4 := −|wi | − 2∥wopp ∥ + wsp , |wi | − 2∥wopp ∥ + wsp Then, we can write the last term as P(X ∈ I1 ) + P(X ∈ I2 ) + P(X ∈ I3 ) + P(X ∈ I4 ). ξ

Let us upper bound Pξ (X ∈ I1 ), and the other terms will follow similarly. Note that I1 is on the negative (t) real line, as the dominating term is −wsp . Thus by Hoeffding’s inequality, we have   (t) 2 √ −(w ) sp (t) (t) (t)⊤ (t) . ξ\i ≤ −(wsp − 2∥wsig ∥ − |wi |) ≲ exp  P(X ∈ I1 ) = P w (t) ξ ξ C∥w⊥\i ∥2 (t)

(t)

′

Then, using |wsp | ≳ θ logC (d) by Lemma E.6(ii) and ∥w⊥\i ∥ ≲ θ logC (d) for C ′ < C − 3 by Lemma E.6(iv),18 we have − log6 (d) . Pξ (X ∈ I1 ) ≲ exp C (t)

Applying this result to each of the four intervals, and using |a(t) | ≍ |wsp | ≲ (log log(d))1/2 by Lemma E.7(i), we obtain ! − log6 (d) (t) (t) (t) −cγ̄ (t) (t) 2 1/2 −C −wi ∂w(t) Lρ ≲ e max (wsp ) |wi | + (log log(d)) exp +d |wi |. i C (a,w) Moreover, applying Lemma F.3(i) with m ≫ d log6 (d)θ−2 we have b ρ − ∂ (t) Lρ | ≪ |a(t) |θ log−2 (d)d−1/2 ≲ θ log−1 (d)d−1/2 , |∂w(t) L w i

such that

(t) b ρ − ∂ (t) Lρ ) ≲ |w(t) |θ log−1 (d)d−1/2 . wi (∂w(t) L i w i

− log2 (d) C

Since this holds for any i > 3, we have (t+1)

∥w⊥

(t)

− w⊥ ∥∞ ≲ ηe−cγ̄

(t)

(t) (t) 2 max (wsp ) ∥w⊥ ∥∞ + ηθ log−1 (d)d−1/2 .

(a,w)

This completes the proof of the lemma. 18 While Lemma E.6(iv) is stated for w

⊥ , it is straightforward to see it holds for w⊥\i .

We are now ready to show the Phase II result. Proposition E.13. Suppose the Phase II scalings are satisfied (Theorem E.1). Under the event Etrain (Definition F.1(v)), upon ( log(d)(log log(d))−1 η −1 θ ≍ polylog−1 (d) TII ≍ log(d)η −1 θ ≍ poly−1 (d) iterations of online minibatch SGD under the ℓρ loss, all neurons (a(TII ) , w(TII ) ) obey the Phase II inductive hypothesis (Theorem E.3) with sgn(fρ(TII ) (x)) = x3 under the event Etest (Definition F.1(vii)). Proof. Recall that the initial conditions for Phase II are given by Theorem E.5. The inductive steps for Theorem E.3 are given as follows: 1. The Definition E.3(i) inductive step is given by Theorem E.9. 2. The Definition E.3(ii) inductive step is given by Theorem E.10. 3. The Definition E.3(iii) inductive step is given by Theorem E.10. 4. The Definition E.3(iv) inductive step is given by Theorem E.11. 5. The Definition E.3(v) inductive step is given by Theorem E.12. 6. The Definition E.3(vi) inductive step is given by Lemma F.13(iii). Let us now analyze the length of Phase II (by definition the number TII of iterations until γ̄ (t) ≍ log log(d)). (t) (t+1) (t) (t) By Equation (64), we have (wsp )2 − (wsp )2 ≍ η(wsp )2 e−γ̄ . Combining with the fact that S + , S − do not vary with t, we can obtain a recurrence for γ̄ (t) by     X X 1  (t+1) 2 (t) 2  (t+1) 2 (t) 2  γ̄ (t+1) − γ̄ (t) = (wsp ) − (wsp ) + (wsp ) − (wsp ) 2p (a,w)∈S + (a,w)∈S −   −γ̄ (t) X X ηe (t) 2 (t) 2   ≍ (wsp ) + (wsp ) 2p + − (a,w)∈S

(a,w)∈S

(t)

ηe−γ̄ (t) (t) (γ+ + γ− ) 2 (t) = ηγ̄ (t) e−γ̄ .

(74)

Write z (t) for γ̄ (t) as shorthand. Let us now perform a continuous approximation. The number of iterations is TII =

X t

X ez(t) t

ηz (t)

(t)

· ηz (t) e−z . (t)

This is a Riemann sum with nonuniform step sizes ηz (t) e−z , wherein the quadrature error bound gives Z log log(d) TII − z (TI )

(t)

1 1 X d ez dz ≤ · (z (t+1) − z (t) )2 −z ηze 2 t dz ηz (t) X d ez(t) (t) ≲ · (ηz (t) e−z )2 , (t) dz ηz t

where we applied Equation (74) for the update rule of z (t) . Substituting Z log log(d) TII − z (TI )

(t)

d ez dz ηz (t)

z (t)

(t)

= e η(z|z(t) )−1| we obtain 2

(t)

X ez |z (t) − 1| (t) 1 ≲ dz · η 2 (z (t) )2 e−2z (t) )2 ηze−z η(z t (t) η X (t) = |z − 1| · e−z 2 t ≲ ηTII log log(d),

(t)

where we used e−z ≤ 1 and z (t) ≲ log log(d). Since η ≪ (log log(d))−1 we have that the approximation error is o(TII ). Hence Z log log(d) 1 TII ≍ dz. (T ) ηze−z z I Proceeding to solve the integral, we have Z 1 1 log log(d) ez Ei(log log(d)) − Ei(z (TI ) ) dz = TII ≍ η z(TI ) z η where

Z z

et dt −∞ t

Ei(z) =

is the exponential integral. The asymptotics of this object are well-understood (Temme, 2026); in particular z Ei(z) ≍ ez for z ≫ 1 and Ei(z) ≍ log(z) for z ≪ 1. Note z (TI ) ≪ 1 since z (TI ) ≲ θ2 log2C (d) by Theorem E.5 and θ ≪ log−C (d). Hence, we have 1 log(d) 1 TII ≍ + log . η log log(d) z (TI ) Now, using z (TI ) ≍ θ2 log2C (d) by Theorem E.5, we have 1 1 log − 2 log ≍ log log(d), θ z (TI ) which is dominated by the loglog(d) log(d) term such that 1 log(d) 1 TII ≍ + log . η log log(d) θ Finally, under the event Etest we have sgn(fρ (x)) = x3 by the same argument as Theorem D.1. This completes the proof of Phase II. We also have the proof of Theorem 3.2. Theorem E.14. Suppose the Phase II scalings are satisfied (Theorem E.1). Then, upon ( log(d)(log log(d))−1 η −1 θ ≍ polylog−1 (d) T ≍ log(d)η −1 θ ≍ poly−1 (d) iterations of online minibatch SGD under the ℓρ loss, we have AccXmaj (fρ(T ) ) ≥ 1 − d−C

AccXmin (fρ(T ) ) ≤ d−C .

Proof. The theorem follows from Theorem E.13. In particular, at the end of Phase II, we have with probability 1 − d−C (i.e., under the events Etrain and Etest ) that sgn(fρ(T ) (x)) = x3 . On this event we have that any majority group point is correctly classified, while any minority group point is incorrectly classified. The inequalities in the bound then result from the d−C probability that the event Etest does not hold.

Technical Lemmas

This section provides some technical lemmas. Section F.1 details certain high-probability events and demarcates each source of randomness in our proofs. Section F.2 regards concentration of uniform samples from the Boolean hypercube. Section F.3 provides norm bounds and concentration for training ReLU neural networks with Lipschitz loss functions. Section F.4 details other miscellaneous helper lemmas.

F.1

High-probability Events

In this section, we specify the high-probability events which will be used as building blocks in many of our proofs. Recall we define the positive neurons by S + := {(a, w) : sgn(a) = 1} and the “negative neurons” by S − := {(a, w) : sgn(a) = −1}. Definition F.1. Define the following events for a constant C > 0: (0)

(0)

(i) The initialization event for the j-th neuron (aj , wj ) with wj rj ∼ Unif({±1}):

∼ Sd−1 (θ) and aj = rj θ where

(0) (0) (0) (j) Einit := ∥wj⊥ ∥ ≍ θ ∩ ∥wj⊥ ∥33 ≲ θ3 d−1/2 ∩ ∥wj⊥ ∥∞ ≲ θ log1/2 (d)d−1/2 . (ii) The initialization event over all p neurons (sampled i.i.d.): p n po \ (j) ∩ Einit := |S + |, |S − | = 1 ± o(log1/2 (d)p−1/2 ) · Einit . 2 j=1

(iii) The concentration event for the t-th minibatch M (t) ∼ Pdm (λ): (t) Ebatch :=

b ρ(t) )2 ≤ log (d) (∂w(t) Lρ(t) − ∂w(t) L i i m

∀i ∈ [d] 2 b ρ(t) )2 ≤ d log (d) ∥w(t) ∥2 . ∩ (∂a(t) Lρ(t) − ∂a(t) L m

(iv) The concentration event over all T minibatches (sampled i.i.d.): Ebatch :=

(t) t=1 Ebatch .

(v) The train event over all neuron initializations and minibatches: Etrain := Einit ∩ Ebatch . (vi) The concentration event for the j-th neuron on a test point x ∼ Pd (λ): ⊤ (j) ⊤ Etest := |wj⊥ (x\i + xi )|, |wj⊥ (x\i − xi )| < C∥wj⊥ ∥ log1/2 (d) (vii) The concentration event over all p neurons on a test point x ∼ Pd (λ): Etest :=

∀i > 3 . (j) j=1 Etest .

Notation. For a point x = (z, s, ξ), we will equivalently write that x satisfies Etest and ξ satisfies Etest , as (j) the event is solely a property of ξ, and similarly for Etest . To show that each of these events occur with high probability, we will use the following lemmas. The first lemma gives the analysis for Definition F.1(i). (0)

(0)

Lemma F.2. For any neuron (a(0) , w(0) ), we have ∥w⊥ ∥ ≍ θ, ∥w⊥ ∥33 ≲ θ3 d−1/2 , and ∥w(0) ∥∞ ≲ θ log1/2 (d)d−1/2 with probability at least 1 − d−C for any fixed C > 0.

u Proof. Write w(0) := θ ∥u∥ where u ∼ N (0, Id ). Choose v = (C + 1) log d. By the standard Gaussian tail √ bound we have P(|ui | ≥ 2v) ≤ 2e−v = 2d−(C+1) , so a union bound over i ∈ [d] gives √ −C P ∃ i : |ui | ≥ 2v ≤ 2d .

Standard χ2 concentration bounds (e.g., Laurent and Massart, 2000) give √ 2 −v −(C+1) . P ∥u∥ ≤ d − 2 dv − 2v ≤ e = d On the complement of these two events, which holds with probability at least 1 − 3d−C , we have r √ 2v log(d) (0) |wi | ≤ θ p ≲θ , √ d d − 2 dv − 2v establishing the ℓ∞ bound. (0) We now show the desired bound for ∥w⊥ ∥. Recall that we write u1:3 := (u1 , u2 , u3 , 0) and u4: := (0, u4 , u5 , . . . ) as shorthand, so that s ∥u4: ∥2 (0) . ∥w⊥ ∥ = θ ∥u1:3 ∥2 + ∥u4: ∥2 Applying χ2 concentration again, each with failure probability at most e−v = d−(C+1) , yields h √ i h p i ∥u1:3 ∥2 ∈ 3 ± 2 3v + 2v and ∥u4: ∥2 ∈ (d − 3) ± 2 (d − 3)v + 2v . This gives the sandwich inequality s s p p (d − 3) − 2 (d − 3)v − 2v (d − 3) + 2 (d − 3)v + 2v (0) p p √ √ θ ≤ ∥w⊥ ∥ ≤ θ . d − 2 (d − 3)v + 2 3v d + 2 (d − 3)v − 2 3v Expanding to leading order with v = (C + 1) log d yields ! r r r d−3 v log(d) (0) ∥w⊥ ∥ ≍ θ 1± ≍θ 1± ≍ θ. d d d (0)

(0)

We now show the desired bound for ∥w⊥ ∥33 . Clearly we have ∥w⊥ ∥33 ≤ ∥w(0) ∥33 , so we can show the result for the latter. Applying χ2 concentration again, with failure probability at most e−v = d−(C+1) ,√yields Pd ∥u∥2 ≥ d2 . Now, ∥u∥33 = i=1 |ui |3 . Since ui ∼ N (0, 1), by direct integration we have E[|ui |3 ] = 2√π2 , so E[∥u∥33 ] ≍ d. Moreover, |ui |3 is subexponential with constant ψ1 -norm, so Bernstein’s inequality implies 3 −v −(C+1) . P(∥u∥3 ≳ d) ≤ e = d

Hence, (0)

∥w⊥ ∥33 ≤ ∥w(0) ∥33 ≤ θ3

∥u∥33 d ≲ θ3 3/2 = θ3 d−1/2 , ∥u∥3 d

as desired. A union bound over all failure events gives total failure probability at most 6d−(C+1) ≤ d−C for d large enough. This completes the proof of the lemma. b ρ gradients about the population Lρ gradients Next, we consider empirical concentration of empirical L (c.f. Glasgow, 2024, Lemma B.12). This constitutes the analysis for Definition F.1(iii). 73

Lemma F.3. Suppose we train via online SGD with batch size m under the ℓρ loss. Then, for any neuron (a, w), with probability at least 1 − d−C for any fixed C > 0, the following hold: 2

b ρ )2 ≤ log (d) a2 for all i ∈ [d]. (i) (∂wi Lρ − ∂wi L m 2

b ρ ∥2 ≤ d log (d) a2 . (ii) ∥∇w Lρ − ∇w L m 2

b ρ )2 ≤ d log (d) ∥w∥2 . (iii) (∂a Lρ − ∂a L m b ρ ] and we defined ∂u L := p ∂L for u ∈ (a, w). Consider Lemma F.3(i) Proof. Recall that Lρ = EM ∼Pdm (λ) [L ∂u and note that ∂ p ℓρ (x) = ℓ′ρ (x)aσ ′ (w⊤ x)xi . ∂wi Since ℓρ is 2-Lipschitz, σ ′ (z) ≤ 1 for the ReLU, and xi ∈ {±1}, we have that ℓ′ρ (x)aσ ′ (w⊤ x)xi is bounded in [−2a, 2a]. Since we are using online SGD the minibatches are independent, and thus Hoeffding’s inequality gives that with probability at least 1 − δ, r 2 b ρ | ≤ 8a log(1/δ) . |∂wi Lρ − ∂wi L m 2

Setting δ = e− log (d) so that log(1/δ) = log2 (d), squaring both sides gives b ρ )2 ≤ (∂wi Lρ − ∂wi L

log2 (d) 2 a m

with failure probability e− log (d) for each fixed i. A union bound over i ∈ [d] gives total failure probability 2 d · e− log (d) ≤ d−C for any fixed C > 0 and d large enough. This immediately gives Lemma F.3(ii). For Lemma F.3(iii), we have ∂ p ℓρ (x) = ℓ′ρ (x)σ(w⊤ x). ∂a √ We have that ℓρ is 2-Lipschitz, σ(z) ≤ z for the ReLU, and w⊤ x ≤√ d∥w∥ by the Cauchy-Schwarz inequality √ and x ∈ {±1}d . Hence ℓ′ρ (x)σ(w⊤ x) is bounded in [−2 d∥w∥, 2 d∥w∥], and we can again use Hoeffding’s inequality to find that with probability at least 1 − δ, r 8d∥w∥2 log(1/δ) b |∂a Lρ − ∂a Lρ | ≤ . m 2

Setting δ = e− log (d) so that log(1/δ) = log2 (d), squaring both sides gives 2

b ρ )2 ≤ d log (d) ∥w∥2 (∂a Lρ − ∂a L m 2

with failure probability e− log (d) ≤ d−C for any fixed C > 0 and d large enough. This completes the proof of the lemma. We are now ready to show that each event in Theorem F.1 occurs with high probability. Lemma F.4. For a constant C > 0 chosen large enough, if log(d) ≪ p ≪ dC and T ≪ dC , then each event in Theorem F.1 occurs with probability at least 1 − d−C . Proof. Since all lemmas hold for any fixed C > 0, union bounds over at most polynomially many events (in d) are handled by applying each lemma with C replaced by C + C0 for an appropriate absolute constant C0 ; we suppress this adjustment throughout. Definition F.1(i) occurs with probability at least 1 − d−C by Theorem F.2. 74

For Definition F.1(ii), recall that sgn(a(0) ) is an independent Rademacher variable for each neuron (a , w(0) ). By a Chernoff bound we have with probability at least 1 − d−C that p p |S + |, |S − | = ± O p log(d) , 2 (0)

wherein p ≫ log(d) implies

p |S + |, |S − | = 1 + o(log1/2 (d)p−1/2 ) · . 2 Tp (j) C A union bound over p ≪ d applications of Theorem F.2 gives j=1 Einit with probability at least 1 − d−C , and the result follows by another union bound. Definition F.1(iii) occurs with probability at least 1 − d−C by Theorem F.3. The condition on Definition F.1(iv) follows by a union bound over T ≪ dC applications of Theorem F.3. The condition on Definition F.1(v) follows by a union bound over Definition F.1(ii) and Definition F.1(iv). ⊤ For Definition F.1(vi), w⊥ ξ is a sum of independent bounded terms for any neuron (a, w). By Hoeffding’s ⊤ inequality, with probability at least 1 − d−C , we have |w⊥ ξ| < C∥w⊥ ∥ log1/2 (d). We need this property to hold for all one-bit-flips of ξ. Indeed, this holds with a slightly larger constant. Since flipping the i-th bit ⊤ (say, from ξ to ξ ′ ) can change the Rademacher sum |w⊥ ξ| by at most 2|wi |, we have ⊤ ′ ⊤ |w⊥ ξ | ≤ |w⊥ ξ| + 2|wi | ≤ C∥w⊥ ∥ log1/2 (d) + 2|wi |. ⊤ ′ ξ | ≤ C ′ ∥w⊥ ∥ log1/2 (d) as desired. If d ≥ e then we may choose C ′ = C + 2 such that |w⊥ The condition on Definition F.1(vii) follows by a union bound over p ≪ dC applications of Definition F.1(vi). This completes the proof of the lemma.

F.2

From Booleans to Gaussians

In this section, we prove some lemmas which enable us to approximate uniform samples from the Boolean hypercube by Gaussians. At a high level, these are different specializations of the well-known Berry-Esseen central limit theorem to our feature learning setting. In Section F.2.1, we overview the Berry-Esseen theorem and some basic Gaussian approximations. In Section F.2.2, we show that the Berry-Esseen error term achieves the optimal d−1/2 rate in our setting. In Section F.2.3, we apply the Berry-Esseen theorem to a certain truncated moment which will be important in our analysis. In Section F.2.4, we use the Lindeberg exchange method to show that a certain population gradient with respect to a weight vector w is approximately parallel to w. F.2.1

Basic Results

One can see that inner products with high-dimensional Rademacher vectors are approximately Gaussian via the following special case of the Berry-Esseen central limit theorem. Theorem F.5. For any v ∈ Rd and µ ∈ R, we have sup

v⊤ ξ − µ ≤ k −

G∼N (µ,∥v∥22 )

k∈R ξ∼Unif({±1}d )

(G ≤ k) ≲

∥v∥33 . ∥v∥32

Following an application of Theorem F.5, we will often want to upper bound the Gaussian term as follows. Lemma F.6. For any k ∈ R we have k P 2 (|G| ≤ k) ≲ exp σ G∼N (µ,σ )

−µ2 2σ 2

k2 σ2

Proof. Let Φ denote the standard Gaussian cumulative distribution function and ϕ denote the standard Gaussian probability density function. Also, let a = −µ/σ and h = k/σ. Noting that Φ′′ (·) is upper bounded by a constant, Taylor’s theorem gives us Φ(a + h) − Φ(a) − hϕ(a) ≍ h2 . In particular, P

G∼N (µ,σ 2 )

(|G| ≤ k) = Φ(a + h) − Φ(a − h) ≲ hϕ(a) + h2 .

The lemma then follows by definition of ϕ, a, and h. We combine Theorem F.5 and Theorem F.6 in the following useful form. Lemma F.7. Suppose v ∈ Rd and we pick k ≲ ∥v∥2 d−1/4 . Then for any µ ∈ R, we have k −µ2 ∥v∥33 |v ⊤ ξ − µ| ≤ k ≲ exp + . P ∥v∥2 2∥v∥22 ∥v∥32 ξ∼Unif({±1}d ) Proof. By Theorem F.5, we have P

ξ∼Unif({±1}d )

|v ⊤ ξ − µ| ≤ k ≲

Using k ≲ ∥v∥2 d−1/4 , we further have

(|G| ≤ k) +

G∼N (µ,∥v∥22 )

∥v∥33 . ∥v∥32

∥v∥33 1 k2 ≲√ ≤ . 2 ∥v∥2 ∥v∥32 d

Applying Theorem F.6 with σ = ∥v∥2 completes the proof of the lemma. F.2.2

Achieving the Optimal Rate ∥v∥3

Theorem F.7 is very close to the result we will need. However, the ∥v∥33 error term can be problematic for us 2 as ∥v∥∞ grows. We can avoid this via a condition that the deviation ∆ from a well-behaved initialization v (0) has small ℓ2 -norm. This ensures that the set of “bad” indices in v which cause the error term to stray from the optimal d−1/2 rate is small enough that it can be integrated out as a constant. Before proving the main result, we introduce a lemma which controls certain vector norms in this regime. Note that the requirements on v (0) are satisfied with high probability when v (0) ∼ Sd−1 (θ).19 Lemma F.8. Suppose we can write v = v (0) + ∆ such that ∥v (0) ∥2 ≍ θ, ∥v (0) ∥∞ ≲ θ log1/2 (d)d−1/2 , ∥v (0) ∥33 ≲ θ3 d−1/2 and ∥∆∥2 ≪ θ log−1/2 (d). Define the “bad” index set B := {i : |∆i | ≥ θd−1/2 }. Define vB := (vi )i∈B and v\B := (vi )i∈B / . Then, we have: (i) |B| ≪ d log−1 (d). (ii) ∥vB ∥2 ≪ θ. (iii) ∥v\B ∥33 ≲ θ3 d−1/2 . (iv) ∥v\B ∥2 ≍ θ. (0)

(0)

19 We will not exactly have v (0) ∼ Sd−1 (θ): in our application we have v (0) = w (0) ∼ Sd−1 (θ), but ⊥ = w4: for a vector w the norm bounds will still hold (with high probability and up to constants).

Proof. For Lemma F.8(i), by the definition of B we have ∆2i ≥ θ2 d−1 for all i ∈ B. Using ∥∆∥2 ≪ θ log−1/2 (d) we have X |B|θ2 d−1 ≤ ∆2i ≤ ∥∆∥22 ≪ θ2 log−1 (d), i∈B

so |B| ≪ d log

−1

(d). For Lemma F.8(ii), using (a + b)2 ≤ 2(a2 + b2 ) for a, b ≥ 0 we have X ∥vB ∥22 = vi2 i∈B

(0)

+ ∆i )2

(0)

(vi

i∈B

≲

(vi )2 +

i∈B

∆2i

i∈B

≤ B∥v (0) ∥2∞ + ∥∆∥22 −1 2

(d) · θ log(d)d−1 + θ2 log−1 (d)

≪ d log ≲ θ2 ,

where we used Lemma F.8(i), ∥v (0) ∥∞ ≲ θ log1/2 (d)d−1/2 , and ∥∆∥2 ≪ θ log−1/2 (d). For Lemma F.8(iii), using the triangle inequality and (a + b)3 ≤ 4(a3 + b3 ) for a, b ≥ 0 we have X |vi |3 ∥v\B ∥33 = i∈B /

≤

(0)

(|vi | + |∆i |)3

i∈B /

≲

(0)

|vi |3 +

i∈B /

|∆i |3

i∈B /

≤ ∥v (0) ∥33 + max |∆i |∥∆∥22 i∈B / −1 3 −1/2 3 −1/2 ≲θ d

+ θ log

3 −1/2

≲θ d

(d)d

where we used ∥v (0) ∥33 ≲ θ3 d−1/2 , |∆i | ≤ θd−1/2 for all i ∈ / B, and ∥∆∥2 ≪ θ log−1/2 (d). For Lemma F.8(iv), we have ∥v\B ∥22 = ∥v∥22 − ∥vB ∥22 = ∥v (0) + ∆∥22 − ∥vB ∥22 = ∥v (0) ∥22 + 2(v (0) )⊤ ∆ + ∥∆∥22 − ∥vB ∥22 . Using ∥v (0) ∥22 ≍ θ2 and the Cauchy-Schwarz inequality we have |∥v\B ∥22 − θ2 | ≲ (v (0) )⊤ ∆ + ∥∆∥22 + ∥vB ∥22 ≤ ∥v (0) ∥2 ∥∆∥2 + ∥∆∥22 + ∥vB ∥22 ≪ θ2 log−1 (d) + θ2 log−1 (d) + θ2 ≪ θ2 , where we also used ∥∆∥2 ≪ θ log−1/2 (d) and Lemma F.8(ii). Thus ∥v\B ∥22 ≍ θ2 (1 ± o(1)) so ∥v\B ∥2 ≍ θ. This completes the proof of the lemma.

Now, we can prove the main lemma of this section. The technique is along the same lines as Glasgow, 2024, Lemma B.4, though the result is modified for our purposes. Lemma F.9. There exists a constant C > 0 such that the following holds. Suppose we can write v = v (0) + ∆ such that ∥v (0) ∥2 ≍ θ, ∥v (0) ∥∞ ≲ θ log1/2 (d)d−1/2 , ∥v (0) ∥33 ≲ θ3 d−1/2 and ∥∆∥2 ≪ θ log−1/2 (d). Moreover, suppose k ≲ θd−1/4 . Then for any µ ∈ R, we have k 1 −µ2 ⊤ |v ξ − µ| ≤ k ≲ exp +√ . P 2 d θ Cθ ξ∼Unif({±1} ) d P P Proof. Define the “bad” index set B := {i : |∆i | ≥ θd−1/2 }. Define SB := i∈B vi ξi and S\B := i∈B / vi ξi . Similarly, define vB := (vi )i∈B and v\B := (vi )i∈B with ξB := (ξi )i∈B and ξ\B := (ξi )i∈B / / . By independence of the ξi ’s, we can condition on SB = z to find |v ⊤ ξ − µ| ≤ k ⇐⇒ |S\B − (µ − z)| ≤ k. By the law of total probability, X ⊤ P |v ξ − µ| ≤ k = P (SB = z) P ξ

ξB

ξ\B

|S\B − (µ − z)| ≤ k ,

(75)

where the sum is over all z in the range of SB , which is finite because SB is a discrete random variable. By Theorem F.7, for a constant c > 0, we have ∥v\B ∥33 k −(µ − z)2 + exp P (|S\B − (µ − z)| ≤ k) ≲ ∥v\B ∥2 2∥v\B ∥22 ∥v\B ∥32 ξ\B k −(µ − z)2 1 ≲ exp +√ , θ cθ2 d where we used Lemma F.8(iii), Lemma F.8(iv), and k ≲ θd−1/4 . Returning to Equation (75), we have k −(µ − SB )2 1 ⊤ +√ . P |v ξ − µ| ≤ k ≲ E exp 2 θ SB cθ ξ d It remains to show that

E exp

−(µ − SB )2 cθ2

≲ exp

−µ2 Cθ2

We have 2 2 2 (µ − SB )2 − (µ2 /2 − SB ) = µ2 − 2µSB + SB − µ2 /2 + SB 2 = µ2 /2 − 2µSB + 2SB

= (µ − 2SB )2 /2 ≥ 0, 2 so (µ − SB )2 ≥ µ2 /2 − SB . In particular, 2 −µ2 SB −(µ − SB )2 ≤ exp . E exp E exp cθ2 2cθ2 SB cθ2 SB h 2 i S We claim that ESB exp cθB2 ≲ 1, which setting C = 2c would prove the lemma. For any λ > 0 and any x

we have 2

2√ 1 eλx 4πλ 4πλ Z ∞ 2 1 λx2 e−t /(4λ) dt =√ e 4πλ −∞ Z ∞ 2 2 1 =√ e−(t−2λx) /(4λ) dt eλx 4πλ −∞ Z ∞ 2 1 =√ etx−t /(4λ) dt, 4πλ −∞

eλx = √

where we completed the square in t and used the standard Gaussian integral Taking expectations over SB and applying Fubini’s theorem, we have Z ∞ 2 1 tS −t2 /(4λ) λSB √ [e ] = dt. E E [e B ]e SB 4πλ −∞ SB

R∞ −∞

e−zt dt =

π/z for any z.

Since SB (as a Rademacher sum) is subgaussian with parameter ∥vB ∥22 , we have ESB [etSB ] ≤ et ∥vB ∥2 /2 , so Z ∞ 1 ∥vB ∥22 1 λS 2 exp −t2 − dt E [e B ] ≤ √ 4λ 2 SB 4πλ −∞ 1 , =p 1 − 2λ∥vB ∥22 where we used the Gaussian integral again, for λ < 1/(2∥vB ∥22 ). Setting λ = 1/(c∥v (0) ∥22 ), and recalling that ∥vB ∥22 ≪ θ2 ≍ ∥v (0) ∥22 by Lemma F.8(ii), we have 2 SB 1 1 ≲ 1. ≲p ≤q E exp 2∥vB ∥22 cθ2 SB 1 − o(1) 1− (0) 2 c∥v

∥2

This completes the proof of the lemma. F.2.3

Approximation of a Truncated Moment

In this section, we can apply Theorem F.5 to a certain truncated moment which will be important in our analysis. The proof technique is a bulk/tail argument on the integral of the difference between the Rademacher and Gaussian cumulative distribution functions. Lemma F.10. For any v ∈ Rd and k ≥ 0, write ∆(v, k) :=

ξ∼Unif({±1}d )

⊤ |v ξ|1 |v ⊤ ξ| ≥ k −

Then, for any v ∈ Rd we have sup ∆(v, k) ≲ ∥v∥∞ log1/2

k≥0

G∼N (0,∥v∥22 )

∥v∥2 ∥v∥∞

[|G|1(|G| ≥ k)] .

Proof. Recall that for any non-negative random variable X and k ≥ 0 we may write Z ∞ E[X 1(X ≥ k)] = k P(X ≥ k) + P(X ≥ u)du. k

(76)

In particular, we have by the triangle inequality that ∆(v, k) ≤ k P |v ⊤ ξ| ≥ k − P(|G| ≥ k) +

Z ∞ k

⊤ P |v ξ| ≥ u − P(|G| ≥ u) du.

For the first term, by Theorem F.5 we have ∥v∥33 . (77) k P |v ⊤ ξ| ≥ k − P(|G| ≥ k) ≲ k ∥v∥32 √ For the second term, we will split the integral at T = 2∥v∥2 log1/2 (∥v∥32 /∥v∥33 ). For the first integral, by Theorem F.5 we have Z T ∥v∥33 ⊤ . (78) P |v ξ| ≥ u − P(|G| ≥ u) du ≲ (T − k) ∥v∥32 k −u2 For the second integral, by the Gaussian tail we have P(|G| ≥ u) ≤ 2 exp 2∥v∥ . Moreover, by Hoeffding’s 2 2 2 −u inequality and independence of each ξi , we have P |v ⊤ ξ| ≥ u ≤ 2 exp 2∥v∥ . Hence, using the standard 2 2 upper bound on the Gaussian tail integral, we obtain Z ∞ Z ∞ −T 2 −u2 ∥v∥22 ⊤ exp exp ≤ . P |v ξ| ≥ u − P(|G| ≥ u) du ≲ 2∥v∥22 T 2∥v∥22 T T By our choice of T we have Z ∞ T

∥v∥22 ∥v∥33 ⊤ · . P |v ξ| ≥ u − P(|G| ≥ u) du ≲ T ∥v∥32

Combining Equation (78) and Equation (79) we have Z ∞ ∥v∥33 ∥v∥22 ∥v∥33 ∥v∥33 ⊤ + · ≍T . P |v ξ| ≥ u − P(|G| ≥ u) du ≲ (T − k) 3 3 ∥v∥2 T ∥v∥2 ∥v∥32 k

(79)

(80)

Combining Equation (77) and Equation (80) we have ∆(v, k) ≲ (k + T )

∥v∥33 . ∥v∥32 ∥v∥3

Now, let us supremize over k ≥ 0. If k ≤ T , we have ∆(v, k) ≲ T ∥v∥33 . On the other hand, if k > T the 2 entirety of the error is contained in the tail (c.f. Equation (79)). Formally, by the triangle inequality we have ∆(v, k) ≤ E |v ⊤ ξ|1 |v ⊤ ξ| ≥ k + E[|G|1(|G| ≥ k)]. By Equation (76) and the tail bounds used previously, we have Z ∞ −k 2 −u2 ∆(v, k) ≲ k exp + exp du 2∥v∥22 2∥v∥22 T The first term is maximized at k = T , i.e., −k 2 −T 2 ∥v∥33 k exp ≤ T exp ≲T . 2 2 2∥v∥2 2∥v∥2 ∥v∥32 For the second term we have by Equation (79) that Z ∞ −u2 ∥v∥22 ∥v∥33 exp du ≲ · . 2∥v∥22 T ∥v∥32 T 80

(81)

(82)

Combining the k ≤ T case with Equation (81) and Equation (82) we have sup ∆(v, k) ≲ T k≥0

∥v∥33 . ∥v∥32

To obtain our final result, we use the definition of T and standard norm inequalities such that ∥v∥32 ∥v∥2 ∥v∥33 1/2 1/2 log ≲ ∥v∥ log . sup ∆(v, k) ≲ ∥v∥2 ∞ ∥v∥32 ∥v∥33 ∥v∥∞ k≥0 This completes the proof of the lemma. F.2.4

Approximation of a Gradient Projection

Finally, we will leverage the Lindeberg exchange method (typically used in the proof of central limit theorems, e.g., Tao, 2015) to show that a certain population gradient with respect to a weight vector w is approximately parallel to w. Lemma F.11. Define W := Eξ∼Unif({±1}d ) ξh(w⊤ ξ) where h(t) := sgn(t)1(|t| ≥ κ). Write W⊥ for the 1/7 ∞ . projection of W onto the subspace perpendicular to w. Then, ∥W⊥ ∥2 ≲ ∥w∥ ∥w∥2 Proof. By the dual formulation and linearity of expectation, we have ⊤ ⊤ ∥W⊥ ∥2 = sup E z ξ · h(w ξ) . z ⊤ w=0,∥z∥=1 ξ

Let us fix some z ∈ Rd such that z ⊤ w = 0 and ∥z∥ = 1 and upper bound the right-hand side. Note that if we instead took the expectation over a standard Gaussian vector g ∼ N (0, I), we would have z ⊤ g and w⊤ g independent such that ⊤ ⊤ ⊤ ⊤ E z g · h(w g) = E z g · E h(w g) = 0, g

in other words, W is exactly parallel to w for Gaussians. We will leverage this fact to show that the distance to a Gaussian analogue of W is small using the Lindeberg exchange method. We will need to Taylor expand h, but it is discontinuous, so we define a smooth relaxation parameterized by τ (to be optimized later): let (k) hτ ∈ C ∞ be such that hτ = h outside [−κ − τ, −κ + τ ] and [κ − τ, κ + τ ] and ∥hτ ∥∞ ≲ τ −k for k = 1, 2, 3. This exists via convolution of h with two mollifiers, since h is just a step function with twoRsteps. Specifically, 1 choose any bounded C ∞ mollifier ρ : R → [0, ∞) such that ρ is supported on [−1, 1] and −1 ρ(t)dt = 1. We rescale this mollifier into [−τ, τ ] as 1 t ρτ (t) := ρ . τ τ Then, write S for the Heaviside step function and define Sτ (t) := (S ∗ ρτ )(t) where ∗ is the convolution operator. We argue that hτ (t) := Sτ (t − κ) − Sτ (−t − κ) has the desired properties. In particular, hτ is C ∞ because ρτ is C ∞ , and it has hτ = h outside [−κ − τ, −κ + τ ] and [κ − τ, κ + τ ] because ρτ is supported on [−τ, τ ]. For the first three derivatives, noting that ρ has bounded derivatives since it is C ∞ , we have 1 t 1 ′ Sτ (t) = ρτ (t) = ρ =⇒ h′τ (t) ≲ , τ τ τ 1 ′ t 1 ′′ ′ Sτ (t) = ρτ (t) = 2 ρ =⇒ h′′τ (t) ≲ 2 , τ τ τ 1 ′′ t 1 ′′′ ′′ =⇒ h′′′ Sτ (t) = ρτ (t) = 3 ρ τ (t) ≲ 3 . τ τ τ

Now, define H(x) := z ⊤ xh(w⊤ x) and Hτ (x) := z ⊤ xhτ (w⊤ x) such that ⊤ ⊤ E[z ξ · h(w ξ)] = E[H(ξ)] ξ

= E[H(ξ)] − E[Hτ (ξ)] + E[Hτ (g)] − E[H(g)] + E[Hτ (ξ)] − E[Hτ (g)] . ξ

(83)

The first two terms are smoothing errors and the final term is the Gaussian analogue error. For the first term, recall that by definition of hτ we have |Hτ (ξ) − H(ξ)| nonzero only when |w⊤ ξ| ∈ [κ − τ, κ + τ ]. Using this fact with Jensen’s inequality, the Cauchy-Schwarz inequality, and isotropy of ξ, we have ⊤ ⊤ E[H(ξ)] − E[Hτ (ξ)] ≤ E |z ξ| · 1 |w ξ| ∈ [κ − τ, κ + τ ] ξ ξ ξ r r ≤ E [(z ⊤ ξ)2 ] E [1 (|w⊤ ξ| ∈ [κ − τ, κ + τ ])] ξ

P (|w⊤ ξ| ∈ [κ − τ, κ + τ ]). ξ

Now, we have two cases. First, if τ < κ, combining Theorem F.5 and Theorem F.6 we find ⊤ P |w ξ| ∈ [κ − τ, κ + τ ] ≲ ξ

τ ∥w∥33 τ2 + . + 2 ∥w∥2 ∥w∥2 ∥w∥32

Second, if τ ≥ κ, combining Theorem F.5 and Theorem F.6 we find ⊤ ⊤ P |w ξ| ∈ [κ − τ, κ + τ ] = P |w ξ| ∈ [0, κ + τ ] ξ

(κ + τ )2 κ+τ ∥w∥33 + + 2 ∥w∥2 ∥w∥2 ∥w∥32 τ2 ∥w∥33 τ + + , ≲ 2 ∥w∥2 ∥w∥2 ∥w∥32 ≲

where we used that τ ≥ κ in the last step. In either case we have E[H(ξ)] − E[Hτ (ξ)] ≲ ξ

3/2

τ 1/2

τ ∥w∥3 + . + 1/2 3/2 ∥w∥ 2 ∥w∥2 ∥w∥2

(84)

The Gaussian smoothing error follows in the same two cases, just without the application of Berry-Esseen: repeating the Cauchy-Schwarz step and using Theorem F.6 we have E[Hτ (g)] − E[H(g)] ≲ g

τ 1/2 1/2 ∥w∥2

τ . ∥w∥2

(85)

Now, we execute the Lindeberg exchange method to bound the Gaussian analogue error. Let us replace coordinates one at a time such that Ki := (ξ1 , . . . , ξi , gi+1 , . . . , gd ). Then we have the telescoping sum E[Hτ (ξ)] − E[Hτ (g)] = ξ

d X i=1

E [Hτ (Ki ) − Hτ (Ki )].

ξ,g

Define qi (t) := Hτ (ξ1 , . . . , ξi−1 , t, gi+1 , . . . , gd ), 82

(86)

then we have Hτ (Ki ) − Hτ (Ki ) = qi (ξi ) − qi (gi ). By a third-order Taylor expansion about zero (which we can do since ρτ is C ∞ ), we have 1 qi (t) = qi (0) + qi′ (0)t + qi′′ (0)t2 + R3 (t). 2 Since ξi and gi are both centered and isotropic, we have q(0) − q(0) = 0, ′ ′ ′ E[qi (0)ξi − qi (0)gi ] = qi (0)(E[ξi ] − E[gi ]) = 0,

1 ′′ 1 ′′ 1 ′′ 2 2 2 2 E[ qi (0)ξi − qi (0)gi ] = qi (0)(E[ξi ] − E[gi ]) = 0, 2 2 2 so the first three terms cancel in expectation. By Taylor’s theorem with remainder, for some ζ1 , ζ2 ∈ (0, 1) we have 3 ′′′ 3 ′′′ E[qi (ξi ) − qi (gi )] = E[R3 (ξi ) − R3 (gi )] ≤ E[|ξi | · |qi (ζ1 ξi )|] + E[|gi | · |qi (ζ2 gi )|]. ⊤ ⊤ Now, writing A := z\i x\i and B := w\i x\i for the contributions from the other coordinates in x ∈ Ki , we ⊤ ⊤ differentiate Hτ (x) := z x · hτ (w x) to find

q ′′′ (t) = ∂i3 Hτ (x)|xi =t = 3zi wi2 h′′τ (wi t + B) + (zi t + A)wi3 h′′′ τ (wi t + B). (k)

Using ∥hτ ∥∞ ≲ τ −k by definition, we have |q ′′′ (t)| ≲

|zi |wi2 (|zi ||t| + |A|)|wi |3 + . τ2 τ3

Since |ξi | = 1 we have

|zi ||wi |3 |zi |wi2 E[|A|]|wi |3 3 ′′′ + + . E[|ξi | |q (ζ1 ξi )|] ≲ 2 3 τ τ τ3 p On the other hand, since E[|gi |3 ] = 8/π and E[gi4 ] = 3 we have 3 ′′′ E[|gi | |q (ζ2 gi )|] ≲

|zi ||wi |3 |zi |wi2 E[|A|]|wi |3 + + . τ2 τ3 τ3

Both sides give the same bound up to constants. By the Cauchy-Schwarz inequality with ∥z∥ = 1 and isotropy of x\i we have ⊤ E[|A|] = E[|z\i x\i |] ≤ ∥z\i ∥ ≤ 1, and |zi | ≤ 1, so

|zi |wi2 |wi |3 + . τ2 τ3 Returning to Equation (86) and summing over the coordinates we have E[q(ξi ) − q(gi )] ≲

E[Hτ (ξ)] − E[Hτ (g)] ≲ ξ

≤

1 X 1 X |zi |wi2 + 3 |wi |3 2 τ i=1 τ i=1 ∥w∥33 ∥w∥24 + , 2 τ τ3

(87)

where we used the Cauchy-Schwarz inequality and ∥z∥ = 1 in the last step. Substituting Equation (84), Equation (85), and Equation (87) into Equation (83), we have ∥W⊥ ∥2 ≲

3/2

τ ∥w∥3 ∥w∥24 ∥w∥33 + + + + . 2 1/2 3/2 ∥w∥2 τ τ3 ∥w∥2 ∥w∥2 τ 1/2

Using the basic inequalities ∥w∥33 ≤ ∥w∥∞ ∥w∥22 and ∥w∥24 ≤ ∥w∥∞ ∥w∥2 we have ∥W⊥ ∥2 ≲

1/2

∥w∥∞ ∥w∥∞ ∥w∥22 τ ∥w∥∞ ∥w∥2 + + . + + 2 1/2 1/2 ∥w∥2 τ2 τ3 ∥w∥2 ∥w∥2 τ 1/2

2/7

5/7

Equalizing the first and last terms we choose τ ≍ ∥w∥∞ ∥w∥2 . Therefore, 1/7

∥W⊥ ∥2 ≲

∥w∥∞

2/7

1/2

3/7

1/7

∥w∥∞ ∥w∥∞ ∥w∥∞ ∥w∥∞ ∥w∥∞ + + + + ≲ , 1/7 2/7 1/2 3/7 1/7 1/7 ∥w∥2 ∥w∥2 ∥w∥2 ∥w∥2 ∥w∥2 ∥w∥2

where the last step follows by ∥ · ∥∞ ≤ ∥ · ∥2 . This completes the proof of the lemma.

F.3

Simultaneous Training of Two-Layer ReLU Neural Networks

In this section, we provide norm bounds and concentration for training ReLU neural networks with Lipschitz loss functions. We begin with a short lemma involving local Lipschitz behavior of the sigmoid function ψ(u) := 1/(1 + e−u ). Lemma F.12. If γ1 , γ2 > 0 or γ1 , γ2 < 0, we have |ψ(γ1 ) − ψ(γ2 )| ≤ exp − min(|γ1 |, |γ2 |) · |γ1 − γ2 |. Proof. By the mean value theorem, for some γ1 ≤ ζ ≤ γ2 , we have |ψ(γ1 ) − ψ(γ2 )| ≤ |ψ ′ (ζ)| · |γ1 − γ2 |, so it suffices to control ψ ′ . In particular, we have ψ ′ (z) = ψ(z)(1 − ψ(z)) =

e−z . (1 + e−z )2

If z ≥ 0 then (1 + e−z )2 ≥ 1 so ψ ′ (z) ≤ e−z . If z < 0 then using the symmetry ψ ′ (z) = ψ ′ (−z) we have ψ ′ (z) ≤ e−|z| . So globally we have ψ ′ (z) ≤ e−|z| . Now, since γ1 and γ2 have the same sign, the segment between them lies in either (−∞, 0] or [0, ∞). In each of these regions, e−|z|is decreasing, and so is maximized at the point with smallest magnitude, i.e., |ψ ′ (ζ)| ≤ exp − min(|γ1 |, |γ2 |) . This completes the proof of the lemma. The next lemma comprises a balancedness condition (c.f. Glasgow, 2024, Lemma B.13). Lemma F.13. Suppose we train via online SGD with batch size m ≫ d log2 (d) and learning rate η < √12 under the ℓρ loss. Then, for any neuron (a(t) , w(t) ), the following hold: (i) ∥∇w(t) Lρ ∥ ≤ 2|a(t) |. (ii) |∂a(t) Lρ | ≤ 2∥w(t) ∥. (t)

(iii) Under the event Ebatch (Definition F.1(iii)), if |a(t) | ≤ ∥w(t) ∥, then |a(t+1) | ≤ ∥w(t+1) ∥. (t)

(iv) Under the event Ebatch (Definition F.1(iii)), we have ∥w(t+1) ∥2 − (a(t+1) )2 ≤ 10η 2 (a(t) )2 + ∥w(t) ∥2 − (a(t) )2 .

Proof. Let us begin with Lemma F.13(i). Recall the expansion of Lρ from Theorem F.3. Since ℓρ is 2-Lipschitz, σ ′ (z) ≤ 1 for the ReLU, and x is isotropic, we have h i 1 (1) ′ (t)⊤ ⊤ ℓ (x) · σ (w x) · x v ≤ 2 sup E[|x⊤ v|] = 2. ∥∇ sup (t) Lρ ∥ = E w ρ |a(t) | v:∥v∥=1 x v:∥v∥=1 x For Lemma F.13(ii), using that ℓρ is 2-Lipschitz and x is isotropic again, we have h i (t)⊤ x) ≤ 2 E[|w(t)⊤ x|] = 2∥w(t) ∥. |∇a(t) Lρ | = E ℓ(1) ρ (x) · σ(w x

For Lemma F.13(iii), by the definition of gradient descent and direct expansion of the square we can write b ρ )2 = (a(t) )2 − 2ηa(t) ∂a(t) L b ρ + η 2 (∂a(t) L b ρ )2 (a(t+1) )2 = (a(t) − η∂a(t) L and

b ρ ∥2 = ∥w(t) ∥2 − 2ηw(t)⊤ ∇w(t) L b ρ + η 2 ∥∇w(t) L b ρ ∥2 . ∥w(t+1) ∥2 = ∥w(t) − η∇w(t) L

b ρ = w ⊤ ∇w L b ρ for any neuron (a, w). Using this together Using 1-homogeneity of the ReLU we have a∂a L with the Cauchy-Schwarz inequality, we have b ρ )2 − ∥∇w(t) L b ρ ∥2 (a(t+1) )2 − ∥w(t+1) ∥2 = (a(t) )2 − ∥w(t) ∥2 + η 2 (∂a(t) L (88) 1 b ρ )2 − b ρ )2 ≤ (a(t) )2 − ∥w(t) ∥2 + η 2 (∂a(t) L (w(t)⊤ ∇w(t) L ∥w(t) ∥2 (a(t) )2 (t) 2 (t) 2 2 2 2 b b = (a ) − ∥w ∥ + η (∂a(t) Lρ ) − (∇ (t) Lρ ) ∥w(t) ∥2 a b ρ )2 η 2 (∂a(t) L (t) 2 (t) 2 = (a(t) )2 − ∥w(t) ∥2 + ∥w ∥ − (a ) ∥w(t) ∥2 ! b ρ )2 η 2 (∂a(t) L (t) 2 (t) 2 . (89) = (a ) − ∥w ∥ 1− ∥w(t) ∥2 (t)

Under the event Ebatch , using also Lemma F.13(ii), we have b ρ )2 ≤ 2(∂a(t) Lρ )2 + 2(∂a(t) Lρ − ∇a(t) L b ρ )2 (∇a(t) L d log2 (d) ≤ 2∥w(t) ∥2 1 + m = 2∥w(t) ∥2 (1 + o(1)), where the last line follows from m ≫ d log2 (d). Substituting into Equation (89) and using η < √12 , for d large enough we have sgn (a(t+1) )2 − ∥w(t+1) ∥2 = sgn (a(t) )2 − ∥w(t) ∥2 , which gives the desired statement of Lemma F.13(iii). Finally, for Lemma F.13(iv), starting from Equation (88) we have b ρ ∥2 − (∂a(t) L b ρ )2 ∥w(t+1) ∥2 − (a(t+1) )2 − ∥w(t) ∥2 + (a(t) )2 = η 2 ∥∇w(t) L b ρ ∥2 ≤ η 2 ∥∇w(t) L b ρ ∥2 . ≤ 2η 2 ∥∇w(t) Lρ ∥2 + ∥∇w(t) Lρ − ∇w(t) L

(t)

Under the event Ebatch , using also Lemma F.13(i) and m ≫ d log2 (d), we have ∥w(t+1) ∥2 − (a(t+1) )2 − ∥w(t) ∥2 + (a(t) )2 ≤ 2η 2 (4(a(t) )2 + (a(t) )2 ) ≤ 10η 2 (a(t) )2 , which is the desired statement of Lemma F.13(iv). This completes the proof of the lemma. The next lemma provides some helpful bounds on the magnitude of the neural network output |fρ (x)|. Lemma F.14. The following hold: (i) Ex [fρ (x)2 ] ≤ (E(a,w)∼ρ [∥aw∥])2 . (ii) Ex [|fρ (x)|] ≤ E(a,w)∼ρ [∥aw∥]. √ (iii) |fρ (x)| ≤ d E(a,w)∼ρ [∥aw∥] for any x ∈ {±1}d . (iv) Px (|fρ (x)| ≥ log(d) E(a,w)∼ρ [∥aw∥]) ≤ d−C for any fixed C > 0. Proof. Let us begin with Lemma F.14(i). By isotropy of x, we have for any neuron (a, w) that Ex [σ(w⊤ x)2 ] ≤ Ex [(w⊤ x)2 ] = ∥w∥2 . Using this fact with the Cauchy-Schwarz inequality, we have " # 2

2 E[fρ (x) ] = E x

= ≤

(a,w)∼ρ

1 X p2

[aσ(w⊤ x)]

aj ak E σ(wj⊤ x) · σ(wk⊤ x) x

j,k

1 X |aj ||ak | · p2

r r ⊤ x)2 · σ(w E E σ(wk⊤ x)2 j x

j,k

≤

1 X |aj ||ak | · ∥wj ∥∥wk ∥ p2 j,k

[∥aw∥])2 ,

(a,w)∼ρ

as desired. For Lemma F.14(ii), using Lemma F.14(i) and Jensen’s inequality, we have q E[|fρ (x)|] ≤ E[fρ (x)2 ] ≤ E [∥aw∥]. x

(a,w)∼ρ

For Lemma F.14(iii), using the triangle inequality, σ(x) ≤ |x|, and |w⊤ x| ≤ ∥w∥1 ≤ x ∈ {±1}d , we have |fρ (x)| =

[aσ(w⊤ x)] ≤

(a,w)∼ρ

[|a||w⊤ x|] ≤

√

d∥w∥2 for

√ d

[∥aw∥].

[∥aw∥]

(a,w)∼ρ

For Lemma F.14(iv), using Lemma F.14(ii) we have log(d)

[∥aw∥] − E[|fρ (x)|] ≥ log(d) x

(a,w)∼ρ

[∥aw∥] −

(a,w)∼ρ

= (log(d) − 1)

(a,w)∼ρ

[∥aw∥].

(a,w)∼ρ

In particular, P |fρ (x)| ≥ log(d) x

[∥aw∥] ≤ P |fρ (x)| − E[|fρ (x)|] ≥ (log(d) − 1)

(a,w)∼ρ

[∥aw∥] .

(a,w)∼ρ

For any i ∈ [d], we have the following bounded differences condition upon flipping the ith bit of x. Using the triangle inequality and 1-Lipschitzness of the ReLU, we obtain |fρ (x)| − |fρ (x − 2xi ei )| ≤ E |a| · |σ(w⊤ x) − σ(w⊤ (x − 2xi ei ))| ≤ 2 E [|awi |]. (a,w)∼ρ

(a,w)∼ρ

Therefore, by McDiarmid’s inequality and Jensen’s inequality,20 we have (log(d) − 1)2 (E(a,w)∼ρ [∥aw∥])2 P |fρ (x)| − E[|fρ (x)|] ≥ (log(d) − 1) E [∥aw∥] ≤ exp − Pd x x (a,w)∼ρ 2 i=1 (E(a,w)∼ρ [|awi |])2 1 ≤ exp − (log(d) − 1)2 2

≤ d−C , which is the desired statement of Lemma F.14(iv). This completes the proof of the lemma.

F.4

Miscellaneous Lemmas

In this section, we provide some miscellaneous helper lemmas. The first lemma is a derivation of the solution to the linear non-homogeneous recurrences of Theorem D.4. Lemma F.15. Consider z, η, µ, T > 0, and let (εt )t≥0 be a sequence with |εt | ≤ δ < 1 such that δηµT ≪ 1. Given a sequence {w(t) }t≥0 satisfying the recurrence w(t+1) − w(t) = ηµ(1 + εt ) · sgn(w(t) )(|w(t) | + z), it holds that |w(T ) | = (1 ± 3δηµT ) · (1 + ηµ)T (|w(0) | + z) − z. Proof. Since |εt | ≤ δ we have 1 − δ ≤ 1 + εt ≤ 1 + δ for all t ≤ T . Write x(t) := |w(t) | + z and note that x(t) > 0 for all t ≥ 0. We will first upper and lower bound x(T ) . For the upper bound, by the triangle inequality we have |w(t+1) | ≤ |w(t) |+ηµ(1+δ)·(|w(t) |+z). Rewriting, we find that x(t+1) ≤ (1 + ηµ(1 + δ))x(t) . Unrolling the recurrence, we have x(T ) ≤ (1 + ηµ(1 + δ))T x(0) . For the lower bound, the increment w(t+1) −w(t) has the same sign as w(t) , so w(t) never changes sign across all t ≥ 0. Therefore, |w(t+1) | ≥ |w(t) | + ηµ(1 − δ) · (|w(t) | + z). Rewriting, we find x(t+1) ≥ (1 + ηµ(1 − δ))x(t) . Unrolling the recurrence, we have x(T ) ≥ (1 + ηµ(1 − δ))T x(0) . Therefore we have established that (1 + ηµ(1 − δ))T x(0) ≤ x(T ) ≤ (1 + ηµ(1 + δ))T x(0) . The ratio of the upper to lower geometric factor satisfies log

(1 + ηµ(1 + δ))T ≤ 2δηµT. (1 + ηµ(1 − δ))T

Applying ex ≤ 1 + x + x2 for x ∈ [0, 1] and δηµT ≪ 1 we obtain (1 + ηµ(1 + δ))T ≤ 1 + 3δηµT. (1 + ηµ(1 − δ))T Hence

x(T ) = (1 ± 3δµηT ) · (1 + ηµ)T x(0) ,

wherein substituting x(T ) := |w(T ) | + z and x(0) = |w(0) | + z yields the result. 20 There is a subtle complication here: unlike Glasgow, 2024 we do not have independence on all coordinates of x. However, for the purposes of McDiarmid’s inequality, we can combine x1 x2 x3 into a single “super-variable” x̃ which is independent from the rest of x. Moreover, the bounded differences condition on x̃ is satisfied by the sum of the bounded differences of x1 , x2 , x3 , so the dependency ends up not changing the bound.

Additional Simulations

In this section, we provide some additional simulations with different hyperparameters.

(a) Phase Ia

(b) Phase Ib

Figure 4: Scaled-up phase transitions. We display the results of a training run with dimension d = 1000, spurious correlation strength λ = 0.1, learning rate η = 0.01, width p = 100, initialization scale θ = 0.001, and batch size m = 5000. For each of the p = 100 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3).

(a) Phase Ia

(b) Phase Ib

Figure 5: Small-initialization phase transitions. We display the results of a training run with dimension d = 1000, spurious correlation strength λ = 0.1, learning rate η = 0.01, width p = 10, initialization scale θ = 0.001, and batch size m = 5000. For each of the p = 10 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3).

(a) No spurious feature

(b) λ = 0.1

(d) λ = 0.2

Figure 6: Scaled-up λ. We display the results of a training run with dimension d = 1000, spurious correlation strength λ = 0.1, learning rate η = 0.01, width p = 100, initialization scale θ = 0.001, and batch size m = 5000. For each of the p = 100 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3).

(a) No spurious feature

(b) λ = 0.1

(d) λ = 0.2

Figure 7: Small-initialization λ. We display the results of a training run with dimension d = 100, learning rate η = 0.05, width p = 10, initialization scale θ = 0.0001, and batch size m = 5000. For each of the p = 10 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3).

(a) λ = 0.1

(b) λ = 0.15

Figure 8: Scaled-up margins. We display the results of a training run with dimension d = 1000, spurious correlation strength λ = 0.1, learning rate η = 0.01, width p = 100, initialization scale θ = 0.001, and batch size m = 5000. For each of the p = 100 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3).

(a) λ = 0.1

(b) λ = 0.15

Figure 9: Small-initialization margins. We display the results of a training run with dimension d = 100, learning rate η = 0.05, width p = 10, initialization scale θ = 0.0001, and batch size m = 5000. For each of the p = 10 neurons, we plot ∥wsig ∥, ∥wopp ∥, ∥wsp ∥, and ∥w⊥ ∥ (defined in Section 2.3).

Limitations, Broader Impacts, and LLM Usage

Limitations. Our analysis is confined to two-layer ReLU neural networks; as is the case more generally in deep learning theory, extending the results to three or more layers would be nontrivial (see, e.g., Nichani et al., 2023). Similarly, we study online minibatch SGD as is common in the literature, and handling batch sizes of 1 or data re-use would require different theoretical techniques Glasgow, 2024. Our results hold only for Boolean data, and not Gaussian data — this is the key fact that enables us to write the population gradients ∇L0 and ∇Lρ in closed form. Finally, we focus on the setting with one signal feature and one spurious feature, whereas in practice there may be many signals and spurious features competing simultaneously Z. Li et al., 2023; Kim et al., 2024. We believe it is an exciting direction to extend our work to multiple features of differing complexities G. Qiu et al., 2024. Broader impacts. We hope this work contributes to the safe and equitable application of machine learning by improving understanding of how neural networks learn and rely on spurious correlations. Our results may help motivate future research on robustness, fairness, and out-of-distribution generalization. A potential negative outcome is that practitioners may over-interpret theoretical guarantees in simplified settings as evidence that real-world systems are robust to spurious correlations. However, no theoretical framework can fully capture the complexity of practical deployment environments, and additional empirical evaluation remains necessary. LLM usage. Large language models (LLMs) including Claude Sonnet 4.6 Thinking, GPT 5.4 Thinking, and Gemini 3 Flash were used for discussion and refinement of some proof ideas. In particular, these models contributed to mathematical derivations in Theorem B.7, Theorem F.9, Theorem F.10, and Theorem F.11. All proofs were written, critically reviewed, and independently verified by the authors, who assume full responsibility for their correctness. Simulations. Our simulations were run on an Nvidia A5000 GPU with 24GB VRAM, but this level of compute is not necessary. Our code is available at https://github.com/tmlabonte/xor.

Record · ID 321844 · SHA-256 457adbe2cfed6d96
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.