Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario Florentin Coeurdoux1 * Grégoire Ferré1
Jean-Philippe Bouchaud1,2,3
1 Capital Fund Management, Paris, France
2 X-CFM Chair of EconophysiX, Ecole polytechnique, Palaiseau, France
arXiv:2604.18450v1 [stat.ML] 20 Apr 2026
3 Académie des Sciences, Paris, France
Abstract Empirical studies of trained models often report a transient regime in which signal is detectable in a finite gradient descent time window before overfitting dominates. We provide an analytically tractable random-matrix model that reproduces this phenomenon for gradient flow in a linear teacher–student setting. In this framework, learning occurs when an isolated eigenvalue separates from a noisy bulk, before eventually disappearing in the overfitting regime. The key ingredient is anisotropy in the input covariance, which induces fast and slow directions in the learning dynamics. In a two-block covariance model, we derive the full time-dependent bulk spectrum of the symmetrized weight matrix through a 2 × 2 Dyson equation, and we obtain an explicit outlier condition for a rank-one teacher via a rank-two determinant formula. This yields a transient Baik–Ben Arous–Péché (BBP) transition [1]: depending on signal strength and covariance anisotropy, the teacher spike may never emerge, emerge and persist, or emerge only during an intermediate time interval before being reabsorbed into the bulk. We map the corresponding phase diagrams and validate the theory against finite-size simulations. Our results provide a minimal solvable mechanism for early stopping as a transient spectral effect driven by anisotropy and noise.
1 Introduction Understanding when gradient-based learning extracts signal from data, and when it instead amplifies noise and overfits, is a central problem in modern learning theory. A recurring empirical observation is that the spectrum of trained weight matrices often passes through a transient regime in which an isolated eigenvalue separates from a noisy bulk before the spectrum becomes more complicated at later stages of training [2, 3, 4, 5]. Although this phenomenon is frequently associated with the practical success of early stopping, its underlying mechanism remains poorly understood. In this paper we isolate a minimal, analytically tractable mechanism for such a transient spectral separation. We study gradient flow in a linear teacher–student model and show that anisotropy in the input covariance can produce a transient BBP transition: an outlier * Corresponding author: [email protected]
1
Transient BBP Transition in Gradient Flow
1
Introduction
may emerge from the bulk at intermediate times (with a non-zero overlap with the teacher direction) and later be reabsorbed as training continues, in which case the teacher direction cannot be reconstructed [1]. The effect is driven by the coexistence of fast and slow directions in the data covariance. Fast directions learn the teacher early, whereas slow directions continue to inject noise and can eventually hide the signal again. Our model is deliberately simple. It is not intended to reproduce the full heavy-tailed phenomenology observed in deep nonlinear networks at convergence; rather, it is designed to capture and explain the bulk-plus-spike regime in a setting where the full dynamics can be analyzed explicitly. The key solvable case is a two-block covariance model in which the singular values of the input matrix take only two values. In that simplified yet realistic setting, the bulk spectrum of the symmetrized weight matrix is described by a 2 × 2 Dyson equation, while the rank-one teacher induces a finite-rank perturbation whose outlier condition can be reduced to a rank-two determinant. Three regimes emerge from our model, depending on signal strength and input covariance anisotropy: (i) a weak-signal regime in which no outlier ever emerges, so the teacher is never spectrally detectable; (ii) a strong-signal regime in which an outlier emerges and remains separated from the bulk at all times; and (iii) an “early-stopping” regime in which an outlier exists only during a finite time window, providing a precise random-matrix formulation of early stopping. Contributions. In the solvable setting above, we (a) derive a 2 × 2 Dyson equation characterizing the full time-dependent bulk spectrum of the symmetrized weight matrix; (b) establish an explicit rank-two BBP-type outlier condition valid at every finite time t; (c) classify the possible learning regimes and give closed-form phase diagrams in the (θ, t) and (θ, λ− ) planes; and (d) identify numerically an optimal early-stopping time, defined through the teacher–eigenvector overlap, that lies strictly inside the transient BBP window. Related works. The BBP phase transition was introduced by Baik, Ben Arous and Péché [1] and has since become a standard tool in spiked random matrix models and finite-rank perturbation theory [6]. Exact learning dynamics in linear and deep-linear models have been studied extensively, including solvable analyses of gradient-flow learning and early stopping [7, 8, 9]. The connection between spectral properties of weight matrices and generalisation has been explored empirically by Martin and Mahoney [2], Thamm et al. [3], Staats et al. [4], Yunis et al. [5], Olsen et al. [10], and others. An inverse BBP transition has been observed in the dynamical Hessian during gradient descent for the problem of phase retrieval [11]. More broadly, random matrix methods have become increasingly useful in the study of high-dimensional learning dynamics and implicit regularisation [12, 13, 14, 15, 16, 17]. Our contribution is to provide an arguably oversimplified but explicit random-matrix mechanism for a transient spectral spike during learning, i.e. a BBP transition followed by an inverse BBP transition, and to tie this transient spectral geometry to a computable optimal stopping time.
2
Transient BBP Transition in Gradient Flow
2
Model and Setup
2 Model and Setup 2.1 Gradient flow on a linear model We consider inputs X ∈ R N × M and targets Y ∈ R N × M in the high-dimensional regime M → γ ≥ 1. N
N, M → ∞,
We train a linear model with weight matrix A ∈ R N × N by minimizing ALS = arg min J ( A),
J ( A) =
A
1 ∥ AX − Y ∥2F . 2
(1)
Although the minimizer is explicit and reads ALS ≡ YX ⊤ ( XX ⊤ )−1 , our interest is not the optimizer itself but the spectral evolution of the gradient-flow trajectory dAt = −∇ J ( At ), dt
A0 = Ainit .
Since
∇ J ( A) = ( AX − Y ) X ⊤ = AXX ⊤ − YX ⊤ , the flow solves the linear matrix ODE Ȧt = − At XX ⊤ + YX ⊤ . Its explicit solution is ⊤
At = Ainit e−tXX + YX ⊤ ( XX ⊤ )−1 I − e−tXX
⊤
.
(2)
This is a standard semigroup evolution, and in the present least-squares setting it provides a natural continuous-time framework for early stopping [9]. At long times it converges to the least-squares estimator A∞ ≡ ALS . The key point for us is that the learning speed is anisotropic: the eigenvalues of XX ⊤ determine how fast the corresponding directions are learned or damped. This observation motivates the two-block covariance model introduced below.
2.2 Random matrix assumptions To model a teacher–student setting, we assume that the targets are generated by a rank-one teacher θ θ Y = Ateach X + Z, Ateach = θ vv⊤ , (3) where v ∈ R N is a fixed unit vector and θ ≥ 0 is the signal amplitude. The noise matrix Z ∈ R N × M has i.i.d. N (0, 1/M ) √ entries, independent of everything else. We take Ainit to be a GOE matrix at the usual 1/ N scale, independent of X and Z. Substituting (3) into (2) yields the decomposition ⊤ ⊤ ⊤ Aθt = Ainit e−tXX + ZX ⊤ ( XX ⊤ )−1 I − e−tXX + θ vv⊤ I − e−tXX . | {z } | {z }
noise part
teacher part
We consider the additive symmetrization Stθ := Aθt + ( Aθt )⊤ , 3
(4)
Transient BBP Transition in Gradient Flow
2
Model and Setup
because it is the simplest construction that preserves the bulk-plus-spike structure of the problem while remaining fully solvable at finite time: the noise contribution becomes a Wigner-type matrix with a piecewise-constant variance profile, and the teacher contribution becomes a finite-rank symmetric perturbation. The more standard multiplicative choice At A⊤ t would introduce a Wishart-type bulk whose time-dependent spectrum requires a substantially heavier analysis; we return to this in the conclusion. Using (4), we obtain h i ⊤ ⊤ Stθ = St0 + θ vv⊤ I − e−tXX + I − e−tXX vv⊤ , (5) where the pure-noise part is ⊤
⊤
⊤ St0 = Ainit e−tXX + e−tXX Ainit + ZX ⊤ ( XX ⊤ )−1 I − e−tXX
⊤
+ I − e−tXX
⊤
( XX ⊤ )−1 XZ ⊤ . (6) θ Our goal is to describe the spectrum of St for all times t ≥ 0 and all θ ≥ 0. Note that we alternatively write St instead of St0 for notational simplicity.
2.3 Two-block structure of the input covariance To obtain an analytically tractable model, we assume that the singular values of X take only two values: a fraction α are equal to λ+ = 1 and a fraction 1 − α are equal to λ− ∈ (0, 1]. Equivalently, XX ⊤ has two eigenvalues, 1 and λ2− , with proportions α and 1 − α. The smaller λ− is, the more anisotropic the learning problem becomes.1 In order to simplify (6), let us write the compact singular value decomposition X = UΛV ⊤ , where U ∈ O( N ), V ∈ R M× N has orthonormal columns, and Λ = diag(1, . . . , 1, λ− , . . . , λ− ) ∈ R N × N . | {z } | {z } (1− α ) N
αN
Conjugating by U and using the rotational invariance of Ainit and of the Gaussian noise, we obtain einit := U ⊤ Ainit U, e := U ⊤ ZV, A Z ve := U ⊤ v, einit distributed as GOE (Gaussian Orthogonal Ensemble) and Z e having i.i.d. Gaussian with A entries of variance 1/M. In this basis, einit e−tΛ2 + ZΛ e −1 I − e−tΛ2 + θ veve⊤ I − e−tΛ2 . U ⊤ Aθt U = A (7) Therefore the symmetrized matrix has the same spectrum as 2 Setθ = Set0 + θ vewt⊤ + wt ve⊤ , wt := I − e−tΛ ve,
(8)
where ⊤ einit e−tΛ2 + e−tΛ2 A einit e −1 I − e−tΛ2 + I − e−tΛ2 Λ−1 Z e⊤ . Set0 = A + ZΛ
(9)
This representation makes the structure transparent: the noise part is a two-block Wigner-type matrix, while the teacher contributes a finite-rank perturbation which is rank one only in the isotropic case λ− = 1 and rank two otherwise. 1 This is where our assumption that γ ≥ 1 is needed. Indeed, when γ < 1 the matrix XX ⊤ has a null space
of dimension (1 − γ) N, which leads to a 3 × 3 Dyson system. See Appendix B.
4
Transient BBP Transition in Gradient Flow
Bulk spectrum via the 2 × 2 Dyson equation
3
3 Bulk spectrum via the 2 × 2 Dyson equation We now characterize the bulk spectrum of the noise matrix in the abstract two-block setting.
3.1 Two-block Wigner-type ensemble Define the block matrices D1,t = diag( at , . . . , at , bt , . . . , bt ), | {z } | {z }
D2,t = diag(ct , . . . , ct , dt , . . . , dt ), | {z } | {z }
(1− α ) N
αN
αN
(1− α ) N
with (for λ+ = 1) 2
−t
at = e ,
bt = e
−λ2− t
,
1 − e−λ− t dt = . λ−
−t
ct = 1 − e ,
(10)
Let W ∈ R N × N be GOE with the usual 1/N scaling, and let Γ ∈ R N × N have i.i.d. entries of variance 1/M, independent of W. Then (9) can be rewritten as St = WD1,t + D1,t W + ΓD2,t + D2,t Γ⊤ .
(11)
In the limit N, M → ∞ with M/N → γ ≥ 1, the off-diagonal variance profile is piecewise constant: N E[Sij2 ] → στ (i),τ ( j) , where τ (i ) ∈ { A, B} is the block label of i and σAA (t) = 4a2t +
2c2t , γ
σBB (t) = 4bt2 +
2d2t , γ
σAB (t) = ( at + bt )2 +
c2t + d2t . γ
(12)
Thus S is a Wigner-type matrix with a two-block variance profile [14, 13].
3.2 The 2 × 2 Dyson equation Let 1 (zI − St )−1 ii , ∑ N →∞ αN i∈ A
m A (z, t) = lim
1 (zI − St )−1 ii . ∑ N → ∞ (1 − α ) N i∈ B
m B (z, t) = lim
Proposition 3.1 (Dyson equation). For each z ∈ C+ and t ≥ 0, the pair (m A (z, t), m B (z, t)) is the unique solution of 1 = −z − α σAA (t) m A (z, t) − (1 − α) σAB (t) m B (z, t), m A (z, t) (13) 1 = −z − α σAB (t) m A (z, t) − (1 − α) σBB (t) m B (z, t), m B (z, t) with the Stieltjes-branch conditions ℑm A (z, t), ℑm B (z, t) < 0 for ℑz > 0 and m A (z, t), m B (z, t) ∼ −1/z as |z| → ∞. The global Stieltjes transform and limiting spectral density are m(z, t) = α m A (z, t) + (1 − α) m B (z, t),
ρ St ( λ ) = −
5
1 lim ℑ m(λ + iη, t). π η ↓0
(14)
Transient BBP Transition in Gradient Flow
3
Bulk spectrum via the 2 × 2 Dyson equation
A short proof is given in Appendix A. Note that when α = 1 or α = 0, (13) reduces to the scalar Dyson equation of a semicircle law. When a = b and c = d, one has σAA = σAB = σBB and the two-block system again collapses to a single semicircle. In the general case the complex behaviour of the spectral dynamics is illustrated in Figure 1 below. To conclude this part we mention that, should we have postulated an n-block structure, we would have obtained n coupled equations for the corresponding set {m A1 (z), m A2 (z), . . . , m An (z)}. The γ < 1 case, which introduces a frozen null-space block, leads to a 3 × 3 Dyson system and is treated in Appendix B; the resulting permanent noise narrows but does not eliminate the transient regime discussed below.
3.3 Empirical validation and dynamical phases We validate the Dyson prediction by simulating the full gradient flow (2) at logarithmically spaced times t ∈ [0.1, 2000], averaging over 20 independent realizations with N = 500, γ = 1, α = 0.5, λ+ = 1, λ− = 0.1. At each time we form St = At + A⊤ t and compare the empirical spectral density with the theoretical density ρSt obtained from the 2 × 2 Dyson equation via the Stieltjes inversion formula (14).
Density
t = 0.1
t = 0.246
t = 1.49
t = 3.66
0.2 0.1 0.0
2
0 2 t = 22.2
2
0 2 t = 54.6
0
5
0 5 t = 2e + 03
t = 330
Density
0.15
Simu Theo
0.10 0.05 0.00
5
0
λ
5
10
10
0 λ
10
20
0 λ
20
20
0 λ
20
Figure 1. Empirical spectral density of St (blue histogram, 20 realizations, N = 500) versus the theoretical density from the 2 × 2 Dyson equation (13)–(14) (black curve) at eight training times. Parameters: λ+ = 1, λ− = 0.1, γ = 1, and α = 0.5. In the signal case θ = 6, the red vertical line marks the outlier location as predicted in Section 4 when an outlier is present. One clearly sees how the outlier first emerges from the bulk before being reabsorbed.
Three dynamical phases are visible in Figure 1. At early times the spectrum is close to a single semicircle, since both blocks still behave similarly and the matrix is dominated by the GOE initialization. At intermediate times the two learning scales separate, the variance profile becomes genuinely two-block, and the bulk deforms accordingly. At very late times the GOE contribution has essentially decayed, while the factor ( XX ⊤ )−1 amplifies the slow block and broadens the support. Across the full dynamical range, the theoretical curve obtained from the Dyson equation agrees well with the empirical histogram. When the rank-one teacher is switched on (θ = 6, red vertical lines), one clearly sees how an outlier 6
Transient BBP Transition in Gradient Flow
4
Rank-two transient BBP transition
first detaches from the bulk at intermediate times and is then reabsorbed at late times; the full analysis of this transient BBP transition is the subject of the next section. Numerical experiments with a full-rank power-law input spectrum, where the covariance eigenvalues are drawn from a power-law distribution, confirm that the same three dynamical phases persist well beyond the two-block setting; we detail this extension in Appendix E.
4 Rank-two transient BBP transition 4.1 Outlier condition for a rank-one teacher We now analyze the effect of the rank-one teacher on St by considering 2
Stθ = St + θ vewt⊤ + wt ve⊤ ,
wt = ( I − e−tΛ )ve.
Because wt is proportional to ve only in the isotropic case λ− = 1, the perturbation is genuinely rank two at finite time, and the standard rank-one BBP formula [6] does not apply directly. We reduce the problem to a 2 × 2 determinantal condition in Appendix C. Proposition 4.1 (Outlier condition). Let L− (t) and L+ (t) denote the lower and upper edges of the limiting bulk spectrum of St . For θ ≥ 0, the perturbed matrix Stθ has an outlier eigenvalue ξ θ,t if and only if it solves 1 − θ ψt (ξ θ,t )
2
= θ 2 ϕt (ξ θ,t )χt (ξ θ,t ),
ξ θ,t ∈ R \ [ L− (t), L+ (t)],
(15)
where the deterministic functions ϕt , ψt , χt are defined in Appendix C, Eq. (21).
Remark 4.2 (Isotropic limit). In the isotropic case λ− = 1, one has wt = f t ve with f t = 1 − e−t . Hence ψt (z) = f t ϕt (z) and χt (z) = f t2 ϕt (z), so (15) reduces to 1 − 2θ f t ϕt (ξ θ,t ) = 0, which is exactly the standard rank-one BBP equation with effective spike strength 2θ f t .
Remark 4.3 (Boundary times). At t = 0, one has wt = 0, hence ψ0 ≡ χ0 ≡ 0, and (15) has no solution outside the bulk. At the other extreme, as t → ∞, one has wt → ve, so ϕt (z), ψt (z), χt (z) → m∞ (z) and (15) reduces to 1 − 2θ m∞ (ξ θ,∞ ) = 0, θ . which is the usual steady-state spike equation for the limiting matrix S∞
4.2 BBP thresholds and signal regimes The upper outlier first appears when (15) acquires a solution at the upper edge L+ (t). The corresponding critical signal level is θc (t) :=
1 ψt ( L+ (t)+ ) +
p
7
ϕt ( L+ (t)+ )χt ( L+ (t)+ )
.
(16)
Transient BBP Transition in Gradient Flow
4
Rank-two transient BBP transition
For fixed signal strength θ, define the fitting set
Tθ := {t > 0 : θ > θc (t)}.
Theorem 4.4 (Three signal regimes). In the limit N, M → ∞ with M/N → γ > 1, exactly one of the following scenarios occurs: 1. Weak-signal regime: Tθ = ∅. No outlier ever separates from the bulk, and the teacher direction is never spectrally detectable. 2. Strong-signal regime: Tθ = (t1 , ∞) for some t1 > 0. An outlier emerges at time t1 and remains separated from the bulk thereafter. 3. Early-stopping regime: Tθ = (t1 , t2 ) for some 0 < t1 < t2 < ∞. An outlier exists only on a finite time window. This is the transient BBP regime associated with early stopping. In the early-stopping regime, t1 and t2 are interpreted as follows: for t < t1 the model is underfitted and the teacher direction cannot be determined; for t1 < t < t2 the teacher is spectrally detectable in the sense that the eigenvector u⋆ corresponding to the spike has a computable, non-zero overlap (u⋆ · v)2 with the teacher direction (see e.g. [18], section 14.2.2); and for t > t2 the outlier is reabsorbed into the bulk, corresponding to a late-time overfitting phase. The optimal stopping time, defined as the topt ∈ (t1 , t2 ) that maximizes (u⋆ · v)2 , is discussed in Section 4.4 below.
12
=1 = 0.1
10 8 c(t)
We illustrate this behaviour by plotting the evolution in time of θc (t) in Figure 2. When λ+ = λ− = 1, the function is monotone, so that only regimes 1 and 2 are possible depending on the value of θ. On the contrary, in the anisotropic regime λ− = 0.1, the function is no longer monotone: θc first decreases (the fast block reveals the teacher) and then increases again (the slow block, amplified by Λ−1 , injects enough noise to hide the signal). This means that, for an intermediate range of values of θ, a spike is visible only during a finite time interval – this is the early-stopping regime, and the minimum of θc is the best detection threshold achievable along the whole training trajectory.
6 4 2 10 1
100
101
Training time t
102
Figure 2. Evolution of θc defined in (16) for α = 0.5 and two values of λ− .
4.3 Phase diagrams and the role of anisotropy The curve θ = θc (t) in the (θ, t) plane directly separates the three regimes of Theorem 4.4. Horizontal cuts at fixed θ give the fitting set Tθ , and the minimum of the boundary curve corresponds to the minimum of θc (t) visible in Figure 3. As the covariance becomes more
8
Transient BBP Transition in Gradient Flow
4
Rank-two transient BBP transition
Signal strength
102
Never out Stays out In-and-out t1 ( ) t2 ( ) topt( )
101
100
10 1
100
101
Training time t
102
103
Figure 3. Phase diagram in the (θ, t) plane for λ− = 0.1, γ = 1 and α = 0.5. The critical curve θ = θc (t) separates three regimes: no outlier (underfitting), transient outlier (early-stopping window), and persistent outlier. For a fixed intermediate value of θ, the outlier emerges at t1 and is reabsorbed at t2 . The overlaid red curve shows the optimal stopping time topt (θ ) defined in Section 4.4: in our experiments it lies strictly inside the transient window (t1 , t2 ) whenever the latter exists.
anisotropic (smaller λ− ), the transient regime broadens; as the signal becomes stronger, the fitting window opens earlier and may become persistent. To examine how anisotropy controls the possible learning regimes, we scan the (θ, λ− ) plane and, for each pair, count the number of sign changes of F (θ, t) := 1 − θψt ( L+ (t)+ )
2
− θ 2 ϕt ( L+ (t)+ )χt ( L+ (t)+ )
over t ∈ [0.05, 3000]. The resulting (θ, λ− ) phase diagram is shown in Figure 4. Two qualitative features are worth emphasizing. First, the yellow “in-and-out” wedge – the regime in which early stopping is the only way to recover the teacher – opens only below a critical anisotropy threshold λ⋆− : for nearly isotropic inputs (λ− close to 1) the transient regime shrinks and eventually disappears, and only the weak- and strong-signal regimes survive. This is the practitioner-facing summary of the mechanism: early stopping is genuinely useful only when the data are sufficiently anisotropic. Second, even for strongly anisotropic inputs, the early-stopping regime is bounded below in θ: below a minimum signal strength, no amount of well-chosen stopping time recovers the teacher. Taken together, Figures 3 and 4 provide a complete map of the parameter region in which early stopping is a principled necessity rather than a mere heuristic.
4.4 Teacher recovery and optimal stopping The regime classification of Theorem 4.4 tells us when the teacher is spectrally detectable, but not how well it is reconstructed at any given time. The natural scalar summary of teacher 9
Covariance anisotropy
Transient BBP Transition in Gradient Flow
4
Rank-two transient BBP transition
Never out Stays out In-and-out c, 1( ) c, 2( ) 10 1
10 2 1 10
100
Signal strength
101
Figure 4. Outlier-regime classification in the (θ, λ− ) plane for λ+ = 1 and γ = 1, α = 0.5, obtained by scanning over t ∈ [0.05, 3000]. Blue: no outlier ever emerges. Red: an outlier emerges and remains separated from the bulk. Yellow: an outlier emerges and is later reabsorbed, corresponding to the transient early-stopping regime.
recovery is the squared overlap qt := (u⋆t · v)2 between the outlier eigenvector u⋆t and the true teacher direction v, with the convention qt = 0 when no outlier exists. This is the quantity that controls the rank-one content of the trained weight matrix along the teacher direction. Because the outlier is a simple real zero of the 2 × 2 Schur complement Mt (z) introduced in Appendix C, the overlap qt can be obtained, via the Woodbury identity, as a residue of v⊤ (zI − Stθ )−1 v at z = ξ θ,t . This residue is a rational function of the deterministic quantities ϕt , ψt , χt and their derivatives at ξ θ,t ; we sketch the derivation in Appendix C and do not expand the expression here, since it is not used in what follows. In the isotropic limit λ− = 1 it collapses to the classical Benaych-Georges–Nadakuditi formula for rank-one BBP. What matters for the discussion below is a qualitative feature of qt : it is continuous on the fitting set Tθ , vanishes on its boundary (as the outlier merges into the bulk edge and the eigenvector delocalizes over the bulk), and is strictly positive in its interior. In the early-stopping regime Tθ = (t1 , t2 ), qt therefore attains at least one interior maximum, which defines an optimal early-stopping time topt (θ ) ∈ arg max qt . t∈(t1 ,t2 )
Numerical evidence. We computed topt (θ ) by one-dimensional maximization of qt over the window (t1 (θ ), t2 (θ )) already located by the phase-diagram computation, for the full range of θ covered by Figure 3. The result is overlaid as the red curve in Figure 3. In our experiments qt is unimodal on (t1 , t2 ), so topt (θ ) is a single point, and it lies strictly inside the transient window for every θ in the early-stopping regime. The curve interpolates 10
Transient BBP Transition in Gradient Flow
5
Conclusion and Extensions
smoothly into the strong-signal regime, tracing the locus of best teacher recovery along the entire (θ, t) plane. In the linear teacher–student setting, qt is, up to a contribution from the bulk of the trained weight matrix, the scalar that controls the rank-one component of the reconstruction error between the trained weights and the teacher Ateach ; a rigorous test-error decomposition is beyond the scope of this paper and is left to future work.
5 Conclusion and Extensions We introduced a solvable random-matrix model in which the spectrum of a symmetrized weight matrix undergoes a transient BBP transition along gradient flow. Depending on signal strength and covariance anisotropy, the teacher spike may never emerge, may emerge and persist, or may exist only during a finite time window. In this last regime, the overlap between the outlier eigenvector and the teacher direction attains an interior maximum along the training trajectory, which numerically identifies an optimal stopping time and gives a minimal theoretical mechanism for early stopping as a transient spectral effect. The driving ingredient is anisotropy of the input covariance. Fast directions reveal the teacher early, whereas slow directions continue to carry noise and can eventually hide the signal again within the fat tail generated by the inverse covariance in the least-squares solution. In this sense, overfitting is not an abstract late-time pathology in our model: it is a concrete consequence of spectral heterogeneity in the data. Connection to empirical observations. Although our model is deliberately simple, it qualitatively reproduces several features reported in empirical spectral studies of neuralnetwork training [2, 3, 4, 5, 10]: the appearance of isolated eigenvalues at intermediate training times, the broadening of the bulk at late times, and the association of the best generalization time with the transient rather than the asymptotic spectral structure. What our model does not capture is the genuinely heavy-tailed bulk and the multi-layer nonlinear coupling observed in deep networks at convergence; these remain outside the scope of a two-block linear teacher–student setup and should be addressed by complementary models. Finally, the present framework suggests a connection with the Neural Tangent Kernel (NTK) perspective [19]. In the infinite-width limit, gradient flow with mean-square loss leads to linear output dynamics governed by the NTK matrix, whose spectrum plays the same role as XX ⊤ in our analysis; see also [20]. This suggests that the transient BBP mechanism identified here may provide a useful toy model for understanding why early stopping is often effective in more complex architectures.
11
Transient BBP Transition in Gradient Flow
References
References [1] Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643–1697, 2005. [2] Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021. [3] Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix analysis of deep neural network weight matrices. Physical Review E, 106(5):054124, 2022. [4] Max Staats, Matthias Thamm, and Bernd Rosenow. Boundary between noise and information applied to filtering neural network weight matrices. Physical Review E, 108(2):L022302, 2023. [5] David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, and Matthew R Walter. Approaching deep learning through the spectral dynamics of weights. arXiv preprint arXiv:2408.11804, 2024. [6] Florent Benaych-Georges and Raj Rao Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Advances in Mathematics, 227(1):494–521, 2011. [7] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014. [8] Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428–446, 2020. [9] Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous-time view of early stopping for least squares regression. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1370–1378. PMLR, 2019. [10] Brian Richard Olsen, Sam Fatehmanesh, Frank Xiao, Adarsh Kumarappan, and Anirudh Gajula. From sgd to spectra: A theory of neural network weight dynamics. arXiv preprint arXiv:2507.12709, 2025. [11] Tony Bonnaire, Giulio Biroli, and Chiara Cammarota. The role of the time-dependent hessian in high-dimensional optimization. Journal of Statistical Mechanics: Theory and Experiment, 2025(8):083401, 2025. [12] Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences, 116(12):5451–5460, 2019. [13] Oskari H Ajanki, László Erdős, and Torben Krüger. Universality for general Wignertype matrices. Probability Theory and Related Fields, 169(3):667–727, 2017. 12
Transient BBP Transition in Gradient Flow
References
[14] Oskari H Ajanki, László Erdős, and Torben Krüger. Stability of the matrix Dyson equation and random matrices with correlations. Probability Theory and Related Fields, 173(1):293–373, 2019. [15] Antoine Maillard, Laura Foini, Alejandro Lage Castellanos, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. High-temperature expansions and message passing algorithms. Journal of Statistical Mechanics: Theory and Experiment, 2019(11):113301, 2019. [16] Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592, 2024. [17] Blake Bordelon and Cengiz Pehlevan. Disordered dynamics in high dimensions: Connections to random matrices and machine learning. arXiv preprint arXiv:2601.01010, 2026. [18] Marc Potters and Jean-Philippe Bouchaud. A first course in random matrix theory: for physicists, engineers and data scientists. Cambridge University Press, 2020. [19] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, pages 8571–8580, 2018. [20] Lechao Xiao, Jeffrey Pennington, and Samuel S Schoenholz. Disentangling trainability and generalization in deep neural networks. In Proceedings of the 37th International Conference on Machine Learning, pages 10462–10472. PMLR, 2020.
13
Transient BBP Transition in Gradient Flow
A
Proof of Proposition 3.1
A Proof of Proposition 3.1 Proof. We fix a time t ≥ 0 and drop all time indexes in the proof for notational simplicity. Write ri := ( D1 )ii , qi := ( D2 )ii , so that ri ∈ { a, b} and qi ∈ {c, d} depending on whether i belongs to block A or block B. From (11), for i < j we have Sij = (ri + r j )Wij + q j Γij + qi Γ ji . Since the families {Wij }i≤ j and {Γij }i,j are independent and centered, the upper-triangular entries of S are independent and centered as well. Moreover, 2
N ESij2 = (ri + r j )2 +
2
qi + q j N 2 (qi + q2j ) −→ (ri + r j )2 + . M γ
Hence the variance profile is piecewise constant, and for off-diagonal entries it is exactly σAA = 4a2 +
2c2 , γ
σBB = 4b2 +
2d2 , γ
σAB = ( a + b)2 +
c2 + d2 , γ
as claimed in (12). The diagonal entries also have variance of order N −1 and fit into the same Wigner-type framework. Now consider the standard resolvent G (z) := (S − zI )−1 ,
z ∈ C+ .
By the general theory of Wigner-type matrices with variance profile sij := E|Sij |2 , the b i (z) solving the quadratic diagonal resolvent entries converge to deterministic limits m vector equation N 1 b j ( z ), − = z + ∑ sij m b i (z) m j =1 b i (z) > 0 for ℑz > 0 [14, 13]. with ℑm Since the variance profile has only two block values, the solution is constant on each block: b i (z) = m b A ( z ) ( i ∈ A ), b i (z) = m b B ( z ) ( i ∈ B ). m m Therefore the quadratic vector equation reduces to
−
1 b A (z) + (1 − α) σAB m b B ( z ), = z + α σAA m b m A (z)
−
1 b A (z) + (1 − α) σBB m b B ( z ). = z + α σAB m b B (z) m
Finally, Proposition 3.1 is written in terms of the resolvent
(zI − S)−1 = − (S − zI )−1 = − G (z). Thus, setting b A ( z ), m A (z) := −m
b B ( z ), m B (z) := −m 14
Transient BBP Transition in Gradient Flow
The case γ < 1: A 3 × 3 Dyson system
B
we obtain exactly 1 = −z − α σAA m A (z) − (1 − α) σAB m B (z), m A (z) 1 = −z − α σAB m A (z) − (1 − α) σBB m B (z). m B (z) Moreover ℑm A (z), ℑm B (z) < 0 for ℑz > 0, and m A (z), m B (z) ∼ −1/z as |z| → ∞. The averaged limit m ( z ) = α m A ( z ) + (1 − α ) m B ( z ) is therefore the Stieltjes transform of the limiting empirical spectral measure in the convention of (14), and the density is recovered by the inversion formula ρS (λ) = −
1 lim ℑ m(λ + iη ). π η ↓0
This proves the proposition.
B The case γ < 1: A 3 × 3 Dyson system When M < N (i.e. γ = M/N < 1), the input covariance XX ⊤ has a null space of dimension ⊤ (1 − γ) N. In the null space, the gradient flow never moves: the exponential e−tXX acts as ⊤ the identity, and the noise term ZX ⊤ ( XX ⊤ )−1 ( I − e−tXX ) vanishes (since X ⊤ annihilates the null space). Consequently, the weight matrix retains its random initialisation in those directions for all time. This introduces a third block in the variance profile of the symmetrised noise matrix, leading to a 3 × 3 Dyson system that generalises the 2 × 2 system of Section 3. We also drop the time index for simplicity.
B.1 Three-block structure We split the N indices into three classes: Block
Fraction
ap
cp
A (fast)
α A = γ/2
e−t
1 − e−t
B (slow)
α B = γ/2
e −λ− t
(1 − e−λ− t )/λ−
C (null)
αC = 1 − γ
1
0
2
2
The null-space block C has aC = 1 (the initialisation is never damped) and cC = 0 (no noise is injected through X).
15
Transient BBP Transition in Gradient Flow
B
The case γ < 1: A 3 × 3 Dyson system
Using the general variance formula σpq = ( a p + aq )2 + (c2p + c2q )/γ, we obtain six variance parameters: 2c2 σAA = 4a2A + A , γ σBB = 4a2B +
2c2B , γ
σCC = 4, σAB = ( a A + a B )2 +
(17)
c2A + c2B , γ
σAC = (1 + a A )2 +
c2A , γ
σBC = (1 + a B )2 +
c2B . γ
Note that σCC = 4 is time-independent: the null-space block is a frozen GOE contribution at all times.
B.2 The 3 × 3 Dyson equation The partial Stieltjes transforms m A (z), m B (z), mC (z) satisfy: γ γ 1 = −z − σAA m A − σAB m B − (1 − γ) σAC mC , m A (z) 2 2 1 γ γ = −z − σAB m A − σBB m B − (1 − γ) σBC mC , m B (z) 2 2 γ γ 1 = −z − σAC m A − σBC m B − (1 − γ) σCC mC , mC (z ) 2 2
(18)
with the Stieltjes-branch conditions ℑ m p (z) < 0 for ℑ z > 0 and m p (z) ∼ −1/z as |z| → ∞. The global Stieltjes transform and spectral density are m(z) =
γ γ m A ( z ) + m B ( z ) + (1 − γ ) m C ( z ), 2 2
ρS (λ) = −
1 lim ℑ m(λ + iη ). π η ↓0
(19)
B.3 Effect of the null space on the transient BBP window Compared to the oversampled regime γ > 1, the null-space block C introduces a frozen GOE component of variance σCC = 4 that competes with the signal at all times. This has two consequences: 1. The bulk spectrum is generically wider (since the frozen initialisation noise never decays), making it harder for an outlier to emerge. 2. The null-space directions do not contribute to ψ or χ (since they carry no learned signal), but they do contribute to ϕ (through mC ), which enters the right-hand side of the outlier equation. As a result, the transient BBP window [t1 , t2 ] is generically narrower for γ < 1 than for γ > 1: more directions carry permanent noise without contributing to learning, raising the effective detection threshold θc (t). 16
Transient BBP Transition in Gradient Flow
C
Outlier equation and BBP transition
C Outlier equation and BBP transition C.1 Weighted finite-rank teacher perturbation In the covariance eigenbasis, the signal contribution to the symmetrized matrix is 2 θ vewt⊤ + wt ve⊤ , wt = ( I − e−tΛ )ve. This perturbation is rank two unless wt is proportional to ve, which happens only in the isotropic case. Thus the standard rank-one BBP formula does not directly apply at finite time [6].
C.2 Rank-two determinant and outlier condition To simplify notation, we drop the tildes and the time subscript and write the perturbed matrix as Sθ = S + θ (wv⊤ + vw⊤ ). Introduce U = [w, v] ∈ R N ×2 ,
V = [v, w] ∈ R N ×2 ,
and let R(z) = (zI − S)−1 . By the matrix determinant lemma, det(zI − Sθ ) = det(zI − S) det I2 − θV ⊤ R(z)U . The 2 × 2 Schur complement is M (z) = I2 − θ
v⊤ R(z)w v⊤ R(z)v w⊤ R(z)w w⊤ R(z)v
!
=
! 1 − θψ(z) −θϕt (z) , −θχ(z) 1 − θψ(z)
where ϕt (z) = v⊤ R(z)v,
ψt (z) = v⊤ R(z)w,
χt (z) = w⊤ R(z)w.
(20)
Concentration of quadratic forms. Since v is asymptotically isotropic in the eigenbasis of the noise matrix and the two blocks have deterministic weights, the quadratic forms concentrate according to ϕt (z) −−−→ α m A,t (z) + (1 − α) m B,t (z), N →∞
ψt (z) −−−→ α f A,t m A,t (z) + (1 − α) f B,t m B,t (z), N →∞
(21)
2 2 χt (z) −−−→ α f A,t m A,t (z) + (1 − α) f B,t m B,t (z), N →∞
with
2
2
f A,t = 1 − e−λ+ t ,
f B,t = 1 − e−λ− t .
Outlier equation. Eigenvalues of Sθ outside the bulk correspond to real zeros of det M (z). Computing the determinant gives 2 det M (z) = 1 − θψt (z) − θ 2 ϕt (z)χt (z) = 0, (22) which is the outlier equation used in Proposition 4.1. 17
Transient BBP Transition in Gradient FlowE
Full-rank power-law spectrum: numerical validation
Eigenvector overlap with the teacher. Given a simple outlier root ξ θ of (22), the projections of the associated eigenvector u⋆ of Sθ onto v and w concentrate to deterministic limits, while the component orthogonal to span(v, w) is delocalized over the bulk. The squared overlap qt = (u⋆ · v)2 with the teacher direction is obtained, via the Woodbury identity, as the residue of the resolvent v⊤ (zI − Sθ )−1 v at z = ξ θ ; this reduces to differentiating det M (z) at ξ θ and yields a rational function of ϕt , ψt , χt and their derivatives at ξ θ . We omit the explicit expression, which is not used in the main text beyond the qualitative properties of qt discussed in Section 4.4.
D Fixed-point algorithm for the Dyson equation Proposition 3.1 characterizes (m A (z), m B (z)) as the unique physical solution of (13) with ℑm A (z), ℑm B (z) < 0 for ℑz > 0. Numerically, we solve this system by fixed-point iteration, (0) (0) initializing m A = m B = −1/z and iterating ( n +1)
mA
=
−1 (n) (n) z + ασAA m A + (1 − α)σAB m B
( n +1)
mB
,
=
−1 (n) (n) z + ασAB m A + (1 − α)σBB m B
.
(23)
Numerical implementation. For z = x + iε with ε > 0, the iteration preserves the physical branch and converges robustly in our experiments. In practice we use ε = 10−2 and stop when the ℓ∞ change between successive iterates is below 10−10 . Bulk edge and phase boundary. The upper edge L+ is estimated from the rightmost point on a fine real grid where the numerical density exceeds a small threshold. For each fixed θ, we then evaluate 2 2 + + F (θ ) = 1 − θψ( L+ + ) − θ ϕ( L+ )χ( L+ ) Doing this on a coarse grid in t and refining each sign change by bisection yields the transition times and the phase diagrams shown in the main text.
E Full-rank power-law spectrum: numerical validation The two-block covariance model of Section 2.3 is analytically tractable because it reduces the Dyson system to two equations. Here we check numerically that the transient BBP phenomenology is not an artefact of that simplification, by replacing the two-block spectrum with a continuous power-law distribution. Setup. We draw the N singular values of the input matrix X independently from the truncated power-law density p(λ) ∝ λ− β ,
λ ∈ [λmin , λmax ],
with β = 1.5, λmin = 0.1, λmax = 5, and N = 400. The rest of the setup follows Section 2 exactly: the teacher is rank-one with signal amplitude θ, the initialisation is GOE at scale √ 1/ N, and the noise matrix has i.i.d. N (0, 1/N ) entries. Working in the eigenbasis of XX ⊤ (with eigenvalues µi = λ2i ), the closed-form gradient-flow solution (2) becomes 1 − e−tµ −tµ At = Q e ·G + · WZ , µ 18
Transient BBP Transition in Gradient FlowE
Full-rank power-law spectrum: numerical validation
where Q is the eigenvector matrix of XX ⊤ , µ = diag(µ1 , . . . , µ N ), G = Q⊤ Ainit , and WZ = Q⊤ ( XY0⊤ ) encodes the teacher-plus-noise contribution. Simulations are run over 8 independent realisations at each of 85 logarithmically spaced times t ∈ [0.05, 1000] and averaged. Results. Figure 5 shows the squared overlap qt = (u⋆t · v)2 between the top eigenvector of St = At + A⊤ t and the teacher direction v, for seven values of the signal amplitude θ.
0.8
(u * v)2
0.6 0.4
= 0.5 = 1.0 = 1.5 = 2.0
= 3.0 = 5.0 = 10.0 random 1/N
102
103
0.2 0.0 10 1
100
101
Training time t
Figure 5. Teacher-direction overlap qt = (u⋆t · v)2 as a function of training time t for a full-rank power-law covariance spectrum (β = 1.5, λ ∈ [0.1, 5], N = 400, 8 realisations averaged). The dotted horizontal line marks the random baseline 1/N. Weak signals (θ ≲ 1) never produce a detectable outlier. Intermediate signals show a clear transient: the overlap rises, peaks, and returns to baseline, reproducing the early-stopping regime of Theorem 4.4. Strong signals produce a persistent outlier with a sustained high overlap.
The three qualitative regimes of Theorem 4.4 are all visible: for θ ≲ 1 the overlap never exceeds the random baseline 1/N (weak-signal regime); for intermediate θ the overlap rises, attains an interior maximum, and decays back to baseline (early-stopping regime); and for large θ the overlap rises and remains elevated throughout training (strong-signal regime). The peak overlap increases with θ and the transient window widens, consistent with the phase-diagram picture of Section 4.3. These observations confirm that the transient BBP mechanism is robust to the specific form of the covariance spectrum and is not a peculiarity of the two-block simplification.
19