L OOP C ORRECTIONS TO THE T RAINING AND G ENER ALIZATION E RRORS OF R ANDOM F EATURE M ODELS
arXiv:2604.12827v1 [cs.LG] 14 Apr 2026
Taeyoung Kim Center for AI and Natural Sciences Korea Institute for Advanced Study Seoul 02455, South Korea [email protected]
A BSTRACT We investigate random feature models in which neural networks sampled from a prescribed initialization ensemble are frozen and used as random features, with only the readout weights optimized. Adopting a statistical-physics viewpoint, we study the training, test, and generalization errors beyond the mean-kernel approximation. Since the predictor is a nonlinear functional of the induced random kernel, the ensemble-averaged errors depend not only on the mean kernel but also on higher-order fluctuation statistics. Within an effective field-theoretic framework, these finite-width contributions naturally appear as loop corrections. We derive the loop corrections to the training, test, and generalization errors, obtain their scaling laws, and support the theory with experimental verification.
Keywords: random feature models, generalization error, finite-width effects, loop corrections, effective field theory, kernel fluctuations
1
I NTRODUCTION
1.1
R ANDOM F EATURE M ODELS
Random feature models provide a mathematically tractable framework for studying the training and generalization properties of neural networks. In these models, a nonlinear feature map is first generated by randomly sampling the parameters of a neural network from a prescribed initialization ensemble and then freezing them, while only the readout layer is optimized. This setting is closely related to the literature on random kitchen sinks, randomized kernel training, and extreme learning machines (Rahimi & Recht, 2007; 2009; Rudi et al., 2017; Huang et al., 2006; Wang et al., 2022). A central object in the analysis of random feature models is the kernel induced by the frozen random features. In the infinite-width limit, random networks often admit deterministic kernel descriptions, and this connection has played a major role in the modern theoretical understanding of wide neural networks (Jacot et al., 2018; Lee et al., 2019). In this regime, the predictor is well approximated by a kernel ridge regression problem associated with the limiting kernel, which has led to substantial progress in understanding training, optimization, and generalization in overparameterized models (Rudi et al., 2017; Adlam & Pennington, 2020). However, realistic models operate at finite width, where the induced kernel fluctuates around its ensemble mean. These fluctuations are not merely a technical nuisance: they can produce nontrivial deviations from the mean-kernel theory and, in sufficiently deep or moderately wide regimes, may remain significant even when the width is large (Hanin & Nica, 2020; Seleznova & Kutyniok, 2022; Hanin, 2024). Since the learned readout depends nonlinearly on the random feature matrix, the training, test, and generalization errors are not determined solely by the mean kernel. Instead, they depend on higher-order statistics of kernel fluctuations. This naturally raises the question of how to systematically quantify finite-width corrections beyond the leading deterministic kernel. 1
1.2
A N E FFECTIVE -T HEORETIC P ERSPECTIVE ON F INITE -W IDTH F LUCTUATIONS
In this paper, we approach this problem from the viewpoint of statistical physics and effective field theory. From this perspective, the frozen random features are ensemble-valued degrees of freedom, and the relevant observables—such as the training error, test error, and generalization gap—should be understood through their ensemble statistics. The standard mean-kernel approximation then plays the role of a leading-order, or tree-level, description, while finite-width effects appear as corrections generated by fluctuations around that mean (Roberts et al., 2022b; Hanin, 2024). This viewpoint is useful because the predictor depends on the inverse of a regularized empirical kernel matrix. Consequently, even if the kernel fluctuations themselves are small, the induced fluctuations of observables are organized nonlinearly through resolvent-type expansions. A systematic treatment therefore requires more than replacing the kernel by its ensemble average: one must also track the covariance structure of the kernel fluctuations and, beyond the Gaussian level, their higher connected cumulants. In the language of field theory, such higher-order contributions are naturally interpreted as loop corrections (Roberts et al., 2022b; Hanin, 2024). Recent works have shown that finite-width random neural networks admit perturbative descriptions in powers of inverse width, and that connected correlation functions provide the natural building blocks for organizing these corrections (Hanin, 2024; Hanin & Nica, 2020). Motivated by this structure, we formulate a loop expansion for random feature ridge regression and apply it to three basic observables: the training error, the test error, and the generalization gap. Our analysis shows that the generalization gap is sensitive not only to fluctuations of the train–train kernel, but also to mixed fluctuation structures coupling train–train and test–train kernel quantities. This reveals a richer finite-width structure than is visible in the mean-kernel limit alone. 1.3
C ONTRIBUTIONS
The main contributions of this paper are as follows. • First, we develop a perturbative framework for random feature models that systematically expands the training error, test error, and generalization gap around the mean-kernel limit. From an effective-theoretic perspective, these finite-width corrections are governed not only by the mean kernel but also by higher-order fluctuation statistics, and can be naturally organized as loop corrections. • Second, we derive explicit loop expansions for the training error, test error, and generalization gap. At leading order, our formulas recover the deterministic mean-kernel prediction, while the subleading terms quantify corrections induced by kernel fluctuations. Our analysis further shows that the generalization gap involves mixed train–test fluctuation terms that are absent from the training error alone. • Third, we study the scaling laws of these correction terms with respect to the model width. This allows us to distinguish dominant Gaussian contributions from genuinely nonGaussian effects and to identify the perturbative regime in which the mean-kernel approximation remains accurate. • Finally, we support the theoretical predictions with experiments, verifying the predicted scaling behavior and showing that the loop-based description captures finite-width deviations from the mean-kernel theory.
2
P RELIMINARIES
In this section, we mathematically define the model under consideration and the problem we aim to analyze. We also explain the motivation for introducing loop corrections from a field-theoretic perspective. 2.1
M ODEL AND P ROBLEM S ETTING
We consider the model defined by the following equation 1. Here, Θ is randomly sampled and then kept frozen, while the model is realized by explicitly determining the weights {wi }. Here, for each 2
layer i, we take Wi ∈ Rni ×ni−1 ,
bi ∈ Rni ,
with n0 = dim(X ), and Θ = (W1 , . . . , WL , b1 , . . . , bL ). The parameter Θ can be regarded as an ordered tuple of parameters, each drawn from a certain statistical ensemble. f (x; {Θ, wi }i=1,...,n ) := ϕT (x; Θ)w.
(1)
The corresponding frozen neural-network random feature ϕ = (ϕ1 , . . . , ϕn )T , determined by Θ, is defined as follow equation 2. ϕ(x; Θ) := WL σ(z (L−1) ) + bL , z (k) = Wk σ(z (k−1) ) + bk , z
(0)
k = 1, . . . , L − 1,
(2)
= x.
Under this setting, given the training dataset DN = {(xj , yj )}j=1...,N , the learning problem reduces to the following ridge regression problem. N X 1
min
(w1 ,...,wn )∈Rn
N j=1
∥f (xj ; · · · ) − yj ∥2 + λ
n X
wi2 .
For notational convenience, by defining Φji := ϕi (xj ; Θ), Φ ∈ RN ×n . (w1 , . . . , wn )T , y = (y1 , . . . , yN )T . it can be written in the simplified form min
1
W∈Rn N
(3)
i=1
∥Φw − y∥2 + λ∥w∥2 .
and w
=
(4)
and its explicit solution is given by w∗ (Θ; DN ) = (ΦT Φ + N λIn )−1 ΦT y.
(5)
Let µ denote the population distribution of the data on the input space X . As a basic assumption, we suppose that the target value is deterministically specified by the input. Under this setting, the test loss is expressed as Z Etest (Θ; DN ) := ∥f (x; {Θ, w∗ (Θ; DN )}) − y∥2 dµ(x). (6) X
2.2
E NSEMBLE AVERAGING AND THE O RIGIN OF L OOP C ORRECTIONS
For each realization of the frozen parameter Θ, the readout weights are determined by the ridge objective 1 ∗ 2 2 w (Θ; DN ) = arg minn ∥Φ(Θ)w − y∥ + λ∥w∥ . (7) w∈R N This induces the predictor fΘ,DN (x) = ϕ(x; Θ)⊤ w∗ (Θ; DN ),
(8)
which is random through the ensemble draw Θ. Accordingly, the training error, test error, and generalization gap are themselves random observables, and the central objects of interest are their ensemble averages over Θ. A naive mean-field training replaces the random kernel KΘ by its ensemble mean K̄ = EΘ [KΘ ]. However, this is not sufficient in general, because the observables depend nonlinearly on KΘ through the resolvent (KΘ + N λIN )−1 . Therefore, ensemble-averaged errors depend not only on the mean kernel but also on higher-order fluctuation statistics of KΘ . This provides the basic motivation for the loop expansion developed below. At leading order, one recovers the mean-kernel prediction, while subleading terms are generated by kernel fluctuation cumulants. In the language of effective field theory, these subleading finite-width contributions are naturally interpreted as loop corrections. 3
3
E FFECTIVE F ORMULATION AND L OOP E XPANSION
In this section, building on the setting introduced in Section 2, we derive the training, test, and generalization errors of the random feature model with leading loop corrections taken into account. We then study their scaling laws and investigate how these loop corrections influence the model’s performance and generalization behavior in following Section 4. While one may in principle include vertices of arbitrarily high order in the loop expansion, their contributions become increasingly negligible according to the scaling laws. We therefore adopt the following proposition from Roberts et al. (2022a), based on the two-point and four-point vertices, as the basic building block for the subsequent analysis. (ℓ)
Proposition 3.1 (Leading non-Gaussian building block). Let zi;α denote the preactivation of neuron i at layer ℓ evaluated on the sample xα ∈ D, and define the stochastic kernel (ℓ) (ℓ) Ĝ(ℓ) α1 α2 := Cb + CW
with mean kernel Let
1
nℓ−1 (ℓ)
X
(ℓ)
σj;α1 σj;α2 ,
nℓ−1 j=1
(9)
i h i h (ℓ) (ℓ) (ℓ) G(ℓ) α1 α2 := E Ĝα1 α2 = E zi;α1 zi;α2 .
(10)
(ℓ) (ℓ) ∆Ĝ(ℓ) α1 α2 := Ĝα1 α2 − Gα1 α2 ,
(11)
and define the four-point vertex by h i 1 (ℓ) (ℓ) (12) V(α1 α2 )(α3 α4 ) := E ∆Ĝ(ℓ) α1 α2 ∆Ĝα3 α4 . nℓ−1 Then, in the wide but finite-width regime, the first departure from Gaussianity appears at order 1 ) through the connected four-point correlator: O( nℓ−1 h i (ℓ) (ℓ) (ℓ) (ℓ) E zi1 ;α1 zi2 ;α2 zi3 ;α3 zi4 ;α4 c 1 h (ℓ) (ℓ) = δi i δi i V + δi1 i3 δi2 i4 V(α1 α3 )(α2 α4 ) (13) nℓ−1 1 2 3 4 (α1 α2 )(α3 α4 ) i (ℓ) + δi1 i4 δi2 i3 V(α1 α4 )(α2 α3 ) . Therefore, to leading order beyond the Gaussian training, the pair G(ℓ) , V (ℓ) provides the basic building block of the effective description: G(ℓ) determines the Gaussian two-point statistics, while 1 V (ℓ) captures the leading non-Gaussian correction of order O( nℓ−1 ). For notational simplicity, in the sequel we assume that all hidden-layer widths and the output feature dimension are equal to n, namely n1 = · · · = nL = n. Accordingly, n will denote the common width parameter controlling the finite-width expansion. The proposition shows that the first finite-width correction to the Gaussian theory is encoded in the connected four-point correlator, whose strength is suppressed by the inverse width. Hence, the twopoint kernel G(ℓ) and the four-point vertex V (ℓ) provide the minimal building blocks for describing the leading loop correction in random feature models. Before turning to the detailed computation of the corrections to the training, test, and generalization errors, we first introduce the following kernel quantities for a fixed parameter Θ: KΘ := ΦΦT ,
(14)
kΘ (x) := ϕ(Θ; x)T ΦT . Then, the random feature model can be written in terms of these kernels as f (Θ; x) = ϕ(Θ; x)T w∗ (Θ; DN ) = ϕ(Θ; x)T ΦT (ΦΦT + N λIN )−1 y −1
= kΘ (x)(KΘ + N λIN ) 4
y.
(15)
To connect Proposition 3.1 with the error expansions below, we fix a layer L and suppress the layer index for simplicity. We identify the random kernel on the training set with the restriction of the stochastic kernel Ĝ to DN × DN , and write KΘ = K̄ + ∆,
K̄αβ = Gαβ ,
EΘ [∆αβ ∆γδ ] =
1 V(αβ)(γδ) , n
(16)
where n denotes the relevant width parameter. For convenience, for any matrix A ∈ RN ×N , define the V -contraction N X (V ⋆ A)αδ := V(αβ)(γδ) Aβγ . (17) β,γ=1
Then
1 (V ⋆ A). (18) n To organize the perturbative expansion, we henceforth assume the standard finite-width power counting in which the kernel fluctuation satisfies ∆ = O(n−1/2 ) in the wide-width regime. Under this scaling, the leading quadratic correction is of order O(n−1 ), while terms with three fluctuation insertions are of order O(n−3/2 ), and higher-order cumulants are correspondingly suppressed. EΘ [∆A∆] =
3.1
E XPANSION OF T HE TRAINING ERROR
We first derive a perturbative expansion of the training error around the mean kernel. For a fixed parameter Θ, the ridge solution yields −1 w∗ (Θ; DN ) = Φ(Θ)⊤ Φ(Θ)Φ(Θ)⊤ + N λIN y, (19) and hence f (Θ; X) = KΘ (KΘ + N λIN )−1 y,
KΘ := Φ(Θ)Φ(Θ)⊤ .
(20)
Accordingly, the training error is given by Etr (Θ) :=
1 ∥Φ(Θ)w∗ (Θ; DN ) − y∥2 . N
(21)
Proposition 3.2 (Loop expansion for training error). Let KΘ = K̄ + ∆,
K̄ := EΘ [KΘ ],
EΘ [∆] = 0,
(22)
and define G0 := (K̄ + γIN )−1 .
γ := N λ,
(23)
Then the expected training error admits the expansion EΘ [Etr ] (0)
(1)
(2)
= Etr + Etr + Etr + O(∆4 )
(24)
γ2 ⊤ 2 = y G0 + L1 + L2 + O(∆4 ) y, N where L1 = G20 EΘ [∆G0 ∆] G0 + G0 EΘ [∆G20 ∆] G0 + G0 EΘ [∆G0 ∆] G20 ,
(25)
and L2 = − G20 EΘ [∆G0 ∆G0 ∆] G0 + G0 EΘ [∆G20 ∆G0 ∆] G0 + G0 EΘ [∆G0 ∆G20 ∆] G0 + G0 EΘ [∆G0 ∆G0 ∆] G20 , (0)
Etr
2
=
γ ⊤ 2 y G0 y, N
(1)
Etr
2
=
γ ⊤ y L1 y, N
(2)
Etr
(26)
2
=
γ ⊤ y L2 y. N
In particular, the leading term is Gaussian and is determined only by the mean kernel K̄, while the higher-order terms describe the loop corrections induced by kernel fluctuations. 5
Proof. Starting from the definition of the training error, we have 1 ∥Φ(Θ)w∗ (Θ; DN ) − y∥2 N 1 ∗⊤ = w Φ(Θ)⊤ Φ(Θ)w∗ − 2y⊤ Φ(Θ)w∗ + ∥y∥2 . N
Etr (Θ) =
Substituting w∗ = Φ(Θ)⊤ (Φ(Θ)Φ(Θ)⊤ + γIN )−1 y, we obtain 1h ⊤ y KΘ (KΘ + γIN )−1 KΘ (KΘ + γIN )−1 y Etr (Θ) = N i − 2y⊤ KΘ (KΘ + γIN )−1 y + ∥y∥2 .
(27)
(28)
Using the identity KΘ (KΘ + γIN )−1 = IN − γ(KΘ + γIN )−1 ,
(29)
the above expression simplifies to γ2 ⊤ y (KΘ + γIN )−2 y. N
(30)
Etr (Θ) = N λ2 y⊤ (KΘ + γIN )−2 y.
(31)
Etr (Θ) = Since γ = N λ, this becomes
Taking expectation over Θ, we get EΘ [Etr ] = N λ2 y⊤ EΘ (KΘ + γIN )−2 y.
(32)
KΘ + γIN = G−1 0 + ∆,
(33)
Now write where G−1 0 = K̄ + γIN . By the resolvent expansion, −1 (G−1 = G0 − G0 ∆G0 + G0 ∆G0 ∆G0 − G0 ∆G0 ∆G0 ∆G0 + · · · . 0 + ∆)
(34)
−1 2 (KΘ + γIN )−2 = (G−1 . 0 + ∆)
(35)
Hence, Expanding this product and taking expectation term by term, the linear contribution vanishes because EΘ [∆] = 0. Therefore, EΘ [(KΘ + γIN )−2 ] = G20 + L1 + L2 + O(∆4 ),
(36)
where the one-loop contribution is L1 = G20 EΘ [∆G0 ∆] G0 + G0 EΘ [∆G20 ∆] G0 + G0 EΘ [∆G0 ∆] G20 ,
(37)
and the cubic contribution is L2 = − G20 EΘ [∆G0 ∆G0 ∆] G0 + G0 EΘ [∆G20 ∆G0 ∆] G0 + G0 EΘ [∆G0 ∆G20 ∆] G0 + G0 EΘ [∆G0 ∆G0 ∆] G20 .
(38)
Substituting this expansion back into the expression for EΘ [Etr ] completes the proof. To illustrate that the training error can be expanded systematically to higher orders in the loop expansion, we include in the proposition terms containing two vertex insertions. However, because the main focus of the present paper is the O n1 -scaling contribution, namely the one-loop correction, we restrict the subsequent expansion and discussion to this order. In the language of effective field theory, G0 plays the role of the bare propagator, while L1 and L2 encode the leading loop corrections generated by fluctuations of the random kernel around its mean. 6
Corollary 3.3 (Training error in terms of G and V ). Under the identification equation 16, the mean kernel and kernel covariance appearing in Proposition 3.2 are given by K̄ = G,
EΘ [∆A∆] =
1 (V ⋆ A) n
(39)
for every A ∈ RN ×N . Hence, with G0 := (G + γIN )−1 ,
(40)
the second-order correction to the training error can be written as 1 2 (41) G0 (V ⋆ G0 )G0 + G0 (V ⋆ G20 )G0 + G0 (V ⋆ G0 )G20 . L1 = n Therefore, 1 γ2 ⊤ 1 2 EΘ [Etr ] = G0 (V ⋆ G0 )G0 + G0 (V ⋆ G20 )G0 + G0 (V ⋆ G0 )G20 y+O 3/2 . y G20 + N n n (42) In particular, the leading non-Gaussian finite-width correction to the training error is of order O n1 . 3.2
E XPANSION OF THE T EST E RROR
We now derive the loop expansion for the test error. Using the kernel representation of the predictor, fΘ,DN (x) = kΘ (x)⊤ (KΘ + N λIN )−1 y,
(43)
the test error is written as Etest (Θ; DN ) =
Z
kΘ (x)⊤ (KΘ + N λIN )−1 y − y(x)
2
dµ(x).
(44)
X
To simplify the notation, define GΘ := (KΘ + γIN )−1 ,
γ := N λ, and introduce
(45)
Z
kΘ (x)kΘ (x)⊤ dµ(x), Z Z bΘ := y(x) kΘ (x) dµ(x), c := y(x)2 dµ(x). CΘ :=
(46)
X
X
(47)
X
Then the test error admits the quadratic form Etest (Θ; DN ) = y⊤ GΘ CΘ GΘ y − 2b⊤ Θ GΘ y + c.
(48)
Proposition 3.4 (Loop expansion for test error). Let KΘ = K̄ + ∆K ,
CΘ = C̄ + ∆C ,
bΘ = b̄ + ∆b ,
(49)
K̄ := EΘ [KΘ ],
C̄ := EΘ [CΘ ],
b̄ := EΘ [bΘ ],
(50)
EΘ [∆K ] = 0,
EΘ [∆C ] = 0,
EΘ [∆b ] = 0.
(51)
where and assume Define further
G0 := (K̄ + γIN )−1 .
(52)
Then the expected test error admits the expansion (0)
(1)
EΘ [Etest ] = Etest + Etest + O(∆3 ),
(53)
(0)
(54)
where the zeroth-order term is Etest = y⊤ G0 C̄G0 y − 2b̄⊤ G0 y + c, 7
and the second-order correction is given by h (1) Etest = y⊤ G0 EΘ [∆K G0 ∆K ] G0 C̄G0 + G0 C̄G0 EΘ [∆K G0 ∆K ] G0 i + G0 EΘ [∆K G0 C̄G0 ∆K ] G0 − G0 EΘ [∆K G0 ∆C ] G0 − G0 EΘ [∆C G0 ∆K ] G0 y − 2b̄⊤ G0 EΘ [∆K G0 ∆K ] G0 y + 2 EΘ [∆⊤ b G0 ∆K G0 ]y. (55) (1)
(2)
Here Etest is the leading Gaussian finite-width correction, while Etest and higher-order terms contain genuinely non-Gaussian contributions involving higher connected cumulants of the joint fluctuations of KΘ , CΘ , and bΘ . Proof. Starting from the definition of the test error, we expand the square: Z 2 Etest (Θ; DN ) = kΘ (x)⊤ GΘ y − y(x) dµ(x) ZX = y⊤ GΘ kΘ (x)kΘ (x)⊤ GΘ y − 2y(x) kΘ (x)⊤ GΘ y + y(x)2 dµ(x).
(56)
X
Using the definitions of CΘ , bΘ , and c, this becomes Etest (Θ; DN ) = y⊤ GΘ CΘ GΘ y − 2b⊤ Θ GΘ y + c.
(57)
Next, write KΘ = K̄ + ∆K , so that
CΘ = C̄ + ∆C ,
bΘ = b̄ + ∆b ,
−1 GΘ = (KΘ + γIN )−1 = (G−1 . 0 + ∆K )
(58) (59)
Applying the resolvent expansion gives GΘ = G0 − G0 ∆K G0 + G0 ∆K G0 ∆K G0 − · · · .
(60)
Substituting this into y⊤ GΘ CΘ GΘ y−2b⊤ Θ GΘ y+c and expanding order by order in the fluctuations, the linear terms vanish after taking expectation because EΘ [∆K ] = EΘ [∆C ] = EΘ [∆b ] = 0. Hence the zeroth-order contribution is (0)
Etest = y⊤ G0 C̄G0 y − 2b̄⊤ G0 y + c.
(61)
Collecting all quadratic terms in the fluctuations yields h (1) Etest = y⊤ G0 EΘ [∆K G0 ∆K ] G0 C̄G0 + G0 C̄G0 EΘ [∆K G0 ∆K ] G0 i + G0 EΘ [∆K G0 C̄G0 ∆K ] G0 − G0 EΘ [∆K G0 ∆C ] G0 − G0 EΘ [∆C G0 ∆K ] G0 y − 2b̄⊤ G0 EΘ [∆K G0 ∆K ] G0 y + 2 EΘ [∆⊤ b G0 ∆K G0 ]y. (62) All remaining two-loop and higher-order corrections are collected into O(∆3 ), which depend on higher connected cumulants of the joint fluctuations. This proves the claim. In the effective field theory language, the zeroth-order term corresponds to the tree-level contribu(1) tion, whereas Etest represents the leading loop correction induced by finite-width fluctuations. For x ∈ X , define the train–test mean kernel vector ⊤ g(x) := Gx1 , . . . , GxN ,
Gxα := G(x, xα ),
(63)
and similarly extend the four-point vertex to mixed train–test indices by V(xα)(βγ) ,
V(xα)(x′ β) , 8
etc.
(64)
Corollary 3.5 (Population operators and mixed fluctuation tensors in terms of G and V ). Let Z Z CΘ = kΘ (x)kΘ (x)⊤ dµ(x), bΘ = y(x)kΘ (x) dµ(x), (65) X
X
and write CΘ = C̄ + ∆C ,
bΘ = b̄ + ∆b .
(66)
Then, to leading order in the finite-width expansion, Z Z 1 C̄αβ = V(xα)(xβ) dµ(x), Gxα Gxβ dµ(x) + n X X
(67)
and Z b̄α =
y(x) Gxα dµ(x).
(68)
X
Moreover, for every A ∈ RN ×N , N
Z
β,γ=1
X
N
Z
β,γ=1
X
1 X Aβγ EΘ [∆A∆C ]αδ = n
EΘ [∆C A∆]αδ =
1 X Aβγ n
1 V(αβ)(xγ) Gxδ + V(αβ)(xδ) Gxγ dµ(x) + O 3/2 , n
(69)
1 Gxα V(xβ)(γδ) + V(xα)(γδ) Gxβ dµ(x) + O 3/2 , n
(70)
and N
EΘ [∆⊤ b A∆]δ =
1 X Aαβ n α,β=1
Z y(x) V(xα)(βδ) dµ(x) + O X
1 . n3/2
(71)
Corollary 3.6 (Test error in terms of G and V ). Under the assumptions of Corollary 3.5, the treelevel test error is (0)
Etest = y⊤ G0 C̄G0 y − 2b̄⊤ G0 y + c,
G0 = (G + γIN )−1 ,
(72)
with C̄ and b̄ given by equation 67 and equation 68. Its leading finite-width correction is h i 1 (1) Etest = y⊤ G0 (V ⋆ G0 )G0 C̄G0 + G0 C̄G0 (V ⋆ G0 )G0 + G0 (V ⋆ (G0 C̄G0 ))G0 y n 1 , n3/2 (73)
− y⊤ G0 MKC [G0 ] y − y⊤ G0 MCK [G0 ] y − 2b̄⊤ G0 (V ⋆ G0 )G0 y + 2 MbK [G0 ]y + O
where N X MKC [A] αδ := Aβγ β,γ=1 N X MCK [A] αδ := Aβγ β,γ=1
Z
V(αβ)(xγ) Gxδ + V(αβ)(xδ) Gxγ dµ(x),
(74)
Gxα V(xβ)(γδ) + V(xα)(γδ) Gxβ dµ(x),
(75)
X
Z X
and N X MbK [A] δ := Aαβ α,β=1
Z y(x) V(xα)(βδ) dµ(x).
Hence the leading correction to the test error is also of order O by the mean kernel G and the four-point vertex V . 9
(76)
X
1 n
, and is completely determined
3.3
E XPANSION OF THE G ENERALIZATION G AP
We now turn to the generalization gap, defined as the difference between the test error and the training error: Egen (Θ; DN ) := Etest (Θ; DN ) − Etr (Θ; DN ). (77) Using the kernel representations derived above, we have and where Z CΘ :=
Etr (Θ; DN ) = N λ2 y⊤ (KΘ + N λIN )−2 y,
(78)
Etest (Θ; DN ) = y⊤ GΘ CΘ GΘ y − 2b⊤ Θ GΘ y + c,
(79)
GΘ := (KΘ + N λIN )−1 , Z ⊤ kΘ (x)kΘ (x) dµ(x), bΘ := y(x) kΘ (x) dµ(x),
X
(80) Z c :=
X
X
y(x)2 dµ(x). (81)
Therefore,
2 ⊤ 2 Egen (Θ; DN ) = y⊤ GΘ CΘ GΘ y − 2b⊤ Θ GΘ y + c − N λ y GΘ y. Proposition 3.7 (Loop expansion for the generalization gap). Let
(82)
KΘ = K̄ + ∆K ,
CΘ = C̄ + ∆C ,
bΘ = b̄ + ∆b ,
(83)
K̄ := EΘ [KΘ ],
C̄ := EΘ [CΘ ],
b̄ := EΘ [bΘ ],
(84)
EΘ [∆K ] = 0,
EΘ [∆C ] = 0,
EΘ [∆b ] = 0.
(85)
where and assume Define further
γ := N λ, G0 := (K̄ + γIN )−1 . Then the expected generalization gap admits the expansion
(86)
(0) (1) EΘ [Egen ] = Egen + Egen + O(∆3 ),
(87)
where the zeroth-order term is (0) Egen = y⊤ G0 C̄G0 y − 2b̄⊤ G0 y + c −
γ2 ⊤ 2 y G0 y, N
(88)
and the second-order correction is given by (1)
(1)
(1) Egen = Etest − Etr ,
(89)
that is, h (1) Egen = y⊤ G0 EΘ [∆K G0 ∆K ] G0 C̄G0 + G0 C̄G0 EΘ [∆K G0 ∆K ] G0 i + G0 EΘ [∆K G0 C̄G0 ∆K ] G0 − G0 EΘ [∆K G0 ∆C ] G0 − G0 EΘ [∆C G0 ∆K ] G0 y − 2b̄⊤ G0 EΘ [∆K G0 ∆K ] G0 y + 2 EΘ [∆⊤ b G0 ∆K G0 ]y − γ 2 y⊤ G20 EΘ [∆K G0 ∆K ] G0 + G0 EΘ [∆K G20 ∆K ] G0 + G0 EΘ [∆K G0 ∆K ] G20 y. (90) (1)
(1)
where Etr is determined by the second-order cumulants of ∆K , while Etest additionally contains mixed connected cumulants involving the joint fluctuations of ∆K , ∆C , and ∆b . In particular, the generalization gap is sensitive not only to fluctuations of the training kernel matrix, but also to mixed fluctuations coupling train–train and test–train kernel quantities. The above proposition clarifies that the generalization gap combines two distinct sources of finitewidth corrections. The first comes from fluctuations of the training kernel KΘ , which already appear in the expansion of the training error. The second, which is specific to the test error, arises from mixed fluctuations involving CΘ and bΘ , and therefore reflects the coupling between training and test kernel quantities. In this sense, the generalization gap captures a richer finite-width structure than the training error alone. 10
Corollary 3.8 (Generalization gap in terms of G and V ). Under the assumptions of Corollaries 3.3 and 3.6, the expected generalization gap admits the expansion 1 (0) (1) EΘ [Egen ] = Egen + Egen + O 3/2 , (91) n where γ2 (0) Egen = y⊤ G0 C̄G0 y − 2b̄⊤ G0 y + c − y⊤ G20 y, (92) N and i γ2 ⊤h 1 2 (1) (1) (93) G0 (V ⋆ G0 )G0 + G0 (V ⋆ G20 )G0 + G0 (V ⋆ G0 )G20 y. Egen = Etest − y N n Equivalently, 1 1 (1) Egen = L̃gen [G, V ] + O 3/2 , (94) n n for an explicit functional L̃gen [G, V ] determined by the contractions of the mean kernel G and the four-point vertex V . Thus, to leading order beyond the mean-kernel approximation, the generalization gap is governed entirely by the pair (G, V ).
4
S PECTRAL A NALYSIS AND S CALING L AWS
In this section, we analyze the modal structure of the error expansions derived in Section 3. Using the eigenbasis of the mean kernel G, we express the tree-level and one-loop terms for the training error, test error, and generalization gap in spectral form. This makes explicit how the leading behavior is controlled by the eigenvalues of the mean kernel, while the finite-width correction is governed by contractions of the four-point vertex V with resolvent factors. As a consequence, the perturbative regime and the corresponding scaling laws can be read off directly from the relative size of the spectral responses of G and V . Let G = U ⊤ DU,
D = diag(ρ1 , . . . , ρN ),
where
Λ = diag
R := (G + γIN )−1 = U ⊤ ΛU,
1 1 ,..., ρ1 + γ ρN + γ
.
For any vector a ∈ RN and matrix A ∈ RN ×N , define ã := U a,
à := U AU ⊤ .
We also define the vertex in the eigenbasis of G by X Ṽ(ij)(kℓ) := Uiα Ujβ Ukγ Uℓδ V(αβ)(γδ) . α,β,γ,δ
Proposition 4.1 (Spectral form of the training error). Let G = U ⊤ DU with Dii = ρi , and let ỹ = U y. Then 1 (0) (1) (95) EΘ [Etr ] = Etr + Etr + O 3/2 , n where N γ2 X ỹi2 (0) Etr = , (96) N i=1 (ρi + γ)2 and (1) Etr =
N γ2 X ỹi ỹj Ṽ(ik)(kj) Nn i,j,k=1
1 1 1 + + . (ρi + γ)2 (ρj + γ)(ρk + γ) (ρi + γ)(ρj + γ)2 (ρk + γ) (ρi + γ)(ρj + γ)(ρk + γ)2 (97) 11
Define the population-projected quantities g̃i (x) :=
N X
Z Uiα Gxα ,
b̃i =
and
y(x)g̃i (x) dµ(x), X
α=1
Z
1 C̃ij = g̃i (x)g̃j (x) dµ(x) + n X
Z Ṽ(xi)(xj) dµ(x), X
where Ṽ(xi)(xj) denotes the mixed train–test representation of the four-point vertex. Proposition 4.2 (Spectral structure of the test error). Let R = (G + γIN )−1 . Then 1 (0) (1) EΘ [Etest ] = Etest + Etest + O 3/2 , n
(98)
with (0)
Etest =
N X
N X ỹi C̃ij ỹj b̃i ỹi −2 + c, (ρ + γ)(ρ + γ) ρ +γ i j i,j=1 i=1 i
(99)
where C̃ and b̃ are determined by the mean kernel G and the mixed train–test vertex contractions of V . Moreover, the one-loop term is of order O n1 and consists of sums of the form mix ỹi ỹj Ṽ(i·)(·j) 1X n (ρi + γ)(ρj + γ)(ρk + γ)
(100)
mix ỹi Ṽ(xi)(··) 1X , n (ρi + γ)(ρk + γ)
(101)
and
so that the spectral amplification is again controlled by small denominators (ρ + γ)−1 , while the coefficient structure is determined by mixed train–test contractions of V . Theorem 4.3 (Spectral scaling of the generalization gap). The expected generalization gap admits the decomposition 1 (0) (1) EΘ [Egen ] = Egen + Egen + O 3/2 , (102) n where N N N X X ỹi C̃ij ỹj b̃i ỹi γ2 X ỹi2 (0) Egen = −2 +c− , (103) (ρi + γ)(ρj + γ) ρ +γ N i=1 (ρi + γ)2 i=1 i i,j=1 and
(1)
(1)
(1) Egen = Etest − Etr .
(104)
In particular, the tree-level gap is governed by the mismatch between the population spectral response and the training resolvent response, while the one-loop correction is governed by the mismatch between mixed train–test vertex contractions and purely training vertex contractions. Hence the generalization gap is the observable most sensitive to the detailed spectral structure of the fourpoint vertex V . Corollary 4.4 (Spectrally aligned population case). Under the additional assumption C̃ =
1 D, N
b̃ =
1 Dỹ, N
the tree-level generalization gap reduces to N
(0) Egen =
1 X γρi ỹ 2 , N i=1 (ρi + γ)2 i
which is maximized at ρi = γ. 12
Proposition 4.5 (One-loop corrections in the spectrally aligned case). Let G = U ⊤ DU , where D = diag(ρ1 , . . . , ρN ),
ỹ = U y,
and assume the spectrally aligned population relations 1 1 D, b̃ := U b̄ = Dỹ. N N Assume further that the kernel covariance is represented by the four-point vertex as C̃ := U C̄U ⊤ =
˜ ik ∆ ˜ kj ] = 1 Ṽ(ik)(kj) . EΘ [∆ n Then the one-loop corrections to the training error, test error, and generalization gap are (1)
Etr =
N γ2 X tr ỹi ỹj Wijk Ṽ(ik)(kj) , Nn
(105)
N 1 X test ỹi ỹj Wijk Ṽ(ik)(kj) , Nn
(106)
N 1 X gap ỹi ỹj Wijk Ṽ(ik)(kj) , Nn
(107)
i,j,k=1
(1)
Etest =
i,j,k=1
and (1) Egen =
i,j,k=1
where
gap test tr Wijk = Wijk − γ 2 Wijk ,
(108)
with tr Wijk =
1 1 1 + + , (ρi + γ)2 (ρj + γ)(ρk + γ) (ρi + γ)(ρj + γ)2 (ρk + γ) (ρi + γ)(ρj + γ)(ρk + γ)2 (109)
and ρj ρi + (ρi + γ)2 (ρk + γ)(ρj + γ) (ρi + γ)(ρj + γ)2 (ρk + γ) (110) ρk 2ρi + − . (ρi + γ)(ρj + γ)(ρk + γ)2 (ρi + γ)2 (ρj + γ) In particular, all one-loop corrections are of order O n1 , and in the spectrally aligned case their test Wijk =
modal structure is determined explicitly by the contraction of the vertex Ṽ against the spectral response factors above.
5
E XPERIMENTAL VALIDATION
In this section, we empirically validate the loop expansion derived above for the training error, test error, and generalization gap in random feature models. Our goal is to verify two main claims: first, that the tree-level theory captures the dominant mean-kernel behavior, and second, that the one-loop correction explains the leading finite-width deviation from the empirical ensemble average. 5.1
E XPERIMENTAL S ETUP
We consider a random feature ridge regression problem with one-dimensional input. For all experiments, the target function is chosen as y(x) = sin(2x), and both training and test inputs are sampled independently from the standard Gaussian distribution. We use Ntrain = 64 training samples and Ntest = 512 test samples, with no additional label noise. 13
Training error Tree Tree + one-loop Empirical
Training error
10 4 9 × 10 5 8 × 10 5 7 × 10 5 6 × 10 5 250
500
750
1000
1250
1500
Width n
1750
2000
Figure 1: Training error versus feature width n. The empirical ensemble average is shown in blue with one-standard-deviation error bars, while the tree-level and one-loop predictions are shown in green and red, respectively. As n increases, the empirical curve moves toward the mean-kernel prediction, and the one-loop term accounts for the leading finite-width discrepancy. Frozen random features are generated using i.i.d. Gaussian weights √ and biases, and the activation d in the input dimension and by function is set to tanh. The feature map is normalized by 1/ √ 1/ n in the width, so that the induced kernel remains O(1) across widths. The readout weights are obtained by ridge regression with regularization parameter λ = 10−5 ,
γ = Ntrain λ.
We consider widths n ∈ {256, 512, 1024, 2048}. For each width, the empirical ensemble-mean training error, test error, and generalization gap are estimated using 50 independent random-feature realizations. The width-dependent reference objects K̄ = E[KΘ ],
C̄ = E[CΘ ],
b̄ = E[bΘ ]
used in the tree-level prediction are estimated using 300 Monte Carlo samples. The tensors entering the one-loop correction are estimated using 500 additional Monte Carlo samples. To ensure numerical stability even in the small-regularization regime, all resolvents are computed using an eigendecomposition-based stabilized inverse with adaptive jitter. Figure 1 compares the empirical ensemble-mean training error with the tree-level and one-loop predictions. Figure 2 shows the analogous comparison for the test error, while Figure 3 reports the generalization gap. To directly examine the predicted finite-width behavior, Figure 4 plots the deviation from tree level in log scale.
6
C ONCLUSION
In this work, we analyzed the ensemble-averaged training, test, and generalization errors of random feature models using the language of effective field theory. Under the proposed assumptions, we derived more accurate error formulas by expanding around the mean-kernel theory and incorporating loop corrections. Through an eigenvalue decomposition of the mean kernel, we further identified scaling laws that depend on the regime of the regularization parameter λ. Our experiments support the validity of the theoretical predictions and compare the scale of finite-width effects with the improvement obtained by including loop corrections. 14
Test error Tree Tree + one-loop Empirical
3 × 10 1
Test error
2 × 10 1
10 1
6 × 10 2 250
500
750
1000
1250
Width n
1500
1750
2000
Figure 2: Test error as a function of feature width n on a logarithmic scale. Empirical results are compared with the tree-level prediction and its one-loop correction. The improved agreement obtained by the one-loop term highlights the role of mixed train–test kernel fluctuations at finite width.
Generalization gap
Generalization gap
Tree Tree + one-loop Empirical
10 1
250
500
750
1000
1250
Width n
1500
1750
2000
Figure 3: Generalization gap as a function of feature width n, shown on a symmetric logarithmic scale. The one-loop prediction follows the empirical trend more closely than the tree-level approximation, indicating a richer finite-width structure in the gap.
A natural direction for future work is to move beyond frozen-parameter random feature models and consider fully trainable neural networks, where all parameters evolve under learning dynamics. In that setting, it may be possible to translate the dynamics of the parameter ensemble into an actionbased formulation and develop a perturbative expansion of the resulting errors in a similar spirit. 15
Finite-width scaling 10 1 |Empirical − Tree|
10 2 Train, α ≈ 0.91 Test, α ≈ 0.95 Gap, α ≈ 0.95
10 3 10 4 10 5 3 × 102 4 × 102
6 × 102
Width n
103
2 × 103
Figure 4: Log–log plot of the deviation between the empirical ensemble average and the tree-level prediction. The fitted slopes support the n−1 -type scaling predicted by the loop expansion. ACKNOWLEDGMENTS Taeyoung Kim is supported by a KIAS Individual Grant (AP102201) at Korea Institute for Advanced Study and supported by the Center for Advanced Computation at Korea Institute for Advanced Study.
R EFERENCES Ben Adlam and Jeffrey Pennington. A multiscale theory of generalization in kernel regression. In ICML, 2020. Boris Hanin. Random fully connected neural networks as perturbatively solvable hierarchies. Journal of Machine Learning Research, 25:1–72, 2024. Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In ICLR, 2020. Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: Theory and applications. Neurocomputing, 70(1–3):489–501, 2006. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, 2018. Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS, 2019. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NeurIPS, 2007. Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In NeurIPS, 2009. D. A. Roberts, S. Yaida, and B. Hanin. The Principles of Deep Learning Theory. Cambridge University Press, 2022a. 16
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge University Press, 2022b. Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Generalization properties of learning with random features. In NeurIPS, 2017. Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the infinite-width limit: Effects of depth and initialization. In ICML, 2022. Jun Wang, Shuo Lu, Shunli Wang, Yue Zhang, and Xue Wang. A review on extreme learning machine. Multimedia Tools and Applications, 81:41611–41660, 2022.
17