arXiv CSopen access

Criticality and Saturation in Orthogonal Neural Networks

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

arXiv:2605.06563v1 [cs.LG] 7 May 2026

Criticality and Saturation in Orthogonal Neural Networks Jan E. Gerken Department of Mathematical Sciences Chalmers University of Technology University of Gothenburg SE-412 96 Gothenburg, Sweden [email protected]

Max Guillen Department of Mathematical Sciences Chalmers University of Technology University of Gothenburg SE-412 96 Gothenburg, Sweden [email protected]

Abstract It has been known for a long time that initializing weight matrices to be orthogonal instead of having i.i.d. Gaussian components can improve training performance. This phenomenon can be analyzed using finite-width corrections, where the infinite-width statistics are supplemented by a power series in 1/width. In particular, recent empirical results by Day et al. show that the tensors appearing in this treatment stabilize for large depth, as opposed to the tensors of i.i.d.-initialized networks. In this article, we derive explicit layer-wise recursion relations for the tensors appearing in the finite-width expansion of the network statistics in the case of orthogonal initializations. We also provide an extension of recently-introduced Feynman diagrams for the corresponding recursions in the i.i.d.-case which are valid to all orders in 1/width. Finally, we show explicitly that the recursions we derive reproduce the stability of the finite-width tensors which was observed for activation functions with vanishing fixed point. This work therefore provides a theoretical explanation for the stability of nonlinear networks of finite width initialized with orthogonal weights, closing a long-standing gap in the literature. We validate our theoretical results experimentally by showing that numerical solutions of our recursion relations and their analytical large-depth expansions agree excellently with Monte-Carlo estimates from network ensembles.

Introduction

Neural network training usually starts from randomly initialized parameters and a good choice of initialization distribution is critical to ensure stable training dynamics to avoid exploding or vanishing gradients. Typically, the initial weight components are sampled i.i.d. from Gaussian distributions whose variance scales inversely with the width of the network [1, 2]. However, it was found that initializing the weight matrices to be orthogonal (i.e. to satisfy 𝑊𝑊 ⊤ = I, yielding eigenvalues all exactly one) improves performance [3, 4]. This phenomenon has been studied theoretically for many years. However, these studies are largely restricted to linear networks [3, 5] or mean-field theory approaches in the large-width limit [6, 7, 8]. Also the neural tangent kernel (NTK) at infinite width [9] has been used to analyze orthogonal initializations. However, at infinite width, the NTKs of networks initialized with orthogonal and Gaussian weights agree [10], rendering the infinite-width NTK an unsuitable tool to understand the observed differences between the two initializations. In this article, we instead use 1/width corrections to the neural network statistics at infinite-width in order to investigate the effect of orthogonal initializations for nonlinear networks at finite width. In this 1

framework [11], the network statistics are studied in terms of the cumulants of the preactivations and derivatives. The cumulants are then decomposed into tensors which are expanded in a power series in 1/width, with the zero-order contribution corresponding to the infinite-width limit. For orthogonal initializations, the first corrections to the statistics of preactivations were worked out analytically and corrections to the remaining tensors appearing in the first correction to the infinite-width training dynamics were sampled numerically [25]. These results show empirically that the network statistics stabilize at large depth for orthogonal initializations, in contrast to the case of Gaussian initializations. In summary, no theoretical analysis was so far able to show that orthogonal initializations of nonlinear, finite-width neural networks lead to improved performance over Gaussian initializations. In this article, we fill this gap. In particular, the finite-width corrections to the tensors governing the network statistics obey layer-wise recursion relations. These have been derived only for the cumulant of four preactivations [25] (the 𝑉4 tensor). Here, we derive the recursion relations for the additional ten fundamental tensors (𝐷, 𝐹, 𝐴, 𝐵, 𝑃, 𝑄, 𝑅, 𝑆, 𝑇 and 𝑈) which govern the first correction to the training dynamics. The orthogonal statistics are captured by Weingarten functions [12] which add additional terms to the recursion relations compared to the Gaussian case. We then show by explicit iteration of our recursions and by expanding their solutions around the large-depth limit that our theoretical results reproduce the empirically observed stability of these tensors. Since these recursion relations are laborious to derive algebraically, we also extend a recently-introduced framework to facilitate these computations using Feynman diagrams [13] to the case of orthogonal weights. This extension, which introduces a novel propagator and charge, simplifies the computation of recursion relations dramatically. We prove the completeness of our Feynman diagrams to all orders in 1width and demonstrate their power by computing the correction to order 1/width2 of the recursion relation of the cumulant of six preactivations (the 𝑉6 tensor). Limitations Although our analysis is limited to MLPs, there are no conceptual obstacles for extensions to other architectures. Our experiments and the large-depth expansion are restricted to the tanh activation function, although our theoretical results are general. Finally, our explicit calculations are mostly restricted to the order 1/width, although our Feynman diagrams are valid to all orders. Our main contributions are: • We derive for the case of orthogonal initializations the recursion relations for the tensors 𝐷, 𝐹, 𝐴, 𝐵, 𝑃, 𝑄, 𝑅, 𝑆, 𝑇, 𝑈 and the first correction to the NTK mean, Θ {1} , at order 1/𝑛, where 𝑛 is the width of the network. We furthermore derive the recursion relation for the tensor 𝑉6 at order 1/𝑛2 . • We provide Feynman rules that simplify the computation of these recursion relations dramatically and prove that they reproduce the correct algebraic expressions to all orders. • We solve our recursion relations by iteration in the single-input case and show that the solution matches empirical results for tanh-MLPs. We analytically compute a large-depth expansion of the solution and show that it supports the saturation observed empirically. We furthermore extend criticality results for the neural network Gaussian process kernel (NNGP) and the NTK from Gaussian to orthogonal initializations.

Related work

Orthogonal initializations of neural network weights have been considered for a long time [14, 3]. An ablation showing performance boosts by orthogonal initializations can be found in Ref. [4].

Theoretical studies of orthogonal initializations mainly consider linear neural networks [3] or use a meanfield theory approach which requires the infinite-width limit. In the latter case, a considerable amount of work has been done on dynamical isometry, which requires orthogonal initializations and supposes that the singular values of the Jacobian of the network are all close to one. With the help of free probability theory, it could be shown that ReLU nonlinearities destroy the dynamical isometry present in deep linear networks with orthogonal initialization, but the sigmoid nonlinearity retains it [6]. These results have been extended to further nonlinearities [7] and to convolutional networks [8]. All these studies focus on initialization but convergence rates for orthogonally initialized networks were also derived in the linear setting [5], confirming the faster convergence in this case. The NTK for orthogonal initializations at infinite width was found to agree with that of Gaussian initializations [10], motivating the study of finite-width corrections. The approach used in this article was introduced in Ref. [15], with further details available in a textbook [11], yielding results about symmetries [16], initialization stability [17] and scaling laws [18, 19] among others. Our Feynmandiagrammatic treatment of the finite-width corrections is based on the rules introduced in [13] which include all tensors appearing in the first correction to the training dynamics. Earlier papers using Feynman diagrams to capture preactivation statistics include [17, 20, 21, 22, 18, 19]. This article is most closely related to Ref. [25], where finite-width corrections were first considered in the orthogonal case. We substantially extend the results in this reference, as outlined above.

Orthogonal neural networks

We consider an 𝐿-layer multilayer perceptron (MLP) N : R𝑛in → R𝑛out , defined recursively by the feed-forward equations 𝑛 ∑︁ −1) 𝑧 𝑖(ℓ ) (𝑥) = 𝑊𝑖(ℓ𝑗 ) 𝜎 𝑧 (ℓ (𝑥) , 𝑖 = 1, . . . , 𝑛, (1) 𝑗 𝑗=1

where 𝑥 ∈ R𝑛in denotes the input, indices 𝑖, 𝑗, . . . label neurons, 𝜎 is a pointwise activation function, and the weight matrices 𝑊 (ℓ ) ∈ R𝑛×𝑛 are sampled independently from the orthogonal group with respect to the Haar measure. For simplicity, we assume that all hidden layers have the same width 𝑛ℓ = 𝑛, and that biases are omitted. √ For a matrix W drawn from the orthogonal Haar measure Haar(O(𝑛)) and scaled by 𝐶𝑊 , the joint moments of its entries are characterized by the orthogonal Weingarten calculus: h i ∑︁ ) (ℓ ) 𝑘 𝜋 𝜎 −1 E 𝑊𝑖(ℓ · · · 𝑊 = (𝐶 ) 𝛿 𝛿 W 𝜋 𝜎 , (2) 𝑊 𝑖 𝑗 𝑖2𝑘 𝑗2𝑘 1 𝑗1 𝜋, 𝜎 ∈ P2 (2𝑘 )

where 𝐶𝑊 is a layer-independent hyperparameter, P2 (2𝑘) denotes the set of pair partitions of {1, . . . , 2𝑘 }, and W is the orthogonal Weingarten function [23, 24]. As illustrative examples, the second- and fourthorder moments are given by ) ) E[𝑊𝑖(ℓ 𝑊𝑖(ℓ ] = 𝐶𝑊 (12)𝑖 (12) 𝑗 W [1] 1 𝑗1 2 𝑗2

(3)

𝑇

(12) (34)𝑖 W [1, 1] W [2] W [2] (12) (34) 𝑗 ª © ª© ª (ℓ ) (ℓ ) (ℓ ) (ℓ ) 2© E[𝑊𝑖1 𝑗1 𝑊𝑖2 𝑗2 𝑊𝑖3 𝑗3 𝑊𝑖4 𝑗4 ] = (𝐶𝑊 ) (13) (24)𝑖 ® W [2] W [1, 1] W [2] ® (13) (24) 𝑗 ® W [2] W [1, 1] ¬ « (14) (23) 𝑗 ¬ « (14) (23)𝑖 ¬ « W [2]

(4)

where we employ the compact notation (12) (34) · · · (2𝑘 − 1 2𝑘)𝑖 ≡ 𝛿𝑖1 𝑖2 𝛿𝑖3 𝑖4 · · · 𝛿𝑖2𝑘−1 𝑖2𝑘 , and the corresponding orthogonal Weingarten functions are given by W [1] =

1 , 𝑛

W [1, 1] =

𝑛+1 , (𝑛 − 1)𝑛(𝑛 + 2) 3

W [2] = −

1 . (𝑛 − 1)𝑛(𝑛 + 2)

(5)

Consequently, in contrast to the standard Gaussian setting where Wickś theorem implies vanishing higher-order cumulants, the cumulants of the orthogonal weights 𝑊, i.e. the connected parts of their joint moments, are nonzero and play a central role in determining the finite-width statistics of the neural network ensemble N . As discussed in [11], effective field theory (EFT) techniques enable the systematic computation of preactivation statistics to all orders in 1/𝑛. Within this framework, the neural network Gaussian process (NNGP) b(ℓ+1) (𝑥1 , 𝑥2 ) = 𝑧 (ℓ+1) (𝑥1 )𝑧 (ℓ+1) (𝑥2 ), defines the expectation 𝐾 (ℓ+1) (𝑥1 , 𝑥2 ) ≡ E[ 𝐾 b(ℓ+1) (𝑥1 , 𝑥2 )]. kernel 𝐾 𝑖1 𝑖2 𝑖1 𝑖2 𝑖𝑖 At leading order in 1/𝑛, this kernel satisfies the recursion 1 (ℓ ) (ℓ ) (ℓ+1) 𝐾 (𝑥1 , 𝑥2 ) = 𝐶𝑊 ⟨𝜎1 𝜎2 ⟩𝐾 (ℓ) + O (6) 𝑛 where we use the shorthand 𝜎𝑎(ℓ ) ≡ 𝜎 𝑧 (ℓ ) (𝑥 𝑎 ) , and ⟨·⟩𝐾 denotes Gaussian expectation with covariance 𝐾. At next-to-leading order, the fourth-order cumulant of the preactivations takes the form E[𝑧 𝑖(ℓ+1) (𝑥1 )𝑧 𝑖(ℓ+1) (𝑥2 )𝑧 𝑖(ℓ+1) (𝑥3 )𝑧𝑖(ℓ+1) (𝑥4 )] 1 2 3 4 1 (ℓ+1) (ℓ+1) (ℓ+1) = (12) (34)𝑖 𝑉1234 + (13) (24)𝑖 𝑉1324 + (14) (23)𝑖 𝑉1423 𝑛 (ℓ+1) (ℓ+1) where we write 𝑉1234 = 𝑉 (ℓ+1) (𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 ). The function 𝑉1234 obeys the recursion h (ℓ+1) =(𝐶𝑊 ) 2 ⟨𝜎1(ℓ )𝜎2(ℓ )𝜎3(ℓ )𝜎4(ℓ ) ⟩𝐾 (ℓ) −⟨𝜎1(ℓ )𝜎2(ℓ ) ⟩𝐾 (ℓ)⟨𝜎3(ℓ )𝜎4(ℓ ) ⟩𝐾 (ℓ) −⟨𝜎1(ℓ )𝜎3(ℓ ) ⟩𝐾 (ℓ)⟨𝜎2(ℓ )𝜎4(ℓ ) ⟩𝐾 (ℓ) 𝑉1234 * + * + i (𝐶 ) 2 ∑︁ 𝑑 2 (𝜎3(ℓ ) 𝜎4(ℓ ) ) 𝑑 2 (𝜎1(ℓ ) 𝜎2(ℓ ) ) 𝑊 (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) −⟨𝜎1 𝜎4 ⟩𝐾 (ℓ) ⟨𝜎2 𝜎3 ⟩𝐾 (ℓ) + 𝑉𝛽1 𝛽2 𝛽3 𝛽4 4 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝐾 (ℓ) 𝛽𝑖 ∈ {1,2,3,4}

(7)

(8)

𝑖=1,...,4

As noted in [25], the recursion (8) differs from its Gaussian counterpart by the last two terms in the square brackets. As we show below, this small change has significant consequences for information processing in orthogonal networks.

3.1

The NTK at finite-width

The layer-ℓ empirical (NTK) associated with the network N is defined as b (ℓ ) (𝑥1 , 𝑥2 ) = Θ 𝑖𝑗

) (ℓ ) 𝜕𝑧 (ℓ 𝑗 (𝑥 2 ) (ℓ ′ ) 𝜕𝑧 𝑖 (𝑥 1 ) , 𝜆𝜃 (ℓ ′ ) (ℓ ′ ) 𝜕𝜃 𝜕𝜃 ′ 𝜇 𝜇 𝜇,ℓ

∑︁

′

(9)

′

) where 𝜆 (ℓ is a layer-dependent training hyperparameter, 𝜃 𝜇(ℓ ) denotes the weight parameters at layer 𝜃 ℓ ′ , and the sum runs over all parameters in layers ℓ ′ ≤ ℓ. The NTK of the full network is given by b=Θ b ( 𝐿) . As is well known, this kernel is a nonlinear, initialization-dependent object that evolves during Θ training. In the small learning-rate regime, the NTK provides a leading-order description of the training dynamics. Consequently, understanding its behavior is important for analyzing neural network learning dynamics [9].

To compute the statistics of the NTK, we employ the chain rule to derive the following forward recursion: (ℓ+1) ∑︁ 𝑛 © (ℓ+1) 𝜆𝑊 (ℓ+1) (ℓ ) (ℓ ) ª b Θ𝑖1 𝑖2 (𝑥1 , 𝑥2 ) = (12)𝑖 𝜆 𝑏 + 𝜎 𝑗,1 𝜎 𝑗,2 ® 𝑛 𝑗=1 « ¬ 𝑛 ∑︁ ) ′(ℓ ) b (ℓ ) + 𝑊𝑖(ℓ+1) 𝑊𝑖(ℓ+1) 𝜎 ′(ℓ 𝑗1 ,1 𝜎 𝑗2 ,2 Θ 𝑗1 𝑗2 (𝑥 1 , 𝑥 2 ), 1 𝑗1 2 𝑗2 𝑗1 , 𝑗2 =1

(10)

(ℓ+1) (ℓ+1) where the standard rescaling 𝜆𝑊 ↦→ 𝜆 𝑊 /𝑛 has been applied to ensure that all network parameters contribute at the same order. As discussed in [11], an appropriate choice of the depth-dependence of the training hyperparameters depends on the activation function under consideration.

c (ℓ ) ≡ Θ b (ℓ ) − E[Θ b (ℓ ) ]. Applying the EFT techniques The fluctuations of the NTK are defined as ΔΘ developed in [11] to the recursionc(10) yields the following leading-order equation for the mean NTK 1 ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ+1) (ℓ+1) (ℓ+1) (ℓ ) (ℓ ) Θ +O Θ12 = 𝜆 𝑏 + 𝜆𝑊 𝜎1 𝜎2 𝐾 (ℓ) + 𝐶𝑊 𝜎1 𝜎2 . (11) 𝐾 (ℓ) 12 𝑛 As in the Gaussian case, the leading-order statistics of the NTK in 1/𝑛 are determined by the crosscumulant between the preactivations and the NTK fluctuation, together with the NTK variance. The former admits the following compact expression c 𝑖(ℓ+1) E𝑐𝜃 [𝑧 𝑖(ℓ+1) (𝑥1 )𝑧 𝑖(ℓ+1) (𝑥2 ) ΔΘ (𝑥3 , 𝑥4 )] 3 𝑖4 1 2 1 (ℓ+1) (ℓ+1) (ℓ+1) = 𝐷 1234 (12) (12) (34)𝑖 +𝐹1324 (13) (24)𝑖 +𝐹1423 (14) (23)𝑖 , 𝑛 where the tensors 𝐷 and 𝐹 can be computed directly from the relation (10), see Appendix A for details. As shown in the Appendix, a careful algebraic analysis yields, for instance, the following leading-order 1/𝑛 recursion relations for the tensor 𝐹 h i (ℓ+1) (ℓ ) 𝐹1324 = (𝐶𝑊 ) 2 ⟨𝜎1(ℓ ) 𝜎2(ℓ ) 𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) − ⟨𝜎1(ℓ ) 𝜎2(ℓ ) ⟩𝐾 (ℓ) ⟨𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) Θ34 4 ∑︁ 1 𝛼𝛾 𝛽 𝛿 (ℓ ) (ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ′(ℓ ) (ℓ ) 2 + (𝐶𝑊 ) ⟨𝜎1 𝜎3 𝑧 𝛼 ⟩𝐾 (ℓ) ⟨𝜎2 𝜎4 𝑧 𝛽 ⟩𝐾 (ℓ) 𝐾 (ℓ ) 𝐾 (ℓ ) 𝐹𝛾3 𝛿4 + O (13) 𝑛 𝛼,𝛽,𝛾, 𝛿=1 As we will show, this recursion can be derived efficiently using a diagrammatic framework based on a consistent set of Feynman rules and graph-theoretic principles, yielding a substantial simplification over conventional algebraic approaches. An analogous construction applies to the tensor 𝐷, as well as to the NTK variance tensors 𝐴 and 𝐵 [11], see Appendix A for details.

Feynman diagrams

A novel diagrammatic framework, inspired by the Feynman-diagram description of interactions among fundamental particles in high-energy physics, was recently introduced for the study of deep neural networks with Gaussian initializations [13, 17]. In this section, we extend this framework to incorporate the orthogonality properties of the network parameters.

4.1

Feynman rules

Without loss of generality, consider the channel (12)𝑖 (34)𝑖 . . . (2𝑘 2𝑘 − 1)𝑖 , corresponding to the pairing (12)(34) . . . (2𝑘 2𝑘 − 1). The Feynman rules reproducing tensors with this channel structure can be organized into two groups. The first group implements the orthogonality constraints as follows: 1. Preactivations and NTKs are denoted by external lines, as illustrated below. c 𝛼𝛽 ≡ 𝛽 ΔΘ 𝛼

𝑧𝛼 ≡ 𝛼

(14)

In (14), a colored line represents a single NTK label. Distinct colors are used for external dotted lines associated with different NTKs. The colors of the dotted lines correspond to the 𝜃-indices appearing in these definitions. Since the 𝜃-indices are contracted in pairs, distinct colors encode the corresponding contraction pattern. 5

2. Define the cubic vertices as 𝛽𝑐

(ℓ)

𝛽

(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼𝑐

𝛼

𝛽𝑐

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛽𝑐

(ℓ)

∼1

(ℓ)

′(ℓ)

Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐 (ℓ)

𝛽𝑐

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐

(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

(15)

𝛼𝑐

Here the superscript 𝑐, referred to as the orthogonality charge, keeps track of the label’s orthogonal character, and lines that do not end in a dot represent internal lines. 3. Draw a square propagator connecting internal lines in all possible ways, consistent with the chosen pairing. The square represents the full expectation value (16) E[ · ]

This procedure generates distinct diagram types, both connected and disconnected. The connected diagrams are further classified as 𝑠-class diagrams, defined by the number 𝑠 of square propagators apppearing in the diagram. The square propagator obeys an additional selection rule: all k-class diagrams formed from pairs of external lines corresponding to different object types vanish. 4. For each 𝑠-class diagram, generate all inequivalent permutations of its 2𝑚 external labels carrying orthogonality charge. Multiply each resulting diagram by 𝑛1 for every uncharged pairing, and by the appropriate 𝑚-class Weingarten function W, determined by the relative ordering 𝜏 of the diagram’s labels with respect to the original pairing 𝜋 = (12) (34) . . . (2𝑘 2𝑘 − 1): W [𝜏, 𝜋] = W [ℓ(𝜏 ◦ 𝜋)], where ◦ denotes ordinary permutation multiplication and ℓ(𝑒) denotes the cycle length of 𝑒. 5. Multiply each 𝑠-class contribution by the Möbius coefficient (−1) 𝑠−1 (𝑠 − 1)!, and sum over all classes. The second group implements the effective field theory techniques developed in [11], applied to the square propagator in the diagrammatic construction of the previous step, through the following set of Feynman rules analogous to those introduced in [13, 17]: 6. We define the bare propagator as ⟨ ⟩𝐾 (ℓ) ≡

(17)

(e) If both dotted and dashed lines of the same color are attached to the propagator, they must appear in pairs carrying the same sample index. The two lines in each pair attach to different vertices. Moreover, if both vertices are drawn in the orientation specified in the Feynman rules, the top-to-bottom ordering of the sample indices (and therefore the colors) of the lines connected to the two vertices must coincide. (f) A pair of dashed lines of the same color connected to the propagator contributes a factor Θ 𝛼𝛽 when the two lines attach to different vertices, where 𝛼 and 𝛽 denote the sample indices of the pair. 7. Quartic vertices are defined analogously, following [13]. Explicitly, 𝛼2

𝛼3

𝛼2

𝛼3

𝛼2

𝛼3

𝛼2

𝛼1

𝛼4

𝛼1

𝛼4

𝛼1

𝛼4

𝛼1

𝛼4

𝛼1

𝛼4

(18) 1 (ℓ+1) 𝑛 𝑉𝛼1 𝛼2 𝛼3 𝛼4

(ℓ+1) 1 𝑛 𝐷 𝛼1 𝛼2 𝛼3 𝛼4

1 (ℓ+1) 𝑛 𝐹𝛼1 𝛼3 𝛼2 𝛼4

1 (ℓ+1) 𝑛 𝐴 𝛼1 𝛼2 𝛼3 𝛼4

1 (ℓ+1) 𝑛 𝐵 𝛼1 𝛼3 𝛼2 𝛼4

8. Higher-order NTK and preactivation tensors are introduced via a natural generalization of the vertices in (18). 9. The square propagator decomposes into all connected and disconnected diagrams built from the bare propagator, quartic vertices, and higher-order vertices, with internal lines remaining undotted. This decomposition respects the selection rules (a)-(f). In Appendix E.3, we prove that the Feynman rules (1)-(9) are complete to all orders in the 1/n expansion. We further derive a simplified formulation valid at leading order in 1/𝑛 in Appendix D. As a consistency check, we recover the results of [17] and [13] for preactivations and the NTK with Gaussian weights by restricting to the leading diagonal Weingarten contributions. These Feynman rules satisfy the following theorem: Theorem 4.1. The Feynman rules stated in items 1-9, uniquely determine the recursion relations governing the layerwise evolution of the orthogonal NTK tensors 𝐷, 𝐹, 𝐴, 𝐵 at order 1/𝑛. Proof. We illustrate the proof using the tensor 𝐹, deriving the recursion relation (13) directly from the Feynman rules above. The analysis for the tensors 𝐷, 𝐴 and 𝐵 is analogous and deferred to Appendix E.1, completing the proof. The Feynman rules (1)-(5) produce the following diagrams for the tensor 𝐹: 3

= W [1, 1]

∑︁ 3𝑐 𝑗,𝑘

+ W [2]

∑︁ 2𝑐 𝑗,𝑘

1𝑐

𝜎𝑗 𝜎𝑗

𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

1𝑐

2𝑐

+ W [2]

𝑗,𝑘

4𝑐 𝜎𝑘′ 𝜎𝑘′

3𝑐

+ W [2]

∑︁ 4𝑐 𝑗,𝑘

4𝑐

∑︁ 2𝑐

1𝑐

𝜎 𝑗 𝜎 ′𝑗

Θ 𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

1𝑐

3𝑐 4𝑐

𝜎𝑘 𝜎𝑘′

2𝑐

(19) 3𝑐

The absence of 2-class diagrams in (19) follows from the fact that the square propagator vanishes when acting on a pair of lines of different type. The 1-class diagrams are generated from the inequivalent permutations of the reference pairing (13) (24), namely {(13) (24), (12) (34), (14) (23)}. The corresponding 𝑘 = 2 Weingarten functions multiplying the subdiagrams are determined by the relative ordering of the reference pairing 𝜏 and the pairing 𝜋 appearing in the subdiagram: W [𝜋, 𝜏] = W [1, 1] for 𝜋 = 𝜏, W [𝜋, 𝜏] = W [2] for 𝜋 ≠ 𝜏. The overall factor of 1 follows from the Möbius relation for a single-block partition.

At first order in 1/𝑛, the Weingarten functions can be approximated by their leading terms in the expansions (30) and (31). In this manner, (19) simplifies to 3

= 1

1 ∑︁ 3 𝑛2 𝑗,𝑘

𝜎 𝑗 𝜎 ′𝑗

2𝑐

𝜎𝑘 𝜎𝑘′

−

1𝑐

1 ∑︁ 2 − 3 𝑛 𝑗,𝑘

𝑐

4𝑐

𝑐

𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

3𝑐 4𝑐

1𝑐

1 ∑︁ 4 − 3 𝑛 𝑗,𝑘

1 ∑︁ 2 𝑛3 𝑗,𝑘

𝑐

Θ 𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

1𝑐

4𝑐

𝑐

𝜎 𝑗 𝜎 ′𝑗

3𝑐

2𝑐

𝜎𝑘 𝜎𝑘′

1 +O 2 𝑛

(20)

3𝑐

1𝑐

We now apply the second set of Feynman rules (6)–(9). As an immediate consequence, the last terms in (20) vanish at order 1/𝑛. A nonzero contribution would require the corresponding diagrams to produce a factor of 𝑛2 , which occurs when 𝑗 ≠ 𝑘 and a square propagator decomposes into two bare propagators forming disconnected subdiagrams. However, such configurations are forbidden by the selection rules (a)–(f), as each subdiagram violates color conservation. Consequently, we are left with 3

1 ∑︁3 = 2 𝑛 𝑗,𝑘

𝑐

𝜎 𝑗 𝜎 ′𝑗

2𝑐

1 ∑︁2 − 3 𝑛 𝑗,𝑘

𝜎𝑘 𝜎𝑘′

1𝑐

4𝑐

𝑐

Θ 𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

1𝑐

3𝑐

1 +O 2 𝑛

(21)

4𝑐

We now expand the diagrams in (21) in terms of the bare propagator and quartic vertices by analyzing the diagonal and off-diagonal neural components of each subdiagram, together with the selection rules (a)-(f). When 𝑖 = 𝑗, the first diagram in (21) contributes as 1 ∑︁ 3 𝑛2 𝑗

𝜎 𝑗 𝜎 ′𝑗

(22)

When 𝑖 ≠ 𝑗, the first and second diagrams reduce respectively to 1 ∑︁ 3 𝑛2 𝑗 , 𝑗 1

𝑧 𝑗1 𝑧 𝑗2

𝜎 𝑗1 𝜎 ′𝑗

𝜎 𝑗2 𝜎 ′𝑗

1 (ℓ) 𝑛 𝐹4

1 ∑︁ 2 , − 3 𝑛 𝑗 ,𝑗 1

Θ 𝜎 ′𝑗 𝜎 ′𝑗

𝜎 𝑗1 𝜎 𝑗1

(23) 4

Substituting (22) and (23) into (21), we obtain 3

= 1

1 ∑︁ 3 𝑛2 𝑗

𝜎 𝑗 𝜎 ′𝑗

−

1 ∑︁ 2 𝑛3 𝑗 , 𝑗 1

𝜎 𝑗 𝜎 ′𝑗

1 ∑︁ 3 𝑛2 𝑗 , 𝑗 1

4 𝜎 𝑗1 𝜎 𝑗1

Θ 𝜎 ′𝑗 𝜎 ′𝑗 2

1 𝑛2

𝜎 𝑗2 𝜎 ′𝑗

1 (ℓ) 𝑛 𝐹4

𝑧 𝑗1 𝑧 𝑗2

𝜎 𝑗1 𝜎 ′𝑗

(24)

The 1/𝑛 scaling of the diagrams follows from straightforward power counting of the respective summations: When 𝑖 = 𝑗, the summation contributes factor of 𝑛, whereas when 𝑖 ≠ 𝑗 it yields a factor of 𝑛2 . Combined with the explicit coefficients on the right-hand side of (24), these factors produce an overall contribution of order 1/𝑛. It is straightforward to verify that the first and second lines on the RHS of (24) reproduce the corresponding lines on the RHS of (13). In the same manner, the full Feynman rules can be used to reproduce the recursion relations governing higher-derivative versions of the NTK, as detailed below. 8

Theorem 4.2. The set of Feynman rules presented above, in conjunction with those defined in Appendix B, reproduces the recursion relations governing the orthogonal dNTK and ddNTK tensors: 𝑃, 𝑄 and 𝑅, 𝑆, 𝑇, 𝑈, respectively, at order 1/𝑛. Proof. See Appendix E.2. The preceding results establish that Feynman diagrams are complete at order 1/𝑛. This construction extends straightforwardly to all orders in 1/𝑛 by allowing the vertices in 18 to admit an arbitrary number of external legs, without requiring any further modifications. The following theorem formalizes this extension. Theorem 4.3. The Feynman rules defined above, augmented by the higher-order generalizations of the tensors 𝑉, 𝐷, 𝐹, 𝐴, 𝐵, 𝑃, 𝑄, 𝑅, 𝑆, 𝑇, 𝑈, provide a complete characterization of the statistics of the orthogonal NTK and its descendants at arbitrary order in 1/𝑛. Proof. See Appendix E.3. This result demonstrates that our diagrammatic formalism enables systematic computation of preactivation and NTK statistics to arbitrary order in 1/𝑛 from a concise set of rules. As a concrete example, we derive in Appendix G the recursion relation for the 𝑉6 tensor at order 1/𝑛2 , in contrast to the cumbersome algebraic manipulations demanded by direct methods.

Applications

In this section, we deduce a set of results for orthogonal neural networks from the Feynman-diagrammatic framework developed above.

5.1

Criticality and stability at finite width

Deep neural network outputs are highly sensitive to initialization: naive architectural choices can cause preactivations to explode or vanish exponentially. At infinite width, this stability is characterized by the susceptibility 𝜒, which controls the growth of both the NNGP and the NTK. This analysis extends to finite-width Gaussian-initialized networks, where criticality at infinite width enforces criticality at finite width for both preactivation and NTK statistics [13, 17]. Next, we combine the ideas developed in [13] with the Feynman rules of Section 4 to show that Theorem 5.1. Criticality of the infinite-width NNGP and NTK implies criticality of orthogonal preactivation and NTK cumulants. Proof. The proof is twofold. First, as established in [13], the stability analysis admits a bootstrap structure: criticality of lower-rank tensors suffices to control higher-rank tensors, whose behavior can then be analyzed systematically using the Feynman rules of Section 4 or Appendix B. Second, at any fixed order in 1/𝑛, orthogonality modifies the recursion of a given tensor only through the introduction of lowerrank tensors, as prescribed by Rule 4 in Section 4.1. This rule introduces Weingarten functions multypling diagrams generated by inequivalent permutations relative to the reference pairing. By definition, these contributions are subleading compared to the identity permutation. Since these lower-rank tensors are already assumed to be critical under the bootstrap hypothesis, the resulting stability analysis for orthogonal tensors coincides with that of the Gaussian case. 9

CW( ) = 0.25

CW( ) = 1.00

( , ) = (0, 0) ( , ) = (0, 1)

100

1.7 × 100 1.6 × 100

()

1.5 × 100 1.4 × 100

10 1

1.3 × 100 1.2 × 100

101

100

101

100

CW( ) = 4.00

105 104 103 102 101 100 0 100 101 102

101

100

) Figure 1: Gradient stability. Monte Carlo estimates of the NTK Θ (ℓ 𝛼𝛽 for a tanh orthogonal network (𝑛 = 50, 𝐿 = 30) across varying 𝐶𝑊 , with the critical case 𝐶𝑊 = 1 shown in the center. Means are computed over 600 initializations; shaded regions denote standard errors (typically not visible). Results are consistent with Theorem 5.1. Experimental 0.5

0.1

0.4

F D B A

0.2 0.3 0.4

0.10

0.3

<n >n

0.2

(a)

<n >n

0.25

0.0

0.15 0.20

0.1

Experimental Prediction 2

0.05

V( )

0.0

0.5

Large- Expansion

Prediction 0.00

K( )

Normalized NTK Tensors

0.1

100

0.30

100

(b)

Figure 2: Orthogonal saturation. (a) Normalized NTK tensors from Monte Carlo simulations of tanh orthogonal networks (𝑛 = 50, 𝐿 = 10) at criticality, compared with theoretical predictions. Means are computed over 600 initializations; shaded regions denote standard deviations (typically not visible), showing quantitative agreement. (b) Large-ℓ expansions of the NNGP and quartic vertex 𝑉, compared with exact solutions and simulations. While discrepancies appear at small ℓ, asymptotic predictions are in precise agreement at large ℓ. The theory captures the regime ℓ > 𝑛, consistent with [25], where orthogonal networks exhibit saturation at large depth, unlike the Gaussian case. In Figure 1, we empirically verify this by sampling the NTK components of a tanh orthogonal MLP (𝑛 = 50, 𝐿 = 30) with inputs drawn i.i.d. from the interval (0, 1) across varying 𝐶𝑊 (see Appendix I). This illustrates the effectiveness of the diagrammatic framework for deriving results to all orders in 1/𝑛.

5.2

Single-input solutions

Once derived within the Feynman diagrammatic framework, the recursion relations can be solved for given initial conditions. We focus on the single-input setting with tanh activation and solve the resulting recursions for the NTK tensors (see Appendix F for an explicit derivation). We consider networks of width 𝑛 = 50 and depth 𝐿 = 10, initialized at criticality 𝐶𝑊 = 1. Gaussian expectations are evaluated numerically in Mathematica. We analyze normalized cumulants, defined as ratios to their infinite-width counterparts; for example, 𝐷˜ (ℓ ) = 𝐷 (ℓ ) /(𝐾 (ℓ ) Θ (ℓ ) ) and 𝐹˜ (ℓ ) = 𝐹 (ℓ ) /(𝐾 (ℓ ) Θ (ℓ ) ). The solutions are shown in Figure 2 (a), where we observe excellent agreement with finite-width simulations, supporting the validity of our framework. See Appendix F.1 for a complete analysis, including NTK derivatives. Our results are consistent with the empirical findings of [25].

5.3

Large-ℓ expansions

The recursion relations derived above admit analytic solutions in the large-depth regime. In this limit, one can substitute the standard ansatz O (ℓ) = ℓ − 𝑝O

∞ ∑︁

𝑐 O𝑝,𝑞

log 𝑝 ℓ ℓ𝑞

(25)

𝑝,𝑞=1

into the recursion equations and determine the coefficients 𝑐 O𝑝,𝑞 systematically once the initial conditions are fixed. We derive large-ℓ expansions for the NNGP and the quartic vertex 𝑉 in the single-input setting, and compare them with exact solutions and Monte Carlo simulations in Figure 2 (b). The analytical expansion agrees well with the empirical observations for large ℓ. Complete derivations of the expansions (25) for the NTK tensors and their derivatives are provided in Appendix H.

Acknowledgments

We would like to thank Philipp Misof, Yonatan Kahn and Zhengkang (Kevin) Zhang for insightful and helpful discussions. This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation.

References [1]

Xavier Glorot and Yoshua Bengio. “Understanding the Difficulty of Training Deep Feedforward Neural Networks”. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, Mar. 2010, pp. 249–256.

[2]

Kaiming He et al. “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, pp. 1026–1034. arXiv: 1502.01852.

[3]

Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks. Feb. 2014. arXiv: 1312.6120.

[4]

Dmytro Mishkin and Jiri Matas. “All You Need Is a Good Init”. In: International Conference on Learning Representations 2016. arXiv, Feb. 2016. arXiv: 1511.06422.

[5]

Wei Hu, Lechao Xiao, and Jeffrey Pennington. “Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks”. In: International Conference on Learning Representations. Sept. 2019. arXiv: 2001.05992.

[6]

Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. “Resurrecting the Sigmoid in Deep Learning through Dynamical Isometry: Theory and Practice”. In: Advances in Neural Information Processing Systems. Vol. 30. Curran Associates, Inc., 2017. arXiv: 1711.04735.

[7]

Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. “The Emergence of Spectral Universality in Deep Networks”. In: Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics. PMLR, Mar. 2018, pp. 1924–1932. arXiv: 1802.09979.

[8]

Lechao Xiao et al. “Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks”. In: Proceedings of the 35th International Conference on Machine Learning. PMLR, July 2018, pp. 5393–5402.

[9]

Arthur Jacot, Franck Gabriel, and Clement Hongler. “Neural Tangent Kernel: Convergence and Generalization in Neural Networks”. In: Advances in Neural Information Processing Systems. Vol. 31. Curran Associates, Inc., 2018. arXiv: 1806.07572.

[10]

Wei Huang, Weitao Du, and Richard Yi Da Xu. “On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization”. In: Twenty-Ninth International Joint Conference on Artificial Intelligence. Vol. 3. Aug. 2021, pp. 2577– 2583. doi: 10.24963/ijcai.2021/355. arXiv: 2004.05867.

[11]

Daniel A. Roberts and Sho Yaida. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge: Cambridge University Press, 2022. isbn: 978-1-316-51933-2. doi: 10.1017/ 9781009023405. arXiv: 2106.10165.

[12]

Don Weingarten. “Asymptotic Behavior of Group Integrals in the Limit of Infinite Rank”. In: Journal of Mathematical Physics 19.5 (May 1978), pp. 999–1001. issn: 0022-2488. doi: 10.1063/1.523807.

[13]

Max Guillen, Philipp Misof, and Jan E. Gerken. Finite-Width Neural Tangent Kernels from Feynman Diagrams. Aug. 2025. doi: 10.48550/arXiv.2508.11522. arXiv: 2508.11522.

[14]

Jiquan Ngiam et al. “Tiled Convolutional Neural Networks”. In: Advances in Neural Information Processing Systems. Vol. 23. Curran Associates, Inc., 2010.

[15]

Sho Yaida. “Non-Gaussian Processes and Neural Networks at Finite Widths”. In: Proceedings of The First Mathematical and Scientific Machine Learning Conference. PMLR, Aug. 2020, pp. 165–192. arXiv: 1910.00019.

[16]

Anindita Maiti, Keegan Stoner, and James Halverson. “Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators”. In: Machine Learning in Pure Mathematics and Theoretical Physics. Chap. Chapter 8, pp. 293–330. doi: 10.1142/9781800613706_0008. arXiv: 2106.00694.

[17]

Ian Banta et al. “Structures of Neural Network Effective Theories”. In: Physical Review D 109.10 (May 2024), p. 105007. doi: 10.1103/PhysRevD.109.105007. arXiv: 2305.02334.

[18]

Alexander Maloney, Daniel A. Roberts, and James Sully. A Solvable Model of Neural Scaling Laws. Oct. 2022. arXiv: 2210.16859.

[19]

Zhengkang Zhang. “Neural Scaling Laws from Large-N Field Theory: Solvable Model beyond the Ridgeless Limit”. In: Machine Learning: Science and Technology 6.2 (Apr. 2025), p. 025010. issn: 2632-2153. doi: 10.1088/26322153/adc872. arXiv: 2405.19398.

[20]

James Halverson, Anindita Maiti, and Keegan Stoner. “Neural Networks and Quantum Field Theory”. In: Machine Learning: Science and Technology 2.3 (Sept. 2021), p. 035002. issn: 2632-2153. doi: 10.1088/2632-2153/abeca3. arXiv: 2008.08601.

[21]

Kevin Grosvenor and Ro Jefferson. “The Edge of Chaos: Quantum Field Theory and Deep Neural Networks”. In: SciPost Physics 12.3 (Mar. 2022), p. 081. issn: 2542-4653. doi: 10.21468/SciPostPhys.12.3.081. arXiv: 2109.13247.

[22]

Mehmet Demirtas et al. “Neural Network Field Theories: Non-Gaussianity, Actions, and Locality”. In: Machine Learning: Science and Technology 5.1 (Jan. 2024), p. 015002. issn: 2632-2153. doi: 10.1088/2632-2153/ad17d3. arXiv: 2307.03223.

[23]

Benoı̂t Collins and Piotr Śniady. “Integration with Respect to the Haar Measure on Unitary, Orthogonal and Symplectic Groups”. In: Communications in Mathematical Physics 264.3 (2006), pp. 773–795. doi: 10.1007/s00220- 0061554-3.

[24]

Benoı̂t Collins and Sho Matsumoto. “On some properties of orthogonal Weingarten functions”. In: Journal of Mathematical Physics 50.11 (2009), p. 113516. doi: 10.1063/1.3251304.

[25]

Hannah Day, Yonatan Kahn, and Daniel A Roberts. “Feature Learning and Generalization in Deep Networks with Orthogonal Weights”. In: Machine Learning: Science and Technology 6.3 (Aug. 2025), p. 035027. issn: 2632-2153. doi: 10.1088/2632-2153/adf278. arXiv: 2310.07765.

Orthogonal NTK tensors

The statistics of the joint distribution of preactivations and the NTK in orthogonal neural networks are characterized by cumulants involving both quantities. At first order in 1/𝑛, these statistics are fully determined by the two cumulants shown in (12) and (36). In this appendix, we derive explicit recursion relations for the tensors 𝐷, 𝐹, 𝐴, and 𝐵 at all orders in 1/𝑛, and then present their first-order 1/𝑛 approximations.

A.1

NTK-preactivation cross-correlator

The cumulant defined in (12) admits the following convenient representation (ℓ+1)

c 𝑖 𝑖 (𝑥3 , 𝑥4 )] E𝑐𝜃 [𝑧 𝑖(ℓ+1) (𝑥1 )𝑧 𝑖(ℓ+1) (𝑥2 ) ΔΘ 3 4 1 2 b (ℓ+1) (𝑥3 , 𝑥4 )] − E𝑐 [𝑧 (ℓ+1) (𝑥1 )𝑧 (ℓ+1) (𝑥2 )]E𝑐 [Θ b (ℓ+1) (𝑥3 , 𝑥4 )] = E𝑐𝜃 [𝑧 𝑖(ℓ+1) (𝑥1 )𝑧 𝑖(ℓ+1) (𝑥2 ) Θ 𝜃 𝑖1 𝜃 𝑖3 𝑖4 𝑖2 𝑖3 𝑖4 1 2

(26)

Substituting the preactivation recursion (1) and the NTK recursion (10), one finds that (ℓ+1)

c 𝑖 𝑖 (𝑥3 , 𝑥4 )] E𝑐𝜃 [𝑧 𝑖(ℓ+1) (𝑥1 )𝑧 𝑖(ℓ+1) (𝑥2 ) ΔΘ 3 4 1 2 (ℓ+1) 𝑛 𝜆𝑊 𝐶𝑊 ∑︁ (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) = (12)𝑖 (34)𝑖 E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 ] − E[𝜎 𝑗,1 𝜎 𝑗,2 ]E[𝜎𝑘,3 𝜎𝑘,4 ] 𝑛2 𝑗,𝑘=1 𝑛 ∑︁ 1 (ℓ ) (ℓ ) (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) (ℓ ) (ℓ ) 2 + (𝐶𝑊 ) E W [1, 1]𝜎 𝑗,1 𝜎 𝑗,2 − 2 E[𝜎 𝑗,1 𝜎 𝑗,2 ] 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 + 2W [2]E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] 𝑛 𝑗,𝑘=1 𝑛 ∑︁ (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] + (13)(24)𝑖 (𝐶𝑊 ) 2 W [1, 1]E[𝜎 𝑗,1 𝑗,𝑘=1 (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b + W [2]E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] + W [2]E[𝜎 𝑗,1 𝜎 𝑗,4 𝜎𝑘,2 𝜎𝑘,3 Θ 𝑗 𝑘,34 ]

+ (14)(23)𝑖 (𝐶𝑊 )

𝑛 ∑︁

(ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b 𝜎 𝑗,4 𝜎𝑘,2 𝜎𝑘,3 Θ 𝑗 𝑘,34 ] W [1, 1]E[𝜎 𝑗,1

𝑗,𝑘=1 (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) b 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] + W [2]E[𝜎 𝑗,1 + W [2]E[𝜎 𝑗,1

(27)

Here W [1, 1] and W [2] denote the 𝑘 = 2 Weingarten functions appearing in (2), and we have used the index symmetry of the NTK sample labels. A direct comparison of (27) and (12) then yields (ℓ+1) 𝑛 1 (ℓ+1) 𝜆𝑊 𝐶𝑊 ∑︁ (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) 𝐷 1234 = E[𝜎 𝜎 𝜎 𝜎 ] − E[𝜎 𝜎 ]E[𝜎 𝜎 ] 𝑗,1 𝑗,2 𝑘,3 𝑘,4 𝑗,1 𝑗,2 𝑘,3 𝑘,4 𝑛 𝑛2 𝑗,𝑘=1 𝑛 ∑︁ 1 (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) (ℓ ) (ℓ ) 2 + (𝐶𝑊 ) E W [1, 1]𝜎 𝑗,1 𝜎 𝑗,2 − 2 E[𝜎 𝑗,1 𝜎 𝑗,2 ] 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 𝑛 𝑗,𝑘=1 (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b (ℓ ) + 2W [2]E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 Θ 𝑗 𝑘,34 ]

(28)

𝑛 ∑︁ 1 (ℓ+1) (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b (ℓ ) 2 𝐹1324 = (𝐶𝑊 ) W [1, 1]E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] 𝑛 𝑗,𝑘=1 (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) b (ℓ ) + W [2]E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] + W [2]E[𝜎 𝑗,1 𝜎 𝑗,4 𝜎𝑘,2 𝜎𝑘,3 Θ 𝑗 𝑘,34 ]

(29)

To obtain the first-order 1/𝑛 corrections to these tensors, we expand the 𝑘 = 2 Weingarten functions in the large-𝑛 regime. Explicitly, 2 2 𝑛+1 1 1 W [1, 1] = = 2 + 4 − 5 +O 6 (30) 𝑛(𝑛 − 1) (𝑛 + 2) 𝑛 𝑛 𝑛 𝑛 1 3 1 1 1 W [2] = − (31) =− 3 + 4 − 5 +O 6 𝑛(𝑛 − 1) (𝑛 + 2) 𝑛 𝑛 𝑛 𝑛 Upon substituting (30) and (31) into (28) and (29), we obtain (ℓ+1) 𝑛 𝑛 ) ) (ℓ ) 1 (ℓ+1) 𝜆𝑊 𝐶𝑊 ∑︁ (𝐶𝑊 ) 2 ∑︁ ′(ℓ ) ′(ℓ ) (ℓ ) c (ℓ c (ℓ c 𝐷 1234 = E[ Δ𝐾 E[Δ𝐾 Δ𝐾 ] + 𝑗,12 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ]Θ34 𝑘,34 2 2 𝑛 𝑛 𝑛 𝑗,𝑘=1 𝑗,𝑘=1 𝑛

𝑛

) (𝐶𝑊 ) 2 ∑︁ (𝐶𝑊 ) 2 ∑︁ (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) c (ℓ E[ Δ𝐾 E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎 𝑗,2 𝜎 𝑗,4 ]Θ34 𝜎 𝜎 ΔΘ ] − 2 𝑗,12 𝑘𝑘,34 𝑘,3 𝑘,4 𝑛2 𝑗,𝑘=1 𝑛3 𝑗=1 𝑛 (𝐶𝑊 ) 2 ∑︁ 1 (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) c (ℓ ) (32) E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 ΔΘ 𝑗 𝑘,34 ] + O 2 −2 3 𝑛 𝑛 𝑗,𝑘=1 𝑛 𝑛 ∑︁ 1 (ℓ+1) (𝐶𝑊 ) 2 ∑︁ (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) c (ℓ ) 𝐹 = E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎 𝑗,2 𝜎 𝑗,4 ]Θ34 + E[𝜎 𝑗,1 𝜎 𝑗,3 𝜎𝑘,2 𝜎𝑘,4 ΔΘ 𝑗 𝑘,34 ] 𝑛 1324 𝑛2 𝑗=1 𝑗,𝑘=1 𝑛 𝑛 ∑︁ (𝐶𝑊 ) 2 ∑︁ (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) (ℓ ) (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑗 𝑘,34 ] E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 ]Θ34 + − 𝑛3 𝑗,𝑘=1 𝑗,𝑘=1 𝑛 𝑛 ∑︁ (𝐶𝑊 ) 2 ∑︁ 1 (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) c (ℓ ) − E[𝜎 𝑗,1 𝜎 𝑗,4 𝜎 𝑗,2 𝜎 𝑗,3 ]Θ34 + E[𝜎 𝑗,1 𝜎 𝑗,4 𝜎𝑘,2 𝜎𝑘,3 ΔΘ 𝑗 𝑘,34 ] + O 2 3 𝑛 𝑛 𝑗=1 𝑗,𝑘=1

(33) ) (ℓ ) (ℓ ) c 𝑖,(ℓ𝛼𝛽 Here we introduce the tensor Δ𝐾 = 𝜎𝑖,(ℓ𝛼) 𝜎𝑖,𝛽 − E[𝜎𝑖,(ℓ𝛼) 𝜎𝑖,𝛽 ], and decompose the NTK tensor into its c One can then directly apply the effective field theory techniques developed mean Θ and fluctuations ΔΘ. in [11] to expand non-Gaussian expectation values with an interaction controlled by the four-point vertex 𝑉4 . After lengthy and technically involved algebraic computations, one arrives at (ℓ+1) (ℓ ) (ℓ+1) c 34 c 12 𝐷 1234 = 𝐶𝑊 ⟨Δ𝐺 ΔΩ ⟩𝐾 (ℓ)

𝐶𝑊 + 4

* ∑︁ 𝛽1 ,𝛽2 ,𝛽3 ,𝛽4 ∈ {1,2,3,4}

(ℓ ) 𝑉(𝛽 1 𝛽2 ) (𝛽3 𝛽4 )

(ℓ ) + c 12 𝑑 2 ( Δ𝐺 )

(ℓ )

c 12 ) 𝑑 2 ( Δ𝐺

𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 )

* 𝐾 (ℓ)

(ℓ+1)

c 34 𝑑 2 ( ΔΩ

𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 )

)

+ 𝐾 (ℓ)

1 , (ℓ ) (ℓ ) 𝑛 𝑑𝑧 𝑑𝑧 (ℓ) 𝛽1 ,𝛽2 ∈ {1,2,3,4} 𝛽1 𝛽2 𝐾 h i (ℓ+1) (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) 2 𝐹1324 = (𝐶𝑊 ) ⟨𝜎1 𝜎2 𝜎3 𝜎4 ⟩𝐾 (ℓ) − ⟨𝜎1(ℓ ) 𝜎2(ℓ ) ⟩𝐾 (ℓ) ⟨𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) Θ34 + * + * 4 ∑︁ 𝑑 (𝜎1(ℓ ) 𝜎3′(ℓ ) ) 𝑑 (𝜎2(ℓ ) 𝜎4′(ℓ ) ) 1 (ℓ ) 2 + (𝐶𝑊 ) 𝐹 + O , 𝛾3 𝛿4 (ℓ ) (ℓ ) 𝑛 𝑑𝑧 𝑑𝑧 (ℓ) (ℓ) 𝛾 𝛼,𝛽,𝛾, 𝛿=1 + (𝐶𝑊 ) 2

∑︁

𝐾

⟨𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) 𝐷 𝛽(ℓ1 )𝛽2 34 + O

𝛿

(34)

(35)

𝐾

) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ+1) (ℓ ) (ℓ ) ) ′(ℓ ) ′(ℓ ) c (ℓ b (ℓ+1) where Δ𝐺 𝜎𝑖, 𝛼 𝜎𝑖,𝛽 + 𝐶𝑊 Θ (ℓ 𝛼𝛽 = 𝜎𝑖, 𝛼 𝜎𝑖,𝛽 − ⟨𝜎𝑖, 𝛼 𝜎𝑖,𝛽 ⟩𝐾 (ℓ) , and Ω𝑖, 𝛼𝛽 = 𝜆 𝑊 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽 with

c 𝑖,(ℓ+1) b (ℓ+1) b (ℓ+1) ΔΩ 𝛼𝛽 = Ω𝑖, 𝛼𝛽 − ⟨Ω𝑖, 𝛼𝛽 ⟩𝐾 (ℓ) . We observe that the recursion relation for the 𝐷 tensor coincides with its Gaussian counterpart, while the recursion relation for the 𝐹 tensor acquires an additional term proportional to the NTK mean. The expression in (13) is obtained from (35) by performing integrations-by-part inside the Gaussian expectation values of derivative terms. 14

A.2

NTK variance

The NTK variance admits the compact representation (ℓ+1)

(ℓ+1)

c 𝑖 𝑖 (𝑥1 , 𝑥2 ) ΔΘ c 𝑖 𝑖 (𝑥3 , 𝑥4 )] E𝑐𝜃 [ΔΘ 1 2 3 4 1 (ℓ+1) (ℓ+1) (ℓ+1) (13) (24)𝑖 +𝐵1423 (14) (23)𝑖 , = 𝐴1234 (12) (34)𝑖 +𝐵1324 𝑛

(36)

where the tensors 𝐴 and 𝐵 are fully determined by 𝐾ˆ and Θ̂. To see this, we rewrite the cumulant (36) in the equivalent form (ℓ+1)

(ℓ+1)

c 𝑖 𝑖 (𝑥1 , 𝑥2 ) ΔΘ c 𝑖 𝑖 (𝑥3 , 𝑥4 )] E𝑐𝜃 [ΔΘ 1 2 3 4 b (ℓ+1) (𝑥1 , 𝑥2 ) Θ b (ℓ+1) (𝑥3 , 𝑥4 )] − E𝑐 [Θ b (ℓ+1) (𝑥1 , 𝑥2 )]E𝑐 [Θ b (ℓ+1) (𝑥3 , 𝑥4 )] = E𝑐𝜃 [Θ 𝜃 𝜃 𝑖1 𝑖2 𝑖3 𝑖4 𝑖1 𝑖2 𝑖3 𝑖4

(37)

Substituting the recursion of the NTK tensor from (10), we obtain that (ℓ+1)

(ℓ+1)

c 𝑖 𝑖 (𝑥1 , 𝑥2 ) ΔΘ c 𝑖 𝑖 (𝑥3 , 𝑥4 )] E𝑐𝜃 [ΔΘ 1 2 3 4 (ℓ+1) 2 ∑︁ (ℓ+1) 𝑛 𝜆𝑊 𝐶𝑊 ∑︁ (𝜆𝑊 ) ) (ℓ ) ) ′(ℓ ) ′(ℓ ) b (ℓ ) c (ℓ c c (ℓ E[ Δ𝐾 Δ𝐾 ] + E[Δ𝐾 = (12)(34)𝑖 𝑗,12 𝑗,34 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] 2 2 𝑛 𝑛 𝑗,𝑘 𝑗,𝑘=1 𝑛 𝑛 ∑︁ 𝜆 (ℓ+1) 𝐶𝑊 ∑︁ ) ′(ℓ ) ′(ℓ ) b (ℓ ) ) ′(ℓ ) b ′(ℓ ) ′(ℓ ) b (ℓ ) 2 c (ℓ 𝜎 𝜎 Θ ] + (𝐶 ) E[Δ𝐾 W [1, 1]E[𝜎 ′(ℓ + 𝑊 2 𝑊 𝑗,34 𝑘,1 𝑘,2 𝑘𝑘,12 𝑗,1 𝜎 𝑗,2 Θ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] 𝑛 𝑗,𝑘=1 𝑗,𝑘=1 𝑛 𝑛 ∑︁ 1 ∑︁ ′(ℓ ) ′(ℓ ) b (ℓ ) ) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) ] + W [2]E[𝜎 ′(ℓ 𝜎 Θ ]E[𝜎 𝜎 Θ E[𝜎 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] 𝑘𝑘,34 𝑘,3 𝑘,4 𝑗,2 𝑗 𝑗,12 𝑗,1 𝑛2 𝑗,𝑘=1 𝑗,𝑘=1 𝑛 ∑︁ ′(ℓ ) ′(ℓ ) b (ℓ ) ) ′(ℓ ) b (ℓ ) + W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 Θ 𝑘 𝑗,34 ]

−

𝑗,𝑘=1

+ (13)(24)𝑖 (𝐶𝑊 )

∑︁ 𝑛

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 Θ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ]

𝑗,𝑘=1

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [1, 1]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] +

𝑗,𝑘=1

𝑛 ∑︁

′(ℓ ) ′(ℓ ) b (ℓ ) ) ′(ℓ ) b (ℓ ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 Θ 𝑘 𝑗,34 ]

𝑗,𝑘=1

+ (14)(23)𝑖 (𝐶𝑊 ) 2

∑︁ 𝑛

′(ℓ ) ′(ℓ ) b (ℓ ) ) ′(ℓ ) b (ℓ ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 Θ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ]

𝑗,𝑘=1

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] +

𝑗,𝑘=1

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [1, 1]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 Θ 𝑘 𝑗,34 ]

𝑗,𝑘=1

(38) A direct comparison with (36) then yields (ℓ+1) 2 (ℓ+1) 𝑛 𝜆𝑊 𝐶𝑊 ∑︁ ) 1 (ℓ+1) (𝜆𝑊 ) ∑︁ c (ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) c (ℓ 𝐴1234 = E[ Δ𝐾 Δ𝐾 ] + E[Δ𝐾 𝑗,12 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] 𝑗,34 2 2 𝑛 𝑛 𝑛 𝑗,𝑘 𝑗,𝑘=1

(ℓ+1) 𝑛 𝜆𝑊 𝐶𝑊 ∑︁

𝑛2

) ′(ℓ ) ′(ℓ ) b (ℓ ) 2 c (ℓ E[Δ𝐾 𝑗,34 𝜎𝑘,1 𝜎𝑘,2 Θ 𝑘𝑘,12 ] + (𝐶𝑊 )

𝑗,𝑘=1

−

∑︁ 𝑛

𝑗,𝑘=1 𝑛 ∑︁

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [1, 1]E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 Θ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ]

1 ) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) ) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) E[𝜎 ′(ℓ W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 Θ 𝑗 𝑗,12 ]E[𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] + 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] 𝑛2 𝑗,𝑘=1 𝑗,𝑘=1

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 Θ 𝑘 𝑗,34 ]

(39)

𝑗,𝑘=1

∑︁ 𝑛 1 (ℓ+1) ) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) 2 𝐵1324 = (𝐶𝑊 ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 Θ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 Θ 𝑘𝑘,34 ] 𝑛 𝑗,𝑘=1 +

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [1, 1]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 Θ 𝑗 𝑘,34 ] +

𝑗,𝑘=1

𝑛 ∑︁

) ′(ℓ ) b (ℓ ) ′(ℓ ) ′(ℓ ) b (ℓ ) W [2]E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 Θ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 Θ 𝑘 𝑗,34 ]

𝑗,𝑘=1

(40) The use of the 1/𝑛-expansions (30), (31) allows the following simplifications 𝑛 1 (ℓ+1) 1 ∑︁ c (ℓ+1) c (ℓ+1) 𝐶𝑊 ∑︁ ′(ℓ ) ′(ℓ ) c (ℓ ) c (ℓ+1) E[ΔΩ 𝑗,12 ΔΩ 𝑗,34 ] + 2 E[ΔΩ 𝐴1234 = 2 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ] 𝑛 𝑛 𝑗,𝑘 𝑛 𝑗,𝑘=1 𝑛 𝑛 2 ∑︁ (ℓ+1) ′(ℓ ) ′(ℓ ) (ℓ ) 𝐶𝑊 ∑︁ (𝐶 ) 𝑊 ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) c 𝑗,34 𝜎 𝜎 ΔΘ c 𝑘𝑘,12 ] + E[ΔΩ E[𝜎 ′(ℓ 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ] + 2 𝑗,1 𝑘,1 𝑘,2 2 𝑛 𝑗,𝑘=1 𝑛 𝑗,𝑘=1 𝑛 2 ∑︁ ) ′(ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) E[𝜎 ′(ℓ − E[𝜎 𝑗,1 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 ]E[𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ] − 𝑗,1 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ]Θ12 Θ34 𝑛 𝑗=1 𝑗,𝑘=1 𝑛 𝑛 ∑︁ ∑︁ 2 2 ) ′(ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) (ℓ ) ) ′(ℓ ) c (ℓ ) ) ′(ℓ ) (ℓ ) E[𝜎 ′(ℓ 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ΔΘ 𝑗 𝑗,34 ]Θ12 − E[𝜎 ′(ℓ 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎 ′(ℓ − 𝑗,1 𝑗,1 𝑗,3 𝜎 𝑗,4 ]Θ34 𝑛 𝑗=1 𝑛 𝑗=1 𝑛 𝑛 1 ∑︁ 1 ∑︁ ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) − E[𝜎 𝑗,1 𝜎𝑘,2 ΔΘ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 ΔΘ 𝑗 𝑘,34 ] − E[𝜎 𝑗,1 𝜎𝑘,2 ΔΘ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 ΔΘ 𝑘 𝑗,34 ] 𝑛 𝑗,𝑘=1 𝑛 𝑗,𝑘=1 𝑛 ∑︁

1 +O 2 𝑛

(41)

𝑛 𝑛 1 ∑︁ 1 (ℓ+1) (𝐶𝑊 ) 2 1 ∑︁ ′(ℓ ) ′(ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ) ′(ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) (ℓ ) − 𝐵1324 = E[𝜎 𝜎 𝜎 𝜎 ]Θ Θ − E[𝜎 ′(ℓ 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ]Θ12 𝑗,1 𝑗,2 𝑘,3 𝑘,4 12 34 𝑗,1 2 𝑛 𝑛 𝑗,𝑘=1 𝑛 𝑗,𝑘=1 𝑛 𝑛 𝑛 1 ∑︁ 1 ∑︁ ′(ℓ ) ′(ℓ ) (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ) ′(ℓ ) c (ℓ ) − E[𝜎 𝑗,1 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ]Θ34 − E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ] 𝑛 𝑗,𝑘=1 𝑛 𝑗,𝑘=1

𝑛 ∑︁

(ℓ ) (ℓ ) ) ′(ℓ ) ′(ℓ ) ′(ℓ ) E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ]Θ12 Θ34 +

(ℓ )

(ℓ ) ) ′(ℓ ) ′(ℓ ) ′(ℓ ) c E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ΔΘ 𝑗 𝑗,34 ]Θ12 +

𝑛 ∑︁

(ℓ )

′(ℓ ) ′(ℓ ) c ) ′(ℓ ) c E[𝜎 ′(ℓ 𝑗,1 𝜎𝑘,2 ΔΘ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 ΔΘ 𝑗 𝑘,34 ]

𝑗,𝑘=1 𝑛 ∑︁

𝑗=1

−

(ℓ )

′(ℓ ) ′(ℓ ) (ℓ ) ) ′(ℓ ) c E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎 𝑗,3 𝜎 𝑗,4 ]Θ34

𝑗=1

𝑗=1 𝑛 ∑︁

𝑛 ∑︁

1 1 ) ′(ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) E[𝜎 ′(ℓ E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ]Θ12 Θ34 − 𝑛 𝑗,1 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎 𝑗,3 𝜎 𝑗,4 ]Θ34 𝑛 𝑗=1 𝑗=1

𝑛 𝑛 1 ∑︁ 1 ∑︁ ′(ℓ ) ′(ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) (ℓ ) − E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ΔΘ 𝑗 𝑗,34 ]Θ12 − E[𝜎 𝑗,1 𝜎𝑘,2 ΔΘ 𝑗 𝑘,12 𝜎𝑘,3 𝜎 𝑗,4 ΔΘ 𝑘 𝑗,34 ] 𝑛 𝑗=1 𝑛 𝑗,𝑘=1 1 (42) +O 2 𝑛 c These expressions Here, we again decompose the NTK tensor into its mean Θ and fluctuations ΔΘ. can be further simplified by noting that not all contributions arising from the off-diagonal entries of the Weingarten matrix appear at order 1/𝑛. In particular, the terms proportional to 1/𝑛 inside the square brackets in (41) scale at most linearly with 𝑛, as follows from (12) and (36), yielding an overall contribution

of order O

1 𝑛2

. This observation leads to the formula

𝑛 1 (ℓ+1) 1 ∑︁ c (ℓ+1) c (ℓ+1) 𝐶𝑊 ∑︁ ′(ℓ ) ′(ℓ ) c (ℓ ) c (ℓ+1) 𝐴1234 = 2 E[ΔΩ 𝑗,12 ΔΩ 𝑗,34 ] + 2 E[ΔΩ 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ] 𝑛 𝑛 𝑗,𝑘 𝑛 𝑗,𝑘=1 𝑛 𝑛 2 ∑︁ (ℓ+1) ′(ℓ ) ′(ℓ ) (ℓ ) 𝐶𝑊 ∑︁ (𝐶 ) 𝑊 ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) c 𝑗,34 𝜎 𝜎 ΔΘ c 𝑘𝑘,12 ] + E[ΔΩ E[𝜎 ′(ℓ 𝜎 𝑗,2 ΔΘ 𝑗 𝑗,12 𝜎𝑘,3 𝜎𝑘,4 ΔΘ 𝑘𝑘,34 ] + 2 𝑗,1 𝑘,1 𝑘,2 2 𝑛 𝑗,𝑘=1 𝑛 𝑗,𝑘=1 𝑛 ∑︁ 1 ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) (43) − E[𝜎 ′(ℓ 𝜎 ΔΘ ]E[𝜎 𝜎 ΔΘ ] + O 𝑗 𝑗,12 𝑘𝑘,34 𝑗,1 𝑗,2 𝑘,3 𝑘,4 2 𝑛 𝑗,𝑘=1

Similarly, the only term that can yield a non-zero contribution at order 𝑛1 in (42) is the first term, ) ′(ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ ) E[𝜎 ′(ℓ 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 ]Θ12 Θ34 , through its off-diagonal components. Likewise, the terms of order 𝑛0 that vanish at order 𝑛1 are precisely those containing a single NTK fluctuation inside the expectation value, in agreement with (12). We therefore obtain 𝑛 𝑛 ∑︁ 1 (ℓ+1) (𝐶𝑊 ) 2 1 ∑︁ ′(ℓ ) ′(ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ ) ) ′(ℓ ) ′(ℓ ) ′(ℓ ) (ℓ ) (ℓ ) 𝐵 = − E[𝜎 𝑗,1 𝜎 𝑗,2 𝜎𝑘,3 𝜎𝑘,4 ]Θ12 Θ34 + E[𝜎 ′(ℓ 𝜎 𝑗,2 𝜎 𝑗,3 𝜎 𝑗,4 ]Θ12 Θ34 𝑗,1 2 𝑛 1324 𝑛 𝑛 𝑗,𝑘=1 𝑗=1 𝑛 ∑︁ 1 ′(ℓ ) ′(ℓ ) c (ℓ ) ′(ℓ ) ′(ℓ ) c (ℓ ) (44) + E[𝜎 𝑗,1 𝜎𝑘,2 ΔΘ 𝑗 𝑘,12 𝜎 𝑗,3 𝜎𝑘,4 ΔΘ 𝑗 𝑘,34 ] + O 2 𝑛 𝑗,𝑘=1 One can apply the effective field theory techniques of [11] to compute the first-order 1/𝑛 corrections to the expectation values appearing in (43) and (44). Using (12) and (36), together with extensive path-integral manipulations and algebraic identities, one finds that (ℓ+1) 𝐴1234

(ℓ+1)

c 12 = ⟨ΔΩ 1 + 4

(ℓ+1) c 34 ⟩𝐾 (ℓ) ΔΩ

* ∑︁

(ℓ ) 𝑉(𝛽 𝛽 ) (𝛽 𝛽 )

𝛽1 ,𝛽2 ,𝛽3 ,𝛽4 ∈ {1,2,3,4}

1 2

3 4

(ℓ+1)

c 12 𝑑 2 ( ΔΩ

)

𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 )

(ℓ+1)

c 34 𝑑 2 ( ΔΩ

𝐾 (ℓ)

)

𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 )

𝐾 (ℓ)

(ℓ+1) + ∑︁ c 12 𝑑 2 ( ΔΩ ) 𝐶𝑊 ⟨𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) 𝐷 𝛽(ℓ1 )𝛽2 34 + (ℓ ) (ℓ ) 2 𝑑𝑧 𝛽1 𝑑𝑧 𝛽2 𝐾 (ℓ) 𝛽1 ,𝛽2 ∈ {1,2,3,4} * (ℓ+1) + ∑︁ c 34 𝑑 2 ( ΔΩ ) 𝐶𝑊 (ℓ ) + ⟨𝜎1′(ℓ ) 𝜎2′(ℓ ) ⟩𝐾 (ℓ) 𝐷 12𝛽 (ℓ ) (ℓ ) 3 𝛽4 2 𝑑𝑧 𝛽3 𝑑𝑧 𝛽4 𝐾 (ℓ) 𝛽3 ,𝛽4 ∈ {1,2,3,4}

(ℓ ) +O +(𝐶𝑊 ) ⟨𝜎1′(ℓ ) 𝜎2′(ℓ ) ⟩𝐾 (ℓ) ⟨𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) 𝐴1234 2

(ℓ+1) 𝐵1324

= (𝐶𝑊 )

1 , 𝑛

⟨𝜎1′(ℓ ) 𝜎2′(ℓ ) 𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) − ⟨𝜎1′(ℓ ) 𝜎2′(ℓ ) ⟩𝐾 (ℓ) ⟨𝜎3′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ)

(ℓ ) +(𝐶𝑊 ) ⟨𝜎1′(ℓ ) 𝜎3′(ℓ ) ⟩𝐾 (ℓ) ⟨𝜎2′(ℓ ) 𝜎4′(ℓ ) ⟩𝐾 (ℓ) 𝐵1324 +O 2

1 . 𝑛

(45)

(ℓ ) (ℓ ) Θ12 Θ34

(46)

Here we observe that the recursion relation for the tensor 𝐴 coincides with its Gaussian counterpart, while that for the tensor 𝐵 acquires an additional term proportional to the NTK mean.

Generalized Feynman rules

1. Preactivations, NTKs, dNTKs and ddNTKs are represented by external lines, as illustrated below. c 𝛼𝛽 ≡ 𝛽 ΔΘ 𝛼

𝑧𝛼 ≡ 𝛼

c𝛿 𝛿 𝛿 ≡ dΘ 0 1 2

𝛿2 𝛿1 𝛿0

𝛿3 𝛿2 𝛿1 𝛿0

dd I Θ 𝛿0 𝛿1 𝛿2 𝛿3 ≡

dd II Θ 𝛿1 𝛿2 𝛿3 𝛿4 ≡

𝛿4 𝛿3 𝛿2 𝛿1

(47)

In the first line of (47), a colored line represents a single NTK label. Distinct colors are used for external dotted lines associated with different NTKs. In the second line, we introduce the diagram2 c 𝛿 𝛿 𝛿 = Í 𝜇𝜈 𝑑 𝑧 𝛿0 𝑑𝑧 𝛿1 𝑑𝑧 𝛿2 , the ddI NTK: dd matic representations of the dNTK: dΘ I Θ 𝛿0 𝛿1 𝛿2 𝛿3 = 0 1 2 𝑑 𝜃 𝜇 𝑑 𝜃𝜈 𝑑 𝜃 𝜇 𝑑 𝜃𝜈 3 2 2 Í Í 𝑑 𝑧 𝛿0 𝑑 𝑧 𝛿1 𝑑 𝑧 𝛿2 𝑑𝑧 𝛿3 𝑑𝑧 𝛿4 𝑑𝑧 𝛿1 𝑑𝑧 𝛿2 𝑑𝑧 𝛿3 𝜇𝜈𝜌 𝑑 𝜃 𝜇 𝑑 𝜃𝜈 𝑑 𝜃𝜌 𝑑 𝜃 𝜇 𝑑 𝜃𝜈 𝑑 𝜃𝜌 , and the ddII NTK: ddII Θ 𝛿1 𝛿2 𝛿3 𝛿4 = 𝜇𝜈𝜌 𝑑 𝜃 𝜇 𝑑 𝜃𝜈 𝑑 𝜃𝜌 𝑑 𝜃𝜈 𝑑 𝜃 𝜇 𝑑 𝜃𝜌 . The colors of the dotted lines correspond to the 𝜃-indices appearing in these definitions. Since the 𝜃-indices are contracted in pairs, distinct colors encode the corresponding contraction pattern. In the diagrammatic notation, triple lines represent third derivatives, double lines represent second derivatives, and single lines represent first derivatives. 2. Define the cubic vertices as 𝛽𝑐

(ℓ)

𝛽

(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

(ℓ)

∼ 𝐶𝑊

𝛼𝑐

𝛽𝑐

(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

′(ℓ)

𝛽𝑐

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

(ℓ)

𝛽𝑐

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐

∼ 𝐶𝑊

(ℓ)

′(ℓ)

∼ 𝐶𝑊

′′(ℓ)

′(ℓ)

(ℓ)

∼1

𝛼

𝛼𝑐 𝛽𝑐

(ℓ)

∼1

𝛼

𝛼𝑐

′′(ℓ)

∼ 𝐶𝑊

𝛽𝑐 ′′(ℓ) (ℓ) 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

′′(ℓ)

Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼𝑐

𝛽 ′(ℓ) ′(ℓ) 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

′′′(ℓ)

′(ℓ)

Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐

𝛽𝑐

𝛽 ′(ℓ) (ℓ) 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼

𝛽𝑐 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐

𝛽 ′(ℓ) ′(ℓ) 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼1

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐 𝛽𝑐

(ℓ)

𝛽𝑐 ′(ℓ) ′(ℓ) 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼𝑐

∼ 𝐶𝑊

𝛼

𝛽𝑐 (ℓ) ′′(ℓ) ′(ℓ) Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐

𝛽𝑐

′(ℓ)

𝛽 ′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼𝑐

∼ 𝐶𝑊

𝛼𝑐 𝛽𝑐

′′(ℓ) (ℓ) 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

𝛼𝑐 𝛽𝑐

′(ℓ)

𝛼𝑐

𝛼

𝛽𝑐

(ℓ)

Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼1

(ℓ)

∼1

′′(ℓ)

′(ℓ)

Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼ 𝐶𝑊

(48)

𝛼𝑐

Here the superscript 𝑐, referred to as the orthogonality charge, keeps track of the label’s orthogonal character, and lines that do not end in a dot represent internal lines. 18

3. Draw a square propagator connecting internal lines in all possible ways, consistent with the chosen pairing. The square represents the full expectation value. (49) E[ · ]

This procedure generates distinct diagram types, both connected and disconnected. The connected diagrams are further classified as 𝑠-class diagrams, defined by the number 𝑠 of square propagators appearing in the diagram. The square propagator obeys an additional selection rule: all k-class diagrams formed from pairs of external lines corresponding to different object types vanish. 4. For each 𝑠-class diagram, generate all inequivalent permutations of its 2𝑚 external labels carrying orthogonality charge. Multiply each resulting diagram by 1/𝑛 for every uncharged pairing, and by the appropriate 𝑚-class Weingarten function W, determined by the relative ordering 𝜏 of the diagram’s labels with respect to the original pairing 𝜋 = (12) (34) . . . (2𝑘 2𝑘 − 1): W [𝜏, 𝜋] = W [ℓ(𝜏 ◦ 𝜋)], where ◦ denotes ordinary permutation multiplication and ℓ(𝑒) denotes the cycle length of 𝑒. 5. Multiply each 𝑠-class contribution by the Möbius coefficient (−1) 𝑠−1 (𝑠 − 1)!, and sum over all classes. The second group implements the effective field theory techniques developed in [11], applied to the square propagator in the diagrammatic construction of the previous step, through the following set of Feynman rules analogous to those introduced in [13]: 6. We define the bare propagator as ⟨ ⟩𝐾 (ℓ) ≡

(50)

where ⟨ ⟩𝐾 (ℓ) denotes a zero-mean Gaussian expectation with covariance specified by 𝐾 (ℓ ) . The expectation value is taken over the decorations of the internal lines attached to the propagator, which obeys the same selection rules described in [13]. These rules are summarized as follows: (a) Propagators may only connect to internal lines emanating from cubic vertices or from the internal quartic vertices introduced below. In particular, propagators cannot be directly connected to other propagators. (b) Dotted lines attached to a propagator do not enter the Gaussian expectation value, as they carry no decorations. (c) Each preactivation line decorated with 𝑧𝑖 acts as a derivative with respect to 𝑧 𝑖 acting on the argument of the Gaussian expectation value. (d) The neural indices of all internal lines connected to a propagator must be identical. (e) If both dotted and dashed lines of the same color are attached to the propagator, they must appear in pairs carrying the same sample index. The two lines in each pair attach to different vertices. Moreover, if both vertices are drawn in the orientation specified in the Feynman rules, the top-to-bottom ordering of the sample indices (and therefore the colors) of the lines connected to the two vertices must coincide. (f) A pair of dashed lines of the same color connected to the propagator contributes a factor Θ 𝛼𝛽 when the two lines attach to different vertices, where 𝛼 and 𝛽 denote the sample indices of the pair. 7. Quartic vertices are defined analogously, following [13]. Explicitly, 𝛼2 𝛼1

1 (ℓ+1) 𝑛 𝑉𝛼1 𝛼2 𝛼3 𝛼4

𝛼3

𝛼2

𝛼4

𝛼1

(ℓ+1) 1 𝑛 𝐷 𝛼1 𝛼2 𝛼3 𝛼4

𝛼3

𝛼4

𝛼1

𝛼2

1 (ℓ+1) 𝑛 𝐹𝛼1 𝛼3 𝛼2 𝛼4

𝛼4

𝛼3

𝛼2 𝛼2 𝛼1

1 (ℓ+1) 𝑛 𝐴 𝛼1 𝛼2 𝛼3 𝛼4

𝛼3

𝛼4

𝛼1

𝛼2

1 (ℓ+1) 𝑛 𝐵 𝛼1 𝛼3 𝛼2 𝛼4

𝛼4

𝑛

𝛼3

𝛼2

𝛼1 1 𝑄 𝛼(ℓ+1) 𝛼4 𝛼 𝛼 𝛼

𝛼1 1 𝑅 𝛼(ℓ+1) 𝛼4 𝛼 𝛼 𝛼

𝛼3

𝛼4

𝑛

1 2 3 4

𝛼2

𝛼1 1 𝑃𝛼(ℓ+1) 𝛼4 𝛼 𝛼 𝛼 3 1 2 4

𝛼3

𝛼2

𝛼1 1 𝑆 𝛼(ℓ+1) 𝛼4 𝛼 𝛼 𝛼 𝑛

1 2 3 4

𝛼2

(51) 𝛼1 1 𝑇𝛼(ℓ+1) 𝛼4 𝛼 𝛼 𝛼 𝑛

𝛼1 1 𝑈𝛼(ℓ+1) 𝛼3 𝛼 𝛼 𝛼 𝑛

1 3 4 2

1 4 2 3

8. Higher-order NTK and preactivation tensors are introduced via a natural generalization of the vertices in (51). 9. The square propagator decomposes into all connected and disconnected diagrams built from the bare propagator, quartic vertices, and higher-order vertices, with internal lines remaining undotted. This decomposition respects the selection rules (a)-(f).

Feynman rules in action

In this appendix, we explicitly apply the Feynman rules (1)-(9) of Section 4 to the tensors 𝑉4 , 𝐹, and 𝐵.

C.1

The quartic vertex: 𝑉4

The Feynman rules (1)-(5) generate the following diagrams for the quartic vertex 𝑉4 : 2

= W [1, 1]

∑︁ 2𝑐 𝑗,𝑘

+ W [2]

∑︁ 4𝑐 𝑗,𝑘

1𝑐

𝜎𝑗 𝜎𝑗

3𝑐

𝜎𝑘 𝜎𝑘

+ W [2]

1𝑐

∑︁ 3𝑐 𝑗,𝑘

4𝑐 𝜎𝑗 𝜎𝑗

𝜎𝑘 𝜎𝑘

2𝑐

− (W [1]) 3𝑐

𝜎𝑗 𝜎𝑗

1𝑐

∑︁ 2𝑐 𝑗,𝑘

1𝑐

𝜎𝑘 𝜎𝑘

2𝑐 4𝑐

𝜎𝑗 𝜎𝑗

𝜎𝑘 𝜎𝑘

3𝑐 4𝑐

(52) The first three terms in (52) are 1-class diagrams, as they are rendered fully connected by the square propagator. They arise from the three inequivalent pairings obtained by permuting the reference pairing (12)(34), namely {(12) (34), (13) (24), (14) (23)}. The 𝑘 = 2 Weingarten functions are selected according to the relative ordering between the pairing 𝜋 appearing in the diagram and the reference pairing 𝜏: W [𝜋, 𝜏] = W [1, 1] for 𝜋 = 𝜏, W [𝜋, 𝜏] = W [2] for 𝜋 ≠ 𝜏. The coefficient multiplying each contribution is fixed by the Möbius formula evaluated on the single-block partition 𝑠 = 1. By contrast, the last diagram in (52) belongs to the 2-class, since the two cubic vertices are disconnected. Each subdiagram admits a unique pairing, yielding the 𝑘 = 1 Weingarten function 𝑊 [1]. The negative coefficient associated with this contribution follows from the Möbius formula applied to the two-block partition 𝑠 = 2. 20

We now apply the second set of Feynman rules, (6)–(9), to the square propagators in (52). Since we are interested in the first-order 𝑛1 correction to the quartic vertex, we approximate the Weingarten functions by their leading terms in the 𝑛1 expansions (30) and (31). Consequently, (52) simplifies to 2𝑐

3𝑐

1𝑐

4𝑐

1 ∑︁ 2 = 2 𝑛 𝑗,𝑘

3𝑐

𝑐

𝜎𝑗 𝜎𝑗

𝜎𝑘 𝜎𝑘

1𝑐

1 ∑︁ 3 − 3 𝑛 𝑗,𝑘

4𝑐

𝑐

𝜎𝑗 𝜎𝑗

2𝑐

𝜎𝑘 𝜎𝑘

1𝑐

4𝑐

1 ∑︁ 2 − 2 𝑛 𝑗,𝑘

𝑐

𝜎𝑗 𝜎𝑗

𝜎𝑘 𝜎𝑘

1𝑐

1 ∑︁ 4 − 3 𝑛 𝑗,𝑘

3𝑐 4𝑐

𝜎𝑗 𝜎𝑗

𝜎𝑘 𝜎𝑘

1 +O 2 𝑛

(53)

To evaluate (53), we distinguish two cases. When 𝑖 = 𝑗, the only nontrivial contribution arises from the first line of (53), which reads 1 ∑︁ 2 𝑛2 𝑗

d𝑗 Δ𝐺

(54)

When 𝑖 ≠ 𝑗, the first line of (53) yields the following diagram: 1 ∑︁ 2 𝑛2 𝑗 , 𝑗 1

𝑧 𝑗1 𝑧 𝑗2

d𝑗 Δ𝐺 1

d𝑗 Δ𝐺 2

(55)

1 (ℓ) 𝑛 𝑉4

whereas the second line produces the following diagram: 1 © ∑︁ 3 − 3 𝑛 𝑗 ,𝑗 1 2 1 «

𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗2 𝜎 𝑗2

∑︁ 4 𝑗1 , 𝑗2

𝜎 𝑗1 𝜎 𝑗1

2 ª

𝜎 𝑗2 𝜎 𝑗2

(56)

® ® 3 ¬

Substituting (54), (55), and (56) into (53), we obtain 2

= 1

1 © ∑︁ 3 − 3 𝑛 𝑗 ,𝑗 1 2 1 «

1 ∑︁ 2 𝑛2 𝑗

d𝑗 Δ𝐺

𝜎 𝑗1 𝜎 𝑗1

1 ∑︁ 2 𝑛2 𝑗 , 𝑗 1

𝜎 𝑗2 𝜎 𝑗2

∑︁ 4

𝑗1 , 𝑗2

𝜎 𝑗1 𝜎 𝑗1

d𝑗 Δ𝐺 1

𝑧 𝑗1 𝑧 𝑗2

d𝑗 Δ𝐺 2

1 (ℓ) 𝑛 𝑉4

𝜎 𝑗2 𝜎 𝑗2

®+O 1 ® 𝑛2 3 ¬ 2 ª

(57) The 1/𝑛 scaling of the diagrams on the right-hand side can be verified by straightforward power counting: the summation in the first diagram yields a factor of 𝑛, while the summations in the remaining diagrams yield factors of 𝑛2 , which combine with the explicit prefactors to produce an overall 1/𝑛 contribution. Consequently, the expression (57) reproduces the recursion relation for the quartic vertex shown in (8), previously derived in [17] through direct algebraic manipulations.

C.2

Joint NTK-preactivation cumulant: 𝐹

We already applied the Feynman rules of Section 4 to 𝐹 in the main text; for completeness, we repeat the derivation here. 21

The Feynman rules (1)-(5) produce the following diagrams for the tensor 𝐹: 3

= W [1, 1]

∑︁ 3𝑐 𝑗,𝑘

+ W [2]

𝜎 𝑗 𝜎 ′𝑗

+ W [2]

1𝑐

∑︁ 2𝑐 𝑗,𝑘

4𝑐

∑︁ 2𝑐 𝑗,𝑘

2𝑐

𝜎𝑘 𝜎𝑘′

3𝑐

𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

+ W [2]

1𝑐

∑︁ 4𝑐 𝑗,𝑘

4𝑐

3𝑐

Θ 𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

4𝑐 𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

2𝑐

(58) 3𝑐

1𝑐

The absence of 2-class diagrams in (58) follows from the fact that the square propagator vanishes when acting on a pair of lines of different type. The 1-class diagrams are generated from the inequivalent permutations of the reference pairing (13) (24), namely {(13) (24), (12) (34), (14) (23)}. The corresponding 𝑘 = 2 Weingarten functions multiplying the subdiagrams are determined by the relative ordering of the reference pairing 𝜏 and the pairing 𝜋 appearing in the subdiagram: W [𝜋, 𝜏] = W [1, 1] for 𝜋 = 𝜏, W [𝜋, 𝜏] = W [2] for 𝜋 ≠ 𝜏. The overall factor of 1 follows from the Möbius relation for a single-block partition. At first order in 1/𝑛, the Weingarten functions can be approximated by their leading terms in the expansions (30) and (31). In this manner, (58) simplifies to 3

1 ∑︁ 3 = 2 𝑛 𝑗,𝑘

𝑐

𝜎 𝑗 𝜎 ′𝑗

2𝑐

𝜎𝑘 𝜎𝑘′

1𝑐

1 ∑︁ 2 − 3 𝑛 𝑗,𝑘

4𝑐 3𝑐

𝑐

𝜎𝑘′ 𝜎𝑘′

𝜎𝑗 𝜎𝑗

1𝑐

4𝑐

3𝑐

𝑐

𝜎𝑗 𝜎𝑗

Θ 𝜎𝑘′ 𝜎𝑘′

1𝑐

1 ∑︁ 4 − 3 𝑛 𝑗,𝑘

4𝑐

𝑐

𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

2𝑐

1 +O 2 𝑛

(59)

3𝑐

1𝑐

We now apply the second set of Feynman rules (6)–(9). As an immediate consequence, the last terms in (59) vanish at order 1/𝑛. A nonzero contribution would require the corresponding diagrams to produce a factor of 𝑛2 , which occurs when 𝑗 ≠ 𝑘 and a square propagator decomposes into two bare propagators forming disconnected subdiagrams. However, such configurations are forbidden by the selection rules (a)–(f), as each subdiagram violates color conservation. Consequently, we are left with 3

1 ∑︁ 3 = 2 𝑛 𝑗,𝑘

𝑐

𝜎 𝑗 𝜎 ′𝑗

2𝑐

𝜎𝑘 𝜎𝑘′

1𝑐

1 ∑︁ 2 − 3 𝑛 𝑗,𝑘

4𝑐

𝑐

𝜎𝑗 𝜎𝑗

1𝑐

Θ 𝜎𝑘′ 𝜎𝑘′

3𝑐

1 +O 2 𝑛

4𝑐

(60) We now expand the diagrams in (60) in terms of the bare propagator and quartic vertices by analyzing the diagonal and off-diagonal neural components of each subdiagram, together with the selection rules (a)-(f). When 𝑖 = 𝑗, the first diagram in (60) contributes as 1 ∑︁ 3 𝑛2 𝑗

𝜎 𝑗 𝜎 ′𝑗

(61) 4

When 𝑖 ≠ 𝑗, the first diagram reduces to 1 ∑︁ 3 𝑛2 𝑗 , 𝑗 1

𝜎 𝑗1 𝜎 ′𝑗

𝑧 𝑗1 𝑧 𝑗2

𝜎 𝑗2 𝜎 ′𝑗

1 (ℓ) 𝑛 𝐹4

(62) 4

whereas the second diagram becomes 1 ∑︁ 2 − 3 𝑛 𝑗 ,𝑗 1

Θ 𝜎 ′𝑗 𝜎 ′𝑗

𝜎 𝑗1 𝜎 𝑗1

(63)

Substituting (61), (62), and (63) into (60), we obtain 3

= 1

1 ∑︁ 3 𝑛2 𝑗

𝜎 𝑗 𝜎 ′𝑗

−

1 ∑︁ 2 𝑛3 𝑗 , 𝑗 1

1 ∑︁ 3 𝑛2 𝑗 , 𝑗

𝜎 𝑗2 𝜎 ′𝑗

1 (ℓ) 𝑛 𝐹4

𝑧 𝑗1 𝑧 𝑗2

Θ 𝜎 ′𝑗 𝜎 ′𝑗

𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗1 𝜎 ′𝑗

1 𝑛2

(64)

The 1/𝑛 scaling of the diagrams follows from straightforward power counting of the respective summations: When 𝑖 = 𝑗, the summation contributes factor of 𝑛, whereas when 𝑖 ≠ 𝑗 it yields a factor of 𝑛2 . Combined with the explicit coefficients on the right-hand side of (64), these factors produce an overall contribution of order 1/𝑛2 . It is straightforward to verify that the first and second lines on the RHS of (64) reproduce the corresponding lines on the RHS of (13).

C.3

NTK variance: 𝐵

The Feynman rules (1)-(5) yield the following diagrams for the tensor 𝐵: 3

= W [1, 1]

∑︁ 3𝑐 𝑗,𝑘

+ W [2]

∑︁ 2𝑐 𝑗,𝑘

+ W [2]

𝜎 ′𝑗 𝜎 ′𝑗

+ W [2]

1𝑐

∑︁ 2𝑐 𝑗,𝑘

4𝑐 Θ 𝜎 ′𝑗 𝜎 ′𝑗

3𝑐

𝜎𝑘′ 𝜎𝑘′

+ W [2]

𝑗,𝑘

𝜎 ′𝑗 𝜎 ′𝑗

𝜎𝑘′ 𝜎𝑘′

3𝑐

∑︁ 4𝑐

+ W [2]

1𝑐

𝑗,𝑘

4𝑐

Θ 𝜎 ′𝑗 𝜎 ′𝑗

3𝑐

Θ 𝜎𝑘′ 𝜎𝑘′

1𝑐

∑︁ 2𝑐

4𝑐

1𝑐

∑︁ 2𝑐 𝑗,𝑘

2𝑐

𝜎𝑘′ 𝜎𝑘′

4𝑐 𝜎 ′𝑗 𝜎 ′𝑗

3𝑐

Θ 𝜎𝑘′ 𝜎𝑘′

1𝑐

4𝑐 𝜎 ′𝑗 𝜎 ′𝑗

𝜎𝑘′ 𝜎𝑘′

2𝑐

(65) 3𝑐

1𝑐

In (65), the application of Feyman rule (3) eliminates the presence of 2-class diagrams. Applying rule (4) to the inequivalent permutations of the reference pairing (13) (24), namely {(13) (24)}, {(12) (34)}, {(14)(23)} generates the 1-class diagrams shown in (65). The corresponding Weingarten coefficients multiplying each subdiagram are determined by the relative ordering between the reference pairing 𝜏 and the pairing 𝜋 defining the subdiagram: W [𝜋, 𝜏] = W [1, 1] for 𝜋 = 𝜏, W [𝜋, 𝜏] = W [2] for 𝜋 ≠ 𝜏. The overall factor of 1 follows from the Möbius formula for a single-block partition. Restricting the analysis to first order in 1/𝑛, the expansions (30) and (31) reduce (65) to 3

= 1

1 ∑︁ 3 𝑛2 𝑗,𝑘

𝑐

𝜎 ′𝑗 𝜎 ′𝑗

𝜎𝑘′ 𝜎𝑘′

1𝑐

1 ∑︁ 2 − 3 𝑛 𝑗,𝑘

𝑐

1𝑐

2𝑐

− 4𝑐

Θ 𝜎 ′𝑗 𝜎 ′𝑗

𝜎𝑘′ 𝜎𝑘′

3𝑐 4𝑐

1 ∑︁ 2 𝑛3 𝑗,𝑘

𝑐

Θ 𝜎 ′𝑗 𝜎 ′𝑗

Θ 𝜎𝑘′ 𝜎𝑘′

1𝑐

1 ∑︁ 2 − 3 𝑛 𝑗,𝑘

3𝑐 4𝑐

𝑐

1𝑐

𝜎 ′𝑗 𝜎 ′𝑗

Θ 𝜎𝑘′ 𝜎𝑘′

3𝑐 4𝑐

1 ∑︁ 2 − 3 𝑛 𝑗,𝑘

3𝑐

𝑐

𝜎 ′𝑗 𝜎 ′𝑗

𝜎𝑘′ 𝜎𝑘′

1𝑐

1 ∑︁ 4 − 3 𝑛 𝑗,𝑘

4𝑐

𝑐

𝜎 ′𝑗 𝜎 ′𝑗

𝜎𝑘′ 𝜎𝑘′

2𝑐

1 +O 2 𝑛

(66)

3𝑐

1𝑐

We now apply the second set of Feynman rules (6)-(9). In particular, the selection rules (a)-(f) eliminate the last four terms in (66). To cancel the prefactor 1/𝑛3 , the corresponding diagrams would have to contribute a factor of 𝑛2 , which could only arise from two disconnected bare propagators. However, such subdiagrams violate the color-conservation property of the propagator. Consequently, we are left with 𝑐 𝑐 3 3𝑐 2 2𝑐 Θ 𝜎 ′𝑗 𝜎 ′𝑗 𝜎 ′𝑗 𝜎 ′𝑗 1 ∑︁ 3 1 ∑︁ 2 1 Θ 𝜎𝑘′ 𝜎𝑘′ 𝜎𝑘′ 𝜎𝑘′ = 2 − 3 +O 2 (67) 𝑛 𝑗,𝑘 𝑛 𝑗,𝑘 𝑛 1

1𝑐

4𝑐

1𝑐

4𝑐

Applying Feynman rules (6)–(9) to the diagonal and off-diagonal neural components on the right-hand side of (67) generates the following diagrams: When 𝑖 = 𝑗, the first term yields

1 ∑︁ 3 𝑛2 𝑗

𝜎 ′𝑗 𝜎 ′𝑗

(68)

When 𝑖 ≠ 𝑗, the first term produces 1 ∑︁ 3 𝑛2 𝑗 , 𝑗 1

𝜎 ′𝑗 𝜎 ′𝑗 1

𝜎 ′𝑗 𝜎 ′𝑗

(69)

1 (ℓ) 𝑛 𝐵4

whereas the second term becomes 1 ∑︁ 2 − 3 𝑛 𝑗 ,𝑗 1

Θ 𝜎 ′𝑗 𝜎 ′𝑗 1

Θ 𝜎 ′𝑗 𝜎 ′𝑗

(70) 4

Substituting (68), (69), and (70) into (67), we obtain 3

1 ∑︁ 3 = 2 𝑛 𝑗

𝜎 ′𝑗 𝜎 ′𝑗

1 © ∑︁ 2 − 3 𝑛 𝑗 ,𝑗 1 2 1 «

1 ∑︁ 3 𝑛2 𝑗 , 𝑗 1

3 ª

4 Θ 𝜎 ′𝑗 𝜎 ′𝑗 1

Θ 𝜎 ′𝑗 𝜎 ′𝑗 2

𝜎 ′𝑗 𝜎 ′𝑗 1

𝜎 ′𝑗 𝜎 ′𝑗

®+O 1 ® 𝑛2 4 ¬

1 (ℓ) 𝑛 𝐵4

(71)

The linear 1/𝑛 scaling of the right-hand side of (71) follows from simple power counting of the associated summations. Diagrams involving a single neural index contribute a factor of 𝑛, whereas those involving two distinct neural indices contribute a factor of 𝑛2 . Combined with the explicit prefactors in (71), each term on the right-hand side yields an overall contribution of order 1/𝑛. It is straightforward to verify that the first and second lines on the RHS of (71) reproduce the corresponding lines on the RHS of (46).

Feynman rules at leading order in 1/𝑛

We present an alternative formulation of the Feynman rules introduced in Section 4, valid at leading order in 1/𝑛, which closely parallels the diagrammatic framework of [13] in the Gaussian setting. For simplicity, we restrict attention to the statistics of preactivations and the NTK. We adopt the same conventions and notation as in [13]: 24

1. Preactivations and NTKs are represented by external lines, as illustrated below. c 𝛼𝛽 ≡ 𝛽 ΔΘ 𝛼

𝑧𝛼 ≡ 𝛼

(72)

where a colored line corresponds to a single NTK label. 2. The propagator is represented by ⟨ ⟩𝐾 (ℓ) ≡

(73)

where ⟨ ⟩𝐾 (ℓ) denotes a zero-mean Gaussian expectation with covariance specified by 𝐾 (ℓ ) . The expectation value is taken over the decorations of the internal lines attached to the propagator, which satisfies the set of selection rules (a)-(f) listed in Rule (6) of Section 4. 3. Cubic vertices are defined as in [13]. Explicitly, 𝛽

d 𝑖,(ℓ)𝛼𝛽 Δ𝐺

𝛼 𝛽

c 𝑖,(ℓ+1) ΔΩ 𝛼𝛽

𝛽

𝐶𝑊 ∼ , 𝑛

𝛽

′(ℓ)

∼

′(ℓ)

𝐶𝑊 , 𝑛

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼

∼

𝐶𝑊 , 𝑛

𝛼

𝛽

′(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

𝛼 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

(ℓ)

1 ∼ , 𝑛

∼

𝐶𝑊 𝑛

(74)

𝛼 (ℓ+1)

b (ℓ+1) −⟨Ω b (ℓ+1) ⟩𝐾 (ℓ) . Lines without b (ℓ+1) = 𝜎 (ℓ ) 𝜎 (ℓ ) +𝐶𝑊 Θ (ℓ ) 𝜎 ′(ℓ ) 𝜎 ′(ℓ ) and ΔΩ c 𝑖, 𝛼𝛽 = Ω where Ω 𝑖, 𝛼 𝑖,𝛽 𝑖, 𝛼𝛽 𝑖, 𝛼𝛽 𝑖, 𝛼𝛽 𝛼𝛽 𝑖, 𝛼 𝑖,𝛽 a dot at one end are internal lines. 4. Quartic vertices are defined analogously, following [13]. Explicitly, 𝛼3

𝛼2

𝛼3

𝛼2

, 𝛼1

1 (ℓ+1) 𝑛 𝑉𝛼1 𝛼2 𝛼3 𝛼4

𝛼2

𝛼3

𝛼2

𝛼4

𝛼1

𝛼3

(ℓ+1) 1 𝑛 𝐷 𝛼1 𝛼2 𝛼3 𝛼4

𝛼4

, 𝛼1

1 (ℓ+1) 𝑛 𝐹𝛼1 𝛼3 𝛼2 𝛼4

𝛼4

𝛼2

(75)

, 𝛼1

1 (ℓ+1) 𝑛 𝐴 𝛼1 𝛼2 𝛼3 𝛼4

𝛼4

𝛼1

1 (ℓ+1) 𝑛 𝐵 𝛼1 𝛼3 𝛼2 𝛼4

𝛼4

5. The orthogonal diagram describing the 2𝑚-point cumulant for the reference pairing 𝜋 = (12) (34) · · · (2𝑚− 1 2𝑚) is obtained from the Gaussian one, using the Feynman rules (74) and (75), by summing over all reconnections of the external labels: gauss orth 𝑉2𝑚, 𝜋 = 𝑉2𝑚, 𝜋 +

𝛽𝜆

∑︁

ℓÖ (𝜆)

𝑛 𝑚−ℓ (𝜆) 𝜏 ∈ C ( 𝜋 ) 𝑗=1 𝜆⊢𝑚, 𝜆≠(1,...,1)

conn 𝑉2𝜆 𝑗

(76)

𝜆

Here: • 𝜆 = (𝜆 1 , . . . , 𝜆ℓ (𝜆) ) ⊢ 𝑚 is a partition of 𝑚, and ℓ(𝜆) is its number of parts; • C𝜆 (𝜋) denotes the set of pairings 𝜏 such that, when compared with 𝜋, the external labels split into blocks of sizes 2𝜆1 , . . . , 2𝜆ℓ (𝜆) ; conn , with 𝑉 conn given by • each block of size 2𝜆 𝑗 contributes a connected correlator 𝑉2𝜆 2 𝑗

𝛽

(ℓ)

𝛽

(ℓ)

𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼

𝐶𝑊 𝑛

𝛼

, 𝛼

(ℓ)

′(ℓ)

Θ 𝛼𝛽 𝜎𝑖, 𝛼 𝜎𝑖,𝛽

∼

𝐶𝑊 𝑛

(77)

• 𝛽𝜆 is the leading coefficient of the orthogonal Weingarten function for pairings in the class 𝜆, namely 𝛽𝜆 𝜏 ∈ C𝜆 (𝜋). (78) W [𝜏, 𝜋] = 𝑚−ℓ (𝜆) + 𝑂 𝑛 − (𝑚−ℓ (𝜆)+1) , 𝑛 Its general expression is given by 𝛽𝜆 =

ℓ Ö

(−1) 𝜆𝑖 −1 𝐶𝜆𝑖 −1

(79)

𝑖=1

1 2𝑘 where 𝐶 𝑘 is the 𝑘-th Catalan number, 𝐶 𝑘 = 𝑘+1 𝑘 . For low orders one finds 𝛽 (2) = −1, 𝛽 (2,1) = −1,

𝛽 (3) = 2,

𝛽 (2,1,1) = −1,

𝛽 (2,2) = 1,

𝛽 (3,1) = 2,

𝛽 (4) = −5.

(80)

We next apply these rules to derive the recursion relations for the tensors 𝐹 and 𝐵, recovering the expressions previously obtained from the general Feynman rules of Section 4, namely (24) and (71). The layer-(ℓ+1) tensor on the LHS of (13) is represented by the third quartic vertex in (75). We enumerate all Feynman diagrams compatible with these external lines at order 1/𝑛. Since the fifth Feynman rule mixes pairing orderings, the analysis splits into two cases. For the ordering 1324, the Gaussian contraction scheme applies: the only admissible cubic vertex connecting a dotted (NTK) line with a solid (preactivation) line is the third vertex in (74). This introduces two internal solid-dashed lines weighted by 𝜎𝑖(ℓ ) 𝜎 ′ 𝑖(ℓ ) , which may carry either identical or distinct channel indices. In the former case, the lines are connected by a single propagator; in the latter, by two propagators and an internal 𝐹-tensor at layer ℓ. For all other orderings, the fifth Feynman rule applies and requires subtracting the contributions associated with the orderings 1234 and 1423. The pairings (12) (34) and (14) (23), when composed with the reference pairing (13)(24), yield the partition 𝜆 = (2), with length ℓ(𝜆) = 1. Consequently, these contributions scale as 1/𝑛, with coefficient 𝛽 (2) = −1. Each pairing contributes a factor 𝑉2 , leading to a product of two such vertices. By the selection rules (a)-(f) of Section 4, the (14) (23) term vanishes at order 1/𝑛 due to violation of color-line preservation. The only non-vanishing contribution therefore arises from (12) (34), yielding a product of two cubic vertices: one involving NTK lines and one involving two preactivation lines, as specified in (77). The diagrams corresponding to (13) are thus given by 3

∑︁ 3 =

𝜎 𝑗 𝜎 ′𝑗

−

𝑗

∑︁ 3 𝑗1 , 𝑗2

𝜎 𝑗1 𝜎 ′𝑗

𝜙 𝑗1 𝜙 𝑗2

1 © ∑︁ 2 𝑛 𝑗 ,𝑗 1 2 1 «

𝜎 𝑗2 𝜎 ′𝑗

𝜎 ′𝑗 𝜎 ′𝑗 2

3 ª

® ® 4 ¬

1 (ℓ) 𝑛 𝐹4

𝜎 𝑗1 𝜎 𝑗1

(81) 4

The sums over channel indices are included in (81). These are the only diagrams contributing at order 1/𝑛; all others are either higher order or excluded by the selection rules. The recursion relation for the tensor 𝐵 at order 1/𝑛 follows analogously. In the Gaussian sector, ordering 1324, the only compatible cubic vertex is the fifth vertex in (74), yielding two diagrams: one with identical channel indices connected by a single propagator, and one with distinct indices connected by 26

two propagators and an internal 𝐵-tensor at layer ℓ. For the remaining orderings, the 1423 contribution vanishes by color-line preservation. The 1234 ordering, constructed from the cubic vertex in (77), instead yields a nonvanishing contribution. The associated prefactor is determined by the cycle structure of the pairing (12)(34) relative to (13) (24), namely 𝜆(2) with 𝛽 (2) = −1. We therefore obtain 3

∑︁ 3 =

𝜎 ′𝑗 𝜎 ′𝑗

𝑗

−

1 © ∑︁ 2 𝑛 𝑗 ,𝑗 1 2 1 «

𝜎 ′𝑗 𝜎 ′𝑗 1

∑︁ 3 𝑗1 , 𝑗2

𝜎 𝑗2 𝜎 ′𝑗

1 (ℓ) 𝑛 𝐵4

𝜎 ′𝑗 𝜎 ′𝑗

𝜎 𝑗1 𝜎 ′𝑗

3 ª

(82)

® ® 4 ¬

The sums over channel indices are included in (82). These exhaust all contributions at order 1/𝑛; all remaining diagrams are either higher order or ruled out by the selection rules. We have verified that the leading-order Feynman rules (1)-(5) remain valid at order 1/𝑛2 , as shown in Appendix G. A useful application of the leading-order 1/𝑛 Feynman rules (1)-(5) is an alternative proof of Theorem 5.1. At this order, orthogonality enters only through the introduction of lower-rank tensors relative to the tensor under consideration, as prescribed by rule (5). Consequently, criticality can be determined by analyzing the first term in (76), corresponding to the Gaussian-like sector. In this way, network stability follows as in the Gaussian case. In particular, if the infinite-width NNGP and NTK are at criticality, then the orthogonal network is also at criticality.

Proof of main theorems

E.1

Proof of Theorem 4.1

Theorem 4.1. The Feynman rules stated in items 1-9, uniquely determine the recursion relations governing the layerwise evolution of the orthogonal NTK tensors 𝐷, 𝐹, 𝐴, 𝐵 at order 1/𝑛. This statement was already proved for the tensors 𝐹 and 𝐵 in Appendix C. To show that the Feynman rules of Appendix B also determine the recursion relations for the tensors 𝐷 and 𝐴 at order 1/𝑛, it suffices to argue that any corrections arising from orthogonality are at least of order 1/𝑛2 , as can be inferred from (34) and (45). We now verify this case by case. E.1.1

Joint NTK-preactivation cumulant

Proof. Orthogonality enters only through Feynman rule (4), which introduces the Weingarten function W [2] for pairings different from the reference pairing. For the tensor 𝐷, the reference pairing is 𝜏 = (12)(34). The diagrams associated with pairings inequivalent to 𝜏 are therefore given by W [2]

∑︁ 3𝑐 𝑗,𝑘

1𝑐

𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

2𝑐

+ W [2]

∑︁ 4𝑐 𝑗,𝑘

4𝑐

1𝑐

𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

2𝑐

(83) 3𝑐

Since the leading term of W [2] scales as 1/𝑛3 , these contributions can survive at order 1/𝑛 only if the corresponding diagrams produce factors of 𝑛2 . This would occur if the square propagator decomposed 27

into two bare propagators with distinct neural indices 𝑗 and 𝑘. However, such configurations violate color conservation in the propagator, as specified by the selection rules (a)–(f). Consequently, these diagrams vanish at order 1/𝑛, implying that the recursion relation for the orthogonal tensor 𝐷 coincides with its Gaussian counterpart at order 1/𝑛. E.1.2

NTK variance

Proof. Similarly to (E.1.1), orthogonality induces the following diagrams for the tensor 𝐴 with reference pairing (12)(34): W [2]

∑︁ 3𝑐 𝑗,𝑘

1𝑐

𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

2𝑐

+ W [2]

∑︁ 4𝑐 𝑗,𝑘

4𝑐

1𝑐

𝜎 𝑗 𝜎 ′𝑗

𝜎𝑘 𝜎𝑘′

2𝑐

(84) 3𝑐

Once again, the leading scaling W [2] ∼ 1/𝑛3 requires the diagrams to contribute a factor 𝑛2 . This would occur if the square propagator decomposed into two bare propagators with distinct neural indices 𝑗 and 𝑘. However, such configurations violate color conservation in the propagators and therefore vanish at order 1/𝑛. Consequently, the recursion relation for the orthogonal tensor 𝐴 coincides with its Gaussian counterpart at order 1/𝑛.

E.2

Proof of Theorem 4.2

Theorem 4.2. The set of Feynman rules presented above, in conjunction with those defined in Appendix B, reproduces the recursion relations governing the orthogonal dNTK and ddNTK tensors: 𝑃, 𝑄 and 𝑅, 𝑆, 𝑇, 𝑈, respectively, at order 1/𝑛. Proof. We prove this statement by showing that the recursion relations for the dNTK and ddNTK tensors coincide with those derived in the Gaussian case. As discussed in the main text, this result is further supported by the numerical simulations presented in Section 5.2 for the single-input setting. To see this, note that orthogonality enters through Feynman rule (4), which introduces the Weingarten function 𝑊 [2] multiplying the 1-class diagrams obtained from inequivalent permutations with respect to the reference pairing. Since the leading scaling of 𝑊 [2] is 1/𝑛3 , such contributions could survive at order 1/𝑛 only if the square propagator decomposed into two bare propagators, so that the corresponding Gaussian expectation values with distinct neural indices produce a factor of 𝑛2 . However, for the preactivation-dNTK cumulant and the mean of the dNTK this configuration is forbidden, as it violates color conservation in the bare propagators. Indeed, since the dNTK and ddNTKs are represented by colored lines as in (47), and the cubic vertices involving these lines always include an internal line carrying at least one color (48), no bare propagator can attach to such vertices. Consequently, the diagrams proportional to W [2] in the recursion relations of the dNTK and ddNTK tensors decouple at order 1/𝑛. Therefore, the recursion relations for the orthogonal dNTK and ddNTK tensors coincide with those obtained in the Gaussian setup.

E.3

Proof of Theorem 4.3

Theorem 4.3. The Feynman rules defined above, augmented by the higher-order generalizations of the tensors 𝑉, 𝐷, 𝐹, 𝐴, 𝐵, 𝑃, 𝑄, 𝑅, 𝑆, 𝑇, 𝑈, provide a complete characterization of the statistics of the orthogonal NTK and its descendants at arbitrary order in 1/𝑛.

Proof. We begin by observing that the restriction to cubic vertices with two external legs and one internal line in our Feynman rules follows from two structural properties of orthogonal weights: the vanishing of all odd moments, and the fact that weight contractions always occur in pairs of neural indices. The square propagator introduced in Section 4 generates all possible pairings of internal lines. This is equivalent to summing over all connected and disconnected diagrams with an even number of external legs. In parallel, the permutation of external labels carrying orthogonal charge, weighted by the corresponding Weingarten functions, implements the orthogonal contractions of the weight parameters appearing in the definition of the observables. The signs associated with each diagram match those prescribed by the Möbius relation (Rule (5)), thereby reproducing the standard combinatorial structure of cumulants. Concretely, applying this procedure to 𝑛 preactivations with a fixed pairing structure generates all evenpoint diagrams, each weighted by the appropriate Weingarten function. Since these weights encode orthogonal contractions, the square propagators can be identified with expectation values over the weights multiplying the corresponding activations. This allows one to rewrite the full expression as products of Í parameter expectation values of preactivations 𝑧 𝛼,𝑖 = 𝑗 𝑊𝑖 𝑗 𝜎𝛼, 𝑗 , each weighted by Möbius factors determined by the associated set partition. This coincides with the standard definition of the cumulant of 𝑛 preactivations. As an illustration, consider six preactivations with reference pairing (12) (34) (56). Following the discussion above, the square propagator generates all possible even-point connected and disconnected diagrams, weighted by the corresponding Weingarten functions W [1, 1, 1], W [2, 1], W [3], W [2], W [1, 1], W [1] (see Appendix G.2 for details). These weights and their associated index contractions coincide with those obtained from the orthogonality of the parameters, allowing one to rewrite the resulting contributions as products of expectation values such as E[𝑧1 𝑧 2 𝑧 3 𝑧 4 𝑧 5 𝑧 6 ]

E[𝑧1 𝑧 2 𝑧3 𝑧 4 ]E[𝑧5 𝑧6 ]

E[𝑧3 𝑧 4 𝑧 5 𝑧 6 ]E[𝑧1 𝑧 2 ]

E[𝑧1 𝑧 2 𝑧 5 𝑧 6 ]E[𝑧3 𝑧 4 ]

E[𝑧1 𝑧 2 ]E[𝑧3 𝑧 4 ]E[𝑧5 𝑧 6 ]

, (85)

Upon attaching the corresponding Möbius prefactors and summing over all partitions, one recovers the contribution of the channel (12) (34) (56) to the standard sextic cumulant. The use of the multileg definitions (47), together with the Feynman rules with coloured lines in (48), allows one to straightforwardly extend this argument to composite observables such as the NTK, dNTK, and ddNTK, which are multilinear in the weights. The second set of Feynman rules implements an effective field theory expansion for non-Gaussian probability distributions. In this framework, the square propagator (full expectation value) decomposes into a Gaussian (free) propagator and interaction vertices arising from the 1/𝑛-expansion of the distribution. As shown in prior work [13], this expansion is governed by a unique cubic vertex structure obtained by Gaussianizing the argument of the square propagator, together with higher-order interaction vertices. In particular, no new cubic vertices appear at higher orders, highlighting the uniqueness of our Feynman rules. Combining these observations, Rules (1)-(5) reproduce the cumulant structure with orthogonal contractions, while Rules (6)-(9) generate the 1/𝑛-expansion of the expectation values entering the cumulant in terms of Gaussian propagators and higher-order interaction vertices. This completes the proof.

Analytical recursion relations: Single-input case

In this appendix, we list the recursion relations obtained from the Feynman diagram approach in the single-input setting for the NTK, dNTK, and ddNTK tensors. The expressions for the tensors 𝐷, 𝐹, 𝐴 and 𝐵 follow directly from (34), (24), (45), and (46), respectively. The relations for the tensors 𝑃, 𝑄, 𝑅, 𝑆, 𝑇, 29

𝑈 can be derived analogously through the systematic application of the Feynman rules in Appendix B. Since these relations coincide with their Gaussian counterparts at order 1/𝑛, one may also use the results of [11]. Explicitly, we obtain

! (ℓ+1) h 𝜆𝑊

𝐷 (ℓ+1) = 𝜒⊥(ℓ ) 𝜒 ∥(ℓ ) 𝐷 (ℓ ) +

𝐶𝑊

2 𝐶𝑊 ⟨𝜎(𝑧)𝜎(𝑧)𝜎(𝑧)𝜎(𝑧)⟩𝐾 (ℓ) − 𝐶𝑊 𝑔 (ℓ )

i 2 + 𝜒 ∥(ℓ ) 𝑉 (ℓ )

i 2 + Θ (ℓ ) 𝐶𝑊 ⟨𝜎(𝑧)𝜎(𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) − 𝐶𝑊 𝑔 (ℓ ) 𝜒⊥(ℓ ) + 2ℎ (ℓ ) 𝜒 ∥(ℓ ) 𝑉 (ℓ ) ,

(86)

2 2 𝐹 (ℓ+1) = 𝜒 ∥(ℓ ) 𝐹 (ℓ ) + 𝐶𝑊 ⟨𝜎(𝑧)𝜎(𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) − ⟨𝜎(𝑧)𝜎(𝑧)⟩𝐾 (ℓ) ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) Θ (ℓ ) , (87)

(ℓ ) 2

𝐴 (ℓ+1) = 𝜒⊥

𝐴 (ℓ ) +

(ℓ+1) 𝜆𝑊

𝐶𝑊 ! (ℓ+1)

+2 +2 + Θ

𝜆𝑊

𝐶𝑊 (ℓ+1) 𝜆𝑊

𝐶𝑊 h (ℓ ) 2

(ℓ )

2 𝐶𝑊 ⟨𝜎(𝑧)𝜎(𝑧)𝜎(𝑧)𝜎(𝑧)⟩𝐾 (ℓ) − 𝐶𝑊 𝑔 (ℓ )

i 2 + 𝜒 ∥(ℓ ) 𝑉 (ℓ )

2 𝐶𝑊 ⟨𝜎(𝑧)𝜎(𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) − 𝐶𝑊 𝑔 (ℓ ) 𝜒⊥(ℓ ) + 2ℎ (ℓ ) 𝜒 ∥(ℓ ) 𝑉 (ℓ )

! 𝜒⊥(ℓ ) 𝜒 ∥(ℓ ) 𝐷 (ℓ ) + 4ℎ (ℓ ) 𝜒⊥(ℓ ) Θ (ℓ ) 𝐷 (ℓ )

2 𝐶𝑊 ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) − 𝜒⊥(ℓ )

i 2 + 2ℎ (ℓ ) 𝑉 (ℓ ) ,

(88)

2 2 2 ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) − ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) Θ (ℓ ) , 𝐵 (ℓ+1) = 𝜒⊥(ℓ ) 𝐵 (ℓ ) + 𝐶𝑊 (89) 2 𝑃 (ℓ+1) = 𝐶𝑊 𝜎 ′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎(𝑧) 𝐾 (ℓ) (Θ (ℓ ) ) 2 + 𝐶𝑊 𝜒⊥(ℓ ) 𝜎 ′′ (𝑧)𝜎(𝑧) 𝐾 (ℓ) 𝐵 (ℓ ) h 2i + 𝐶𝑊 𝜒⊥(ℓ ) 𝜎 ′′ (𝑧)𝜎(𝑧) 𝐾 (ℓ) + 𝜒⊥(ℓ ) 𝑃 (ℓ ) ,

𝑄

(ℓ+1)

2 = 𝐶𝑊

′′

′

𝜎 (𝑧)𝜎 (𝑧)𝜎 (𝑧)𝜎(𝑧) 𝐾 (ℓ) (Θ

(ℓ ) 2

) +

(ℓ+1) 𝜆𝑊

𝐶𝑊

(90)

𝐹 (ℓ+1) + 2ℎ (ℓ ) 𝜒 ∥(ℓ ) Θ (ℓ ) 𝐹 (ℓ )

h 2i + 𝐶𝑊 𝜒⊥(ℓ ) 𝜎 ′′ (𝑧)𝜎(𝑧) 𝐾 (ℓ) + 𝜒⊥(ℓ ) 𝑄 (ℓ ) 2 (ℓ+1) 𝑅 (ℓ+1) = 𝜒⊥(ℓ ) 𝑅 (ℓ ) + 𝜆 𝑊 𝐶𝑊 ⟨𝜎 ′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎(𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 2 2 + 𝐶𝑊 ⟨𝜎 ′′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 3 (ℓ+1) + 𝜒⊥(ℓ ) 𝜆𝑊 ⟨𝜎 ′′ (𝑧)𝜎(𝑧)⟩𝐾 (ℓ) + 𝐶𝑊 Θ (ℓ ) ⟨𝜎 ′′′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) 𝐵 (ℓ ) + 𝑃 (ℓ ) (ℓ+1) + 𝜒⊥(ℓ ) 𝜆𝑊 ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) + 𝐶𝑊 Θ (ℓ ) ⟨𝜎 ′′ (𝑧)𝜎 ′′ (𝑧)⟩𝐾 (ℓ) 𝑃 (ℓ ) ,

(91)

(92)

2 (ℓ+1) 𝑆 (ℓ+1) = 𝜒⊥(ℓ ) 𝑆 (ℓ ) + 𝜆 𝑊 𝐶𝑊 ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 2 2 + 𝐶𝑊 ⟨𝜎 ′′ (𝑧)𝜎 ′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 3 h i (ℓ+1) + 𝜒⊥(ℓ ) 𝜆𝑊 ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) + 𝐶𝑊 Θ (ℓ ) ⟨𝜎 ′′ (𝑧)𝜎 ′′ (𝑧)⟩𝐾 (ℓ) 𝐵 (ℓ ) ,

(93)

2 (ℓ+1) 𝑇 (ℓ+1) = 𝜒⊥(ℓ ) 𝑇 (ℓ ) + 2𝐶𝑊 𝜆𝑊 ⟨𝜎 ′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎(𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 2 2 + 𝐶𝑊 ⟨𝜎 ′′ (𝑧)𝜎 ′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 3 (ℓ+1) 2 (ℓ ) + (𝜆 𝑊 ) Θ ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)𝜎(𝑧)𝜎(𝑧)⟩𝐾 (ℓ)

2 𝐹 (ℓ ) (ℓ+1) + 𝜆𝑊 ⟨𝑧𝜎 ′ (𝑧)𝜎(𝑧)⟩𝐾 (ℓ) + 𝐶𝑊 Θ (ℓ ) ⟨𝑧𝜎 ′′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) (𝐾 (ℓ ) ) 2 (ℓ+1) + 2𝜒⊥(ℓ ) 𝜆𝑊 ⟨𝜎 ′′ (𝑧)𝜎(𝑧)⟩𝐾 (ℓ) + ⟨𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ)

′′′

′

′′

𝑄 (ℓ ) ,

(94)

2 2 𝑈 (ℓ+1) = 𝜒⊥(ℓ ) 𝑈 (ℓ ) + 𝐶𝑊 ⟨𝜎 ′′ (𝑧)𝜎 ′′ (𝑧)𝜎 ′ (𝑧)𝜎 ′ (𝑧)⟩𝐾 (ℓ) (Θ (ℓ ) ) 3 ,

(95)

+ 𝐶𝑊 Θ

(ℓ )

⟨𝜎 (𝑧)𝜎 (𝑧)⟩𝐾 (ℓ) + ⟨𝜎 (𝑧)𝜎 (𝑧)⟩𝐾 (ℓ)

where the susceptibilities 𝜒 |(ℓ| ) , 𝜒⊥(ℓ ) , and the auxiliary functions 𝑔 (ℓ ) (𝐾), ℎ (ℓ ) (𝐾) are defined as follows: 𝜒 |(ℓ| ) = ℎ (ℓ ) (𝐾) =

F.1

𝐶𝑊 (ℓ ) (ℓ ) ⟨𝑧 𝜎 (𝑧)𝜎 ′(ℓ ) (𝑧)⟩𝐾 (ℓ) , 𝐾 (ℓ )

𝐶𝑊 ⟨((𝑧 (ℓ ) ) 2 − 𝐾 (ℓ ) ) (𝜎 ′(ℓ ) (𝑧)𝜎 ′(ℓ ) (𝑧))⟩𝐾 (ℓ) , 4(𝐾 (ℓ ) ) 2

𝜒⊥(ℓ ) = 𝐶𝑊 ⟨𝜎 ′(ℓ ) (𝑧)𝜎 ′(ℓ ) (𝑧)⟩𝐾 (ℓ) 𝑔 (ℓ ) (𝐾) = ⟨𝜎 (ℓ ) 𝜎 (ℓ ) ⟩𝐾 (ℓ)

(96)

Solution to the single-input recursions

In this subsection, we analyze the solutions of the recursions (86)–(95), shown in Figure 9. As in Section 5.2, we consider a square neural network with tanh activation, width 𝑛 = 50, and depth 𝐿 = 10, and perform both numerical and symbolic computations in Mathematica. The input vector 𝑥 0 ∈ R50 is taken to be 0.934738 0.377754 0.254697 0.256718 0.178558 © ª © ª © ª © ª © ª 0.26696 ® 0.127474 ® 0.560475 ® 0.869761 ® 0.411167 ® ® ® ® ® ® 0.784097 ® 0.879907 ® 0.508664 ® 0.0406222® 0.100846 ® ® ® ® ® ® 0.656448 ® 0.710555 ® 0.0271565® 0.431362 ® 0.220264 ® ® ® ® ® ® 0.305308 ® 0.509949 ® 0.426426 ® 0.906228 ® 0.215917 ® ® ® ® ® ® 𝑥0 = ® ⊕ 0.312682 ® ⊕ 0.457646 ® ⊕ 0.55979 ® ⊕ 0.490943 ® . 0.401958 ® ® ® ® ® 0.894594 ® 0.0854376® 0.913778 ® 0.275852 ® 0.596323 ® ® ® ® ® ® 0.0559893 ® 0.869372 ® 0.40436 ® 0.553722 ® 0.0799147® ® ® ® ® ® 0.000643274® 0.114232 ® 0.407187 ® 0.235762 ® 0.205998 ® ® ® ® ® ® 0.0274513 0.0851646 0.0644401 0.751627 0.0372218 « ¬ « ¬ « ¬ « ¬ « ¬

(97)

The normalized tensors are defined by 𝐵˜ (ℓ ) =

𝐵 (ℓ ) 2 , Θ (ℓ )

𝑅 𝑅˜ (ℓ ) =

(ℓ ) 𝐾 (ℓ )

, (ℓ ) 3

𝐴˜ (ℓ ) =

𝐴 (ℓ ) 2 , Θ (ℓ )

𝑆 𝑆˜ (ℓ ) =

(ℓ ) 𝐾 (ℓ )

, (ℓ ) 3

𝑃˜ (ℓ ) =

𝑃 (ℓ ) 2 , Θ (ℓ )

𝑇 𝑇˜ (ℓ ) =

(ℓ ) 𝐾 (ℓ )

, (ℓ ) 3

𝑄 (ℓ ) 𝑄˜ (ℓ ) = 2 , Θ (ℓ ) 𝑈 𝑈˜ (ℓ ) =

(ℓ ) 𝐾 (ℓ )

Θ (ℓ )

3 .

(98)

We report results for the normalized tensors in the following order: the NTK-mixed tensors 𝐷˜ and 𝐹˜ (Figure 4); the NTK variance tensors 𝐴˜ and 𝐵˜ (Figure 5); the dNTK tensors 𝑃˜ and 𝑄˜ (Figure 6); the

Exact Solution Large- Expansion

0.5

1.6 1.4 ()

K( )

0.4 0.3

1.2

0.2

1.0

0.1

0.8 1

Exact Solution Large- Expansion 1

Figure 3: The NNGP and NTK. Solution of the single-input recursion relations (6) and (11). We consider a tanh network of width 𝑛 = 50 and depth 𝐿 = 10 with inputs drawn from (0, 1). Blue points denote the exact solutions of (141) and (11), while orange boxes represent the large-ℓ expansions (157) and (159), evaluated at integer ℓ. The tensor magnitudes remain smaller than their Gaussian counterparts and exhibit early-layer saturation. ˜ 𝑇, ˜ and 𝑈˜ (Figure 8). The corresponding large-ℓ ddI NTK tensor 𝑅˜ (Figure 7); and the ddII NTK tensors 𝑆, expansions, derived in Appendix H, are shown for comparison. For ease of comparison, we provide a combined plot of all NTK, dNTK, and ddNTK tensors in Figure 9. The results show that all tensors attain smaller magnitudes than their Gaussian counterparts and exhibit early-layer saturation, consistent with the empirical findings of [25], which are quantitatively reproduced here.

The 𝑉6 tensor at order 1/𝑛2

In this appendix, we compute the recursion relation governing the layer evolution of the sextic vertex 𝑉6 using both algebraic and diagrammatic methods.

G.1

Algebraic derivation

The sextic vertex is defined from the six-point cumulant of the preactivations as follows E𝑐𝜃 [𝑧 𝑖(ℓ+1) . . . 𝑧𝑖(ℓ+1) ] = E 𝜃 [𝑧 𝑖(ℓ+1) . . . 𝑧𝑖(ℓ+1) ] 1 , 𝛼1 1 , 𝛼1 6 , 𝛼6 6 , 𝛼6 (ℓ+1) (ℓ+1) (ℓ+1) (ℓ+1) − E 𝜃 [𝑧 𝑖1 , 𝛼1 . . . 𝑧𝑖4 , 𝛼4 ]E 𝜃 [𝑧 𝑖5 , 𝛼5 𝑧 𝑖6 , 𝛼6 ] + 14 other (4,2) subdivisions (ℓ+1) (ℓ+1) (ℓ+1) (ℓ+1) (ℓ+1) + 2 E 𝜃 [𝑧 𝑖(ℓ+1) 𝑧 ]E [𝑧 𝑧 ]E [𝑧 𝑧 ] + 14 other (2,2,2) subdivisions 𝜃 𝑖3 , 𝛼3 𝑖4 , 𝛼4 𝜃 𝑖5 , 𝛼5 𝑖6 , 𝛼6 1 , 𝛼1 𝑖2 , 𝛼2 1 (ℓ+1) (99) = 2 𝛿𝑖1 𝑖2 𝛿𝑖3 𝑖4 𝛿𝑖5 𝑖6 𝑉 𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 + 14 other (2,2,2) subdivisions 𝑛ℓ To obtain an explicit expression for 𝑉 𝛼(ℓ+1) , we evaluate the expectation values of the orthogonal 1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 weights defining the preactivations 𝑧 𝑖, 𝛼 . In the case of four weights, we introduce the basis 𝑒 1 = (12) (34), 32

0.6

Exact Solution Large- Expansion

0.05

0.4

0.10

0.2

0.15

F( )

D( )

0.00

0.20

Exact Solution Large- Expansion

0.0 0.2

0.25 0.4

0.30 1

0.6

Figure 4: The NTK mixed tensors. Solution of the single-input recursion relations (86) and (87). We consider a tanh network of width 𝑛 = 50 and depth 𝐿 = 10 with inputs drawn from (0, 1). Blue points denote the exact solutions of (86) and (87), while orange boxes represent the large-ℓ expansions (160) and (161), evaluated at integer ℓ. The tensor magnitudes remain smaller than their Gaussian counterparts and exhibit early-layer saturation. 𝑒 2 = (13)(24), 𝑒 3 = (14) (23). The general definition of the Weingarten function, W [𝜏, 𝜋] = W [ℓ(𝜏 ◦ 𝜋)], where ◦ denotes ordinary permutation multiplication and ℓ(𝑒) denotes the cycle length of 𝑒, implies that W [𝑒 𝑖 , 𝑒 𝑗 ] = W [1, 1] if 𝑒 𝑖 = 𝑒 𝑗 , and W [𝑒 𝑖 , 𝑒 𝑗 ] = W [2] if 𝑒 𝑖 ≠ 𝑒 𝑗 . Therefore, in the basis (𝑒 1 , 𝑒 2 , 𝑒 3 ), the 𝑘 = 2 Weingarten matrix takes the form W [1, 1] W [2] W [2] © ª 𝑀 = W [2] W [1, 1] W [2] ® W [2] W [1, 1] ¬ « W [2]

(100)

For six weights, the corresponding basis consists of 15 elements, namely 𝑒˜1 = (12) (34) (56),

𝑒˜2 = (12) (35) (46),

𝑒˜3 = (12) (36) (45),

𝑒˜4 = (13) (24) (56),

𝑒˜5 = (13) (25) (46),

𝑒˜6 = (13) (26) (45),

𝑒˜7 = (14) (23) (56),

𝑒˜8 = (14) (25) (36),

𝑒˜9 = (14) (26) (35),

(101)

𝑒˜10 = (15) (23) (46), 𝑒˜11 = (15) (24) (36), 𝑒˜12 = (15) (26) (34), 𝑒˜13 = (16) (23) (45), 𝑒˜14 = (16) (24) (35), 𝑒˜15 = (16) (25) (34). In this case, the cycle lengths can take only three possible values, namely (1, 1, 1), (2, 1), and (3), depending on the structure resulting from the product of three permutations. Consequently, in the basis (101), the 𝑘 = 3 Weingarten matrix is represented by a 15 × 15 matrix whose first row reads W [1, 1, 1], W [2, 1], W [2, 1], W [2, 1], W [3], W [3], W [2, 1], W [3], W [3], W [3], W [3], W [2, 1], W [3], W [3], W [2, 1]

(102)

Using (100) and (102), we then obtain (ℓ+1) 𝑉 𝛼(ℓ+1) = 𝑉 𝛼G1(ℓ+1) 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 + Δ𝑉 𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6

where 𝑉 𝛼G1(ℓ+1) 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 is defined as 𝑛 ∑︁ 1 G (ℓ+1) (ℓ+1) 3 (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) 𝑉 𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 = (𝐶𝑊 ) W [1, 1, 1]E[𝜎𝑖,1 𝜎𝑖,2 𝜎 𝑗,3 𝜎 𝑗,4 𝜎𝑘,5 𝜎𝑘,6 ] 2 𝑛ℓ 𝑖, 𝑗,𝑘=1 33

(103)

0.0

0.1

0.5

0.3

B( )

A( )

0.2

0.4

1.0 1.5

0.5 0.6 0.7

2.0

Exact Solution Large- Expansion 1

Figure 5: The NTK variance tensors. Solution of the single-input recursion relations (88) and (89). We consider a tanh network of width 𝑛 = 50 and depth 𝐿 = 10 with inputs drawn from (0, 1). Blue points denote the exact solutions of (88) and (89), while orange boxes represent the large-ℓ expansion (162) and (163), evaluated at integer ℓ. The magnitude of the tensors remains smaller than its Gaussian counterpart and exhibits early-layer saturation.

(ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) − W [1, 1] · W [1] E[𝜎𝑖,1 𝜎𝑖,2 𝜎 𝑗,3 𝜎 𝑗,4 ]E[𝜎𝑘,5 𝜎𝑘,6 ] + E[𝜎 𝑗,3 𝜎 𝑗,4 𝜎𝑘,5 𝜎𝑘,6 ]E[𝜎𝑖,1 𝜎𝑖,2 ] (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) 3 + E[𝜎𝑖,1 𝜎𝑖,2 𝜎𝑘,5 𝜎𝑘,6 ]E[𝜎 𝑗,3 𝜎 𝑗,4 ] + 2(W [1]) E[𝜎𝑖,1 𝜎𝑖,2 ]E[𝜎 𝑗,3 𝜎 𝑗,4 ]E[𝜎𝑘,5 𝜎𝑘,6 ] (104)

and Δ𝑉 𝛼(ℓ+1) is given by 1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 𝑛 ∑︁ 1 (ℓ+1) (ℓ+1) 3 Δ𝑉 𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 = (𝐶𝑊 ) W [2, 1]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ),𝑖 𝜎𝛼(ℓ3 ), 𝑗 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ), 𝑗 𝜎𝛼(ℓ6 ),𝑘 ] 𝑛ℓ2 𝑖, 𝑗,𝑘=1 + W [2, 1]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ),𝑖 𝜎𝛼(ℓ3 ), 𝑗 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ), 𝑗 ] + W [2, 1]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑖 𝜎𝛼(ℓ4 ), 𝑗 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ),𝑘 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑖 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ), 𝑗 𝜎𝛼(ℓ6 ),𝑘 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑖 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ), 𝑗 ] + W [2, 1]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ), 𝑗 𝜎𝛼(ℓ4 ),𝑖 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ),𝑘 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑘 𝜎𝛼(ℓ4 ),𝑖 𝜎𝛼(ℓ5 ), 𝑗 𝜎𝛼(ℓ6 ),𝑘 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑘 𝜎𝛼(ℓ4 ),𝑖 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ), 𝑗 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ), 𝑗 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ),𝑖 𝜎𝛼(ℓ6 ),𝑘 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑘 𝜎𝛼(ℓ4 ), 𝑗 𝜎𝛼(ℓ5 ),𝑖 𝜎𝛼(ℓ6 ),𝑘 ] + W [2, 1]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑘 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ),𝑖 𝜎𝛼(ℓ6 ), 𝑗 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ), 𝑗 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ),𝑖 ] + W [3]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑘 𝜎𝛼(ℓ4 ), 𝑗 𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ),𝑖 ] + W [2, 1]E[𝜎𝛼(ℓ1 ),𝑖 𝜎𝛼(ℓ2 ), 𝑗 𝜎𝛼(ℓ3 ),𝑘 𝜎𝛼(ℓ4 ),𝑘 𝜎𝛼(ℓ5 ), 𝑗 𝜎𝛼(ℓ6 ),𝑖 ] (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) − W [2] · W [1] E[𝜎𝛼1 ,𝑖 𝜎𝛼2 , 𝑗 𝜎𝛼3 ,𝑖 𝜎𝛼4 , 𝑗 ] + E[𝜎𝛼1 ,𝑖 𝜎𝛼2 , 𝑗 𝜎𝛼3 , 𝑗 𝜎𝛼4 ,𝑖 ] E[𝜎𝛼(ℓ5 ),𝑘 𝜎𝛼(ℓ6 ),𝑘 ] (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) − W [2] · W [1] E[𝜎𝛼3 ,𝑖 𝜎𝛼4 , 𝑗 𝜎𝛼5 ,𝑖 𝜎𝛼6 , 𝑗 ] + E[𝜎𝛼3 ,𝑖 𝜎𝛼4 , 𝑗 𝜎𝛼5 , 𝑗 𝜎𝛼6 ,𝑖 ] E[𝜎𝛼(ℓ1 ),𝑘 𝜎𝛼(ℓ2 ),𝑘 ] (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) − W [2]W [1] E[𝜎𝛼1 ,𝑖 𝜎𝛼2 , 𝑗 𝜎𝛼5 ,𝑖 𝜎𝛼6 , 𝑗 ] + E[𝜎𝛼1 ,𝑖 𝜎𝛼2 , 𝑗 𝜎𝛼5 , 𝑗 𝜎𝛼6 ,𝑖 ] E[𝜎𝛼3 ,𝑘 𝜎𝛼4 ,𝑘 ]

(105)

The term 𝑉 𝛼G1(ℓ+1) 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 captures the contribution of the diagonal components of the Weingarten matrices and reduces to the Gaussian case in the infinite-width limit, whereas Δ𝑉 𝛼(ℓ+1) encodes the off1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 diagonal contributions. 34

3.5

Exact Solution Large- Expansion

3.0

2.5

2.0

Q( )

2.0

P( )

Exact Solution Large- Expansion

3.0

1.5

1.5 1.0

1.0 0.5

0.5

0.0

0.0 1

Figure 6: The dNTK tensors. Solutions of the single-input recursion relations (90) and 91. We consider a tanh network of width 𝑛 = 50 and depth 𝐿 = 10 with inputs drawn from (0, 1). Blue points denote the exact solution of (90) and 91, while orange boxes represent the large-ℓ expansions (164) and (165), evaluated at integer ℓ. The magnitude of the tensors remains smaller than its Gaussian counterpart and exhibits early-layer saturation. We now compute 𝑉 𝛼(ℓ+1) at order 1/𝑛2 . To this end, we expand the 𝑘 = 3 Weingarten functions 1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 as follows 1 1 6 W [1, 1, 1] = 3 + 5 + 𝑂 6 , 𝑛 𝑛 𝑛 1 1 1 W [2, 1] = − 4 + 5 + 𝑂 6 , (106) 𝑛 𝑛 𝑛 2 1 W [3] = 5 + 𝑂 6 . 𝑛 𝑛 Combining (106), (30) and (31), we obtain 1 G (ℓ+1) 1 6 (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) 𝑉 𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 = 2 𝑉 𝛼𝐺1(ℓ+1) 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 + 2 ⟨𝜎𝛼1 𝜎𝛼2 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼5 𝜎𝛼6 ⟩𝐾 (ℓ) 2 𝑛ℓ 𝑛ℓ 𝑛ℓ 3·2 − 2 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) 𝑛ℓ 1 𝐺 (ℓ+1) = 2 V𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 𝑛ℓ

(107)

The factor 6 in the first equality of (107) originates from W [1, 1, 1] in (104), while the factor 2 · 3 arises from the three contributions of the form W [1, 1] · W [1]. The tensor 𝑉 𝛼𝐺1(ℓ+1) 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 denotes the Gaussian six-point cumulant at order 𝑛12 . The corresponding recursion relation was derived explicitly in [17] and reads (ℓ+1) 3 D E 𝐶𝑊 (ℓ ) (ℓ ) (ℓ ) 1 𝐺 (ℓ+1) c c c 𝑉 = Δ𝐺 Δ𝐺 Δ𝐺 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 𝐾 (ℓ) 𝑛ℓ2 1 2 3 4 5 6 𝑛ℓ2 " ) ) (ℓ+1) 3 c (ℓ ) 𝑑 2 Δ𝐺 c (ℓ c (ℓ ∑︁ 𝑑 2 Δ𝐺 𝐶𝑊 𝛼1 𝛼2 Δ𝐺 𝛼5 𝛼6 𝛼3 𝛼4 (ℓ ) 𝑉𝛽1 𝛽2 𝛽3 𝛽4 + (ℓ ) (ℓ ) 4 𝑛ℓ 𝑛ℓ −1 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 𝐾 (ℓ) 𝑑𝑧 𝛽 𝑑𝑧 𝛽 𝐾 (ℓ) 𝛽𝑖 ∈ { 𝛼1 ,..., 𝛼6 } 3 4

0 1

R( )

2 3 4 5 6

Exact Solution Large- Expansion

7 1

Figure 7: The dI NTK tensor. Solution of the single-input recursion relation (92). We consider a tanh network of width 𝑛 = 50 and depth 𝐿 = 10 with inputs drawn from (0, 1). Blue points denote the exact solution of (92), while orange boxes represent the large-ℓ expansion (166), evaluated at integer ℓ. The magnitude of the tensor remains smaller than its Gaussian counterpart and exhibits early-layer saturation. 0.8

0.0

Exact Solution Large- Expansion

0.7

0.2

0.5

7.5

0.4

10.0

T( )

0.3

12.5

0.2

15.0

0.1

17.5

0.0

20.0 1

U( )

5.0

0.6

S( )

0.0

2.5

0.6 0.8

Exact Solution Large- Expansion 1

Exact Solution Large- Expansion

1.0 10

Figure 8: The ddII NTK tensors. Solutions to the single-input recursion relations (93), (94), and (95). We consider a tanh network with width 𝑛 = 50 and depth 𝐿 = 10, with inputs drawn from (0, 1). Blue points show the exact solutions of (93), (94), and (95), while orange boxes represent the corresponding large-ℓ expansions (167), (168), and (169), evaluated at integer ℓ. The tensor magnitude remains below its Gaussian counterpart and exhibits early-layer saturation. +

) c (ℓ c (ℓ ) 𝑑 2 Δ𝐺 𝛼1 𝛼2 Δ𝐺 𝛼3 𝛼4 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 3 ∑︁ 𝐶 (ℓ+1)

𝐾 (ℓ)

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 5 6

𝑉𝛽(ℓ1 𝛽) 2 𝛽3 𝛽4 𝛽5 𝛽6 2 8 𝑛ℓ −1 𝛽 ∈ { 𝛼 ,..., 𝛼 } 𝑖 1 6

(ℓ+1) 𝐶𝑊

(ℓ )

c𝛼𝛼 𝑑 4 Δ𝐺 1 2

) c (ℓ c (ℓ ) 𝑑 2 Δ𝐺 𝛼3 𝛼4 Δ𝐺 𝛼5 𝛼6 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 )

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 3 4

𝐾 (ℓ)

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 1 2

𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝐾 (ℓ) (ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 5 6

𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ5 ) 𝑑𝑧 𝛽(ℓ6 ) 𝐾 (ℓ) " ) 2 c (ℓ ) 2 c (ℓ ) c (ℓ 𝑑 Δ𝐺 𝛼5 𝛼6 𝑑 4 Δ𝐺 𝑑 Δ𝐺 𝛼1 𝛼2 𝛼3 𝛼4 (ℓ )

∑︁

𝑖

c𝛼𝛼 𝑑 2 Δ𝐺 1 2

𝑉 (ℓ ) 𝑉 16 𝑛ℓ2 −1 𝛽 ∈ { 𝛼 ,... 𝛼 𝛽}1 𝛽2 𝛽3 𝛽4 𝛽5 𝛽6 𝛽7 𝛽8

𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝐾 (ℓ)

𝑊

𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝑑𝑧 𝛽(ℓ7 ) 𝑑𝑧 𝛽(ℓ8 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ5 ) 𝑑𝑧 𝛽(ℓ6 ) 𝐾 (ℓ)

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 3 4

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 5 6

𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝑑𝑧 𝛽(ℓ7 ) 𝑑𝑧 𝛽(ℓ8 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ5 ) 𝑑𝑧 𝛽(ℓ6 ) 𝐾 (ℓ)

(ℓ )

c𝛼𝛼 𝑑 4 Δ𝐺 5 6

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 1 2

(ℓ )

c𝛼𝛼 𝑑 2 Δ𝐺 3 4

𝑑𝑧 𝛽(ℓ3 ) 𝑑𝑧 𝛽(ℓ4 ) 𝑑𝑧 𝛽(ℓ7 ) 𝑑𝑧 𝛽(ℓ8 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ1 ) 𝑑𝑧 𝛽(ℓ2 ) 𝐾 (ℓ) 𝑑𝑧 𝛽(ℓ5 ) 𝑑𝑧 𝛽(ℓ6 ) 𝐾 (ℓ) 36

# .

(108)

Normalized Tensors

0.1

A( ) B( ) D( ) F( ) P( ) Q( ) R( ) S( ) T( ) U( )

0.0 0.1 0.2 0.3 0.4 0.5 1

Figure 9: Orthogonal saturation. Exact solutions of the single-input recursion relations at order 1/𝑛 (see Theorems 4.1 and 4.2). We consider a tanh network of width 𝑛 = 50 with inputs drawn from the real interval (0, 1). All tensors attain smaller values than their Gaussian counterparts and exhibit early-layer saturation. These predictions are consistent with the empirical results of [25]. (ℓ )

c 𝛼𝛽 = 𝜎𝛼(ℓ ) 𝜎 (ℓ ) − ⟨𝜎𝛼(ℓ ) 𝜎 (ℓ ) ⟩𝐾 (ℓ) . where Δ𝐺 𝛽 𝛽 Likewise, the off-diagonal contribution ΔV𝛼(ℓ+1) can be systematically computed as follows: 1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6 1. In the 𝑛1 -expansion of W [2, 1], only the terms of order 𝑛15 and 𝑛14 contribute nontrivially. The former becomes relevant when all neural indices are distinct, yielding 1 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 𝑛ℓ2 +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) + ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) (109) The latter splits into two cases: • The first case arises when all neural indices are distinct: 3 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 2 𝑛ℓ +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) + ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) (110) • The second case occurs when exactly two neural indices coincide. In this case, we find two types of contributions: one that does not involve the four-point cumulant, 1 − 2 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 𝑛ℓ 37

+⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼6 𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) + ⟨𝜎𝛼2 𝜎𝛼5 𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) (111) and one that explicitly involves 𝑉4 : 1 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) − 𝑛ℓ 𝑛ℓ −1 +⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼6 |𝜎𝛼3 𝜎𝛼4 ⟩𝑉 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) + ⟨𝜎𝛼2 𝜎𝛼5 |𝜎𝛼3 𝜎𝛼4 ⟩𝑉 (ℓ) ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) (112) where we have introduced the notation ⟨𝜎𝛼 𝜎𝛽 |𝜎𝛿 𝜎𝜖 ⟩𝑉 as shorthand for * 2 (ℓ ) (ℓ ) + + * ∑︁ 𝑑 (𝜎𝛼 𝜎𝛽 ) 𝑑 2 (𝜎𝛿(ℓ ) 𝜎𝜖(ℓ ) ) 1 (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) 𝑉𝛽1 𝛽2 𝛽3 𝛽4 ⟨𝜎𝛼 𝜎𝛽 |𝜎𝛿 𝜎𝜖 ⟩𝑉 (ℓ) = 4 𝑑𝑧 (ℓ ) 𝑑𝑧 (ℓ ) 𝑑𝑧 (ℓ ) 𝑑𝑧 (ℓ ) (ℓ) 𝛽𝑖 ∈ { 𝛼,𝛽, 𝛿, 𝜖 }

𝛽1

𝛽2

𝐾

𝛽3

𝛽4

𝐾 (ℓ)

(113) 2. In the 𝑛1 -expansion of W [3], only the term of order 𝑛15 contributes. This occurs when all neural indices are distinct: 2 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 𝑛ℓ2 +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼3 ⟩𝐾 (ℓ) ⟨𝜎𝛼4 𝜎𝛼5 ⟩𝐾 (ℓ) + ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼5 ⟩𝐾 (ℓ) (114) 3. In the 𝑛1 -expansion of W [2], only the terms of order 𝑛14 and 𝑛13 contribute nontrivially. The 𝑛14 term arises when all neural indices are distinct 1 − 2 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) 𝑛ℓ 38

+⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) + ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) (115) The 𝑛13 contribution, in turn, splits into two cases: • one in which all neural indices are distinct: 1 − 2 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 𝑛ℓ +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) + ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) (116) • and one in which exactly two neural indices coincide. In the latter case, we again obtain two types of contributions: one that does not involve the four-point cumulant, 1 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) 𝑛ℓ2 +⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) +⟨𝜎𝛼1 𝜎𝛼5 𝜎𝛼2 𝜎𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) + ⟨𝜎𝛼1 𝜎𝛼6 𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼4 ⟩𝐾 (ℓ) (117) and one that explicitly involves 𝑉4 1 ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) 𝑛ℓ 𝑛ℓ −1 +⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) +⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) |𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) (118) Combining all contributions, we obtain (ℓ+1) 3 (𝐶𝑊 ) 1 (ℓ+1) Δ𝑉 𝛼1 ... 𝛼6 = − ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 2 2 𝑛ℓ 𝑛ℓ

+ ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) −

(ℓ+1) 3 (𝐶𝑊 )

𝑛ℓ 𝑛ℓ −1

⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ)

+ ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) |𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) 39

+ ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) |𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) |𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) +

(ℓ+1) 3 2(𝐶𝑊 )

𝑛ℓ2

⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ)

+ ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) (ℓ+1) 3 2(𝐶𝑊 )

⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ2 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 2 𝑛ℓ (ℓ ) (ℓ ) + ⟨𝜎𝛼1 𝜎𝛼3 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ5 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ)

+ ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ)

(119) or, more compactly (ℓ+1) 3 (𝐶𝑊 ) ) 1 (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ+1) c (ℓ ) c (ℓ ) c (ℓ ) c (ℓ Δ𝑉 𝛼1 ... 𝛼6 = − ⟨Δ𝐺 𝛼1 𝛼2 Δ𝐺 𝛼3 𝛼5 ⟩𝐾 (ℓ) ⟨𝜎𝛼4 𝜎𝛼6 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼1 𝛼2 Δ𝐺 𝛼4 𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼3 𝜎𝛼5 ⟩𝐾 (ℓ) 2 2 𝑛ℓ 𝑛ℓ (ℓ )

(ℓ )

c 𝛼 𝛼 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 c 𝛼 𝛼 Δ𝐺 c 𝛼 𝛼 Δ𝐺 c 𝛼 𝛼 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 4 3 5 6 3 6 1 2 4 5 1 2 ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ c (ℓ ) c (ℓ ) c (ℓ ) + ⟨Δ𝐺 𝛼1 𝛼3 Δ𝐺 𝛼5 𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼4 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼2 𝛼4 Δ𝐺 𝛼5 𝛼6 ⟩𝐾 (ℓ) ⟨𝜎𝛼1 𝜎𝛼3 ⟩𝐾 (ℓ) (ℓ )

(ℓ )

c 𝛼 𝛼 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) c 𝛼 𝛼 Δ𝐺 c 𝛼 𝛼 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 c 𝛼 𝛼 Δ𝐺 + ⟨Δ𝐺 1 4 2 3 2 3 1 4 5 6 5 6 c 𝛼 𝛼 Δ𝐺 c 𝛼 𝛼 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 c 𝛼 𝛼 Δ𝐺 c 𝛼 𝛼 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 2 1 6 5 1 5 3 4 2 6 3 4 ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ ) c (ℓ ) c (ℓ ) c (ℓ + ⟨Δ𝐺 𝛼1 𝛼6 Δ𝐺 𝛼3 𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼2 𝛼5 Δ𝐺 𝛼3 𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ)

(ℓ+1) 3 (𝐶𝑊 ) ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ ) c (ℓ ) c (ℓ ) c (ℓ − ⟨Δ𝐺 𝛼1 𝛼2 | Δ𝐺 𝛼3 𝛼5 ⟩𝑉 (ℓ) ⟨𝜎𝛼4 𝜎𝛼6 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼1 𝛼2 | Δ𝐺 𝛼4 𝛼6 ⟩𝑉 (ℓ) ⟨𝜎𝛼3 𝜎𝛼5 ⟩𝐾 (ℓ) 𝑛ℓ 𝑛ℓ −1 ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ c (ℓ ) c (ℓ ) c (ℓ ) + ⟨Δ𝐺 𝛼1 𝛼2 | Δ𝐺 𝛼3 𝛼6 ⟩𝑉 (ℓ) ⟨𝜎𝛼4 𝜎𝛼5 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼1 𝛼2 | Δ𝐺 𝛼4 𝛼5 ⟩𝑉 (ℓ) ⟨𝜎𝛼3 𝜎𝛼6 ⟩𝐾 (ℓ) (ℓ )

(ℓ )

c 𝛼 𝛼 | Δ𝐺 c 𝛼 𝛼 ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 c 𝛼 𝛼 | Δ𝐺 c 𝛼 𝛼 ⟩𝑉 (ℓ) ⟨𝜎𝛼(ℓ ) 𝜎𝛼(ℓ ) ⟩𝐾 (ℓ) + ⟨Δ𝐺 2 4 1 3 1 3 2 4 5 6 5 6 ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ ) c (ℓ c (ℓ ) c (ℓ ) + ⟨Δ𝐺 𝛼1 𝛼4 | Δ𝐺 𝛼5 𝛼6 ⟩𝑉 (ℓ) ⟨𝜎𝛼2 𝜎𝛼3 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼2 𝛼3 | Δ𝐺 𝛼5 𝛼6 ⟩𝑉 (ℓ) ⟨𝜎𝛼1 𝜎𝛼4 ⟩𝐾 (ℓ) ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ c (ℓ ) c (ℓ ) c (ℓ ) + ⟨Δ𝐺 𝛼1 𝛼5 | Δ𝐺 𝛼3 𝛼4 ⟩𝑉 (ℓ) ⟨𝜎𝛼2 𝜎𝛼6 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼2 𝛼6 | Δ𝐺 𝛼3 𝛼4 ⟩𝑉 (ℓ) ⟨𝜎𝛼1 𝜎𝛼5 ⟩𝐾 (ℓ) ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) c (ℓ c (ℓ ) c (ℓ ) c (ℓ ) + ⟨Δ𝐺 𝛼1 𝛼6 | Δ𝐺 𝛼3 𝛼4 ⟩𝑉 (ℓ) ⟨𝜎𝛼2 𝜎𝛼5 ⟩𝐾 (ℓ) + ⟨Δ𝐺 𝛼2 𝛼5 | Δ𝐺 𝛼3 𝛼4 ⟩𝑉 (ℓ) ⟨𝜎𝛼1 𝜎𝛼6 ⟩𝐾 (ℓ) (ℓ+1) 3 2(𝐶𝑊 )

⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) 2 𝑛ℓ (ℓ ) (ℓ ) + ⟨𝜎𝛼1 𝜎𝛼4 ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) +

+ ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ3 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ4 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ) + ⟨𝜎𝛼(ℓ1 ) 𝜎𝛼(ℓ6 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ2 ) 𝜎𝛼(ℓ4 ) ⟩𝐾 (ℓ) ⟨𝜎𝛼(ℓ3 ) 𝜎𝛼(ℓ5 ) ⟩𝐾 (ℓ)

(120)

G.2

Diagrammatric derivation

We now derive the recursion relation for the sextic vertex (120) using the Feynman rules of Appendix B. Taking the reference pairing to be (12) (34) (56), the application of rule (3) generates the 1-class diagram 3𝑐

4𝑐 𝜎𝑗 𝜎𝑗

2𝑐

5𝑐

(121)

, 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

6𝑐

1𝑐

as well as the corresponding 2-class diagrams 3𝑐

5𝑐

4𝑐

6𝑐

𝜎𝑗 𝜎𝑗

1𝑐

𝜎𝑗 𝜎𝑗

5𝑐

2𝑐

𝜎𝑗 𝜎𝑗

3𝑐

2𝑐

𝜎𝑘 𝜎𝑘

𝜎𝑖 𝜎𝑖

6𝑐

1𝑐

5𝑐

4𝑐

, 𝜎𝑖 𝜎𝑖

2𝑐

𝜎𝑘 𝜎𝑘

1𝑐

𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

3𝑐

4𝑐

(122)

6𝑐

and the 3-class diagram 5𝑐

6𝑐 𝜎𝑘 𝜎𝑘

3𝑐

2𝑐

(123)

. 𝜎𝑖 𝜎𝑖

𝜎𝑗 𝜎𝑗

4𝑐

1𝑐

The application of rule (4) introduces additional diagrams, each accompanied by a Weingarten function determined by the cycle structure resulting from the composition of the corresponding pairing with the reference pairing. As discussed in the previous subsection, for 1-class diagrams there are three possible cycle types, namely (1, 1, 1), (2, 1), and (3). The fifteen 3-class diagrams can thus be organized as follows: one diagram of type (1, 1, 1) 3𝑐

4𝑐 𝜎𝑗 𝜎𝑗

2𝑐

W [1, 1, 1]

5𝑐

(124)

, 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

6𝑐

1𝑐

six diagrams of type (2, 1) 3𝑐

𝜎𝑗 𝜎𝑗

2𝑐

W [2, 1]

5𝑐

3𝑐 4𝑐

1𝑐

𝜎𝑗 𝜎𝑗

2𝑐

W [2, 1]

, 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

6𝑐

4𝑐

, 𝜎𝑖 𝜎𝑖

1𝑐

6𝑐

𝜎𝑘 𝜎𝑘

5𝑐

2𝑐

𝜎𝑗 𝜎𝑗

3𝑐

W [2, 1]

4𝑐 5𝑐

W [2, 1]

𝜎𝑘 𝜎𝑘

6𝑐

1𝑐 2𝑐

W [2, 1]

6𝑐 5𝑐

2𝑐 3𝑐

𝜎𝑗 𝜎𝑗

6𝑐

W [2, 1]

𝜎𝑘 𝜎𝑘

1𝑐

𝜎𝑘 𝜎𝑘

1𝑐

, 𝜎𝑖 𝜎𝑖

5𝑐

, 𝜎𝑖 𝜎𝑖

6𝑐 𝜎𝑗 𝜎𝑗

5𝑐

𝜎𝑗 𝜎𝑗

4𝑐

, 𝜎𝑖 𝜎𝑖

3𝑐

2𝑐

4𝑐

3𝑐

(125)

, 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

4𝑐

1𝑐

and eight diagrams of type (3) 2𝑐

5𝑐 𝜎𝑗 𝜎𝑗

3𝑐

4𝑐 𝜎𝑗 𝜎𝑗

6𝑐

4𝑐

3𝑐

W [3]

, 5𝑐

(126)

. 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

1𝑐

6𝑐 2𝑐

W [3] 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

1𝑐

3𝑐 𝜎𝑗 𝜎𝑗

3𝑐

, 𝜎𝑖 𝜎𝑖

6𝑐

𝜎𝑗 𝜎𝑗

W [3]

2𝑐

4𝑐

5𝑐

4𝑐

𝜎𝑘 𝜎𝑘

1𝑐

5𝑐 2𝑐

W [3] 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

1𝑐

3𝑐 𝜎𝑗 𝜎𝑗

3𝑐

, 𝜎𝑖 𝜎𝑖

6𝑐

5𝑐

𝜎𝑗 𝜎𝑗

W [3]

2𝑐

6𝑐

4𝑐

𝜎𝑘 𝜎𝑘

1𝑐

5𝑐 2𝑐

W [3] 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

1𝑐

5𝑐 𝜎𝑗 𝜎𝑗

4𝑐

, 𝜎𝑖 𝜎𝑖

6𝑐 2𝑐

4𝑐

W [3]

, 𝜎𝑘 𝜎𝑘

1𝑐

𝜎𝑗 𝜎𝑗

3𝑐

4𝑐

W [3] 𝜎𝑖 𝜎𝑖

6𝑐

2𝑐

𝜎𝑘 𝜎𝑘

5𝑐

1𝑐

Similarly, the nine 2-class diagrams generated by rule (4) can be classified according to the associated 𝑘 = 2 Weingarten factor. Explicitly, three diagrams are weighted by W [1, 1] 3𝑐

5𝑐

4𝑐 𝜎𝑗 𝜎𝑗

W [1, 1] · W [1]

𝜎𝑗 𝜎𝑗

5𝑐

2𝑐

1𝑐

3𝑐

2𝑐

W [1, 1] · W [1]

, 𝜎𝑖 𝜎𝑖

6𝑐

𝜎𝑘 𝜎𝑘

6𝑐

, 𝜎𝑖 𝜎𝑖

1𝑐

𝜎𝑘 𝜎𝑘

4𝑐

1𝑐

2𝑐 𝜎𝑗 𝜎𝑗

5𝑐

4𝑐

W [1, 1] · W [1]

(127)

, 𝜎𝑖 𝜎𝑖

𝜎𝑘 𝜎𝑘

3𝑐

6𝑐

and six diagrams are weighted by W [2] 3𝑐

3𝑐

4𝑐 𝜎𝑗 𝜎𝑗

𝜎𝑗 𝜎𝑗

5𝑐

6𝑐

2𝑐

W [2] · W [1]

, 𝜎𝑖 𝜎𝑖

2𝑐

W [2] · W [1]

𝜎𝑘 𝜎𝑘

𝜎𝑖 𝜎𝑖

6𝑐

1𝑐 5𝑐

6𝑐

5𝑐 5𝑐

2𝑐

W [2] · W [1]

4𝑐

2𝑐

W [2] · W [1]

𝜎𝑘 𝜎𝑘

𝜎𝑖 𝜎𝑖

1𝑐

4𝑐

𝜎𝑘 𝜎𝑘

3𝑐

1𝑐

2𝑐

1𝑐

𝜎𝑗 𝜎𝑗

2𝑐 𝜎𝑗 𝜎𝑗

5𝑐

6𝑐

4𝑐

W [2] · W [1]

, 𝜎𝑖 𝜎𝑖

6𝑐 𝜎𝑗 𝜎𝑗

3𝑐

1𝑐

𝜎𝑘 𝜎𝑘

1𝑐

𝜎𝑗 𝜎𝑗

𝜎𝑖 𝜎𝑖

4𝑐

W [2] · W [1]

𝜎𝑘 𝜎𝑘

𝜎𝑖 𝜎𝑖

3𝑐

6𝑐

𝜎𝑘 𝜎𝑘

(128)

5𝑐

The single 3-class diagram contributes as 5𝑐

6𝑐 𝜎𝑘 𝜎𝑘

3𝑐

2𝑐

(W [1])

3 𝜎𝑖 𝜎𝑖

(129)

𝜎𝑗 𝜎𝑗

1𝑐

4𝑐

Möbius factors are incorporated through rule (5): 1-class diagrams are multiplied by 1, 2-class diagrams by −1, and 3-class diagrams by 2. After summing over all neural indices and all diagrams described above, one recovers (103), with 𝑉 G given by (104) and Δ𝑉 defined in (105). We now apply the Feynman rules (6)–(9) to the square propagator. In particular, we focus on the layer evolution of the sextic vertex at order 1/𝑛2 . To this end, the Weingarten functions are expanded as in (30), 43

(31), and (106). The bare propagators, together with the corresponding quartic and sextic vertices, are then introduced in accordance with the selection rules (a)–(f). After a careful, albeit straightforward, implementation of these rules, we arrive at the following diagrammatic decomposition of the tensor 𝑉6(ℓ+1) : 3

4 d𝑗 Δ𝐺

2 1 ∑︁

1 (ℓ+1) 𝑉 = 𝑛ℓ2 123456 𝑛ℓ3

𝑗

d𝑗 Δ𝐺

𝜙 𝑗1 𝜙 𝑗2

d𝑗 Δ𝐺 3

𝜙 𝑗2 𝜙 𝑗3

ª ® ® ® ® ® ® ® ® 5 ® + 𝑃(12, 34, 56) ®® ® ® 6 ® ® ® ® ® ® ® ¬

d𝑗 Δ𝐺 2 𝜙 𝑗2 𝜙 𝑗1 d𝑗 Δ𝐺 1

d𝑗 Δ𝐺 1

ª ® ® 5 ® ® + 𝑃(12, 34, 56) ® ® ® 6 ® ® ¬

d𝑗 Δ𝐺 2 d𝑗 Δ𝐺 1

4 d𝑗 Δ𝐺 2

2 1 ∑︁ 𝑛ℓ3 𝑗1 , 𝑗2 , 𝑗3 1

𝜙 𝑗2 d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

6 1

+ d𝑗 Δ𝐺 2

d𝑗 Δ𝐺 2

𝑗1 , 𝑗2

+ d𝑗 Δ𝐺 2

𝑗1 , 𝑗2

d𝑗 Δ𝐺 2

𝜎 𝑗1 𝜎 𝑗1

3 d𝑗 Δ𝐺 2

d𝑗 Δ𝐺 2

∑︁ 6

𝜎 𝑗1 𝜎 𝑗1

𝑗1 , 𝑗2

𝜎 𝑗1 𝜎 𝑗1

∑︁ 3

∑︁ 5

𝜎 𝑗1 𝜎 𝑗1

𝑗1 , 𝑗2

+ d𝑗 Δ𝐺 2

d𝑗 Δ𝐺 2

4 𝜎 𝑗1 𝜎 𝑗1

∑︁ 4

d𝑗 Δ𝐺 2

∑︁ 3 𝑗1 , 𝑗2

5 d𝑗 Δ𝐺 2

d𝑗 Δ𝐺 2

𝜎 𝑗1 𝜎 𝑗1

𝑗1 , 𝑗2

𝜎 𝑗1 𝜎 𝑗1

∑︁ 6 d𝑗 Δ𝐺 2

d𝑗 Δ𝐺 2

𝑗1 , 𝑗2

d𝑗 Δ𝐺 2

𝜎 𝑗1 𝜎 𝑗1

∑︁ 2 𝑗1 , 𝑗2

d𝑗 Δ𝐺 2

𝑗1 , 𝑗2

𝑗1 , 𝑗2 , 𝑗3

d𝑗 Δ𝐺 2

𝑗1 , 𝑗2

+ 2

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

∑︁

∑︁ 𝑗1 , 𝑗2 , 𝑗3

𝑗1 , 𝑗2 , 𝑗3

d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

∑︁

∑︁ 𝑗1 , 𝑗2 , 𝑗3

𝑗1 , 𝑗2 , 𝑗3

d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

∑︁ 𝑗1 , 𝑗2 , 𝑗3

5 d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

𝑗1 , 𝑗2 , 𝑗3

+ 1

d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

∑︁

𝑗1 , 𝑗2 , 𝑗3

d𝑗 Δ𝐺 1

𝑗1 , 𝑗2 , 𝑗3

4 d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

6 𝜎 𝑗2 𝜎 𝑗2

+ 1

𝜙 𝑗1 𝜙 𝑗3

𝜎 𝑗2 𝜎 𝑗2

d𝑗 Δ𝐺 3

𝜎 𝑗2 𝜎 𝑗2

∑︁

d𝑗 Δ𝐺 3

𝜎 𝑗2 𝜎 𝑗2

5 d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

+ 1

d𝑗 Δ𝐺 3

d𝑗 Δ𝐺 2

𝜎 𝑗2 𝜎 𝑗2

d𝑗 Δ𝐺 2

𝜎 𝑗2 𝜎 𝑗2

∑︁ 𝑗1 , 𝑗2 , 𝑗3

𝜙 𝑗1 𝜙 𝑗3

3 d𝑗 Δ𝐺 1

d𝑗 Δ𝐺 2

𝜎 𝑗2 𝜎 𝑗2

d𝑗 Δ𝐺 2

𝜎 𝑗2 𝜎 𝑗2

4 d𝑗 Δ𝐺 1

∑︁ 2

𝜎 𝑗2 𝜎 𝑗2

∑︁

𝜎 𝑗1 𝜎 𝑗1

∑︁ 2

𝜎 𝑗1 𝜎 𝑗1

5 𝜎 𝑗1 𝜎 𝑗1

∑︁ 5

4 d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

𝜎 𝑗2 𝜎 𝑗2

∑︁ 𝑗1 , 𝑗2 , 𝑗3

𝜙 𝑗1 𝜙 𝑗3

d𝑗 Δ𝐺 3

𝑗1 , 𝑗2 , 𝑗3

d𝑗 Δ𝐺 1

𝜙 𝑗1 𝜙 𝑗3

𝜎 𝑗2 𝜎 𝑗2

+ 𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗3 𝜎 𝑗3

∑︁ 𝑗1 , 𝑗2 , 𝑗3

4 𝜎 𝑗1 𝜎 𝑗1

𝑗1 , 𝑗2 , 𝑗3

+ 𝜎 𝑗3 𝜎 𝑗3

∑︁ 𝑗1 , 𝑗2 , 𝑗3

1 2

4 𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗3 𝜎 𝑗3

𝜎 𝑗2 𝜎 𝑗2

∑︁ 𝑗1 , 𝑗2 , 𝑗3

+ 𝜎 𝑗3 𝜎 𝑗3

∑︁ 𝑗1 , 𝑗2 , 𝑗3

1 2

5 𝜎 𝑗1 𝜎 𝑗1

𝑗1 , 𝑗2 , 𝑗3

+ 1

4 𝜎 𝑗2 𝜎 𝑗2

6 𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗3 𝜎 𝑗3

𝜎 𝑗2 𝜎 𝑗2

∑︁

4 𝜎 𝑗2 𝜎 𝑗2

5 𝜎 𝑗1 𝜎 𝑗1

6 𝜎 𝑗2 𝜎 𝑗2

4 𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗3 𝜎 𝑗3

𝜎 𝑗2 𝜎 𝑗2

∑︁

𝜎 𝑗2 𝜎 𝑗2

∑︁

+ d𝑗 Δ𝐺 1

d𝑗 Δ𝐺 3

𝜎 𝑗2 𝜎 𝑗2

ª ® ® ® ® ® 3 ® ® ® ® ® 5 ® ® ® ® ® ¬

𝜎 𝑗3 𝜎 𝑗3

∑︁ 𝑗1 , 𝑗2 , 𝑗3

6 𝜎 𝑗1 𝜎 𝑗1

𝜎 𝑗3 𝜎 𝑗3

ª ® ® ® ® ® 3 ® ® ® ® ® 5 ® ® ® ® ® ¬

(130)

Remarkably, these diagrams exactly reproduce the algebraic expression derived in the previous subsection at order 1/𝑛2 , see (107). Moreover, (130) provides a direct validation of the leading-order (1/n) Feynman rules presented in Appendix D. Specifically, the first three lines of (130) follow from the Gaussian-like Feynman rules (1)-(4). The remaining contributions, namely the two groups of terms in parentheses multiplied by −1/𝑛4 and 2/𝑛5 , arise from rule (5): the former corresponds to 𝜆 = (2, 1) with 𝛽 (2,1) = −1, while the latter corresponds to 𝜆 = (3) with 𝛽 (3) = 2. In both cases, the cubic vertices defined in (77) appear as 46

disconnected components, as prescribed by rule (5).

G.3

Single-input case

In the single-input setting, the recursion relation for 𝑉6(ℓ ) takes a simple and compact form. Explicitly, 𝑉6G is readily obtained as (ℓ+1) 3 1 G (ℓ+1) (𝐶𝑊 ) 6 4 2 2 3 𝑉 = ⟨𝜎 ⟩ − 3⟨𝜎 ⟩ ⟨𝜎 ⟩ + 2(⟨𝜎 ⟩ ) 𝐾 𝐾 𝐾 𝐾 𝑛ℓ2 6 𝑛ℓ2 (ℓ+1) 2 (𝐶𝑊 ) (ℓ ) (ℓ ) 2 ′ 2 3 ′′ ′ 2 2 2 ′′ 𝑉 𝜒 | | 3⟨𝜎 (𝜎 ) ⟩𝐾 + ⟨𝜎 𝜎 ⟩𝐾 − ⟨(𝜎 ) ⟩𝐾 ⟨𝜎 ⟩𝐾 − ⟨𝜎 ⟩𝐾 ⟨𝜎𝜎 ⟩𝐾 +6 𝑛ℓ 𝑛ℓ −1 4 3𝐶 (ℓ+1) (ℓ ) 2 (ℓ ) 2 ′′′′ 1 + 𝑊2 ( 𝜒 | | ) (𝑉4 ) 3⟨(𝜎 ′′ ) 2 ⟩𝐾 + 4⟨𝜎 ′ 𝜎 ′′′ ⟩𝐾 + ⟨𝜎𝜎 ⟩𝐾 + 2 ( 𝜒 |(ℓ| ) ) 3𝑉6(ℓ ) 2𝑛ℓ −1 𝑛ℓ −1 (131)

whereas Δ𝑉6 is given by (ℓ+1) 3 (𝐶𝑊 ) 1 𝑛ℓ (ℓ+1) 4 2 2 3 2 2 2 Δ𝑉 = − 12⟨𝜎 ⟩𝐾 ⟨𝜎 ⟩𝐾 + 28(⟨𝜎 ⟩𝐾 ) − 12 ⟨𝜎 |𝜎 ⟩𝑉 (ℓ) ⟨𝜎 ⟩𝐾 𝑛ℓ −1 𝑛ℓ2 6 𝑛ℓ2 (ℓ+1) 3 (𝐶𝑊 ) 𝑛ℓ 12 (ℓ ) 2 4 2 2 3 (ℓ ) 2 − 12⟨𝜎 ⟩𝐾 ⟨𝜎 ⟩𝐾 + 28(⟨𝜎 ⟩𝐾 ) − = 𝑉 ( 𝜒 | | ) ⟨𝜎 ⟩𝐾 (ℓ+1) 2 𝑛 𝑛ℓ2 ℓ −1 (𝐶𝑊 ) (132) Therefore, the orthogonal six-point cumulant reads (ℓ+1) 3 1 (ℓ+1) (𝐶𝑊 ) 6 4 2 2 3 𝑉 = ⟨𝜎 ⟩ − 15⟨𝜎 ⟩ ⟨𝜎 ⟩ + 30(⟨𝜎 ⟩ ) 𝐾 𝐾 𝐾 𝐾 𝑛ℓ2 6 𝑛ℓ2 (ℓ+1) 2 (𝐶𝑊 ) (ℓ ) (ℓ ) 2 ′ 2 3 ′′ ′ 2 2 2 ′′ +6 𝑉 𝜒 | | 3⟨𝜎 (𝜎 ) ⟩𝐾 + ⟨𝜎 𝜎 ⟩𝐾 − 3⟨(𝜎 ) ⟩𝐾 ⟨𝜎 ⟩𝐾 − 3⟨𝜎 ⟩𝐾 ⟨𝜎𝜎 ⟩𝐾 𝑛ℓ 𝑛ℓ −1 4 (ℓ+1) 3𝐶𝑊 ′′′′ 1 (ℓ ) 2 (ℓ ) 2 ′′ 2 ′ ′′′ + ( 𝜒 | | ) (𝑉4 ) 3⟨(𝜎 ) ⟩𝐾 + 4⟨𝜎 𝜎 ⟩𝐾 + ⟨𝜎𝜎 ⟩𝐾 + 2 ( 𝜒 |(ℓ| ) ) 3𝑉6(ℓ ) (133) 2 2𝑛ℓ −1 𝑛ℓ −1

We use Mathematica to solve (133) for a rectangular tanh MLP of width 𝑛 = 50, depth 𝐿 = 30 and input vector as in (97), both exactly and asymptotically; see Appendix H for details. The results for the normalized tensor 𝑉˜6(ℓ ) = 𝑉6(ℓ ) /(𝐾 (ℓ ) ) 3 are shown in Figure 10.

Large-ℓ expansion

In this appendix, we solve the single-input recursion relations (86)-(95) and (133) for tanh networks in the large-ℓ regime. We adopt a standard renormalization-inspired ansatz and expand a generic tensor O (ℓ ) as O (ℓ ) = ℓ − 𝑝O

𝑘 ∑︁

𝑐 𝑖,O 𝑗

log 𝑗 (ℓ) ℓ𝑖

𝑖, 𝑗=0

where 𝑝 O is the critical exponent and 𝑘 controls the truncation order. Throughout, we set 𝑘 = 5.

(134)

16 14

V(6 )

12 10 8 6 4

Exact Solution Large- Expansion 0

Figure 10: The sextic vertex. Solution to the single-input recursion relation (133) at order 1/𝑛2 . We consider a tanh network with width 𝑛 = 50 and depth 𝐿 = 30, with inputs drawn from (0, 1). Blue points show the exact solution of (133), while orange boxes represent the large-ℓ expansion (170), evaluated at integer ℓ. The tensor magnitude remains below its Gaussian counterpart and exhibits early-layer saturation, consistent with the behavior of lower-rank orthogonal tensors. The coefficients 𝑐 𝑖,O 𝑗 in (134) depend on both the input data and the network parameters through the initial conditions 𝐾 (1) = 𝐷

(1)

𝑛 𝐶𝑊 ∑︁ 𝑥𝑖 · 𝑥𝑖 , 𝑛 𝑖=1

= 0,

𝑄 (1) = 0,

𝐹

(1)

= 0,

𝑅 (1) = 0,

Θ (1) = 𝜆 𝑏(1) +

(1) ∑︁ 𝑛 𝜆𝑊

𝑥𝑖 · 𝑥𝑖 ,

2 𝑉 (1) = −2 𝐾 (1) ,

𝑛 𝑖=1

𝐴

(1)

= 0,

𝑆 (1) = 0,

= 0,

𝑃 (1) = 0,

𝑇 (1) = 0,

𝑈 (1) = 0.

𝐵

(1)

(135)

In addition, they depend parametrically on the choice of activation function. We compute these coefficients using Mathematica, combining symbolic manipulation with numerical evaluation. We report below the critical exponents 𝑝 O and the corresponding coefficients for all tensors considered in this work. We consider a square neural network with activation function tanh, and fix 𝐶𝑊 = 1, thereby focusing on the critical regime [11]. As discussed in [11], for the tanh nonlinearity the training hyperparameters can be chosen as 𝜆 𝑏(ℓ ) =

1 ℓ

(ℓ ) 𝜆𝑊 =1

(136)

Since the recursion relations (86)-(95) connect quantities at consecutive layers, i.e. at depths ℓ and ℓ + 1, the following large-ℓ asymptotic expansions are useful: 𝑖 ∞ 1 1 ∑︁ −𝑎 1 = (137) (1 + ℓ) 𝑎 ℓ 𝑎 𝑖=0 𝑖 ℓ 𝑖 ∞ ∑︁ 1 1 log(1 + ℓ) = log(ℓ) − − (138) 𝑖 ℓ 𝑖=0 The Gaussian expectations appearing in (86)-(95) are evaluated by expanding the activation function as 𝜎(𝑧) =

∞ ∑︁

𝜎𝑚

𝑧𝑚 𝑚!

(139)

𝑚=0

where the coefficients 𝜎𝑚 = 𝜎 (𝑚) (0) are determined by derivatives of 𝜎 at the origin. For 𝜎 = tanh, these coefficients are known in closed form. 48

For the trucantion order 𝑘 = 5 in (134), retaining a finite number of terms in the expansion (139) is sufficient to obtain a consistent large-ℓ expansion. In practice, we include terms up to 𝑚 = 20, which ensures stability of all coefficients at the desired order. Concretely, the non-vanishing coefficients are 𝜎1 = 1

𝜎3 = −2

𝜎5 = 16

𝜎7 = −272

𝜎13 = 22368256

𝜎15 = −1903757312

𝜎17 = 209865342976

𝜎19 = −29088885112832

𝜎9 = 7936

𝜎11 = −353792 (140)

with 𝜎𝑚 = 0 for 𝑚 even.

H.1

The NNGP

To illustrate the procedure, we derive the expansion of the NNGP kernel 𝐾 (ℓ ) . Substituting the ansatz (134) into the recursion (6) yields 1 5 log ℓ 5 log ℓ0 1 + 2 − 𝐾 (ℓ ) = 2ℓ ℓ 24 24 2 1 25(log ℓ) 5 log ℓ (−5 − 10 log ℓ0 ) 53 + 25 log ℓ0 + 25(log ℓ0 ) 2 + 3 + + 288 288 288 ℓ 125 75 2 5 log ℓ 23 + log ℓ + (log ℓ ) 3 2 0 0 8 8 1 © 125(log ℓ) 25(log ℓ) (−25 − 30 log ℓ0 ) + 4 + + 3456 6912 432 ℓ « −8597 − 9200 log ℓ0 − 3125(log ℓ0 ) 2 − 1250(log ℓ0 ) 3 + 34560 2 287 + 650 log ℓ + 100(log ℓ ) 2 25(log ℓ) 3 4 0 0 3 1 © 625(log ℓ) 125(log ℓ) (−65 − 60 log ℓ0 ) + 5 + + 41472 124416 27648 ℓ « +

8125 2 3 log ℓ −10897 − 21525 2 log ℓ0 − 2 (log ℓ0 ) − 1250(log ℓ0 )

20736 −2479663 + 653820 log ℓ0 + 322875(log ℓ0 ) 2 + 81250(log ℓ0 ) 3 + 18750(log ℓ0 ) 4 . (141) + 1244160 𝐾 = − 5 log(ℓ ). Consistency of the expansion fixes the critical exponent to 𝑝 = 1, where we defined 𝑐 1,0 0 𝐾 24 ensuring that the recursion (6) admits a solution within the ansatz. The first two lines of (141) reproduce the known large-ℓ expansion of the midpoint kernel up to order ℓ −3 [11]. The scale ℓ0 is fixed by the initial condition (135) after replacing ℓ = 1 in (141).

H.2

The quartic vertex

Having obtained the large-ℓ expansion of 𝐾 (ℓ ) , we now compute the corresponding expansion of the quartic vertex 𝑉4(ℓ ) from (8). We again adopt the ansatz (134) together with the initial conditions (135). Using the large-ℓ identities (137), (138), the tanh coefficients (140), and fixing the critical exponent to 𝑝 𝑉 = 2, we obtain 1 5 1 1 (ℓ ) (8 + 5 log ℓ0 ) 𝑉4 = − 2 + 3 − log ℓ + 12 12 2ℓ ℓ 49

25 1 1 2 𝑉 + 4 − (log ℓ) + log ℓ (−167 + 75 log ℓ0 ) + 𝑐 2,0 . 96 144 ℓ

(142)

Although the ansatz formally includes coefficients 𝑐𝑉 𝑖, 𝑗 for 𝑖, 𝑗 = 0, . . . , 5, the recursion (8) only constrains a subset of these coefficients within our truncation scheme. In particular, the Gaussian expectation values in the first two terms on the left-hand side of (8), which depend on 𝐾 (ℓ ) , contribute at most at order ℓ −5 . Since 𝑝 𝑉 = 2, this implies that only coefficients with 𝑖 = 0, 1, 2 can be determined. The coefficient 𝑐𝑉 2,0 is fixed by the initial conditions (135) after replacing ℓ = 1 in (142).

H.3

The NTK

We next determine the large-ℓ behavior of the frozen NTK Θ (ℓ ) from (10). Substituting the expansion of 𝐾 (ℓ ) from (141) and proceeding as in the previous cases, we find that the critical exponent is 𝑝 Θ = 0, yielding i 3 1h 5 1 + − (log ℓ) 2 + log ℓ(27 + 10 log ℓ0 ) + 𝑐Θ 1,0 2 ℓ 24 24 h 25 5 447 + 205 log ℓ0 1 (log ℓ) 3 + (log ℓ) 2 (31 + 15 log ℓ0 ) + + 2 − 288 288 288 ℓ −313 − 175 log ℓ − 50(log ℓ ) 2 5 0 0 + log ℓ + log ℓ 𝑐Θ 288 12 1,0 i 1 − (4 + 5 log ℓ0 )𝑐Θ 1,0 12 125 1 h 125 (log ℓ) 4 + (log ℓ) 3 (2 + log ℓ0 ) + 3 − 3456 864 ℓ −183829 − 116850 log ℓ0 − 26500(log ℓ0 ) 2 + 69120 25(143 + 93 log ℓ + 25(log ℓ ) 2 ) 25 0 0 + (log ℓ) 2 − + (log ℓ) 2 𝑐Θ 1,0 3456 144 2 99 + 65 log ℓ0 + 25(log ℓ0 ) Θ 𝑐 1,0 + 144 17031 + 10790 log ℓ + 2650(log ℓ ) 2 + 500(log ℓ ) 3 0 0 0 + log ℓ 6912 5 i Θ (13 + 10 log ℓ0 ) 𝑐 1,0 + log ℓ − 144 h 625 1 125(log ℓ) 4 (103 + 50 log ℓ0 ) (log ℓ) 5 + + 4 − 41472 82944 ℓ 287109682 + 282873525 log ℓ0 + 85689000(log ℓ0 ) 2 + 67184640 3 12240000(log ℓ0 ) + 67184640 25(23863 + 16110 log ℓ + 4050(log ℓ ) 2 ) 0 0 + (log ℓ) 3 − 746496 3 125 Θ + (log ℓ) 𝑐 1728 1,0 21949 + 18100 log ℓ0 + 6125(log ℓ0 ) 2 Θ − 𝑐 1,0 17280 1250(log ℓ0 ) 3 Θ − 𝑐 1,0 17280 2029301 + 1492050 log ℓ + 457875(log ℓ ) 2 0 0 + (log ℓ) 2 746496

Θ (ℓ ) =

78750(log ℓ ) 3

25(49 + 30 log ℓ0 ) Θ 𝑐 1,0 746496 3456 −126292817 − 95368260 log ℓ − 30944250(log ℓ ) 2 0 0 + log ℓ 22394880 −5130000(log ℓ ) 3 − 675000(log ℓ ) 4 0 0 + log ℓ 22394880 i 5(362 + 245 log ℓ + 75(log ℓ ) 2 ) 0 0 𝑐Θ + log ℓ 1,0 . 1728 + (log ℓ) 2

−

(143)

As in the previous cases, the truncation order 𝑘 together with the scaling 𝑝 Θ = 0 restricts the set of Θ coefficients 𝑐Θ 𝑖, 𝑗 that can be determined within this approximation. The coefficient 𝑐 1,0 is determined by the initial conditions (135) after replacing ℓ = 1 in (143).

H.4

The NTK-preactivation mixed cumulant

We next determine the preactivation-NTK mixed cumulants 𝐷 (ℓ ) and 𝐹 (ℓ ) . Applying the same procedure as above yields the critical exponents 𝑝 𝐷 = 2 and 𝑝 𝐹 = 1, together with the large-ℓ expansions " 4 5 1 1 𝐷 𝐷 (ℓ ) = − 2 + 3 (log ℓ) 3 + (log ℓ) 2 (51 − 10 log ℓ0 ) + 𝑐 1,0 36 3ℓ ℓ 54 # 1 𝑉 + log ℓ −274 − 290 log ℓ0 − 75(log ℓ0 ) 2 − 192𝑐Θ , (144) 1,0 − 288𝑐 2,0 144 " # 1 1 5 1 1 (ℓ ) 2 Θ (log ℓ) + log ℓ (−21 − 5 log ℓ0 ) + 𝐹 =− + 2 27 + 5 log ℓ0 − 8𝑐 1,0 2ℓ ℓ 48 24 16 " 25 1 25 (log ℓ) 3 − (log ℓ) 2 (10 + 3 log ℓ0 ) + 3 288 ℓ 288 1 log ℓ 1061 + 365 log ℓ0 + 50(log ℓ0 ) 2 − 120𝑐Θ + 1,0 288 # 1 2 Θ Θ + −3321 − 575 log ℓ0 − 25(log ℓ0 ) + 432𝑐 1,0 + 120 log ℓ0 𝑐 1,0 288 " 1 625 5 𝐹 (log ℓ) 4 − (log ℓ) 3 (284 + 125 log ℓ0 ) + 𝑐 3,0 + 4 3456 ℓ 13824 (log ℓ) 2 + 23833 + 12990 log ℓ0 + 3000(log ℓ0 ) 2 − 3600𝑐Θ 1,0 13824 log ℓ + 1107057 − 450550 log ℓ0 − 84000(log ℓ0 ) 2 − 5000(log ℓ0 ) 3 69120 # Θ Θ −188640𝑐 1,0 + 36000 log ℓ0 𝑐 1,0 . (145) 𝐷 and 𝑐 𝐹 are fixed by the initial conditions (135) together with the previously The coefficients 𝑐 1,0 3,0 Θ determined parameters log ℓ0 , 𝑐𝑉 2,0 , and 𝑐 1,0 .

H.5

The NTK-variance

We now turn to the NTK variance tensors 𝐴 (ℓ ) and 𝐵 (ℓ ) . Using the expansion of 𝐷 (ℓ ) in (144), we find the critical exponents 𝑝 𝐴 = 1 and 𝑝 𝐵 = 1 together with " 1 5 1 16 𝐴 + 2 − (log ℓ) 4 + (log ℓ) 3 (−61 + 10 log ℓ0 ) + 𝑐 1,0 𝐴 (ℓ ) = 3ℓ ℓ 54 27 4 + log ℓ −15 + 10 log ℓ0 − 9𝑑𝑑1,0 + 12𝑐Θ 1,0 9 # 1 𝑉 + (log ℓ) 2 330 + 370 log ℓ0 + 75(log ℓ0 ) 2 + 192𝑐Θ , (146) 1,0 + 288𝑐 2,0 72 " 5 9 1 1 𝐵 𝐵 (ℓ ) = + 2 − (log ℓ) 3 + (log ℓ) 2 (27 + 10 log ℓ0 ) + 𝑐 1,0 2ℓ ℓ 12 8 # 3 Θ + log ℓ −21 + 8𝑐 1,0 4 " 125 5 1 (log ℓ) 4 + (log ℓ) 3 (121 + 50 log ℓ0 ) + 3 − 288 144 ℓ 1 𝐵 − (4 + 5 log ℓ0 ) 𝑐 1,0 6 1 + (log ℓ) 2 −1683 − 435 log ℓ0 − 100(log ℓ0 ) 2 + 420𝑐Θ 1,0 72 1 Θ Θ 2 + −18516 − 690 log ℓ0 − 125(log ℓ0 ) 2 + 5664𝑐Θ + 240 log ℓ 𝑐 − 576(𝑐 ) 0 1,0 1,0 1,0 288 !# 2 Θ Θ 5043 + 3330 log ℓ0 + 50(log ℓ0 ) − 1344𝑐 1,0 − 960 log ℓ0 𝑐 1,0 5 𝐵 + log ℓ 𝑐 1,0 + 6 144 " 1 125 125(log ℓ) 4 (211 + 90 log ℓ0 ) + 4 − (log ℓ) 5 + 384 6912 ℓ 214 + 170 log ℓ0 + 75(log ℓ0 ) 2 𝐵 𝑐 1,0 144 5 (log ℓ) 3 17317 + 7310 log ℓ0 + 1650(log ℓ0 ) 2 − 2880𝑐Θ − 1,0 3456 2 15676647 + 5933100 log ℓ0 + 221500(log ℓ0 ) + 22500(log ℓ0 ) 3 + 69120 4405920 + 1816800 log ℓ0 + 60000(log ℓ0 ) 2 Θ − 𝑐 1,0 69120 339840 + 172800 log ℓ0 Θ 2 + (𝑐 1,0 ) 69120 25 𝐵 174843 + 113770 log ℓ0 + 16900(log ℓ0 ) 2 + 2500(log ℓ0 ) 3 + (log ℓ) 2 𝑐 1,0 + 48 2304 ! 36000 + 21600 log ℓ0 Θ − 𝑐 1,0 2304 +

+ log ℓ − +

5 𝐵 (17 + 15 log ℓ0 ) 𝑐 1,0 72

−625380 − 280395 log ℓ0 − 72950(log ℓ0 ) 2 − 1250(log ℓ0 ) 3 3456 52

159888 + 64080 log ℓ0 + 18000(log ℓ0 ) 2 Θ 8640 Θ 2 + 𝑐 1,0 − (𝑐 ) 3456 3456 1,0

!# .

(147)

𝐴 and 𝑐 𝐵 are again fixed by the initial conditions (135). The parameters 𝑐 1,0 1,0

H.6

The dNTK-preactivation mixed cumulant

The asymptotic behavior of the preactivation-dNTK cumulant tensors 𝑃 (ℓ ) and 𝑄 (ℓ ) follows from the same procedure. This yields the critical exponents 𝑝 𝑃 = 0 and 𝑝 𝑄 = 0, together with the large-ℓ expansions i 3 1h 5 1 1 + (log ℓ) 2 + log ℓ(−16 − 5 log ℓ0 ) + (31 + 5 log ℓ0 − 24𝑐Θ 1,0 ) 4 ℓ 16 8 16 5 1 h 25 (log ℓ) 4 + (log ℓ) 3 (137 + 20 log ℓ0 ) + 2 − 576 576 ℓ (log ℓ) 2 + −5748 − 1785 log ℓ0 − 100(log ℓ0 ) 2 + 240𝑐Θ 1,0 576 log ℓ 26199 + 5205 log ℓ0 + 350(log ℓ0 ) 2 + 576 log ℓ Θ + −5592𝑐Θ 1,0 − 480 log ℓ0 𝑐 1,0 576 1 𝐵 + −40257 − 6300 log ℓ0 − 125(log ℓ0 ) 2 − 576𝑐 1,0 576 i 1 Θ Θ 2 + 9624𝑐Θ + 840 log ℓ 𝑐 − 576(𝑐 ) 0 1,0 1,0 1,0 576 h 55 1 5 + 3 − (log ℓ) 5 + (log ℓ) 4 (2009 + 220 log ℓ0 ) 3456 13824 ℓ (log ℓ) 3 −128583 − 62430 log ℓ0 − 3700(log ℓ0 ) 2 − 1920𝑐Θ + 1,0 20736 (log ℓ) 2 + 396027 + 182700 log ℓ0 + 50800(log ℓ0 ) 2 13824 (log ℓ) 2 Θ 3000(log ℓ0 ) 3 − 66192𝑐Θ + 1,0 + 3840 log ℓ0 𝑐 1,0 13824 log ℓ + 1355239 − 1599000 log ℓ0 − 260500(log ℓ0 ) 2 23040 log ℓ 𝐵 + −10000(log ℓ0 ) 3 − 19200𝑐 1,0 − 516720𝑐Θ 1,0 23040 log ℓ 2 Θ Θ 2 + 269600 log ℓ0 𝑐Θ 1,0 + 24000(log ℓ0 ) 𝑐 1,0 + 72960(𝑐 1,0 ) 23040 i

𝑃 (ℓ ) = −

𝑃 + 𝑐 3,0 . (148) h i 17 1 5 1 1 𝑄 (ℓ ) = − + (log ℓ) 2 + log ℓ(−101 − 30 log ℓ0 ) + (93 + 10 log ℓ0 − 36𝑐Θ 1,0 ) 12 ℓ 8 24 12 1 h 25 5 + 2 − (log ℓ) 4 + (log ℓ) 3 (183 + 40 log ℓ0 ) 288 576 ℓ (log ℓ) 2 + −6013 − 2205 log ℓ0 − 200(log ℓ0 ) 2 + 480𝑐Θ 1,0 576 log ℓ + 20897 + 7085 log ℓ0 + 750(log ℓ0 ) 2 576 log ℓ Θ −4392𝑐Θ + 1,0 − 960 log ℓ0 𝑐 1,0 576

1 −53509 − 8045 log ℓ0 − 150(log ℓ0 ) 2 576 i 1 Θ Θ 2 + 11424𝑐Θ + 1800 log ℓ 𝑐 − 1152(𝑐 ) 0 1,0 1,0 1,0 576 h 305 5 1 (log ℓ) 5 + (log ℓ) 4 (4273 + 1220 log ℓ0 ) + 3 − 3456 13824 ℓ (log ℓ) 3 + −209617 − 115500 log ℓ0 − 15950(log ℓ0 ) 2 + 11280𝑐Θ 1,0 20736 (log ℓ) 2 1546071 + 268150 log ℓ0 + 81125(log ℓ0 ) 2 + 17280 (log ℓ) 2 Θ 9375(log ℓ0 ) 3 − 227460𝑐Θ + 1,0 − 28200 log ℓ0 𝑐 1,0 17280 log ℓ + −697585 − 864585 log ℓ0 − 137400(log ℓ0 ) 2 6912 log ℓ 𝐹 + −6250(log ℓ0 ) 3 + 27648𝑐 3,0 − 62712𝑐Θ 1,0 6912 log ℓ 2 Θ Θ 2 129840 log ℓ0 𝑐Θ + 1,0 + 22800(log ℓ0 ) 𝑐 1,0 + 10368(𝑐 1,0 ) 6912 i +

𝑄 + 𝑐 3,0 .

(149)

𝑃 and 𝑐 𝑄 follow from the initial conditions (135). The coefficients 𝑐 3,0 3,0

H.7

The ddI NTK mean

Applying this framework to the dI NTK mean tensor 𝑅 (ℓ ) fixes the critical exponent to 𝑝 𝑅 = −1, yielding 𝑅 (ℓ ) = − 0.43821 − 1.75 ℓ + 0.9375(log ℓ) 2 + log ℓ (−6.67172 − 2.5 log ℓ0 ) + 0.520833 log ℓ0 − 6𝑐Θ 1,0 1h + −30.8931 − 0.173611(log ℓ) 4 − 39.3424 log ℓ0 − 2.08333(log ℓ0 ) 2 ℓ 𝐵 Θ Θ 2 + (log ℓ) 3 (4.17447 + 1.30208 log ℓ0 ) − 3𝑐 1,0 + 49.7536𝑐Θ 1,0 + 7.5 log ℓ0 𝑐 1,0 − 9(𝑐 1,0 ) + (log ℓ) 2 −27.3051 − 16.1153 log ℓ0 − 1.5625(log ℓ0 ) 2 + 3.125𝑐Θ 1,0 + log ℓ 75.8169 + 33.0744 log ℓ0 + 3.125(log ℓ0 ) 2 i Θ + log ℓ −43.9268𝑐Θ − 7.5 log ℓ 𝑐 0 1,0 1,0 h 1 + 2 (log ℓ) 5 0.256462 + 0.336123 log ℓ0 + 0.0434028(log ℓ0 ) 2 − 0.0347222𝑐Θ 1,0 ℓ + (log ℓ) 4 −0.898367 − 1.47402 log ℓ0 − 0.580512(log ℓ0 ) 2 − 0.036169(log ℓ0 ) 3 Θ + 1.24491𝑐Θ 1,0 + 0.260417 log ℓ0 𝑐 1,0 𝐵 + (log ℓ) 3 −7.80677 − 6.49926 log ℓ0 − 1.31955(log ℓ0 ) 2 + 0.138889𝑐 1,0 Θ 2 Θ Θ 2 − 7.24624𝑐Θ 1,0 − 4.46181 log ℓ0 𝑐 1,0 − 0.347222(log ℓ0 ) 𝑐 1,0 + 0.416667(𝑐 1,0 )

+ (log ℓ) 2 −111.626 + 21.3942 log ℓ0 + 13.3665(log ℓ0 ) 2 + 1.30208(log ℓ0 ) 3 𝐵 𝐵 Θ − 1.125𝑐 1,0 − 0.416667 log ℓ0 𝑐 1,0 − 3.46714𝑐Θ 1,0 − 1.02641 log ℓ0 𝑐 1,0 2 Θ 2 − 9.375(𝑐Θ 1,0 ) − 1.25 log ℓ0 (𝑐 1,0 ) + log ℓ 131.914 − 39.0856 log ℓ0 − 32.2968(log ℓ0 ) 2 − 2.17014(log ℓ0 ) 3 𝐵 Θ 2 Θ + log ℓ +𝑐 1,0 − 6.96453𝑐Θ 1,0 + 32.6793 log ℓ0 𝑐 1,0 + 6.25(log ℓ0 ) 𝑐 1,0

𝐵 Θ 2 Θ 3 𝑃 + log ℓ −2𝑐 1,0 𝑐 1,0 + 11(𝑐Θ 1,0 ) − 2(𝑐 1,0 ) − 2𝑐 3,0 i 𝑅 . + 𝑐 3,0

(150)

𝑅 is determined analogously from the initial conditions (135). The parameter 𝑐 3,0

H.8

The ddII NTK mean

Applying the same analysis to the dII NTK mean tensors 𝑆 (ℓ ) , 𝑇 (ℓ ) and 𝑈 (ℓ ) , yields the critical exponents 𝑝 𝑆 = −1, 𝑝 𝑇 = −1, and 𝑝𝑈 = 0, together with 𝑆 (ℓ ) =

3 5 1 3 ℓ − (log ℓ) 2 + log ℓ (27 + 10 log ℓ0 ) + (45 + 16𝑐Θ 1,0 ) 4 16 16 32 h 5 1 25 (log ℓ) 4 − (log ℓ) 3 (61 + 10 log ℓ0 ) + ℓ 576 288 1 + (log ℓ) 2 2763 + 1560 log ℓ0 + 100(log ℓ0 ) 2 − 240𝑐Θ 1,0 576 1 Θ log ℓ −7137 + 360 log ℓ0 − 200(log ℓ0 ) 2 + 5232𝑐Θ + 1,0 + 480 log ℓ0 𝑐 1,0 576 1 𝐵 + 6 + 765 log ℓ0 + 50(log ℓ0 ) 2 + 576𝑐 1,0 + 2400𝑐Θ 1,0 576 i Θ 2 − 480 log ℓ0 𝑐Θ 1,0 + 576(𝑐 1,0 ) 5 1 h 335 (log ℓ) 5 − (log ℓ) 4 (1328 + 335 log ℓ0 ) + 2 3456 ℓ 3456 5 (log ℓ) 3 8685 + 4286 log ℓ0 + 480(log ℓ0 ) 2 − 912𝑐Θ + 1,0 3456 1 + (log ℓ) 2 −412047 − 75465 log ℓ0 − 14625(log ℓ0 ) 2 − 1000(log ℓ0 ) 3 6912 Θ + 164520𝑐Θ 1,0 + 27360 log ℓ0 𝑐 1,0 1 + log ℓ 1165869 + 298080 log ℓ0 + 1200(log ℓ0 ) 2 + 2500(log ℓ0 ) 3 13824 𝐵 Θ 2 Θ + 52992𝑐 1,0 − 458208𝑐Θ 1,0 − 96480 log ℓ0 𝑐 1,0 − 9600(log ℓ0 ) 𝑐 1,0 i 2 𝑆 + 108288(𝑐Θ 1,0 ) + 𝑐 3,0 ,

(151)

𝑇 (ℓ ) = − 18.981 + 1.88889 ℓ − 1.84028(log ℓ) 2 + 0.196759 log ℓ0 + log ℓ 5.80387 + 1.18056 log ℓ0 + 2.83333 𝑐Θ 1,0 1h 4 3 + 440.657 + 0.868056(log ℓ) + (log ℓ) −8.44101 − 1.04167 log ℓ0 ℓ + 8.02013 log ℓ0 − 0.491898(log ℓ0 ) 2 + (log ℓ) 2 46.0699 + 3.38594 log ℓ0 − 2.5 𝑐Θ 1,0 i Θ − 32.6732 𝑐Θ + log ℓ −158.347 − 13.6717 log ℓ + 8.12626 𝑐 0 1,0 1,0 h 1 + 2 (log ℓ) 5 −0.1989 − 0.0241419 log ℓ0 + 0.0694444 𝑐Θ 1,0 ℓ + (log ℓ) 4 −2.50973 − 1.10439 log ℓ0 − 0.108507(log ℓ0 ) 2 − 0.376245 𝑐Θ 1,0 + (log ℓ) 3 −75.8647 + 8.00378 log ℓ0 + 0.962168(log ℓ0 ) 2 + 1.63315 𝑐Θ 1,0 − 0.347222 log ℓ0 𝑐Θ 1,0 + (log ℓ) 2 −256.807 − 13.8107 log ℓ0 − 2.88587(log ℓ0 ) 2 − 10.1288 𝑐Θ 1,0 Θ + 3.15131 log ℓ0 𝑐 1,0 55

+ log ℓ −297.85 + 308.089 log ℓ0 + 8.69265(log ℓ0 ) 2 − 0.204958(log ℓ0 ) 3 𝑄 Θ Θ 2 + 229.472 𝑐Θ 1,0 − 10.9002 log ℓ0 𝑐 1,0 + (𝑐 1,0 ) − 4 𝑐 3,0 i + 𝑐𝑇3,0 , 27 1 h 45 9 + − (log ℓ) 2 + log ℓ (37 + 10 log ℓ0 ) 𝑈 (ℓ ) = 8 ℓ 16 16 i 9 − 47 + 5 log ℓ0 − 12𝑐Θ 1,0 8 h 5 1 5 (log ℓ) 5 − (log ℓ) 4 (27 + 10 log ℓ0 ) + 2 128 ℓ 64 1 (log ℓ) 3 1119 + 540 log ℓ0 + 100(log ℓ0 ) 2 − 240𝑐Θ + 1,0 192 3 Θ + (log ℓ) 2 −837 − 260 log ℓ0 + 432𝑐Θ 1,0 + 160 log ℓ0 𝑐 1,0 128 3 Θ 2 log ℓ −2733 + 405 log ℓ0 + 100(log ℓ0 ) 2 + 1104𝑐Θ − 1,0 − 192(𝑐 1,0 ) 64 i

(152)

+ 𝑐𝑈 2,0 1h 5 + 3 − (log ℓ) 5 (819 + 230 log ℓ0 ) 2304 ℓ 5 + (log ℓ) 4 1593 + 1135 log ℓ0 + 225(log ℓ0 ) 2 − 300𝑐Θ 1,0 1152 1 + (log ℓ) 3 −187047 − 81075 log ℓ0 − 14850(log ℓ0 ) 2 − 2000(log ℓ0 ) 3 3456 Θ + 43560𝑐Θ 1,0 + 18000 log ℓ0 𝑐 1,0 1 (log ℓ) 2 460782 + 110370 log ℓ0 + 21525(log ℓ0 ) 2 + 500(log ℓ0 ) 3 + 2304 Θ 2 Θ Θ 2 − 173088𝑐Θ 1,0 − 41760 log ℓ0 𝑐 1,0 − 9600(log ℓ0 ) 𝑐 1,0 + 20160(𝑐 1,0 ) 1 30188151 − 884850 log ℓ0 + 309000(log ℓ0 ) 2 + 65000(log ℓ0 ) 3 + 46080 Θ 2 Θ Θ 2 − 12139200𝑐Θ 1,0 − 302400 log ℓ0 𝑐 1,0 − 60000(log ℓ0 ) 𝑐 1,0 + 1497600(𝑐 1,0 ) 2 Θ 3 + 57600 log ℓ0 (𝑐Θ 1,0 ) − 92160(𝑐 1,0 ) −1946727 − 1035240 log ℓ + 48450(log ℓ ) 2 + 15500(log ℓ ) 3 0 0 0 + log ℓ 4608 526176 + 348480 log ℓ0 + 4800(log ℓ0 ) 2 Θ 64512 + 46080 log ℓ0 Θ 2 + 𝑐 1,0 − (𝑐 1,0 ) 4608 4608 1 i 5 𝑈 − (4 + 5 log ℓ ) 𝑐 (153) + 𝑐𝑈 0 2,0 . 6 2,0 6

𝑆 , 𝑐 𝑇 , and 𝑐𝑈 are fixed by the initial conditions (135). The coefficients 𝑐 3,0 3,0 2,0

H.9

The sextic vertex

Finally, for the sextic vertex 𝑉6(ℓ ) , we obtain the critical exponent 𝑝 𝑉6 = 3, and the large-ℓ expansion 𝑉6(ℓ ) =

2 1 + 4 (−0.573233 + 2.5 log ℓ − 1.25 log ℓ0 ) . 3 ℓ ℓ

(154)

Within our approximation scheme, 𝑉6(ℓ ) is entirely determined by the lower-rank tensors 𝑉4(ℓ ) and 𝐾 (ℓ ) .

𝑐 𝑖,𝐾 𝑗

𝑝𝐾

𝑖\ 𝑗 0 1 2 3 4

1 0

1 2

2 0 0

5 24

0.571128 0.598432 0.54664 −1.97764

0.389134 0.585901 0.666942

3 0 0 0 125 3456

4 0 0 0 0

0.0999518

625 41472

25 288

0.20704 0.401984

5 0 0 0 0 0

Table 1: Coefficients of the large-ℓ expansion for the tensor 𝐾 (ℓ ) .

H.10

Application

We apply the above results to a square neural network of width 𝑛 = 50, using the same input vector as in Section F. This choice fixes the initial conditions 𝐾 (1) = 0.238565

Θ (1) = 1.23857

𝑉4(1) = −0.113827

(155)

These determine the scale ℓ0 via log(ℓ0 ) = −2.74141, as well as the coefficients 𝑐 O in (141)-(153): 𝑐𝑉 2,0 = 0.861763, 16 , 3 𝑅 𝑐 3,0 = −49.1347, 𝐴 𝑐 1,0 =−

𝑐Θ 1,0 = 0.829493,

4 𝐷 𝑐 1,0 = , 3

𝐹 𝑐 3,0 = 6.49751,

𝐵 𝑐 1,0 = 7.19961,

𝑃 𝑐 3,0 = 39.7952,

𝑄 𝑐 3,0 = 47.0357,

𝑆 𝑐 3,0 = −16.4737,

𝑐𝑇3,0 = −372.59,

𝑐𝑈 2,0 = −196.103

(156)

Substituting these values into (141)–(153) yields the corresponding large-ℓ expansions. We now present these expressions explicitly. The NNGP takes the form 5 1 1 + 2 0.571128 + log ℓ 𝐾 (ℓ ) = 2ℓ ℓ 24 25 1 2 (log ℓ) + 3 0.598432 + 0.389134 log ℓ + 288 ℓ 1 + 4 (0.54664 + 0.585901 log ℓ ℓ 125 2 3 +0.20704(log ℓ) + (log ℓ) 3456 1 + 5 (−1.97764 + 0.666942 log ℓ ℓ 625 2 3 4 +0.401984(log ℓ) + 0.0999518(log ℓ) + (log ℓ) . (157) 41472 For completeness, the parameters 𝑝 𝐾 and 𝑐 𝑖,𝐾 𝑗 are summarized in Table 1. The quartic vertex reads

5 −0.475589 − log ℓ 12 1 25 2 + 4 0.861763 − 2.58754 log ℓ − (log ℓ) , 96 ℓ

1 1 𝑉4(ℓ ) = − 2 + 3 2ℓ ℓ

with coefficients given in Table 2. The NTK takes the form 3 1 5 (ℓ ) 2 Θ = + 0.829493 − 0.0172557 log ℓ − (log ℓ) 2 ℓ 24 57

(158)

𝑐𝑉 𝑖, 𝑗

𝑝𝑉

0 − 12 −0.475589 0.861763

𝑖\ 𝑗 0 1 2

1 0 5 − 12 −2.58754

2 0 0 − 25 96

3 0 0 0

4 0 0 0

5 0 0 0

Table 2: Coefficients of the large-ℓ expansion for the tensor 𝑉 (ℓ ) . 𝑐Θ 𝑖, 𝑗

𝑝Θ

𝑖\ 𝑗 0 1 2 3 4

0 3 2

0.829493 0.271725 −0.280312 −1.08234

1 0 −0.0172557 −0.380142 −0.00940075 −0.722724

2 0 5 − 24 −0.175715 −0.405269 −0.125245

3 0 0 25 − 288 −0.107265 −0.279448

4 0 0 0 125 − 3456 −0.0513459

5 0 0 0 0 625 − 41472

Table 3: Coefficients of the large-ℓ expansion for the tensor Θ (ℓ ) . 25 1 2 3 + 2 0.271725 − 0.380142 log ℓ − 0.175715(log ℓ) − (log ℓ) 288 ℓ 1 + 3 (−0.280312 − 0.00940075 log ℓ ℓ 125 2 3 4 −0.405269(log ℓ) − 0.107265(log ℓ) − (log ℓ) 3456 1 + 4 (−1.08234 − 0.722724 log ℓ ℓ 625 5 (log ℓ) , (159) −0.125245(log ℓ) − 0.279448(log ℓ) − 0.0513459(log ℓ) − 41472 2

with coefficients summarized in Table 3. The NTK-preactivation mixed cumulants are 4 1 4 5 2 3 (ℓ ) − 3.12564 log ℓ + 2.17817(log ℓ) + (log ℓ) , 𝐷 =− 2 + 3 54 3ℓ ℓ 3 1 1 5 𝐹 (ℓ ) = − + 2 0.416062 − 0.303872 log ℓ + (log ℓ) 2 2ℓ ℓ 48 1 + 3 (−6.41357 + 1.16879 log ℓ ℓ 25 −0.154146(log ℓ) 2 + (log ℓ) 3 288 1 + 4 (6.49751 + 22.7949 log ℓ ℓ 625 2 3 4 +0.562927(log ℓ) + 0.0848911(log ℓ) + (log ℓ) , 13824 with coefficients listed in Tables 4 and 5. The NTK variance tensors are 16 1 16 5 (ℓ ) 2 3 4 𝐴 = + − − 19.7601 log ℓ + 3.98303(log ℓ) − 3.2746(log ℓ) − (log ℓ) , 3ℓ ℓ 2 3 54 58

(160)

(161)

(162)

𝑐 𝑖,𝐷𝑗

𝑝𝐷 2

0 − 43

𝑖\ 𝑗 0 1

4 3

1 0 −3.12564

2 0 2.17817

3 0

4 0 0

5 54

5 0 0

Table 4: Coefficients of the large-ℓ expansion for the tensor 𝐷 (ℓ ) . 𝑐 𝑖,𝐹 𝑗

𝑝𝐹

𝑖\ 𝑗 0 1 2 3

0 − 12 0.416062 −6.41357 6.49751

1 0 −0.303872 1.16879 22.7949

2 0 5 48

−0.154146 0.562927

3 0 0 25 288

4 0 0 0

0.0848911

625 13824

5 0 0 0 0

Table 5: Coefficients of the large-ℓ expansion for the tensor 𝐹 (ℓ ) . 𝐵 (ℓ ) =

9 5 1 + 2 7.19961 − 10.773 log ℓ − 0.0517672(log ℓ) 2 − (log ℓ) 3 2ℓ ℓ 12 1 + 3 (−36.2955 − 12.3472 log ℓ ℓ 125 2 3 4 −12.4116(log ℓ) − 0.55801(log ℓ) − (log ℓ) 288 1 + 4 (24.5959 − 70.6998 log ℓ ℓ 125 2 3 4 5 −14.6047(log ℓ) − 10.5449(log ℓ) − 0.646109(log ℓ) − (log ℓ) , 384

(163)

with coefficients reported in Tables 6 and 7. The dNTK tensors read 5 3 1 2 (ℓ ) −0.163431 − 0.286616 log ℓ + (log ℓ) 𝑃 =− + 4 ℓ 16 1 + 2 (−38.8818 + 19.1203 log ℓ ℓ 25 2 3 4 (log ℓ) −2.44277(log ℓ) + 0.713296(log ℓ) − 576 1 + 3 (39.7952 + 130.51 log ℓ ℓ 55 5 +10.9595(log ℓ) + 0.63484(log ℓ) + 0.508496(log ℓ) − (log ℓ) , 3456 17 1 5 𝑄 (ℓ ) = − + 2.97701 − 0.781566 log ℓ + (log ℓ) 2 12 ℓ 8 1 + 2 (−48.596 + 9.80987 log ℓ ℓ 25 −1.86301(log ℓ) 2 + 0.636662(log ℓ) 3 − (log ℓ) 4 288 1 + 3 (47.9186 + 108.564 log ℓ ℓ 305 2 3 4 5 +63.8277(log ℓ) − 0.168637(log ℓ) + 0.33582(log ℓ) − (log ℓ) , 3456 2

(164)

(165)

𝑐 𝑖,𝐴 𝑗

𝑝𝐴 1

𝑖\ 𝑗 0 1

0 16 3 − 16 3

1 0 −19.7601

2 0 3.98303

3 0 −3.2746

4 0 5 − 54

5 0 0

Table 6: Coefficients of the large-ℓ expansion for the tensor 𝐴 (ℓ ) . 𝑐 𝑖,𝐵 𝑗

𝑝𝐵

𝑖\ 𝑗 0 1 2 3

1 0 −10.773 −12.3472 −70.6998

9 2

7.19961 −36.2955 24.5959

2 0 −0.0517672 −12.4116 −14.6047

3 0 5 − 12 −0.55801 −10.5449

4 0 0 125 − 288 −0.646109

5 0 0 0 − 125 384

Table 7: Coefficients of the large-ℓ expansion for the tensor 𝐵 (ℓ ) . with coefficients given in Tables 8 and 9. The ddI NTK tensor is 𝑅 (ℓ ) = −6.84299 − 1.75 ℓ + 0.181818 log ℓ + 0.9375(log ℓ) 2 1 + (57.7277 − 10.7504 log ℓ ℓ +7.72305(log ℓ) 2 + 0.604921(log ℓ) 3 − 0.173611(log ℓ) 4 +

1 (−49.1347 − 77.9822 log ℓ ℓ2 −101.16(log ℓ) 2 + 3.35098(log ℓ) 3 − 0.034582(log ℓ) 4 − 0.367605(log ℓ) 5 ,

(166)

with coefficients listed in Table 10. The ddII NTK tensors are 𝑆 (ℓ ) = 5.46299 + +

3 5 ℓ − 0.0258836 log ℓ − (log ℓ) 2 4 16

1 (10.2607 − 11.0739 log ℓ ℓ 25 −1.66866(log ℓ) − 0.583088(log ℓ) + (log ℓ) 4 576 2

1 (−16.4737 + 39.1868 log ℓ ℓ2 335 5 −31.861(log ℓ) − 0.309359(log ℓ) − 0.592631(log ℓ) + (log ℓ) , 3456 2

(167)

𝑇 (ℓ ) = −17.1701 + 1.88889 ℓ + 2.56748 log ℓ − 1.84028(log ℓ) 2 1 + (387.871 − 114.126 log ℓ ℓ +34.7139(log ℓ) 2 − 5.58537(log ℓ) 3 + 0.868056(log ℓ) 4 +

1 (−372.59 − 1045.22 log ℓ ℓ2

−256.202(log ℓ) 2 − 88.4311(log ℓ) 3 − 0.609713(log ℓ) 4 − 0.0751136(log ℓ) 5 , (168) 27 1 45 𝑈 (ℓ ) = + −26.2564 + 5.39205 log ℓ − (log ℓ) 2 8 ℓ 16 60

𝑐 𝑖,𝑃 𝑗

𝑝𝑃

0 − 34 −0.163431 −38.8818 39.7952

𝑖\ 𝑗 0 1 2 3

1 0 −0.286616 19.1203 130.51

2 0 5 16

−2.44277 10.9595

3 0 0 0.713296 0.63484

4 0 0 25 − 576 0.508496

5 0 0 0 55 − 3456

Table 8: Coefficients of the large-ℓ expansion for the tensor 𝑃 (ℓ ) . 𝑐 𝑖,𝑄𝑗

𝑝𝑄

𝑖\ 𝑗 0 1 2 3

0 − 17 12 2.97701 −48.596 47.9186

1 0 −0.781566 9.80987 108.564

2 0 5 8

−1.86301 63.8277

3 0 0 0.636662 −0.168637

4 0 0 25 − 288 0.33582

5 0 0 0 305 − 3456

Table 9: Coefficients of the large-ℓ expansion for the tensor 𝑄 (ℓ ) . +

1 (−196.103 + 108.191 log ℓ ℓ2 5 −3.04052(log ℓ) + 0.995277(log ℓ) + 0.0161773(log ℓ) + (log ℓ) 5 64 2

1 (218.985 − 21.8079 log ℓ ℓ3 +93.3553(log ℓ) 2 − 11.5693(log ℓ) 3 − 0.331592(log ℓ) 4 − 0.409017(log ℓ) 5 ,

(169)

with coefficients given in Tables 11-13. Finally, the sextic vertex reads 𝑉6(ℓ ) =

2 1 (2.85353 + 2.5 log ℓ) . + ℓ3 ℓ4

(170)

with coefficients reported in Table 14.

Experimental setup

I.1

Stability analysis

In this section, we provide details of the experiments in Section 5 and present additional results validating the stability at criticality of the NNGP 𝐾, the four-point vertex 𝑉4 , and the NTK tensors 𝐷, 𝐹, 𝐴, and 𝐵 as the network depth ℓ increases. We estimate the preactivation and NTK tensors via Monte Carlo sampling. For this purpose, individual networks are initialized in JAX using neural-tangents stax layers. Computing NTK-related quantities 𝜕𝑧

(ℓ)

(𝑥 )

requires the Jacobian 𝑖𝜕𝜃𝜇 𝛼 , which we obtain via JAX’s automatic differentiation. Since forming the full Jacobian is computationally and memory intensive for wide networks, we instead compute layer-wise statistics sequentially and propagate them to deeper layers using the chain rule. Sampling across initializations is parallelized with vmap. Orthogonal weight initialization is implemented using JAX’s orthogonal initializer. As tanh is not directly available in this setup, we approximate it using ElementwiseNumerical with degree 80. 61

𝑐 𝑖,𝑅 𝑗

𝑝𝑅

−1

𝑖\ 𝑗 0 1 2 3

0 − 74 −6.84299 57.7277 −49.1347

1 0 0.181818 −10.7504 −77.9822

2 0 0.9375 7.72305 −101.16

3 0 0 0.604921 3.35098

4 0 0 −0.173611 −0.034582

5 0 0 0 −0.367605

Table 10: Coefficients of the large-ℓ expansion for the tensor 𝑅 (ℓ ) . 𝑐 𝑖,𝑆 𝑗

𝑝𝑆

−1

𝑖\ 𝑗 0 1 2 3

0 3 4

5.46299 10.2607 −16.4737

1 0 −0.0258836 −11.0739 39.1868

2 0 5 − 16 −1.66866 −31.861

3 0 0 −0.583088 −0.309359

4 0 0 25 576

−0.592631

5 0 0 0 335 3456

Table 11: Coefficients of the large-ℓ expansion for the tensor 𝑆 (ℓ ) . All experiments are conducted on tanh MLPs without biases. Inputs are drawn with i.i.d. standard normal components, resulting in 0.934738 0.377754 0.254697 0.256718 0.178558 ª © ª © ª © ª © ª © 0.26696 ® 0.127474 ® 0.560475 ® 0.869761 ® 0.411167 ® ® ® ® ® ® 0.784097 ® 0.879907 ® 0.508664 ® 0.0406222® 0.100846 ® ® ® ® ® ® 0.656448 ® 0.710555 ® 0.0271565® 0.431362 ® 0.220264 ® ® ® ® ® ® 0.305308 ® 0.509949 ® 0.426426 ® 0.906228 ® 0.215917 ® ® ® ® ® ® 𝑥0 = ® ⊕ 0.312682 ® ⊕ 0.457646 ® ⊕ 0.55979 ® ⊕ 0.490943 ® , 0.401958 ® ® ® ® ® 0.894594 ® 0.0854376® 0.913778 ® 0.275852 ® 0.596323 ® ® ® ® ® ® 0.0559893 ® 0.869372 ® 0.40436 ® 0.553722 ® 0.0799147® ® ® ® ® ® 0.000643274® 0.114232 ® 0.407187 ® 0.235762 ® 0.205998 ® ® ® ® ® ® 0.0274513 0.0851646 0.0644401 0.751627 0.0372218 ¬ « ¬ « ¬ « ¬ « ¬ «

(171)

0.752504 0.23285 0.815355 0.193679 0.986304 ª ª © ª © ª © © ª © 0.396331 ® 0.632048® 0.475999 ® 0.183045® 0.0212271® ® ® ® ® ® 0.829442 ® 0.2954 ® 0.493653 ® 0.92565 ® 0.805301 ® ® ® ® ® ® 0.163461 ® 0.12661 ® 0.267819 ® 0.199642® 0.696779 ® ® ® ® ® ® 0.0583318® 0.440952® 0.0133166® 0.38384 ® 0.0208987® ® ® ® ® ® 𝑥1 = ® ⊕ 0.949114® ⊕ 0.814708 ® ⊕ 0.184987® ⊕ 0.71065 ® , 0.0385021 ® ® ® ® ® 0.885504 ® 0.824362® 0.315126 ® 0.518954® 0.335234 ® ® ® ® ® ® 0.160447 ® 0.373311® 0.47199 ® 0.078869® 0.908037 ® ® ® ® ® ® 0.583711 ® 0.386399® 0.992467 ® 0.456603® 0.94978 ® ® ® ® ® ® 0.318431 0.333712 0.570161 0.101208 0.389812 ¬ ¬ « ¬ « ¬ « ¬ « «

(172)

𝑐𝑇𝑖, 𝑗

𝑝𝑇 𝑖\ 𝑗 0 1 2 3

−1

0 1.88889 −17.1701 387.871 −372.59

1 0 2.56748 −114.126 −1045.22

2 0 −1.84028 34.7139 −256.202

3 0 0 −5.58537 −88.4311

4 0 0 0.868056 −0.609713

5 0 0 0 −0.0751136

Table 12: Coefficients of the large-ℓ expansion for the tensor 𝑇 (ℓ ) . 𝑐𝑈 𝑖, 𝑗

𝑝𝑈

𝑖\ 𝑗 0 1 2 3

0 27 8

−26.2564 −196.103 218.985

1 0 5.39205 108.191 −21.8079

2 0 − 45 16 −3.04052 93.3553

3 0 0 0.995277 −11.5693

4 0 0 0.0161773 −0.331592

5 0 0 5 64

−0.409017

Table 13: Coefficients of the large-ℓ expansion for the tensor 𝑈 (ℓ ) .

0.268954 0.0628893 0.631468 0.901974 0.345452 ª © ª © ª © ª © ª © 0.486888® 0.679924 ® 0.779267 ® 0.196523 ® 0.888144® ® ® ® ® ® 0.22212 ® 0.792366 ® 0.0763499® 0.939642 ® 0.360429® ® ® ® ® ® 0.915653® 0.986086 ® 0.669761 ® 0.794076 ® 0.207685® ® ® ® ® ® 0.795563® 0.936489 ® 0.155254 ® 0.225599 ® 0.600649® ® ® ® ®, ® 𝑥2 = ®⊕ ®⊕ ®⊕ ®⊕ ® 0.797374® 0.273049 ® 0.343043 ® 0.66191 ® 0.377979® 0.826671® 0.360604 ® 0.907627 ® 0.912069 ® 0.827548® ® ® ® ® ® 0.11106 ® 0.921974 ® 0.0647726® 0.0218007® 0.871934® ® ® ® ® ® 0.853911® 0.820319 ® 0.0988953® 0.303759 ® 0.364661® ® ® ® ® ® 0.298177 0.53683 0.761094 0.0650933 0.384977 ¬ « ¬ « ¬ « ¬ « ¬ «

(173)

0.54987 0.810621 0.814678 0.773795 0.0682295 ª ª © ª © ª © © ª © 0.140096 ® 0.604593 ® 0.0561101® 0.101687® 0.0049892® ® ® ® ® ® 0.340508 ® 0.842464 ® 0.193425 ® 0.224264® 0.133457 ® ® ® ® ® ® 0.359852 ® 0.882483 ® 0.413272 ® 0.370917® 0.778419 ® ® ® ® ® ® 0.613812 ® 0.93913 ® 0.622813 ® 0.731496® 0.169455 ® ®. ® ® ® ® 𝑥3 = ® ®⊕ ®⊕ ®⊕ ®⊕ 0.19288 ® 0.872499 ® 0.668977 ® 0.234507® 0.0719057® 0.134531 ® 0.317102 ® 0.771141 ® 0.906523® 0.1728 ® ® ® ® ® ® 0.49299 ® 0.289739 ® 0.884678 ® 0.431445® 0.906299 ® ® ® ® ® ® 0.591431 ® 0.0178811® 0.516841 ® 0.115241® 0.322143 ® ® ® ® ® ® 0.847235 0.275449 0.54484 0.160385 0.893789 ¬ « ¬ ¬ « ¬ « ¬ « «

(174)

For the two-dimensional tensors 𝐾 and Θ, computations are restricted to the input pair (𝑥0 , 𝑥1 ). Monte Carlo estimation of the kernels. We consider an ensemble of 𝑁net neural networks with orthogonal weight initialization. The empirical NNGP and NTK are estimated as (ℓ )

𝐾 𝛼𝛽 =

𝑁net 1 ∑︁ 𝑧 (ℓ ) 𝑧 (ℓ ) , 𝑁net 𝐼=1 𝐼;𝑖, 𝛼 𝐼;𝑖,𝛽

(175)

6 𝑐𝑉 𝑖, 𝑗

𝑝 𝑉6 3

0 2 2.85353

𝑖\ 𝑗 0 1

1 0 2.5

2 0 0

3 0 0

4 0 0

5 0 0

Table 14: Coefficients of the large-ℓ expansion for the tensor 𝑉6(ℓ ) .

CW( ) = 0.25

CW( ) = 1.00

CW( ) = 4.00

10 3

|K |

10 6

100

10 1

10 9 10 12

( , ) = (0, 0) ( , ) = (0, 1)

10 15 10 18

10 2 101

100

101

100

101

Figure 11: Variance stability. Components of the Monte Carlo estimate of the NNGP 𝐾 𝛼𝛽 for a tanh (ℓ ) MLP, shown as a function of layer depth ℓ for identical and distinct inputs, across three values of 𝐶𝑊 . Hidden layers have width 50. Means are computed over 600 initializations for both non-critical (left, right) and critical (middle) cases. Error bars are shown in all panels (see text).

(ℓ ) Θ 𝛼𝛽 =

(176)

where 𝑖 denotes a fixed channel and 𝐼 indexes network initializations. Error bars indicate standard errors, √ computed as the sample standard deviation divided by 𝑁net . Variance stability. Figure 11 demonstrates variance stability under the same setup used for the gradient analysis in Figure 1. At the critical value 𝐶𝑊 = 1, both the orthogonal NNGP and NTK remain stable. While variance stability has been studied numerically for Gaussian initializations in [17], our results show that the corresponding criticality conditions transfer directly to the orthogonal setting. Furthermore, at criticality, the empirical NNGP remains stable beyond the perturbative regime ℓ < 𝑛. In Figure 12 (a), we compare its diagonal component with both the single-input exact solution (Section F.1) and the large-ℓ expansion (Section H). The Monte Carlo estimate agrees quantitatively with the exact solution across all depths, and matches the large-ℓ expansion at large ℓ. This confirms the stability of the orthogonal NNGP in the regime ℓ > 𝑛, consistent with the observations of [25]. Monte–Carlo estimation of the four-point cumulant 𝑉4 . We estimate the quartic vertex 𝑉4 to leading order as 𝑛ℓ ª ∑︁ 1 𝑛ℓ −1 (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) (ℓ ) ® 𝑧 𝐼;𝑖, 𝛼 𝑧 𝐼;𝑖,𝛽 𝑧 𝐼; 𝑗,𝛾 𝑧 𝐼; 𝑗, 𝛿 ® − 𝑛ℓ −1 𝐾 𝛼𝛽 𝐾 𝛾 𝛿 + O . 𝑁net ® 𝑛 (𝑛 − 1) 𝑖, 𝑗=1 𝑛 𝐼=1 ℓ ℓ 𝑖≠ 𝑗 « ¬ 𝑁net © 1 ∑︁

(ℓ ) 𝑉 𝛼𝛽𝛾 𝛿 =

(177)

where 𝐾 is defined in (175). Here we exploit channel symmetry and estimate (ℓ )

𝐾 𝛼𝛽 =

𝑛ℓ 𝑁net 1 ∑︁ 1 ∑︁ 𝑧 (ℓ ) 𝑧 (ℓ ) . 𝑁net 𝐼=1 𝑛ℓ 𝑖=1 𝐼;𝑖, 𝛼 𝐼;𝑖,𝛽

(178)

Experimental

0.00

0.5

0.05 0.10

0.3

0.2

V( )

K( )

0.4 >n

0.15

0.20

0.1 0.0

Large- Expansion

Prediction

0.25 0

0.30

100

(a)

100

(b)

Figure 12: Stability beyond the perturbative regime. Comparison of the diagonal components of the Monte Carlo estimate, single-input exact solution, and large-ℓ expansion for the NNGP 𝐾 and quartic vertex 𝑉 in a tanh MLP with orthogonal initialization. Hidden layers have width 50; means are computed over 600 initializations. (a) The NNGP estimates are in quantitative agreement with the exact solution at both small and large depths. The large-ℓ expansion is inaccurate at small ℓ, as expected, but becomes accurate after a few layers. Stability persists up to ℓ = 100, well beyond the perturbative regime ℓ < 𝑛. (b) The quartic vertex 𝑉 exhibits analogous scaling behavior. To estimate uncertainties, we repeat the Monte Carlo procedure 𝑁stats times and report the mean and standard deviation. Results up to ℓ = 30 are shown in Figures 13 (critical) and 14 (non-critical). In the critical case, fitted asymptotic power laws confirm the stability of 𝑉4 . Moreover, at criticality, the quartic vertex 𝑉 remains well-behaved beyond the perturbative regime ℓ < 𝑛. In Figure 12 (b), we compare its diagonal component with the single-input exact solution (Section F.1) and the large-ℓ expansion (Section H). The Monte Carlo estimates closely track the exact solution across all depths and align with the asymptotic expansion at large ℓ. This provides further evidence for the stability of orthogonal networks in the regime ℓ > 𝑛, consistent with [25]. Monte–Carlo estimation of the tensors 𝐷, 𝐹, 𝐴 and 𝐵. Similarly to 𝑉, these tensors are estimated to leading order as 𝑛ℓ 𝑁net 𝑛ℓ −1 ∑︁ 1 ∑︁ 1 (ℓ ) (ℓ ) c (ℓ ) 𝑧 𝑧 ΔΘ +O 𝑁net 𝐼=1 𝑛ℓ2 𝑖, 𝑗=1 𝐼;𝑖, 𝛼 𝐼;𝑖,𝛽 𝐼; 𝑗 𝑗,𝛾 𝛿 𝑛

(179)

𝐹 𝛼𝛾𝛽 𝛿 =

𝑛ℓ 𝑁net 𝑛ℓ −1 ∑︁ 1 1 ∑︁ (ℓ ) (ℓ ) c (ℓ ) 𝑧 𝑧 ΔΘ + O 𝑁net 𝐼=1 𝑛ℓ2 𝑖, 𝑗=1 𝐼;𝑖, 𝛼 𝐼; 𝑗,𝛽 𝐼;𝑖 𝑗,𝛾 𝛿 𝑛

(180)

(ℓ ) 𝐴 𝛼𝛽𝛾 𝛿 =

𝑛ℓ 𝑁net (ℓ ) (ℓ ) 𝑛ℓ −1 ∑︁ 1 ∑︁ c 𝐼;𝑖𝑖, 𝛼𝛽 ΔΘ c 𝐼; 𝑗 𝑗,𝛾 𝛿 + O 1 ΔΘ 2 𝑁net 𝐼=1 𝑛ℓ 𝑖, 𝑗=1 𝑛

(181)

(ℓ ) 𝐵 𝛼𝛾𝛽 𝛿 =

𝑛ℓ 𝑁net (ℓ ) (ℓ ) 1 ∑︁ 𝑛ℓ −1 ∑︁ c 𝐼;𝑖 𝑗, 𝛼𝛽 ΔΘ c 𝐼;𝑖 𝑗,𝛾 𝛿 + O 1 , ΔΘ 2 𝑁net 𝐼=1 𝑛ℓ 𝑖, 𝑗=1 𝑛

(182)

(ℓ ) 𝐷 𝛼𝛽𝛾 𝛿 =

(ℓ )

c 𝑖(ℓ𝑗,)𝛼𝛽 was introduced in Section 3.1. The computation is repeated 𝑁stats where the NTK fluctuation ΔΘ times to estimate the mean and standard deviation. For each tensor, both critical and non-critical cases are evaluated using the same input configurations as for 𝑉, up to ℓ = 30. Results for 𝐷, 𝐹, 𝐴, and 𝐵 are shown in Figures 15–22. In the critical case, regressions of the asymptotic power laws (shown in orange) 65

CW(`) = 1.0

10 2

10 3

10 3 y = 0.2` −1.92 `start = 10

10 1

y = 0.19` −1.97 `start = 10

10 1

|V 0123 |

|V 0103 |

|V 0023 |

10 3

10 3 100

101

y = 0.19` −1.96 `start = 10

10 1 10 2

10 2

y = 0.21` −1.96 `start = 10

10 1 10 2

|V 0101 |

|V 0000 |

10 1

y = 0.25` −1.92 `start = 10

10 1

|V 0022 |

y = 0.34` −1.89 `start = 10

100

101

100

101

Figure 13: Stability of the four-point cumulant 𝑉4 at criticality. Selected components of the Monte Carlo (ℓ ) estimate 𝑉 4 for a tanh MLP are shown as a function of layer depth ℓ at the critical value 𝐶𝑊 = 1. Hidden layers have width 50. An asymptotic power-law fit is shown in orange, with the fit starting at ℓstart . Estimates are obtained from 𝑁net = 600 initializations, with means and error bars computed over 𝑁stats = 10 repetitions (see text). confirm the expected power-law behavior. This provides consistent evidence that the NTK tensors 𝐷, 𝐹, 𝐴, and 𝐵 are stabilized under the same criticality conditions as in the infinite-width limit. Similarly, at criticality, the NTK tensors remain well-behaved beyond the perturbative regime ℓ < 𝑛. In Figure 23, we compare the diagonal components of the tensors D and F with their single-input exact solutions (Section F.1) and large-ℓ expansions (Section H). The Monte Carlo estimates closely track the exact solutions across all depths and align with the asymptotic expansions at large ℓ. This provides further evidence for the stability of orthogonal networks in the regime ℓ > 𝑛, consistent with [25].

I.2

Compute resources

All experiments were conducted on a laptop equipped with an AMD Radeon Pro 5500M (4 GB VRAM). Computations were performed on a single GPU. The stability analysis for the NNGP and 𝑉4 was completed within several hours for all three values of 𝐶𝑊 . Gradient-dependent tensors are more computationally demanding; accordingly, the layer width was reduced to 50, and each tensor (for three values of 𝐶𝑊 ) required up to O (10) hours to compute.

CW(`) = 0.25

CW(`) = 4.00 101

10 5

100

10 10

10 1

|V αβγδ |

10 15

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

10 20 10 25 10 30 10 35 100

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

10 2 10 3 10 4 10 5

101

100

101

Figure 14: Instability of the four-point cumulant 𝑉4 away from criticality.Selected components of the (ℓ ) (ℓ ) Monte Carlo estimate 𝑉 4 are shown for 𝐶𝑊 < 1 (left) and 𝐶𝑊 > 1 (right). Estimates are computed from 𝑁net = 600 initializations, with means and error bars obtained from 𝑁stats = 10 repetitions (see text).

CW(`) = 1.0 10 3 |D 0022 |

|D 0101 |

|D 0000 |

10 2 10 3

10 3

10 4

10 2

10 1

10 2

10 4

y = 0.064` −1.45 `start = 10

10 5 10 6

10 5 y = 3` −3.46 `start = 10

10 6

y = 0.023` −1.02 `start = 10

10 2

|D 0023 |

|D 0103 |

|D 0123 |

10 3 10 4

10 3

10 4 10 5

y = 0.0011` −0.68 `start = 10

y = 0.011` −0.83 `start = 10

101

10 4

y = 0.017` −1.02 `start = 10

101

Figure 15: Stability of the 𝐷 tensor at criticality. Selected components of the Monte Carlo estimate 𝐷 (ℓ ) for a tanh MLP are shown as a function of layer depth ℓ at the critical value 𝐶𝑊 = 1. Hidden layers have width 50. An asymptotic power-law fit is shown in orange, with the fit starting at ℓstart . Estimates are obtained from 𝑁net = 600 initializations, with means and error bars computed over 𝑁stats = 10 repetitions (see text).

CW(`) = 0.25

CW(`) = 4.00 105

10 6

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

104

10 10

103

10 14

102

|D αβγδ |

10 18 10 22

101

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

10 26 10 30 10 34 100

100 10 1 10 2 10 3

101

100

101

Figure 16: Instability of the 𝐷 tensor away from criticality. Selected components of the Monte Carlo (ℓ ) (ℓ ) estimate 𝐷 are shown for 𝐶𝑊 < 1 (left) and 𝐶𝑊 > 1 (right). Estimates are computed from 𝑁net = 600 initializations, with means and error bars obtained from 𝑁stats = 10 repetitions (see text).

CW(`) = 1.0 10 1

y = 0.37` −0.89 `start = 10

10 2

|F 0022 |

|F 0101 |

|F 0000 |

10 1

y = 0.12` −0.88 `start = 10

10 2

y = 0.26` −0.95 `start = 10

10 2

y = 0.12` −0.87 `start = 10

10 2

|F 0123 |

|F 0023 |

|F 0103 |

10 1

y = 0.22` −0.93 `start = 10

101

10 2

y = 0.17` −0.92 `start = 10

101

Figure 17: Stability of the 𝐹 tensor at criticality. Selected components of the Monte Carlo estimate 𝐹 for (ℓ ) a tanh MLP are shown as a function of layer depth ℓ at the critical value 𝐶𝑊 = 1. Hidden layers have width 50. An asymptotic power-law fit is shown in orange, with the fit starting at ℓstart . Estimates are obtained from 𝑁net = 600 initializations, with means and error bars computed over 𝑁stats = 10 repetitions (see text).

CW(`) = 0.25

CW(`) = 4.00 105

10 4

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

104

10 9

103

10 14 |F αβγδ |

102

10 19

101

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

10 24 10 29 10 34 100

100 10 1 10 2 10 3

101

100

101

Figure 18: Instability of the 𝐹 tensor away from criticality. Selected components of the Monte Carlo (ℓ ) (ℓ ) estimate 𝐹 are shown for 𝐶𝑊 < 1 (left) and 𝐶𝑊 > 1 (right). Estimates are computed from 𝑁net = 600 initializations, with means and error bars obtained from 𝑁stats = 10 repetitions (see text).

CW(`) = 1.0 10 1

|A 0022 |

|A 0101 |

|A 0000 |

10 2

10 2 10 2

y = 0.096` −0.08 `start = 10

y = 0.054` −0.06 `start = 10

10 2

|A 0123 |

10 2

|A 0023 |

|A 0103 |

10 2

y = 0.024` −0.16 `start = 10

y = 0.027` −0.11 `start = 10

y = 0.027` −0.53 `start = 10

y = 0.028` −0.66 `start = 10

101

Figure 19: Stability of the 𝐴 tensor at criticality. Selected components of the Monte Carlo estimate 𝐴 for (ℓ ) a tanh MLP are shown as a function of layer depth ℓ at the critical value 𝐶𝑊 = 1. Hidden layers have width 50. An asymptotic power-law fit is shown in orange, with the fit starting at ℓstart . Estimates are obtained from 𝑁net = 600 initializations, with means and error bars computed over 𝑁stats = 10 repetitions (see text).

CW(`) = 0.25

CW(`) = 4.00

10 5

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

1010

10 9

108

10 17

106

|A αβγδ |

10 13

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

10 21 10 25 10 29 10 33 100

104 102 100 10 2 101

100

101

Figure 20: Instability of the 𝐴 tensor away from criticality. Selected components of the Monte Carlo (ℓ ) (ℓ ) estimate 𝐴 are shown for 𝐶𝑊 < 1 (left) and 𝐶𝑊 > 1 (right). Estimates are computed from 𝑁net = 600 initializations, with means and error bars obtained from 𝑁stats = 10 repetitions (see text).

CW(`) = 1.0 6 × 10 1

4 × 10 1

2 × 10 1

3 × 10 1

|B 0101 |

|B 0000 |

3 × 10 1 2 × 10 1

10 1

y = 0.94` −0.54 `start = 10

|B 0022 |

4 × 10 1

10 1

6 × 10 2 4 × 10 2

2 × 10 1

y = 0.35` −0.56 `start = 10

3 × 10 1

2 × 10 1

10 1

y = 0.65` −0.59 `start = 10

2 × 10 1

10 1

6 × 10 2 4 × 10 2

|B 0123 |

|B 0023 |

|B 0103 |

2 × 10 1

y = 0.33` −0.57 `start = 10

101

6 × 10 2

y = 0.46` −0.57 `start = 10

101

6 × 10 2

y = 0.35` −0.60 `start = 10

101

Figure 21: Stability of the 𝐵 tensor at criticality. Selected components of the Monte Carlo estimate 𝐵 for (ℓ ) a tanh MLP are shown as a function of layer depth ℓ at the critical value 𝐶𝑊 = 1. Hidden layers have width 50. An asymptotic power-law fit is shown in orange, with the fit starting at ℓstart . Estimates are obtained from 𝑁net = 600 initializations, with means and error bars computed over 𝑁stats = 10 repetitions (see text).

CW(`) = 0.25

CW(`) = 4.00

10 4

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

1010

10 8

108

10 12 |B αβγδ |

10 16

106

(α, β, γ, δ) = (0, 0, 0, 0) (α, β, γ, δ) = (0, 1, 0, 1) (α, β, γ, δ) = (0, 0, 2, 2) (α, β, γ, δ) = (0, 1, 0, 3) (α, β, γ, δ) = (0, 0, 2, 3) (α, β, γ, δ) = (0, 1, 2, 3)

10 20 10 24 10 28 10 32 100

104 102 100 101

100

101

Figure 22: Instability of the 𝐵 tensor away from criticality. Selected components of the Monte Carlo (ℓ ) (ℓ ) estimate 𝐵 are shown for 𝐶𝑊 < 1 (left) and 𝐶𝑊 > 1 (right). Estimates are computed from 𝑁net = 600 initializations, with means and error bars obtained from 𝑁stats = 10 repetitions (see text).

0.00 0.05 0.10 0.15 0.20 0.25 0.30

Large- Expansion

Prediction

0.5 0.4 0.3 <n

F( )

D( )

Experimental

0.2

0.1 0.0 0

100

(a)

0.1

100

(b)

Figure 23: Stability beyond the perturbative regime. Comparison of the diagonal components of the Monte Carlo estimate, single-input exact solution, and large-ℓ expansion for the NTK tensors 𝐷 and 𝐹 in a tanh MLP with orthogonal initialization. Hidden layers have width 50; means are computed over 600 initializations. (a) The tensor 𝐷 estimates are in quantitative agreement with the exact solution at both small and large depths. The large-ℓ expansion is inaccurate at small ℓ, as expected, but becomes accurate after a few layers. Stability persists up to ℓ = 100, well beyond the perturbative regime ℓ < 𝑛. (b) The tensor 𝐹 exhibits analogous scaling behavior.

Record · ID 168333 · SHA-256 22bfa76f37fa35ea
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.