arXiv CSopen access

Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics Claudio Nordio Draft research note∗

arXiv:2606.09744v1 [cs.LG] 8 Jun 2026

June 9, 2026

Abstract We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers.

1

Introduction

Gradient descent in neural networks is usually formulated as a dynamical system in weight space. The state of the network is represented by a collection of trainable parameters, while activations, outputs and predictions are treated as derived quantities. This perspective is natural from the optimization viewpoint, but it is not necessarily the most direct description of the learning process. The present work explores an alternative formulation1 . Rather than following the evolution of the weights directly, we ask whether gradient descent can be reformulated in terms of dynamical fields constructed from the activations of the network. More specifically, we investigate whether the learning dynamics admits a closed representation in terms of activation fields and a finite hierarchy of auxiliary fields generated by the training process itself. The motivation originates from the simplest possible setting: a feed-forward ReLU network with a single hidden layer, fixed readout and quadratic loss. In this case, following an NTK-like approach [2], the weight variables can be eliminated in favour of a closed dynamical description of the activations. The resulting evolution is governed by a collective kernel acting directly on the training residuals. This kernel naturally factorizes into two contributions: a geometric component determined by the training inputs and a dynamical component determined by the activation patterns of the network. This observation raises a natural question. Does a similar description survive increasing depth? The answer is not obvious. As additional hidden layers are introduced, new dynamical structures emerge. Nevertheless, the residual dynamics retains a simple layer-wise form. The resulting kernels can be expressed through activation overlaps and conjugate-field correlators, while the explicit dependence on the weights remains highly constrained. ∗

Circulated for discussion and feedback; comments are welcome at [email protected]. This draft research note was prepared independently of the recent preprint by Cha et al., The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks (arXiv:2605.06258v1) [1]. The relationship between the two approaches will be discussed in a subsequent revision. 1

1

The analysis developed in this paper is restricted to feed-forward ReLU networks with fixed readout. Within this setting, we derive residual dynamics for depths ranging from one hidden layer to arbitrary depth. For one hidden layer, the dynamics closes on the activation fields. For two hidden layers, closure requires the introduction of conjugate fields. Starting from three hidden layers, a hierarchy of quadratic operators emerges. These operators take the form 

(ℓ+1) Gαβ ℓ = W

T

Dℓαβ W (ℓ+1) ,

(1)

where 

Dℓαβ = diag aαℓ+1 ⊙ aβℓ+1



(2)

is a co-activation projector. As will be shown, these objects admit a natural geometric interpretation as pullback Gram metrics induced by deeper layers. Some structural properties emerge throughout the derivation. First, increasing depth does not generate arbitrarily high-order collective statistics for the residuals: the residual kernels remain expressible in terms of second-order activation overlaps and fourth-order correlators of activation and conjugate fields. Second, explicit weight dependence never appears beyond quadratic order. Whenever weights re-enter the dynamics, they do so through the Gram operators Gαβ ℓ . Finally, gradient descent can be represented through the coupled evolution of activation fields, conjugate fields and induced pullback metrics2 , n

o

S = uαℓ , bαℓ , Gαβ . ℓ

(3)

The remainder of the paper is devoted to the explicit construction of this dynamical representation.

2

General Setup and Notation

We consider a feed-forward network with L hidden layers and ReLU activations. Training examples are labelled by α = 1, . . . , M.

(4)

The input associated with example α is denoted by uα0 .

(5)

For each hidden layer ℓ = 1, . . . , L, we define the pre-activations vℓα = W (ℓ) uαℓ−1 ,

(6)

and the activations uαℓ = ϕ(vℓα ),

ϕ(x) = max(x, 0).

(7)

The corresponding activation indicators are aαℓ = 1(vℓα > 0).

(8)

Strictly speaking, the operators Gαβ depend on intermediate weights W (ℓ+1) , which are themselves time ℓ integrals of past gradient updates. A strict closure would require augmenting the state with the corresponding gradient variables. Here we highlight the explicit weight dependence entering the field dynamics as confined to the induced quadratic transport operators Gαβ ℓ . 2

2

The output layer is assumed to be fixed throughout the paper. For the deepest hidden layer we define Ŷ α =

1 X α u . N i L,i

(9)

The residual associated with example α is rα = Ŷ α − Y α ,

(10)

M 1 X (rα )2 . M α=1

(11)

and the loss function is L=

The weights evolve according to gradient descent W (ℓ) (t + 1) = W (ℓ) (t) − η

∂L . ∂W (ℓ) (t)

(12)

We use the notation D

1 X α β x y N i i i

E

xα y β =

(13)

for empirical averages over neuron coordinates. Similar layer-wise overlap structures appear in field-theoretic treatments of deep learning theory [5]. In particular, (ℓ)

D

E

Qαβ = uαℓ uβℓ =

1 X α β u u . N i ℓ,i ℓ,i

(14)

We denote by ⊙ the component-wise, or Hadamard, product. For two vectors x and y, (x ⊙ y)i = xi yi .

(15)

Our goal is to progressively replace the weight-space description by a dynamical representation involving three classes of objects: 1. activation fields, uαℓ ,

(16)

bαℓ ,

(17)

Gαβ ℓ .

(18)

2. conjugate fields,

3. induced Gram metrics,

The induced Gram metrics will emerge only from depth three onward and will be shown to take the form 

(ℓ+1) Gαβ ℓ = W

T

Dℓαβ W (ℓ+1) .

(19)

Here the co-activation operator is defined as 



Dℓαβ = diag aαℓ+1 ⊙ aβℓ+1 . Equivalently, in components,

3

(20)

h

Dℓαβ

i ij

= δij aαℓ+1,i aβℓ+1,i .

(21)

Since the ReLU activation indicators satisfy aαℓ+1,i ∈ {0, 1},

(22)

the operator Dℓαβ is symmetric, 

Dℓαβ

T

= Dℓαβ .

(23)

It is also positive semidefinite. Indeed, for every vector x, xT Dℓαβ x =

X

aαℓ+1,i aβℓ+1,i x2i ≥ 0.

(24)

i

Finally, it is idempotent, Dℓαβ

2

= Dℓαβ ,

(25)

aαℓ+1,i aβℓ+1,i

2

= aαℓ+1,i aβℓ+1,i .

(26)



because 

Therefore Dℓαβ is an orthogonal projector onto the subspace of neurons simultaneously active for the pair of examples (α, β). Since Dℓαβ is symmetric and positive semidefinite, the operator Gαβ ℓ defined in Eq. (19) is itself symmetric and positive semidefinite. As will become clear in the following sections, it admits a natural interpretation as the pullback of the co-activation geometry defined on layer ℓ + 1. The remainder of the paper is devoted to the explicit construction of the dynamical system generated by the fields n

o

S = uαℓ , bαℓ , Gαβ . ℓ

3

(27)

One Hidden Layer: Exact Closure of the Activation Dynamics

We begin with the simplest non-trivial architecture, consisting of a single hidden layer with fixed readout. This case serves as the prototype for all subsequent constructions.

3.1

Forward Dynamics

The hidden activations are uαi = ϕ(viα ).

(28)

X

Wij uα0,j .

(29)

aαi = 1(viα > 0).

(30)

with viα =

j

The activation indicators are Since the ReLU activation preserves the sign of the pre-activation, viα > 0 ⇐⇒ uαi > 0,

4

(31)

the activation indicators can equivalently be written as aαi = 1(uαi > 0).

(32)

This observation will be used repeatedly below. Although the pre-activation variables viα still appear in intermediate calculations, the final dynamical equations will depend only on the activation fields uαi . The network output is 1 X α Ŷ α = u , (33) N i i and the residuals are

3.2

rα = Ŷ α − Y α .

(34)

M 1 X (rβ )2 , M β=1

(35)

Gradient of the Loss

Starting from L= we obtain

M ∂L 2 X ∂ Ŷ β = rβ . ∂Wij M β=1 ∂Wij

(36)

∂ Ŷ β 1 ∂uβi = . ∂Wij N ∂Wij

(37)

uβi = ϕ(viβ ),

(38)

∂uβi = aβi uβ0,j . ∂Wij

(39)

M ∂L 2 X = rβ aβi uβ0,j . ∂Wij M N β=1

(40)

Using Eq. (33),

Since we have

Substituting into Eq. (36),

The gradient-descent update therefore becomes ∂L , ∂Wij

(41)

M 2η X rβ aβi uβ0,j . M N β=1

(42)

∆Wij = −η or explicitly ∆Wij = −

5

3.3

Activation Dynamics

The variation of the hidden activation is ∆uαi = aαi ∆viα ,

(43)

X

∆Wij uα0,j .

(44)

M X β 2η X rβ aβi u0,j uα0,j . M N β=1 j

(45)

where ∆viα =

j

Substituting Eq. (42), ∆viα = −

We now introduce the overlap matrix of the input layer, Qαβ =

1 X α β u u . N j 0,j 0,j

(46)

∆viα = −

M 2η X (0) Q aβ rβ . M β=1 αβ i

(47)

M 2η α X (0) Qαβ aβi rβ . ai M β=1

(48)

(0)

Eq. (45) becomes

Substituting into Eq. (43), we obtain ∆uαi = −

Using Eqs. (33) and (34), the residual can be written as rβ =

1 X β u − Y β. N k k

(49)

Substituting Eq. (49) into Eq. (48) gives 2η ∆uαi = − aαi M

M X

!

(0) Qαβ aβi

β=1

1 X β u −Yβ . N k k

(50)

Together with Eq. (32), Eq. (50) depends only on the activation fields uβ and on the fixed targets Y β . Thus the weight variables have disappeared from the final activation dynamics.

3.4

Co-Activation Structure

We define the co-activation overlap (1)

Aαβ =

1 X α β a a . N i i i

(51)

The superscript reminds us that the quantity is associated with the first hidden layer. The collective kernel is then defined as (0) (1) Jαβ = Qαβ Aαβ . (52) The kernel naturally separates into two factors. The first factor, (0)

Qαβ ,

(53)

depends exclusively on the geometry of the training inputs. The second factor, (1)

Aαβ ,

(54)

depends on the activation regions explored during learning and therefore evolves dynamically. 6

3.5

Residual Dynamics

Averaging Eq. (48) over the hidden neurons gives ∆Ŷ α =

1 X ∆uαi . N i

(55)

Substituting Eq. (48), ∆Ŷ

α

M 2η X (0) =− Q M β=1 αβ

!

1 X α β β a a r . N i i i

(56)

Using Eq. (51), ∆Ŷ

α

M 2η X =− Jαβ rβ . M β=1

(57)

∆rα = ∆Ŷ α .

(58)

Since the targets are fixed, Therefore, rα (t + 1) = rα (t) −

M 2η X Jαβ (t)rβ (t). M β=1

(59)

In vector form, 2η J(t) r(t). r(t + 1) = I − M 



(60)

These equations provides a closed dynamics for the residuals.

3.6

Prediction on Unseen Inputs

Consider a new input pattern labelled by γ. Its activation field is 



uγi = ϕ

X

Wij uγ0,j  .

(61)

j

Repeating the previous calculation gives ∆uγi = −

M 2η γ X (0) ai Qγβ aβi rβ , M β=1

(62)

1 X γ β u u . N j 0,j 0,j

(63)

where (0)

Qγβ =

Equivalently, substituting the explicit expression for the residuals, 2η ∆uγi = − aγi M

M X

!

(0) Qγβ aβi

β=1

1 X β u −Yβ . N k k

(64)

The previous equation is driven by the activation fields and targets of the training set and does not require the introduction of additional trainable parameters. Prediction is therefore obtained by augmenting the dynamical system with the activation field associated with the unseen example.

7

3.7

Interpretation

The single-hidden-layer case provides an exact realization of the dynamical reformulation proposed in this work. The weight variables disappear from the final activation dynamics, yielding the closed evolution in Eq. (50). The residual dynamics is governed by the collective kernel (52), which factorizes into an input-geometric overlap Q(0) and a dynamical co-activation overlap A(1) . The next question is whether an analogous closure survives when a second hidden layer is introduced. In the following section we will see that the answer requires the introduction of a new family of dynamical fields.

4

Two Hidden Layers: Emergence of Conjugate Fields

4.1

Forward Dynamics

We consider a network with two hidden layers. The activation fields are uα1 = ϕ W (1) uα0 ,





(65)

uα2 = ϕ W (2) uα1 .





(66)

1 X α u . N i 2,i

(67)

1 X α u − Y α. N i 2,i

(68)

and

The fixed readout is Ŷ α = The residuals are therefore rα =

4.2

Gradient with Respect to W (2)

The gradient with respect to the second-layer weights is ∂L

= (2)

∂Wij

2 X β β β r a2,i u1,j . MN β

(69)

2η X β β β r a2,i u1,j . MN β

(70)

Thus the gradient-descent update is (2)

∆Wij = −

4.3

Dynamics of u2

The variation of the second hidden layer is ∆uα2,i = aα2,i

X

(2)

∆Wij uα1,j .

(71)

j

Substituting Eq. (70), we obtain ∆uα2,i = −

2η α X (1) β β a Q a r , M 2,i β αβ 2,i

where 8

(72)

(1)

Qαβ =

E D 1 X α β u1,j u1,j = uα1 uβ1 . N j

(73)

Using Eq. (68) for the residuals, Eq. (72) can be written as 2η ∆uα2,i = − aα2,i

!

X

M

(1) Qαβ aβ2,i

β

1 X β u −Yβ . N k 2,k

(74)

This is the first occurrence of the overlap matrix Q(1) , associated with the first hidden layer.

4.4

Gradient with Respect to W (1)

We now compute the gradient with respect to the first-layer weights. By the chain rule, ∂L

= (1)

∂Wij

2 X β ∂ Ŷ β r . (1) M β ∂W

(75)

ij

Using the fixed readout, β

∂ Ŷ β

= (1)

∂Wij

1 X ∂u2,k . N k ∂W (1)

(76)

ij

Since !

uβ2,k = ϕ

X

(2) Wkm uβ1,m

,

(77)

= aβ2,k Wki aβ1,i uβ0,j .

(78)

2 X β β β X (2) β Wki a2,k . r a1,i u0,j MN β k

(79)

m

we have ∂uβ2,k (1) ∂Wij

(2)

Substituting into Eq. (75), we obtain ∂L

=

(1) ∂Wij

At this point, the combination X

(2)

Wki aβ2,k

(80)

k

appears naturally. This motivates the definition of the first conjugate field, bβ1,i =

(2)

Wki aβ2,k .

(81)

bβ1 = (W (2) )T aβ2 .

(82)

X k

In vector form,

With this definition, the gradient becomes ∂L

= (1)

∂Wij

2 X β β β β r a1,i b1,i u0,j . MN β

Therefore 9

(83)

(1)

∆Wij = −

4.5

2η X β β β β r a1,i b1,i u0,j . MN β

(84)

Dynamics of u1

The variation of the first hidden layer is ∆uα1,i = aα1,i

X

(1)

∆Wij uα0,j .

(85)

j

Substituting Eq. (84), we obtain ∆uα1,i = −

2η α X (0) β β β a Q a b r . M 1,i β αβ 1,i 1,i

(86)

Using Eq. (68) for the residuals, this becomes 2η ∆uα1,i = − aα1,i

!

X

M

4.6

(0) Qαβ aβ1,i bβ1,i

β

1 X β u −Yβ . N k 2,k

(87)

Dynamics of b1

We now compute the dynamics of the conjugate field. Starting from bα1 = (W (2) )T aα2 ,

(88)

∆bα1 = (∆W (2) )T aα2 + (W (2) )T ∆aα2 .

(89)

we obtain

Neglecting threshold-crossing events at first order in η, ∆aα2 = 0.

(90)

∆bα1 = (∆W (2) )T aα2 .

(91)

Therefore

In components, ∆bα1,i =

X

(2)

∆Wki aα2,k .

(92)

2η X β β X α β r u1,i a2,k a2,k . MN β k

(93)

Aαβ =

1 X α β a a . N k 2,k 2,k

(94)

∆bα1,i = −

2η X (2) β β A u r . M β αβ 1,i

(95)

k

Substituting Eq. (70), we get ∆bα1,i = − Define (2)

Then

10

Using Eq. (68), this can be written explicitly as !

2η ∆bα1,i = −

(2) Aαβ uβ1,i

X

M

β

1 X β u −Yβ . N k 2,k

(96)

Collecting Eqs. (74), (87), and (96), we obtain the closed dynamical system 2η ∆uα2,i = − aα2,i M

!

X

(1) Qαβ aβ2,i

β

2η ∆uα1,i = − aα1,i M

1 X β u −Yβ , N k 2,k !

X

1 X β u −Yβ , N k 2,k

(0) Qαβ aβ1,i bβ1,i

β

(97)

!

2η X (2) β ∆bα1,i = − A u M β αβ 1,i

1 X β u −Yβ . N k 2,k

The right-hand sides depend only on the fields u1 , u2 , b1 and on the fixed targets Y β . Thus the two-hidden-layer network admits a closed description in terms of the state S (2) = {u1 , u2 , b1 }.

4.7

(98)

Residual Dynamics

Averaging the equation for ∆uα2 over the output layer gives ∆rα = −

2η X (2) β K r . M β αβ

(99)

Therefore rα (t + 1) = rα (t) −

2η X (2) K (t)rβ (t). M β αβ

(100)

The two-layer kernel decomposes naturally into a contribution generated by the output layer and a contribution generated by the hidden layer, (2)

[2]

[1]

Kαβ = Kαβ + Kαβ .

(101)

The output-layer contribution is [2]

(1)

(2)

Kαβ = Qαβ Aαβ .

(102)

D E 1 X α β u1,i u1,i = uα1 uβ1 , N i

(103)

1 X α β a a . N k 2,k 2,k

(104)

Here (1)

Qαβ = and

(2)

Aαβ = The hidden-layer contribution is [1]

(0) (1)

Kαβ = Qαβ Sαβ ,

11

(105)

where (1)

Sαβ =

D E 1 X α β α β a1,i a1,i b1,i b1,i = aα1 aβ1 bα1 bβ1 . N i

(106)

Collecting the two contributions yields (2)

(1)

(2)

(0) (1)

Kαβ = Qαβ Aαβ + Qαβ Sαβ .

(107)

which is the first example of the layer-wise kernel decomposition that will reappear at arbitrary depth.

5

Three Hidden Layers

5.1

Forward Dynamics

We consider a network with three hidden layers. The activation fields are uα1 = ϕ W (1) uα0 .





(108)

uα2 = ϕ W (2) uα1 .





(109)

uα3 = ϕ W (3) uα2 .





(110)

1 X α u . N i 3,i

(111)

The network output is Ŷ α =

5.2

Emergence of the Conjugate Fields

Repeating the gradient calculation with respect to W (2) , one naturally encounters the field bα2 = (W (3) )T aα3 .

(112)

Subsequently, the gradient with respect to W (1) generates bα1 = (W (2) )T (aα2 ⊙ bα2 ) .

(113)

This is the natural generalization of the two-layer construction.

5.3

Dynamics of u3

The calculation is identical to the two-layer case. One finds 



X (2) β 2η 1 X β ∆uα3,i = − aα3,i Qαβ a3,i  u3,j − Y β  . M N j β

5.4

(114)

Dynamics of u2

The gradient of W (2) yields 



X (1) β β 2η 1 X β ∆uα2,i = − aα2,i Qαβ a2,i b2,i  u3,j − Y β  . M N j β

12

(115)

5.5

Dynamics of u1

The gradient of W (1) gives 



X (0) β β 1 X β 2η Qαβ a1,i b1,i  u3,j − Y β  . ∆uα1,i = − aα1,i M N j β

(116)

Up to this point, the derivation proceeds exactly as in the two-layer network.

5.6

Dynamics of b2

Starting from bα2 = (W (3) )T aα3 ,

(117)

∆bα2 = (∆W (3) )T aα3 + (W (3) )T ∆aα3 .

(118)

we obtain

Neglecting threshold-crossing events, ∆aα3 = 0,

(119)

∆bα2 = (∆W (3) )T aα3 .

(120)

and therefore

Substituting the gradient update gives 



2η X (3) β  1 X β ∆bα2,i = − u − Y β . A u M β αβ 2,i N j 3,j

(121)

This expression is completely analogous to the corresponding equation in the two-layer case.

5.7

Dynamics of b1

A genuinely new phenomenon appears at this stage. Starting from bα1 = (W (2) )T (aα2 ⊙ bα2 ) ,

(122)

∆bα1 = (∆W (2) )T (aα2 ⊙ bα2 ) + (W (2) )T ∆ (aα2 ⊙ bα2 ) .

(123)

we obtain

Neglecting again threshold-crossing events, ∆aα2 = 0,

(124)

∆bα1 = (∆W (2) )T (aα2 ⊙ bα2 ) + (W (2) )T (aα2 ⊙ ∆bα2 ) .

(125)

which leaves

The first term can be rewritten in terms of the fields already introduced. The second term requires the substitution of Eq. (121): 



 1 X  2η X (3) (W (2) )T (aα2 ⊙ ∆bα2 ) = − Aαβ (W (2) )T aα2 ⊙ uβ2  uβ − Y β  . M β N j 3,j

13

(126)

At this point we use 



(127)



(128)

uβ2 = aβ2 ⊙ W (2) uβ1 . It follows that 

aα2 ⊙ uβ2 = diag aα2 ⊙ aβ2 W (2) uβ1 . We therefore define 



D2αβ = diag aα2 ⊙ aβ2 .

(129)

Substituting into Eq. (126) yields 



(W (2) )T aα2 ⊙ uβ2 = (W (2) )T D2αβ W (2) uβ1 .

(130)

At this point the closure mechanism of the two-layer network fails. The quantity (W (2) )T D2αβ W (2) cannot be expressed solely in terms of the activation fields u1 , u2 , u3 and the conjugate fields b1 , b2 . Instead, the dynamics generates a new quadratic operator involving the weights. We therefore introduce (2) T αβ ) D2 W (2) . Gαβ 1 = (W

(131)

(D2αβ )T = D2αβ ,

(132)

D2αβ ⪰ 0,

(133)

(D2αβ )2 = D2αβ ,

(134)

Since

and

the operator D2αβ is an orthogonal projector. Consequently, αβ T (Gαβ 1 ) = G1 ,

(135)

Gαβ 1 ⪰ 0.

(136)

and

The operator Gαβ 1 can therefore be interpreted as a pullback Gram metric induced by the co-activation geometry of the second hidden layer.

14





X (2) β 2η 1 X β ∆uα3,i = − aα3,i Qαβ a3,i  u3,j − Y β  , M N j β

2η ∆uα2,i = − aα2,i M









1 X β (1) Qαβ aβ2,i bβ2,i  u3,j − Y β  , N j β

X

X (0) β β 2η 1 X β ∆uα1,i = − aα1,i Qαβ a1,i b1,i  u3,j − Y β  , M N j β

∆bα2,i = −

2η X M





(3) β  1 X β

Aαβ u2,i

β

N

(137)

u3,j − Y β  ,

j





2η X (3) αβ β  1 X β ∆bα1,i = − A G u u − Y β . M β αβ 1 1,i N j 3,j S (3) = {u1 , u2 , u3 , b1 , b2 , G1 }.

(138)

Since all residuals are functions of the deepest activation fields, the equations above define the dynamics in terms of the stated variables.

6

Four Hidden Layers: Emergence of a Hierarchy of Pullback Metrics

6.1

Forward Dynamics

We introduce the activation fields uα1 ,

uα2 ,

uα3 ,

Ŷ α =

1 X α u . N i 4,i

uα4 .

(139)

The network output is

6.2

(140)

Conjugate Fields

Repeating the construction of the previous section, we obtain bα3 = (W (4) )T aα4 ,

(141)

bα2 = (W (3) )T (aα3 ⊙ bα3 ) ,

(142)

bα1 = (W (2) )T (aα2 ⊙ bα2 ) .

(143)

and

At this point the recursive structure of the conjugate fields is already apparent.

15

6.3

Dynamics of the Activation Fields

As in the previous sections, the activation dynamics is 



X (3) β 1 X β 2η Qαβ a4,i  u4,j − Y β  , ∆uα4,i = − aα4,i M N j β 



X (2) β β 2η 1 X β ∆uα3,i = − aα3,i Qαβ a3,i b3,i  u4,j − Y β  , M N j β 

(145)



X (1) β β 1 X β 2η Qαβ a2,i b2,i  u4,j − Y β  , ∆uα2,i = − aα2,i M N j β 

(146)



X (0) β β 2η 1 X β ∆uα1,i = − aα1,i Qαβ a1,i b1,i  u4,j − Y β  . M N j β

6.4

(144)

(147)

Dynamics of b3

Exactly as in the previous section, one finds 



2η X (4) β  1 X β ∆bα3,i = − u − Y β . A u M β αβ 3,i N j 4,j

(148)

This equation is already closed.

6.5

Dynamics of b2

Starting from bα2 = (W (3) )T (aα3 ⊙ bα3 ) ,

(149)

∆bα2 .

(150)

we compute

As in the previous section, a crucial contribution arises from (W (3) )T (aα3 ⊙ ∆bα3 ) .

(151)

Substituting Eq. (148), one encounters terms of the form (W (3) )T aα3 ⊙ uβ3 .





(152)

uβ3 = aβ3 ⊙ W (3) uβ2 ,

(153)

(W (3) )T D3αβ W (3) uβ2 ,

(154)



(155)

Using

we obtain

where 

D3αβ = diag aα3 ⊙ aβ3 .

16

This naturally motivates the definition (3) T αβ Gαβ ) D3 W (3) . 2 = (W

(156)

The evolution of b2 therefore contains contributions of the form 



X β β 1 Gαβ u − Y β . 2 u2 N j 4,j

(157)

The operator G2 is the direct analogue of the metric G1 introduced in the three-layer network.

6.6

Dynamics of b1

We now consider bα1 = (W (2) )T (aα2 ⊙ bα2 ) .

(158)

∆bα1

(159)

Computing

produces two qualitatively distinct contributions. The first is identical to the mechanism already encountered in the three-layer case and generates (2) T αβ Gαβ ) D2 W (2) . 1 = (W

(160)

The second originates from the newly generated dynamics of b2 , which itself contains the operator Gαβ 2 .

(161)

Consequently, the evolution of b1 depends simultaneously on both pullback metrics, ∆b1 = F (u1 , G1 , G2 ) .

(162)

The closure of the dynamics therefore requires both operators.

6.7

Closed Dynamical State

Collecting Eqs. (144)–(162), and using Eq. (34) to express the residuals in terms of the deepest activation field, we obtain a closed dynamical system depending on u1 , u2 , u3 , u4 ,

b1 , b2 , b3 ,

G1 , G2 .

The corresponding state is S (4) = {u1 , u2 , u3 , u4 , b1 , b2 , b3 , G1 , G2 }.

(163)

The appearance of G2 shows that the pullback metric introduced in the three-layer case is not an isolated phenomenon. Increasing depth generates a hierarchy of pullback Gram metrics associated with successive layers. Since all residuals are functions of the deepest activation fields, the equations above define the dynamics in terms of the stated variables.

17

7

Arbitrary Depth

The explicit calculations performed for three and four hidden layers reveal a recursive structure that naturally extends to arbitrary depth. We therefore consider a network with L hidden layers, u1 , . . . , uL .

(164)

The output is 1 X α u . N i L,i

(165)

1 X α u − Y α. N i L,i

(166)

Ŷ α = and the residuals are rα =

7.1

Activation Fields

For every hidden layer ℓ = 1, . . . , L, the activation dynamics takes the form ∆uαℓ,i = −

2η α X (ℓ−1) β β β Q aℓ,i bℓ,i r . a M ℓ,i β αβ

(167)

For the deepest layer we use the convention bαL ≡ 1,

(168)

which reproduces the previously derived equation for uL . The overlap matrices are (ℓ)

Qαβ =

7.2

1 X α β u u . N i ℓ,i ℓ,i

(169)

Conjugate Fields

The recursive structure observed for L = 2, 3, 4 suggests the definition bαL = 1,

(170)

and 

bαℓ = W (ℓ+1)

T

aαℓ+1 ⊙ bαℓ+1 ,

ℓ = 1, . . . , L − 1.

(171)

The corresponding dynamics is ∆bαℓ,i = −

2η X (ℓ+1) β β α A uℓ,i r + Tℓ,i , M β αβ

(172)

1 X α β a a . N i ℓ,i ℓ,i

(173)

where (ℓ)

Aαβ =

18

and Tℓ denotes the contribution generated by the dynamics of deeper conjugate fields. For L = 2, T1 = 0.

(174)

For L ≥ 3, Tℓ generates the pullback operators discussed below.

7.3

Pullback Gram Metrics

The calculations of Sections 5 and 6 show that terms of the form T 



(175)

uβℓ+1 = aβℓ+1 ⊙ W (ℓ+1) uβℓ ,

(176)



W (ℓ+1)

aαℓ+1 ⊙ uβℓ+1

appear systematically in the dynamics of bℓ . Using

one obtains 

W (ℓ+1)

T

Dℓαβ W (ℓ+1) uβℓ ,

(177)



(178)

where 

Dℓαβ = diag aαℓ+1 ⊙ aβℓ+1 . This motivates the definition T

Dℓαβ W (ℓ+1) .



T



(ℓ+1) Gαβ ℓ = W

(179)

Since Dℓαβ = Dℓαβ

,

Dℓαβ ⪰ 0,

(180) (181)

and 

Dℓαβ

2

= Dℓαβ ,

(182)

the operator Dℓαβ is an orthogonal projector. Consequently, 

αβ Gαβ ℓ = Gℓ

T

,

(183)

and Gαβ ℓ ⪰ 0.

(184)

The operator Gαβ ℓ can therefore be interpreted as a pullback Gram metric induced by the co-activation geometry of layer ℓ + 1. 19

7.4

Residual Kernel

The residual dynamics retains the same form as in the shallow cases: ∆rα = −

2η X (L) β K r . M β αβ

(185)

Inspection of the cases L = 1, 2, 3, 4 suggests the layer-wise decomposition (L)

Kαβ =

L X

(ℓ−1) (ℓ) Sαβ .

Qαβ

(186)

1 X α β α β a a b b . N i ℓ,i ℓ,i ℓ,i ℓ,i

(187)

ℓ=1

with (ℓ)

Sαβ = For the deepest layer,

bL = 1,

(188)

(L)

(189)

and therefore (L)

Sαβ = Aαβ .

Eq. (186) suggests for the kernels the recursive structure previously obtained for L = 1, 2, 3, 4.

8

Structural Properties

The explicit calculations of the previous sections reveal several structural features that appear to persist with increasing depth.

Proposition 1. Layer-wise kernel decomposition For every depth considered, the residual kernel admits a decomposition into layer-wise contributions, (L)

Kαβ =

L X

(ℓ−1) (ℓ) Sαβ .

Qαβ

(190)

ℓ=1

Each term couples an activation overlap (ℓ−1)

Qαβ

=

1 X α u uβ , N i ℓ−1,i ℓ−1,i

(191)

or, equivalently, (ℓ−1)

Qαβ

D

E

= uαℓ−1 uβℓ−1 ,

(192)

1 X α β α β a a b b , N i ℓ,i ℓ,i ℓ,i ℓ,i

(193)

to a conjugate-field correlator (ℓ)

Sαβ = or equivalently (ℓ)

D

E

Sαβ = aαℓ aβℓ bαℓ bβℓ . 20

(194)

The residual dynamics therefore remains organized as a sum of contributions associated with individual layers.

Proposition 2. Fourth-order closure of the collective kernel For all depths analyzed in this work, the collective residual kernel can be expressed in terms of second-order activation overlaps and fourth-order conjugate-field correlators. Indeed, every contribution to the kernel (L)

Kαβ =

L X

(ℓ−1) (ℓ) Sαβ

Qαβ

(195)

ℓ=1

is constructed from activation overlaps D

(ℓ)

E

Qαβ = uαℓ uβℓ ,

(196)

and conjugate-field correlators (ℓ)

D

E

Sαβ = aαℓ aβℓ bαℓ bβℓ .

(197)

Consequently, increasing depth enlarges the hierarchy of dynamical fields and pullback metrics, but does not generate progressively higher-order collective correlators in the residual kernel itself. The complexity of the learning dynamics is therefore transferred to the evolution of the fields uℓ ,

bℓ ,

Gℓ ,

rather than to the statistical order of the kernel observables. In this sense, depth increases the geometric complexity of the dynamical state while preserving a remarkably low-order collective structure for the residual dynamics.

Proposition 3. Gradient descent as a dynamics of fields and pullback metrics The explicit constructions of Sections 3–7 suggest that the learning dynamics admits a closed description in terms of a hierarchy of collective dynamical variables: for one hidden layer, the dynamics closes on the activation fields alone; for two hidden layers, closure requires the introduction of conjugate fields; starting from three hidden layers, closure additionally requires the hierarchy of pullback Gram metrics 

(ℓ+1) Gαβ ℓ = W

T

Dℓαβ W (ℓ+1) .

(198)

The resulting dynamical state may therefore be written schematically as S (L) = {uℓ , bℓ , Gℓ }.

(199)

where ℓ = 1, . . . , L. In this representation, gradient descent is described through the coupled evolution of activation fields, conjugate fields and pullback metrics. The role of increasing depth is not to generate higher-order collective correlators, but rather to enlarge the hierarchy of induced transport operators required to close the dynamics. Consequently, the complexity of deep learning appears to be encoded primarily in the geometry of the transport operators Gℓ , while the residual dynamics itself retains a comparatively simple collective structure.

21

9

Discussion

The analysis developed in this work is restricted to fixed-readout ReLU networks trained with quadratic loss. Within this setting, gradient descent can be represented through the coupled, closed evolution of activation fields, conjugate fields and a hierarchy of pullback Gram metrics. The collective residual kernels remain expressible in terms of second-order activation overlaps and fourth-order conjugate-field correlators, and explicit weight dependence re-enters only through quadratic pullback Gram operators.

9.1

Relation to NTK-like descriptions

Our approach follows an NTK-like strategy [2]: rather than treating the weights as the primary dynamical variables, one rewrites the evolution in terms of the induced dynamics on activations and residuals. Unlike the standard infinite-width NTK limit, the present formulation keeps track of finite-width activation patterns and their induced co-activation structure.

9.2

Pullback metrics and layerwise geometry

For depth L ≥ 3, the attempt to close the dynamics in terms of activation and conjugate fields produces operators of the form 

(ℓ+1) Gαβ ℓ = W

T

Dℓαβ W (ℓ+1) .

(200)

The co-activation projectors 

Dℓαβ = diag aαℓ+1 ⊙ aβℓ+1



(201)

select the subspace of neurons simultaneously active for a pair of examples. Since Dℓαβ is symmetric, positive semidefinite and idempotent, the operator Gαβ ℓ is a positive semidefinite Gram matrix. It can be interpreted as the pullback of the co-activation geometry of layer ℓ + 1 onto layer ℓ.

9.3

Possible relation to WeightWatcher and heavy-tailed spectra

Recent semi-empirical theories associated with the WeightWatcher project [3] has shown that the spectra of matrices of the form W T W often display heavy-tailed behaviour in trained networks with strong generalization properties ([4] and subsequent works). WeightWatcher analyzes trained models through such spectral diagnostics and connects them to Heavy-Tailed Self-Regularization. The present formalism suggests a possible dynamical mechanism behind this phenomenon. The relevant objects in the learning dynamics are the pullback metrics 

(ℓ+1) Gαβ ℓ = W

T

Dℓαβ W (ℓ+1) .

(202)

The co-activation projectors 

Dℓαβ = diag aαℓ+1 ⊙ aβℓ+1



(203)

select the subspace of neurons simultaneously active for a pair of examples. During training, similar inputs may progressively share activation subspaces, while unrelated inputs may separate into distinct activation regions. In this scenario, the projectors Dℓαβ become increasingly αβ concentrated on informative lower-dimensional subspaces. Since Gαβ ℓ is the pullback of Dℓ (ℓ+1) through W , this concentration acts as a filter on gradient transport. The directions selected by the co-activation geometry are amplified through repeated gradient updates, while directions

22

orthogonal to them receive comparatively little accumulated signal. Integrating the gradient flow, Z t

W (t) = W (0) +

Ẇ (τ ) dτ,

(204)

0

the learned weights may therefore inherit the anisotropic structure generated by the family of projectors Dℓαβ and metrics Gαβ ℓ . Under this interpretation, the large eigenvalues observed in W T W would emerge from the cumulative action of the transport operators Gαβ ℓ . Repeated propagation through activationconditioned subspaces may progressively concentrate gradient flow along a restricted set of collective directions. The resulting anisotropic accumulation could then manifest itself in the spectral structure of the learned weight matrices. In this picture, heavy-tailed spectra would reflect the cumulative effect of repeated low-rank transport through activation-conditioned subspaces. The present formalism therefore suggests a possible dynamical mechanism linking activation geometry, gradient transport and the spectral properties of trained networks.

References [1] Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan, and Donghun Lee. The weight gram matrix captures sequential feature linearization in deep networks, 2026. [2] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018. [3] Charles H. Martin. Weightwatcher project. https://weightwatcher.ai. Accessed 2026-0608. [4] Charles H. Martin and Michael W. Mahoney. Traditional and heavy-tailed self regularization in neural network models. Journal of Machine Learning Research, 22(165):1–73, 2021. [5] Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory. Cambridge University Press, 2022.

23