Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

P ERCOLATION DYNAMICS IN O PTIMIZATION : VARIANCE C ASCADES AND D ISCRETE S CALE I NVARIANCE

arXiv:2609.02373v1 [cs.LG] 2 Sep 2026

Sai Niranjan Ramachandran* †

Suvrit Sra* †

September 3, 2026

A BSTRACT We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model. SGD collapses deep neural networks toward sparse, low-rank representations generated by architectural symmetry. We show how this collapse progresses over time, and that the mechanism extends to Adam and AdamW.

1

Introduction

Unlike classical statistical learning deep learning systems rely on a complex interaction between the dataset, architecture, and choice of optimizer. Furthermore, they are known to achieve strong generalization without explicit regularization, a phenomenon attributed to implicit biases introduced during training. Recent work has established that Stochastic Gradient Descent (SGD) is a central source of this implicit bias collapsing networks onto invariant sets that behave like much simpler subnetworks [Wei et al., 2008, Chen et al., 2023], however the dynamics by which these invariant sets are reached are poorly understood. This gap limits our ability to mechanistically explain anomalous training behaviors. Delayed generalization in transformer networks, known as grokking [Power et al., 2022, Liu et al., 2022], is one example. A network may memorize the training data perfectly, but it only discovers a generalizing solution after thousands of epochs, challenging the classical view of smooth optimization. Similar patterns can also be seen in exact deep linear networks [Saxe et al., 2013] and task shifts [Goodfellow et al., 2013, Kirkpatrick et al., 2017]. This suggests that the topological structure of these dynamics contains crucial information about how optimization progresses. To chart this evolution, we describe the optimization continuously with a stochastic differential equation (SDE) that records how parameters drift and diffuse over time [Li et al., 2017]. Architectural symmetries produce invariant sets which draw parameter trajectories [Chen et al., 2023]. Independent parameters entering these sets come together and fuse into equivalence classes, and the converging weights collide and merge, folding the parameter space into a simpler topology in the settings we ∗ †

School of Computation, Information and Technology, Technical University of Munich, Germany Munich Center for Machine Learning (MCML)

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

examine [Naitzat et al., 2020]. We represent this dynamics as a percolation process where isolated components link and fuse, potentially causing sudden macroscopic transitions [Achlioptas et al., 2009, Chen et al., 2014]. We develop this framework for SGD, where the mechanism is cleanest to state and prove, but most large-scale training uses adaptive optimizers; we show the same trapping mechanism and cascade structure extend to Adam and AdamW under an explicit heavy-tailed noise model, and use this extension to justify the AdamW-trained grokking experiment in Section 6. Building on this percolation framework, our contributions are as follows, 1. Percolation Model of Topological Condensation: We construct a framework that maps stochastic gradient flow near invariant sets onto a graph that tracks merges and splits (Reeb graph), which is then renormalized into a percolation process. Architectural symmetries cause discrete, simultaneous block-merges instead of continuous edge attachment, and we determine the precise condition under which this discontinuity survives as the network grows large, rather than being a finite-size artifact that vanishes in that limit. We also show that the trapping mechanism extends to Adam and AdamW under a heavy-tailed noise model. 2. Variance Divergence and Topological Cascades: We use relative variance across training trajectories to isolate discrete microtransitions, and derive that multi-body block-merges yield Discrete Scale Invariance (DSI) [Sornette, 1998], shaping the phase transition into a geometrically scaling cascade. 3. Empirical Observations: We test this framework across environments and largely confirm the predicted cascade structure. In toy models, we track parameters clustering under shifting data distributions. We examine tabular classification on UCI datasets, vision benchmarks, and grokking in Transformers on modular arithmetic.

2

Stochastic Collapse in SGD

Let θ ∈ Rd denote a model’s parameter vector and L(θ) a nonconvex loss, for example a neural network’s empirical risk. At iteration t, optimizing L with learning rate η over a mini-batch Bt yields the discrete update: θ t+1 = θ t − η∇LBt (θ t ). By separating the exact full-batch gradient ∇L(θ t ) from the zero-mean batch noise, we approximate this discrete sequence continuously using a Stochastic Gradient Flow (SGF) modeled by an Itô SDE [Li et al., 2017]: p dθ t = −∇L(θ t )dt + ηΣ(θ t )dWt , (1) where Wt is a standard d-dimensional Wiener process and Σ(θ) is the position-dependent noise covariance matrix of the mini-batch gradients. Architectural symmetries, such as permutation invariances among neurons, generically create flat, degenerate regions in the parameter space. We formally define these regions as invariant sets, following [Chen et al., 2023]. Definition 2.1 (Invariant Set). A set S ⊂ Rd is an invariant set for an optimization dynamic if any trajectory initialized within S never escapes. Formally, θ 0 ∈ S =⇒ θ t ∈ S for all t > 0. When structural symmetries constrain these invariant regions into affine subspaces, invariance of the discrete SGD dynamics is exactly preserved by the continuous SGF dynamics. Definition 2.2 (Invariant Sets of Continuous Processes). For a stochastic process {θ t ∈ Rd : t ≥ 0}, a Borel-measurable set A ⊂ Rd is invariant if, for every initial point θ 0 ∈ A, the probability that the process stays in A for all t ≥ 0 is exactly 1: P [θ t ∈ A for any t ≥ 0 | θ 0 ∈ A] = 1. Proposition 2.3 (Affine Invariance in SGF). Consider an SGD process where the individual sample gradients ∇ℓ(·; xi , yi ) are Lipschitz continuous and bounded. If a subset A ⊆ Rd forms an affine invariant set of the discrete SGD process, then A also forms an invariant set of the continuous SGF process. Proof Sketch [Chen et al., 2023]. Bounded, L-Lipschitz individual gradients enforce a Lipschitz population drift ∇L(θ). Applying the Powers-Stormer inequality to the noise covariance shows that Σ(θ) remains Frobenius-Lipschitz continuous. Together, these conditions guarantee a unique 2

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

strong solution for the global SDE. To establish structural invariance, we map the dynamics through a projection operator P onto the affine subset A. Because A traps the discrete SGD process, the local gradients align (P ∇ℓ = ∇ℓ) within the subspace, so the projected SDE matches the original SDE almost surely, trapping any trajectory initialized inside A. Once parameters drift near these invariant sets, the model predicts a qualitative change in the optimization dynamics. If the mini-batch variance Σ(θ) decays as θ approaches A, an inward pull emerges that can trap trajectories near A. We formalize this as stochastic attractivity, Definition 2.4 (Stochastic Attractivity via Drift Dominance). An invariant set A ⊂ Rd of the stochastic process {θ t ∈ Rd : t ≥ 0} is locally stochastically attractive if there exists a basin N (A, ϵ) such that for all θ ∈ N (A, ϵ), the inward deterministic gradient drift strictly overpowers the outward stochastic diffusion. Formally, applying the infinitesimal generator A to the transverse distance (i) (i) (i) process Yt = ∥θ t − πA (θ t )∥22 yields,  ⊤ (i) (i) (i) A Yt = −2 θ t − πA (θ t ) ∇θ(i) L(θ t ) + ηTr(Σii (θ t )) ≤ 0. (2) When transverse noise overpowers gradient drift, stochastic attractivity draws trajectories toward simplified invariant sets and can drive stochastic collapse even when those sets contain saddle points or local maxima [Ziyin et al., 2022, Chen et al., 2023]. SGF is known to behave as a non-equilibrium system driven by anisotropic, position-dependent noise [Chaudhari and Soatto, 2018, Mandt et al., 2017, Mori et al., 2022], becoming trapped in prolonged metastable states rather than relaxing smoothly to equilibrium, which motivates the non-equilibrium analysis developed below.

3

Topological Dynamics of SGF via Attractor Equivalence

Definition 3.1 (Subnetwork Partition). Partition the global parameter vector into K distinct subnetworks, θ = [θ (1) , . . . , θ (K) ], such that each θ (i) ∈ Rdi evolves over a marginal function space C([0, ∞), Rdi ). To model how a network collapses as a sequence of topological phase transitions, we track the restricted path measures of these subnetworks over a filtered probability space (Ω, F, {Ft }t≥0 , P), isolating the marginal path measures of distinct structural sub-components via the network’s symmetries and the low-rank simplicity bias documented to decouple parameter blocks during training [Şimşek et al., 2021, Huh et al., 2021, Shah et al., 2020]. Assumption 3.2 (Block-Diagonal Dominance). For two subnetworks θ (i) and θ (j) decoupled by architectural symmetries and the low-rank simplicity bias, and occupying distinct invariant sets, the cross-covariance blocks of the diffusion matrix satisfy ∥Σij (θ)∥2 ≤ δ for a sufficiently small δ > 0. Assumption 3.2 bounds off-diagonal stochastic interference, licensing the treatment of subnetwork trajectories as conditionally independent SDEs prior to collapsing together. To formalize structural binding, we project these decoupled trajectories into a shared quotient space by tracking their squared (i) (i) (i) transverse distance Yt = ∥θ t −πA (θ t )∥22 to a target invariant set A. Under stochastic attractivity, the inward deterministic gradient dominates the transverse diffusion, opposing outward drift. We show this is enough to trap the subnetwork within the local basin. Theorem 3.3 (Local Supermartingale Trapping). Given a stochastically attractive invariant set (i) A, there exists a local basin N (A, ϵ) such that the stopped transverse process Yt∧τϵ operates as (i) a non-negative local supermartingale, where τϵ = inf{t ≥ t0 : Yt ≥ ϵ} marks the boundary of escape. Proof Sketch. By Definition 2.4, stochastic attractivity structurally enforces the pointwise drift (i) condition A Yt ≤ 0 prior to escape. Expanding the stopped dynamics via Itô’s lemma directly isolates this non-positive drift from the stochastic noise. Because sample-path continuity forces the stopped process to remain uniformly bounded by ϵ (Lemma B.2, Appendix B), the stochastic integral is a genuine bounded-integrand martingale rather than merely a local one [Karatzas and Shreve, 1991], and the conditional expectation annihilates it exactly. 3

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Corollary 3.4 (Probabilistic Non-Escape). For any stochastically attractive invariant set A, the probability of a trajectory initialized at θ 0 escaping the ϵ-neighborhood is strictly bounded by the initial transverse distance. Proof Sketch. This follows directly from the local supermartingale property established in Theorem 3.3 via Doob’s maximal inequality (detailed in Appendix B). Theorem 3.3 shows that decoupled subnetworks are trapped within shared invariant subspaces, suppressing transverse variance and driving their path measures toward synchronization. We formalize this structural binding as Attractor Equivalence, quantifying whether synchronized trajectories survive (i) indefinitely without triggering the spatial escape time τϵ = inf{t ≥ t0 : Yt ≥ ϵ}.   (i,j) (i) (j) Definition 3.5 (Attractor Equivalence, ∼A ). Let Plink (t) := P τϵ = ∞ ∩ τϵ = ∞ | Ft denote the joint probability that decoupled subnetworks θ (i) and θ (j) never breach the basin boundary. We define θ (i) ∼A θ (j) at time t if and only if their path measures couple as the transverse drift vanishes: (i,j) lim i Plink (t) = 1. h (i) (j) E Yt +Yt |Ft →0

(3)

Theorem 3.6 (Metastable Equivalence). The relation ∼A establishes a time-parameterized equivalence class over the restricted path measures, conditioned on the survival of τϵ . Proof Sketch. Reflexivity and symmetry follow from the algebraic properties of the shared quotient space. For transitivity, we apply Doob’s maximal inequality [Revuz and Yor, 1999] to the local (i) supermartingales, bounding the escape probability by 1 − ϵ−1 E[Yt | Ft ]. As expected transverse distances collapse to zero, this bound forces individual escape probabilities to vanish, so the intersection of survival events converges to certainty and the path measures transitively bind to the same invariant set, provided τϵ remains untriggered. We aggregate these pairwise equivalences into a topological network, treating distinct path measures as nodes linked by transient edges of ∼A . Because this coupling hinges on the survival of τϵ , it is reversible, and we project it onto a continuous Reeb graph [Edelsbrunner and Harer, 2008, Carlsson, 2009]. Let S = {1, . . . , K} index the discrete subnetworks, tracking equivalence classes [i]t = {j ∈ S | θ (i) ∼A θ (j) at time t}. A topological transition is any discrete change in the membership of [i]t . We project these classes onto a continuous Reeb graph, constructed as the quotient space R := (S × [0, ∞))/ ∼R , where (i, t) ∼R (j, t) identifies two subnetworks whenever θ (i) ∼A θ (j) at time t. Theorem 3.7 (Non-Equilibrium Condensation and Fragmentation). The Reeb graph R realizes these topological transitions via two mechanisms: (i)

(j)

• Condensation (∪): As transverse drift vanishes (E[Yt + Yt ] → 0), stochastic attractivity (i,j) drives the joint link probability to certainty (Plink (t) → 1), enforcing ∼A . The quotient map binds the disjoint sets ([i]t = [i]t−δ ∪ [j]t−δ ), collapsing distinct path measures into a single topological merge node. • Fragmentation (∅): If an injected variance deviation breaches the local basin, the spatial (i,j) stopping time triggers (τϵ ≤ t), the maximal bounds collapse (Plink (t) ≪ 1), ∼A is severed, and the equivalence class [i]t ∩ [j]t = ∅ splits the node into decoupled branches. Proof Sketch. We map the stochastic path measures directly to the topological level sets of R [Edelsbrunner and Harer, 2008]. For condensation, Doob’s maximal bound squeezes the joint link probability to 1, satisfying ∼A and forcing the quotient projection πR to map the independent branches into a single vertex with in-degree > 1. For fragmentation, a stochastic large deviation [Freidlin and Wentzell, 2012] breaches N (A, ϵ), realizing τϵ ≤ t. This terminates the joint survival event, invalidating ∼A and forcing the quotient map to partition the trajectories into a critical point with out-degree > 1. 4

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

The Reeb topology thus governs how the network collapses in this model, with every merge and split rewiring the global covariance matrix by fusing or severing its off-diagonal blocks.

4

Percolative Collapse and Discrete Scale Invariance

4.1

Architectural Symmetry and Graph Renormalization

The equivalence classes defining the Reeb graph’s vertices correspond to topological attractors within the loss landscape, since permutation invariances among functionally equivalent neurons partition the parameter space into lower-dimensional geometric subspaces [Chen et al., 2023]. Definition 4.1 (Approximate Q-Symmetry). Let Q ∈ Rd×d satisfy QQ⊤ = Id or Q = Q⊤ . A loss functional L(θ) exhibits approximate Q-symmetry around A = {θ ∈ Rd | Qθ = θ} if, for any ϵ > 0, there exists δ > 0 such that |L(Qθ) − L(θ)| < ϵ · d(θ, A)

∀θ s.t. d(θ, A) < δ,

(4)

where d(θ, A) = inf a∈A ∥θ − a∥2 . Theorem 4.2 (Affine Trapping and Transverse Diffusion). If L satisfies approximate Q-symmetry around A, deterministic gradient drift is restricted to A. The affine geometry of A decouples the transverse stochastic distance Yt = d(θ t , A) into a curvature-free diffusion process. Proof Sketch. Along a normal vector n ⊥ A, approximate symmetry forces ∇n L(θ) = 0, restricting deterministic drift to the subspace. Because A is affine, all principal curvatures vanish (HA = 0), so geometric potential drift cannot couple to the transverse diffusion, permitting a well-posed, one-dimensional stationary distribution. The full Fokker-Planck derivation is in Appendix C. Stochastic gradient noise continuously breaks and reforms edges on the transient Reeb graph; a temporal renormalization over the macroscopic training timescale extracts a monotonic topological progression (Appendix C). Theorem 4.3 (Reeb Graph Renormalization to Percolation Graph). Let Rt denote the transient Reeb graph. Under the timescale separation τf ast ≪ τslow , applying a temporal renormalization operator over τslow collapses microscopic Brownian fluctuations, transforming Rt into a monotonic macroscopic percolation graph Gτ = (V, Eτ ) governed by the continuous expected edge density p ∈ [0, 1]. Proof Sketch. On τf ast , transverse diffusion Yt satisfies local detailed balance (Jss (ϵ) = 0), canceling transient fragmentations and condensations. Renormalizing over τslow integrates out this noise, isolating the macroscopic condensation trend (σ̇ < 0), which advances p monotonically from 0 to 1. See Appendix C. We track the connectivity of Gτ via the structural order parameter, the fractional size of the largest connected component: |Cmax (p)| O(p) = . (5) N Unlike classical continuous phase transitions, where component growth proceeds via independent single-edge attachments [Stauffer and Aharony, 1994], symmetry-induced percolation breaks this continuous mechanism. Theorem 4.4 (Non-Erdős-Rényi Discontinuity). Driven by Sn multi-body invariant sets, the neural percolation graph Gτ prohibits continuous edge attachment, forcing simultaneous group-wise bindings that produce a jump ∆O(p) = (n − 1)i/N in the order parameter at each microtransition merging components of size i. This jump is a genuine discontinuity in the thermodynamic limit (as network size N → ∞) precisely when the merging components are already macroscopic at the moment of the merge (i = Θ(N )). When microscopic (i = O(1)), the jump vanishes as N → ∞ and the transition is asymptotically continuous (Appendix C, Corollary C.18). 5

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Proof Sketch. Erdős-Rényi dynamics allow uniform infinitesimal growth (C1 → C1 + 1), but the symmetric permutation group Sn topologically restricts edge formation, identically binding n indistinguishable path measures and enforcing discontinuous block-merges of size ni (Theorem C.22) with finite-N jump ∆O(p) = (n − 1)i/N . Whether this jump survives N → ∞ depends on the scaling of i with N . The full regime analysis is in Appendix C. Because stochastic noise shifts the density at which these discrete merges occur, single-trajectory observations obscure the critical thresholds. We isolate structural shocks via the relative variance of the order parameter across an ensemble of training trajectories:   E (O(p) − E[O(p)])2 Rv (p) = . (6) E[O(p)]2 Theorem 4.5 (Divergence of Relative Variance). At a structural microtransition, the relative variance Rv (p) diverges proportionally to the squared jump amplitude (∆O)2 , preceding the critical global connectivity threshold pc . Proof Sketch. Near a transition density pi , stochastic fluctuations induce bimodal coexistence between pre-jump and post-jump states. At balanced probability mass, the expectation denominator remains bounded while the variance numerator scales directly with (∆O)2 , yielding a sharp, localized divergence in Rv (p) [Chen et al., 2014]. The discrete microtransitions flagged by Rv (p) are incompatible with the continuous scale invariance of classical phase transitions, since confining network growth to rigid multi-body block merges breaks the continuous Lie group of dilations into a discrete subgroup, motivating a generalized notion of scaling. Definition 4.6 (Generalized Discrete Scale Invariance (DSI)). An observable f (x) exhibits DSI if it satisfies self-similarity under a discrete set of preferred magnification factors λ(n) = nσ , where σ is the critical exponent of the underlying continuous percolation transition [Sornette, 1998]. Theorem 4.7 (Symmetric Hyper-Condensation Yields DSI). The Sn topological constraint restricts component growth to the discrete mapping C1 → nC1 , locking critical microtransition densities into a geometric cascade: pc − pni 1 lim = (n) , where λ(n) ≡ nσ . (7) i→∞ pc − pi λ This cascade forecasts the global structural collapse pc within the model. Proof Sketch. Inverting the continuous baseline growth relation, understood as describing the (ω) ensemble-averaged transition density pi := Eω [pi ] (Appendix C, Definition C.24) rather than any single realization, isolates pi = pc − Aσamp i−σ . Substituting the discrete block-merge constraint ni yields pni . The ratio of their distances from pc reduces to n−σ . The full thermodynamic limit derivation is in Appendix C.

5

Extension to Adam and AdamW

The theory of Sections 3–4 is stated for SGD. We extend the trapping mechanism and DSI cascade to Adam [Kingma and Ba, 2015] and AdamW under an explicit set of conditions, with full derivations in Appendix D. Adam and AdamW maintain a joint state (θ t , mt , vt ), and the elementwise squaring in vt narrows the admissible symmetry group from general orthogonal Q to coordinate permutations Pπ , a class that already contains the Sn neuron-permutation symmetry used throughout Section 4, so the joint invariant set à = {(θ, m, v) : Pπ θ = θ, Pπ m = m, Pπ v = v} extends Theorem C.2 without modifying the geometry of A. Assumption 5.1 (Heavy-Tailed Gradient Noise). There exist p ∈ (1, 2] and σ > 0 such that E|gt,i |p ≤ σ p for every coordinate i and every t, matching the heavy-tailed regime documented for attention-based architectures [Zhang et al., 2020]. 6

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

This permits infinite variance when p < 2, so we truncate gt at ĝt := gt · 1{|gt | ≤ τt }, with τt := (σ p t/ log(1/δ))1/p , at the cost of a bias bt = O(τt1−p ). The truncated momentum and second-moment recursions are exponential moving-average P∞ filters; using the one-sided z-transform [Oppenheim and Schafer, 2010] X(z) := Z{xt } := t=0 xt z −t , which satisfies Z{xt−1 }(z) = z −1 X(z), gives   1 − β2 ∂ 2 Φ(z, u) 1 − β1 Ĝ(z), V̂ (z) = − , M̂ (z) = 1 − β1 z −1 1 − β2 z −1 ∂u2 u=0 where Ĝ(z) = Z{ĝt } and Φ(z, u) = Z{E[eiuĝt ]}. Squaring commutes with neither expectation nor the z-transform, so V̂ (z) cannot be built from Ĝ(z) directly; differentiating Φ(z, u) twice at u = 0 instead recovers E[ĝt◦2 ] through a linear operation on u, and linear operations commute with the summation defining the transform, so this yields V̂ (z) without ever forming Ĝ(z)2 . These filters give memory scales τ1 = 1/(1 − β1 ) and τ2 = 1/(1 − β2 ). Definition 5.2 (Low-Correlation Stopping Time). Let τcorr denote the lag beyond which the autocovariance of ĝt◦2 is negligible. Define Et := {τcorr (ĝs ) < τ2 ∀s ≤ t} and τE := inf{t ≥ t0 : Et fails}. The realized v̂t tracks its target m∗i (θ) = (∂i L(θ))2 + Σii (θ) up to two errors. Temporally, a drift term O(Lm τ2 ∆max√ ), with Lm = 6L2 the Lipschitz constant of m∗i derived from Proposition A.2 and ∆max = ητt /( vmin + ϵ) a single Adam step, plus a filtered-noise variance O((1 − β2 )τt4 ) controlled via Wiener-Khinchin on Et . Cross-sectionally, coordinates in a permutation-symmetric block share a common projection point on A, √ so the same Lipschitz bound keeps the block-restricted preconditioner P (θ t )|block within O(Lτt ϵ) of a scalar multiple c(θ t )I, deterministically and regardless of correlation between block coordinates. Together these give a residual Rt = O(bt + √ Lm τ2 ∆max + Lτt ϵ) with high probability, perturbing the preconditioned drift and diffusion away from a pure scalar rescaling of the original SGF. Assumption 5.3 (Non-Degeneracy and Macroscopic Block Scaling). v̂i (θ t ) ≥ vmin > 0 for every coordinate under analysis, for t ≤ τϵblk ∧ τE . For the Sn -symmetric block, n(N )/N → c ∈ (0, 1] as N → ∞. Theorem 5.4 (Conditional Trapping under Adam and AdamW). Suppose A is stochastically attractive (i) for the rescaled drift c(θ)∇L(θ), with drift margin dominating the residual Rt . Then Yt∧τ blk ∧τE is ϵ a non-negative local supermartingale. Proof Sketch. Substituting P (θ t ) = c(θ t )I + Et into the exact update decomposes the SDE into a term rescaled by the positive scalar c(θ t ), which preserves the affine geometry of Ã, plus the residual (i) Rt . Itô’s lemma on Yt then reproduces the transverse generator with an extra term whose sign is preserved by c(θ t ) > 0 and dominated by the drift margin, keeping the generator non-positive prior to τϵblk ∧ τE . Full derivation in Appendix D. Theorem 5.5 (Conditional DSI Cascade under Adam and AdamW). Under Assumption 5.3 and the heavy-tailed noise model above, on t ≤ τϵblk ∧ τE the Adam- or AdamW-trained trajectory admits a percolation graph satisfying Generalized Discrete Scale Invariance with the same magnification factor λ(n) = nσ as Theorem 4.7. Proof Sketch. Theorem 5.4 lets the Reeb graph and percolation construction of Sections 3 and 4 run on the stopped process with τϵ replaced by τϵblk ∧ τE . The scalar c(θ t ) cancels in the ratio defining λ(n) , and under the Θ(N ) block scaling of Assumption 5.3, the merge is placed in the regime where the discontinuity survives the thermodynamic limit. Full derivation in Appendix D. Both theorems’ full proofs are in Appendix D, which also reports a preliminary diagnostic check of the stopping-time and non-degeneracy conditions on the AdamW-trained grokking run of Section 6.

6

Empirical Validation

We test whether percolative collapse holds under unconstrained, non-convex optimization by tracking phase transitions via a macroscopic order parameter O and its detrended variance fluctuations Rv , 7

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

computed across an ensemble of training seeds to satisfy the bimodal-coexistence requirement of Theorem 4.5 (see Appendices E, F for derivations and baselines). 6.1

Stochastic Condensation and Reactive Fragmentation

In a controlled toy setting adapting Chen et al. [2023], pairwise subnetwork merges are predicted at magnification λ = 2. A constrained K = 3 baseline and unconstrained K = 6 SGD optimization both match this prediction (λ ≈ 2.00) via localized Rv divergences, and a task shift at t = 3000 confirms the predicted reversibility, producing a reactive fragmentation peak (Figure 1, Appendix E). 6.2

Universality, Grokking, and Benchmark Generalization

Scaling to unconstrained, high-dimensional networks requires replacing distance-based clustering with Spectral Effective Rank.3 In a Transformer trained on modular arithmetic, delayed generalization (grokking, [Power et al., 2022]) coincides with a 3-peak DSI cascade immediately preceding the performance spike (λ = 2.11, phase-randomized spectral null false positive rate (FPR) = 0.1%), though we do not claim this cascade is the sole driver of the transition. The same cascade structure appears, with fractional scaling factors consistent with pairwise or higher-order merges, across UCI tabular classification and vision benchmarks (Figure 2, Appendix F).

Figure 1: Topological Condensation and Fragmentation: (Top) Kinematic verification of the theoretical λ = 2 DSI baseline. (Middle & Bottom) Empirical SGD dynamics. The condensation phase (t < 3000) exhibits sequential DSI variance peaks (p1 , p2 ) with a log-linear progression close to λ ≈ 2.00 (fit through two points; see main text). The task shift at t = 3000 reverses the stability inequality and is followed by a reactive fragmentation peak (p3 ). Zoom for clarity.

7

Related Work

Stochastic Dynamics and Implicit Bias: Traditionally SGD is modeled as driving networks toward low-rank manifolds [Blanc et al., 2020, Nacson et al., 2022, Chen et al., 2023] through anomalous diffusion and glassy dynamics [Baity-Jesi et al., 2018, Geiger et al., 2019, Kunin et al., 2024, Chen et al., 2020]. We instead track the transient path to these manifolds through network percolation on a continuous Reeb graph. Graph and Percolation Perspectives on Training: Recent work applies percolation tools to training, studying connectivity under dropout [Devlin and Sanders, 2025] or synaptic invariants that forecast 3 Spectral Effective Rank measures soft dimensionality collapse via the exponentiated Shannon entropy of a weight matrix’s singular value spectrum, generalizing hard, exact-zero collapse to the case where weights rarely vanish exactly.

8

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Figure 2: Universality and Empirical Generalization: (a) Transformer Grokking (Modular Arithmetic): Delayed generalization co-occurring with continuous dimensionality collapse and a log-linear DSI cascade. (b, c) UCI Heart Disease and FMNIST: DSI variance cascades in tabular regression (b) and image classification (c). Extended evaluations are in Appendix F. performance [Li et al., 2022a]. Our percolation graph is instead constructed from a renormalized Reeb graph (Theorem 4.3) that tracks collapse into shared invariant sets (Definition 3.5), with edges driven by architectural symmetry rather than random deletion or correlation. Neuron and Weight Condensation: Condensation serves as a complimentary lens on stochastic collapse focusing on initialization scale rather than symmetry. [Luo et al., 2021, Zhou et al., 2022, 2023]. Our mechanism allows to construct an explicit mechanism for the transient dynamics.

8

Limitations, Conclusion and Future Work

We formalize SGD’s transient dynamics as a symmetry-induced percolation process and prove it exhibits Discrete Scale Invariance (Theorem 4.7), confirmed exactly in toy models (Appendix E) and consistent with the same cascade across tabular, vision, and grokking settings (Section 6). Future work includes testing whether these discontinuities persist at practical network widths (Corollary C.18), extending the framework to curved invariant manifolds and extreme hyperparameter regimes, and using DSI variance spikes (Rv ) to guide learning rate scheduling. The DSI cascade also connects to empirical scaling laws (Remark C.26), suggesting a route to deriving new scaling relations directly from the percolation model. We also provide a preliminary extension for adaptive optimizers in Appendix D, validated through grokking though a systematic study of how this extends to large scale models remains open.

9

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

References Dimitris Achlioptas, Raissa M D’Souza, and Joel Spencer. Explosive percolation in random networks. Science, 323(5920):1453–1455, 2009. Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, and Gérard Ben Arous. Comparing dynamics: Deep neural networks versus glassy systems. Journal of Statistical Mechanics, 2018: 033301, 2018. Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on Learning Theory, pages 483–513. PMLR, 2020. Gunnar Carlsson. Topology and data. Bulletin of the American Mathematical Society, 46(2):255–308, 2009. Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In International Conference on Learning Representations, 2018. Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts SGD dynamics towards simpler subnetworks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Guozhang Chen, Cheng Qu, and Pulin Gong. Anomalous diffusion dynamics of learning in deep neural networks. arXiv preprint arXiv:2009.10588, 2020. Wei Chen, Malte Schröder, Raissa M D’Souza, Didier Sornette, and Jan Nagler. Microtransition cascades to percolation. Physical review letters, 112(15):155701, 2014. Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 9722–9732. PMLR, 2021. Finley Devlin and Jaron Sanders. Dropout neural network training viewed from a percolation perspective. arXiv preprint arXiv:2512.13853, 2025. Herbert Edelsbrunner and John Harer. Topological persistence and simplification. Discrete & Computational Geometry, 39(1–3):427–463, 2008. Mark I Freidlin and Alexander D Wentzell. Random perturbations of dynamical systems. Springer Science & Business Media, 2012. Crispin Gardiner. Stochastic Methods: A Handbook for the Natural and Social Sciences. Springer Berlin Heidelberg, 2009. Mario Geiger, Stefano Spigler, Stéphane d’Ascoli, Levent Sagun, and Marco Baity-Jesi. The jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100:012115, 2019. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013. Jeff Z HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory, pages 2315–2357. PMLR, 2021. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, pages 30016–30030, 2022. 10

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape degeneracy and stagewise development in transformers. Transactions on Machine Learning Research, 2024. Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2021. URL https://openreview.net/forum?id=3a0Cbtb1Hs. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Ioannis Karatzas and Steven E. Shreve. Brownian Motion and Stochastic Calculus, volume 113 of Graduate Texts in Mathematics. Springer-Verlag, 2nd edition, 1991. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521–3526, 2017. Daniel Kunin, Javier Sagastuy-Breña, Lauren Gillespie, Eshed Margalit, and Hidenori Tanaka. The limiting dynamics of SGD: Modified loss, phase-space oscillations, and anomalous diffusion. Neural Computation, 36:151–197, 2024. Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pages 2101–2110. PMLR, 2017. Yang Li, Kai Ma, Han Lu, Xin Chen, and Renjie Song. Neural capacitance: A new perspective of neural network selection via edge dynamics. arXiv preprint arXiv:2201.04194, 2022a. Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss? -a mathematical framework. In International Conference on Learning Representations, 2022b. Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 34651–34663, 2022. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer ReLU neural networks at infinite-width limit. Journal of Machine Learning Research, 22(71):1–47, 2021. Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873–4907, 2017. Takashi Mori, Liu Ziyin, Kangqiao Liu, and Masahito Ueda. Logarithmic landscape and power-law escape rate of SGD. In International Conference on Machine Learning, pages 15959–15978. PMLR, 2022. Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pages 16270–16295. PMLR, 2022. Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of data in deep learning. Frontiers in Applied Mathematics and Statistics, 6:554449, 2020. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. 11

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Alan V. Oppenheim and Ronald W. Schafer. Discrete-Time Signal Processing. Prentice Hall, Upper Saddle River, NJ, 3rd edition, 2010. Athanasios Papoulis and S. Unnikrishna Pillai. Probability, Random Variables, and Stochastic Processes. McGraw-Hill, New York, 4th edition, 2002. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4. Sai Niranjan Ramachandran and Suvrit Sra. Trees to flows and back: Unifying decision trees and diffusion models. In Forty-third International Conference on Machine Learning, 2026. Daniel Revuz and Marc Yor. Continuous martingales and Brownian motion, volume 293. Springer Science & Business Media, 1999. Oliver Riordan and Lutz Warnke. Explosive percolation is continuous. Science, 333:322–324, 2011. Hannes Risken. The Fokker-Planck Equation: Methods of Solution and Applications. Springer-Verlag, 1996. Gennady Samorodnitsky and Murad S. Taqqu. Stable Non-Gaussian Random Processes: Stochastic Models with Infinite Variance. Chapman & Hall, New York, 1994. Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013. Harshay Shah, Kaustubh Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 9573–9585, 2020. Didier Sornette. Discrete scale invariance and complex dimensions. Physics Reports, 297(5):239–270, 1998. Dietrich Stauffer and Ammon Aharony. Introduction to Percolation Theory. Taylor & Francis, 1994. Vimal Thilak, Etai Littwin, Shuanghai Zhai, Omid Saremi, Roni Joshua, and Leonard Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. arXiv preprint arXiv:2206.04817, 2022. Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari. Dynamics of learning near singularities in layered networks. Neural Computation, 20(3):813–843, 2008. Susan Wei, Daniel Murfet, Ming Gong, Huan Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that’s good. IEEE Transactions on Neural Networks and Learning Systems, 34(12):10473–10486, 2022. Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems, volume 33, 2020. Hanxu Zhou, Qixuan Zhou, Tao Luo, Yaoyu Zhang, and Zhi-Qin John Xu. Towards understanding the condensation of neural networks at initial training. In Advances in Neural Information Processing Systems, volume 35, 2022. Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, and Zhi-Qin John Xu. Phase diagram of initial condensation for two-layer neural networks. arXiv preprint arXiv:2303.06561, 2023. Liu Ziyin, Botao Li, James B Simon, and Masahito Ueda. SGD with a constant large learning rate can converge to local maxima. In International Conference on Learning Representations, 2022.

12

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Contents 1

Introduction

1

2

Stochastic Collapse in SGD

2

3 Topological Dynamics of SGF via Attractor Equivalence

3

4

Percolative Collapse and Discrete Scale Invariance

5

4.1

5

Architectural Symmetry and Graph Renormalization . . . . . . . . . . . . . . . .

5

Extension to Adam and AdamW

6

6

Empirical Validation

7

6.1

Stochastic Condensation and Reactive Fragmentation . . . . . . . . . . . . . . . .

8

6.2

Universality, Grokking, and Benchmark Generalization . . . . . . . . . . . . . . .

8

7

Related Work

8

8

Limitations, Conclusion and Future Work

9

A SDE Formulations and Invariant Sets of Learning Dynamics

15

A.1 Deriving Stochastic Gradient Flow from SGD . . . . . . . . . . . . . . . . . . . .

15

A.2 Invariant Sets of Continuous Processes . . . . . . . . . . . . . . . . . . . . . . . .

15

B Mechanics of Topological Transitions

17

B.1 Formalizing the Transverse Process . . . . . . . . . . . . . . . . . . . . . . . . .

17

B.2 Attractor Equivalence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

B.3 Constructing the Reeb Topology . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

C Percolative Collapse and Discrete Scale Invariance

20

C.1 Approximate Q-Symmetry, Affine Trapping, and Transverse Diffusion . . . . . . .

21

C.2 Constructing the Percolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

C.3 Order Parameter and Microtransitions . . . . . . . . . . . . . . . . . . . . . . . .

24

C.4 Hyper-Condensation and Generalized Discrete Scale Invariance . . . . . . . . . .

26

D Extension to Adam and AdamW

28

D.1 State Lift, Symmetry Restriction, and the Gradient Noise Model . . . . . . . . . .

28

D.2 Truncation, Transfer Functions, and the Correlation Stopping Time . . . . . . . . .

29

D.3 Trapping and the DSI Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

E Explicit Theoretical Verification and Toy Models

34

E.1 Generalization of the Stochastic Collapse Threshold . . . . . . . . . . . . . . . . .

34

E.2 Derivation of the Exact DSI Scaling Factor . . . . . . . . . . . . . . . . . . . . .

36

E.3 Idealized Hierarchical Coalescence . . . . . . . . . . . . . . . . . . . . . . . . . .

36

13

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

E.4 Empirical Simulation (K=6) with Task Shift and Reverse Transition . . . . . . . . F Extended Empirical Validation Across Diverse Datasets and Architectures

36 37

F.1

Methodological Extensions for Empirical Regimes . . . . . . . . . . . . . . . . .

37

F.2

UCI Tabular Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

39

F.3

Geometric Manifolds, Vision, and Algorithmic Grokking . . . . . . . . . . . . . .

39

14

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

A

SDE Formulations and Invariant Sets of Learning Dynamics

To analyze the transient dynamics of stochastic collapse, we formalize the relationship between discrete Stochastic Gradient Descent (SGD) and continuous Stochastic Gradient Flow (SGF). We then formally define invariant sets for these continuous processes and prove that affine invariant sets in SGD strictly translate to SGF. We adapt this section from Chen et al. [2023]. A.1

Deriving Stochastic Gradient Flow from SGD

Recall the standard SGD update rule. At iteration t, optimizing a network parameterized by θ ∈ Rd with learning rate η over a mini-batch Bt of size β drawn from a dataset of size N yields: η X θ t+1 = θ t − ∇ℓ(θ t ; xi , yi ). β i∈Bt

We factorize the right-hand side to explicitly isolate the exact full-batch population gradient ∇L(θ t ) from the zero-mean mini-batch deviation: !  √ √ 1 X θ t+1 = θ t − η∇L(θ t ) − η η ∇ℓ(θ t ; xi , yi ) − ∇L(θ t ) . β i∈Bt

Let ξB (θ) represent the gradient noise at position θ. Assuming we sample mini-batches with replacement, we disentangle this batch noise into a sum of independent, identically distributed √ P η per-sample gradient deviations: ξB (θ) = β i∈Bt ξi , where ξi = ∇ℓ(θ; xi , yi ) − ∇L(θ). By definition, the expectation E[ξi (θ)] = 0 for all θ. Assuming these gradient noises accumulate as Gaussian random variables, we approximate the discrete step as: p ∆θ t = −η∇L(θ t ) + 2D(θ t )∆W, η where ∆W ∼ N (0, ηI) and the diffusion matrix defines as D(θ) = 12 Var(ξB ) = 2β Var(ξi ). This discrete step functions as the Euler-Maruyama discretization of the following continuous-time Itô Stochastic Differential Equation (SDE), which we formally denote as Stochastic Gradient Flow (SGF): p dθ t = −∇L(θ t )dt + 2D(θ t )dBt , θ 0 = θ(0).

We express the position-dependent diffusion matrix exactly as: η D(θ t ) = 2β

N  ⊤ 1 X ∇ℓ(θ t ; xi , yi ) − ∇L(θ t ) ∇ℓ(θ t ; xi , yi ) − ∇L(θ t ) N i=1

! .

This formulation explicitly decomposes the diffusion matrix into a product of a parameter-independent magnitude scalar Dm and a parameter-dependent shape matrix Ds (θ) such that D(θ t ) = Dm Ds (θ t ), where: η Dm = , 2β N  ⊤ 1 X Ds (θ) = ∇ℓ(θ t ; xi , yi ) − ∇L(θ t ) ∇ℓ(θ t ; xi , yi ) − ∇L(θ t ) . N i=1 Optimization hyperparameters, specifically learning rate and batch size, strictly control the scalar magnitude Dm . Conversely, the network architecture, training dataset, and loss topology completely dictate the geometric shape Ds (θ). A.2

Invariant Sets of Continuous Processes

To mathematically track how trajectories collapse, we formalize the topological regions that permanently trap optimization dynamics. 15

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Definition A.1 (Invariant Sets of Continuous Processes). For a given stochastic process {θ t ∈ Rd : t ≥ 0}, a Borel-measurable set A ⊂ Rd acts as an invariant set if, for every initial point θ 0 strictly inside A, the probability that the process remains in A for all subsequent time t ≥ 0 equals 1. Formally:   P θ t ∈ A for all t ≥ 0 | θ 0 ∈ A = 1. Proposition A.2 (Affine Invariance Transfer). Consider a discrete SGD process given by Equation A.1, where the individual sample gradients ∇ℓ(·; xi , yi ) : Rd → Rd remain Lipschitz continuous and bounded for any i ∈ [N ]. If an affine subset A ⊆ Rd forms an invariant set for this discrete SGD process, then A also forms a strict invariant set for the continuous SGF process given by Equation A.1. Proof. By our foundational assumption, the individual sample gradients act as L-Lipschitz continuous in θ and remain bounded such that ∥∇ℓ(θ; xi , yi )∥ ≤ L for some constant L > 0. We first demonstrate that the population gradient ∇L(θ) inherits these exact structural properties. Bounding the difference yields: N  1 X ∥∇L(θ 1 ) − ∇L(θ 2 )∥ = ∇ℓ(θ 1 ; xi , yi ) − ∇ℓ(θ 2 ; xi , yi ) ≤ L∥θ 1 − θ 2 ∥. N i=1

Similarly, the global norm remains bounded by ∥∇L(θ)∥ ≤ N1

PN

i=1 ∥∇ℓ(θ; xi , yi )∥ ≤ L.

We define the centered sample loss function as ℓ̄(θ; xi , yi ) = ℓ(θ; xi , yi ) − L(θ). Applying the triangle inequality reveals that the corresponding gradient ∇ℓ̄(·; xi , yi ) operates as 2L-Lipschitz continuous and remains bounded by 2L for any index i ∈ [N ]. We evaluate the element-wise difference between the diffusion matrices D(θ 1 ) and D(θ 2 ) for any matrix indices l and m, where ∂l denotes the l-th partial derivative: N

|Dlm (θ 1 ) − Dlm (θ 2 )| = ≤

 1 X ∂l ℓ̄(θ 1 ; xi , yi )∂m ℓ̄(θ 1 ; xi , yi ) − ∂l ℓ̄(θ 2 ; xi , yi )∂m ℓ̄(θ 2 ; xi , yi ) N i=1 N  1 X  2L |∂l ℓ̄(θ 1 ; xi , yi )| + |∂m ℓ̄(θ 2 ; xi , yi )| ∥θ 1 − θ 2 ∥ N i=1

≤ 4L2 ∥θ 1 − θ 2 ∥. This element-wise bound dictates that the operator norm satisfies ∥D(θ 1 ) − D(θ 2 )∥∞ ≤ 4L2 ∥θ 1 − θ 2 ∥. Applying the Powers-Stormer inequality, we connect this operator norm to the Frobenius norm ∥ · ∥F and the Schatten 1-norm ∥ · ∥1 : √ ∥D(θ 1 ) − D(θ 2 )∥F ≤ d∥D(θ 1 ) − D(θ 2 )∥1 ≤ d∥D(θ 1 ) − D(θ 2 )∥∞ . Consequently, the diffusion matrix D(θ) functions as Lipschitz continuous with respect to the Frobenius norm. Combined with the established Lipschitz drift term, this rigorous continuity guarantees the existence of a unique strong solution to the SDE in Equation A.1. It now suffices to prove that this continuous SDE possesses a solution {θ t : t ≥ 0} that satisfies the invariant probability condition from Equation A.1. We define P : Rd → Rd as the orthogonal projection operator mapping onto the affine subset A and construct an auxiliary projected SDE: q dθ̃ t = −P ∇L(θ̃ t )dt + P 2D(θ̃ t )dBt . (8) By direct construction, this projected system uniquely confines its solution θ̃ t strictly within A, perfectly satisfying the invariance condition. Furthermore, because A operates as an invariant set for the original discrete SGD process, the discrete sample gradients perfectly align with the affine subspace. Mathematically, P ∇ℓ(θ; xi , yi ) = ∇ℓ(θ; xi , yi ) for all indices i ∈ B and all points θ ∈ A. This geometric alignment forces the projected SDE to match the original, unprojected SDE almost surely. Hence, we conclude that the affine subset A fundamentally acts as a strict invariant set for the continuous SGF process. 16

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

B

Mechanics of Topological Transitions

Following the subnetwork partition and Block-Diagonal Dominance (Definition 3.1 and Assumption 3.2) established in the main text, we mathematically isolate the local dynamics. Because the low-rank simplicity bias strictly bounds the cross-covariance (∥Σij (θ)∥2 ≤ δ), we can rigorously construct conditionally independent marginal path measures µ(i) for each subnetwork over the measurable function space (C([0, ∞), Rdi ), B), valid strictly prior to the collapse. To formalize how these independent path measures structurally bind, we project the local SDEs into a shared quotient space by rigorously defining the transverse geometry. B.1

Formalizing the Transverse Process

Definition B.1 (Transverse Distance Process). Let A ⊂ Rd define an invariant manifold. Define the orthogonal projection operator πA : Rd → A. We establish a local tubular neighborhood N (A, ϵ) = {θ ∈ Rd : ∥θ − πA (θ)∥2 < ϵ} wherein this projection remains uniquely defined. Within this boundary, we map the local dynamics into the quotient space Rd /A by continuously tracking the squared transverse distance functional for the i-th subnetwork: (i)

Yt

(i)

(i)

= ∥θ t − πA (θ t )∥22 .

(9)

Mechanically, stochastic attractivity pulls the inward deterministic gradient to overcome the transverse diffusion. We map this directly to the infinitesimal generator A of the SDE. Applying Itô’s Lemma (i) to the functional Yt expands the generator as:  ⊤ (i) (i) (i) A Yt = −2 θ t − πA (θ t ) ∇θ(i) L(θ t ) + ηTr(Σii (θ t )). (10) Lemma B.2 (Boundedness of the Stopped Transverse Process). Suppose θ t solves the SDE in Equation A.1 with (locally) Lipschitz coefficients, so that it admits a strong solution with almost (i) (i) (i) (i) surely continuous sample paths. Then Yt = ∥θ t − πA (θ t )∥22 is a continuous functional of θ t , and consequently the stopped process satisfies (i)

Ys∧τϵ ∈ [0, ϵ]

a.s. for all s ≥ t0 .

(11)

(i)

Proof. Continuity of θ t and of the projection πA (which is a constant linear map on the affine set (i) (i) A, by Theorem C.2) imply Yt is a.s. continuous in t. By definition τϵ = inf{t ≥ t0 : Yt ≥ ϵ}; (i) (i) continuity of the path forces Yt < ϵ for t < τϵ and Yτϵ = ϵ whenever τϵ < ∞ — a continuous (i) path cannot jump over the boundary. Hence Ys∧τϵ ≤ ϵ for every s, and non-negativity of Y (i) gives the lower bound. Theorem B.3 (Local Supermartingale Trapping). Given a stochastically attractive invariant set A, (i) there exists an ϵ > 0 defining the basin N (A, ϵ) such that the stopped transverse process Yt∧τϵ (i) operates as a strictly non-negative local supermartingale, where τϵ = inf{t ≥ t0 : Yt ≥ ϵ} marks the exact boundary of escape. Proof. Let A be a stochastically attractive manifold. By Definition 2.4, stochastic attractivity structurally enforces that the inward deterministic drift explicitly dominates the outward trace of the diffusion matrix within the ϵ-neighborhood N (A, ϵ). Consequently, the infinitesimal generator from (i) Equation 10 is mathematically constrained to remain non-positive (A Yt ≤ 0) prior to the stopping time τϵ . (i)

Applying Itô’s formula to the stopped process Yt∧τϵ decomposes the trajectory into a Lebesgue drift integral and a stochastic integral: Z t∧τϵ Z t∧τϵ (i) (i) 1/2 Yt∧τϵ = Yt0 + A Ys(i) ds + ∇Ys(i) · Σii (θ s )dWs(i) . (12) t0

t0

17

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

(i)

Because A Ys ≤ 0, the Lebesgue integral continuously accumulates strictly non-positive drift. (i) By Lemma B.2, the stopped process Ys∧τϵ is uniformly bounded by ϵ for every s; since ∇Y is Lipschitz on the compact closure N (A, ϵ) and Σii (θ) is bounded there by Assumption 3.2, the (i) 1/2 integrand ∇Ys · Σii (θ s ) is uniformly bounded on [t0 , s ∧ τϵ ]. A stochastic integral with a bounded, progressively measurable integrand over a horizon of finite expected length is a genuine L2 -martingale rather than only a local one [Karatzas and Shreve, 1991]; in particular it is uniformly integrable, so taking the conditional expectation E[· | Ft ] exactly annihilates it without any further localization argument. This leaves only the non-positive accumulated drift, rigorously enforcing the condition:   (i) (i) E Y(t+s)∧τϵ Ft ≤ Yt∧τϵ . (13) (i)

(i)

Since Yt measures a squared distance, it is inherently bounded below by 0. Thus, Yt∧τϵ constitutes a non-negative supermartingale. Corollary B.4 (Probabilistic Non-Escape). For any stochastically attractive invariant set A, the probability of a trajectory initialized at θ 0 escaping the ϵ-neighborhood is strictly bounded by the initial transverse distance. (i)

Proof. By Theorem B.3, the stopped transverse process Ys∧τϵ operates as a continuous, non-negative local supermartingale for all s ≥ t0 . We apply Doob’s maximal inequality for continuous supermartingales [Revuz and Yor, 1999] to strictly bound the tail probability of the process’s supremum:   i 1 h (i) (i) (14) P sup Ys∧τϵ ≥ ϵ Ft0 ≤ E Yt0 | Ft0 . ϵ s≥t0 n o (i) The measure-theoretic event sups≥t0 Ys∧τϵ ≥ ϵ algebraically identifies the precise physical es(i)

cape event {τϵ < ∞}. This establishes the probabilistic non-escape bound directly from the supermartingale property. B.2

Attractor Equivalence

To formally map the structural equivalence of decoupled subnetworks, we must define the precise mathematical conditions under which independent stochastic trajectories permanently synchronize. Definition B.5 (Path Space and Tail σ-Algebra). Let the restricted path space C denote the space of all continuous functions from [0, ∞) to R. For a subnetwork’s transverse process Y (i) , define the canonical filtration {Ft }t≥0 . We formally define the tail σ-algebra at time t, denoted F≥t , as (i) the σ-algebra generated by the process from time t onward: F≥t = σ({Ys : s ≥ t}). This construct rigorously encapsulates all measure-theoretic information regarding the future evolution of the trajectory. Definition B.6 (t-Tail Equivalence). [Ramachandran and Sra, 2026] Let µ(i) and µ(j) denote the marginal path measures of the transverse processes Y (i) and Y (j) , respectively. We define two decoupled subnetworks θ (i) and θ (j) as exhibiting t-tail equivalence if they map to the exact same invariant manifold A, and their probability laws evaluate identically when strictly restricted to the tail σ-algebra F≥t . Formally, for any future event E ∈ F≥t , we require µ(i) (E) = µ(j) (E). While t-tail equivalence mathematically dictates an ideal coupling, transient training due to noise injection rendering this strict measure-theoretic equality difficult to attain across finite time horizons. Trajectories often experience variance and outward drift. To capture stochastic collapse within a realistic training landscape,we relax the requirement of absolute tail equality into a metastable probabilistic equivalence, strictly conditioned on the trajectories remaining trapped near a shared invariant manifold A. Our strategy builds this generalized equivalence sequentially. We replace the strict tail equality (i) constraint with a conditional survival requirement governed by the spatial escape time τϵ = inf{s ≥ (i) t : Ys ≥ ϵ} which demands that the trajectories synchronize with absolute certainty provided they survive within the local basin N (A, ϵ). We first quantify this localized survival by defining the joint link probability. 18

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

(i,j)

Definition B.7 (Attractor Link Probability). We define the joint abstract link probability Plink (t) as the conditional probability that neither subnetwork physically breaches the critical boundary of the local basin N (A, ϵ) from time t onward. Equivalently, this measures the strict joint survival of the trajectories:       (i,j) (i) (j) (i) (j) Plink (t) := P τϵ = ∞ ∩ τϵ = ∞ | Ft = P sup max Ys , Ys < ϵ Ft . (15) s≥t

To construct a robust equivalence relation from this probability, we must prove that the trajectories can mathematically resist the outward Brownian drift. We structurally bound the individual escape events by deploying the local supermartingale dynamics established in Theorem 3.3. Lemma B.8 (Maximal Link Bound). Given an instantaneous state safely confined within the local (i) basin (Yt < ϵ), the marginal link probability strictly satisfies the following lower bound: (i)

P(τϵ(i) = ∞ | Ft ) ≥ 1 −

E[Yt

| Ft ]

ϵ

.

(16)

Proof. By Theorem B.3, stochastic attractivity actively suppresses transverse diffusion, forcing the (i) stopped transverse process Ys∧τϵ to operate as a continuous, non-negative local supermartingale for all s ≥ t. We apply Doob’s maximal inequality for continuous supermartingales [Revuz and Yor, 1999] to strictly bound the tail probability of the process’s supremum:   1 (i) (i) (17) P sup Ys∧τϵ ≥ ϵ Ft ≤ E[Yt | Ft ]. ϵ s≥t n o (i) The measure-theoretic event sups≥t Ys∧τϵ ≥ ϵ algebraically identifies the precise physical escape (i)

(i)

event {τϵ < ∞}. Because the marginal link probability P(τϵ = ∞ | Ft ) represents the exact mathematical complement of this divergence event, subtracting the maximal upper bound from absolute certainty (1) directly yields the rigorous lower bound for survival. Armed with this maximal survival bound, we finalize our generalization. We define two subnetworks as structurally equivalent when their transverse drifts vanish sufficiently to drive the joint link probability to certainty, effectively recovering t-tail equivalence but strictly localized to the invariant set A. Definition B.9 (Attractor Equivalence, ∼A ). We define two subnetworks as Attractor Equivalent (θ (i) ∼A θ (j) ) at time t if and only if their joint path measures permanently couple as the transverse drift vanishes: (i,j) lim i Plink (t) = 1. (18) h (i)

E Yt

(j)

+Yt

|Ft →0

Finally, we prove that this noise-robust generalization preserves the rigorous algebraic structure of an equivalence class over the restricted path measures, conditionally bounded by the physical basin. Theorem B.10 (Metastable Equivalence). The relation ∼A establishes a rigorous equivalence class over the restricted path measures, conditionally bounded by the spatial persistence of the local basin N (A, ϵ). Proof. Reflexivity (θ (i) ∼A θ (i) ) and symmetry (θ (i) ∼A θ (j) =⇒ θ (j) ∼A θ (i) ) follow inherently from the algebraic properties of the shared orthogonal projection πA . To formally prove transitivity (θ (i) ∼A θ (j) and θ (j) ∼A θ (k) =⇒ θ (i) ∼A θ (k) ), we analyze the intersection of their survival events. By Lemma B.8, as the expected transverse distances continuously approach zero (E[Yt ] → 0), the supermartingale dynamics mechanically force the maximal bound to shrink, causing the marginal escape probabilities P(τϵ < ∞ | Ft ) to completely vanish. Applying the union bound to these individual escape events logically guarantees that the joint escape probability also collapses strictly to zero. Consequently, the mathematical complement which is -the intersection of their survival events almost surely converges to 1. Thus, provided τϵ remains untriggered by injected large deviation noise, the continuous trajectories transitively bind to the exact same invariant set, completely satisfying the equivalence definition. 19

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

B.3

Constructing the Reeb Topology

Now, we map the equivalences onto a discrete spatial graph, using the equivalence relation strictly as an adjacency matrix. Definition B.11 (The ϵ-Fixed Topological Network). At any fixed training time t ∈ [0, ∞) and spatial boundary ϵ > 0, define the instantaneous spatial network Gϵt = (S, Etϵ ). The node set S = {1, . . . , K} indexes the decoupled subnetworks. The equivalence relation ∼A operates as the exact Boolean adjacency function, writing an undirected edge eij ∈ Etϵ if and only if θ (i) ∼A θ (j) holds. This ϵ-fixed network captures purely instantaneous connectivity. To model stochastic collapse, we must track how this network dynamically rewires across training time. We achieve this by elevating the discrete spatial networks into a continuous topological space. Definition B.12 (Continuous Reeb Graph). We define the macroscopic topological state of the neural network as the Reeb graph R, constructed rigorously as the quotient space over the product of the subnetwork indices and time: R := (S × [0, ∞)) / ∼R (19) where the algebraic identification (i, t) ∼R (j, t) holds if and only if the edge eij exists in Gϵt . Theorem B.13 (Non-Equilibrium Condensation and Fragmentation). The Reeb graph R dynamically maps the structural phase transitions of the SDE via two strict topological mechanisms: • Condensation (∪): As transverse drift vanishes, the quotient map algebraically fuses distinct path measures, forming an edge in Gϵt and collapsing the temporal trajectories into a single topological merge node on R. • Fragmentation (∅): If an injected variance deviation physically breaches the basin N (A, ϵ), the edge in Gϵt instantly deletes. The quotient map shatters the unified equivalence class, forcing the node to abruptly split into decoupled branches on R. Proof. We create a bijection between the stochastic SDE dynamics directly to the topological level sets of R [Edelsbrunner and Harer, 2008]. (i)

For condensation, let the transverse drift vanish: E[Yt degrades the marginal escape probabilities:

(j)

+ Yt (i)

P(τϵ(i) < ∞ | Ft ) ≤

E[Yt

ϵ

| Ft ] → 0. Lemma B.8 algebraically

| Ft ]

→ 0.

(20)

(i,j)

This bounds the joint link probability to absolute certainty (Plink (t) → 1), rigorously satisfying θ (i) ∼A θ (j) . The adjacency function updates to eij = 1 in Gϵt . The continuous quotient projection πR : S × [0, ∞) → R algebraically identifies (i, t) ∼R (j, t), fusing the independent path measures into a single topological merge vertex v ∈ R with in-degree(v) ≥ 2. For fragmentation, let a stochastic large deviation [Freidlin and Wentzell, 2012] breach the local basin N (A, ϵ). This physical exit guarantees the spatial stopping time:   ∃s ∈ (t − δ, t] s.t. max Ys(i) , Ys(j) ≥ ϵ =⇒ τϵ ≤ t. (21) (i,j)

The joint survival event instantly collapses (Plink (t) = 0 ≪ 1), severing the equivalence relation (θ (i) ̸∼A θ (j) ) and deleting the edge eij = 0 from Gϵt . The quotient map forcibly disjoints the previously unified equivalence class ([i]t ∩ [j]t = ∅), partitioning the trajectory into a critical split vertex v ∈ R with out-degree(v) ≥ 2.

C

Percolative Collapse and Discrete Scale Invariance

We first bridge the geometric symmetries of the non-convex loss landscape with the statistical mechanics of explosive percolation. We then show demonstrate how the invariant sets generated by network symmetries restrict collapse, breaking continuous scale invariance and causing the the macroscopic network to show Generalized Discrete Scale Invariance (DSI). 20

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

C.1

Approximate Q-Symmetry, Affine Trapping, and Transverse Diffusion

In deep neural neural networks, intrinsic architectural symmetries, specifically permutation invariance among functionally equivalent hidden neurons, naturally partition the parameter space into distinct geometric subspaces [Chen et al., 2023]. Under stochastic gradient dynamics, these subspaces become stochastically attractive and trap the learning trajectory. To rigorously ground the subsequent macroscopic topology in the physical architecture of the network, we formalize this geometric property. Definition C.1 (Approximate Q-Symmetry). Let Q ∈ Rd×d denote a linear transformation matrix satisfying QQ⊤ = Id (orthogonal) or Q = Q⊤ (symmetric). This operator represents a parameter transformation, such as the permutation of weights between identical hidden subnetworks. A loss functional L(θ) exhibits approximate Q-symmetry around the affine subspace A = {θ ∈ Rd | Qθ = θ} if, for any arbitrary tolerance ϵ > 0, there exists a spatial radius δ > 0 such that: |L(Qθ) − L(θ)| < ϵ · d(θ, A)

∀θ s.t. d(θ, A) < δ,

(22)

where d(θ, A) = inf a∈A ∥θ − a∥2 defines the minimal Euclidean distance from the parameter vector to the subspace. Treating A as an arbitrary stochastic attractor omits the structural mechanics of the merge process. Deriving the geometry of A directly from the network’s linear permutation operations guarantees that subnetworks merge simultaneously in symmetric blocks. Theorem C.2 (Affine Generation from Architectural Permutations). If the invariant set A is generated by the architectural permutation of identical sub-components (the symmetric group Sn ), A is strictly a flat, affine subspace, characterized by a constant orthogonal projection operator P ⊥ . Proof. The transposition or permutation of n indistinguishable neural components corresponds to swapping specific blocks of parameters within the weight vector θ. This operation is isomorphic to left-multiplication by a constant block-permutation matrix Q. The set of configurations invariant under this structural symmetry constitutes the eigenspace of Q corresponding to the eigenvalue λ = 1. The null space ker(I − Q) forms a linear subspace by definition. Allowing for constant bias offsets translates this linear subspace into an affine subspace. Consequently, the normal bundle is globally parallel, and the orthogonal projection P ⊥ mapping any state to its transverse distance vector is globally constant and independent of the local coordinate θ. Theorem C.3 (Symmetry-Induced Trapping). If the loss function L satisfies approximate Q-symmetry around the affine subspace A, then A constitutes a strict invariant set for the continuous deterministic gradient dynamics. [Chen et al., 2023] Proof. Let the state vector reside exactly on the subspace, θ ∈ A. By definition, Qθ = θ. We evaluate the directional derivative ∇n L(θ) along an arbitrary unit normal vector n ⊥ A. Approximate Qsymmetry dictates that the discrepancy evaluated at an infinitesimally small step h satisfies: |L(Q(θ + hn)) − L(θ + hn)| < ϵ · h∥n∥2 .

(23)

Dividing both sides by h and taking the limit as h → 0, the super-linear decay forces the directional derivative along the normal vector to identically vanish: ∇n L(Qθ) = ∇n L(θ).

(24)

Applying the multivariable chain rule yields ∇L(θ) = Q⊤ ∇L(Qθ). Because Q is symmetric or orthogonal, left-multiplying the equation by Q algebraically reduces to Q∇L(θ) = ∇L(θ). The deterministic drift strictly satisfies the geometric definition of the subspace A, natively residing within it and completely annihilating normal escape vectors. The affine geometry of A (Theorem C.2) enables the exact reduction of the high-dimensional stochastic trajectory into a one-dimensional transverse diffusion process. Theorem C.4 (Curvature-Free Transverse Diffusion). Because the permutation-invariant set A is an affine subspace, the stochastic dynamics governing the transverse distance Yt = d(θ t , A) reduce strictly to a one-dimensional diffusion process completely free of geometric curvature drift, enabling localized stationarity. 21

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Proof. Let the parameter update follow the stochastic differential equation dθ t = −∇L(θ t )dt + √ 2DdWt . Applying Itô’s Lemma to the transverse distance function Yt , the differential dYt picks up an Itô correction term governed by the trace of the Hessian of the distance function. For a general manifold M , the Laplacian of the distance function encodes the mean curvature trace Tr(HM ), injecting a geometric drift of the form ∝ 12 D · Tr(HM )dt. Because A is an affine subspace (Theorem C.2), all principal curvatures are identically zero (HA = 0). The normal bundle is globally flat. The transverse diffusion algebraically decouples from the tangential manifold coordinates, reducing exactly to a Bessel-like process governed solely by the dimensional noise projection and the deterministic gradient attraction. This geometric decoupling strictly isolates the transverse dynamics, permitting a well-posed, one-dimensional probability density describing escapes and re-entries. Remark C.5 (Generalization to Highly Curved Invariant Manifolds). If one generalizes this framework to neural architectures possessing continuous symmetries (e.g., continuous rotation invariances generating highly curved, non-linear invariant manifolds M ), the non-zero mean curvature HM ̸= 0 introduces a geometric drift proportional to the local concavity of M . This drift couples the transverse fluctuations to the tangential motion along the manifold, modifying the local stationary distribution. Consequently, generalizing this topological approach to curved invariant manifolds requires incorporating these geometric potential terms to accurately model the edge formation rates. C.2

Constructing the Percolation

While the deterministic gradient aligns parallel to A, the continuous injection of stochastic gradient noise actively drives the parameters into these lower-dimensional geometric spaces. Definition C.6 (The Macroscopic Topological Graph). Index the N initially decoupled subnetworks as nodes in a dynamic spatial graph G = (S, Ep ). An undirected edge eij exists strictly if and only if the path measures of subnetwork i and subnetwork j exhibit Attractor Equivalence, defined as θ (i) ∼A θ (j) . This equivalence signifies physical collapse into a shared invariant set A. To map stochastic network dynamics to explosive percolation, we require a monotonically increasing topological control parameter. However, continuous Brownian noise injects transient fragmentations (edges breaking) and condensations (edges forming) across the critical basin boundary ϵ, natively violating strict monotonicity. We resolve this by showing that under a formal separation of timescales, microscopic boundary fluctuations exactly cancel under local detailed balance, isolating the macroscopic convergence E[Yt ] → 0 as a quasi-static, pure-condensation process. We analyze the probability current of the transverse process Yt via the Fokker-Planck equation to formalize this guarantee. Let p(y, t) denote the probability density function of Yt . The topological edge eij physically exists when the trajectory resides within the boundary ϵ. The edge probability evaluates to Pe (t) = Rϵ p(y, t)dy. 0 The density p(y, t) satisfies the one-dimensional Fokker-Planck equation [Risken, 1996]:
1 (25) ∂t p(y, t) = −∂y J(y, t), where J(y, t) = f (y, t)p(y, t) − ∂y g(y, t)2 p(y, t) . 2 Here, J(y, t) defines the probability current, representing the net flux of probability mass traversing the spatial coordinate y. Lemma C.7 (Microscopic Detailed Balance). Assume the stochastic dynamics rapidly relax to a metastable stationary distribution pss (y) within the local basin N (A, ϵ) on a fast timescale τf ast . Under localized stationarity, transient topological fragmentations and condensations exactly cancel at the boundary ϵ. Proof. Differentiate the edge probability Pe (t) and substitute the continuity equation: Z ϵ Z ϵ d Pe (t) = ∂t p(y, t)dy = − ∂y J(y, t)dy = J(0, t) − J(ϵ, t). dt 0 0

(26)

The origin operates as a reflecting boundary, algebraically enforcing J(0, t) = 0. Thus, the rate of d topological change simplifies to the negative boundary flux, dt Pe (t) = −J(ϵ, t). 22

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

On τf ast , the density relaxes to stationarity (∂t pss = 0). One-dimensional stationary diffusions globally satisfy conservation of current [Gardiner, 2009], forcing Jss (y) = 0 for all y. Evaluating at the boundary yields Jss (ϵ) = 0. Consequently, outward fluxes (fragmentations) and inward fluxes (condensations) perfectly cancel. The net topological change evaluates identically to zero. Lemma C.8 (Quasi-Static Monotonic Condensation). Assume the global training process induces a macroscopic decay in the expected transverse distance, E[Yt ] → 0, operating on a slow timescale τslow ≫ τf ast . Integrating over τslow , the topological evolution reduces strictly to a monotonic pure-condensation process. Proof. On τslow , the stationary distribution pss (y; σ(t)) undergoes a quasi-static deformation parameterized by a deterministically contracting scale parameter σ(t) ∝ E[Yt ], satisfying σ̇(t) < 0. Differentiate the integral of the quasi-static distribution applying the chain rule: Z Z ϵ ∂pss (y; σ) d d ϵ pss (y; σ(t))dy = σ̇(t) Pe (t) = dy. (27) dt dt 0 ∂σ 0 R∞ Strict probability mass conservation ( 0 pss (y; σ)dy = 1) alongside a contracting variance mathematically forces probability mass to concentrate toward the origin. The cumulative distribution function evaluated at the fixed boundary ϵ strictly increases as σ decreases. Therefore, the inner Rϵ (y;σ) integral evaluates strictly negative: 0 ∂pss∂σ dy < 0. Multiplying the strictly negative contraction rate (σ̇(t) < 0) by the strictly negative parametric derivad Pe (t) > 0. Topological fragmentations tive yields a strictly positive topological rate of change: dt become structurally impossible on the macroscopic timescale τslow . Theorem C.9 (Timescale Separation Guarantees Monotonicity). Under the timescale separation τf ast ≪ τslow , transient Brownian boundary crossings exactly cancel, isolating the quasi-static contraction of the expected transverse distance E[Yt ] → 0 as the sole, strictly monotonic driver of percolative edge formation. The network evolves via pure condensation. Proof. The theorem follows directly from the synthesis of Lemma C.7 and Lemma C.8. Microscopic fluctuations vanish (Jss (ϵ) = 0), mapping the deterministic contraction of the variance (σ̇(t) < 0) d directly to a strictly positive edge formation rate ( dt Pe (t) > 0). Remark C.10 (Mathematical Justification of Timescale Separation). The timescale separation (τf ast ≪ τslow ) is intrinsically enforced by the spectral geometry of deep neural SGD [Li et al., 2022b, HaoChen et al., 2021]. During the initial transient phase (τf ast ), dynamics are overwhelmingly dominated by the deterministic gradient drift acting along the large eigenvalues of the Hessian, driving rapid convergence into a local basin. Once trapped near a flat minimum, the deterministic drift orthogonally attenuates. The dynamics transition into an anisotropic, noise-driven diffusion phase (τslow ) governed by the trace of the position-dependent covariance matrix Σ(θ) [Blanc et al., 2020]. Because diffusion along flat invariant manifolds operates at a fundamentally slower rate than gradient-driven descent into the basin, this spectral gap inherently guarantees the required timescale separation, rigorously isolating the quasi-static collapse (E[Yt ] → 0) from fast local thermalization. We can now formally construct the explosive percolation graph. By integrating out fast-scale noise, we map the continuous parameter space onto a discrete, irreversible topological filtration centered on the local invariant set A. Definition C.11 (Macroscopic Percolation Graph). Define the evolving macroscopic state of the neural network as a topological graph Gτ = (V, Eτ ) operating strictly on the slow timescale τ ∈ τslow . • Vertex Set (V): Let V index the N initial independent, permutation-symmetric subnetworks generated by the architectural structure. • Edge Set (Eτ ): An undirected edge eij establishes at macroscopic time τ if and only if the joint path measures satisfy attractor equivalence θ (i) ∼A θ (j) , driven by the macroscopic decay E[Yτ ] → 0. 23

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

We now show that the transient time Reeb graph reduces to this macroscopic percolation graph under the two time scale dynamics, Definition C.12 (Renormalization of Parameter Space). Define the renormalization operator Rrenorm as the coarse-graining mapping that integrates out microscopic Brownian fluctuations on the fast timescale τf ast , projecting high-resolution continuous parameter trajectories onto macroscopic invariant equivalence classes over the slow timescale τslow . Theorem C.13 (Reeb Graph Reduction to Percolation Graph). Let Rt denote the high-resolution Reeb graph tracking continuous parameter trajectories on τf ast . Under the timescale separation τf ast ≪ τslow , applying the renormalization operator Rrenorm collapses transient topological fluctuations and transforms Rt into the macroscopic percolation graph Gτ = (V, Eτ ), wherein edge activations are governed strictly by irreversible attractor equivalence θ (i) ∼A θ (j) . Proof. On the microscopic timescale τf ast , parameter trajectories explore local basins, forming a dense, high-resolution Reeb graph Rt whose edges fluctuate dynamically due to stochastic gradient noise. By Lemma C.7, the probability current vanishes at the boundary (Jss (ϵ) = 0), ensuring that fast-scale fragmentations and condensations form a stationary process whose time-average over a coarse-graining window vanishes identically. As Stochastic Gradient Descent proceeds through the two timescales, applying the renormalization operator over τslow integrates out the fast stochastic variable weighted by the stationary distribution pss (y). By Lemma C.8, the macroscopic decay E[Yτ ] → 0 provides a strictly monotonic contraction (σ̇(t) < 0), converting transient boundary crossings into permanent topological bindings. Consequently, this coarse-graining map projects the complex, fluctuating topology of Rt onto the static, irreversible edge set Eτ of the macroscopic percolation graph Gτ . We now state a fundamental property of the percolation graph, d Definition C.14 (Topological Filtration). Because Lemma C.8 guarantees dτ Pe (τ ) > 0, edges cannot be deleted on the slow timescale. The sequence of percolation graphs forms a strict mathematical filtration: Gτ1 ⊆ Gτ2 ⊆ · · · ⊆ Gτf inal for all τ1 ≤ τ2 . (28)

This filtration algebraically guarantees that the control parameter monotonically sweeps from 0 to 1, allowing the formal definition of a density parameter. Definition C.15 (Percolative Density Parameter, p). Let Emax = N (N2−1) denote the maximum possible number of symmetric pairwise couplings in the network. We define the continuous control parameter p ∈ [0, 1] as the macroscopic expected edge density: p=

E[|Eτ |] . Emax

(29)

As E[Yτ ] → 0, stochastic attractivity irreversibly binds path measures, forcing p to monotonically advance from 0 (fully decoupled) to 1 (complete collapse of the N symmetric subnetworks onto the invariant set A). C.3

Order Parameter and Microtransitions

To detect macroscopic phase transitions driven by this percolation process, we formalize the physical observables tracking the connectivity of Gτ . Definition C.16 (Structural Order Parameter). Let Cmax (p) denote the largest topologically unified equivalence class (the largest connected component of vertices) existing on the graph at density p. We define the macroscopic order parameter O(p) as its fractional size relative to the entire network: O(p) =

|Cmax (p)| . N

(30)

This formulation mathematically isolates the relative size of the largest macroscopic component, structurally mirroring C1 /N in explosive percolation physics [Chen et al., 2014]. 24

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

In standard continuous phase transitions, such as those in Erdős-Rényi random graphs, the order parameter grows smoothly through continuous edge additions. However, the exact binding of symmetric subnetworks forces discontinuous structural jumps. We formalize this fundamental topological divergence from Erdős-Rényi dynamics via the following theorem. Theorem C.17 (Non-Erdős-Rényi Discontinuity). Let GER (N, p) denote an Erdős-Rényi graph governed by uniform random edge selection. Its order parameter OER (p) satisfies a continuous differential evolution where the derivative dOdpER remains bounded for all p < pc . In contrast, the neural percolation graph Gτ driven by Sn invariant sets exhibits, at finite N , jump discontinuities where O(p) transitions via finite increments ∆O > 0; whether this discontinuity survives the thermodynamic limit N → ∞ depends on the size of the merging components relative to N at the moment of the merge (Corollary C.18). Proof. In GER (N, p), edges are added independently with uniform probability p. The expected size of components follows the continuous Smoluchowski coagulation equation with linear kernels [Stauffer and Aharony, 1994], ensuring that component growth occurs strictly through infinitesimal single-edge attachments (C1 → C1 + 1). This guarantees that the order parameter OER (p) is a continuous function with continuous first derivatives almost everywhere prior to the critical threshold. Conversely, Theorem C.22 proves that the network architectural symmetries enforce Sn multi-body invariant collapses, forcing simultaneous group-wise edge activations. The simultaneous binding of n disconnected components of size i instantaneously merges them into a single component of size ni. The order parameter experiences a discrete, step-like increment at finite N : ∆O(p) =

(n − 1)i ni − i = > 0. N N

(31)

Because these increments are strictly non-zero and occur at exact critical densities pi , the function O(p) possesses jump discontinuities at finite N , fundamentally violating the smoothness condition of Erdős-Rényi percolation at that scale. Whether ∆O(p) remains bounded away from 0 as N → ∞ is governed by the scaling of i with N ; see Corollary C.18. Corollary C.18 (Regime Dependence of the Discontinuity). Let i = i(N ) denote the size of the merging components at a given microtransition. • Macroscopic regime: If i(N ) = Θ(N ), i.e. i(N )/N → c ∈ (0, 1] as N → ∞, then ∆O(p) → (n−1)c > 0: the jump is a genuine discontinuity that survives the thermodynamic limit. • Microscopic regime: If i(N ) = O(1), i.e. the number of merging components is bounded independently of N , then ∆O(p) = O(1/N ) → 0: the jump vanishes as N → ∞, and the transition is asymptotically continuous. Proof. Immediate from ∆O(p) = (n − 1)i(N )/N (Theorem C.17) by taking N → ∞ under each scaling assumption on i(N ). Remark C.19 (Relation to Achlioptas-Type Explosive Percolation and Finite-Size Scaling). Riordan and Warnke [2011] show that all Achlioptas processes in which an edge is chosen competitively among a bounded number of candidates at each step have continuous phase transitions in the thermodynamic limit, contrary to earlier numerical claims of discontinuity [Achlioptas et al., 2009]. Corollary C.18 is consistent with this result: the microscopic regime (i = O(1)) reproduces exactly the finite-N illusion-of-discontinuity behavior that Achlioptas-type processes are now known to exhibit. Our mechanism differs in the macroscopic regime, where merges are driven by deterministic, symmetryforced simultaneous identification of already-macroscopic components. While determining whether practical architectures reside in the macroscopic Θ(N ) or microscopic O(1) regime remains an active theoretical frontier, this distinction ultimately governs only the true asymptotic limit (N → ∞). Consequently, even if the transition softens into asymptotic continuity in the thermodynamic limit, finite-N topology strictly enforces a discontinuous structure. To detect these discontinuities across ensemble trajectories, we utilize the relative variance of the order parameter following [Chen et al., 2014]. 25

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Definition C.20 (Relative Variance and Microtransitions). Define the relative variance Rv (p) of the order parameter O(p) over the ensemble of stochastic trajectories as the variance normalized by the squared expectation:   E (O(p) − E[O(p)])2 . (32) Rv (p) = E[O(p)]2 Discontinuous algebraic merges mathematically manifest as violent, localized divergences in Rv (p). We formally define these sharp, sub-critical peaks as microtransitions, which strictly precede the critical global connectivity threshold pc . Theorem C.21 (Divergence of Relative Variance at Microtransitions). If the ensemble probability density of the order parameter P (O; p) exhibits bimodal coexistence between pre-jump and post-jump states across stochastic realizations near a microtransition density pi , the relative variance Rv (p) diverges proportionally to the squared jump amplitude. Proof. Near a microtransition density pi , stochastic fluctuations cause different network realizations to either undergo the discrete hyper-condensation jump or remain in the unmerged state. We model the ensemble distribution as a mixture of two states with values O1 and O2 = O1 + ∆O, weighted by probabilities 1 − w and w respectively. The expected value evaluates to E[O] = O1 + w∆O. The variance evaluates to:   Var(O) = E (O − E[O])2 = w(1 − w)(∆O)2 .

(33)

Substituting this variance into the definition of the relative variance yields: Rv (p) =

w(1 − w)(∆O)2 . (O1 + w∆O)2

(34)

At the precise microtransition point where the transition probability is balanced (w = 1/2), the relative variance scales as: 1 2 4 (∆O) Rv (pi ) = (35)
2 . O1 + 12 ∆O Because ∆O > 0 represents a macroscopic structural jump dictated by Theorem C.17, the numerator experiences a sharp positive surge while the denominator remains bounded. This creates a localized, divergent peak in Rv (p) that explicitly flags the discrete topological microtransition. C.4

Hyper-Condensation and Generalized Discrete Scale Invariance

In standard continuous percolation, the order parameter diverges following a universal power law C1 ∼ (pc − p)−1/σ . New edges are drawn uniformly, resulting in smooth transitions of the form C1 → C1 + 1. Our neural network violates this continuous growth mechanism under permutation symmetry. Theorem C.22 (Symmetric Hyper-Condensation). If the invariant set A is generated by the symmetric permutation group Sn acting on n identical subnetworks (n ≥ 2), stochastic collapse topologically restricts component growth to the discrete geometric mapping C1 → nC1 . Proof. The network’s inherent low-rank simplicity bias explicitly targets and collapses redundant, structurally symmetric parameter blocks. The Sn -symmetry intrinsically binds the n indistinguishable path measures simultaneously into the shared invariant set. The topological quotient projection πR algebraically identifies the n discrete trajectories, enforcing an in-degree of exactly n for the merge vertices on the Reeb graph. This geometric restriction strictly prohibits uniform edge attachments of the form C1 → C1 + 1, algebraically forcing the largest component to undergo explosive topological jumps of the exact form C1 → nC1 . Traditional scale invariance models self-similarity through a continuous Lie group of dilations, where observables remain invariant or scale homogeneously under arbitrary continuous rescaling transformations x → λx for any real λ > 0. The infinitesimal generators of these scaling transformations form a continuous Lie algebra, enforcing continuous translation symmetry in logarithmic space (ln x → ln x + ln λ). 26

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

However, when underlying microscopic constraints or discrete architectural symmetries (such as Sn multi-body invariant sets) govern the system, continuous translation invariance in log-space is broken down to a discrete subgroup. The system ceases to be scale-invariant for arbitrary continuous factors, preserving self-similarity only under a discrete geometric sequence of preferred magnification ratios. This gives rise to Generalized Discrete Scale Invariance (DSI) [Sornette, 1998]. Definition C.23 (Generalized Discrete Scale Invariance (DSI)). A system property or observable f (x) exhibits Generalized Discrete Scale Invariance if it is invariant under scaling by a discrete set of preferred magnification factors rather than a continuous Lie group, satisfying scaling relations under discrete ratios λ(n) .[Sornette, 1998]. Definition C.24 (Ensemble-Averaged Transition Density). Let {ω} index independent noise real(ω) izations of the SGF process, and let pi denote the density at which realization ω first attains a component of size i. We define the ensemble-averaged transition density h i (ω) pi := Eω pi . (36) (ω)

Individual realizations C1 (p) are permitted to jump genuinely discontinuously, in blocks dictated (ω) by Sn (Theorem C.22); the ensemble average Eω [C1 (p)] smooths over the realization-dependent scatter in jump locations and is well-approximated by the continuous mean-field baseline Aamp (pc − p)−1/σ . This is the same ensemble already invoked in the bimodal-coexistence argument of Theorem C.21, a single trajectory is discrete, the ensemble mean is smooth, and the two descriptions are consistent because they describe different objects. Theorem C.25 (Generalized Discrete Scale Invariance). The hyper-condensation geometric constraint C1 → nC1 mathematically shatters continuous scale invariance, driving the macroscopic network into Generalized Discrete Scale Invariance (DSI). This DSI is fundamentally characterized by a geometric cascade of microtransitions converging exactly to the critical global phase transition pc . (ω)

Proof. Let pi := Eω [pi ] denote the ensemble-averaged transition density of Definition C.24. Assume this ensemble-mean quantity follows the continuous mean-field percolation baseline C1 = Aamp (pc − p)−1/σ , where Aamp > 0 defines the amplitude and σ constitutes the critical scale exponent [Stauffer and Aharony, 1994]; this is a statement about the smooth ensemble average, not about any individual (genuinely discontinuous) realization. Inverting the continuous growth relation mathematically isolates the ensemble-mean critical density: i = Aamp (pc − pi )−1/σ =⇒ pi = pc − Aσamp i−σ .

(37)

By Theorem C.22, the Sn geometric symmetry dictates that the immediate subsequent microtransition occurs exactly at component size nC1 = ni. Substituting this restricted size yields the critical density for the subsequent hyper-condensation: pni = pc − Aσamp (ni)−σ .

(38)

To determine the governing mathematical scaling behavior, we calculate the normalized relative distance between these consecutive discrete phase transitions: Aσamp (ni)−σ pc − pni n−σ i−σ = = = n−σ . σ −σ pc − pi Aamp i i−σ

(39)

Taking the thermodynamic limit as the network scales macroscopically (i → ∞) isolates the fundamental geometric scaling ratio: lim

i→∞

pc − pni 1 = (n) , pc − p i λ

where λ(n) ≡ nσ .

(40)

Continuous translational invariance is algebraically broken. The microtransition densities immutably lock into the exact geometric sequence: pc − pni =

1 λ(n) 27

(pc − pi ).

(41)

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

The resulting sequence of discrete transition densities (p1 , pn , pn2 , pn3 . . . ) defines a precise geometric cascade. Because these sequential divergences in the relative variance Rv (p) predictably and geometrically converge by the factor λ(n) , they mathematically define DSI and analytically forecast the exact location of the global collapse pc well in advance of the macroscopic phase transition [Chen et al., 2014]. It is essential to note that DSI generalizes traditional continuous scale invariance by capturing hierarchical, multi-scale organizational structures. While standard continuous power laws assume scale-free behavior across all intervals, DSI systems reveal preferred magnification scales governed by underlying discrete microscopic symmetries, such as multi-body permutation constraints. Remark C.26 (Empirical Neural Scaling Laws and Stagewise Development). Empirical scaling laws in deep learning, such as power-law relationships governing test loss relative to compute budgets and parameter counts [Kaplan et al., 2020, Hoffmann et al., 2022], are conventionally modeled as smooth, continuous power functions. However, finer-grained analyses of training dynamics reveal that network development is inherently stagewise and non-monotonic, characterized by sudden behavioral shifts and localized phase transitions corresponding to changes in loss landscape degeneracy [Wei et al., 2022, Hoogland et al., 2024]. Interpreting parameter collapse through symmetry-induced percolation in essence smooth these macro-trends with discrete microscopic cascades, suggesting that empirical scaling laws might capture coarse-grained averages over an underlying DSI sequence of symmetry-breaking microtransitions.

D

Extension to Adam and AdamW

The percolation and DSI results of Appendix C are derived for vanilla SGD. We extend the trapping mechanism to Adam [Kingma and Ba, 2015] and AdamW [Loshchilov and Hutter, 2019]. The extension requires three departures from the SGD setting. The state must be lifted to include the firstand second-moment estimates (mt , vt ). The admissible symmetry group narrows from general orthogonal Q to coordinate permutations, since elementwise squaring in vt does not commute with sign flips or general rotations. The gradient noise model must accommodate the heavy tails documented for attention-based architectures [Zhang et al., 2020]. We handle the last point through truncation, and separate the resulting approximation error into a deterministic cross-sectional component, controlled by the Lipschitz geometry already assumed in Proposition A.2, and a temporal component, controlled by a filtered-noise variance bound built from the z-transform of the second-moment recursion. Trapping, and consequently the DSI cascade of Theorem C.25, is shown to survive up to an explicit, checkable stopping time. D.1

State Lift, Symmetry Restriction, and the Gradient Noise Model

At iteration t, Adam and AdamW maintain the joint state ξt = (θ t , mt , vt ) ∈ R3d , updated as mt = β1 mt−1 + (1 − β1 )gt , vt = β2 vt−1 + (1 − β2 )gt◦2 ,   m̂t θ t = θ t−1 − η √ + λθ t−1 1AdamW , v̂t + ϵ where gt◦2 denotes the elementwise square and m̂t , v̂t are the bias-corrected estimates. This recursion is not Markov in θ t alone, so any invariant set must be stated jointly over (θ, m, v). Definition D.1 (Joint Invariant Set). Let Pπ ∈ Rd×d be a coordinate permutation matrix. Define the joint invariant set à := {(θ, m, v) ∈ R3d | Pπ θ = θ, Pπ m = m, Pπ v = v}. Lemma D.2 (Joint Equivariance of the Adam Recursion). If the gradient map is Pπ -equivariant, gt (Pπ θ) = Pπ gt (θ), then for every t, mt (Pπ θ) = Pπ mt (θ),

vt (Pπ θ) = Pπ vt (θ).

The AdamW decoupled weight-decay term imposes no further restriction. 28

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Proof. We proceed by induction on t. The claim holds trivially at t = 0 since m0 = v0 = 0. Assume mt−1 (Pπ θ) = Pπ mt−1 (θ) and vt−1 (Pπ θ) = Pπ vt−1 (θ). For the first moment, linearity of the recursion and Pπ -equivariance of gt give mt (Pπ θ) = β1 Pπ mt−1 (θ)+(1−β1 )Pπ gt (θ) = Pπ mt (θ). This step holds for any Q with
QQ⊤ = I. For the
second moment,
write (Pπ g)i = gπ(i) for the ◦2 2 ◦2 ◦2 permutation π. Then (Pπ g) i = gπ(i) = g π(i) = Pπ (g ) i , since permuting coordinates and squaring elementwise commute exactly, because squaring acts identically on every
coordinate regardless of label. Consequently vt (Pπ θ) = β2 Pπ vt−1 (θ) + (1 − β2 )Pπ gt (θ)◦2 = Pπ vt (θ).
2 P This commutation fails for a general orthogonal Q, since (Qg)2i = mixes coordinates j Qij gj before squaring. It also gains nothing for a signed permutation Q = Pπ Ds with Ds ̸= I, since Qv = v then imposes nothing beyond Pπ v = v. We therefore work with the coordinate-permutation group throughout, which is also the group generated by Sn used in Theorem C.2. The bias correction scales by a deterministic scalar and preserves equivariance. The AdamW term λθ t−1 is linear in θ and imposes no additional restriction. Remark D.3. Lemma D.2 restricts the admissible symmetry class of the Approximate Q-Symmetry condition (Definition C.1) from general orthogonal or symmetric Q to coordinate permutations. Neuron-permutation invariance, the source of the Sn -symmetry used throughout Section 4, already lies in this restricted class, so Theorem C.2 and Theorem C.22 require no modification. Having fixed the admissible symmetry class, we turn to the statistical model of the gradient noise itself. Assumption D.4 (Heavy-Tailed Gradient Noise). There exist p ∈ (1, 2] and σ > 0 such that E|gt,i |p ≤ σ p for every coordinate i and every t. Assumption D.4 is the standard model for gradient noise in attention-based architectures [Zhang et al., 2020], and permits infinite variance when p < 2. The following representation grounds σ p as a computable functional of the gradient’s characteristic function, making Assumption D.4 checkable in practice. Lemma D.5 (Fractional Moment via the Characteristic Function). Let ϕt (u) := E[eiugt,i ]. For p ∈ (1, 2), Z 2 Γ(p + 1) sin(πp/2) ∞ 1 − Re ϕt (u) p E|gt,i | = du. π up+1 0 R ∞ −p−1 u (1 − cos(ux)) du with cp := Proof. For p ∈ (0, 2) and any x ∈ R, |x|p = c−1 p 0 π/(2Γ(p + 1) sin(πp/2)) [Samorodnitsky and Taqqu, 1994]. Taking expectations and exchanging with the u-integral by Tonelli’s theorem, valid since the integrand is non-negative, gives R ∞ −p−1 E|gt,i |p = c−1 u (1 − E[cos(ugt,i )]) du, and E[cos(ugt,i )] = Re ϕt (u) gives the stated p 0 form. Remark D.6. The restriction p ∈ (1, 2) excludes the finite-variance case p = 2, where the identity of 2 Lemma D.5 degenerates as sin(πp/2) → 0. There, E[gt,i ] = −ϕ′′t (0) directly. Convergence of the p integral requires 1 − Re ϕt (u) = O(u ) as u → 0, the standard local regularity condition replacing an ordinary second derivative when p < 2. D.2

Truncation, Transfer Functions, and the Correlation Stopping Time

This heavy-tailed model needs to be tamed before either moment of gt can be manipulated algebraically. Under Assumption D.4 with p < 2, gt may have infinite variance, so neither the momentum nor second-moment recursion admits the usual moment estimates directly. We truncate gt at a schedule tied to its p-th moment bound, restoring finite moments of every order at the cost of a controlled bias. Definition D.7 (Truncated Gradient). For a schedule τt > 0, define ĝt := gt · ⊮{|gt | ≤ τt }, with  1/p σp t τt := log(1/δ) for a target confidence level 1 − δ. Lemma D.8 (Truncation Bias). Under Assumption D.4, |E[gt ] − E[ĝt ]| = O(τt1−p ). 29

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Proof. Write E[gt ] − E[ĝt ] = E[gt 1{|gt | > τt }]. By Hölder’s inequality with exponents p, p/(p − 1) and Markov’s inequality applied to |gt |p ,  p (p−1)/p σ p 1/p (p−1)/p = σ p τt1−p . E[gt ⊮{|gt | > τt }] ≤ (E|gt | ) P(|gt | > τt ) ≤σ τtp We replace gt by ĝt throughout the mt , vt recursions, writing m̂t , v̂t for the resulting sequences and bt := O(τt1−p ) for the bias above. Since ĝt ∈ [−τt , τt ] almost surely, m̂t , v̂t have finite moments of every order, and |m̂t,i | ≤ τt , v̂t,i ∈ [0, τt2 ] almost surely, since both are convex combinations, with weights summing to at most 1, of terms bounded by τt , τt2 respectively. With ĝt now bounded, the z-transforms of its associated filters are well-defined on the entire region |z| > 1, which we use next to characterize the momentum and second-moment recursions as linear filters. Definition D.9 (Shift Operator and z-Transform). For a sequence {xt }t≥0 , define the shift operator (Sx)t := xt−1 , with x−1 := 0, and the one-sided z-transform [Oppenheim and Schafer, 2010] X(z) := Z{xt } :=

∞ X

xt z −t ,

t=0

defined on the region |z| > ρ for the smallest ρ ≥ 0 for which the series converges absolutely. Lemma D.10 (Shift Property). Z{Sx}(z) = z −1 X(z). P P −t Proof. Z{Sx}(z) = xt−1 z −t = t≥0 t≥1 xt−1 z , since the t = 0 term vanishes because P x−1 = 0, giving z −1 s≥0 xs z −s = z −1 X(z) after reindexing s = t − 1. Let Ĝ(z) := Z{ĝt }. Since |ĝt | ≤ τt with τt = O(t1/p ), the series converges for all |z| > 1. The momentum recursion is linear in ĝt , so its transfer function follows immediately from the shift property. 1 − β1 Lemma D.11 (Momentum Filter). M̂ (z) = Ĝ(z), with a pole at z = β1 and effective 1 − β1 z −1 memory τ1 := 1/(1 − β1 ). Proof. Applying Z{·} to m̂t = β1 m̂t−1 +(1−β1 )ĝt and using Lemma D.10, M̂ (z) = β1 z −1 M̂ (z)+ (1 − β1 )Ĝ(z). Solving for M̂ (z) gives the stated form. The pole at z = β1 corresponds to impulse response (1 − β1 )β1k , decay time 1/(1 − β1 ). The vt recursion is nonlinear in ĝt . Squaring in time corresponds to convolution in z rather than multiplication, so Z{ĝt◦2 } ̸= Ĝ(z)2 and Lemma D.11’s argument does not transfer directly. We extract Z{E[ĝt◦2 ]} instead via the characteristic function. Lemma D.12 (Second-Moment Filter via the Characteristic Kernel). Let ϕ̂t (u) := E[eiuĝt,i ] and Φ(z, u) := Z{ϕ̂t (u)}. Then   ∂ 2 Φ(z, u) 1 − β2 V̂ (z) = − , τ2 := 1/(1 − β2 ). 1 − β2 z −1 ∂u2 u=0 Proof. Since ĝt is bounded, ϕ̂t is twice continuously differentiable with −∂u2 ϕ̂t (u)|u=0 = 2 E[ĝt,i ]. Differentiation in u and summation over t act on independent indices and commute, so 2 −∂u Φ(z, u)|u=0 = Z{E[ĝt◦2 ]}. Substituting into v̂t = β2 v̂t−1 + (1 − β2 )ĝt◦2 and applying Lemma D.10 as above gives the stated form. The filter of Lemma D.12 tracks its quasi-stationary target well only if the underlying noise decorrelates faster than the filter’s own memory τ2 . We specify the sense in which ĝt is treated as locally stationary, which licenses both the autocorrelation analysis here and the drift estimate below. Assumption D.13 (Slow Truncation Growth). τt varies slowly enough that ĝt is stationary over any window of length τ2 . 30

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

2 2 Under Assumption D.13, the truncated autocorrelation Rĝ◦2 (k) := Cov(ĝt,i , ĝt+k,i ) is well-defined and independent of t within a τ2 -window. Definition D.14 (Autocorrelation and Power Spectrum). Define the power spectrum Sĝ◦2 (eiω ) := P −iωk , related to Rĝ◦2 by the Wiener-Khinchin theorem [Papoulis and Pillai, 2002]. Let k Rĝ ◦2 (k)e τcorr denote the lag beyond which Rĝ◦2 is negligible. Definition D.15 (Low-Correlation Event and Stopping Time). Et := {τcorr (ĝs ) < τ2 ∀s ≤ t}, τE := inf{t ≥ t0 : Et fails}. Lemma D.16 (τE is a Stopping Time). τE is a stopping time with respect to {Ft }t≥0 .

Proof. τcorr (ĝs ) for s ≤ t is Ft -measurable, so Et ∈ Ft and {τE ≤ t} = D.3

S

c s≤t Es ∈ Ft .

Trapping and the DSI Cascade

Equipped with this stopping time, we bound how well the realized v̂t,i tracks its quasi-stationary target m∗i (θ) := (∂i L(θ))2 + Σii (θ). The remaining ingredient concerns how far Adam’s own per-step update can move a coordinate, which requires the second moment to stay bounded away from zero so the update’s denominator does not vanish. Assumption D.17 (Non-Degeneracy). v̂i (θ t ) ≥ vmin > 0 for every coordinate i under analysis, for all t ≤ τϵblk ∧ τE . We split the discrepancy into a component driven by θ t ’s own drift and a component driven by filtered noise,
P
P 2 v̂t,i −m∗i (θ t ) = s wt−s m∗i (θ s ) − m∗i (θ t ) + s wt−s ĝs,i − m∗i (θ s ) , wk := (1−β2 )β2k . | {z } | {z } drift of target

filtered fluctuation

The first term is bounded deterministically, using only Assumption D.17 and the Lipschitz structure already established in Proposition A.2. The second requires Assumption D.13 and the stopping time of Definition D.15. Lemma D.18 (Window Drift Bound). Let L denote the Lipschitz-and-boundedness constant of Proposition A.2, so ∥∇L∥ ≤ L and ∇L is L-Lipschitz. Then m∗i is Lm -Lipschitz with Lm = 6L2 , and, using Assumption D.17, X

wt−s m∗i (θ s ) − m∗i (θ t )

≤ Lm τ2 ∆max ,

s

∆max := √

ητt vmin + ϵ

.

Proof. For the Lipschitz constant, |(∂i L(θ 1 ))2 − (∂i L(θ 2 ))2 | = |∂i L(θ 1 ) − ∂i L(θ 2 )| · |∂i L(θ 1 ) + ∂i L(θ 2 )| ≤ L∥θ 1 − θ 2 ∥ · 2L = 2L2 ∥θ 1 − θ 2 ∥, using |∂i L| ≤ ∥∇L∥ ≤ L and Lipschitz continuity of ∂i L with the same constant L. By the argument of Proposition A.2, |Σii (θ 1 ) − Σii (θ 2 )| ≤ ∥D(θ 1 ) − D(θ 2 )∥∞ ≤ 4L2 ∥θ 1 − θ 2 ∥. Summing gives Lm = 2L2 + 4L2 = 6L2 . p √ Each per-step update satisfies |∆θs,i | = η|m̂s,i |/( v̂s,i + ϵ) ≤ ητt /( vmin + ϵ) = ∆max , using |m̂s,i | ≤ τs ≤ τt (Section D, truncation bound, τt increasing) and Assumption D.17. Hence ∥θ s − θ t ∥ ≤ |t − s|∆max by the triangle inequality over the telescoping sum of steps. Since wk is supported with effective width τ2P , |m∗i (θ s ) − m∗i (θ t )| ≤ Lm |t − s|∆max ≤ Lm τ2 ∆max for s within the effective filter window, and s wt−s = 1 gives the stated bound. Having bounded the drift-of-target term deterministically, we turn to the filtered-fluctuation term. This requires ruling out the possibility that the filter simply passes correlated noise through unattenuated. Lemma D.19 (Filtered Noise Variance). On the event Et (Definition D.15), " # X

2 Var wt−s ĝs,i − m∗i (θ s ) = O (1 − β2 )τt4 . s

Proof. ByRthe Wiener-Khinchin relation of Definition D.14, the variance of the filtered process π 1 equals 2π |H2 (eiω )|2 Sĝ◦2 (eiω ) dω with H2 (z) = (1 − β2 )/(1 − β2 z −1 ). On Et , τcorr < τ2 , −π 31

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

2 so ĝs,i − m∗i (θ s ) is well-approximated as uncorrelated across the effective filter width, and the P 2
1−β2 2 2 variance reduces to the independent-input case, Var = k wk Var(ĝt,i ) = 1+β2 Var(ĝt,i ) = P 4 2 O((1 − β2 )τt ), using k wk = (1 − β2 )/(1 + β2 ), a direct geometric-series computation, and 2 4 Var(ĝt,i ) ≤ E[ĝt,i ] ≤ τt4 from truncation.

Combining the deterministic and stochastic components gives a single bound on the total tracking error. Corollary D.20
(Temporal Tracking Bound). On Et , for any δ ′ > 0, P |v̂t,i − m∗i (θ t )| ≥
′ 4 ′2 Lm τ2 ∆max + δ ≤ O (1 − β2 )τt /δ . Proof. Combine Lemma D.18, which is deterministic, with Chebyshev’s inequality applied to Lemma D.19 via the triangle inequality on the decomposition above. Corollary D.20 bounds the error accrued from a single coordinate’s own second-moment estimate drifting away from its target over time. A second, independent source of error arises across coordinates within a symmetric block. Even if every v̂t,i tracked its own target perfectly, differently positioned coordinates in the same block would have different targets unless their trajectories have already converged. We bound this cross-sectional discrepancy next. It requires no additional stochastic assumption. Let A be generated by the full permutation group on an Sn -symmetric block, as in Theorem C.2. The key geometric fact enabling a deterministic bound is that every coordinate in such a block projects onto the same point of A. Lemma D.21 (Common Projection Point). For every coordinate i in the block, πA (θ)i = θ̄blk := P 1 k∈block θk . n Proof. A restricted to the block is {θ ∈ Rn : θi = θj ∀i, j}, since A is the fixed-point set of the full permutationPgroup on the block (Theorem C.2). The orthogonal projection of θ onto this subspace minimizes k (θk − c)2 over the diagonal point c1. Differentiating in c gives c = θ̄blk . Because every coordinate in the block shares this projection point, two coordinates trapped near A cannot be far from each other. The Lipschitz gradient bound already in force elsewhere in this paper then keeps their second moments from being far apart either. (k)

Lemma D.22 (Pathwise Block Homogeneity). Let τϵblk := mink∈block τϵ . For t ≤ τϵblk and any i, j in the block, almost surely, √ |v̂t,i − v̂t,j | ≤ 4Lτt ϵ,

√ ∥Et ∥∞ ≤ κ · 4Lτt ϵ,

1 , κ := √ √ 2 vmin ( vmin + ϵ)2

p where Et denotes the deviation of P (θ t )|block = diag(1/( v̂(θ t ) + ϵ)) from c(θ t )I, c(θ t ) the block average, and κ is evaluated using Assumption D.17. Proof. By Lemma D.21, πA (θ(i) ) = πA (θ(j) ) = θ̄tblk for i, j in the block. By Lemma B.2, (i) (j) Yt , Yt ∈ [0, ϵ] almost surely for t ≤ τϵblk . Hence q q √ (i) (j) |θt,i − θt,j | ≤ |θt,i − θ̄tblk | + |θ̄tblk − θt,j | = Yt + Yt ≤ 2 ϵ. By Proposition A.2, gt is L-Lipschitz, being an average of L-Lipschitz sample gradients, so |ĝt,i − √ ĝt,j | ≤ L|θt,i − θt,j | ≤ 2L ϵ whenever both are untruncated. The truncation indicator can only reduce this gap since |ĝ| ≤ |g| pointwise on the same realization, with the boundary case bounded separately by τt and absorbed into the√stated constant. Using |a2 − b2 | =P|a − b||a + b| ≤ 2τt |a − b| 2 2 2 for |a|, |b| ≤ τt , |ĝs,i − ĝs,j | ≤ 4Lτt ϵ for every s ≤ t. Since v̂t,i = s wt−s ĝs,i with wt−s ≥ 0 √ 2 2 summing to at most 1, |v̂t,i − v̂t,j | ≤ sups≤t |ĝs,i − ĝs,j | ≤ 4Lτt ϵ. Since c(θ t ) is a convex √ combination of the v̂t,k , |v̂t,i − c(θ t )| obeys the same bound. The map v 7→ 1/( v + ϵ) has derivative bounded by κ on [vmin , ∞), and the mean value theorem transfers the bound to ∥Et ∥∞ . 32

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Together, Corollary D.20 and Lemma D.22 bound every source of discrepancy between the exact Adam preconditioner and its scalar mean-field approximation. We now assemble them into a single reduced diffusion. Theorem D.23 (Preconditioned SGF Reduction). Under Assumptions D.4, D.13, and D.17, on t ≤ τϵblk ∧ τE , p dθ t = −η c(θ t )∇L(θ t ) dt + η c(θ t ) Σ(θ t ) dWt + Rt , √
where, with probability at least 1 − O((1 − β2 )τt4 /δ ′2 ), ∥Rt ∥ = O bt + Lm τ2 ∆max + δ ′ + Lτt ϵ . Proof. By Corollary D.20, v̂t,i tracks m∗i (θ t ) up to the stated drift-plus-fluctuation bound, giving the mean-field preconditioner P (θ t ) up to this error. Substituting P (θ t ) = c(θ t )I + Et , with ∥Et ∥ bounded by Lemma D.22, into the exact preconditioned update decomposes drift and diffusion into a term scaled by the scalar c(θ t ), which commutes with every Q and preserves the affine geometry of à by the argument of Theorem C.3, plus a residual Rt collecting the truncation bias bt (Lemma D.8), the temporal tracking error (Corollary D.20), and the cross-sectional deviation ∥Et ∥ (Lemma D.22). This reduced SGF has the same structural form as the vanilla-SGD flow of Section 2, up to the residual Rt and the positive scalar rescaling c(θ t ). We now check that the trapping argument of Theorem B.3 survives this perturbation. Theorem D.24 (Conditional Local Supermartingale Trapping). Suppose A is stochastically attractive for the reduced drift c(θ)∇L(θ) (Definition 2.4), with drift margin dominating ∥Rt ∥ from Theorem (i) D.23. Then Yt∧τ blk ∧τE is a non-negative local supermartingale. ϵ

(i)

Proof. Applying Itô’s lemma to Yt

under the reduced SDE gives (i) (i) (i)
⊤ (i) A Yt = −2c(θ t ) θt − πA (θt ) ∇θ(i) L(θ t ) + ηc(θ t )2 Tr(Σii (θ t )) + ⟨∇Yt , Rt ⟩.

The first two terms retain the sign established by stochastic attractivity, scaled by c(θ t ) > 0. By (i) hypothesis the drift margin dominates ∥Rt ∥, so A Yt ≤ 0 prior to τϵblk ∧ τE . Boundedness of the stopped process follows as in Lemma B.2, and the remainder proceeds as in Theorem B.3, with the stochastic integral a genuine martingale by [Karatzas and Shreve, 1991]. (i)

Corollary
D.25 (Conditional Non-Escape). P(τϵblk < τE | Ft0 ) ≤ ϵ−1 E[Yt0 | Ft0 ] + O (1 − β2 )τt4 /δ ′2 . Proof. Doob’s maximal inequality [Revuz and Yor, 1999] applied to Theorem D.24 on the event that τE has not occurred, combined via a union bound with the failure probability of Corollary D.20. This conditional trapping result is the direct analogue, under Adam and AdamW, of Corollary B.4. The final ingredient before carrying it through to the percolation graph and DSI cascade is combinatorial. It keeps the merging block large enough, relative to the network, that its discontinuity survives the thermodynamic limit of Corollary C.18. Assumption D.26 (Macroscopic Block Scaling). For the Sn -symmetric block, n(N )/N → c ∈ (0, 1] as N → ∞. Theorem D.27 (Conditional DSI Cascade under Adam and AdamW). Under Assumptions D.4, D.13, D.17, and D.26, on t ≤ τϵblk ∧ τE the Adam- or AdamW-trained trajectory admits a percolation graph Gτ satisfying Generalized Discrete Scale Invariance with the same magnification factor λ(n) = nσ as in Theorem C.25, with escape probability bounded as in Corollary D.25. (i)

Proof. By Theorem D.24, Yt∧τ blk ∧τE is a non-negative local supermartingale, and the reduced ϵ dynamics of Theorem D.23 rescale the drift and diffusion of the original transverse process by the identical positive scalar c(θ t ), leaving the affine geometry of à unchanged. The arguments of Theorem B.13, Theorem C.17, and Theorem C.25 therefore apply verbatim to the stopped process with τϵ replaced by τϵblk ∧ τE . In the ratio (pc − pni )/(pc − pi ) of Theorem C.25, the scalar c(θ t ) cancels, and under the Θ(N ) block scaling of Assumption D.26, Corollary C.18 places the merge in the regime where the exponent λ(n) = nσ survives the thermodynamic limit. 33

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Remark D.28. On the AdamW-trained grokking run of Section 6, over 60000 recorded steps (≈ 60 multiples of τ2 ), τcorr = 1 step against τ2 ≈ 1000, so Et held throughout. For the tracked coordinate local pair, vmin ≈ 4.7 × 10−13 , below ϵ; the network-wide minimum was 3.2 × 10−23 , confirming Assumption D.17 is scoped to the block under analysis rather than uniform across parameters, since some coordinate will read near-zero regardless. The pair had not merged by the end of the run, so corr(|θi − θj |, |vi − vj |) = −0.11 reflects the bound of Lemma D.22 holding as an inequality rather than a monotone relationship at this stage, not a violation. The raw ratio max |g|/std(g) ≈ 77 is consistent with a low tail index p under Lemma D.5, though a direct estimate of p from the characteristic function was not computed here.

E

Explicit Theoretical Verification and Toy Models

We construct two distinct experimental environments to validate the underlying framework of symmetry-induced topological collapse. First, we isolate the topological mechanics in a controlled kinematic setting to verify the algebraic predictions of the Generalized Discrete Scale Invariance (DSI) cascade. Second, we deploy an empirical neural network optimized via Stochastic Gradient Descent (SGD) to verify that these topological phase transitions—both condensation and reactive fragmentation—govern non-convex empirical loss landscapes. E.1

Generalization of the Stochastic Collapse Threshold

To derive the exact mathematical condition under which a subnetwork irreversibly collapses into an invariant set, we analyze the local dynamics of continuous stochastic gradient flow (SGF) Li et al. [2017]. Consider a network with multiplicative parameter interactions, such as adjacent layers connected by weights win and wout . Assuming the subnetwork learns a target signal µ under a Mean Squared Error (MSE) objective, the local loss is L(win , wout ) = 14 (win wout − µ)2 . Under continuous d 2 2 − win ) = 0. gradient descent, the difference of squared weights is a conserved quantity: dt (wout Assuming a balanced initialization, the parameters satisfy win = wout = w. The transverse dynamics consequently collapse onto a one-dimensional effective loss landscape: 1 L(w) = (w2 − µ)2 . (42) 4 Near the structurally collapsed state at the origin (w = 0), the gradient is L′ (w) = w3 − µw, establishing w = 0 as a stationary invariant manifold where the first derivative vanishes. The local curvature is given by the second derivative L′′ (0) = −µ, defining a deterministic repulsive drift away from the manifold. Simultaneously, the gradient noise covariance matrix D(w) inherently scales with the magnitude of the active weights Blanc et al. [2020]. Near the manifold, the transverse diffusion term scales quadratically, D(w) ≈ 12 ζ 2 w2 , yielding purely multiplicative noise. Substituting the exact gradient p and the local noise approximation into the continuous SDE dwt = −∇w L(wt )dt + 2D(wt )dBt yields: dwt = −(wt3 − µwt )dt + ζwt dBt . (43) To determine whether the network undergoes collapse, we evaluate the Fokker-Planck equation for localized stationarity (∂t p = 0). Forcing the probability current to vanish yields the steady-state Gibbs distribution pss (w) ∝ e−κΨ(w) governed by the modified effective potential Chen et al. [2023]:   w2 2µ Ψ(w) = − − 2 ln(w). (44) 2 ζ2 The global normalization partition function diverges when the logarithmic exponent forces a nonintegrable singularity at the origin, requiring 2µ ζ 2 − 2 ≤ −1. Theoretical Threshold: The invariant manifold becomes strictly stochastically attractive—collapsing the transverse probability mass into a Dirac delta distribution δ(w)—if and only if the multiplicative noise variance overcomes the deterministic drift: ζ2 (45) µ≤ . 2 34

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Figure 3: Experimental validation of symmetry-induced topological collapse and DSI scaling laws. Top Row (Experiment 1): Kinematic simulation of hierarchical coalescence (K = 3), displaying discrete jumps in the order parameter (a), divergence peaks in relative variance exactly at topological merge points (b), and forced continuous spline trajectories (c). Middle Row (Experiment 2): Empirical SGD dynamics (K = 6), showing macroscopic dimensionality collapse (a), distinct precursory variance peaks p1 , p2 during the condensation phase, and a reactive fragmentation peak p3 following a task shift at t = 3000 (b), alongside the raw stochastic weight trajectories (c). Bottom Row: Log-linear scaling progressions extracting the DSI magnification factor for Experiment 1 (Left) and Experiment 2 (Right), both yielding the theoretically predicted λ ≈ 2.00.

35

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Because the empirical SGD diffusion amplitude scales proportionally with the learning rate η, maintaining a sufficiently high learning rate explicitly satisfies this threshold, permanently trapping the parameters within the lower-dimensional invariant subspace Chen et al. [2023]. E.2

Derivation of the Exact DSI Scaling Factor

Once parameters are trapped in the invariant manifold, the network topology is governed by the symmetric permutation group Sn , forcing identical subnetworks to merge in discrete blocks of size n. This algebraic constraint replaces continuous scale invariance with Discrete Scale Invariance (DSI) Sornette [1998]. To derive the scaling factor, we assume the continuous mean-field percolation baseline for the size of the largest macroscopic component, C1 = Aamp (pc − p)−1/σ , where pc is the global collapse threshold and σ is the universal critical exponent Stauffer and Aharony [1994]. Inverting this relation (ω) isolates the ensemble-averaged critical density pi := Eω [pi ] (Definition C.24) corresponding to a subnetwork component of expected size i, pi = pc − Aσamp i−σ .

(46)

Applying the Sn block-merge constraint forces the subsequent microscopic transition to occur precisely at the discrete multiple ni. This yields the subsequent critical density pni = pc − Aσamp (ni)−σ . We compute the ratio of their relative distances to the critical threshold: Aσamp n−σ i−σ pc − pni = = n−σ . pc − p i Aσamp i−σ

(47)

Theoretical Prediction: This derivation extracts the geometric magnification factor λ(n) = nσ that governs the precursory cascade. For pairwise identical merges (n = 2) operating under standard mean-field percolation (σ = 1), the mathematical framework dictates a rigid scaling factor of λ = 2. Consequently, the sequential divergence times tk detected by the macroscopic relative variance Rv (t) must strictly satisfy the log-linear progression tk+1 = 2tk . E.3

Idealized Hierarchical Coalescence

Theoretical Justification: In non-convex empirical loss landscapes, topological phase transitions are tightly coupled with geometric distortions, complicating the isolation of pure structural scaling. To verify the mathematical validity of the derived DSI scaling factor, we isolate the topological variable in a controlled environment. Empirical Verification: We construct a synthetic kinematic system of K = 3 decoupled parameters. We enforce the algebraic constraint of sequential, binary hierarchical coalescence (n = 2) by routing continuous spline trajectories injected with decaying Gaussian noise. This controlled environment tests whether a system adhering purely to the Sn block-merge topology intrinsically generates the predicted geometric scaling law in its macroscopic variance. Results: The binding of parameters 1 and 2 at t1 = 700, followed by parameter 3 at tc = 1400 (mapped in Figure 3, Top Row, Panel c), causes the macroscopic order parameter ⟨O(t)⟩ (Figure 3, Top Row, Panel a) to collapse via discrete jump discontinuities. Driven by the bimodal ensemble states at these critical thresholds, the relative variance Rv (t) (Figure 3, Top Row, Panel b) registers localized divergences exactly at the topological merge points. Plotting these critical transition times on a logarithmic scale against their sequential index k (Figure 3, Bottom Left) yields an empirical scaling ratio of λ = 1400/700 = 2.00. This two-point log-linear fit yields a slope of λ = 2.00, matching the theoretical derivation λ(2) = 2 exactly. E.4

Empirical Simulation (K=6) with Task Shift and Reverse Transition

Theoretical Justification: Having verified the isolated topological scaling, we show that the noise characteristics generated by standard neural network optimization can satisfy these rigid topological constraints without external kinematic forcing. Furthermore, the theoretical threshold (µ ≤ ζ 2 /2) dictates that the collapse is dynamically reversible: dropping the noise variance while increasing the signal curvature should render the invariant manifold repulsive, forcing reactive fragmentation. 36

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Empirical Verification: We deploy a PyTorch environment, initializing a K = 6 shallow neural network with a GELU activation function, optimized via SGD on an MSE objective. We manipulate the theoretical stochastic collapse condition by separating the training into two distinct phases. In the condensation phase (t < 3000), a high learning rate injects multiplicative noise to satisfy µ ≤ ηζ 2 /2. At t = 3000, we execute a task shift: we inject a high-frequency target signal (increasing the deterministic curvature µ) while simultaneously dropping the learning rate (decreasing the noise amplitude ζ 2 ). This tests both forward DSI condensation and reverse reactive fragmentation. Results: During the initial high learning rate phase, the aggressive stochastic diffusion neutralizes the deterministic gradient drift (Figure 3, Middle Row, Panel c). This natively forces the independent parameters to collapse into shared, lower-dimensional invariant subspaces. The SGD noise dynamically shifts the merge thresholds across ensemble realizations, creating a distinct multi-phase precursory cascade. We detect sequential divergence peaks in the smoothed relative variance Rv (t) (Figure 3, Middle Row, Panel b) strictly within the high learning rate regime. Extracting these transition indices and computing their scaling progression (Figure 3, Bottom Right) yields an extracted magnification factor of λ ≈ 2.00, matching the theoretical derivation. Following the task shift at t = 3000, the drastic reduction in learning rate and the addition of the new target signal immediately violate the trapping threshold. The invariant manifold becomes stochastically repulsive. The macroscopic topology registers a reverse transition: the relative variance Rv (t) experiences an immediate divergence peak as the previously bound parameters discontinuously fragment to map the new target dimensionality.

F

Extended Empirical Validation Across Diverse Datasets and Architectures

Having verified the rigid algebraic predictions of Discrete Scale Invariance (DSI) in controlled kinematic and low-dimensional SGD environments, we now generalize the framework to standard deep learning paradigms. We deploy unconstrained neural networks across a diverse suite of datasets, spanning tabular data, geometric manifolds, vision tasks, and algorithmic grokking. F.1

Methodological Extensions for Empirical Regimes

Transitioning from isolated theoretical toy models to highly non-convex, unconstrained deep learning environments requires adapting our measurement observables. All empirical experiments were run on a Tesla T4 GPU and require ∼ 1 day worth of compute. Soft Dimensionality Collapse (Spectral Effective Rank): In high-dimensional empirical networks, parameter constraints rarely force weights to absolute zero (hard topological collapse). Instead, structural merges manifest as spectral concentration, where the energy of the weight matrix collapses into a smaller subset of principal components. We map the macroscopic order parameter ⟨O(t)⟩ to the Effective Rank of the target layer’s weight matrix. Computing the Singular Value Decomposition (SVD), P we extract the singular values σk , normalize them into a probability distribution pk = σk / σi , and compute the exponential of the Shannon entropy:  X  O(t) = exp − pk log pk . (48) This continuous metric reliably tracks the macroscopic dimensionality collapse as the top singular values diverge and absorb the network’s capacity. Signal Processing, Detrended Log-Variance, and Ablation Controls: Real-world SGD exhibits massive baseline heteroskedasticity; the raw variance is known to decay as the network settles into a basin [Mandt et al., 2017, Mori et al., 2022]. To isolate the delicate microscopic variance divergences (Rv (t)) from this global decay, we operate strictly in log-variance space, log(Var[O(t)]). We subsequently subtract a global linear secant to perfectly detrend the signal, preventing artificial boundary artifacts during macroscopic smoothing. This isolates the scale-invariant structural fluctuations from the standard optimization noise. To establish a rigorous baseline, we construct a spectral null model testing the null hypothesis that observed cascades are mere artifacts of this optimization noise. We compute 1000 phase-randomized spectral surrogates which preserve the empirical power spectrum of the variance trajectory while destroying localized temporal structures to calculate empirical False Positive Rates (FPR). Additionally, we compute 500 bootstrap resamples per configuration to ensure 37

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

statistical stability, aggregating variance across 20 independent initializations for the UCI tabular suite and 15 independent initializations for high-dimensional vision and algorithmic grokking datasets. Spectral Relaxation of the Percolation Exponent: In the idealized K = 3 kinematic model (Appendix E), exact topological counting dictates a rigid integer magnification factor (λ = 2) derived strictly from pairwise binary block-merges (n = 2) under standard mean-field percolation (σ = 1). However, mapping unconstrained empirical networks necessitates the continuous Effective Rank order parameter. We prove that integrating the collapse through this continuous spectral entropy mathematically relaxes the rigid integer constraint, compressing the structural jump precisely by the fractional variance mass of the merging components. Theorem F.1 (Spectral Relaxation of DSI Scaling). Let a neural network undergo an n-body symmetry-induced merge where n equivalent sub-components, carrying a total fractional spectral variance mass m ∈ (0, 1], collapse into a single component conserving the mass m. Under the Spectral Effective Rank order parameter O(t), the integer DSI magnification factor relaxes to a continuous fractional scaling law strictly governed by λef f = nm·σ . P order Proof. Let pk = σk / σi denote the normalized singular value spectrum. The macroscopic P parameter evaluates the exponentiated Shannon entropy: O = exp(H) = exp (− pk ln pk ). Prior to the structural transition, the n-body symmetry distributes the variance mass equally, assigning m n to each of the n pre-merge components. Let Hrest denote the constant spectral entropy of the non-participating components. The pre-merge entropy evaluates to: n m m X Hpre = − ln + Hrest = −m ln m + m ln n + Hrest . n n j=1 During the topological collapse, the n components structurally fuse into a single geometric component containing the integrated mass m. The post-merge entropy evaluates to: Hpost = −m ln m + Hrest . To extract the macroscopic scaling of the component sizes across the discrete transition, we evaluate the ratio of the Effective Rank: Opre exp(−m ln m + m ln n + Hrest ) = exp(m ln n) = nm . = Opost exp(−m ln m + Hrest ) Whereas discrete topological counting measures an absolute component reduction factor of n, the continuous spectral mapping registers an effective reduction of strictly nm . Following the inversion of the mean-field baseline (Theorem C.25), this geometric scaling directly modulates the critical percolation exponent, extracting σef f = mσ. Consequently, the generalized discrete scaling ratio evaluates precisely to λef f = nmσ . Remark F.2 (Empirical Extraction of Variance Mass and Topological Order). Theorem F.1 transforms the fractional magnification factor from an empirical artifact into a mathematical diagnostic, enabling the extraction of the active variance mass (m) under a hypothesized topological order (n). Assuming standard mean-field dynamics (σ = 1): • Modular Arithmetic (Transformer Grokking): The cascade yields λ = 2.11. Because this strictly exceeds the absolute upper bound for any pairwise spectral relaxation (maxm∈(0,1] 2m = 2), Theorem F.1 mathematically guarantees the transition is driven by higher-order permutation symmetries. Assuming a 3-body simultaneous topological merge (n = 3), inversion (3m = 2.11) isolates m ≈ 0.68. We hypothesize that the dismantling algorithmic circuits localized precisely ∼ 68% of the structural variance. • UCI Abalone: The tabular dataset yields λ = 1.28. Assuming a pairwise collapse (n = 2), inverting the spectral relaxation relation (2m = 1.28) yields m ≈ 0.36. Under this hypothesis, the transition geometrically monopolized ∼ 36% of the active spectral variance. • UCI Heart Disease: The tabular dataset yields an empirical λ = 1.71. If driven by a pairwise merge (n = 2), inverting the relation (2m = 1.71) isolates m ≈ 0.77. This suggests the topological merge absorbed ∼ 77% of the active spectral variance. 38

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

• FashionMNIST (Vision Benchmark): The dataset yields λ = 1.57. Assuming a pairwise collapse (n = 2), inverting the relation (2m = 1.57) isolates m ≈ 0.65. We hypothesize the transition absorbed ∼ 65% of the active spectral variance. • Conjecture on Network and Task Complexity: We cautiously hypothesize that highly deep neural models trained on complex representations form distinct macroscopic subnetwork symmetries, yielding clean log-linear cascades. Conversely, shallower networks on low-dimensional tabular data generate variance fluctuations frequently indistinguishable from baseline stochastic gradient noise. F.2

UCI Tabular Datasets

We evaluate shallow Multi-Layer Perceptrons (MLPs) optimized via SGD on a suite of standard tabular datasets from the UCI Machine Learning Repository. Across all datasets, the macroscopic order parameter (Effective Rank) and the training loss exhibit smooth, continuous decay. However, the detrended variance successfully shatters this continuity, revealing discrete DSI microtransitions. Table 1 aggregates the empirical DSI fits and spectral null-model evaluations across the suite. Heart Disease provides a statistically significant mapping of internal topological constraints (FPR = 4.9%). Conversely, German Credit serves as a critical negative control. Although the network exhibits a 4-peak variance sequence (λ = 1.99, R2 = 0.86), spectral null testing yields a False Positive Rate of 80.2%. This negative result establishes the necessity of the signal processing pipeline; without it, standard optimization noise mimics geometric scaling artifacts. Table 1: Empirical DSI cascades and statistical null-model evaluations for UCI tabular datasets (aggregated over 20 independent initializations). Dataset Heart Disease Abalone German Credit Digits

Peaks

λ

R2

FPR (%)

4 6 4 1

1.71 1.28 1.99 N/A

0.98 0.97 0.86 N/A

4.9% 15.6% 80.2% N/A

Figure 4: Abalone: The macroscopic continuous decay of effective rank (a) is underscored by a highly structured 6-peak DSI cascade yielding a strong log-linear fit of R2 = 0.97 with a fractional scaling factor λ = 1.28 (b). The collapse is driven by a dominant top singular value (c). F.3

Geometric Manifolds, Vision, and Algorithmic Grokking

To verify that these scaling laws are not strictly an artifact of tabular data or shallow MLPs, we extend the empirical validation to geometric manifolds (Moons, Swiss Roll), vision benchmarks (MNIST, FashionMNIST), and algorithmic tasks (Modular Arithmetic). Critically, the Modular Arithmetic task utilizes a Transformer architecture optimized via AdamW with explicit weight decay. This algorithmic setting inherently induces grokking, a dynamic phenomenon wherein deep neural networks initially overfit the training distribution before abruptly 39

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Figure 5: Digits: Variance fluctuations dynamically map the topological condensation of the network during early-stage SGD optimization.

Figure 6: German Credit: The network exhibits a 4-peak cascade (λ = 1.99, R2 = 0.86), but yields a spectral null False Positive Rate of 80.2%, acting as a negative control.

Figure 7: Heart Disease: Dimensionality collapse (c) is mirrored by a sequential 4-peak variance divergence (λ = 1.71, R2 = 0.98, FPR = 4.9%) mapping the internal topological constraints.

40

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

acquiring perfect generalization after extensive continued optimization Power et al. [2022]. Mechanistic analyses reveal that this delayed performance spike fundamentally masks a representational phase transition; the network progressively dismantles high-rank, dense memorization circuits to construct low-dimensional, structured algorithmic manifolds Nanda et al. [2023], Liu et al. [2022]. Driven by explicit regularization specifically the weight decay utilized in our AdamW optimizer and inherent stochastic gradient noise, the optimization trajectory continuously penalizes complex parameterizations, forcing the weights to systematically escape sharp memorizing minima Thilak et al. [2022]. Consequently, we expect this prolonged macroscopic dimensionality reduction to happen through a sequence of discrete topological collapses. The Transformer’s subsequent delayed generalization is consistent with this hypothesis. Across 15 independent initializations, the final train loss converges to 0.005 ± 0.004, validation loss drops to 0.030 ± 0.033, and effective rank collapses to 10.93 ± 0.99. The detrended variance isolates a perfectly log-linear 3-peak DSI cascade yielding λ = 2.11 and R2 = 1.00. Spectral null testing confirms robust statistical significance with a False Positive Rate of 0.1%. We test on modulo 17 for 30000 epochs with a decay parameter of 0.5 and learning rate of 0.003. The high-dimensional vision benchmarks mirror this discretized capacity reduction. FashionMNIST exhibits consecutive variance peaks highlighting structural scaling, yielding a 3-peak cascade with λ = 1.57, R2 = 1.00, and a spectral null FPR of 10.7%. Conversely, MNIST produces a 2-peak sequence (p1 , p2 ) mapping early-stage topological condensation without forming a full extended cascade. Evaluations on geometric manifolds map the collapse phase as the MLP parameterizes spatial embeddings. Moons isolates a single primary microtransition (p1 ) dynamically navigating the nonlinear 2D geometry. Swiss Roll similarly identifies a primary structural microtransition (p1 ) charting the 3D spatial embedding. These initial divergences capture the precise epoch of spatial alignment consistent with discrete topological condensation during early-stage optimization.

Figure 8: Moons (Geometric Manifold): Collapse phase mapped by variance fluctuations as the MLP parameterizes the 2D non-linear manifold.

41

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Figure 9: Swiss Roll (Geometric Manifold): Sequential structural microtransitions dynamically navigate the 3D spatial embedding of the data.

Figure 10: MNIST (Vision): Variance fluctuations produce a 2-peak sequence (p1 , p2 ) dynamically mapping the topological condensation of the network during early-stage SGD optimization.

42

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Figure 11: FashionMNIST (Vision): Detection of consecutive variance peaks highlighting the discretized nature of capacity reduction (λ = 1.57, R2 = 1.00, FPR = 10.7%).

Figure 12: Modular Arithmetic (Grokking): A Shallow Transformer trained with AdamW experiences a delayed topological collapse. The macroscopic decay of the attention embeddings’ effective rank is composed of a perfectly log-linear 3-peak DSI cascade (λ = 2.11, R2 = 1.00, FPR = 0.1%).

43

Related documents

• Information geometry of perturbed gradient flow systems on hypergraphs: A perspective towards nonequilibrium physics#1052040
• Percolation Markov Decision Processes#1014951
• Thresholds and Fluctuations of Submultiplexes in Random Multiplex Networks#612877
• Thresholds and Fluctuations of Submultiplexes in Random Multiplex Networks#611400
• Topological Uncertainty and Higher-Order Interactions in Spatial Networks#1012625