Conceptio › Archive › arXiv CS

arXiv CSopen access

Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

knowledge-representation reasoning

artificial intelligence, reasoning, knowledge representation

arXiv:2605.04971v1 [cs.LG] 6 May 2026

Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking

Kyungwon Jeong Hyntel [email protected]

Won-Gi Paeng Hyntel [email protected]

Honggyo Suh Hyntel [email protected]

Abstract Weight matrices in deep networks exhibit geometric continuity—principal singular vectors of adjacent layers point in similar directions. While this property has been widely observed, its origin remains unexplained. Through experiments on toy MLPs and small transformers, we identify two mechanisms: residual connections create cross-layer gradient coherence that aligns weight updates across layers, and symmetry-breaking nonlinearities constrain all layers to a shared coordinate frame, preventing the rotation drift that would otherwise destabilize weight structure. Crucially, a nonlinear but rotation-preserving activation fails to retain continuity, isolating symmetry breaking—not nonlinearity itself—as the active ingredient. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. In transformers, continuity is projection-specific: Q, K, Gate, and Up (which read from the residual stream) develop input-space (v1 ) continuity; O and Down (which write to it) develop output-space (u1 ) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity.

Introduction

Recent work has revealed that weight matrices of adjacent layers in large language models share similar structure [1, 2]. We observe that this similarity extends to principal singular vectors, which point in similar directions across adjacent layers, forming smooth trajectories across depth—a property we term geometric continuity. This geometric continuity has practical implications—it enables layer pruning [3, 4] and cross-layer parameter sharing [2]. To quantify this, we decompose each layer’s weight matrix via SVD and measure the cosine similarity of the leading right singular vectors (v1 ) between adjacent layers in pretrained Llama-3.1-8B [5] (Appendix A). We find that continuity is space-specific: Q, K, Gate, and Up projections show high v1 (input-space) continuity, O and Down show high u1 (output-space) continuity, while V develops only low continuity. Despite these observations, a fundamental question remains unanswered: why does geometric continuity emerge? Is it an inevitable consequence of training neural networks, or does it require specific architectural or data conditions? In this work, we investigate the origin of geometric continuity through ablation experiments on toy MLPs (MNIST) and small transformers (WikiText-103). Our key findings are: 1. Two necessary conditions in toy MLPs (Figure 1): residual connections provide crosslayer gradient coherence, and symmetry-breaking nonlinearities (activation, normalization) constrain all layers to a shared coordinate frame, preventing rotation drift that would Preprint.

Figure 1: Geometric continuity of weight v1 across layers. 3D PCA of principal right singular vectors from a 16-layer MLP trained on MNIST. (a) Before training: random. (b) Res+ReLU: smooth trajectory. (c) Res+None: activation removed, low continuity. (d) NoRes+ReLU: residual removed, weak continuity. Both residual connections and symmetry-breaking nonlinearity are necessary. otherwise destabilize weight structure. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. Weight v1 emerges as the principal direction of each layer’s cumulative gradient over training. A nonlinear-but-rotation-preserving activation (Section 4.3) fails to produce continuity, confirming that symmetry breaking—not nonlinearity itself—is the active ingredient. 2. Projection-specific continuity in transformers: Q, K, Gate, and Up (which read from the residual stream) develop input-space (v1 ) continuity; O and Down (which write to it) develop output-space (u1 ) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity. Direct ablation confirms the Gate dependence: removing the MLP activation drops Gate v1 from 0.66 to 0.37 with only a modest perplexity change (∼3%).

Related Work

Cross-layer weight and Jacobian structure. Several works have independently observed that deep networks exhibit cross-layer geometric structure. Min and Wang [1] propose DOCS, a metric based on column-wise cosine similarity distributions, revealing that adjacent layers form similarity clusters. Wang et al. [2] exploit shared SVD bases across layers for LLM compression. Most closely related to our work, Li and Papyan [6] discover that Residual Jacobians exhibit top singular vector alignment across depth that vanishes when skip connections are removed—directly corresponding to our Condition 1 (residual connections). Their analysis focuses on Jacobians rather than weight matrices (which can diverge under nonlinearity) and identifies only the residual condition; our Res+None experiment shows that residual alone is insufficient for weight continuity, motivating the additional role of symmetry-breaking nonlinearity (Condition 2). Razzhigaev et al. [7] show that adjacent transformer layer outputs are nearly linearly related (Procrustes similarity ∼0.99), with the linear relationship weakening when residual connections are removed. Our work complements these observations by identifying why this structure emerges from a mechanistic perspective. Implicit structural regularization. Ji and Telgarsky [8] prove that, in deep linear networks trained ⊤ under gradient descent, the top rank-1 components of adjacent layers align: |vl+1 ul | → 1 (signalflow alignment between adjacent layers), providing a theoretical precedent for cross-layer structural regularity. Marion et al. [9] show that deep residual networks are implicitly regularized toward neural ODE solutions, explaining smooth cross-layer transformations. Beaglehole et al. [10] establish that the left singular structure of weight matrices aligns with pre-activation tangent features (a gradientderived quantity), a closely related phenomenon to our finding that weight v1 reflects accumulated gradient directions. Symmetry and symmetry breaking. Saxe et al. [11] derive exact learning dynamics for deep linear networks under orthogonal invariance of adjacent weight products, providing a foundation for analyzing rotational degrees of freedom in such models. Godfrey et al. [12] formalize this by 2

characterizing the intertwiner group for each activation type: linear networks admit the full general linear group as symmetry, while element-wise activations such as ReLU reduce it to a discrete subgroup (permutations and positive rescaling) that preserves coordinate axes. Marcotte et al. [13] extend this analysis to residual architectures, proving that an isolated residual block preserves the same conservation laws as its non-residual counterpart, while consecutive residual blocks admit none of these laws across them—formally identifying the cross-block flat direction along which weight-SVD drift can occur. Our finding that activation is necessary for stable gradient-to-weight transfer can be understood through this lens: the reduction from continuous rotational symmetry to a discrete subgroup eliminates the rotational degrees of freedom that would otherwise allow weight frames to drift, enabling stable imprinting of gradient directions onto weights. Layer similarity and pruning. Gromov et al. [3] show that up to half of LLM layers can be pruned with minimal performance loss, and Men et al. [4] identify redundant layers via Block Influence scores. These results are consistent with geometric continuity—adjacent layers that share similar principal directions perform similar transformations and are thus prunable.

Background

Singular value decomposition. Any weight matrix W ∈ Rm×n admits the decomposition W = UΣV⊤ , where U ∈ Rm×m and V ∈ Rn×n are orthogonal, and Σ is diagonal with singular values σ1 ≥ σ2 ≥ · · · ≥ 0. We call v1 (first column of V) the principal right singular vector (input space) and u1 (first column of U) the principal left singular vector (output space). Geometric continuity. For a sequence of weight matrices W(1) , . . . , W(L) across layers, we define geometric continuity as the mean absolute cosine similarity between adjacent layers’ principal singular vectors: L−1 1 X (l) (l+1) Continuity(v1 ) = cos v1 , v1 (1) L−1 l=1

The absolute value handles the intrinsic sign ambiguity of singular vectors: (u, v) and (−u, −v) represent thep same SVD component. A value near 1 indicates that v1 evolves smoothly across layers; a value near 2/(πd) (where d is the vector dimension) indicates random, unstructured directions. σ 2 -weighted continuity. The v1 -only metric captures alignment of the dominant direction but ignores whether that direction is truly dominant. We additionally define a σ 2 -weighted variant P that considers all singular vectors, weighted by their contribution to the Frobenius norm ∥W∥2F = k σk2 : 2 (l) L−1 K σk 1 XX (l) (l+1) Continuityσ2 = cos v , v (2) k k 2 P L−1 (l) l=1 k=1 σ j j Here K = min(m, n) is the number of non-zero singular vectors of W ∈ Rm×n (e.g., K = 256 for our toy MLP experiments); the weighting is anchored to layer l (asymmetric in l ↔ l + 1) for simplicity. This matches the explained variance ratio used in PCA and aligns with the σ 2 -based definition of effective rank below. When σ1 dominates, this reduces to v1 continuity; when the spectrum is flat, all directions contribute equally. We use this metric in Section 4.3 to distinguish how activation versus LayerNorm break rotational symmetry: activation concentrates structure in v1 , while LayerNorm distributes it across multiple directions. Effective rank.

The effective rank [14] of a matrix with singular values σ1 , . . . , σn is: ! X σ2 erank(W) = exp − pi log pi , pi = P i 2 j σj i

(3)

This measures the effective dimensionality of the matrix. A rank-1 matrix has erank = 1; a matrix with all equal singular values has erank = n. 3

Table 1: Ablation results (MNIST, 16-layer MLP, 3 seeds, mean). v1 /u1 = cross-layer cosine similarity of the 1st right/left singular vector; σ 2 -WA = σ 2 -weighted alignment across all P (t) right singular vectors; Ḡv1 = continuity of the cumulative gradient Ḡl = t Gl ; ∆GW = cont(v1 (Ḡl )) − cont(v1 (Wl )) measures coherence lost between the cumulative gradient and the final weight. Res+Radial uses the rotation-equivariant nonlinearity σrad (x) = x · tanh(∥x∥). Bold = notably high; underline = notably low that contrasts with high gradient coherence. Full metrics in Appendix E. Configuration

Acc

G v1

G σ 2 -WA

Ḡ v1

W v1

W σ 2 -WA

∆GW

W u1

Res+GELU Res+SiLU Res+ReLU Res+Tanh Res+None Res+Radial

.978 .977 .977 .971 .912 .929

.901 .831 .792 .846 .927 .941

.887 .809 .782 .756 .928 .934

.976 .977 .961 .912 .859 .859

.964 .964 .959 .842 .217 .225

.249 .242 .249 .285 .059 .059

+.013 +.013 +.003 +.070 +.641 +.633

.955 .962 .940 .061 .218 .241

Res+None+LN Res+ReLU+LN

.918 .980

.885 .876

.872 .863

.879 .956

.578 .905

.303 .252

+.301 +.051

.622 .914

NoRes+GELU NoRes+ReLU NoRes+None

.924 .848 .873

.046 .067 .057

.047 .068 .055

.102 .137 .054

.175 .378 .048

.093 .089 .050

−.073 −.241 +.006

.549 .701 .059

Gauge symmetry and rotation orbits. Linear products of weight matrices possess a continuous symmetry: for any orthogonal R, (Wl R⊤ )(RWl−1 ) = Wl Wl−1 leaves the product unchanged. The set of all weight pairs related by such rotations forms an orbit of the rotation group SO(d) (the group of d-dimensional rotations). Within an orbit the combined function is identical, so the loss landscape is flat along orbit directions and gradient descent can drift along them without resistance. Section 4.3 shows that activation breaks this gauge symmetry, eliminating the drift.

What Causes Geometric Continuity?

We investigate the origin of geometric continuity through ablation experiments on a toy MLP, systematically isolating the contribution of each architectural and data factor. 4.1

Experimental Setup

We use a residual MLP with D=256 hidden dimensions and L=16 layers. Each layer computes 256×256 hl+1 = hl + act(Wl hl ), where Wl ∈ R√ has no bias and is initialized with PyTorch’s default Kaiming uniform scheme (bound = 1/ 256 ≈ 0.0625, std ≈ 0.036). An embedding layer maps inputs from R784 to R256 , and a linear head maps to 10 classes. We train on MNIST (60K train / 10K test, standard split) with Adam (lr = 10−3 , batch size 128) for 20 epochs. Unless otherwise noted, all results report means over 3 seeds (42, 123, 777); the baseline Res+ReLU achieves 97.7% test accuracy. This toy model allows independent removal of residual connections, swapping of activation functions, or changing the data distribution, while preserving the essential structure of deep residual networks. We measure continuity metrics—v1 , u1 , and σ 2 -weighted alignment (Section 3), with v2 reported in Appendix E—for both weight and gradient matrices across the 16 layers W1 , . . . , W16 . All toy experiments run on a single NVIDIA A100 GPU in under 10 minutes per configuration. 4.2

Condition 1: Residual Connections

Residual connections create gradient coherence from initialization. In a residual network, the gradient with respect to layer l’s input satisfies: ∂L ∂L ∂fl = I+ (4) ∂hl ∂hl+1 ∂hl 4

Figure 2: Rotation drift and continuity collapse without activation (both configurations use small initialization σ=0.0001, MNIST, 50 epochs). (a) Per-layer weight v1 rotation angle from epoch 1 reference (left two panels): Res+None layers all rotate to ∼85–90 (mutual misalignment), while Res+ReLU layers rotate only ∼25–35 coherently. (b) Inter-layer weight (red) and gradient (blue) v1 continuity, and test accuracy (green dashed, right y-axis) over training (right two panels): Res+None weight continuity peaks at epoch 1 (∼1.0) then collapses to ∼0.18 with gradient continuity following, and accuracy shows a transient dip at epoch 15 (0.92 → 0.61) that recovers to 0.92 without restoring continuity; Res+ReLU remains stable near 0.94 continuity and 0.98 accuracy throughout. The identity term I ensures that adjacent layers receive similar error signals. We verify this empirically with Res+ReLU: gradient v1 cross-layer continuity is 0.95 at the first backward pass, before any weight update, while weight v1 continuity starts at 0.04 (random) and gradually increases to 0.96 during training (Figure 12 in Appendix C). Without residual, gradient coherence is lost. Removing residual connections (hl+1 = act(Wl hl )) eliminates gradient coherence: gradient v1 continuity drops to 0.05–0.07 (random level), and weight v1 continuity reaches only 0.05–0.38 (Table 1), far below the 0.96 obtained with residual connections. Interestingly, NoRes+ReLU exhibits high u1 continuity (0.70) despite low v1 continuity (0.38), suggesting a forward-signal pathway distinct from the residual-mediated gradient coherence we focus on; characterizing it is left to future work. 4.3

Condition 2: Symmetry Breaking (Activation and LayerNorm)

The critical experiment: Res+None. The most revealing experiment removes the activation function while keeping residual connections (Res+None: hl+1 = hl + Wl hl ). This configuration achieves gradient v1 continuity = 0.93 ± 0.02 (high—residual still provides coherence) but weight v1 continuity = 0.22 ± 0.04 (far below Res+ReLU’s 0.96). Despite receiving coherent gradient signals, the network develops only weak weight continuity—far below the 0.96 achieved with activation. The gap is even starker in σ 2 -weighted alignment: gradient σ 2 -WA reaches 0.928 across all singular vector directions, yet weight σ 2 -WA is only 0.059—rotational degeneracy blocks transfer not just in the dominant direction but across the entire spectrum. This demonstrates that gradient coherence alone is insufficient for sustained weight continuity—symmetry breaking is needed to stabilize the transfer. Rotational symmetry breaking. In deep non-residual linear networks, adjacent layers admit an exact rotational degeneracy: Wl Wl−1 = (Wl R⊤ )(RWl−1 ) for any orthogonal R, so individual SVDs are underdetermined and flat directions appear in the loss landscape [11]. Adding residual connections partially breaks this symmetry: (I + Wl R⊤ )(I + RWl−1 ) ̸= (I + Wl )(I + Wl−1 ) in general, as the identity terms couple the rotation R to the product. Marcotte et al. [13] formalize this: consecutive residual blocks share no conservation laws spanning across them (their Theorem 4.7), establishing that the cross-block direction is unprotected by symmetry. Consistent with this, we observe empirically that sufficient near-flatness remains to allow drift: Res+None’s weight continuity forms transiently then collapses (Figure 2, quantified below). Element-wise activations break this remaining symmetry more sharply: Godfrey et al. [12] characterize ReLU’s intertwiner group as permutations and positive rescaling—a discrete subgroup that preserves coordinate axes. Since act(W2 R · R⊤ W1 x) ̸= act(W2 W1 x) in general, the rotational degrees of freedom are eliminated, and each layer’s weight is constrained to a shared coordinate frame. We distinguish this direct ⊤ continuity (vl ↔ vl+1 ) from Ji and Telgarsky [8]’s signal-flow alignment (|vl+1 ul | → 1), which can dissociate. 5

Two-phase dynamics: formation versus retention. This drift dissociates two distinct dynamical phases that the residual + activation pair both support, but only activation retains. To make the two phases observable in isolation, we train Res+None with very small initialization (σ=0.0001) so the first epoch’s ∆W immediately dominates W(0) . Phase 1 (formation): at epoch 1, weight v1 continuity reaches 0.99 (Figure 2b, Res+None panel)—gradient-to-weight transfer does occur without activation, driven entirely by residual-mediated gradient coherence. Phase 2 (retention vs. drift): by epoch 15, Res+None’s weight continuity has collapsed to ∼ 0.18 as each layer’s rotation frame drifts independently along its SO(d) orbit (Figure 2a, Res+None panel). Crucially, weight continuity collapses first (epochs 5–10) and gradient continuity follows (epochs 10–15)—a positive feedback loop where weight drift induces hidden-state divergence, which in turn destroys gradient coherence. The collapse coincides with a transient accuracy dip (epoch 15: 0.92 → 0.61) that recovers to 0.92 without restoring continuity (Figure 2b, dashed green): the network finds a functional but structurally different solution. Res+ReLU with identical initialization maintains v1 > 0.94 throughout 50 epochs (Figure 2, Res+ReLU panels). Thus residual is sufficient for Phase 1, but Phase 2 retention requires symmetry breaking. Symmetry, not nonlinearity, is the active ingredient. The above argument predicts that any rotation-equivariant activation should fail to pin the frame, despite being nonlinear. We test this with the radial activation σrad (x) = x · tanh(∥x∥), which acts only on the magnitude (preserving SO(d) exactly: σrad (Rx) = Rσrad (x)). Even though Res+Radial is nonlinear, weight v1 continuity drops to 0.225 ± 0.040 (vs. 0.217 for Res+None and 0.959 for Res+ReLU; Appendix E). To P (t) localize this drift more precisely, we track the cumulative raw gradient Ḡl = t Gl and define ∆GW = cont(v1 (Ḡl )) − cont(v1 (Wl )), the gap between coherence created in the cumulative gradient and coherence retained in the weight. For Res+ReLU, ∆GW = 0.003 (full transfer); for Res+None and Res+Radial, ∆GW = 0.641 and 0.633 (cumulative gradient is coherent at ∼ 0.86 in both cases, but drift erases it before reaching the weight). This ∼ 200× separation in ∆GW isolates symmetry breaking—specifically, breaking of the per-layer SO(d) symmetry—as the active ingredient: nonlinearity per se is insufficient. In Res+None and Res+Radial the gradient direction is itself rotated freely by per-layer drift; in Res+ReLU it is anchored to a fixed coordinate frame and accumulates faithfully into the weight. Activation type matters. Different activations reduce the symmetry group to different extents [12], directly determining weight continuity (Table 1). GELU, SiLU, and ReLU all reduce symmetry to permutations (and rescaling for ReLU), none of which affect singular vector directions—yielding similar continuity (∼0.96). Tanh, being an odd function (tanh(−x) = − tanh(x)), retains a sign-flip symmetry (w ↔ −w per neuron) that introduces additional directional degeneracy, yielding lower v1 continuity (0.842); additionally, Tanh’s output saturation suppresses backward gradient flow in the output direction, collapsing u1 to 0.061 (detailed mechanism in Appendix E). Linear (None) preserves the full rotational symmetry, yielding only 0.217. LayerNorm: a different kind of symmetry breaking. LayerNorm also breaks rotational symmetry, but differently from activation. Activation concentrates continuity in v1 (Res+ReLU: W v1 = 0.959, σ 2 -WA = 0.249), while LayerNorm distributes it across multiple directions (Res+None+LN: W v1 = 0.631, σ 2 -WA = 0.303). When combined (Res+ReLU+LN), activation’s v1 -concentrated mode dominates and LN’s contribution is largely absorbed (σ 2 -WA = 0.252, essentially unchanged). The critical requirement is symmetry breaking in general; task structure modulates continuity within an architecture but does not substitute for it (Appendix F). 4.4

How Gradient Accumulation Forms Weight Structure

Gradient continuity precedes weight and reflects the long-term gradient average. Gradient v1 continuity is 0.95 at the first backward pass (before any weight update), while weight v1 starts at 0.04 and rises to 0.96 during training (Figure 12, Appendix C)—gradient continuity precedes (T ) (0) weight continuity, consistent with a causal direction from gradient to weight. Since Wl = Wl − P (t) η t Gl , weight structure is the cumulative result of gradient updates; computing exponential (t) (t−1) (t) moving averages Ḡl = β Ḡl + (1−β)Gl at various smoothing levels confirms this, with weight v1 aligning most strongly with the long-term gradient average (0.42 for raw gradient, rising to 0.84 at EMA β=0.999). 6

From gradient coherence to weight coherence. Combining Conditions 1 and 2 yields the full (t) (t) mechanism. Residual connections ensure Gl ≈ Gl+1 (Section 4.2), so cumulative gradients are P (t) (0) also cross-layer coherent. Once η∥ t Gl ∥ dominates ∥Wl ∥, weight v1 converges to the cumulative gradient v1 , inheriting its cross-layer coherence; symmetry breaking then stabilizes this transfer by pinning each layer to a shared coordinate frame (Section 4.3). We verify the ∥∆W∥/∥W(0) ∥ prediction with Res+ReLU trained via Adam β2 =0, ε=1 (Appendix D), which produces small updates: at default Kaiming initialization, β2 =0 yields near-zero continuity (v1 = 0.047); progressively shrinking the initialization to σ=0.0001 recovers v1 = 0.958, matching Adam default (Table 3). At σ=0.0001 continuity is high across all singular directions (v2 =0.921, u1 =0.948, σ 2 -WA=0.949), because ∆W dominates W(0) in every direction. Test accuracy, however, drops to 0.885 (vs. 0.977 for Adam default), illustrating that continuity reflects the direction of weight structure while performance also requires sufficient magnitude.

Transformer Experiments

We extend our analysis to transformers to test whether the toy MLP findings generalize and to uncover transformer-specific phenomena.

Figure 3: Residual stream read/write structure of a transformer block. Each projection either reads from the residual stream (green up arrows: Q, K, V, Gate, Up) or writes to it (red down arrows: O, Down). Nonlinearities (softmax, σ) are shown as blue boxes. Section 5.2 uses this structure to predict each projection’s continuity space. 5.1

Setup

We train a small Llama-style transformer (D=256, 8 layers, 4 heads, SiLU + RMSNorm, ∼34M parameters) on WikiText-103 [15] using the GPT-2 tokenizer. All linear and embedding layers are initialized with N (0, 0.022 ), matching standard LLM practice. Training uses 10,000 steps with batch size 128, sequence length 128, seed 42. We run 4 activation ablations and 2 failure-mode controls (Table 2). 5.2

Projection-Specific Continuity in Transformers

Our most striking transformer finding is that continuity is projection-specific: each projection develops continuity in the space facing the residual stream, with its strength determined by a nearby nonlinearity (Figure 3). The Llama-style MLP computes: MLP(x) = Wdown (SiLU(Wgate x) ⊙ Wup x)

(5)

The activation is applied only to Gate’s output. Attention computes Q, K, V linearly, with softmax √ providing nonlinearity in softmax(QK⊤ / d). Ablating MLP activation drops Gate continuity. Table 2 isolates the MLP-side effect: removing the MLP activation drops Gate v1 from 0.662 to 0.366 while only modestly affecting perplexity (41.9 → 43.2, ∼3% increase). The attention-side projections (Q, K, V) are architecturally independent of the MLP activation, so their continuity is largely unchanged; isolating the softmax mechanism for Q, K would require architectural modification and is left to future work. 7

Table 2: Transformer continuity results. Top: small Llama-style transformer ablations (10K steps, init N (0, 0.022 )). Middle: pretrained Llama-3.1-8B (mean ± std over 32 layers). The projection-specific pattern sharpens at scale: Q/K/Gate develop strong input-space (v1 ) continuity, O/Down develop strong output-space (u1 ) continuity, and V remains low in both spaces. Bottom: removing residual or norm causes training failure. Config

PPL

Q v1

K v1

Gate v1

Up v1

O u1

Down u1

V v1

Res+SiLU+Norm Res+ReLU+Norm Res+GELU+Norm Res+None+Norm

41.9 42.8 41.9 43.2

0.405 0.398 0.370 0.492

0.627 0.671 0.609 0.668

0.662 0.820 0.804 0.366

0.225 0.308 0.350 0.359

0.129 0.339 0.147 0.221

0.554 0.532 0.507 0.517

0.209 0.188 0.115 0.237

Llama-3.1-8B

—

0.82

0.77

0.80

0.63

0.89

0.74

0.14

NoRes+SiLU+Norm NoNorm (any)

1700 NaN

0.036 —

0.062 —

0.045 —

0.036 —

0.066 —

0.052 —

0.054 —

V and O both show modest continuity in this small-transformer regime, but O u1 sharpens substantially in pretrained Llama-3.1-8B (µ = 0.89; Section 5.3)—suggesting that output-space continuity for O consolidates with scale and longer training, while V’s low continuity persists across scales. Perplexity stays near baseline because the bilinear Gate⊙Up multiplication and RMSNorm remain active; removing residual or RMSNorm (Table 2, bottom) causes training failure, underscoring that activation is dispensable for function while residual and norm are not. 5.3

Projection-Specific Pattern in Pretrained LLMs

The full projection-specific pattern emerges clearly in pretrained Llama-3.1-8B [5] (32 layers, d=4096; Table 2, Appendix A): Q, K, Gate, and Up show high v1 mean adjacent-layer cosine similarity (µ = 0.82, 0.77, 0.80, 0.63) while V shows the lowest (µ = 0.14); O and Down show high u1 continuity (µ = 0.89, 0.74). At small scale, several projections (notably O and Up) are still consolidating—only Gate, K, and Down are clearly above baseline (Table 2)—while V remains low at both scales. These patterns are consistent with our mechanism. Q, K develop input-space (v1 ) continuity because softmax acts on their output in QK-space. Gate develops input-space continuity from the MLP activation. Up shows intermediate input-space continuity because the activated Gate masks its gradient via element-wise multiplication: Up’s gradient is suppressed where σ(g) ≈ 0, while Gate’s gradient is active whenever σ ′ (g) is nonzero. This asymmetric masking is a plausible contributor, though the precise mechanism warrants dedicated analysis. O and Down develop output-space (u1 ) continuity because they receive post-nonlinearity signals. V shows neither because no nonlinearity is applied to it. The projection-specific continuity pattern also holds for five additional models spanning 1.5B–70B parameters (Appendix B), confirming scale invariance across two orders of magnitude.

Discussion

Continuity as a byproduct of learning. Our central finding is that geometric continuity is not an artifact of a specific optimizer or training trick, but a structural byproduct of learning in residual networks with symmetry-breaking nonlinearity. Within a single training run, continuity rises rapidly and saturates well before convergence (Appendix C). Two mechanisms combine: residual connections create cross-layer gradient coherence, which aligns weight updates across layers; symmetry-breaking nonlinearities then constrain all layers to a shared coordinate frame, preventing the rotation drift that would otherwise destabilize the accumulated structure (Section 4.3). Projection-specific nonlinearity determines projection-specific structure. The transformer experiments reveal a precise correspondence: each projection’s continuity appears in the space—input (v1 ) or output (u1 )—that faces the residual stream, and is stabilized by the nonlinearity acting on the projection. Q, K, Gate, and Up read from the residual stream and show input-space continuity, with 8

softmax (Q/K), the MLP activation (Gate), and gated multiplication (Up) preventing rotation drift. O and Down write to the residual stream and show output-space continuity from the post-nonlinearity signals flowing into them. V—which lacks any direct nonlinearity—shows only low continuity regardless of configuration. This provides a mechanistic explanation for the space-specific continuity we observe in pretrained LLMs (Appendix A), connecting architectural design to weight geometry. Implications for model compression. Our results provide a mechanistic basis for layer pruning [3, 4] and cross-layer parameter sharing [2]. Since continuity arises from learning itself, well-trained models will naturally exhibit the cross-layer similarity that enables compression. The projectionspecific pattern further suggests that compression strategies should respect the space in which each projection is structured: Q, K, Gate, and Up admit a shared input-space basis across layers, while O and Down admit a shared output-space basis; V, which develops only low continuity in either space, has no obvious cross-layer sharing to exploit. Limitations. Our mechanistic claim for projection-specific continuity rests on one direct ablation (Gate); the Q/K claims are indirect since softmax cannot be ablated without changing the attention mechanism, and V has no nonlinearity to remove. Our transformer experiments use a small model (34M parameters, 8 layers) trained on WikiText-103. While the same patterns appear in pretrained Llama-3.1-8B and models up to 70B parameters (Appendix A, B), we have not verified the training dynamics (rotation drift) at larger scales. Transformer experiments use a single seed; toy MLP results with 3 seeds show low variance, but transformer variance remains uncharacterized.

Conclusion

We investigated the origin of geometric continuity in weight matrices through controlled experiments on toy MLPs, with transformer validation in pretrained LLMs. Two necessary conditions emerge: residual connections provide cross-layer gradient coherence, and symmetry-breaking nonlinearities constrain all layers to a shared coordinate frame, preventing rotation drift that would otherwise destabilize weight structure. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. Without symmetry breaking, continuity does not stably develop; even when it forms transiently under small initialization, it collapses as layers’ rotation frames drift apart. A rotation-equivariant control activation—nonlinear yet SO(d)-preserving—fails to retain continuity, isolating symmetry breaking, not nonlinearity itself, as the active ingredient. In transformers, the same mechanism produces a projection-specific pattern that emerges with scale: in pretrained LLMs, Q, K, Gate, and Up (reading from the residual stream) develop input-space (v1 ) continuity; O and Down (writing to it) develop output-space (u1 ) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity. These findings establish geometric continuity as a structural property of learned residual networks, connecting architectural design choices to the emergent weight geometry relevant to downstream applications. Future work. Our framework opens three directions. (1) Designing continuity profiles: if symmetry breaking determines continuity, partial symmetry breakers (e.g., activations that preserve specific rotational subgroups) could be used to engineer targeted continuity patterns—concentrating structure in v1 alone or distributing it across multiple directions on demand. (2) Drift as a forgetting indicator: monitoring rotation drift during fine-tuning may detect when a model loses pretrained structure, providing a diagnostic for catastrophic forgetting and a signal for continual learning. (3) Orbit-aware model merging: if multiple fine-tuned checkpoints occupy different points on a shared SO(d) orbit, projecting them onto a common orbit representative could reduce destructive interference in model merging and souping.

References [1] Zeping Min and Xinshang Wang. DOCS: Quantifying weight similarity for deeper insights into large language models. In International Conference on Learning Representations, 2025. 9

[2] Jingcun Wang, Yu-Guang Chen, Ing-Chao Lin, Bing Li, and Grace Li Zhang. Basis sharing: Cross-layer parameter sharing for large language model compression. In International Conference on Learning Representations, 2025. [3] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024. [4] Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in large language models are more redundant than you expect. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. [5] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [6] Jianing Li and Vardan Papyan. Residual alignment: Uncovering the mechanisms of residual networks. In Advances in Neural Information Processing Systems, 2023. [7] Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer is secretly linear. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. [8] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2019. [9] Pierre Marion, Yu-Han Wu, Michael E Sander, and Gérard Biau. Implicit regularization of deep residual networks towards neural ODEs. In International Conference on Learning Representations, 2024. [10] Daniel Beaglehole, Ioannis Mitliagkas, and Atish Agarwala. Feature learning as alignment: a structural property of gradient descent in non-linear neural networks. arXiv preprint arXiv:2402.05271, 2024. [11] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014. [12] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. In Advances in Neural Information Processing Systems, 2022. [13] Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Transformative or conservative? conservation laws for ResNets and transformers. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. [14] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 15th European Signal Processing Conference (EUSIPCO), 2007. [15] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [16] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019. [17] An Yang, Baosong Yang, Beichen Zhang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [18] Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. [19] LG AI Research. EXAONE 4.0 technical report. arXiv preprint, 2025.

Figure 4: 3D PCA of principal right singular vectors (v1 ) across 32 layers. Colors indicate layer index (blue to yellow). Q, K, Up, Gate show smooth trajectories; V, O, Down are scattered; OV composite (WO WV ) shows moderate structure.

Geometric Continuity in Pretrained LLMs

A.1

Experimental Setup

Model and Weight Matrices. We analyze Llama-3.1-8B [5], a 32-layer transformer with hidden dimension d = 4096 and intermediate dimension df f = 14336. We extract seven weight matrices per layer: • Attention: WQ ∈ R4096×4096 , WK , WV ∈ R1024×4096 (GQA with 8 KV heads), WO ∈ R4096×4096 • MLP: Wup , Wgate ∈ R14336×4096 , Wdown ∈ R4096×14336 Sign Alignment. Singular vectors are only defined up to sign: if Wv = σu, then W(−v) = −σu, so both (u, v) and (−u, −v) are valid singular vector pairs. Since SVD algorithms return signs arbitrarily, we apply sequential sign alignment to ensure consistent orientation across layers. For each layer l > 0, we flip the sign if the dot product with the previous layer is negative: (l) (l) (l−1) (l) v1 ← sign ⟨v1 , v1 ⟩ · v1 (6) This alignment is essential for visualizing smooth trajectories in PCA plots, as arbitrary sign flips would create artificial discontinuities in the visualization. (The scalar continuity metric defined in Section 3 is sign-invariant by construction, via the absolute value, so it does not rely on this alignment.) A.2

Geometric Continuity

PCA Visualization. To analyze the geometric structure of principal vectors across layers, we (l) perform PCA on the collection {v1 }31 l=0 . Note that this constitutes a second SVD: the first extracts principal directions from each weight matrix, while the second (PCA) analyzes how these directions are distributed across layers. Specifically, we: (1) L2-normalize each vector, (2) center by subtracting the mean, P and (3) apply SVD to obtain principal components. The explained variance ratio EVRi = σi2 / j σj2 quantifies what fraction of the cross-layer variance is captured by each PC. As a baseline, 11

Figure 5: 3D PCA of principal left singular vectors (u1 ) across 32 layers. Colors indicate layer index (blue to yellow). O, Down show smooth trajectories; Q, K, V, Gate, Up are scattered—opposite to Fig. 4; OV composite (WO WV ) shows high continuity similar to O. 32 random unit vectors in R4096 yield expected 3-PC EVR of ∼10%. Thus, observed EVR > 50% in 3 PCs indicates that the layer-wise vectors lie on a low-dimensional manifold—far from random. The 3D PCA visualizations (Figs. 4 and 5) quantitatively reveal space-specific continuity. Different projections exhibit geometric continuity in different spaces: right singular vectors (v1 ) for Q, K, Gate, Up—these projections show continuity in input space, reflecting how they “read” from the residual stream; left singular vectors (u1 ) for O, Down—these projections show continuity in output space, reflecting how they “write” to downstream computations. The value projection shows low continuity in both spaces, motivating composite WO WV analysis. Right Singular Vectors (Fig. 4). Q, K, Up, and Gate form smooth curved trajectories with high explained variance (Q: 55%, K: 51%, Up: 42%, Gate: 53% in first 3 PCs), confirming consistent inputspace geometry across layers. V shows weak structure (19% EVR), notably lower than Q/K/Up/Gate but higher than O/Down. O and Down show scattered, unstructured distributions (EVR ≈ 11%), indicating layer-specific principal input directions. The OV composite shows moderate structure (24% EVR), indicating partial recovery from V’s lack of coherence. Left Singular Vectors (Fig. 5). O exhibits a remarkably dominant first PC (EVR = 0.56) with total 76% in 3 PCs, forming a nearly linear progression from layer 0 to 31. Down shows clear trajectory structure (52% EVR) in contrast to its scattered right-vector pattern. Q, K, V, Gate, and Up are now scattered (EVR ≈ 13%), opposite to their right-vector behavior. The OV composite exhibits high continuity (61% EVR) with a smooth trajectory similar to O, demonstrating geometric recovery through composition. V Projection. The value projection WV shows near-zero continuity in both input and output spaces, consistent with the main text finding that V lacks direct nonlinearity (Section 5.2). Layer-wise Continuity. The cosine similarity analysis (Fig. 6) reveals three groups: highcontinuity projections (O: µ = 0.89, Q: 0.82, Gate: 0.80, K: 0.77), intermediate (Down: 0.74, Up: 0.63), and low-continuity (V: 0.14). Continuity is measured as defined in Section 3. Space-Specific Continuity. The central observation is space-specific continuity: projections show high continuity in one space (input or output) but not both. Q, K, Gate, Up show input-space 12

Figure 6: Layer-wise continuity (cosine similarity) between adjacent principal singular vectors. Each projection uses its high-continuity space: right vectors (v1 ) for Q, K, Up, Gate; left vectors (u1 ) for O, Down; V and VO use right vectors (arbitrary, as both spaces show low continuity). High-continuity projections maintain µ > 0.75, while V (µ = 0.14) shows near-orthogonal adjacent vectors. (v1 ) continuity; O, Down show output-space (u1 ) continuity; V shows neither. In the main text (Section 5.2), we trace this pattern to the nonlinearity acting on each projection: softmax for Q/K, SiLU for Gate, and no nonlinearity for V.

Additional Transformer Models

We replicate our analysis on five additional models spanning 1.5B–70B parameters. All exhibit the same space-specific continuity pattern. GPT-2 XL (1.5B).

OpenAI [16], GELU activation, no gating (Fig. 7).

Qwen3-8B (8B).

Alibaba [17], SwiGLU, grouped-query attention (Fig. 8).

Gemma-3-12B (12B).

Google [18], 48 layers, alternating local/global attention (Fig. 9).

EXAONE-4.0-32B (32B). Llama-3.1-70B (70B).

LG AI Research [19], bilingual Korean-English (Fig. 10).

Meta [5], same architecture as 8B at larger scale (Fig. 11).

Across all models (1.5B–70B), the same space-specific continuity pattern holds, confirming scale invariance.

Gradient Accumulation Mechanism

Figure 12 tracks the temporal evolution underlying the claims in Section 4.4. Gradient v1 continuity is already high at the first backward pass (∼0.95, before any weight update), while weight v1 continuity rises from 0.04 to 0.96 over training, and the final weight v1 aligns most strongly with the long-term gradient EMA (β=0.999: ∼0.85).

Adam β2 =0 Configuration

Adam β2 controls the second-moment estimate that provides per-parameter adaptive learning rates. Setting β2 =0 with ε=1 disables the adaptive scaling that normally amplifies updates: the update 13

(a) Right singular vectors (v1 )