arXiv CSopen access

Signed-Permutation Coordinate Transport for RMSNorm Transformers

Signed-Permutation Coordinate Transport for RMSNorm Transformers

arXiv:2606.31963v1 [cs.LG] 30 Jun 2026

John Sweeney Sideplane AI [email protected] https://sideplane.ai

Abstract Modern LLM workflows increasingly move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-k neuron sets, attribution lists, and merge alignments. This is only well posed after fixing the model’s residual-stream gauge. We show that the native discrete gauge is architecturedependent: LayerNorm residual charts have permutation gauge Sd , up to a global sign flip, while RMSNorm residual charts with generic per-channel gain have signed-permutation gauge Bd = Sd ⋉ {±1}d . Thus permutation-only alignment is symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove a sharp population failure mode: with decorrelated source coordinates, raw signed-correlation matching has a structural permutationaccuracy ceiling equal to the fraction of positive signs in the true gauge up to O(d2−d ), whereas sign-marginalized matching removes this obstruction. We then make coordinate-preserving transport, rather than function-level merging, the primary object: composing saved-checkpoint local Bd gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching, and the gain is not explained by merely routing through the base. The recovered gauge transfers tools that permutationonly alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under Bd recovery versus 1.08 under Sd ; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under Sd . Coordinate-preserving merge tests show the same mechanism. The same covariance governs stateful training: signed transport of AdamW state preserves the resumed trajectory, while permutation-only state transport starts from a functionally identical checkpoint but follows a different trajectory. Finally, we give gauge-sweep audits for indexlevel interpretability claims: coordinate names are reproducible only relative to an explicit gauge.

1

Introduction

The question “which neuron is this?” is ill-posed without a gauge specification. If a residual coordinate of a transformer is relabeled, the input-output function is unchanged but every index-level object attached to that coordinate moves with the relabeling. This is not merely philosophical: modern workflows publish coordinate-indexed artifacts–ranked neuron or attribution sets, knowledge-neuron edits, SAE dictionaries, steering directions, LoRA updates, and merge alignments. A coordinateindexed claim should therefore pass a simple audit: exact gauges should preserve behavior, move raw coordinate names, and restore agreement after mapping back. Standard alignment tools such as Git Re-Basin and weight matching search over permutations [Ainsworth et al., 2023, Entezari et al., 2022]. That is correct for LayerNorm residual streams, but widely used open-weight LLMs such as Llama 2 and Qwen2.5 use RMSNorm [Touvron et Preprint.

al., 2023, Yang et al., 2024a, Zhang and Sennrich, 2019]. RMSNorm lacks mean-centering. Its residual stream therefore admits not only permutations but independent sign flips of every coordinate. The relevant discrete group is the hyperoctahedral group Bd of order 2d d!, not Sd of order d!. In a 4096-dimensional residual stream, the omitted sign component fixes only one of 24096 gauge cosets and forgoes 4096 binary degrees of freedom; Corollary 2.4 shows that mis-applying it to a learned 2 weight update introduces squared error of magnitude 2 ∥∆W ∥F in expectation, the same order as the update’s own energy. Many analyses are already gauge-invariant; our target is workflows that explicitly move or publish coordinate-indexed artifacts. This paper is about the consequences of using the right group. We first prove that LayerNorm has Sd , up to a global sign flip, while RMSNorm has full signed-permutation gauge Bd for generic gain γ, and that Bd is the maximal coordinate-preserving subgroup of O(d) in the native γ-explicit parameterization. Broader orthogonal freedom returns only after absorbing γ into adjacent weights; this is useful for compression or dense fusion, but it changes the coordinate system and does not preserve sparse coordinate identity. We then recover the gauge from activations. The natural algorithm is the activation-matching linear assignment used by earlier neuron-alignment and Git Re-Basin work [Li et al., 2016, Tatro et al., 2020, Ainsworth et al., 2023], solved by Hungarian-style assignment methods [Kuhn, 1955], except that the cost must marginalize over sign: match columns by | hi , h′j |, then recover the sign from the matched correlation. The absolute value is required by the Bd symmetry: under a negative gauge sign, the correct coordinate is maximally anti-correlated. We prove this ceiling in the decorrelatedcoordinate population case: when half the true signs are negative, the correct entries for those coordinates are the most negative entries in their rows, so signed-correlation maximization avoids them even with unlimited probes. Recorded-gauge LLM activations show the same obstruction: signed-cost recovery stays near 50%, while sign-marginalized recovery reaches 100%. The constructive payoff is transport when an atlas exists. Instead of matching only two endpoints, we compose saved-checkpoint local Bd gauges along a training trajectory. On a gauge-instrumented Qwen2.5-1.5B fine-tuning benchmark (1500 steps, with a randomly sampled Bd gauge applied at step 0 and its induced correspondence recorded as ground truth), endpoint matching recovers 60.3% of coordinates while transport-via-base recovers 91.1%; endpoint-via-base recovers only 11.4%, so the improvement is not a base-reference artifact but comes from many small, high-margin local gauges. The result is Bd -specific: Sd -only transport drops signs and reverses transferred tools; continuous-rotation atlases preserve function but lose coordinate identity (Theorem 2.2); only Bd preserves both, so steering vectors and adapters transfer cleanly under Bd recovery. The same sign failure mode appears in coordinate-preserving merge tests. In a Llama-2-7B gaugescramble merge, signed alignment eliminates the peak-above-chord barrier (6.14 → 0.00), while exact permutation-only alignment is actively harmful. In independently trained 10M-parameter RMSNorm transformers, sign mismatch appears between seeds without an external basis change, and signed alignment improves the barrier from 0.66 to 0.42 over permutation-only. These merge results are scoped to the coordinate-preserving regime: when coordinate identity is the object, RMSNorm alignment must include signs. The same gauge covariance governs AdamW optimizer state: signed transport preserves the resumed training trajectory, while permutation-only state diverges despite starting from a functionally identical checkpoint. Finally, we use gauge sweeps as a falsifiable negative control for coordinate-indexed interpretability. Coordinates are not always the right units; directions, subspaces, CKA, and behavior-level interventions are often better and already gauge-invariant [Raghu et al., 2017, Morcos et al., 2018, Kornblith et al., 2019]. But the field still produces coordinate-indexed artifacts–knowledge neurons, top-k steering coordinates, SAE feature dictionaries, sparse ablation sets–and these are reproducible only relative to a gauge. Cross-run reproducibility for such claims requires a gauge anchor (e.g., a same-base atlas), Bd -correct READ/WRITE rules for any artifact crossing parameterizations, and the audit as a negative control: under an exact random gauge, raw indices should move, mapped indices should agree, and behavior should remain fixed.

2

Native residual-stream gauge

We use row-vector notation. A residual activation matrix H ∈ Rn×d is gauged by right multiplication H 7→ HG. A signed permutation G ∈ Bd has exactly one nonzero entry, ±1, in each row and column. 2

A permutation is the special case with all signs +1. We use “gauge” in the reparameterization sense: a choice of coordinates within a function-preserving symmetry orbit. In this paper the relevant group is the global discrete signed-permutation action Bd on residual coordinates. Theorem 2.1 (Maximal native discrete gauge). Consider a transformer with residual dimension d, tokenwise LayerNorm or RMSNorm, standard affine projections Y = XW ⊤ + b, and attention/MLP sublayers that interact with residual coordinates only through linear maps, elementwise nonlinearities, and the normalization. Then the following groups act by function-preserving residual-coordinate relabelings:  {±P : P ∈ Sd } ∼ = Sd × C2 , LayerNorm, G= Bd , RMSNorm with generic per-channel gain γ. For LayerNorm, the C2 factor is one global sign flip with β 7→ −β, so the per-index content is still Sd . Moreover, for generic RMSNorm γ, Bd is the maximal subgroup of O(d) that preserves native coordinate identity without absorbing γ into adjacent weights. Repeated γ values can add rotations inside common equal-γ subspaces across RMSNorms sharing the residual chart; if any relevant gain vector is generic, the native orthogonal gauge reduces to Bd . Proof sketch. Linear maps admit any orthogonal residual gauge under READ/WRITE weight transforms. A READ map uses W G = W G, giving HG(W G )⊤ = HW ⊤ ; a WRITE map uses W G = G⊤ W , bG = bG, giving (XW ⊤ + b)G. Q, K, V projections are invariant; only the output projection writes the gauged contribution. Normalization restricts G. RMSNorm is norm-invariant, but its diagonal gain requires G⊤ diag(γ)G to remain diagonal; for generic γ, the orthogonal solutions are exactly signed permutations. LayerNorm is permutation-invariant but excludes independent coordinate signs; the remaining signed elements are −P , with β 7→ −βP (Lemma B.3). Full module rules and induction over blocks are in Appendices B and F. The native-vs.-absorbed distinction is the main boundary with recent orthogonal methods. SliceGPT and QuaRot exploit the identity RMSNorm(XQ)Q⊤ = RMSNorm(X) after moving gains into neighboring matrices [Ashkboos et al., 2024a,b]; generalized linear mode connectivity and rotationbased fusion use related O(d) structure [Theus et al., 2025, Zhang et al., 2025]. These methods are complementary. They are appropriate when a dense rotated basis is acceptable. Transporting a top-k coordinate set, an SAE dictionary attached to residual coordinates, an AdamW diagonal state, or an index-level claim requires the native coordinate-preserving subgroup, which is Bd (Proposition 6.1 and Remark 4.3). Theorem 2.2 (Discrete content of an orthogonal map). For any Q ∈ O(d), define P ⋆ = arg min ∥Q − P ∥F . P ∈Bd

Generically (outside a Lebesgue-null set of Q where Hungarian ties or matched zeros occur), P ⋆ is obtained by Hungarian assignment on |Q|: choose X σ ⋆ = arg max |Qσ(j),j |, σ∈Sd

j

then set the sign of column j to sign(Qσ⋆ (j),j ). The residual R = Q(P ⋆ )−1 ∈ O(d) captures continuous drift, and ρ(Q) = d−1 tr(Q⊤ P ⋆ ) equals one iff Q ∈ Bd . Theorem 2.2 separates the two alignment objects. The Bd component is a discrete scaffold that preserves coordinate identity; the residual R is a dense basis change. In limited-data settings this matters: estimating an arbitrary dense R requires paired activation information at scale, while the discrete component of a known orthogonal map is recovered from |Q| alone. Proof is in Appendix E. Proposition 2.3 (Gauge-covariance of learned weight updates). Let θ be a transformer parameterization satisfying the assumptions of Theorem 2.1, and let G ∈ G be an admissible discrete gauge. Write TG for the modulewise READ/WRITE/RMSNorm transform of Theorem 2.1 acting on weights. Then TG is linear: for any weight delta ∆θ on the same parameter set, however produced, TG (θ + ∆θ) = TG (θ) + TGlin (∆θ), where TGlin applies the same modulewise READ/WRITE rules to ∆θ as a free linear map. In particular, for a low-rank LoRA factor ∆W = BA on a residual-stream projection, WRITE: AG = A, B G = G⊤ B.

READ: AG = AG, B G = B; 3

Table 1: Bd transport rules for coordinate-indexed artifacts. G ∈ Bd acts on residual coordinates from the right (H 7→ HG); READ matrices read from the residual, WRITE matrices write into it. In every row, Sd preserves the artifact’s support but loses orientation by Lemma 2.5. Artifact

Correct Bd transport

What Sd misses

Steering vector v Cached residual/patch h Linear probe direction w SAE encoder Wenc SAE decoder Wdec LoRA READ ∆W = BA LoRA WRITE ∆W = BA AdamW/SGDm first moment mW AdamW second moment vW

v 7→ vG h 7→ hG w 7→ wG Wenc 7→ Wenc G Wdec 7→ G⊤ Wdec A 7→ AG, B 7→ B A 7→ A, B 7→ G⊤ B mW transforms with W (per role) vW 7→ T|G| (vW ) (perm only; signs square out)

sign orientation cached-state orientation decision-boundary orientation feature-read orientation reconstruction orientation input-basis signs output-basis signs first-moment orientation none—v is sign-blind by construction

Corollary 2.4 (Sign-omission destroys learned updates). For an update ∆W on a coordinate-indexed module under RMSNorm gauge G = P S with permutation P and signs S = diag(s), sj ∈ {±1}, the squared Frobenius update error of Sd -only transport (correct permutation, signs forced to +1) is X 2 2 2 ∥∆W P S − ∆W P ∥F = 4 ∆W:,P −1 (j) 2 , ES [·] = 2 ∥∆W ∥F . j:sj =−1

For random signs, the expected error equals twice the update’s own energy. Empirical confirmation. In our Qwen2.5-1.5B SST-2 LoRA experiment (Table 5), Sd -permonly transport gives accuracy 0.523 versus the unadapted base 0.525–i.e., a −0.5% adapter-gain preservation, consistent with the sign-omission collapse predicted by Corollary 2.4. The signed transport recovers 0.944, matching the source adapter (0.945) and the upper-bound reference (0.944). Lemma 2.5 (Artifact orientation). Let G = P diag(s) ∈ Bd with s ∈ {±1}d . For any residualaligned vector artifact v ∈ Rd –a steering vector, cached activation, linear-probe direction, SAE encoder row, or LoRA READ-side row–the correct Bd transport is vG, while permutation-only (Sd ) transport is vP (WRITE-side matrices and SAE decoder columns transform via G⊤ ; see Table 1). These satisfy P 2 X j sj (vP )j 2 2 ∥vG − vP ∥2 = 4 (vP )j , cos ∠(vG, vP ) = P . 2 j (vP )j j:s =−1 j

For balanced signs and a vP not concentrated on a few coordinates, the cosine is near zero: Sd preserves the support of v but loses its orientation. Lemma 2.5 is the vector specialization of Corollary 2.4 and unifies all Bd -specific tool-transfer outcomes in this paper: each artifact in Table 1 has a coordinate-indexed orientation that Sd alignment cannot represent, and each gauge-covariance row in Table 5 is one instance of the same orientation law. Gauge-invariance validation. We verify Theorem 2.1 by applying random gauges and measuring max logit deviation on held-out prompts or images. LayerNorm models pass permutation gauges and fail independent sign-flip gauges; RMSNorm/T5LayerNorm models pass full signed gauges. Biases must transform as WRITE vectors. Table 2 reports cross-architecture checks; Figure 1 in the appendix gives the corresponding off-group validation cases. The audit does not invalidate the behavior-level conclusions of these analyses. It separates invariant claims from coordinate claims: the refusal direction still changes refusal behavior, and the steering vectors still steer, but the coordinate names attached to those objects are not invariant artifacts until a gauge is specified.

3

Sign-marginalized gauge recovery

Given paired probe activations Hs , Ht ∈ Rn×d , we want a recovery action matrix G ∈ Bd such that Ht G ≈ Hs . Center columns and set C = Hs⊤ Ht . For a correspondence π and signs si , minimizing X 2 ∥Hs [:, i] − si Ht [:, π(i)]∥2 i

4

Table 2: Gauge invariance validation. Errors on the order of 10−4 are numerical tolerance. Model

Norm

Gauge

Max logit error

Qwen2.5-7B Llama-3.1-8B TinyLlama-1.1B BERT-base T5-small ViT-B/16

RMSNorm RMSNorm RMSNorm LayerNorm T5LayerNorm LayerNorm

Bd Bd Bd Sd Bd Sd

1.1 × 10−4 3.0 × 10−5 8.6 × 10−5 2.3 × 10−5 7.6 × 10−5 4.3 × 10−6

is equivalent to maximizing problem reduces to

P

i si Ci,π(i) . For fixed π, the optimal sign is si = sign(Ci,π(i) ), so the

π ⋆ = arg max

X

π∈Sd

|Ci,π(i) |,

s⋆i = sign(Ci,π⋆ (i) ).

i

This is a standard linear assignment on −|C|. We solve it with SciPy’s linear_sum_assignment routine, a modified Jonker-Volgenant implementation; the assignment problem and shortestaugmenting-path algorithms are classical [Kuhn, 1955, Jonker and Volgenant, 1987, Crouse, 2016, Virtanen et al., 2020]. Permutation/sign convention. Write Pπ for the row-action permutation with (Pπ )i,π(i) = 1, so (hPπ )π(i) = hi . Matching returns source-indexed signs si = sign(Ci,π(i) ) and Ss = diag(si ). The forward gauge is Gfwd = Ss Pπ , so Ht ≈ Hs Gfwd ; the recovery action is Grec = Pπ⊤ Ss = G−1 fwd , so Ht Grec ≈ Hs . Equivalently, with target-indexed St = Pπ⊤ Ss Pπ , Gfwd = Pπ St ; later formulas writing G = P diag(s) use target-indexed signs. This section reports recovery matrices; Section 4 switches to forward-direction subscripts. Theorem 3.1 (Structural ceiling in the decorrelated-coordinate case). Suppose two activation matrices are related by a true signed permutation (π ⋆ , s⋆ ), and source coordinates are uncorrelated with positive variances. Let X X π̂signed = arg max Ci,π(i) , π̂abs = arg max |Ci,π(i) |. π

π

i

i

As N → ∞, π̂signed ’s accuracy equals the positive-sign fraction except in the one-negative-sign case; under iid signs its expectation is 1/2 + O(d2−d ). The sign-marginalized estimator π̂abs converges to the true permutation under the assignment-margin condition of Lemma G.1. The theorem explains why permutation-only RMSNorm alignment can fail even with unlimited probes. If the true sign is negative, the correct entry is the row’s most negative correlation; signedcorrelation matching is designed to avoid it. Correlated residual coordinates can change the exact ceiling, but not the sign-marginalization requirement: under a negative gauge sign, the true coordinate is anti-correlated with its source coordinate, so signed-correlation matching is optimizing the wrong objective. In recovery experiments where a recorded Bd basis change provides the target correspondence, naive Hungarian on signed cost recovers only 49.4%–49.9% of the permutation and about 50.1% of signs on 7B/8B RMSNorm models, matching the predicted sign obstruction; sign-marginalized matching recovers 100% on the same rows (Table 15). The same recovery succeeds with as few as 22 probe tokens on TinyLlama (Appendix H.2). Gradient-times-activation attribution is also restored: signs cancel in (hG) ⊙ ((∇h L)G) = (h ⊙ ∇h L)P , so the recovered permutation maps top-k attribution indices back exactly. Probe-budget regimes. The experiments use three regimes, summarized in Table 3: settings with a recorded gauge correspondence are probe-robust; same-base fine-tuning trajectory transport works with small local probes because consecutive checkpoints have high assignment margins; natural alignment between independently trained or differently trained models is probe-hungry and is reported as stability against a large-probe reference rather than as exact accuracy. 5

Table 3: Probe budget by result family. The ∼500-token protocol is the default unless a larger natural-alignment run is explicitly stated. Result family

Probe budget

Setting

Gauge-instrumented recovery Gauge-scramble merge Transport trajectory recovery Qwen Bd vs. Sd steering TinyLlama SAE/steering transfer Independent-seed RMSNorm merge Natural cross-model stability

22–500 tokens ∼500 tokens ∼500 tokens/local match ∼500 tokens 11,134 tokens ∼500 tokens ∼36K tokens (saturated)

known reference basis basis change + 50-step fine-tune same-base fine-tunes same-base fine-tunes basis-change tool transfer same data, different seeds no reference basis

Table 4: Cross-run residual-stream recovery on Qwen2.5-1.5B. Numbers are permutation recovery percentages, mean ± s.d.; combined permutation+sign accuracy differs by at most 0.13%. Transport composes local Bd gauges along the trajectory; endpoint is one final-to-final match. Both methods use the same probe set for each estimated edge; transport uses more saved checkpoints, so this is a low-probe local-composition test rather than an equal-total-query comparison. Pair type Cross-seed (9 pairs) Cross-dataset (9 pairs) Overall (18 pairs) Endpoint-via-base Hybrid via base Transport-via-base

4

200 steps Endpoint Transport 31.7 ± 13.3 90.9 ± 15.7 61.3 ± 33.6

96.2 ± 6.6 96.2 ± 6.4 96.2 ± 6.3

17.1 ± 6.0 37.6 ± 11.1 96.2 ± 6.3

1500 steps Endpoint Transport 43.9 ± 18.6 76.8 ± 36.1 60.3 ± 32.6

91.0 ± 8.4 91.2 ± 9.6 91.1 ± 8.7

11.4 ± 3.6 26.7 ± 6.0 91.1 ± 8.7

Parallel transport and tool transfer

Endpoint matching estimates one gauge; transport composes many local ones. For a trajectory t0 , t1 , . . . , tT , estimate Gtk →tk+1 by sign-marginalized matching with Htk Gtk →tk+1 ≈ Htk+1 , then set G0→T = Gt0 →t1 Gt1 →t2 · · · GtT −1 →tT . Within this section we use the forward-direction subscript Ga→b with Ha Ga→b ≈ Hb ; the recovery action of Section 3 is its transpose. For two runs a, b with a shared base, compose through the base:  −1 (a) (b) Ĝa→b = G0→T G0→T . If each local match is correct, the Bd product is exact. We evaluate on a gauge-instrumented Qwen2.5-1.5B fine-tuning benchmark built from real 1500step trajectories. A randomly sampled Bd gauge is applied at step 0, and its induced coordinate correspondence is recorded as ground truth; subsequent checkpoints come from real fine-tuning updates. Checkpoints are spaced every two optimizer steps, so a 1500-step base-to-final map composes 750 local matches; a cross-run transport-via-base map composes two such chains with the same fixed probe set reused on every edge. At 1500 steps, transport-via-base recovers 91.1% ± 8.7% of residual coordinates across 18 run pairs, versus 60.3% ± 32.6% for direct endpoint matching (Table 4). The cross-seed subset is hardest: endpoint matching falls to 43.9%, while transport recovers 91.0%. Cross-dataset pairs are easier for endpoint matching because the seed is shared, but transport still improves the mean. Per-pair results show consistently positive cross-seed gains, while cross-dataset pairs can favor endpoint matching when endpoint matching is already near-ceiling (Figure 2). The lower panel answers a key ablation: the transport gain is not obtained by simply referencing the base. Composing two endpoint maps through the base gives 11.4% at 1500 steps, worse than direct endpoint matching. The improvement appears only when both base-to-final maps are themselves transported by local composition. Additional Llama-3.2-1B and FFN results are in Appendix J; FFN transport is less stable over long horizons, which we discuss as a limitation. Two facts pin down why same-base trajectories are the natural setting. 6

Table 5: Learned, coordinate-indexed artifacts are gauge-covariant. In same-base and exact basischange tests on RMSNorm models, Sd (permutation-only) transport fails by Corollary 2.4; Bd (signed-permutation) recovery matches the reference upper bound to within evaluation noise. Qwen steering rows are a single WikiText cross-seed evaluation pair, detailed in Table 21. Artifact

Setting

Raw

Sd

Bd

Coordinate identity Steering (sentiment) Steering (refusal) SAE reconstruction NMSE LoRA adapter acc. LoRA gain preserved LoRA adapter acc. LoRA gain preserved

Qwen2.5 same-base Qwen WikiText cross-seed Qwen WikiText cross-seed TinyLlama Bd basis Qwen SST-2 Bd basis Qwen SST-2 Bd basis TinyLlama Bd basis TinyLlama Bd basis

60.3% ep. – – – 0.622 23.0% 0.770 14.3%

– 17.2% −32.2% effect 1.08 0.523 −0.5% 0.795 26.2%

91.1% tr. 95.8% +150.5% effect 0.004 0.944 99.7% 0.950 100.0%

Proposition 4.1 (No canonical function-level transport between independent endpoints). Let G be a nontrivial gauge group acting on parameter space Θ with θ ∼ θ′ iff f (·; θ) = f (·; θ′ ). No map A : Θ × Θ → G can simultaneously satisfy (i) function-level factoring, A(θ1 , θ2 ) = A(θ1 , θ2 H) for all H ∈ G, and (ii) gauge-equivariance, A(θ1 , θ2 H) = A(θ1 , θ2 )H −1 , since the two would force H = I for every H ∈ G. Hence no canonical coordinate transport between independently specified parameterizations exists from function-level information alone. Proposition 4.2 (Atlas consistency iff zero holonomy). Let V be a finite set of model checkpoints and E a connected set of edges, each labeled Ĝij ∈ Bd . There exist gauge choices Ri ∈ Bd such that Ĝij = Ri−1 Rj for every edge if and only if every cycle has identity holonomy: Ĝi1 i2 · · · Ĝik i1 = I. A collection of pairwise gauges {Ĝij } ⊂ Bd is a transport system only when it factors through per-checkpoint gauges: there must exist Ai ∈ Bd such that Ĝij = A−1 i Aj for every pair, equivalently Ĝii = I, Ĝji = Ĝ−1 Ĝ Ĝ = Ĝ , and on every triple. By Proposition 4.2, zero cycle holonomy is ij jk ik ij exactly this cocycle condition. Same-base trajectory transport supplies such an atlas by construction: the shared base is a function-level anchor that breaks the gauge ambiguity in Proposition 4.1, and consecutive-checkpoint composition along a trajectory traces a tree, on which the cocycle holds trivially. For independently specified endpoints, no atlas is defined by the function-level data; pairwise high-scoring matches alone are therefore not transport, only candidate pieces of an atlas, and become one only if they pass cycle consistency. 4.1

Learned coordinate-indexed artifacts are gauge-covariant

Proposition 2.3 predicts that any learned weight update–LoRA factors, full task vectors, steering directions, SAE dictionaries–transforms by the same READ/WRITE rules as the base weights. Corol2 lary 2.4 predicts Sd -only transport introduces update error of magnitude 2 ∥∆W ∥F in expectation, collapsing transferred adapters to (or below) the unadapted base. Table 5 confirms the unified law across four artifact classes (real same-base trajectories and exact basis-change tests), spanning coordinate transport, steering vectors, SAE dictionaries, and LoRA adapters. The Qwen2.5-1.5B LoRA row is the sharpest empirical instance of the corollary: under Sd transport the adapter accuracy falls to 0.523, indistinguishable from the unadapted base (0.525) and below the raw transfer baseline (0.622). Under Bd recovery the same adapter retains 99.7% of its gain, matching the upper-bound reference. All conditions use the same trained adapter precision, so the gap is not a dtype comparison; it is the sign-omission failure predicted by Corollary 2.4 for a learned coordinate-indexed update under an exact RMSNorm reparameterization. The TinyLlama SAE row exhibits the same failure at the level of an entire dictionary: Sd recovery yields reconstruction NMSE 1.08, while Bd recovery matches the upper-bound reference at 0.004. Across these four artifact-class tests the pattern is identical: Bd preserves what Sd destroys. 7

Table 6: Stateful resumption: optimizer state is gauge-covariant. Held-out logit relative MSE between a gauge-equivalent reference resume and two optimizer-state transports. The pre-resume checkpoint is already gauge-equivalent, so the post-resume gap measures state transport. Model/opt. Qwen2.5-0.5B/AdamW

4.2

Setup

Bd state

Pre

fp32, b = 32, 500+500

5.24 × 10

−12

Sd state −10

5.30 × 10

Sd /Bd −3

3.50 × 10

6.6 × 106

Stateful resumption: optimizer state is gauge-covariant

The same orientation law (Lemma 2.5) governs stateful training: optimizer state is part of the checkpoint’s coordinate-dependent representation. AdamW stores a first moment mW (an exponential moving average of gradients) and a second moment vW (an EMA of gradient squares). Under an exact Bd reparameterization G = P diag(s) of the residual coordinates, the gradient transforms with W , so mW inherits the same READ/WRITE/NORM rules; vW is the elementwise square of an inherited tensor, so its sign action squares out: vW 7→ T|G| (vW ) = TP (vW ). Remark 4.3√(Bd is the maximal gauge for native AdamW). AdamW’s elementwise update θ 7→ θ − ηmW /( vW + ϵ) presupposes diagonal storage of the second moment. For G ∈ Bd , the gauge action preserves diagonality because G diag(vW )G⊤ = diag(vW P ), so an elementwise vW tensor remains a valid representation. For continuous orthogonal Q ∈ O(d) \ Bd , Q diag(vW )Q⊤ is dense; AdamW’s diagonal storage cannot represent it without information loss. Hence Bd is the maximal subgroup of O(d) under which AdamW state is gauge-covariant given native data structures. We test this mechanism by warming AdamW on a real Qwen2.5-0.5B fine-tuning task, saving the model and optimizer state, applying a random Bd gauge to the model, and resuming from alternative transported optimizer states. Bd transport applies TG to mW and T|G| to vW ; Sd -only transport applies the correct permutation but forces all signs to +1. Each resumed run is compared to a gauged copy of the reference resume from the original chart. The checkpoint is already gauge-equivalent before resumption, so the post-resume difference isolates optimizer state rather than model mismatch. Table 6 reports the long-horizon fp32 run. After 500 warmup steps and 500 resumed steps, Bd preserves the reference trajectory to numerical tolerance, whereas Sd changes it by a factor of 6.6 × 106 in held-out logit relative MSE. This is a trajectory-identity result, not a loss-improvement claim: the final losses can be similar, but permutation-only optimizer state no longer represents the same point in the stateful training process. Additional AdamW and SGD-momentum resumption variants are reported in Appendix K. Scope. The random Bd gauge is applied to test the orientation law, and the trainable scope is restricted to keep optimizer state tractable. The result should be read as a stateful transport test: even when two checkpoints are functionally identical, resuming with permutation-only optimizer state changes the fp32 training trajectory.

5

Coordinate-preserving merging

We evaluate merging in the coordinate-preserving setting: interpolation compares or averages corresponding coordinates, so RMSNorm signs are part of the correspondence. This is the setting where the discrete gauge is the relevant object; arbitrary function-level merging between divergent models can depend on dense changes beyond a signed permutation. We use peak-above-chord barriers: for interpolation loss L(α) with endpoint chord (1 − α)L(0) + αL(1), the barrier is max L(α) − [(1 − α)L(0) + αL(1)]. α

This measures interior excess above the endpoint chord, avoiding monotone endpoint-gap artifacts. In the Llama-2-7B experiment, we reparameterize the base model by a random Bd basis change, fine-tune the rebased copy for 50 steps on code, and interpolate weights between the original base and the aligned endpoint. The basis change is recorded by construction, so the experiment isolates whether the alignment respects the symmetry: a permutation-only alignment applies the correct permutation but forces all signs to +1, leaving destructive sign mismatch; signed alignment eliminates 8

Table 7: Coordinate-preserving merge results. Across all three settings, signs are the variable that permutation-only alignment cannot represent. Setting

Unaligned

Sd perm-only

Bd signed

6.14 1.68 6.36

7.69 0.66 6.47

0.00 0.42 0.00

Llama-2-7B gauge-scramble, 50-step FT 10M RMSNorm, independent seeds, same data TinyLlama-1.1B, 0-step gauge-equivalent merge

Table 8: Gauge audits for coordinate-indexed artifacts. Each row applies an exact function-preserving reparameterization in the native gauge for the named coordinate system. Raw coordinate names move; mapped coordinates recover the original object. Pipeline

Invariant check

Raw coordinates

Gauge-fixed

BERT knowledge neurons [Dai et al., 2022] Llama-2 refusal direction [Arditi et al., 2024] CAA steering vectors [Rimsky et al., 2024] Gradient×activation attribution

same FFN function ablation 100% → 15% effect error < 10−4 , CKA=1 logits invariant

0/2 neurons median 0/10 top coords. ≈0/10 top coords. 0/10 top attributions

2/2 mapped 10/10, 100% gauge rec. mapped effect preserved 10/10 mapped

the barrier. The 10M row is a small-scale natural independent-seed sanity check; the 7B and 1.1B rows are exact-symmetry tests that isolate the sign variable at production and mid scales. In the 10M independent-seed experiment with no basis change, two RMSNorm transformers trained from different seeds develop about 50% sign mismatch without an imposed gauge, and Bd alignment gives a 75% barrier reduction versus unaligned and a 37% relative improvement over Sd .

6

Gauge audits expose coordinate non-reproducibility

This section is not a claim that coordinates are always the right units; it is an audit for pipelines that output coordinate names. Index-level interpretability claims name a residual dimension, an MLP hidden unit, a top-k set, or an SAE feature index. Such claims are not wrong, but they are relative to a gauge. The audit is simple: apply an exact gauge from the architecture’s symmetry group, check that behavior or invariant quantities are unchanged, and then compare both raw and gauge-mapped coordinate names. This is a gauge-equivariance test analogous to verifying that a saliency method is not sensitive to arbitrary label choices. Table 8 shows the pattern across existing coordinate-indexed workflows. The represented object is unchanged, but raw coordinate names are not reproducible; once the gauge is applied, the same object is recovered. These audits do not dispute behavior-level conclusions; they separate invariant objects from gauge-relative coordinate names. Proposition 6.1 (Sparse coordinate interventions force Bd ). For Q ∈ O(d), the following are equivalent: (i) for every coordinate projector PS , QPS Q⊤ is another coordinate projector; (ii) every fixed-budget sparse coordinate intervention transports to another sparse coordinate intervention with the same budget; (iii) Q ∈ Bd . Proof sketch. Signed permutations relabel coordinate projectors. Conversely, apply the condition to singleton projectors ei e⊤ i : each Qei must be ±ej , and orthogonality forces distinct targets, so Q is a signed permutation. Proposition 6.1 formalizes why dense orthogonal maps are not substitutes for Bd when the object is a sparse coordinate operation: a dense rotation preserves a function after reparameterization but turns a top-k coordinate edit into a dense rank-k projection. Concrete interpretability examples are in Appendix N; limitations and broader impacts are in Appendix O.

7

Conclusion

In the native generic-gain coordinate-preserving setting, RMSNorm transformers are coordinateidentifiable up to signed permutation, not permutation alone (Theorem 2.1); the missing sign compo2 nent is a quantitative prediction (Corollary 2.4: squared error 2 ∥∆W ∥F ) confirmed on a Qwen2.51.5B SST-2 LoRA: Sd collapses accuracy to the unadapted base (0.523), Bd recovers the source 9

adapter (0.944). Exact gauge audits turn this into a reproducibility test: under random gauge, raw indices move while behavior and mapped coordinates stay fixed. For sparse coordinate-indexed objects–SAEs, steering vectors, neuron sets, attribution indices, LoRA adapters, or coordinatepreserving merges–signs are part of the identity map; “Which neuron is this?” is well-posed once Bd is named.

References Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023. URL https: //openreview.net/forum?id=CQsmMYmlP5T. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https://arxiv.org/abs/2406.11717. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In International Conference on Learning Representations (ICLR), 2024a. URL https://arxiv.org/abs/2401.15024. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, volume 37, 2024b. doi: 10.52202/079017-3180. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html. Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv:1907.02911, 2019. URL https://arxiv.org/abs/1907.02911. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/ 2023/monosemantic-features/index.html. David F. Crouse. On implementing 2d rectangular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679–1696, 2016. doi: 10.1109/TAES.2016.140952. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 8493–8502, 2022. doi: 10.18653/v1/2022.acl-long.581. URL https://aclanthology.org/2022. acl-long.581/. Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1309–1318, 2018. URL https://proceedings.mlr. press/v80/draxler18a.html. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031, 2021. doi: 10.1162/tacl_a_00410. URL https: //aclanthology.org/2021.tacl-1.60/. Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/privileged-basis/ index.html. Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=dNigytemkL. C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In International Conference on Learning Representations, 2017. doi: 10.48550/arXiv.1611.01540. URL https://arxiv.org/abs/1611.01540. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations (ICLR), Oral, 2025. doi: 10.48550/arXiv.2406.04093. URL https://openreview.net/ forum?id=tcsZt9ZNKD.

10

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems, 2018. doi: 10.48550/arXiv.1802.10026. URL https://arxiv.org/abs/1802.10026. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj. Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. In International Conference on Learning Representations, 2024. doi: 10.48550/arXiv.2310.05719. URL https://arxiv.org/abs/2310.05719. Roy Jonker and Anton Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38(4):325–340, 1987. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519–3529, 2019. Harold W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2):83–97, 1955. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1511.07543. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018. Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. In International Conference on Learning Representations (ICLR), Poster, 2026. URL https://openreview. net/forum?id=EjInprGpk9. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in Neural Information Processing Systems, 30, 2017. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.828. URL https://aclanthology.org/2024.acl-long.828/. Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ fb2697869f56484404c8ceee2985b01d-Abstract.html. George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. ZipIt! merging models from different tasks without training. In International Conference on Learning Representations, 2024. URL https://iclr.cc/virtual/2024/poster/18869. N. Joseph Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. In Advances in Neural Information Processing Systems, volume 33, pages 15300–15311, 2020. Alexander Theus, Alessandro Cabodi, Sotiris Anagnostidis, Antonio Orvieto, Sidak Pal Singh, and Valentina Boeva. Generalized Linear Mode Connectivity for Transformers. In Advances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=KurYdcCbjv. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/abs/2307.09288. Neha Verma and Maha Elbayad. Merging text transformer models from different initializations. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=nWnYSLncXa. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa,

11

Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17:261–272, 2020. doi: 10.1038/s41592-019-0686-2. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/. Mitchell Wortsman, Gabriel Ilharco, Samir Y. Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23965–23998. PMLR, 2022. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, 2023. doi: 10.48550/arXiv.2306.01708. URL https://arxiv.org/abs/2306.01708. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen2.5 technical report, 2024a. URL https://arxiv.org/abs/2412.15115. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in LLMs, MLLMs, and beyond: Methods, theories, applications and opportunities. arXiv preprint, 2024b. URL https://arxiv.org/abs/2408.07666. Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, and Jundong Li. Beyond the permutation symmetry of transformers: The role of rotation for model fusion. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 77090–77106. PMLR, 2025. URL https://proceedings.mlr.press/v267/zhang25dk.html.

A

Additional symmetry-boundary figures

Figure 1: Symmetry-boundary validation for the native gauge. LayerNorm models are invariant to permutations but fail under independent sign flips; biases must transform under the residual gauge; RMSNorm models pass full signed-permutation gauges. Dashed lines indicate the ∼ 10−4 numerical tolerance threshold.

B

Proof of Theorem 2.1

Roadmap: we establish module-level gauge transform rules (norms, linear layers, attention), then compose them via residual addition and a layerwise induction to prove Theorem 2.1. We present the decoder-only case for concreteness, using the row-vector convention: residual activations H ∈ Rn×d are matrices with tokens as rows, and the gauge acts by right multiplication H G = HG. Definition B.1 (Gauge action). For permutation gauge G = P ∈ Sd , P is a permutation matrix with Pi,π(i) = 1. For signed permutation gauge G = P S ∈ Bd , additionally S = diag(s1 , . . . , sd ) with si ∈ {−1, +1}. In both cases, the gauge action on residual activations H ∈ Rn×d is H G := HG. 12

Lemma B.2 (RMSNorm equivariance). Define h ⊙γ RMSNorm(h; γ) = q P 1 2+ϵ h i i d for row vector h. With γjG := γπ−1 (j) (permute only, no sign), we have RMSNorm(hG; γ G ) = RMSNorm(h; γ) · G. P P Proof. Mean square is invariant under signed permutations: d1 i (hG)2i = d1 i h2i . Thus RMS(hG) = RMS(h), so RMSNorm(hG; γ G ) = hG/ RMS(h) ⊙ γ G . The permuted γ G aligns elementwise scaling with the permuted coordinates, giving (h/ RMS(h) ⊙ γ)G. Why γ permutes but does not flip signs. The sign flip is absorbed into the activation h, not γ. Specifically, (hG)j = sj hπ−1 (j) carries the sign, while γjG = γπ−1 (j) only permutes. The elementwise product (hG ⊙ γ G )j = sj hπ−1 (j) γπ−1 (j) = sj (h ⊙ γ)π−1 (j) = ((h ⊙ γ)G)j . Lemma B.3 (LayerNorm equivariance). Define h − µ(h) 1X + β, µ(h) = hi . LayerNorm(h; γ, β) = γ ⊙ σ(h) d i For permutation P ∈ Sd with γjP = γπ−1 (j) and βjP = βπ−1 (j) , LayerNorm(hP ; γ P , β P ) = LayerNorm(h; γ, β) · P. More generally, ϵP with ϵ ∈ {±1} is equivariant if β P is replaced by ϵβ P . Signed permutations with nonconstant sign pattern are not equivariant in general. P P Proof. For permutation P , µ(hP ) = d1 j (hP )j = d1 j hπ−1 (j) = µ(h). Thus (hP −µ(hP )) = (h − µ(h))P and LayerNorm is Sd -equivariant. For ϵ = −1, µ(−hP ) = −µ(h) and the normalized centered vector changes by the same global sign, giving LayerNorm(−hP ; γ P , −β P ) = LayerNorm(h; γ, β)(−P ). P For signed permutation with nonconstant sign flips, µ(hG) = d1 j sj hπ−1 (j) ̸= ±µ(h) in general. Thus (hG − µ(hG)) ̸= (h − µ(h))G, breaking equivariance. Lemma B.4 (Post-norm wrapper). Let G be an admissible gauge. Suppose ∆ is gauge-equivariant, ∆G (HG) = ∆(H)G, and N is gauge-equivariant, N G (HG) = N (H)G. Then Φ(H) = N (H + ∆(H)) is gauge-equivariant: ΦG (HG) = Φ(H)G. Proof. ΦG (HG) = N G (HG + ∆G (HG)) = N G ((H + ∆(H))G) = N (H + ∆(H))G. Lemma B.5 (Linear layer transforms under HuggingFace convention). HuggingFace Linear computes Y = XW ⊤ + b with W ∈ Rdout ×din . Under gauge X G = XG: • READ (input from residual, output unchanged): set W G = W G, bG = b. Then Y G = X G (W G )⊤ + bG = XGG⊤ W ⊤ + b = XW ⊤ + b = Y . • WRITE (output to residual): set W G = G⊤ W , bG = bG. Then Y G = X(W G )⊤ + bG = XW ⊤ G + bG = (XW ⊤ + b)G = Y G. Lemma B.6 (Attention invariance). With Q, K, V projections as READ layers, the projected queries, keys, and values √ are unchanged under the gauge reparameterization. Therefore attention weights softmax(QK ⊤ / dk ) are identical, and only the output projection (WRITE) produces a gauged residual contribution. Proof. Let H G = HG be the gauged input to attention. Q-projection is READ: Q = HWQ⊤ + bQ becomes QG = H G (WQG )⊤ + bQ = HGG⊤ WQ⊤ + bQ = HWQ⊤ + bQ = Q. Similarly K G = K and V G = V . Q/K/V biases are in head space, not residual space, so they remain unchanged under gauge. Therefore attention weights are unchanged. The O-projection is WRITE, producing ∆G = ∆G. 13

Proof of Theorem 2.1. By induction on layers. Let HℓG = Hℓ G at layer ℓ. Base case: token G embeddings E = Wemb [tokens] are row vectors. With Wemb = Wemb G, we have E G = EG = G H0 G. For the inductive step, given Hℓ = Hℓ G, let ∆ be the attention or MLP update. If the block is pre-norm, H ′ = Hℓ + ∆(N (Hℓ )); Lemma B.2 or B.3 and Lemma B.6 (or Lemma B.5 for MLP) give H ′G = H ′ G. If post-norm, H ′ = N (Hℓ + ∆(Hℓ )); ∆G (HℓG ) = ∆(Hℓ )G and Lemma B.4 G yields H ′G = H ′ G. Applying this to attention and MLP gives Hℓ+1 = Hℓ+1 G. G For the output, the LM head is READ from the final residual. With Whead = Whead G, G ⊤ ⊤ logitsG = HLG (Whead )⊤ = HL GG⊤ Whead = HL Whead = logits . Tied embeddings: When Whead = Wemb (weight tying), both transform identically as W G = W G, preserving the tie. This is automatic since both are READ layers from/to the residual stream.

Two constraints determine the gauge group. For RMSNorm, the diagonal learned gain γ must remain diagonal under gauge. For a generic gain vector, or a jointly generic family of gain vectors sharing a residual chart, the simultaneous stabilizer in O(d) is exactly the signed permutation group Bd . Exact tied coordinates across all relevant gains can admit block-orthogonal stabilizers, but these do not preserve sparse coordinate identity; all results use the always-present Bd subgroup. For LayerNorm, mean-centering P breaks independent sign symmetry. For signed permutation G with signs si , the mean µ(xG) = d1 j sj xπ−1 (j) generally differs from ±µ(x) unless all signs are equal. Thus generic affine LayerNorm admits {±P : P ∈ Sd }, where the negative component requires β 7→ −βP ; its per-index relabeling content is Sd . Both groups are large: |Sd | = d! and |Bd | = d! · 2d . The factor of 2d distinguishes the architectures, but both imply neuron indices are non-identifiable without fixing a gauge. Remark B.7 (Generic γ and practical degeneracies). For RMSNorm RMSNorm(x) = x/ ∥x∥ ⊙ γ, the admissible gauge G must preserve diagonality under conjugation: G⊤ diag(γ)G remains diagonal (possibly permuted). For generic γ (distinct entries), the only orthogonal matrices with this property are signed permutations, i.e., Bd ; if γi = γj for some i ̸= j, a single RMSNorm layer also admits rotations within the equal-γ subspace. Exact repeats in learned γ can occur in real checkpoints, especially in low-precision formats such as bf16/fp16, which can enlarge the admissible symmetry beyond Bd for a given RMSNorm layer. For a transformer-wide residual chart, the continuous part is the intersection of these per-layer stabilizers across all RMSNorms sharing the chart. We do not rely on this enlargement; all results use the always-present discrete subgroup Bd . Algorithm behavior with continuous symmetry: Our activation-based alignment (§3) selects one representative from the equivalence class—specifically, the discrete signed permutation that maximizes activation cross-correlation. Within equal-γ subspaces where continuous rotations are valid, this representative is determined by the probe activations and assignment tie-breaking. This choice is reproducible for a fixed probe set even when the underlying symmetry group is larger than Bd . Similarly, for LayerNorm with repeated (γi , βi ) pairs, additional permutations within tied subspaces are valid.

C

Proof of atlas consistency

Proof of Proposition 4.2.

If Ĝij = Ri−1 Rj , then every cycle product telescopes:

Ĝi1 i2 Ĝi2 i3 · · · Ĝik i1 = Ri−1 Ri2 Ri−1 · · · Ri−1 Ri1 = I. 1 2 k Conversely, fix a root r. For any vertex i, choose a path from r to i and define Ri as the ordered product of edge labels along that path. Zero holonomy makes Ri path-independent: two paths from r to i form a cycle after reversing one path, so their products agree. For an edge (i, j), the path to i followed by that edge is a path to j; path-independence therefore gives Rj = Ri Ĝij , hence Ĝij = Ri−1 Rj .

D

Proofs for artifact covariance

Proof of Proposition 2.3. The parameter map TG is modulewise multiplication by fixed matrices G, G⊤ , or by the corresponding permutation of norm gains. Hence it is affine in the base parameters 14

and linear on increments:

TG (θ + ∆θ) − TG (θ) = TGlin (∆θ). For a LoRA update ∆W = BA, the READ rule is ∆W G = ∆W G = B(AG), so AG = AG, B G = B. The WRITE rule is ∆W G = G⊤ ∆W = (G⊤ B)A, so AG = A, B G = G⊤ B. Write G = P S, with S = diag(s). Let M = ∆W P . Then X X 2 2 2 2 ∥∆W P S − ∆W P ∥F = ∥M S − M ∥F = (sj − 1)2 ∥M:,j ∥2 = 4 ∥(∆W P ):,j ∥2 .

Proof of Corollary 2.4.

j

j:sj =−1

2

P

2

Since P is orthogonal, j ∥(∆W P ):,j ∥2 = ∥∆W ∥F . Under iid uniform signs, each column 2 contributes with probability 1/2, giving 2 ∥∆W ∥F in expectation. Proof of Lemma 2.5.

The norm identity is the vector version of the preceding proof with m = vP : X 2 2 ∥vG − vP ∥2 = ∥mS − m∥2 = 4 m2j . j:sj =−1

The cosine follows from ⟨vG, vP ⟩ = ⟨mS, m⟩ =

E

2 j sj mj and ∥vG∥2 = ∥vP ∥2 = ∥m∥2 .

P

Proofs for decomposition and matching ceiling

Proof of Theorem 2.2.

For any signed permutation P ∈ Bd , 2

2

2

∥Q − P ∥F = ∥Q∥F + ∥P ∥F − 2 tr(Q⊤ P ) = 2d − 2 tr(Q⊤ P ), so minimizing Frobenius distance is equivalent to maximizing tr(Q⊤ P ). Write P as a permutation σ plus signs ϵj ∈ {±1}, with column j mapped to row σ(j). Then X tr(Q⊤ P ) = ϵj Qσ(j),j . j

P For a fixed σ, the optimal sign is ϵj = sign(Qσ(j),j ), giving objective j |Qσ(j),j |. Maximizing this over permutations is exactly a linear assignment problem on |Q|. Generically the assignment has a unique maximizer; ties are the only source of non-uniqueness. Since both Q and P ⋆ are orthogonal, R = Q(P ⋆ )−1 is orthogonal. Finally, ∥R − I∥F = Q(P ⋆ )−1 − I F = ∥Q − P ⋆ ∥F by right-invariance of the Frobenius norm. The scalar ρ(Q) = d−1 tr(Q⊤ P ⋆ ) is one iff Q = P ⋆p ∈ Bd by the distance identity above. For Haar-random Q, the largest entry in each column is O(plog d/d) in expectation, and assignment can improve constants but not order, giving Eρ(Q) = O( log d/d). Proof of Theorem 3.1. Let the target columns satisfy vπ⋆ (i) = s⋆i ui in population, up to sampling noise, and assume E[ui uj ] = 0 for i ̸= j with variance σi2 > 0 on coordinate i. Assignment is b = N −1 H ⊤ Ht , which unchanged by positive rescaling, so work with the empirical covariance C s obeys bi,j −−−−→ s⋆i σi2 1{j = π ⋆ (i)}. C N →∞

All off-diagonal entries vanish in population. If s⋆i = +1, the correct entry is the unique positive entry in row i, so signed-correlation maximization can select it. If s⋆i = −1, the correct entry isPthe most negative population entry in its row, while all incorrect entries are zero; a maximizer of i Ci,π(i) therefore avoids that entry whenever an off-diagonal assignment is available. Hungarian’s global one-to-one constraint must displace some other row in exchange: when there are k ≥ 2 negative signs, swapping any two negative-sign rows yields strict gain σi21 + σi22 > 0 over the truth, so the maximizer strictly avoids all k negatives. The event k ≤ 1 has probability O(d2−d ) under iid uniform signs. Hence signed-correlation matching has limiting accuracy equal to the positive-sign fraction except on this event, giving expectation 1/2 + O(d2−d ) under independent uniform signs. 15

bi,π⋆ (i) | → σ 2 > 0 and |C bi,j | → 0 for j ̸= π ⋆ (i). Under standard sub-Gaussian By contrast, |C i p b−C = O( log d/N ); if this probe assumptions, concentration plus a union bound gives C ∞

perturbation is smaller than the assignment margin, Lemma G.1 preserves the true assignment. Thus sign-marginalized matching converges to the true permutation. E.1

FFN hidden unit symmetry

The residual gauge acts on the d-dimensional residual stream. MLP hidden units live in a separate dffn -dimensional space. We first show that FFN hidden activations are invariant under the residual gauge, then prove they admit their own independent permutation symmetry. Lemma E.1 (FFN hidden invariance under residual gauge). Under any admissible residual gauge G (i.e., G ∈ {±P : P ∈ Sd } for LayerNorm models or G ∈ Bd for RMSNorm models), the FFN hidden-layer activations are invariant, not just equivariant. That is, if H G = HG and weights transform as READ/WRITE (Table 9), then the dffn -dimensional hidden activations are identical: AG = A. ⊤ ⊤ Proof. For a plain 2-layer FFN, let ∆MLP (H) = σ(HWup + bup )Wdown + bdown under row-vector G G convention. Under the residual gauge, Wup is READ, Wup = Wup G, bup = bup ; Wdown is WRITE, G Wdown = G⊤ Wdown , bG down = bdown G. The preactivation into the hidden units is G ⊤ ⊤ ⊤ ⊤ ⊤ Z G = H G (Wup ) + bG up = (HG)(Wup G) + bup = HGG Wup + bup = HWup + bup = Z.

Since σ is elementwise, AG = σ(Z G ) = σ(Z) = A. The hidden activations are literally identical. ⊤ ⊤ ⊤ Gated MLP (Llama/SwiGLU). For ∆MLP (H) = (σ(HWgate + bgate ) ⊙ (HWup + bup ))Wdown + bdown : Both Wgate and Wup are READ layers, so the same GG⊤ = I cancellation applies to both branches. Thus both the gated and linear pre-activations are unchanged, their Hadamard product is unchanged, and the hidden activations are invariant.

Output equivariance. The down-projection is WRITE, so: G ⊤ G ⊤ ⊤ ⊤ ∆G MLP = A(Wdown ) + bdown = A(G Wdown ) + bdown G = AWdown G + bdown G = ∆MLP · G.

The FFN contribution is equivariant (picks up G on output), while internal hidden activations are invariant. Corollary E.2. Methods that output top-k FFN hidden unit indices (e.g., “knowledge neurons”) are gauge-variant under the FFN symmetry (Sdffn for plain FFNs, Bdffn for gated FFNs), but are not affected by the residual gauge G. Thus such methods require gauge-fixing for the FFN symmetry, which is independent of the residual-stream symmetry analyzed in our main theorems. Theorem E.3 (FFN permutation symmetry). For each FFN (MLP) layer with intermediate dimension dffn and elementwise activation σ, the permutation group Sdffn acts on the hidden unit indices while leaving the layer output invariant. Specifically, for P ∈ Sdffn , define ′ Wup = P ⊤ Wup ,

b′up = bup P,

′ Wdown = Wdown P,

b′down = bdown .

Then FFN′ (x) = FFN(x) for all inputs x. For gated FFNs (Llama/SwiGLU), the same permutation ′ P applies to both branches: Wgate = P ⊤ Wgate and b′gate = bgate P . For plain (non-gated) FFNs, sign flips are not a symmetry because common activations (GELU, SiLU, ReLU) are not odd functions. Gated FFNs admit additional sign symmetry; see Proposition E.4. ⊤ ⊤ Proof. Let FFN(x) = σ(xWup + bup )Wdown + bdown (row-vector convention), with Wup ∈ dffn ×dmodel dmodel ×dffn R and Wdown ∈ R . ⊤ Define the hidden activations h := σ(xWup + bup ) ∈ R1×dffn . With the transformed weights: ′ ⊤ ⊤ (Wup ) = (P ⊤ Wup )⊤ = Wup P

so the new preactivation is: ′ ⊤ ⊤ ⊤ x(Wup ) + b′up = xWup P + bup P = (xWup + bup )P.

16

Since σ is elementwise, it commutes with coordinate permutations: ⊤ ⊤ h′ = σ((xWup + bup )P ) = σ(xWup + bup )P = hP.

For the output: ′ ⊤ (Wdown )⊤ = (Wdown P )⊤ = P ⊤ Wdown

so: ′ ⊤ ⊤ y ′ = h′ (Wdown )⊤ + bdown = (hP )(P ⊤ Wdown ) + bdown = h(P P ⊤ )Wdown + bdown . ⊤ Since P is a permutation matrix, P P ⊤ = I, so y ′ = hWdown + bdown = y. Proposition E.4 (Signed-permutation subgroup of gated FFNs). For gated FFNs (SwiGLU/GLU as in Llama/Mistral/Qwen), Bdffn is an exact orthogonal hidden-coordinate symmetry subgroup, not merely Sdffn .

For P ∈ Sdffn and S = diag(s1 , . . . , sdffn ), define ′ Wgate = P ⊤ Wgate , ′ Wdown = Wdown P S,

b′gate = bgate P,

′ Wup = SP ⊤ Wup ,

b′up = bup P S,

b′down = bdown .

Then FFN′ (x) = FFN(x) for all x, and the post-gating hidden vector transforms as A′ (x) = A(x)P S. ⊤ ⊤ Proof. Let u(x) = xWup + bup (linear branch) and g(x) = xWgate + bgate (gate branch). Under the transform: ′ ⊤ u′ (x) = x(Wup ) + b′up

= x(SP ⊤ Wup )⊤ + bup P S ⊤ = (xWup + bup )P S = u(x)P S,

and g ′ (x) = g(x)P , hence σ(g ′ (x)) = σ(g(x))P since σ is elementwise. The post-gating hidden vector becomes: A′ (x) = σ(g ′ (x)) ⊙ u′ (x) = (σ(g(x))P ) ⊙ (u(x)P S) = (σ(g(x)) ⊙ u(x))P S = A(x)P S. For the output: ′ FFN′ (x) = A′ (x)(Wdown )⊤ + bdown

= A(x)P S(Wdown P S)⊤ + bdown ⊤ = A(x)Wdown + bdown = FFN(x),

since P S is orthogonal and (P S)(P S)⊤ = I. Remark E.5 (Additional diagonal scaling in gated FFNs). The signed-permutation subgroup is not the full parameter symmetry of a gated FFN. For any invertible diagonal D in hidden space, the ′ ′ transform Wup = DWup , b′up = bup D, Wdown = Wdown D−1 , with the gate branch unchanged, also preserves the FFN function. This larger scaling symmetry is non-orthogonal unless D has ±1 diagonal entries, so it is separate from the residual-stream coordinate-preserving group studied in the main text. Remark E.6 (Why Gated FFNs Admit Sign Flips). The key point is that the hidden-unit sign matrix acts on the linear branch u(x) (via Wup ), not inside the nonlinear gate σ(g(x)). Although σ (SiLU/GELU) is not odd, the sign acts on u(x) which is then multiplied by σ(g(x)). This is distinct from plain FFNs where the nonlinearity directly acts on the pre-activation. Remark E.7 (Independence of Symmetries). The residual gauge G ∈ Bd and FFN hidden transform act on orthogonal index sets and can be applied simultaneously. The residual gauge acts on the dmodel columns of Wup ; the FFN hidden transform acts on the dffn rows. For the signed hidden subgroup, write K = P S. The combined transform is: G,P Wgate = P ⊤ Wgate G,

G,K Wup = K ⊤ Wup G,

G,K Wdown = G⊤ Wdown K.

Biases transform as bgate 7→ bgate P , bup 7→ bup K, and bdown 7→ bdown G. These commute because G and K act on disjoint dimensions: G permutes/signs the d-dimensional residual space while K acts on the dffn -dimensional hidden space. 17

Remark E.8 (Why Plain (Non-Gated) FFNs Have No Sign Symmetry). For plain 2-layer MLPs of the ⊤ form h = σ(xWup + b), sign flips would require σ(−z) = −σ(z) (oddness). ReLU(x) = max(0, x) satisfies ReLU(−x) ̸= − ReLU(x) for x > 0; similarly GELU and SiLU are not odd. Thus for plain FFNs, sign flips do not preserve outputs, limiting the orthogonal hidden-coordinate subgroup to Sdffn . Plain ReLU FFNs also have the familiar positive diagonal scaling symmetry from ReLU homogeneity; GELU/SiLU plain FFNs generally do not. Gated FFNs (Proposition E.4) admit this because the sign matrix acts on the linear branch while the nonlinear gate branch is only permuted. E.2

SAE gauge invariance

Sparse autoencoders (SAEs) learn feature dictionaries from residual-stream activations. We show that the SAE training objective is invariant under residual gauge, so SAEs cannot canonically choose a basis. d Proposition E.9 (SAE objective is gauge-invariant). Let D = {hi }N i=1 ⊂ R be residual activations (row-vector convention), and consider an SAE with encoder and decoder ⊤ z(h) = ϕ(hWenc + benc ),

⊤ ĥ(h) = z(h)Wdec + bdec ,

trained with objective L(θ; D) =

X

2

h − ĥ(h) 2

h∈D

+ λ ∥z(h)∥1 .

Let G be any admissible residual gauge (a permutation, possibly composed with the global sign flip, for LayerNorm models, or a signed permutation for RMSNorm models), and define DG = {hG : h ∈ D} and parameters G Wenc = Wenc G,

bG enc = benc ,

G Wdec = G⊤ Wdec ,

bG dec = bdec G.

Then for all h ∈ D, z G (hG) = z(h),

ĥG (hG) = ĥ(h)G,

and consequently L(θG ; DG ) = L(θ; D). ⊤ Proof. Using GG⊤ = I, z G (hG) = ϕ(hGG⊤ Wenc + benc ) = z(h). Then ĥG (hG) = ⊤ G z(h)Wdec G + bdec G = ĥ(h)G. Thus (hG − ĥ (hG)) = (h − ĥ(h))G, so the squared error is preserved because G is orthogonal, and the ℓ1 term is unchanged because z G (hG) = z(h).

Corollary E.10 (SAE feature indices are non-identifiable). Independently of the residual gauge, for any feature permutation P ∈ Sdfeat , the SAE objective is invariant under Wenc 7→ P ⊤ Wenc , benc 7→ benc P , and Wdec 7→ Wdec P , with bdec unchanged. Thus SAE feature indices are noncanonical without alignment.

F

Transform rules for all module types

All rules use the row-vector convention: H G = HG where H has shape [n, d]. HuggingFace Linear computes Y = XW ⊤ + b with W ∈ Rdout ×din . Table 9: Gauge transform rules by module type. Module

Type

Weight

Bias

Tok/Pos embed Q, K, V proj O projection MLP up/gate MLP down RMSNorm LayerNorm LM head

Embed READ WRITE READ WRITE Norm Norm READ

W 7→ W G W 7→ W G W 7→ G⊤ W W 7→ W G W 7→ G⊤ W γ 7→ γ[π −1 ] γ 7→ γ[π −1 ] W 7→ W G

– unchanged b 7→ bG unchanged b 7→ bG – β 7→ ϵβ[π −1 ] for G = ϵP ; other signs break –

18

Index form: let π be the permutation with Pi,π(i) = 1, and sj the signs. READ maps satisfy W G [:, j] = sj W [:, π −1 (j)] (permute/sign columns). WRITE maps satisfy W G [i, :] = si W [π −1 (i), : ] (permute/sign rows). WRITE bias transforms as bG j = sj bπ −1 (j) . RMSNorm transforms as G γj = γπ−1 (j) (permute only, no sign). Why Q, K, V are unchanged under reparameterization. This is the key insight that makes the proof work with multi-head attention. With H G = HG and WQG = WQ G (READ rule): QG = H G (WQG )⊤ = HG · G⊤ WQ⊤ = HWQ⊤ = Q. The gauge cancels at the READ boundary, so attention patterns are identical.

G

Hungarian optimality

The sign-marginalized cost is optimal for recovering signed permutations. Let ui be column i of H, vj column j of H G , and C = H ⊤ H G . We seek a permutation π and signs s minimizing min

π∈Sd , s∈{±1}d

d X

2

ui − si vπ(i) 2 .

i=1

Expanding by columns, d X

2

ui − si vπ(i) 2 =

i=1

 X X 2 2 ∥ui ∥2 + vπ(i) 2 − 2si u⊤ v = const − 2 si Ci,π(i) . π(i) i i

i

For any fixed permutation π, the optimal sign is si = sign(Ci,π(i) ), yielding X |Ci,π(i) |. max π∈Sd

i

This is exactly a linear assignment problem on cost −|Ci,j |, solvable by the Hungarian algorithm. G.1

Assignment stability under a margin

Lemma G.1 (Assignment stability under a margin). Let C ∈ Rd×d and define FC (π) = Pd ⋆ i=1 |Ci,π(i) | for π ∈ Sd . Assume the maximizer π = arg maxπ FC (π) is unique and define the margin m := FC (π ⋆ ) − max⋆ FC (π) > 0. π̸=π

d×d

If E ∈ R satisfies ∥E∥∞ = maxi,j |Ei,j | ≤ ϵ and 2dϵ < m, then π ⋆ remains the unique maximizer for C + E. Proof.

For any scalars a, b, ||a + b| − |a|| ≤ |b|. Thus for any π, |FC+E (π) − FC (π)| ≤

d X

|Ei,π(i) | ≤ dϵ.

i=1

Therefore FC+E (π ⋆ ) ≥ FC (π ⋆ ) − dϵ and for any π ̸= π ⋆ , FC+E (π) ≤ FC (π) + dϵ ≤ FC (π ⋆ ) − m + dϵ. Hence FC+E (π ⋆ ) − FC+E (π) ≥ m − 2dϵ > 0.

H

Experimental details

Models. Across experiments we use 11 models spanning 4 architectures and scales from ∼10M to 8B parameters. The symmetry-boundary checks of Table 2 use the seven models listed there; the transport, optimizer-state, LoRA, steering, SAE, and merging experiments add four further RMSNorm models. 19

Production-scale RMSNorm models. Qwen2.5-7B has 28 layers, d = 3584, RMSNorm, and Q/K/V head-space biases only (boundary check). Llama-3.1-8B has 32 layers, d = 4096, RMSNorm, and no biases (boundary check). Llama-2-7b-chat-hf has 32 layers, d = 4096, RMSNorm, and no biases (boundary check, refusal reanalysis, gauge-scramble merge). Mid-scale RMSNorm models. TinyLlama-1.1B has 22 layers, d = 2048, RMSNorm, and no biases (boundary check, SAE/steering/LoRA transfer). Qwen2.5-1.5B has 28 layers, d = 1536, RMSNorm, and Q/K/V head-space biases only (transport trajectory recovery, Bd /Sd steering, LoRA, gauge-scramble). Qwen2.5-0.5B has 24 layers, d = 896, RMSNorm, and Q/K/V head-space biases only (stateful optimizer resumption). Llama-3.2-1B has 16 layers, d = 2048, RMSNorm, and no biases (transport replication, steering replication; Appendix J). Qwen Q/K/V biases live in attention head space and remain unchanged under the residual gauge, as in Lemma B.6; the reported transform rules do not rely on residual-space WRITE biases in these modules. Small-scale and other architectures. We use a custom 10M Llama-style transformer with 4 layers, d = 256, RMSNorm (independent-seed merging; trained from scratch on identical data, different seeds); BERT-base (110M, 12 layers, d = 768, LayerNorm; Sd per-index symmetry; boundary check); T5-small (60M, 6+6 layers, d = 512, T5LayerNorm; Bd symmetry; boundary check); and ViT-B/16 (86M, 12 layers, d = 768, LayerNorm; Sd per-index symmetry; boundary check). Existing assets and licenses. All external checkpoints and datasets are used through their original public distributions and are not redistributed with this paper. The Qwen2.5, TinyLlama, BERT-base, T5-small, and ViT checkpoints report Apache-2.0 license metadata on their Hugging Face model cards; Meta Llama checkpoints are gated and used under the corresponding Llama 2, Llama 3.1, or Llama 3.2 community license terms. WikiText is used under its CC-BY-SA-3.0/GFDL license metadata; SST-2/GLUE is used through the public GLUE distribution and original benchmark terms. ParaRel prompt templates used by the knowledge-neuron audit scripts are credited to Elazar et al. [2021]. Arditi et al.’s refusal-direction code and splits are credited in Appendix H.3 and used only for the replication/audit reported there. Core software dependencies are PyTorch, Hugging Face Transformers/Datasets [Wolf et al., 2020], NumPy, and SciPy; their standard open-source licenses are BSD-style or Apache-2.0, and SciPy’s assignment solver is cited where used. Gauge transforms. We generate random gauges using NumPy with deterministic seeds: the permutation is a random permutation of [0, d − 1] with seed s; the signs are a random choice from {−1, +1}d with seed s + 1000; multi-gauge sweeps use seeds 42–61 (K = 20 gauges). All transforms are applied in-place to minimize memory overhead. Steering vectors. We extract sycophancy steering vectors using Contrastive Activation Addition (CAA): generate contrastive prompt pairs (sycophantic vs. honest responses), extract residual stream activations at layer 10, and compute the mean difference vector across prompt pairs. Steering effect is measured as the logit difference change when adding the vector at inference. Evaluation metrics. Logit invariance is evaluated on a fixed invariance batch (5 short prompts for language models; 5 random images for ViT), and we report the maximum absolute logit deviation across the batch. This batch is separate from the alignment probe set used for Hungarian gauge recovery. The logit error is the maximum absolute difference in output logits between original and gauge-transformed models, with threshold ∼ 10−4 for gauge invariance. Top-k overlap counts shared indices between top-k coordinates by activation magnitude with k = 10. Linear CKA uses standard centering across samples (HSIC with linear kernel), i.e., 2

CKA(X, Y ) =

Xc⊤ Yc F , ∥Xc⊤ Xc ∥F ∥Yc⊤ Yc ∥F

where Xc , Yc are centered across samples. Refusal direction reanalysis (Arditi et al., 2024). Setup. We reproduce Arditi et al.’s released pipeline on meta-llama/Llama-2-7b-chat-hf (full protocol in Appendix H.3). Candidates. We (ℓ) (ℓ) (ℓ) form directions ri = E[hi | harmful] − E[hi | harmless] from block-input residual streams. 20

Selection. We follow their bypass/induce criterion with KL filter 0.1 and layer cutoff ℓ < 0.8L; refusal score is last-token log-odds on R = {306}. Checks. On 20 harmful validation prompts (greedy decoding), ablating the selected direction reduces refusal from 20/20 to 3/20. We match their released direction artifact (layer 14, position −1) with cosine 0.984 in the same residual space (d = 4096). Hungarian alignment. The sign-marginalized recovery procedure uses SciPy’s linear sum assignment solver [Virtanen et al., 2020]; SciPy documents this routine as a modified Jonker–Volgenant implementation described by Crouse [2016], building on shortest-augmenting-path assignment algorithms [Jonker and Volgenant, 1987]. Our gauge-derived cost matrices are near-permutation, and the dense assignment step completes in a few seconds for d = 4096 in our implementation (worst-case O(d3 )). For d ≥ 8192, we restrict each row to its top-m matches in |C| (we use m = 200) and greed⊤ ily match; this avoids cubic assignment but still forms the dense cross-correlation C = Horig Hgauged . On a d = 4096 ablation, greedy top-200 recovers 94.7% permutation / 97.5% sign vs 100% exact. Probe set: 30 diverse sentences totaling ∼500 tokens. Evaluation uses held-out test sentences (disjoint from probe set). Reproducibility. Unless otherwise stated, evaluation and gauge-recovery experiments load models in float32 for numerical stability. The Qwen LoRA row uses a bf16-trained adapter; all raw, Sd , Bd , and reference evaluations for that row use the same trained checkpoint precision, so the comparison isolates gauge transport rather than dtype. All random seeds are fixed and reported. JSON result files include full hyperparameters and timestamps. For transport experiments, we enable deterministic CUDA algorithms (torch.use_deterministic_algorithms) to ensure exact reproducibility of composed gauge estimates; this disables TF32 and may reduce speed. H.1

Compute resources

All reported experiments are single-worker PyTorch jobs, with no model parallelism, data parallelism, or distributed training. Exact float32 7B/8B boundary and recovery rows require a CUDA GPU with at least 48 GB VRAM; half-precision variants fit on 24 GB-class GPUs but are not the exact rows. The 7B/8B gradient-times-activation attribution sweep used a single H200 worker. The 1B–1.5B evaluation, LoRA, TinyLlama tool-transfer, Qwen2.5-1.5B transport, and Llama-3.2-1B transport runs fit on one 24 GB-class GPU. We recommend 100–300 GB local disk for Hugging Face caches and temporary checkpoints. The archived timers for the transport families alone sum to about 154 single-GPU hours. Including the shorter un-timed boundary, merge, LoRA/SAE, attribution, and optimizer-state runs, a fromscratch reproduction of all reported tables should budget roughly 200–250 single-GPU hours, plus model-download time. H.2

Probe-budget sensitivity

This appendix collects probe-budget data referenced in Section 3 and Table 3. Known-gauge exact-recovery regime: probe-robust. On TinyLlama-1.1B with a sampled Bd gauge recorded as ground truth, sign-marginalized Hungarian recovery achieves 100% permutation/sign/combined accuracy across all probe budgets we tested. Harder same-base long-stride recovery: larger probes improve direct recovery. On Qwen2.51.5B same-base fine-tunes with a randomly sampled Bd gauge applied at step 0 and recorded as ground truth, direct long-stride recovery saturates as the local-match probe budget grows. Permutation and sign recovery both reach 100% at ∼8K tokens; below this, finite-sample noise in the per-step cross-correlation matrix produces some near-degenerate matches. The pattern matches Theorem 3.1: the sign component is robust to small probe budgets (the marginalization step needs only enough samples to determine the sign of each matched correlation), while the permutation component requires the cross-correlation matrix to be far from rank-degenerate, which scales with sample size. Natural cross-model regime: substantially probe-hungrier. For independently-trained models with no ground-truth gauge, the right notion is stability: agreement of the recovered alignment with a 21

Table 10: Compute resources for the reported experiment families. Wall-clock times are measured where the archived artifacts retain timers; otherwise we give the run configuration and a conservative single-GPU reproduction budget. Result family

Worker and memory

Run time / run size

Symmetry-boundary, gauge-fixing, and attribution audits

Single CUDA worker. Float32 7B/8B rows require ≥48 GB VRAM; H200 used for the 7B/8B grad×act attribution sweep. Smaller models fit on 24 GB-class GPUs.

Boundary/gauge recovery uses 5–30 prompts. Budget tens of minutes per 7B/8B model, dominated by loading and forward/backward passes.

Qwen2.5-1.5B transport and steering transfer

Single CUDA worker, float32. The archived 200-step log records about 6.2 GB allocated after model load and about 6.2–6.3 GB before optimizer creation, so this fits on a 24 GB-class GPU.

Archived timers: 22.8 h for the 200-step run and 115.6 h for the 1500-step run; each covers 9 trajectories and 18 cross-run pairs.

Llama-3.2-1B replication

transport

Single CUDA worker, float32. The run log records about 5.0 GB allocated after model load and about 5.1 GB before optimizer creation.

Archived timers: 5.7 h for the 200-step run and 10.3 h for the 1500-step run; each covers 9 trajectories and 18 cross-run pairs.

Qwen/TinyLlama LoRA, SAE, and steering tool transfer

Single 24 GB-class CUDA worker. The Qwen LoRA run uses bf16, batch size 4, gradient accumulation 4, LoRA rank 16, and 800 training steps; TinyLlama SAE/steering uses an 11,134-token probe.

Exact wall-clock timers were not retained in the final JSONs. Budget one to a few GPU-hours per condition.

Optimizer-state tion

Single 24 GB-class worker for Qwen2.50.5B and TinyLlama runs.

Main fp32 run: batch size 32, 500 warmup steps plus 500 resumed steps. Appendix variants range from 30+10 to 200+200 steps.

Single CUDA worker. TinyLlama and 10M rows fit on 24 GB-class GPUs; 7B gauge-scramble merge should use bf16 on 24 GB or float32 on a larger single GPU.

Gauge-scramble and independent-seed merge runs use small probe sets and 0–50 fine-tuning steps.

Merge runs

resump-

Table 11: Probe-sensitivity on TinyLlama-1.1B with a sampled Bd gauge recorded as ground truth. Recovery is invariant to probe budget across this range. Sentences

Tokens

Perm

Sign

Combined

2 3 5 10 20 30

22 31 49 101 202 292

100% 100% 100% 100% 100% 100%

100% 100% 100% 100% 100% 100%

100% 100% 100% 100% 100% 100%

large-probe reference. We measure this on a representative natural-alignment problem (Qwen2.51.5B against a same-architecture variant trained on different data); this is a probe-budget stability study, separate from the coordinate-preserving merge experiments in Section 5. Stability against the 36K-token reference grows from ∼6% at 39 tokens to ∼94% at 27K tokens. Practical guidance. For known-gauge settings (e.g., gauge-scramble experiments), a few hundred tokens suffice. For natural alignment between independently-trained models on the same architecture, plan for 10K–30K tokens of natural-domain text. For transport along a fine-tuning trajectory, a small per-step budget (∼500 tokens) suffices because each local match has a high assignment margin and the chain composes group-exactly when each step is correct. 22

Table 12: Probe-budget saturation for a harder direct long-stride recovery on Qwen2.5-1.5B samebase fine-tunes (randomly sampled Bd gauge applied at step 0 and recorded as ground truth, 200 fine-tuning steps). Probe set

Tokens

Perm

Sign

Combined

builtin_5 builtin_15 builtin_35 wiki_50 wiki_200

39 118 270 7,873 33,678

47.5% 63.5% 63.6% 100% 100%

100% 100% 100% 100% 100%

47.5% 63.5% 63.6% 100% 100%

Table 13: Natural-alignment stability vs. probe budget. Agreement is the fraction of coordinates assigned the same matched index as the wiki_200 reference; sign-flip fraction is the share of matches with negative sign.

H.3

Probe set

Tokens

Agreement

Sign-flip frac.

builtin_5 builtin_15 builtin_35 wiki_10 wiki_50 wiki_100 wiki_150 wiki_200 (ref)

39 118 270 1,830 6,619 18,032 27,044 36,010

5.7% 11.5% 15.7% 73.2% 82.9% 89.4% 93.6% 100%

47.3% 44.5% 42.1% 12.8% 8.8% 5.9% 4.2% 4.8%

Arditi et al. refusal direction replication

Code path. We reproduce Arditi et al.’s released refusal-direction pipeline unmodified, adding a thin wrapper that contributes only the gauge analysis on top of their selection code. Prompts and splits. We use Arditi’s published harmful/harmless train and validation splits with instructions_only=true and no shuffling. For Llama-2-7b-chat-hf: 128 harmful + 128 harmless train prompts to estimate mean activations, and 32+32 validation prompts for direction selection. Behavioral refusal uses the first 20 harmful validation prompts. Prompt formatting. Instructions are formatted with the Llama-2 chat template [INST]{instruction}[/INST] (no system prompt, trailing whitespace), matching Arditi’s model class. Definition of h and candidate directions. We hook the transformer block input (residual stream before attention/MLP; pre-RMSNorm for Llama-2) using a forward pre-hook on each model.model.layers[l]. We record activations at the end-of-instruction token positions, i.e., the tokenized suffix [/INST] with indices −|EOI|, . . . , −1. Candidate directions are rpos,ℓ = E[hpos,ℓ | harmful] − E[hpos,ℓ | harmless], then L2-normalized. Direction selection (Arditi criteria). For each (pos, ℓ) we compute (1) a refusal ablation score on harmful prompts using Arditi’s global 3-site ablation (block input + attention output + MLP output at every layer), (2) a steering score on harmless prompts by adding the direction at the source layer only, and (3) a KL divergence score between baseline and ablated last-token logits on harmless prompts. We discard the last 20% of layers, candidates with KL > 0.1, and candidates with steering < 0. We select the candidate with the lowest ablation refusal score, where refusal score is the log-odds of the model’s refusal token(s) at the last prompt token (Llama-2 uses token id 306, “I”). For this run the selected direction is layer 12, position −2. Cached direction comparison (cosine 0.984). Arditi’s released cached direction for Llama-2-7bchat-hf is at layer 14, position −1. We compute cosine similarity between their cached vector and our candidate at the same layer/position (both L2-normalized in the same residual space, d = 4096), obtaining 0.984. Our selected direction differs only in (ℓ, pos), consistent with sample variance from subset sizes above. 23

Behavioral refusal rate (100% → 15%). We generate completions for 20 harmful validation prompts with greedy decoding (do_sample=False, max_new_tokens=512; no temperature/top-p). Refusal is the fraction of completions containing any of the following fixed substrings: • "i’m sorry", "i am sorry", "i apologize" • "as an ai", "as a language model", "as an assistant" • "i cannot", "i can’t", "i am unable to", "i’m unable to" • "i am not able to", "i’m not able to" Baseline refusals are 20/20; global direction ablation reduces this to 3/20 (15%). Table 14: Gauge invariance validation across architectures and scales. All models achieve max logit error ≲ 1.3 × 10−4 , consistent with the discrete gauge symmetry. LayerNorm models admit per-index Sd (plus the global sign flip); RMSNorm/T5LayerNorm models admit full Bd (signed permutations). Model

I

Type

Norm

Gauge

Max error

Production-scale models (7–8B parameters) Qwen2.5-7B Decoder-only RMSNorm Llama-3.1-8B Decoder-only RMSNorm Llama-2-7b-chat Decoder-only RMSNorm

Bd Bd Bd

1.13 × 10−4 3.00 × 10−5 1.28 × 10−4

Smaller models and other architectures TinyLlama-1.1B Decoder-only RMSNorm BERT-base Encoder-only LayerNorm T5-small Encoder-decoder T5LayerNorm ViT-B/16 Vision LayerNorm

Bd Sd Bd Sd

8.58 × 10−5 2.29 × 10−5 7.63 × 10−5 4.29 × 10−6

Multi-behavior steering results

Table 15: Signed-cost versus sign-marginalized Hungarian recovery under recorded Bd basis changes on 7B/8B RMSNorm models. All rows use layer 10, seed 42, and 30 probe texts. Signed-cost Hungarian maximizes raw correlations; sign-marginalized Hungarian matches by absolute correlation and then recovers the sign from the matched entry. Model

Signed-cost Perm. Sign

Sign-marginalized Perm. Sign

Qwen2.5-7B Llama-3.1-8B Llama-2-7B-chat

49.4% 49.9% 49.5%

100.0% 100.0% 100.0%

50.1% 50.1% 50.1%

100.0% 100.0% 100.0%

Table 16: Gauge equivariance across three steering behaviors (TinyLlama-1.1B-Chat; mean over all layers). Error (mean) is the mean absolute difference in baseline-subtracted steering effect (scale 2.0 minus 0.0), where steering effect is the mean per-token log-probability margin between positive and negative continuations. Top-10 overlap is the mean of | top-10(|v|) ∩ top-10(|v G |)| across layers. Behavior Sentiment Sycophancy Refusal

Extraction Raw text CAA Raw text

Error (mean)

Top-10 overlap

−5

0.05/10 0.00/10 0.00/10

1.15 × 10 2.85 × 10−5 1.98 × 10−5

We test gauge equivariance across three steering behaviors. For each, we extract steering vectors using either Contrastive Activation Addition (CAA) with chat-templated prompts, or raw contrastive text pairs. All three behaviors show the same pattern: steering effects are preserved (error < 10−4 ) while index-level summaries have near-zero overlap. Sentiment uses contrastive pairs of positive vs. negative affect statements. Sycophancy uses CAA prompts following Rimsky et al. [2024]: agreement-seeking vs. honest responses to opinion questions. Refusal uses contrastive pairs of compliant vs. refusing responses to benign requests. 24

J

Gauge tracking via parallel transport

We evaluate cross-run alignment by composing transports through a shared base checkpoint. Setup. We fine-tune Qwen2.5-1.5B for 200 and 1500 steps on three datasets (WikiText, code, math) with seeds 42, 5042, 10042, yielding 9 runs and 18 cross-run pairs. Unless noted otherwise, Appendix J reports Qwen2.5-1.5B results; we include additional Llama-3.2-1B summaries in Tables 18 and 20. The checkpoint stride is two optimizer steps: 200-step runs use 101 checkpoints and 100 local edges per base-to-final chain; 1500-step runs use 751 checkpoints and 750 local edges. A cross-run transport-via-base map composes two chains, so it uses 200 or 1500 local matches per reported pair. The same fixed probe set is reused for each estimated edge; held-out probes are used only for the overfitting check below. For each run we estimate a base-to-final map either by endpoint matching (a one-shot best-fit signed permutation on probe activations at the endpoints) or by composing local gauges along the training trajectory (transport). For a pair (a, b) we compare direct endpoint matching to base-composition ablations (endpoint-via-base, hybrid) and transport-via-base. We apply a randomly sampled signed-permutation gauge at step 0 and record the induced residual-stream correspondence as ground truth, but report permutation recovery accuracy for compactness since sign recovery is essentially perfect in these runs (permutation accuracy and total (perm+sign) accuracy differ by ≤ 0.13% across all pairs). As a probe-overfitting check, endpoint-direct accuracy on a held-out batch is slightly lower (by 2.8 percentage points on average at 1500 steps). Cross-run ablations. Table 17 reports residual-stream cross-run permutation recovery for five ways of constructing the cross-run map: direct final-to-final endpoint matching, composing endpoint gauges through the base (endpoint-via-base), two hybrid compositions (one side transport, one side endpoint), and composing transported gauges through the base (transport-via-base). Transport-via-base is consistently best at both 200 and 1500 steps. Pair-type breakdown. Table 4 decomposes cross-run recovery into cross-seed and cross-dataset pairs. The largest gains occur on cross-seed pairs, where direct endpoint matching is unreliable. Crossdataset pairs are more heterogeneous: when endpoint matching is already near-ceiling, transport can underperform due to compounding local alignment errors (Figure 2). Table 17: Residual-stream cross-run gauge recovery on Qwen2.5-1.5B. Endpoint-direct matches the two final checkpoints directly; the via-base and hybrid methods first align each run to the shared base (by endpoint matching or transport) and then compose through the base. Numbers report permutation recovery accuracy (%, mean±s.d.) over 18 run pairs (3 datasets×3 seeds). Method Endpoint-direct (final ↔ final) Endpoint-via-base (endpoint·endpoint) Hybrid (A transport, B endpoint) Hybrid (A endpoint, B transport) Transport-via-base (transport·transport)

200 steps

1500 steps

61.3 ± 33.6 17.1 ± 6.0 37.4 ± 12.2 37.7 ± 9.9 96.2 ± 6.3

60.3 ± 32.6 11.4 ± 3.6 26.9 ± 6.0 26.5 ± 6.1 91.1 ± 8.7

Within-run recovery. Table 18 summarizes base→final recovery within each run. Residual-stream transport remains accurate even at 1500 steps, while FFN transport can degrade over long trajectories; Table 19 provides the per-run breakdown. We also include an additional within-run summary for Llama-3.2-1B. J.1

Intervention transfer: cross-run steering vector transfer

Gauge recovery accuracy measures coordinate-level alignment, but does this translate to improved interpretability? We test whether steering vectors extracted from one independently trained model transfer effectively to another model trained from the same base. Setup. Using the same cross-run setup, we extract sentiment steering vectors and apply them across runs. We report results for Qwen2.5-1.5B (28 layers; residual layers {7, 14, 21}) and Llama-3.2-1B (16 layers; residual layers {4, 8, 12}). Let Tr denote the transport base→final map for run r and 25

Figure 2: Per-pair transport advantage ∆ = Transport−Endpoint (percentage points) for cross-seed and cross-dataset pairs on Qwen2.5-1.5B. Each dot is a run pair; vertical bars denote group means. Transport consistently improves cross-seed recovery, while cross-dataset gains are heterogeneous and can be negative when endpoint matching is already near-ceiling. Table 18: Within-run gauge recovery (base → final) averaged over 9 fine-tuning runs (3 datasets × 3 seeds). Transport composes local gauges along the training trajectory; endpoint matches activations only at the endpoints. FFN transport is evaluated only for Qwen2.5-1.5B. Metric Qwen2.5-1.5B residual transport Qwen2.5-1.5B residual endpoint Qwen2.5-1.5B FFN transport Qwen2.5-1.5B FFN endpoint Llama-3.2-1B residual transport Llama-3.2-1B residual endpoint

200 steps

1500 steps

98.1 ± 5.0 38.3 ± 11.8 96.3 ± 4.0 75.7 ± 17.0 93.5 ± 5.2 37.8 ± 9.0

95.3 ± 6.6 27.9 ± 6.4 72.6 ± 25.3 30.7 ± 6.7 81.6 ± 31.2 20.9 ± 8.0

Er the endpoint base→final map (one-shot matching of t0 and tT ). Then GA→B = TA−1 TB and −1 endpoint-via-base is EA EB . For each pair of models A → B, we compare: • Fresh: Steering vector extracted directly from model B (upper bound) • Transported: Vector from A transformed via transport gauge GA→B −1 • Endpoint-mapped: Vector from A transformed via endpoint-via-base mapping EA EB

We measure steering effect preservation as 100 × ∆method /∆fresh . We treat a steering direction as detectable only if ∆fresh > 0.1; otherwise preservation is not reported and is excluded from averages. Permutation-only comparison. Table 21 reports the Qwen2.5-1.5B rows used in Table 5. This comparison uses the WikiText cross-seed pair (source seed 42, target seed 5042) after 1500 fine-tuning steps. The Bd rows compose signed-permutation gauges; the Sd rows use the same permutation maps with all signs fixed to +1. Preservation is averaged over residual layers {7, 14, 21} when the fresh target effect is detectable; for refusal, layer 7 is excluded because the fresh effect is below the detectability threshold. FFN steering detectability. In the Qwen2.5-1.5B FFN steering-transfer evaluation we probe three pre-chosen layers {7, 14, 21} (for consistency with the residual-stream experiments). However, the fresh in-target FFN intervention is often negligible at layers 7 and 21, making preservation ratios ill-conditioned. We therefore treat a layer as not detectable when ∆fresh ≤ 0.1 and exclude it from preservation averages. Table 22 reports detectability rates for transparency: layer 14 is detectable in the majority of pairs, whereas layers 7 and 21 are rarely detectable. Reporting policy for FFN steering. All FFN steering-transfer preservation numbers are reported for layer 14 only, where the fresh in-target intervention is most consistently detectable (Table 22). 26

Table 19: Per-run within-run recovery on Qwen2.5-1.5B after 1500 fine-tuning steps. Dataset/seed

Res. endpoint

Res. transport

FFN endpoint

FFN transport

code 10042 code 42 code 5042 math 10042 math 42 math 5042 wiki 10042 wiki 42 wiki 5042

28.0 26.4 25.7 28.6 13.5 32.1 33.1 27.5 35.9

97.1 99.3 99.2 91.5 100.0 97.5 79.3 99.1 95.1

22.4 37.6 29.7 35.8 23.8 42.5 26.2 28.5 30.3

77.1 92.8 97.5 68.7 70.5 13.3 80.7 91.0 61.9

Table 20: Cross-run steering transfer after aligning source and target runs. We report Qwen2.5-1.5B and Llama-3.2-1B. Steering preservation is 100×∆method /∆fresh , where ∆ is the baseline-subtracted steering effect (log-probability margin), and the win-rate is over detectable cases (∆fresh > 0.1). For Qwen FFN we report only layer 14 (detectable in 14/18 pairs at 200 steps, 16/18 at 1500 steps). Values above 100% indicate the transported vector exceeds the freshly extracted one; negative values indicate a reversed effect. Component

Transport

Endpoint

Win-rate

103.6 80.5 84.8 38.3 99.1 68.4

33.2 18.4 25.2 -6.2 53.5 38.9

54/54 50/52 13/14 12/16 54/54 40/52

Qwen residual, 200 steps Qwen residual, 1500 steps Qwen FFN layer 14, 200 steps Qwen FFN layer 14, 1500 steps Llama-3.2 residual, 200 steps Llama-3.2 residual, 1500 steps

Scope of steering transfer. The cross-run steering-transfer experiment evaluates sentiment steering; broader behaviors and architectures are not evaluated in this experiment. FFN gauge tracking is less stable than residual-stream tracking over long trajectories in some runs (Table 19), suggesting more robust FFN matching (or periodic re-anchoring) is required for long-horizon FFN transport.

K

Optimizer-state resumption details

The stateful-resumption run compares resumed trajectories, not endpoint losses. For AdamW, Bd transport applies signs to the first moment mW according to each parameter’s READ/WRITE role, while the second moment vW is transported by the unsigned permutation because signs square out. The Sd ablation applies the correct permutation but omits signs in mW . The main Qwen2.5-0.5B row uses float32, AdamW with learning rate 2 × 10−5 , weight decay 0.01, gradient clipping at 1.0, sequence length 128, 2048 training texts, four held-out evaluation batches, and a trainable suffix of four decoder layers (trainable_scope=last_n, last_n_layers=4). The fp32 rows are the primary tests of exact gauge covariance.

L

SAE and steering tool-transfer details

The TinyLlama-1.1B tool-transfer experiment isolates the sign variable with a known gauge. We train an SAE and extract CAA steering vectors on model A, apply a random Bd gauge to obtain functionally equivalent model B, recover the gauge from activations alone, and transfer the tools to B. The probe set contains 100 texts (11,134 tokens). Full Bd recovery obtains 100% joint permutation/sign accuracy. The transferred SAE has normalized reconstruction MSE 0.004, matching the recovered-gauge reference. The signed-cost Sd recovery pipeline hits the structural ceiling of Theorem 3.1: it recovers only the positive-sign half of the coordinates and gives SAE NMSE 1.08. Separately, steering vectors use the matched permutation with signs forced to +1, isolating orientation loss; across sentiment, sycophancy, and refusal this degrades or inverts all three effects, ranging from −14% to −80%. 27

Table 21: Qwen2.5-1.5B signed-permutation versus permutation-only steering transfer on a samebase cross-seed pair. Step-0 and transport columns report coordinate recovery. Steering preservation is 100 × ∆method /∆fresh . Behavior

Gauge

Step 0

Transport

Steering preservation

Sentiment Sentiment Refusal Refusal

Bd Sd Bd Sd

100.0% 48.2% 100.0% 48.2%

92.7% 27.3% 92.6% 25.9%

95.8% 17.2% +150.5% -32.2%

Table 22: Detectability of FFN steering directions in the cross-run transfer evaluation (number of pairs with ∆fresh > 0.1). Only layer 14 yields consistently detectable FFN steering. FFN layer 7 14 21

M

200 steps

1500 steps

3/18 14/18 3/18

0/18 16/18 3/18

Natural sign mismatch in independent training

Question: The merge experiments in §5 include gauge-scramble settings. Does sign mismatch also occur between independently trained models? Answer: Yes, in this small-scale run. Two RMSNorm models trained from different seeds develop 50% sign mismatch, and signed alignment outperforms perm-only by 37%. Setup. We train two identical 4-layer Llama-style transformers (256 hidden, RMSNorm, ∼10M params) on WikiText [Merity et al., 2017] for 1000 steps, differing only in initialization seed. We then estimate the gauge relating them via activation-based Hungarian matching. This is a low-dimensional same-data example, not evidence that ∼ 500 probe tokens suffice for natural alignment of large LLMs; larger natural LLM alignment remains probe-hungry as in Appendix H.2. Sign mismatch occurs without an imposed gauge in this run. The models exhibit 50% sign disagreement: half of matched neuron pairs learned opposite sign conventions. This is expected— RMSNorm models have Bd symmetry, and independent training runs break this symmetry differently. Signed alignment helps. Table 25 shows merge barrier peaks. Perm-only alignment (Sd ) reduces the peak barrier by 61%, but signed alignment (Bd ) achieves 75% reduction—a 37% relative improvement from accounting for signs. Takeaway. This small-scale natural run shows that sign mismatch can also arise between independently trained RMSNorm models. M.1

Additional 0-step gauge-equivalent merge: TinyLlama-1.1B

The same gauge-scramble merge setup also applies to a smaller RMSNorm decoder, TinyLlama/TinyLlama-1.1B-Chat-v1.0, with 0 fine-tuning steps. Thus the two endpoints are exactly gauge-equivalent, and a symmetry-respecting alignment should eliminate the peak barrier. Table 26 shows that perm-only alignment with the correct permutation but signs forced to +1 does not reduce the peak barrier relative to unaligned averaging, while signed alignment (ours; Bd ) eliminates it. M.2

Minimal example: Sd -only alignment can worsen averaging under Bd

We give a minimal example illustrating the cancellation mechanism behind “perm-only worse than unaligned” when the true symmetry group includes sign flips. 28

Table 23: Optimizer-state resumption variants. Entries are held-out logit relative MSE against the gauge-equivalent reference resume. “Other variants” ranges over raw optimizer state, wrong signs, random signs, and zeroed moments. Model/opt.

Setup

Pre

Bd

Sd

Other variants

Qwen2.5-0.5B AdamW Qwen2.5-0.5B SGD µ = 0.9 TinyLlama-1.1B AdamW Qwen2.5-0.5B AdamW Qwen2.5-0.5B AdamW

fp32 b = 1, 30+10 fp32 b = 1, 50+10 fp32 b = 1, 30+10 fp32 b = 8, 200+200 fp32 b = 32, 500+500

1.86 × 10−12 9.28 × 10−12 5.64 × 10−13 1.51 × 10−12 5.24 × 10−12

1.05 × 10−12 5.11 × 10−12 3.25 × 10−13 5.75 × 10−12 5.30 × 10−10

2.08 × 10−2 3.83 × 10−2 3.77 × 10−3 3.72 × 10−4 3.50 × 10−3

2.54 × 10−3 –7.57 × 10−2 2.72 × 10−2 –7.10 × 10−2 1.35 × 10−3 –1.50 × 10−2 3.48 × 10−4 –4.69 × 10−4 3.88 × 10−3 –7.04 × 10−3

Table 24: TinyLlama-1.1B tool transfer under a sampled Bd gauge recorded as ground truth. Condition

Perm/sign recovery

SAE NMSE

100% / 100% 50% / – 100% / 100%

0.004 1.08 0.004

Bd recovery (ours) Sd signed-cost recovery Recovered-gauge reference

Proposition M.1 (Permutation-only alignment can increase midpoint loss under Bd ). There exist two Bd -gauge-equivalent parameterizations of the same function such that averaging after a permutationonly (Sd ) alignment step yields strictly higher midpoint loss than unaligned averaging. Proof. Consider the two-layer linear network fθ (x) = xAB with A = B = I2 and squared-error 2 loss to targets y = x under isotropic inputs with E ∥x∥2 < ∞. Let     0 1 0 1 P = , S = diag(−1, 1), G := P S = ∈ Bd , 1 0 −1 0 so GG⊤ = I2 and G⊤ = −G. Define gauge-transformed parameters by AG = AG and B G = G⊤ B, so fθG (x) = xAG B G = x and θ and θG are gauge-equivalent. For unaligned averaging, midpoint parameters are A1/2 = 12 (I2 + G) and B1/2 = 12 (I2 + G⊤ ), giving A1/2 B1/2 = 41 (I2 + G + G⊤ + GG⊤ ) = 12 I2 . Thus f1/2 (x) = 12 x and expected midpoint 2 loss is 14 E ∥x∥2 . For Sd -only alignment, a permutation-only alignment can remove the permutation component P but cannot apply the required sign flip. Composing with P yields sign-only mismatch GP = diag(1, −1). Averaging θ with θGP gives A1/2 = 12 (I2 + GP ) = diag(1, 0) and B1/2 = diag(1, 0), so f1/2 (x) = 2 2 (x1 , 0). The expected midpoint loss is E[x22 ] = 12 E ∥x∥2 , strictly larger than 14 E ∥x∥2 .

N

Extended related work

Permutation symmetries and mode connectivity. Hidden-unit permutations have long been known to create equivalent parameterizations and connected loss landscapes [Brea et al., 2019, Freeman and Bruna, 2017, Draxler et al., 2018, Garipov et al., 2018, Tatro et al., 2020, Entezari et al., 2022]. Git Re-Basin turns this into an alignment procedure over Sd [Ainsworth et al., 2023]. Our contribution is the architecture-dependent residual-stream group for transformers: per-index Sd for LayerNorm and Bd for RMSNorm, together with the ceiling theorem showing why the missing sign component is structurally unrecoverable by signed-correlation matching. Continuous and rotation-based methods. SliceGPT, QuaRot, generalized linear mode connectivity, and rotation-aware fusion exploit broader O(d) freedom after reparameterization [Ashkboos et al., 2024a,b, Theus et al., 2025, Zhang et al., 2025]. These methods solve different alignment problems. They can be stronger for coordinate-indifferent compression or fusion; they are not coordinate-preserving transports of sparse objects in the native γ-explicit model. Theorem 2.2 gives the interface: extract the Bd component when coordinates matter, use the residual dense rotation when they do not. 29

Table 25: Merge barrier peaks for independently trained RMSNorm models (two independent seeds). Signed alignment provides 37% additional barrier reduction over perm-only. Alignment

Barrier peak (∆NLL)

Reduction

1.68 0.66 0.42

– 61% 75%

Unaligned Perm-only (Sd ) Signed (Bd )

Table 26: TinyLlama-1.1B gauge-scramble merge (0-step fine-tuning; endpoints are gauge-equivalent). Perm-only alignment with the correct permutation but signs forced to +1 leaves a large peak barrier, while signed alignment (ours; Bd ) eliminates it. Alignment

Barrier peak (∆NLL)

Unaligned Perm-only (correct permutation) Signed (ours)

6.36 6.47 0.00

Model merging and interpretability. Merging methods include weight matching, model soups, task arithmetic, TIES, ZipIt!, and optimal-transport fusion [Ainsworth et al., 2023, Wortsman et al., 2022, Ilharco et al., 2023, Yadav et al., 2023, Stoica et al., 2024, Singh and Jaggi, 2020, Imfeld et al., 2024, Verma and Elbayad, 2024, Yang et al., 2024b]. Our merge results are not a universal merging recipe; they identify a necessary discrete correction for RMSNorm coordinate-preserving alignment. In interpretability, invariant representation metrics remain valid without gauge fixing [Raghu et al., 2017, Morcos et al., 2018, Kornblith et al., 2019]; direction and subspace analyses are similarly invariant within a fixed chart [Elhage et al., 2023]. Our audit targets the complementary class of analyses that output indices. SAE dictionary studies identify feature indices with semantic labels [Bricken et al., 2023, Gao et al., 2025]: Bricken et al. [2023] report “feature A/1/2357” as a base64-input detector and characterize hundreds of similarly-indexed features with concept labels. Knowledge-neuron pipelines locate factual associations at specific MLP hidden units: Dai et al. (9) (10) [2022] identify w2141 and w1122 as carrying the Ireland–Dublin association, and report similar coordinate triples for other facts. Direction-level steering work [Rimsky et al., 2024, Arditi et al., 2024] makes single-model claims that are gauge-invariant in isolation but require a coordinate map to be compared or transferred across runs or related models, which is exactly what Proposition 2.3 characterizes. These examples have different native gauges: BERT-style FFN neuron claims are relative to hiddenunit permutations, while residual-stream artifacts in RMSNorm Llama/Qwen/TinyLlama charts live on a Bd orbit of size 2d d!. The dependence is not hypothetical. Paulo and Belrose [2026] report only ∼ 30% SAE feature overlap across training seeds on Llama 3 8B; our TinyLlama transfer experiment shows that on a single model, ignoring the residual Bd component drives SAE reconstruction NMSE from 0.004 to 1.08–a 270× degradation under an exact symmetry of the model. Other sources of cross-seed disagreement also exist (optimization noise, data order, feature splitting), but the residual gauge is a formally identifiable component, and the audit specifies the controls that isolate it.

O

Detailed limitations and broader impacts

Detailed limitations. First, our transport experiments measure recovery along real Qwen2.5-1.5B fine-tuning trajectories against coordinate correspondences induced by randomly sampled Bd gauges applied at step 0 and recorded as ground truth; the complementary regime–coordinate alignment between independently specified parameterizations from function-level information alone–is shown by Proposition 4.1 to admit no canonical answer, so its absence here reflects the structure of the symmetry rather than an algorithmic gap. Second, transport involves stride and probe-distribution choices, and FFN transport is less stable over long horizons than residual-stream transport. Third, exact repeated γ values create global stabilizer intersections, while near-repeats reduce assignment margins; activation matching then selects a probe-dependent Bd representative. Fourth, our audits target the subset of interpretability and adaptation workflows that output coordinate names. Coordinate-level claims 30

complement invariant analyses, and coordinate-level claims on RMSNorm models require gauge specification. Broader impacts. Gauge-correct transport can improve reproducibility and checkpoint-editing audits, but can also lower the cost of transferring behavioral interventions, including undesirable steering directions, between related checkpoints. We therefore present it as an audit and verification tool, and emphasize gauge-invariant reporting when coordinate identity is unnecessary.

31