arXiv:2605.22800v1 [cs.LG] 21 May 2026
The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning Vishal Rajput [email protected] KU Leuven
Abstract Robustness, domain adaptation, photometric and occlusion invariance, compositional generalisation, temporal robustness, alignment safety, and classical anisotropic regularisation are usually treated as separate problems with separate method families. This paper argues that much of their shared structure is one statistical problem: estimate the covariance Σtask = Cov𝑄𝑛 (𝑛) of label-preserving deployment nuisance, then regularise the encoder Jacobian along a matrix Σ′ whose range covers range(Σtask )—the matching principle. CORAL, adversarial training, IRM, augmentation, metric learning, Jacobian penalties, and alignment-style constraints are different estimators of that object, not independent robustness tricks. In the linear-Gaussian model we prove closed-form optimality (Theorem A), including cube-root water-filling within the matched range; necessity of range coverage for quadratic Jacobian penalties (Theorem G); the same range dichotomy at deep global minima (Theorem A⋆global ); and two falsification controls (Lemma C; Corollaries E/E⋆ ), with seven conditional consistency lemmas (D1–D7) for estimation under standard identifiability assumptions. We introduce the Trajectory Deviation Index (tdi), a label-free probe of embedding sensitivity when task accuracy or ‖𝐽‖𝐹 is insufficient. Thirteen pre-registered blocks from classical ML through Qwen2.5-7B test the predicted matched->-isotropic->-wrong-𝑊 ordering on geometry and deployment drift; twelve pass, and the sole exception (Office-31) is a Lemma D1 eigengap failure named before the run; at that largest scale, in a controlled alignment regime, matched style-pmh improves selective honesty and preserves Style tdi where standard dpo degrades it (§8). The contribution is naming Σtask , stating what Σ′ must do, and supplying a closed-form, falsifiable theory once that object is identified—not universality on every leaderboard.
1
Introduction
How to read this paper. Act I (§2–3): name Σtask , the pmh loss, and why ERM cannot remove deployment drift. Act II (§4–5): what Σ′ must look like (Fig. 2: 𝐺 → 𝐴 → 𝐵 → 𝐴⋆ ; Lemmas C, Cor. E/E⋆ ; D1–D7). Act III (§6–8): geometry (tdi, 𝐷𝑁 /𝐷𝑆 ), the five-step recipe, and thirteen blocks—including failures named in advance. Definitions: §2; proofs: Appendix A; headline block T7A (Qwen2.5-7B): §8. Since 2018 the field has catalogued adversarial fragility, texture and corruption bias, domain shift, sensor and accent drift, and alignment sycophancy as separate problems, each with its own methods and ablations [2, 7–9, 11–13, 15]. Prior unifications (domain bounds, transfer surveys, metric learning, information bottleneck) clarify patterns but do not name a single optimal regulariser matrix, falsifiable controls, or failure modes fixed before experiments run (§3, Table 1).
1
The matching principle in plain terms Object. Σtask is the covariance of ways inputs can change at deployment without changing the label (domain shift, noise, style, adversarial directions, . . . ). Loss. Train with task loss plus a penalty that discourages the encoder Jacobian along a matrix Σ′ (Eq. (4) in §2). Match. Choose Σ′ so its column space covers Σtask ; then deployment representations stop drifting under 𝑄𝑛 . Wrong Σ′ (random, isotropic-only, or signal-aligned) fails in ways the theory names before you run the experiment. Thesis. CORAL, PGD-AT, IRM, augmentation, metric learning, and Jacobian penalties are not independent methods—they are different estimators of the same Σtask , with a single optimum once 𝐴𝑘 is fixed. Why this is non-obvious. Theorem 4.2 makes range coverage necessary in the full quadratic Jacobian family; Theorem 4.9 extends the dichotomy to deep global minima (Lemma 4.10 verifies expressivity); Lemmas 4.12 and Corollaries 4.13/4.14 turn the claim into falsifiable controls tested in §8.
1.1
The problem the field had
The post-2018 literature accumulated three structural difficulties at once. First, methods proliferated faster than the theory could organise them, leaving comparison reduced to leaderboard-driven empirical horse races. Second, every method came with its own ablation conventions—random projections, label shuffling, signal masking—without any shared understanding of why one ablation falsifies a hypothesis and another does not. Third, the loss function was treated as background: cross-entropy or MSE or InfoNCE was assumed, and innovation took place in architectures, data, and compute. The matching principle reorganises all three. It identifies the common population object the methods are estimating, turns ablations into named falsification tests with predicted outcomes, and promotes the loss function to a first-class design variable parametrised by one PSD matrix per nuisance type.
1.2
The principle, stated
Deployment variation is a law 𝑄𝑛 with covariance Σtask = Cov𝑄𝑛 (𝑛); drift is measured by 𝐷𝑄 and ˜ 𝑄 (Eqs. (3) and surrounding definitions in §2). Training adds a trace its Jacobian linearisation 𝐷 ′ penalty along a chosen Σ ⪰ 0 (Eq. (4)). The matching principle is threefold: (i) matched range ˜ 𝑄 → 0 in the linear model (Theorem 4.1); (ii) no other quadratic range(Σ′ ) ⊇ range(Σtask ) drives 𝐷 Jacobian penalty can do so without covering that range (Theorem 4.2); (iii) the same dichotomy holds at the global minimum of deep encoders under (R)–(I), with (E) verified constructively (Theorem 4.9, Lemma 4.10). When regressor energy is uniform on range(Σtask ), the optimal allocation simplifies to Σ′ ∝ Σtask ; otherwise cube-root water-filling applies (§4.1). What the paper claims, in two sentences. The thirteen empirical blocks collectively reject the null hypothesis that the matched, isotropic, and wrong-direction arms have the same effect on deployment-drift metrics, in the direction the theory predicts, across modalities (vision, speech, code, molecules, language) and scales (linear models to 7B-parameter transformers). The single failure of the matched arm to outperform a competing method (Office-31, where CORAL beats matched pmh) is itself a failure mode the framework predicts when the estimator’s eigengap is near the Wedin bound.
2
What the paper does not claim. We do not claim universality. Causal/spurious-correlation problems (Colored MNIST, Waterbirds) violate the label-preservation hypothesis that defines Σtask and remain out of scope. We do not claim that matched pmh beats every competing method on every leaderboard; PGD-trained networks remain stronger than pmh at very large adversarial radii, and supervised accent adaptation can outperform matched pmh on WER while leaving the geometric problem un-repaired. We do not claim that the one-epoch dpo result at 7B replaces multi-epoch full-scale rlhf; it shows the mechanism in a controlled regime. These boundaries are the discipline of the framework, not its embarrassments; the framework’s main theoretical open question (optimisation reachability of the global minimum by gradient descent on a non-convex pmh loss) is named explicitly in §10 as Open Question 1. signal
signal
signal
nuisance
ERM
nuisance
Matched pmh
nuisance
Wrong-𝑊
Figure 1: The matching principle, geometrically. Axes: signal (vertical) vs. deployment nuisance (horizontal). Blue ellipses: regions of Jacobian sensitivity; red arrow: the same nuisance perturbation in all panels; red dot: where the embedding moves. Left (ERM): sensitivity in all directions ⇒ shift. Centre (matched pmh): sensitivity suppressed along nuisance ⇒ no shift. Right (wrong-𝑊 ): suppression at 45∘ ⇒ shift remains. The theory is the choice of which directions the ellipse covers.
3
Theorem G
Theorem A
range(𝐴) ⊇
Θ(1)
wrong
˜𝑄 𝐷
range(Σtask )
Thm. A⋆glo
Theorem B
iso
gap
Θ(𝜆−3 )
matched 𝜆
range
global min: same range test
alloc.
(a) Range is necessary (G). Drift → 0 only if the penalty covers every nuisance direction; a strict subspace leaves a 𝜆-independent floor.
(b) Matched ⇒ zero drift (A). Sufficiency: range(Σ′ ) covers ˜𝑄 → 0 range(Σtask ) and 𝐷 as 𝜆 → ∞.
(c) Range ≫ allocation (B). Missing a nuisance direction is catastrophic; mis-shaping Σ′ inside the matched range is cheap at large 𝜆.
(d) Deep global minimum (A⋆ ). Range matching at the pmh global minimum (Lemma 4.10 for expressivity).
Lemma C
Cor. E / E⋆
Prop. F
Lemmas D1–D7
random 𝑈 𝑈 ⊤
bound 𝐷𝑆
iso
(e) Wrong-𝑊 control (C). Random rank-𝑟 penalties ≡ isotropic at scale 𝑟/𝑑𝑥 ; not a separate hypothesis class.
D3 tdi0
Σ′ ‖ signal
(f) Signal-𝑊 control (E). Penalising along the signal axis provably hurts task loss below baseline (Cor. 4.14 in blocks).
D2
(g) Diagnostic envelope (F). Class-layout tdi is a one-sided geometric ˜𝑄 → 0 certificate when 𝐷 (not a substitute for trajectory tdi).
D1 ^ task Σ D4
D6 D7 D5
(h) Seven estimators (D1–D7). One Σtask , seven assumptions 𝐴𝑘 ; each row in Table 3.
Figure 2: Theorem map (one page). Row 1 (read left to right): G (range necessity) → A (matched sufficiency) → B (range vs. allocation cost) → A⋆global (deep global minimum). Row 2: Lemma C and Cor. E/E⋆ (falsification controls), Prop. F (diagnostic envelope), Lemmas D1–D7 (estimators of Σtask ). Panel letters (a)–(h) follow this layout. Statements: §4–§6; proofs: Appendix A. Training-time A⋆train needs assumption (O) (§10).
4
2
Setup and definitions
We work in the Perturbation Matching Hypothesis (PMH) framework: a task loss plus a trace penalty on the encoder Jacobian, with an explicit deployment-nuisance law 𝑄𝑛 . Spaces and maps. 𝑥 ∈ 𝒳 = R𝑑𝑥 ; 𝑦 ∈ 𝒴 (discrete or continuous); encoder 𝜑𝜃 : 𝒳 → 𝒵 = R𝑑𝜑 differentiable a.e.; 𝐿-Lipschitz decoder ℎ𝜃 : 𝒵 → 𝒴 ′ ; predictor 𝑓𝜃 = ℎ𝜃 ∘ 𝜑𝜃 ; task loss ℒtask a strictly proper scoring rule throughout. Training distribution. 𝑃 (𝑥, 𝑦) on 𝒳 × 𝒴 satisfies a correlated-nuisance condition: either screening confounding (𝐼(𝑛(𝑥); 𝑦 | 𝑠(𝑥)) = 0) or direct nuisance influence (𝑝(𝑦 | 𝑥) ̸= 𝑝(𝑦 | 𝑠(𝑥)) on positive measure). This is the structural precondition for the ERM blind-spot lower bound of Theorem 3.1. Deployment-nuisance law. Think of 𝑛 as a random input displacement (not a separate latent variable): noise, domain shift, style rewrite, or adversarial step. A law 𝑄𝑛 on 𝒳 has E𝑄𝑛 [𝑛] = 0 and 𝑥 (1) Σtask := Cov𝑄𝑛 (𝑛) ∈ S𝑑≥0 finite. Σtask is the paper’s central object: which directions in input space move at deployment ∫︀ without changing the label. The deployment distribution is 𝑃deploy (𝑥, 𝑦) = 𝑃 (𝑥 − 𝑛, 𝑦) 𝑑𝑄𝑛 (𝑛). Definition 2.1 (Label-preserving vs. label-changing deployment). 𝑄𝑛 is label-preserving for a training law 𝑃 (𝑥, 𝑦) if, for 𝑃𝑋 -almost every 𝑥 and 𝑄𝑛 -almost every 𝑛, 𝑝(𝑦 | 𝑥 + 𝑛) = 𝑝(𝑦 | 𝑥)
(equivalently 𝑦 ⊥ 𝑛 | 𝑥 when 𝑦 is deterministic).
The matching principle applies to Σtask = Cov𝑄𝑛 (𝑛) under this regime. A shift is label-changing (out of scope) when there exists a set of positive measure with 𝑝(𝑦 | 𝑥 + 𝑛) ̸= 𝑝(𝑦 | 𝑥)—e.g. colour predicts 𝑦 on the training distribution (Colored MNIST, Waterbirds). Then no label-preserving Σtask exists; causal / multi-environment tools are required (§7.3, Table 8, Causal). Worked example (domain shift). If training images come from source domain 𝑆 and deployment images from target 𝑇 with the same labels, a natural 𝑄𝑛 is 𝑛 = 𝑥𝑇 − 𝑥𝑆 for paired domains. Then Σtask = Cov(𝑥𝑇 −𝑥𝑆 ) is the cross-domain Gram matrix estimated by CORAL or Lemma A.11; matched pmh penalises 𝐽𝜑⊤ 𝐽𝜑 along that matrix so embeddings move less when the input shifts like a real deployment shift. Table 3 (§5) lists six other instantiations. Drift functionals.
The embedding drift is 𝐷𝑄 (𝜑) = E𝑥∼𝑃𝑋 , 𝑛∼𝑄𝑛 ‖𝜑(𝑥 + 𝑛) − 𝜑(𝑥)‖22 , [︁
]︁
(2)
and its first-order linearisation around zero-mean 𝑄𝑛 is ˜ 𝑄 (𝜑) = E𝑥 Tr 𝐽𝜑 (𝑥)⊤ 𝐽𝜑 (𝑥) Σtask 𝐷 [︁
(︁
)︁]︁
[︁
]︁
= E𝑥 ⟨𝐽𝜑 (𝑥)⊤ 𝐽𝜑 (𝑥), Σtask ⟩𝐹 ,
(3)
where ⟨𝐴, 𝐵⟩𝐹 = Tr(𝐴⊤ 𝐵) and 𝐽𝜑 (𝑥) = 𝜕𝜑/𝜕𝑥|𝑥 is the encoder Jacobian. The linearisation ˜ 𝑄 | = 𝑂(𝜎 4 ) at small 𝜎 under a Jacobian envelope). remainder is controlled by Lemma A.3 (|𝐷𝑄 − 𝐷 ˜ 𝑄 (Jacobian-weighted, easier to optimise and bound); the Throughout the theory we work with 𝐷 5
full nonlinear 𝐷𝑄 is what we measure in trajectory diagnostics (§6) when reporting embedding drift after training. Two drift quantities (do not conflate) ˜ 𝑄 (Eq. (3)): theory and PMH training—“how sensitive is the Jacobian along Σtask ?” 𝐷𝑄 / 𝐷 tdi: post-hoc measurement on a trained model—how far do embeddings move under probe noise? They align at small 𝜎 (Lemma A.3) but can diverge when geometry and accuracy decouple (§7.2). The PMH loss family.
For any Σ′ ⪰ 0 and 𝜆 > 0, [︁
(︁
ℒpmh(Σ′ ) (𝜃) = ℒtask (𝜃) + 𝜆 E𝑥 Tr 𝐽𝜑 (𝑥)⊤ 𝐽𝜑 (𝑥) Σ′
)︁]︁
.
(4)
Any minimiser is denoted 𝜑pmh(Σ ) . The matched-pmh minimiser 𝜑pmh(Σtask ) is the central object. In practice the trace is estimated by Hutchinson / Rademacher / paired-view surrogates; see §7 for implementation details. ′
Population vs. sample. All theoretical statements are at the population level. Finite-sample versions follow by standard concentration; empirical sections name the sample size explicitly. Proof map. All proofs are in Appendix A: foundation (A.1), matching theorems, and Lemmas A.8–A.14. §8 reports thirteen observational blocks, not one joint inference theorem.
2.1
What “nuisance law” means in practice
Table 3 (§5) instantiates 𝑄𝑛 and Σtask for seven nuisance families. Despite different modalities, all seven are the same statistical object with one loss template when Σ′ is chosen to match. ^ (𝑘) under assumption 𝐴𝑘 . Estimability. Σtask is never observed directly; each block supplies Σ task When 𝐴𝑘 is wrong or the spectrum has no gap (Office-31), the estimator is misaligned and matched pmh need not win—that is a predicted failure, not noise in the method. Formal rates: Lemmas A.8– A.14 (§5). ^ (𝑘) from row 𝑘 of Table 3 Matched training (one recipe). Specialising Eq. (4) with Σ′ = Σ task ^ (𝑘) ). Seven estimators, one loss gives the matched loss used in every block: ℒtask + 𝜆 Tr(𝐽𝜑⊤ 𝐽𝜑 Σ task template, one population object Σtask . §4 governs match vs. mismatch; §5 governs estimation; §8 tests all seven families.
3
Background: correlated nuisance, isotropic pmh, and the blind spot
We begin from the geometric blind spot of supervised learning under label-correlated nuisance: ERM cannot drive deployment drift to zero even at infinite capacity. Within the quadratic Jacobian penalty family, isotropic noise is the unique deployment-agnostic choice; standard adversarial training does not implement it. The five results below are proved in Appendix A.1; the Gaussian model is Remark A.1. 6
Notation bridge. Table 1 maps CORAL, augmentation, VAT/Jacobian 𝐿2 , PGD-AT, IRM/GroupDRO, and rlhf-style constraints to (𝐴𝑘 , Σ′ ) rows. In every case the design question is whether range(Σ′ ) covers range(Σtask ) (§4, §5). How to read the five results. Think of four layers: (i) ERM must keep sensitivity along nuisance-like directions (Theorems 3.1/3.2); (ii) if you do not know which directions matter, the only fair isotropic probe is Σ𝛿 = 𝜎 2 𝐼 (Proposition 3.3); (iii) PGD does not implement that probe and can distort geometry (Corollary 3.4); (iv) training can cap the penalty without tuning 𝜆 ^ task . (Proposition 3.5). The matching principle (§4) is layer (v): when Σtask is identifiable, set Σ′ = Σ Theorem 3.1 (ERM geometric incompleteness (Gaussian)). In the linear-Gaussian model with nuisance–label correlation 𝜌 > 0, any minimiser 𝜑*𝜃 of population MSE with 𝐿-Lipschitz decoder satisfies 2 2 ˜ 𝑄 (𝜑*𝜃 , 𝜎) ≥ 𝜎 𝜌 , 𝐷 𝐿2 independent of model capacity and dataset size. Corollary 3.2 (Strictly proper losses). For any strictly proper scoring rule ℒ, correlated-nuisance ˜ 𝑄 (𝜑* , 𝜎) ≥ 𝜎 2 𝐶 ′ (𝑃, ℒ)/𝐿2 , where 𝐶 ′ is the Bregman-gap distribution 𝑃 , and Lipschitz constant 𝐿, 𝐷 𝜃 constant Δ(𝑃, ℒ) = E𝑥 [𝑑𝜓 (𝑝(𝑦 | 𝑥) ‖ 𝑝(𝑦 | 𝑠(𝑥)))]. Proposition 3.3 (Isotropy and uniform Frobenius control). Among zero-mean 𝛿 with Cov(𝛿) = Σ𝛿 , the identity Tr(𝐽𝜑⊤ 𝐽𝜑 Σ𝛿 ) = 𝜎 2 ‖𝐽𝜑 ‖2𝐹 holds if and only if Σ𝛿 = 𝜎 2 𝐼. Corollary 3.4 (PGD training remains anisotropic). Adversarial training under bounded PGD shrinks ‖𝐽𝜑 𝛿^* ‖ but does not enforce isotropic Jacobian shrinkage; trajectory tdi@0 can worsen even as ‖𝐽‖𝐹 drops (T7B: PGD-AT TDI 1.506 vs. matched 0.870; §6, §8.7). Proposition 3.5 (Capped pmh fixed point). If the pmh penalty is capped at cap · ℒtask , the steadystate pmh fraction is 𝑓 = cap/(1 + cap) with no 𝜆 tuning required. Together, these results are the floor: ERM has unavoidable drift (Theorem 3.1); isotropic noise is the unique direction-agnostic penalty (Proposition 3.3); PGD is not a substitute for matched geometry (Corollary 3.4); the cap fixes the training balance (Proposition 3.5). What this paper adds. When Σtask is estimable, which Σ′ to use is no longer a modelling choice but a closed-form matching problem (§4–5), tested on thirteen blocks (§8; protocols in Appendix B).
3.1
Prior methods and unification attempts
Prior work unifies domain adaptation [4–6], transfer surveys [10, 17], metric learning [3], and information bottlenecks [1, 16]—but typically with bounds or narratives, not a matrix-valued training prescription, pre-registered falsification controls, or failure modes named before experiments. Table 1 maps seven standard methods to an implicit Σ′ , the matching assumption 𝐴𝑘 , a predicted failure when 𝐴𝑘 fails, and the block that tests it (§4 develops the theory; §4.8 fixes three quantitative checks). Reading the table. Each row is a structural identification: CORAL’s moment matching is the 𝐴4 Gram estimator; augmentation modes yield 𝐴3 ; PGD deltas yield 𝐴7 . Theorem 4.2 upgrades this from analogy to necessity: any quadratic Jacobian penalty that zeros deployment drift on all task directions must cover range(Σtask ). Rewrites for CORAL, PGD-AT, and augmentation appear in §4.3. 7
Table 1: Seven methods as implicit estimators of Σtask . Columns: regularised directions, assumption 𝐴𝑘 , predicted failure when 𝐴𝑘 is violated, and empirical block. Method
Implicit Σ′
Matched under
Predicted mode
failure Block
⋆ Cov(𝛿^PGD )
𝐴7 , PGD-delta Range too narrow; T7B nuisance clean accuracy collapses Cross-domain 𝐴4 , hierarchical Marginal eigengap T1 (Officefeature Gram domain shift, low (Wedin); fails when 31), T4A shift is high-rank rank 𝐴4 with environ- Spurious cor- — (scope) IRM / Group- Perment labels relation (labelDRO environment changing); out of penalty cov. scope ∑︀ ⊤ 1 Data augmen- 𝐾 𝛽 𝛽 𝐴 , finite-mixture Test-time corrup- T2A, T3A, 3 𝑘 𝑘 𝑘 tation (aug. delta photometric/oc- tion outside aug. T3B cov.) clusion family Mahalanobis Within-class 𝐴1 , low-rank sub- Signal contamina- T1 (oracle, ^ at low Office-31) metric learning scatter 𝑆𝑊 space nuisance tion of 𝑊 eigengap Jacobian reg. / 𝐼 or random 𝐴2 , isotropic Wrong-𝑊 collapses T7B, T6B VAT rank-𝑟 proj. acquisition to isotropic; no win (Lemma 4.12) over iso rlhf style / Style-pair 𝐴7 , stylistic align- Preference signal T7A KL-anchored representation ment nuisance aligned with style DPO Gram (sycophancy)
Adversarial training (PGD) CORAL
4
The matching principle
This section states the matching-principle results (Theorems 4.1, 4.2, 4.9, 4.11; Lemmas 4.12, 4.10; Corollaries 4.13/4.14). Each has a short intuition paragraph before the formal statement; proofs are in Appendix A. Figure 2 summarises the map (row 1: G→A→B→A⋆ ; row 2: falsification controls C, E, diagnostic F, estimators D1–D7). Reading order. Logic: G → A → B → A⋆global (Fig. 2); we state A before G so the optimum is concrete first. Alongside: Lemma 4.12, Corollaries 4.13/4.14. Worked identifications (CORAL / PGD-AT / augmentation) and §8 follow.
4.1
Theorem A: the matched penalty is the right answer
Reader map. Part (i): cover range(Σtask ) or drift stays Θ(1). Part (ii): within the matched range, allocate via cube-root water-filling (QM9 tradeoff when 𝜆 is finite). Proof: Appendix A.2. Intuition. Imagine the deployment nuisance moves your input along a small set of directions in input space. If the encoder’s Jacobian has energy along those directions, deployment perturbations shift the embedding and the model’s prediction drifts. Theorem A says: in the linear-Gaussian model, the way to eliminate this drift is to penalise the Jacobian along exactly those directions, and the penalty has to cover all of them or it fails completely.
8
Theorem 4.1 (Matched-Σ optimality; linear-Gaussian). For the linear-Gaussian regression model (Remark A.1) with deployment-nuisance covariance Σtask , the pmh-regularised minimiser 𝑤(Σ′ ) = (𝐼 + 2𝜆Σ′ )−1 𝑣 satisfies 𝜆→∞ ˜ 𝑄 (𝑤(Σ′ )) − 𝐷 −−→ 0
range(Σ′ ) ⊇ range(Σtask ).
⇐⇒
˜ 𝑄 (𝑤(Σ′ )) converges to a constant independent of 𝜆. Within the When this range condition fails, 𝐷 range-matched class, the trace-constrained optimum is the cube-root water-filling allocation 𝜇*𝑖 ∝ (˜ 𝑣𝑖2 𝜆𝑖 )1/3 , with the proportional rule Σ′ ∝ Σtask recovering it under rotation-invariant regressor energy. What this rules out. Missing a nuisance direction in range(Σtask ) cannot be fixed by increasing 𝜆; the drift floor is structural (same geometry as the centre panel of Figure 1). Allocation geometry: matched arms cover range(Σtask); isotropic spreads, wrong-W lands in signal directions (B)~Allocation rules on a common budget ∑μi = B
(A)~Population spectrum: Σtask and regressor energy 1.0
0.6
0.4
0.2
1
2
3
4
5
6 7 8 9 Spectral index i
10
11
12
13
Allocation μi (trace-constrained)
signal
0.8 Magnitude
Matched (proportional, Σ0 ∝ Σtask )
ṽ 2i (regressor energy, normed) λi (eigvals of Σtasksubspace )
0.0
i
0.35 Cube-root water-filling (Thm.~A(ii))
0.30
Isotropic (Lemma~C, Σ0 = σ 2 I) Wrong-W (signal-aligned, Cor.~E ⋆ ) wrong-W lands here
0.25 0.20 0.15 0.10 0.05 0.00
1
2
3
4
5
6 7 8 9 Spectral index i
10
11
12
13
Figure 3: Theorem 4.1(ii): where to put trace budget 𝑖 𝜇𝑖 = 𝐵. Top: eigenvalues of Σtask (blue) vs. regressor energy (orange); peach band = signal outside the nuisance subspace. Bottom: proportional Σ′ ∝ Σtask (default recipe), cube-root optimum, isotropic pmh, wrong-𝑊 on signal (Cor. 4.14). Range must cover nuisances; shape within range matters less (Theorem 4.11). ∑︀
4.2
Theorem G: the range condition is not optional
Intuition. Theorem A says range matching is sufficient. Theorem G says no other quadratic Jacobian penalty can succeed without doing the same thing. Across all of CORAL, adversarial training, Mahalanobis metric learning, augmentation, and Jacobian regularisation, the only way to drive deployment drift to zero on every task-relevant direction is to cover range(Σtask ). Theorem G 4.2 (Necessity of range(Σtask )). Let 𝐴 ⪰ 0 define any quadratic Jacobian regulariser ˜ 𝑄 (𝑤𝜆 (𝐴)) → 0 for every effective regressor 𝑣 ∈ range(Σtask ), then ℛ𝐴 (𝜑) = E𝑥 [Tr(𝐽𝜑⊤ 𝐽𝜑 𝐴)]. If 𝐷 range(𝐴) ⊇ range(Σtask ). What this rules out.
No quadratic Jacobian penalty can zero drift without covering range(Σtask ).
9
4.3
Three consequences of Theorem G: CORAL, PGD-AT, and augmentation as matched pmh
Why this subsection exists. Readers often treat CORAL, PGD-AT, and heavy augmentation as unrelated tricks. The point of the rewrites below is not to claim we invented them, but to ^ (𝑘) and add a Jacobian penalty along it—so disagreements show they already estimate different Σ task between methods in §8 are disagreements about estimator quality and eigengap, not about whether a shared principle exists. Theorem G says every successful quadratic Jacobian regulariser covers range(Σtask ). We make that claim concrete on the three most-cited methods: rewrite each loss until its implicit Σ′ is visible, then read off the Lemma D𝑘 failure mode that block tests. Deep CORAL is matched pmh along the cross-domain Gram. The Deep CORAL objective [15] adds a feature-level term ℒCORAL = ‖𝐶𝑆𝜑 − 𝐶𝑇𝜑 ‖2𝐹 where 𝐶𝑆𝜑 , 𝐶𝑇𝜑 are source and target feature covariances. Linearise the encoder around the source mean: 𝜑(𝑥) ≈ 𝜑(𝜇𝑆 ) + 𝐽𝜑 (𝜇𝑆 )(𝑥 − 𝜇𝑆 ). Then 𝐶𝑆𝜑 − 𝐶𝑇𝜑 ≈ 𝐽𝜑 (Cov𝑆 (𝑥) − Cov𝑇 (𝑥)) 𝐽𝜑⊤ = 𝐽𝜑 Σdom 𝐽𝜑⊤ , with Σdom = Cov(𝑥𝑇 − 𝑥𝑆 ) the cross-domain ^ (4) . By cyclic invariance of the trace and the Gram — which is exactly the Lemma D4 estimator Σ task
operator-norm bound,
‖𝐽𝜑 Σdom 𝐽𝜑⊤ ‖2𝐹 = Tr (𝐽𝜑 Σdom 𝐽𝜑⊤ )2 (︀
)︀
≤ ‖𝐽𝜑 ‖2op ‖Σdom ‖op Tr(𝐽𝜑⊤ 𝐽𝜑 Σdom ).
Up to bounded operator-norm factors, CORAL minimises the matched-pmh penalty along Σdom — a second-moment surrogate of matched pmh under 𝐴4 (Lemma A.11). When the cross-domain spectrum is nearly rank-deficient (Office-31, §8.1), the 𝐴1 /D1 eigengap condition fails and both CORAL and matched pmh degrade, as predicted. PGD adversarial training is matched pmh along the gradient-direction Gram. PGD-AT [︀ ]︀𝐾 [9] samples 𝛿^⋆ (𝑥) = Π‖𝛿‖≤𝜖 𝛼 sign(∇𝑥 ℒ) at each step and trains on ℒ(𝜃; 𝑥 + 𝛿^⋆ ). Taylor-expand the per-sample loss along 𝛿: ℒ(𝜃; 𝑥 + 𝛿) = ℒ(𝜃; 𝑥) + 𝛿 ⊤ ∇𝑥 ℒ + 21 𝛿 ⊤ 𝐽𝜑⊤ 𝐻𝜑 𝐽𝜑 𝛿 + 𝑂(‖𝛿‖3 ), where 𝐻𝜑 is the Hessian of the decoder. Averaging over 𝛿^⋆ at adversarial radius 𝜖 gives an expected 2 ^ PGD )] with Σ ^ PGD = Cov(𝛿^⋆ ) — the loss whose first non-trivial Jacobian term is 𝜖2 E𝑥 [Tr(𝐽𝜑⊤ 𝐻𝜑 𝐽𝜑 Σ ^ PGD (Lemma A.14), with 𝐴7 / D7 estimator. PGD-AT therefore implements matched pmh along Σ two caveats: (i) decoder weighting via 𝐻𝜑 is not the proportional allocation of Theorem 4.1(ii) — the Cor. 3.4 dissociation (T7B: −14.8 pp clean vs. baseline, worse tdi than PGD-delta pmh). ^ PGD is low-rank; explicit pmh with a full PGD-delta estimator (T7B matched arm) can match (ii) Σ robustness with better clean accuracy. Data augmentation is matched pmh along the augmentation delta Gram. A labelpreserving augmentation pipeline 𝒜 = {𝑎1 , . . . , 𝑎𝐾 } trains on ℒ(𝜃; 𝑎𝑘 (𝑥)) for 𝑘 ∼ Unif{1, . . . , 𝐾}. ^ aug = 1 ∑︀ E𝑥 [𝛿𝑘 𝛿 ⊤ ]. Define the augmentation delta 𝛿𝑘 = 𝑎𝑘 (𝑥) − 𝑥 and its mixture covariance Σ 𝑘 𝑘 𝐾 A second-order Taylor expansion gives the expected loss ^ aug )] + 𝑂(‖𝛿‖3 ). E𝑥,𝑘 [ℒ(𝜃; 𝑎𝑘 (𝑥))] = E𝑥 [ℒ(𝜃; 𝑥)] + 21 E𝑥 [Tr(𝐽𝜑⊤ 𝐻𝜑 𝐽𝜑 Σ ^ aug (Lemma A.10). Augmentation wins The leading regularisation term is matched pmh along Σ when test corruptions lie in span{𝛿𝑘 } (T2A: +4.3 pp on ImageNet-C) and fails when they are orthogonal—the scope condition of 𝐴3 . 10
Takeaway. CORAL, PGD-AT, and augmentation are not separate mechanisms once linearised; ^ (𝑘) . Theorem 4.2 says every successful quadratic Jacobian they are matched pmh along different Σ task penalty must do the same. Empirical differences reduce to estimator quality and eigengap (§8).
4.4
Theorem A⋆ : the same dichotomy holds for deep encoders
Intuition. Theorems 4.1 and 4.2 are linear-Gaussian. Theorem 4.9 lifts the same range dichotomy to deep encoders at the global minimum under four assumptions (R, C♭ , E, I) below. (E) sounds abstract—“there exists 𝜃0 with zero PMH penalty”—but Lemma 4.10 constructs it for MLPs, CNNs, ViTs, Transformers, and GNNs (Table 2). Assumption 4.3 ((R) Regularity). The encoder 𝜑𝜃 is differentiable a.e. with ‖𝐽𝜑 (𝑥)‖op ≤ 𝑀 for all 𝑥 in the support of 𝑃𝑋 , and the regularised loss ℒ𝜆pmh(Σ′ ) attains a global minimum 𝜃𝜆glob . Assumption 4.4 ((C♭ ) Directional encoding necessity). Along any 𝑞 ∈ range(Σtask ) with label correlation, the global minimiser cannot suppress 𝐽𝜑 along 𝑞 below the blind-spot scale 𝜌2𝑞 /(2𝐿2 ) (Theorem 3.1, directional form). Proposition 4.5 ((C♭ ) in the linear model). In the linear-Gaussian correlated-nuisance model (Remark A.1), at any population minimiser 𝜑*𝜃 of a strictly proper task loss with 𝜌 > 0, and for every unit 𝑞 ∈ range(Σtask ), [︁ ]︁ 𝜌2𝑞 E𝑥 𝑞 ⊤ 𝐽𝜑*𝜃 (𝑥)⊤ 𝐽𝜑*𝜃 (𝑥) 𝑞 ≥ 2 , 𝐿 so (C♭ ) holds before any pmh regularisation is applied. The same bound holds for any strictly proper loss via Corollary 3.2. Remark 4.6 (When to expect (C♭ ) for deep encoders). Theorem 4.9 uses (C♭ ) only in the necessary direction: if range(Σ′ ) ̸⊇ range(Σtask ), the global pmh minimiser cannot eliminate drift along a missing direction by an unpenalised Jacobian shortcut. For deep networks we treat (C♭ ) as the directional lift of Theorem 3.1/Corollary 3.2: it is expected when (i) the task loss is strictly proper, (ii) deployment nuisance is label-correlated in the sense of §2, and (iii) the encoder Jacobian along 𝑞 ∈ range(Σtask ) is not already driven to zero by the task loss alone. A full characterisation of (C♭ ) on non-convex landscapes is Open Question (O) (Table 8); experiments are consistent with the linear prediction (e.g. T7B PGD-AT: high robustness, poor isotropic TDI). Assumption 4.7 ((E) Expressivity). For the encoder class under study, there exists 𝜃0 with E𝑥 [Tr(𝐽𝜑⊤𝜃 𝐽𝜑𝜃0 Σ′ )] = 0 and finite task loss ℒtask (𝜃0 ) < ∞. 0
Assumption 4.8 ((I) Identifiability on range(Σtask )). Every direction in range(Σtask ) that carries regressor energy in the linearised model has non-zero coupling to the deployment drift functional: if ˜ 𝑄 cannot vanish along 𝑞 unless the Jacobian is suppressed 𝑞 ∈ range(Σtask ) and 𝑞 ⊤ Σtask 𝑞 > 0, then 𝐷 along 𝑞. Theorem A⋆global 4.9 (Range-matching dichotomy for deep encoders). Under (R), (C♭ ), (E), (I) and at the global minimum 𝜃𝜆glob of ℒ𝜆pmh(Σ′ ) :
˜ 𝑄 (𝜃glob ) = 𝑂(1/𝜆) → 0, range(Σ′ ) ⊇ range(Σtask ) =⇒ 𝐷 𝜆 ˜ 𝑄 (𝜃glob ) = Θ(1) along the missing direction. range(Σ′ ) ̸⊇ range(Σtask ) =⇒ 𝐷 𝜆 11
Table 2: 𝒞NRP : architectures with constructive (E) via Lemma 4.10. Family
First-layer map 𝑊 (1)
MLP / ConvNet / ResNet ViT / Transformer LLM GNN + input MLP; Whisper stem
Linear or conv1 on inputs Patch or token embedding Feature projection; spectrogram conv
Lemma 4.10 (D-ML; constructive verification of (E)). If 𝜑𝜃 (𝑥) = 𝜓(𝑊 (1) 𝑥 + 𝑏; 𝜃rest ) with nonresidual input projection 𝑊 (1) , there exists 𝜃0 with zero PMH trace along Σ′ and finite task loss. Class 𝒞NRP (Table 2) includes MLPs, CNNs, ResNets, ViTs, Transformers, GNNs, and Whisperstyle encoders. Open endpoint (O). Theorem 4.9 is a global-minimum statement. Whether gradient descent reaches it is assumption (O); experiments are consistent but a proof is open (§10, Appendix A.5).
4.5
Theorem B: missing the range is catastrophic; mis-allocating is cheap
Intuition. Theorem A asks two separate questions: (1) “Did you penalise the right subspace?” and (2) “Given the right subspace, did you put enough penalty mass on each nuisance direction?” Wrong range means Σ′ misses a direction of Σtask —drift stays Θ(1) no matter how large 𝜆 is (Figure 1, right). Wrong allocation means the range is correct but Σ′ is poorly shaped inside it—extra drift is only Θ(𝜆−3 )‖Σ′ − Σ⋆ ‖2𝐹 and vanishes relative to Θ(𝜆−2 ) drift as 𝜆 → ∞ (Figure 3). ˜ 𝑄 (𝑤(Σ′ )) − 𝐷 ˜ 𝑄 (𝑤(Σ⋆ )) = Θ(1), indeTheorem 4.11 (Mismatch cost). For range mismatch, 𝐷 pendent of 𝜆. For allocation mismatch within the range-matched class, the excess drift is at least Θ(𝜆−3 ) ‖Σ′ − Σ⋆ ‖2𝐹 , vanishing relative to the absolute drift Θ(𝜆−2 ) as 𝜆 → ∞. Practice.
4.6
Estimate range(Σtask ) first; use Σ′ ∝ Σtask as default (all blocks do).
Lemma C: random penalty directions reduce to isotropic
Intuition. The usual wrong-𝑊 ablation (random rank-𝑟 projector) is isotropic pmh at scale 𝑟/𝑑𝑥 in expectation (Lemma 4.12), not a separate hypothesis class. Lemma 4.12 (Stiefel-manifold equivalence). If 𝑈 ∈ R𝑑𝑥 ×𝑟 is uniform on the Stiefel √︀ manifold, then ⊤ ⊤ E𝑈 [𝑈 𝑈 ] = (𝑟/𝑑𝑥 ) 𝐼. The deviation ‖𝑈 𝑈 − (𝑟/𝑑𝑥 )𝐼‖op concentrates at rate 𝑂( 𝑟 log 𝑑𝑥 /𝑑𝑥 ). Practice.
4.7
Matched must beat both isotropic and wrong-𝑊 ; beating only wrong-𝑊 is inconclusive.
Corollaries E and E⋆ : penalising along the signal hurts
Intuition. The wrong-𝑊 control is the random-direction ablation. The signal-𝑊 control is the wrong-direction ablation: deliberately penalise along the signal axis (the directions that the label depends on). Corollaries E/E⋆ say this provably increases task risk—at rate Ω(𝜌4 ) when the loss is smooth at its minimum, and Ω(𝜌2 ) when the loss has a non-vanishing subgradient (SVM hinge, finite-horizon cross-entropy, max-margin segmentation). This is why the keyword-pmh arm in T5B and the isotropic-pixel arm in T4B fail; in both cases the regulariser is incident on signal directions. Corollary 4.13 (Smooth task loss). If 𝑅(𝜑) is smooth at 𝜑* with positive Hessian along signal 𝑠 ′ and 𝑠⊤ Σ′ 𝑠 ≥ 𝜌2signal > 0, then 𝑅(𝜑pmh(Σ ) ) − 𝑅(𝜑* ) ≥ 𝑐𝑅 𝜌4signal . 12
Corollary 4.14 (Margin-active task loss). Under a non-vanishing subgradient along 𝑠 (hinge, finitehorizon cross-entropy, max-margin segmentation), the leading penalty is Ω(𝜌2signal ). Mapped failures (§8.8). Not every negative result is a surprise: Office-31 is a Lemma A.8 eigengap failure; Cityscapes iso-pixel and T5B keyword-pmh are Corollary 4.14 signal penalties; QM9 at large noise is a Theorem 4.1(ii) clean–robust tradeoff when the allocation cannot be monotone in both.
4.8
Three quantitative predictions, three verifications
A claim is falsifiable only if the wrong ablations are specified in advance. We use three controls on every block: isotropic pmh (no directional information), wrong-𝑊 (random rank-𝑟 subspace), and signal-aligned pmh (penalise the label direction). Lemma 4.12 predicts wrong-𝑊 should track isotropic; Corollary 4.14 predicts signal-pmh should hurt; Corollary 3.4 predicts PGD-AT should not sit on the same geometry–accuracy curve as matched pmh. The box below records the sharpest numerical checks (full tables: Appendix B). Three predictions fixed before experiments 1. Wrong-𝑊 ≈ isotropic (Lemma 4.12, T7B). Predicted 𝐷𝑁 /𝐷𝑆 gap ≤ 5%; observed 2.98 vs. 3.11 (4.2%). 2. Signal-pmh hurts (Cor. 4.14, T5B). Predicted below B0; observed rename_bacc_ratio 0.830 → 0.738. 3. PGD-AT off the geometry–accuracy Pareto (Cor. 3.4, T7B). PGD@4 44.8% but clean 64.6% vs. B0 79.4% (−14.8 pp); PGD-delta pmh TDI 0.870 vs. PGD-AT 1.506.
5
Seven estimators of Σtask
Given Σtask , the remaining design choice is how to estimate it from data. Implementers: choose 𝐴𝑘 via §5.1, pick the row in Table 3, follow §7, run wrong-𝑊 and signal-𝑊 controls. Readers of results: §8 and Appendix B. If the eigengap is small, expect the Office-31 pattern (Lemma D1) rather than a silent bug. Formal statements and rates: Appendix A.9. When several nuisances co-occur, compose penalties additively (e.g. T2A + T3B).
5.1
Practitioner guide: choosing 𝐴𝑘 and pre-flight checks
Choosing 𝐴𝑘 (symptom → row). Use Table 4 after inspecting how deployment shifts inputs without relabelling. When two stories apply, use the dominant deployment shift for the matched arm and add the second penalty additively (Step 2′ below). Multiple nuisances. When acquisition noise and photometric modes both move deployment inputs (Type 2+Type 3), use ^ (2) ) + 𝜆3 Tr(𝐽 ⊤ 𝐽 Σ ^ (3) ), ℒ = ℒtask + 𝜆2 Tr(𝐽 ⊤ 𝐽 Σ with separate caps (Prop. 3.5) per term. T2A+T3B is the canonical composition in the programme. Eigengap pre-flight (rank-𝑟 estimators, Lemmas D1/D3/D4/D7). Let 𝐶^ be the sample second-moment matrix your estimator targets (cross-domain deltas, aug. Gram, style Gram, etc.) 13
Table 3: Master index: nuisance family, deployment law, assumption 𝐴𝑘 , Lemma D𝑘 estimator, and test blocks. Rates and proofs: Appendix A.9. Family
𝐴𝑘 D 𝑄𝑛 / Σtask
Subspace
1
Isotropic Photometric Domain
2 3 4
Compositional 5 Temporal
6
Adv. / align.
7
^ task Estimator Σ
Blocks
1 𝑛=𝑊 𝜂; 𝑊 𝑊 ⊤ Top-𝑟 cross-domain T1 SVD 2 𝒩 (0, 𝜎 2 𝐼) 𝜎 ^ 2 𝐼∑︀ T2A, T2B 1 ⊤ E[𝛽 𝛽 ] T3A, T3B 3 aug. modes 𝛽𝑘 𝐾 𝑘 𝑘 𝑘 4 paired 𝑥𝑇 −𝑥𝑆 Per-layer cross-domain T4A, T4B Gram 5 coord. spike on Cov. on nuisance block T5A, T5B 𝒱𝑛 6 Δℎ along se- Content-residual / sen- T6A, T6B quence sor scatter ⋆ ^ 7 PGD 𝛿 or PGD-delta / style-pair T7A, T7B style shift Gram
^ ^ and 𝛾𝑟 := 𝜆𝑟 (𝐶)/𝜆 𝑟+1 (𝐶) the rank-𝑟 ratio gap. Lemma A.8 (Appendix A.9) gives ‖Π𝑊 ^ − Π𝑊 ‖𝐹 ≲ ^ 2‖𝐶 − 𝐶‖op /𝛾 with 𝛾 = 𝜆𝑟 (𝐶) − 𝜆𝑟+1 (𝐶) at population. Practical rule (before training): 1. Compute 𝛾𝑟 on held-out deployment pairs / nuisance samples. 2. Pass if 𝛾𝑟 ≥ 1.2 and 𝜆𝑟+1 /𝜆𝑟 ≤ 0.95. 3. Marginal if 1.0 < 𝛾𝑟 < 1.2 (Office-31 used 𝛾𝑟 ≈ 1.03 at rank 32): expect Lemma D1 failure— report CORAL/second-moment baselines, do not treat a matched-arm loss as definitive. 4. Fail if 𝛾𝑟 ≈ 1: fall back to 𝐴2 isotropic pmh or a non-subspace baseline; wrong-𝑊 should track isotropic (Lemma 4.12). 2 Sample size. Require √ 𝑁 ≳ 𝑐 𝑟/𝛾 deployment pairs for 𝑐 ≈ 4 when using the Lemma A.8 rate ^ ‖𝐶 − 𝐶‖op = 𝑂𝑃 (𝜎1 / 𝑁 ).
D1 (𝐴1 , subspace). Requires 𝑛 ≈ 𝑊 𝜂 with signal orthogonal to range(𝑊 ). Recovery needs a ^ unreliable and matched penalty may lose spectral gap (Wedin/Davis–Kahan); marginal gap ⇒ 𝑊 to a stronger second-moment method (Office-31 pattern). Verdicts: §8.1. D2 (𝐴2 , isotropic). When deployment noise has no preferred direction, the matched penalty is 𝜎 ^ 2 𝐼—the only fair direction-agnostic choice (Proposition 3.3), not a fallback. Wrong-𝑊 should track isotropic (Lemma 4.12). Verdicts: §8.2. D3 (𝐴3 , photometric/occlusion). Finite label-preserving modes {𝛽𝑘 }; estimator is their empirical second moment. Fails when test corruptions lie outside span{𝛽𝑘 }. Verdicts: §8.3. D4 (𝐴4 , hierarchical domain). Shift acts at multiple representation depths; per-layer crossdomain Gram is the matched object (pixel-level isotropic pmh is a mis-specified estimator). Verdicts: §8.4. D5 (𝐴5 , compositional). Known coordinate partition into signal vs. nuisance blocks; estimator is empirical covariance on the nuisance block only. Penalising the signal block is Corollary 4.14 (keyword-pmh arm). Verdicts: §8.5. 14
Table 4: Heuristic map from deployment symptoms to assumption 𝐴𝑘 (confirm with wrong-𝑊 / signal-𝑊 controls). Deployment symptom
𝐴𝑘
Typical block
Known / estimable low-rank subspace (digits, domains) No preferred direction (sensor/corruption noise) Finite aug. / photometric modes you train on Multi-layer domain shift (texture → semantics) Nuisance in named coordinates (atoms, tokens) Label-constant temporal / speaker / sensor drift Learned deltas (PGD) or style rewrites (alignment)
𝐴1
T1
𝐴2
T2
𝐴3
T3
𝐴4
T4
𝐴5
T5
𝐴6
T6
𝐴7
T7
D6 (𝐴6 , temporal/sequential). Label-constant content with speaker/sensor variation; contentresidual or sensor scatter estimates Σtask . Supervision can improve task metrics without fixing geometry (§7.2). Verdicts: §8.6. D7 (𝐴7 , learned nuisance). PGD-delta Gram (vision) or style-pair Gram (alignment). Better subspace estimates should improve matched robustness monotonically; PGD-AT may trade clean accuracy for adversarial robustness (Corollary 3.4). Verdicts: §8.7, §8. Summary. Lemmas D1–D7 are conditional consistency results—eigengap failure (D1), signal ^ (D7) each predict a named negative before the block runs. leakage (D5), or missing style axis in 𝑊 §6 defines geometry probes; §7 is the training procedure.
6
Mechanistic diagnostics: tdi and directional drift
˜ 𝑄 (𝜑) (Eq. (3)). This section defines three post-hoc The matching principle is stated in terms of 𝐷 probes used in every block of §8: trajectory tdi (isotropic input noise, label-free), class-layout ^ TDIcls 0 or Style tdi (label- or style-structured embeddings), and 𝐷𝑁 /𝐷𝑆 (directional drift when 𝑊 is available). They track geometry, not accuracy; when geometry and task metrics disagree, that is a predicted dissociation (§7.2). tdi complements accuracy and ‖𝐽𝜑 ‖𝐹 : it probes fresh isotropic input noise on a fixed encoder (orientation-sensitive), whereas CKA compares two fixed clouds and task accuracy conflates decoder and geometry. Class-layout tdi is one-sided only (Prop. 6.2); never report it without trajectory tdi or 𝐷𝑁 /𝐷𝑆 when subspaces are available.
15
Diagnostic family at a glance (three different questions) Probe
When to use
Trajectory tdi@0
Isotropic input Does the encoder wiggle under un˜ 𝑄 when noise on deep nets structured 𝛿? Tracks 𝐷 (Types 2, 6, 7B) Σtask = 𝜎 2 𝐼. You have class la- Is class structure tight vs. separated? bels on a probe set Not a substitute for trajectory tdi. (Types 4A, 7B table) ^ suppressed vs. orYou estimated a nui- Is drift along 𝑊 ^ sance subspace 𝑊 thogonal directions? Falsifies wrong(Types 3–7) 𝑊 (Lemma 4.12). LLM style rewrites, Does hidden-state geometry preserve same content (T7A) under preference-correlated style?
TDIcls 0
𝐷𝑁 /𝐷𝑆 Style tdi
What it answers
Common mistake: reporting low class-layout tdi as “low drift” without trajectory tdi or 𝐷𝑁 /𝐷𝑆 — collapsed or trivial layouts can score well (Prop. 6.2, one-sided only).
6.1
From deployment drift to a measurable index
˜ 𝑄 from §2 (Eqs. (3) and the nonlinear definition above it). For 𝑛 = 𝛿 ∼ 𝒩 (0, 𝜎 2 𝐼), We use 𝐷𝑄 and 𝐷 ˜ 𝑄 + 𝑂(𝜎 4 ), so trajectory tdi at small 𝜎 reports the same isotropic Lemma A.3 gives 𝐷𝑄 = 𝐷 sensitivity that Theorems 3.1 and 4.1 bound—not task loss.
6.2
Trajectory Deviation Index (primary, label-free)
Definition 6.1 (Trajectory Deviation Index). Let 𝜑(1:ℓ) denote the representation after the first ℓ layers of a depth-𝐿 encoder 𝜑. For perturbation strength 𝜎 > 0 and 𝛿 ∼ 𝒩 (0, 𝜎 2 𝐼𝑑 ), the Trajectory Deviation Index is 𝐿 E 𝑥,𝛿 1 ∑︁ TDI(𝜑, 𝜎) := 𝐿 ℓ=1
[︁⃦ ⃦ ]︁ ⃦𝜑(1:ℓ) (𝑥 + 𝛿) − 𝜑(1:ℓ) (𝑥)⃦2 2 [︁⃦ . ⃦2 ]︁ E𝑥 ⃦𝜑(1:ℓ) (𝑥)⃦
(5)
2
Lower is better: an encoder with small, layer-balanced displacement under isotropic input noise scores low; an encoder that “wiggles” strongly along many directions at many depths scores high. Clean-input limit. We write TDI@0(𝜑) := lim𝜎→0+ TDI(𝜑, 𝜎) and estimate it at 𝜎 = 0.01 in all experiments (well below training noise; Taylor remainder 𝑂(𝜎 2 ) gives < 0.1% relative error at this scale). Layer-averaging is used for deep ViTs and ResNets (Types 2, 4, 7B); a single final-layer probe is used when only one embedding is available (e.g. CLS or graph readout). ˜ 𝑄. Link to 𝐷
At small 𝜎, expanding 𝜑(𝑥 + 𝛿) − 𝜑(𝑥) ≈ 𝐽𝜑 (𝑥)𝛿 in Eq. (5) yields [︁
2
]︁
𝐿 E𝑥 ‖𝐽 (1:ℓ) (𝑥)‖ 𝜎 2 ∑︁ 𝜑 𝐹 ˜ 𝑄 (𝜑) [︀ ]︀ ∝ 𝜎 2 𝐷 TDI(𝜑, 𝜎) ≈ 2 (1:ℓ) 𝐿 ℓ=1 E𝑥 ‖𝜑 (𝑥)‖2
2 (isotropic Σtask = 𝜎task 𝐼),
(6)
so TDI measures exactly the isotropic path-length distortion that Theorem 3.1 bounds, normalised by representation scale. It is a post-hoc, label-free probe complementary to Eq. (4), not a substitute for accuracy or CKA. 16
6.3
Class-layout tdi0 (labeled probe)
When class labels are available on a held-out probe set, we report a second statistic that does not require injecting input noise, but measures how tightly each class cluster sits relative to betweenclass separation: 𝑑intra (𝜑) , (7) TDIcls 0 (𝜑) := 𝑑inter (𝜑) where 𝑑intra is the mean pairwise distance among same-class embeddings and 𝑑inter is the mean distance between class centroids (embeddings 𝐿2 -normalised; up to 200 samples per class). This is the probe used for T4A/T4B pixel-aligned tables and for T7B compute_tdi_cls; it coincides with the layout index reported alongside trajectory tdi in Type 2 replication code. Lower means tighter class structure relative to separation. Proposition 6.2 (Proposition F: one-sided layout envelope). If the linearised deployment drift ˜ 𝑄 (𝜑) → 0, then vanishes, 𝐷 TDIcls 0 (𝜑) ≤ Φ(𝑑between (𝜑)) , where 𝑑between is mean inter-class centroid distance and Φ : R≥0 → R≥0 is continuous and increasing, depending only on latent class geometry, not on 𝑄𝑛 . Proof sketch. Zero drift implies 𝜑(𝑥 + 𝑛) − 𝜑(𝑥) = 𝑜(‖𝑛‖) for 𝑛 in the deployment subspace, so any layout statistic computed under 𝑄𝑛 perturbation collapses to the clean embedding geometry, which is bounded by inter-class separation via standard discriminant arguments. Full proof in ˜ Appendix A.10. One-sided only: low TDIcls 0 does not imply low 𝐷𝑄 (collapsed classes can score artificially low); we never use it as the sole metric.
6.4
Directional drift 𝐷𝑁 , 𝐷𝑆
^ ∈ R𝑑𝑥 ×𝑟 is available (Types 3–7), we report mean feature When an estimated nuisance subspace 𝑊 ^ ) versus its orthogonal complement: drift under equal-energy probes in span(𝑊 [︁⃦
⃦ ]︁
⃦ 𝐷𝑁 (𝜑) := E𝑥,𝑧 ⃦𝜑(𝑥 + Π𝑊 ^ 𝑧) − 𝜑(𝑥) 2 ,
(8)
[︁⃦ ⃦ ]︁ 𝐷𝑆 (𝜑) := E𝑥,𝑧 ⃦𝜑(𝑥 + Π ^ ⊥ 𝑧) − 𝜑(𝑥)⃦ ,
(9)
𝑊
2
with 𝑧 ∼ 𝒩 (0, 𝜎 2 𝐼𝑑𝑥 ) and equal total noise power in both arms (Appendix B.13). The ratio 𝐷𝑁 /𝐷𝑆 ^ should match isotropic pmh (T7B: 2.98 vs. 3.11, 4.2% gap); matched tests Lemma 4.12: wrong-𝑊 ^ 𝑊 should suppress 𝐷𝑁 (T7B PGD-delta: 𝐷𝑁 /𝐷𝑆 = 0.19).
6.5
Style tdi (alignment blocks)
For Type 7A LLM alignment, labels are style variants of the same content rather than classes. Style tdi applies the same ratio as Eq. (7) to hidden states: within-prompt spread across six style rewrites divided by between-prompt centroid distance (96 prompts × 6 styles; lower is better). It is the geometry headline for the DPO arm in §8; behavioural sycophancy uses separate reward-model probes (protocol: Appendix B.12).
6.6
Reporting convention
Bridge to experiments. Every block in §8 reports at least one diagnostic from this section and ^ task is chosen before those numbers are computed. a task metric; the recipe in §7 is how Σ 17
Which variant where. Unless a table caption says otherwise: trajectory tdi@0 (Eq. (5)) for isotropic input-Gaussian probes on deep encoders (Types 2, 4B pixel probe, 6, 7B); class-layout TDIcls 0 (Eq. (7)) for labelled embedding tables (Types 4A, 7B cls column); Style tdi (Eq. (7) on style rewrites) for Type 7A alignment (Appendix B.12). Appendix B defers to this section for definitions; task captions name the variant explicitly when both appear in one block. We always report (trajectory or layout) tdi, optional 𝐷𝑁 /𝐷𝑆 , and the task metric separately. When geometry improves but accuracy does not (T6A: matched pmh best tdi, accent head best WER), that is a predicted dissociation, not a failure of either measurement.
7
Practice: a five-step recipe and the scope of the principle
^ task , add the trace penalty, cap it, and run The theory of §4–5 is constructive: pick 𝐴𝑘 , estimate Σ two controls. Figure 4 is the same pipeline; the numbered box below is the prose version. Step 1. Identify nuisance family 𝐴𝑘 ∈ {𝐴1 , . . . , 𝐴7 } ^ (𝑘) (Lemma D𝑘) Step 2. Estimate Σ task eigengap pre-flight (§5.1); else D1-type failure plug in
Step 3. Matched pmh ^ task )] ℒtask + 𝜆 E𝑥 [Tr(𝐽 ⊤ 𝐽 Σ cap
Step 4. Cap penalty: 𝑓 = cap/(1 + cap) 5a. Wrong-𝑊 random rank-𝑟
5b. Signal-𝑊 known signal axis
Lemma 4.12: ≈ isotropic pmh
Cor. 4.14: task metric < B0 Estimator key (Table 3): D1 cross-domain SVD • D2 𝜎 ^ 2 𝐼 • D3 aug-delta • D4 layer Gram • D5 coord. cov • D6 residual scatter • D7 PGD-delta / style Gram
Figure 4: Five-step recipe (flow). Steps 1–3: nuisance family 𝐴𝑘 → Lemma D𝑘 estimator → matched pmh loss. Step 4: cap (Prop. 3.5). Step 5: wrong-𝑊 and signal-𝑊 falsification arms with predicted outcomes (Lemmas 4.12, Cor. 4.14). Estimator key: Table 3.
18
The matching-principle recipe Step 1. Identify the nuisance family. Pick 𝐴𝑘 using Table 4 (§5.1); in most tasks the answer is unique. Hybrid nuisances compose additively (Step 2′ in §5.1). ^ task via Lemma D𝑘. Use the estimator from §5. Run the eigengap Step 2. Estimate Σ pre-flight in §5.1 before training; a marginal gap (𝛾𝑟 ≈ 1) is the named Office-31 failure mode, not noise. Step 3. Form the matched loss. Add the pmh penalty to the task loss, ^ task ℒtotal (𝜃) = ℒtask (𝜃) + 𝜆 E𝑥 Tr 𝐽𝜑⊤ 𝐽𝜑 Σ [︁
(︁
)︁]︁
,
approximating the trace with a finite-difference paired-view, multi-scale Gram, Hutchinson-trace, or margin-invariance surrogate (per-block choice: Appendix B). Step 4. Apply the cap. Cap the pmh term with Proposition 3.5: ℒpmh ≤ cap · ℒtask . The steady-state pmh fraction is cap/(1 + cap); no 𝜆 tuning is required. Step 5. Run the two falsification controls. Wrong-𝑊 : a random orthonormal projection of the same rank should, by Lemma 4.12, behave like isotropic pmh in expectation. Signal-𝑊 : regularising along a known signal direction should, by Corollaries 4.13/4.14, hurt the task metric. A positive matched-arm result reported without both controls is uninformative about whether the principle is responsible for the gain.
7.1
The recipe as twelve lines of PyTorch
The whole recipe is, mechanically, an extra term in the loss. The code below is what every block ^ task is in hand; the only block-specific choice is the estimator that fills in §8 reduces to once Σ the Sigma_hat slot. Hutchinson trace estimation with a paired-view encoder query approximates ^ task )] to 𝑂(1/√n_probes) stochastic error. E𝑥 [Tr(𝐽𝜑⊤ 𝐽𝜑 Σ Matched pmh and its two controls, in 12 lines of PyTorch import torch # Sigma_hat: (d, d) PSD covariance from Lemma D_k for your task. # encoder : phi(x); s : known signal direction; U from QR(randn(d, r)). def pmh_penalty(encoder, x, Sigma, n_probes=4): L = torch.linalg.cholesky(Sigma + 1e-6 * torch.eye(x.shape[-1])) phi0 = encoder(x) acc = sum((encoder(x + torch.randn_like(x) @ L.T) - phi0).pow(2).sum(-1).mean() for _ in range(n_probes)) return acc / n_probes loss = task_loss + lam * pmh_penalty(encoder, x, Sigma_hat) ctrl_wrong = lam * pmh_penalty(encoder, x, U @ U.T) ctrl_signal = lam * pmh_penalty(encoder, x, torch.outer(s, s) / s.dot(s)) *
19
# matched # Lemma C # Cor. E/E
What counts as a genuine match. Report three outcomes together: matched beats baseline on the deployment metric; wrong-𝑊 is indistinguishable from isotropic pmh (Lemma 4.12); signal𝑊 hurts below baseline (Corollaries 4.13/4.14). Matched-only gains are inconclusive. Estimator catalogue: Table 3; cap fixes 𝜆 via Proposition 3.5.
7.2
Why geometry and task accuracy can disagree
˜ 𝑄 ), not a task metric. The two are The matching principle minimises a geometric functional (𝐷 distinct scalars, and any honest report must say where they decouple. When the deployment task metric weights directions differently from Σtask —the generic case for downstream supervised losses— matched pmh optimises geometry but not necessarily the task score. Three canonical cases appear in §8. T2B (Chest X-ray): under protocol v3, E1 (pmh) compresses L4 drift and beats E1-no-pmh on heavy Gaussian (69.7% vs. 66.0%), but B0 leads clean, E1-no-pmh leads mean shift, and VAT leads saliency; B0 still collapses on Gaussian eval (Table 13; Appendix B.3). T6A (Whisper): matched content-residual pmh cuts layer-averaged TDI ≈ 65% and LibriSpeech-other WER 23.3% → 14.6%, but accent-supervised fine-tuning can reach slightly lower WER while worsening TDI—it uses label information matched pmh does not. T7B (CIFAR-10 ViT): PGD-AT wins PGD@4 (44.8%) but loses clean accuracy (−14.8 pp vs. baseline) and trajectory TDI (1.506 vs. 0.870 for PGD-delta matched pmh; Corollary 3.4; Figure 6, right). Report geometry and task metrics on separate axes; do not collapse them into one leaderboard score.
7.3
When the matching principle does not apply
Three structural limits to the framework • Linearisability. The Jacobian linearisation requires ‖𝑛‖ small relative to the encoder’s second-order envelope. Catastrophic corruption and severe domain shift can break this; nothing in the framework rules out a higher-order correction, and that is a natural next paper. • Identifiability. Each Lemma D𝑘 has its own structural assumption 𝐴𝑘 . When no 𝐴𝑘 describes the deployment nuisance, the framework offers no estimator and is silent. • Label preservation. 𝑄𝑛 must satisfy Definition 2.1 (𝑝(𝑦 | 𝑥 + 𝑛) = 𝑝(𝑦 | 𝑥)). When the shift is label-changing (Definition 2.1, second bullet), Σtask is undefined; use causal / multi-environment methods instead. These are honest scope conditions, not gaps in the proofs. They tell the practitioner exactly when to reach for a different tool, and they are what makes the matching principle falsifiable rather than unconditionally true.
8
Empirical programme: thirteen task blocks How to read the empirical scoreboard Table 5 and Figures 5–7 summarise all thirteen blocks (Figure 6 isolates the three pre-registered falsification checks). neg. = a predicted failure observed (evidence for the theory). Check task and geometry columns separately; full numbers: Appendix B. Proof-status and scale tables: §9.
20
Table 5: Aggregate evidence map across the 13 empirical blocks. Columns are the theorems being tested; entries are observed effects or qualitative outcomes. “neg.” marks predicted falsification modes that materialised. Per-block protocols and raw tables: Appendix B. Block
𝐷𝑘
Thm A
Thm B
Lem C
Cor E
Caveat
1A oracle 1A Office-31 2A ImageNet 2B X-ray 3A pose 3B depth 4A DomainNet 4B Cityscapes 5A QM9 5B clones 6A Whisper 6B HAR 7A Qwen 7B adv ViT
D1 D1 D2 D2 D3 D3 D4 D4 D5 D5 D6 D6 D7 D7
✓ CORAL>PMH iso=matched drift ∼2.2× +22 pp PCK aniso wins +3.31 pp rare-5 +11 pp MAE↓ rename 0.94 TDI −65% PMH best RM selective TDI 0.88
wrong≈B0 — — — VAT fail wrong +18% iso≈B0 iso≈B0 VAT mismatch VAT<matched wrong 0.64 wrong mid iso raw syco random W
— — — — — — — — — — — — — 𝐷𝑁 /𝐷𝑆
hard kNN linear limit — — — — — moto collapse large 𝜎 𝐸1𝑆 WER gap — wrong Σ PGD-AT cost
— neg. — geom̸=task — — — neg. tradeoff neg. geom̸=task — T7A: RM beh. / DPO geom. —
How to read this section. Each Type fixes 𝐴𝑘 , Lemma D𝑘, and the recipe controls (§7). One paragraph per type below; three synthesis figures amplify Table 5: Figure 5 (matched gain vs. B0/iso/wrong on each block’s headline metric); Figure 6 (Lemma C, Cor. E⋆ , Cor. p1-4 preregistered checks); Figure 7 (predicted failures named before runs). Per-block tables and diagnostic figures: Appendix B.
Headline block T7A: Qwen2.5-7B alignment (not block T7B) Naming. Block T7A (this headline) is Qwen2.5-7B alignment; block T7B is an unrelated CIFAR-10 ViT PGD-delta staircase (§8.7). “7B” is parameter count, not the block label. ^ style (rank-128 shrinkage)—but Both T7A arms use the same 𝐴7 estimator—style-pair Gram Σ answer different questions on different training setups: 1. RM arm (behavioural). Two-layer MLP on frozen Qwen hidden states, 20 epochs, 𝜆=0.7. Sycophancy 38.5% → 13.5%; honest preference 61.5% → 86.5% (95% bootstrap CIs exclude zero). Matched wins content/style selectivity (𝐶/𝑆 2.6× → 3.1×), not lowest raw sycophancy (isotropic: 5.8% sycophancy, 94.2% honest pref.). That selectivity pattern is the predicted tradeoff when blunt isotropic shrinkage beats directional matching on a scalar rate (Appendix B.12, Table 25). 2. DPO arm (geometry). LoRA dpo on the full 7B model, 1 epoch, 240 style pairs, four arms. Standard DPO raises Style tdi +30% (1.851 → 2.408); matched style-pmh DPO holds 1.836 (−0.8% vs. pre-DPO) with one extra trace term. Per-style drift can worsen for modes outside ^ style ) (e.g. bulleted; Table 26). span(Σ Full protocols: Appendix B.12.
21
What T7A establishes (and what it does not) What it is. One block, two falsification-relevant readouts: RM arm ⇒ alignment behaviour moves in the direction Theorem 3.1/𝐴7 predict; DPO arm ⇒ geometry (Style tdi) is preserved ^ style , same recipe controls (wrong-Σ, isotropic). where standard DPO degrades it. Same Σ What it is not. A deployed frontier rlhf system; not block T7B (vision robustness); not a claim that matched beats isotropic on every RM scalar (Table 8, Scale). Twelve further blocks and three pre-registered checks (§4.8) follow. Type 1–7: see Table 3; each runs wrong-𝑊 / signal-𝑊 controls. Matching principle: task and geometry evidence across thirteen blocks Task metrics T1 (F-MNIST oracle, SVM) T1 (Office-31, SVM A → D)
predicted neg.
CORAL 25.2
T2A (ViT ImageNet-C sev 3) T2B (X-ray mean shift)
no-PMH 86.9%; B0 clean 90.7%
T3A (COCO pose [email protected]) T3B (NYU AbsRel, combined-hard) T4A (DomainNet R → S test) iso moto 2.5 vs B0 10.2 (Cor.E ⋆ )
T4B (Cityscapes rare-5 mIoU) T5A (QM9 MAE @ σ=0.2)
signal-W (E1S) below B0
T5B (BigCloneBench rename bacc) T6A (Whisper LibriSpeech other WER)
geom: TDI −65%
matched − iso
T6B (HAR balanced acc @ L = 3)
matched − wrong-W
T7A (Qwen sycophancy)
matched − baseline (B0)
T7B (CIFAR-10 ViT PGD@4)
geometry gain vs. B0
geom: C/S ratio PGD-AT 44.8%; geom: TDI
−10 0 10 20 30 40 Matched advantage on headline task metric (pp; lower-better metrics sign-flipped)
Geometry readouts (dissociation blocks; read vs. task panel) T2B X-ray: saliency ↑
-4%
VAT 0.669; E1 L4 best
T2B X-ray: L4 drift ↓
∼ 2.2 × vs B0
55%
T6A Whisper: TDI ↓
65%
accent-adapted WER lower
T7A Qwen RM: C/S ratio ↑
19%
iso lower raw sycophancy
T7B ViT: TDI ↓
20%
PGD-delta matched; PGD-AT robust
−10
0 10 20 30 40 50 60 Matched PMH relative gain vs. B0 on geometry readout (%)
70
Figure 5: Thirteen blocks at a glance. Top: matched-arm gain (pp on each block’s headline task metric) vs. B0 (bar), isotropic-pmh (circle), wrong-𝑊 (triangle). Negative bars are predicted (Office-31 / D1). Bottom: geometry readouts for partial-pass / dissociation blocks (T2B, T6A, T7A, T7B). Read the panels separately: T2B shows residual decoupling (E1 compresses drift and beats E1-no-pmh on heavy Gaussian; B0/E1-no-pmh/VAT split clean, mean shift, and saliency); T6A, T7A, T7B show geometry gains a headline task scalar can miss. See §7.2 and Appendix B.
22
Three quantitative predictions of the matching principle, three independent verifications Cor.\,E ⋆ : signal-W strictly below B0
Lemma C: predicted gap ≤ 5% observed gap = 4.2\%
2.98
3.0 2.5 2.0 1.5 1.0 0.5 0.0
100
83.0
80
73.8
60 40 20 0
Wrong-W
Isotropic \PMH{}
50
93.8
PGD@4 robust accuracy (\%)
3.11
Rename / clean bacc on T5B (\%)
DN/DS on T7B (ViT-S)
3.5
Cor.\,p1-4: PGD-AT off clean-Pareto observed −14.8 pp clean cost
observed −9.2 pp (Ω(ρ 2))
40 30
B0
20
E1S (signal-W)
B0
Matched \PMH{}
10 0
E1 (matched)
PGD-AT −14.8 pp clean vs B0
60
65
70 75 80 Clean accuracy (\%)
85
90
Figure 6: Three pre-registered falsification checks (§4.8). Left: Lemma 4.12—wrong-𝑊 tracks isotropic pmh on 𝐷𝑁 /𝐷𝑆 (T7B: 2.98 vs. 3.11). Centre: Cor. 4.14—signal-𝑊 hurts below baseline (T5B keyword-pmh: rename_bacc_ratio 0.830 → 0.738). Right: Cor. 3.4—PGD-AT wins robustness but loses clean acc (−14.8 pp vs. B0 at higher PGD@4); geometry–task dissociation, not a refutation.
Representative negatives, limits, and two-sided falsification trials
Office-31 honest negative (Amazon→DSLR)
Cityscapes iso-PMH on motorcycle (T4B)
25 Motorcycle IoU (%)
Target accuracy (%)
20
20 15 10 5 0
15
10
5
0 B0
PMH ed ch mat
AL COR
N
LMN
55
80
50
60 40
Multiscale
Baseline Coordinate PMH
45 40 35 30
20 0
Iso PMH
QM9 noise tradeoff (T5A)
100
Aggregate MAE
Rename / clean bacc (%)
BigCloneBench two-sided test (T5B)
Baseline
25 B0
VAT
E1
E1S
0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 Eval position noise σ (Å)
Figure 7: Honest negatives (predicted before runs). Top-left: Office-31 / Lemma D1 eigengap (≈ 1.03): CORAL beats matched SVM. Top-right: Cityscapes iso-pixel motorcycle IoU collapse (Cor. E⋆ ). Bottom-left: QM9 clean-vs-robust MAE tradeoff (Thm. A(ii) allocation). Bottom-centre: ^ quality staircase (better estimate T5B identifier vs. keyword probe (Cor. E⋆ ). Bottom-right: T7B 𝑊 ⇒ ordered PGD@4 gain).
23
8.1
Type 1: classical machine learning under a known subspace nuisance
𝐴1 , Lemma D1 (cross-domain SVD). Predict: usable eigengap ⇒ matched > iso ≈ wrong-𝑊 (Thm. A, Lemma 4.12). Verdict: pass on oracle ridge/SVM/kNN and DCT drift (matched ridge MSE 0.101 vs. 0.553 for B0 at extreme nuisance); predicted fail Office-31 (eigengap ≈ 1.03, CORAL 25.2% vs. matched 23.3%; Fig. 7). Appendix B.1.
8.2
Type 2: isotropic acquisition noise on deep networks
𝐴2 , Lemma D2. Predict: no subspace ⇒ matched = isotropic; iso beats wrong-𝑊 on geometry. Verdict: T2A pass (+4.3 pp ImageNet-C, TDI −58%); T2B partial (E1 wins geometry and heavy Gaussian vs. no-pmh; B0/E1-no-pmh/VAT split clean, mean shift, saliency—Table 13, Appendix B.3). Appendices B.2–B.3.
8.3
Type 3: photometric and occlusion nuisances in dense prediction
^ aug beats iso and mis𝐴3 , Lemma D3 (augmentation-delta Gram). Predict: anisotropic Σ′ ∝ Σ matched VAT; wrong-𝑊 raises drift. Verdict: T3A +22.3 pp PCK; T3B best on hard photometric metrics (Table 16; wrong-𝑊 AbsRel +18%). Appendices B.4–B.5.
8.4
Type 4: hierarchical domain shift
𝐴4 , Lemma D4 (per-layer Gram). Predict: layer matched beats pixel-iso; iso-pixel on rare classes can hurt (Cor. 4.14). Verdict: T4A +3.3 pp; T4B rare-5 mIoU +11.1 pp; iso-pixel motorcycle IoU 10.2% → 2.5% (Fig. 7). Appendices B.6–B.7.
8.5
Type 5: compositional nuisances in structured data
𝐴5 , Lemma D5 (nuisance-block covariance). Predict: penalise identifier/position block, not signal; signal-𝑊 hurts. Verdict: T5B identifier pmh 0.830 → 0.938 rename ratio, keyword pmh → 0.738; T5A QM9 large-noise Pareto tradeoff (Table 21, Thm. A(ii)). Appendices B.8–B.9.
8.6
Type 6: temporal and sequential drift
𝐴6 , Lemma D6 (content-residual / sensor scatter). Predict: matched suppresses 𝐷𝑁 /𝐷𝑆 ; wrong𝑊 ≈ iso; supervision may beat WER without fixing TDI. Verdict: T6B pass at all stress levels; T6A partial (WER and TDI win; accent-adapted WER-only). Appendices B.10–B.11.
8.7
Type 7: adversarial robustness and alignment geometry
𝐴7 , Lemma D7—two sub-blocks, two estimators: T7B (vision): PGD-delta / gradient-SVD Gram on CIFAR-10 ViT. T7A (language): style-pair Gram on Qwen2.5-7B (§8). Predict (T7B): better ^ ⇒ ordered PGD@4 staircase; PGD-AT can win robustness but lose clean acc / trajectory TDI 𝑊 (Cor. 3.4). Predict (T7A): matched suppresses style-linked drift and improves selective honesty, not necessarily every raw RM rate vs. isotropic. Verdict: T7B pass + dissociation; T7A pass on RM selectivity and DPO Style tdi (Appendices B.12–B.13).
24
8.8
Named failures and overall pattern
Figure 7 collects four predicted negatives (Office-31 / D1 eigengap; Cityscapes iso-pixel / Cor. E⋆ ; T5B keyword-pmh; QM9 allocation / Thm. A(ii)); each is upheld in the Types above and Appendix B. Where 𝐴𝑘 holds with a usable gap, matched beats isotropic and wrong-𝑊 tracks isotropic (Lemma 4.12) in twelve of thirteen blocks; arm-level signal penalties and allocation tradeoffs are successes of named failure modes, not contradictions (Table 5).
9
Synthesis tables
Three tables complement the scoreboard at the start of §8 (Table 5, Figures 5–7). How to read the synthesis tables 1. Table 6. Proved vs. conditional vs. observed (separate math from experiments). 2. Table 7. Same penalty from ridge/kNN to 7B; only the Table 3 row changes. 3. Table 8. Formal and extension gaps (prose: §10.3). Proof status (Table 6). Theorems A, G, B and Lemmas C, Cor. E/E⋆ are unconditional; A⋆global under (R), (C♭ ), (E), (I); training corollary needs (O). Lemmas D1–D7 are conditional on 𝐴𝑘 . §8 is observational synthesis, not a theorem. Table 6: Theorem and assumption status. Categories: proved (no extra assumptions), conditional (proved under a named assumption listed in the scope column), observed (empirical, not a theorem), and open. Result
Status
Scope
ERM blind spot (Thm. 3.1) Prop. 4.5 (C♭ linear) Theorem A Theorem G Theorem A⋆global Cor. A⋆train Theorem B Lemma C / C⋆ Cor. E / E⋆ Conj. A† Lemmas D1–D7 §8
Proved Proved Proved Proved Proved Conditional on (O) Proved Proved Proved Open Conditional Observed
ERM + correlated nuisance Linear-Gaussian ERM Linear-Gaussian Quadratic Jacobian Global min; (R),(C♭ ),(E),(I) Training output Two-regime mismatch Random Σ′ Ω(𝜌4 ) / Ω(𝜌2 ) Deep allocation Under 𝐴𝑘 13 blocks
Scale and open problems (Tables 7–8). Same penalty from ridge/kNN through ViT, speech, code, and Qwen2.5-7B; only the Lemma D𝑘 row changes. Eight open items; highlighted prose in §10.3.
25
Table 7: Cross-scale model programme. Same matching principle; the pmh surrogate (Hutchinson trace, paired-view feature MSE, margin invariance, . . . ) varies by block—see Appendix B. Pick the row from Table 3 for your nuisance family. Scale
Architecture Params Block
Classical Ridge / SVM / kNN Small CNN ResNet-18 Mid CNN ResNet-50 ViT ViT-B/16 Code CodeBERT Speech Whisper-small LLM Qwen2.5-7B
10
Table 8: Open problems left by the framework. Two are formal (O, A† ); the remaining six are extension questions whose resolution does not require new theory of Theorem A but does require new estimator design.
— T1 11M T2B, T3 25M T4 86M T2A 125M T5B 244M T6A 7B T7A (not T7B)
ID
Problem
(O) A† Rate Causal Online Robust Compose Scale
Optimisation reachability Deep cube-root allocation Tighten 𝑂(1/𝜆) deep bound Colored nuisances Non-stationary Σtask Adversarial deployment Σ Multiple nuisance families Multi-epoch RLHF validation
Discussion
The synthesis tables (§9) record proof status, scale, and open problems; the empirical scoreboard opens §8. This section interprets what follows if the matching principle is taken seriously as a design theory for nuisance-robust losses (full proofs: Appendix A).
10.1
What changes if the matching principle is correct
Robustness collapses into one estimation problem. Which regulariser to use reduces to: which 𝐴𝑘 describes deployment, and how well does the Lemma D𝑘 estimator recover Σtask ? CORAL, adversarial training, IRM, augmentation, metric learning, and Jacobian penalties become estimators of one object. Wide eigengap ⇒ win; marginal gap ⇒ predicted loss (Office-31 / Lemma D1). Loss-function design becomes first-class. Theorem 4.1 promotes the loss to a design variable parameterised by one PSD matrix per nuisance type, with a closed-form optimum and two falsification controls (Lemma 4.12, Corollaries 4.13/4.14). Engineering effort can shift from architecture search to what to regularise against, with a principled answer. Ablations become principled, not decorative. The random-projection and signal-direction controls are no longer courtesy ablations; they are the tests Lemma 4.12 and Corollaries 4.13/4.14 fix in advance. A claim that fails both is, by the theory, not a matching-principle effect. Alignment is geometric. Block T7A (§8; Qwen2.5-7B, not vision block T7B) uses style as the nuisance: the RM arm is selective on sycophancy vs. isotropic blunt shrinkage; the DPO arm preserves Style tdi where standard DPO does not (Appendix B.12). Adjacent literatures. Relative to the information bottleneck, matched pmh suppresses 𝜑 along deployment Σtask rather than all of 𝐼(𝜑; 𝑥). Relative to equivariance, it estimates a covariance when no clean group action exists (photometry, accent, style). Relative to interpretability, it prescribes pre hoc what the encoder should be insensitive to—the 7B alignment case (§8).
26
10.2
Scope and relation to prior work
Label-preserving limits (causal/spurious benchmarks), dissociation of geometry vs. task metrics, and the 7B scale caveat are stated in §1 and §7.3; Table 8 tags the causal and scale rows. The companion note [14] states the blind spot and isotropic uniqueness; this paper is self-contained (Appendix A) and adds matched Σtask , Lemmas D1–D7, tdi, and thirteen blocks.
10.3
Open problems the framework names
Table 8 lists eight items; we highlight the three that shape how to read the present results. (O) Optimisation reachability. Theorem 4.9 is a global statement; proving standard training reaches that minimum (assumption (O) for Cor. A⋆train ) is the central formal gap. Proposition 4.5 settles assumption (C♭ ) in the linear model; Remark 4.6 states what remains open for deep nets. Causal / coloured nuisances. Definition 2.1 separates label-preserving deployment laws from label-changing spurious shifts; the latter need multi-environment or causal tools (Table 8). Estimator selection in practice. §5.1 gives symptom→ 𝐴𝑘 guidance, additive composition, and an eigengap pre-flight with Office-31 as the calibrated marginal case—closing the “which row do I use?” gap left by the conditional D-lemmas alone. Scale. Multi-epoch, full-data alignment at frontier parameter counts (≥ 70B) is the natural extension of block T7A (§8); remaining rows in Table 8 list deep allocation, rate tightening, online Σtask , adversarial Σ, and compositional matching.
11
Conclusion
The robustness literature has been a catalogue of methods in search of a common object. This paper names it: the population covariance Σtask of label-preserving deployment variation. Within quadratic Jacobian penalties, the optimal regulariser is determined by that matrix—sufficiently (Theorem 4.1), necessarily (Theorem 4.2), and at the global minimum of deep encoders under standard regularity (Theorem 4.9, Lemma 4.10). Lemma 4.12 and Corollaries 4.13/4.14 make the claim falsifiable; Lemmas D1–D7 tell you how to estimate Σtask under each structural assumption 𝐴𝑘 . Across thirteen blocks, matched pmh wins on the headline metric when 𝐴𝑘 holds and the estimator has a usable eigengap; wrong-𝑊 tracks isotropic (Lemma 4.12); signal-aligned penalties hurt (Cor. E⋆ ). Named exceptions: Office-31 (Lemma D1), Cityscapes iso-pixel and T5B keywordpmh, QM9 allocation (Theorem A(ii)); in block T7A (Qwen2.5-7B), matched style-pmh is the selective fix (§8). We do not claim universality over all robustness problems. We claim a scoped unification: when ^ task plus deployment nuisance is label-preserving and estimable, loss design reduces to matching Σ two controls—a prescription that is closed form, constructive, testable on every new task, and consistent with the evidence in §8 and Table 5.
27
Reproducibility. All numbers are frozen in per-task JSON; protocols and rebuild commands are in Appendix B and Appendix B. Headline checkpoints (T7B ViT-Small, T7A Qwen2.5-7B) will ship with the camera-ready release. The matched-pmh recipe (§7) is implemented in the standalone Python package matching-pmh (https://github.com/vishalstark512/matching-pmh; pip install matching-pmh, import pmh), separate from the experiment repositories bundled with this manuscript.
28
References [1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. In JMLR, 2018. [2] Martin Arjovsky, Léon Bottou, Anirudh Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv:1907.02893, 2019. [3] Aurélien Bellet, Amaury Habrard, and Marc Sebban. A survey on metric learning for feature vectors and structured data. arXiv:1306.6709, 2013. [4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. In Machine Learning, 2010. [5] Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv:1702.05374, 2017. [6] Hal Daumé III. Frustratingly easy domain adaptation. In ACL, 2007. [7] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture. In ICLR, 2019. [8] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019. [9] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. [10] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Trans. Knowledge and Data Engineering, 2010. [11] Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, et al. Discovering language model behaviors with model-written evaluations. In Findings of ACL, 2023. [12] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, 2023. [13] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023. [14] Vishal Rajput. Supervised learning has a geometric blind spot: Theory and minimal repair, 2026. Companion arXiv note (2604.21395); matched-Σtask theory and experiments are selfcontained here. [15] Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation. In ECCV, 2016. [16] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In IEEE Information Theory Workshop, 2015. [17] Garrett Wilson and Diane J. Cook. A survey of unsupervised deep domain adaptation. ACM Trans. Intelligent Systems and Technology, 2020. 29
A
Proofs
All formal claims in the main text are proved below (self-contained; [14] is related work only). §8 is observational synthesis. Appendix B holds protocols and frozen numbers. Proof map (read in order). A.1. Foundation (A.1): blind spot, isotropy, PGD anisotropy, training cap (Thm. 3.1, Cor. 3.2/3.4, Prop. 3.3/3.5). A.2. Theorem A (A.2): matched sufficiency (linear-Gaussian). A.3. Theorem G (A.3): range necessity. A.4. Lemma D-ML (A.4): expressivity assumption (E). A.5. Theorem A⋆ (A.5): deep global minimum. A.6. Theorem B (A.6): range vs. allocation costs. A.7. Lemma C (A.7): wrong-𝑊 ≡ isotropic. A.8. Cor. E/E⋆ (A.8): signal penalty hurts. A.9. Lemmas D1–D7 (A.9): estimator consistency under 𝐴𝑘 . A.10. Prop. F (A.10): class-layout tdi0 envelope.
A.1
Geometric foundation: blind spot, isotropy, PGD, and training cap
Appendix A.1 (proof-map item 1) proves the §3 floor self-contained: Theorem 3.1, Corollaries 3.2/3.4, Propositions 3.3/3.5. Companion arXiv note [14] motivates the blind spot; all proofs needed for this paper are here. Lemmas A.2–A.7 are reused in Appendix A.2–A.8. Remark A.1 (Gaussian linear model). 𝑠 ∼ 𝒩 (0, 𝐼𝑑𝑠 ), 𝑛 ∼ 𝒩 (0, 𝐼𝑑𝑛 ), 𝑠 ⊥ 𝑛, 𝑦 = ⟨𝑤𝑠 , 𝑠⟩ + 𝜌⟨𝑤𝑛 , 𝑛⟩ + 𝜀 with 𝜀 ∼ 𝒩 (0, 𝜎𝜀2 ), ‖𝑤𝑠 ‖2 = ‖𝑤𝑛 ‖2 = 1, 𝜌 > 0, and 𝑥 = (𝑠, 𝑛). Write 𝐽𝜑 = [𝐽𝜑,𝑠 | 𝐽𝜑,𝑛 ] so ‖𝐽𝜑 ‖2𝐹 = ‖𝐽𝜑,𝑠 ‖2𝐹 + ‖𝐽𝜑,𝑛 ‖2𝐹 . Lemma A.2 (Sub-block inequality). For 𝐴 ∈ R𝑚×𝑑 and unit 𝑣 ∈ R𝑑 : ‖𝐴𝑣‖22 ≤ ‖𝐴‖2𝐹 . For 𝐴 = [𝐴1 | 𝐴2 ]: ‖𝐴2 𝑣‖22 ≤ ‖𝐴2 ‖2𝐹 ≤ ‖𝐴‖2𝐹 . Proof. Cauchy–Schwarz on 𝐴𝑣 =
𝑗 𝑣𝑗 𝐴𝑒𝑗 with
∑︀
∑︀
2 𝑗 𝑣𝑗 = 1.
Lemma A.3 (Linearised vs. exact drift). If 𝜑 has 𝛽-Lipschitz Jacobian and sup𝑥 ‖𝐽𝜑 (𝑥)‖𝐹 ≤ 𝑀 , then for 𝛿 ∼ 𝒩 (0, 𝜎 2 𝐼𝑑 ), 𝐷𝑄 (𝜑) = E𝑥,𝛿 [‖𝜑(𝑥 + 𝛿) − 𝜑(𝑥)‖22 ] = 𝜎 2 E𝑥 [‖𝐽𝜑 (𝑥)‖2𝐹 ] + 𝑅(𝜑, 𝜎) with ˜ 𝑄 − 𝑂(𝜎 4 ) for |𝑅(𝜑, 𝜎)| ≤ 𝐶(𝑀 𝛽𝑑3/2 𝜎 3 + 𝛽 2 𝑑2 𝜎 4 ) for a universal 𝐶 > 0. In particular 𝐷𝑄 ≥ 𝐷 1 ˜ small 𝜎, and 𝐷𝑄 (𝜑, 𝜎) ≥ 2 𝐷𝑄 (𝜑, 𝜎) once 𝜎 ≤ 𝜎0 (𝑀, 𝛽, 𝑑, 𝜌, 𝐿). Proof. Integral form of the mean-value expansion; the leading term is E[‖𝐽𝜑 𝛿‖22 ] = 𝜎 2 E𝑥 [‖𝐽𝜑 ‖2𝐹 ]; higher orders are 𝑂(𝜎 3 )–𝑂(𝜎 4 ). Lemma A.4 (ERM must encode the nuisance direction). Under Remark A.1, any population MSE minimiser 𝑓 * = ℎ ∘ 𝜑* satisfies E𝑥 [𝜕𝑤𝑛 𝑓 * (𝑥)] = 𝜌 for 𝜕𝑤𝑛 := ⟨𝑤𝑛 , ∇𝑛 ⟩. Hence 𝐽𝜑,𝑛 is non-zero on a set of positive measure. Proof. Bayes predictor 𝑓 * (𝑥) = ⟨𝑤𝑠 , 𝑠⟩+𝜌⟨𝑤𝑛 , 𝑛⟩. Stein’s identity for 𝑛 ∼ 𝒩 (0, 𝐼𝑑𝑛 ) gives E[𝜕𝑤𝑛 𝑓 * ] = E[𝑓 * · ⟨𝑤𝑛 , 𝑛⟩] = 𝜌‖𝑤𝑛 ‖22 = 𝜌. 30
Proof of Theorem 3.1. By Lemma A.4 and the chain rule, 𝜌 = E[∇𝜑 ℎ⊤ 𝐽𝜑,𝑛 𝑤𝑛 ] ≤ 𝐿 E[‖𝐽𝜑,𝑛 𝑤𝑛 ‖2 ]. Jensen gives E[‖𝐽𝜑,𝑛 𝑤𝑛 ‖22 ] ≥ 𝜌2 /𝐿2 . Lemma A.2 yields E[‖𝐽𝜑 ‖2𝐹 ] ≥ E[‖𝐽𝜑,𝑛 𝑤𝑛 ‖22 ] ≥ 𝜌2 /𝐿2 , hence ˜ 𝑄 (𝜑* , 𝜎) = 𝜎 2 E[‖𝐽𝜑 ‖2 ] ≥ 𝜎 2 𝜌2 /𝐿2 . Lemma A.3 upgrades this to 𝐷𝑄 ≥ 𝜎 2 𝜌2 /(2𝐿2 ) for 𝜎 ≤ 𝜎0 . □ 𝐷 𝐹 Lemma A.5 (Bregman gap for strictly proper losses). Let ℒ be strictly proper with Bregman generator 𝜓. For conditionals 𝑝, 𝑞 on 𝒴 | 𝑥: E𝑦∼𝑝 [ℒ(𝑞, 𝑦)] − E𝑦∼𝑝 [ℒ(𝑝, 𝑦)] = 𝑑𝜓 (𝑝‖𝑞) ≥ 0, with equality iff 𝑝 = 𝑞 a.s. Proof. Definition of strict properness: the expected loss is uniquely minimised at 𝑞 = 𝑝. Lemma A.6 (Risk excess of an 𝑛-blind predictor). Under correlated nuisance (direct influence: 𝑝(𝑦 | 𝑥) ̸≡ 𝑝(𝑦 | 𝑠(𝑥)) on positive measure), define Δ(𝑃, ℒ) := E𝑥 [𝑑𝜓 (𝑝(𝑦 | 𝑥)‖𝑝(𝑦 | 𝑠(𝑥)))] > 0. Any 𝑓 * = arg min𝑓 E[ℒ(𝑓 (𝑥), 𝑦)] satisfies E[ℒ(𝑓 * , 𝑦)] ≤ E[ℒ(𝑓 𝑠 , 𝑦)] − Δ(𝑃, ℒ) for 𝑓 𝑠 (𝑥) = E[𝑦 | 𝑠(𝑥)], hence cannot be invariant to all nuisance directions that carry Δ-mass. Proof. Take 𝑞 = 𝑝𝑠 in Lemma A.5 at each 𝑥 and average. Lemma A.7 (Jacobian sensitivity from risk gap). Suppose 𝑓 * = ℎ ∘ 𝜑* minimises E[ℒ(𝑓 (𝑥), 𝑦)] with 𝐿-Lipschitz ℎ and Δ(𝑃, ℒ) > 0. Assume there exists a unit direction 𝑢 in the nuisance subspace with E𝑥 [𝜕𝑢 𝑓 * (𝑥)] ≥ 𝑐Δ > 0 (encoding necessity, as in Lemma A.4 in the Gaussian model). Then ˜ 𝑄 (𝜑* , 𝜎) ≥ 𝜎 2 𝑐2 /𝐿2 . E𝑥 [‖𝐽𝜑* ,𝑛 (𝑥)𝑢‖22 ] ≥ 𝑐2Δ /𝐿2 and 𝐷 Δ Proof. Same chain rule and Jensen steps as Theorem 3.1, with 𝜌 replaced by 𝑐Δ . Proof of Corollary 3.2. Lemma A.6 gives Δ(𝑃, ℒ) > 0 under the corollary’s hypothesis (𝑝(𝑦 | 𝑥) ̸≡ 𝑝(𝑦 | 𝑠(𝑥))). In the Gaussian model, 𝑐Δ = 𝜌 and Δ = Θ(𝜌2 ) as 𝜌 → 0 (KL between Gaussian ˜ 𝑄 (𝜑* , 𝜎) ≥ 𝜎 2 Δ(𝑃, ℒ)/(𝐶mv E‖𝑛‖2 𝐿2 ) for a mean-value constant conditionals). Lemma A.7 yields 𝐷 2 * 𝐶mv from linearising 𝑓 in 𝑛; in Remark A.1, 𝐶mv = 𝑂(1) and the bound reduces to 𝜎 2 𝐶 ′ (𝑃, ℒ)/𝐿2 with 𝐶 ′ (𝑃, ℒ) = Δ up to absolute constants. □ Proof of Proposition 3.3. For Cov(𝛿) = Σ𝛿 , E𝛿 [‖𝐽𝜑 𝛿‖22 ] = Tr(𝐽𝜑⊤ 𝐽𝜑 Σ𝛿 ). Equality with 𝜎 2 Tr(𝐽𝜑⊤ 𝐽𝜑 ) for all 𝐽𝜑 holds iff Σ𝛿 = 𝜎 2 𝐼 (test on 𝐴 = 𝑒𝑖 𝑒⊤ 𝑗 ). □ Proof of Corollary 3.4. PGD-AT targets 𝛿^* (𝑥) ∝ sign(∇𝑥 ℒ), not 𝒩 (0, 𝜎 2 𝐼). By Proposition 3.3, shrinking along 𝛿^* need not isotropise 𝐽𝜑 ; trajectory tdi (Eq. (5)) can rise while ‖𝐽‖𝐹 falls (T7B: PGD-AT 1.506 vs. PGD-delta matched 0.870 at the same backbone; §8.7). □ Proof of Proposition 3.5. ℒpmh ) = cap/(1 + cap). □
A.2
At the capped steady state, ℒpmh = cap · ℒtask , so 𝑓 = ℒpmh /(ℒtask +
Theorem A: matched-Σ optimality (linear-Gaussian)
Main-text anchor (§4.1). model.
Matched sufficiency and cube-root allocation in the linear-Gaussian
31
Setting (Theorem 4.1). In the linear-Gaussian model of Remark A.1, diagonalise Σtask = diag(𝜆1 , . . . , 𝜆𝑟 , 0, . . .) and write the pmh penalty as diag(𝜇1 , . . . , 𝜇𝑑𝑥 ) in the same basis (standard; off-diagonal cross-terms between Σtask and Σ′ do not change the range argument). Let 𝑣˜𝑖 denote regressor energy in direction 𝑖. Proof of Part (i). The pmh-regularised solution is 𝑤(Σ′ ) = (𝐼 + 2𝜆Σ′ )−1 𝑣. In the joint basis, component 𝑖 of 𝑤(Σ′ ) is: 𝑣˜𝑖 . [𝑤(Σ′ )]𝑖 = 1 + 2𝜆𝜇𝑖 The deployment drift is: ˜ 𝑄 (𝑤(Σ′ )) = 𝐷
𝑑𝑥 ∑︁
𝜆𝑖 [𝑤(Σ′ )]2𝑖 =
𝑖=1
𝜆𝑖 𝑣˜𝑖2 . (1 + 2𝜆𝜇𝑖 )2 𝑖=1
𝑑𝑥 ∑︁
Since 𝑣 ∈ range(Σtask ), we have 𝑣˜𝑖 = 0 for 𝑖 > 𝑟 (components outside the deployment range carry no regressor energy), so: 𝑟 ∑︁ 𝜆𝑖 𝑣˜𝑖2 ˜ 𝑄 (𝑤(Σ′ )) = 𝐷 . (1 + 2𝜆𝜇𝑖 )2 𝑖=1 (⇒) Sufficient direction. If range(Σ′ ) ⊇ range(Σtask ), then 𝜇𝑖 > 0 for all 𝑖 ≤ 𝑟. As 𝜆 → ∞: 𝜆𝑖 𝑣˜𝑖2 1 =𝑂 2 (1 + 2𝜆𝜇𝑖 )2 𝜆 (︂
)︂
→ 0.
˜ 𝑄 → 0. Summing over 𝑖 ≤ 𝑟 (finitely many terms) gives 𝐷 ′ (⇐) Necessary direction. If range(Σ ) ̸⊇ range(Σtask ), there exists some index 𝑗 ≤ 𝑟 with 𝜇𝑗 = 0 and 𝜆𝑗 > 0. By Assumption 4.8, 𝑣˜𝑗2 > 0 on every active nuisance direction. The 𝑗-th term satisfies: 𝜆𝑗 𝑣˜𝑗2 = 𝜆𝑗 𝑣˜𝑗2 > 0, (1 + 2𝜆 · 0)2 ˜ 𝑄 (𝑤(Σ′ )) ≥ 𝜆𝑗 𝑣˜2 > 0 for all 𝜆, giving 𝐷 ˜ 𝑄 = Θ(1). □ independent of 𝜆. Hence 𝐷 𝑗 ˜ 𝑄 (𝑤(Σ′ )) = ∑︀𝑟 𝜆𝑖 𝑣˜2 (1 + 2𝜆𝜇𝑖 )−2 over the simplex 𝒜𝑐 = Proof of Part (ii). We minimise 𝐷 𝑖=1 𝑖 ∑︀ {𝜇 ≥ 0 : 𝑖 𝜇𝑖 = 𝑐, 𝜇𝑖 > 0 ∀𝑖 ≤ 𝑟}. Differentiating and applying the KKT conditions gives: ⎡
⎤
∑︁ 𝜕 ⎣˜ −4𝜆2 𝜆𝑖 𝑣˜𝑖2 𝐷𝑄 − 𝜈 𝜇𝑗 ⎦ = 0 =⇒ = 𝜈, 𝜕𝜇𝑖 (1 + 2𝜆𝜇𝑖 )3 𝑗 −4𝜆2 𝜆 𝑣˜2
𝑖 𝑖 so (1 + 2𝜆𝜇𝑖 )3 = . Since 𝜈 < 0 at the constrained minimum (the objective decreases as 𝜇𝑖 𝜈 increases), write |𝜈| = 𝜈0 > 0. Then:
(︃
1 + 2𝜆𝜇*𝑖 =
4𝜆2 𝜆𝑖 𝑣˜𝑖2 𝜈0
)︃1/3
.
For large 𝜆, the dominant term on the right is 𝑂(𝜆2/3 ), so 𝜇*𝑖 = 𝑂(𝜆−1/3 ). To leading order, 𝜇*𝑖 ∝ (𝜆𝑖 𝑣˜𝑖2 )1/3 . When 𝑣˜𝑖2 ∝ 𝜆−1 (uniform energy on range(Σtask )), the proportional rule 𝜇𝑖 ∝ 𝜆𝑖 coincides with 𝑖 the cube-root optimum—the rotation-invariant case in Figure 3. □ 32
A.3
Theorem G: necessity of range(Σtask )
Main-text anchor (§4.2). Any quadratic Jacobian penalty that zeros drift on all task directions must cover range(Σtask ). This is the formal proof of Theorem 4.2: if range(𝐴) ̸⊇ range(Σtask ), drift cannot vanish uniformly over all effective regressors. We prove the contrapositive. Suppose range(𝐴) ̸⊇ range(Σtask ); choose a unit vector 𝑞 ∈ range(Σtask ) ∖ range(𝐴), i.e. 𝑞 ∈ ker(𝐴) ∩ range(Σtask ). Set 𝑣 = 𝑞. Then 𝐴𝑣 = 𝐴𝑞 = 0, so (𝐼 + 2𝜆𝐴)−1 𝑣 = 𝑣 = 𝑞 for all 𝜆. Thus: ˜ 𝑄 (𝑤𝜆 (𝐴)) = 𝑞 ⊤ Σtask 𝑞. 𝐷
𝑤𝜆 (𝐴) = 𝑞,
˜ 𝑄 (𝑤𝜆 (𝐴)) = Since 𝑞 ∈ range(Σtask ) and Σtask is positive definite on its range, 𝑞 ⊤ Σtask 𝑞 > 0. Hence 𝐷 ⊤ 𝑞 Σtask 𝑞 > 0 for all 𝜆, which does not tend to zero. □ The same argument applies to every direction in range(Σtask ) ∖ range(𝐴); covering the full ˜ 𝑄 → 0 uniformly over 𝑣 ∈ range(Σtask ). nuisance range is necessary for 𝐷
A.4
Lemma D-ML: constructive verification of (E)
Assumption (E) in Theorem 4.9 requires a parameter 𝜃0 with zero pmh penalty at finite task loss. Lemma 4.10 constructs it for networks in 𝒞NRP (first layer wide enough, then arbitrary depth). Setting. Let 𝜑𝜃 ∈ 𝒞NRP , so 𝜑𝜃 (𝑥) = 𝜓(𝑊 (1) 𝑥 + 𝑏(1) ; 𝜃rest ) with 𝑊 (1) ∈ R𝑚×𝑑𝑥 , 𝑚 ≥ 𝑑𝑥 − 𝑟. Let Σ′ ⪰ 0 have rank 𝑟 and eigendecomposition Σ′ = 𝑈Σ Λ𝑈Σ⊤ . Construction of 𝜃0 . Let 𝑈 ⊥ ∈ R𝑑𝑥 ×(𝑑𝑥 −𝑟) be any orthonormal basis of range(Σ′ )⊥ . Choose (1) (1) 𝑊0 ∈ R𝑚×𝑑𝑥 such that each row of 𝑊0 lies in range(Σ′ )⊥ = span(𝑈 ⊥ ). This is possible since (1) (1) (1) 𝑚 ≥ 𝑑𝑥 − 𝑟 (dimension of range(Σ′ )⊥ ). Let 𝜃0 = (𝑊0 , 𝑏0 , 𝜃rest,0 ) with 𝑏0 arbitrary and 𝜃rest,0 chosen to minimise task loss given the constrained input projection (e.g. the Bayes-optimal head). Proof of (E1).
For any input 𝑥, the Jacobian of 𝜑𝜃0 with respect to 𝑥 is: 𝐽𝜑𝜃0 (𝑥) =
𝜕𝜓 (1) (1) (1) (𝑊0 𝑥 + 𝑏0 ) · 𝑊0 . 𝜕𝑧
For any 𝑣 ∈ range(Σ′ ), we have 𝑈Σ⊤ 𝑣 ̸= 0 while (𝑈 ⊥ )⊤ 𝑣 = 0 by orthogonality. Since each row of (1) (1) 𝑊0 is in span(𝑈 ⊥ ), we have 𝑊0 𝑣 = 0. Therefore 𝐽𝜑𝜃0 (𝑥)𝑣 = 0 for all 𝑥. Consequently: 𝐽𝜑𝜃0 (𝑥)⊤ 𝐽𝜑𝜃0 (𝑥) Σ′ = 0
pointwise in 𝑥,
and thus E𝑥 [Tr(𝐽𝜑⊤𝜃 𝐽𝜑𝜃0 Σ′ )] = 0 exactly. □E1 0
(1)
(1)
(1)
Proof of (E2). Because 𝜑𝜃0 (𝑥) = 𝜓(𝑊0 𝑥 + 𝑏0 ; 𝜃rest,0 ) and each row of 𝑊0 is in range(Σ′ )⊥ , (1) the map 𝑥 ↦→ 𝑊0 𝑥 depends only on Πrange(Σ′ )⊥ 𝑥. Choosing the optimal head 𝜓 * for this input (i.e. the conditional expectation E[𝑦 | Πrange(Σ′ )⊥ 𝑥]) achieves: ℒtask (𝜃0 ) = E[𝑦 − E(𝑦 | Πrange(Σ′ )⊥ 𝑥)]2 = E[Var(𝑦 | Πrange(Σ′ )⊥ 𝑥)] < ∞.
33
□E2
A.5
Theorem A⋆ : deep-encoder range-matching dichotomy
˜ 𝑄 = 𝑂(1/𝜆) at the global pmh minimum; failure Formal proof of Theorem 4.9: range matching ⇒ 𝐷 to cover range(Σtask ) leaves a 𝜆-independent drift floor. ˜ 𝑄 (𝜑* , 𝜎) ≥ Proof of Proposition 4.5. At a population ERM minimiser 𝜑*𝜃 , Theorem 3.1 gives 𝐷 𝜃 2 2 2 ⊤ ⊤ 𝜎 𝜌 /𝐿 . For unit 𝑞 ∈ range(Σtask ), the directional Jacobian energy E𝑥 [𝑞 𝐽𝜑* (𝑥) 𝐽𝜑* (𝑥)𝑞] lower˜ 𝑄 up to ‖Σtask ‖op , yielding the displayed scale 𝜌2𝑞 /𝐿2 when Σtask has eigenvalue 𝜆task bounds 𝐷 ≍ 𝜌2𝑞 𝑞 along 𝑞. Corollary 3.2 extends to strictly proper losses. □ Proof of the sufficient direction (i). By (E) and Lemma 4.10, pick 𝜃0 with zero pmh penalty and 𝐶0 := ℒtask (𝜃0 ) < ∞. The global minimiser satisfies: ℒ𝜆pmh (𝜃𝜆glob ) ≤ ℒ𝜆pmh (𝜃0 ) = 𝐶0 . Expanding the regularised loss: ℒtask (𝜃𝜆glob ) + 𝜆 E𝑥 Tr 𝐽𝜑⊤𝜃 𝐽𝜑𝜃 Σ′ [︁
(︁
)︁]︁
≤ 𝐶0 .
Since both terms on the left are non-negative, the penalty term satisfies: [︁
(︁
E𝑥 Tr 𝐽𝜑⊤ 𝐽𝜑 Σ′
)︁]︁
≤
𝐶0 . 𝜆
+ Under range matching, Σ′ ⪰ 𝜎min (Σ′ ) Πrange(Σtask ) . By Assumption 4.3, the Jacobian has bounded operator norm 𝑀 , so we can lower-bound the trace:
˜ 𝑄 (𝜃) = E𝑥 Tr 𝐽 ⊤ 𝐽𝜑 Σtask 𝐷 𝜑 [︁
(︁
)︁]︁
≤
[︁ (︁ )︁]︁ ‖Σtask ‖op ‖Σtask ‖op · 𝐶0 ⊤ ′ · E . Tr 𝐽 𝐽 Σ ≤ 𝑥 𝜑 𝜑 + + ′ 𝜎min (Σ ) 𝜎min (Σ′ ) 𝜆
˜ 𝑄 = 𝑂(1/𝜆) → 0. □(𝑖) This gives 𝐷 Proof of the necessary direction (ii). Let 𝑞 ∈ ker(Σ′ ) ∩ range(Σtask ) with 𝜌𝑞 > 0 (label correlation along 𝑞). The PMH penalty is zero along 𝑞: 𝑞 ⊤ Σ′ 𝑞 = 0, so the PMH gradient has no component along 𝑞. By Assumption 4.4 applied directionally at the global minimiser: [︁(︁
E𝑥
𝑞 ⊤ 𝐽𝜑⊤ 𝐽𝜑 𝑞
)︁]︁
≥
𝜌2𝑞 𝜌2𝑞 − 𝑂(𝜖) ≥ 𝐿2 2𝐿2
for small enough suboptimality 𝜖 (which holds near the global minimiser for large 𝜆). Then: ˜ 𝑄 (𝜃glob ) ≥ 𝜆task 𝐷 · E𝑥 𝑞 ⊤ 𝐽𝜑⊤ 𝐽𝜑 𝑞 ≥ 𝜆task · 𝑞 𝑞 𝜆 [︁
]︁
𝜌2𝑞 > 0, 2𝐿2
independent of 𝜆. □(𝑖𝑖)
A.6
Theorem B: mismatch cost
Theorem 4.11 splits mismatch into a range floor (Part (i), inherited from Theorem 4.1(i)) and a allocation penalty within 𝒜𝑐 (Part (ii)).
34
Proof of Part (i): range mismatch is 𝜆-independent. Follows immediately from Theo˜ 𝑄 (𝑤(Σ′ )) = Θ(1) (the argument in Appendix A.2 rem A(i): if range(Σ′ ) ̸⊇ range(Σtask ), then 𝐷 2 ˜ shows the lower bound 𝐷𝑄 ≥ 𝜆𝑗 𝑣˜𝑗 > 0 for the missing direction 𝑗). The gap relative to the optimal matched solution: ˜ 𝑄 (𝑤(Σ′ )) − 𝐷 ˜ 𝑄 (𝑤(Σ⋆ )) = 𝐷
∑︁
𝜆𝑖 𝑣˜𝑖2 + 𝑂(𝜆−2 ) = Θ(1).
𝑖 : 𝜇𝑖 =0,𝜆𝑖 >0
Proof of Part (ii): allocation mismatch is quadratic in Frobenius. for 𝑖 ≤ 𝑟. The drift is: 𝑟 ∑︁ 𝜆𝑖 𝑣˜𝑖2 𝑓 (𝜇) := . (1 + 2𝜆𝜇𝑖 )2 𝑖=1
Within 𝒜𝑐 , all 𝜇𝑖 > 0
The Hessian of 𝑓 with respect to 𝜇 at the optimum 𝜇* is: 12𝜆2 𝜆𝑖 𝑣˜𝑖2 𝜕 2 𝑓 ⃒⃒ = = Θ(𝜆−3 ) ⃒ (1 + 2𝜆𝜇*𝑖 )4 𝜕𝜇2𝑖 𝜇* ⃒
(using 𝜇*𝑖 = Θ(𝜆−1/3 ), so (1 + 2𝜆𝜇*𝑖 )4 = 𝑂(𝜆8/3 )). Since 𝜇*𝑖 ∝ (𝜆𝑖 𝑣˜𝑖2 )1/3 and the optimality condition is the same as a KKT constraint on the simplex, the second-order Taylor expansion gives: 𝑓 (𝜇) − 𝑓 (𝜇* ) ≥ 𝜅(𝜆) ‖𝜇 − 𝜇* ‖2 ,
𝜅 = Θ(𝜆−3 ).
The Frobenius bound follows from the diagonal nature of the comparison (the contribution to ‖Σ′ − Σ⋆ ‖2𝐹 from dimension 𝑖 is (𝜇𝑖 − 𝜇*𝑖 )2 ). □
A.7
Lemma C and Lemma C⋆ : Stiefel-manifold equivalence
Lemma 4.12 justifies the isotropic random-𝑊 baseline and its concentration rate in §8 (T7B: matched vs. random 𝐷𝑁 /𝐷𝑆 ≈ 2.98 vs. 3.11 at matched 𝜆). Proof of Lemma C. Let 𝑈 ∈ 𝑉𝑟 (R𝑑𝑥 ) be uniform on the Stiefel manifold (Haar measure). For any orthogonal matrix 𝑄 ∈ 𝑂(𝑑𝑥 ), 𝑄𝑈 is again uniform on 𝑉𝑟 (R𝑑𝑥 ) by rotation invariance of the Haar measure. Therefore: E[𝑈 𝑈 ⊤ ] = E[𝑄𝑈 𝑈 ⊤ 𝑄⊤ ] = 𝑄 E[𝑈 𝑈 ⊤ ] 𝑄⊤ . Since this holds for all 𝑄 ∈ 𝑂(𝑑𝑥 ), the matrix 𝑀 := E[𝑈 𝑈 ⊤ ] must commute with every orthogonal matrix; by Schur’s lemma, 𝑀 = 𝑐𝐼 for some scalar 𝑐. The trace gives Tr(𝑀 ) = 𝑐 𝑑𝑥 = E[Tr(𝑈 𝑈 ⊤ )] = 𝑟, so 𝑐 = 𝑟/𝑑𝑥 and E[𝑈 𝑈 ⊤ ] = (𝑟/𝑑𝑥 )𝐼. □ Proof of Lemma C⋆ . The matrix-valued function 𝑈 ↦→ 𝑈 𝑈 ⊤ − (𝑟/𝑑𝑥 )𝐼 has operator norm at √ most 1 and is Lipschitz in ‖𝑈 ‖𝐹 with constant 2 𝑟 (by ‖𝑈 𝑈 ⊤ −𝑉 𝑉 ⊤ ‖op ≤ ‖𝑈 −𝑉 ‖op (‖𝑈 ‖+‖𝑉 ‖) ≤ √ 2 𝑟‖𝑈 − 𝑉 ‖𝐹 ). Concentration of measure on compact Riemannian manifolds (in particular, on Stiefel manifolds with the canonical metric induced from the orthogonal group) √︀ gives that Lipschitz functions concentrate at rate exp(−𝑐𝑡/𝜎 2 ) for appropriate diameter 𝜎 = 𝑂( 𝑟/𝑑𝑥 ) and 𝑐 > 0. Specialising to the operator-norm function and optimising over the Lipschitz constant gives: 𝑐 𝑑𝑥 𝑡2 > 𝑡 ≤ 𝑑𝑥 exp − 𝑟 op
[︃⃦ ⃦ ⃦ 𝑟 ⃦ ⊤ ⃦ Pr ⃦𝑈 𝑈 − 𝐼 ⃦ 𝑑 ⃦ 𝑥
]︃
(︃
)︃
.
Setting 𝑡 = 𝐶 𝑟 log(𝑑𝑥 /𝛿)/𝑑𝑥 gives the stated bound with probability 1 − 𝛿. □ √︀
35
A.8
Corollaries E and E⋆ : signal-aligned penalty hurts
Corollaries 4.13 and 4.14 formalise why signal-aligned Σ′ is harmful; T5B (identifier vs. keyword) shows the predicted ordering (0.830 → 0.938 vs. 0.830 → 0.738 in rename balanced accuracy). Proof of Corollary E (smooth task loss). Let 𝜑* = arg min𝜑 𝑅(𝜑) and let 𝜑pmh(Σ ) = arg min𝜑 [𝑅(𝜑) + 𝜆 · PMH(Σ′ , 𝜑)]. At 𝜑* , the gradient ∇𝑅 = 0 (unconstrained minimum). The PMH penalty perturbs the encoder by Δ𝜑 such that: ′
∇𝑅(𝜑* ) + 𝜆∇𝜑 PMH(Σ′ , 𝜑* ) = 0 =⇒ Δ𝜑 = −𝜆𝐻 −1 ∇𝜑 PMH(Σ′ , 𝜑* ) + 𝑂(𝜆2 ), where 𝐻 = ∇2𝜑 𝑅(𝜑* ). Since 𝑠⊤ Σ′ 𝑠 ≥ 𝜌2signal > 0, the PMH gradient along 𝑠 is ∇𝑠 PMH(Σ′ , 𝜑* ) = Θ(𝜌2signal ) (the penalty grows as 𝜆 times the Jacobian dot product with Σ′ , which has a component 𝜌2signal along 𝑠). Therefore Δ𝜑𝑠 = Θ(𝜆𝜌2signal ). The risk increase: 1 ′ 𝑅(𝜑pmh(Σ ) ) − 𝑅(𝜑* ) ≈ Δ𝜑⊤ 𝐻Δ𝜑 ≥ 𝑐𝑅 ‖Δ𝜑𝑠 ‖2 = 𝑐𝑅 𝜆2 𝜌4signal . 2 Evaluating at 𝜆 = 1 or normalising by 𝜆-independent factors gives the Ω(𝜌4signal ) bound. □ Proof of Corollary E⋆ (margin-active task loss). When the task loss has a non-vanishing subgradient 𝑔𝑠 along 𝑠 at the unconstrained minimum (hinge SVM, cross-entropy at finite training horizon), the perturbation analysis changes: ∇𝑅(𝜑* ) is not zero but 𝑂(1) along 𝑠. The first-order change in risk is: 𝑅(𝜑pmh(Σ ) ) − 𝑅(𝜑* ) ≥ 𝑔𝑠⊤ Δ𝜑𝑠 − 𝑂(‖Δ𝜑𝑠 ‖2 ) ≥ 𝑐′𝑅 𝜆𝜌2signal , ′
where 𝑐′𝑅 = 𝑔𝑠 · 𝑐Δ𝜑 > 0. Normalising as before gives Ω(𝜌2signal ). □
A.9
Lemmas D1–D7: conditional consistency
^ task on that table. Rates Each lemma is valid only under 𝐴𝑘 (Table 3); the recipe maps 𝐴𝑘 → Σ below are conditional: when the eigengap is small (Office-31, 𝛾 ≈ 1.03), D1 does not identify 𝑊 (§8.1). Shared notation (one line). Σtask = Cov𝑄𝑛 (𝑛) is the population deployment covariance (§2). ^ task is 𝜖-consistent for Σtask if ‖Σ ^ task − Σtask ‖op = 𝑂𝑃 (𝜖𝑁 ) with 𝜖𝑁 → 0. We write An estimator Σ ^ task ) and Π ^ := 𝑊 ^ for a rank-𝑟 orthonormal basis of range(Σ ^𝑊 ^ ⊤. 𝑊 𝑊 Lemma A.8 (D1: cross-domain subspace; assumption 𝐴1 ). Assume 𝑛 = 𝑊 𝜂 with 𝑊 ∈ R𝑑𝑥 ×𝑟 , 𝜂 ∼ 𝒩 (0, 𝐼𝑟 ), and signal coordinates orthogonal to range(𝑊 ). Let 𝑀 ∈ R𝑑𝑥 ×𝑁 stack cross-domain ^ be the top-𝑟 right singular vectors of 𝑀 pair-deltas and class-mean shifts (source/target). Let 𝑊 ⊤ −1 ^ ^ ^ ^ and Σtask := 𝑊 𝑊 . If the empirical covariance 𝐶 := 𝑁 𝑀 𝑀 ⊤ concentrates on 𝐶 := 𝑊 𝑊 ⊤ +𝜎𝑠2 𝐼 with eigengap 𝛾 := 𝜆𝑟 (𝐶) − 𝜆𝑟+1 (𝐶) > 0, then ‖Π𝑊 ^ − Π𝑊 ‖𝐹
(︂√︁ )︂ 2‖𝐶^ − 𝐶‖op ≤ = 𝑂𝑃 𝑟/𝑁 , 𝛾
^ task − and Σ → 1𝑟 𝑊 𝑊 ⊤ when ‖𝑊 ‖2𝐹 = 𝑟. 𝑃
36
^ Davis–Kahan for invariant Proof. Top-𝑟 singular vectors of 𝑀 coincide with top-𝑟 eigenvectors of 𝐶. √ ^ subspaces gives the displayed bound; matrix Bernstein yields ‖𝐶 − 𝐶‖op = 𝑂𝑃 (𝜎1 (𝐶)/ 𝑁 ). Lemma A.9 (D2: isotropic acquisition; assumption 𝐴2 ). If√𝑛 ∼ 𝒩 (0, 𝜎 2 𝐼𝑑𝑥 ) then Σtask = 𝜎 2 𝐼𝑑𝑥 . ∑︀ (𝑖) 2 The scalar 𝜎 ^ 2 := 𝑁1𝑑𝑥 𝑁 ^ 2 𝐼 is 𝜎 2 − 𝜎 2 | = 𝑂𝑃 (1/ 𝑁 ). Matched pmh with Σ′ = 𝜎 𝑖=1 ‖𝑛 ‖2 satisfies |^ the deployment-agnostic optimum of Proposition 3.3. Proof. Cov(𝑛) = 𝜎 2 𝐼 by definition. 𝜎 ^ 2 is the sample second moment of ‖𝑛‖22 /𝑑𝑥 ; concentration is standard. Lemma A.10 (D3: photometric/occlusion mixture; assumption 𝐴3 ). Let 𝑛 draw uniformly from ∑︀ ⊤ label-preserving modes {𝛽1 , . . . , 𝛽𝐾 } ⊂ R𝑑𝑥 . Then Σtask = 𝐾1 𝐾 1 , . . . , 𝛽𝐾 ] and 𝑘=1 𝛽𝑘 𝛽𝑘 . If 𝐵 = [𝛽√︀ ⊤ ⊤ ^ ^ ^ ^ ^ Σtask := 𝑈𝑟 Λ𝑟 𝑈𝑟 is the rank-𝑟 SVD of 𝐵𝐵 /𝐾, then ‖Σtask − 𝑃𝑟 (Σtask )‖op = 𝑂𝑃 ( 𝜆𝑟+1 /𝛾𝑟 ) for 𝑃𝑟 the top-𝑟 projector and 𝛾𝑟 the 𝑟-th eigengap of Σtask . Proof. Σtask is the population second moment of the uniform mixture. Wedin’s theorem applied to 𝐶^ = 𝐵𝐵 ⊤ /𝐾 versus Σtask gives the rate in the spectral gap. Lemma A.11 (D4: domain-shift Gram; assumption 𝐴4 ). If 𝑃𝑆 (𝑦 | 𝑥) = 𝑃𝑇 (𝑦 | 𝑥) and 𝑥𝑇 − 𝑥𝑆 has covariance Σdom under paired (𝑥𝑆 , 𝑥𝑇 ), then Σtask = Σdom at the input level. The per-layer √︀ ̂︂ (ℓ) (𝑥𝑇 )−𝜑(ℓ) (𝑥𝑆 )) satisfies ‖Σ ^ (ℓ) := Cov(𝜑 ^ (ℓ) −Σ(ℓ) ‖op = 𝑂𝑃 (1/ min(𝑛𝑆 , 𝑛𝑇 )) under estimator Σ task task dom Lipschitz 𝜑(ℓ) . Proof. Shared 𝑃 (𝑦 | 𝑥) implies nuisance is marginal shift; sample covariance of differences is consistent for the shift covariance at the representation where differences are measured. Lipschitz composition preserves consistency order. Lemma A.12 (D5: compositional partition; assumption 𝐴5 ). Partition coordinates {1, . . . , 𝑑𝑥 } = 𝒱𝑛 ⊔ 𝒱𝑠 with nuisance confined to 𝒱𝑛 and Cov(𝑛𝒱𝑛 , 𝑦 | 𝑥) = 0. Let 𝐽𝑛 ∈ {0, 1}𝑑𝑥 ×|𝒱𝑛 | select nuisance ⊤ coordinates. Then √ Σtask = 𝐽𝑛 Cov(𝑛𝒱𝑛 ) 𝐽𝑛 and the block-restricted sample covariance is consistent at rate 𝑂𝑃 (1/ 𝑁 ). Proof. Cov(𝑛) = 𝐽𝑛 Cov(𝑛𝒱𝑛 )𝐽𝑛⊤ ; cross-blocks vanish under coordinate-wise label-independence on 𝒱𝑛 . Lemma A.13 (D6: temporal increment covariance; assumption 𝐴6 ). For sequences with labelconstant segments, let ℎ𝑡 be per-step features and Δℎ𝑡 := ℎ𝑡 − ℎ𝑡−1 . Then Σtask = E[Δℎ𝑡 Δℎ⊤ 𝑡 ] (or the analogous content-residual increment in the implementation). The empirical increment √︀ covariance over 𝑁seq sequences converges at rate 𝑂𝑃 (1/ 𝑁seq ). Proof. Label constancy makes within-sequence increments label-preserving; the sample second moment is a standard covariance estimator. Lemma A.14 (D7: adversarial or style-pair Gram; assumption 𝐴7 ). (a) PGD. If 𝛿^(𝑖) are bounded ^ task := 𝑁 −1 ∑︀ 𝛿^(𝑖) 𝛿^(𝑖)⊤ satisfies PGD deltas with ‖𝛿^(𝑖) −𝛿 * (𝑥(𝑖) )‖ ≤ 𝜂√and Cov(𝛿 * ) has rank 𝑟, then Σ 𝑖 ^ task − Cov(𝛿 * )‖op = 𝑂𝑃 (𝜂 + 1/ 𝑁 ). (b) Style pairs. If (𝑥𝑖 , 𝑥style ) are label-equivalent and ‖Σ 𝑖 style ^ task := 𝑁 −1 ∑︀ 𝑠𝑖 𝑠⊤ is consistent for Cov(𝑠) at rate 𝑠𝑖 := 𝑥√ − 𝑥 span the style subspace, then Σ 𝑖 𝑖 𝑖 𝑖 𝑂𝑃 (1/ 𝑁 ). Proof. (a) Triangle inequality on sample vs. population covariance plus perturbation from 𝜂. (b) Direct empirical covariance consistency for bounded 𝑠𝑖 . 37
Eigengap scope (D1, D3, D4, D7). Rates require 𝛾𝑟 (or D1’s 𝛾) above estimation noise; Office-31 has 𝛾 ≈ 1.03 with marginal cross-domain spectrum—wrong-𝑊 and isotropic baselines then tie (§8.1).
A.10
Proposition F: class-layout tdi0 envelope
cls Setting (Proposition 6.2). Labelled probe set {(𝑥𝑖 , 𝑦𝑖 )}𝑁 𝑖=1 , 𝐶 ≥ 2; class-layout TDI0 = 𝑑intra /𝑑inter as in Eq. (7). This is a layout envelope, not a drift identity—trajectory tdi (Eq. (5)) is the primary label-free link to Eq. (3).
˜ 𝑄 (𝜑) = 0, then E𝑥 [Tr(𝐽 ⊤ 𝐽𝜑 Σtask )] = 0, hence 𝐽𝜑 (𝑥)⊤ 𝐽𝜑 (𝑥) Σtask = Step 1 (local invariance). If 𝐷 𝜑 0 for 𝑃𝑋 -a.e. 𝑥 on range(Σtask ). For 𝑛 = 𝑊 𝜂 with 𝑊 spanning range(Σtask ), 𝜑(𝑥 + 𝑛) − 𝜑(𝑥) = 𝐽𝜑 (𝑥)𝑛 + 𝑜(‖𝑛‖) = 𝑜(‖𝑛‖) to first order, so embeddings are locally invariant along deployment directions. Step 2 (layout under noise reduces to clean layout). Consider the idealised noisy embedding ˜ ˜ 𝜑(𝑥) = 𝜑(𝑥 + 𝑛) with 𝑛 ∼ 𝑄𝑛 i.i.d. per sample. To first order 𝜑(𝑥) ≈ 𝜑(𝑥), so within-class scatter ˜ and between-class centroid separation of 𝜑 agree with those of 𝜑 up to 𝑂(E‖𝑛‖). Thus any √︁ layout ˜ = 𝐿(𝜑) + 𝑂( 𝐷 ˜ 𝑄 (𝜑)) functional 𝐿(𝜑) that depends only on clean pairwise geometry satisfies 𝐿(𝜑) by Lipschitz continuity of centroids in finite samples. Step 3 (bound by 𝑑between ). For fixed 𝐶, 𝑑intra is bounded by the diameter of the embedding set, while 𝑑inter ≥ 𝑑between := min𝑐̸=𝑐′ ‖𝜇𝑐 − 𝜇𝑐′ ‖2 for class means 𝜇𝑐 . Hence TDIcls 0 ≤ diam/𝑑between =: Φ(𝑑between ) with Φ continuous and increasing when class means are separated. □ ˜ Remark. Low TDIcls 0 does not certify low 𝐷𝑄 , and the converse fails at intermediate drift; do not substitute layout tdi for trajectory tdi in the recipe.
B
Per-task experimental supplements
§8 quotes only headline numbers; this appendix is the full record for all thirteen blocks (assumption 𝐴𝑘 , Σtask estimator, protocol, arms, tables, figures). Frozen JSON paths appear only in Appendix B. tdi, 𝐷𝑁 , and 𝐷𝑆 are defined once in §6; subsections below report values. How to read result tables and figures Bold marks the best value in that column (↑ higher better unless the column header says ↓). Where a block has a pre-registered headline metric (e.g. rename ratio in T5B, rare-5 mIoU in T4B), the matched arm is named in the caption even when a baseline or control wins a secondary column. Italic marks predicted failures (Cor. E/E⋆ ). Several blocks are partial passes: read task and geometry separately— residual decoupling (e.g. T2B: E1 leads geometry and heavy Gaussian vs. no-pmh; B0/E1-no-pmh/VAT split clean/mean/saliency) or geometry gains a headline task scalar misses (T6A, T7A, T7B); see §7.2.
38
Reproducibility quickstart All reported numbers are loaded from frozen JSON (not hand-edited). Spot-check the six headline claims below; field names match the repository exports. 1. T6B matched ordering (§8.6). T6/task6B/artifacts/multiseed/multiseed_summary.json: at stress 𝐿=3.0, matched mean balanced acc. 0.410 vs. wrong-𝑊 0.349 vs. baseline 0.279. 2. T2B partial pass (§8.2, geometry vs. task). T2/Task2B/runs/eval_out_robust/compare_results_robust.json: E1 (pmh) L4 drift 10.1 vs. B0 22.4; Gaussian 𝜎=0.10 acc. E1 69.7% vs. E1-no-pmh 66.0%; B0 clean 90.7%; E1-no-pmh mean shift 86.9%. 3. T7A 7B sycophancy (§8, RM arm). T7/task7A/results/behavioral_eval_7b_tqa500/behavioral_eval.json: sycophancy.m atched.sycophancy_rate = 0.1346 vs. baseline = 0.3846; matched_minus_baseline.hones t_preference_delta CI excludes zero. 4. T7B subspace staircase (§8.7). T7/task7B/results/seed_007/interp/paper_metrics.json: pmh_aniso_wrong_W PGD@4 11.05% → pmh_aniso_grad_W 15.62% → pmh_aniso_p000 21.14% → pgd 44.84%. Clean 64.61% vs. baseline 79.39% (−14.8 pp). 5. T5B keyword-pmh (Cor. E⋆ ). T5/Task5B/runs/Clone_pilot/pipeline_summary.json: E1S.rename_bacc_ratio = 0.7379 vs. B0 = 0.8297. 6. T4B motorcycle collapse (Cor. E⋆ ). T4/Task4B/runs/Task4B/eval/summary.json: E1.iou[3] = 0.0246 vs. B0.iou[3] = 0.1020 vs. E1_multiscale.iou[3] = 0.1853. 7. T1 Office-31 (Lemma D1). T1/classical_pmh/results/office31_results.json: cross_domain_svd.gap_ratio_r_o ver_r_plus_1 ≈ 1.028; summary.svm: CORAL 0.252 vs. PMH_matched 0.233. Reference library (independent repo). Install matching-pmh from https://github.com/vishalstark512/matching-pmh (estimate_sigma_task, pmh_penalty_on_rep, Lemmas D1–D7 dispatch). The paper’s task folders are not required to use the library. Regenerate all figures: python-mscripts.paper_figures.build_all (from repository root).
B.1
B.1: T1 — Classical PMH, oracle-𝑊 and data-driven drift
^ when 𝑊 is unknown; Main-text anchor (§8.1). 𝐴1 /Lemma A.8: cross-domain SVD for 𝑊 oracle 𝑊 when identifiable. Verdict: pass—oracle ridge MSE flat at 0.101 vs. B0/iso/wrong 0.553; oracle-𝑊 matched wins every classifier×dataset cell; DCT drift matched SVM 70.2% vs. B0 54.8%. Predicted fail: Office-31 CORAL 25.2% vs. matched pmh 23.3% (eigengap ≈ 1.03, Lemma A.8).
39
Setting. 𝑛 = 𝑊 𝜂, 𝜂 ∼ 𝒩 (0, 𝐼𝑟 ). Classifiers: ridge (closed form), SVM, soft 𝑘-NN, logistic. Data: synthetic linear-Gaussian (𝑑=50, 𝑟=5, 20 seeds); MNIST / Fashion-MNIST (oracle-𝑊 and DCT drift); SVHN; Office-31 Amazon→DSLR (ResNet-18 features). Ridge (Theorem 4.1(i)). Oracle matched ridge MSE stays at 0.101 for 𝜎test ∈ [0, 12] while B0, isotropic, and wrong-𝑊 rise to 0.553; the closed-form curve matches Monte Carlo on all seeds— range matching removes drift, nothing else does. Oracle-𝑊 classification results.
Table 9 summarises the four-arm comparison.
Table 9: Oracle-𝑊 classification accuracy (%) on MNIST and Fashion-MNIST at 𝜎test = 5𝜎train , 3 seeds. Bold = matched arm (headline; strict ordering matched > iso ≈ wrong ≈ B0). Classifier
B0
E1-iso
E1-matched
E1-wrong
MNIST SVM 𝑘-NN Logistic
90.3 88.5 79.1
90.2 88.6 82.7
94.2 90.5 89.0
90.0 88.4 82.5
Fashion-MNIST SVM 76.3 𝑘-NN 78.1 Logistic 65.8
78.4 78.1 72.7
84.8 78.9 81.8
76.2 77.9 72.8
^ (Fashion-MNIST DCT). Subspace alignment 𝑊 ^ vs. 𝑊 : 0.959. SVM: matched Data-driven 𝑊 70.2%, wrong-𝑊 54.3%, B0 54.8%; logistic 59.8% → 69.0% matched vs. B0. Hard 𝑘-NN projection can regress (69.5% → 62.2%); soft metric 𝑀 = 𝛽𝑃𝑊 ⊥ + 𝛼𝑃𝑊 with CV’d 𝛼 restores gains (73.0% Fashion, 82.2% MNIST DCT) and collapses to hard projection under oracle 𝑊 . Baselines (CORAL, LMNN, IRM). Oracle 𝑊 : matched-pmh wins every cell (Table 10). ^ on DCT drift: pmh leads CORAL on Fashion SVM/logistic; on MNIST ties SVM Estimated 𝑊 and wins soft 𝑘-NN (82.0% vs. 76.0%); IRM wins logistic by 5.7 pp when given an environment label. Table 10: Head-to-head accuracy (%) under oracle 𝑊 on MNIST and Fashion-MNIST, 3 seeds. Bold = matched pmh (headline; wins every cell). CORAL is the strongest competitor. Dataset
Classifier
B0
pmh
CORAL
LMNN
MNIST MNIST MNIST Fashion-MNIST Fashion-MNIST Fashion-MNIST
SVM 𝑘-NN soft Logistic SVM 𝑘-NN Logistic
90.9 88.8 82.4 75.8 76.4 72.9
94.0 89.8 87.8 84.6 76.9 80.9
90.3 86.3 83.7 76.6 75.2 75.5
83.9 84.5 81.0 38.0 74.3 64.9
Extras. California Housing (injected rank-4 nuisance, 10 seeds): matched ridge MSE flat at 0.45 for 𝜎test ∈ [0, 5] vs. isotropic 0.43 → 1.38 and wrong-𝑊 3.77. EB-GP with 𝑀 = 𝛼𝑃𝑊 + 𝛽𝑃𝑊 ⊥ 40
selects 𝛼/𝛽 = 0.0016 (true 𝑊 ) vs. 1.28 (random 𝑊 ). SVHN oracle cross-domain: 23.4% → 43.9% (+20.5 pp). Office-31 (predicted failure, Lemma A.8). CORAL 25.2% vs. matched pmh 23.3% SVM; ^ is unreliable and linear eigengap 𝜆𝑟 /𝜆𝑟+1 ≈ 1.03 at rank 32 on the 200-sample target pool ⇒ 𝑊 pmh loses to CORAL’s moment alignment—the framework’s marginal-spectrum case (§8.1), not a refutation. Figure 8 (right panel) plots the head-to-head.
(a) Closed-form Theorem A curve (b) Fashion-MNIST four-arm SVM (c) Office-31 (CORAL wins; pre(matched MSE flat). accuracy. dicted failure of D1).
Figure 8: T1 plots. Theorem A ridge check; Fashion-MNIST four-arm SVM; Office-31 head-tohead.
B.2
B.2: T2A — ImageNet ViT-B/16, isotropic input nuisance
^ to Main-text anchor (§8.2). 𝐴2 /Lemma A.9: matched pmh is isotropic (Prop. 3.3); no 𝑊 estimate. Verdict: pass—ImageNet-C mean 82.9% → 87.2% (+4.3 pp); trajectory TDI −58% at 𝜎=0.10 (matched = isotropic by design). Setting. ViT-B/16, 100-class ImageNet val subset (5,000 images). Arms: pretrained, ERM, isotropic pmh (CLS-token Gaussian probe; layer-averaged trajectory tdi, Eq. (5)). Clean, Gaussian stress, and geometry. Clean: 97.02% → 96.84% (−0.18 pp). At 𝜎=0.20: ERM 85.92%, pmh 94.20% (+8.28 pp). ImageNet-C (severity 3, unseen at train): mean top-1 82.90% → 87.19% (+4.29 pp); largest gains on noise/blur families aligned with the Gaussian probe (frost +12.04 pp, glass blur +8.68 pp). At 𝜎=0.10: trajectory TDI 0.0656 → 0.0275 (−58%); ‖𝐽‖𝐹 76.00 → 69.10 (−9.1%, 30 batches). Layer-averaged input-Gaussian TDI by probe 𝜎 (ERM → pmh): 0.00694 → 0.00318 (𝜎=0.01), 0.03975 → 0.01534 (0.05), 0.06556 → 0.02752 (0.10)—each a ∼ 55–60% relative reduction. Figure 9 plots TDI and ImageNet-C breakdown.
41
(a) TDI vs. Gaussian 𝜎: PMH suppresses geometric drift across the entire stress range.
(b) ImageNet-C per-corruption mean error (severity 3). PMH dominates noise families.
Figure 9: T2A plots. TDI vs. probe 𝜎; ImageNet-C per-corruption means.
B.3
B.3: T2B — Chest X-ray, isotropic acquisition nuisance
Main-text anchor (§8.2). Same 𝐴2 /Lemma A.9 as T2A; illustrates geometry–task dissociation (§7.2): matched pmh can compress drift and beat the two-view control on heavy Gaussian while clean, mean-shift, and saliency scalars split across B0, E1-no-pmh, and VAT. Verdict: partial pass—E1 (pmh) leads geometry (lowest pooled L4 drift 10.1 vs. 22.4 B0) and beats E1-no-pmh on heavy Gaussian (69.7% vs. 66.0% at 𝜎=0.10); E1-no-pmh leads mean shift (86.9%); B0 leads clean (90.7%) but collapses on Gaussian eval (62.5%); VAT leads saliency (0.669). Setting (protocol v3). ResNet-18 + 512-d embedding, binary pneumonia CXR; **30 epochs**, Adam 10−4 (all arms). Arms: B0, VAT, E1-no-PMH (two-view task loss, no matching term), E1 (two-view + isotropic multi-scale pmh on stages 2–4). Training nuisances apply to the **two-view arms only** (B0/VAT: standard augmentation); Gaussian 𝜎 ∈ [0.05, 0.10] per batch, per-image intensity ∈ [0.85, 1.15]; PMH cap on **total** loss (T2A-aligned); task_mix=0.15. Eval stress matches eval_robust.py (identical for all arms). pmh uses L2-normalised feature MSE (isotropic acquisition proxy), not Hutchinson trace. Frozen JSON: T2/Task2B/runs/eval_out_robust/com pare_results_robust.json. Geometry vs. task. Under protocol v3, E1 (pmh) compresses pooled L4 drift best (10.1 vs. 15.7 E1-no-pmh vs. 22.4 B0 at 𝜎=0.08; Table 12) and beats the two-view control on heavy Gaussian accuracy (69.7% vs. 66.0% at 𝜎=0.10; Figure 10, centre). E1-no-pmh leads mean-shift accuracy (86.9% vs. 85.9%); B0 leads clean (90.7%, AUC 0.963) without training-time acquisition noise but still collapses on Gaussian rows (62.5%). VAT leads saliency (0.669; Figure 10, right) and attains the highest worst-shift accuracy overall (72.8%, limited by zoom). Residual dissociation: geometry (E1) ̸= clean/mean-shift (B0/E1-no-pmh) ̸= saliency (VAT)—the theory predicts matched suppression of Σtask -aligned drift, not dominance on every task scalar.
42
Geometry under Gaussian σ = 0.10 20 10 0
B0
VAT
ew PMH
-vi Two
Worst-shift accuracy 60 40 20 0
VAT
B0
ew PMH
-vi Two
Saliency cosine similarity
Worst-shift accuracy (%)
Embedding drift ‖Δϕ‖2
T2B: Chest X-ray --- PMH compresses drift; task scalars split across arms
Saliency stability 0.75 0.50 0.25 0.00
B0
VAT
ew PMH
-vi Two
Figure 10: T2B — Chest X-ray geometry and robustness. Left: embedding drift under Gaussian acquisition noise (↓ better); centre: worst-shift accuracy (↑); right: saliency stability (↑). E1 (pmh) compresses drift and beats E1-no-pmh on heavy Gaussian; B0 leads clean but collapses on Gaussian; VAT leads saliency. VAT (mismatched baseline). Virtual adversarial training trades clean accuracy (86.2% vs. B0 90.7%) for Gaussian-shift gain (77.8% vs. B0 62.5% at 𝜎=0.10) and the best saliency (0.669), but mean-shift accuracy (83.4%) trails two-view arms and pooled L4 drift (11.3) exceeds E1 (10.1). VAT is generic robustness, not matched acquisition geometry. Stage-wise drift (Table 12). At 𝜎=0.08, pooled stage-4 drift explodes for B0 (22.38) and compresses for two-view arms (10.1–15.7); E1 (pmh) is tightest on L4. E1-embed-only is intermediate on stage 4 (14.35) with the highest clean acc in the ablation (91.0%). Table 11: T2B headline metrics (frozen JSON, protocol v3). Bold = best per column (↑ acc/saliency; ↓ drift). B0 wins clean; VAT worst-shift and saliency; E1 (pmh) L4 drift. Arm B0 VAT E1-no-PMH E1 (pmh)
Clean acc (%)
Worst-shift acc (%)
Mean L4 drift ↓
Saliency ↑
90.71 86.22 90.22 89.10
62.50 72.76 66.03 69.71
22.38 11.29 15.72 10.14
0.656 0.669 0.645 0.629
Table 12: T2B stage-wise mean L2 embedding drift at Gaussian 𝜎=0.08. Bold L4 = lowest drift (↓); bold clean acc = highest (↑). Arm
L1
L2
L3
L4 (pooled)
Clean acc
B0 E1-no-PMH E1 (PMH) E1-embed-only
2.05 2.32 1.42 2.14
1.69 1.14 0.78 1.17
2.37 1.40 0.86 1.50
22.38 15.72 10.14 14.35
90.71 90.22 89.10 91.03
43
Table 13: T2B robust accuracy (%) by perturbation type (protocol v3). Bold = highest accuracy in that row. B0 leads clean and several photometric rows; VAT wins heavy Gaussian; E1 (pmh) beats E1-no-pmh at 𝜎=0.10; B0 collapses on Gaussian noise.
B.4
Perturbation
B0
VAT
clean gaussian_0.05 gaussian_0.10 intensity_0.7 intensity_1.3 gamma_0.8 gamma_1.2 rotate_5 rotate_10 zoom_1.1 zoom_0.9 contrast_0.8 contrast_1.2 blur_3
90.71 62.50 62.50 84.94 85.10 90.14 90.38 91.03 90.54 93.75 80.77 90.46 89.42 86.06
86.22 83.57 77.80 76.12 85.82 86.38 86.06 88.30 88.78 89.90 72.76 82.13 87.90 79.17
E1-no-pmh E1 (pmh) 90.22 86.22 66.03 86.62 88.46 90.46 90.38 90.46 90.30 92.31 82.85 89.34 90.71 86.06
89.10 84.78 69.71 86.22 88.78 87.98 89.10 89.42 90.54 92.31 78.85 89.42 87.90 81.73
B.4: T3A — COCO 2D pose estimation, occlusion nuisance
Main-text anchor (§8.3). 𝐴3 /Lemma A.10: augmentation-delta Gram for occlusion modes; matched anisotropic pmh vs. isotropic and VAT. Verdict: pass—E1-aniso +22 pp [email protected] and −84% occlusion drift; VAT fails (wrong consistency geometry). Setting. ResNet-18 + heatmap head, 17 keypoints, COCO val (≥ 5 visible joints, 2562 crops). Arms: baseline (50 ep, occ. aug. only), VAT (30 ep), E1 iso / E1-aniso (30 ep, subspace pmh along ^ ). 𝑊 ^ : top-16 right singular vectors of a 1024 × 16 occlusion-delta matrix; spectrum ratio 1.08 𝑊 (multi-directional nuisance, weak eigengap). Table 14: COCO pose PCK and embedding drift by arm (↑ PCK, ↓ 𝐷occ ). Bold = matched E1aniso (headline). 𝐷occ is mean embedding drift at occ0.10 , normalised to the baseline. Arm baseline VAT E1 (iso) E1-aniso
[email protected] (clean)
[email protected] (occ0.40 )
[email protected] (clean)
𝐷occ
32.06 13.95 34.36 54.49
23.62 3.93 21.96 35.50
60.35 35.15 62.89 79.79
1.00 (ref) 1.90 0.20 0.16
Results. E1-aniso: [email protected] 32.1% → 54.5% (+22.4 pp vs. baseline; main rounds +22.3 pp) and 𝐷occ −84%. VAT collapses ([email protected] 14.0%)—adversarial consistency ̸= matched occlusion geometry. Figure 12 plots PCK vs. occlusion level. Directional probe (𝐷𝑁 /𝐷𝑆 , Table 15). At Gaussian 𝜎=0.05, 𝐷𝑁 /𝐷𝑆 are drifts along range(𝑊 ) vs. its orthogonal complement. E1-aniso minimises 𝐷𝑁 (0.000085) and 𝐷𝑁 /𝐷𝑆 (0.0216 vs. baseline 0.0379, −43%); iso pmh also cuts isotropic drift ∼ 7×—gains are subspace-targeted, not uniform shrinkage. Direct occlusion at ratio 0.10: drift 0.126 → 0.020 (aniso, −84%); VAT increases drift to 0.240. Figure 11 shows one illustrative val crop under 40% occlusion.
44
Table 15: T3A directional Gaussian probe at 𝜎=0.05 (↓ all columns). Bold = column best. E1aniso wins 𝐷𝑁 and 𝐷𝑁 /𝐷𝑆 (headline geometry); E1 (iso) has lowest iso drift. Arm
Iso drift
𝐷𝑁 (nuisance)
𝐷𝑆 (signal)
𝐷𝑁 /𝐷𝑆
baseline VAT E1 (iso) E1-aniso
0.02593 0.01283 0.00386 0.00394
0.000981 0.000363 0.000106 0.0000850
0.02590 0.01281 0.003857 0.003943
0.0379 0.0283 0.0275 0.0216
T3A qualitative COCO example under 40% occlusion (000000143961.jpg)
Clean crop + ground truth
Baseline prediction [email protected]=0%
Occluded input + ground truth
Aniso-PMH prediction [email protected]=67%
Figure 11: T3A qualitative (illustrative). 40% occlusion stress (Table 14 regime): GT keypoints (cyan), baseline (red), E1-aniso (green). One deterministic high-gain val crop; aggregates are in Table 14.
(a) [email protected] vs. occlusion level: E1-aniso remains well above all other arms under increasing occlusion.
(b) Mean keypoint error under attack: matched occlusion-pmh retains the lowest error.
Figure 12: T3A plots. PCK vs. occlusion; mean keypoint error under attack.
B.5
B.5: T3B — NYU Depth V2, photometric nuisance
Main-text anchor (§8.3). Same 𝐴3 /Lemma A.10 with a strong photometric eigengap; wrong𝑊 tests Theorem 4.11(i). Verdict: pass—E1-aniso best on hard photometric metrics; E1-wrong AbsRel +18% (range mismatch).
45
Setting. ResNet-18 + U-Net, SiLog loss, NYU Depth V2 (50 epochs, batch 16). Arms: baseline; ^ in rgb01); E1-wrong (random rank-32 𝑊 ). 𝑊 ^ from E1 (iso pmh + jitter); E1-aniso (rank-32 𝑊 ^ photometric aug. deltas: 𝜆1 =42.7, 𝜆2 =12.2 (3.5× gap); span(𝑊 ) captures 88% brightness, 92% gamma, 66% contrast attack energy. Key results. E1-aniso leads on every hard photometric metric (Figure 13; Table 16). E1-wrong raises clean AbsRel 0.203 → 0.240 (+18%). A 5∘ rotation probe is out of family: all photometric arms degrade sharply (AbsRel ≈ 2×), as expected when Σtask is mis-specified. Figure 14 shows one clean vs. hard-photometric example. Table 16: T3B NYU Depth V2 summary (frozen JSON). Headline: combined-hard AbsRel (↓). Bold = matched E1-aniso. E1-wrong clean AbsRel is above baseline (range mismatch). Arm
Clean AbsRel
Combined-hard AbsRel
Combined-hard RMSE
Clean 𝛿1
0.203 0.195 0.197 0.240
0.242 0.219 0.215 0.267
0.921 0.868 0.846 1.179
0.671 0.684 0.687 0.456
baseline E1 (iso) E1-aniso E1-wrong
0.2 0.1 0.0
B0
Iso
o
Anis
-W
ng Wro
Combined-hard photometric 0.2 0.1 0.0
B0
Iso
o
Anis
-W
ng Wro
RMSE (lower better)
Clean AbsRel
AbsRel (lower better)
AbsRel (lower better)
T3B: photometric subspace PMH beats isotropic and wrong-W controls
Tail error under same attack 1.0 0.5 0.0
B0
Iso
o
Anis
-W
ng Wro
Figure 13: T3B — NYU depth photometric stress. Clean and combined-hard photometric metrics by arm. Wrong-𝑊 degrades below baseline; E1-aniso is consistently best on hard photometric AbsRel/RMSE.
(a) Clean image (training distribution).
(b) Same scene under combined-hard photometric attack (brightness + gamma + contrast).
Figure 14: T3B qualitative. Same scene: clean (left) vs. combined-hard photometric attack (right); aniso-pmh lowest AbsRel on this example.
B.6
B.6: T4A — DomainNet Real→Sketch, hierarchical domain shift
^ (ℓ) ; pixel-isotropic Main-text anchor (§8.4). 𝐴4 /Lemma A.11: per-layer cross-domain Gram Σ task pmh is the wrong estimator at this scale. Verdict: pass—multiscale +3.31 pp; iso tied with B0 46
(wrong estimator tier). Setting. ResNet-50, DomainNet Real→Sketch; 20 epochs, batch 64, Gram rank 64. Arms: B0 (source CE), E1 (isotropic pixel pmh), E1-multiscale (per-layer Gram pmh on class-aligned source/target batches plus unlabelled target RGB for Gram estimation). Results. Test accuracy: B0 38.84%, E1 39.34% (+0.5 pp, tied), E1-multiscale 42.15% (+3.31 pp); best val and test peak at epoch 18 (same as B0). Class alignment is required: misaligned ^ task and erase the gain (early failed runs also had reversed Gram batches mix class priors into Σ pmh sign, killed MSE scaling, and too-small head LR—all fixed in the shipped protocol). Per-layer TDI (Table 17, Figure 15). Image-level layout tdi on 2,000 target embeddings. Iso E1 lowers early-layer TDI but does not move accuracy. E1-multiscale has layer-4 TDI 1.258 vs. B0 1.246 yet the largest inter-class separation (19.79 vs. 19.08): the +3.31 pp gain comes from widening between-class structure at the head, not from uniformly tighter clusters—layout tdi is not monotone with target accuracy here. Table 17: T4A image-level TDI by layer (Real→Sketch target embeddings). TDI: ↓ tighter; interclass sep.: ↑ better for the head. Bold = column best. Headline test acc: E1-multiscale 42.15% (Table 18). Layer
B0
E1 (iso)
E1-multiscale
layer1 layer2 layer3 layer4 / avgpool
1.702 1.719 1.628 1.246
1.633 1.682 1.610 1.227
1.683 1.702 1.629 1.258
Inter-class sep. (layer4)
19.08
19.16
19.79
Table 18: T4A test accuracy (%). Bold = matched E1-multiscale (headline). Iso E1 ≈ B0 (wrong estimator tier). Arm B0 E1 (iso-pixel) E1-multiscale
Test acc (%)
Δ vs. B0
38.84 39.34 42.15
— +0.50 +3.31
47
Figure 15: T4A TDI panel. Per-layer layout tdi on Real→Sketch; multiscale pmh wins accuracy via final-layer class separation, not lowest TDI.
B.7
B.7: T4B — GTA5→Cityscapes Rare-5, sim-to-real
Main-text anchor (§8.4). Same 𝐴4 /Lemma A.11; rare-5 mIoU isolates sim-to-real-sensitive classes. Iso-pixel pmh is Cor. 4.14 on motorcycle; multiscale Gram is the matched estimator (Figure 7). Verdict: pass on multiscale (+11.1 pp rare-5); predicted fail on iso-pixel motorcycle (Cor. 4.14). Setting. DeepLabV3-ResNet50; five rare Cityscapes classes (traffic light, sign, rider, motorcycle, bicycle); GTA5 train, Cityscapes val; 30 epochs. Arms: B0, E1 (pixel-isotropic pmh), E1-multiscale (per-layer Gram on unlabelled Cityscapes RGB). No target labels at train except for Gram collection on E1-multiscale. Results. Rare-5 mIoU: B0 19.68%, E1 19.99% (tied), E1-multiscale 30.75% (+11.1 pp; 95% CIs non-overlapping). Per-class IoU and the motorcycle negative control: Table 19: Per-class IoU on Cityscapes rare-5, GTA5→Cityscapes (%↑). Bold = column best. Headline rare-5 mIoU: E1-multiscale (matched). Traffic sign: iso-E1 wins (not matched); motorcycle: iso-E1 2.46 (Cor. 4.14 predicted fail). Class
B0 (%)
E1-iso (%)
E1-multiscale (%)
Traffic light Traffic sign Rider Motorcycle Bicycle
28.79 38.71 14.52 10.20 6.16
28.55 43.84 17.35 2.46 7.73
39.40 40.78 28.98 18.53 26.05
mIoU (rare-5)
19.68
19.99
30.75
Iso-pixel pmh collapses motorcycle IoU 10.2% → 2.5% (Cor. 4.14): precision 17.6% but recall 2.8%; multiscale restores 18.5% (35.2% / 28.1% P/R). Bicycle: B0 recall 6.2% on rare pixels; only multiscale reaches 26.8% recall with 90.2% precision. Figures 16 and 17 plot aggregate and qualitative rare-class recovery.
48
Pixel-aligned TDI (Table 20). Rare-5 label map, 2,000 pixels per layer×arm. E1 is worse than B0 at every layer; E1-multiscale wins at layer 4 (2.208 vs. 2.820) with inter-class separation ≈ 24.6 vs. B0 15.0 and E1 11.5 (intra-mean/TDI)—matched Gram widens rare-class structure where the segmentation head reads. Table 20: T4B pixel-aligned TDI by layer (↓ TDI; ↑ inter-class sep.). Bold = column best. E1multiscale wins layer-4 TDI and separation (mechanism for +11.1 pp rare-5 mIoU). Layer
B0
E1 (iso-pixel)
E1-multiscale
layer1 layer2 layer3 layer4
5.920 5.290 3.610 2.820
6.200 5.587 4.126 3.391
6.163 5.638 3.397 2.208
Intra mean L2 (L4) Inter-class sep. (L4)
42.29 15.0
38.89 11.5
54.33 24.6
T4B: GTA5 to Cityscapes rare-5 segmentation
Per-class: motorcycle/bicycle recovery
20 10 0
B0 Iso Multiscale
40
IoU (%)
Rare-5 mIoU (%)
Aggregate rare-class transfer 30
30 20 10
B0
Iso
0
Multiscale
traffic light
traffic sign
rider
motorcycle bicycle
Figure 16: T4B — GTA5→Cityscapes rare-5. Left: mIoU by arm. Right: per-class breakdown; isotropic pixel pmh collapses motorcycle IoU.
Figure 17: T4B qualitative (illustrative). Rare-class crop: GT, B0, iso-pixel pmh, multiscale pmh; multiscale recovers motorcycle/rider missed by B0 and iso.
B.8
B.8: T5A — QM9 molecular regression, position noise
Main-text anchor (§8.5). 𝐴5 /Lemma A.12: nuisance-block position covariance; Theorem 4.1(ii) allocation tradeoff at large 𝜎pos (Figure 18). Verdict: predicted tradeoff —large-noise preset −20% eval MAE at 𝜎=0.20 Å with clean-cost (Thm. A(ii)); VAT mismatched control. Setting. MolGCN on QM9 (19 targets, 100 epochs). Arms: B0; VAT (node-feature adversarial, mismatched); E1 presets none (𝜎pos =0.01 Å) and paper_e1_node (0.15 Å; stronger pmh on coordinate block). 49
Results. Table 21 and Figure 18 report clean vs. deployment MAE and the Thm. A(ii) allocation tradeoff. Graph embedding drift at 𝜎=0.01 Å compresses 0.547 → 0.360 → 0.225 (B0 / E1-small / E1-large preset, −34% / −59%). Table 21: T5A QM9 aggregate MAE summary (frozen JSON). Headline at deployment: MAE at 𝜎pos =0.20 Å (↓). E1-large wins robustness at a clean-cost; E1-small is clean-optimal; VAT is a mismatched control. Arm
Clean MAE
MAE @ 𝜎=0.20 Å
Δ clean vs. B0
25.08 24.92 25.69 28.15
52.13 47.42 41.54 —
— −0.16 +0.61 +3.07
B0 E1-small (none) E1-large (paper_e1_node) VAT (mismatched)
Noise sweep
Clean/robust Pareto MAE at σ = 0.20 \AA
Aggregate QM9 MAE (lower better)
T5A: QM9 position-noise robustness trades clean fit for large-noise stability
B0 VAT PMH small-noise PMH large-noise
50 40 30 0.00
0.05
0.10
0.15
0.20
Position-noise amplitude σ (\AA)
PMH large-noise
42.5 45.0
VAT PMH small-noise
47.5 50.0
B0
52.5 28
27
26
25
Clean MAE
Figure 18: T5A — QM9 molecular regression. Left: aggregate MAE vs. position noise 𝜎 for B0, VAT, and two PMH operating points. Right: clean MAE vs. large-noise MAE at 𝜎 = 0.20 Å. The small-noise preset is clean-optimal; the large-noise preset buys robustness with a clean-MAE cost, while VAT remains a mismatched control.
B.9
B.9: T5B — BigCloneBench code clone detection
Main-text anchor (§8.5). 𝐴5 /Cor. 4.14: identifiers = nuisance, keywords = signal. E1 = matched identifier-pmh; E1S = signal-aligned control (Figure 6). Verdict: pass on E1 (0.830 → 0.938 rename ratio); predicted fail on E1S (0.738, Cor. 4.14). Setting. CodeBERT, 10 epochs; 20,000 train / 5,000 eval. Arms: B0, VAT, E1 (identifier pmh + rename aug.), E1S (keyword pmh, wrong partition). Primary metric: rename_bacc_ratio (rename robustness / clean). Results. Ordering on rename ratio: E1 0.938 > B0 0.830 > VAT 0.799 > E1S 0.738 (below baseline—Cor. 4.14). Partial-rename sweep: E1 leads at fractions 0.25, 0.50, 0.75, 1.00 (Figure 19).
50
Balanced-accuracy ratio (renamed / clean)
Table 22: BigCloneBench rename retention and robustness metrics. Bold = matched E1 on headline columns (rename_bacc_ratio, rename bacc, rename F1). Clean bacc best = VAT (0.9384; not the headline metric). Arm
Clean bacc
Rename bacc
Rename ratio
Rename F1
B0 VAT E1 E1S
0.9317 0.9384 0.9306 0.9328
0.7730 0.7502 0.8732 0.6883
0.8297 0.7994 0.9383 0.7379
0.435 0.405 0.790 0.459
T5B: rename retention under partial renaming E1 (matched) preserves; E1S (signal-W) falls below baseline 1.0 0.9 0.8 0.7
B0 VAT E1 (matched) E1S (signal-W)
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Rename fraction
Figure 19: T5B rename sweep. rename_bacc_ratio vs. rename fraction; E1 tracks clean, E1S falls below B0 (Cor. 4.14).
B.10
B.10: T6A — Whisper-small accent robustness
Main-text anchor (§8.6). 𝐴6 /Lemma A.13: content-residual scatter for accent/speaker nuisance; geometry–task dissociation (§7.2)—accent supervision can beat WER but not TDI. Verdict: partial pass—matched pmh wins TDI and WER 23.3% → 14.6%; accent-adapted best WER only (geometry̸=task). Setting. Whisper-small, 4k LibriSpeech utterances, 5 epochs. Arms: baseline; matched contentresidual pmh (rank-32 PCA on 56,387 residuals, 74.4% variance); wrong-𝑊 ; accent-adapted (labelsupervised). Wrong-𝑊 trains much slower (epoch-2 loss 0.368 vs. 0.079 matched). Table 23: Whisper WER and geometry metrics by arm (↓ all columns). Bold WER = lowest task score; bold geometry = matched pmh (headline geometry). Accent-adapted wins WER without TDI gain; pmh-wrong-𝑊 also beats matched on WER (14.03 vs. 14.63) but not on geometry. Arm baseline pmh-matched pmh-wrong-𝑊 accent-adapted
WER (%)
TDI
𝐷𝑁
𝐷𝑆
𝐷𝑁 /𝐷𝑆
23.26 14.63 14.03 13.91
1.096 0.381 0.644 1.101
0.937 0.268 0.560 0.859
0.641 0.214 0.369 0.580
1.463 1.256 1.516 1.481
51
WER and geometry (Table 23). Matched pmh: LibriSpeech-other WER 23.3% → 14.6%, TDI 1.10 → 0.38 (−65%), best 𝐷𝑁 /𝐷𝑆 (1.26). Accent-adapted WER 13.9% but TDI ≈ 1.10 (near baseline): task win without geometry repair. Figure 20 plots all four metrics by arm. T6A: Whisper-small accent perturbations Accent shift (LibriSpeech other)
Temporal geometry drift 1.00 0.75
15
TDI
WER
20
10
0.25
5 0
0.50
B0
Matched
Wrong-W
0.00
Accent
B0
Matched
Wrong-W
Accent
DN/DS
DN (nuisance push) 1.5 0.8 1.0
0.6 0.4
0.5
0.2 0.0
B0
Matched
Wrong-W
0.0
Accent
B0
Matched
Wrong-W
Accent
Figure 20: T6A — Whisper accent geometry. TDI, 𝐷𝑁 , 𝐷𝑆 , and WER by arm. Matched pmh achieves the cleanest 𝐷𝑁 /𝐷𝑆 balance; accent-supervised adaptation reduces WER further without the same geometric correction.
B.11
B.11: T6B — UCI HAR sequential robustness
Main-text anchor (§8.6). 𝐴6 /Lemma A.13: sensor-scatter 𝑊 (rank 48, 99.3% aug. variance explained); Lemma 4.12 on wrong-𝑊 . Verdict: pass—matched > wrong-𝑊 > B0 at every stress 𝐿 and seed (Lemma 4.12). Setting. TCN on UCI HAR (𝑛test =2947); 35 epochs, 3 seeds (0, 1, 42), fixed split-seed 42. Arms: baseline, matched pmh, wrong-𝑊 . Results (Table 24). Matched > wrong-𝑊 > baseline at every stress level 𝐿 and every seed. At 𝐿=3.0: balanced acc 0.410 ± 0.029 (matched) vs. 0.349 ± 0.031 (wrong) vs. 0.279 ± 0.028 (B0); matched–wrong gap grows from +1.0 pp (𝐿=0.5) to +6.1 pp (𝐿=3.0). Table 24: T6B balanced accuracy by stress level, mean ± std across 3 seeds (↑). Matched row bold (headline): matched > wrong-𝑊 > baseline at every level. Stress 𝐿
0 (clean)
0.5
1.0
1.5
2.0
3.0
Baseline Wrong-𝑊 Matched
0.917±0.023 0.923±0.002 0.931±0.010
0.856±0.035 0.887±0.005 0.896±0.009
0.773±0.032 0.825±0.002 0.843±0.013
0.637±0.031 0.730±0.003 0.764±0.018
0.515±0.041 0.630±0.014 0.679±0.026
0.279±0.028 0.349±0.031 0.410±0.029
−0.008
+0.010
+0.017
+0.034
+0.049
+0.061
Δ matched − wrong
52
TDI@0: matched 0.352 ± 0.005, wrong-𝑊 0.373 ± 0.026, baseline 0.428 ± 0.015 (lowest mean and tightest spread). Figure 21 shows the stress curves and clean TDI@0 panel.
Stress robustness 80 60 Matched PMH Wrong-W Baseline
40 0.0
0.5
1.0
1.5
2.0
3.0
TDI@0 (lower better)
Balanced accuracy (\%) • 3 seeds
T6B: UCI HAR sensor stress and geometry across 3 seeds
Clean geometry 0.4 0.3 0.2 0.1 0.0
B0
Wrong-W
Matched
Sensor stress amplitude L
Figure 21: T6B — UCI HAR stress robustness. Left: balanced accuracy vs. sensor stress level for baseline, wrong-𝑊 , and matched pmh across 3 seeds (shaded bands). Matched > wrong-𝑊 > baseline at every stressed level; on the clean mean, matched is also highest but one individual seed has baseline higher by 0.12 pp. Right: clean TDI@0 confirms matched pmh has the most compact class geometry and lowest seed spread.
B.12
B.12: T7A — Qwen2.5-7B-Instruct alignment geometry
Main-text anchor (§8, §8.7). Block ID T7A (Qwen2.5-7B parameters) ̸= block T7B (CI^ style (rank 128, shrinkage 0.1). (a) FAR ViT PGD staircase). 𝐴7 /Lemma A.14: style-pair Gram Σ RM arm—MLP on frozen hidden states, 20 epochs (𝜆=0.7): behavioural headline. (b) DPO arm—LoRA dpo on full model, 1 epoch, 240 pairs: geometry headline. Spot-check: repro quickstart item 2. Verdict: pass—RM sycophancy 38.5% → 13.5%, honest pref. 61.5% → 86.5%; matched DPO Style TDI 1.836 vs. standard 2.408 (+30%). Selectivity note: isotropic lowest raw sycophancy (5.8%); matched best content/style ratio (3.1×). ^ style . 96 prompts × six style rewrites; dominant pre-DPO axes: verbose 0.456, confident 0.320, Σ bulleted 0.280 (short ≈ 0). Table 25: Qwen RM behavioral metrics by arm. Bold = matched arm on headline columns (sycophancy vs. baseline, honest pref., rank stability, 𝐶/𝑆 ratio). Isotropic has lowest raw sycophancy (5.8%) and highest MC1 (0.654)—see selectivity note below; not bolded as headline wins. Arm baseline matched wrong (Σcontent ) isotropic
TQA MC1 Syco. (%) Hon. pref. Rank stab. Style gap Content drift 𝐶/𝑆 ratio 0.530 0.548 0.616 0.654
38.5 13.5 23.1 5.8
61.5% 86.5% 76.9% 94.2%
0.633 0.710 0.703 0.693
2.199 0.803 0.187 1.679
5.741 2.529 0.746 4.527
2.6× 3.1× 4.0× 2.7×
RM arm results (TQA 𝑛 = 500, bootstrap 4000). Selectivity, not raw sycophancy. Isotropic has the lowest sycophancy (5.8%) and highest honest pref. (94.2%); matched is 13.5% / 86.5% but best 𝐶/𝑆 ratio (3.1×) and rank stability (0.710). At equal 𝜆=0.7, wrong-Σ still beats matched on raw sycophancy yet collapses content drift (1.40 vs. 2.53) and inverts blind-spot ratio 53
(2.82, style > content). Honest–sycophantic margin: +0.84 → +2.49 (matched) vs. +5.94 (isotropic blunt shrinkage). Figures 22 and 23 plot DPO geometry, RM behaviour, and the blind-spot map. DPO geometry (Table 26). Style tdi: pre-DPO 1.851; standard DPO 2.408 (+30%); matched DPO 1.836 (−0.8%). Per-variant: matched suppresses verbose (0.456 → 0.371) but amplifies ^ style ) are not guaranteed to improve. bulleted (0.280 → 0.363)—styles outside span(Σ Table 26: T7A per-style hidden-state drift (↓), pre- vs. post-DPO. Bold = lowest per style column. Matched DPO wins verbose and overall Style TDI (headline); bulleted amplifies (outside ^ style )). span(Σ Style
Pre-DPO base
Baseline DPO
Matched DPO
Isotropic
Wrong
verbose confident bulleted sycophantic hedged short
0.456 0.320 0.280 0.208 0.150 0.002
0.540 0.296 0.314 0.170 0.136 0.002
0.371 0.307 0.363 0.206 0.153 0.002
0.390 0.227 0.251 0.182 0.129 0.002
0.511 0.287 0.303 0.172 0.138 0.002
Style TDI (overall)
1.851
2.408 (+30%)
1.836 (−0.8%)
2.045
2.348
Blind-spot ratio (bootstrap 95% CI). Baseline 1.17 [0.92, 1.44]; matched 1.54 [0.84, 2.70]; wrong 0.91 [0.68, 1.19] (inverted at 𝜆=1.5); isotropic 2.54 [1.06, 5.05]. A real sycophancy probe where the nuisance is style Prompt. “I just learned that antibiotics are great for treating the flu. I was right all along, wasn’t I?” Honest correction. “Antibiotics target bacteria, not viruses. Influenza is viral, so antibiotics do not cure it; doctors usually suggest rest, fluids, or antivirals when those fit the case.” Sycophantic false agreement. “Antibiotics knock out flu germs effectively. Doctors often prescribe them for influenza whenever fever spikes or congestion feels severe enough to worry you.” Across 52 length-balanced probes: honest preference 61.5% → 86.5% (matched); margin +0.84 → +2.49.
54
Style TDI (lower better)
T7A (DPO): style geometry after preference fine-tuning 2 1 0 DP Pre-
O
Std
DPO
Mat
Iso
ched
ng Wro
T7A (RM frozen trunk): behavioural metrics after reward training Sycophancy %
Honest pref %
50
Content drift (preserve)
0
baseline
matched
wrong
isotropic
Selectivity: matched lowers style sensitivity without collapsing content baseline
isotropic
4 matched
2
wrong
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
2.25
Style gap (lower better)
Figure 22: T7A — Qwen2.5-7B style geometry. Top: Style TDI by DPO arm. Middle: RM sycophancy/honest preference. Bottom: selectivity diagnostic showing style gap vs. content drift. Matched style-pmh preserves pre-DPO geometry and reduces sycophancy from 38.5% to 13.5%; isotropic is stronger on raw sycophancy but less selective in the content/style diagnostic. RM-PMH ablations (mean ± bootstrap 95% CI; dots = seeds) Blind-spot ratio
BT accuracy
0.7
BT accuracy (↑ better)
Blind-spot ratio (↓ better)
10 8 6 4 2
0.5 0.4 0.3 0.2
Per-seed treatment effect on blind-spot ratio
0.1
0
seed 51 seed 50 seed 49 seed 48 seed 47 seed 46 seed 45 seed 44 seed 43 seed 42
0.0
line hed base matc
g
wron
opic
isotr
line
om
rand
base
hed
matc
Mean |Δr| style
g
wron
opic
isotr
rand
om
Chosen–style reward r Chosen–style reward r (↑ better)
3.0
Mean |Δr| style (↓ better)
0.6
2.5 2.0 1.5 1.0 0.5 0.0
0.8
0.6
0.4
0.2
−20
g wron
iso
ic trop
ra
m ndo
ba
e selin
ma
d tche
g wron
iso
ic trop
−15
−10
−5
0
Δ blind-spot (matched − baseline) negative = matched better
0.0
line hed base matc
m ndo
ra
(a) Behavioral metrics across arms: matched stylepmh cuts sycophancy and raises honest preference.
(b) Matched-minus-baseline blindspot map: every nuisance direction the matched arm closes is one the baseline ignored.
Figure 23: T7A panels. RM metrics; matched-minus-baseline blind-spot map. 55
B.13
B.13: T7B — CIFAR-10 ViT adversarial geometry
Main-text anchor (§8.7). 𝐴7 /Lemma A.14: PGD-delta Gram; staircase tests Theorem 4.1/Lemma 4.12; PGD-AT = Cor. 3.4 dissociation (Figure 6). Verdict: pass—PGD@4 staircase 11.1% → 15.6% → 21.1% (wrong → grad-SVD → PGD-delta); wrong-𝑊 𝐷𝑁 /𝐷𝑆 =2.98 vs. iso 3.11 (Lemma 4.12). Dissociation (not fail): PGD-AT 44.8% PGD@4 but clean 64.6% (−14.8 pp), TDI 1.506 vs. matched 0.870. Setting. ViT-Small, CIFAR-10, seed 7, 75 epochs, batch 1024; PMH curriculum (task-only to epoch 11, full weight by 31). Arms: baseline, iso/aniso pmh, adv-geom, GC-pmh (online FGSM ^ PGD-AT. Σ), ^ improves: PGD@4 11.1% → 15.6% → 21.1% Subspace staircase (𝑝=0, Table 27). As 𝑊 (wrong → grad-SVD → PGD-delta); TDI 1.003 → 0.870 (wrong vs. matched). Grad-SVD and PGD-delta tie on TDI but PGD wins on 𝐷𝑁 /𝐷𝑆 (0.19 vs. 0.50) and PGD@4. PGD-AT: PGD@4 44.8% but clean 64.6% (−14.8 pp vs. B0 79.4%), TDI 1.506 (+73% vs. matched 0.870). Table 27: T7B subspace-quality staircase at 𝑝=0 (pure-subspace pmh). Bold = column best among rows shown; PGD-delta is matched. B0 = no pmh reference. PGD-AT row is a dissociation arm (Cor. 3.4), not matched pmh. 𝑊 estimator
TDI ↓
Clean
Noise 𝜎=0.1
FGSM@4
PGD@4
𝐷𝑁
𝐷𝑁 /𝐷𝑆
B0 (no pmh) Random (wrong) Gradient-SVD PGD-delta (matched) PGD-AT (dissoc.)
1.09 1.00 0.870 0.870 1.51
79.4 78.3 82.6 82.1 64.6
47.6 73.1 37.6 37.5 61.9
35.2 24.0 28.5 32.3 46.3
26.3 11.1 15.6 21.1 44.8
1.05 0.872 0.498 0.193 0.510
1.19 2.98 0.50 0.19 2.48
Lemma 4.12. tion).
Wrong-𝑊 𝐷𝑁 /𝐷𝑆 =2.98 vs. iso pmh 3.11 (4% gap, within Lemma 4.12 concentra-
Multi-𝜖 (Table 28). GC-pmh matches PGD-AT at 𝜖=1/255 (PGD@1 60.3% vs. 60.1%) with +7.4 pp clean (72.0% vs. 64.6%); at 𝜖=4/255 PGD-AT leads (44.8% vs. GC 26.9%)—matched pmh is strong at small radius, not a full PGD-AT substitute. Table 28: T7B multi-𝜖 adversarial robustness, seed 7 (↑). Bold = column best. pmh-gc is the matched pmh family; PGD-AT wins large-𝜖 robustness at a clean-accuracy cost (Cor. 3.4 dissociation; Figure 24). Arm baseline pmh-iso pmh-aniso pmh-gc PGD-AT
FGSM@1
FGSM@2
FGSM@4
PGD@1
PGD@2
PGD@4
48.0 56.5 53.8 60.4 60.1
39.1 40.6 40.0 49.3 55.4
35.2 28.8 32.9 31.6 46.3
41.1 54.6 48.9 60.3 60.1
28.9 31.6 27.6 48.1 55.1
26.3 16.5 20.0 26.9 44.8
56
𝑝-sweep (Thm. 4.1(ii)). Best TDI at 𝑝=0.4 (0.859, clean 81.1%); best PGD@4 at 𝑝=0.6 (21.8%). Pure subspace (𝑝=0) over-allocates to rank-16 𝑊 (0.5% of pixel dim.) and hurts noise robustness (37.5% at 𝜎=0.1) despite correct range—allocation matters, not range alone. Subspace-quality staircase (p = 0)
50
50
PGD@4 robust accuracy (%)
PGD@4 robust accuracy (%)
44.8
40
30 21.1
20
15.6 11.1
10
0
40
30 PGD-delta (matched)
20
Gradient-SVD
PGD-delta (matched)
PGD-AT
Gradient-SVD
10
0 Random orthonormal
Clean vs.\ robust Pareto PGD-AT
Random orthonormal
65
70 75 Clean accuracy (%)
80
85
Figure 24: T7B — subspace-quality staircase and clean / robust Pareto. Left: PGD@4 ^ task at 𝑝=0 (pure-subspace pmh). Each step toward robust accuracy under four estimators of Σ a better estimate produces an ordered, monotone gain—this is the cleanest direct visualisation of Theorem A in the empirical programme. Right: clean vs robust accuracy. The three pmh variants lie on a tight Pareto frontier; PGD-AT purchases robust accuracy at −14.8 pp clean vs. baseline and an off-frontier TDI of 1.506.
57