Proceedings of Machine Learning Research vol :1–21, 2026
SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
Jian Gu
JIAN . GU @ MONASH . EDU
Monash University
Aldeida Aleti
ALDEIDA . ALETI @ MONASH . EDU
arXiv:2606.32022v1 [cs.LG] 30 Jun 2026
Monash University
Chunyang Chen
CHUN - YANG . CHEN @ TUM . DE
Technical University of Munich
Hongyu Zhang
HYZHANG @ CQU . EDU . CN
Chongqing University
Abstract Residual-stream analysis asks how a language model’s computation evolves across depth, but this question requires a stable semantic coordinate system. Intermediate decoding is meaningful only when readout coordinates remain comparable across layers. In language models, this comparability is mediated by the token interface: if embedding-side anchors and unembedding-side readout disagree on the semantic span under analysis, apparent hidden-state motion may reflect measurement drift rather than computation. We introduce Semantic Reference Frames (SemRF), an anchor-based formalism that separates semantic measurement from residual-stream dynamics. A SemRF fixes the anchors used in the analysis and measures residual states against them. Pseudo-inverse tying supplies an exact anchor-synchronization case; under restricted bi-invertibility on the chosen span, SemRF gives stable semantic-basis coordinates, distortion bounds, and controlled near-identity frame changes. With the frame fixed, residual computation becomes a depthwise semantic trajectory. The same anchors induce a semantic Voronoi diagram: semantic distance, equivalently semantic evidence such as logits, assigns each layer to a coarse cell, while coordinates retain within-cell motion and margins. We define layerwise steps, target-direction contribution profiles, and imbalance diagnostics, then use the observed Voronoi trace to define a margin-relaxed semantic tube. The canonical trace is the minimum-action path inside this tube. In a nonempty tube with at least one positive quadratic weight, it is unique and satisfies a discrete spline equation away from active constraints. Excess action controls step mismatch, curvature mismatch, interior deviation, and target-direction contribution-profile mismatch. Finally, low curvature energy implies piecewiselinear compressibility. This yields a local notion of knowledge density: within a fixed admissible frame and tolerance, lower trace complexity means that the depth-by-coordinate trajectory can be summarized by fewer semantic knots. Through the residual network’s parameter-to-trajectory map, this also gives a conditional link to parameter efficiency: among admissible parameter settings that fit the same data, lower-action and lower-complexity traces use fewer effective semantic degrees of freedom. The guarantees require controlled interface error, small projection residual, and explicit tube constraints. Keywords: semantic reference frames, semantic dynamics, language models, mechanistic interpretability, knowledge density © 2026 J. Gu, A. Aleti, C. Chen & H. Zhang.
G U A LETI C HEN Z HANG
1. Introduction Transformer language models update a shared residual state through layerwise writes. This suggests a trajectory view: each layer moves the same state, and intermediate residual vectors carry partial semantic evidence. LogitLens and Tuned Lens decode hidden states into vocabulary space across depth Nostalgebraist (2020); Belrose et al. (2023), but such trajectories are meaningful only when readout coordinates remain comparable across layers. If embedding-side anchors and unembeddingside readout disagree on the semantic span under analysis, apparent semantic motion may reflect measurement drift rather than computation. We introduce Semantic Reference Frames (SemRF), an anchor-based formalism for separating measurement from dynamics. A SemRF fixes the semantic anchors used for the analysis and measures residual states against them. The main structural assumption is restricted bi-invertibility of the token interface on this anchor span. Pseudo-inverse tying supplies an exact anchor-synchronization case; when the restricted interface condition holds, semantic coordinates are stably measurable and interface mismatch yields explicit distortion bounds. SemRF specifies when semantic readout, diagnosis, and comparison are well posed. Once measurement is controlled, residual evolution becomes a semantic trajectory. The same semantic anchors induce a semantic Voronoi diagram: each state is assigned to the anchor with smallest semantic distance, equivalently largest semantic evidence such as a logit. This is the sense in which we use “Voronoi” throughout; ordinary Euclidean nearest-neighbor Voronoi is only the special case induced by a Euclidean distance score. The diagram gives cells, faces, margins, and a coarse Voronoi trace; continuous coordinates record within-cell motion, cell crossings, and reversals along a fixed target direction. We combine the two views through a margin-relaxed semantic tube: the observed Voronoi trace defines a coarse route, and a path action selects the minimum-action path inside the tube as the canonical trace. The terminology is close to Semantic Tube Prediction Huang et al. (2026), but here the tube is an analysis constraint induced by measured Voronoi cells rather than a training objective or geodesic prior. When tube constraints are active, the comparison is no longer just endpoint interpolation. The scope of our research is local and diagnostic: after choosing semantic anchors, checking interface and projection errors, and fixing tube constraints, depthwise semantic computation becomes a measurement-and-trajectory problem. Within this setting, we define layerwise semantic steps, targetdirection contribution profiles, and imbalance measures; prove stability under nearby admissible frames; and show that excess action controls deviations from the canonical trace. Low-curvature traces are piecewise-linearly compressible, yielding a local notion of knowledge density: for a fixed admissible frame and tolerance, lower trace complexity means that the depth trajectory can be summarized by a few semantic knots rather than by all layer-by-coordinate states. Parameterefficiency claims are read through the induced trajectory map θ 7→ z0:L (x; θ), not through a direct identification between hidden-state locations and unique parameter values: after interface alignment stabilizes the semantic basis, parameter settings can be compared by the complexity of the semantic transport they realize on the same data. Our contributions are as follows: 1. Controlled semantic measurement. We formalize semantic reference frames as anchor-based coordinate systems for residual-stream analysis, identify restricted interface conditions under which semantic coordinates are stably measurable, and derive distortion bounds on the chosen anchor span. 2
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
2. Stable semantic dynamics and canonical traces. We define layerwise steps, target-direction contribution profiles, and imbalance diagnostics; prove cross-frame stability; and use an anchor-induced semantic Voronoi diagram to define a margin-relaxed semantic tube whose minimum-action path is the canonical trace. 3. Deviation control and local knowledge density. We show that excess action controls specified deviations from the canonical trace, and that low curvature energy yields piecewise-linear compressibility, giving a frame-relative interpretation of knowledge density and a conditional bridge to parameter efficiency.
2. Related Work We position SemRF relative to work on intermediate readout, LM semantics, residual-stream analysis, geometric control, and efficiency. Intermediate Vocabulary Readout. LogitLens and Tuned Lens recover token-level information from intermediate residual states Nostalgebraist (2020); Belrose et al. (2023). Probing, contextualgeometry, and latent-visualization studies examine how semantic content is organized in hidden representations Petroni et al. (2019); Reif et al. (2019); Ethayarajh (2019). SemRF isolates the measurement issue behind these methods: trajectory-level claims require coordinates that remain comparable across depth, not a readout recalibrated independently at each layer. Semantics and Interface Alignment. Recent work formulates LM semantics through vocabularydefined semantics, semantic transition analysis, latent semantic alignment, and cross-model semantic transfer Gu et al. (2024, 2025, 2026b). Pseudo-inverse tying connects semantic stability to synchronization between the input embedding and output unembedding on the relevant span Gu et al. (2026a). SemRF builds on this line by making the semantic anchors explicit, using them to form anchor-induced Voronoi cells, and turning restricted bi-invertibility and admissibility into quantitative conditions for comparing anchor-restricted coordinates across depth. Residual-Stream Analysis. Mechanistic interpretability views Transformer computation as successive writes to a shared residual stream Elhage et al. (2021a); Olsson et al. (2022); Elhage et al. (2021b). This perspective supports circuit analyses, causal mechanism identification, sparse autoencoders, superposition models, and residual-stream factorization Conmy et al. (2023); Wu et al. (2023); Huben et al. (2023); Bricken et al. (2023); Templeton et al. (2024); Lawson et al. (2024); Elhage et al. (2022). Related work also studies representation stabilization and information transport through depth Yu and Ananiadou (2023); Janiak et al. (2024); Shai et al. (2024); Meng et al. (2022). SemRF asks when depth-indexed semantic motion reflects computation rather than readout drift. Representation Geometry and Control. Work on linear features, representation steering, internalupdate equivalence, isotropy, and embedding geometry studies when representations admit stable geometric interpretation Park et al. (2023); Zou et al. (2023); Goldwaser et al. (2025); Hu et al. (2026); Liang et al. (2021); Ethayarajh (2019). Semantic Tube Prediction uses a geometric trajectory prior as a JEPA-style learning signal Huang et al. (2026). SemRF uses the related idea as an analysis device: its semantic tube is the margin-relaxed set of paths that follow the trusted Voronoi trace induced by semantic anchors in a validated frame. 3
G U A LETI C HEN Z HANG
Parameter Efficiency and Knowledge Density. Scaling laws and capability-density analyses study efficiency at the level of parameters, data, and compute Kaplan et al. (2020); Hoffmann et al. (2022); Xiao et al. (2025). SemRF uses a narrower notion: after admissible measurement is fixed, parameter settings that solve the same data can be compared by the action and complexity of the semantic traces they induce. This trace-level ranking is complementary to global efficiency metrics, not a replacement for them.
3. Coordinates and Measurement We begin with the measurement problem. Let hℓ,t ∈ Rd denote the residual-stream state at depth ℓ ∈ {0, . . . , L} and position t, and write hℓ+1,t = hℓ,t + uℓ,t (hℓ,t ),
∆hℓ,t := uℓ,t (hℓ,t ).
We analyze a fixed position t and suppress it. Depth ℓ is discrete time. A semantic reference frame (SemRF) consists of semantic anchors B and a coordinate map ϕ : Rd → RK . It induces coordinates zℓ := ϕ(hℓ ) and increments ∆zℓ := zℓ+1 − zℓ . This section asks when these coordinates are stable measurements of the chosen anchor span. 3.1. Coordinates and Vocabulary Readout d A SemRF starts by fixing the semantic anchors that define the analysis. Let B = {bk }K k=1 ⊂ R , K ≥ 2, be these anchors, and let z = ϕ(h) ∈ RK , (1)
be the corresponding semantic coordinates of a residual state h ∈ Rd . Definition 1 (SemRF coordinates) A semantic coordinate system is a pair (ϕ, B) consisting of anchors B and a coordinate map ϕ : Rd → RK . The coordinate zi = ϕi (h) is interpreted as the semantic evidence of state h for anchor bi . Semantic-basis coordinates. Throughout the paper, z = ϕ(h) denotes the coordinate vector in the semantic basis fixed by B. Margins, Voronoi assignments, steps, action values, and compressibility quantities are all computed in this single reference frame. Notation. ⟨·, ·⟩ is the standard Euclidean inner product in the relevant ambient coordinate space, and ∥·∥ denotes the induced ℓ2 norm on vectors and the corresponding operator norm on matrices. Vocabulary anchors and interface readout. Vocabulary readout is the motivating example. Let ⊤ ∈ Win ∈ RV ×d be the input embedding and WU ∈ RV ×d the unembedding. Writing E := Win Rd×V , the i-th column ei is the input-side anchor for token i. Output-side anchors can be obtained from B̃ := WU+ ∈ Rd×V . A hidden state is measured by its evidence for the chosen anchors, for example through logit coordinates or cosine similarity Gu et al. (2024, 2025). Semantic Voronoi cells and margins. The same semantic anchors in B induce a semantic Voronoi diagram through semantic distances, equivalently through monotone semantic evidence scores. In the logit case, larger logit evidence means smaller semantic distance; for cell assignment one may set di (h) := −ϕi (h). Thus the cell of anchor bi is Vi := {h : ϕi (h) ≥ ϕj (h) for all j}. 4
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
The induced Voronoi index is κ(h) := arg max ϕi (h), i
with ties broken by a fixed rule. A Voronoi face is where two anchors tie, ϕi (h) = ϕj (h). The margin mar(h) := ϕ(1) (h) − ϕ(2) (h), where ϕ(1) (h) and ϕ(2) (h) are the largest and second-largest coordinates, is the semantic-distance gap to the nearest competing face. Thus κ(h) gives a coarse semantic state, while ϕ(h) and its margin retain within-cell information. Throughout the paper, “Voronoi” refers to this semanticdistance/evidence diagram. It reduces to ordinary Euclidean Voronoi only for special distance-derived scores; the results below use only the coordinate dominance inequalities zi ≥ zj , which are convex halfspace constraints. 3.2. Interface Alignment and Admissibility We now state when vocabulary readout becomes a controlled measurement procedure on the chosen anchor span. The first two propositions record exact and approximate synchronization cases; the assumption that follows is the local condition used by the later bounds. Proposition 2 (Exact interface synchronization) Suppose there exist matrices Z ∈ RV ×d and an invertible T ∈ Rd×d such that (i) the embedding and unembedding are parameterized as Win = ZT −1 ∈ RV ×d and WU = ZT ⊤ ∈ RV ×d , and (ii) Z has orthonormal columns: Z ⊤ Z = Id . Then the unembedding pseudoinverse recovers the input-side anchor matrix: ⊤ WU+ = T −⊤ Z ⊤ = Win .
Equivalently, the output-side pseudoinverse anchors coincide with the embedding anchors, so semantic coordinates defined via the restricted unembedding pseudoinverse are consistent across the input and output interfaces. Proposition 3 (Approximate interface synchronization) Let EB ∈ Rd×K have full column rank and write EB+ = (EB⊤ EB )−1 EB⊤ . Let η := ∥UB⊤ −EB+ ∥ measure the discrepancy between a restricted readout and the anchor pseudoinverse. Then ∥UB⊤ EB − IK ∥ ≤ η ∥EB ∥. Moreover, if UB⊤ = ẼB+ for a nearby full-column-rank synthesis matrix ẼB with ∥EB − ẼB ∥ ≤ δ, then ∥UB⊤ EB − IK ∥ ≤ ∥ẼB+ ∥ δ. Assumption 1 (Token-interface bi-invertibility) Let B = {bk }K k=1 be the anchor set used by ϕ. There exist an anchor synthesis matrix EB ∈ Rd×K and a restricted readout matrix UB ∈ Rd×K such that ∥UB⊤ EB − IK ∥ ≤ ε, with EB full column rank and ∥EB+ ∥ ≤ CE , ∥UB ∥ ≤ CU for fixed frame constants CE , CU . 5
G U A LETI C HEN Z HANG
This local assumption rules out spurious coordinate drift created by the token interface. It requires stable synthesis and readout on the chosen anchor span, not a global semantic basis. Definition 4 (Admissible SemRF regime) Fix tolerances ηint , ηproj > 0. We call a prediction event (θ, x, y) admissible for a frame (ϕ, B) when the interface mismatch satisfies ∥UB⊤ EB − IK ∥ ≤ ηint , and the relative projection residual of each analyzed state satisfies ∥hℓ − EB EB+ hℓ ∥ ≤ ηproj , ∥hℓ ∥
ℓ = 0, . . . , L.
The same requirement may also be imposed on residual writes ∆hℓ when step-level conclusions are needed. All quantitative results below are conditional on admissibility. Large interface mismatch means unstable readout; large projection residual means the chosen anchors miss active computation. Targetdirection statements additionally require a fixed unit direction, either the target-token coordinate axis or a unit contrast direction. 3.3. Diagnostics and Distortion Bounds The same diagnostics control how far measured coordinates can deviate from intended anchor-span coordinates. Diagnostics. The key observables are the interface mismatch ∥UB⊤ EB − IK ∥ and the relative projection residual ∥r(h)∥ r(h) := h − EB EB+ h, . ∥h∥ The first tests stability of synthesis and readout on the anchor span; the second tests whether the analyzed state lies mostly inside that span. Anchor choice therefore determines the semantic alternatives, interface conditioning, and reliability of later trajectory bounds. Proposition 5 (Anchor-span interface distortion) Assume Assumption 1 and use the restricted linear readout coordinates ϕrd (h) = UB⊤ h. For any coordinate vector c ∈ RK , let hc := EB c be its recomposition into the anchor subspace. Then ∥ϕrd (hc ) − c∥ = ∥(UB⊤ EB − IK )c∥ ≤ ε ∥c∥. More generally, for any decomposition h = EB c + r, ∥ϕrd (h) − c∥ ≤ ε ∥c∥ + ∥UB ∥ ∥r∥. Corollary 6 (Trajectory distortion under bounded drift) Let {hℓ }L ℓ=0 be any residual trajectory and suppose each state decomposes as hℓ = EB cℓ + rℓ . Let zℓ = ϕrd (hℓ ) and ∆cℓ := cℓ+1 − cℓ . Then ∥zℓ − cℓ ∥ ≤ ε∥cℓ ∥ + ∥UB ∥∥rℓ ∥,
∥∆zℓ − ∆cℓ ∥ ≤ ε(∥cℓ+1 ∥ + ∥cℓ ∥) + ∥UB ∥(∥rℓ+1 ∥ + ∥rℓ ∥).
These bounds identify the regime in which measured semantic motion is attributable to the residual trajectory rather than to interface error or off-span residuals. Section 4 then studies which dynamical features remain stable under admissible changes of frame. 6
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
4. Layerwise Semantic Dynamics With a checked SemRF, residual-stream computation induces a depthwise trajectory in the semanticbasis coordinates fixed in Section 3. For a prediction event (θ, x, y), write z0:L for this trajectory. This section defines step observables, proves their stability under admissible frame changes, and introduces the tube-constrained action used in Section 5. 4.1. Step Structure and Path Imbalance The semantic anchors induce Voronoi cells, so each layer has a coarse state κℓ := κ(hℓ ) and margin mar(hℓ ) to the nearest Voronoi face. The coordinates zℓ retain the within-cell motion and the evidence changes that drive cell crossings. For a trajectory {hℓ }L ℓ=0 , define the induced semantic field at layer ℓ by Fℓ (h) := ϕ h + uℓ (h) − ϕ(h) ∈ RK . (2) Along the realized trajectory, Fℓ (hℓ ) = ∆zℓ . Assumption 2 (Local linearization) Along trajectories of interest, ϕ admits a first-order approximation with bounded Jacobian Jϕ : ϕ(h + δ) = ϕ(h) + Jϕ (h)δ + r(h, δ) with ∥r(h, δ)∥ ≤ Cr ∥δ∥2 . Proposition 7 (Induced-step linearization) Under Assumption 2, the induced semantic step satisfies ∆zℓ = Jϕ (hℓ ) ∆hℓ + εℓ , ∥εℓ ∥ ≤ Cr ∥∆hℓ ∥2 . Thus each semantic step is the first-order image of the residual write, up to a controlled quadratic remainder. For a fixed context x, let ey be a fixed unit direction in the chosen coordinates, either the targettoken coordinate axis when included in the anchor set or a unit contrast direction. Define the signed target-direction contribution aℓ := ⟨∆zℓ , ey ⟩. The contribution profile {aℓ } sums to the net target-direction displacement, L−1 X
aℓ = ⟨zL − z0 , ey ⟩,
ℓ=0
and records where target-direction displacement is accumulated, delayed, or reversed. We use semantic imbalance for trajectories in which later layers partially undo earlier progress before reaching the same endpoint. Three diagnostics make this precise: motion energy
L−1 X
2
∥∆zℓ ∥ ,
curvature energy
ℓ=0
L−1 X
∥∆2 zℓ ∥2 ,
ℓ=1
where ∆2 zℓ := zℓ+1 − 2zℓ + zℓ−1 is the discrete second difference, together with the cumulative target-direction backtracking penalty L−1 X
−⟨∆zℓ , ey ⟩ + .
ℓ=0
7
G U A LETI C HEN Z HANG
The first tracks total semantic travel, the second tracks oscillation across depth, and the third isolates target-direction reversal. They flag detours, low-margin crossings, and oscillation between alternatives, without by themselves proving that a crossing is unnecessary. 4.2. Cross-Frame Stability of Dynamics We next show that admissible frame changes preserve the coordinate geometry and step structure used below. Proposition 8 (Anchor-span coordinate equivalence) Let EB ∈ Rd×K have full column rank and consider two restricted coordinate maps ϕ(h) = UB⊤ h and ϕ̃(h) = ŨB⊤ h. Assume both satisfy the restricted interface condition with errors ε, ε̃ < 1: ∥UB⊤ EB − IK ∥ ≤ ε,
∥ŨB⊤ EB − IK ∥ ≤ ε̃.
Then on the anchor subspace span(EB ) the two coordinate systems are related by a near-identity linear transform. For any h = EB c, ϕ̃(h) = IK + ∆ ϕ(h),
∥∆∥ ≤
ε + ε̃ . 1−ε
Corollary 9 (Relational stability) Under the hypotheses of Proposition 8, let η :=
ε + ε̃ . 1−ε
For any u, v ∈ span(EB ), ⟨ϕ̃(u), ϕ̃(v)⟩ − ⟨ϕ(u), ϕ(v)⟩ ≤ (2η + η 2 ) ∥ϕ(u)∥ ∥ϕ(v)∥, and ∥ϕ̃(u) − ϕ̃(v)∥ ≤ (1 + η) ∥ϕ(u) − ϕ(v)∥. If η < 1, the matching lower bound also holds: (1 − η) ∥ϕ(u) − ϕ(v)∥ ≤ ∥ϕ̃(u) − ϕ̃(v)∥. Thus pairwise similarities and distances are stable when the interface errors are small. Assumption 3 (Lipschitz stability) The coordinate map ϕ is Lϕ -Lipschitz. For all h, h′ ∈ Rd , ∥ϕ(h) − ϕ(h′ )∥ ≤ Lϕ ∥h − h′ ∥. Theorem 10 (Semantic-step stability bound) Under Assumption 3, for any residual update ∆hℓ , ∥∆zℓ ∥ = ∥ϕ(hℓ + ∆hℓ ) − ϕ(hℓ )∥ ≤ Lϕ ∥∆hℓ ∥. In particular, if ∥∆hℓ ∥ ≤ U for all ℓ, then ∥∆zℓ ∥ ≤ Lϕ U for all ℓ. 8
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
Theorem 11 (Frame-to-step stability transfer) Consider two linear-readout coordinate maps ϕ(h) = UB⊤ h and ϕ̃(h) = ŨB⊤ h built on the same anchor synthesis EB ∈ Rd×K , and assume both satisfy the restricted bi-invertibility condition (Assumption 1) with errors ε, ε̃ < 1. Define Cframe :=
ε + ε̃ . σmin (EB )
Then on the anchor subspace span(EB ) the two coordinate systems are uniformly close, and for any h = EB c, ∥ϕ(h) − ϕ̃(h)∥ ≤ Cframe ∥h∥. Moreover, for any anchor-subspace update ∆h = EB ∆c, ∥(ϕ(h + ∆h) − ϕ(h)) − (ϕ̃(h + ∆h) − ϕ̃(h))∥ ≤ Cframe ∥∆h∥. Together, these results show that, under admissibility, nearby frames preserve the anchor-span geometry and induced step profiles used below. Continuous semantic steps can then be compared across depth without being dominated by readout drift. Voronoi assignments are reliable when their margins dominate frame distortion; for example, if ∥ϕ̃(h) − ϕ(h)∥∞ ≤ δ and mar(h) > 2δ, then κ̃(h) = κ(h). 4.3. Endpoint-Matched Path Action We now introduce the comparison path. Endpoint matching makes pathwise deviation comparable. The semantic tube keeps the path on the trusted coarse Voronoi route up to margin relaxations, without copying the observed coordinates exactly. obs be an observed teacher-forced trajecProposition 12 (Contribution-profile telescoping) Let z0:L cmp tory and let z0:L be any endpoint-matched comparison path. Assume
z0obs = z0cmp ,
zLobs = zLcmp .
Define gℓ := ⟨∆zℓobs − ∆zℓcmp , ey ⟩. Then L−1 X ℓ=0
gℓ = 0,
j−1 X
gℓ = ⟨zjobs − zjcmp , ey ⟩,
j = 1, . . . , L.
ℓ=0
Thus cumulative contribution-profile mismatch equals intermediate target-direction displacement between the observed path and any endpoint-matched comparison path. obs be the observed trajectory and let τ := Definition 13 (Margin-relaxed semantic tube) Let z0:L ℓ κ(hobs ℓ ) be its Voronoi index. Choose a trusted layer set I ⊆ {1, . . . , L − 1}, typically the layers whose margins dominate measurement error, and relaxation radii ρℓ ≥ 0. The semantic tube induced by the observed Voronoi trace is z0 = z0obs , zL = zLobs , obs Tρ (z ) := z0:L : . zℓ,τℓ ≥ zℓ,j − ρℓ ∀ℓ ∈ I, ∀j
9
G U A LETI C HEN Z HANG
The constraints are linear dominance constraints in the semantic coordinates, so the tube is a closed convex polyhedron. For a realized trajectory and nonnegative radii, it contains z obs . If I = ∅, the construction reduces to endpoint matching; inactive constraints leave the endpoint-only baseline unchanged. Definition 14 (Path action) For a prediction event (θ, x, y), let ey ∈ RK be the fixed unit direction used for target-direction diagnostics. We define an action S[z] on depth-discrete trajectories z0:L by S[z] := α
L−1 X
L−1 X
L−1 X
ℓ=0
ℓ=1
ℓ=0
∥∆zℓ ∥2 + β
∥∆2 zℓ ∥2 + γ
−⟨∆zℓ , ey ⟩ + ,
(3)
with α, β ≥ 0 not both zero, and γ ≥ 0. The terms penalize excess motion, oscillation, and target-direction backtracking. If no target-direction diagnostic is used, set γ = 0 and omit the backtracking term. The action scores feasible comparison paths rather than training the model; Section 5 minimizes it inside the semantic tube.
5. Traces and Compressibility We now pass from descriptive path comparison to a selected baseline. Minimizing the path action from Section 4 over the margin-relaxed semantic tube produces the canonical trace. Unless stated otherwise, Euler–Lagrange statements concern the quadratic regime γ = 0 on layers where tube inequalities are inactive. 5.1. Minimum-Action Canonical Traces TF with endpoints Under teacher forcing, the model induces an observed semantic trajectory z0:L TF TF TF z0 = z0 and zL = zL . With the semantic tube Tρ (z ) from Definition 13, define
z ⋆ ∈ arg
min
z0:L ∈Tρ (z TF )
S[z].
(4)
We call z ⋆ the canonical trace: the diagnostic minimum-action comparison path that follows the trusted coarse Voronoi route up to the relaxation radii. Theorem 15 (Canonical-trace characterization) Consider (4) with the action (3). The semantic tube from Definition 13 is a nonempty closed convex polyhedron for realized trajectories and nonnegative radii. If (α, β) ̸= (0, 0) with α, β ≥ 0, the quadratic part is strictly convex on endpointconstrained path variables. Hence the canonical trace is unique for γ = 0 and remains unique after adding the convex backtracking term for γ > 0. For γ = 0, writing ∆2 zℓ := zℓ+1 − 2zℓ + zℓ−1 ,
∆4 zℓ := zℓ+2 − 4zℓ+1 + 6zℓ − 4zℓ−1 + zℓ−2 ,
the coordinates satisfy, on every inactive interior layer, β ∆4 zℓ⋆ = α ∆2 zℓ⋆ ,
ℓ = 2, . . . , L − 2.
At active tube constraints, KKT multipliers for the corresponding linear dominance inequalities are added. 10
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
Here α prices total semantic travel and β penalizes oscillatory correction. If the tube has no active constraints, the quadratic canonical trace is straight endpoint interpolation. Otherwise z ⋆ is obtained from a sparse convex quadratic program with a banded Hessian and layer-local linear inequalities. 5.2. Excess Action and Deviation Control Excess action is a scalar deviation diagnostic: the action gap dominates quadratic deviation energy, with strong-convexity modulus stated in Proposition 21. Proposition 16 (Excess-action lower bound) Let z ⋆ be a minimizer of (4) for the action (3) with γ ≥ 0, and let z ∈ Tρ (z TF ) be any feasible trajectory. Writing eℓ := zℓ − zℓ⋆ , one has ⋆
S[z] − S[z ] ≥ α
L−1 X
2
∥∆eℓ ∥ + β
ℓ=0
L−1 X
∥∆2 eℓ ∥2 .
ℓ=1
When γ = 0 and the active normal-cone pairing vanishes in the direction z − z ⋆ , the inequality is an equality. Corollary 17 (Deviation controls from excess action) In the setting of Proposition 16, define ∆S := S[z] − S[z ⋆ ]. Then α
L−1 X
∥∆zℓ − ∆zℓ⋆ ∥2 ≤ ∆S,
β
ℓ=0
L−1 X
∥∆2 zℓ − ∆2 zℓ⋆ ∥2 ≤ ∆S.
ℓ=1
If the endpoint-constrained problem has strong-convexity modulus µ > 0, then L−1 X
∥zℓ − zℓ⋆ ∥2 ≤
ℓ=1
2∆S . µ
For z = z obs , define gℓ := ⟨∆zℓobs − ∆zℓ⋆ , ey ⟩. If α > 0, then max
1≤j≤L
j−1 X
gℓ ≤
ℓ=0
L−1 X ℓ=0
r |gℓ | ≤
L ∆S 1/2 . α
Thus excess action controls step mismatch, curvature mismatch, interior displacement, and cumulative target-direction contribution-profile mismatch whenever the corresponding weights are positive. 5.3. Compressibility and Trace Complexity The preceding bounds quantify deviation from the canonical trace. We now ask when such a trace admits a compact description: low curvature means that few semantic knots suffice to approximate the full depthwise trajectory. Definition 18 (Trajectory compressibility) A semantic trajectory z0:L ∈ (RK )L+1 is (m, ε)compressible if there exists a piecewise-linear trajectory z̃0:L with at most m linear segments, whose breakpoints are the semantic knots, such that L
1 X ∥zℓ − z̃ℓ ∥2 ≤ ε2 . L+1 ℓ=0
11
G U A LETI C HEN Z HANG
Definition 19 (Frame-relative trace complexity) For a fixed admissible SemRF and tolerance ε, define mε (z) as the smallest m for which z is (m, ε)-compressible. Up to knot locations, the trace needs O(K mε (z)) real values rather than (L + 1)K. We use local knowledge density only in this frame-relative sense: within the same frame and tolerance, smaller trace complexity means a more compact description of the same depthwise semantic transport. Connection to parameter efficiency. For a fixed architecture and dataset D, parameters θ induce teacher-forced residual trajectories and hence SemRF trajectories z0:L (x; θ). Interface alignment, including the pseudo-inverse tying case in Section 3, stabilizes the semantic basis so that parameter settings can be compared through their induced semantic traces rather than through moving readout frames. Define the average trace complexity 1 X m̄ε (θ; D) := mε (z ⋆ (x; θ)) , |D| x∈D
where z ⋆ (x; θ) is the canonical trace inside the tube induced by the trajectory of θ on x. For a fixed frame dimension, a conditional semantic-density proxy is KDε (θ; D) :=
L+1 . m̄ε (θ; D) + 1
Equivalently, over a feasible family ΘD of admissible parameters satisfying the same data constraints, one may use 1 X θ† ∈ arg min S[z ⋆ (x; θ)] + λ m̄ε (θ; D) θ∈ΘD |D| x∈D
with λ ≥ 0 as a semantic-efficiency selector. Lower action and lower trace complexity then indicate higher local knowledge density and a more parameter-efficient semantic realization, relative to the parameter-to-trajectory map θ 7→ z0:L (x; θ) and the fixed frame, data, and tolerance. Theorem 20 (Curvature-to-compressibility bound) Let z0:L be any trajectory with curvature energy L−1 X E2 (z) := ∥∆2 zℓ ∥2 . ℓ=1
There exists a piecewise-linear z̃0:L with at most m segments such that L
1 X c L3 E2 (z) ∥zℓ − z̃ℓ ∥2 ≤ , L+1 m4 ℓ=0
where c > 0 is a universal constant independent of K and L. Hence z is (m, ε)-compressible whenever 1/4 3 c L E2 (z) m≥ . ε2 For a quadratic canonical trace with γ = 0 and β > 0, one has E2 (z ⋆ ) ≤ S[z ⋆ ]/β. Theorem 20 therefore gives (m, ε)-compressibility whenever 3 1/4 c L S[z ⋆ ] m≥ . β ε2 12
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
This justifies the local knowledge-density claim: once measurement, tolerance, and tube constraints are fixed, low-curvature admissible traces need fewer values than the full depth-by-coordinate representation. The Voronoi trace gives the coarse route, while the canonical trace and excess action quantify the remaining continuous motion.
6. Conclusion SemRF isolates a local regime in which residual-stream semantics is well posed. On a validated anchor span, coordinates are stable, the same anchors induce a semantic Voronoi diagram, and admissible frame changes preserve the geometry needed for path comparison. The observed Voronoi sequence defines a margin-relaxed semantic tube; minimizing path action inside it gives a diagnostic canonical trace. With at least one positive quadratic weight, the trace is unique, satisfies a discrete spline equation away from active constraints, and reduces to straight interpolation only when no coarse-state constraint is active. Excess action controls the corresponding step, curvature, interior, and target-direction deviations. Low curvature yields piecewise-linear compressibility, giving a local, frame- and tolerance-relative notion of knowledge density. Through the parameter-induced trajectory map, this connects to parameter efficiency by ranking lower-action and lower-complexity semantic transport within the same admissible frame and data constraints. These are local guarantees: they require controlled interface error, small projection residual, explicit tube constraints, and a fixed unit direction whenever target-direction diagnostics are used.
References Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemanticfeatures/index.html. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià GarrigaAlonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021a. Nelson Elhage, Christopher Olah, Neel Nanda, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021b. URL https://transformer-circuits. pub/2021/framework/index.html. Accessed 2026-02-22. 13
G U A LETI C HEN Z HANG
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. Kawin Ethayarajh. How contextual are contextualized word representations. Comparing the geometry of BERT, ELMo, and GPT-2 Embeddings, 2, 2019. Adrian Goldwaser, Michael Munn, Javier Gonzalvo, and Benoit Dherin. Equivalence of context and parameter updates in modern transformer blocks. arXiv preprint arXiv:2511.17864, 2025. Jian Gu, Aldeida Aleti, Chunyang Chen, and Hongyu Zhang. Vocabulary-defined semantics: Latent space clustering for improving in-context learning. arXiv preprint arXiv:2401.16184, 2024. Jian Gu, Aldeida Aleti, Chunyang Chen, and Hongyu Zhang. A semantic-aware layer-freezing approach to computation-efficient fine-tuning of language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8019–8033, 2025. Jian Gu, Aldeida Aleti, Chunyang Chen, and Hongyu Zhang. Rethinking weight tying: Pseudoinverse tying for stable lm training and updates, 2026a. Jian Gu, Aldeida Aleti, Chunyang Chen, and Hongyu Zhang. Beyond neural incompatibility: Crossscale knowledge transfer in large language models through latent semantic alignment. In Findings of the Association for Computational Linguistics: ACL 2026, pages 21893–21905, 2026b. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 10, 2022. Zhimin Hu, Lanhao Niu, and Sashank Varma. The representational geometry of number. arXiv preprint arXiv:2602.06843, 2026. Hai Huang, Yann LeCun, and Randall Balestriero. Semantic tube prediction: Beating llm data efficiency with jepa. arXiv preprint arXiv:2602.22617, 2026. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023. Jett Janiak, Jacek Karwowski, Chatrik Singh Mangat, Giorgi Giglemiani, Nora Petrova, and Stefan Heimersheim. Characterizing stable regions in the residual stream of llms. arXiv preprint arXiv:2409.17113, 2024. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Tim Lawson, Lucy Farnik, Conor Houghton, and Laurence Aitchison. Residual stream analysis with multi-layer saes. arXiv preprint arXiv:2409.04185, 2024. 14
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
Yuxin Liang, Rui Cao, Jie Zheng, Jie Ren, and Ling Gao. Learning to remove: Towards isotropic pre-trained bert embedding. In International conference on artificial neural networks, pages 448–459. Springer, 2021. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. Nostalgebraist. Interpreting gpt: The logit lens. LessWrong post, 2020. URL https://www.le sswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-l ens. Accessed 2026-02-22. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2463–2473, 2019. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. Advances in neural information processing systems, 32, 2019. Adam Shai, Lucas Teixeira, Alexander Oldenziel, Sarah Marzen, and Paul Riechers. Transformers represent belief state geometry in their residual stream. Advances in Neural Information Processing Systems, 37:75012–75034, 2024. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-mon osemanticity/index.html. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205–78226, 2023. Chaojun Xiao, Jie Cai, Weilin Zhao, Biyuan Lin, Guoyang Zeng, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, and Maosong Sun. Densing law of llms. Nature Machine Intelligence, pages 1–11, 2025. Zeping Yu and S Ananiadou. Exploring the residual stream of transformers, 2023. 15
G U A LETI C HEN Z HANG
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.
Appendix A. Additional Proofs The additional proofs to our research are as follows, grouped by the corresponding technical role. A.1. Coordinate Measurement and Interface Synchronization Proof of Proposition 2. Because Z ⊤ Z = Id and T is invertible, WU+ = (ZT ⊤ )+ = T −⊤ Z ⊤ . ⊤ = T −⊤ Z ⊤ , so W + = W ⊤ . Using Win = ZT −1 gives Win in U
Proof of Proposition 3. Let EB+ = (EB⊤ EB )−1 EB⊤ . Since EB has full column rank, EB+ EB = IK . Hence UB⊤ EB − IK = (UB⊤ − EB+ )EB , so if η := ∥UB⊤ − EB+ ∥, then
∥UB⊤ EB − IK ∥ ≤ η ∥EB ∥.
If instead UB⊤ = ẼB+ for a nearby full-column-rank synthesis matrix ẼB , then ẼB+ ẼB = IK and therefore UB⊤ EB − IK = ẼB+ (EB − ẼB ), which gives ∥UB⊤ EB − IK ∥ ≤ ∥ẼB+ ∥ δ.
Proof of Proposition 5. For hc = EB c, ϕrd (hc ) − c = (UB⊤ EB − I)c, so the first claim follows from Assumption 1. For the general decomposition h = EB c + r, ϕrd (h) − c = (UB⊤ EB − I)c + UB⊤ r, and the stated bound follows by the triangle inequality. Proof of Corollary 6. The state bound is the second bound of Proposition 5, applied at layer ℓ with hℓ = EB cℓ + rℓ . For the step bound, write ∆zℓ − ∆cℓ = (zℓ+1 − cℓ+1 ) − (zℓ − cℓ ), and apply the state bound at layers ℓ + 1 and ℓ.
16
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
A.2. Layerwise Dynamics and Frame Stability Proof of Proposition 7. Apply Assumption 2 with h = hℓ and δ = ∆hℓ : ϕ(hℓ + ∆hℓ ) − ϕ(hℓ ) = Jϕ (hℓ )∆hℓ + r(hℓ , ∆hℓ ), and rename the remainder εℓ . Proof of Proposition 8. Let A := UB⊤ EB and à := ŨB⊤ EB . Since ∥A − I∥ ≤ ε < 1, the Neumann-series bound gives A invertible with ∥A−1 ∥ ≤ 1/(1 − ε). For h = EB c, ϕ̃(h) = Ãc = ÃA−1 ϕ(h).
ϕ(h) = Ac,
Hence ϕ̃ = (I + ∆)ϕ on the anchor span with ∆ := ÃA−1 − I = (Ã − A)A−1 . Using ∥Ã − A∥ ≤ ∥Ã − I∥ + ∥A − I∥ ≤ ε̃ + ε, one obtains ∥∆∥ ≤
ε + ε̃ . 1−ε
Proof of Corollary 9. By Proposition 8, on the anchor span one has ϕ̃ = (I + ∆)ϕ with ∥∆∥ ≤ η :=
ε + ε̃ . 1−ε
For any u, v ∈ span(EB ), ⟨ϕ̃(u), ϕ̃(v)⟩ − ⟨ϕ(u), ϕ(v)⟩ = ⟨∆ϕ(u), ϕ(v)⟩ + ⟨ϕ(u), ∆ϕ(v)⟩ + ⟨∆ϕ(u), ∆ϕ(v)⟩. Applying the Cauchy–Schwarz inequality and the operator-norm bound on ∆ gives ⟨ϕ̃(u), ϕ̃(v)⟩ − ⟨ϕ(u), ϕ(v)⟩ ≤ (2η + η 2 ) ∥ϕ(u)∥ ∥ϕ(v)∥. For distances, set w := ϕ(u) − ϕ(v). Then ϕ̃(u) − ϕ̃(v) = (I + ∆)w, so ∥(I + ∆)w∥ ≤ (1 + η)∥w∥. If η < 1, the reverse triangle inequality also gives ∥(I + ∆)w∥ ≥ ∥w∥ − ∥∆w∥ ≥ (1 − η)∥w∥. This proves the stated distance bounds. Proof of Theorem 10. The statement is immediate from Assumption 3: ∥∆zℓ ∥ = ∥ϕ(hℓ + ∆hℓ ) − ϕ(hℓ )∥ ≤ Lϕ ∥∆hℓ ∥. 17
G U A LETI C HEN Z HANG
The uniform bound follows by substituting ∥∆hℓ ∥ ≤ U . Proof of Theorem 11. For h = EB c, ϕ(h) − ϕ̃(h) = (UB⊤ − ŨB⊤ )EB c = ((UB⊤ EB − I) − (ŨB⊤ EB − I))c. Hence ∥ϕ(h) − ϕ̃(h)∥ ≤ (ε + ε̃)∥c∥. Since ∥h∥ = ∥EB c∥ ≥ σmin (EB )∥c∥, it follows that ∥ϕ(h) − ϕ̃(h)∥ ≤
ε + ε̃ ∥h∥ = Cframe ∥h∥. σmin (EB )
For the step bound, use linearity: (ϕ(h + ∆h) − ϕ(h)) − (ϕ̃(h + ∆h) − ϕ̃(h)) = (UB⊤ − ŨB⊤ )∆h. Writing ∆h = EB ∆c and repeating the previous argument yields ∥(ϕ(h + ∆h) − ϕ(h)) − (ϕ̃(h + ∆h) − ϕ̃(h))∥ ≤
ε + ε̃ ∥∆h∥ = Cframe ∥∆h∥. σmin (EB )
Proof of Proposition 12. Let z cmp be any endpoint-matched comparison path, so z0obs = z0cmp and zLobs = zLcmp . By definition, L−1 X
gℓ =
L−1 X
ℓ=0
⟨∆zℓobs − ∆zℓcmp , ey ⟩ = ⟨zLobs − z0obs − (zLcmp − z0cmp ), ey ⟩ = 0.
ℓ=0
For every j ∈ {1, . . . , L}, the same telescoping computation gives j−1 X
gℓ = ⟨zjobs − z0obs − (zjcmp − z0cmp ), ey ⟩ = ⟨zjobs − zjcmp , ey ⟩,
ℓ=0
because the comparison path shares the same initial state as the observed path.
A.3. Trace Optimality and Compressibility Endpoint-constrained convexity. For fixed endpoints, collect the interior variables into z := (z1 , . . . , zL−1 ) ∈ (RK )L−1 . The quadratic part is Q[z] = α z⊤ (D1⊤ D1 ⊗ IK )z + β z⊤ (D2⊤ D2 ⊗ IK )z + boundary terms, where D1 , D2 are endpoint-constrained difference matrices. Their nullspaces are trivial on zeroendpoint perturbations, so the quadratic Hessian is positive definite whenever (α, β) ̸= (0, 0); convex backtracking and tube constraints preserve uniqueness over the feasible set. 18
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
Proposition 21 (Strong-convexity modulus) In the endpoint-constrained regime, the quadratic part of the action is µ0 -strongly convex in the interior variables, where π π µ0 := 8α sin2 + 32β sin4 . 2L 2L Proof of Proposition 21. The Hessian is H = 2α(D1⊤ D1 ⊗ IK ) + 2β(D2⊤ D2 ⊗ IK ). The smallest eigenvalues of D1⊤ D1 and D2⊤ D2 are 4 sin2 (π/(2L)) and 16 sin4 (π/(2L)), respectively; summing the contributions gives the bound. Proof of Theorem 15. With interior variables z = (z1 , . . . , zL−1 ), the tube is convex and, for γ = 0, the objective has Hessian H = 2α(D1⊤ D1 ⊗ IK ) + 2β(D2⊤ D2 ⊗ IK ). By the convexity facts above, H is positive definite whenever (α, β) ̸= (0, 0); adding the convex backtracking term preserves uniqueness over the convex tube. On inactive interior layers, varying z ⋆ + τ v with v0 = vL = 0 gives L−1 L−1 X X ⋆ 0=α ⟨∆zℓ , ∆vℓ ⟩ + β ⟨∆2 zℓ⋆ , ∆2 vℓ ⟩. ℓ=0
ℓ=1
Discrete summation by parts yields the coefficient −α∆2 zℓ⋆ + β∆4 zℓ⋆ , hence β∆4 zℓ⋆ = α∆2 zℓ⋆ . Active Voronoi-face constraints add KKT normal-cone terms. Proof of Proposition 16. Write S = Q + γB, where Q is the quadratic part and B the convex backtracking term. Let C := Tρ (z TF ) and e := z − z ⋆ . Optimality gives s ∈ ∂B(z ⋆ ) and n ∈ NC (z ⋆ ) such that ∇Q(z ⋆ ) + γs + n = 0. Since ⟨n, e⟩ ≤ 0 for feasible z ∈ C, convexity of B gives γ B[z] − B[z ⋆ ] ≥ γ⟨s, e⟩ = −⟨∇Q(z ⋆ ), e⟩ − ⟨n, e⟩. Therefore S[z] − S[z ⋆ ] ≥ Q[z] − Q[z ⋆ ] − ⟨∇Q(z ⋆ ), e⟩ − ⟨n, e⟩ ≥ Q[z] − Q[z ⋆ ] − ⟨∇Q(z ⋆ ), e⟩. Expanding the quadratic part yields Q[z] − Q[z ⋆ ] − ⟨∇Q(z ⋆ ), e⟩ = α
L−1 X
L−1 X
ℓ=0
ℓ=1
∥∆eℓ ∥2 + β
∥∆2 eℓ ∥2 .
This proves the lower bound. When γ = 0 and the active normal-cone pairing vanishes, the bound is tight. Proof of Corollary 17. The step and curvature bounds are immediate from Proposition 16. Strong convexity on endpoint-constrained variables gives L−1
S[z] ≥ S[z⋆ ] +
µ µX ∥zℓ − zℓ⋆ ∥2 . ∥z − z⋆ ∥2 = S[z⋆ ] + 2 2 ℓ=1
19
G U A LETI C HEN Z HANG
For the contribution-profile bound, let eℓ := zℓobs − zℓ⋆ . Then e0 = eL = 0 and gℓ = ⟨∆eℓ , ey ⟩. Since ∥ey ∥ = 1, L−1 L−1 1/2 X X X √ L−1 . |gℓ | ≤ ∥∆eℓ ∥ ≤ L ∥∆eℓ ∥2 ℓ=0
ℓ=0
ℓ=0
Proposition 16 bounds the final factor by α−1/2 ∆S 1/2 , and every partial sum is bounded by the total variation.
Lemma 22 (Block interpolation bound) Let e0 , . . . , en ∈ RK satisfy e0 = en = 0. Then n X
∥ek ∥2 ≤ 2n4
k=0
n−1 X
∥∆2 ek ∥2 .
k=1
Proof Pn of Lemma 22. Write gk := ek − ek−1 for k = 1, . . . , n. Since e0 = en = 0, one has k=1 gk = 0. Hence, for each k, n 1X gk = (gk − gi ). n i=1
If i < k, then gk − gi =
Pk−1
2 j=i ∆ ej , while if i > k, then gk − gi = −
∥gk ∥ ≤
n−1 X
∥∆2 ej ∥ ≤
√
n−1
Pi−1
2 j=k ∆ ej . Therefore
n−1 1/2 X ∥∆2 ej ∥2 .
j=1
j=1
Summing the first differences gives, for every k, ∥ek ∥ =
k X
gi ≤ n3/2
n−1 1/2 X ∥∆2 ej ∥2 .
i=1
j=1
Squaring and summing over k = 0, . . . , n yields n X
2
∥ek ∥ ≤ (n + 1)n
3
n−1 X
2
2
∥∆ ej ∥ ≤ 2n
j=1
k=0
4
n−1 X
∥∆2 ej ∥2 ,
j=1
which proves the claim. Proof of Theorem 20. Let the interval {0, . . . , L} be partitioned into m contiguous blocks Ij = [sj , sj+1 ] with lengths nj := sj+1 − sj ≤ ⌈L/m⌉. Let z̃ be the piecewise-linear interpolant that agrees with z at all block endpoints. On each block, define the interpolation error eℓ := zℓ − z̃ℓ . Then esj = esj+1 = 0 and ∆2 eℓ = ∆2 zℓ on interior indices of the block. Applying Lemma 22 on each block gives sj+1 −1 X X ∥eℓ ∥2 ≤ 2n4j ∥∆2 zℓ ∥2 . ℓ∈Ij
ℓ=sj +1
20
S EMANTIC R EFERENCE F RAME FOR R ESIDUAL -S TREAM DYNAMICS
Summing over blocks and using nj ≤ ⌈L/m⌉ gives L X ℓ=0
l L m4 L4 ∥zℓ − z̃ℓ ∥2 ≤ 2 E2 (z) ≤ c1 4 E2 (z) m m
for a universal constant c1 . Dividing by L + 1 yields the stated bound after absorbing fixed numerical factors into the universal constant c.
21