Journey Operators for Structured Multi-Axis Composition
Mahesh Godavarti A Carrot, Inc
arXiv:2607.26775v1 [cs.LG] 29 Jul 2026
Abstract Many kinds of data have structure along one or more axes: words in a sentence, pixels in an image, nodes in a tree, frames in audio, or cells in a 3D volume. Along one axis, order matters: “the dog bit the man” is different from “the man bit the dog.” Across independent axes, however, neither composition nor movement should depend on the order of axes: in an image, composing right then down should give the same result as composing down then right, and moving right then down should describe the same relative position as moving down then right. We develop a framework for modeling this kind of multi-axis structure. Each data item carries its content together with a small transformation for each axis. A path connecting two positions defines a journey; the journey operator is the product of per-axis transformations along that path, governing both how data composes along the path and how relative position is described. When the transformations are fixed, our framework recovers Rotary Position Embedding (RoPE) and its multi-dimensional variants. When they depend on the data, the model gains a content-adaptive positional inductive bias. We show exactly when these paths are well-defined: both composition and movement across axes are path-independent precisely when the axis transformations commute. We also prove that, under the stated toral-frame symmetry, cocycle, bilinearity, and norm-preservation assumptions, the resulting pairwise scoring rule must take the form of block-wise rotations, explaining why RoPE-like methods arise naturally. Finally, we use this theory to design JoFormer, a model for value aggregation, and relate it to attention and state-space models (SSMs). Initial experiments across vision, language, and length generalization suggest that these inductive biases can have observable consequences in practice.
1
Introduction
Any data that can be indexed—sequences, images, audio, volumetric arrays, trees—composes along multiple axes. In the one-dimensional (1D) case, a sequence like [a b c d] can be formed by composing sub-sequences—for example, by concatenating [a b] with [c d], or [a] with [b c d]. Algebraic tools such as non-commutative semigroups or free groups have long provided principled ways to model such 1D compositional structure [Rudolph and Giesbrecht, 2010]. In contrast, there exists no broadly accepted a b algebraic framework for modeling two-dimensional (2D) composition. Consider the array c d : it can be composed vertically, by stacking [a b] over [c d], or horizontally, by placing ac next to db . Such multiple valid composition paths do not fit neatly into existing algebraic systems, which are typically designed for linear (1D) structured data. Even in the 1D case, aligning algebraic formalisms with the needs of modern machine learning architectures can be problematic—for example, representing tokens as matrices instead of vectors breaks the core assumptions of the attention mechanism in transformers, which relies fundamentally Preprint.
on vector operations. This tension reveals a deeper issue: the absence of an algebraic framework that can both align with vector-based learning architectures and naturally support composition along multiple axes. We present such an algebraic framework, built on one operation that models composition in 1D and extends naturally to multiple axes. In 1D, each data point carries both content and a single orthogonal transformation—its axis-step generator. Composing two data points applies the first point’s axis-step generator to rotate the second’s content, and multiplies their axis-step generators together. This is associative but non-commutative: the first element’s axis-step generator acts on the second’s content, not vice versa, so order matters. Composing an entire sequence accumulates these transformations: each value is transformed by the product of all preceding axis-step generators (Section 2.1). In nD, each datum carries one axis-step generator per axis. Composing two data points along a given axis applies that axis’s axis-step generator to rotate the second point’s content into the first’s frame, accumulates the axis-step generator for that axis, and leaves all other axes unchanged. For multi-axis data like images, we want two kinds of path independence: composition itself should not depend on the order of axes (composing right-then-down equals down-then-right), and any movement should depend only on the initial and final positions, not on the path taken. This is the inductive bias—it encodes the assumption that independent axes of real-world data commute. Our framework captures both: when per-axis axis-step generators commute (the flat regime), composition is path-independent and the journey—the relative transformation between two positions—depends only on displacement (Theorem 2.2). In 1D with a single axis-step generator, this recovers RoPE; content-dependent axis-step generators yield data-dependent composition (JoFormer-projected). This compositional structure is absent from standard attention, which aggregates values without non-commutative composition. Bringing composition into attention—applying the journey operator to values before aggregation—is the architecture (JoFormer) developed in Section 3. Initial experiments across vision, language, and length generalization are consistent with the framework’s value (Section 4). With fixed rotations, the framework is a generalized DFT with learned frequencies. RoPE [Su et al., 2021], 2D-RoPE [Heo et al., 2024], tree PEs [Shiv and Quirk, 2019], and RotatE [Sun et al., 2019] emerge as special cases; the value-path formula provides common notation for standard attention, SSMs, and JoFormer (Section F.2).
Contributions. 1. We introduce a compositional framework (Section 2.1): in 1D, each data point carries content and a single axis-step generator (an orthogonal transformation); in nD, each data point carries one axis-step generator per axis. Composition along any axis is a single associative operation. 2. From this framework we derive several theoretical results: (a) path-independent composition requires commuting axis-step generators (Theorem 2.2); (b) under natural axioms (bilinearity, toral frame invariance, cocycle, norm preservation), the only compatible mechanism is block-diagonal SO(2)d/2 rotation (Theorem 2.4); if the full orthogonal group O(d) is the symmetry instead, positional structure collapses (Theorem 2.5); (c) in the flat (commutative) regime, value transport via the journey operator is translation equivariant (Theorem 2.8); (d) the DFT is a special case of the compositional embedding, and learned frequencies yield a generalized Fourier transform (Theorem C.1). 3. These constraints lead to JoFormer (Section 3), an architecture that brings composition into attention via data-dependent projected angles and value-side rotation. 4. A value-path hierarchy (Section F.2) places standard attention, SSMs, and JoFormer on a common spectrum—the SSM recurrence unrolls into the same value-path formula (Theorem F.1)—clarifying what each family composes and what it discards. 5. Experiments on vision, language modeling, and length generalization suggest the framework has practical value (Section 4). 2
2
Algebraic Framework and Theoretical Analysis
2.1
Compositional Data Points
Everything in this paper flows from one operation. In 1D, each data point is a pair (v, R): content v ∈ Rd plus an orthogonal transformation R ∈ G ≤ O(d) (the group of norm-preserving d × d matrices)—its axis-step generator. Given two such points (a, A) and (b, B)—where a, b ∈ Rd are contents and A, B ∈ G are axis-step generators—their composition is: (a, A) ◦ (b, B) := (a + Ab, AB).
(1)
The content of the composite is a plus b rotated into a’s frame by A; the axis-step generator of the composite is the product AB. This operation is associative with identity (0, I), but non-commutative: the first element’s axis-step generator acts on the second’s content, so order matters. Axis-specific composition. For data indexed by D axes, each datum carries content and one axisstep generator per axis: (v, R1 , . . . , RD ). Given two data points with contents a, b ∈ Rd at axis-k positions nk and mk respectively, composing along axis k applies the first point’s axis-k generator to rotate the second’s content and accumulates that axis-step generator, leaving other axes unchanged: nD nD (a, R1n1 , . . . , Rknk , . . . , RD ) ◦k (b, R1n1 , . . . , Rkmk , . . . , RD ) nD ). = (a + Rknk b, R1n1 , . . . , Rknk +mk , . . . , RD
Crucially, ◦k is defined only when the two operands agree on all axes i ̸= k—they must share the same exponents (n1 , . . . , nk−1 , nk+1 , . . . , nD ). Composition therefore operates over tiles: 1D slices of the grid at fixed coordinates on the remaining D−1 axes. One cannot, for instance, compose elements from different rows of a 2D grid along the column axis. When Rk Rl = Rl Rk for all axis pairs, the regime is flat: composition is path-independent (composing along axis k then l gives the same result as l then k), and the relative transformation between two positions depends only on their displacement, not on which path connects them (Theorem 2.2). Both properties follow from the single condition of commuting axis-step generators. When axis-step generators do not commute, the regime is curved: composition depends on the order of axes, and relative transformations depend on absolute position. Sequence unrolling.
In 1D, composing a sequence [e1 , . . . , eT ] with et = (vt , Rt ) gives: e1 ◦ e2 ◦ · · · ◦ eT =
T i−1 X Y i=1 j=1
T Y Rj vi , Rj .
(2)
j=1
P Q The content part i ( j<i Rj )vi is a weighted sum where each value is rotated by all preceding transformations—position is encoded through accumulated rotation. The journey operator. The journey operator from position j to position k is defined as the composition of axis-step generators along the path connecting them. In 1D, traveling from j to k (with j > k) means composing the generators at positions k, k+1, . . . , j−1 in sequence. Qi−1 Absolute operators. Separately, define the absolute operator at position i as Ai = j=1 Rj —so A P1 = I, A2 = R1 , A3 = R1 R2 , and so on. The unrolled content (2) can then be rewritten as i Ai vi : each value vi is expressed in the global frame (position 1’s frame) via Ai . Journey in terms of absolute operators. We can show that the path-defined journey equals −1 A−1 k Aj . To see this: the absolute operator Aj maps vj into the global frame, and Ak maps −1 from the global frame into position k’s frame. Their composition Pj→k = Ak Aj therefore transforms vj into k’s frame—exactly what the path-defined journey does. When all axis-step generators are identical (Rt = R), the absolute operator is Ai = Ri−1 and the journey reduces to Pj→k = R−(k−1) Rj−1 = Rj−k , a pure function of displacement. In nD, the absolute operator at nD position n is An = R1n1 · · · RD ; the formal multi-axis treatment is in Section 2.2. 3
Value-path formula (1D). The unrolled content gives a sum of rotated values, each weighted equally. In an attention mechanism, different positions contribute differently: position j’s value is weighted by an attention weight αkj (how much position k attends to position j). Combining the journey operator with selective attention yields the value-path formula: X ck = αkj Pj→k vj , (3) j −1 where Pj→k = Ak Aj is the journey from Q j to k. This notation unifies standard attention (P = I, no compositional structure), SSMs (P = Axt , content-dependent recurrence with uniform weights), and JoFormer (P = A−1 k Aj with full attention weights).
2D composition and value-path formula. For a 2D grid of data points e(i,j) = (v(i,j) , Rx , Ry ), P we compose along rows first (1D composition along x): each row i gives j Rxj v(i,j) by the 1D unrolling (2). Composing the row results along columns (1D composition along y) applies Ryi to each row’s content: X X X X Ryi Rxj v(i,j) = Ryi Rxj v(i,j) = A(i,j) v(i,j) , i
j
i,j
i,j
where A(i,j) = Ryi Rxj is the absolute operator at position (i, j). When the axis-step generators i j commute, composing columns-then-rows gives the same result (since RyP Rx = Rxj Ryi ). This generalizes directly to D axes: composing along each axis in turn yields n An vn with An = QD nk ′ ′ k=1 Rk , independent of the order of axes when generators commute. The journey from (i , j ) to (i, j) is: −j −i i′ j′ ′ ′ P(i′ ,j ′ )→(i,j) = A−1 (i,j) A(i ,j ) = Rx Ry Ry Rx . ′
′
When the axis-step generators commute (Rx Ry = Ry Rx ), this simplifies to Ryi −i Rxj −j —a pure function of displacement. The value-path formula becomes: X ′ ′ c(i,j) = α(i,j),(i′ ,j ′ ) Ryi −i Rxj −j v(i′ ,j ′ ) . (4) (i′ ,j ′ )
This generalizes directly to D axes: the absolute operator at position n = (n1 , . . . , nD ) is An = QD nk k=1 Rk , and the journey (with commuting axis-step generators) is the multi-axis displacement QD n′ −n operator Pn′ →n = k=1 Rk k k . The hierarchy is developed in Section F.2. 2.2
Path Transport and Commutativity
We now formalize how composing axis-step generators along different paths may or may not yield the same result. Definition 2.1 (Multi-axis composition and path transport). For D axes with axis-step generators R1 , . . . , RD ∈ G ≤ O(d), a position n = (n1 , . . . , nD ) ∈ ZD has absolute operator An = nD R1n1 · · · RD (as in Section 2.1). For a word π = k1 · · · km over axes {1, . . . , D} (a path), define the path transport: T (π) = Rkm · · · Rk1 . (5) The path transport T (π) composes generators step by step; different orderings of the same axis steps may yield different composites—the transport is path-dependent in general. Under the canonical axis ordering, the composite transport from m to n evaluates to A−1 n Am , but this equals the transport along every other path only when generators commute. This motivates the central theorem. Theorem 2.2 (Path independence ⇔ commuting generators). If you compose along a multi-axis grid using per-axis axis-step generators, the result is independent of the order you take the steps (path-independent) if and only if the axis-step generators commute. When they commute, the journey operator Pm→n = A−1 n Am depends only on displacement—it is fully determined by composition. Formal statement and proof in Section A.1. Remark 2.3 (Connection to interchange laws). In higher category theory, the interchange law (x ◦i y) ◦j (z ◦i w) = (x ◦j z) ◦i (y ◦j w) states that two composition operations are compatible. Path independence is the geometric manifestation of this algebraic condition: the interchange law holds for group-valued elements iff generators commute iff transport is path-independent. 4
2.3
Journey Operator and Score/Value Attention
As derived in Section 2.1, composing axis-step generators along each axis yields absolute operators Ai , Aj ∈ G ≤ O(d) at each position. In the flat (commutative) regime established by Theorem 2.2, the journey operator arising from composition reduces to: Pj→i = A−1 i Aj . This operator structures both the score and value sides of attention: Att(i, j) = qi⊤ Pj→i kj = (Ai qi )⊤ (Aj kj ), P P Value: ci = A−1 i j αij Aj vj = j αij Pj→i vj .
Score:
(6) (7) (8)
⊤ For orthogonal Ai : (Ai qi ) (Aj kj ) = qi⊤ A−1 i Aj kj = qi Pj→i kj , confirming consistency. Score computation rotates q and k by their respective absolute operators; value computation rotates each vj by Aj then inverse-rotates the output by A−1 i . RoPE as a special case. For a sequence, At = Rt with R = diag(R(θ1 ), . . . , R(θd/2 )) being a block-diagonal rotation. Then Pj→i = R−i Rj = Rj−i , and the score becomes (Ri qi )⊤ (Rj kj ) = qi⊤ Rj−i kj —exactly RoPE [Su et al., 2021]. Standard RoPE uses only the score side; our framework ⊤
additionally rotates values. 2D-RoPE as a special case. For a 2D grid with generators Rx , Ry acting on disjoint subspaces (first d/4 planes for x, remaining for y): A(m,n) = Rxm Ryn = diag(R(ω1 m), . . . , R(ωd/4 m), R(ϕ1 n), . . . , R(ϕd/4 n)). Since the generators act on disjoint planes, Rx Ry = Ry Rx —path independence is satisfied by construction. 2.4
Toral Classification
We classify all bilinear, norm-preserving, cocycle-compatible attention mechanisms under toral frame symmetry T = SO(2)d/2 . The result: the score matrix must be a block-diagonal SO(2)d/2 rotation—each of d/2 planes gets an independent angle θk (s) − θk (s′ ). This recovers RoPE [Su et al., 2021], 2D-RoPE [Heo et al., 2024], and tree PEs [Shiv and Quirk, 2019] as special cases. If the full orthogonal group O(d) is the symmetry instead, positional structure collapses entirely. Full axioms (bilinearity, toral frame invariance, cocycle, norm preservation), formal statements, and proofs are in Section A.2. Theorem 2.4 (Toral Classification). Under bilinearity, toral frame invariance, cocycle compositionality, and norm preservation, the score matrix must be a block-diagonal rotation: each of d/2 planes gets an independent angle θk (s) − θk (s′ ), and the full operator is their direct sum. This is the unique form compatible with the symmetry contract. Formal statement and proof in Section A.3. Theorem 2.5 (O(d) Impossibility). If the full orthogonal group O(d) is the frame symmetry (instead of the torus), every score matrix reduces to a scalar multiple of the identity—positional structure is completely lost. Proof in Section A.4. 2.5
Flat/Curved Dichotomy and the V Rotation Prediction
Definition 2.6 (Flat and curved). A multi-axis compositional framework is flat if its per-axis axis-step generators commute ([Ri , Rj ] = 0 for all i ̸= j), and curved otherwise. All block-diagonal SO(2)d/2 methods are flat; dense rotation matrices (like LieRE) are generically curved. In the flat regime, Pj→i vj = R(∆p)vj depends only on relative displacement—the journey operator is fully determined by composition. Definition 2.7 (Translation equivariance). For a shift u ∈ ZD , define the translation operator (Tu v)m = vm−u . An aggregation rule F mapping value fields v to output fields c is translation equivariant if F (Tu v)n+u = F (v)n for all n, u: shifting inputs shifts outputs by the same amount. Theorem 2.8 (Relative-displacement equivariance of value transport). When per-axis generators commute (flat regime) and attention weights depend only on relative displacement, applying the journey operator A−1 n Am to values before aggregation gives a translation-equivariant rule: shifting all positions by the same amount shifts outputs identically. When generators do not commute, the value transport becomes path-dependent—it depends on absolute positions, not just displacement. Formal statement and proof in Section A.5. 5
Remark 2.9 (Scope: operator-level, not whole-layer equivariance). In a standard transformer, √ attention weights αij = softmaxj (qi⊤ Pj→i kj / d) depend on token content through qi = WQ xi and kj = WK xj , so they are not purely displacement-dependent. Theorem 2.8 therefore does not imply that the full attention layer is translation equivariant. What the theorem does justify is the valueside operator choice: once absolute operators are determined by composition (Section 2.1), A−1 i Aj is the canonical cocycle-compatible journey that yields displacement-dependent value transport when weights happen to be displacement-only. In the general content-dependent case, no full equivariance guarantee remains, but the value transform is still norm-preserving and uses the same toral operator family. We test the flat-regime hypothesis experimentally in Section 4.1. 2.6
Value-Path Hierarchy and Relation to SSMs
The value-path formula (3), introduced in Section 2.1 as a direct consequence of the composition operation, provides common notation for standard attention, SSMs, and JoFormer. This is a notationlevel bridge rather than an equivalence: Mamba’s transition matrices are not generally toral, attention weights are normalized, and the forward SSM kernel differs from the attention journey by convention. The hierarchy, SSM bridge proposition, and detailed analysis are in Section F.2.
3
JoFormer Architecture
The algebraic framework motivates a concrete architecture: the JoFormer (Journey-based Transformer), which implements data-dependent journey operators on both the score and value sides. 3.1
Design Principles
Three inductive biases follow from the theory: 1. Commutativity: use block-diagonal SO(2)d/2 rotations (flat, satisfying path independence). Under the full modeling contract (bilinearity, toral-frame invariance, cocycle, norm preservation), this is the only compatible structure (Theorem 2.4). The choice of T = SO(2)d/2 as the symmetry group is an assumption of that contract, not a consequence. 2. V rotation: apply Pj→i to values (justified by the flat/curved analysis—the journey operator arising from composition gives displacement-dependent value transport when axis-step generators commute). 3. Data dependence: compute angles θ(x) from content, enriching the value-path beyond fixed positional functions—an SSM-like data-dependent value transform, though without the recurrent product over intermediate states. What the theory does and does not justify. The core claim is that standard attention lacks a structural inductive bias on the value side: commutative summation discards compositional context. Theorems 2.2 and 2.8 prove that the journey operator—arising directly from composition—fills this gap, giving displacement-dependent value transport exactly in the flat/commutative regime with relative attention weights. JoFormer-fixed and JoFormer-learned preserve this contract. JoFormerprojected keeps the same norm-preserving toral operator family but makes angles content-dependent; it is therefore a theory-motivated architecture, not a direct translation-equivariance corollary. A detailed breakdown of which theorem supports which variant is in Table 8 (Section F.1). 3.2
Architecture Variants
All variants share the same attention computation: √ αij ∝ exp (R(θi )qi )⊤ (R(θj )kj )/ d , P Value: ci = R(θi )−1 j αij R(θj ) vj .
Score:
The variants differ in how angles θi are computed: 6
(9) (10)
(l)
JoFormer-fixed. Angles are linear in position: θk (t) = ωk · t (same as RoPE frequencies). This is equivalent to RoPE on Q/K plus V rotation and inverse rotation on output. Implements the journey value path Pj→i = R−i Rj = Rj−i . JoFormer-learned. Per-layer learned frequency vectors ω (l) ∈ Rd/2 define θ(l) (t) = t · ω (l) . This is equivalent to RoPE with per-layer learned frequencies instead of fixed geometric spacing. Enables layer-specific frequency selection while maintaining the linear-in-position structure. JoFormer-projected.
Per-layer MLP angle projectors compute angles from the residual stream: (l)
(l)
θ(l) (x) = W2 GELU(W1 LN(x)),
(11)
(l) (l) with W1 ∈ Rd×d , W2 ∈ Rd/2×d . Angles are content-dependent: computed fresh at each layer from the current representation. This provides an SSM-like data-dependent value transform: Pj→i
depends on the current residual representations at positions i and j, but does not implement the recurrent product over intermediate states. The MLP adds ∼ 1.5d2 parameters per layer (small relative to the 12d2 for attention + FFN). In practice, JoFormer-projected requires softmax attention (softplus is unstable with data-dependent angles, as unbounded weights accumulate across layers) and a lower learning rate (≤ 2×10−4 ). Attention ordering for vision. JoFormer uses K ⊤ Q (not Q⊤ K) attention ordering in vision applications. Since qi⊤ Pj→i kj = (Ai qi )⊤ (Aj kj ) = q̂i⊤ k̂j , both orderings yield equivalent scores; we adopt K ⊤ Q as a convention matching the reference-factored structure k̂j⊤ q̂i .
4
Experiments
Each experiment is a single-seed sanity check for a different theoretical hypothesis; none are benchmark claims. MNIST (Section E.1): monoidal compression. CIFAR-100/ImageNet: V rotation. Wikipedia LM: value-path hierarchy. Length generalization: projected angles. 4.1
CIFAR-100 V Rotation and Scaling
Setup. Two experimental setups test V rotation for vision. (A) ViT-Tiny (D=384, 12 layers, 6 heads, ∼14.9M params, patch 4×4, CIFAR-100, 200 epochs, Adam lr= 10−4 , cosine annealing, H100, seed=42) using LieRE’s framework [Ostmeier et al., 2024]. (B) Smaller ViT (4 layers, 4 heads, patch 4×4, CIFAR-100, 300 epochs, cosine lr= 10−3 , dropout=0.1, weight decay, mixup, cutout) at D ∈ {32, 64, 128, 256} with fully deterministic GPU-resident training. Both setups use exact reproducibility (fixed seeds, no DataLoader workers, torch-op augmentation on GPU). Methods. Nine PE variants organized on two axes: frequency type (fixed RoPE vs. learned) and V rotation (Q/K only vs. Q/K/V with inverse rotation on output). The axial factorization splits dimensions into disjoint y- and x-subspaces, guaranteeing commutativity. Table 1: V rotation effect on CIFAR-100 (paired comparisons, Setup B, D=256, deterministic single run). V rotation helps in all five paired comparisons; the synergy with learnable frequencies is largest. Approach Axial learned Axial learned (per-layer) Axial fixed Combined fixed Combined learned (per-layer)
Q/K only
Q/K/V
monoidal_axial: 60.92% 60.63% rope2d: 61.39% rope2dv2: 55.53% 60.85%
joformer_axial: 63.27% 62.76% joformer_old: 61.85% joformer_fixed: 56.66% 61.17%
∆ +2.35% +2.13% +0.46% +1.13% +0.32%
Results: V rotation is consistently beneficial in these paired single-seed CIFAR protocols. V rotation helps consistently across D=32–256 (Table 6 in Section E.3), with V rotation and learnable frequencies synergistic (+1.88% at D=256). In the ViT-Tiny LieRE framework, axial-dense V rotation gains +0.51%, while LieRE64’s dense (curved) rotation drops 1.23% (Section E)—consistent with the flat-regime hypothesis. 7
4.2
ImageNet ViT-S Scale Validation
ViT-S (D=384, 12 layers, 6 heads, ∼22M params, DeiT-III recipe [Touvron et al., 2022], 300 epochs, single seed; details in Section E). Table 2: ImageNet-1K ViT-S (DeiT-III, 300 epochs, single seed). JoFormer adds V rotation + inverse rotation to RoPE2D. Method
Top-1 Acc
Top-5 Acc
RoPE2D (axial, Q/K only) JoFormer (axial, Q/K/V)
80.71% 81.11%
95.26% 95.53%
∆
+0.40%
+0.27%
The +0.40% gap is consistent from epoch 180 onward. Single-seed: a scale sanity check, not a conclusive ImageNet improvement. 4.3
JoFormer Wikipedia Language Modeling
Full English Wikipedia (∼983M byte-pair encoding (BPE) tokens, vocab=8K, block_size=512, 200K iters; details in Section E). Table 3: Wikipedia LM validation perplexity (PPL; 200K iters, vocab=8K, full wiki). JoFormerprojected leads in all configurations tested (single seed). Lower = better. Config
RoFormer
JoF-fixed
JoF-learned
JoF-projected
n100, L2 n200, L2 n200, L4 n250, L4 n500, L2 n500, L4
7.24 5.82 5.36 5.10 5.02 4.67
6.67 5.51 5.07 4.85 4.82 4.42
6.43 5.42 5.01 4.74 4.73 4.37
6.15 5.17 4.72 4.55 4.58 4.32
JoFormer-projected leads in all configurations, consistent with the value-path hierarchy (which does not prove an optimization ordering). 4.4
Length Generalization
163M params (D=768, 16 layers, 8 heads), OpenWebText, trained on block_size=512. Architecture: 5 windowed layers (window=32) + 1 full-attention NoPE layer. JoFormer-projected uses 200K total iterations (150K fixed-angle + 50K projected-angle fine-tuning); the RoPE baseline trains for 150K iterations. The additional 50K iterations are needed to convert fixed angles to content-dependent projected angles (details in Section E). Table 4: Length generalization from 512 training length (163M params, OWT, single seed). Ratio = PPL@4096 / PPL@512; lower ratio = better extrapolation. JoFormer-projected uses 200K total iterations (150K fixed + 50K projected fine-tuning); RoPE trains for 150K. Model JoFormer-projected JoFormer fixed RoPE
PPL@512
PPL@1024
PPL@2048
PPL@4096
Ratio
25.55 26.32 26.82
25.17 34.32 41.90
24.73 64.74 90.10
26.01 115.96 168.67
1.02× 4.41× 6.29×
JoFormer-projected extrapolates well (1.02× ratio vs. 6.29× for RoPE); JoFormer-fixed is intermediate (4.41×), suggesting data-dependent angles—not just V rotation—are consistent with improved length generalization. The staged recipe worked more reliably than training projected angles from scratch in our setup. 8
5
Related Work
RoPE [Su et al., 2021], 2D-RoPE [Heo et al., 2024], and ComRoPE [Yu et al., 2025] operate on Q/K only; our compositional framework extends to the value side via the journey operator, with commutativity as the iff condition for displacement-dependent value transport (Theorem 2.8). Shaw et al. [Shaw et al., 2018] inject relative information into values via learned additive embeddings; the proposed value operation instead uses the same orthogonal cocycle transport as scores, inheriting norm preservation and the target-frame interpretation. LieRE [Ostmeier et al., 2024] uses dense (curved) rotations—expressive, but composition is path-dependent. S4 [Gu et al., 2022] and Mamba [Gu and Dao, 2023] implement non-trivial value-path kernels; our hierarchy (Section F.2) provides common notation. ALiBi [Press et al., 2022] uses additive biases; toral scores span a d-dimensional trigonometric polynomial space (Section A.7)—a function-class separation, not a dominance claim.
6
Discussion and Conclusion
We started from one operation: composing data points that carry both content and axis-step generators. This operation defines journey operators along paths, and we showed that journey operators can be expressed in terms of absolute operators as A−1 k Aj . From this foundation, a chain of results follows. Path-independent composition requires commuting axis-step generators (Theorem 2.2). Under natural axioms, the only compatible mechanism is block-diagonal SO(2)d/2 rotation (Theorem 2.4), explaining why RoPE-like methods arise naturally. In the flat regime, value transport via the journey operator is displacement-dependent (Theorem 2.8), giving a concrete design rule: apply the journey to values, not just scores. Standard attention discards this compositional structure entirely. JoFormer brings it back by applying the journey operator to values before aggregation. The value-path formula (3) places standard attention, SSMs, and JoFormer on a common spectrum, clarifying what each composes and what it discards (Section F.2). For practitioners, V rotation is a zero-parameter change to RoPE: apply the same rotation to values and inverse-rotate the output. Our experiments across vision (∼2% on CIFAR-100, +0.40% on ImageNet), language modeling, and length generalization suggest the framework has practical value. All comparisons are single-seed sanity checks, not benchmark claims. Open directions include extending the toral classification to non-abelian frame groups, making the value-path hierarchy quantitative, and finding simpler length-generalization recipes than the staged JoFormer-projected training. Limitations. The toral classification assumes multiplicity-free representations; extending it to non-abelian frame groups remains open. The theory motivates projected angles and value-side rotation but does not prove that they help—they are theory-motivated, not theory-proved (Table 8). Full-layer equivariance requires additional assumptions beyond operator-level analysis (Theorem 2.9). All experimental comparisons are single-seed sanity checks at small-to-moderate scale; we do not claim state-of-the-art results, and the approach has not been validated on large-scale pretraining. Broader Impact. This work is primarily theoretical—it provides an algebraic framework for understanding and designing positional mechanisms in transformers. We do not foresee specific negative societal consequences beyond those common to general-purpose sequence and vision architectures.
References Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, 2024. 9
Sophie Ostmeier, Brian Coste, Mischa Jung, Justin Luitjens, Boris Bornstein, Luciano Del Corro, Dhruv Mahajan, Patrice Simard, Arindam Nori, and Nanyun Peng. LieRE: Generalizing rotary position encodings. In International Conference on Machine Learning, 2024. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length generalization. In International Conference on Learning Representations, 2022. Sebastian Rudolph and Eugenie Giesbrecht. Compositional matrix-space models of language. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 907–916, 2010. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 464–468, 2018. Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. In Advances in Neural Information Processing Systems, volume 32, 2019. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. RotatE: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019. Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge of the ViT. arXiv preprint arXiv:2204.07118, 2022. Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, and Chun Yuan. ComRoPE: Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
10
A
Full Proofs
A.1
Proof of Path Independence (Theorem 2.2)
Formal statement.
Let R1 , . . . , RD ∈ G be axis-step generators. The following are equivalent:
(i) T (π) depends only on the count vector (|π|1 , . . . , |π|D ) of π (path independence). (ii) Every elementary square commutes: T (ij) = T (ji) for all i ̸= j. (iii) Per-axis generators commute: Ri Rj = Rj Ri for all i ̸= j. When these hold, the relative operator mD −nD m1 −n1 · · · RD Pm→n = A−1 n Am = R1 depends only on the displacement m − n (source minus target)—the journey is fully determined by the composition structure. Proof. (iii)⇒(i): Any two words π, π ′ with the same count vector are related by a sequence of adjacent transpositions (this is the well-known fact that adjacent transpositions generate the symmetric group, applied to each pair of adjacent letters in the word). Consider an adjacent transposition swapping letters at positions p, p + 1: π = . . . kp kp+1 . . . becomes π ′ = . . . kp+1 kp . . .. Since T (π) = Rkm · · · Rkp+1 Rkp · · · Rk1 and the transposition only affects the product Rkp+1 Rkp , we need Rkp+1 Rkp = Rkp Rkp+1 . If kp = kp+1 this is trivial; otherwise it follows from (iii). Composing all adjacent transpositions: T (π) = T (π ′ ). (i)⇒(ii): Words ij and ji have the same count vector (1i , 1j ) (one step along each axis). By (i): T (ij) = T (ji). (ii)⇒(iii): By definition, T (ij) = Rj Ri (rightmost letter acts first) and T (ji) = Ri Rj . Equality T (ij) = T (ji) gives Rj Ri = Ri Rj , i.e., Ri Rj = Rj Ri . nD Path independence of relative operators: When (iii) holds, An = R1n1 · · · RD is well−1 defined (independent of multiplication order by commutativity). Then Pm→n = An Am = −nD mD mD −nD , depending only on m − n. Conversely, if RD · · · R1−n1 R1m1 · · · RD = R1m1 −n1 · · · RD Pm→n depends only on m − n, then in particular Pej →ei +ej = P0→ei . The LHS is (Ri Rj )−1 Rj = Rj−1 Ri−1 Rj ; the RHS is Ri−1 . Equality gives Rj−1 Ri−1 Rj = Ri−1 , hence Ri Rj = Rj Ri .
A.2
Toral Classification: Full Development
This subsection provides the full axioms, scope discussion, and supporting material for the toral classification summarized in Section 2.4. In this subsection, Bs,s′ denotes the pairwise score matrix (the journey Ps′ →s ); it is not the absolute operator As used elsewhere. Scope of the classification. The theorem does not derive the torus from first principles. It characterizes all bilinear, norm-preserving, cocycle-compatible attention mechanisms after selecting the maximal connected abelian (commutative) compact frame group T = SO(2)d/2 acting with one non-repeated 2D plane per factor. The axiom choice has a design rationale: bilinearity restricts to multiplicative PEs; toral frame invariance selects T (full O(d) collapses positional structure, Theorem 2.5); cocycle enforces compositionality; norm preservation prevents rescaling. The result is conditional: it determines what is compatible with this contract, not what is optimal. The novelty is not Schur’s lemma itself, but the identification of this symmetry contract as exactly the contract under which journey-based score/value transport reduces to RoPE-style block rotations. The four axioms below formalize a concrete question: if the only unobservable frame changes are independent rotations of each 2D coordinate plane (i.e., the frame group is the torus T = SO(2)d/2 ), what is the most general attention mechanism consistent with that symmetry? Axiom 1 (Bilinearity). For each position pair (s, s′ ), there exists Bs,s′ ∈ Rd×d such that Att(q, k, s, s′ ) = q ⊤ Bs,s′ k. Axiom 2 (Toral Frame Invariance). For all M ∈ T = SO(2)d/2 : Att(M q, M k, s, s′ ) = Att(q, k, s, s′ ). This states that rotating Q and K by the same toral element does not change scores. 11
Axiom 3 (Cocycle). Bs,s = Id (self-attention is unmodified) and Bs,s′ Bs′ ,s′′ = Bs,s′′ (composition). Equivalently: non-degeneracy (each Bs,s′ invertible) plus transitivity. Axiom 4 (Norm preservation). Each Bs,s′ preserves the Euclidean norm: ∥Bs,s′ x∥ = ∥x∥ for all x. This prevents positional encoding from rescaling attention scores. Lemma A.1 (Cocycle factorization). Under Axiom 3, fix a reference position s0 and define Rs = Bs,s0 . Then Bs,s′ = Rs Rs−1 ′ . −1 Proof. By the cocycle property: Bs,s0 = Bs,s′ Bs′ ,s0 , hence Bs,s′ = Bs,s0 Bs−1 ′ ,s = Rs Rs′ . 0
The derivation applies standard commutant and Schur machinery to multiplicity-free representations; the contribution is identifying toral frame invariance as the right contract, from which RoPE, 2DRoPE, and tree PEs emerge as special cases. Orientation convention. Here Bs,s′ = Ps′ →s = A−1 s As′ under the notation contract. The t factorization Bs,s′ = Rs Rs−1 uses gauge variables R = B ′ s s,s0 . With reference s0 = 0 and At = R : −1 −s Rs = Bs,0 = As A0 = R , so the gauge angle is θk (s) = −ωk s (negated relative to the absolute ′ angle). The formula θk (s) − θk (s′ ) = ωk (s′ − s) then matches Ps′ →s = Rs −s . If repeated isotypic components or non-abelian frame groups are allowed, the commutant—and therefore the classification—changes. Theorem 2.4 recovers RoPE [Su et al., 2021] (gauge angle θk (t) = −ωk t, giving score angle ωk (s′ −s)), 2D-RoPE [Heo et al., 2024] (axis-partitioned planes), simple depth-indexed tree encodings [Shiv and Quirk, 2019], and dot-product attention (θk ≡ 0) as special cases. A.3
Proof of the Toral Classification (Theorem 2.4)
Formal statement. Assume d is even. Let T = SO(2)d/2 act on Rd as the direct sum of the standard 2D representation of each factor, one factor per coordinate plane (each SO(2) factor acts on exactly one 2-plane; there are no repeated isotypic components). Under Axioms 1–4 with this frame group, the pairwise score matrix is: Bs,s′ = diag R θ1 (s) − θ1 (s′ ) , . . . , R θd/2 (s) − θd/2 (s′ ) , (12) ϕ − sin ϕ and θk : S → R assigns an angle per rotation plane per position. where R(ϕ) = cos sin ϕ cos ϕ Proof. Step 1 (Commutant structure): The commutant of T = SO(2)d/2 in Rd×d consists of blockdiagonal matrices with 2 × 2 blocks. Each block carries the standard 2D representation of the corresponding SO(2) factor, which is irreducible over R. By Schur’s lemma, the real commutant of an irreducible real representation is R, C, or H. For the 2D representation of SO(2), the commutant is spanR {I2 , J} ∼ = C where J = 01 −1 0 . Off-diagonal blocks between different SO(2) factors vanish because distinct irreducible representations have zero intertwining operators. Step 2 (Frame invariance ⇒ commutant): Axiom 2: M ⊤ Bs,s′ M = Bs,s′ for all M ∈ T . Since each M ∈ T is orthogonal (M ⊤ = M −1 ), this becomes M −1 Bs,s′ M = Bs,s′ , i.e., Bs,s′ M = M Bs,s′ for all M ∈ T . Thus Bs,s′ lies in the commutant Comm(T ) =p{X : XM = M X ∀ M ∈ T }. Each 2 × 2 block has the form ak I2 + bk J = ρk R(ψk ) with ρk = a2k + b2k . Step 3 (Norm preservation): Axiom 4: ∥Bs,s′ x∥ = ∥x∥. Applied to each 2 × 2 block: ρk = 1, giving pure rotation R(ψk ). Step 4 (Cocycle factorization): Axiom 3: Bs,s′ = Rs Rs−1 for some Rs ∈ T . Each block: ′ R(ψk (s, s′ )) = R(θk (s))R(−θk (s′ )) = R(θk (s) − θk (s′ )). A.4
O(d) Impossibility
Formal statement. Under full orthogonal symmetry H = O(d) with d ≥ 2, frame invariance forces Bs,s′ = αs,s′ Id . Attention collapses to position-dependent rescaling (by Schur’s lemma on the irreducible standard representation). 12
Proof of Theorem 2.5. The standard representation of O(d) on Rd is irreducible for d ≥ 2 (any invariant subspace is either {0} or Rd , since O(d) acts transitively on unit vectors). By Schur’s lemma, its real commutant is R · Id . Frame invariance forces Bs,s′ ∈ Comm(O(d)) = R · Id . A.5
Proof of V-Rotation Equivariance (Theorem 2.8)
Formal statement. Let positions lie in ZD with orthogonal axis-step generators R1 , . . . , RD and nD absolute operators (from composition) An = R1n1 · · · RD . Assume attention weights are functions only of relative displacement, αnm = α(m − n). Define value aggregation with transport: X α(m − n) A−1 cn = n Am v m . m
(a) If the generators commute, then A−1 n Am depends only on m − n (by Theorem 2.2), and the aggregation rule is translation equivariant in the sense of Theorem 2.7. −1 (b) If some generators do not commute, there exist m, n, u such that A−1 n+u Am+u ̸= An Am , so value transport is path-dependent (it depends on absolute positions, not only displacement). The transport is still a valid operation arising from composition; it simply does not reduce to a pure function of displacement. mD −nD Proof. (a) Commutativity gives A−1 = R1m1 −n1 · · · RD , depending only on m − n. For n Am P translation equivariance: F (Tu v)n+u = m α(m − n − u) A−1 Am vm−u . Substituting m′ = n+u P −1 ′ −1 m − u: = m′ α(m − n) An+u Am′ +u vm′ . By commutativity, A−1 n+u Am′ +u = An Am′ (both ′ depend only on m − n). Hence F (Tu v)n+u = F (v)n .
(b) Suppose Ra , Rb do not commute (relabel axes so a < b). Let n = eb , m = ea + eb , and −1 u = −eb . Under the canonical order An = Rb , Am = Ra Rb , so A−1 n Am = Rb Ra Rb . After −1 −1 −1 shifting: An+u Am+u = A0 Aea = Ra . These are equal iff Rb Ra Rb = Ra , i.e., Ra Rb = Rb Ra — contradicting the assumption. 0 Remark A.2 (Noncommuting counterexample). Take D = 2 with Ra = R(π/4), Rb = 10 −1 (a −1 reflection). The journey from (1, 1) to (0, 1) is A−1 A = R R R , a reflection-conjugated a b b (0,1) (1,1) rotation. After shifting by u = −eb : the journey from (1, 0) to (0, 0) is A−1 0 Aea = Ra , a plain rotation. These differ: the same displacement gives different value operators depending on absolute position. A.6
SSM Bridge (Theorem F.1)
Proof. Mamba’s recurrence ht = Axt ht−1 + Bxt xt unrolls by induction: hT = PT QT QT t=1 s=t+1 Axs Bxt xt . Define Pt→T = s=t+1 Axs and vt = Bxt xt . Then hT = PT t=1 Pt→T vt , which has the form of (3) with αT t = 1 (uniform, unnormalized weights) and content-dependent path operators. When Axt ∈ SO(2)d/2 , each prefix product is a block-diagonal rotation. The associative scan Pt→T = AT · AT −1 · · · At+1 can be computed in O(log N ) parallel depth using the standard parallel prefix algorithm. A.7
ALiBi Expressivity Separation
Proposition A.3 (Representational limitations of additive linear biases). Let d ≥ 4 be even. Compare: • ALiBi [Press et al., 2022]: AttALiBi (q, k, n) = q ⊤ k − m · n, where n = s − s′ is the signed relative position and m > 0 is a per-head slope. • Toral mechanism (Theorem 2.4): Atttoral (q, k, n) = q ⊤ Rθn k, where Rθ = diag(R(θ1 ), . . . , R(θd/2 )). Then: (a) Score dimensionality gap. For fixed q, k, the ALiBi score is affine in n: AttALiBi = q ⊤ k − mn. The toral score is a trigonometric polynomial: Atttoral (q, k, n) =
d/2 X b=1
13
αb cos(nθb ) + βb sin(nθb ),
where αb = q2b−1 k2b−1 + q2b k2b and βb = q2b−1 k2b − q2b k2b−1 . The ALiBi positional signal spans a 1-dimensional function space; the toral signal spans a d-dimensional space. (b) Strict separation. For any period p ≥ 3, the periodic attention pattern ϕ(n) = cos(2πn/p) is representable by the toral mechanism but not by any ALiBi head. (c) Approximate subsumption. On any bounded context |n| ≤ N , the toral mechanism can ϵ-approximate any ALiBi attention pattern for any ϵ > 0. (d) Value-path deficiency. ALiBi modifies only attention logits, leaving Pj→k = I for all j, k. The value P aggregation under ALiBi is order-blind: standard attention with ALiBi computes ck = j αkj vj with no positional modulation of values, regardless of the slope m. This separation does not imply that either mechanism is uniformly better for all tasks; it identifies positional functions representable by one class and not the other. Proof. (a) The b-th 2 × 2 rotation block of Rθn contributes αb cos(nθb ) + βb sin(nθb ) to q ⊤ Rθn k, with αb , βb as stated. For distinct θb , the d/2 cosine and d/2 sine terms are linearly independent functions of n, spanning a d-dimensional space. ALiBi contributes only the single linear function n 7→ −mn. (b) Set θ1 = 2π/p and q = k = e1 ∈ Rd . Then Atttoral (e1 , e1 , n) = cos(2πn/p), which equals ϕ(n) for all n. For ALiBi, q ⊤ k − mn = cos(2πn/p) for all n would require a bounded periodic function to equal an unbounded affine function—impossible. (c) The key identity is sin(nθ)/θ → n uniformly on |n| ≤ N as θ → 0, with error |n − sin(nθ)/θ| ≤ p |n|3 θ2 /6. Choose θ1 = 6ϵ/(mN 3 ) and q, k in block 1 so that β1 = −m/θ1 and α1 = 0. Then |β1 sin(nθ1 ) − (−mn)| = m|n − sin(nθ1 )/θ1 | ≤ mN 3 θ12 /6 = ϵ for all |n| ≤ N . The constant c0 = q ⊤ k is matched by the remaining blocks. (d) ALiBi adds a scalar bias to the attention logit; it does not transform value vectors. Hence Pj→k = I identically. A.8
Cocycle from Primitives
The cocycle axiom (Bs,s = I, Bs,s′ Bs′ ,s′′ = Bs,s′′ ) follows from two simpler primitives: (P1) Non-degeneracy: each Bs,s′ is invertible. (P2) Composability: Bs,s′′ = Bs,s′ · Bs′ ,s′′ . Proof: Set s′ = s in (P2): Bs,s′′ = Bs,s Bs,s′′ ; invertibility of Bs,s′′ gives Bs,s = I. Conversely, cocycle ⇒ composability is by definition, and Bs,s′ · Bs′ ,s = Bs,s = I gives invertibility with inverse −1 Bs,s ′ = Bs′ ,s .
B
Additional Theoretical Results
This section collects additional theoretical results that complement the core framework. B.1
DFT Sufficiency for White Sources
Proposition B.1 (DFT sufficiency for white sources). Fix K = d/2 distinct frequencies θ1 , . . . , θK ∈ [0, 2π) and let N → ∞. If the source is white (S(ω) = S0 for all ω), then any such set asymptotically achieves the same mutual information I(X; Y ) → d2 log(1 + N S0 /σ 2 ). In particular, equispaced DFT frequencies θk = 2πk/K achieve this bound. Proof. Since K is fixed and the frequencies are distinct, their pairwise separation δ > 0 is conPK stant. By the large-sieve inequality, ΦH θ Φθ /N → IK as N → ∞, so I(X; Y ) → k=1 log(1 + N S(θk )/σ 2 ). When S(ω) = S0 is constant, every term equals log(1 + N S0 /σ 2 ) regardless of the θk , giving I(X; Y ) → d2 log(1 + N S0 /σ 2 ). B.2
MSE Gain of Learned Frequencies
Proposition B.2 (MSE gain of learned frequencies). Fix K = d/2 and let N → ∞. For a source with non-uniform power spectral density (PSD) S(ω) whose K largest values occur at distinct ∗ frequencies θ1∗ , . . . , θK , the asymptotic MSE gap between optimal learned frequencies and fixed DFT 14
frequencies satisfies: MSEDFT − MSEopt ≥ 0
(N → ∞),
with equality if and only if the DFT frequencies already sample the K largest values of S(ω). Proof. With K fixed and frequencies distinct, the large-sieve gives ΦH θ Φθ /N → IK , so the MSE 2 PK S(θk ) for estimating X from Y = Φθ X + Z reduces to MSEθ = k=1 σ2σ+N S(θk ) . Since each term is a ∗ ∗ decreasing function of S(θk ), the MSE is minimized by choosing θ1 , . . . , θK at the K largest values of S(ω). The DFT places frequencies at θk = 2πk/K, which generically miss these peaks. The inequality follows, with equality exactly when the DFT grid coincides with the spectral maxima. Together, Theorems B.1 and B.2 explain the MNIST results (Section E.1): for white sources the DFT suffices, but for structured signals (images), learned frequencies select task-relevant spectral peaks that the DFT grid generically misses. B.3
Context-Sensitive Separation
Proposition B.3 (Context-sensitive separation). Let x = (x1 , . . . , xN ) and x′ be two sequences that differ only at position s, with xs ̸= x′s . Fix positions j < s < k. (a) Content-independent operators. If Pj→k = Rk−j , then the value-path transform applied to ′ vj at output position k is the same for both sequences: Pj→k = Pj→k . Intervening content is invisible to the value path. Qk−1 (b) Content-dependent operators. If Pj→k = t=j R̃xt where R̃xt = h(xt ) ∈ GL(d) is ′ a learned function of token content, and R̃xs ̸= R̃x′s , then Pj→k ̸= Pj→k . For any vj d outside a proper subspace of R (i.e., for generic inputs), the value-path outputs differ: ′ Pj→k vj ̸= Pj→k vj . Proof. Part (a) is immediate: Rk−j depends only on positions, not content. ′ For part (b), factor the shared portions: Pj→k = L R̃xs M and Pj→k = L R̃x′s M , where L = Qs−1 Qk−1 ′ R̃ and M = t=s+1 R̃xt are common invertible factors. Then Pj→k − Pj→k = L R̃xs − t=j xt R̃x′s M . Since L, M ∈ GL(d) and R̃xs ̸= R̃x′s , the difference is a nonzero linear map, so its kernel is a proper subspace of Rd .
This result motivates JoFormer-projected: content-dependent journey operators allow the value path to encode contextual information that fixed-angle operators cannot capture, providing a formal separation between the JoFormer-fixed and JoFormer-projected variants. B.4
Shift-Invariance and Order-Awareness
We show that the compositional embedding simultaneously achieves two desirable properties: invariance to uniform shifts (for translation-invariant tasks) and sensitivity to element ordering (for sequence-dependent tasks). Definition B.4 (m-Representations). For a sequence of length N with embeddings a1 , . . . , aN ∈ Rd , choose a window length m. For each window starting at position k, form the window embedding: sk =
m X
Ri−1 ak+i−1 ,
(13)
i=1
where R ∈ O(d) is a fixed block-diagonal rotation. Partition sk into K = d/2 blocks sk,1 , . . . , sk,K and define the magnitude vector: vk = ∥sk,1 ∥, . . . , ∥sk,K ∥ ∈ RK . (14) Under circular boundary conditions (indices modulo N ), the circular m-representation is v ◦ = PN k=1 vk . 15
Theorem B.5 (Shift-invariance). For orthonormal R ∈ O(d), the circular m-representation v ◦ is invariant under uniform circular shifts of the input along any axis. Proof. A uniform circular shift by one replaces at with at+1 mod N in each window. The shifted window at position k contains elements ak+1 , . . . , ak+m (mod N ), which is the original window at N position k + 1. So s′k = sk+1 mod N , and the Pmultiset of window embeddings {sk }k=1 is unchanged. ◦ Hence the sum of magnitude vectors v = k vk is invariant. Theorem B.6 (Order-awareness). Let R ∈ O(d) be a block-diagonal rotation matrix with K = d/2 blocks having angles θ1 , . . . , θK , at least one of which satisfies θℓ /π ∈ / Q. For elements a1 , . . . , am ∈ Rd P in general position, the magnitude vector v = (∥s(1) ∥, . . . , ∥s(K) ∥) of the window m embedding s = i=1 Ri−1 ai distinguishes different permutations: if σ ̸= id, then generically v(a1 , . . . , am ) ̸= v(aσ(1) , . . . , aσ(m) ). (ℓ)
Proof. In the ℓ-th block with angle θℓ , represent the block component of ai as zi Pm i−1 (ℓ) zi with ωℓ = eiθℓ . i=1 ωℓ
∈ C, so s(ℓ) =
For a fixed non-identity permutation σ, define the collision set Cσ = {(a1 , . . . , am ) ∈ Rdm : P (ℓ) ∥s(ℓ) ∥2 = ∥s′(ℓ) ∥2 for all ℓ}, where s′(ℓ) = i ωℓi−1 zσ(i) . Each equation is polynomial, so Cσ is an algebraic variety. It suffices to show Cσ ̸= Rdm . Choose block ℓ with θℓ /π ∈ / Q and set ω = ωℓ . The difference of squared magnitudes is a Hermitian form: |s|2 − |s′ |2 = z∗ H z,
Hij = ω i−j − ω σ
−1
(i)−σ −1 (j)
.
For σ ̸= id, there exist i, j with i − j ̸= σ −1 (i) − σ −1 (j) (otherwise σ −1 (i) = i + c for constant c, forcing c = 0, contradicting σ ̸= id). Since θℓ /π ∈ / Q, the element ω has infinite order, so Hij ̸= 0 and the Hermitian form is not identically zero. Hence Cσ is a proper algebraic subvariety of measure zero. The union over all m! − 1 non-identity permutations is still measure zero. B.5
Structured Concatenation
Definition B.7 (Axis-k concatenation). Given two compositional embeddings X = nD nD ) that share the same axis(a; R1n1 , . . . , Rknk , . . . , RD ) and Y = (b; R1n1 , . . . , Rkmk , . . . , RD step generators Ri and the same exponents on all axes i ̸= k, their concatenation along axis k is: nD X ⊕k Y = a + Rknk b; R1n1 , . . . , Rknk +mk , . . . , RD . (15) Theorem B.8 (Concatenation invertibility). Given X ⊕k Y , the axis dimensions (nk , mk ), and invertibility of Rk , either component is uniquely recoverable from the other: (a) Given a (the value of X): b = (Rknk )−1 (X ⊕k Y )value − a . (b) Given b (the value of Y ): a = (X ⊕k Y )value − Rknk b. Proof. From (15), the value component is c = a + Rknk b. Since Rk ∈ GL(d), the matrix Rknk is invertible, so given a: b = (Rknk )−1 (c − a), and given b: a = c − Rknk b.
C
Classical Transforms and Fixed-Operator Limits
PN −1 The monoidal embedding E = t=0 Rt vt directly recovers Fourier-style spectral features when R is a fixed block rotation. Fixed-sign Hadamard/Walsh constructions are useful analogies and limiting cases, but they do not satisfy the same connected toral SO(2)d/2 contract used in Theorem 2.4. 16
C.1
Discrete Fourier Transform
Proposition C.1 (DFT as compositional embedding). The discrete Fourier transform is a special case of the compositional embedding: 1. In 1D, composing n data points with a shared block-diagonal rotation R = n−1 ⊤ diag(R( 2πk n ))k=0 yields an embedding whose k-th block encodes (ℜ(Xk ), ℑ(Xk )) , where Xk is the k-th DFT coefficient. 2. In 2D, applying the 1D construction along rows then columns recovers the full 2D DFT. 3. Replacing fixed frequencies θk = 2πk/n with learned θk gives a generalized Fourier transform with task-optimal spectral components. Proof. One-dimensional case. Consider n input embeddings (vi , R) for i = 1, . . . , n, all sharing the same transformation R. Their composition yields (V, Rn ) where V =
n X
Ri−1 vi .
(16)
i=1
Let R be a real 2n × 2n block-diagonal matrix consisting of n rotation blocks. The k-th block (k = 0, . . . , n−1) is ! − sin 2πk cos 2πk n n Rk = , R = diag(R0 , R1 , . . . , Rn−1 ), sin 2πk cos 2πk n n so Rn = I2n (each Rkn is a rotation by 2πk). For input vectors, take vi = (ai , 0, ai , 0, . . . , ai , 0)⊤ ∈ R2n , repeating (ai , 0)⊤ across each block. Because R is block-diagonal, Ri−1 rotates the k-th block of vi by angle (i−1) · 2πk/n: 2πk(i − 1) ai cos n Ri−1 vi = . 2πk(i − 1) block k ai sin n Summing over i, the k-th block of V is Pn 2πk(i − 1) i=1 ai cos n V = k = 0, 1, . . . , n−1. (17) , P block k 2πk(i − 1) n i=1 ai sin n Pn j2π(i−1)k/n Defining the DFT coefficients Xk = i=1 ai e , we have V block k = (ℜ(Xk ), ℑ(Xk ))⊤ . The compositional embedding exactly encodes all n DFT coefficients. Two-dimensional case. Consider an n × n array ai,j . Apply the 1D construction along each row i, yielding output Wi whose k-th block is ! (row) n X ℜ Xi,k (row) Wi = , where X = ai,m ej2π(m−1)k/n . i,k (row) block k ℑ Xi,k m=1 Then apply the same construction along the column index i for each frequency k. The result for the p-th block is n n X n X X (row) j2π(i−1)p/n Yp,k = Xi,k e = ai,m exp j 2π (m−1)k + (i−1)p , n i=1
i=1 m=1
which is the (p, k) entry of the 2D DFT. The double sum is separable: first n independent 1D DFTs along the rows, then n along the columns. This is exactly what the compositional embedding with the same rotation R along both axes achieves. Learned frequencies. Setting θk = 2πk/N recovers the standard DFT. Learned θk select taskoptimal spectral components—a generalized Fourier transform where the basis frequencies are optimized for the downstream task rather than fixed to uniform spacing. 17
C.2
Discrete Cosine Transform
The discrete cosine transform (DCT) can be obtained similarly to the DFT by choosing a real-valued cosine basis. The 1D DCT-II of length n is: Ck =
n X
ai cos
π(i − 1)(k − 1) n
i=1
,
k = 1, . . . , n.
This is the real part of a DFT applied to an even extension of the sequence. In our framework, one sets the operator R to perform rotations with half-frequency increments (angles πk/n instead of 2πk/n) so that the embedding sum yields Ck on the output vector. Because these cosine rotations commute, the transform is separable across dimensions. In particular, an n × n 2D DCT is achieved by applying the 1D DCT along the rows and then along the columns (or vice versa), using the same commuting operators. C.3
Discrete Sine Transform
The discrete sine transform (DST) uses a sine basis instead of cosines. The DST-II is: Sk =
n X
πik ai sin , n+1 i=1
k = 1, . . . , n.
This corresponds to the imaginary part of the DFT for an odd extension of the data. In our embedding model, the DST is realized by choosing operators that introduce half-sample shifts and sign flips to generate sine terms. Since these sine-based rotations also commute, the multi-dimensional DST is performed by applying the 1D DST along each axis. C.4
Hadamard Transform
The Hadamard transform can be written as Hn x = n = 2m ).
Pn
i=1 R
i−1
vi for specific R and vi (assuming
Definition of R.
Let R = diag(1, −1, 1, −1, . . . , 1, −1), so R2 = I.
Definition of vi .
For input x = (x1 , . . . , xn )⊤ , define xi · (ei + ei+n/2 ) if 1 ≤ i ≤ n/2, vi = xi · (ei−n/2 − ei ) if n/2 < i ≤ n,
where ej is the j-th standard basis vector. 1 Base case (n = 2). H2 = 11 −1 ,R = v1 + Rv2 = (x1 + x2 , x1 − x2 )⊤ = H2 x.
1 0 0 −1
, v1 = x1 (1, 1)⊤ , v2 = x2 (1, −1)⊤ . Then
Inductive step. Assume the claim holds for n = 2k . For n′ = 2n, write x = (x(1)⊤ , x(2)⊤ )⊤ . By Sylvester’s construction, H2n x = (Hn (x(1) + x(2) ), Hn (x(1) − x(2) ))⊤ . Both terms can be written P as i Rni−1 vi by the inductive assumption, completing the induction. C.5
Walsh Transform
The Walsh transform Wn = P Hn is obtained by reordering the Hadamard matrix rows so that sign changes (sequency) increase monotonically, where P is a Gray-code permutation matrix. Given the Hadamard embedding {R, vi }, define R′ = P RP −1 and vi′ = P vi . Then X X i−1 R′ vi′ = P Ri−1 vi = P Hn (P −1 x) = Wn x. i
i
18
C.6
Learnable Generalization
All classical transforms above use fixed operators; the compositional framework generalizes them P ′ by learning R (or equivalently the angles θk ) from data. The magnitude ∥E∥2 = t,t′ vt⊤ Rt −t vt′ depends only on pairwise relative positions (shift-invariant), yet permuting the sequence changes ∥E∥2 (order-sensitive). This explains why learned monoidal embeddings outperform DFT at low dimension (Section E.1): they select exactly the frequencies needed for the task, rather than committing to uniform spacing.
D
Application Sketches
The multi-axis compositional framework extends naturally beyond sequences and images. We sketch four application domains. D.1
Audio and Spectrograms
Audio spectrograms are naturally arranged on a time–frequency grid. Assign one composition operator ◦t for the temporal axis and another ◦f for the frequency axis, giving each spectrogram cell a compositional embedding: X f t Eaudio = Rtime Rfreq vt,f . t,f ′
′
t −t f −f When Rtime and Rfreq commute, the journey operator P(t′ ,f ′ )→(t,f ) = Rtime Rfreq depends only on the time–frequency displacement, enabling models to incorporate both temporal and spectral locality in a principled way. The same construction applies to EEG or other time–frequency representations.
D.2
Video
A video is an H ×W ×T grid (two spatial dimensions plus time). Introducing a temporal composition operator ◦time in addition to the spatial operators gives a 3-axis embedding: X j i t Evideo = Rrow Rcol Rtime vi,j,t . i,j,t
By Theorem 2.2, when the three generators commute, composing spatially within each frame and then advancing in time yields the same result as first advancing each pixel in time and then composing spatially. This coherence ensures consistent representations of motion regardless of composition order. D.3
Multimodal Fusion
For multimodal data (e.g., text + image + audio), assign one composition axis per modality with generator Rmod , plus intra-modality axes for sequence or spatial structure. The cross-modal journey −1 Ptext→image = Rmod · A−1 text Aimage
provides a principled alignment operator between modalities, with commutativity ensuring that fusing text-then-audio gives the same result as audio-then-text. D.4
Multi-Axis Alignment
Given two compositional embeddings X and Y , the optimal alignment along axis k can be found via inner product maximization: s∗ = arg max ⟨X, Rks Y ⟩. (18) s
This is a cross-correlation computed in the rotated embedding space. For D-dimensional alignment, sD apply shifts s1 , . . . , sD on each axis: Y ′ = R1s1 · · · RD Y . Because the Ri commute, the order of shifts does not matter. This enables operations like sequence alignment or embedding fusion by sliding one structure relative to another. 19
E
Extended Experimental Details
E.1
MNIST Monoidal Embedding
Setup. The monoidal embedding demonstrates the compositional framework on image classification. Each pixel pij of a 28×28 image is represented as a data point with content pij ·e (pixel intensity times a d-dimensional basis vector e) and two axis-step generators: Rx for the horizontal axis and Ry for the x vertical axis. Both axis-step generators are block-diagonal rotations Rx = diag(R(θ1x ), . . . , R(θd/2 )), y y Ry = diag(R(θ1 ), . . . , R(θd/2 )), sharing the same block structure so that Rx Ry = Ry Rx (flat regime). The full image embedding is the 2D composition over all pixels: Eimage =
27 X 27 X
pij Ryi Rxj e.
i=0 j=0
Each pixel’s content is rotated by the product of its row and column axis-step generators, and the results are summed into a single d-dimensional vector. Because the axis-step generators commute, this composition is path-independent: composing rows-then-columns gives the same embedding as columns-then-rows. P y x In each 2D block (identified with C), the k-th component reduces to E (k) = i,j pij ei(jθk +iθk ) —a 2D Fourier-like transform with learned frequencies (θkx , θky ). With fixed frequencies θkx = 2πk/d, θky = 2πk/d, this recovers the standard 2D DFT; with learned frequencies, the model selects task-optimal spectral components. The d-dimensional embedding Eimage is fed into an MLP classifier (128 hidden units, ReLU, 10class softmax) trained end-to-end with cross-entropy loss. Training: Adam lr= 10−3 , 50 epochs, batch=128, no augmentation. The only learnable parameters in the embedding itself are the d angles d/2 (θkx , θky )k=1 ; all representational capacity comes from the compositional structure. Table 5: MNIST accuracy by embedding method and dimension (single seed). Monoidal embeddings learn task-optimal spectral components; DFT uses fixed Fourier frequencies. Method
d=2
d=8
d = 32
d = 784
Monoidal (learned) DFT (fixed)
55.2% 21.0%
86.4% 75.3%
96.5% 95.5%
97.5% —
MLP baseline
97.2% (128 hidden, d = 784)
Results. The advantage is most dramatic at low dimension: at d = 2, monoidal achieves 55.2% (well above chance 10%) while DFT achieves only 21.0%. At d = 32, monoidal reaches 96.5%— within 0.7% of the full MLP baseline (97.2%) despite using 24× fewer input dimensions. This is consistent with the core insight: learned angles select task-relevant spectral components that fixed Fourier bases cannot match. The advantage diminishes at high d (where frequency allocation is less constrained), suggesting that the monoidal structure’s benefit is in efficient spectral selection. E.2
CIFAR-100 LieRE Experiment
Architecture: ViT-Tiny with D=384, 12 layers, 6 heads, head_dim=64, ∼14.9M params. Patch size: 4 × 4 → 8 × 8 = 64 patches + CLS token. Training: Adam lr= 10−4 , cosine annealing 200 epochs, bf16-mixed precision, seed=42, NVIDIA H100 PCIe. Augmentation: RandomCrop(32, padding=4), RandomHorizontalFlip, Normalize([0.5071, 0.4867, 0.4408], [0.2675, 0.2565, 0.2761]). Extended results at 400 epochs: Method
Q/K only
Q/K/V
V effect
Axial-dense LieRE64
70.61% 70.64%
71.12% 69.41%
+0.51% −1.23%
20
Setup B full reproducibility. Optimizer: AdamW (β1 =0.9, β2 =0.999, ε=10−8 , weight decay=0.1). Batch size: 128. MLP ratio: 4× (hidden dim = 4D). Dropout: 0.1; no droppath or stochastic depth. Schedule: cosine annealing from lr=10−3 to 0 over 300 epochs, no warmup. Mixup: α=0.8; label smoothing 0. Augmentation: random crop (pad 4, reflect), random horizontal flip (50%), RandAugment (n=2, m=9), random erasing (50% probability, 16×16 zeroed patch). Normalization: mean = [0.4914, 0.4822, 0.4465], std = [0.2470, 0.2435, 0.2616]. Seed: 42; torch.manual_seed + cuda.manual_seed_all. Deterministic: entire dataset GPUresident; batches via torch.randint (no DataLoader workers). Train/test: standard CIFAR-100 split (50K/10K), no validation hold-out. E.3
CIFAR-100 Scaling Results
Table 6: CIFAR-100 accuracy scaling: joformer_axial (learned axial + V rotation) vs. rope2d (fixed axial, Q/K only) and all 9 variants. Setup B, 300 epochs cosine, single seed. Model
D=32
D=64
D=128
D=256
joformer_axial joformer (combined + V) joformer_old (fixed + V) monoidal_axial (learned, no V) rope2d (fixed, no V) learned (additive PE)
52.77 52.43 50.54 51.75 50.91 52.23
61.33 58.83 59.13 59.10 59.23 57.26
66.67 66.19 66.10 64.81 64.22 60.74
63.27 62.10 61.85 60.92 61.39 55.49
In these paired single-seed CIFAR protocols, the joformer_axial advantage over rope2d is consistent at ∼2% across all scales: +1.86% (D=32), +2.10% (D=64), +2.45% (D=128), +1.88% (D=256). At D=128, the top three models are all V-rotation variants (66.67, 66.19, 66.10), separated from non-V models (64.81, 64.22) by a clear ∼2% gap. In this single-seed grid, joformer_axial leads at every checkpoint from epoch 50 onward—not a late-stage effect. Key finding: V rotation × learnable frequencies is synergistic. Neither V rotation alone (joformer_old vs. rope2d: +0.46% at D=256) nor learnable frequencies alone (monoidal_axial vs. rope2d: −0.47%) provides the full benefit. Together (joformer_axial vs. rope2d: +1.88%), they interact synergistically. The mechanism: V rotation gives the model a richer position-dependent value transformation; learnable frequencies adapt to leverage it. Per-layer frequencies don’t help. Shared frequencies across layers outperform per-layer variants (e.g., joformer_axial 63.27% vs. per-layer 62.76% at D=256). D=256 non-monotonicity. The D=256 models are not uniformly better than D=128 under this fixed training recipe, so the table should be read as a paired PE comparison at each width rather than as an optimized scaling law. Consistent positional encoding helps the residual stream maintain coherent position information across layers. E.4
ImageNet ViT-S
Model: ViT-S (D=384, 12 layers, 6 heads, patch=16, img=224). Recipe: DeiT-III (AdamW lr= 10−3 , wd=0.05, cosine schedule + 5-epoch warmup, 300 epochs, batch=1024, RandAugment(9, 0.5), Mixup 0.8, CutMix 1.0, Random Erasing 0.25, Label Smoothing 0.1, Stochastic Depth 0.1, AMP fp16). Hardware: 2× NVIDIA H100 PCIe. JoFormer: K ⊤ Q ordering, V rotation = Rj vj followed by output inverse Ri−1 ci . Training time: ∼72 hours for RoPE2D, ∼94 hours for JoFormer in this ViT-S setup. E.5
Wikipedia Language Modeling
Data: Full English Wikipedia dump (28.8M lines, ∼983M BPE tokens with vocab=8000). Architecture: standard transformer with rotary attention, FFN ratio 4×, pre-norm (LayerNorm before attention and FFN). Training: AdamW (lr= 2 × 10−4 , β1 =0.9, β2 =0.95, wd=0.01), cosine schedule 21
to lrmin = 2 × 10−5 , batch=32, block_size=512, gradient accumulation=1, AMP bf16. JoFormerprojected MLP: input LayerNorm → Linear(d, d) → GELU → Linear(d, d/2) (angles). Evaluation: validation PPL on held-out 1% of data. E.6
Length Generalization
Architecture: 5 windowed layers (window=32) + 1 full-attention NoPE layer. The windowed layers use either RoPE or JoFormer variants; the NoPE layer has no positional encoding. JoFormer-projected recipe (staged; in our setup this worked more reliably than training projected angles from scratch): Stage 1: JoFormer-fixed (RoPE angles + V rotation), lr= 5 × 10−4 , 100K iters. Stage 2: Continue fixed-angle training at lr= 2 × 10−4 , 50K iters. Stage 3: Fine-tune to projected angles (zero-initialize angle projector weights so initial angles match fixed), lr= 5 × 10−5 , 50K iters. The first two stages (150K total) match the RoPE baseline iteration count; stage 3 is a fine-tuning phase that converts fixed angles to content-dependent projected angles. Evaluation: 200 iterations per length, fixed seed, lengths {512, 1024, 2048, 4096, 8192}. E.7
Parameter and Wall-Clock Overhead
Table 7: JoFormer parameter overhead relative to RoPE2D baseline. PE params are additional learnable parameters in the positional encoding; V rotation adds no new parameters. ImageNet ViT-S training times on 2× H100 PCIe, 300 epochs (unoptimized rotation ops). Variant RoPE2D (baseline) JoFormer-fixed (V rot.) JoFormer-learned (axial) JoFormer-projected
Extra PE params
Wall-clock (ImageNet)
0 0
∼72h ∼94h ∼94h —
2nheads × dhead /2 per layer MLP: d→d→d/2 per layer
The ImageNet ViT-S training overhead reflects unoptimized rotation ops in a standard ViT pipeline. JoFormer-projected was tested only on Wikipedia LM (not ImageNet); its MLP projector adds O(d2 ) parameters per layer.
F
Architecture Variants and Value-Path Hierarchy
F.1
Variant-Scope Table
Table 8: Which theorem supports which architecture variant. Only fixed and learned linear-position variants inherit the relative-displacement theorem; projected angles are a separate, theory-motivated architecture. Variant
What is proved
What is not proved
JoFormer-fixed
Covered by Theorem 2.8 when weights are relative-displacement functions. Value operator depends only on displacement (arises from composition). Same as fixed if angles remain linear in position with learned frequencies. Still uses commuting block-diagonal rotations and norm-preserving value transport.
Does not prove a performance gain.
JoFormer-learned JoFormerprojected
22
Frequencies are learned, but the theorem does not choose optimal frequencies. Not a translation-equivariant relativedisplacement PE, because angles depend on content/residual state. It is an architecture motivated by the value-path view, not a direct corollary of Theorem 2.8.
F.2
Value-Path Hierarchy and SSM Bridge
The journey operator on the value side (8) gives the attention value-path formula: X ck = αkj Pj→k vj .
(3)
j
For SSMs, the value-path kernel is oriented in the forward recurrence direction. This kernel is not necessarily the same object as the attention journey Pj→k = A−1 k Aj unless one adapts the address convention (e.g., At = R−t ). We therefore write SSM forward kernels as Kj→k in this subsection to distinguish them from the attention journey P . The value-path formula reveals a hierarchy: (a) Standard Transformer: Pj→k = I for all j, k. Values are aggregated without positional modulation—the model is order-blind on the value side. (b) Linear SSM (S4 [Gu et al., 2022]): Kj→k = Rk−j . Values carry position-dependent phase shifts, enabling the recurrence ht = Rht−1 + vt . Qk (c) Selective SSM (Mamba [Gu and Dao, 2023]): Kj→k = t=j+1 Axt . Each transition depends on input content. (d) JoFormer (attention journey): Pj→k = A−1 k Aj with full attention weights αkj . Combines content-dependent value paths with selective attention. Proposition F.1 (SSM recurrences in value-path notation). Mamba’s selective recurrence ht = PT QT Axt ht−1 + Bxt xt unrolls to: hT = t=1 s=t+1 Axs Bxt xt . Defining the forward kernel QT P Kt→T = s=t+1 Axs and vt = Bxt xt , this gives the value-path formula hT = t Kt→T vt with uniform attention weights. When Axt ∈ SO(2)d/2 , the prefix products are block-diagonal rotations computable via parallel scan in O(log N ) depth. Note: Mamba’s actual Axt matrices are not generally in SO(2)d/2 , and attention weights are normalized (unlike the uniform weights here). The forward kernel Kt→T and the attention journey −t Pt→T = A−1 T At coincide when At = R ; otherwise they differ by a sign convention. This embedding highlights structural similarities but is not a full equivalence. JoFormer uses the attention journey Pj→i = A−1 i Aj on values, combining full attention selectivity (the score side) with content-dependent value paths (the value side). It does not implement the full product over intermediate states, but rather a single content-dependent rotation per layer.
NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract states five contributions (interchange prediction, toral classification, JoFormer architecture, value-path hierarchy, experiments) all substantiated with theorems and experimental results in the paper body. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 6 explicitly lists four limitations: single-seed experiments, three-stage training recipe complexity, stability requirements, and speed penalty. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 23
Justification: All axioms are explicitly stated (Axioms 1–4). Main theorems have proof sketches in the body and complete proofs in Appendix A. The norm-preservation axiom is stated before the toral classification theorem. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results? Answer: [Yes] Justification: Appendix E provides complete hyperparameters, architectures, training recipes, hardware, seeds, and stability requirements for all experiments. 5. Open access to data and code Question: Does the paper provide open access to the data and code? Answer: [No] Justification: Code will be released upon acceptance. All experiments use public datasets (CIFAR-100, ImageNet-1K, OpenWebText, MNIST, English Wikipedia). 6. Experimental setting/details Question: Does the paper specify all the training and test details necessary to understand the results? Answer: [Yes] Justification: Full details in Appendix E: optimizer, learning rate, schedule, batch size, augmentation, architecture, epochs/iterations, hardware, seed, and critical stability parameters. 7. Experiment statistical significance Question: Does the paper report error bars or other appropriate statistical significance information? Answer: [No] Justification: Most experiments use a single seed due to computational constraints. We acknowledge this explicitly in Section 6 and frame experiments as controlled tests of algebraic predictions rather than definitive performance comparisons. CIFAR-100 uses deterministic training ensuring exact reproducibility. 8. Experiments compute resources Question: Does the paper provide sufficient information on compute resources? Answer: [Yes] Justification: Hardware specified in Appendix E: NVIDIA H100 PCIe for CIFAR-100 and ImageNet, GPU details for language modeling experiments. 9. Code of ethics Question: Does the research conform with the NeurIPS Code of Ethics? Answer: [Yes] Justification: This is foundational research on positional encoding theory with no direct negative societal applications. 10. Broader impacts Question: Does the paper discuss potential societal impacts? Answer: [N/A] Justification: This paper develops algebraic theory for attention mechanisms—foundational research with no direct path to negative societal impacts. 11. Safeguards Question: Does the paper describe safeguards for responsible release? Answer: [N/A] Justification: No models or datasets with misuse potential are released. 24
12. Licenses for existing assets Question: Are existing assets properly credited with licenses respected? Answer: [Yes] Justification: CIFAR-100, ImageNet-1K, OpenWebText, MNIST, and Wikipedia are properly cited. LieRE is credited. 13. New assets Question: Are new assets well documented? Answer: [N/A] Justification: No new datasets or models are released in this submission. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments, does the paper include instructions and compensation details? Answer: [N/A] Justification: No human subjects or crowdsourcing involved. 15. Institutional review board (IRB) approvals Question: Does the paper describe potential risks and IRB approvals? Answer: [N/A] Justification: No human subjects research. 16. Declaration of LLM usage Question: Does the paper describe LLM usage if it is a core method component? Answer: [N/A] Justification: LLMs were not used as a component of the research methodology.
25