The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups Przemyslaw Musialski∗ June 19, 2026
arXiv:2606.20547v1 [cs.LG] 18 Jun 2026
Abstract We place the attention token on the group: a token is an element gi of a matrix Lie group G — a bare transformation, with no feature payload and no external action ρ(g) carrying it. To our knowledge this is the first attention construction whose tokens are bare matrix Lie group elements. Their score is the closedform algebra norm of the relative pose rather than a learned kernel, and it reaches the affine full-frame groups that every irrep- or surjective-exp-based method must exclude. We call it Lie-Algebra Attention. Once tokens are group elements, the rest follows with none of the usual representation-theoretic machinery. The relative geometry of a pair is canonical, gi−1 gj , so the pairwise invariant wij = log(gi−1 gj ) is intrinsic rather than designed. Equivariance under the diagonal G-action is then tautological rather than enforced, and the cocycle condition for relative poses holds automatically. The attention score is the negative squared algebra norm of the relative pose, sij = −∥ log(gi−1 gj )∥2λ /τ . This is the canonical proximity kernel on wij under a blockweighted Frobenius inner product. It uses no irreducible representations, no spherical harmonics, no Clebsch– Gordan products, and no learned kernel. The construction applies to any matrix Lie group with a faithful finitedimensional representation. We give closed-form instantiations for SO(2), SE(2), SO(3), SE(3), Aff(2), and the spatial affine extension Aff(3). The affine cases form the non-compact non-abelian regime with scale and shear that no vector-token attention method reaches: neither the irrep tradition, which has no non-trivial finitedimensional unitary irreps, nor the surjective-exp tradition (LieConv-style lifting). Three sequence-completion experiments, on SE(2), SO(3), and Aff(2), bear this out. The closed-form score matches a learned MLP kernel on the same invariant and outperforms it on SE(2), using 50 to 80× fewer score parameters; the learned kernel does not improve on it. A vector-token baseline, by contrast, breaks invariance by five to twelve orders of magnitude. Prior group-element attention (LieTransformer) attaches a feature vector to each token and learns the kernel; the inversion here is to keep the token a bare group element and read the score off in closed form.
1
Introduction
This paper changes what an attention token is. In equivariant models for spatial reasoning (point clouds, molecules, robotic poses, 2D layouts, protein structure), a token is almost always a feature vector v ∈ V , and the symmetry group G acts on it externally through a representation ρ(g) : V → V . Equivariance is then something to enforce, through machinery built for that purpose: irreducible representations and Clebsch–Gordan tensor products (Tensor Field Networks [1], the SE(3)-Transformer [2], Equiformer [3]), steerable kernels [7], multivector sandwich products (the Geometric Algebra Transformer [9]), or auxiliary frames (AlphaFold’s Invariant Point Attention [14]). In every such case the token carries a transformation action; it is not itself a transformation. We make the token a transformation instead: it is an element gi ∈ G of a matrix Lie group — the group element is the token, carrying nothing else. Spatial reasoning then operates on transformations directly, rather than on vector embeddings that carry transformation actions, collapsing the standard distinction between the data and the symmetry acting on it. The consequences are immediate and structural. The relative geometry of two tokens is canonical, gi−1 gj , and the pairwise invariant wij = log(gi−1 gj ) ∈ g is intrinsic, not designed. Equivariance of the output is tautological rather than trained. Consistency of all pairwise relative poses, the cocycle condition, is automatic. None of the representation-theoretic machinery above is required. With tokens on the group, the attention score is read off in closed form. The natural diagonal-invariant scalar of a token pair is a function of wij = log(gi−1 gj ); because wij is itself invariant, any fixed bilinear form on g applied to it is invariant too, and the block-weighted squared norm sij = −∥wij ∥2λ /τ is the canonical proximity ∗ New Jersey Institute of Technology; [email protected]
1
choice among them — a closed-form kernel with no irreducible representations, no spherical harmonics, no Clebsch–Gordan products, and no learned kernel network. Equivariance and the cocycle follow as theorems with one-line proofs (Section 2). The simplicity is not an implementation shortcut; it is what the representationtheoretic framing of equivariance has obscured. We call the resulting construction Lie-Algebra Attention. We make three contributions, one per pillar of the construction: • The bare group-element token. The token is an element of a matrix Lie group — a transformation with no feature payload and no external action ρ(g) — rather than a vector the group acts on. Equivariance, the intrinsic pairwise invariant wij = log(gi−1 gj ), and cocycle consistency follow from the group structure with one-line proofs (Section 2); a vector-token control that reintroduces a payload breaks equivariance by five to twelve orders of magnitude (Section 5). • The closed-form canonical score. The score is the negative squared algebra norm of the relative pose, sij = −∥wij ∥2λ /τ — the canonical proximity kernel on the invariant, read off in closed form with no irreducible representations, Clebsch–Gordan products, steerable kernels, or a learned kernel network. A learned MLP kernel on the same invariant, carrying 50 to 80× more score parameters, does not improve on it (Section 5). • The affine full-frame regime. Closed-form instantiations for six groups — SO(2), SE(2), SO(3), SE(3), Aff(2), Aff(3) — including the non-compact non-abelian affine cases with scale and shear. These lie beyond the irrep tradition (no non-trivial finite-dimensional unitary irreps) and the surjective-exp tradition (LieConv); Aff(2) is validated empirically (Section 5.3) and Aff(3) is given in closed form as the spatial affine analogue, with empirical validation deferred to follow-up work (Section 3). Every prior tradition keeps the token a vector and the group action external, and the limitation each inherits is a symptom of that shared ontology: the irrep tradition is restricted by unitarity to compact groups; geometric algebra keeps tokens in flat ambient space, off the group manifold; capsule networks [12] use unconstrained pose matrices with no Lie-algebra structure; AlphaFold IPA gets the squared-distance kernel right for the position part but handles rotation separately via frame transformation; LieTransformer [16] attends over (g, v) pairs and learns the kernel, and the LieConv [17] lifting it builds on requires a surjective exponential, excluding Aff(n); RoPE [20] uses SO(2) rotations only for positional encoding. Placing the token strictly on the group is what removes the restriction in each case. A detailed comparison is in Section 6. The paper is organized as follows. Section 2 develops the construction for any matrix Lie group, using Aff(2) as the running example. Section 3 specializes it to the five remaining groups SO(2), SE(2), SO(3), SE(3), and the spatial affine Aff(3) (Aff(2) being the running example). Section 4 gives the transformer architecture in concrete terms. Section 5 validates it on SE(2), SO(3), and Aff(2). Section 6 positions the construction against prior work. Section 7 discusses limitations; Section 8 concludes.
2
The Construction
This section develops the construction at the level of any matrix Lie group G with a faithful finite-dimensional representation, using Aff(2) as the running concrete example. The premise to keep in front of every step is the one from Section 1: the token is an element of G, not a vector that G acts on. Three properties of any matrix Lie group are all we use: (i) closed-form exponential and logarithm on a principal chart U ⊂ G, (ii) the cancellation identity (agi )−1 (agj ) = gi−1 gj , and (iii) a block decomposition of the Lie algebra under the natural orthogonal action of O(n). From these the invariant, the score, equivariance, and the cocycle all follow as consequences. No representation-theoretic machinery enters anywhere.
2.1
Setting: tokens as group elements
A token in this construction is an element g ∈ G of a matrix Lie group — a transformation in its own right, not a vector in a space on which G acts. The standard equivariant-ML setup has a token v ∈ V and a representation ρ(g) : V → V acting on it; we discard V and ρ entirely. For the running example G = Aff(2) — the group of invertible affine maps of the plane — a token is a framed point: an origin together with two basis vectors. As a homogeneous matrix, A t Aff(2) = : A ∈ GL(2, R), t ∈ R2 , 0 1 where A encodes rotation, scale, and shear of the frame, t encodes the origin, and the group has six degrees of freedom. Group composition and inversion are closed-form: (A1 , t1 ) · (A2 , t2 ) = (A1 A2 , A1 t2 + t1 ),
2
(A, t)−1 = (A−1 , −A−1 t).
For general matrix G, multiplication is matrix multiplication of the representation, and inversion is matrix inversion. No further structure is required.
2.2
Block decomposition of the algebra
The Lie algebra g of G is the tangent space at the identity, and the bracket is the matrix commutator [X, Y ] = XY − Y X. For G = Aff(2), X v aff(2) = : X ∈ gl(2, R), v ∈ R2 ∼ = R6 , 0 0 with bracket [(X1 , v1 ), (X2 , v2 )] = ([X1 , X2 ], X1 v2 − X2 v1 ). The non-trivial bracket reflects the non-abelian interaction between linear part and translation. The algebra decomposes into geometrically distinct subspaces under the natural orthogonal action on the matrix representation. For aff(2) the decomposition has four irreducible blocks: (1)
R ⊕ Sym0 (2) , aff(2) = |{z} R2 ⊕ so(2) ⊕ |{z} | {z } | {z } translation
rotation
iso. scale
aniso. + shear
with dimensions 2 + 1 + 1 + 2 = 6. Each block carries a distinct geometric meaning and is invariant under the relevant orthogonal action. For other groups the decomposition is simpler: so(3) is one block, se(n) splits into translation + rotation. The full list is in Table 1. Orthonormal basis. Each block has a natural √ matrix representative. The unnormalized generators of the linear-part blocks each have Frobenius √ norm 2; to obtain an orthonormal basis under the Frobenius inner product (Section 2.4), we divide each by 2. For Aff(2), the translation generators 0 0 0 0 0 1 Ty = 0 0 1 Tx = 0 0 0 , 0 0 0 0 0 0 √ are already orthonormal, and the linear-part generators, normalized by 1/ 2, are 1 0 −1 1 1 0 1 1 0 1 0 R= √ , S=√ , D1 = √ , D2 = √ 0 2 1 2 0 1 2 0 −1 2 1
1 . 0
All six basis elements satisfy tr(Bi⊤ Bj ) = δij . Every element of aff(2) is written W = tx Tx + ty Ty + θR + sS + q1 D1 + q2 D2 , with coordinates (tx , ty , θ, s, q1 , q2 ) ∈ R6 . √ rotation angle (so that the The orthonormal coordinates absorb the 2 normalization. If φ is the physical √ unnormalized rotation block is φ times √ the antisymmetric matrix in R), then θ = 2 φ. Similarly, if σ is the physical isotropic-scale factor, then s = 2 σ. The translation coordinates tx , ty are unchanged.
2.3
Exponential and logarithm
The exponential map sends algebra elements to group elements. For algebras with a block-triangular structure v (X, v) 7→ ( X 0 0 ) — which covers Aff(2), SE(2), SE(3) — the exponential has the block closed form X
exp(X, v) = e , V (X) v ,
Z 1 V (X) =
erX dr =
0
X k≥0
Xk . (k + 1)!
When X is invertible, V (X) = X −1 (eX − I). For Aff(2), X ∈ gl(2, R) is a 2 × 2 matrix and Cayley–Hamilton gives X 2 − (trX) X + (det X) I = 0, so every higher power X k with k ≥ 2 reduces to a linear combination of I and X. Summing the exponential series collapses to eX = α(X) I + β(X) X, 3
where α, β are scalar functions of trX and det X. The same block-triangular form gives closed-form V (ω) via the Rodrigues route for SE(2) and SE(3) (see Section 3). The principal logarithm inverts exp on a chart U ⊂ G — an open neighborhood of the identity on which log is single-valued. For Aff(2), UAff(2) = (A, t) : A has no eigenvalue on R≤0 , and on this chart
log(A, t) = log A, V (log A)−1 t .
The maps exp and log are smooth on their domains and are each other’s inverses on U. They provide the bridge between G, where tokens live, and g, where standard linear operations apply. For the compact groups SO(2), SO(3), the chart is θ ∈ (−π, π) for the rotation angle. For SE(n) the chart inherits from the rotation part. The full per-group specification is in Section 3.
2.4
Block-weighted inner product
The algebra g sits inside gl(m, R) for some m via the matrix representation. The Frobenius inner product on gl(m, R) restricts to a positive-definite inner product on g: ⟨W1 , W2 ⟩ = tr(W1⊤ W2 ). In the orthonormal basis of Section 2.2, for Aff(2) this takes the diagonal form ∥W ∥2 = t2x + t2y + θ2 + s2 + q12 + q22 . Weighted form. Respecting the block structure and the isotropy within each block, the most general O(n)invariant block-diagonal positive-definite inner product is ∥W ∥2λ = λt (t2x + t2y ) + λθ θ2 + λs s2 + λq (q12 + q22 ),
(2)
with positive weights λ = (λt , λθ , λs , λq ) ∈ R4>0 for Aff(2). The Frobenius form is λ = (1, 1, 1, 1). For other groups the weight count drops: two weights for SE(n), one weight for SO(n). The full count per group is in Table 1. Remark 1 (No canonical Ad-invariant metric on aff(2)). The algebra aff(2) is not semisimple; it contains the nonzero translation ideal t = R2 . We show no Ad-invariant positive-definite metric exists. Let B be any Adinvariant symmetric bilinear form. Take the isotropic-scale generator sI for s ̸= 0; for v1 , v2 ∈ t, Ad-invariance gives B(sv1 , v2 ) + B(v1 , sv2 ) = 0 =⇒ 2s B(v1 , v2 ) = 0, hence B|t×t = 0. The cross-pairing B|t×gl(2) vanishes by the same generator: for v ∈ t and Y ∈ gl(2), [sI, v] = s v while [sI, Y ] = 0 since sI is central in gl(2), so Ad-invariance gives s B(v, Y ) = 0, hence B(v, Y ) = 0. Therefore t lies in the radical of every Ad-invariant symmetric bilinear form, and no such form is non-degenerate. The Frobenius form is non-degenerate and positive-definite but not Ad-invariant on aff(2). The construction does not require Ad-invariance of the metric: it requires that wij itself be invariant (Lemma 2), so any fixed bilinear form on g applied to wij yields an invariant scalar. We use the Frobenius form because it is positive-definite, gives equal weight to all coordinates in the orthonormal basis of Section 2.2, and admits the block-weighted refinement (2). For SE(2), SE(3), an analogous no-Ad-invariant-metric result holds via a structurally different argument (using translation generators acting on rotation through the bracket); see Remarks in Section 3. For SO(3) (and compact simple algebras generally) the Killing form is non-degenerate and the metric is canonical up to a positive scalar. For SO(2) the algebra is abelian and the Killing form vanishes identically; any positive scalar gives an Ad-invariant metric on the 1-dimensional algebra.
2.5
The invariant wij
With tokens on the group, the pairwise invariant is not something we have to design. It is intrinsic to the token type. Given two tokens gi , gj ∈ G, their relative pose is the group element gi−1 gj . For Aff(2): −1 gi−1 gj = A−1 i Aj , Ai (tj − ti ) . When gi−1 gj ∈ U, mapping to the algebra gives a dim(g)-vector encoding how j differs from i in each geometric component: wij = log gi−1 gj ∈ g. 4
Lemma 2 (Diagonal invariance). wij is invariant under the diagonal action gi 7→ agi for every a ∈ G. Proof. (agi )−1 (agj ) = gi−1 a−1 a gj = gi−1 gj , and log depends only on the group element. The cancellation needs only the group axioms. It holds for every matrix Lie group, and so does the lemma. Remark 3 (Antisymmetry). On the chart, log(g −1 ) = − log(g), so wji = −wij and ∥wji ∥2λ = ∥wij ∥2λ . The −1 symmetry is chart-local; it requires both gij and gij to lie in U. For Aff(2), the coordinates of wij in the basis of Section 2.2 are the relative translation, relative rotation, relative isotropic scale, and relative anisotropic-scale-and-shear between gi and gj . Each of the four block coordinates is itself G-invariant.
2.6
The score sij = −∥wij ∥2λ /τ
Once wij is the intrinsic invariant of a token pair, the score follows as its canonical readout. The natural diagonalinvariant scalar built from wij is its squared norm under the block-weighted inner product of Section 2.4. Because wij is itself invariant (Lemma 2), any fixed bilinear form on g applied to it is invariant as well. The block-weighted squared norm is therefore the canonical proximity readout, with the block-diagonal O(n)-isotropic shape fixed in Section 2.4. The word canonical carries two senses here. The functional form, a squared algebra norm of the invariant, is canonical given wij . The block weights λ are a minimal learned refinement, since no canonical Ad-invariant metric exists on the non-compact cases (Remark 1). Section 5 finds that a learned kernel on the same invariant does not improve on this form. The attention score between tokens i and j is the negative of that, divided by a temperature: sij = −∥wij ∥2λ / τ,
τ ∈ R>0 .
(3)
This is a proximity kernel: tokens with similar poses receive high scores; tokens with large geometric differences receive low scores. Each of the block weights controls how strongly one type of geometric difference contributes. By Lemma 2 the score is invariant under the diagonal G action for every choice of λ and τ , since wij is invariant and the norm is a deterministic function of wij . Multi-head structure. With H heads, each head k = 1, . . . , H carries its own weight vector λk ∈ RK >0 , where K is the number of blocks in the algebra (Table 1). With a per-head temperature τk , the total number of score parameters is H(K + 1): for Aff(2) (K = 4), 5H parameters; for SE(n) (K = 2), 3H; for SO(n) (K = 1), 2H. Only the ratios λk /τk matter; τk can be absorbed into λk . One head may weight translation heavily, attending by spatial proximity. Another may weight rotation, attending by orientation similarity. The block decomposition of Section 2.2 is what makes this separation possible. Domain. The score is defined for token pairs where gi−1 gj ∈ U . On or near the chart boundary, the global form of Section 2.7 applies but the closed-form score does not.
2.7
Equivariant output (global and local form)
Equivariance, in this construction, is not a property to be enforced by representation machinery. It is tautological: the form of every equivariant map is fixed by the token type. The following theorem states it. Theorem 4 (Equivariant output, global form). Every map {gi } 7→ {ĝi } satisfying ĝi (a·) = a · ĝi (·) for all a ∈ G has the form ĝi = gi ∆i , ∆i ∈ G invariant under the diagonal G action. (4) Proof. Define ∆i = gi−1 ĝi . Under gi 7→ agi and ĝi 7→ aĝi : gi−1 ĝi 7→ (agi )−1 (aĝi ) = gi−1 ĝi , so ∆i is invariant. The reconstruction ĝi = gi ∆i recovers the original map. Local form. When ∆i ∈ U, write δi = log(∆i ) ∈ g.
∆i = exp(δi ), Since ∆i is invariant, so is δi . The output becomes
ĝi = gi exp(δi ).
5
(5)
This is the form used in practice: the model predicts δi ∈ Rdim g from invariant features and the output pose is gi exp(δi ). The exponential map of G is not surjective in general: not every group element is the exponential of an algebra element. The local form (5) covers tokens whose corrections lie on the chart; the global form (4) applies to the rest.
2.8
Cocycle preservation
The relative poses between output tokens must be internally consistent: ĝij ĝjk must equal ĝik for all triples i, j, k. This is the cocycle condition. Because the tokens are group elements, the output formula (4) satisfies it automatically. The model cannot emit a globally inconsistent set of poses, even adversarially. The consistency lives in the token type, not in the loss. For ĝi = gi ∆i , the output relative pose is ĝij = ĝi−1 ĝj = ∆−1 i gij ∆j . Composing two consecutive relative poses: −1 gjk ∆k = ∆−1 ĝij ĝjk = ∆−1 i gik ∆k = ĝik . i gij ∆j ∆j | {z } =e
The local form ĝi = gi exp(δi ) gives the same conclusion: ĝij ĝjk = exp(−δi ) gik exp(δk ) = ĝik . The cocycle holds for every choice of ∆i (or δi ), with no constraint imposed on the model. Consistency is structural, not learned.
3
Six Instantiations
The construction of Section 2 applies to any matrix Lie group with a faithful finite-dimensional representation. This section instantiates it explicitly for SO(2), SE(2), SO(3), SE(3), and Aff(3); the remaining group, Aff(2), is the running worked example throughout Section 2. The compact and Euclidean cases SO(2), SE(2), SO(3), SE(3) are all instances of one token type. Together they reproduce the regimes that prior methods handle one group at a time. The affine cases Aff(2) and Aff(3) are non-compact and non-abelian, with scale and shear. No irrep method can reach them, since they have no non-trivial finite-dimensional unitary irreps, and no surjectiveexp method can either, since the LieConv-style group lifting requires a surjective exponential that the affine cases lack. Yet they fall out of the same construction with only a larger block count. They are the point of the section. The six instantiations cover all groups relevant to 2D and 3D spatial reasoning. The proofs of invariance, equivariant output, and cocycle preservation in Section 2 apply to all six instantiations without modification. They use only the group axioms and the cancellation (agi )−1 (agj ) = gi−1 gj .
3.1
SO(2) — Planar rotations
Group. SO(2) = {R(θ) : θ ∈ R}, dimension 1. 0 −1 Algebra. so(2) = {ωJ : ω ∈ R} where J = . Dimension 1. 1 0 Exp/log. exp(ωJ) = R(ω); log(R(θ)) = θJ for θ ∈ (−π, π). Block decomposition. One block: rotation. so(2) = |{z} R . √rotation Orthonormal basis. R0 = √12 J. Coordinate: δθ = 2 ω. Score. One weight: 2 sij = −λθ δθij /τ.
Block-weight count per head: 1 (plus temperature). The score operates on the same invariant as RoPE (the angular difference) but with quadratic, not oscillatory, dependence.
3.2
SE(2) — Rigid motions of the plane
Group. SE(2) = SO(2) ⋉ R2 , dimension 3. Algebra. ωJ se(2) = 0
v 0
2
: ω ∈ R, v ∈ R 6
∼ = R3 .
Bracket: [(ω1 , v1 ), (ω2 , v2 )] = (0, ω1 Jv2 − ω2 Jv1 ). ω J and V (θ)−1 = θ2 cot(θ/2) I − θ2 J, on the chart Exp/log. Closed-form via V (ω) = sinω ω I + 1−cos ω θ ∈ (−π, π). Block decomposition. Two blocks: se(2) = |{z} R2 ⊕ |{z} R . translation
rotation
Orthonormal basis. Tx , Ty (translation, unit norm), R0 = √12 J (rotation, normalized). Score. Two weights: sij = − λt (δt2x + δt2y ) + λθ δθ2 /τ. Block-weight count per head: 2 (plus temperature). Experimentally validated in Section 5. Remark 5 (No Ad-invariant positive-definite metric on se(2)). The translation ideal t = R2 is a nonzero abelian ideal of se(2). Let B be any Ad-invariant symmetric bilinear form. For X = (0, vX ), Y = (0, vY ) ∈ t and Z = (ω, 0) ∈ so(2) with ω ̸= 0: since translations commute, [X, Y ] = 0, so Ad-invariance gives B(Y, [X, Z]) = 0. The bracket evaluates to [X, Z] = (0, −ωJvX ) ∈ t. Since J is invertible, {JvX : vX ∈ R2 } = R2 . Therefore B|t×t = 0. A positive-definite form cannot vanish on a non-zero subspace, so no Ad-invariant positive-definite metric exists. The Frobenius form is non-degenerate and positive-definite but not Ad-invariant.
3.3
SO(3) — Spatial rotations
Group. SO(3) = {R ∈ R3×3 : R⊤ R = I, det R = 1}, dimension 3. Algebra. so(3) = {X ∈ R3×3 : X ⊤ = −X}, spanned by 0 0 0 1 0 0 0 Lx = 0 0 −1 , Ly = 0 0 0 , Lz = 1 0 −1 0 0 0 1 0
−1 0 0
0 0 . 0
√ Each has ∥Lα ∥F = 2. Bracket: [Lx , Ly ] = Lz and cyclic. Exp/log. Rodrigues formula: exp(θ n̂× ) = I + sin θ n̂× + (1 − cos θ) n̂2× , where n̂ ∈ S 2 is the axis and θ ∈ [0, π). Log on the chart θ ∈ (0, π): R − R⊤ tr(R) − 1 , n̂× = . θ = arccos 2 2 sin θ Block decomposition. One block: rotation. so(3) = |{z} R3 . The algebra is simple; it does not decompose rotation
further. √ Orthonormal basis. Rα = √12 Lα for α ∈ {x, y, z}. Coordinates: δθα = 2 ωα . Score. One weight: sij = −λθ (δθx2 + δθy2 + δθz2 )/τ. Block-weight count per head: 1 (plus temperature). This is the squared geodesic distance on SO(3), up to a positive scalar. Remark 6 (Canonical Ad-invariant metric on so(3)). so(3) is simple and compact. The Killing form K(X, Y ) = tr(adX adY ) is negative-definite and Ad-invariant. Computed in the standard 3D representation: K(Lx , Lx ) = tr(ad2Lx ) = −2 = tr(L2x ), so K(X, Y ) = tr(XY ). For antisymmetric matrices, tr(XY ) = −tr(X ⊤ Y ), hence K(X, Y ) = −tr(X ⊤ Y ). The canonical Ad-invariant positive-definite metric is therefore ⟨X, Y ⟩ = −K(X, Y ) = tr(X ⊤ Y ), which is the Frobenius form. The metric is determined up to a positive scalar, and the block weight λθ is a global scale only.
7
3.4
SE(3) — Rigid motions of 3-space
Group. SE(3) = SO(3) ⋉ R3 , dimension 6: R SE(3) = 0
t 1
: R ∈ SO(3), t ∈ R3 .
Algebra. se(3) =
Ω 0
v 0
: Ω ∈ so(3), v ∈ R3
∼ = R6 .
Bracket: [(Ω1 , v1 ), (Ω2 , v2 )] = ([Ω1 , Ω2 ], Ω1 v2 − Ω2 v1 ). Exp/log. Block structure identical to Aff(2) and SE(2): exp(Ω, v) = eΩ , V (Ω) v , with eΩ via Rodrigues and
θ − sin θ 2 1 − cos θ Ω+ Ω , 2 θ θ3 where θ is the rotation angle, defined coordinate-free by q θ = − 12 tr(Ω2 ) = ∥ω∥2 , V (Ω) = I +
√ with ω ∈ R3 the axial vector√ satisfying Ω = [ω]× . Note that ∥Ω∥F = θ 2, so the Frobenius norm of Ω is not the rotation angle; the factor 2 corresponds to the orthonormal normalization of Section 2.2. Inverting on the chart θ ∈ (0, π): 1 1 + cos θ −1 1 V (Ω) = I − 2 Ω + − Ω2 . θ2 2θ sin θ Block decomposition. Two blocks: se(3) = |{z} R3 ⊕ |{z} R3 . translation
rotation
Orthonormal basis. Translation: Tx , Ty , Tz (unit norm in the homogeneous embedding). Rotation: Rα =
√1 Lα . 2
Score. Two weights: sij = − λt (δt2x + δt2y + δt2z ) + λθ (δθx2 + δθy2 + δθz2 ) /τ. Block-weight count per head: 2 (plus temperature). Same structure as SE(2), one dimension higher.
Remark 7 (No Ad-invariant positive-definite metric on se(3)). Same structure as SE(2). For X = (0, vX ), Y = (0, vY ) ∈ t and Z = (Ω, 0) ∈ so(3): [X, Y ] = 0 and [X, Z] = (0, −ΩvX ) ∈ t. So B|t×t (vY , ΩvX ) = 0. ⊥ ⊥ For fixed vX ̸= 0, {ΩvX : Ω ∈ so(3)} = vX . Every non-zero w ∈ R3 lies in vX for some vX (choose any 3 vX ⊥ w), so the union over vX is all of R . Therefore B|t×t = 0, and no Ad-invariant positive-definite metric exists on se(3).
3.5
Aff(3) — Spatial affine frames
The affine construction extends directly to spatial affine frames. An Aff(3) token is A t 3 3 Aff(3) = GL(3, R) ⋉ R = : A ∈ GL(3, R), t ∈ R , 0 1 with composition and inverse (A1 , t1 )(A2 , t2 ) = (A1 A2 , A1 t2 + t1 ), Its algebra is
(A, t)−1 = (A−1 , −A−1 t).
aff(3) = gl(3, R) ⋉ R3 ,
with bracket [(X1 , v1 ), (X2 , v2 )] = ([X1 , X2 ], X1 v2 − X2 v1 ). 8
Under the natural orthogonal action, the algebra decomposes as R ⊕ Sym0 (3) , aff(3) = |{z} R3 ⊕ so(3) ⊕ |{z} | {z } | {z } translation
rotation
iso. scale
aniso. + shear
with dimensions 3 + 3 + 1 + 5 = 12. Thus Aff(3) has the same four geometric block types as Aff(2), but the traceless symmetric block grows from dimension 2 to dimension 5. Writing wij = log(gi−1 gj ) = (uij , ωij , ηij , qij ) ∈ R3 ⊕ R3 ⊕ R ⊕ R5 in orthonormal block coordinates, the block-weighted norm is 2 ∥wij ∥2λ = λt ∥uij ∥2 + λθ ∥ωij ∥2 + λs ηij + λq ∥qij ∥2 ,
and the score is sij = −∥wij ∥2λ /τ . The number of score weights per head is therefore four, exactly as for Aff(2). The exponential and logarithm use the same block-triangular form as in Aff(2): Z 1 X exp(X, v) = e , V (X) v , V (X) = erX dr. 0
On the principal chart
UAff(3) = {(A, t) : A has no eigenvalue on R≤0 }, log(A, t) = log A, V (log A)−1 t .
Remark 8 (No Ad-invariant positive-definite metric on aff(3)). Let D = (I3 , 0) be the isotropic-scale generator. Then [D, (0, v)] = (0, v) for every translation v ∈ R3 . Ad-invariance of a symmetric bilinear form B gives, for v1 , v2 ∈ t, B([D, v1 ], v2 ) + B(v1 , [D, v2 ]) = 0 =⇒ 2 B(v1 , v2 ) = 0, hence B|t×t = 0. For X ∈ gl(3), [D, X] = 0, so B([D, v], X) + B(v, [D, X]) = B(v, X) + 0 = 0, giving B(v, X) = 0. The translation ideal lies in the radical of every Ad-invariant symmetric bilinear form; no Adinvariant positive-definite metric exists on aff(3). The construction uses the fixed positive-definite Frobenius form and its four-block refinement above.
3.6
Summary table
Group
Dim
Blocks
SO(2) SE(2) SO(3) SE(3) Aff(2) Aff(3)
1 3 3 6 6 12
rotation (1) trans. (2) + rot. (1) rotation (3) trans. (3) + rot. (3) trans. (2) + rot. (1) + scale (1) + shear (2) trans. (3) + rot. (3) + scale (1) + aniso.+shear (5)
Weights/head 1 2 1 2 4 4
Exp/log
Ad-inv. metric?
R(ω) / θ V (ω) closed-form Rodrigues Rodrigues + V (Ω) Cayley–Hamilton matrix exp/log + V (X)
yes (any λ > 0; abelian, Killing = 0) no (radical) yes (Killing) no (radical) no (radical) no (radical)
Table 1: Six instantiations of the construction. For all groups, the score is sij = −∥wij ∥2λ /τ with wij = log(gi−1 gj ). The per-group block count is not a design choice: it equals the number of O(n)-irreducible components of the algebra under the standard representation (Schur’s lemma applied to g as an O(n)-module). For Aff(2) the irreducibles are V ⊕R⊕R⊕Sym20 V (translation V = R2 , rotation Λ2 V ∼ = R, isotropic scale R, anisotropic+shear Sym20 V ∼ = R2 ) — four blocks, four weights. The group dictates K; we do not. For compact simple algebras (so(3)), the Killing form provides a canonical metric and the block weight is a global scale only. For SO(2), the algebra is 1-dimensional and abelian; the Killing form is zero, but any positive scalar defines an Ad-invariant metric (the choice is absorbed by λθ ). For non-compact groups with a nonzero translation ideal (SE(2), SE(3), Aff(2)), no Ad-invariant positive-definite metric exists. Here the block weights carry geometric content. They determine the relative importance of translation versus rotation, and of scale and shear for Aff(2). Aff(2) is the case no vector-token attention method reaches. The irrep-and-Clebsch–Gordan tradition is restricted by unitarity to compact groups, and Aff(2) is non-compact non-abelian (no non-trivial finitedimensional unitary irreps). LieTransformer [16] attends over (g, v) pairs and learns the kernel; the LieConv [17] 9
lifting it builds on requires a surjective exponential, which Aff(2) fails. PONITA-style position-orientation reductions partially collapse the rotation block and have no place for scale or shear. Placing the token strictly on the group unlocks the full affine case, and with it the entire four-block decomposition above. That same decomposition extends verbatim to Aff(3) (Section 3.5). The construction is dimension-uniform across Aff(n) for n ≥ 2, via the same O(n)-irreducible split.
4
Architecture
The construction of Section 2 fits inside a standard transformer skeleton with one substitution. The backbone is a vanilla transformer with pre-layer normalization, multi-head attention, a feed-forward block, and residual connections. It operates directly on group-element tokens. The attention score is the algebra-norm score (3) on the invariant wij , and the per-token output is the local-form correction (5). Each token is nothing but its group element; all geometry enters through wij . The backbone carries none of the representation-theoretic machinery enumerated in Section 1 — no irrep layers, no auxiliary frames, no equivariant message-passing primitives. The equivariance lives in the token type and the score; the network adds nothing geometric on top.
4.1
Set-input transformer
The input is a permutation-invariant set {gi }N i=1 ⊂ G of N group-valued tokens. All tokens are initialized with the same learned vector, (0) hi = h0 ∈ Rd , i = 1, . . . , N, so that all token-distinguishing information must enter through the geometric relations wij . No positional encoding is used; spatial position lives in the group element gi itself and is accessed through wij . The distinction matters for the claim that tokens carry no payload. The token is the bare group element gi . The vector hi is a hidden state: identical across tokens at initialization, it is filled in only from the relations wij during the forward pass, never a feature attached to a particular token. Diagonal entries of the attention map are masked, sii = −∞, preventing trivial self-attention from wii = 0. In the experiments of Section 5 the architecture uses L = 3 layers, embedding dimension d = 32, and H = 4 attention heads, on 7-token model inputs (one element held out of each length-8 sequence).
4.2
Forward pass
The pairwise invariant is computed once at the input and reused across all L layers: wij = log gi−1 gj ∈ Rdim g , (i, j) ∈ {1, . . . , N }2 . Each transformer layer is a pre-LN block: 1 h(ℓ+ 2 ) = h(ℓ) + Attn LN(h(ℓ) ), w , 1 1 h(ℓ+1) = h(ℓ+ 2 ) + FFN LN(h(ℓ+ 2 ) ) ,
where the FFN is two linear layers with hidden width 4d and a GELU non-linearity. The attention module Attn(·, w) produces per-token updates as follows. Scores. For each head k = 1, . . . , H: (k) (k) (k) sij = −∥wij ∥2λk / τk , αij = softmaxj sij , (k)
with the diagonal masked sii = −∞ before softmax. The block-weighted norm ∥ · ∥2λk is the per-head instance of (2); the number of weights per head equals the block count of Table 1. Values. Each pair (i, j) contributes a value vector Vij = WV hj ; wij ∈ Rd , formed by concatenating the source hidden state hj with the relative pose wij and projecting through a single linear map WV ∈ Rd×(d+dim g) . The result is reshaped into H heads of dimension d/H and combined with the attention weights α(k) to produce the layer output. An output linear WO then projects this output back to Rd . We use Vij = WV [hj ; wij ] instead of the standard Vij = WV hj . This makes the value pathway carry the geometric difference vector, not just the source hidden state. This preserves directional information that the score loses when it takes a squared norm: ∥wij ∥2λ = ∥ − wij ∥2λ , but the values Vij and Vji are distinct. 10
4.3
Multi-head and block-weight parameterization
Each head k owns its own block-weight vector λk ∈ RK >0 and temperature τk > 0, where K is the algebra’s block count (Table 1). Positivity is enforced by a softplus reparameterization: λk = softplus(λ̃k ) + ε,
τk = softplus(τ̃k ) + ε,
with λ̃k , τ̃k ∈ R the unconstrained learned parameters and ε = 10−3 a small floor. The unconstrained parameters are initialized at zero, giving λk ≈ τk ≈ ln 2 ≈ 0.69 at initialization — isotropic across blocks, on the order of unity. The total number of score parameters in L layers with H heads is L · H · (K + 1). For the SE(2) experiments (K = 2, H = 4, L = 3), this is 36.
4.4
Output head (L)
The per-token correction is produced by a two-layer MLP applied to the final hidden state hi : (L) (L) δi = ϕδ hi = W2 GELU W1 hi + b1 + b2 ∈ Rdim g , with W1 ∈ Rd×d , W2 ∈ Rdim g×d . The output pose is the local form of Theorem 4: ĝi = gi exp(δi ). (L)
Since δi is a function of the invariant features hi , it is itself invariant under the diagonal G action. The composition ĝi is therefore equivariant. Task-specific output heads — for example, a per-token gap-detection head in the sequence-completion experiments of Section 5 — are added on top of this base in the same invariant-features-in, scalar-out pattern.
4.5
Equivariance and parameter count
Equivariance is structural rather than learned. The input wij is invariant by Lemma 2. Every subsequent operation (score, softmax, value, FFN, layer norm) is a function of invariant features and produces invariant features. The output head produces invariant δi ; the final composition ĝi = gi exp(δi ) is equivariant by Theorem 4. No training signal enforces equivariance and no equivariance-error term appears in the loss. For the SE(2) instance (Section 5.1), the parameter counts are (per model, total parameters / score-only parameters); the per-group score-parameter counts for all three experiments are given in Section 5: Model G — algebra-norm score (3) C — learned MLP kernel on the same wij A — vanilla scaled-dot-product on absolute coordinates
Total
Score-only
∼33 000 ∼35 000 ∼40 000
36 1 932 —
The score-parameter ratio between G and C is ∼54×. Model C uses the same invariant wij as G but replaces the closed-form score (3) with a per-head MLP ψ : Rdim g → R; everything else is identical. Model A uses the standard scaled-dot-product score on absolute coordinate features (e.g., (cos θi , sin θi , tx,i , ty,i ) for SE(2)) and serves as the equivariance-breaking control. The architecture is otherwise identical across G, C, A: same depth, width, head count, FFN, and output head.
5
Experiments
We validate the construction on three groups spanning every block-count level of Section 1: SE(2) (two blocks), SO(3) (one block), and Aff(2) (four blocks). The task in each case is sequence completion: a constant-step sequence of N =8 group elements is generated, one interior token is removed and the remaining seven are permuted; the model must reconstruct the missing pose from the unordered set. Three models are compared in each experiment: • G — algebra-norm score sij = −∥wij ∥2λ /τ from (3). • C — same invariant wij , score replaced by a per-head MLP ψ : Rdim g → R (one hidden layer of 32 units, ReLU). This is the LieTransformer-style move of learning the kernel on the relative-pose invariant; against G it isolates fixed-versus-learned scoring, with the invariant and the rest of the architecture held constant. 11
• A — the vector-token ontology held fixed: tokens are flat absolute features in Rd (specified per experiment) and the score is vanilla scaled-dot-product attention. This is the standard equivariant-ML schema (token = vector, no intrinsic invariant) operating on the same data, and serves as the direct ontology-comparison control. It isolates the group-token-versus-vector-token ontology; the finer distinction between a bare group element and a (g, v) pair (the contrast with LieTransformer) is argued in Section 6, not ablated here. All three architectures are otherwise identical: the transformer of Section 4 with L=3 layers, H=4 heads, d=32, the gap-detection head, the local-form correction head, and the same value pathway Vij = WV [hj ; wij ] on G and C. Training: Adam at 10−3 , batch 64, gradient clip 2.0, 200 epochs, 5,000 training instances, 500 each for validation and test. Each configuration is run with three random seeds; we report mean ± std across seeds. Metrics (common). Pose error Eg = ∥ log(ĝ −1 gj )∥2 on the test set in physical coordinates (Frobenius fallback near the chart boundary; in all three experiments the fallback was triggered by no test instance). Flanking accuracy: indicator that the predicted base-token index î = arg maxi pi is one of the two true neighbors {j − 1, j + 1}. Equivariance error: apply 10 random global a ∈ G per instance and measure ∥ log((aĝ)−1 ĝ(a · S))∥2 , averaged.
5.1
SE(2) sequence completion
Task. Generate gk = g0 · hk on SE(2) with g0 sampled across the plane and h = exp(ωh , vh ) small enough that |ωh | < π/8 keeps every relative pose on the principal chart. Score parameters. G uses 2 block weights + temperature per head = 3 score parameters per head, for L · H · 3 = 36 total. C’s per-head MLP R3 → R has ∼161 score parameters per head, 1,932 total. C/G ratio: 54×. Absolute features for A. vi = (cos θi , sin θi , tx,i , ty,i ) ∈ R4 . Results (Table 2, Figure 1). Model
Pose error ↓
Flanking acc. ↑
Equivariance error ↓
G C A
0.003 ± 0.001 0.005 ± 0.001 0.069 ± 0.002
1.000 ± 0.000 1.000 ± 0.000 0.999 ± 0.001
4.0 × 10−10 1.3 × 10−12 1.3 × 10−1
Table 2: Sequence completion on SE(2). Mean ± std over 3 seeds on 500 test instances.
Sequence completion on SE(2) pose error
equivariance error 10 1 10 3 10 5 10 7
10 2
10 9 10 11 G
C
A
G
C
A
Figure 1: SE(2) sequence completion: pose error and equivariance error per model (log scale). G is 34% better than C at 54× fewer score parameters; both vastly outperform the equivariance-blind A, whose equivariance error is eleven orders of magnitude larger. G uses 36 score parameters and reaches pose error 0.003; C uses 1,932 (54× more) and reaches 0.005. The closed-form algebra-norm score is 34% better than the learned MLP on the same invariant, with 54× fewer parameters. G’s and C’s equivariance errors are at the float32 numerical floor; A’s is ∼ 10−1 , confirming that the task tests invariance and that G/C are equivariant by construction. 12
5.2
SO(3) sequence completion
Task. Generate gk = g0 · hk on SO(3) with g0 sampled Haar-uniformly and h = exp([ωh ]× ) with ∥ωh ∥ ≤ π/8, so that powers hk for k ≤ 7 stay on the principal chart. Same sequence-completion task: remove one interior token, reconstruct. Score parameters. so(3) has one block (the algebra is simple), so G uses one block weight λθ plus temperature per head = 2 score parameters per head, L · H · 2 = 24 total. C’s per-head MLP R3 → R has ∼161 score parameters per head, 1,932 total. C/G ratio: 80×. Absolute features for A. vi = vec(Ri ) ∈ R9 (flattened rotation matrix). Results (Table 3, Figure 2). Model G C A
Pose error ↓
Flanking acc. ↑
Equivariance error ↓
(1.8 ± 0.1) × 10−4 (1.3 ± 0.5) × 10−4 0.067 ± 0.002
0.998 ± 0.000 0.998 ± 0.001 0.965 ± 0.006
1.6 × 10−14 2.7 × 10−14 7.4 × 10−2
Table 3: Sequence completion on SO(3). Mean ± std over 3 seeds on 500 test instances.
Sequence completion on SO(3) pose error
equivariance error
10 1 10 3
10 2
10 5 10 7 10 9
10 3
10 11 10 13
10 4 G
C
A
G
C
A
Figure 2: SO(3) sequence completion: pose error and equivariance error per model (log scale). G with 24 score parameters matches C at 80× fewer parameters (both at the 10−4 floor); G’s and C’s equivariance errors sit at ∼ 10−14 in float32, while A’s is ∼ 10−1 . Both G and C reach pose error ∼10−4 (statistically indistinguishable within seed noise); C’s third seed lands lower and accounts for the wider C-std, but the mean difference is well inside the spread of either model. Equivariance errors for G and C sit at ∼ 10−14 , effectively float64 precision in float32 arithmetic. These are the cleanest of the three groups, because the SO(3) Rodrigues path is a single closed-form branch with no eigenvalue casesplits. A fails by twelve orders of magnitude on equivariance and by 380× on pose; its flanking accuracy also drops to 0.965, the only configuration where the gap head is not effectively saturated.
5.3
Aff(2) sequence completion
Task. Generate gk = g0 · hk on Aff(2) with g0 = (A0 , t0 ) sampled generously across the group (rotation, anisotropic scale and shear of bounded magnitude, translation ∼ N (0, 9I2 )). The step h = exp(Xh , vh ) has its linear-part magnitude bounded so that Akh for k ≤ 7 has no eigenvalue on R≤0 (chart safety on the principal log chart of GL(2, R)). Score parameters. aff(2) has four blocks ((1)), so G uses 4 block weights + temperature per head = 5 score parameters per head, L · H · 5 = 60 total. C’s per-head MLP R6 → R has ∼257 score parameters per head, 3,084 total. C/G ratio: 51×. Absolute features for A. vi = (Ai,11 , Ai,12 , Ai,21 , Ai,22 , tx,i , ty,i ) ∈ R6 . Results (Table 4, Figure 3).
13
Model
Pose error ↓
Flanking acc. ↑
Equivariance error ↓
G C A
0.007 ± 0.002 0.007 ± 0.001 0.79 ± 0.10
1.000 ± 0.000 1.000 ± 0.000 0.953 ± 0.010
2.3 × 10−5 1.4 × 10−9 1.29
Table 4: Sequence completion on Aff(2). Mean ± std over 3 seeds on 500 test instances.
Sequence completion on Aff(2) pose error
100
equivariance error 100 10 2
10 1
10 4 10 6
10 2
10 8 G
C
A
G
C
A
Figure 3: Aff(2) sequence completion: pose error and equivariance error per model (log scale). G with 60 score parameters matches C at 51× fewer parameters; A’s pose error is 117× worse than G’s, and its equivariance error is 1.29 — five orders of magnitude above G’s and nine above C’s. G and C reach pose error ≈ 0.007, indistinguishable within seed noise, despite C using 51× more score parameters. A collapses. Its pose error is 0.79 (117× G’s). Its equivariance error is 1.29, well above the inputs themselves, so the model produces an essentially independent output under global transforms. The gap-detection head also degrades, to 0.95 flanking accuracy. This is the only group where lack of structural invariance also breaks token-identification performance. G’s equivariance error of 2.3 × 10−5 is several orders of magnitude above the SO(3) and SE(2) values. This is a float32 accumulation artifact of the branched Cayley–Hamilton path for the GL(2) matrix exp and log, with its real-distinct, real-repeated, complex-conjugate, and near-zero eigenvalue cases. It is not a structural failure. The construction is exactly equivariant in arithmetic for Aff(2) as for the other groups.
5.4
Interpretation
The three experiments deliver three claims about the construction, backed by every block-count level of the instantiation table. The closed-form score is canonical, not merely compact. Model C gives a learned MLP kernel one to two orders of magnitude more score parameters on the same invariant wij , 50 to 80× G’s count (24 vs 1,932 on SO(3), 36 vs 1,932 on SE(2), 60 vs 3,084 on Aff(2)). It does not improve on it. G outperforms C on SE(2) by ∼34% and matches it within seed noise on SO(3) and Aff(2), with the SO(3) point estimate marginally favouring C. The MLP has every bit of capacity needed to represent G’s kernel and more, yet cannot reliably turn the surplus into a better attention pattern. On these tasks a far higher-capacity learned kernel does not beat the closed-form norm — consistent with the block-weighted algebra norm being the canonical readout of the invariant rather than a compression of some better learned score. The 50 to 80× parameter saving is a corollary of that, not the headline. These tasks are deliberately simple — flanking accuracy saturates at 1.000 on SE(2) and Aff(2) — so the parity bounds what a learned kernel buys in this regime rather than settling it in general; a harder, non-saturated task is left to follow-up. Structural equivariance. G’s and C’s equivariance errors stay at the float32 numerical floor across all three groups: ∼ 10−14 on SO(3), ∼ 10−10 –10−12 on SE(2), ∼ 10−5 –10−9 on Aff(2). The differences across groups track the length of the matrix exp/log path: single-branch Rodrigues for SO(3), two-block triangular for SE(2), four-branch Cayley–Hamilton for Aff(2). They do not reflect any structural property. The construction is exactly equivariant in arithmetic for every matrix Lie group it covers. A breaks equivariance by five (on Aff(2), relative 14
to G’s higher floor) to twelve (on SO(3), relative to G’s 10−14 ) orders of magnitude — confirming that the tasks test invariance and that G’s equivariance is structural, not trained. Reaching the affine regime. The three experiments cover every block-count level of the construction’s instantiation: K = 1 (SO(3)), K = 2 (SE(2)), K = 4 (Aff(2)); the closed-form algebra-norm score works at every level with no architectural change, only the number of block weights (K + 1 per head including temperature). The Aff(2) experiment is the load-bearing one: its four-block decomposition — translation, rotation, isotropic scale, anisotropic-and-shear — realizes in a transformer the non-compact non-abelian affine regime that no irrep or surjective-exp attention method reaches, the first such demonstration we are aware of. On the constant-step data the per-instance invariants are collinear, so the four block weights enter only through a single scalar and are not separately identified. The experiment shows the construction runs and is equivariant in this regime, not that the block weighting itself is exercised. A task separating the blocks is left to the same follow-up.
6
Related Work
Every prior tradition keeps the token a vector and lets the group act on it externally; the structural limitation each inherits is a symptom of that shared ontology. We compare to nine traditions below, organized by token type, group operations, and the limitation each ontology imposes. The construction here uses none of these enforcement mechanisms, because placing the token on G makes equivariance tautological rather than something to enforce. Tradition
Token domain
Group ops
Structural limitation
Irrep / harmonic EGNN Geometric algebra Capsules Frame-augmented (IPA) LieTransformer PONITA (position-orient.) Mironenco–Forré (Lie decomp.) RoPE / RiemannFormer
vector in irrep space (xi , hi ) ∈ Rn × Rd multivector in flat Rd unconstrained 4 × 4 matrix vector + frame Ti ∈ SE(3) (g, v) ∈ G × Rd (x, n) ∈ Rn × S n−1 function on Aff + (n) vector in Rd
representation distances + coordinate updates sandwich product heuristic hybrid learned kernel on pairs homogeneous-space Cartan/polar, Haar-MC operational
no non-compact irreps point tokens; no frames/poses token off the manifold no Lie structure position-only kernel surjective-exp lift (LieConv) rotation block reduced; SE(n) only CNN; not transformer not manifold-valued
This work
gi ∈ G on the group
closed-form algebra norm
—
Table 5: Attention methods by token type: every prior token is a vector carrying an external group action; this work alone places the token on the group. Listed by token domain, group operations, and structural limitation; sources cited in subsections.
6.1
Irrep / harmonic analysis
TFN [1], SE(3)-Transformer [2], Equiformer [3], MACE [5], group-equivariant CNNs [6], and steerable CNNs [7] act on tokens that live in vector spaces carrying group representations. Equivariance is achieved via Clebsch– Gordan tensor products of irreps, restricted by unitarity to compact groups. The translation part of SE(n) is handled via relative positions (an abelian normal subgroup factoring trick); the rotation part uses irreps of SO(n). Non-trivial unitary irreps of non-compact non-abelian groups are infinite-dimensional, so the standard recipe does not extend to Aff(n). The tokens are vectors with ρ(g) acting on them, not points on the group manifold. The line continues to the current state of the art — EquiformerV3 [4] is a 2026 irrep SE(3) graphattention transformer.
6.2
E(n)-equivariant graph networks
EGNN [8] is a central point-token baseline for molecules and point clouds. Its token is (xi , hi ): a point xi ∈ Rn plus invariant features hi ∈ Rd .1 Messages depend on invariant squared distances ∥xi − xj ∥2 , and coordinate updates are built from relative displacement vectors xi −xj , giving E(n) equivariance without irreps or spherical harmonics. This is close in spirit to using native geometric quantities, but the token is still a point with attached features, not a frame or transformation. There is no group-valued token gi , no relative pose gi−1 gj , and no 1 EGNN’s h is a feature attached to the token. The hidden state h of Section 4 reuses the symbol but is not a token payload: it is identical i i across tokens at initialization and acquires content only from the relations wij (Section 4.1).
15
unified rotation/translation/shear logarithm. EGNN is therefore the natural Euclidean point baseline, whereas the construction here is the corresponding frame/pose-token construction.
6.3
Geometric algebra
GATr [9], DriveGATr [11], and the algebra-choice paper [10] represent tokens as multivectors in a fixed-dimension Clifford algebra. The group acts via the sandwich product. Tokens live in flat ambient space, not on the group: motors representing SE(3) are a 6-dimensional submanifold of the 8-dimensional even subalgebra of Cl(3, 0, 1), and tokens are unconstrained 16-dimensional multivectors. The geometry is in the algebra structure, not the token space.
6.4
Capsule networks
Matrix Capsules with EM routing [12] represent each entity by a 4 × 4 pose matrix updated by voting and EM routing. The pose matrices are unconstrained reals, not Lie group elements; the voting protocol provides soft training pressure toward affine-like composition, but no architectural constraint enforces a group structure. No Lie-algebra machinery, no exp/log, no equivariance theorem. Spatial Transformer Networks [13] use Aff(2) as a learnable warping parameter, not a token representation.
6.5
Frame-augmented (IPA)
AlphaFold’s Invariant Point Attention [14] attaches a frame Ti ∈ SE(3) to each vector token si . The attention score uses a squared-distance kernel between points expressed in pairs of local frames, which is SE(3)-invariant. This is the closest published precedent for the construction: a squared-distance kernel between geometric quantities. The limitation is that only the position part is scored this way. Rotation is handled separately via frame transformation, and the frame is auxiliary. The primary token remains the vector si . The construction unifies rotation and translation in a single quantity, wij = log(gi−1 gj ), and places the token on the group rather than in Rd . A recent instance of the same pattern, ActionFlow [15], carries SE(3) poses as metadata on feature tokens for robot policies, with a point-distance score.
6.6
LieTransformer
LieTransformer [16] performs self-attention over pairs of elements in G×Rd for an arbitrary Lie group, building on the LieConv [17] group lifting. Attention is computed on group-element pairs — the closest existing recipe to native group-attention. Two differences set it apart from the construction here. The features are (g, v) pairs, not bare group elements. And the kernel is a learned MLP on the relative-pose invariant, not the closed-form algebra norm; this is the fixed-versus-learned axis that Model C isolates in Section 5. The affine groups lie outside its reach for a separate, inherited reason: the LieConv lifting requires a surjective exponential, which Aff(n) lacks. The construction places the token strictly on G and applies on the principal-log chart of any matrix Lie group. The relative-pose log-map invariant itself appears earlier still, as the convolution-kernel coordinate of LieConv [17]. The cancellation-based equivariance argument is likewise anticipated by LieTransformer (the closed-form exp / log on SO(n)/SE(n) is classical); what is new here is the bare group-element token, the closed-form algebra-norm score, and the affine reach beyond surjective-exp groups.
6.7
Position-orientation networks (PONITA)
PONITA [18] is the closest published work to the SE(2) and SE(3) cases of the construction. It operates on the homogeneous space SE(n)/SO(n − 1) ∼ = Rn × S n−1 — positions equipped with a single orientation vector — and uses message passing with attention weight-shared over relative position-orientation pairs, derived from homogeneous-space theory. Provably SE(n)-equivariant by construction. Two differences. First, the token is a position-with-single-orientation pair, not a full frame on the group; the rotation block of se(n) is partially reduced. Second, the invariants used for weight-sharing are geometric quantities chosen per task (relative distance, relative orientation angles, alignments) rather than read off a single algebra log; the unified algebra-log form wij = log(gi−1 gj ) ∈ se(n) used here recovers these as special cases, see Section 3.1 (RoPE-class) and the IPA discussion above. The construction also extends naturally to Aff(2) where the position-orientation reduction is unavailable.
16
6.8
Lie-group decompositions for non-compact equivariance
Mironenco and Forré [19] target non-compact non-abelian groups including G+ = GL+ (n, R) and the orientationpreserving affine groups Rn ⋊ G+ , which contain Aff + (2). The exponential map for Aff(n) is not surjective, so the standard “lift to algebra, exponentiate” recipe fails. The paper’s contribution is to use the Cartan/polar decomposition to factor G+ globally as Pos(n) × SO(n), a symmetric-positive-definite part times a compact rotation part, on which Haar measure factors. The architecture is a group-equivariant CNN over images, not a transformer over framed primitives. The construction in this paper takes the same group as a target but uses a different mechanism — attention over invariants on the principal-log chart — and produces a different kind of model. Mironenco–Forré give a rigorous measure-theoretic treatment of the Aff(n) symmetry via the Cartan/polar decomposition; the construction here is the first to embed Aff(2) tokens in a transformer architecture. Reductive Lie Neurons [28] reach GL(n) with Lie-algebra-valued features in a general (non-attention) network, extending the non-compact reach beyond Mironenco–Forré’s convolution; neither is an attention construction on group-element tokens, so the affine case scored by a closed-form invariant attention norm remains open.
6.9
RoPE and geometric reframings
RoPE [20] uses SO(2) rotations applied to pairs of feature dimensions for positional encoding. The construction at Section 3.1 subsumes the kernel-form question by giving the score −λδθ2 /τ . This depends on the same invariant, but with quadratic rather than oscillatory dependence. LieRE [22] generalizes RoPE to higher-dimensional P Lie groups by learning a basis of skew-symmetric matrices Ai and applying the rotation R(p) = exp( i pi Ai ) to keys and queries; the token remains a feature vector, the Lie rotation is positional encoding acting on it. RiemannFormer [21] imports geometric vocabulary (tangent vectors, parallel transport, Lie-algebra positional encoding) while keeping tokens as vectors in Rd ; the paper explicitly states it avoids the continuous geometry. All three keep the vector-token ontology — the construction differs by actually placing the token on the manifold.
6.10
Concurrent and recent work (2024–2026)
Several 2024–2026 systems approach the same territory from the vector-token side and stop at the schema boundary; we group them by recency rather than tradition, as each extends a family already covered above. The Geometry-Aware Attention mechanism (GTA) [23] applies group representations to queries, keys, and values to align coordinate frames between multi-view tokens. The tokens remain vectors, and the group acts on them by ρ(g). GTA already uses the relative pose, transforming keys and values by ρ(gi gj−1 ), but it does so as an external representation action on a vector token, with no logarithm and no norm. The Clebsch–Gordan Transformer [24] builds global equivariant attention through irreducible representations and Clebsch–Gordan tensor products — the irrep tradition in its purest transformer form, restricted to compact groups by unitarity. The Vanilla Group Equivariant Vision Transformer [25] renders ViT modules (patch embedding, self-attention, positional encoding, sampling) equivariant through enforced inductive biases; tokens are feature vectors throughout. None of these crosses to token-as-group-element; in each, the limit we highlight tracks the vector-token ontology they retain. Two further 2025–2026 entries sit in the same frame: GRAPE [26] encodes position by a group action applied to feature tokens, and François & Ravera [27] reformulate attention through symmetry-reduced invariants on feature vectors. On the finite-group side, Group-Algebraic Tensors [29] make equivariance an intrinsic algebraic property via a ⋆G tensor algebra (Lean-4 formalised). The same move, treating equivariance as structure rather than something to enforce, takes the Lie-group form here, by a different mechanism.
6.11
Summary
To our knowledge, no published work places the token strictly on a matrix Lie group and scores attention by the algebra norm of the relative pose. The distinction from every prior method, including the 2024–2026 cluster above, is not the kernel choice but the token type. Every prior approach is a vector token with an external group action. Lie-Algebra Attention is a group-element token with no external action at all. The construction in this paper is, in that precise sense, new.
7
Discussion and Limitations
Chart restriction. The construction lives on the principal-log chart U ⊂ G on which log is single-valued. Some token pairs have a relative pose gi−1 gj near or on the chart boundary. Examples are a rotation by π on SO(3),
17
or an Aff(2) element whose linear part has a negative-real-axis eigenvalue. For these the score is undefined or numerically unstable. In Section 5.1 and Section 5.2 the data distribution naturally stayed off the boundary thanks to the constraints |ωh | < π/8 and ∥ωh ∥ ≤ π/8, respectively; in Section 5.3 we used a rejection sampler that discards step generators whose iterated linear part Akh for k ≤ 7 has an eigenvalue on the non-positive real axis. For aggressive use cases that frequently cross the boundary, two options apply: chart-switching to an alternate atlas, or the global form of Theorem 4, which uses ∆i ∈ G directly without the exp parametrization at the cost of giving up the closed-form algebra-norm score. The chart issue is well-understood in the Lie-group literature and standard fixes carry over; for this work, the chart restriction is a real constraint that the practitioner should be aware of but not a structural obstacle. Block-isotropy assumption. The block-weighted norm (2) reduces the K 2 entries of a full bilinear form on a K-block algebra to K block weights (one per block, isotropic within), motivated by the O(n)-symmetry of the Frobenius form on each block. A non-isotropic kernel would have more parameters and more freedom to fit. Whether the extra capacity helps is an empirical question. Model C (full MLP score on the same invariant wij , with 50 to 80× more score parameters than G) does not consistently outperform G across the three sequencecompletion experiments. It loses by 34% on SE(2) and stays level with G within seed noise on SO(3) and Aff(2), marginally ahead on SO(3). The order-of-magnitude larger score does not help. That tests the squarednorm form, not the block-isotropy reduction itself. The constant-step invariants are collinear (Section 5.4), so the blocks scale together. Neither G’s weights nor C’s kernel can exploit block shape here. A targeted ablation — the block-weighted form against a non-isotropic kernel on data whose invariants span the blocks — would isolate the block structure; we leave this for follow-up work. Symmetric score. The score sij = −∥wij ∥2λ /τ is symmetric in i, j, since on the chart wji = −wij and ∥−wij ∥λ = ∥wij ∥λ ; directional information enters only through the value pathway Vij = WV [hj ; wij ], where Vij ̸= Vji . For the permutation-invariant set and undirected-sequence tasks of this paper the symmetric score is appropriate, but a directed or autoregressive application would need an added asymmetric score term. Empirical scope. This preprint validates the construction empirically on SE(2), SO(3), and Aff(2) at the toy-task scale of sequence completion — one experiment per block-count level of Section 1, with the math for SO(2) and SE(3) in Section 3 and independently verified symbolically, and the spatial affine extension Aff(3) included similarly in Section 3 as the analogue of Aff(2). The Aff(3) instantiation is given in closed form: its reach into the 3D non-compact non-abelian affine regime is mathematically established here, and what is deferred is the empirical evaluation of 3D affine-token applications, not the construction’s ability to express them. Larger-scale and applied empirical extensions are deferred similarly; the release here makes the priority claim on the construction itself rather than on benchmark dominance.
8
Conclusion
We placed the token on the group: a token is a bare matrix Lie group element — a transformation, nothing attached to it and nothing acting on it from outside. Lie-Algebra Attention is what follows from that premise. The intrinsic invariant wij = log(gi−1 gj ) gives a canonical closed-form algebra-norm score, equivariance becomes a tautology, and the cocycle is automatic. The backbone stays a vanilla transformer with no equivariant machinery. We instantiated the construction in closed form for the six matrix Lie groups relevant to 2D and 3D spatial reasoning — SO(2), SE(2), SO(3), SE(3), Aff(2), Aff(3) — and validated it across SE(2), SO(3), and Aff(2), every block-count level of the construction: a learned MLP kernel with 50 to 80× more score parameters on the same invariant does not improve on the closed-form score, and equivariance holds at the numerical floor, five to twelve orders of magnitude tighter than a vector-token baseline. The contribution is not a new attention kernel but a different premise about what a token is; the canonical score is its readout, not a design choice. What the premise reaches that no prior attention method does is the affine full frame: Aff(2) and Aff(3), non-compact, non-abelian, with scale and shear. This lies beyond the reach of irrep methods, which have no non-trivial finitedimensional unitary irreps, and of surjective-exp methods. We demonstrate the Aff(2) case here and give the spatial Aff(3) case in closed form for follow-up work to evaluate.
References [1] N. Thomas et al. Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds. arXiv:1802.08219, 2018. [2] F. B. Fuchs, D. E. Worrall, V. Fischer, M. Welling. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. NeurIPS 2020.
18
[3] Y.-L. Liao, T. Smidt. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. arXiv:2206.11990, 2022. [4] Y.-L. Liao, A. J. Hoffman, S. C. Shen, A. Duval, S. W. Norwood, T. Smidt. EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers. arXiv:2604.09130, 2026. [5] I. Batatia et al. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields. NeurIPS 2022. [6] T. Cohen, M. Welling. Group Equivariant Convolutional Networks. ICML 2016. [7] M. Weiler, G. Cesa. General E(2)-Equivariant Steerable CNNs. NeurIPS 2019. [8] V. G. Satorras, E. Hoogeboom, M. Welling. E(n) Equivariant Graph Neural Networks. ICML 2021. [9] J. Brehmer, P. de Haan, S. Behrends, T. Cohen. Geometric Algebra Transformer. NeurIPS 2023. [10] P. de Haan, T. Cohen, J. Brehmer. Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers. AISTATS 2024. [11] S. Xu, D. Chen, K. Wong, C. Zhang, K. Fallah, R. Urtasun. Efficient Equivariant Transformer for Self-Driving Agent Modeling (DriveGATr). arXiv:2604.01466, 2026. [12] G. E. Hinton, S. Sabour, N. Frosst. Matrix Capsules with EM Routing. ICLR 2018. [13] M. Jaderberg, K. Simonyan, A. Zisserman, K. Kavukcuoglu. Spatial Transformer Networks. NeurIPS 2015. [14] J. Jumper et al. Highly Accurate Protein Structure Prediction with AlphaFold. Nature 596, 583–589, 2021. [15] N. Funk, J. Urain, J. Carvalho, V. Prasad, G. Chalvatzaki, J. Peters. ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matching. arXiv:2409.04576, 2024. [16] M. Hutchinson, C. Le Lan, S. Zaidi, E. Dupont, Y. W. Teh, H. Kim. LieTransformer: Equivariant Self-Attention for Lie Groups. ICML 2021. [17] M. Finzi, S. Stanton, P. Izmailov, A. G. Wilson. Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. ICML 2020. arXiv:2002.12880. [18] E. J. Bekkers, S. Vadgama, R. D. Hesselink et al. Fast, Expressive SE(n) Equivariant Networks Through WeightSharing in Position-Orientation Space. ICLR 2024. [19] M. Mironenco, P. Forré. Lie Group Decompositions for Equivariant Neural Networks. ICLR 2024. [20] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864, 2021. [21] Z. Ji. RiemannFormer: A Framework for Attention in Curved Spaces. arXiv:2506.07405, 2025. [22] S. Ostmeier et al. LieRE: Lie Rotational Positional Encodings. arXiv:2406.10322, 2024. [23] T. Miyato, B. Jaeger, M. Welling, A. Geiger. GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers. arXiv:2310.10375, 2024. [24] O. Howell et al. Clebsch–Gordan Transformer: Fast and Global Equivariant Attention. arXiv:2509.24093, 2025. [25] J. Fu, Q. Xie, D. Meng, Z. Xu. Vanilla Group Equivariant Vision Transformer: Simple and Effective. arXiv:2602.08047, 2026. [26] Y. Zhang, Z. Chen, Y. Liu, Z. Qin, H. Yuan, K. Xu, Y. Yuan, Q. Gu, A. C.-C. Yao. Group Representational Position Encoding. arXiv:2512.07805, 2025. [27] J. François, L. Ravera. Toward Manifest Relationality in Transformers via Symmetry Reduction. arXiv:2602.18948, 2026. [28] C. Kim, S. Zhao, M. Zhu, T.-Y. Lin, M. Ghaffari. Equivariant Neural Networks for General Linear Symmetries on Lie Algebras (Reductive Lie Neurons). arXiv:2510.22984, 2025. [29] P. Hoyos, S. Ubaru, D. Huh, V. Kalantzis, K. L. Clarkson, M. Kilmer, H. Avron, L. Horesh. Group-Algebraic Tensors: Provably-optimal Equivariant Learning and Physical Symmetry Discovery. arXiv:2605.20440, 2026.
19