arXiv CSopen access

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

· arxiv_cs

arXiv CS · Papers · License: Open Access

machine learning, deep learning, neural networks

C OMPLEX KDA: U NDERSTANDING AND E NHANCING THE E XPRESSIVITY OF K IMI D ELTA ATTENTION Julien Siems∗♡ , Riccardo Grazzi∗♢ , Korbinian Pöppel∗⃝ , Jaisidh Singh♠□ , Arber Zela△ , Timur Carstensen♡⃝ , Jenia Jitsev†♣Ψ⃝ , Frank Hutter♡O , Volkan Cevher△ , Antonio Orvieto□ , Aaron Klein⃝ Equal contribution, ♡ University of Freiburg, ♢ Microsoft Research, University of Tübingen, Zuse School ELIZA, □ MPI-IS Tübingen, ELLIS Institute Tübingen, △ EPFL, ⃝ OpenEuroLLM, † Jülich Supercomputing Center (JSC), ♣ LAION, Ψ Open-Ψ (Open-Sci) Collective, O PriorLabs [email protected] [email protected] [email protected]

∗

arXiv:2609.24797v1 [cs.LG] 21 Sep 2026

♠

A BSTRACT Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in [−1, 1] and the delta-rule coefficient β in [0, 2]. We call the resulting model Complex KDA (CKDA). It preserves KDA’s stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the state-tracking expressivity of DeltaProduct2 . We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on S3 , S4 , and periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior. Our code is open-source as are our models.

I NTRODUCTION

Linear recurrent neural networks (RNNs) offer efficient sequence modeling with linear scaling in sequence length and a fixed-size recurrent state. Their efficiency and expressivity depend on the structure of their state-transition matrices: diagonal transitions, as used in Mamba-1/2 (Gu & Dao, 2024; Dao & Gu, 2024), GLA (Yang et al., 2024a), and mLSTM (Beck et al., 2024), support fast computation, while non-diagonal transitions using the delta-rule (Schlag et al., 2021) introduce a rank-one correction that mixes information across state coordinates (Yang et al., 2024b; Peng et al., 2025; Hatamizadeh et al., 2026). Gated delta-rule models have become recurrent backbones of large language models, with two variants differing in how they parameterize the gate in the statetransition At = (I − βt kt kt⊤ ) Diag(αt ). Gated DeltaNet (GDN) (Yang et al., 2025) uses a scalar gate, αt = αt 1, and is adopted in several recent language models (Qwen Team, 2026; Qiu et al., 2026; Merrill et al., 2026b). Kimi Delta Attention (KDA) (Kimi Team, 2025) allows a separate gate value per channel and has likewise been adopted in recent models (Upstage Solar Team, 2026; Z.ai, 2026; inclusionAI, 2026; Kimi Team et al., 2026). Both retain diagonal-plus-rank-one transitions, motivating us to understand what the shift from a scalar to a full diagonal adds to their expressivity. We study this expressivity question through state tracking, which requires composing inputdependent updates over time, as in parity, modular addition or general permutation composition. 1

Rπ/2 p

x=0

HDp

+90◦

k p

Dp p = (1, 0)⊤

H = I − 2kk⊤

D = Diag(−1, 1)

Rπ/2 = HD

D = Diag( , 1)

Figure 1: Visualization of CKDA applying a rotation to a vector p within a single recurrent update (I − βkk⊤ ) Diag(−1, 1) by combining a signed diagonal gate with a Householder reflection (β = √ 2). k = (1, 1)⊤ / 2 allows a 90◦ rotation, while varying k allows any planar rotation angle. =1

= 0.5

= 0.25

= 0.5

= 1

0 1

D= I

1 0 1

Figure 2: Spectral view of the two-dimensional construction in Equation 2. For A = (I − 2kk⊤ )D with k = (cos θ, sin θ)⊤ and θ ∈ [0, π/2]. For the channel-wise gate D = Diag(α, 1), negative α allows a complex-conjugate pair, which moves toward the unit circle as α approaches −1 and recovers a rotation. For the scalar gate D = αI, the spectrum remains real (see also Figure 11). Equipped with increasingly complex orthogonal transitions, Linear RNNs can solve harder statetracking problems with one layer: 1D reflections for parity, 2D rotations for modular addition and higher-dimensional orthogonal representations for permutation composition. DeltaProductk (Siems et al., 2025) controls this complexity by composing k delta-rule transitions per token, each an identity-plus-rank-one matrix. In particular, one layer can solve modular addition via 2D rotations when k = 2, using two delta-rule updates per token but increasing the computational cost relative to DeltaProduct1 . We show that KDA’s channel-wise gate provides the structure needed to break the symmetry of the Householder–diagonal transition, making complex eigenvalues possible, whereas GDN’s scalar gate cannot break this symmetry. However, KDA’s standard nonnegative gate still restricts the transition to a real spectrum. We introduce Complex KDA (CKDA) by combining two existing range extensions: gate entries in [−1, 1] (Sarrof et al., 2024) and βt ∈ [0, 2] (Grazzi et al., 2025). As a consequence, CKDA can realize any 2D rotation while maintaining non-expansiveness and efficient recurrent computation. CKDA realizes planar rotations using gate entries of opposite sign to supply a coordinate reflection, which combines with the freely oriented Householder reflection at βt = 2 (Figures 1 and 2). Our main contributions are: • We characterize the spectrum of CKDA’s transitions and prove that every orthogonal diagonalplus-rank-one matrix is exactly a CKDA transition. We also study the expressivity of products of CKDA matrices (Section 4). • One CKDA layer tracks every finite subgroup of SO(3), including S3 , S4 , and A5 (Theorem 3). Three layers solve arbitrary finite-group word problems and, with β > 2, simulate weighted finite automata in polynomial precision (Theorem 5). These bounds match DeltaProduct2 and use one layer fewer than (Gated) DeltaNet. • We rule out one-layer S5 tracking for CKDA and DeltaProductk (k ≤ 3) with non-expansive transitions and finite reachability (Theorem 4). Finite reachability (the set of RNN states is finite) holds in most group-tracking constructions considered in the literature and simplifies the analysis. • Combining both extensions yields the strongest length extrapolation among the tested KDA range settings on S3 , S4 , and periodic waveform continuation. At 1.3B parameters and 100B tokens in language modeling, CKDA performs on par with KDA, while outperforming Transformers and other linear RNN architectures. Our implementation is a minor modification of KDA recurrence kernels from FLA (Yang & Zhang, 2024) and retains 96–97% of standard KDA’s throughput. 2

BACKGROUND & R ELATED W ORK

Linear RNNs. Linear recurrent neural networks process sequences through stacked layers with affine state updates. For one head, following Grazzi et al. (2025), we write Hi = A(xi )Hi−1 + B(xi ),

ŷi = dec(Hi , xi ),

i = 1, . . . , t.

(1)

Here Hi ∈ Rn×dv is the recurrent state; the learned maps A, B, and dec specify the transition applied to its columns, additive update, and output. Architectures differ mainly in the structure imposed on A. GLA (Yang et al., 2024a) uses diagonal transitions, while Mamba-2 (Dao & Gu, 2024) and mLSTM (Beck et al., 2024) use scalar-times-identity transitions within each head. Diagonal-plus-low-rank (DPLR) transitions permit channel mixing. DeltaNet (Schlag et al., 2021; Yang et al., 2024b) uses I − βi ki ki⊤ : for unit keys, βi = 0, 1, 2 give identity, projection, and Householder reflection (Householder, 1958; Grazzi et al., 2025), respectively. Gated DeltaNet (Yang et al., 2025) adds scalar decay, whereas KDA (Kimi Team, 2025) uses a channel-wise positive gate, Ai = (I − βi ki ki⊤ ) Diag(αi ); both retain diagonal-plus-rank-one transitions and efficient WYbased chunk-wise implementations. Related delta architectures introduce channel-wise learning rates and separate erase/write gates (Hatamizadeh et al., 2026), or asymmetric rank-one corrections (Peng et al., 2025). Complex and rotation-based recurrences. S4 (Gu et al., 2022) and LRU (Orvieto et al., 2023) use complex DPLR and complex-diagonal representations, respectively. Earlier RNNs combine nonlinear activations with unitary (Arjovsky et al., 2016) or Householder-parameterized (Mhammedi et al., 2017) orthogonal recurrent matrices, unlike the affine updates we consider. RotRNN (Biegun et al., 2024) uses rotation-based linear recurrences, while Orvieto et al. (2024) analyze the benefits of complex eigenvalues for finite-window memory reconstruction. DeltaProduct (Siems et al., 2025) realizes planar rotations using two Householder factors per token, requiring an additional rank-one update. Separately, higher-order and block-diagonal LRUs (Dubinin et al., 2026) enrich state mixing outside the delta-rule family. Selective RoPE (Movahedi et al., 2026) adds input-dependent rotations to gated attention, Mamba-3 (Lahoti et al., 2026) uses complex dynamics, and Adaptive Unitary SSMs (Karuvally et al., 2025) use input-dependent unitary transitions sharing a diagonalizing basis. MDN (Huang et al., 2026) adds an auxiliary momentum state, yielding second-order dynamics with complex-conjugate eigenvalues. Semidirect Fourier Delta Attention (Zhang, 2026) combines explicit phase/decay gates with delta updates and chunk-WY algorithms; its phases admit real 2 × 2 rotation implementations and realize cyclic counters even without delta updates. CKDA instead retains KDA’s first-order, real diagonal-plus-rank-one recurrence: signed gates αi ∈ [−1, 1]n (Sarrof et al., 2024) and βi ∈ [0, 2] (Grazzi et al., 2025) let coordinate and delta-rule reflections compose into noncommuting rotation families, without auxiliary recurrent states or explicit phase gates. See Table 3 for a comparison between CKDA, MDN, and SFDA. Formal languages and recurrent expressivity. Finite monoids recognize exactly the regular languages; non-commutative group problems require order-sensitive composition, with S5 linked to the computational complexity class NC1 through permutation branching programs (Barrington, 1986). We consider every group element as an input, not only generators. Under the finite-precision assumptions of Shakerinava et al. (2026), single-layer input-dependent complex-diagonal SSMs track exactly the finite abelian groups, excluding non-abelian tracking despite complex eigenvalues. Alternative transitions use bilinear interactions (Ebrahimi & Memisevic, 2026), selective SSM parameterizations for automaton emulation (Terzic et al., 2025a), or fixed-point iterations (Movahedi et al., 2025). Our multilayer results adapt finite-state and rational weighted-automaton constructions (Siems et al., 2025; Peng et al., 2025; Merrill et al., 2026a), replacing the two-layer DeltaNet clock with one CKDA layer. Circuit-complexity bounds depend on precision and evaluation assumptions (Merrill et al., 2024; 2026a); algebraic analyses show that discretization and evaluation order can change expressivity (Nowak et al., 2026). Recent work highlights a gap between the expressive capacity of linear RNNs and their robustness in long-horizon state tracking, emphasizing limitations in error correction that can allow perturbations to accumulate and compromise state representations (Dankowiakowski & Ronca, 2025; Chung et al., 2026). Beyond abstract group-word benchmarks. Siems et al. (2026); Merrill et al. (2026b) study permutation tracking in next-token-style code traces; Shin et al. (2026) report improved extrapolation when permitting negative transition eigenvalues in their action-conditioned video Shell Game. 3

M OTIVATION : F ROM S YMMETRY TO ROTATION

We begin by isolating the mechanism through which channel-wise signed gating changes the transition geometry: We show that scalar gates commute with every matrix, whereas signed diagonal gates can combine with the Householder update to produce rotations. Symmetry. A KDA state-transition matrix A = Hk D, with Hk := I −βkk⊤ and D := Diag(α), is the product of two symmetric matrices. A is symmetric if and only if the factors commute, since A⊤ = DHk . Entrywise, the commutation condition Hk D = DHk becomes (Hk D − DHk )ij = βki kj (αi − αj ) = 0

for all i, j.

Scalar Gate. For Gated DeltaNet, αi = α for every coordinate, so the commutation condition above is always true. Hence A = α(I − βkk⊤ ) is symmetric and has a real spectrum. Moreover, over several recurrent steps the scalar gates factor entirely out: let At = αt (I − βt kt kt⊤ ), then ! T Y AT · · · A1 = αt I − βT kT kT⊤ · · · I − β1 k1 k1⊤ . t=1

Thus extending the scalar gate to [−1, 1] can change the overall scale and introduce a global sign, but it cannot contribute an additional independently oriented transformation (Figure 2 bottom row). Diagonal Gate. For a channel-wise gate, differences αi − αj need not vanish. Whenever β ̸= 0 and the key has nonzero components on two coordinates with different gate values, the two symmetric factors do not commute and the resulting transition is nonsymmetric. Noncommutation alone, however, is not sufficient to produce a non-real spectrum. For strictly positive gates, Hk D is similar to the symmetric matrix D 1/2 Hk D 1/2 , so all its eigenvalues are real. The same conclusion holds when some gates are zero, by continuity. Hence standard nonnegative KDA still has a real spectrum. As we show next, allowing the diagonal gate to change sign removes this restriction. We see the resulting geometry in a simple 2D case. Consider the KDA transition with β=2 given by Aα,θ = Hk Dα ,

Hk := I − 2kk⊤ ,

k = (cos θ, sin θ)⊤ ,

Dα := Diag(α, 1).

Expanding the product gives Aα,θ =

−α cos 2θ −α sin 2θ

− sin 2θ . cos 2θ

(2)

Since Aα,θ ∈ R2×2 , its eigenvalues are non-real precisely when its discriminant is negative, ∆ := tr(Aα,θ )2 − 4 det(Aα,θ ) = (1 − α)2 cos2 (2θ) + 4α < 0. Hence, for α ≥ 0, the spectrum is necessarily real, whereas for α < 0, a complex-conjugate pair can exist. √ Whenever the eigenvalues are non-real, their product is det(Aα,θ ) = −α, and hence |λ| = −α. Thus, as α moves from 0 toward −1, the complex pair moves outward toward the unit circle (see Figure 2). At the endpoint α = −1, we have Diag(−1, 1) = I − 2e1 e⊤ 1 = He1 , and hence A−1,θ = Hk He1 . The transition is therefore the composition of two reflections whose mirror lines differ by an angle θ. Their composition is a rotation by 2θ. This perspective extends coordinate by coordinate. A diagonal matrix can be decomposed as ⊤ Diag(α) = I − (1 − α1 )e1 e⊤ I − (1 − α2 )e2 e⊤ 2 · · · I − (1 − αn )en en . 1

(3)

Thus each factor is an axis-aligned generalized Householder transformation with normal ei and learning rate βi = 1 − αi . This can also be viewed as a special case of the generalized Householder composition studied by Siems et al. (2025). In particular, when αi = −1, the corresponding factor is an exact coordinate reflection. Hence a KDA state transition At = (I − βt kt kt⊤ ) Diag(αt ) combines one freely oriented generalized Householder transformation with n axis-aligned ones. CKDA. We define CKDA as KDA with αt ∈ [−1, 1]n and βt ∈ [0, 2]; our implementation uses βt = 2σ(bt ) and a signed gate based on rt,i = 2σ(at,i ) − 1, following Sarrof et al. (2024) and Grazzi et al. (2025) with the magnitude floor and backward convention specified in Appendix E. 4

Table 1: Expressivity of linear recurrent models CKDA

GDN

(I − βkk⊤ )D(α)

α(I − βkk⊤ )

Param. ranges

β ∈ [0, 2], α ∈ [−1, 1]d α ∈ [0, 1], β ∈ [0, 2]

Complex pairs

≤ 1 (Thm. 8) 0 if β ≤ 1 or α ≥ 0

S2 (parity) Zn , Dn (n > 2) S4 , A5

✓ 1 via [U3]† ✓ 1 (Thm. 3)† ✓ 1 (Thm. 3)†

✓ 1 via [U3] ✓ 2[D7] × 1 FP[U2] ✓ 4[D1] × 1 FP[U2]

Sn (n ≥ 5)

✓ 3 (Thm. 5) × 1 FR (Thm. 4)

Regular languages

✓ Lq via [D2] ✓ 3 (Thm. 5) allow β > 2 ✓ 3 (Thm. 5)

(unstable) WFAs over Q (PP)

Gated DeltaProduct Qk ⊤

D(α) − k(v ⊙ k)⊤

α ∈ [0, 1], βj ∈ [0, 2]

α ∈ [0, 1]d , v ∈ [0, 2]d

≤ ⌊k/2⌋

j=1 (I − βj kj kj )

✓ 1 (k ≥ 1) via [U3] ✓ 1 (k ≥ 2) [D7] ✓ 1 (k ≥ 2) [D4] ✓ 1: k ≥ n − 1 [D1] ✓ 4[D1] × 1 FP[U2] ✓ 3: 2 ≤ k ≤ n − 2 [D1] × 1 FR: k ≤ 3 (Thm. 4) ✓ Lq [D2] ✓ 4 via [M11] β ∈ [0, 4] ✓ 4 [M11] β∈R

β ∈ [0, ∞)

✓ Lq (k ≥ 1) [D2] ✓ 3 (k ≥ 2); 1 (k ≥ 4q ) β ∈ [0, 4] via [RL3, M11] ✓ 3 (k ≥ 2); 1 (k ≥ Bq ) β ∈ R via [M11, ML12]

RWKV-7 / GDN-2

✓ 1 [RL1]‡ ✓ 2[D7]‡ × 1 FP[U2] ✓ 4[D1]‡ × 1 FP[U2] ✓ 4[D1]‡ × 1 FP[U2] ✓ Lq via [D2]‡ ✓ 4 [R3] ✓ 4 [M6] unrestricted

✓ L: depth L is sufficient; × L A: depth L is impossible under A. Lq : a sufficient finite depth depending on the language’s q -state DFA. Unstable / amber: expansive transitions allowed. † CKDA: β ∈ {0, 2}, α ∈ {±1}d (α = 1 for parity). ‡ RWKV-7/GDN-2 constructions use D(α) = γI , v = γβ1, with γ ∈ [0, 1], β ∈ [0, 2] (stable: ∥A∥2 ≤ 1); reflections use γ = 1, β = 2. RWKV-7/GDN-2: unit keys, nonnegative gates (RWKV-7’s factor c = 2 is absorbed into v ). Both allow expansive transitions by default; GDN-2 entries use its extended erase range [0, 2]d . D(α) = Diag(α); q : automaton size; Bq = 8q 2 + 5q + 1 (2q + 1 coordinates). Group inputs: all elements; complex pairs count multiplicity per head. FP: fixed precision (exact datatypes). PP: polynomial bit length over a fixed algebraic number field (App. B.4). FR: per-head finite reachability under exact affine updates (App. B.3). U (Grazzi et al., 2025), D (Siems et al., 2025), R (Peng et al., 2025), M (Merrill et al., 2026a): [D1]/[RL1]: Theorem/Lemma 1; “via”: derived bounds. Comparison details: App. D.5. (a) Orthogonal columns a1 a2 A = [a1 , a2 ]

k⊥

(b) Reflect both columns e2 a1

−e1

H = I − 2kk⊤

−e1

HA = S S = Diag(−1, 1)

Figure 3: Intuition for Theorem 1 in 2D. A single reflection aligns a 2D orthogonal frame with the coordinate axes, giving HA = S and hence A = HS.

S TRUCTURE AND S PECTRUM OF C OMPLEX KDA

The planar construction shows how signed gating enables rotations within a rank-one diagonal-pluslow-rank (DPLR) transition. In this section, we study CKDA in arbitrary dimensions. Our first finding is that CKDA captures the entire orthogonal rank-one DPLR (DPR1) family. Theorem 1 (DPR1 and CKDA). Every orthogonal DPR1 matrix A = D + uv ⊤ ∈ Rn×n with diagonal D can be written in the form A = (I − 2kk⊤ )S, ∥k∥2 = 1, S = Diag(si ), si ∈ {−1, +1}. We call this form a signed-Householder matrix, i.e. a CKDA with αi ∈ {+1, −1}, β = 2. Geometric intuition in 2D (Figure 3). The columns of an orthogonal matrix A form a perpendicular pair of unit vectors. Choose a reflection H across a line through the origin that sends the first column onto the first coordinate axis. Since reflections preserve lengths and perpendicularity, the second column must then lie on the second coordinate axis. Thus HA = S for a diagonal sign matrix S, and H 2 = I gives A = HS. In higher dimensions, aligning one column does not automatically align the others; the DPR1 structure guarantees that a suitable single reflection still aligns the entire frame. Thus replacing KDA’s structured rank-one term by a non-symmetric uv ⊤ , such as in RWKV-7, adds no orthogonal transitions. The next result generalizes the 2D case of the motivation section to higher dimensions: each CKDA transform can be viewed as an element-wise scaling followed by a transform which can be a rotation only in a 2D subspace. 5

(12)

(23)

(13)

Figure 4: Example of the S3 permutation task, analogous to the shell game with 3 hidden objects. Proposition 2 (Sign-magnitude decomposition). Let ∥k∥2 = 1, β ∈ [0, 2], and αi ∈ [−1, 1]. Write e Diag(|α|), where A e = (I − βkk⊤ )S and S = Diag(sign α), A = (I − βkk⊤ ) Diag(α) = A choosing sign(0) = 1 for this factorization. Let E± = ker(S ∓ I) and decompose k = k− + k+ with k± ∈ E± , ∥k− ∥ = cos θ, and ∥k+ ∥ = sin θ, θ ∈ [0, π/2]. When both components are nonzero, e to U = span{k− , k+ }, in the orthonormal basis k− /∥k− ∥, k+ /∥k+ ∥, is the restriction of A β cos2 θ − 1 −β sin θ cos θ B(β, θ) = . β sin θ cos θ 1 − β sin2 θ e agrees with S. The block is a rotation by 2θ when β = 2, and its eigenvalues are non-real On U ⊥ , A exactly when β 2 cos2 (2θ) < 4(β − 1). If either component vanishes, Ã has only real eigenvalues. Therefore, after the initial coordinate-wise scaling, a 2D rotation with complex eigenvalues can occur only if (i) at least two gate coordinates differ in sign, (ii) the key vector spans coordinates with opposite signs and (iii) β > 1. The 2D example in the previous section fits exactly these criteria. The analysis of the full CKDA transition adds complexity and requires a separate argument: Theorem 8 in the appendix proves that any CKDA matrix also has at most one non-real conjugate eigenvalue pair, requiring both β > 1 and a negative gate entry. Thus both range extensions are required for non-real eigenvalues. Moreover, this restriction extends beyond CKDA: every nonexpansive DPR1 matrix has at most one non-real conjugate eigenvalue pair on the unit circle, counted with algebraic multiplicity (Theorem 9). Products of CKDA transitions can represent any square non-expansive matrix, despite the one-plane restriction on each individual transition. Generalized Householder products already have this universality (Grazzi et al., 2025, Prop. 1); Proposition 10 shows that CKDA needs at most max{1, 2n − 2} factors in dimension n. For orthogonal matrices, max{1, n − 1} factors suffice and this bound is sharp. In contrast, products of RoPE and Selective RoPE rotations remain block-diagonal rotations in the same fixed coordinate planes (Su et al., 2024; Movahedi et al., 2026).

5 S TATE -T RACKING E XPRESSIVITY Tracking a state-transition system. Let S be a state space with initial state s0 and an update Ta : S → S for each input symbol a. A recurrent model tracks this system if a fixed decoder f recovers its state from the model’s hidden state ht for every input sequence: st = Tat (st−1 )

(t ≥ 1),

f (ht ) = st

(t ≥ 0).

Several hidden states may decode to the same target state. Systems include finite-group products (see Figure 4), deterministic automata, and weighted finite automata (WFAs). Our constructions allow sufficiently expressive feed-forward decoders; our arithmetic and precision conventions are defined in Appendix B.4. A contextualized summary of our results is in Table 1. Single-layer expressivity. Consider a finite group G. A faithful orthogonal representation assigns each group element a distinct orthogonal matrix, so that composing group elements corresponds exactly to multiplying their matrices. We call a construction that uses these matrices directly as its transitions a realization, and write d for their dimension. Tracking does not require a faithful representation: a many-to-one decoder can map distinct hidden states to the same group element, and the input transitions need not themselves form a representation. Any finite group is isomorphic to a subgroup of a permutation group and can be realized by one Linear RNN layer using permutation matrices as transitions. However, CKDA cannot model arbitrary permutations in a single transition1 . Theorem 3 (Single-layer finite-group expressivity). A single CKDA layer (one head) tracks every finite group isomorphic to a subgroup of SO(3). Specifically, it realizes every finite cyclic (Zn ) and dihedral group (Dn ) in d = 2 and A4 , S4 in d = 3, and tracks A5 in d = 4. These constructions use orthogonal transitions and a fixed exact datatype. 1 We consider the case where inputs range over all of G. If inputs are restricted to identity and swaps, one-layer DeltaNet suffices (Grazzi et al., 2025).

Proof sketch. The planar rotations from the motivation section, together with reflections, give the cyclic and dihedral (such as S3 ) groups. Every three-dimensional rotation is a product of two Householder reflections, but, apart from the identity and 180◦ rotations, CKDA requires a rotation axis with a zero coordinate. Reorienting the cube satisfies this condition for S4 and its subgroup A4 , whereas no orientation works for the icosahedral rotations of A5 . In four dimensions, however, A5 can be tracked using a many-to-one decoder. See Sections C.1 to C.4 for the constructions and proofs. Limits of a single CKDA layer. Increasing dimension and arbitrary decoders do not remove every obstruction. The next result shows that non-expansive transitions with only one possible complex eigenvalue pair, such as CKDA (Theorem 8) and (Gated) DeltaProductk with k ≤ 3, cannot track S5 in one layer, even with any finite number of independent heads. Section C.6 gives the proof. Theorem 4 (Spectral obstruction to S5 tracking). Suppose every head transition A satisfies ∥A∥2 ≤ 1 and has at most one non-real conjugate eigenvalue pair on the unit circle, counted with algebraic multiplicity. Then a single recurrent layer with any finite number of independent heads cannot track S5 when the updates of each head reach only finitely many states. Proof sketch. Finite reachability and non-expansion let us remove the additive terms and restrict each head to orthogonal updates, preserving its spectral bound. The resulting matrices generate a finite group that maps onto S5 through the decoder. From a five-cycle and its conjugate by a transposition, we construct two transition-matrix products R1 and R2 whose blocks in each head are either identities or planar rotations with the same angle and order divisible by five. Their commutator C = R1 R2 R1−1 R2−1 satisfies C 10 = I by Lemma 23. Yet it decodes to a three-cycle, whose tenth power is not the identity, a contradiction. Finite reachability is natural for exact group tracking: all constructions considered here and in the cited results satisfy it (Section B.4). It enables the orthogonal reduction, simplifying the analysis without requiring a unique hidden state per group element. Multi-layer expressivity. The same planar rotations that strengthen one-layer tracking also reduce the depth needed to simulate general state-transition systems. Theorem 5 (Multi-layer expressivity). Three CKDA layers solve every finite group-word problem with a fixed exact datatype. If β > 2 is allowed, three layers also recognize every regular language with a fixed exact datatype and compute every WFA over Q in polynomial precision using exact arithmetic over a fixed algebraic number field containing the clock and normalized-key parameters (Sections B.4 and D). We adapt the existing clock–buffer–accumulator construction (Peng et al., 2025; Siems et al., 2025; Merrill et al., 2026a). A clock tracks position modulo a fixed period to schedule updates. The buffer factors each block transition; the accumulator applies one factor per step, and a completion readout corrects the delay. CKDA implements the clock with a planar rotation in one layer, saving one layer relative to DeltaNet/GDN which instead uses 4 layers for the same result.

E XPERIMENTS

We evaluate whether CKDA’s added expressivity improves state tracking and periodic waveform extrapolation, and assess its language-modeling performance and computational efficiency.

KDA, Triton CKDA, Triton

Throughput (k tok/s)

Implementation. KDA stores gate magnitudes in logspace, so signed gates require separating signs from magnitudes. We implement the same sign transformation in three ways: (1) compiled PyTorch operations that absorb cumulative signs into keys and queries, without changing kernels (gauge); (2) Triton (Tillet et al., 2019) kernels that fuse these signs into normalization and its backward pass; and (3) a TileLang-backed (Wang et al., 2026) hybrid that combines these Triton changes with TileLang backward kernels. The kernel implementations retain approximately 96–97% of KDA throughput (Figure 5); Section E gives the derivation, implementation details, and benchmark protocol. The ungated DeltaProduct2 baseline achieves competitive throughput while performing two delta-rule updates per token, each

2600 2400 2200 2000 1800

16×2k

KDA, TileLang CKDA, hybrid

8×4k

CKDA, gauge DeltaProduct 2 , tuned

4×8k

2×16k

Batch × length (32k tok/step)

Figure 5: Forward–backward kernel throughput on an H100 in BF16, with 16 heads, dk = dv = 128, and 32k tokens per step. DeltaProduct2 uses no forget gate. Implementation and timing details are in Section E.

Scaled accuracy

1.0

0.5 0.0

128

256 384 Sequence length

512 1

KDA [0, 1], [0, 2]

CKDA

128

256 384 Sequence length

KDA [ 1, 1], [0, 1]

512 1

KDA [0, 1], [0, 1]

128

DeltaProduct2

256 384 Sequence length

512

CKDA theory init

abc

2 1

7 11

Group element

(b) Gate α, head 11 ( ) ( ) e

abc

10 15

Group element

−1

1.0

Im(λ)

(a) Rate β by head ( ) ( )

Gate coordinate

Head

Cumulative variance

Figure 6: One-layer KDA variants on the S3 , S4 , and A5 word problems, by the eigenvalue range allowed for the diagonal and for the Householder component. Combining a signed gate with a reflection-enabled Householder update gives the strongest extrapolation on S3 and S4 among the tested KDA range settings; S4 accuracy still declines at long lengths. The A5 result uses the separate theory-initialized setup described in the text (see Figure 16 for baseline results).

0.5 0.0

2 3 4 Components

(d) Spectrum, head 11 e (ab) (abc)

target e±2πi/3

−1

0 Re(λ)

Figure 7: Learned CKDA transitions on the S3 word problem. Here e, (ab), and (abc) denote the identity, transpositions, and 3-cycles, respectively. (a) Learned update strength β by head; head 11 approaches the reflection limit β = 2. (b) Its channel-wise gate becomes nearly sign-valued, with coordinates close to ±1. (c) PCA of the keys: the first three principal components explain approximately 95% of the variance. (d) The resulting transition spectra show complex eigenvalues. The learned model recovers the mechanism predicted by our theory. with its own value: its measured throughput includes processing an additional value input relative to CKDA. This is due to its identity-plus-rank-one factors, which avoid CKDA’s channel-wise gating computations; the gated variant adds only scalar decay. We therefore view CKDA as an alternative way to obtain the expressivity similar to DeltaProduct2 , without claiming inherent efficiency gains over it. We also use spread gate initialization, with approximately equal proportions of positive and negative entries, alongside the original, positive-only standard initialization (In Appendix G.1, we also propose a variant for β). Finally, the SiLU activation inherited from DeltaNet (Yang et al., 2024b) on key and query projections can bias keys toward positive entries (Section F). We remove SiLU for KDA state-tracking experiments and ablate this choice for language modeling. 1. State-Tracking. We train one-layer models on S3 , S4 , and A5 group word problems (Merrill et al., 2024; Terzic et al., 2025b; Movahedi et al., 2025), predicting cumulative products from inputs spanning each group. Training lengths reach 32; we report the best of three seeds and scale accuracy from chance (0) to perfect (1). Training details are in Appendix G.2. CKDA (α ∈ [−1, 1]n , β ∈ [0, 2]) extrapolates well on S3 and S4 while other settings of KDA fail (Figure 6; Figure 16 in Appendix I), consistent with Theorem 3. Extending only the gate or β yields long-length S3 scaled accuracy near 0.2, matching parity-only discrimination of the two A3 cosets. A successful CKDA head learns β ≈ 2, nearly sign-valued gates, and complex-conjugate eigenvalues near the unit circle (Figure 7), recovering the mechanism of Section 3 and proposition 2. Standard training fails to learn A5 , whereas a smaller, fully trainable model initialized near our quaternion construction (Appendix C.4.2) achieves length extrapolation. Training retains ordinary next-state cross-entropy and a standard MLP readout, consistent with representation discovery being an optimization obstacle. The architecture and training schedule also differ, so this comparison does not isolate the effect of initialization. 2. Audio continuation. To test whether complex eigenvalues help preserve phase beyond symbolic state tracking, we train single-layer models to continue a synthetic periodic groove after a halfbar cue, followed by zero inputs. Whereas the autoregressive audio experiments of Mamba-1 (Gu & Dao, 2024) and SaShiMi (Goel et al., 2022) use prediction feedback that can induce nonlinear state dynamics, we compute the continuation in a single forward pass without output feedback. 8

(b) Numerical comparison

KDA α 2 [0, 1], β 2 [0, 1] //

KDA α 2 [−1, 1], β 2 [0, 2] //

Causal Transformer //

GRU

264 8

Sequence step (broken axis) KDA: α 2 [0, 1], β 2 [0, 1] KDA: α 2 [0, 1], β 2 [0, 2]

264

Waveform MSE

(a) Waveform continuations 10−1

10−9

KDA: α 2 [−1, 1], β 2 [0, 1] KDA: α 2 [−1, 1], β 2 [0, 2]

136

200

264

Sequence length

GRU Causal Transformer

Figure 8: Periodic waveform continuation with single-layer models. (a) Predictions shortly after the half-bar cue and far beyond the training horizon; gold marks the cue, grey the target, and dark blue the prediction. (b) Waveform MSE versus sequence length; the dashed line marks the maximum training length (136). CKDA extrapolates accurately where the other KDA variants and causal Transformer fail, while the GRU achieves the lowest error. Table 2: Language modeling and zero-shot common-sense reasoning at 1.3B parameters on 100B tokens of FineWeb-Edu in comparison to values from Hatamizadeh et al. (2026). ppl ↓

accuracy (%) ↑

recall (%) ↑

Model

Wiki. LMB. LMB. PIQA Hella. Wino. ARC-e ARC-c OBQA SIQA BoolQ

Avg. SQuAD SWDE

Recurrent models Mamba-2 Gated DeltaNet KDA Mamba-3 (SISO) Mamba-3 (MIMO) Gated DeltaNet-2 KDA, bounded gate (ours) CKDA (ours)

FDA

16.79 16.40 16.81 16.30 16.45 15.90 15.73 15.78

12.38 11.89 11.68 12.99 11.66 11.41 10.53 10.08

45.24 49.62 48.13 45.06 47.82 48.09 50.46 51.66

72.58 72.31 72.09 72.31 72.36 72.80 72.85 73.12

55.51 56.50 55.75 55.58 56.49 56.84 59.32 59.23

55.33 56.75 55.72 56.20 55.78 57.85 61.33 58.88

70.68 68.81 70.83 70.45 72.38 72.43 73.23 73.99

35.26 35.15 35.92 34.56 38.07 38.23 38.40 39.08

31.00 30.20 30.40 31.00 30.00 31.60 28.80 28.60

40.63 40.53 40.99 41.76 40.89 40.58 41.35 41.61

60.19 58.78 60.67 55.90 57.74 59.54 61.10 60.34

51.82 52.07 52.28 51.42 52.39 53.11 54.09 54.06

– – – – – – 38.17 39.68

– – – – – – – – – – – – 49.59 30.40 48.42 31.49

Attention or hybrid models Transformer (SWA-hybrid) Mamba-2 (hybrid) Gated DeltaNet (hybrid) KDA (hybrid) Mamba-3 (SISO, hybrid) Mamba-3 (MIMO, hybrid) Gated DeltaNet-2 (hybrid) KDA + attn 3:1 (ours) CKDA + attn 3:1 (ours)

19.22 17.46 16.00 16.01 15.54 15.81 15.62 15.04 15.34

13.72 11.29 10.82 10.66 10.65 10.92 10.43 9.93 9.90

48.32 48.05 48.71 49.21 49.19 49.82 50.90 51.60 52.57

70.21 71.47 70.06 71.06 71.01 71.98 72.20 73.88 73.56

56.12 57.52 57.50 56.89 58.75 58.19 58.46 59.57 59.68

55.85 56.17 56.83 57.77 57.30 57.06 58.56 60.62 60.14

69.23 70.50 70.41 71.59 70.54 70.54 71.89 72.69 72.56

33.84 34.73 35.15 35.07 36.35 38.48 36.69 37.20 36.86

25.00 29.80 30.60 30.00 32.00 29.40 33.00 27.00 28.20

39.74 40.35 40.97 40.53 41.20 40.99 41.50 42.43 42.58

59.42 59.31 60.00 62.03 57.86 57.98 62.57 60.31 63.15

50.86 51.99 52.25 52.68 52.69 52.72 53.97 53.92 54.37

– – – – – – – 42.69 37.77

– – – – – – – – – – – – – – 64.81 64.79 63.91 58.89

The task requires inferring phase from the cue and maintaining a periodic continuation after the cue ends. Among the four KDA range settings, only CKDA with both range extensions accurately continues the waveform (Figure 8). At length 264, beyond the maximum training length of 136, it retains 38.1 dB SNR, while the causal Transformer falls to 2.8 dB. Audio samples are available online. See Section G.3 for experimental details. These results connect learned rotational dynamics to periodic extrapolation, although the GRU (Cho et al., 2014), a nonlinear RNN, remains more accurate on this task. 3. Language modeling. We first train small language models of 340 million parameters on 15 billion tokens from the Nemotron-CC (Su et al., 2025) dataset, using CKDA as the transformer token mixer along with softmax/dense attention, DeltaProduct, GDN, and KDA as baselines. For architectures that introduce more projections, we reduce the latent dimension of the SwiGLU MLP by 256 to ensure comparable parameter counts. Detailed configurations are provided in Table 4. In the Nemotron-CC block of Table 6, CKDA with α ∈ [−1, 1], β ∈ [0, 2], and spread initialization achieves the highest observed average downstream accuracy: 52.30%, versus 51.32% for standard KDA. It does not have the lowest validation perplexity. In the small-model FineWeb ablation at 45B tokens, CKDA with gate and β spread achieves the highest observed average downstream accuracy of 52.21%, compared with 51.85% for KDA without SiLU on the keys (Table 6); the differences are modest and configuration-dependent. Validation curves are provided in Figure 18. Separately, we train 1.3B-parameter models on 100B tokens of FineWeb-Edu (Lozhkov et al., 2024), following the training recipe of Hatamizadeh et al. (2026). Our recurrent and hybrid CKDA models achieve similar average downstream accuracy to the corresponding KDA controls and the reported Mamba3 (Lahoti et al., 2026) and GDN-2 (Hatamizadeh et al., 2026) baselines (Table 2). The hybrid 9

KDA

CKDA

KDA + attn 3:1

CKDA + attn 3:1

0.00

N = 47M

N = 124M

N = 302M

N = 588M

N = 983M

N = 1.7B

0.02

val loss (nats)

0.04 0.06 0.00 0.02 0.04 0.06 6B

12B

20B

30B

50B 6B

12B

20B

30B

50B 6B

12B

20B

30B

50B

Layer

Figure 9: Advantage of CKDA recurrent and hybrid variants against the Transformer baseline (including QK-Norm, similar to Ajroldi et al. (2026)) in nats across scales. Grey area is the approximate noise floor. See Appendix H for a detailed scaling analysis of these numbers. 23 20 15 10 5 0

(a) Pr(α < 0)

20%

(b) Pr(β > 1)

50%

20 40 60 80 100 0%

10%

20 40 60 80 100 0%

Training tokens (B)

Figure 10: Extended-range use in non-hybrid CKDA 1.3B with standard initialization: per-layer fractions of negative gates (a), rates β > 1 (b), and transitions with complex eigenvalues (c). All three emerge during training. Negative gates occur most frequently in the early layers while β > 1 occurs in every layer, but particularly in the first and second. Colors use square-root scaling. Hybrid CKDA results in Figure 19, statistics over the final checkpoint across all layers in Figure 20. recipes differ: ours use full gated attention (Qiu et al., 2025) without RoPE (Kazemnejad et al., 2023) at a 3 : 1 recurrent-to-attention ratio, following Kimi Team et al. (2026), whereas the reported hybrid baselines use a 1 : 1 ratio with sliding-window attention (Sun et al., 2026). Our KDA controls also use the safe sigmoid gate of Kimi Team et al. (2026), which may contribute to the differences from the KDA results as reported in Hatamizadeh et al. (2026). Comparing architecture scaling laws trained on Nemotron-CC, CKDA (and KDA) outperform a Transformer baseline across scales (Ajroldi et al., 2026), see Figure 9. We don’t see a crossing point favoring Transformers going to larger over-training ratios (data to model parameter ratio) as observed for xLSTM in Beck et al. (2026), although small model behavior indicates a slight advantage reduction going to such regimes. To test whether the mechanism of Section 3 also emerges in language modeling, we track the fraction of negative gate entries, fraction of β > 1, and of complex transitions in every layer of CKDA during training of the 1.3B parameter models (Figure 10). The results are on our validation split of FineWeb-Edu at sequence length 4096. We find that despite the standard initialization keeping the gates close to 1, the model learns to use the negative gates early on during training primarily in the first layers. Similarly, the model learns to use the extended β range, first in the initial layers of the model but then also in deeper layers. The combination leads to complex eigenvalue pairs in the state-transition primarily in the first two layers of the trained model. We leave a mechanistic analysis of how the gate and β are being used in the model for future work.

C ONCLUSION

We showed that signed channel-wise gates and β ∈ [0, 2] enable planar rotations within a single non-expansive diagonal-plus-rank-one KDA transition. CKDA captures every orthogonal matrix of this form and strengthens state-tracking expressivity. Among tested KDA settings, it extrapolates 10

best on S3 , S4 , and periodic waveforms, with comparable downstream accuracy at 1.3B parameters and near-baseline throughput. Learned transitions exhibit the predicted mechanism in both state tracking and language modeling. Limitations. CKDA remains constrained by its rank-one transition structure: a non-expansive transition with a rank-one DPLR correction supports at most one persistent complex-conjugate eigenvalue pair. Our expressivity results also do not imply learnability. Notably A5 is representable but was not learned from random initialization under our standard setup. In addition, the general WFA construction requires β > 2, sacrificing guaranteed non-expansiveness, and uses exact arithmetic over a fixed algebraic number field. CKDA offers an alternative route to the expressivity capabilities established for DeltaProduct2 through signed channel-wise gating and a single rankone correction. This structural distinction does not establish an inherent computational advantage over (Gated) DeltaProduct2 , and the DeltaProduct2 baseline achieves competitive throughput in our kernel benchmarks. Future Work. CKDA preserves the structure of the additive updates, leaving the structure of the input to the recurrence untouched. Future work could combine CKDA with the separate gates of GDN-2 (Hatamizadeh et al., 2026) that control the additive term and the forgetting of the previous state. While we show that the extended gate and β ranges are learned to be used during training, their functional roles remain unclear.

AI U SE S TATEMENT The central idea of extending KDA’s gate to signed values and combining it with an extended range for the Householder coefficient originated with the authors independently of AI assistance after reading the Kimi K3 report. This included the hypothesis that the combination could produce complex eigenvalues, which was subsequently developed with assistance from Claude Opus 4.8. The change of variables described in Appendix E, which supports signed gates without modifying the existing KDA kernels, was proposed by Claude Opus 5. The authors contextualized and implemented this idea in relation to similar changes of variables used by other models. The integration into the chunk-wise parallel form was proposed by the authors and implemented by the same model. The TileLang kernels were modified using ChatGPT Astra (medium) and then checked through consistency with the naive Python recurrence and the state-tracking experiments by the authors. The orthogonal DPLR characterization (Theorem 1) and norm-constrained DPLR result (Theorem 9) were developed with assistance from ChatGPT 5.6 Sol (High). ChatGPT 5.6 Sol and 6 Astra produced the first complete versions of the proofs for the single-layer group-word-problem expressivity results (Theorems 3 and 4; proofs in Appendix C). The authors subsequently checked, corrected, and substantially edited these proofs. The adaptation in Appendix D showing that three layers suffice to recognize general weighted finite automata, based on (Merrill et al., 2026a), was proposed by ChatGPT 5.6 Sol and edited with assistance from Claude Opus 5 and ChatGPT 6 Astra. Claude Opus 5 was used to assist in implementation of language modeling adaptations, implementations, and experimentation. The authors reviewed all AI-assisted work and take responsibility for the final content.

ACKNOWLEDGMENTS We would like to thank Alicia Curth for extensive feedback over the whole duration of the project on the manuscript and figures. We are also grateful to Jan Tönshoff, Simon Schrodi, Baohe Zhang, and Adrian Barfuß who provided constructive feedback throughout this project. Frank Hutter acknowledges financial support by the Hector Foundation. The authors acknowledge support from ELLIS and MPI-IS Tübingen. Jaisidh Singh is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. Arber Zela and Volkan Cevher were funded by the Swiss National Science Foundation (SNSF) under grant number 2000-1-240094. This research was partially supported by the European Commission under the grant No. 101195233 (OpenEuroLLM). We acknowledge EuroHPC Joint Undertaking for awarding us access to Leonardo at CINECA, Italy, JUWELS and JUPITER at JSC, Germany, Deucalion at MACC, Portugal, MareNostrum5 at BSC, Spain, and the DLC2 Cluster at 11

University of Freiburg, Germany. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.

AUTHOR C ONTRIBUTIONS Julien Siems initiated and led the project, proposed combining signed channel-wise gating with an extended Householder range to enable complex eigenvalues, implemented CKDA, developed the initial state-tracking experiments, conducted preliminary language modeling experiments, iterated on kernel changes using Codex, and coordinated the collaboration. Julien Siems and Riccardo Grazzi developed the core theoretical intuition connecting signed gating, Householder transformations, rotations, and DPLR dynamics. Riccardo Grazzi led the theoretical contributions, identified the A5 construction, and verified most theoretical results. Riccardo Grazzi and Korbinian Pöppel led the formalization, verification, and revision of the finite-group and multilayer expressivity results. Arber Zela conducted the state-tracking experiments, building on Julien Siems’s initial versions, and contributed to the empirical evaluation and ablations. Jaisidh Singh conducted the small-scale language-modeling experiments. Timur Carstensen conducted the interpretability and additional language-modeling experiments and extended the chunk-wise parallel Triton kernels to include the CKDA gate. Korbinian Pöppel proposed the audio experiments, which Julien Siems conceptualized and carried out. Korbinian Pöppel conducted all large-scale experiments and scaling-law analyses. Aaron Klein enabled the large-scale language-modeling experiments, provided project supervision and crucial early support for the broader empirical study which catalyzed the start of the project. Antonio Orvieto contributed to the theoretical framing and characterization of the transition dynamics, suggested language model evaluations, and provided extensive manuscript feedback. Jenia Jitsev provided feedback on the scaling law analysis and manuscript feedback. Volkan Cevher and Frank Hutter supervised the project and reviewed the manuscript.

R EFERENCES Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan, Jörg Franke, Sampo Pyysalo, Jenia Jitsev, and Aaron Klein. Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss, August 2026. URL http://arxiv.org/abs/2608.28308. arXiv:2608.28308 [cs.LG]. Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pp. 1120–1128. PMLR, 2016. Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. arXiv preprint arXiv:2304.09433, 2023. David A Barrington. Bounded-width polynomial-size branching programs recognize exactly those languages in NC1 . In Proceedings of the eighteenth annual ACM symposium on Theory of computing, pp. 1–5, 1986. Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. Advances in Neural Information Processing Systems, 37: 107547–107603, 2024. Maximilian Beck, Kajetan Schweighofer, Sebastian Böck, Sebastian Lehner, and Sepp Hochreiter. xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=bpbU549sSg. 12

Kai Biegun, Rares Dolga, Jake Cunningham, and David Barber. RotRNN: Modelling long sequences with rotations. arXiv preprint arXiv:2407.07239, 2024. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, Apr. 2020. doi: 10.1609/aaai.v34i05.6239. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239. Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb. Distillation Scaling Laws. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=1nEBAkpfb9. Yutian Chen, Zhiyuan Li, Yucheng Wang, and Ming Wei. FlashKDA: Flash Kimi Delta Attention. https://github.com/MoonshotAI/FlashKDA, 2026. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, eighth workshop on syntax, semantics and structure in statistical translation, pp. 103–111, 2014. Jihyun Choi and Jae-Hyouk Lee. Binary icosahedral group and 600-cell. Symmetry, 10(8):326, 2018. Jiwan Chung, Heechan Choi, and Seon Joo Kim. Rethinking state tracking in recurrent models through error control dynamics. arXiv preprint arXiv:2605.07755, 2026. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803. 05457. Keith Conrad. Simplicity of An . Expository notes, University of Connecticut, n.d. URL https: //kconrad.math.uconn.edu/blurbs/grouptheory/Ansimple.pdf. Accessed September 14, 2026. Federico Danieli, Pau Rodriguez, Miguel Sarabia, Xavier Suau, and Luca Zappella. ParaRNN: Unlocking parallel training of nonlinear RNNs for large language models. In The Fourteenth International Conference on Learning Representations, 2026. Adam Dankowiakowski and Alessandro Ronca. Metric automata theory: A unifying theory of RNNs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024. Gianna M Del Corso, Federico Poloni, Leonardo Robol, and Raf Vandebril. When is a matrix unitary or hermitian plus low rank? Numerical Linear Algebra with Applications, 26(6):e2266, 2019. Igor Dubinin, Antonio Orvieto, and Felix Effenberger. Improved state mixing in higher-order and block diagonal linear recurrent networks. arXiv preprint arXiv:2602.12021, 2026. Reza Ebrahimi and Roland Memisevic. Revisiting bi-linear state transitions in recurrent neural networks. Advances in Neural Information Processing Systems, 38:69615–69642, 2026. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602. Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It’s raw! audio generation with statespace models. In International conference on machine learning, pp. 7616–7633. PMLR, 2022. 13

Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues. In International Conference on Learning Representations (ICLR), 2025. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=uYLFoz1vlAC. Ali Hatamizadeh, Yejin Choi, and Jan Kautz. Gated DeltaNet-2: Decoupling erase and write in linear attention. arXiv preprint arXiv:2605.22791, 2026. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models, March 2022. URL http: //arxiv.org/abs/2203.15556. arXiv:2203.15556 [cs]. Alston S Householder. Unitary triangularization of a nonsymmetric matrix. Journal of the ACM (JACM), 5(4):339–342, 1958. Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng Lin, Zunchang Liu, Xiaowen Chu, Zeke Xie, and Bojun Cheng. MDN: Parallelizing stepwise momentum for delta linear attention. arXiv preprint arXiv:2605.05838, 2026. inclusionAI. Ling-3.0-flash. Hugging Face model card, 2026. URL https://huggingface. co/inclusionAI/Ling-3.0-flash. Accessed: 2026-09-01. Eugen J. Ionascu. Rank-one perturbations of diagonal operators. Integral Equations and Operator Theory, 39(4):421–440, 2001. ISSN 1420-8989. doi: 10.1007/BF01203323. URL https: //doi.org/10.1007/BF01203323. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: //kellerjordan.github.io/posts/muon/. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs, stat], January 2020. URL http://arxiv.org/abs/2001. 08361. arXiv: 2001.08361. Arjun Karuvally, Franz Nowak, T. Anderson Keller, Carmen Amo Alonso, Terrence Sejnowski, and Hava T Siegelmann. Bridging expressivity and scalability with adaptive unitary SSMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=s4zitEu2R8. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The Impact of Positional Encoding on Length Generalization in Transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 24892–24928. Curran Associates, Inc., 2023. doi: 10.52202/075280-1082. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 4e85362c02172c0c6567ce593122d31c-Paper-Conference.pdf. Kimi Team. Kimi Linear: An expressive, efficient attention architecture. Technical report, Moonshot AI, 2025. arXiv:2510.26692. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. 14

Aakash Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, J Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved sequence modeling using state space principles. In The Fourteenth International Conference on Learning Representations, 2026. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, pp. 552–561, 2012. URL https://cdn.aaai.org/ocs/4492/ 4492-21843-1-PB.pdf. Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extraction meets the semi-structured web. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3047–3056, 2019. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/ HuggingFaceFW/fineweb-edu. Johan Ernest Mebius. A matrix-based proof of the quaternion representation theorem for fourdimensional rotations. arXiv preprint math/0501249, 2005. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In International Conference on Machine Learning (ICML), 2024. William Merrill, Hongjian Jiang, Yanhong Li, Anthony Widjaja Lin, and Ashish Sabharwal. Why are linear RNNs more parallelizable? In Forty-third International Conference on Machine Learning, 2026a. William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, Chuan Li, Kyle Lo, Saumya Malik, D. J. Matusz, Benjamin Minixhofer, Jacob Morrison, Luca Soldaini, Finbarr Timbers, Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi, and Ashish Sabharwal. OLMo Hybrid: From theory to practice and back. In Third Conference on Language Modeling, 2026b. Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using Householder reflections. In International Conference on Machine Learning, pp. 2401–2409. PMLR, 2017. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, 2018. doi: 10.18653/v1/D18-1260. URL https://aclanthology.org/D18-1260/. Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, and Tri Dao. M2RNN: Non-linear RNNs with matrix-valued states for scalable language modeling. arXiv preprint arXiv:2603.14360, 2026. Sajad Movahedi, Felix Sarnthein, Nicola Muca Cirone, and Antonio Orvieto. Fixed-point RNNs: Interpolating from diagonal to dense. Advances in Neural Information Processing Systems, 38: 44873–44908, 2025. Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, and Volkan Cevher. Selective rotary position embedding. In International Conference on Learning Representations (ICLR), 2026. Yoshihiro Nakamura. One-dimensional perturbations of isometries. Integral Equations and Operator Theory, 9(2):286–294, 1986. 15

Franz Nowak, Ryan Cotterell, and Reda Boumasmoud. An algebraic view of the expressivity of recurrent language models. arXiv preprint arXiv:2606.01765, 2026. Marc Olive. Effective computation of so (3) and o (3) linear representation symmetry classes. Mathematics and Mechanics of Complex Systems, 7(3):203–237, 2019. Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning (ICML), 2023. Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith. Universality of linear recurrences followed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues. In Forty-first International Conference on Machine Learning, 2024. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525–1534, 2016. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144/. Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. RWKV-7 ”goose” with expressive dynamic state evolution. In Second Conference on Language Modeling, 2025. Korbinian Pöppel, Maximilian Beck, and Sepp Hochreiter. FlashRNN: I/O-Aware Optimization of Traditional RNNs on modern hardware. In The Thirteenth International Conference on Learning Representations, 2025. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview. net/forum?id=1b7whO4SfY. Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, and Dayiheng Liu. On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability, 2026. URL https://arxiv.org/abs/2608.30320. Qwen Team. Qwen3.5-Omni technical report. arXiv preprint arXiv:2604.15804, 2026. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, 2018. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, 2011. URL https://cdn.aaai.org/ocs/2418/ 2418-10878-1-PB.pdf. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. Yash Sarrof, Yana Veitsman, and Michael Hahn. The expressive capacity of state space models: A formal language perspective. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Peter Scherk. On the decomposition of orthogonalities into symmetries. Proceedings of the American Mathematical Society, 1(4):481–491, 1950. 16

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International conference on machine learning, pp. 9355–9366. PMLR, 2021. Robert Schreiber and Beresford Parlett. Block reflectors: Theory and computation. SIAM Journal on Numerical Analysis, 25(1):189–205, 1988. Mehran Shakerinava, Behnoush Khavari, Siamak Ravanbakhsh, and Sarath Chandar. The expressive limits of diagonal SSMs for state-tracking. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=5bg5Ru5OML. Joonghyuk Shin, Yicong Hong, Jaesik Park, and Xun Huang. Can video world models track unobserved world states? 2026. Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2502.10297. Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Kirill Kalinin, Hitesh Ballani, and Babak Rahmani. Learning State-Tracking from Code Using Linear RNNs. arXiv preprint arXiv:2602.14814, 2026. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2459–2475, 2025. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Pingwei Sun, Yuxuan Hu, Jianchao Tan, Xue Wang, Jiaqi Zhang, Yifan Lu, Yerui Sun, Yuchen Xie, and Xunliang Cai. FG2 -GDN: Enhancing long-context gated delta networks with doubly fine-grained control. arXiv preprint arXiv:2604.19021, 2026. Xiaobai Sun and Christian Bischof. A basis-kernel representation of orthogonal matrices. SIAM journal on matrix analysis and applications, 16(4):1184–1196, 1995. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023. Aleksandar Terzic, Michael Hersche, Giacomo Camposampiero, Thomas Hofmann, Abu Sebastian, and Abbas Rahimi. On the expressiveness and length generalization of selective state space models on regular languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20876–20884, 2025a. Aleksandar Terzic, Nicolas Menet, Michael Hersche, Thomas Hofmann, and Abbas Rahimi. Structured sparse transition matrices to enable state tracking in state-space models. Advances in Neural Information Processing Systems, 38:83072–83111, 2025b. Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19, 2019. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen 17

Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. URL http://arxiv.org/abs/2307.09288. arXiv:2307.09288 [cs.CL]. Upstage Solar Team. arXiv:2607.20062.

Solar open 2 technical report.

Technical report, Upstage, 2026.

Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, and Kartik Ahuja. Skaling: Chinchilla’s Exponents Meet Kaplan’s Coupling, August 2026. URL http://arxiv.org/abs/2608. 07222. arXiv:2608.07222 [cs.CL]. Lei Wang, Yu Cheng, Yining Shi, Zhiwen Mo, Zhengju Tang, Wenhao Xie, Tong Wu, Lingxiao Ma, Yuqing Xia, Jilong Xue, Fan Yang, and Zhi Yang. Tilelang: Bridge programmability and performance in modern neural kernels. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=Jb1WkNSfUB. Songlin Yang and Yu Zhang. FLA: A Triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. URL https://github.com/fla-org/ flash-linear-attention. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024a. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522, 2024b. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with Delta Rule. In International Conference on Learning Representations (ICLR), 2025. Z.ai. GLM-5.3-Flash: Frontier intelligence, flash cost. 3-flash, August 2026. Accessed: 2026-09-15.

https://z.ai/blog/glm-5.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472/. Tiantian Zhang. Semidirect fourier delta attention: Phase-controlled delta memory with constructive chunk-WY kernels. arXiv preprint arXiv:2607.11897, 2026.

S UPPLEMENTARY M ATERIAL The supplementary material is structured as follows: • Section A characterizes orthogonal DPLR matrices, analyzes the spectrum of CKDA, bounds persistent rotations, and studies products of CKDA transitions. • Section B defines the state-tracking tasks, recurrent model, realization and tracking conventions, and exact algebraic datatypes and precision assumptions. • Section C proves the single-layer finite-group results: the planar and cube realizations, the three-dimensional obstruction and four-dimensional tracker for A5 , the affine and independenthead reductions, and the S5 spectral obstruction. • Section D explains the clock–buffer–accumulator mechanism, proves the three-layer automata result, and gives the factor counts and comparison-table details. • Section E derives the sign transformation that enables reuse of existing KDA kernels and reports the fused implementation’s throughput. • Section F discusses how the SiLU key activation affects learning group-word problems. • Section G gives experimental details for state tracking, periodic waveform continuation, and language modeling. • Section H describes the scaling-law methodology and results. • Section I presents additional state-tracking baselines and language-modeling validation curves. Our code is open-source on GitHub https://github.com/OpenEuroLLM/ComplexKDA and is based on Flash-Linear-Attention (Yang & Zhang, 2024). Notation. Matrices and vectors are denoted by bold uppercase and lowercase letters, respectively. We write Q, R, and C for the rational, real, and complex numbers. The matrix Id is the d×d identity, with the subscript omitted when its size is clear, and ei is the ith standard coordinate vector. For a scalar z, |z| denotes its absolute value or complex modulus; for a finite set X, |X| denotes its cardinality. We use z or z ∗ for complex conjugation, A⊤ for transpose, and A∗ for conjugate transpose. For a vector v, Diag(v) is the diagonal matrix with entries v, and ⊙ denotes the element-wise (Hadamard) product. The direct sum A ⊕ B places A and B on the diagonal of a block-diagonal matrix. We write ker A, im A, and rank A for the kernel, image, and rank of A. The notation span{v1 , . . . , vk } denotes the linear span of the indicated vectors, and V ⊥ denotes the orthogonal complement of a subspace V . For v ∈ Rn , we similarly write v ⊥ := {x ∈ Rn : v ⊤ x = 0}; when v ̸= 0, this is a hyperplane through the origin. For a subspace V , A|V denotes the restriction of A to vectors in V . If V is invariant under A, meaning AV ⊆ V , this restriction is a linear map from V to itself. The vector norm ∥v∥ = ∥v∥2P is Euclidean, while ∥A∥ = ∥A∥2 is the induced operator norm. The Frobenius norm is ∥A∥F = ( i,j |Aij |2 )1/2 . For a real symmetric matrix, A ⪰ 0 and A ≻ 0 mean positive semidefinite and positive definite, respectively. We write σ(A) for the spectrum (the set of eigenvalues), ρ(A) for the spectral radius, and tr(A) and det(A) for the trace and determinant. For an alphabet Σ, Σ∗ is the set of all finite words, including the empty word. Transitions act on column states, so the product At · · · A1 applies A1 first.

C HARACTERIZATION OF A CKDA TRANSITION

A.1

O RTHOGONAL DPLR MATRICES REDUCE TO SIGNED H OUSEHOLDERS

Rank-one perturbations of isometries and low-rank representations of orthogonal transformations have a classical literature; see, e.g., Scherk (1950); Nakamura (1986); Schreiber & Parlett (1988); Sun & Bischof (1995). The following result gives a more specific rigidity statement for diagonalplus-rank-one matrices: orthogonality forces the diagonal factor to reduce, up to the rank-one correction, to a sign matrix. 19

Table 3: Spectral structure of complex-capable delta recurrences CKDA Transition

(I − βkk⊤ )D(α)

MDN (quadrant-constrained) h i ⊤ α(I−βηkk ) −βµI αηkk⊤ µI

SFDA (default complex) (I − βkk∗ )Λ

First order

First order in (S, M ); generically second order in S

d real

2d real

β ∈ [0, 2] α ∈ [−1, 1]d

α ∈ (0, 1), µ ∈ [e−1 , 1) 0≤β ≤1−α η ∈ (0, 2)

β ∈ [0, 1] α ∈ [0, 1]m θ ∈ [−θmax , θmax ]m

Non-real conjugate pairs

≤1 (Thm. 8)

≤1

Up to m after realification

Origin of complex modes

Signed gate–delta interaction

Non-real |λ| = 1 modes†

✓ Any angle in a two-dimensional block

× None under the stated constraints

✓ Undamped phase modes

Unit-circle support†

Full circle

Only {+1}

{e±iθ : |θ| ≤ θmax }; full circle if θmax ≥ π

Whole-state isometries†

Orthogonal DPR1; need not commute

None in the interior; only I in the closure

Diagonal unit phases (β = 0, α = 1); commute

Non-expansive by construction

✓ ∥A∥2 ≤ 1

× Not guaranteed in the augmented Euclidean norm

Recurrence order State per value column Ranges compared

Memory–momentum coupling

First order over C m complex ≃ 2m real

Explicit phases plus delta correction

✓ ∥A∥2 ≤ 1

Time subscripts are suppressed; keys have unit norm, d ≥ 2, m ≥ 1, and θmax > 0. D(α) = Diag(α) and Λ = Diag(αj eiθj ). Dimensions and pair counts concern the transition acting on one value column, not the vectorized whole memory. † Unit-circle support and isometry entries include the boundaries/closures of the stated parameter families; sigmoid/tanh parameterizations need not attain these endpoints at finite logits. Isometry means A∗ A = I on the entire state space. MDN uses the constraint family in its Sec. 3.2; its experiments also report other momentum floors. SFDA refers specifically to its complex Eq. (13): its exact realification has rotation–decay blocks plus a correction of real rank at most two. Its isometry statement does not apply to an extension with β = 2 or to enlarged reflection/control families, and does not assert that general SFDA transitions commute. All spectral statements are single-step.

Theorem 6 (Characterization of orthogonal DPLR matrices; restatement of Theorem 1). Let A = D + uv ⊤ ∈ Rn×n , where D = Diag(d1 , . . . , dn ), and suppose A⊤ A = I. Then there exist a unit vector k ∈ Rn and a diagonal sign matrix S = Diag(s1 , . . . , sn ), with si ∈ {−1, +1}, such that A = (I − 2kk⊤ )S. Consequently, A has at most one non-real conjugate pair of eigenvalues. Proof. If A is diagonal, orthogonality makes it a diagonal sign matrix. Take k = e1 and let H = I − 2e1 e⊤ 1 , which flips the first coordinate. Then S := HA is also a diagonal sign matrix, and H 2 = I gives A = HS. Its eigenvalues are already ±1. If instead A is not diagonal, so u, v ̸= 0, we seek column sign flips S that turn AS into a reflection. Since reflections are symmetric, we first ask whether these flips can make the rank-one term symmetric. Orthogonality gives A⊤ A = AA⊤ = I, so the ith column and row both have squared norm one: 1 = ∥di ei + vi u∥2 = d2i + 2di ui vi + ∥u∥2 vi2 , 1 = ∥di ei + ui v∥2 = d2i + 2di ui vi + ∥v∥2 u2i . Subtracting the two equations yields ∥u∥2 vi2 = ∥v∥2 u2i , hence |vi | = c|ui | with c := ∥v∥/∥u∥ > 0. Thus ui = 0 exactly when vi = 0; on these coordinates, the preceding equations also give d2i = 1. 20

We can therefore choose signs so that Sv = −cu, making the rank-one term a negative multiple of uu⊤ , as in a Householder reflection. Explicitly, take S = Diag(s1 , . . . , sn ) with −cui /vi , ui ̸= 0, si = di , ui = 0. Each si is ±1. Consequently, Q := AS = DS + u(Sv)⊤ = DS − cuu⊤ is symmetric and orthogonal, hence Q2 = I. A reflection moves every vector along its normal direction. To find that direction, consider the displacement r = x − Qx of any vector x. Since Qr = Qx − Q2 x = Qx − x = −r, every displacement is reversed by Q. Substituting the formula for Q gives DSr − cu(u⊤ r) = −r, or coordinatewise, (1 + di si )ri = cui (u⊤ r). We can solve these equations coordinatewise. Indeed, |Qii | ≤ 1 because Q is orthogonal, so for ui ̸= 0 we have 1 + di si = 1 + Qii + cu2i > 0. For ui = 0, the same denominator equals 1 + d2i = 2. Thus r = c(u⊤ r)w, where wi := ui /(1 + di si ). Every displacement therefore lies along the same nonzero vector w. Since A is not diagonal, Q ̸= I; otherwise A = S. Hence some displacement is a nonzero multiple of w, and Qr = −r implies Qw = −w. For arbitrary x, write x − Qx = tw. Symmetry determines the coefficient: t∥w∥2 = w⊤ (x − Qx) = w⊤ x − (Qw)⊤ x = 2w⊤ x. Therefore Qx = x − 2w(w⊤ x)/∥w∥2 , so Q = I − 2kk⊤ with k := w/∥w∥. Since S 2 = I, we obtain A = QS = (I − 2kk⊤ )S. Finally, Ax = Sx − 2k(Sk)⊤ x. The correction singles out k and Sk, so consider U := span{k, Sk}. The matrix S exchanges these two vectors and is orthogonal, so it preserves both U and U ⊥ . The formula for Ax shows that A also preserves both spaces and agrees with S on U ⊥ . Its eigenvalues there are therefore ±1. All non-real eigenvalues lie in the restriction to U, whose dimension is at most two, allowing at most one conjugate pair. Remark. The characterization above can also be derived from (Ionascu, 2001, Proposition 3.1) which characterizes normal rank-one perturbations of normal operators, specialized to the finitedimensional real orthogonal case; here we state it explicitly in the signed-Householder form relevant to CKDA. A.2

T HE S IGN -M AGNITUDE D ECOMPOSITION OF CKDA

Proposition 7 (Expanded Proposition 2). Let ∥k∥ = 1, β ∈ [0, 2], and αi ∈ [−1, 1]. Define e Diag(|α|), A = (I − βkk⊤ ) Diag(α) = A

e := (I − βkk⊤ )S, A

S := Diag(sign α),

with sign(0) = 1. Let E− := ker(S + I) be the space spanned by the coordinate vectors whose signs are flipped by S, with dimension m− , and E+ := ker(S − I) with dimension m+ , and write k = k− + k + ,

k− ∈ E− ,

k+ ∈ E+ ,

∥k− ∥ = cos θ,

∥k+ ∥ = sin θ,

where θ ∈ [0, π/2] is the angle of k from E− . When nonzero, let e− := k− /∥k− ∥, e+ := k+ /∥k+ ∥, and let P be an orthogonal matrix beginning with the relevant vectors e− , e+ and then completed by orthonormal bases of E− first and then E+ . Then π : 2 θ=0: π θ= : 2

0<θ<

e = P B(β, θ) ⊕ (−Im −1 ) ⊕ Im −1 P ⊤ , A − + e = P (β − 1) ⊕ (−Im −1 ) ⊕ Im P ⊤ , A − + e = P (1 − β) ⊕ (−Im ) ⊕ Im −1 P ⊤ . A − + 21

= + 0.50

= + 0.00

= 0.50

= 1.00

= 0.50

= 1.00

1 1

= 1.50 Im

1 1

= 1.50 Im

= + 0.00

1 1

= 1.00 Im

1 1

0 1 1

= 2.00 Im

1 1

= 2.00 Im

Scalar gate: D = I

= + 0.50

1 1

= 0.50 Im

1 1

0 1

= + 1.00

= 0.00 Im

Channel-wise gate: D = Diag( , 1)

= + 1.00

0 Re

0 1

0 Re

Figure 11: Extension of Figure 2 varying both β and α. The figure demonstrates that both components need to be extended from their standard ranges to obtain complex eigenvalues. Here ⊕ denotes the direct sum of matrix blocks, which places the two operands as the blocks of a block-diagonal matrix. The only nontrivial block is B(β, θ) =

β cos2 θ − 1 β sin θ cos θ

−β sin θ cos θ , 1 − β sin2 θ

B(2, θ) =

cos 2θ sin 2θ

− sin 2θ . cos 2θ

e has one non-real conjugate pair exactly when the eigenvalues of B(β, θ) are More specifically, A non-real, i.e. when β 2 cos2 (2θ) < 4(β − 1), which only holds when β > 1. When β = 2, B(β, θ) becomes a pure rotation by 2θ. ⊥ ⊥ Proof. For x ∈ E− ∩ k− and y ∈ E+ ∩ k+ ,

e = −x, Ax

e = y, Ay

so only the span of the nonzero components of k remains. When both are nonzero, k = cos θ e− + sin θ e+ ,

Se− = −e− ,

Se+ = e+ ,

e on e− , e+ gives B(β, θ). The endpoint cases follow by setting k = k− or and evaluating A k = k+ , and the form of B(2, θ) follows from the double-angle identities. A.3

T HE W INDOW FOR C OMPLEX E IGENVALUES

We now consider the eigenvalues of the full CKDA transition. Theorem 8 (Complex Eigenvalues in Diagonal Householder Product). The matrix A = (I − βkk⊤ ) Diag(α) has at most one pair λ, λ∗ of non-real eigenvalues, and such a pair can occur only if β > 1 and αi < 0 for at least one i. This means it is not sufficient to extend β ∈ [0, 2] alone within KDA as in Grazzi et al. (2025), nor is it sufficient to extend only the range of αi while keeping β ≤ 1. This is similar to the requirement shown in (Siems et al., 2025, Prop. 1.3), where both βs in a product of two generalized Householders need to be greater than 1 for the product to obtain complex eigenvalues. Proof. Let Hkβ := I −βkk⊤ and D := Diag(α). The matrix Hkβ has eigenvalues (1−β, 1, . . . , 1), with k as an eigenvector for 1−β and k⊥ as the eigenspace for 1. Thus we can choose an orthogonal 22

matrix Q with k as its first column such that Q⊤ Hkβ Q = Λ := Diag(1 − β, 1, . . . , 1). Defining B := Q⊤ DQ, which is symmetric, we obtain Q⊤ AQ = ΛB.

(4)

First suppose β < 1. Then Λ ≻ 0, and Λ− /2 (ΛB)Λ /2 = Λ /2 BΛ /2 , 1

which is symmetric. Hence all eigenvalues are real. For β = 1, write B= Then

b11 b

b⊤ . B22

0 0 . b B22 This matrix has eigenvalues 0 together with the eigenvalues of the symmetric matrix B22 , so its spectrum is again real. Therefore non-real eigenvalues require β > 1. ΛB =

Now suppose β > 1. Define Σ := Diag(β − 1, 1, . . . , 1),

J := Diag(−1, 1, . . . , 1),

so that Λ = J Σ. Applying the similarity transformation with Σ /2 gives 1

/2 /2 Σ− /2 (ΛB)Σ /2 = J Σ | BΣ {z } =: T , 1

(5)

where S is symmetric. Hence T ∗ J = J T . Let T x = λx with λ ∈ / R. Then x∗ J T x = λx∗ J x = x∗ Sx ∈ R.

(6)

Since x J x ∈ R and λ ∈ / R, this implies x J x = 0. Similarly, for two eigenvectors T x = λx and T y = µy, (µ − λ)x∗ J y = 0. (7) Thus eigenvectors corresponding to eigenvalues in the upper half-plane are mutually J -orthogonal and J -isotropic. ∗

∗

The same statement holds for the corresponding generalized eigenspace. Let X span the generalized eigenspace associated with all eigenvalues in the upper half-plane, and write T X = XR. Setting G := X ∗ J X, the identity T ∗ J = J T gives R∗ G = GR. The spectrum of R lies in the upper half-plane, while that of R∗ lies in the lower half-plane. Hence the corresponding Sylvester equation has the unique solution G = 0, so the generalized eigenspace is totally J -isotropic. Since J = Diag(−1, 1, . . . , 1) has only one negative direction, every totally J -isotropic subspace has dimension at most one. Indeed, if a vector in such a subspace has first coordinate zero, then n X x∗ J x = |xi |2 = 0, i=2

and hence x = 0. Projection onto the first coordinate is therefore injective. Thus there is at most one eigenvalue in the upper half-plane, counting algebraic multiplicity, and since A is real, at most one corresponding complex-conjugate pair. Finally, if αi ≥ 0 for all i, then D ⪰ 0. The matrices Hkβ D and D /2 Hkβ D /2 1

have the same characteristic polynomial, while the latter is symmetric. Hence all eigenvalues of A are real. Therefore non-real eigenvalues require αi < 0 for at least one i. 23

A.4

N ON - EXPANSIVE DPLR MATRICES

Theorem 9 (Persistent rotations in non-expansive DPLR). Let A = D + R ∈ Rn×n , where D = D ⊤ is diagonal and rank(R) ≤ r, and suppose ∥A∥2 ≤ 1. Then A has at most r non-real conjugate pairs of eigenvalues on the unit circle. In particular, a non-expansive rank-one DPLR transition has at most one such pair. Proof. Since D is symmetric, the difference between A and its transpose comes entirely from the low-rank term: rank(A − A⊤ ) = rank(R − R⊤ ) ≤ rank(R) + rank(R⊤ ) ≤ 2r. We will show that each non-real unit-circle pair contributes two independent vectors to this range. Let Ax = λx with x ∈ Cn \{0} and |λ| = 1, and write x∗ for conjugate transpose. This eigenvector loses no norm under A; we first show that A⊤ reverses its action. Since ∥A⊤ ∥2 = ∥A∥2 ≤ 1, ∥A⊤ x − λ̄x∥2 = ∥A⊤ x∥2 + ∥x∥2 − 2 Re(λ̄x∗ Ax) = ∥A⊤ x∥2 + ∥x∥2 − 2 Re(λ̄λ∥x∥2 ) = ∥A⊤ x∥2 − ∥x∥2 ≤ 0. A squared norm cannot be negative, so A⊤ x = λ̄x. Consequently, (A − A⊤ )x = (λ − λ̄)x. If λ is non-real, the coefficient is nonzero, and x = (λ − λ̄)−1 (A − A⊤ )x belongs to the range of A − A⊤ . To count these vectors with multiplicity, observe that A also preserves x⊥ : whenever x∗ y = 0, we have x∗ Ay = (A⊤ x)∗ y = λx∗ y = 0. Thus span{x} and its orthogonal complement are both invariant, and the restriction to the complement remains a contraction. Splitting off this onedimensional block and repeating produces independent eigenvectors for every unit-circle eigenvalue, counted with algebraic multiplicity. If there are c non-real conjugate pairs on the unit circle, we therefore obtain 2c independent vectors in the range of A − A⊤ . A real matrix has the same rank over R and C, so 2c ≤ 2r, giving c ≤ r. A.5

E XPRESSIVITY OF PRODUCTS OF CKDA TRANSITIONS

The diagonal gate reduces the number of factors needed to represent a non-expansive matrix compared with the 3n generalized-Householder construction of Grazzi et al. (2025, Prop. 1, item 2). Proposition 10 (Expressivity of CKDA products). Let n ≥ 1 and A ∈ Rn×n satisfy ∥A∥2 ≤ 1. Then A is a product of at most max{1, 2n − 2} CKDA transitions, each with a unit key, β ∈ [0, 2], and diagonal gate entries in [−1, 1]. If A is orthogonal, at most max{1, n − 1} factors suffice, and this orthogonal bound is sharp. In particular, the permutation matrix of an n-cycle cannot be expressed using fewer than n − 1 CKDA factors. Proof. The zero matrix is a single zero-gate transition, and for n = 1 any scalar A ∈ [−1, 1] is a single diagonal gate with β = 0. The scalar A = −1 also shows sharpness of the orthogonal bound in dimension one. Henceforth let n ≥ 2 and A ̸= 0. Orthogonal matrices. By Householder QR decomposition, any orthogonal matrix can be written as Q = H1 · · · Hn−1 S, where S is a diagonal sign matrix and each Hi is a reflection or an identity padding factor. Absorbing S into the last factor gives n − 1 CKDA transitions. General contractions. Take a singular value decomposition A = U ΣV ⊤ , with Σ = Diag(σ1 , . . . , σn ) and σi ∈ [0, 1]. Apply Householder QR to both orthogonal factors: U = H1 · · · Hn−1 SU ,

V = G1 · · · Gn−1 SV .

Each Hi and Gi is a reflection or an identity padding factor, and SU , SV are diagonal sign matrices. Thus, with D := SU ΣSV , whose entries lie in [−1, 1], A = H1 · · · Hn−2 (Hn−1 D)Gn−1 · · · G1 . 24

Every reflection is a CKDA transition with β = 2 and identity gate, and an identity factor uses β = 0. Absorbing D into Hn−1 gives (n − 2) + 1 + (n − 1) = 2n − 2 CKDA factors. Sharpness for orthogonal matrices. Each CKDA factor is diagonal plus a matrix of rank at most one. Inductively, a product of m such factors is diagonal plus a matrix of rank at most m: multiplying D + R, with rank(R) ≤ m, by E + uv ⊤ gives the diagonal part ED and correction ER + uv ⊤ (D + R), of rank at most m + 1. Now let Cn be the cyclic permutation matrix defined by Cn ei = ei+1 for i < n and Cn en = e1 . For any diagonal D = Diag(d1 , . . . , dn ), the equation (Cn − D)x = 0 reads xi−1 = di xi

(i = 2, . . . , n),

xn = d1 x1 .

Starting from xn , the first n − 1 equations determine all other coordinates; the last equation can only impose an additional constraint. Thus dim ker(Cn − D) ≤ 1, so rank(Cn − D) ≥ n − 1 for every diagonal D. Recall that a product of m CKDA factors differs from some diagonal matrix by a correction of rank at most m. Thus any such factorization of Cn would give a diagonal D satisfying n − 1 ≤ rank(Cn − D) ≤ m, forcing m ≥ n − 1. Since Cn is orthogonal, the upper bound is attained. Remark. This bound is also related to the Hermitian-plus-low-rank characterization of Del Corso et al. (2019, Theorem 12); the proof above gives a direct contraction-specific rank argument.

S TATE - TRACKING PRELIMINARIES AND ARITHMETIC CONVENTIONS

This section fixes the tasks, recurrent model, and arithmetic conventions used in the single-layer results of Section C and the multilayer results of Section D. We follow the state-tracking setup of Grazzi et al. (2025); Siems et al. (2025), making explicit the distinction between tracking a state, recognizing a language, and computing a weighted-automaton output. B.1

S TATE - TRANSITION SYSTEMS AND TRACKING

Definition B.1 (State tracking). A state-transition system consists of a finite input alphabet Σ, a state space S, an initial state s0 ∈ S, and a map Ta : S → S for each a ∈ Σ. For a word w = a1 · · · aT ∈ Σ∗ , its states satisfy st = Tat (st−1 ). For a recurrent model with hidden updates Fa and initial state h0 , a hidden state is reachable if it is obtained from h0 by processing some finite input word. This includes h0 itself, reached by the empty word. The model tracks the system if a fixed decoder d satisfies d(h0 ) = s0 ,

d(Fa (h)) = Ta (d(h))

for every reachable h and every a ∈ Σ.

Thus d(ht ) = st at every prefix of every word, including the empty word. The decoder need not be injective, and S need not be finite. For each target system, the network dimensions, parameters, initial state, and decoder are fixed independently of input length. Under the polynomial-precision convention of Section B.4, each scalar is represented exactly using a number of bits bounded by a fixed polynomial in the input length. Groups and prefix products. We write e for a group identity and |G| for the number of elements of G. The group Sn consists of permutations of n objects, and An is its subgroup of even permutations, those expressible as an even number of swaps of two objects. We write Zn for the cyclic group of n elements and Dn for the dihedral group of 2n elements. The order of an element h is the smallest positive integer m with hm = e, when such an integer exists. We use cycle notation for permutations: (12) swaps 1 and 2, while (1245) sends 1 7→ 2 7→ 4 7→ 5 7→ 1, leaving unlisted objects fixed. A swap is called a transposition. Disjoint cycles act on separate objects; for example, (1245)(36) also swaps 3 and 6. Products are composed from right to left, so (12)(23) = (123) applies (23) first. Disjoint cycles commute, but overlapping cycles need not: reversing this product gives (23)(12) = (132). 25

For a finite group G with identity e, take Σ = S = G, s0 = e, and Tg (s) = gs. The desired state is st = gt · · · g1 . Throughout our group results, inputs range over all of G. Restricting inputs to a chosen set of generators gives a different task. For example, tracking Zn means computing the running sum modulo n. We use “solving the group-word problem of G” to mean tracking this system in the sense of Definition B.1, that is, recovering every prefix product. Deterministic finite automata. A DFA is a tuple (Q, Σ, δ, q0 , F ), where Q is a finite state set, q0 ∈ Q is the initial state, F ⊆ Q is the accepting set, and δ : Q × Σ → Q is the transition function. Indexing coordinates by Q, define the column-state transition matrix by Ma eq = eδ(q,a) ,

p0 = eq0 ,

pt = Mat pt−1 .

Each column of Ma is one-hot. Tracking recovers the current state qt ; recognition only asks whether qt ∈ F . For example, a two-state parity automaton swaps its states on input 1, preserves them on input 0, and accepts in its even state. The functions Q → Q induced by words form a finite monoid under composition: composition is associative and the empty word acts as the identity. If all symbol maps are bijective, this monoid is a group. Rational weighted finite automata. A WFA over Q has a finite alphabet Σ, an initial vector α0 ∈ Qn , a matrix Ma ∈ Qn×n for each symbol, and a final vector ω ∈ Qn . Its weighted state and scalar output are p0 = α0 ,

pt = Mat pt−1 ,

yt = ω ⊤ pt = ω ⊤ Mat · · · Ma1 α0 .

(8)

Unlike a DFA state, pt can range over an infinite set. Tracking pt suffices to compute yt by applying the final vector ω. The converse need not hold: distinct weighted states can have the same scalar output, so computing yt need not recover pt . B.2

R ECURRENT MODEL , INITIALIZATION , AND READOUT

All matrices are real, and Id denotes the d × d identity matrix. A linear RNN head has a matrixvalued state and an affine update St = A(xt )St−1 + B(xt ),

(9)

St ∈ Rn×dv .

The functions are position-independent: time dependence comes through the layer input xt . In CKDA, A(xt ) = (I − βt kt kt⊤ ) Diag(αt ),

B(xt ) = βt kt vt⊤ ,

∥kt ∥2 = 1.

Unless stated otherwise, βt ∈ [0, 2] and αt ∈ [−1, 1]n . These theoretical ranges include their endpoints and zero gates. The practical gate parameterization and its magnitude floor are described separately in Section E. The regular-language and WFA simulation in Section D explicitly allows βt > 2. The S5 lower bound allows arbitrary additive matrices B(xt ), and therefore also covers the CKDA additive term. Heads and layers. Independent heads have separate recurrent states and receive the same layer input; their updates do not depend on the previous states of other heads. A joint decoder may use their outputs. Stacked layers may apply nonlinear feed-forward maps between recurrent layers, and may pass the current input along with the recurrent readout. Initial state and decoder. We permit a prescribed initial state; in particular, the group constructions use S0 = Id and B(xt ) = 0 at every step. A head is usually read through St⊤ qt , followed by a feed-forward decoder, with qt determined by the current layer input. For finite orthogonal state sets, Lemma 12 shows that one fixed query retains all information required by a state decoder. We allow sufficiently expressive feed-forward maps to implement finite lookups, as in Grazzi et al. (2025); Siems et al. (2025); Merrill et al. (2026a). In the WFA construction, the readout is a linear functional of the accumulator with coefficients selected from a finite lookup. 26

B.3

R EALIZATION AND DECODED TRACKING

Group representations. The groups O(d) = {Q : Q⊤ Q = Id } and SO(d) = {Q ∈ O(d) : det Q = 1} contain the orthogonal matrices and the orientation-preserving ones, respectively. The notation ⟨Ah : h ∈ G⟩ denotes the group generated by the indicated matrices: all finite products of them and their inverses, including Id . Definition B.2 (Homomorphism, isomorphism, and faithful representation). For groups G and H, a map ϕ : G → H is a homomorphism if ϕ(ab) = ϕ(a)ϕ(b) for every a, b ∈ G. It is an isomorphism if it is also bijective; we write G ∼ = H when such a map exists. An orthogonal representation is a homomorphism ρ : G → O(d), and it is faithful if it is injective. A faithful representation identifies G with the matrix group ρ(G). For a homomorphism ϕ : G → H, its kernel is ker ϕ = {a ∈ G : ϕ(a) = eH }, and the fiber ϕ−1 (b) = {a ∈ G : ϕ(a) = b} collects elements with the same image. The kernel is trivial, ker ϕ = {eG }, exactly when ϕ is injective. For a subgroup N ⊂ G, a left coset hN = {hn : n ∈ N } is a copy of N obtained by multiplying each element on the left by h. Distinct cosets partition G. Their number is the index [G : N ], which equals |G|/|N | when G is finite. In particular, index two means that the two cosets each contain half the elements of G. Every nonempty fiber of ϕ is a coset of its kernel: if ϕ(a0 ) = b, then ϕ−1 (b) = a0 ker ϕ. Thus each attained value has exactly | ker ϕ| preimages when the kernel is finite. A subgroup N ⊂ G is normal if hN h−1 = N for every h ∈ G. Kernels are normal: for a ∈ ker ϕ, ϕ(hah−1 ) = ϕ(h)eH ϕ(h)−1 = eH . A surjective homomorphism also maps normal subgroups to normal subgroups of its codomain. A nontrivial group is simple if its only normal subgroups are {e} and the whole group. We use that A5 is a noncyclic simple group of order 60 (Conrad, n.d.). Permitted transitions, realization, and tracking. For the orthogonal group constructions we specialize to S0 = Id , St = Agt St−1 . For a unit vector u ∈ Rd , the Householder matrix Hu = Id − 2uu⊤ reflects across the hyperplane u⊥ . A sign diagonal is a diagonal matrix with entries in {−1, +1}. We consider all orthogonal transition matrices of a CKDA layer: SHd := {Hu D : ∥u∥ = 1, D a sign diagonal}. Its elements are signed-Householder matrices. This family contains Id = He1 He1 , since the first coordinate vector e1 gives the sign diagonal He1 = Diag(−1, 1, . . . , 1). The family is not generally closed under multiplication. A realization represents the accumulated group element directly by a matrix. Tracking allows several hidden matrices to represent the same group element, provided a fixed decoder recovers the correct product. Definition B.3 (Signed-Householder realization and tracking). For a finite group G, choose transitions Ah ∈ SHd , h ∈ G, and let R be the states reachable from Id under these updates. They realize G in dimension d if h 7→ Ah is a faithful representation, and track G if a decoder p : R → G satisfies p(Id ) = e, p(Ah S) = h p(S) (h ∈ G, S ∈ R). (10) The tracker has finite reachability, or is finite-state, if R is finite (Definition B.4). For finite-state tracking, the update rule itself forces the decoder to preserve multiplication. This lets us use group structure even when the decoder was initially allowed to be an arbitrary function. Lemma 11 (Finite tracking induces a group homomorphism). For a finite-state signed-Householder tracker, R = Γ := ⟨Ah : h ∈ G⟩, and its decoder p : Γ → G is a surjective homomorphism. Such a tracker has a bijective decoder exactly when its transitions form a realization. Proof. The finite set R contains the identity and is closed under multiplication. For each Q ∈ R, two powers coincide; invertibility gives Qm = Id for some m ≥ 1. Hence Q−1 = Qm−1 ∈ R, so R = Γ. Writing Q1 = Aht · · · Ah1 and iterating the tracking rule gives p(Q1 Q2 ) = 27

ht · · · h1 p(Q2 ) = p(Q1 )p(Q2 ). Also p(Ah ) = h, proving surjectivity. If p is bijective, its inverse is a faithful representation with p−1 (h) = Ah . Conversely, a realization ρ has reachable set ρ(G) and decoder ρ−1 . Query projection. The following lemma relates these matrix-state definitions to the query readout in Section B.2. Lemma 12 (A fixed query distinguishes finite orthogonal states). For every finite subgroup Γ ⊂ O(d), there is a unit query q ∈ Rd such that S 7→ S ⊤ q is injective on Γ. For a finite-state tracker with decoder p, the output decoder π(S ⊤ q) := p(S) is therefore well defined and satisfies π(q) = e,

π((Ah S)⊤ q) = h π(S ⊤ q).

(11)

Distinct reachable query outputs have positive separation, so the precision needed to distinguish them is independent of sequence length. Proof. Choose a nonzero vector q outside the fixed-point subspaces of all nonidentity matrices in Γ, and normalize it. Each fixed-point subspace of S ∈ Γ \ {Id } is proper, and a finite union of proper linear subspaces cannot cover Rd . Let S1 , S2 ∈ Γ. If S1⊤ q = S2⊤ q, then S2 S1⊤ q = q. By the choice of q, the only matrix in Γ that fixes q is Id . Hence S2 S1⊤ = Id , which gives S1 = S2 . The output tracking rule follows from that of p. The distinct outputs form a finite set, which has a positive minimum pairwise distance whenever it has at least two elements; a singleton needs no distinction. The query can be chosen with algebraic coordinates: first choose a rational vector outside the finitely many excluded proper subspaces, then normalize it. Positive separation concerns the readout of exact reachable states; it does not by itself bound errors accumulated in the recurrence. B.4

P RECISION AND FINITE REACHABILITY

In practice, floating-point recurrent implementations round arithmetic results within each update and pass the resulting state to the next step. A simplified model rounds once per update: bt = round A(xt )S bt−1 + B(xt ) . S Actual implementations may round several times, depending on the arithmetic and fused operations. This generally makes the update nonlinear, and need not give the same result as rounding the exact unrolled state at the end. We therefore need to specify both the datatype and how arithmetic is evaluated. Simplifying the arithmetic. Merrill et al. (2024, Section 2.1 and Appendix A) evaluate iterated sums and products in the unrolled computation exactly before casting to O(log T )-bit floats. Following this approach, Grazzi et al. (2025, Appendices A.4 and B) use a fixed finite datatype for their lower bounds. Merrill et al. (2026a, Sections 2.1 and 2.4) instead use exact rational arithmetic with bounds on representation size, preserving associativity and distributivity. Our arithmetic convention. Our constructive results use polynomial precision over a fixed real algebraic number field K ⊂√R. This extends the rational setting of√Merrill et al. (2026a) to include irrational constants, such as 3/2 in a rotation through 2π/3 or 1/ 2 in a normalized transposition key. Neither fixed-width floats nor rational numbers with polynomially many bits represent these exactly. Fix a basis η1 , . . . , ηr of K over Q and store r X x= cj ηj , cj ∈ Q. j=1

The numerators and denominators of the cj have polynomial bit length in the input length T . The field, basis, and network parameters are fixed independently of T . Arithmetic is exact: each ηi ηj is a fixed rational combination of basis elements, so only the rational coefficients need updating. Floating-point approximations require a separate error analysis; a small rotation-angle error can accumulate over repeated updates. Our exact constructions do not establish numerical robustness at arbitrary lengths. 28

Finite reachability. A finite target state space does not force the model to visit finitely many hidden states: different words with the same group product may lead to different states that decode to that product. Even the DFA encoding ht = 2−t eqt retains the current state while visiting infinitely many hidden states. We distinguish this from the following property of the exact recurrence. Definition B.4 (Finite reachability). The reachable set of a recurrent model is R = {h0 } ∪ {Fat ◦ · · · ◦ Fa1 (h0 ) : t ≥ 1, ai ∈ Σ}. The model has finite reachability, or is finite-state, if R is finite under its exact updates. Our finite-group and DFA constructions use only finitely many exact values, including intermediate computations. They admit a fixed finite datatype D ⊂ K, possibly containing irrationals, as in Grazzi et al. (2025, Appendix A.4) and Siems et al. (2025, Appendix B). It contains all values needed by the construction, without needing to be closed under arbitrary arithmetic. For the block simulation of Peng et al. (2025, Appendix D.2) and our adaptation, finiteness follows from the finite clocks, buffers, DFA states, and partial factor products. Thus these constructions have finite reachability without rounding. What the lower bounds assume. We assume finite reachability in the S5 obstruction to simplify the analysis: together with non-expansion, it lets us remove the additive term and restrict to orthogonal transitions (Propositions 20 and 21). Finitely many rounded states do not imply finitely many exact states, so choosing a finite datatype alone does not justify this reduction. Grazzi et al. (2025, Thms. 1–2 and Appendix B.2) do not require finite exact reachability. Under their finite-datatype casting convention, they rule out tracking S2 ∼ = Z2 (parity) with any fixed number of layers when every transition has only nonnegative real eigenvalues. Their single-layer argument also rules out tracking Zn for every n > 2 when all transition eigenvalues are real. On a repeated input, the cast state of one layer eventually becomes constant in the first case, or alternates between at most two values in the second, preventing the required counting. Our S5 result allows a non-real conjugate pair per head, so the real-spectrum argument does not apply directly. Replacing finite reachability by a suitable datatype and casting assumption remains open.

S INGLE - LAYER FINITE - GROUP EXPRESSIVITY

We use the tracking and realization definitions of Section B.3 and the exact arithmetic convention of Section B.4. Inputs range over every element of the target group. We first construct planar and cube realizations, then prove the three-dimensional obstruction and four-dimensional tracker for A5 . Finally, we extend the lower-bound setup to affine updates and independent heads before proving the spectral obstruction for S5 . C.1

A XIS CRITERION FOR THREE - DIMENSIONAL ROTATIONS

A rotation in R3 fixes its axis and rotates the perpendicular plane. The following proposition characterizes exactly when such a rotation is a signed-Householder matrix. Its proof uses only the elementary fact that the product of reflections in two planes is a rotation about their line of intersection. Proposition 13 (Axis test). Let R ∈ SO(3) be a rotation about the axis spanned by a unit vector a ∈ R3 . 1. The identity and every half-turn belong to SH3 . 2. If its angle is different from 0 and π, then R = Hu D for some sign diagonal D if and only if its rotation axis a is contained in a coordinate plane. Explicitly, this means that a ⊥ ei for some coordinate axis Rei , or equivalently that a has at least one zero coordinate. Proof. For part 1, the matrix 2aa⊤ − I fixes a and sends every x ⊥ a to −x, so it is the half-turn about Ra. Moreover, 2aa⊤ − I = (I − 2aa⊤ )(−I) = Ha (−I), which has the required form. The identity case was established when introducing the permitted family. 29

For part 2, suppose R = Hu D. Since det Hu = −1 and det R = 1, the diagonal D has either one or three negative entries. In the latter case D = −I and R = 2uu⊤ − I, the half-turn from part 1, contradicting the assumption on the angle. Hence D has exactly one negative entry. If that entry is ⊥ in position i, then D = I − 2ei e⊤ i , which is precisely the reflection across the coordinate plane ei . ⊥ Both factors are now reflections in planes: D reflects in e⊥ i and Hu reflects in u . Their product ⊥ ⊥ rotates about the line ei ∩ u . This is the axis Ra, so a ⊥ ei ; in other words, a lies in the coordinate plane e⊥ i .

Conversely, every rotation in SO(3) is a product of reflections in two planes through its axis, separated by half the rotation angle. If a ⊥ ei , we may choose one plane to be the coordinate ⊤ plane e⊥ i . Its reflection is Di = I − 2ei ei ; the other reflection is therefore Hu := RDi , and R = Hu Di . C.2

C YCLIC AND DIHEDRAL GROUPS

We write Zn for the cyclic group of n elements and Dn for its dihedral extension of 2n elements, represented by planar rotations and reflections. The following realizations also give finite-state tracking. Proposition 14 (planar groups). Every cyclic group Zn and every dihedral group Dn has a signedHouseholder realization in two dimensions. Proof. In the plane, take D0 = Diag(1, −1), reflection in the horizontal axis. If Sα denotes reflection in the line making angle α with the horizontal axis, then Sα D0 is a rotation through 2α. Choosing α = πk/n realizes every rotation through 2πk/n, so every element in the standard representation of Zn has the required form. The dihedral group adds reflections of the regular n-gon. Each is already a Householder matrix, so choose D = I. Thus the same is true for the standard planar representation of Dn . C.3

R EALIZING S4 VIA THE CUBE ROTATIONAL SYMMETRIES

The group S4 has a familiar geometric realization: it is isomorphic to the group of rotational symmetries of a cube. We now show that rotating the cube to a suitable orientation makes all of these rotations admissible at once. Since restricting a faithful representation to a subgroup preserves both faithfulness and the permitted matrix form, the same construction will also realize A4 ⊂ S4 . Figure 12 shows the face and body-diagonal axes; edge axes give only half-turns. 90◦

Cube

Face axis: order 4

120◦

Vertex axis: order 3

Figure 12: Smallest symmetry-preserving rotations (excluding half-turns) for the cube, grouped by axis. The red point marks the axis viewed end-on; visible edges are solid and hidden edges are dashed. Proposition 15 (cube realization). The group S4 , realized as the rotational symmetry group of a cube, has a signed-Householder realization in three dimensions. Restricting it to the subgroup A4 gives a realization of A4 in the same dimension. Proof. The orientation-preserving symmetry group of a cube is isomorphic to S4 : it permutes the four unoriented body diagonals faithfully. Its non-half-turn rotation axes are 30

• the three face axes, supporting rotations through 90◦ . • the four body-diagonal axes, supporting rotations through 120◦ . All remaining non-identity rotations are half-turns and are automatically admissible by the axis test (Proposition 13). Place the cube first in its standard orientation. The three face axes and four body-diagonal axes are represented, respectively, by F = {(1, 0, 0), (0, 1, 0), (0, 0, 1)},

B = {(1, 1, 1), (1, 1, −1), (1, −1, 1), (−1, 1, 1)}.

Every vector in B has three nonzero coordinates, so this orientation does not pass the axis test of Proposition 13. We can change the orientation of these axes while preserving the group being represented. Write ρ(h) for the matrix of a cube symmetry in the standard orientation. Reorienting the whole cube by a rotation R replaces each symmetry matrix by Rρ(h)R⊤ , an operation called conjugation. Using the same R for every symmetry preserves multiplication, since (Rρ(h)R⊤ )(Rρ(k)R⊤ ) = Rρ(hk)R⊤ . At the same time, each rotation axis u moves to Ru. Our task is therefore to find one rotation that places all seven axes in coordinate planes. We choose a 45◦ rotation about the third coordinate axis: √ 1/ √2 R = −1/ 2 0 

√ 1/√2 1/ 2 0

 0 0 ∈ SO(3). 1

Applying R to each of the seven axes gives (1, −1, 0) (1, 1, 0) √ √ RF = , , (0, 0, 1) , 2 2 √ √ √ √ RB = {( 2, 0, 1), ( 2, 0, −1), (0, − 2, 1), (0, 2, 1)}. Thus the rotation does not spoil the three face axes: they still have a zero coordinate, as do all four transformed body diagonals. The axis test (Proposition 13) therefore shows that every cube rotation is a signed-Householder matrix. Restriction to A4 proves the remaining claim. C.4

H OW TO TRACK A5

The orientation-preserving symmetry group of the icosahedron is isomorphic to A5 . We prove that A5 has no finite-state signed-Householder tracker in three dimensions, then construct one in four dimensions. The latter uses a many-to-one decoder, so the accumulated hidden matrix carries more information than the group element being tracked. An icosahedron has 12 vertices, 20 faces, and 30 edges. Pairing each feature with its opposite gives the rotation axes below. An axis has order k if the rotations about it form a cyclic group of k elements; equivalently, its smallest positive rotation has angle 2π/k and returns to the identity after k applications. Thus there are 6 axes of order 5,

10 axes of order 3,

15 axes of order 2.

The order-2 axes do not matter: the axis test (Proposition 13) already makes every half-turn admissible. The difficulty comes from the other sixteen axes. Figure 13 illustrates these two relevant axis types. By the axis test (Proposition 13), representing all odd-order rotations would require three coordinate planes to cover all sixteen axes. We prove the stronger, orientation-independent fact that any plane covers at most four. 31

120◦ 72◦

Icosahedron

Face axis: order 3

Vertex axis: order 5

Figure 13: Smallest symmetry-preserving rotations (excluding half-turns) for the icosahedron, grouped by axis. The red point marks the axis viewed end-on; visible edges are solid and hidden edges are dashed. C.4.1

T HE OBSTRUCTION TO FINITE - STATE TRACKING IN THREE DIMENSIONS

Lemma 16 (icosahedral plane bound). Among the six order-5 axes and ten order-3 axes of an icosahedron, no plane through the center contains more than four axes. √ Proof. Let φ = (1 + 5)/2 be the golden ratio. In the usual coordinates, representative vectors for the six unoriented vertex axes (the order-5 axes) are V = {(0, 1, ±φ), (1, ±φ, 0), (φ, 0, ±1)} . The face centers form the vertices of the dual dodecahedron. Representatives of their ten unoriented axes (the order-3 axes) are T = {(1, 1, 1), (1, 1, −1), (1, −1, 1), (−1, 1, 1)} ∪ {(0, φ, ±(φ − 1)), (φ, ±(φ − 1), 0), (φ − 1, 0, ±φ)}. It suffices to check that no three axes in either family are coplanar. Indeed, substituting the displayed vectors and using φ2 = φ + 1 gives the following possible absolute determinants of three distinct representatives: family | det(v1 , v2 , v3 )| √ √ V 1√+ 5, 3 + 5√ √ 5 − 1, 2, 1 + 5, 2 5, 4 T All these determinants are nonzero. Thus a plane contains at most two vertex axes and at most two face axes, hence at most four axes in total. The bound is tight: the plane x = 0 contains (0, 1, φ), (0, 1, −φ), (0, φ, φ − 1), (0, φ, −(φ − 1)).

Using the group facts recalled in Section B.3, we now turn the geometric bound into a tracking obstruction. We also use the classification of finite subgroups of SO(3) (Olive, 2019, Sec. 3): cyclic groups, dihedral groups, and the rotational symmetry groups of the tetrahedron, cube, and icosahedron, isomorphic to A4 , S4 , and A5 . These occur up to an orthogonal change of coordinates. Theorem 17 (No finite-state three-dimensional A5 tracker). The group A5 admits no finite-state signed-Householder tracking in dimension three, even with an arbitrary decoder. In particular, it admits no signed-Householder realization in that dimension. Proof. Suppose a finite-state signed-Householder tracker exists, with transitions Ah and decoder p. The homomorphism property of finite tracking (Lemma 11) gives a finite hidden group Γ = ⟨Ah : h ∈ A5 ⟩ ⊂ O(3) and a surjective homomorphism p : Γ → A5 satisfying p(Ah ) = h. Turn the hidden matrices into rotations. In three dimensions, multiplying an orientation-reversing orthogonal matrix by −I3 makes it orientation-preserving. Define ψ(Q) := (det Q)Q,

H := ψ(Γ) ⊂ SO(3). 32

h = (123) (a) Cups

(b) Labels F (Qi )

k = (124)

Q0 = I 4 L0 = e R0 = e

ℓ = (14)(23)

Q1 = g(h) L1 = (123) R1 = (132)

Q2 = g(k)Q1 L2 = (14)(23) R2 = (13)(24)

Same cups: f (Q3 ) = f (Q0 ) = e

Q3 = g(ℓ)Q2 L3 = e R3 = (12)(34)

Different hidden states: Q3 ̸= Q0

Figure 14: Many-to-one tracking, illustrated by h = (123), k = (124), and ℓ = (kh)−1 = (14)(23). The cups show identities at successive positions; object 5 is fixed and omitted. Each box groups the two labels F (Qi ) = (Li , Ri ) of one hidden matrix, where Li = f (Qi ) and Ri = f− (Qi ). An input a gives Q 7→ g(a)Q and (L, R) 7→ (aL, a−1 R). The cups and L return to their initial values, but R3 = (12)(34) ̸= R0 = e; hence Q3 ̸= Q0 , although both decode to e. Indeed, det ψ(Q) = (det Q)4 = 1. This map also preserves multiplication: ψ(Q1 Q2 ) = (det Q1 )(det Q2 )Q1 Q2 = ψ(Q1 )ψ(Q2 ). Thus H is a finite rotation group. The map identifies at most a matrix and its negative, since its kernel K is contained in {±I3 }. These identifications cannot change the decoded group element. Since K is a kernel, it is normal in Γ. As p is surjective, its image p(K) is therefore normal in A5 . This image has at most two elements, so simplicity of A5 forces p(K) = {e}. Consequently the decoder descends to a well-defined surjective homomorphism p̄ : H → A5 ,

p̄(ψ(Q)) := p(Q).

To check that this is well defined, if ψ(Q1 ) = ψ(Q2 ), then Q1 Q−1 2 ∈ K, so p(Q1 ) = p(Q2 ). Identify the resulting rotation group. The classification of finite subgroups of SO(3) now forces H to be an icosahedral rotation group. The tetrahedral and cubic groups have only 12 and 24 elements, so cannot map onto A5 . A cyclic or dihedral group has a cyclic normal subgroup of index at most two. Its image under p̄ would be a cyclic normal subgroup of A5 with at most two cosets. Simplicity forces that image to be trivial or all of A5 : the former has 60 cosets, while the latter is not cyclic. These cases are therefore also impossible. It follows that |H| = 60, so the surjective map p̄ : H → A5 is an isomorphism. In particular, the matrices ψ(Ah ), one for each h ∈ A5 , exhaust H, because p̄(ψ(Ah )) = h. Each is still signedHouseholder: if Ah = Huh Dh , then ψ(Ah ) = Huh (det Ah )Dh , and the factor in parentheses is again a sign diagonal. Apply the geometric obstruction. We have obtained an orientation of the icosahedron in which every rotation is signed-Householder. But its six order-5 axes and ten order-3 axes cannot all lie in the three coordinate planes: the bound in Lemma 16 allows at most four axes per plane, hence at most 12 of the 16 axes altogether. A nonidentity rotation about any remaining axis fails the signed-Householder axis test (Proposition 13), a contradiction. Finally, a signed-Householder realization would itself give a finite-state tracker, with the inverse representation as decoder. It is therefore excluded as well. C.4.2

A FOUR - DIMENSIONAL CONSTRUCTION THAT TRACKS A5

The preceding theorem shows that a non-injective decoder cannot rescue a finite orthogonal hidden group in three dimensions. Four dimensions do admit such a construction. We first show how to compute iterated products in SO(3) using signed-Householder transitions in SO(4). An encoder g converts each input rotation into a permitted transition, and a fixed decoder f recovers the accumulated product from the hidden matrix. The hidden matrix need not itself equal the encoding of that product: what matters is that decoding each hidden update gives the correct three-dimensional update. For inputs belonging to a finite rotation group, the construction also has only finitely many reachable hidden matrices. 33

The underlying SO(4) geometry is the classical left/right quaternionic decomposition of fourdimensional rotations (Mebius, 2005); the A5 case uses the binary icosahedral lift (Choi & Lee, 2018). Please see Figure 14 for an intuitive example of the construction. Proposition 18 (Tracking rotations with signed-Householder transitions). There exist an injective encoder g : SO(3) → SO(4) and a decoder f : SO(4) → SO(3) such that every g(A) is signedHouseholder, f (I4 ) = I3 , and for every A ∈ SO(3),

f (g(A)S) = A f (S)

(12)

S ∈ SO(4).

Consequently, for every n ≥ 1 and every sequence of rotations A1 , . . . , An ∈ SO(3), f g(A1 ) · · · g(An ) = A1 · · · An .

(13)

In particular, f (g(A)) = A. Explicitly, let A = Ru,θ be the three-dimensional rotation through angle θ ∈ [0, π] about the axis spanned by a unit vector u ∈ R3 , with the direction of rotation specified by the right-hand rule around u. The encoder can be chosen as cos(θ/2) g(A) = (I4 − 2aA a⊤ )D , a := , D0 := Diag(−1, 1, 1, 1). (14) 0 A A sin(θ/2) u Thus all encoded transitions use the same diagonal sign matrix D0 . To define the decoder, write a matrix S ∈ SO(4) in 1 + 3 block form: α r⊤ S= , α ∈ R, r, s ∈ R3 , C ∈ R3×3 . s C Then the decoder is the fixed quadratic map f (S) = αC − sr ⊤ − C[r]× ,

(15)

where [r]× is the 3 × 3 cross-product matrix, defined by [r]× x = r × x for every x ∈ R . 3

Moreover, for every finite subgroup H ⊂ SO(3), the group satisfies

ΓH := ⟨g(A) : A ∈ H⟩ ⊂ SO(4)

|ΓH | ≤ 2|H|2 .

(16)

Here ΓH is the group generated by the encoded transitions. In particular, starting from I4 , products of encoded inputs from H reach at most 2|H|2 distinct hidden matrices. Proof. We first verify the encoder, then construct the decoder through its action on a threedimensional space of matrices, and finally prove the finite-state bound. The encoded transitions. The vector aA in equation 14 has unit norm, so I4 − 2aA a⊤ A is a Householder reflection. The fixed matrix D0 is also a reflection: it changes the sign of the first coordinate. Their product is therefore orthogonal with determinant +1, and has the required signed-Householder form. At θ = 0, the construction gives g(I3 ) = I4 independently of the chosen axis. At θ = π, the two choices u and −u replace aA by its negative and hence give the same Householder matrix. Thus the encoder is well defined for every rotation. Multiplying the two factors gives c −su⊤ g(Ru,θ ) = , su I3 + (c − 1)uu⊤

c := cos θ,

s := sin θ.

(17)

This matrix rotates the plane spanned by the first coordinate vector e0 = (1, 0, 0, 0)⊤ and (0, u)⊤ through angle θ, and fixes the orthogonal complement of that plane. A decoder that preserves multiplication. Let e0 , . . . , e3 be the standard basis of R4 , and define ⊤ Bij := ei e⊤ j − ej ei ,

J1 := B01 + B23 ,

J2 := B02 + B31 ,

J3 := B03 + B12 .

The three skew-symmetric matrices J1 , J2 , J3 are linearly independent. For a coefficient vector x ∈ R3 , write their linear combination as 3 X 0 x⊤ . J (x) := xi Ji = −x −[x]× i=1

We will show that conjugation X 7→ QXQ⊤ by any Q ∈ SO(4) maps this three-dimensional space to itself. Its action on the coefficient vector x will be the decoder f (Q). To verify this invariance, it suffices to check two elementary types of rotation. First, for TR := Diag(1, R) with R ∈ SO(3), block multiplication and the identity R[x]× R⊤ = [Rx]× give ⊤ TR J (x)TR = J (Rx).

Second, let Uϕ rotate the first two coordinate directions in R4 , and let Rϕ rotate the last two coordinate directions in R3 :   ! cos ϕ − sin ϕ 0 0 1 0 0  sin ϕ cos ϕ 0 0 , Rϕ := 0 cos ϕ − sin ϕ . Uϕ :=  0 0 1 0 0 sin ϕ cos ϕ 0 0 0 1 Multiplication on the three basis matrices gives Uϕ J1 Uϕ⊤ = J1 , Uϕ J2 Uϕ⊤ = cos ϕ J2 + sin ϕ J3 , Uϕ J3 Uϕ⊤ = − sin ϕ J2 + cos ϕ J3 . Equivalently, Uϕ J (x)Uϕ⊤ = J (Rϕ x). Rotations among the last three coordinates are included in TR . Conjugating Uϕ by suitable TR gives rotations between the first coordinate and either of the other two coordinates. Thus these matrices generate all coordinate-plane rotations in R4 . Every matrix in SO(4) is a product of such rotations, as follows, for example, by eliminating its entries with Givens rotations. Invariance therefore holds for every Q ∈ SO(4). There is consequently a unique 3 × 3 matrix M (Q) such that QJ (x)Q⊤ = J (M (Q)x)

(x ∈ R3 ).

(18)

On the two elementary types above, this matrix is respectively R and Rϕ , both in SO(3). Applying two conjugations in succession gives (Q1 Q2 )J (x)(Q1 Q2 )⊤ = J M (Q1 )M (Q2 )x . Uniqueness of the coefficients implies that M preserves multiplication. Since the elementary rotations generate SO(4), it also follows that M (Q) ∈ SO(3) for every Q ∈ SO(4). It remains to identify this induced matrix with the explicit decoder. Write Q in the block form used in the statement. The top-right block of QJ (x)Q⊤ is ⊤ −(r ⊤ x)s⊤ + (αx⊤ − r ⊤ [x]× )C ⊤ = (αC − sr ⊤ − C[r]× )x . Here we used r ⊤ [x]× = ([r]× x)⊤ . The top-right block of J (M (Q)x) is (M (Q)x)⊤ , so M (Q) = f (Q) as defined in equation 15. We have therefore proved f (Q) ∈ SO(3),

f (I4 ) = I3 ,

f (Q1 Q2 ) = f (Q1 )f (Q2 ).

(19)

Decoding the accumulated product. For the blocks in equation 17, αC − sr ⊤ = uu⊤ + c(I3 − uu⊤ ),

−C[r]× = s[u]× ,

where the second identity uses u⊤ [u]× = 0. Hence Rodrigues’ rotation formula gives f (g(A)) = uu⊤ + c(I3 − uu⊤ ) + s[u]× = Ru,θ = A.

(20)

This also proves injectivity: if g(A) = g(B), applying f gives A = B. Combining equation 19 and equation 20 proves equation 12 for every hidden matrix in SO(4). Repeated application gives equation 13. Finitely many hidden states for a finite input group. We introduce a second triple of matrices K1 := B01 − B23 ,

K2 := B02 − B31 , 35

K3 := B03 − B12 ,

and write their linear combinations as K(x) :=

3 X

xi Ki =

i=1

0 −x

x⊤ . [x]×

The same elementary rotations act on this triple by ⊤ TR K(x)TR = K(Rx),

Uϕ K(x)Uϕ⊤ = K(R−ϕ x).

Thus its span is also invariant. The induced matrix is an auxiliary map f− : SO(4) → SO(3) that preserves multiplication. Reading the top-right block as before gives QK(x)Q⊤ = K(f− (Q)x),

f− (Q) = αC − sr ⊤ + C[r]× .

(21)

Substituting equation 17 now reverses the sign of the sine term in Rodrigues’ formula, so f− (g(A)) = Ru,−θ = A−1 .

(22)

The two maps together determine a hidden matrix up to sign. To see this, suppose f (Q) = f− (Q) = I3 . Then conjugation by Q fixes all six matrices Ji , Ki . Their sums and differences span all 2 the Bij , so Q commutes with every Bij . It therefore also commutes with −Bij , the orthogonal projector onto the coordinate plane spanned by ei , ej . Thus Q preserves every coordinate plane, and also every coordinate axis, obtained by intersecting two such planes. This makes Q diagonal. Commutation with Bij then forces its i-th and j-th diagonal entries to agree. Hence Q is a scalar matrix, and orthogonality gives Q = ±I4 . Conversely, both of these matrices act trivially by conjugation. Define the joint map F (Q) := (f (Q), f− (Q)). It preserves multiplication and satisfies F (Q) = (I3 , I3 ) exactly when Q = ±I4 . Its kernel therefore has two elements, so every nonempty fiber contains exactly two matrices in SO(4), differing by sign. Now let H ⊂ SO(3) be finite. For every encoded generator, F (g(A)) = (A, A−1 ) ∈ H × H. Since H × H is closed under multiplication and inverses, F (ΓH ) ⊆ H × H. There are at most |H|2 such pairs and at most two hidden matrices for each pair. Consequently |ΓH | ≤ 2|H|2 , proving equation 16. Applying Proposition 18 to the icosahedral group gives our four-dimensional A5 tracker. Corollary 19 (Four-dimensional A5 tracker). The group A5 admits finite-state signed-Householder tracking in dimension four, implemented by one CKDA layer. Its sixty transitions use the encoder in equation 14, with the fixed diagonal gate Diag(−1, 1, 1, 1), and its decoder is the quadratic map in equation 15. There are at most 7200 reachable hidden matrices, and the decoder is many-to-one on this set. Proof. Identify A5 with the icosahedral rotation group and apply Proposition 18 with H = A5 . Starting from S0 = I4 , the transitions Ah = g(h) track the group with decoder f and zero additive terms. They are distinct and orthogonal, and the reachable set has size at most 2 · 602 = 7200. For the many-to-one claim, choose noncommuting h, k ∈ A5 and set ℓ = (hk)−1 . The two maps from the proposition satisfy f (Ah Ak Aℓ ) = e,

f− (Ah Ak Aℓ ) = h−1 k −1 hk ̸= e.

Thus this reachable product differs from I4 , although both decode to the identity. C.5

A DDITIVE TERMS AND INDEPENDENT HEADS

Under the finite-reachability convention of Section B.3, for group tracking with non-expansive updates, the additive term Bg can be removed. Restricting to a suitable invariant subspace then gives orthogonal update matrices, without increasing eigenvalue multiplicities or the number of state columns. 36

Proposition 20 (Removal of the additive term and orthogonal restriction). Let G be a finite group and let Tg (S) = Ag S + Bg , g ∈ G, act on S ∈ Rn×m , with ∥Ag ∥2 ≤ 1 for every g. Suppose the reachable set R from S0 is finite and admits a decoder satisfying d(S0 ) = e and d(Tg (S)) = g d(S) for all g ∈ G and S ∈ R. Then there is a finite-state tracker of G with states in Rr×m , where bt = A bg S bt−1 with orthogonal matrices A bg . These matrices represent 0 ≤ r ≤ n, and updates S t the restrictions of the original Ag to a common invariant subspace in an orthonormal basis. Every bg is an eigenvalue of Ag with no increase in algebraic multiplicity. Both decoders eigenvalue of A may be many-to-one. Proof. We first choose a word whose linear part has minimum Frobenius norm and repeat it until it acts as a projection without changing the decoded value. Minimality and non-expansion then force the original matrices to act orthogonally on its image. The resulting affine updates permute a finite set of states; subtracting their average removes the additive term, and expressing the centered states in an orthonormal basis of the image gives the orthogonal tracker. Step 1: Find a repeatable word that leaves the decoded value unchanged. Let V be the span of all columns of differences X − Y with X, Y ∈ R. Since Ag (X − Y ) = Tg (X) − Tg (Y ), each Ag preserves V . For a word u = g1 · · · gt , write Tu = Tgt ◦ · · · ◦ Tg1 , Pu = Agt · · · Ag1 , and gu = gt · · · g1 . Thus d(Tu (S)) = gu d(S). Only finitely many linear actions Pu |V occur. Indeed, there are finitely many maps R → R, and if two word maps agree there, subtracting their values shows that their linear parts agree on the columns spanning V . Choose u minimizing the Frobenius norm ∥Pu |V ∥F . Put N = |R|!. The orbit S, Tu (S), Tu2 (S), . . . enters a cycle within |R| − 1 steps: among its first |R| + 1 states, two coincide. Every cycle length is at most |R| and divides N . Thus another N applications after the first N return to the same state, giving Tu2N = TuN on R. Let z be u repeated N times. Then Tz = TuN and Tz2 = Tz on R. Subtracting this identity at two reachable states shows that its linear part E := Pz |V satisfies E 2 = E. It still has minimum Frobenius norm: further applications of a non-expansive matrix cannot increase that norm, and E is itself a linear word action. Decoding at S0 gives gz2 = gz because d(S0 ) = e. Multiplying by gz−1 gives gz = e. Thus applying z may merge states, but never changes their decoded group element. Step 2: Obtain orthogonal updates on the surviving subspace. The identity E 2 = E makes E a projection onto im E. It is an orthogonal projection because it cannot increase lengths. Indeed, for w ∈ im E, let z be the orthogonal projection of w onto ker E. Then E(w − z) = w, so ∥w∥2 = ∥E(w − z)∥2 ≤ ∥w − z∥2 = ∥w∥2 − ∥z∥2 . Hence z = 0 and im E ⊥ ker E. In words, removing a kernel component shortens the vector, and a non-expansive map cannot reconstruct the longer vector. Write r = dim im E and choose any orthonormal basis w1 , . . . , wr of im E. Since E is an orthogonal projection, ∥E∥2F = r. The product EAg E is another linear word action on V , so minimality gives r X r ≤ ∥EAg E∥2F = ∥EAg wi ∥2 ≤ r. i=1

The sum has r terms, each at most one, so all must equal one: no basis vector can lose length. Any unit vector w ∈ im E can be included in such a basis. Consequently, 1 = ∥EAg w∥ ≤ ∥Ag w∥ ≤ 1. Equality for the orthogonal projection implies Ag w ∈ im E, and the remaining equality gives ∥Ag w∥ = ∥w∥. Thus every Ag preserves im E and acts orthogonally there. Step 3: Center the states and construct the orthogonal tracker. Let R∗ := Tz (R). Every state in R∗ is fixed by Tz , so its differences satisfy X − Y = E(X − Y ) and have columns in im E. For each input g, define Tg∗ := Tz ◦ Tg : apply g, then z. This sends R∗ into itself. For X, Y ∈ R∗ , Step 2 gives Tg∗ (X) − Tg∗ (Y ) = EAg (X − Y ) = Ag (X − Y ). This preserves distances, so distinct states stay distinct. A one-to-one P map of a finite set into itself is a permutation. Consequently, Tg∗ fixes the average S := |R∗ |−1 S∈R∗ S: affine maps preserve 37

averages, and a permutation leaves the average unchanged. Since differences S − S have columns in im E, we obtain Tg∗ (S) − S = Ag (S − S), S ∈ R∗ . Thus subtracting this common average turns every update into multiplication by the original Ag . Choose U ∈ Rn×r whose columns form an orthonormal basis of im E. For a vector in this subspace, U ⊤ returns its coordinates in that basis and U reconstructs the vector. Define bg = U ⊤ Ag U , bt = A bg S bt−1 . b0 = U ⊤ Tz (S0 ) − S , A S S t

bg and shows that A bg is orthogonal: it is the same length-preserving action Step 2 gives Ag U = U A on im E, expressed in orthonormal coordinates. For every S ∈ R∗ , the centering identity above gives bg U ⊤ (S − S) = U ⊤ T ∗ (S) − S . A g b Thus the new updates preserve the finite set U ⊤ (R∗ − S). On this set, define d(X) = d(U X + S): reconstruct the original state and apply its decoder. Since gz = e, we have d(Tz (S0 )) = e and d(Tg∗ (S)) = g d(S). Consequently, bS bA b b0 ) = e, bg X) = d T ∗ (U X + S) = g d(X). d( d( g This is the required tracker with orthogonal matrices and no additive term. Its states have r rows and the same m columns. If r = 0, the centered set is a singleton, forcing G = {e}; the zero-dimensional state suffices. bg Finally, completing the columns of U to a basis of Rn puts Ag in block triangular form with A bg as a factor: every as a diagonal block. Its characteristic polynomial therefore contains that of A b eigenvalue of Ag occurs in Ag at least as many times. Proposition 21 (Orthogonal reduction for independent heads). Let G be a finite group and consider H < ∞ independent heads with exact affine updates (h) Tg(h) (S (h) ) = A(h) + Bg(h) , g S

∥A(h) g ∥2 ≤ 1,

for g ∈ G and h = 1, . . . , H. Suppose each head has a finite reachable set from its fixed initial state and a joint decoder tracks G. Then there is a finite-state tracker of G with homogeneous orthogonal updates on the same heads, allowing zero-dimensional heads. Each reduced head transition is a restriction of its original transition to a common invariant subspace, so eigenvalue multiplicities do not increase in any head. The joint transition matrices generate a finite group Γ with a surjective homomorphism p : Γ → G taking the transition for each input g to g. QH Proof. Let Rh be the finite reachable set of head h. The joint reachable set satisfies R ⊆ h=1 Rh and is therefore finite, since H < ∞. Stack the head states, padding their columns with zeros if (1) (H) needed, so the joint linear part is Ag = diag(Ag , . . . , Ag ). Apply Proposition 20 and express the resulting centered states in the original coordinates. Their reachable set is finite, and every Ag acts orthogonally on its column span W . Let Ph project onto head h and put Wh = Ph W . Block diagonality makes each Wh invariant. For w ∈ W , H X

2 2 2 ∥A(h) g Ph w∥ = ∥Ag w∥ = ∥w∥ =

h=1

H X

∥Ph w∥2

h=1 (h)

forces equality in each head, since no head can increase the norm. Thus Ag |Wh is orthogonal. Express these restrictions in orthonormal bases of the Wh . Their eigenvalues, with algebraic multiplicities, are inherited from the original head transitions. Projecting the centered states onto all heads preserves their finite joint reachable set and its decoder: the projections together determine the entire centered state. Each reduced head transition permutes its projected finite reachable set. Its columns span Wh , so the action on that set determines the matrix. Consequently only finitely many joint matrix products b0 and decoder occur; being orthogonal, they form a finite group Γ. For the reduced initial state S b b b d, define p(P ) = d(P S0 ). Every element of Γ is represented by an input word, and the tracking bg ) = g. Thus p is a surjective homomorphism. identity gives p(P Q) = p(P )p(Q) and p(A 38

Stacking the reduced heads gives one block-diagonal orthogonal update, whose spectrum is the multiset union of the head spectra. Non-real eigenvalue pairs therefore add across heads: this combination does not preserve a bound on their number per head or the original single-head transition family. C.6

A SPECTRAL OBSTRUCTION TO FINITE - STATE TRACKING OF S5

The previous construction tracks A5 , the 60 even permutations of five objects. We now show that the same spectral constraint cannot support S5 , which contains all 120 permutations. The model must handle every permutation and their compositions, even when several hidden states represent the same output. Theorem 22 (Spectral obstruction to S5 tracking; restatement of Theorem 4). Suppose every head transition A satisfies ∥A∥2 ≤ 1 and has at most one non-real conjugate eigenvalue pair on the unit circle, counted with algebraic multiplicity. Then a single recurrent layer with any finite number of independent heads cannot track S5 when inputs range over all of S5 and the exact affine updates of each head reach only finitely many states from its fixed initial state. This holds for arbitrary state dimensions, input-dependent additive matrices, and many-to-one joint state decoders. Proof idea. After the orthogonal reduction, we construct two conjugate transition-matrix products that act in each head either as the identity or as rotations of one plane through the same angle. Their commutator, the product R1 R2 R1⊤ R2⊤ , returns to the identity after ten repetitions by Lemma 23. Its decoded permutation is a three-cycle, whose tenth power is not the identity. This contradiction applies to any number of heads and does not require an injective decoder. We prove the rotation lemma after the main argument. Proof of Theorem 4. Suppose a tracker exists. By Proposition 21, we may remove the additive terms and restrict each head to orthogonal transitions, without increasing eigenvalue multiplicities. The resulting block-diagonal matrices generate a finite group Γ, with a surjective homomorphism p : Γ → S5 satisfying p(Ag ) = g. This reduction applies to one head as well as to several. Construct the rotations in each head. Let c = (12345) and t = (12). Since p(Ac ) = c, the order of Ac is divisible by five; write it as 5a r, with a ≥ 1 and 5 does not divide r. Choose E > 0 divisible by 2r and congruent to 1 modulo 5; such a choice exists because 2r is coprime to 5. Set R1 = AE c ,

R2 = At R1 A⊤ t .

The even exponent turns all real eigenvalues ±1 into 1, and its factor r removes every order factor coprime to five. Thus each head of R1 is either the identity or a planar rotation of nontrivial powerof-five order. The corresponding head of R2 is its orthogonal conjugate, with the same angle and order. Meanwhile, p(R1 ) = cE = c,

p(R2 ) = tct−1 = (13452) =: b.

Compare the matrix product with its decoded permutation. Let Rj,h denote the block of Rj in head h, and define ⊤ ⊤ Ch = R1,h R2,h R1,h R2,h ,

C = R1 R2 R1⊤ R2⊤ = diag(C1 , . . . , CH ).

Each pair of head rotations belongs to a finite group, as the image of Γ on that head. By Lemma 23, Ch10 = I in every nonidentity head, and the same holds in identity heads. Hence C 10 = I. But direct permutation composition gives p(C) = cbc−1 b−1 = (142), so e = p(C 10 ) = (142)10 = (142) ̸= e, a contradiction. Lemma 23 (A constraint on two planar rotations). Let R1 , R2 ∈ O(n) belong to a finite group. Suppose they are planar rotations through the same angle, possibly in different planes, and their common order is a multiple of five. Angles are measured in [0, π], ignoring direction. Then (R1 R2 R1⊤ R2⊤ )10 = I. 39

Proof. First prove a three-dimensional version. Let X, Y ∈ SO(3) satisfy the lemma’s assumptions, and write θ for their common angle. In particular, their traces are equal: tr X = tr Y = 1 + 2 cos θ. We claim that D := XY X Y satisfies D 5 = I3 . If their axes coincide, the rotations commute and D = I3 . If their axes differ, we use the finite rotation-group classification recalled before Theorem 17. In a cyclic or dihedral rotation group, all rotations of order greater than two share an axis. Tetrahedral and octahedral rotations have orders at most four. Therefore only the rotations of an icosahedron remain possible. ⊤

⊤

In this case, X, Y rotate about distinct axes through √ opposite vertices, and θ is 2π/5 or 4π/5. We now choose these axes explicitly. Let φ = (1 + 5)/2. Place the icosahedron at the origin with vertices (0, ±1, ±φ), (±1, ±φ, 0), (±φ, 0, ±1), where the signs are independent. Each pair of opposite vertices defines one of the six vertex axes. By symmetry, we may take the axis of X through (0, 1, φ). Rotation by 2π/5 about this axis cycles the other five vertex axes, so we may take the axis of Y through (1, φ, 0). Therefore we can choose unit axis vectors (0, 1, φ)⊤ (1, φ, 0)⊤ u = ±p , v = ±p . 1 + φ2 1 + φ2 Choose their signs so that X, Y both rotate through θ with the right-hand rule. The signs do not affect the squared inner product: (u⊤ v)2 =

φ2 1 = , 2 2 (1 + φ ) 5

where the last equality follows from φ2 = φ + 1 and (1 + φ2 )2 = 5φ2 . It remains to show that tr D = 1 + 2 cos θ. Write x = cos θ. Rodrigues’ formula expresses the two rotations as X = xI3 + (1 − x)uu⊤ + sin θ[u]× ,

Y = xI3 + (1 − x)vv ⊤ + sin θ[v]× ,

where [w]× z = w × z. Substituting into D = XY X ⊤ Y ⊤ and taking the trace gives 2 2 tr D = 2 − (1 − x)2 1 − (u⊤ v)2 − 1 = 2 − 54 (1 − x)2 − 1. Thus the factor 4/5 comes from the angle between the two vertex axes. √ It remains to substitute the rotation angle: since θ ∈ {2π/5, 4π/5}, we have x = (−1 ± 5)/4. Both values satisfy 4x2 + 2x − 1 = 0, which implies 54 (1 − x)2 = 1 − 2x. Therefore tr D = (1 + 2x)2 − 1 = 1 + 2x = tr X = tr Y , where the second equality again uses 4x2 + 2x − 1 = 0. For a three-dimensional rotation, the trace 1 + 2 cos θ uniquely determines the angle in [0, π]. Hence D also rotates through 2π/5 or 4π/5, so D 5 = I3 as claimed. Apply this calculation to the original planar rotations. Their two rotating planes span a space U of dimension at most four. Both matrices preserve U and leave every vector in U ⊥ unchanged. We can therefore restrict to U and, if needed, add coordinates on which both matrices act as the identity. This gives matrices in SO(4) with the same angles and orders, still generating a finite group. The proof of Proposition 18 constructs two maps f, f− : SO(4) → SO(3) that preserve multiplication. Each sends a planar rotation to a three-dimensional rotation with the same angle: in a basis aligned with the rotating plane, the images rotate through θ and −θ, respectively. Moreover, if both maps send a matrix to the identity, that matrix must be I4 or −I4 : f (P ) = f− (P ) = I3

=⇒

P ∈ {I4 , −I4 }.

For either map, the images of R1 , R2 thus satisfy the three-dimensional claim. Writing C = R1 R2 R1⊤ R2⊤ , we obtain f (C 5 ) = f− (C 5 ) = I3 . 5 Therefore C = ±I4 , and squaring gives C 10 = I4 . Since the original matrices fix U ⊥ , the same identity holds in the original dimension. 40

Consequences for CKDA and Gated DeltaProduct. With unit keys, CKDA transitions with β ∈ [0, 2] and αi ∈ [−1, 1] are non-expansive, and Theorem 9 bounds their non-real unit-circle pairs by one. Gated DeltaProduct transitions with βj ∈ [0, 2] and scalar αg ∈ [0, 1] are also non-expansive. Expanding the product of k Householder transformations gives rank(Ag − αg I) ≤ k. Since αg is real, at most k eigenvalues can be non-real, counted with algebraic multiplicity. They occur in conjugate pairs, so k ≤ 3 allows at most one such pair per head. Theorem 4 therefore rules out both architectures under finite reachability in each head, including input-dependent additive matrices and arbitrary joint decoders. For sufficiency at k = 4, use the permutation-matrix construction of Siems et al. (2025, Thm. 1). Every permutation of five objects is a product of at√most four transpositions. Each transposition (ij) ⊤ , where kij = (ei − ej )/ 2 is a unit key. Thus four Householder transforhas matrix I5 − 2kij kij mations suffice, padding shorter products with βj = 0 and choosing βj = 2 for the transpositions. Set the scalar gate to α = 1, the additive term to zero at every step, and the initial state to I5 . The state is then exactly the permutation matrix of the running group product, so a single head tracks S5 with precisely 120 reachable states and orthogonal transitions. Any k > 4 is also sufficient by padding with identities. Together with the lower bound, this proves that four is the minimum number of Householder transformations needed for single-layer Gated DeltaProduct tracking of S5 under these assumptions.

T HREE - LAYER SIMULATION OF FINITE AND WEIGHTED AUTOMATA

We prove Theorem 5 using the tasks and arithmetic conventions of Sections B.1 and B.4. The construction has three recurrent layers: a clock, a buffer, and an accumulator. CKDA implements the clock in one layer, saving one layer relative to the cited four-layer DeltaNet construction. D.1

T HE CLOCK , BUFFER , AND ACCUMULATOR

The idea is to give the recurrence several steps to apply a complicated transition. Choose m large enough that any block transitions (the product of all transition matrices can be factored into at most m permitted matrices. Following Siems et al. (2025, Thm. 3 and its proof), collect m consecutive inputs and factor their combined transition, padding with identities when necessary. Apply these factors one at a time while reading the next block: • The clock records the position modulo 2m, telling the later layers which buffer slot and matrix factor to use. • The buffer retains the last 2m tokens. Its feed-forward lookup selects the factors for the previous block while retaining the inputs arriving in the current block. • The accumulator applies the selected factor at each step, carrying forward the product from earlier blocks. For example, with m = 3, inputs 4, 5, 6 provide three steps to apply the three factors representing inputs 1, 2, 3. The readout corrects this delay: it completes the unapplied factors and then applies the current block’s prefix. The first block is handled directly by lookup. The same organization applies to the column-state matrix products in Section B.1. The buffer and completion readout are those in Peng et al. (2025, Appendix D) and Siems et al. (2025, Thm. 3, Lems. 2–3), extended to rational WFAs by Merrill et al. (2026a, Thms. 6 and 11); we refer to these works for their constructions. The buffer uses DeltaNet updates, which are also CKDA updates with identity diagonal gate. Below we give the CKDA clock, the factor counts determining m, and the resulting simulation argument. D.2

A ONE - LAYER CLOCK

DeltaNet obtains the clock rotation from two alternating reflections, using a parity layer to select between them (Merrill et al., 2026a, Lem. 7). CKDA combines the two reflections within one transition: its signed diagonal supplies one of them. 41

Lemma 24 (One-step modular counter). For every integer m ≥ 1 there is a one-layer CKDA head whose readout determines i mod 2m at every position i. Proof. With β = 2, α = (1, −1) and k = (− sin φ2 , cos φ2 )⊤ the transition (I − 2kk⊤ ) Diag(α) is a rotation by φ. Set φ = π/m and c0 = e1 . Then ci = (cos(πi/m), sin(πi/m))⊤ visits 2m distinct points. Choose a fixed unit query q such that q ⊤ (ci − cj ) ̸= 0 for 0 ≤ i < j < 2m. Such a query exists because only finitely many lines are excluded. The single scalar readout q ⊤ ci then distinguishes all clock states, and a finite lookup returns the residue. This is similar to the DeltaProductk k ≥ 2 counter in Siems et al. (2025, Lem. 2(ii)). Its rotation parameters are algebraic; the fixed query can also be chosen algebraic by the same argument as in Section B.3. The clock uses the fixed exact datatype of Section B.4. D.3

ACCUMULATOR FACTORIZATIONS

The accumulator applies one matrix factor per recurrent step. We therefore bound the number of factors required for permutation, deterministic, and rational transitions. These bounds determine the block length m ≥ 1 in Section D.1: any combined block transition must admit a factorization with at most m factors. Proposition 25 (Factor-count upper bounds for the accumulator). Let n ≥ 1 be the automaton-state dimension. A deterministic transition is an n × n matrix with one entry equal to 1 in each column and all other entries zero. The following numbers of recurrent matrix factors are sufficient; these construction-dependent bounds need not be optimal: matrix family or primitive

RWKV-7 DeltaNet/GDN

one transposition arbitrary permutation one DFA column-copy primitive deterministic transition, explicit construction deterministic transition, up to positive scale arbitrary rational n × n matrix

1 n−1 1 n n 2n

CKDA

1 1 n−1 n−1 4 2 4n 2n − 2 3n max{1, 2n − 2} (8n2 + 5n + 1)† 7n2 + 3n

The GDN entries 4 and 4n retain a conservative allowance of four factors per DFA column-copy primitive, defined below. The positive-scale row uses an exact-real singular value decomposition (SVD) for GDN and CKDA: it factors ρM for some ρ > 0, where M is the transition, and gives no fixed rounded-datatype guarantee. The rational-matrix row permits scratch coordinates (auxiliary storage): RWKV-7 operates in dimension 2n, and DeltaNet/GDN and CKDA in dimension 2n + 1. The dagger marks the cited DeltaNet/GDN budget with unrestricted unit-key learning rate β ∈ R; the CKDA budget uses β ≥ 0 and diagonal entries in [−1, 1]. The explicit CKDA deterministic construction needs only β ∈ [0, 3]; permutations use β = 2 reflections. A product with no factors is the identity; the swap and column-copy rows require n ≥ 2. Proof. Write ej for the jth standard basis vector and I for the identity in the current state dimension. For a scalar γ and vector u, write Hγ (u) = I − γuu⊤ . A CKDA factor Hγ (u)D applies a diagonal gate D first and a generalized Householder transformation second. For u ̸= 0, its unit key is k = u/∥u∥2 and its learning rate is β = γ∥u∥22 . Rational transition entries do not imply rational normalized keys. Transpositions and permutations. A transposition (a swap of coordinates i ̸= j) is H1 (ei − ej ), so a permutation costs at most n − 1 factors (Siems et al., 2025, Thm. 3, proof). RWKV-7 also implements each swap in one factor (Peng et al., 2025, Lem. 4). For CKDA, this worst-case count is sharp by the n-cycle example in Proposition 10. Deterministic transitions: explicit construction. For distinct coordinates s, d, the DFA column-copy primitive merges state d into state s: on a column state x, it sends (xs , xd ) to (xs + xd , 0) and leaves other coordinates unchanged. Set Dd = I − ed e⊤ d , the diagonal gate that clears coordinate d. The merge has the factorization ⊤ Md→s := I − ed e⊤ d + es ed = H3 (es )Dd H1/6 (3es + ed ). 42

In the (s, d) coordinates, this identity is simply −2 0 −1/2 0 0 −1/2

−1/2 1 = 5/6 0

1 . 0

The effective learning rates are 3 and 5/3, so each column copy uses two CKDA factors. Separating Dd = H1 (ed ) gives three GDN factors. RWKV-7 uses one factor per identity, swap, or column copy and at most n such primitives per deterministic transition (Peng et al., 2025, Lems. 3–4), yielding the RWKV-7 and GDN bounds. For CKDA, use one merge per noncycle vertex and ℓ − 1 swaps per cycle of length ℓ. To see this, let f : {1, . . . , n} → {1, . . . , n} be the deterministic state map of the underlying deterministic automaton, so M ei = ef (i) . In the graph with arrows i → f (i), every component is a cycle with trees feeding into it. Let k count the vertices on cycles and c count the cycles, including fixed points. Right-multiplying by Md→s replaces column d by column s. Thus, starting from I, rightmultiply by Mi→f (i) for each noncycle vertex i, processing vertices in decreasing distance from their cycle. This replaces column i by column f (i), which is still the original ef (i) . The remaining cycle columns are then arranged by swaps within each cycle, leaving completed noncycle columns unchanged. The column-state recurrence applies the resulting factors from right to left. Since a merge costs two CKDA factors and a swap costs one, the total is 2(n − k) + (k − c) = 2n − k − c ≤ 2n − 2, | {z } | {z }

noncycle merges

cycle swaps

because k ≥ c ≥ 1. For n = 1, the transition is the identity and needs no factors. Deterministic transitions: exact-real SVD. Choose 0 < ρ ≤ 1/ max(1, ∥M ∥2 ), where ∥ · ∥2 is the spectral norm, and write the SVD ρM = U ΣV ⊤ . Here U , V are orthogonal and Σ is diagonal with entries in [0, 1]. Positive scaling preserves the position of the nonzero entry in a one-hot DFA state, so that state remains exactly decodable. For GDN, the two orthogonal factors cost at most n reflections each, and the diagonal costs another n generalized Householders, giving 3n factors (Grazzi et al., 2025, Prop. 1, item 2). For CKDA, Proposition 10 applies to the contraction ρM and gives at most max{1, 2n − 2} factors by absorbing the signed diagonal contraction into an adjacent Householder factor. Arbitrary rational matrices. Let M ∈ Qn×n be the target matrix and x ∈ Rn the input state. The RWKV-7 entry follows from the 2n-factor construction in Merrill et al. (2026a, Lems. 5–6). In M their row-state convention, the factor product is ( M 0 0 ). Applying these same factors in reverse execution order to a column state maps (x, 0) to (M x, 0), so scratch coordinates remain zero at block boundaries. For CKDA, adapt the 8n2 + 5n + 1-factor DeltaNet program of Merrill et al. (2026a, Lems. 11–12), using n main coordinates x, n scratch coordinates s, and one temporary coordinate t. First clear s and t. A unit coordinate addition (transvection) adds coordinate s to coordinate d ̸= s of this augmented state, leaving all others unchanged. It uses three factors: I + e d e⊤ s = H1/3 (es + 2ed )H1/2 (es )H2 (es + ed ). This is the column-state transpose of Merrill et al. (2026a, Lem. 10). Each scaled addition si ← si + Mij xj , for 1 ≤ i, j ≤ n, uses t ← t + xj , t ← Mij t, si ← si + t, then t ← 0, where Mij is the (i, j) entry of M . CKDA absorbs this clear into the diagonal gate of the next Householder factor, reducing the cost from eight to seven factors. For a rational scaling coefficient a > 0, define the coordinate sign flip Sj = I − 2ej e⊤ j and use H1+a (ej )Sj to scale coordinate j by a. For a < 0, use H1+|a| (ej ), and for a = 0 use H1 (ej ). Each is one admissible CKDA factor with nonnegative learning rate. The cited GDN scaling H1−a (ej ) instead needs a negative learning rate for a > 1 (Merrill et al., 2026a, Lem. 9). The initial clear of scratch and temporary coordinates is absorbed into the first Householder factor. At the end, the final temporary clear and all main-coordinate clears are absorbed into the first of n three-factor unit additions that copy scratch back to main. The total is therefore n2 (3 + 1 + 3) | {z }

scaled additions; clears absorbed

+ |{z} 3n = 7n2 + 3n.

copy back

The program maps (x, s, t) to (M x, M x, 0) for arbitrary initial scratch and temporary contents. The DFA simulation below uses the explicit CKDA construction and the fixed exact datatype of Section B.4. D.4

T HREE - LAYER EXPRESSIVITY

We now combine the clock with the cited buffer and completion readout. The exact finite-datatype and polynomial-precision conventions below are those defined in Section B.4. Theorem 26 (Three layers suffice; restatement of Theorem 5). Three CKDA layers solve every finite group-word problem under the exact finite-datatype convention above. Allowing β > 2 also gives every regular language under that convention and every WFA over Q in polynomial precision, using exact arithmetic over the fixed algebraic number field specified above. Proof. Use the clock–buffer–accumulator construction of Merrill et al. (2026a, Thm. 11), summarized in Section D.1, replacing its two clock layers by Lemma 24. The remaining layers stream the permutation, DFA, or rational WFA factors from Proposition 25, with m = max(1, n − 1), m = max(1, 2n − 2), or m = 7n2 + 3n, respectively. The exact finite-datatype and polynomialstorage guarantees follow from Section B.4. D.5

C OMPARISON WITH OTHER TRANSITION FAMILIES

Table 1 compares transition families under the expressive decoder and precision conventions used in the cited constructions. Its group rows allow every group element as an input token. In particular, the one-layer swap-tracking result of Peng et al. (2025, Lem. 2) does not give one-layer tracking of arbitrary S5 inputs. The displayed depths are upper bounds from constructions; a one-layer obstruction and a three-layer construction leave the two-layer case undecided. Qk Spectral restrictions. For scalar-gated DeltaProduct, write A = α j=1 (I − βj kj kj⊤ ). Every vector orthogonal to all keys is an eigenvector with real eigenvalue α. Therefore at most min(d, k) eigenvalues can be non-real, giving at most ⌊min(d, k)/2⌋ conjugate pairs. For RWKV-7 and GDN2 with nonnegative decay and erase√ gates, the transition has the form D − u(v ⊙ u)⊤ with v ≥ 0. When v > 0, conjugation by Diag( v) makes this matrix symmetric; zero entries follow by continuity. Hence its spectrum is real, and the one-layer impossibility result of Grazzi et al. (2025, Thm. 2 and App. B.2), under their finite-precision assumptions, applies to every group row containing an element of order greater than two. The unrestricted WFA parameterizations are not subject to this nonnegative-gate assumption. RWKV-7 and GDN-2. The shared RWKV-7/GDN-2 entries are transfers between transition families, not additional theorems claimed by the GDN-2 paper. For the finite-state constructions, the table uses unit keys, decay entries in [0, 1], and erase entries in [0, 2], including the endpoints. In RWKV-7 notation, this absorbs the factor c = 2 into the erase gate. The c = 1 adaptation in Peng et al. (2025, Appendix D.3) instead rescales the state and uses normalization and growing exponent storage; our fixed-datatype entries refer to the c = 2 construction. Both contain DeltaNet when the decay is one and the erase gate is constant. This transfers the group constructions of Siems et al. (2025, Thms. 1 and 7). For GDN-2, use its update in Hatamizadeh et al. (2026, Eq. (10)) with the erase range extended to [0, 2]. It also implements the DFA column-copy primitive that sends state j √ to state i: take D = I, k = (ei − ej )/ 2, and erase gate 2ej , obtaining I + (ei − ej )e⊤ j . Thus it supports the identity, swap, and copy primitives in the four-layer regular-language construction of Peng et al. (2025, Thm. 3). The WFA entry instead uses unrestricted parameters, as in Merrill et al. (2026a, Thm. 6); GDN-2 inherits their DeltaNet construction by allowing an unrestricted constant erase gate. DeltaProduct and automata bounds. The DeltaProduct automata entries combine existing constructions. Siems et al. (2025, Lem. 2(ii)) provide its one-layer clock, which replaces the two clock layers in the four-layer DeltaNet simulation of Merrill et al. (2026a, Thm. 11). This gives three 44

layers when k ≥ 2; when k = 1, the original four-layer bound applies. For the one-layer entries, Peng et al. (2025, Lem. 3) factor a q-state deterministic transition into q identity, swap, or copy operations. The DFA primitive here copies a column of the identity, not a coordinate of a column-state vector. The identity in Proposition 25 implements it using three Householder factors, so the existing 4q factor allowance remains sufficient. The largest effective unit-key learning rate in that identity is 3; in particular, β ∈ [0, 4] suffices for the stated regular-language bounds. Merrill et al. (2026a, Lem. 12) gives Bq = 8q 2 + 5q + 1 factors for a rational q × q matrix with scratch coordinates and unrestricted, possibly negative, learning rates. A DeltaProduct transition with enough factors can apply these entire products in one step. Thus these table entries are consequences of the cited results, not new factorization or simulation claims. In particular, Siems et al. (2025, Thm. 6) concerns products of RWKV-7 matrices, so it is not a direct source for the Householder-product regular-language bound. When all learning rates remain in [0, 2], the regular-language result of Siems et al. (2025, Thm. 2) instead gives a finite depth that depends on the language; the scalar gate supplies resets. All finite-precision positive entries use the exact finite-datatype convention of Section B.4, while our WFA clock adaptations use exact arithmetic over a fixed algebraic number field, with polynomial bit length as explained in Section B.4. The WFA weights and outputs are rational; this does not require all network parameters to be rational. The clock obstruction. Under the finite-precision assumptions of Grazzi et al. (2025, Thm. 2 and Appendix B.2), a one-layer recurrence with only real transition eigenvalues cannot count modulo q > 2. This applies to GDN with A = α(I − βkk⊤ ), including α ∈ [−1, 1], and to DeltaNet at α = 1. In the cited GDN construction, a parity layer followed by alternating reflections supplies the clock. CKDA instead composes the two reflections within one transition. This saves one layer relative to that construction; it does not prove that every GDN automata simulation requires four layers.

E FFICIENT I MPLEMENTATION OF THE S IGNED G ATE

KDA stores decay magnitudes in log-space. To support signed gates, we absorb cumulative signs into the keys and queries, allowing the reuse of existing kernels in flash-linear-attention (Yang & Zhang, 2024). Fusing these sign flips into key and query normalization then avoids separate transformation passes. Kimi K3’s safe sigmoid (Kimi Team et al., 2026) uses g = −5σ(a) and α = exp(g) for the scaled, biased gate preactivation a, ensuring α ≥ ϵ := e−5 (indices suppressed). Our signed modification sets r = tanh(a/2) = 2σ(a) − 1 and α = s[ϵ + (1 − ϵ) |r|], with s = +1 for r ≥ 0 and s = −1 otherwise, and computes g = log |α| ∈ [−5, 0] in FP32 without additional clamping. At a = 0, α = ϵ and the signed gate is discontinuous; backpropagation detaches s and uses PyTorch’s zero subgradient for |r| at zero. Write Diag(αi ) = Si Di , where Si = Diag(σi ) with σi ∈ {±1}n and Di = Diag(|αi |); the sign of a zero gate entry can be chosen arbitrarily. To cancel the signs accumulated by the state, let Pi := S1 · · · Si with P0 = I. These diagonal sign matrices satisfy Pi⊤ = Pi = Pi−1 . Lemma 27 (Sign absorption). Let Hi = (I − βi ki ki⊤ )Si Di Hi−1 + βi ki vi⊤ with readout oi = Hi⊤ qi . In the transformed coordinates H̃i := Pi Hi , k̃i := Pi ki and q̃i := Pi qi , the same computation becomes H̃i = (I − βi k̃i k̃i⊤ )Di H̃i−1 + βi k̃i vi⊤ , oi = H̃i⊤ q̃i . Moreover, H̃0 = H0 , ∥k̃i ∥ = ∥ki ∥, and Hi = Pi H̃i . Proof. Left-multiply the recurrence by Pi and substitute Hi−1 = Pi−1 H̃i−1 . Since diagonal matrices commute, Si Di Pi−1 = Pi Di , so the transition becomes Pi (I − βi ki ki⊤ )Pi Di = (I − βi (Pi ki )(Pi ki )⊤ )Di = (I − βi k̃i k̃i⊤ )Di . The additive term becomes βi Pi ki vi⊤ = βi k̃i vi⊤ , and the readout satisfies Hi⊤ qi = H̃i⊤ Pi qi = H̃i⊤ q̃i . Orthogonality gives the norm identity and the inverse transformation; P0 = I gives the initial condition. 45

Relation to rotary embeddings. Absorbing cumulative transformations into keys and queries is familiar from RoPE and RetNet (Su et al., 2024; Sun et al., 2023), and from input-dependent rotary formulations of SSMs and Gated DeltaNet (Lahoti et al., 2026; Movahedi et al., 2026). Here, diagonal signs commute with arbitrary channel-wise decay, preserving all n independent decay magnitudes. The transformation introduces no parameters or additional recurrent state.

Three implementations of the sign transformation. The implementation labeled gauge uses compiled PyTorch operations, with no kernel changes. It computes cumulative signs and materializes q̃i = Pi qi and k̃i = Pi ki using torch.compile, then calls existing KDA kernels on |αi |. It requires no kernel modification; the benchmark uses a TileLang-backed recurrence for this path. The Triton implementation computes signs by an integer parity scan and jointly normalizes and transforms keys and queries. This fusion uses (Pi x)/∥Pi x∥ = Pi (x/∥x∥); backward normalization and sign application are fused into the final query/key gradient writes. The TileLang-backed hybrid combines these Triton components with TileLang kernels for the WY backward computation; it is not an all-TileLang implementation. The algebraic recurrence is identical in all three cases. The final state is recovered as Hi = Pi H̃i , and its incoming gradient is transformed by the same signs. Our models use equal key and value head counts; grouped value attention with value-head-specific signs requires expanding keys and queries.

Recurrence-kernel benchmark. Figure 5 measures forward and backward on an H100 with BF16 inputs, 16 heads, and dk = dv = 128, holding the number of logical tokens per step at 32,768. Projections, optimizer updates, and final-state output are excluded. Each implementation is wrapped in torch.compile; kernel compilation and autotuning precede 20 warm-up iterations per shape and 50 timed iterations per round. Bars report mean throughput over four rounds. The normal and signed kernel paths both use normalization fusion, and DeltaProduct2 uses its tuned configuration without a forget gate at the same head dimensions; its two updates per token are not counted as extra logical tokens. The timed recurrence processes two separately projected value inputs per token, one for each delta-rule update, so the throughput includes the recurrence work associated with this additional value input relative to CKDA. Computing the projections is outside the timed region. Triton and hybrid measurements use separate H100 allocations with the same default dot-precision policy, without explicitly setting TRITON F32 DEFAULT. The PyTorch-gauge series comes from an earlier allocation without normalization fusion and is therefore an implementation reference rather than a fully matched fusion ablation. Note that we do not benchmark FlashKDA (Chen et al., 2026) because it contains only inference kernels, while we benchmark the forward and backward pass.

H OW THE KEY ACTIVATION AFFECTS LEARNING GROUP - WORD PROBLEMS

The key path of DeltaNet (Yang et al., 2024b), inherited by KDA, is x → Wk → SiLU → ℓ2 . Since SiLU ≥ −0.2785, normalization turns that floor into an angular prior: a unit key with a negative component of size c requires ∥SiLU(z)∥ ≤ 0.2785/c. Every unit-key direction remains reachable, so this is a possible optimization bias rather than a representational restriction. At β = 2, √ √ 3 3 1 1 S3 needs no negative component at all: k ∈ {( 2 , 0, 2 ), ( 2 , 0, 2 ), e3 } realizes all five non-identity elements, the two of order 3 reusing the first two keys. The cube realization of S4 in Appendix C.3 uses mixed-sign keys; this does not establish that every higher-dimensional realization or decoded tracker must do so. Dubinin et al. (2026) report that BD-LRU solves S4 with much higher sample efficiency compared to DeltaProduct. But DeltaProduct shares the same key path as DeltaNet, while BD-LRU’s entrywise gating has none to constrain. We hypothesize that the SiLU key activation contributes to optimization differences. The ablation in Figure 16 is group-dependent: removing SiLU improves the displayed A5 result, whereas the displayed S4 result is stronger with SiLU. It therefore does not establish SiLU as the cause of the cited S4 gap. 46

(a) α: standard

(b) α: gate spread

Density

15 10 5 0 −1.0

−0.5

0.0 gate value α

0.5

1.0

−1.0

−0.5

0.0 gate value α

0.5

1.0

1.5

2.0

(d) β: spread

Density

1.5 1.0 0.5 0.0 0.0

0.5

1.0 rate β

1.5

2.0

0.0

0.5

1.0 rate β

Figure 15: Standard and spread initializations for the signed extended-range layer.

E XPERIMENTAL D ETAILS

G.1

I NITIALIZATION

The gate starts from a log-uniform decay parameter dt and applies the corresponding inverse parametrization, so that the initial magnitude is |α| = exp(−dt ). For a signed gate, standard initialization uses α = + exp(−dt ), whereas gate spread independently flips the sign of approximately half of the channels, α = s exp(−dt ) with s ∈ {−1, +1}. The rate uses β = c σ(b), where σ is the sigmoid, b is the rate preactivation, c = 1 for the standard range, and c = 2 for the extended range. Beta spread rescales the projection and adds opposite biases to different heads, β = 2σ(γz ± b0 ), where z is the unshifted projection output, producing two modes near 0.5 and 1.5. The resulting empirical distributions are shown in Figure 15. G.2

S TATE -T RACKING

Standard models use one layer with 12 heads of dimension 16, trained for 60k steps with batch size 1024 and Muon (Jordan et al., 2024) at learning rate 5 × 10−3 . We use the length curriculum 4, 6, 8, 16, 32 and report the best of three seeds. The curriculum aids learning, consistent with Beck et al. (2024); Siems et al. (2026), and Muon improves it further. For a group G, scaled accuracy (a − 1/|G|)/(1 − 1/|G|) maps chance to 0 and perfection to 1; lengths above 32 test extrapolation. For the theory-initialized A5 experiment, we use A5 ∼ = 2I/±1 to assign each element a unitquaternion lift qg ∈ R4 following Appendix C.4.2. Two heads are initialized with kg = qg , α = (−1, 1, 1, 1), and β ≈ 2, yielding Tg ≈ (I − 2qg qg⊤ ) diag(−1, 1, 1, 1). One head suffices theoretically; two improve empirical robustness. The model has four heads of dimension 4, hidden dimension 32, and a standard MLP readout. All parameters, including initialized embeddings and KDA projections, remain trainable. We train for 3k steps on all 60 elements using next-state cross-entropy and lengths 4, 6, 7, 8, 10, 12, 16, 32, without auxiliary representation loss, restrictedgenerator curriculum, or maxout readout. The robustness limitation discussed in Section 6 does not preclude exact infinite-horizon tracking in exact arithmetic (Chung et al., 2026). Complementarily, Dankowiakowski & Ronca (2025) show that linear-recurrence SSMs cannot robustly recognize all star-free languages, including FLIP-FLOP, whereas nonlinear xLSTM can. State-dependent nonlinear RNNs can implement error-correcting dynamics unavailable to affine recurrences (Beck et al., 2024; Pöppel et al., 2025; Danieli et al., 2026; Mishra et al., 2026). G.3

P ERIODIC WAVEFORM CONTINUATION

Data and task. We synthesize a fixed two-bar groove at 124 BPM with 32 sixteenth-note frames, render stereo audio at 4096 Hz, convert it to mono, and resample each frame to 64 waveform values. The resulting periodic sequence Y ∈ R32×64 is normalized independently along each waveform coordinate. Each training example is circularly shifted by a phase s ∼ Uniform{0, . . . , 31}, with 47

targets yt = Y(s+t) mod 32 and inputs xt =

yt , 0,

t < 8, t ≥ 8.

The model thus observes eight frames (one half-bar, approximately 0.97 seconds); inputs and targets have shape B × L × 64, and mean-squared error is computed only after the cue (t ≥ 8). Using shifts of one fixed groove isolates phase inference and periodic extrapolation rather than general-purpose audio generation. Architectures. All models use one sequence-mixing layer with hidden size 128, a bias-free 64 → 127 input projection followed by an appended constant coordinate, and a shared 128 → 512 → 64 GELU readout. KDA models use eight heads with key and value dimensions 16, no short convolution, and no post-recurrent SiLU activation. We compare all four combinations of α ∈ [0, 1] or [−1, 1] and β ∈ [0, 1] or [0, 2], using the corresponding spread initialization for extended ranges and default initialization for standard ranges. KDA models have approximately 182k parameters; the GRU baseline has one 128-dimensional recurrent layer and 206k parameters. The causal Transformer has one eight-head pre-norm self-attention block, causal masking, sinusoidal positional encodings, and a 128-dimensional feed-forward block, totaling 207k parameters. Training. All models are trained from scratch for 2500 updates with batch size 64, seed 0, and the sequence-length curriculum 12, 16, 24, 40, 72, 136. The curriculum occupies the first 40% of training, reaching length 136 at update 835 and retaining it thereafter. Muon optimizes two-dimensional matrices inside the sequence layer with learning rate 0.02, momentum 0.95, Nesterov momentum, and five Newton–Schulz iterations; AdamW optimizes the input projection, readout, biases, and other parameters with learning rate 0.003. Both learning rates use 250 warm-up updates followed by cosine decay to 10% of their initial values. Weight decay is 10−12 , and the global gradient norm is clipped to 1. Evaluation. After undoing training normalization, we average error over all 32 cue phases, continuation steps, and waveform coordinates at sequence lengths 16, 24, 32, 40, 56, 72, 88, 104, 120, 136, 152, 168, 184, 200, 216, 232, 248, 264. Waveform signalto-noise ratio is SNR = 10 log10 E[y2 ]/E[(ŷ − y)2 ] , with both expectations taken over the continuation region. Qualitative waveforms in Figure 8 show phase 0 over steps 0–24 and the extrapolation window 232–264. G.4

L ANGUAGE MODELING .

Training. For the 1.3B parameter language modeling experiments, we follow the recipe of Yang et al. (2025); Hatamizadeh et al. (2026), training on 100B tokens of FineWeb-edu (Lozhkov et al., 2024), using the Llama-2 tokenizer (Touvron et al., 2023), a global batch size of 0.5M tokens, a linear warmup of 1B tokens to peak learning rate 4e − 4, followed by a cosine decay to 10% of the peak learning rate all optimized with the AdamW optimizer (Loshchilov & Hutter, 2019). For our CKDA variants we use the standard init with no SiLU on keys. Note that our hybrid variants are using full attention at a 3:1 recurrent-to-attention ratio compared to the 1:1 recurrent + sliding window comparison. For additional ablations on the impact of α and β ranges as well as the impact of SiLU activation for keys, we train on Nemotron-CC (Su et al., 2025) and a smaller version of FineWeb-edu. Also here, see Table 6, CKDA outperforms an attention baseline as well as DeltaProduct (Siems et al., 2025), GDN (Yang et al., 2025) and vanilla KDA (Kimi Team, 2025) on downstream evaluations. The small-model experiments use the GPT-2 tokenizer, with the model vocabulary padded to 65,536 = 216 entries. Language-model evaluation. We use the Language Model Evaluation Harness (Gao et al., 2024). The evaluation tasks include WikiText (Merity et al., 2017), LAMBADA (OpenAI version) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), the Winograd Schema Challenge (Levesque et al., 2012), WinoGrande (Sakaguchi et al., 2021), COPA (Roemmele et al., 2011), PIQA (Bisk et al., 2020), ARC-Easy, ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), 48

Table 4: Configurations used in the small-scale Nemotron-CC language modeling experiments. The intermediate size (the latent dimension of the SwiGLU MLP block) is reduced for GDN, DeltaProduct, and CKDA. We use AdamW with a global batch size of 64 and learning rate 5×10−4 , following a warmup–stable–decay (WSD) schedule: 2000 warmup steps and cooldown over the last 20% of the token budget. Weight decay is set to 0.1 and (β1 , β2 ) = (0.9, 0.95). Config Dense Attention GDN DeltaProduct KDA CKDA Hidden size Num. hidden layers Num. attention heads Intermediate size Hidden activation RoPE θ Max. position embeddings Vocabulary size

768 28 12 3072 SwiGLU 10,000 2048 65,536

768 28 12 2816 SwiGLU N/A 2048 65,536

Total parameters (millions)

341.55

342.33

342.07

344.87

Table 5: Architectures of the scaling ladder, with the 1.3B/100BT FineWeb-Edu replication beside it. Every arm is parameter-matched to its column’s target: the intermediate size (the latent dimension of the SwiGLU MLP) absorbs the difference between each mixer’s own parameter budget, so the MLP width differs between arms and the totals are very close. The ladder trains on NemotronCC under AdamW with (β1 , β2 ) = (0.9, 0.95) and weight decay 0.1; multiple global batch sizes / learning rates are possible for different total token budget. The FineWeb-Edu column follows the published Gated DeltaNet recipe instead, which is why its vocabulary, head dimension, tying and schedule differ. Config

47M

124M

302M

588M

983M

1.7B

1.3B (FineWeb-Edu)

384 12 6 64 50,304 yes Nemotron-CC WSD 0.002 32 / 64 2000 20%

576 18 9 64 50,304 yes Nemotron-CC WSD 0.001 64 2000 20%

896 20 14 64 50,304 yes Nemotron-CC WSD 0.001 64 / 128 2000 20%

1280 20 20 64 50,304 yes Nemotron-CC WSD 0.0005 / 0.001 64 / 128 / 256 2000 20%

1536 24 24 64 50,304 yes Nemotron-CC WSD 0.0005 64 / 128 / 256 / 512 2000 20%

2048 24 32 64 50,304 yes Nemotron-CC WSD 0.0005 64 / 128 / 256 / 512 2000 20%

2048 24 16 128 32,000 no FineWeb-Edu cosine 0.0004 128 1908 –

Intermediate size Transformer++ KDA (safe sigmoid) CKDA KDA + attn 3:1 CKDA + attn 3:1

1536 1472 1472 1408 1408

2304 2240 2240 2176 2176

3584 3520 3520 3456 3456

5120 4992 4992 4928 4928

6144 6016 6016 5952 5952

8192 8064 8064 7936 7936

5632 5440 5440 5312 5312

Total parameters (millions) Transformer++ KDA (safe sigmoid) CKDA KDA + attn 3:1 CKDA + attn 3:1

47.64 48.03 48.03 47.27 47.27

124.55 125.45 125.45 124.14 124.14

302.01 303.66 303.66 302.96 302.96

588.73 586.32 586.32 587.75 587.75

983.31 980.00 980.00 984.36 984.36

1713.74 1709.71 1709.71 1712.29 1712.29

1364.30 1362.63 1362.63 1362.26 1362.26

Hidden size Num. hidden layers Num. attention heads Head dimension Vocabulary size Tied embeddings Training corpus LR schedule Peak learning rate Global batch size (sequences) Warmup steps Cooldown fraction of budget

Constant across every column Hidden activation Context length

SwiGLU 4,096

SWDE (Lockard et al., 2019), FDA (Arora et al., 2023), and SQUADv2 (Rajpurkar et al., 2018). Table 8 expands the benchmark abbreviations. In Table 7 we also report RULER results. We use the standard methodology and prompts, but we found that slight variations of the prompt (e.g. adding a ”:” to ”The answer is”) lead to largely different numbers and bad S-NIAH numbers can to a large degree be attributed to a lack of instruction following (no answer is given, the prompt is repeated). These should therefore be treated with caution.

S CALING L AW RESULTS

In addition to language modeling performance at a fixed scale, we also test for scaling behavior for varying model size and training data size (Kaplan et al., 2020; Hoffmann et al., 2022). We follow the recipe of Ajroldi et al. (2026) on varying the model and data size of training with a linear warmup49

Table 6: Language modeling results. The best-performing configuration for each evaluation is highlighted in bold, and the second best is underlined; markings consider only the Nemotron-CC rows. Nemotron-CC runs use 15B training tokens and FineWeb runs use 45B tokens; both blocks average the same ten accuracy tasks. Validation perplexity is measured on each block’s own training distribution, and differences in training data and token budget prevent the cross-block comparison from isolating architecture effects. Abbreviations are expanded in Table 8. Perplexity (↓)

Architecture

Accuracy (↑)

Val.

Wiki.

Lamb.

Hella.

W.grad

W.grande

Lamb.

COPA

PIQA

ARC-e.

ARC-c.

BBQA-Wiki

OBQA

Avg.

Dense Attention

17.99

24.15

24.54

42.36

63.37

53.75

39.92

70.00

68.44

59.81

29.61

48.79

33.00

50.90

DeltaProduct k = 2 DeltaProduct k = 2, β ∈ [0, 2]

18.87 19.04

28.07 28.38

31.90 30.43

39.50 39.40

62.64 61.54

52.09 50.67

32.00 33.20

65.00 64.00

68.17 66.65

56.61 57.45

27.39 27.47

46.79 46.85

34.20 33.60

48.44 48.08

GDN GDN β ∈ [0, 2]

18.29 18.10

27.18 27.22

25.28 25.21

42.30 42.15

66.67 66.30

51.14 51.54

36.13 36.00

65.00 63.00

68.28 67.79

57.87 58.12

28.41 29.10

47.47 47.40

34.60 35.00

49.79 49.64

KDA α ∈ [0, 1] β ∈ [0, 1] KDA α ∈ [0, 1] β ∈ [0, 2]

17.45 17.55

25.29 25.51

19.67 20.53

43.87 43.79

66.30 63.74

52.72 52.57

39.71 39.10

67.00 66.00

68.55 68.34

59.81 59.05

30.03 30.12

51.21 49.48

34.00 33.60

51.32 50.58

KDA α ∈ [−1, 1] β ∈ [0, 2] (with ablations) standard spread spread, no SiLU on keys spread, β ∈ [0, 1]

17.74 17.49 17.49 17.45

25.86 25.34 25.30 25.26

22.50 20.21 19.64 20.17

43.22 44.04 43.65 43.91

69.96 68.86 65.93 64.10

51.38 53.67 53.12 52.41

38.13 39.67 39.74 39.28

62.00 68.00 72.00 67.00

69.75 69.53 68.55 68.82

60.40 61.32 61.07 61.91

28.75 31.23 30.12 30.29

51.20 51.10 50.48 50.34

36.20 35.60 34.00 35.80

51.10 52.30 51.87 51.39

17.02 17.03

23.46 23.43

13.66 13.67

47.08 47.19

64.84 68.13

54.22 55.09

45.29 45.57

68.00 67.00

70.84 69.80

52.82 52.02

26.02 26.11

54.32 55.79

32.60 31.80

51.60 51.85

17.00 17.01 17.10 17.12 17.16

23.29 23.20 23.92 23.76 23.80

14.18 14.21 13.83 14.34 14.33

46.76 46.72 46.98 46.40 46.70

68.86 70.70 67.77 67.77 68.86

54.22 56.04 54.70 52.33 54.62

44.42 44.11 44.75 44.58 44.23

66.00 65.00 68.00 71.00 70.00

70.08 70.73 70.13 70.84 70.95

52.57 52.31 52.31 53.07 51.85

27.30 25.26 26.02 27.39 26.88

56.12 56.71 53.49 54.22 54.99

31.80 31.40 31.80 31.80 33.00

51.81 51.90 51.60 51.94 52.21

FineWeb, 45B tokens KDA α ∈ [0, 1] β ∈ [0, 2] KDA α ∈ [0, 1] β ∈ [0, 2], no SiLU on keys CKDA α ∈ [−1, 1] β ∈ [0, 2] (with ablations) standard standard, no SiLU on keys spread spread, no SiLU on keys spread, β spread init

Table 7: RULER needle retrieval at 1.3B parameters / 100B FineWeb-Edu tokens, 4K training sequences; accuracy (%) over 500 samples per cell. Our hybrid rows are evaluated only at or below 4,096 tokens, the context their attention layers trained at. It remains to be investigated why KDA without extended gates is much worse here. Model

S-NIAH-1 1K

S-NIAH-2 2K

S-NIAH-3 8K

KDA, bounded gate (ours) 32.2 65.8 94.0 CKDA (ours) 100.0 100.0 100.0

68.8 87.8

73.6 100.0 97.6 21.0 7.4 46.6 9.8 48.6 26.2 26.0 100.0 99.8 88.6 33.2 94.4 88.2 51.0 56.0 46.6 37.0

ours, hybrid - 3:1 full NoPE gated attention KDA + attn 3:1 (ours) 100.0 100.0 100.0 100.0 CKDA + attn 3:1 (ours) 99.8 96.8 81.2 53.4

99.8 93.2

Hatamizadeh et al. (2026), recurrent Mamba-2 100.0 100.0 97.0 Gated DeltaNet 99.8 100.0 100.0 KDA 100.0 100.0 99.2 Mamba-3 (SISO) 100.0 99.0 63.4 Mamba-3 (MIMO) 100.0 99.8 93.0 Gated DeltaNet-2 100.0 100.0 100.0

99.6 99.6 62.6 21.0 59.2 38.6 100.0 100.0 87.2 32.0 89.8 54.2 100.0 100.0 89.0 30.6 77.4 63.2 99.8 99.0 59.4 25.2 60.2 35.6 99.8 98.8 64.2 27.2 89.2 72.4 100.0 100.0 93.0 39.2 92.0 89.8

55.8 97.6 70.6 27.8 35.6 97.8

99.2 98.6

MK-NIAH-1

99.2 70.2 99.4 98.0 95.8 73.8 60.2 63.8 83.2 62.0 97.6 91.0 86.8 95.0 94.4 94.4 14.4 60.6 26.2 12.2 29.2 31.8

29.0 58.0 54.0 44.8 49.4 72.6

21.2 37.0 44.2 27.4 19.2 51.4

21.4 27.8 28.0 20.2 18.0 37.8

Table 8: Full names of abbreviated evaluation benchmarks. Abbreviation Full Name Wiki. Lamb. Hella. W.grad W.grande COPA PIQA ARC-e. ARC-c. BBQA OBQA Avg.

WikiText LAMBADA OpenAI HellaSwag Winograd Schema Challenge WinoGrande Choice of Plausible Alternatives Physical Interaction QA AI2 Reasoning Challenge (Easy) AI2 Reasoning Challenge (Challenge) BigBench QA OpenBook QA Average across accuracy tasks

stable-decay (WSD) schedule with AdamW (Loshchilov & Hutter, 2019), using model sizes of 50

47M , 124M , 302M , 588M , 983M and 1.71B parameters and data scales of 6B, 12B, 20B, 30B, 50B tokens (leaving out the largest data scales here). To save compute, we rely on their found optimal batch size and learning rates - assuming invariance of these towards a change of sequence mixing backbone or hybrid model architecture. We follow the scaling model of Videau et al. (2026); Busbridge et al. (2025) in the parametrization of the loss for a parametric fit. H.1

S CALING - LAW METHODOLOGY

The grid. Each architecture is trained on a shared (N, D) grid of six model sizes (47M–1.7B nonembedding parameters, N from 4.73 × 107 to 1.71 × 109 ) crossed with five token budgets (D ∈ {6, 12, 20, 30, 50} BT), on the high-quality subset of Nemotron-CC tokenized with the GPT-NeoX20B tokenizer at a sequence length of 4096. So that a difference between arms is attributable to the mixer and not to parameter count, every non-attention arm is parameter-matched to the attention baseline at its rung by adjusting dffn alone; depth, dmodel , head count and head dimension are held fixed across arms. Batch size b⋆ and peak learning rate η ⋆ are taken per cell from the grids of Ajroldi et al. (2026), so that the ladder inherits a tuned, published schedule rather than one of our own choosing. We depart from that reference in exactly two respects, both applied uniformly across the grid: β2 = 0.95 at every rung, where the reference switches from 0.99 below 300M and so would place a hyperparameter change inside our N axis; and no bias terms anywhere outside the (C)KDA layers. Endpoints. Every cell is trained under a warmup–stable–decay schedule, and the loss we fit is the validation loss (over 0.838 B tokens) of the annealed endpoint. Within a rung, cells sharing b⋆ and η ⋆ are run as a single constant-rate trunk with a cooldown branched from it at 0.8D and annealed linearly over the remaining 20%, rather than as independent runs to save compute. Functional forms. We fit the non-separable Skaling form of Videau et al. (2026); Busbridge et al. (2025), the one Ajroldi et al. (2026) find extrapolates reliably on this corpus: LSkaling (N, D) = E + AN −α + BD−β

(23)

Three treatments of the irreducible loss. Every fit below gives each architecture its own A, α, B, β, k and chained-cell offset, and the three differ in how the irreducible loss E is modeled. Table 9 fits each arm in complete isolation, so E is free and separate per arm; Table 10 pins E to the value of Ajroldi et al. (2026), as they have a more extensive, compute heavy grid. Since the irreducible loss should be independent of the model - assuming each can perfectly fit at infinite scales - this is a valid assumption. Estimation. Parameters are estimated by trust-region least squares on the residuals in nats, under a Huber loss with scale 0.01 nats. The scale is set at the measured run-to-run reproducibility of the ladder, where two bit-identical configurations differ by up to 0.002 nats: cells agreeing to within noise then enter quadratically, while a single genuine outlier cannot dominate the fit. Because both forms are strongly multi-modal in (A, α, B, β, k), each fit is run from 600 randomized starts, loguniform over the prefactors and uniform over the exponents, and the lowest-cost optimum is retained. Uncertainty. Intervals are obtained by bootstrap rather than from a covariance matrix. We resample the endpoints with replacement, stratified within each arm so that every replicate has the same design as the data and no arm can lose the cells that identify it, and refit. Each replicate is warmstarted from the full-data optimum and given a small number of additional random restarts, which keeps the cost tractable while guarding against a resample whose optimum genuinely lies elsewhere. We report the 2.5th and 97.5th percentiles of the replicates as a subscript on each estimate. These intervals describe sampling variability of this ladder under the fitted model; they do not carry the systematic uncertainty in b⋆ and η ⋆ , which are held fixed at the values from Ajroldi et al. (2026). 51

H.2

S CALING L AW R ESULTS

All trained model architectures can be well fitted to the chosen parametric form (Busbridge et al., 2025; Videau et al., 2026), see Tables 9 and 10. A held-out point on the largest scale can be predicted with low error for each model, see Table 11. Re-using the irreducible loss found in Ajroldi et al. (2026) (which is smaller than our fitted values) leads to a slight shift of fitted parameters, namely the data scaling exponent and coupling exponent are smaller. This means that larger scale experiments could lead to a non-negligible change still in the fitted parameters. While the validation loss slightly favors KDA over CKDA, the downstream evaluations show a mixed picture here, see Table 13. Given the confidence bounds, CKDA and KDA perform on par in vanilla language modeling. All measured validation losses are shown in Table 12. The derived compute-optimal scaling for the model architectures is showing in Figure 17. The observed decline of the advantage of the recurrent / hybrid models over the Transformer going to large over-training regimes at small scales (see 9) is not clearly visible in the scaling exponents, though there are differences. A detailed exploration of this remains to be investigated in future work. Table 9: Fitted parameters with 1.7B/50BT held out, all parameters fit per architecture in an isolated way. RMSE is in-sample, over that arm’s remaining cells; the held-out error is in Table 11. Note that our data is missing the regime of about 10× more compute (100B and 300B tokens) - leading to a slightly different offset compared to Ajroldi et al. (2026). Model Arch. Transformer++ Transformer++ (QK-norm) KDA CKDA KDA + attn 3:1 CKDA + attn 3:1 Ajroldi et al. (2026)

RMSE

1.1628[−0.1500, 1.3895] 1.0869[0.6970, 1.2665] 1.2091[1.0517, 1.3015] 1.2111[0.9258, 1.3227] 1.1765[0.8513, 1.3069] 1.2473[1.0476, 1.3609]

2.76 × 103[6.61×102 , 1.10×105 ] 5.57 × 103[2.06×103 , 1.57×104 ] 2.85 × 104[2.06×104 , 4.96×104 ] 2.57 × 104[1.47×104 , 6.13×104 ] 2.49 × 104[1.32×104 , 7.21×104 ] 2.18 × 104[1.34×104 , 5.07×104 ]

0.401[0.326, 0.453] 0.429[0.385, 0.461] 0.528[0.515, 0.545] 0.523[0.498, 0.547] 0.520[0.490, 0.541] 0.521[0.502, 0.544]

5.23 × 103[4.97×102 , 5.78×104 ] 7.61 × 103[1.47×103 , 5.81×104 ] 1.35 × 104[6.31×103 , 2.92×104 ] 1.33 × 104[3.31×103 , 3.56×104 ] 1.24 × 104[4.07×103 , 3.34×104 ] 1.12 × 104[5.22×103 , 2.63×104 ]

0.382[0.234, 0.475] 0.391[0.297, 0.486] 0.425[0.380, 0.465] 0.425[0.352, 0.476] 0.419[0.356, 0.474] 0.421[0.376, 0.463]

0.500[0.263, 0.621] 0.450[0.362, 0.524] 0.436[0.395, 0.461] 0.440[0.373, 0.484] 0.434[0.361, 0.480] 0.457[0.384, 0.492]

0.00697 0.00432 0.00206 0.00269 0.00262 0.00212

0.9638±0.0870

1.96 × 104±1.40×104

0.483±0.030

4.54 × 103±2.90×103

0.349±0.022

0.393±0.033

0.00473

Table 10: E pinned to 0.9638, the value of Ajroldi et al. (2026), fitted on the same corpus and heldout split over a ladder reaching 1.7B parameters and 300B tokens. Model Arch.

E∗

RMSE

Transformer++ Transformer++ (QK-norm) KDA CKDA KDA + attn 3:1 CKDA + attn 3:1

* * * * * *

3.99 × 103[7.51×102 , 1.43×104 ] 7.28 × 103[2.95×103 , 1.27×104 ] 5.07 × 104[3.24×104 , 7.53×104 ] 4.53 × 104[2.35×104 , 7.21×104 ] 4.06 × 104[2.16×104 , 5.91×104 ] 4.10 × 104[2.64×104 , 6.65×104 ]

0.403[0.325, 0.458] 0.431[0.390, 0.457] 0.533[0.512, 0.552] 0.527[0.499, 0.547] 0.523[0.497, 0.540] 0.526[0.507, 0.548]

3.17 × 103[5.47×102 , 2.13×104 ] 6.38 × 103[1.67×103 , 2.43×104 ] 7.46 × 103[3.42×103 , 1.44×104 ] 6.77 × 103[2.56×103 , 1.56×104 ] 7.33 × 103[3.03×103 , 1.86×104 ] 5.11 × 103[2.31×103 , 1.22×104 ]

0.342[0.272, 0.423] 0.372[0.317, 0.429] 0.373[0.341, 0.401] 0.369[0.326, 0.404] 0.374[0.337, 0.413] 0.358[0.325, 0.393]

0.454[0.384, 0.562] 0.423[0.384, 0.476] 0.383[0.364, 0.406] 0.387[0.362, 0.417] 0.388[0.366, 0.416] 0.394[0.367, 0.419]

0.00697 0.00439 0.00244 0.00302 0.00286 0.00263

0.9638∗±0.0870

1.96 × 104±1.40×104

0.483±0.030

4.54 × 103±2.90×103

0.349±0.022

0.393±0.033

0.00473

Ajroldi et al. (2026)

Table 11: Held-out error at 1.7B/50BT against the parametric loss fit, at the largest model/data regime we are training here. Arm Transformer++ Transformer++ (QK-norm) KDA CKDA KDA + attn 3:1 CKDA + attn 3:1

Measured

E per model

E pinned to Ajroldi et al. (2026)

2.1630 2.1374 2.1046 2.1061 2.0971 2.1002

+0.0007[−0.0081, +0.0089] +0.0006[−0.0053, +0.0053] −0.0007[−0.0047, +0.0022] +0.0007[−0.0046, +0.0065] −0.0031[−0.0078, +0.0006] −0.0016[−0.0058, +0.0018]

−0.0027[−0.0059, +0.0027] −0.0018[−0.0048, +0.0024] −0.0046[−0.0070, −0.0016] −0.0029[−0.0059, +0.0032] −0.0063[−0.0088, −0.0033] −0.0059[−0.0086, −0.0027]

Table 12: Validation loss at every scaling experiment point. Best per row in bold. The final column is the best cell of the hyperparameter sweep of Ajroldi et al. (2026) at the same (N, D), a dash indicates that no result was reported”—the cells contain dashes. Cell

Transformer++

Transformer++ (QK-norm)

KDA

CKDA

KDA + attn 3:1

CKDA + attn 3:1

Ajroldi et al. (2026)

47M/6BT 47M/12BT 47M/20BT 47M/30BT 47M/50BT

2.9844 2.9109 2.8782 2.8578 2.8271

2.9463 2.8833 2.8504 2.8335 2.8045

2.9247 2.8664 2.8403 2.8215 2.7974

2.9267 2.8671 2.8397 2.8220 2.7962

2.9126 2.8522 2.8250 2.8063 2.7809

2.9167 2.8560 2.8257 2.8075 2.7814

2.9272 2.8747 2.8417 2.8198 2.7974

124M/6BT 124M/12BT 124M/20BT 124M/30BT 124M/50BT

2.7471 2.6799 2.6389 2.6074 2.5682

2.7188 2.6541 2.6124 2.5851 2.5488

2.6869 2.6212 2.5837 2.5576 2.5286

2.6880 2.6220 2.5850 2.5581 2.5288

2.6718 2.6072 2.5705 2.5442 2.5146

2.6719 2.6083 2.5708 2.5462 2.5154

2.7129 2.6497 2.6048 2.5736 2.5409

302M/6BT 302M/12BT 302M/20BT 302M/30BT 302M/50BT

2.6126 2.5382 2.4942 2.4598 2.4160

2.5789 2.5044 2.4560 2.4287 2.3882

2.5243 2.4505 2.4077 2.3768 2.3441

2.5258 2.4550 2.4089 2.3784 2.3453

2.5146 2.4446 2.3956 2.3651 2.3323

2.5174 2.4437 2.3967 2.3667 2.3335

2.5723 2.4939 2.4480 2.4158 2.3778

588M/6BT 588M/12BT 588M/20BT 588M/30BT 588M/50BT

2.5209 2.4346 2.3805 2.3436 2.3032

2.4892 2.4051 2.3534 2.3172 2.2770

2.4496 2.3626 2.3143 2.2775 2.2386

2.4532 2.3673 2.3191 2.2815 2.2417

2.4430 2.3556 2.3051 2.2676 2.2284

2.4424 2.3555 2.3049 2.2683 2.2295

2.4798 2.3988 2.3476 2.3112 2.2696

983M/6BT 983M/12BT 983M/20BT 983M/30BT 983M/50BT

2.4551 2.3682 2.3114 2.2721 2.2298

2.4282 2.3403 2.2844 2.2464 2.2070

2.3905 2.3009 2.2484 2.2098 2.1686

2.3963 2.3059 2.2532 2.2141 2.1722

2.3810 2.2920 2.2369 2.1981 2.1571

2.3850 2.2959 2.2405 2.2017 2.1608

2.4250 2.3363 2.2815 2.2430 2.2003

1.7B/6BT 1.7B/12BT 1.7B/20BT 1.7B/30BT 1.7B/50BT

2.4001 2.3100 2.2488 2.2077 2.1630

2.3725 2.2829 2.2218 2.1812 2.1374

2.3359 2.2496 2.1882 2.1477 2.1046

2.3402 2.2507 2.1898 2.1494 2.1061

2.3284 2.2399 2.1795 2.1395 2.0971

2.3299 2.2433 2.1828 2.1429 2.1002

– – – 2.1802 2.1353

Table 13: Downstream accuracy, averaged over the nine accuracy tasks of Table 2 at every scaling experiment cell. Below 302M most of the suite is at chance – HellaSwag near 27, ARC-c near 23, OpenBookQA near 25 – so the average at the lower rungs is largely majority-class behavior on BoolQ and PIQA. Cell

Transformer++

Transformer++ (QK-norm)

KDA

CKDA

KDA + attn 3:1

CKDA + attn 3:1

47M/6BT 47M/12BT 47M/20BT 47M/30BT 47M/50BT

37.97 38.14 38.59 39.13 38.64

37.75 37.62 38.22 37.49 38.94

39.08 38.04 39.06 40.16 39.42

38.49 38.43 37.84 39.36 38.96

37.70 38.11 37.72 39.22 38.96

39.12 38.60 39.42 39.46 38.49

124M/6BT 124M/12BT 124M/20BT 124M/30BT 124M/50BT

40.47 40.57 42.41 42.01 42.98

40.06 41.74 42.16 41.52 44.01

41.22 42.41 43.41 43.53 44.02

42.25 41.29 43.67 43.64 44.23

41.06 41.96 43.11 43.68 43.59

40.69 41.68 42.88 43.48 43.51

302M/6BT 302M/12BT 302M/20BT 302M/30BT 302M/50BT

42.17 44.10 44.50 45.31 46.21

42.39 44.15 45.24 45.19 46.21

44.76 45.89 46.12 47.73 47.69

44.56 45.90 47.19 47.99 48.25

44.50 46.77 47.01 48.20 48.82

44.22 45.53 47.02 47.97 49.14

588M/6BT 588M/12BT 588M/20BT 588M/30BT 588M/50BT

44.01 45.84 47.16 47.53 48.45

44.15 46.67 46.79 47.98 49.83

45.75 47.82 48.57 50.18 51.43

45.35 47.67 48.66 49.45 50.44

45.96 48.53 48.75 49.50 51.24

45.46 47.87 49.54 50.79 51.30

983M/6BT 983M/12BT 983M/20BT 983M/30BT 983M/50BT

45.22 47.99 48.90 50.16 51.80

46.19 48.36 49.12 50.42 51.57

46.80 49.29 50.51 51.51 53.13

47.14 49.48 51.22 51.95 54.10

47.78 50.40 51.49 52.60 54.10

46.86 49.33 50.95 52.32 54.25

1.7B/6BT 1.7B/12BT 1.7B/20BT 1.7B/30BT 1.7B/50BT

46.66 49.02 50.13 51.56 53.08

47.61 49.59 51.75 53.29 55.10

48.25 51.10 52.80 55.13 56.87

48.84 51.34 53.38 54.59 57.21

48.35 51.18 52.60 54.91 56.47

48.29 50.95 52.53 54.35 55.82

Scaled accuracy

1.0

0.5

0.0

128

256 384 Sequence length

512 1

128

256 384 Sequence length

Gated DeltaNet, SiLU off

Gated DeltaNet, SiLU on

512 1

DeltaProduct nh =2, SiLU off

128

256 384 Sequence length

512

DeltaProduct nh =2, SiLU on

Figure 16: Baseline results for Figure 6 Transformer++ (QK-norm) KDA CKDA

KDA + attn 3:1 CKDA + attn 3:1 compute-optimal front (fit)

2.8

Validation Loss

2.6

2.4

2.2

2.0 1018

1019

1020

1021

Training Compute C = 6ND [FLOPs]

Figure 17: Validation loss vs. Compute for the language model scaling experiments on NemotronCC.

A DDITIONAL E XPERIMENTAL R ESULTS

3.6

Validation loss

3.4

Language modeling comparison KDA KDA KDA GDN

[ 1, 1] [0, 2] [0, 1] [0, 2] [0, 1] [0, 1] [0, 1] [0, 1]

2.90

GDN [0, 1] [0, 2] DeltaProduct nh = 2 DeltaProduct nh = 2 [0, 2] Dense Attention

3.2

3.0

2.8

Tokens

12B

[ 1, 1] [0, 2] ablations

2.88 2.87 2.86

KDA

2.89

Validation loss

15B

2.85

spread spread, [0, 1] shipped

spread, no silu on keys KDA [0, 1] [0, 1]

14.5B

Tokens

15B

Figure 18: Language modeling results: validation loss versus tokens. (Left) Comparison across baselines. (Right) Comparison across ablations shown for the end of the cooldown phase, as loss values are quite close together. 55

(a) Pr(α < 0)

Layer

17 15

(b) Pr(β > 1)

20%

10%

50%

20 40 60 80 100 0%

10 5 0

Training tokens (B)

Hybrid Layer

(a) Gate α distribution

(b) Rate β distribution

10−1

10−4

6 × 10−2 4 × 10−−22 3 × 10 2 × 10−2 10−2 6 × 10−3 4 × 10−3

Probability/bin

23 20 15 10 5 0 17 15

Probability/bin

Non-hybrid Layer

Figure 19: Hybrid CKDA 1.3B, see Figure 10.

5 0

−1

1 β

50% Pr(complex pair)

Figure 20: Final layer profiles of gate and β for the CKDA 1.3B non-hybrid and hybrid models after training on 100B tokens.

Record · ID 1028692 · SHA-256 a312087290a2c009
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.