A Held-Out Transition-Pair Falsifier for Long-Horizon Non-Abelian State Tracking
arXiv:2606.07254v1 [cs.LG] 5 Jun 2026
Jeonghoon Lee Attractor Dynamics [email protected] Abstract State tracking exposes a sharp limitation of sequence models: the relevant signal is often not a summary of observed tokens, but an ordered latent state that evolves through noncommutative transformations. We introduce a held-out transition-pair falsifier for finite nonAbelian group tracking. The protocol forbids selected ordered generator pairs during training and requires the same local patterns during evaluation, blocking one direct local-transition memorization pathway. In a controlled 𝑆3 × 𝑆3 benchmark, a projected recurrent state model trained only on length-8 sequences produces error-free final-state predictions (perfect 250/250 per horizon) through evaluation horizons up to 1,048,576 tokens across five seeds. Matched native-readout baselines, including bag, GRU, and a single-configuration structured state-space model, remain near floor under the same protocol. Projection-matched GRU, structured SSM, and bag baselines equipped with analogous finite-group prototype readouts also remain near chance under the same split. Mechanism diagnostics show that hard projection coincides with low homomorphism error, low state-consistency drift, and non-trivial commutator separation, while softened projection collapses final-state accuracy. Clean-split audits verify zero verbatim reduced-word overlap and zero structural-template overlap between training and evaluation partitions. The evidence is scoped to this controlled finite-group falsifier rather than to a general architecture ranking. Within that regime, explicit projected non-commutative state composition acts as a useful inductive bias for long-horizon hidden-state tracking.
1
Introduction
Long-context systems are increasingly used as agents, tool users, and workflow controllers. In such settings, failure is often not a missing token but a corrupted hidden state: the system loses track of what has already been done, which branch of a process it is in, or which latent condition is currently true. A benchmark that can be solved by local-template memorization does not test this failure mode. Recent sequence-model evaluation increasingly emphasizes token-level prediction quality, longcontext retrieval, or in-context learning. These framings reward models that summarize observed tokens well. They underweight a different family of computational requirements: tasks whose relevant signal is not a summary of any visible token sequence, but an ordered latent state that evolves through composition operations that may not commute. Finite non-Abelian group tracking is a canonical controlled example. Given a sequence of generator symbols (𝑎𝑡1 , 𝑎𝑡2 , … , 𝑎𝑡𝐿 ) drawn from a group G, the target is the accumulated product 𝐻𝐿 = 𝑎𝑡1 ⋅𝑎𝑡2 ⋅…⋅𝑎𝑡𝐿 . When G is non-Abelian, the order matters: 𝑎⋅𝑏 ≠ 𝑏 ⋅𝑎 for at least one pair. A correct model must therefore preserve order in its internal state across the full sequence.
1
Standard length-extrapolation evaluation of such tasks is vulnerable to a subtle confound. A model that has memorized local transition patterns observed during training can produce correctlooking outputs at longer horizons by interpolating from those patterns, without performing genuine non-commutative state composition. The model may appear to extrapolate in length while in fact relying on observed (𝑎𝑖 , 𝑎𝑗 ) → next-state transitions. A model can appear to extrapolate in length while still relying on local transition patterns seen during training. Our goal is to remove that path. This paper makes three contributions: 1. A held-out transition-pair falsifier. We define a protocol that forbids one or more specified ordered generator-pairs from any training sequence, and requires those same pairs to occur in every evaluation sequence. Under this split, any model that solves the task by memorizing the specific local transition templates excluded from training must fail at evaluation: the required ordered pair was excluded from training, so no observed template covers it. Baseline failure under this protocol supports the interpretation that the direct local-template pathway has been blocked. 2. A projected recurrent state model interface. We describe a class of sequence models that maintain a continuous-valued non-commutative recurrent hidden state and produce symbolic group elements via a temperature-controlled projection onto the target finite group. We show that, under the falsifier protocol, the hard-projected variant of this model class preserves exact final-state accuracy through evaluation horizons up to approximately 105 times the training horizon on a worked 𝑆3 × 𝑆3 benchmark. 3. Mechanism diagnostics under projection temperature. We report a four-axis diagnostic family: final-token accuracy, exact homomorphism error, state-consistency drift, and commutator gap. Across a sweep of projection temperatures, these diagnostics identify a coherent boundary at which the model’s representation departs from grouphomomorphic behavior. A complementary clean-split overlap audit verifies that the training and evaluation partitions of the data are non-trivially distinct under both verbatim reduced-word and structural-template criteria. The result is intentionally narrow. The paper contributes a falsifier and evidence inside that falsifier, rather than a universal ranking of sequence-model families. Concurrent work by Sung [8] reports a related non-Abelian length-extrapolation result on 𝑆10 variable binding using a different mechanism. The contribution of this paper is accordingly narrow but concrete: under a protocol that blocks one direct local-transition memorization pathway, explicit projected noncommutative state composition yields exact final-state predictions at evaluation horizons up to roughly 105 times the training length on the controlled 𝑆3 × 𝑆3 benchmark.
2
Related Work
2.1
State tracking and sequence-model expressivity
Non-commutative group composition has long served as a diagnostic of computational expressivity. Barrington’s theorem [1] established that the word problem for finite non-Abelian groups is complete for the circuit-complexity class 𝑁 𝐶 1 , and Krohn–Rhodes decomposition [2] factored finite-state computation into group and aperiodic primitives. Modern theoretical work on neural sequence architectures has connected these algebraic facts to architectural limits. Merrill, Petty,
2
and Sabharwal [3] analyze classes of state-space sequence models and relate their expressivity to low-depth circuit classes, motivating finite group state tracking as a diagnostic. Shakerinava et al. [4] study input-dependent complex-valued diagonal SSMs and show that a single-layer DCD SSM cannot track any non-Abelian group at finite precision; more generally, a k-layer DCD SSM can express group state tracking if and only if the group admits a length-k subnormal series with Abelian factors. Ebrahimi et al. [5] empirically compare Transformers and recurrent models on in-distribution state-tracking tasks, finding that Transformers require substantially more training data as state-space size and sequence length grow and exhibit negligible or even detrimental sharing of learned state-tracking mechanisms across sequence lengths. Prior work in this line turns state tracking into a diagnostic of inductive bias. Our contribution is complementary: a stricter empirical falsifier (the held-out transition-pair split) and a constructive model-side result within the regime that the falsifier exposes. The theoretical separations above motivate the benchmark family; the present paper tests a particular empirical failure mode inside it. Recent constructive work also seeks to recover state-tracking capability by enriching recurrent or state-space transitions. Terzić et al. [6] propose PD-SSM, a structured sparse SSM whose transition matrix is parameterized as a product of a column-one-hot matrix P and a complexvalued diagonal matrix D, enabling finite-state automaton tracking with efficient parallel scans. Mishra et al. [7] revisit nonlinear recurrent models with matrix-valued hidden states and report state-tracking generalization together with scalable language-modeling results. The present paper is narrower than these architecture papers: it isolates a held-out transition-pair falsifier and reports a projected-readout result on one controlled finite-group benchmark.
2.2
Non-commutative group tasks as controlled benchmarks
Permutation groups have repeatedly been used as controlled benchmarks for sequence-model state tracking, both at the level of 𝑆𝑛 for small n and at the level of product groups exposing direct-product structure. We use 𝑆3 × 𝑆3 as a compact controlled setting under our generatorand-split design. The choice is not motivated by a claim that 𝑆3 × 𝑆3 is in any sense the smallest meaningful target. Rather, it simultaneously exposes non-commutativity, held-out local-template generalization (in the generator-pair embodiment used here), non-trivial commutator structure, and same-multiset different-product distinguishability, the four properties our protocol and diagnostics rely on, within a state space (order 36) small enough to admit transparent statistical analysis. The protocol extends naturally to other product non-Abelian groups, including 𝑆3 × 𝑆5 , 𝑆5 × 𝑆5 , dihedral products, and direct products containing simple non-Abelian factors. The main controlled result of this paper is restricted to 𝑆3 × 𝑆3 ; Appendix F additionally reports a preliminary 𝑆5 stress test, but we do not make 𝑆5 the main claim.
2.3
Projection, symbolic readout, and mechanism diagnostics
Existing recurrent and state-space models map their hidden state to a continuous distribution over a class-label set, to a continuous regression target, or to a discrete output via softmax over a vocabulary. To the best of our knowledge, prior work has not described a readout that projects a continuous-valued non-commutative recurrent hidden state onto a target finite group to produce a symbolic accumulated group element, with a tunable softness parameter that admits coherent mechanism diagnostics. The projection operator we describe (Section 4) makes the model’s symbolic output explicit and admits the diagnostic family described in Section 5.3. The four diagnostics are final-token accuracy, exact homomorphism error, state-consistency drift, and commutator gap. Together, they probe whether the model’s representation behaves approximately as a group homomorphism under the hard-projection regime and how that behavior degrades as projection softens. 3
2.4
Holonomic and gauge-based non-Abelian state tracking
Concurrent work by Sung [8] proposes the Holonomic Network, which performs non-Abelian state tracking by maintaining a hidden state on the orthogonal manifold 𝑆𝑂(𝑁 ) through an input-dependent multiplicative update of the form ℎ𝑡 = 𝑈 (𝑥𝑡 )ℎ𝑡−1 , where 𝑈 (𝑥𝑡 ) = exp(𝒜(𝑥𝑡 )) with 𝒜(𝑥𝑡 ) ∈ 𝔰𝔬(𝑁 ). The final state is obtained as the path-ordered product (holonomy), motivated by an effective Chern–Simons gauge theory. On an 𝑆10 variable-binding task, the model reports perfect generalization when trained on sequences of length up to 50 and evaluated at length 5000. We view Holonomic Networks as closely related prior work within the broader family of noncommutative recurrent architectures. Both approaches rely on structured, order-preserving state updates on a manifold. However, the present paper differs in several concrete respects. First, we introduce a held-out transition-pair falsifier that explicitly removes specific local transition templates from training while requiring them at evaluation; Sung’s experiments focus primarily on length extrapolation rather than this form of local-template blocking. Second, we report projection-matched baselines (GRU, structured SSM, and bag) equipped with the same finite-group prototype readout, evaluated under the identical held-out split. Third, our evaluation reaches horizons up to 1,048,576 tokens, more than two hundred times longer than the 5,000-token horizon reported in [8]. Fourth, our mechanism uses a continuous non-commutative state followed by an explicit projection operator 𝜋 ∶ 𝑆 → 𝐺 onto a finite group at readout, whereas the Holonomic Network maintains its state directly on 𝑆𝑂(𝑁 ) without an explicit finite-group projection step. Scan parallelism arising from associativity is not a distinguishing feature here, as Holonomic Networks also exploit this property. The contribution of this work lies instead in the held-out transition-pair falsifier, the use of projection-matched baselines, the explicit finite-group readout, and error-free million-token evaluation under that falsifier.
3
Problem Setting
3.1
Target group
We take 𝐺 = 𝑆 3 × 𝑆3 ,
|𝐺| = 36,
Σ = {𝑎0 , 𝑎1 , 𝑎2 , 𝑎3 }.
where a_0, a_1 generate the first S_3 factor and a_2, a_3 generate the second S_3 factor. G is non-Abelian: each factor S_3 is non-Abelian, so the direct product contains non-commuting pairs (e.g., pairs within the first factor or pairs within the second factor).
3.2
Update sequence
An update sequence of length L is a finite ordered tuple 𝑤 = (𝑎𝑡1 , 𝑎𝑡2 , … , 𝑎𝑡𝐿 ),
𝑎𝑡𝑖 ∈ Σ.
The accumulated group state is the cumulative product 𝐻𝐿 (𝑤) = 𝑎𝑡1 ⋅ 𝑎𝑡2 ⋅ … ⋅ 𝑎𝑡𝐿 ∈ 𝐺. 4
evaluated under the group operation. The task is to predict H_L(w) exactly, as a symbolic group element.
3.3
Held-out transition-pair split
Let 𝑃forbid = {(𝑎0 , 𝑎2 ), (𝑎2 , 𝑎0 )} ⊂ Σ × Σ. Training sequences are generated such that no training sequence contains any pair (𝑎𝑖 , 𝑎𝑖+1 ) ∈ 𝑃𝑓 𝑜𝑟𝑏𝑖𝑑 as consecutive generators. Evaluation sequences are generated such that every evaluation sequence contains at least one occurrence of each pair in 𝑃require = 𝑃forbid = {(𝑎0 , 𝑎2 ), (𝑎2 , 𝑎0 )}. The two ordered pairs (a_0, a_2) and (a_2, a_0) are distinct as local transition templates even when the corresponding generators act on different factors of the product group. In the present generator convention, a_0 belongs to the first S_3 factor and a_2 to the second; the two factors commute element-wise as a property of the direct product, so the two adjacent two-token products coincide as group elements. The purpose of holding out these specific ordered templates is therefore not to assert that the two adjacent two-token products are themselves non-commuting, but to remove a specific local transition template from training while requiring it at evaluation. Non-commutativity of the target task is probed at the full-sequence level (where order across many positions determines H_L) and explicitly by the commutator and same-multiset different-product diagnostics in Gate C and Gate E.
3.4
Why this is a falsifier
The split functions as a falsifier in the following sense. Any model that predicts H_L by interpolating from observed local transition templates (𝑎𝑖 , 𝑎𝑗 ) → next-state must fail under this split: the required ordered template never appears in training, so no observed template covers it. Therefore baseline failure under this protocol is positive evidence that the protocol blocks one direct local-transition memorization pathway. The protocol does not, by itself, foreclose every conceivable memorization or interpolation strategy; it forecloses the most direct one and exposes whether a candidate model can succeed without it. Conversely, success under this protocol, particularly at evaluation horizons much longer than the training horizon, is evidence consistent with non-commutative state composition beyond the direct local-transition pathway. The full-sequence non-commutativity is what distinguishes correct prediction of H_L over million-token horizons from any constant or direct templateinterpolation strategy.
3.5
Relation to a broader transition-pattern protocol
The present paper reports the update-pair embodiment (k = 2) of a more general held-out transition-pattern protocol in which P_forbid and P_require may consist of ordered generatortuples of arbitrary length 𝑘 ≥ 2, reversed-order templates, commutator templates, inversecancellation templates, reduced-word templates, or same-multiset different-product templates. We restrict the present empirical scope to the k = 2 update-pair case to keep the falsification reading direct and the statistical analysis transparent. Extension to higher-order and structural patterns is straightforward in principle.
5
4
Model Interface
We describe the model interface in a deliberately carrier-agnostic form. The state-tracking benchmark and falsifier protocol are defined independently of any particular continuous carrier.
4.1
Projected recurrent state model
A projected recurrent state model consists of (i) a continuous-valued recurrent hidden state, (ii) a non-commutative composition rule that combines per-token updates, and (iii) a projection operator that maps the continuous hidden state to a symbolic element of the target finite group G. Formally, at sequence position t, 𝑠𝑡 = 𝐹 (𝑠𝑡−1 , 𝑥𝑡 ),
𝑦𝐿̂ = 𝜋(𝑠𝐿 ) ∈ 𝐺.
where s_t is the continuous-valued hidden state, F is a learned per-token update map, and 𝜋 ∶ 𝑆 → 𝐺 is the projection operator. Because the per-token update is composed by an associative composition operation, the full state sequence may equivalently be computed in a scan-parallel form 𝑢𝑡 = 𝜙(𝑥𝑡 ),
𝑠 𝐿 = 𝑢 1 ⊙ 𝑢2 ⊙ … ⊙ 𝑢 𝐿 ,
𝑦𝐿̂ = 𝜋(𝑠𝐿 ).
where ⊙ is an associative but non-commutative composition operation. Order is preserved; sequential depth reduces to 𝑂(log 𝐿).
4.2
Hard projection and soft projection
The projection operator 𝜋 admits two regimes: • Hard projection 𝜋hard (𝑠) = arg min𝑔∈𝐺 𝑑(𝑠, 𝜄(𝑔)) returns the unique nearest finite-group representative under a specified distance, where 𝜄 ∶ 𝐺 → 𝑅 embeds elements of G into a representation space R. • Soft projection with temperature 𝑇 > 0 produces either a probability distribution 𝑝𝑇 (𝑔|𝑠) ∝ exp(−𝑑(𝑠, 𝜄(𝑔))/𝑇 ) over elements of G, or a convex combination of embedded representatives ∑𝑔 𝑝𝑇 (𝑔|𝑠) ⋅ 𝜄(𝑔). As 𝑇 → 0, soft projection approaches hard projection. As 𝑇 → ∞, it approaches a uniform mixture. The two regimes are reported separately throughout this paper. Hard-projected outputs are symbolic group elements; soft-projected outputs are distributions over G.
4.3
Public interface and implementation boundary
This paper specifies the public interface needed to define the benchmark, run the held-out transition-pair falsifier, interpret the projected finite-group readout, and reproduce the diagnostic calculations. The model interface consists of a continuous recurrent state, an associative order-sensitive composition rule, and a projection operator 𝜋 ∶ 𝑆 → 𝐺 that maps the final state to a symbolic element of the target finite group. The exact continuous carrier, the internal carrier embedding, and the carrier-level constraint function used by the hard-projected model are not part of this technical preprint. The empirical claims in this paper should therefore be read as claims about the projected state-tracking interface under the stated falsifier protocol, not as a full disclosure of the carrier construction. The carrier-level construction is treated separately from the protocol and readout contribution studied here. 6
5
Experimental Protocol
5.1
Gate A: baseline competence under matched protocol
Purpose. To establish that the baseline models used elsewhere in the paper are not generically incapable of solving easy state-tracking tasks under the present protocol. Without this gate, any subsequent baseline failure on the non-commutative task could be attributed to under-trained or mis-configured baselines. Configuration. Three baseline models are trained under matched protocol: a bag-of-tokens model with continuous native readout, a Gated Recurrent Unit (GRU) with continuous native readout, and a single-configuration structured state-space model with continuous native readout. The tasks are (i) easy commutative and (ii) 6-class noncommutative. Training uses 𝑛train = 1000, 𝑛val = 200, 𝑛test = 200, epochs = 8, batch_size = 64, lr = 0.003, and five seeds {20260525, 20260526, 20260527, 20260528, 20260529}. The structured state-space model is run in a single configuration (𝑑model = 64, 𝑛layers = 2, 𝑑state = 16, 𝑑conv = 4, expand = 2). A disjoint-template overlap audit verifies zero verbatim-token, template-ID, and template-family overlap across the easy-task and held-out-task partitions used for each baseline. Gate A is a separate 6-class diagnostic control used to verify baseline competence; it is not the 𝑆3 × 𝑆3 held-out-pair task reported in Gate B. Its hard task is a 6class non-commutative final-state diagnostic chosen to be informative about ordinary baselines’ competence at order-sensitive prediction; its easy task is a matched 6-class commutative control.
5.2
Gate B: long-horizon held-out transition-pair performance
Purpose. The main empirical claim of this paper. Configuration. Both the proposed projected-readout model and the three baselines are trained on 𝑆3 × 𝑆3 update sequences under the held-out transition-pair split (Section 3.3), with 𝑛train = 200, 𝑛val = 80, 𝑛test = 50 per seed (250 total per horizon), train_seq_len = 8, epochs = 5, batch_size = 40, and five seeds: {20260525, 20260526, 20260527, 20260528, 20260529}. Evaluation is conducted at horizons 𝐿eval ∈ {4096, 16384, 65536, 524288, 1048576}. With 𝐿train = 8, the longest evaluation horizon yields a length extrapolation ratio of 𝐿eval /𝐿train = 131072 ≥ 105 . For the projected-readout model, training is conducted at a small but nonzero projection temperature; long-horizon evaluation is reported under hard projection. Local action supervision over Σ and presentation supervision enforcing 𝑎|𝑎| = 𝑒 for each generator are applied during training. 5.2.1
Projection-matched baselines
To control for the possibility that baseline degradation under the present protocol is a readout artifact rather than an architecture difference, we add three projection-matched baselines: a GRU, a single-configuration structured state-space model (same configuration as in Section 5.1), and a bag-of-tokens encoder, each equipped with a learned prototype-projection readout over the 36 elements of 𝐺 = 𝑆3 × 𝑆3 . The prototype readout is trained with final group-state 7
cross entropy only; no local-action or presentation-supervision signals are added. The held-out transition-pair split, seed list, training budget, evaluation horizons, and 𝑛test = 50 per seed match Section 5.2.
5.3
Gate C: mechanism diagnostics under projection temperature
Purpose. To probe whether the projected-readout model’s representation behaves approximately as a group homomorphism, and how that behavior degrades as projection softens. Configuration. Four diagnostics are evaluated across a sweep of projection temperature 𝑇 ∈ {0.25, 0.50, 0.75, 1.00, 1.50, 2.00, 3.00}. at evaluation lengths {2048, 8192}: • Final-token accuracy. The probability that the model’s prediction of H_L equals the ground-truth group element. • Exact homomorphism error. 𝐸(𝑢,𝑣) [𝑑(𝜋(𝑠(𝑢𝑣)), 𝜋(𝑠(𝑢)) ⋅𝐺 𝜋(𝑠(𝑣)))] over selected word pairs. • State-consistency drift. For a carrier admitting a constraint function C(s) = 0 that an exact per-token update would preserve, we define state-consistency drift as 𝐿
𝐷 = 𝐿−1 ∑ ‖𝐶(𝑠𝑡 )‖. 𝑡=1
where L is the evaluation horizon, s_t is the model’s continuous-valued hidden state at position t, and ‖ ⋅ ‖ is a specified matrix or vector norm. The functional form of C is a property of the carrier. The specific C used to compute the drift values reported in Section 6.3 is not disclosed in the present paper. • Commutator gap. For selected pairs (𝑥, 𝑦) ∈ Σ × Σ whose group commutator [x, y] is non-identity, the distance between the model’s representation of [x, y] and its representation of the group identity.
5.4
Gate E: leakage and triviality firewall
Purpose. To certify that the held-out evaluation partition does not contain trivially leaked content from the training partition, and that the perturbations the diagnostics evaluate are genuinely non-commutative-specific. Configuration. Five group-product specificity rates are reported over the clean-split data structure: • contextual_commutator: rate at which deterministic commutator construction yields distinct group products in context; • contextual_inverse_shuffle: rate at which inverse-cancellation order perturbation changes the group product; • held_out_generator_pair: rate at which paired held-out generator orders (ab vs ba) yield distinct group products; • reversed_word_difference: rate at which reversed-order perturbation changes the group product; • same_multiset_different_product: rate at which two sequences with identical generator multisets yield distinct group products. 8
Each rate is reported with a bootstrap 95% interval over five seeds. An overlap audit additionally verifies zero verbatim reduced-word overlap and zero structural-template overlap between training and evaluation partitions. These rates are properties of the data generation process, not direct neural-model accuracies. Gate E is a leakage and triviality firewall, not a model performance result.
6
Results
6.1
Gate A: baseline competence
Model
Task
Seeds
Mean acc
95% interval
bag GRU structured SSM (single config) bag GRU structured SSM (single config)
easy commutative easy commutative easy commutative
5 5 5
1.0000 1.0000 0.8095
[1.0000, 1.0000] [1.0000, 1.0000] [0.7635, 0.8555]
6-class noncommutative 6-class noncommutative 6-class noncommutative
5 5 5
0.1600 0.1690 0.1900
[0.1395, 0.1835] [0.1355, 0.2015] [0.1380, 0.2650]
Chance accuracy on this 6-class diagnostic control is 1/6 ≈ 0.1667. The bag and GRU baselines solve the easy commutative control at the largest reported length with mean accuracy 1.0000; the single-configuration structured SSM solves it at 0.8095 ± seed variation. On the 6-class non-commutative final-state diagnostic, all three baselines remain at or near chance (0.16–0.19). The baselines used in the present paper are therefore not generically incompetent at state-tracking-style sequence tasks; they fail the 6-class non-commutative diagnostic under the same training protocol.
9
Gate A: baseline competence under matched protocol
Mean accuracy (5 seeds)
1.0 0.8 0.6 0.4 0.2 0.0
bag
chance = 1/6 0.1667
GRU
structured SSM
easy commutative (positive control)
6-class noncommutative held-out (diagnostic)
Figure 1: Gate A baseline competence under matched protocol. The dotted line marks the 1/6 chance level for the separate 6-class diagnostic control. Error bars indicate 95% bootstrap intervals over five seeds.
6.2
Gate B: long-horizon held-out transition-pair performance
Main result (expanded 𝑛test = 50). Model Hard-projected (ours) Hard-projected (ours)
Eval length
Seeds
n per seed
Exact / total
Mean final acc
95% lower bound
524288
5
50
250 / 250
1.0000
0.9854
1048576
5
50
250 / 250
1.0000
0.9854
The 95% lower bound is a two-sided Clopper-Pearson interval lower endpoint with alpha = 0.05. Short-horizon supplement (same 𝑛test = 50 protocol). Model Hard-projected (ours) Hard-projected (ours) Hard-projected (ours)
Eval length
Seeds
n per seed
Exact / total
Mean final acc
95% lower bound
4096
5
50
250 / 250
1.0000
0.9854
16384
5
50
250 / 250
1.0000
0.9854
65536
5
50
250 / 250
1.0000
0.9854
Statistical reporting note. The expanded Gate B run uses 50 evaluations per seed across five fixed seeds (250 samples per horizon). All 250 evaluations at both 524288 and 1048576 are error-free (perfect 250/250 at each horizon). The two-sided Clopper-Pearson 95% lower 10
bound is 0.9854 at both horizons. The earlier n_test = 8 pilot evidence is superseded by this expanded sampling. Broader sampling at the longest horizons remains future work. Soft / unprojected regime. The error-free behavior under hard projection must not be conflated with soft or unprojected behavior of the same model. Under soft evaluation at T = 1.0 on the same evaluation set, the model produces all-token accuracy ≈ 0.23 / ≈ 0.13 at 524288 / 1048576 and final-token accuracy 0.0000. The two regimes correspond to the two projection forms in Section 4.2 and are reported separately throughout. The hard-projected model produces error-free final-state predictions (250/250 at each horizon) across two evaluation horizons separated from the training horizon by approximately four to five orders of magnitude. The expanded sampling makes the main long-horizon result substantially less sensitive to small-n pilot variance than the earlier n_test = 8 pilot. (a) Short-horizon regime
(b) Million-token regime
0.8
1.0 250/250 exact
0.6
hard-projected (ours) native readout (pilot) chance = 1/36 0.0278
0.4 0.2
Final-token accuracy
Final-token accuracy
1.0
0.0
0.8
250/250 exact hard-projected (ours) GRU + prototype SSM + prototype bag + prototype GRU (native, pilot) chance = 1/36 0.0278
0.6 0.4 0.2 0.0
4,096
16,384 Evaluation length (tokens)
65,536
524,288 1,048,576 Evaluation length (tokens)
Figure 2: Gate B held-out-pair falsifier. Panel (a) shows the short-horizon supplement. Panel (b) shows the million-token expanded test with projection-matched baselines (GRU, bag, structured SSM with prototype-projection readout) and the retained native-readout pilot reference. The dotted line marks chance accuracy for 𝑆3 × 𝑆3 , 1/36 ≈ 0.0278. Curves are shown only for model/horizon combinations implemented in the present harness. 6.2.1
Native-readout baselines (retained from earlier pilot)
The native-readout baseline pilot from the earlier n_test = 8 configuration is retained below as context. These rows are not re-evaluated at 𝑛test = 50 in this paper and are reported only to preserve continuity with the original pilot.
Model / regime bag, native readout bag, native readout GRU, native readout GRU, native readout structured SSM, native readout structured SSM, native readout structured SSM, native readout
Eval length
Seeds
n per seed
524288 1048576 524288 1048576 4096
5 5 5 5 5
8 8 8 8 8
0.0000 0.0000 0.0000 0.0500 0.0000
pilot only pilot only pilot only pilot only pilot only
16384
5
8
0.0000
pilot only
65536
5
8
0.0000
pilot only
11
Mean final acc
Read
6.2.2
Projection-matched baselines
To test the readout-artifact hypothesis, three baselines with learned prototype readout over the 36 elements of 𝐺 = 𝑆3 × 𝑆3 were trained and evaluated under the same protocol as Section 5.2 (held-out transition-pair split, five fixed seeds, 𝑛test = 50 per seed, evaluation at 524288 and 1048576). The chance accuracy is 1/36 ≈ 0.0278.
Model
Readout
GRU
prototype projection prototype projection prototype projection prototype projection prototype projection prototype projection
GRU Structured SSM Structured SSM Bag Bag
Eval length
Exact / total
Mean acc
95% lower bound
524288
7 / 250
0.0280
0.0113
1048576
2 / 250
0.0080
0.0010
524288
9 / 250
0.0360
0.0166
1048576
3 / 250
0.0120
0.0025
524288
15 / 250
0.0600
0.0340
1048576
6 / 250
0.0240
0.0089
None of the projection-matched baselines approaches the hard-projected result. Under the present tested configurations, the readout-artifact hypothesis is not supported. We emphasise that this does not establish exhaustive baseline impossibility; it tests fixed, pre-registered baseline configurations under a matched held-out-pair protocol and projection-readout control.
0.08
Projection-matched baselines (held-out-pair, Gate B) GRU + prototype SSM + prototype bag + prototype chance = 1/36 0.0278
0.07 Final-token accuracy
0.06 0.05 0.04 0.03 0.02 0.01 0.00 524,288
Evaluation length (tokens)
1,048,576
Figure 3: Projection-matched baselines under the held-out-pair protocol at 𝐿eval ∈ {524,288, 1,048,576}. All baseline cells are far below the hard-projected 250/250 result (all ≤ 15/250); the largest, bag at 524,288 (15/250), is modestly above the 1/36 chance line (dotted), while the rest sit at or near it.
12
6.3
Gate C: mechanism diagnostics across projection temperature 𝑇
Final acc
Homomorphism error
State-consistency drift
Commutator gap
0.25 0.50 0.75 1.00 1.50 2.00 3.00
1.0000 0.0400 0.0200 0.0600 0.0600 0.0400 0.0200
0.000583 0.192048 1.225186 2.960337 5.006641 5.478391 5.661507
0.032646 0.830739 0.825461 0.824627 0.828726 0.831072 0.832476
8.4844 8.1760 6.6129 4.2619 1.1692 0.3160 0.0448
We observe a coherent boundary behavior beginning at 𝑇 = 0.50: final-token accuracy collapses from 1.0000 at 𝑇 = 0.25 to 0.0400; homomorphism error rises by more than two orders of magnitude over the same step; state-consistency drift rises sharply and then plateaus near 0.83 across the softened regime; and the commutator gap decays progressively from 8.48 to 0.045 as temperature increases. In the lowest-temperature regime reported in the Gate C sweep (T = 0.25), which approximates the hard-projection regime used in Gate B, the model’s projected representation exhibits very low homomorphism error and very low drift, while preserving a large commutator gap. This is the joint behavior expected of an approximately group-homomorphic representation: the projection of a product equals the product of projections (low 𝐸homo ); the continuous state remains close to the carrier’s constraint manifold under 𝐶 (low drift); and non-commuting pairs are separated from the identity (large commutator gap). All three diagnostics co-degrade as projection softens. The coincidence of these signals, rather than any single one alone, constitutes the mechanism-level evidence the diagnostic family is designed to deliver.
6.4
Gate E: leakage and triviality firewall
Specificity check
Mean (5 seeds)
Bootstrap 95% interval
1.0000 0.7936 0.3402 0.5554 1.0000
[1.0000, 1.0000] [0.7826, 0.8046] [0.3248, 0.3556] [0.5454, 0.5656] [1.0000, 1.0000]
contextual_commutator contextual_inverse_shuffle held_out_generator_pair reversed_word_difference same_multiset_different_product
The clean-split overlap audit additionally verifies zero verbatim reduced-word overlap and zero structural-template overlap between the training and evaluation partitions of the held-out transition-pair split. The contextual commutator and same-multiset different-product checks return 1.0000: the data-generation process reliably produces inputs in which commutator construction and samemultiset-different-product reorderings yield distinct group products. Contextual inverse-shuffle and reversed-word-difference return mid-range rates (≈ 0.79, ≈ 0.56), consistent with non-trivial but not maximal sensitivity of these perturbations under the present construction. The held-out generator-pair check returns ≈ 0.34, reflecting the proportion of contextualized constructions in which perturbing the held-out generator order changes the full sequence product. This is a contextual full-product check, not a claim that the adjacent two-token products (𝑎0 , 𝑎2 ) and (𝑎2 , 𝑎0 ) are themselves distinct in the direct-product convention. None of these numbers is 13
(a) Final-token accuracy
(b) Homomorphism error (log)
0.8
100
0.6
10 1
Ehomo
Final-token accuracy
1.0
0.4
10 2
0.2 10 3
0.0
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
1.0 1.5 2.0 2.5 (c) State-consistency drift
3.0
0.5
1.0 1.5 2.0 2.5 (d) Commutator gap (log)
3.0
0.5
1.0 1.5 2.0 2.5 Projection temperature T
3.0
101
Commutator gap
Drift D
0.5
100
10 1 0.5
1.0 1.5 2.0 2.5 Projection temperature T
3.0
Figure 4: Gate C mechanism diagnostics across projection temperature. Final-token accuracy collapses beginning at 𝑇 ⋆ = 0.5; homomorphism error and state-consistency drift rise; and commutator gap decays across the softened regime. Homomorphism error and commutator gap are shown on log scale. A dotted vertical line marks the boundary at 𝑇 ⋆ = 0.5 on all four panels.
14
a neural-model accuracy. Gate E certifies that the held-out evaluation partition contains genuinely non-commutative perturbations and is free of trivial leakage.
6.5
Robustness: same-factor held-out pair
The held-out pair used for the main result, {(𝑎0 , 𝑎2 ), (𝑎2 , 𝑎0 )}, draws its two generators from different 𝑆3 factors of the direct product, which commute element-wise; the protocol therefore removes a local transition template rather than a directly non-commuting adjacent pair (Section 3.3). To check that the result does not depend on this cross-factor choice, we repeat the protocol with the held-out pair drawn from within a single 𝑆3 factor, where the two generators genuinely do not commute. We run two such splits independently: a first-factor split {(𝑎0 , 𝑎1 ), (𝑎1 , 𝑎0 )} and a second-factor split {(𝑎2 , 𝑎3 ), (𝑎3 , 𝑎2 )}, each with the same five seeds, 𝑛test = 50 per seed (250 total per horizon), training length 𝐿train = 8, and evaluation horizons 𝐿eval ∈ {4096, 65536, 524288, 1048576}. For each split a per-seed audit confirms that the held-out pair occurs zero times in the training partition. Held-out split
Model
Exact / total
Mean acc
95% LB
First factor First factor First factor First factor
Hard-projected (ours) GRU + projection Structured SSM + projection Bag + projection
250 / 250 2 / 250 3 / 250 6 / 250
1.0000 0.0080 0.0120 0.0240
0.9854 0.0010 0.0025 0.0089
Second factor Second factor Second factor Second factor
Hard-projected (ours) GRU + projection Structured SSM + projection Bag + projection
250 / 250 3 / 250 2 / 250 8 / 250
1.0000 0.0120 0.0080 0.0320
0.9854 0.0025 0.0010 0.0139
The hard-projected model is error-free (250/250) at every horizon for both in-factor splits, identical to the cross-factor main result, while the projection-matched GRU, structured SSM, and bag baselines all remain near the 1/36 chance line. The headline 1,048,576-token horizon is shown above; the 4096, 65536, and 524288 horizons exhibit the same pattern (hard-projected 250/250 throughout; every projection-matched baseline cell at or below 15/250). The result is therefore not an artifact of the cross-factor template choice: it holds equally when the held-out adjacent pair is genuinely non-commuting, and on either factor of the product group.
Same-factor held-out transition-pair falsifier First factor
Mean final-token accuracy
1.0
Second factor
0.8 hard-projected (ours) GRU + projection structured SSM + projection bag + projection chance = 1/36 0.0278
0.6 0.4 0.2 0.0
6
4,09
6
36 ,288 ,576 65,5 524 1,048 Evaluation sequence length
4,09
36 ,288 ,576 65,5 524 1,048 Evaluation sequence length
Figure 5: Same-factor held-out robustness on 𝑆3 × 𝑆3 . Across both the first-factor and secondfactor in-factor held-out splits, the hard-projected model is error-free at every evaluation horizon while the projection-matched GRU, structured SSM, and bag baselines remain near the 1/36 chance line. 15
7
Discussion
7.1
What this demonstrates
Within a controlled state-tracking regime, an explicit projected non-commutative statecomposition interface provides a useful inductive bias for preserving ordered hidden states across horizons orders of magnitude longer than those seen in training. The held-out transitionpair falsifier makes this claim precise by blocking one direct local-transition memorization pathway. Under this protocol, the projected-readout model maintains exact final-state accuracy at million-token horizons, while matched baselines remain near chance. This does not rule out every possible form of memorization or interpolation, but it closes the most direct local-template pathway targeted by the split.
7.2
What this does not demonstrate
The result is deliberately narrow. It does not establish general superiority of the proposed interface over Transformers, recurrent networks, or state-space models on open-ended sequence tasks. It also does not demonstrate that conventional sequence models are incapable of noncommutative state composition in principle, only that the matched baselines tested here fail under the specific constraints of this falsifier. The present evidence is scoped to a controlled finite-group benchmark with explicit projection; transfer to natural language, code, or real-world workflows remains future work.
7.3
Why this matters
State-tracking evaluation in current practice does not always actively block memorization pathways. As model families and benchmarks both grow, the gap between “appears to extrapolate” and “actually composes” widens. We view the held-out transition-pair falsifier as a small but reusable evaluation primitive that can be applied wherever a sequence-modeling claim turns on order-sensitive hidden-state composition. The projection-temperature diagnostic family is a complementary primitive that distinguishes representations behaving approximately as group homomorphisms from representations producing correct outputs by other means.
8
Limitations
A few limitations matter for how these results should be read. 1. Solvable target group. The main result is on 𝑆3 ×𝑆3 . Appendix F reports a preliminary 𝑆5 non-solvable stress test with both native-readout and projection-matched baselines, but it remains a single non-solvable group under a non-released carrier and carries its own carrier-embedding caveat; it is not a general non-solvable-group tracking result. 2. Synthetic, controlled evaluation. The benchmark is a finite-group state-tracking task. Real-world workflows, natural language, and code are out of scope, and no downstream application (e.g. workflow first-divergence localization or natural-language reasoning) is tested here. 3. Hard projection is essential. Exactness in Gate B holds under hard projection (𝑇 → 0). Soft and unprojected variants of the same model collapse at the same horizons, and the two regimes are reported separately rather than being interchangeable. 16
4. Baseline coverage is partial. Projection-matched GRU, structured SSM, and bag baselines were run under the same held-out split, same seeds, and 𝑛test = 50 on 𝑆3 × 𝑆3 (Section 6.2.2), on the two in-factor robustness splits (Section 6.5), and on 𝑆5 (Appendix F). A broader sweep over state-space variants, attention-based architectures, and hybrids is left to future work. The projection-matched comparison does not by itself exhaust the readout-vs-architecture isolation question; it controls the readout artifact under fixed baseline configurations. 5. No direct comparison to recent non-Abelian-state architectures under the same held-out split. The Holonomic Network [8], PD-SSM [6], and M2 RNN [7] are not run under the held-out transition-pair falsifier in this paper. A feasibility audit found that PD-SSM has public reference code but would require a custom data adapter and readout mapping for a fair matched comparison (deferred to a companion comparison), while no official reference implementation was located for M2 RNN or the Holonomic Network (literature-only positioning for now). Direct comparison therefore remains future work; the present evidence should be read as a protocol-and-readout result rather than as a model-level ordering over these architecture families. 6. Carrier-level implementation boundary. The continuous-carrier form of the projected recurrent state model is not described here. The benchmark, the falsifier, and the diagnostics can be reproduced without those details. Carrier construction is outside the scope of this protocol-and-readout preprint and is treated separately in an architecture report. The claim is not unsupervised discovery of the group law. The model uses local action supervision and presentation-level constraints; the evidence concerns the resulting projected state-composition interface under the held-out falsifier. Two further notes. The earlier pilot used 𝑛test = 8 per seed; the long-horizon evaluation reported here was expanded to 𝑛total = 250 per horizon on 𝑆3 × 𝑆3 , giving a Clopper–Pearson 95% lower bound of 0.9854 under the tested distribution. Broader sampling is still future work. This is a technical preprint: it prioritizes reproducibility of the protocol, data generation, overlap audit, and diagnostics over end-to-end model reproducibility.
9
Conclusion
We introduced a held-out transition-pair falsifier for non-Abelian state tracking and showed that a projected recurrent state model preserves exact final-state accuracy across million-token evaluation horizons after short-horizon training, while matched native-readout baselines remain near floor under the same protocol. Mechanism diagnostics across a projection-temperature sweep exhibit a coherent boundary consistent with approximately group-homomorphic behavior under hard projection. The result is deliberately narrow. Its point is simple: when the state is order, explicit projected non-commutative structure can be a useful inductive bias for milliontoken horizons.
17
A
Full data generation protocol
This appendix specifies the data generation procedure used in Gate B and Gate E. 1. Group representation. 𝑆3 × 𝑆3 is represented as ordered pairs of S_3 elements under permutation-matrix arithmetic, with the group operation given componentwise. 2. Generator definitions. Σ = {𝑎0 , 𝑎1 , 𝑎2 , 𝑎3 }, where a_0, a_1 are two generators of the first S_3 factor and a_2, a_3 are two generators of the second factor. Generators are selected such that each factor’s generators together generate the full S_3. Note that, by the direct-product structure, any generator from the first factor commutes with any generator from the second factor. 3. Sequence generation. Sequences are drawn by sampling generators independently from Σ and then post-processing to (a) reject training sequences containing any forbidden ordered pair as consecutive generators, and (b) inject required ordered pairs into evaluation sequences at sampled positions. 4. Rejection sampling for forbidden pairs. Training sequences are sampled by independent generator draw, then any sequence containing a forbidden pair is rejected and resampled. Rejection rate at 𝐿train = 8 is moderate and is recorded in the seed manifest. 5. Required-pair insertion. Evaluation sequences are sampled and then deterministically modified to include each required pair in at least one consecutive position; the modification is constructed to preserve the unconstrained nature of the remaining positions. A frozen seed manifest is emitted with each Gate B and Gate E rollup.
B
Overlap audit details
The clean-split overlap audit computes two overlap statistics between the training and evaluation partitions: • Verbatim reduced-word overlap. The fraction of evaluation reduced-word signatures that also appear verbatim in the set of training reduced-word signatures, where each sequence is reduced under the relations of the target group prior to comparison. • Structural-template overlap. The fraction of evaluation structural templates (length, generator-multiset, and ordered-pair multiset) that also appear in the training set under the same template criterion. Both statistics are reported per check and per seed. The clean-split data used in Gate E reports zero overlap under both criteria. A separate audit run on the development data of an earlier prototype identified a non-zero reduced-word overlap on three of five checks; that prototype is not reported as the present paper’s evidence, and was demoted to structural-probe status prior to the clean-split rebuild.
C
Baseline configurations • Bag-of-tokens baseline. Per-token embeddings averaged across the sequence, followed by a linear continuous native readout over the class label set. • Gated Recurrent Unit (GRU). Standard single-layer or two-layer GRU, hidden size matched to baseline budget, with a continuous native readout. • Structured state-space baseline. Single configuration: 𝑑model = 64, 𝑛layers = 2, 𝑑state = 16, 𝑑conv = 4, expand = 2. Continuous native readout.
All baselines share training protocol, 𝑛train / 𝑛val / 𝑛test , optimizer, learning rate, batch size, and seed list with the projected-readout model on each task.
18
D
Prior empirical context (motivation only)
Earlier internal experiments on related but distinct finite-group tracking tasks (including permutation tasks on S_3, S_5, and several program-state benchmarks) motivated the stricter held-out transition-pair falsifier introduced in this paper. Those earlier results are not the main evidence reported here and are not assumed by the present claims; they are referenced only as motivation for the present protocol design.
E
Public Interface of the Model
This appendix specifies the public-facing interface of the projected recurrent state model used in this paper, enough to follow Sections 4–6 without seeing the carrier-level implementation. The exact carrier, embedding map, and constraint function are outside the scope of this protocoland-readout preprint.
E.1 Computation graph At sequence position 𝑡, the model maintains a continuous-valued recurrent hidden state 𝑠𝑡 and computes: • input: a generator token 𝑥𝑡 ∈ Σ; • per-token update: 𝑢𝑡 = 𝜑(𝑥𝑡 ), where 𝜑 is a learned map from Σ into the continuous state space; • associative non-commutative composition: 𝑠𝐿 = 𝑢1 ⊙ 𝑢2 ⊙ … ⊙ 𝑢𝐿 , equivalently realized by a recurrent accumulation 𝑠𝑡 = 𝑠𝑡−1 ⊙ 𝑢𝑡 ; • projection: 𝜋 ∶ 𝑆 → 𝐺, mapping the continuous state to a symbolic element of the target finite group; • output: 𝑦𝐿̂ = 𝜋(𝑠𝐿 ) ∈ 𝐺, the predicted accumulated group element.
E.2 Training-signal composition The model is trained with the following loss components: • final-state cross entropy on the predicted symbolic group element 𝑦𝐿̂ against the groundtruth 𝐻𝐿 ; • local action supervision: for each generator 𝑎 ∈ Σ, a target update on the continuous state consistent with the action of 𝑎 on the target group; • presentation supervision: the defining relations of 𝐺 are imposed on the representation (e.g., 𝑎|𝑎| = 𝑒 for each generator 𝑎); • projection/readout consistency, when used, regularises the agreement between hard and soft projection at low temperature.
E.3 Implementation boundary This appendix describes the public computation graph and training-signal family used to interpret the reported experiments. It does not specify the exact continuous carrier, internal embedding map, or carrier constraint function. Those implementation details are outside the
19
scope of this protocol-and-readout preprint. The benchmark, the held-out transition-pair falsifier, and the diagnostic family of Section 5.3 are defined and reproducible independently of these carrier-level details.
F
Preliminary S_5 Non-Solvable Stress Evidence
This appendix reports a preliminary stress test on the non-solvable group S_5 (|𝐺| = 120). It is included as preliminary scope evidence and is not part of the main paper claim.
F.1
Group setup and split
The target group is the symmetric group S_5. Generators (Cayley): r = (1 2 3 4 5) of order 5 and s = (1 2) of order 2. These generate S_5 and do not commute. Generator set convention: positive only, Σ = {𝑟, 𝑠}. The held-out transition-pair split is P_forbid = P_require = {(r, s), (s, r)}. Chance accuracy for the full group-state classification is 1/120 ≈ 0.0083.
F.2
Configuration
Both a short-horizon configuration (𝐿train = 8, 𝐿eval ∈ {512, 2048, 8192}) and an extended configuration (𝐿train = 16, 𝐿eval ∈ {512, 2048, 8192, 65536}) were executed with 𝑛test = 50 per seed across the same five seeds. The aggregate table below combines the two configurations for the three shared shorter horizons.
F.3
Results
Model Hard-projected (ours) Hard-projected (ours) Hard-projected (ours) Hard-projected (ours) GRU native readout GRU native readout GRU native readout GRU native readout
Eval length
Exact / total
Mean acc
95% lower bound
Chance
512 2048 8192 65536 512 2048 8192 65536
500 / 500 500 / 500 500 / 500 250 / 250 3 / 500 4 / 500 3 / 500 0 / 250
1.0000 1.0000 1.0000 1.0000 0.0060 0.0080 0.0060 0.0000
0.9926 0.9926 0.9926 0.9854 0.0012 0.0022 0.0012 0.0000
0.0083 0.0083 0.0083 0.0083 0.0083 0.0083 0.0083 0.0083
Projection-matched baselines on 𝑆5 To remove the readout as a confound on this stress setting, we additionally ran prototypeprojection GRU, structured SSM, and bag baselines over the 120 elements of 𝑆5 under the same held-out split, the same five seeds, and 𝑛test = 50 per seed (250 total per horizon). All three projection-matched baselines remain near the 1/120 ≈ 0.0083 chance reference; the largest single aggregate cell is bag + projection at 3/250.
20
Model
Eval length
Exact / total
Mean acc
512 2048 8192 65536 512 2048 8192 65536 512 2048 8192 65536
0 / 250 1 / 250 1 / 250 0 / 250 0 / 250 1 / 250 1 / 250 1 / 250 2 / 250 3 / 250 1 / 250 2 / 250
0.0000 0.0040 0.0040 0.0000 0.0000 0.0040 0.0040 0.0040 0.0080 0.0120 0.0040 0.0080
GRU + projection GRU + projection GRU + projection GRU + projection Structured SSM + projection Structured SSM + projection Structured SSM + projection Structured SSM + projection Bag + projection Bag + projection Bag + projection Bag + projection
Preliminary S5 non-solvable stress test 1.0
Final-token accuracy
0.8 0.6
hard-projected (ours) GRU (native readout) chance = 1/120 0.0083
0.4 0.2 0.0 103
104 Evaluation length (tokens)
Figure 6: Preliminary 𝑆5 non-solvable stress test. The hard-projected model remains exact across the executed horizons, while native-readout GRU remains near chance. Chance line at 1/120 ≈ 0.0083.
F.4
Honest caveat
Perfect S_5 accuracy under the present projected-readout interface raises a natural architectural question: does the non-released carrier instantiate or approximate a representation of S_5 in a way that makes this stress setting especially favourable? That mechanism analysis is outside the scope of the present preprint. The present empirical claim is bounded by the protocol: the hard-projected model solves the reported held-out split on S_5 under the stated configuration, while a GRU with native readout remains near chance. This appendix should not be read as evidence that arbitrary non-solvable groups are solved by the interface; it is a single stress test under a non-released implementation. The result should not be framed as a broad complexity-class separation or a universal claim about all sequence models. Both native-readout and projection-matched baselines (GRU, structured SSM, bag) remain near chance on this split, so the gap is not a readout artifact under the tested configurations; but this remains a single non-solvable group under a non-released 21
Projection-matched baselines on S5 (held-out-pair)
0.025
GRU + projection Structured SSM + projection Bag + projection chance = 1/120 0.0083
Final-token accuracy
0.020 0.015 0.010 0.005 0.000
512
2,048
8,192 Evaluation length (tokens)
65,53
6
Figure 7: Projection-matched baselines on 𝑆5 under the held-out-pair protocol. Prototypeprojection GRU, structured SSM, and bag baselines all remain near the 1/120 ≈ 0.0083 chance line, mirroring the native-readout GRU and supporting that the hard-projected 𝑆5 result is not explained by the tested projection-readout artifact hypothesis. carrier, not a general non-solvable-tracking result.
F.5
Wall-clock summary
Per-seed evaluation wall-clock times for the hard-projected row were short and broadly comparable to the GRU baseline (illustrative median values: approximately 0.03 s at 512, 0.10 s at 2048, 0.41 s at 8192, and 3.3 s at 65536 on the reported CUDA device). A more thorough timing study is left to future work.
G
Reproducibility notes
G.1
Eval-set provenance
For the main 𝑆3 × 𝑆3 result, the runner uses the same deterministic generator call for every phase, and recomputed token+label SHA-256 hashes are identical across phases for each (seed, eval_length) cell; the audit certifies identity by deterministic regeneration. For the same-factor robustness splits (Section 6.5) and the 𝑆5 projection-matched runs (Appendix F), the actual evaluation token and label arrays are additionally persisted as files, each with a recorded SHA-256, a token-encoding specification, and required-pair insertion positions, supporting independent byte-for-byte replay without regeneration. A separate per-seed audit of the length-8 training partitions confirms zero occurrences of the held-out pair in training for the main split, the two in-factor splits, and the 𝑆5 split.
G.2
Structured-SSM dependency note
The structured SSM baseline used mamba-ssm 2.3.2.post1+cu11torch2.6 on an NVIDIA A100 device. A small compatibility shim (triton.set_allocator no-op) was added at the import site to bypass an attribute the installed Triton package did not expose. The shim af-
22
fects only import/runtime compatibility; it does not alter data, labels, split policy, or reported outcomes. A version manifest is maintained alongside the artifacts.
References [1] David A. Barrington. Bounded-Width Polynomial-Size Branching Programs Recognize Exactly Those Languages in NC^1. Journal of Computer and System Sciences, 38(1):150–164, 1989. doi:10.1016/0022-0000(89)90037-8. [2] Kenneth Krohn and John Rhodes. Algebraic Theory of Machines. I. Prime Decomposition Theorem for Finite Semigroups and Machines. Transactions of the American Mathematical Society, 116:450–464, 1965. doi:10.2307/1994127. [3] William Merrill, Jackson Petty, and Ashish Sabharwal. The Illusion of State in State-Space Models. ICML 2024; arXiv:2404.08819, 2024. [4] Mehran Shakerinava, Behnoush Khavari, Siamak Ravanbakhsh, and Sarath Chandar. The Expressive Limits of Diagonal SSMs for State-Tracking. arXiv:2603.01959, 2026. [5] M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal, and Roland Memisevic. On the “Induction Bias” in Sequence Models. arXiv:2602.18333, 2026. [6] Aleksandar Terzić, Nicolas Menet, Michael Hersche, Thomas Hofmann, and Abbas Rahimi. Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models. NeurIPS 2025 Spotlight; arXiv:2509.22284, 2025. [7] Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, and Tri Dao. 𝑀 2 RNN: NonLinear RNNs with Matrix-Valued States for Scalable Language Modeling. arXiv:2603.14360, 2026. [8] Ilmo Sung. Robust Reasoning as a Symmetry-Protected Topological Phase. arXiv:2601.05240, 2026.
Code and Data Availability The public release package includes benchmark generation code, held-out split construction, overlap-audit scripts, Gate E specificity checks, result CSVs, figure scripts, evaluation-set hashes, and projection-matched baseline configurations. These artifacts are intended to make the falsifier protocol, clean-split audit, baseline controls, and reported figures independently inspectable. The code, configurations, figures, result tables, and evaluation-set hashes are available at https: //github.com/jeonghoon-ad/heldout-transition-pair-falsifier. The long-horizon evaluation token/label arrays (524,288 and 1,048,576 tokens) are archived at https://doi.org/10.5281/ze nodo.20506128.
23
Component Benchmark generator Held-out split manifests Overlap-audit scripts Gate E specificity audit Result CSVs Figure scripts Proj.-matched baseline configs Evaluation-set hashes Hard-projected model carrier Carrier constraint function Core model training code
Released? Yes Yes Yes Yes Yes Yes Yes Yes No No No
Purpose Reproduce held-out splits Verify train/eval construction Check leakage / structural overlap Reproduce data-level firewall checks Reproduce reported tables Reproduce reported plots Reproduce baseline controls Verify provenance of evaluations Outside this technical preprint Outside this technical preprint Not in the public release
The release does not include the non-released carrier implementation used by the hard-projected model. Full end-to-end reproduction of that implementation is outside the scope of this technical preprint (see Section 8). Reproducibility notes on the structured-SSM dependency are listed in Appendix G. Some methods described in this paper are the subject of a pending patent application filed by the author.
Acknowledgements The author acknowledges the broader theoretical work on state-tracking expressivity by Merrill, Petty, Sabharwal, Shakerinava, Ebrahimi, and collaborators referenced above, whose work framed the question that this paper attempts a narrow constructive empirical response to. AI tools were used for technical exploration, code development, data analysis, figure preparation, and manuscript writing and editing.
24