When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation Shuhan Wang shuhan.wang.25@ucl.ac.uk

Yilin Luo yilin.luo.25@ucl.ac.uk

Nan Xu† nan.xu.25@ucl.ac.uk

Chi Wang Cheung† c.cheung.25@ucl.ac.uk

arXiv:2608.13365v1 [cs.LG] 13 Aug 2026

†

Equal contribution.

Abstract Rotation-based post-training quantisation commonly applies an orthogonal transform across an entire attention head to reduce outlier-induced error. RoPE instead partitions each head into two-dimensional frequency pairs, raising a specific design question: can a transform that respects this decomposition improve on full-head mixing? Prior work has established the per-pair rotations that commute with RoPE. We state the converse result that, for distinct frequencies, no other single-head orthogonal map commutes with RoPE. For the head-shared parameterisation used in our experiments, we then derive the rotation angle that minimises the larger channel variance under a pooled-covariance, position-averaged surrogate and verify that the implementation attains its analytic minimum. The evaluated head-shared pairwise configuration does not improve accuracy in the tested dynamic W4A4KV4 setting. Across four checkpoints, replacing the full-head Hadamard with this configuration increases perplexity at both short and long context lengths. Composing the pairwise rotation with the Hadamard satisfies the selected ±0.05-PPL interval criterion under the default estimator. Estimating the shared angle from K alone improves pairwise-only on every checkpoint without eliminating its gap relative to full-head mixing. The analytic objective controls a position-averaged second moment of a covariance reconstructed from concatenated calibration observations, whereas the dynamic quantiser determines its step from a tokenwise group range. The pairwise transform also provides only two-channel support for redistributing a peak. Along a controlled interpolation from two-channel to full-head mixing, K range, relative quantisation error, and perplexity degradation decrease as support increases. These results show that optimality for a structured surrogate need not reduce quantisation error when the surrogate and mixing support are misaligned with the quantiser’s scale-setting statistic.

1

Introduction

Low-bit post-training quantisation depends strongly on how activation outliers are presented to the quantiser (Dettmers et al., 2022). Rotation-based methods insert orthogonal transforms that redistribute activation energy while preserving the unquantised network function (Ashkboos et al., 2024b; Liu et al., 2025; Shao et al., 2025). These methods build on weight-side incoherence and transformer invariance results (Chee et al., 2023; Tseng et al., 2024; Ashkboos et al., 2024a); §8 reviews this development. For the online transform R3 applied to post-RoPE queries and keys, the relevant design choice is the mixing support: should the transform mix all channels in an attention head, or only the two channels in each RoPE frequency pair? We study this question under dynamic W4A4KV4 quantisation. RoPE (Su et al., 2024) provides a natural motivation for pairwise mixing because it partitions every head into fixed two-dimensional subspaces. A rotation within these subspaces respects the frequency-pair decomposition and can be commuted through RoPE. Applying the same post-RoPE orthogonal map to Q and K already preserves their inner products, so commutativity is a structural constraint rather than a correctness requirement. We test whether imposing this constraint improves quantisation accuracy. 1

For a single head with distinct RoPE frequencies, the commuting orthogonal maps are exactly the independent planar rotations within the frequency pairs. FPTQuant (van Breugel et al., 2025) previously used this family by folding the transform into WQ and WK ; our theoretical contribution is the converse characterisation that excludes other single-head commuting orthogonal maps. The evaluated implementation further restricts the family by sharing one angle for each layer and frequency pair across attention heads. For this head-shared parameterisation, we derive the closed-form angle ϕ∗k that minimises the larger channel variance under a pooled-covariance, position-averaged surrogate. On Llama-3.2-3B, the implementation uses the checkpoint’s deployed RoPE frequencies and attains the analytic minimum of that surrogate. This construction distinguishes failure to optimise the stated local objective from failure of the optimised surrogate to improve quantisation accuracy. The experiments exhibit the second outcome. Across the evaluated Llama and Mistral checkpoints, replacing the full-head Hadamard with the head-shared pairwise configuration increases perplexity in every short-context comparison. The same ordering holds at every evaluated long-context point, including 128K where available. Composing the pairwise rotation with the Hadamard satisfies the selected interval criterion under the default estimator. These observations establish that the optimised local surrogate does not provide the range reduction achieved by full-head mixing in the evaluated setting. We examine two factors associated with this discrepancy. First, the default angle estimator pools Q and K calibration statistics even though K is the only stream quantised after R3 . Estimating the shared angle from K alone improves the pairwise-only configuration on all four checkpoints, although its perplexity remains higher than that of the full-head Hadamard. Second, the analytic surrogate and the quantiser operate on different statistics: the closed-form rotation minimises a position-averaged second moment within each pair, while the dynamic INT4 quantiser determines its step from a tokenwise range over a larger group. Pairwise rotations also have two-channel mixing support. In an interpolation that expands block-Hadamard support from one pair to a full head, K range, relative quantisation error, and perplexity degradation decrease with support size. These findings support three contributions. Contributions. • An exact local benchmark. For a single head with distinct RoPE frequencies, the RoPE-commuting orthogonal family consists of independent pairwise rotations. We attribute this family to van Breugel et al. (2025) and establish the converse characterisation. For the head-shared implementation, we derive the minimax-variance member of a pooled-covariance, position-averaged surrogate and verify attainment of its analytic minimum. • A scoped negative result for head-shared pairwise mixing. Under dynamic W4A4KV4, the evaluated head-shared RoPE-aligned configuration yields higher perplexity than full-head Hadamard mixing across four checkpoints and all evaluated contexts. On Llama-3.2-3B, the same ordering holds for the verified surrogate optimum ϕ∗k . K-only estimation improves pairwise-only without reversing its ordering relative to full-head mixing. • Evidence on proxy alignment and mixing support. The theoretical surrogate controls a pairwise, position-averaged second moment, whereas the dynamic quantiser uses a tokenwise group range to determine its scale. Along one controlled support interpolation, broader mixing is associated with lower K range, lower quantisation error, and smaller end-to-end degradation.

2

Problem setup: RoPE-aligned versus full-head Q/K mixing

We study the online transform R3 applied to queries and keys after RoPE. Using the same orthogonal map for both streams preserves their inner products. We impose commutativity as the structural constraint of interest and compare two support sizes: mixing within each two-dimensional RoPE frequency pair and mixing across an entire attention head. Frequency pairs and mixing support. Fix one even-dimensional LK head, dh = 2K, with distinct RoPE frequencies θk . Let G(ϕ) ∈ SO(2) be a planar rotation and Rm = k=1 G(mθk ) the RoPE map at position 2

m (Su et al., 2024). Implementations use the rotate_half layout, in which frequency pair k occupies channels {k, k +dh /2} rather than the adjacent coordinates {2k−1, 2k}. The layouts differ by a fixed channel permutation, so the per-pair statements are invariant to this relabelling. The deployed implementation applies the rotation in each checkpoint’s native layout, as documented in Appendix A. We refer to the k-th two-dimensional subspace as a RoPE frequency pair and use band only in equations and formal statements. A full-head Hadamard mixes all dh channels in a head, while a pairwise transform mixes only the two channels in one frequency pair. Their mixing supports are therefore dh and 2, respectively. What the deployed quantiser measures. At this stage of the evaluated quantisation procedure, K is quantised and the post-RoPE Q stream is not. For each token t and channel group g, dynamic asymmetric INT4 computes the range ut,g − ℓt,g and sets ∆t,g = (ut,g − ℓt,g )/(2B − 1) with B = 4. The scale-setting statistic is therefore a tokenwise range over a group that spans multiple frequency pairs. It differs from a per-pair second moment averaged over positions. The local variance objective is empirically non-trivial. A descriptive calibration pass finds nonnegligible within-pair anisotropy and correlation on every evaluated checkpoint: mean |ρ| is 0.135–0.175, between 72% and 78% of (layer, head, pair) triples have |ρ| > 0.05, and the mean absolute log variance ratio is 0.36–0.58. The pairwise variance objective therefore has structure to optimise; Table 6 reports the complete diagnostics.

3

The single-head RoPE centraliser and a pooled-covariance optimum

With distinct frequencies, commutativity restricts each head to independent planar rotations within its RoPE frequency pairs. We first characterise this single-head family and then define the exact surrogate optimised by the head-shared implementation. The resulting angle minimises the larger channel variance under a pooled-covariance, position-averaged objective. This result provides a local analytic benchmark for the stated surrogate; it does not characterise the tokenwise range used by the quantiser or end-to-end perplexity. The calibration collector treats samples, sequence positions, and attention heads as observation rows and accumulates their raw first and second moments. The default estimator adds the Q and K accumulators before applying a single centring operation; the alternative estimators modify only this stream-level aggregation rule. Each estimator therefore produces one covariance matrix for every layer and frequency pair. We write this matrix as Σ(k) , suppressing the layer index. It is the covariance of the concatenated observation rows, rather than an average of separately centred head-wise or stream-wise covariance matrices. The analysis reuses this matrix at every deployed position. With the finite phase averages L−1

Ck =

L−1

1 X cos(2mθk ), L m=0

Sk =

1 X sin(2mθk ), L m=0

(1)

we define the pooled-covariance, position-averaged surrogate L−1 X ⊤ e (k) = 1 Σ R(k) Σ(k) (R(k) m ) . L m=0 m

(2)

(k)

If Σm denotes the actual pre-RoPE covariance conditional on position m, the corresponding positionconditional quantity would instead be L−1

1 X (k) (k) (k) ⊤ R Σ (Rm ) . L m=0 m m (k)

The two expressions coincide under the additional condition Σm = Σ(k) at every deployed position. We neither assume nor empirically establish this position-stationarity condition. All optimality statements below therefore refer to the pooled-covariance surrogate in Eq. (2). 3

Write σ12 , σ22 for the diagonal of Σ(k) and ρ for its correlation. Within this surrogate, finite phase averaging e (k) = 1 (σ 2 −σ 2 )Sk +ρσ1 σ2 Ck is generally non-zero when the channel variances can induce in-pair structure: Σ 1 2 12 2 differ and Sk ̸= 0, including when the pooled pre-RoPE covariance is uncorrelated. The surrogate structure can therefore arise from variance asymmetry and finite RoPE phases rather than from the optimiser-related outlier effect documented by Ahmadian et al. (2023). LK Definition 1 (Band-aligned subgroup). Bdh := { k=1 G(ϕk ) : ϕk ∈ (−π, π]} ∼ = SO(2)dh /2 . Lemma 2 (Single-head RoPE centraliser). For pairwise-distinct RoPE angles θk ∈ (0, π), the centraliser of {Rm }m∈Z in O(dh ) is Bdh . The lemma characterises one attention head. Excluding cross-head mixing, its H-head direct-product extension permits H M K M R3 = G(ϕh,k ), h=1 k=1

with head-dependent angles. The evaluated implementation imposes the additional constraint ϕh,k ≡ ϕk within each layer because it aggregates calibration rows across heads and broadcasts the resulting angle to every head. The experiments therefore evaluate a strict head-shared subfamily of the head-dependent direct-product family. e (k) , the following angle Theorem 3 (Closed-form optimum for the pooled-covariance surrogate). For fixed Σ (k) ⊤ e equalises the two diagonal entries of G(ϕk )Σ G(ϕk ) :  i h e (k) − Σ e (k) , 2Σ e (k) . (3) ϕ∗k = wrap[−π/4,π/4) 21 atan2 Σ 11 22 12 It satisfies min max

ϕk j∈{1,2}

h

e (k) G(ϕk )⊤ G(ϕk )Σ

i

= jj

σ12 + σ22 . 2

This optimum is defined with respect to the pooled-covariance surrogate; finite phase averaging preserves the trace of Σ(k) . Proofs, the 2 × 2 Hadamard comparison, and the multi-head scope of Lemma 2 are in Appendix C. When the surrogate objective is recomputed from covariances reconstructed after raw-moment aggregation and from the deployed checkpoint frequencies, every evaluated ϕ∗k configuration attains its analytic minimum. The maximum excess is 6.90 × 10−7 under a verification tolerance of 5 × 10−5 (Appendix B, §J). This verification concerns the covariance of concatenated rows; it does not establish simultaneous equalisation of head-specific covariances or optimality over head-dependent angles. Attention preservation constrains how the angle is applied, but not how it is estimated: Proposition 4 (Attention preservation requires shared Q/K band angles). For R3Q , R3K ∈ Bdh , the original relative-position attention score is preserved exactly iff their angles agree in every band modulo 2π. The matched per-pair construction of van Breugel et al. (2025) implicitly satisfies this condition. The proposition constrains application of the estimator: Q and K must receive the same angle within a matched head and frequency pair, although that angle may be estimated from either stream. Because only K is quantised at this stage, a K-derived estimator is more directly aligned with the measured quantisation pathway. We therefore test whether K-only estimation changes perplexity (§5). An observed difference establishes sensitivity to the estimator statistic; it does not, by itself, identify K-cache quantisation error as the causal pathway. This Q/K-sharing requirement is distinct from the implementation’s additional choice to share angles across attention heads.

4

Evaluation design

To test whether local optimality translates into lower quantisation error, we compare three matched online Q/K transforms: 4

• full-head Hadamard: the baseline R3 , which mixes all dh channels in a head; • pairwise-only: the head-shared RoPE-commuting rotation used in place of the Hadamard; • pairwise + Hadamard: the same head-shared pairwise rotation composed with the Hadamard. In the appendix and artifact code, these transforms are named hadamard, eq_only, and eq_layered, respectively. We evaluate Llama-3.2-1B (Meta, 2024a), Llama-3.2-3B (Meta, 2024b), Llama-3.1-8B (Grattafiori et al., 2024), and Mistral-7B-v0.3 (Mistral AI, 2024), whose architecture follows Jiang et al. (2023). We use GPTQ weights (Frantar et al., 2023) with dynamic per-token W4A4KV4 activation and KV-cache quantisation. Calibration uses 128 WikiText-2 (Merity et al., 2017) sequences of length 2048 and DartQuant’s whip objective (Shao et al., 2025) for the shared offline rotations. Within each paired comparison, all configurations use the same offline rotations, GPTQ Hessian, calibration sample, and random seed; only R3 varies. Longcontext evaluation follows a concatenated-stream protocol on Proof-Pile (Azerbayev et al., 2022) and PG19 (Rae et al., 2020). The primary outcomes are perplexity, mean K range after R3 , and relative K quantisation error. We report paired differences relative to the same-seed full-head Hadamard baseline, with positive values indicating higher perplexity. Pairwise + Hadamard satisfies the manuscript’s selected ±0.05-PPL interval criterion when its complete 90% paired confidence interval lies within [−0.05, +0.05], corresponding to the interval form of the two one-sided tests procedure (Lakens, 2017). This threshold is an explicit reporting criterion, not a task-independent minimum important difference; all equivalence statements are therefore conditional on the selected margin. Appendix B, Table 10, reports sensitivity at ±0.02 and ±0.10 PPL. A cross-checkpoint equivalence statement is made only when every checkpoint satisfies the criterion. We compare three estimators within the head-shared parameterisation. The default estimator ϕ̂k adds raw Q and K moment accumulators across attention heads and centres the resulting observation population once before constructing one covariance for each layer and frequency pair; it then applies Eq. 3 with (Ck , Sk ) = (1, 0). Because this aggregation weights streams by physical head count, grouped-query attention (Ainslie et al., 2023) makes 75–80% of the concatenated observations Q-derived even though only K is quantised. The K-only estimator constructs the covariance from K observations alone. On Llama-3.2-3B, the streambalanced estimator assigns Q and K equal stream-level weights to their uncentred moments before the common centring operation. The position-averaged surrogate optimum ϕ∗k is evaluated only on Llama-3.23B, using the checkpoint’s deployed scaled RoPE frequencies and a branch-matched ϕ̂k control to isolate the finite-phase correction. Unless otherwise stated, confidence intervals are unadjusted, pointwise paired t intervals. The angleestimator analyses are exploratory effect-estimation analyses and are interpreted through estimated magnitudes and directional consistency across checkpoints. The twelve long-context comparisons between ϕ∗k and its branch-matched control form a separate multiplicity family and use Holm adjustment of two-sided pvalues (Appendix B, §J). Seed counts appear in each table caption; Appendix A documents the environments, inference conventions, and reproduction commands. The support interpolation uses the same 3B/WikiText-2 dynamic W4A4KV4 setting and pair-interleaved block-Hadamard transforms with block size b ∈ {2, 4, 8, 16, 32, 64, 128}. It measures K range and relative quantisation error after R3 . Only b = 2 belongs to the commuting family; larger blocks vary support along a specific non-commuting transform path. Layout and orientation controls use one seed and are reported in Appendix B.

5

The evaluated head-shared configuration does not match fullhead mixing

Four-checkpoint short-context test. Table 1 answers the primary empirical question for the evaluated head-shared parameterisation. Under the default estimator, pairwise-only has higher mean perplexity than the full-head Hadamard on every checkpoint, with differences ranging from +1.33 PPL on Llama-3.2-1B to +0.05 PPL on Mistral-7B-v0.3. For pairwise + Hadamard, the complete 90% paired confidence interval lies within the selected ±0.05-PPL criterion on every checkpoint. The replacement and composition results therefore support distinct conclusions: pairwise-only does not reproduce the perplexity of full-head mixing, 5

Table 1: Short-context paired perplexity differences relative to each checkpoint’s full-head Hadamard baseline on WikiText-2 under dynamic W4A4KV4 with whip calibration, n = 6. Positive values indicate higher perplexity. Under the default estimator, pairwise + Hadamard satisfies the selected ±0.05-PPL interval criterion on every checkpoint. K-only estimation improves pairwise-only on every checkpoint without reducing its perplexity below the full-head Hadamard baseline. pairwise-only model

pairwise + Hadamard

default est.

K-only est.

default est.

K-only est.

+1.3339 +0.4835 +0.1898 +0.0538

+0.8053 +0.3169 +0.1238 +0.0467

−0.0099 +0.0053 −0.0002 −0.0015

+0.0461 +0.0326 +0.0127 +0.0003

Llama-3.2-1B Llama-3.2-3B Llama-3.1-8B Mistral-7B-v0.3

Table 2: Matched Llama-3.2-3B short-context configurations on WikiText-2 under dynamic W4A4KV4 with whip calibration, n = 6, with the angle taken from the default estimator ϕ̂k or from the positionaveraged surrogate optimum ϕ∗k as indicated. The reported difference is configuration − full-head Hadamard; positive values indicate higher perplexity. The ϕ∗k configurations use the checkpoint’s deployed scaled RoPE frequencies. configuration full-head Hadamard identity (no R3 ) pairwise-only, ϕ̂k pairwise-only, ϕ∗k pairwise + Hadamard, ϕ̂k pairwise + Hadamard, ϕ∗k

mean PPL

gap vs Hadamard

90% CI

10.1121 10.5764 10.5956 10.6499 10.1174 10.1301

n/a +0.4643 +0.4835 +0.5378 +0.0053 +0.0180

n/a [+0.4294, +0.4992] [+0.4542, +0.5128] [+0.5071, +0.5686] [−0.0026, +0.0132] [+0.0050, +0.0311]

whereas the composed transform satisfies the stated interval criterion. The latter conclusion is conditional on the selected margin. Estimator ablation within the head-shared parameterisation. Relative to the default estimator, Konly estimation reduces pairwise-only perplexity on all four checkpoints, including reductions of 0.53 PPL on Llama-3.2-1B and 0.17 PPL on Llama-3.2-3B; all four unadjusted 90% intervals for the K-only-minus-default contrast lie below zero. On Llama-3.2-3B, the stream-balanced pairwise-only estimate lies between the default and K-only estimates. Relative to identity on that checkpoint, the interval for the default pairwise-only estimator includes zero, whereas the K-only and stream-balanced estimates and their unadjusted intervals are negative. For pairwise + Hadamard, the K-only-minus-default estimates are positive on Llama-3.2-1B, Llama-3.2-3B, and Llama-3.1-8B, with intervals above zero, while the Mistral-7B-v0.3 interval includes zero. These exploratory comparisons are not adjusted for family-wise multiplicity; we therefore interpret their magnitudes and directional consistency rather than treating individual intervals as confirmatory decisions. None of the evaluated estimators reduces pairwise-only perplexity below the full-head Hadamard baseline. Complete per-seed results are reported in Appendix B, §L. The pooled-covariance surrogate optimum on Llama-3.2-3B. The matched Llama-3.2-3B comparison gives the same ordering when the estimated angle is replaced by the verified position-averaged surrogate optimum. Pairwise-only with ϕ∗k has the highest mean perplexity among the reported pairwise-only configurations despite attaining the analytic objective value, and its unadjusted paired interval relative to identity lies entirely above zero. When composed with the Hadamard, both ϕ∗k and ϕ̂k satisfy the selected ±0.05PPL interval criterion relative to full-head Hadamard. Branch-matched, alternative-objective, and identity comparisons are reported in Appendix B. These results motivate an examination of why the exact surrogate optimum does not transfer to the deployed quantiser.

6

6

Evidence on the discrepancy between local optimality and quantised performance

6.1

Empirical discrepancy

The discrepancy involves three distinct quantities: the pooled-covariance surrogate optimised in theory, the tokenwise group range used by the quantiser, and measured perplexity. The evaluated head-shared configuration attains the first objective but remains less accurate than full-head mixing. We therefore examine how the surrogate differs from the scale-setting statistic and how both relate to mixing support.

6.2

Objective mismatch

Theorem 3 concerns the pooled-covariance surrogate. Dynamic INT4 instead uses a tokenwise max − min range over groups that span multiple frequency pairs (§2). A quadratic moment averaged over RoPE phases can decrease while ∆t,g remains unchanged because the two statistics apply different operations over different supports. In the matched 3B comparison, ϕ∗k attains the surrogate minimum and has the largest pairwiseonly perplexity gap. This observation bounds the analytic certificate: it does not extend to the scale-setting statistic of the deployed quantiser. Recent methods address related objective mismatches directly. Hu et al. (2025) measure utilisation of the quantisation space, and Wang et al. (2025) constrain key-quantisation error relative to a query subspace. Our result provides a negative instance in which exact optimisation of a pooled, position-averaged second moment does not control the quantiser’s tokenwise group range. Finite phase averaging also affects the interpretation of exact surrogate optimality. For high-frequency pairs, both sums (Ck , Sk ) in Eq. 1 are close to zero, so the surrogate covariance approaches isotropy and many angles are nearly optimal. The selected angular representative can then depend on a small finite-sum remainder. At L = 2048, 17 of the 64 deployed 3B pairs satisfy |(Ck , Sk )| < 0.01, and the corresponding mean offset | 12 atan2(Sk , Ck )| is 0.75 radians. This conditioning issue limits the deployment interpretation of the surrogate optimum while leaving the numerical attainment check unchanged (Appendix B, §J).

6.3

Mixing support

Mixing support imposes a complementary structural constraint. If an orthogonal transform √ distributes one peak with equal magnitude across b channels, norm preservation gives the lower bound |x|/ b on the resulting peak magnitude. A pairwise rotation has b = 2, whereas a full-head Hadamard has b = dh ; at dh = 128, the corresponding bounds differ by approximately 8×. This statement concerns an outlier-range proxy and does not imply a bound on perplexity. To examine mixing support along a controlled transformation path, we vary b within a pair-interleaved block-Hadamard family while retaining the calibration and evaluation protocol (Table 3). The point estimates for K range, relative K quantisation error, and perplexity difference decrease monotonically as b increases. The b ≤ 32 perplexity intervals lie above the full-head Hadamard baseline, whereas the b = 64 and b = 128 intervals include zero. At b = 128, all three outcomes approach the full-head endpoint, and the paired perplexity interval includes zero. More specifically, relative K quantisation error matches at the reported precision, while mean K range differs by 0.0011. This pattern documents a joint progression of support size and the three measured outcomes along one transformation family; it does not isolate support independently of the accompanying change in block structure. The dependence of outlier suppression on block size is established elsewhere and is not claimed here. DuQuant (Lin et al., 2024a) rotates within blocks and then permutes channels to balance outliers across them; PeRQ (Sanjeet et al., 2026) gives a non-asymptotic analysis in which the suppression a block-Hadamard transform achieves is governed by how ℓ1 mass is distributed over blocks, and recovers part of the lost suppression using the same permutation device; Jia et al. (2026) select block-diagonal rotation with tokenwise INT4 as the preferred KV-cache configuration under serving constraints. What Table 3 contributes is the post-RoPE Q/K instance of that axis under a dynamic per-token quantiser, with K range, relative K quantisation error and perplexity measured together along one interpolation path.

7

Table 3: Mixing-support interpolation for Llama-3.2-3B on WikiText-2 under dynamic W4A4KV4, using three paired seeds. The perplexity difference is the block-Hadamard configuration minus the same-seed fullhead Hadamard baseline; K diagnostics are measured after R3 . Only b = 2 commutes with RoPE.

6.4

b

√ 1/ b

mean K range

rel. K quant. error

PPL gap [90% CI]

2 4 8 16 32 64 128

0.7071 0.5000 0.3536 0.2500 0.1768 0.1250 0.0884

14.8156 13.1189 11.5768 10.7337 10.4475 10.1357 9.6431

0.02306 0.01805 0.01386 0.01183 0.01123 0.01054 0.00954

+0.4158 [+0.3720, +0.4596] +0.2262 [+0.2069, +0.2456] +0.1373 [+0.1242, +0.1503] +0.0751 [+0.0505, +0.0996] +0.0544 [+0.0266, +0.0822] +0.0153 [−0.0286, +0.0592] +0.0039 [−0.0236, +0.0313]

full-head Hadamard

n/a

9.6420

0.00954

0

What the evidence supports

Together, these analyses support a bounded interpretation. K-only estimation shows that the result is sensitive to the statistic from which the shared angle is estimated. Along the block-Hadamard interpolation, larger mixing support is accompanied by lower K range, lower relative quantisation error, and a smaller perplexity difference. This interpolation covers one checkpoint, one corpus, one context length, and one transformation path, and it leaves the RoPE-commuting family for b > 2. It therefore supports an association within the evaluated intervention rather than a general causal attribution. The evidence identifies two relevant mismatches, namely the estimator’s second-moment proxy and the transform’s limited support, without assigning exclusive causality to either.

7

Robustness across models, contexts, and tasks

The preceding analysis uses a short-context Llama-3.2-3B setting. We next assess whether the pairwisereplacement ordering is stable across context lengths, corpora, checkpoints, and evaluation formats. These evaluations test robustness of the ordering and provide no additional identification of its cause. Consistent with the short-context ordering in §5, pairwise-only has higher mean perplexity than full-head Hadamard at every evaluated long-context point under the default estimator. Table 4 includes all evaluated checkpoint–corpus combinations: Llama-3.2-3B and Llama-3.2-1B are evaluated on Proof-Pile and PG19 through 128K, and Llama-3.1-8B and Mistral-7B-v0.3 are evaluated on both corpora through 64K. Because the scored token set depends on context length, the table summarises within-length differences and does not model perplexity as a function of length. Complete perplexities and the position-averaged contrasts are reported in Appendix B; exact per-length window counts are specified in Appendix A, §A.4. Table 4: Ranges of evaluated within-length pairwise-only perplexity differences relative to full-head Hadamard under the default estimator and dynamic W4A4KV4, n = 3. Positive values indicate higher perplexity for pairwise-only. Each range summarises point estimates across context lengths and is not a confidence interval. checkpoint / corpus

tested lengths

gap range

longest endpoint

3B / Proof-Pile 3B / PG19 1B / Proof-Pile 1B / PG19 8B / Proof-Pile 8B / PG19 Mistral / Proof-Pile Mistral / PG19

2K–128K 2K–128K 2K–128K 2K–128K 2K–64K 2K–64K 2K–64K 2K–64K

+0.143 to +0.207 +0.566 to +1.199 +0.416 to +0.574 +1.819 to +3.861 +0.056 to +0.076 +0.206 to +0.480 +0.034 to +0.056 +0.113 to +0.238

+0.193 at 128K +1.169 at 128K +0.513 at 128K +3.585 at 128K +0.074 at 64K +0.394 at 64K +0.056 at 64K† +0.238 at 64K†

† Exploratory 64K evaluation beyond the 32K context documented for Mistral-7B-v0.3 (Mistral AI, 2024).

8

Generation-based evaluations provide limited corroboration. In the synthetic retrieval evaluation following the NIAH format used by RULER (Hsieh et al., 2024), full-head Hadamard has higher retrieval accuracy than pairwise-only at context lengths with appreciable separation from the FP16 result; shorter-context results and the Llama-3.1-8B evaluation do not distinguish the configurations. LongBench-v2 (Bai et al., 2025) scores are at or below the four-choice random baseline and therefore provide no evidence for differences among the transformations at this quantisation setting. Appendix B reports the complete task results. An exploratory static per-channel control produces lower short-context perplexity for pairwise-only than for full-head Hadamard. The control estimates fixed scales from target-corpus windows that overlap the scored text and changes scale granularity from grouped per-token to per-channel quantisation. It therefore demonstrates that the observed ordering depends on the evaluated quantiser protocol, but it neither identifies the source of that dependence nor estimates out-of-sample deployment performance. Related evidence that static and dynamic quantisers can respond differently to the same modification is reported by Chen et al. (2026b). No generation-based task evaluation was conducted for the static control, so this result is limited to perplexity under the stated protocol; complete curves and scale diagnostics are in Appendix B.

8

Related work

From incoherence processing to online rotations. Rotation-based post-training quantisation originates on the weight side: QuIP (Chee et al., 2023) makes weights and Hessians incoherent with random orthogonal transforms, and QuIP# (Tseng et al., 2024) replaces those transforms with the randomised Hadamard transform. The online setting studied here inherits that construction. The offline rotations rest on the computational invariance of RMSNorm transformers established by Ashkboos et al. (2024a), which permits an orthogonal map to be folded into the weights; the online transforms are instead inserted together with their inverses. QuaRot (Ashkboos et al., 2024b) inserts Hadamard transforms into the activation path under W4A4KV4, SpinQuant (Liu et al., 2025) learns the offline rotations, and DartQuant (Shao et al., 2025) calibrates them efficiently; we reproduce its released calibration defaults exactly, with the consequences recorded in Appendix A. Later learned and closed-form transforms retain full-head support (Sun et al., 2025; Xiang & Zhang, 2025; Akhondzadeh et al., 2025; Chen et al., 2026a), including structured factorisations whose composition still spans the head (Xu et al., 2025). Mixing support and block rotations. Confining a transform to blocks reduces cost at the expense of mixing support, and the resulting dependence on block size has been studied directly. DuQuant (Lin et al., 2024a) rotates within blocks and permutes channels so that outlier mass is balanced across them; GSR (Choi et al., 2025) uses a training-free block-diagonal Walsh–Hadamard rotation, applied to the offline residualstream transform rather than to the online Q/K path, at W2A16 and W2A4; PeRQ (Sanjeet et al., 2026) analyses block-Hadamard suppression non-asymptotically through the distribution of ℓ1 mass over blocks; and Jia et al. (2026) adopt block-diagonal rotation with tokenwise INT4 for KV-cache serving. §6 places the RoPE frequency pair at the low-support end of that axis for post-RoPE Q/K. Table 5 summarises the design axes of the nearest families. Transform objectives. Many rotation designs optimise a variance or outlier proxy. OSTQuant (Hu et al., 2025) measures utilisation of the quantisation space and permits scaling alongside the orthogonal transform, while SQuat (Wang et al., 2025) constrains key-quantisation error relative to a query subspace. Closest to the present construction in analytic form, Xiao et al. (2025) derive closed-form Givens angles for W4A4 outlier smoothing under alignment and uniformity criteria, without a RoPE-pair or commutativity constraint. The present study instead derives and numerically verifies the optimum of a pooled-covariance surrogate for the implemented head-shared parameterisation, then evaluates the corresponding configurations under dynamic W4A4KV4 quantisation. RoPE-structured transforms and concurrent work. The per-pair RoPE-commuting structure predates this work. FPTQuant (van Breugel et al., 2025) folds a scaled per-pair map into WQ and WK . ParoQuant (Liang et al., 2026) optimises Givens rotations with channel scaling over selected channel pairs in a weight-only quantisation procedure. Pavlov (2026) propose spectral-energy rescaling and describe the 9

Table 5: Design axes of the nearest transform families. Mixing support is the number of channels b over which a single peak can be redistributed. FPTQuant and ParoQuant compose the pairwise rotation with channel scaling, and DuQuant and PeRQ compose the block rotation with a permutation; the present work varies block size and mixing support without introducing channel scaling or permutation. method

placement

mixing support

fitting

quantiser

FPTQuant (van Breugel et al., 2025) ParoQuant (Liang et al., 2026) ART / URT (Xiao et al., 2025) IIS-R (Pavlov, 2026) DuQuant (Lin et al., 2024a) PeRQ (Sanjeet et al., 2026) GSR (Choi et al., 2025) RotateKV (Su et al., 2025) This work

offline WQ , WK weight-only offline, activation offline, weight side activation activation offline R1 pre-RoPE cache online post-RoPE

RoPE pair, b=2 channel pair, b=2 Givens pairs full head; per-pair block, permuted block, permuted block, b= group grouped heads b=2 to b=dh

trained trained untrained calibrated calibrated calibrated untrained calibrated untrained

static INT4 weight-only W4A4 low bit width W4A4 INT4 W2A16/W2A4 2-bit KV dynamic W4A4KV4

per-pair RoPE-commuting form as a secondary extension, folded into WQ , WK and evaluated at 1024 tokens. Yu et al. (2025) study trainable commuting angle matrices as a positional-encoding design. Lemma 2 supplies the converse result: with distinct frequencies, every single-head orthogonal map that commutes with RoPE has the per-pair form. The empirical comparison evaluates the stricter head-shared subfamily rather than the complete head-dependent direct-product family. Appendix B also reports a restricted probe of a trained and scaled FPTQuant-style component within our evaluation procedure; this probe does not constitute a reproduction of FPTQuant. Scaling and cache-side methods. Scaling reduces a peak without redistributing it across channels and is therefore complementary: SmoothQuant (Xiao et al., 2023) and Outlier Suppression+ (Wei et al., 2023) on the activation path, AWQ (Lin et al., 2024b) on the weight side, and OmniQuant (Shao et al., 2024) on both. Closest in motivation to this work is Q-ROAR (Qiao & Huang, 2026; Qiao et al., 2025), which analyses RoPE channel bands under position interpolation and reports gains from per-band weight rescaling in quantised long-context models; it varies the scaling axis under interpolated positions, whereas we vary the rotation axis under native RoPE, so the null reported here does not bear on that result (§9). On the cache side, KVQuant (Hooper et al., 2024) introduced pre-RoPE key quantisation, establishing that the post-RoPE stream is the harder one to quantise, and with KIVI (Liu et al., 2024) sets the low-bit baselines; RotateKV (Su et al., 2025) is the nearest rotation-based method, applying a pre-RoPE grouped-head rotation with channel reordering to a 2-bit cache; PolarQuant (Wu et al., 2025) and CommVQ (Li et al., 2025) act on the stored cache in a RoPE-pair-aware form, and TurboQuant (Zandieh et al., 2026) is a data-oblivious online vector quantiser.

9

Limitations and conclusion

Limitations. 1. Quantiser and method scope. The result concerns native-RoPE dynamic W4A4KV4 with the evaluated calibration procedure. Learned rotations, channel scaling, other bit widths, static deployment, and positional scaling (Peng et al., 2024) remain outside this scope. Q-ROAR (Qiao & Huang, 2026; Qiao et al., 2025) studies per-band weight rescaling under position interpolation, whereas our evaluation varies rotations under native RoPE; the present result therefore does not evaluate the Q-ROAR setting. The evaluated pairwise parameterisation shares each layer–pair angle across attention heads. Head-dependent angles ϕh,k and transforms that mix across heads are untested. Among the evaluated head-shared estimators, K-only estimation yields lower perplexity than the default estimator but is not established as optimal. The surrogate optimum ϕ∗k is evaluated only on Llama-3.2-3B. 2. Evaluation and statistical scope. Perplexity is the primary outcome. Several NIAH comparisons are saturated near the FP16 result, and LongBench-v2 is non-discriminative at the evaluated quantisation

10

setting. Most long-context evaluation points use three seeds. Every equivalence statement is conditional on the evaluated corpora, seeds, and selected ±0.05-PPL interval criterion. 3. Explanatory scope. The support interpolation covers Llama-3.2-3B on WikiText-2 at short context, three seeds, and one block-Hadamard transformation path; it leaves the commuting family for b > 2, and its layout and orientation controls use one seed. The static comparison is exploratory and does not identify a causal mechanism. Conclusion. RoPE defines a precise pairwise structure for online Q/K rotations. For a single head with distinct frequencies, the commuting orthogonal family consists of independent rotations within the RoPE frequency pairs, a structure established by prior work (van Breugel et al., 2025); our converse result completes this single-head characterisation. For the implemented head-shared parameterisation, the pooled-covariance, position-averaged surrogate has a closed-form minimiser, and the implementation attains the associated surrogate minimum. This objective-specific guarantee does not improve accuracy under dynamic W4A4KV4 quantisation in the evaluated setting. Across the tested checkpoints and context lengths, replacing the full-head Hadamard with the evaluated head-shared pairwise configuration yields higher perplexity. K-only estimation reduces this gap without eliminating it. Along the evaluated support-interpolation path, K range, relative quantisation error, and perplexity degradation decrease as mixing support approaches the full-head endpoint. This pattern is consistent with mixing support contributing to the observed difference, but it does not identify a mechanism beyond that path. The evidence separates optimality for a pooled-covariance surrogate, improvement in the quantiser’s tokenwise range statistic, and end-to-end model quality. The broader implication is that structural alignment alone does not determine whether a rotation reduces quantisation error. A structured transform should be judged by whether its optimisation objective and its mixing support match the scale-setting rule of the quantiser in which it will be deployed.

References Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Zhen Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. arXiv:2305.19268. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2305.13245. Mohammad Sadegh Akhondzadeh, Aleksandar Bojchevski, Evangelos Eleftheriou, and Martino Dazzi. KurTail: Kurtosis-based LLM quantization. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. arXiv:2503.01483. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In International Conference on Learning Representations (ICLR), 2024a. arXiv:2401.15024. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2024b. arXiv:2404.00456. Zhangir Azerbayev, Edward Ayers, and Bartosz Piotrowski. Proof-Pile: A pre-training dataset of mathematical texts. Hoskinson Center for Formal Mathematics / EleutherAI, 2022. URL https://github. com/zhangir-azerbayev/proof-pile. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on

11

realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3639–3664, 2025. doi: 10.18653/v1/2025.acl-long.183. arXiv:2412.15204. Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2307.13304. Jiale Chen, Vage Egiazarian, Roberto L. Castro, Torsten Hoefler, and Dan Alistarh. WUSH: Near-optimal adaptive transforms for LLM quantization. In International Conference on Machine Learning (ICML), 2026a. arXiv:2512.00956. Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. PrefixQuant: Eliminating outliers by prefixed tokens for large language models quantization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026b. arXiv:2410.05265. Euntae Choi, Sumin Song, Woosang Lim, and Sungjoo Yoo. Grouped sequency-arranged rotation: Optimizing rotation transformation for quantization for free. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Student Research Workshop, 2025. arXiv:2505.03810. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 30318–30332. Curran Associates, Inc., 2022. doi: 10.52202/068431-2198. arXiv:2208.07339. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023. arXiv:2210.17323. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2401.18079. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024. arXiv:2404.06654. Xing Hu, Yuan Cheng, Dawei Yang, Zhixuan Chen, Zukang Xu, Jiangyong Yu, Chen Xu, Zhihang Yuan, Zhe Jiang, and Sifan Zhou. OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In International Conference on Learning Representations (ICLR), 2025. arXiv:2501.13987. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2021. arXiv:2104.02112. Jinda Jia, Jisen Li, Zhongzhu Zhou, Jung Hwan Heo, Jue Wang, Tri Dao, Shuaiwen Leon Song, Ben Athiwaratkun, Chenfeng Xu, Tianyi Zhang, and Xiaoxia Wu. SAW-INT4: System-aware 4-bit KV-cache quantization for real-world LLM serving. arXiv preprint arXiv:2604.19157, 2026. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. 12

Daniël Lakens. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4):355–362, 2017. doi: 10.1177/1948550617697177. Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Colorado Reed, Chong Wang, and Chuang Gan. CommVQ: Commutative vector quantization for KV cache compression. In International Conference on Machine Learning (ICML), 2025. arXiv:2506.18879. Yesheng Liang, Haisheng Chen, Zihan Zhang, Song Han, and Zhijian Liu. ParoQuant: Pairwise rotation quantization for efficient reasoning LLM inference. In International Conference on Learning Representations (ICLR), 2026. arXiv:2511.10645. Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2024a. Oral; arXiv:2406.01721. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), 2024b. arXiv:2306.00978. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations (ICLR), 2025. arXiv:2405.16406; OpenReview: ogO6DGE6FZ. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 32332–32344, 2024. arXiv:2402.02750. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), 2017. arXiv:1609.07843. Meta. Llama 3.2 1B model card. Hugging Face model card, 2024a. URL https://huggingface.co/ meta-llama/Llama-3.2-1B. Meta. Llama 3.2 3B model card. Hugging Face model card, 2024b. URL https://huggingface.co/ meta-llama/Llama-3.2-3B. Mistral AI. Mistral-7B-v0.3 model card. Hugging Face model card, 2024. URL https://huggingface.co/ mistralai/Mistral-7B-v0.3. Gorgi Pavlov. Influence-inspired spectral rotations for extreme low-bit LLM quantization. arXiv preprint arXiv:2605.25203, 2026. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations (ICLR), 2024. arXiv:2309.00071. Ye Qiao and Sitao Huang. Q-ROAR: Outlier-aware rescaling for RoPE position interpolation in quantized long-context LLMs. In AAAI Conference on Artificial Intelligence (AAAI), Student Abstract Track, 2026. arXiv:2509.14391. Ye Qiao, Haocheng Xu, Xiaofan Zhang, and Sitao Huang. Rethinking RoPE scaling in quantized LLM: Theory, outlier, and channel-band analysis with weight rescaling. arXiv preprint arXiv:2510.00028, 2025. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations (ICLR), 2020. arXiv:1911.05507; introduces the PG-19 long-range language-modelling benchmark.

13

Sai Sanjeet, Ian Colbert, Pablo Monteagudo-Lago, Giuseppe Franco, Yaman Umuroglu, and Nicholas J. Fraser. Pushing the limits of block rotations in post-training quantization. arXiv preprint arXiv:2601.22347, 2026. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations (ICLR), 2024. arXiv:2308.13137. Yuantian Shao, Yuanteng Chen, Peisong Wang, Jianlin Yu, Jing Lin, Yiwu Yao, Zhihui Wei, and Jian Cheng. DartQuant: Efficient rotational distribution calibration for LLM quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2511.04063. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom. 2023.127063. arXiv:2104.09864. Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan. RotateKV: Accurate and robust 2-bit KV cache quantization for LLMs via outlier-aware adaptive rotations. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2025. arXiv:2501.16383. Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. FlatQuant: Flatness matters for LLM quantization. In International Conference on Machine Learning (ICML), 2025. arXiv:2410.09426. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks. In International Conference on Machine Learning (ICML), 2024. arXiv:2402.04396. Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, and Markus Nagel. FPTQuant: Functionpreserving transforms for LLM quantization. In ICML 2025 Workshop on Efficient Systems for Foundation Models (ES-FoMo-III), 2025. Oral; arXiv:2506.04985. Subsequently published at ICML 2026 (main conference). Hao Wang, Ligong Han, Kai Xu, and Akash Srivastava. SQuat: Subspace-orthogonal KV cache quantization. In Conference on Language Modeling (COLM), 2025. arXiv:2503.24358. Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1648–1665, 2023. arXiv:2304.09145. Songhao Wu, Ang Lv, Xiao Feng, Yufei Zhang, Xun Zhang, Guojun Yin, Wei Lin, and Rui Yan. PolarQuant: Leveraging polar transformation for efficient key cache quantization and decoding acceleration. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2502.00527. Jingyang Xiang and Sai Qian Zhang. DFRot: Achieving outlier-free and massive activation-free for rotated LLMs with refined rotation. In Conference on Language Modeling (COLM), 2025. arXiv:2412.00648. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 38087–38099, 2023. arXiv:2211.10438. Jinying Xiao, Bin Ji, Shasha Li, Xiaodong Liu, Jun Ma, Chao Wang, Wei Li, Ye Zhong, Xuan Xie, Nyima Tashi, and Jie Yu. Outlier smoothing with closed-form rotations for W4A4 large language model quantization. arXiv preprint arXiv:2511.22316, 2025. Bingxin Xu, Zhen Dong, Oussama Elachqar, and Yuzhang Shang. ButterflyQuant: Ultra-low-bit LLM quantization through learnable orthogonal butterfly transforms. arXiv preprint arXiv:2509.09679, 2025.

14

Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, and Chun Yuan. ComRoPE: Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4508–4517, 2025. arXiv:2506.03737. Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. TurboQuant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations (ICLR), 2026. arXiv:2504.19874.

A

Reproducibility

The commands, paths, and identifiers below describe the companion code artifact. Records retained only in the execution environment are identified explicitly; reproducing from raw outputs requires those records in addition to the compact artifact.

A.1 Common implementation and quantisation configuration All results are produced with a DartQuant-style W4A4KV4 post-training quantisation procedure (Shao et al., 2025). The procedure applies four rotations, following the R1 –R4 taxonomy of Ashkboos et al. (2024b) and Liu et al. (2025): an offline residual-stream rotation R1 , an offline value/output-projection rotation R2 , an online post-RoPE Q/K rotation R3 , and an online down-projection Hadamard transform R4 . Weights are quantised with GPTQ (Frantar et al., 2023). Unless a static-scale control is explicitly specified, activations and the KV cache use asymmetric per-token dynamic quantisation, with scales recomputed on every forward pass. The RoPE-aligned R3 transformation is integrated at the calibration-statistics stage and at the online Q/K transformation stage. In all reported comparisons, R4 remains a Hadamard transform. The default configuration is Llama-3.2-3B with head dimension dh = 128, corresponding to 64 RoPE frequency pairs. Calibration uses 128 WikiText-2 sequences of length 2048. Unless otherwise stated, the reference seed is 0 (–seed 0). All quantised evaluations use –quantizer_type int4 –w_bits 4 –a_bits 4 –k_bits 4 –v_bits 4. RoPE channel layout. The evaluated checkpoints use the Hugging Face rotate_half layout, in which RoPE frequency pair k occupies channels {k, k + dh /2}, not the adjacent pair {2k−1, 2k} used in the statements of §3. The two layouts differ by a fixed permutation of channels, and all per-pair statements are invariant under it, but the implementation must apply the rotation in the layout the checkpoint actually uses. It does so throughout: the band rotation is applied across the two halves of the channel axis in dartquant_v2/equalising_r3.py, and the runtime collector (dartquant_v2/equalising_r3_runtime.py) forms its covariances under the same (k, k+dh /2) pairing. The submitted head-resolved calibration summary is docs/data_db/paper/table1_rho.csv; the artifact verifier checks its complete three-decimal paper values but does not reconstruct it from omitted raw activations. For the mixing-support interpolation (§A.7), dartquant_v2/block_hadamard_r3.py applies an explicit permutation 2k + s 7→ k + s dh /2 before forming blocks, so that the b = 2 block spans the two channels of one frequency pair. Cutting blocks directly on raw half-half channel indices would instead pair channels belonging to two different bands, which would silently break the b = 2 endpoint’s membership in the commuting family; the permutation exists to prevent that. The reader-facing transformation names correspond to the following code identifiers: • full-head Hadamard: hadamard, with no equalising-R3 flags; • pairwise-only: eq_only, using –equalising_r3 –equalising_r3_no_hadamard; • pairwise + Hadamard: eq_layered or eq_r3, using –equalising_r3 without –equalising_r3_no_hadamard. The variance objective is always selected explicitly with –equalising_r3_objective variance in the corresponding configurations, because the implementation otherwise defaults to linf.

15

The primary Llama-3.2-3B evaluations use NVIDIA A100 nodes with 40 or 80 GB of memory. The crossmodel evaluations and the separate six-seed Llama-3.2-3B evaluation use H100/H200-class nodes with A100 fallback. Both hardware pools use torch 2.6.0+cu124 and transformers 4.49.0. The A100 evaluations verify fast-hadamard-transform 1.1.0; the second pool uses a source build whose version tag is not confirmed. Each result directory records the source revision, software versions, and GPU model in env.json.

A.2 Implementation of the RoPE-aligned Q/K rotation A.2.1 Calibration statistics and angle estimation. With R1 and R2 folded into the weights, one calibration forward pass records the outputs of qproj and kproj in every layer. For each stream and frequency pair, the collector folds the calibration-sample, sequence-position, and attention-head axes into one observation P P P 2 P P axis and accumulates n, a, b, a , b2 , and ab in CPU float64. Under the default rule, the Q and K totals are added elementwise and the covariance is then formed once as n X ⊤ bk = 1 xi x⊤ Σ i − x̄x̄ , n i=1

xi = (ai , bi )⊤ .

This operation computes the covariance of the concatenated Q/K observation rows. It is not an arithmetic average of separately centred head-wise or stream-wise covariance matrices; when group means differ, the common centring operation retains the corresponding between-group contribution. Because each physical head contributes the same number of calibration positions, the default Q:K observation ratios are 3:1 for Llama-3.2-3B and 4:1 for the other evaluated checkpoints. The procedure produces one covariance matrix for every layer and frequency pair. The pooling rule is selected by –equalising_r3_pool: • rows adds the raw Q and K sufficient statistics and then centres once, equivalently computing the covariance of their concatenated observation rows; • k_only constructs the covariance from K observations alone; • balanced averages the per-stream first and uncentred second moments with equal stream weights and then centres once. It is therefore not, in general, the arithmetic mean of the two centred stream covariances. This option changes only the statistic used to estimate the angle. Every configuration still applies one shared angle to Q and K and therefore satisfies Proposition 4; the estimator is compared in App. B, §L. For  the default variance objective, the implementation uses the closed-form Jacobi equalising angle ϕ̂k =  1 2 2 atan2 σ − σ , 2ρ σ σ , which is Eq. 3 evaluated at (Ck , Sk ) = (1, 0). The alternative objectives k 1,k 2,k 1,k 2,k 2 linf (max-channel L∞ ), l2max (sum of per-channel max-squared), and quantile do not have a closed form in the implementation and are evaluated over a 200-point discretisation of (−π/4, π/4]. Candidate observations are drawn from a per-pair top-k set ranked by |a| + |b|. The stored angle array has shape (n_layers, dh /2) = (n_layers, 64) and is broadcast across attention heads and sequence positions. Thus ϕℓ,h,k = ϕℓ,k for every head h. The implementation therefore evaluates a head-shared subset of the within-head commuting family and does not estimate head-dependent angles. A.2.2 Online application. The implementation applies the pairwise 2×2 rotation G(ϕ̂k ) immediately after RoPE produces (q, k). This is implemented through a runtime modification of the Q/K processing K module. The same per-pair angle is applied to both streams. Distinct angles ϕQ k ̸≡ ϕk (mod 2π) would Q K leave attention with a non-cancelling G(ϕk − ϕk ) factor (Proposition 4) and therefore would not preserve the original attention geometry. The online application occurs after the KV-cache quantiser has been configured. This placement is identical across all compared configurations except for the choice of R3 . The offline rotations, GPTQ Hessian, calibration sample, and random seed are held fixed within every paired comparison.

A.3 Short-context evaluation protocol Short-context perplexity is evaluated on WikiText-2 under dynamic W4A4KV4 quantisation. Within each model and seed, the compared transformations use the same offline rotations, GPTQ Hessian, calibration 16

sequences, and random seed. Reported differences are therefore paired within seed and isolate the online R3 transformation. The reference invocation below compares the full-head Hadamard and pairwise + Hadamard configurations for seeds 0–2. for SEED in 0 1 2; do # Full-head Hadamard baseline python dartquant_v2/run_quantize.py \ --model /path/to/meta-Llama-3.2-3B \ --loss whip --quantizer_type int4 --cal_dataset wikitext2 \ --w_bits 4 --a_bits 4 --k_bits 4 --v_bits 4 \ --seed $SEED --ppl_eval --ppl_eval_dataset wikitext2 \ --output_dir out/multiseed/whip/hadamard_seed$SEED # Pairwise + Hadamard python dartquant_v2/run_quantize.py \ --model /path/to/meta-Llama-3.2-3B \ --loss whip --quantizer_type int4 --cal_dataset wikitext2 \ --w_bits 4 --a_bits 4 --k_bits 4 --v_bits 4 \ --seed $SEED --ppl_eval --ppl_eval_dataset wikitext2 \ --equalising_r3 --equalising_r3_objective variance \ --output_dir out/multiseed/whip/eq_r3_seed$SEED done The scheduler-neutral script scripts/reproduction/run_extended_short_context.sh records this protocol and prints its commands by default. Paired differences are computed as ∆(eq_r3 − hadamard) within seed, and the reported standard deviations use the sample (n−1) convention. The full-head Hadamard baseline varies by approximately 0.1 PPL across seeds in representative long-context evaluations (the 2K cross-seed sample standard deviation is 0.082–0.289 across the reported corpora, Table 11), which motivates the use of within-seed comparisons for differences of a few hundredths of a perplexity point (the largest per-seed 3B whip difference in Table 7 is +0.0168). The cross-model short-context evaluation applies the same protocol to Llama-3.2-1B, Llama-3.1-8B, and Mistral-7B-v0.3 by changing –model. The submitted compact view docs/data_db/paper/table2_null_ short.csv records the seed-specific differences and an explicitly named sample-standard-deviation field. The verifier recomputes that field from the three displayed seeds. Sample standard deviations use the n−1 convention throughout the manuscript. The additional nine-seed Llama-3.2-1B comparison uses scripts/reproduction/run_extended_short_ context.sh and evaluates the full-head Hadamard and pairwise + Hadamard configurations for seeds 0– 8. The angle-objective comparison uses scripts/reproduction/run_angle_objectives.sh. Comparisons that use references produced under a different source state are identified explicitly as such in Table 9 and are not used for equivalence conclusions.

A.4 Long-context evaluation protocol Long-context perplexity is evaluated with the concatenated-stream protocol on Proof-Pile and PG19. The compared transformations are full-head Hadamard, pairwise-only, and pairwise + Hadamard. Within each model, seed, corpus, and context length, all three configurations score identical token sequences, permitting paired perplexity comparisons. The standard context-length set is 2048, 4096, 8192, 16384, 32768, and 65536 tokens. Llama-3.2-3B and Llama-3.2-1B are additionally evaluated at 131072 tokens. Native RoPE is used without YaRN or position interpolation. Mistral results beyond its documented 32K context window are labelled exploratory. SEEDS="0 1 2" \ LENGTHS=2048,4096,8192,16384,32768,65536 \ DATASETS=proof_pile,pg19 \ CONFIGURATIONS="hadamard pairwise_only pairwise_plus_hadamard" \

17

MODEL=/path/to/meta-Llama-3.2-3B \ bash scripts/reproduction/run_long_context.sh The portable driver prints commands unless EXECUTE=1; no workload-manager commands are embedded. It invokes the following evaluation command for each specified configuration: python dartquant_v2/run_quantize.py \ --model /path/to/meta-Llama-3.2-3B \ --loss whip --quantizer_type int4 --cal_dataset wikitext2 \ --w_bits 4 --a_bits 4 --k_bits 4 --v_bits 4 --seed $SEED \ --ppl_eval --ppl_eval_dataset wikitext2 \ --eval_long_context \ --long_context_datasets proof_pile \ --long_context_lengths 2048,4096,8192,16384,32768,65536 \ --long_context_stride 256 --long_context_max_docs 64 \ --long_context_output <out_dir>/long_context_eval.json \ <configuration flags> The configuration-specific flags are empty for full-head Hadamard; –equalising_r3 –equalising_r3_no_hadamard –equalising_r3_objective variance for pairwise-only; and –equalising_r3 –equalising_r3_objective variance for pairwise + Hadamard. For memory efficiency, the language-model head is applied to hidden states in blocks of 1024 tokens. Each requested context length is evaluated independently; if a context length exceeds available memory, the output records an explicit error for that length and evaluation proceeds for the remaining lengths. The reported Llama-3.2-3B and Llama-3.2-1B 128K evaluations fit on a 40 GB A100. Llama-3.1-8B and Mistral-7B-v0.3 are reported through 64K. The concatenated-stream evaluator scores the first min(24, ⌊N/L⌋) non-overlapping windows from the tokenised 64-document stream at each context length L. Both compared configurations therefore score byteidentical tokens at a given length. The scored token set changes with context length: Proof-Pile contributes 24/24/24/14/7/3/1 windows at 2K–128K, whereas PG19 contributes 24 windows at every reported length through 128K (a prefix growing from 49K to 3.15M tokens). Consequently, absolute perplexities should not be compared across context lengths; the analysis uses within-length paired differences.

A.5 Pooled-covariance surrogate optimum: evaluation and verification The position-averaged configurations add –equalising_r3_position_averaged and specify the averaging length with –equalising_r3_position_length L. In long-context evaluations, L is set to the context length being scored (2048, 32768, or 131072 in the reported comparisons), so no evaluation uses an angle averaged for a different deployment length. The branch-matched controls phi_hat_canon_only and phi_hat_canon_layered apply the same branch reduction to ϕ̂k . Comparisons between ϕ∗k and these controls therefore isolate the finite-position-averaging correction rather than the branch convention. RoPE frequencies are obtained from the instantiated checkpoint rather than reconstructed from a base parameter. At runtime, the implementation searches the rotary-embedding modules for a materialised inv_freq buffer of length dh /2. Execution terminates with an explicit error if no compatible buffer is found or if more than one distinct compatible buffer is present. This procedure includes checkpoint-specific frequency scaling, including the Llama-3 rope_type rule. Each position-averaged evaluation records the applied angles, per-pair covariances reconstructed after raw-moment aggregation, and deployed frequency vector. A separate verification procedure iterates over the registered configuration list rather than a filesystem glob, recomputes the pooled-covariance, positionaveraged surrogate objective from these stored quantities, and compares the achieved per-pair minimax variance with the analytic minimum λ̄k of Theorem 3. It writes a per-pair CSV and a summary carrying phi_star_verified, the worst-case excess, the tolerance, and the frequency source. Missing required verification artifacts cause the procedure to terminate with an error; configurations that do not use ϕ∗k are excluded by construction.

18

All 30 position-averaged verification records satisfy the surrogate-optimality criterion. The largest excess over the analytic minimum is 6.90 × 10−7 , compared with a tolerance of 5 × 10−5 , and every record identifies the live rotary module as the frequency source. Completeness of the evaluation set is determined by the same predicate the evaluation harness uses to decide whether a configuration must still be evaluated, so the criterion is identical for the harness and for the verification procedure. The registered primary evaluation set contains 81 manifest-defined configurations, all of which produced valid completion artifacts. A further 14 supplementary configurations (nine long-context controls and five component evaluations) also completed successfully. Results obtained in an earlier evaluation that reconstructed frequencies from the base parameter are excluded. That construction omitted checkpoint-specific RoPE scaling and therefore implemented a different transformation. No reported value is derived from that evaluation.

A.6 Static-quantisation control The static per-channel control is enabled with DARTQUANT_STATIC_ACT=1, DARTQUANT_STATIC_CALIB=64, and DARTQUANT_STATIC_MATCH_EVAL_LEN=1. Under this condition, ActQuantizer.find_params uses one scale per channel, estimated across the token dimension. The calibration-derived scales are then held fixed during evaluation. For each corpus–context-length combination, the static state is reset and the scales are re-estimated from up to 64 non-overlapping windows of the target length drawn from the evaluation corpus. Calibration and scoring text therefore overlap. This protocol also changes the quantiser geometry from the default grouped per-token rule to a static per-channel rule. The comparison is consequently interpreted as a quantiser-regime contrast rather than as an identification of a causal mechanism. SEEDS="0 1 2 3 4 5" \ DARTQUANT_STATIC_MATCH_EVAL_LEN=1 \ DARTQUANT_STATIC_ACT=1 DARTQUANT_STATIC_CALIB=64 \ DARTQUANT_STATIC_SCOPE=both \ LENGTHS=2048,4096,8192,16384,32768,65536 \ DATASETS=proof_pile \ CONFIGURATIONS="hadamard pairwise_only" \ MODEL=/path/to/meta-Llama-3.2-3B \ bash scripts/reproduction/run_long_context.sh This expands to the same run_quantize.py invocation as §A.4, with the environment variables exported and the two configuration flag sets given there. The environment variable DARTQUANT_STATIC_SCOPE selects which quantiser uses calibration-derived fixed scales: both holds the activation and KV-cache scales fixed; act holds the activation scales fixed while the KV-cache scales remain per-token dynamic; and kv holds the KV-cache scales fixed while the activation scales remain dynamic. The both setting reproduces the original joint static modification. The runtime modification is inactive unless the corresponding environment variable is set and does not alter the upstream DartQuant source files.

A.7 Mixing-support interpolation The mixing-support evaluation uses pair-interleaved block-Hadamard transformations with block sizes b ∈ {2, 4, 8, 16, 32, 64, 128}. The primary comparison includes three paired perplexity seeds and three Kdiagnostic seeds for each block size. Three reference configurations are evaluated at the same source state for each seed, and single-seed channel-layout and orientation controls are reported descriptively. Only b = 2 belongs to the RoPE-commuting family. Larger block sizes deliberately relax commutativity to vary the number of channels over which a peak can be redistributed. Each evaluation records WikiText-2 perplexity, mean and maximum K range after R3 , the quantiser step diagnostic, and relative K quantisation error. The complete command set is generated without an execution-system dependency by scripts/ reproduction/generate_mixing_support_manifest.py:

19

python scripts/reproduction/generate_mixing_support_manifest.py \ --output mixing_support_manifest.csv The manifest contains exactly 33 unique cells: seven block sizes at three seeds, three reference transformations at three seeds, and three single-seed layout/orientation controls. Each row records the model, seed, layout, orientation, and complete method arguments and may be executed using the same run_quantize.py prefix as §A.4. The submitted compact aggregate and its paper values are verified entirely from files in the companion code artifact with python scripts/verify_artifact.py The verifier requires all three seeds for the primary interpolation and the corresponding references, checks the published intervals and both support endpoints, and verifies the reported monotone point-estimate trend. The single-seed layout and orientation controls are excluded from the primary intervals. Table 28 is generated directly from docs/data_db/clean/mixing_support.csv.

A.8 Implementation constraints and reproducibility notes • Model-path requirement. The DartQuant evaluation path selects model-specific handling using the substring meta in –model. A local checkpoint path must therefore contain meta (for example, .../meta-Llama-3.2-3B) for the intended evaluation branch to be used. • Behaviour of the released R2 calibration defaults. The released DartQuant R2 calibrator (Shao et al., 2025) (calibrater/r2_base_qr.py, as of the public release used here; later revisions may differ) defaults to –nsamples 128 –bsz 128 –accumulation_steps 2. The complete calibration set therefore forms one batch per epoch, and the condition (batch_idx+1) % accumulation_steps is never satisfied. No optimiser update is applied, and R2 remains at its random-Hadamard initialisation. The present implementation reproduces this behaviour intentionally (the R2 batch size is set to the dataset size, with a comment recording why) so that the experimental substrate matches the released method. The R2 loss trace is therefore expected to remain constant, and the saved R2 equals its initial value. Because R2 is identical across transformations within a seed, it cancels in all paired comparisons. • Empty per-document sliding-window evaluations. Under the per-document sliding-window protocol (concat=False), a requested window longer than every document yields no scored window, and the average NLL is vacuous. Earlier code reported the value ppl = 1.0 in this case, which produced a spurious GovReport (Huang et al., 2021) ≥ 32K crossover. The current output records n_docs_scored and n_short_skipped, and the reported long-context results use the concatenated-stream protocol, which always supplies scored tokens at the requested lengths. The earlier GovReport values at 32K and above obtained from the empty-window case are excluded. • Concatenated-stream window cap. At each context length L, the evaluator scores the first min(24, ⌊N/L⌋) non-overlapping windows from the 64-document token stream. This cap limits evaluation time and is fixed by the implementation. It does not affect within-length pairing, because compared transformations score identical tokens, but it changes the scored token set across lengths (§A.4) and therefore precludes interpreting absolute perplexity as a length trend, as disclosed in §7. • 128K memory requirement. The 131072-token evaluation is the most memory-intensive condition. Llama-3.2-3B and Llama-3.2-1B fit on a 40 GB A100 when the language-model head is applied in 1024token blocks; no 80 GB card is required. The principal memory requirement arises from the forward pass and KV cache rather than from the language-model head. If a requested length exceeds available memory, the result file records an error for that length while preserving results for the remaining lengths. The Llama-3.1-8B and Mistral-7B-v0.3 evaluations are reported through 64K. • Non-additivity of the fixed-scale conditions. The joint activation-and-KV fixed-scale result is not the sum of the activation-only and KV-only results. At 2K, the activation-only difference is +1.73 PPL and the KV-only difference is −1.02 PPL (sum +0.71), whereas the joint difference is −4.69 PPL 20

in the preliminary unmatched three-seed analysis (App. B, §D). This non-additivity is descriptive. Because the protocol is transductive and changes quantiser geometry, it does not isolate an interaction mechanism or attribute the result to either stage. The preliminary 2K joint result also has substantial seed variability (sample standard deviation 3.67 at n=3); the matched six-seed estimate in Table 18 (−5.60 PPL at 2K) is the reported reference.

A.9 Artifact and source-state provenance The companion code artifact is a compact release rather than a copy of the execution environment. It contains the implementation, portable command drivers and manifest generators, final compact summaries, selected seed-level tables, a selective paper-number verifier, and content digests. It does not contain raw-output aggregation utilities and does not claim to rebuild every table from raw logs. It excludes machine names, scheduler identifiers, private development revisions, model weights, evaluation corpora, and superseded execution notes. Consequently, the supplied tables can be inspected and the documented configurations can be re-executed, but the ZIP alone does not reconstruct the original scheduler history. A.9.1 Position-averaged and component evaluations. The evaluated set comprises 81 primary configurations, 23 repeated evaluations, nine additional long-context controls, and five component or precheck configurations. The companion package represents these experiments through the configuration generators and portable drivers. Their relation to the reported tables is documented in REPRODUCIBILITY.md. • Configuration generation. scripts/reproduction/generate_primary_configuration_manifest. py emits the 81 primary rows without invoking a scheduler. It records the model, configuration, seed, corpus, context-length, and method arguments used by the evaluation harness. • Scheduler-neutral commands. scripts/reproduction/run_extended_short_context.sh records the additional short-context and position-averaged configurations, while scripts/reproduction/run_ angle_objectives.sh records the angle-objective comparison, and scripts/reproduction/run_long_ context.sh records the context-length and fixed-scale commands. These scripts print commands by default and require an explicit execution setting before launching experiments. • Verification. Position-averaged runs request the per-band objective dump, which records the achieved objective, the analytic minimum, the numerical tolerance, and the checkpoint frequency source. • Compact data. The compact CSVs under docs/data_db/paper/ preserve the fields documented for each mapped table in REPRODUCIBILITY.md. Coverage is table-specific: some files contain displayed per-seed values, while others contain only final summaries. A.9.2 Angle-estimator evaluation. The angle-estimator analysis (App. B, §L) contains 77 Llama-3.2-3B configurations and 103 cross-model configurations; all listed processes terminated successfully. The companion package contains compact final views for this analysis; it does not include the raw-output aggregator. Three details govern their interpretation. • Calibration-row budget. The activation-bank row budget is derived from an explicit byte budget rather than from available host memory, because the realised row count affects the subsampling used to derive R1 and R2 . Each evaluated configuration validates and records the model-specific row counts, ensuring that a change in the resulting rotations cannot occur without being recorded. • Combination of forward and reverse work lists. The cross-model manifest is the union of forward and reverse evaluations over the same fixed list, producing 103 unique configurations with no conflicting duplicates. When a configuration appears in both sets, the result fields are identical and the longer execution record is retained. One result reuses a previously validated evaluation produced under the same source state, byte budget, and realised row counts; the aggregation script records this reuse explicitly. • Source-state separation. The retained execution records distinguish the revision captured when a work list was emitted from the source state that executed it. Those development identifiers are omitted from the compact package because they are not required to use the released source. The release instead fixes the supplied implementation by file-level SHA-256 digests.

21

A.9.3 Package integrity. The archive-generation procedure excludes Git metadata, local outputs, scheduler-specific development launchers, and private denylist material. It writes MANIFEST.sha256 over the included files. The included archive builder can additionally emit an optional sidecar SHA-256 digest next to the ZIP. The preparation scanner and its release-specific denylist are not included in the companion package. These digests verify the supplied contents; they do not claim identity with private scheduler records that are intentionally absent. A.9.4 Scope limitation. The companion package does not contain the raw scheduler logs or superseded status files. In the retained execution archive, twelve unsuccessful status records coexist with later schemavalid outputs; all affected configurations were subsequently evaluated successfully, and all 81 registered primary configurations are complete under the valid-output criterion. The private records are retained outside the distributed ZIP and can be consulted separately if provenance verification requires them.

B

Complete experimental data

This appendix reports the complete numerical results supporting the main text. Sections §A–§H present the detailed Llama-3.2-3B evaluations; §I presents the cross-model results for Llama-3.2-1B, Llama-3.1-8B, and Mistral-7B-v0.3; and §J–§M report the position-averaged, component, estimator, and mixing-support analyses. App. A specifies the corresponding implementation and evaluation protocols. Throughout this appendix, paired differences are computed within seed against the full-head Hadamard configuration unless another comparator is stated explicitly.

Common environment and experimental configuration • Hardware and software. The primary Llama-3.2-3B evaluations use NVIDIA A100 nodes with 40 or 80 GB of memory. The cross-model evaluations and the separate six-seed Llama-3.2-3B evaluation use H100/H200-class nodes with A100 fallback. Both environments use torch 2.6.0+cu124 and transformers 4.49.0. The A100 environment verifies fast-hadamard-transform 1.1.0; the second environment uses a source build whose version tag is not confirmed. Each result directory records the source revision, software versions, and GPU model in env.json. • Quantisation. Unless otherwise stated, all evaluations use W4A4KV4 INT4 (–quantizer_type int4 –w_bits 4 –a_bits 4 –k_bits 4 –v_bits 4), GPTQ weight quantisation, and DartQuant asymmetric per-token dynamic quantisation for activations and the KV cache. §D reports the static perchannel control. • Calibration. The standard calibration set contains 128 WikiText-2 sequences of length 2048. The closed-form R3 angle is estimated once from this calibration pass (§4). • Statistical reporting. Seed-level variability is reported with the sample (n−1) standard deviation. Population standard deviations emitted by the evaluation procedure are converted by multiplying by p n/(n − 1). Unless stated otherwise, confidence intervals are 90% paired t-intervals computed within seed. The twelve position-averaged long-context contrasts in §J constitute one comparison family; their two-sided p-values use Holm adjustment. The estimator comparisons in §L are exploratory and report unadjusted 90% paired intervals. Equivalence conclusions are stated relative to the selected margin and do not imply a task-independent minimum important difference. • Transformations. The reader-facing names are full-head Hadamard, pairwise-only, and pairwise + Hadamard. Their literal code identifiers are hadamard (no R3 flags), eq_only (–equalising_r3 –equalising_r3_no_hadamard –equalising_r3_objective variance), and eq_layered ≡ eq_r3 (–equalising_r3 –equalising_r3_objective variance; the closed form composed before Hadamard), respectively. The variance objective is selected explicitly in all corresponding evaluations, because the flag otherwise defaults to linf. • Long-context evaluation. Long-context perplexity uses the concatenated-stream protocol described in App. A, with min(24, ⌊N/L⌋) non-overlapping windows at each context length (the window cap

22

is fixed by the implementation, not a tunable flag). Proof-Pile and PG19 are evaluated with native RoPE and no YaRN. Mistral-7B-v0.3 results beyond its documented 32K context window are labelled exploratory. • Models. The evaluated checkpoints are Llama-3.2-1B (dh =64), Llama-3.2-3B (dh =128, the primary checkpoint), Llama-3.1-8B (dh =128), and Mistral-7B-v0.3 (dh =128, a second pretraining family). The local Mistral path contains the substring meta, as required by the DartQuant evaluation branch. • Evaluation coverage. The default-estimator comparisons are complete for all reported checkpoints and contexts. The position-averaged surrogate optimum ϕ∗k (§J) is evaluated only on Llama-3.2-3B and uses the checkpoint’s deployed scaled RoPE frequencies. The pooling analysis (§L) evaluates the default Q/K-weighted and K-only estimators on all four checkpoints, the stream-balanced estimator on Llama-3.2-3B, and ϕ∗k pooling variants on Llama-3.2-3B; non-default pooling is evaluated only at short context, so no long-context claim is made about it. The L∞ and high-quantile objectives are likewise restricted to 2K short-context evaluation. The matched static-scale comparison is reported through 64K (§H). LongBench-v2 results are reported only for the aggregated Llama-3.2-3B evaluation of §G: the corresponding cross-model evaluations were run at three seeds and produced the same nondiscriminative value, but were never aggregated, so no cross-model LongBench-v2 value is reported and no conclusion is drawn from it. • Position-averaged configurations. The code identifiers eq_only_star and eq_layered_star denote pairwise-only and pairwise + Hadamard with ϕ∗k , frequencies read from the checkpoint’s live rotary module. phi_hat_canon_only and phi_hat_canon_layered denote the branch-matched ϕ̂k controls, which apply the same branch reduction so that the difference between ϕ∗k and the control isolates the position-averaging correction.

A. Intra-band calibration statistics A descriptive calibration analysis retains the attention-head dimension and estimates one covariance matrix for every (layer, head, frequency-pair) combination. For each stream, Table 6 reports the mean absolute correlation |ρ|, the proportion of combinations with |ρ| > 0.05, and the mean absolute log variance ratio | log(σ12 /σ22 )|. This analysis is distinct from the deployed angle estimator (App. A, §A.2.1), which aggregates raw moments across attention-head observation rows before constructing one covariance matrix and one angle for each (layer, frequency-pair) combination. Results are reported for all four checkpoints. model

stream

Llama-3.2-1B

Q K Q K Q K Q K

Llama-3.2-3B Llama-3.1-8B Mistral-7B-v0.3

|ρ|

frac |ρ|>0.05

| log var-ratio|

0.165 0.169 0.141 0.159 0.135 0.154 0.148 0.175

0.780 0.769 0.734 0.748 0.723 0.738 0.735 0.755

0.466 0.577 0.402 0.534 0.367 0.504 0.356 0.479

Table 6: Intra-band calibration statistics by checkpoint and stream. The table reports the mean absolute correlation, the proportion of (layer, head, frequency-pair) combinations with |ρ| > 0.05, and the mean absolute log variance ratio. These statistics describe the head-resolved calibration distribution and are distinct from the concatenated-row angle estimator used in the evaluated transformations.

B. Short-context paired comparisons Table 7 reports the three-seed Llama-3.2-3B comparison between pairwise + Hadamard and full-head Hadamard on WikiText-2 under dynamic W4A4KV4. The paired difference is ∆PPL = PPL(pairwise +

23

Hadamard) − PPL(full-head Hadamard). The corresponding Llama-3.2-1B, Llama-3.1-8B, and Mistral-7Bv0.3 results appear in §I. mean PPL

paired PPL difference

full-head Hadamard

pairwise + Hadamard

by seed 0/1/2

mean ± sample sd

10.0823

10.0864

−0.0078/+0.0034/+0.0168

+0.0041 ± 0.0123

Table 7: Llama-3.2-3B short-context paired comparison on WikiText-2 under dynamic W4A4KV4 with whip calibration, n = 3; ± is sample sd. The difference is pairwise + Hadamard minus full-head Hadamard; positive values indicate higher perplexity for pairwise + Hadamard. The 90% confidence interval lies within the selected ±0.05-PPL interval criterion.

Additional seed counts. An independent six-seed evaluation of ϕ̂k , conducted in the cross-model software environment and paired within that environment, yields a corrected 90% confidence interval of [−0.005, +0.023] PPL under whip calibration. These values are not pooled with the three-seed results because the software environments differ. An additional nine-seed Llama-3.2-1B evaluation yields a mean paired difference of +0.0002 PPL, sample standard deviation 0.0283, and 90% confidence interval [−0.0173, +0.0178]. comparison

n

mean difference

sample sd

90% CI

Llama-3.2-1B, pairwise + Hadamard with ϕ̂k

9

+0.0002

0.0283

[−0.0173, +0.0178]

Table 8: Additional nine-seed short-context interval-criterion evaluation for Llama-3.2-1B on WikiText-2 under dynamic W4A4KV4 with whip calibration. The paired difference is pairwise + Hadamard minus fullhead Hadamard. The 90% confidence interval lies within the selected ±0.05-PPL margin under the TOST interval formulation.

Angle-objective comparison. Table 9 compares the variance, L∞ , and high-quantile angle objectives. Comparisons labelled matched use configurations evaluated from the same source revision and execution state. Comparisons labelled different use a previously recorded full-head Hadamard or pairwise + Hadamard reference; a validation evaluation measured a 0.110-PPL difference attributable to source-state drift, so different-source-state comparisons are interpreted directionally and are not used to establish equivalence. configuration

objective

comparator

pairwise-only pairwise-only pairwise-only pairwise-only pairwise-only pairwise + Hadamard pairwise + Hadamard pairwise + Hadamard pairwise + Hadamard

variance L∞ L∞ high-quantile high-quantile L∞ L∞ high-quantile high-quantile

full-head Hadamard full-head Hadamard pairwise-only, variance full-head Hadamard pairwise-only, variance full-head Hadamard pairwise + Hadamard, variance full-head Hadamard pairwise + Hadamard, variance

source state different different matched different matched different different different different

mean difference

sample sd

90% CI

+0.4945 +0.4108 −0.0837 +0.4264 −0.0681 +0.0110 +0.0005 −0.0066 −0.0172

0.0476 0.0257 0.0717 0.0344 0.0600 0.0225 0.0058 0.0232 0.0032

[+0.4143, +0.5747] [+0.3676, +0.4541] [−0.2045, +0.0372] [+0.3684, +0.4845] [−0.1692, +0.0331] [−0.0269, +0.0489] [−0.0093, +0.0102] [−0.0457, +0.0324] [−0.0226, −0.0118]

Table 9: Llama-3.2-3B short-context comparison of angle objectives under dynamic W4A4KV4 with whip calibration, n = 3. Matched-source-state rows compare pairwise-only objectives within the same execution state. Rows using a different-source-state reference are reported as directional checks and are not used for equivalence inference.

Equivalence-margin sensitivity. Table 10 evaluates the criterion-relative conclusion at margins of ±0.02, ±0.05, and ±0.10 PPL. Equivalence at margin m is established when the complete sample-sd 90% paired confidence interval lies within [−m, +m]. The manuscript uses m = 0.05 PPL as its primary reporting 24

criterion. This value is not presented as an externally validated minimum important difference; conclusions are conditional on the selected margin, and the additional columns show their sensitivity to alternative thresholds. model

mean ∆

90% CI

±0.02

±0.05

±0.10

Llama-3.2-1B (n = 9) Llama-3.2-3B Llama-3.1-8B Mistral-7B-v0.3

+0.0002 +0.0041 +0.0063 −0.0012

[−0.017, +0.018] [−0.017, +0.025] [−0.002, +0.015] [−0.006, +0.004]

equivalent not established equivalent equivalent

equivalent equivalent equivalent equivalent

equivalent equivalent equivalent equivalent

Table 10: Sensitivity of the TOST conclusion for pairwise + Hadamard versus full-head Hadamard under whip calibration. The Llama-3.2-1B row uses the additional n = 9 evaluation; the other rows use n = 3. Equivalence at margin m is established when the full 90% confidence interval lies within [−m, +m]. All four checkpoints satisfy the selected ±0.05-PPL criterion; the table shows that this conclusion is margindependent for Llama-3.2-3B.

C. Long-context evaluation under dynamic quantisation Table 11 reports concatenated-stream perplexity for Llama-3.2-3B on Proof-Pile and PG19 under dynamic per-token W4A4KV4, with three paired seeds. The compared transformations are full-head Hadamard, pairwise-only, and pairwise + Hadamard. Results are complete from 2K through 128K on both corpora; the 128K evaluations fit on the 40 GB A100 environment, so no 80 GB hardware is needed. Within every corpus and context length, the compared transformations score identical tokens. mean PPL L

full-head Hadamard

PPL difference vs Hadamard

pairwise-only

pairwise + Hadamard

pairwise-only

pairwise + Hadamard

Proof-Pile 2K 4K 8K 16K 32K 64K 128K

4.536±.082 3.516±.052 3.606±.047 3.411±.039 3.290±.037 3.233±.028 2.933±.021

4.696±.095 3.659±.062 3.770±.065 3.582±.047 3.481±.046 3.439±.036 3.126±.032

4.536±.073 3.514±.044 3.605±.046 3.414±.036 3.294±.036 3.234±.027 2.937±.020

+0.161±.018 +0.143±.014 +0.164±.020 +0.171±.009 +0.191±.010 +0.207±.011 +0.193±.016

+0.000±.015 −0.002±.008 −0.001±.005 +0.003±.004 +0.004±.002 +0.001±.001 +0.004±.002

PG19 2K 4K 8K 16K 32K 64K 128K

10.953±.289 15.986±.414 17.148±.442 16.527±.407 14.735±.352 13.054±.272 12.484±.324

11.518±.324 16.964±.491 18.213±.476 17.726±.448 15.888±.405 14.169±.326 13.653±.410

10.937±.282 15.974±.399 17.163±.424 16.560±.426 14.751±.380 13.059±.290 12.488±.341

+0.566±.060 +0.979±.080 +1.065±.035 +1.199±.042 +1.153±.054 +1.115±.056 +1.169±.085

−0.015±.041 −0.012±.016 +0.015±.019 +0.033±.026 +0.016±.029 +0.005±.018 +0.004±.016

Table 11: Llama-3.2-3B concatenated-stream perplexity on Proof-Pile and PG19 under dynamic W4A4KV4 with whip calibration, n = 3; ± is sample sd. Differences are computed relative to the same-seed fullhead Hadamard configuration; positive values indicate higher perplexity. Absolute perplexity should not be compared across context lengths because the scored token set changes with length; exact per-length window counts are reported in Appendix A, §A.4. Pairwise-only yields higher perplexity at every reported evaluation point through 128K.

Frequency source for the position-averaged configurations. Only position-averaged evaluations that obtain RoPE frequencies from the checkpoint’s live rotary module are included. Reconstructing frequencies 25

from the base parameter omits checkpoint-specific scaling and defines a different transformation; values obtained from that construction are excluded from all reported results, and every ϕ∗k value reported here comes from the deployed-frequency evaluations of §J.

D. Static per-channel control and fixed-scale component comparison The preliminary three-seed analysis in Table 12 uses a static per-channel quantiser in which activation and KV scales are estimated from 64 calibration forwards and retained during evaluation (DARTQUANT_STATIC_SCOPE ∈ {both, act, kv}), on the same Llama-3.2-3B / Proof-Pile concatenated-stream setting as §C. Calibration length is not matched to evaluation length in this analysis. The matched six-seed comparison in Table 18 removes that length mismatch, but it continues to estimate scales from windows that are subsequently scored and changes the scale geometry from grouped per-token to per-channel quantisation. The static comparisons are consequently treated as descriptive contrasts between evaluation protocols. They do not identify the process responsible for the dynamic-quantisation result or quantify out-of-sample performance under static quantisation. mean PPL L

full-head Hadamard

pairwise-only

PPL difference vs Hadamard pairwise + Hadamard

joint activation-and-KV fixed-scale configuration 2K 22.430±3.47 17.738±1.21 4K 14.922±1.37 12.657±0.32 8K 14.808±1.30 13.129±0.15 16K 12.791±1.15 12.305±0.45 32K 11.714±1.19 12.960±0.73 64K 11.413±1.18 15.179±2.46

21.077±2.65 14.681±1.91 15.105±1.71 13.547±1.42 12.324±1.15 11.681±0.93

pairwise-only

pairwise + Hadamard

−4.692±3.67 −2.265±1.41 −1.679±1.38 −0.485±1.47 +1.246±1.74 +3.766±2.79

−1.353±.87 −0.241±.58 +0.297±.42 +0.757±.32 +0.610±.05 +0.268±.42

Table 12: Preliminary unmatched static per-channel comparison on Llama-3.2-3B with whip calibration, n = 3; ± is sample sd. Activation and KV scales are estimated during calibration and held fixed during evaluation. Calibration length is confounded with evaluation length; the apparent 64K reversal is therefore not used for inference. Table 13 compares activation-only and KV-only fixed-scale conditions under the same preliminary threeseed protocol. Neither condition reproduces the joint activation-and-KV result, and the three differences are not additive. pairwise-only − full-head Hadamard PPL difference fixed-scale condition activation-only KV-only

2K

4K

8K

16K

32K

64K

+1.733±.33 −1.019±.06

+1.292±.28 −0.729±.12

+1.028±.26 −0.795±.12

+0.935±.27 −0.710±.10

+0.964±.27 −0.321±.09

+1.038±.27 +0.278±.12

Table 13: Preliminary comparison of activation-only and KV-only fixed-scale conditions on Llama-3.2-3B with whip calibration, n = 3; ± is sample sd. Values are pairwise-only minus full-head Hadamard. Neither single-component condition reproduces the joint activation-and-KV fixed-scale difference.

E. Quantiser-step and post-quantisation-error diagnostic The quantiser diagnostic instruments ActQuantizer.find_params without changing its numerical output. For each transformation, it records the selected per-channel step ∆c and the relative post-quantisation error ∥y − ŷ∥2 /∥y∥2 for the tensors whose scales are being determined. Each reported condition contains 39,480 quantiser evaluations. Under dynamic per-token quantisation, the values are reported for seed 0 and the call-level observations are dependent; they therefore provide a descriptive readout rather than independent statistical replication. Under the joint static per-channel condition, values are aggregated across three seeds. The static condition 26

increases error for both transformations and substantially reduces the diagnostic advantage of the full-head Hadamard. Table 14 reports the corresponding step-size and error diagnostics for the dynamic and joint static conditions. configuration

∆c

sd ∆c

max ∆c

post-quant rel-MSE

quantiser evaluations

0.01348 0.01634

39,480 39,480

0.02418 0.02391

39,480 39,480

dynamic per-token rule (seed 0) full-head Hadamard 0.2843 0.2188 1.800 pairwise-only 0.3093 0.2774 2.329 joint activation-and-KV fixed-scale rule (3 seeds) full-head Hadamard 0.3393 0.281 1.647 pairwise-only 0.3288 0.265 2.875

Table 14: Quantiser-step and relative-error diagnostic for Llama-3.2-3B with whip calibration. Dynamic values are from seed 0 and call-level observations are dependent; fixed-scale values aggregate three seeds. The diagnostic is consistent with partial absorption of the transformation-dependent range difference by dynamic scale estimation, but it is not an independent statistical test.

F. Lower-bit evaluation Table 15 reports WikiText-2 perplexity for Llama-3.2-3B at W4A3KV4 and W3A3KV3 using three paired seeds. Absolute perplexity is highly variable at these bit widths, although within-seed differences remove part of the shared variation. The models are substantially degraded, so the results are descriptive and do not support a general claim across bit widths (§9). configuration

seed 0

seed 1

seed 2

mean

PPL difference vs Hadamard

W4A3KV4 full-head Hadamard pairwise + Hadamard pairwise-only

85.373 83.947 93.287

56.406 56.776 64.522

55.569 55.133 62.502

65.783 65.286 73.437

n/a −0.497 +7.654

W3A3KV3 full-head Hadamard pairwise + Hadamard pairwise-only

155.760 159.480 205.163

146.080 152.329 191.455

165.384 164.379 213.939

155.741 158.729 203.519

n/a +2.988 +47.778

Table 15: Llama-3.2-3B lower-bit evaluation on WikiText-2 with whip calibration, n = 3. The quantised models are substantially degraded; the paired differences are reported descriptively and do not establish a general bit-width trend.

G. Long-context task evaluations NIAH. Table 16 reports depth-averaged NIAH retrieval accuracy for Llama-3.2-3B under dynamic quantisation. Each value averages five needle depths and 20 probes per depth. Quantised results are averaged over seeds {0, 1, 2}; the FP16 reference uses seed 0. A context length is described as discriminative when the quantised configurations are sufficiently separated from the near-saturation FP16 value to permit an informative comparison; this designation is a descriptive diagnostic rather than a preregistered statistical rule. The depth-averaged values used in this table are available in docs/data_db/paper/appG_niah_3b.csv; individual probe records are not included in the companion code artifact. LongBench-v2. The same Llama-3.2-3B quantised model instances are evaluated on the 384 LongBenchv2 examples within the 64K evaluation window. Answers are scored by option log probability. The random baseline for four-choice questions is 0.25. All reported means are at or below this baseline, so the benchmark 27

L

FP16

full-head Hadamard

pairwise-only

pairwise + Hadamard

4K 8K 16K 32K 64K

0.990 0.980 0.990 1.000 0.990

0.973 0.983 0.957 0.973 0.983

0.967 0.970 0.953 0.923 0.923

0.980 0.977 0.953 0.950 0.933

diagnostic status near FP16 saturation near FP16 saturation near FP16 saturation discriminative discriminative

Table 16: Llama-3.2-3B NIAH depth-averaged retrieval accuracy with whip calibration. The 32K and 64K evaluations provide sufficient separation from the FP16 result to distinguish the quantised configurations; shorter contexts are near saturation and are therefore non-discriminative. does not discriminate among the transformations at this quantisation point. Table 17 reports the complete per-seed results. configuration

overall accuracy

full-head Hadamard − configuration

per-seed accuracy

0.207±.011 0.224±.021 0.239±.049

n/a −0.016±.030 −0.031±.043

.195/.214/.214 .245/.203/.224 .203/.219/.294

full-head Hadamard pairwise-only pairwise + Hadamard

Table 17: Llama-3.2-3B LongBench-v2 accuracy with whip calibration, n = 3; ± is sample sd. Scores are at or below the 0.25 four-choice random baseline; the evaluation is therefore non-discriminative for the compared transformations.

H. Interpretation of the matched static-scale comparison The matched-length six-seed analysis removes the calibration-length mismatch in Table 12. At 64K, the mean pairwise-only-minus-Hadamard difference is −0.18 PPL with sample standard deviation 0.89; this comparison does not satisfy the selected ±0.05-PPL criterion. A matched 128K result is not reported because the concatenated stream contains only one 128K window, which would further increase the overlap between scale estimation and scoring. Such overlap is present at every reported length. For each target length in Table 18, scales are estimated from windows of that length and retained during scoring. The protocol also replaces the deployed grouped dynamic geometry with static per-channel quantisation, and the number of calibration tokens varies with context length. The results are therefore used only to assess sensitivity to quantiser protocol. They are not used as evidence of deployment performance or as identification of a causal mechanism. Table 18: Matched static per-channel comparison on Llama-3.2-3B/Proof-Pile with whip calibration, n = 6; ± is sample sd. Positive differences indicate higher perplexity for pairwise-only. Scales are estimated during target-length calibration and held fixed during evaluation. Calibration overlaps the scored windows, and the per-channel geometry differs from the deployed grouped dynamic quantiser. joint activation-and-KV fixed scale length

full-head Hadamard PPL

pairwise-only PPL

2K 4K 8K 16K 32K 64K

19.76 17.48 15.98 14.63 13.32 12.56

14.16 11.68 13.25 12.36 12.76 12.38

pairwise-only − full-head Hadamard joint

activation-only

KV-only

−5.60±2.69 −5.80±5.28 −2.74±1.26 −2.27±2.11 −0.55±1.59 −0.18±.89

+1.50±.23 +1.26±.26 +1.30±.53 +1.20±.34 +1.12±.24 +1.33±.14

−1.12±.22 −0.91±.14 −1.16±.14 −1.25±.17 −1.14±.18 −0.97±.28

Figure 1 compares the dynamic and matched static pairwise-only differences across context lengths. It visualises sensitivity to quantiser protocol and does not support a direct performance comparison between the two regimes. 28

paired perplexity difference (pairwise-only full-head Hadamard)

Exploratory quantiser-regime comparison 4 2 0 2 4 6 8 10 12

above 0: full-head Hadamard has lower perplexity

joint activation/KV fixed-scale, matched (n = 6) activation-only fixed-scale (n = 6) KV-only fixed-scale (n = 6) dynamic per-token, PG19 (n = 3) dynamic per-token, Proof-Pile (n = 3)

below 0: pairwise-only has lower perplexity

2K

4K

8K

16K 32K context length

64K

128K

Figure 1: Exploratory comparison across quantiser regimes. Dynamic curves use the deployed per-token rule with three paired seeds; fixed-scale curves use matched, transductive per-channel calibration with six paired seeds. Positive values indicate higher perplexity for pairwise-only than for full-head Hadamard. Figure 2 summarises the associated scale-setting diagnostics. The measurements are descriptive because the dynamic values use one seed and the static protocol is transductive.

I. Cross-model evaluation The cross-model evaluation applies the same W4A4KV4 INT4 protocol to Llama-3.2-1B, Llama-3.1-8B, and Mistral-7B-v0.3, using GPTQ weights, dynamic per-token activation/KV quantisation, 128 WikiText-2 calibration sequences of length 2048, and seeds {0, 1, 2}. Table 19 reports the short-context comparison between pairwise + Hadamard and full-head Hadamard. The Llama-3.1-8B and Mistral-7B-v0.3 confidence intervals lie within the selected ±0.05-PPL interval criterion. The original three-seed Llama-3.2-1B result is inconclusive and is superseded for equivalence inference by the independent nine-seed result in Table 8. mean PPL model Llama-3.2-1B Llama-3.1-8B Mistral-7B-v0.3

paired PPL difference

full-head Hadamard

pairwise + Hadamard

mean ± sample sd

by seed 0/1/2

14.4268 7.8251 5.8921

14.4163 7.8314 5.8909

−0.0105 ± 0.0359 +0.0063 ± 0.0049 −0.0012 ± 0.0031

−0.0490/−0.0048/+0.0222 +0.0040/+0.0030/+0.0120 −0.0034/+0.0023/−0.0025

Table 19: Cross-model short-context paired differences on WikiText-2 under dynamic W4A4KV4 with whip calibration, n = 3; ± is sample sd. The difference is pairwise + Hadamard minus full-head Hadamard. The Llama-3.1-8B and Mistral-7B-v0.3 intervals satisfy the selected ±0.05-PPL interval criterion; the three-seed Llama-3.2-1B result is superseded for inference by Table 8. Table 20 reports the long-context pairwise-only difference relative to full-head Hadamard. Every withinlength mean is positive. The Mistral differences are small and are included as cross-model context rather than as individually decisive results. Absolute perplexity and slopes are not compared across lengths.

29

Descriptive scale-setting diagnostics Dynamic per-token quantisation (deployed)

dynamic scale rule Δc re-estimated for each token

post-RoPE R3 comparison full-head Hadamard versus pairwise-only observed channel ranges differ

seed-0 diagnostic Δc : 0.284 vs 0.309 relative error: 0.0135 vs 0.0163

dynamic perplexity pairwise + Hadamard is practically equivalent; pairwise-only yields higher perplexity

Static per-channel quantisation (transductive contrast)

fixed-scale rule calibration-derived Δc held fixed during evaluation

fixed-scale diagnostic larger relative increase for full-head Hadamard 1.79× vs 1.46× relative MSE: 0.0242 vs 0.0239 Δc : 0.339 vs 0.329

fixed-scale comparison 2K pairwise-only − Hadamard difference = −5.60 PPL descriptive regime pattern

The pattern is consistent with multiple explanations; transductive calibration and the change in quantiser geometry preclude mechanism identification.

Figure 2: Descriptive scale-setting diagnostic. The dynamic ∆c values are from seed 0, and individual quantiser evaluations are not independent replicates. The fixed-scale comparison is transductive and changes quantiser geometry. The figure therefore illustrates a quantiser-regime pattern rather than identifying a mechanism. Cross-model NIAH. Depth-averaged NIAH retrieval accuracy is also evaluated on the corresponding quantised model instances (seed 0; accuracy, so higher is better). Llama-3.2-1B provides sufficient separation at long context and follows the perplexity ordering: at 16K/32K/64K, full-head Hadamard obtains 0.68/0.59/0.51, pairwise-only obtains 0.53/0.47/0.36, and pairwise + Hadamard obtains 0.69/0.62/0.54. The FP16 value is approximately 1.0 at 16K. Llama-3.1-8B remains near FP16 saturation at all evaluated lengths, with all transformations between approximately 0.95 and 0.99, and therefore does not distinguish them. This is the saturation counterpart of §G. The complete model-by-length-by-depth results, including Mistral-7Bv0.3, are recorded in docs/data_db/paper/appI_breadth_niah.csv.

J. Pooled-covariance surrogate optimum with deployed RoPE frequencies (Llama-3.2-3B) This section reports the only ϕ∗k evaluations used in the manuscript. All configurations use Llama-3.23B, dynamic W4A4KV4, whip calibration, a single recorded software environment, and one source state. Inverse frequencies are read from the checkpoint’s live rotary module, including llama3 frequency scaling. Reconstructing frequencies from the base parameter would misstate 63 of the 64 frequency pairs, with errors of up to 32× in the lowest-frequency pairs. Verification of the analytic optimum. For every position-averaged configuration, the verification procedure reconstructs the surrogate objective from covariances formed after raw-moment aggregation and from the deployed frequency vector, then compares the achieved per-pair minimax variance with the analytic minimum λ̄k of Theorem 3. All 30 verification records satisfy the criterion. The maximum excess over the analytic minimum, across all layers, frequency pairs, and evaluated configurations, is 6.90 × 10−7 , compared with a tolerance of 5 × 10−5 .

30

pairwise-only − full-head Hadamard PPL difference model / corpus 1B / Proof-Pile 1B / PG19 8B / Proof-Pile 8B / PG19 Mistral / Proof-Pile Mistral / PG19

2K

4K

8K

16K

32K

64K

128K

+0.525 +1.819 +0.057 +0.206 +0.043 +0.113

+0.416 +3.102 +0.056 +0.371 +0.035 +0.159

+0.456 +3.401 +0.072 +0.421 +0.035 +0.196

+0.515 +3.861 +0.075 +0.480 +0.034 +0.223

+0.539 +3.636 +0.076 +0.459 +0.038 +0.189

+0.574 +3.455 +0.074 +0.394 +0.056 +0.238

+0.513 +3.585 n/a n/a n/a n/a

Table 20: Cross-model concatenated-stream perplexity differences under dynamic W4A4KV4 with whip calibration, n = 3. Positive values indicate higher perplexity for pairwise-only than for full-head Hadamard. Every reported within-length mean is positive. Mistral-7B-v0.3 differences are small and are interpreted as cross-model context rather than as individually decisive comparisons. Applying the same verification procedure to alternative transformations yields mean minimax excesses of 0.433 for identity, 0.375 for the per-pair H2 , and 0.0246 for ϕ̂k (raw or branch-matched alike, the branch shift being objective-preserving). The corresponding value for ϕ∗k is 1.26 × 10−8 . The implementation therefore reproduces the analytic ordering. End-to-end perplexity does not follow the same ordering: the positionaveraged surrogate optimum attains the analytic minimum but yields the largest short-context pairwise-only perplexity difference. §6 gives the conditioning reason why the position-averaging correction is largest, and least meaningful, on precisely the frequency pairs where |(Ck , Sk )| ≈ 0. Conditioning of the finite-position correction. At L = 2048, 17 of the 64 deployed frequency pairs satisfy |(Ck , Sk )| < 0.01. For these pairs, | 12 atan2(Sk , Ck )| has mean 0.75 radians and maximum 1.51 radians. At L = 8192, 25 of 64 pairs satisfy the same threshold. When both finite sums are close to zero, the position-averaged covariance is close to isotropic and many angles are nearly optimal for the variance objective. The selected branch can therefore depend strongly on the small finite-sum remainder while the objective remains nearly flat over angle. This qualifies the deployment interpretation of ϕ∗k without affecting the analytic verification. Short-context comparison (2K WikiText-2). Table 21 reports per-seed WikiText-2 perplexity for the full-head Hadamard, identity, per-pair H2 , pairwise-only, and pairwise + Hadamard configurations. Table 22 reports the corresponding paired differences, which back Table 2 of the body. The branch-matched controls are evaluated for seeds 0–2; comparisons involving those controls therefore use the shared three-seed subset rather than the six-seed means. configuration

s0

s1

s2

s3

s4

s5

mean

sd

full-head Hadamard identity (no R3 ) per-pair H2 pairwise-only, ϕ̂k pairwise-only, ϕ∗k pairwise + Hadamard, ϕ̂k pairwise + Hadamard, ϕ∗k

10.3690 10.8305 10.8245 10.8570 10.9367 10.3724 10.3690

9.7748 10.2184 10.2020 10.2054 10.2610 9.7916 9.8161

10.1030 10.5697 10.5264 10.5943 10.6261 10.0952 10.1120

10.5908 11.1342 11.0978 11.1317 11.1793 10.6043 10.6202

9.9134 10.3311 10.3243 10.3851 10.4607 9.9101 9.9188

9.9216 10.3747 10.3443 10.4001 10.4358 9.9309 9.9447

10.1121 10.5764 10.5532 10.5956 10.6499 10.1174 10.1301

0.3114 0.3478 0.3436 0.3437 0.3452 0.3119 0.3083

branch-matched controls (seeds 0–2) branch-matched pairwise-only, ϕ̂k branch-matched pairwise + Hadamard, ϕ̂k

10.9052 10.3747

10.2581 9.7889

10.6349 10.1170

n/a n/a

n/a n/a

n/a n/a

10.5994 10.0935

0.3250 0.2936

Table 21: Per-seed WikiText-2 perplexity for Llama-3.2-3B under dynamic W4A4KV4 with whip calibration. The primary transformations use six seeds; branch-matched controls use seeds 0–2. The table supports the short-context comparison in Table 2.

31

configuration

comparator

n

mean difference

sample sd

identity (no R3 ) per-pair H2 pairwise-only, ϕ̂k pairwise-only, ϕ∗k pairwise + Hadamard, ϕ̂k pairwise + Hadamard, ϕ∗k

full-head Hadamard full-head Hadamard full-head Hadamard full-head Hadamard full-head Hadamard full-head Hadamard

6 6 6 6 6 6

+0.4643 +0.4411 +0.4835 +0.5378 +0.0053 +0.0180

0.0424 0.0355 0.0356 0.0374 0.0096 0.0159

[+0.4294, +0.4992] [+0.4119, +0.4703] [+0.4542, +0.5128] [+0.5071, +0.5686] [−0.0026, +0.0132] [+0.0050, +0.0311]

branch-matched pairwise-only, ϕ̂k pairwise-only, ϕ∗k branch-matched pairwise + Hadamard, ϕ̂k pairwise + Hadamard, ϕ∗k

pairwise-only, ϕ̂k branch-matched pairwise-only, ϕ̂k pairwise + Hadamard, ϕ̂k branch-matched pairwise + Hadamard, ϕ̂k

3 3 3 3

+0.0472 +0.0085 +0.0071 +0.0055

0.0061 0.0207 0.0129 0.0188

[+0.0369, +0.0575] [−0.0264, +0.0435] [−0.0147, +0.0290] [−0.0262, +0.0372]

90% CI

Table 22: Paired short-context perplexity differences for the position-averaged analysis. Standard deviations use the sample convention, and intervals are 90% paired t-intervals. Both pairwise + Hadamard comparisons against full-head Hadamard lie within the selected ±0.05-PPL interval criterion. Three of the four branchcontrol comparisons also satisfy this criterion; the interval for branch-matched minus raw pairwise-only ϕ̂k extends to +0.0575 PPL. Long-context comparison. Table 23 reports mean concatenated-stream perplexity at 2K, 32K, and 128K. Each ϕ∗k configuration uses the averaging length at which it is evaluated. Independent evaluations of the full-head Hadamard, pairwise-only ϕ̂k , and pairwise + Hadamard ϕ̂k configurations agree with the corresponding Table 11 values to all printed digits across all 18 model-seed-corpus-length combinations. The two result sets are therefore directly comparable at the reported precision. Table 24 reports the paired ϕ∗k comparisons and the Holm adjustment of their two-sided p-values. pairwise-only L

full-head Hadamard

pairwise + Hadamard

ϕ̂k

branch-matched ϕ̂k

ϕ∗k

ϕ̂k

branch-matched ϕ̂k

ϕ∗k

Proof-Pile 2K 32K 128K

4.5357 3.2902 2.9331

4.6963 3.4812 3.1257

4.7358 3.4887 3.1207

4.7342 3.4931 3.1093

4.5358 3.2938 2.9370

4.5347 3.2972 2.9356

4.5381 3.2944 2.9388

PG19 2K 32K 128K

10.9528 14.7352 12.4838

11.5183 15.8883 13.6527

11.5813 15.9415 13.5616

11.5683 15.8977 13.5370

10.9373 14.7511 12.4875

10.9760 14.7505 12.4933

10.9546 14.7574 12.4912

Table 23: Mean concatenated-stream perplexity for Llama-3.2-3B under dynamic W4A4KV4 with whip calibration, n = 3. Position-averaged surrogate configurations use the context length at which they are evaluated. Absolute perplexity should not be compared across lengths. These values support the longcontext comparisons in §7.

K. Local FPTQuant-style Q/K component (Llama-3.2-3B) Table 25 reports a local FPTQuant-style (van Breugel et al., 2025) per-pair Q/K component implemented within the dynamic W4A4KV4 evaluation procedure used in this paper. All compared components use 128 optimisation steps, learning rate 0.02, and 128 calibration sequences divided into 112 optimisation and 16 held-out sequences. Q and K use the same fake-quantisation settings. The table reports the objective at the first and final optimisation steps together with held-out final-token logit MSE. Scope. This component is an independent reimplementation of an FPTQuant-style local map and is not a reproduction of FPTQuant. It therefore does not bear on the results reported for that method. A single-seed comparison at 32, 128, and 512 optimisation steps yields perplexities 10.9755/11.0653/12.0509. Repeating the 128-step condition at the same seed changes both the optimisation trace and perplexity by more than 0.12 PPL, which exceeds the observed 32 → 128 step difference. The deterministic portions of the procedure reproduce exactly, localising this variability to the component optimisation. For this reason, the manuscript 32

90% CI

p

0.0177 0.0133 0.0083 0.0154 0.0342 0.0088

[−0.0315, +0.0283] [−0.0180, +0.0268] [−0.0253, +0.0025] [−0.0389, +0.0129] [−0.1015, +0.0139] [−0.0395, −0.0097]

.8906 .6225 .1395 .2805 .1568 .0404

pairwise + Hadamard: ϕ∗k − branch-matched ϕ̂k Proof-Pile 2K +0.0034 0.0153 Proof-Pile 32K −0.0028 0.0036 Proof-Pile 128K +0.0032 0.0028 PG19 2K −0.0214 0.0043 PG19 32K +0.0070 0.0066 PG19 128K −0.0021 0.0050

[−0.0223, +0.0292] [−0.0089, +0.0033] [−0.0016, +0.0080] [−0.0286, −0.0141] [−0.0042, +0.0181] [−0.0106, +0.0063]

.7335 .3095 .1888 .0132 .2102 .5387

corpus

L

mean difference

sample sd

pairwise-only: ϕ∗k − branch-matched ϕ̂k Proof-Pile 2K −0.0016 Proof-Pile 32K +0.0044 Proof-Pile 128K −0.0114 PG19 2K −0.0130 PG19 32K −0.0438 PG19 128K −0.0246

Table 24: Paired comparisons between ϕ∗k and branch-matched ϕ̂k , n = 3. Standard deviations use the sample convention and intervals are 90% paired t-intervals. Eleven of twelve intervals lie within the selected ±0.05-PPL criterion; the PG19 32K pairwise-only comparison is inconclusive under that criterion. The twelve two-sided comparisons form one multiplicity family. Neither of the two smallest unadjusted p-values (.0132, .0404) passes the first Holm threshold 0.05/12 = .0042, so no adjusted comparison rejects its corresponding null hypothesis of zero mean difference. does not interpret these values as an optimisation-budget trend.

L. Angle-estimator comparison This section reports the full angle-estimator analysis underlying the four-checkpoint summary in the main text. The only varied quantity is the aggregation rule used to estimate the shared (layer, frequency-pair) angle from calibration statistics, selected by –equalising_r3_pool: • default Q/K-weighted estimator (rows): raw Q and K sufficient statistics are summed and centred once, weighting the streams by physical head count; • K-only estimator (k_only): the angle is estimated from K statistics alone; • stream-balanced estimator (balanced): per-stream first and uncentred second moments are averaged so that Q and K receive equal stream-level weight, followed by one centring operation. Every configuration applies one shared angle to Q and K. Proposition 4 therefore applies within every head. The angle is also shared across heads, so the evaluated transformations occupy the head-shared subset of the head-dependent direct-product family. The implementation path for the default Q/K-weighted estimator is unchanged. The comparisons in this section are exploratory. All reported intervals are unadjusted 90% paired t-intervals, and no family-wise error adjustment is applied. We therefore report effect estimates and cross-checkpoint consistency without assigning confirmatory significance to individual estimator contrasts. Multiplicity-adjusted testing is restricted to the twelve two-sided contrasts in §J. Comparability with the reported short-context values. Six configurations from Table 2 were evaluated again within this source state. Across all six, the largest absolute difference from the previously reported value is 3.3 × 10−5 PPL, equal to the effect of four-decimal rounding. The estimator comparisons are therefore made within a common source state and are directly comparable with the reported values at the stated precision. Table 26 reports the per-seed perplexities, and Table 27 reports the paired estimator contrasts.

33

component

seed

PPL

first loss

last loss

held-out MSE

rotation

0 1 2 0 1 2 0 1 2

11.0218 10.3707 10.6160 11.2190 10.4694 10.8642 10.9445 10.5346 10.8299

1.3816 2.4586 1.7880 1.3816 2.4586 1.7880 1.3816 2.4586 1.7880

0.7670 0.8684 1.4216 3.5531 1.1914 1.5660 1.7670 0.6165 1.2372

1.9030 1.7253 1.6383 2.2056 2.5308 3.8017 2.0365 1.9659 2.5088

scaling

rotation + scaling

Table 25: Per-seed results for the local FPTQuant-style Q/K component on WikiText-2 under dynamic W4A4KV4 with whip calibration. Comparisons use the full-head Hadamard seeds from Table 21. The reported optimisation objective is not monotone and does not track perplexity across seeds; no optimisationbudget conclusion is drawn. configuration

s0

s1

s2

s3

s4

s5

mean

sd

pairwise-only ϕ̂k , default Q/K-weighted ϕ̂k , stream-balanced ϕ̂k , K-only ϕ∗k , default Q/K-weighted ϕ∗k , K-only

10.8570 10.8137 10.7260 10.9367 10.8161

10.2054 10.2037 10.0706 10.2610 10.1952

10.5943 10.5043 10.4145 10.6261 10.5095

11.1317 11.0659 10.9185 11.1793 11.0157

10.3851 10.3208 10.2224 10.4607 10.3437

10.4001 10.3420 10.2218 10.4358 10.3334

10.5956 10.5417 10.4290 10.6499 10.5356

0.3437 0.3326 0.3293 0.3452 0.3173

pairwise + Hadamard ϕ̂k , default Q/K-weighted ϕ̂k , stream-balanced ϕ̂k , K-only ϕ∗k , default Q/K-weighted ϕ∗k , K-only

10.3724 10.3753 10.3811 10.3690 10.3747

9.7916 9.8046 9.8188 9.8161 9.7992

10.0952 10.0980 10.1310 10.1120 10.0952

10.6043 10.6096 10.6337 10.6202 10.5902

9.9101 9.9359 9.9387 9.9188 9.9287

9.9309 9.9579 9.9646 9.9447 9.9442

10.1174 10.1302 10.1447 10.1301 10.1220

0.3119 0.3048 0.3086 0.3083 0.3023

Table 26: Per-seed WikiText-2 perplexity for the Llama-3.2-3B angle-estimator comparison under dynamic W4A4KV4 with whip calibration. Full-head Hadamard and identity are evaluated in the same source state and agree with the corresponding Table 21 values to the stated tolerance. Cross-model estimator effects. Table 1 of the body reports the four-checkpoint comparison with six paired seeds. For pairwise-only, all four K-only-minus-default estimates are negative, and their unadjusted 90% intervals exclude zero: −0.5286 [−0.5747, −0.4825] on Llama-3.2-1B, −0.1666 [−0.1921, −0.1412] on Llama-3.2-3B, −0.0661 [−0.0797, −0.0525] on Llama-3.1-8B, and −0.0071 [−0.0091, −0.0051] on Mistral7B-v0.3. For pairwise + Hadamard, the corresponding estimates are +0.0559 [+0.0348, +0.0771], +0.0272 [+0.0193, +0.0352], +0.0129 [+0.0027, +0.0231], and +0.0018 [−0.0013, +0.0050]. The first three intervals lie above zero, whereas the Mistral-7B-v0.3 interval includes zero. The repeated direction across checkpoints supports estimator sensitivity within the evaluated configurations, but these exploratory comparisons do not constitute separately multiplicity-controlled advantage claims. Software-environment separation. The estimator analysis, including its cross-model evaluations, uses the A100 environment employed for the primary Llama-3.2-3B results. §I uses the second hardware environment. Absolute full-head Hadamard perplexity therefore differs between the two result sets: 14.482 versus 14.427 for Llama-3.2-1B, 7.919 versus 7.825 for Llama-3.1-8B, and 5.847 versus 5.892 for Mistral-7B-v0.3. Results from the two environments are not pooled. Within the estimator-analysis environment, the default Q/K-weighted pairwise + Hadamard differences independently reproduce the magnitudes reported in §I at n = 6 and lead to the same conclusion under the selected ±0.05-PPL interval criterion. The signs agree on Llama-3.2-1B and Mistral-7B-v0.3 but not on Llama-3.1-8B (−0.0002 here versus +0.0063 in §I); both values

34

configuration

comparator

n

mean difference

sample sd

pooling-rule comparisons, Llama-3.2-3B pairwise-only, ϕ̂k , K-only pairwise-only, ϕ̂k , stream-balanced pairwise-only, ϕ∗k , K-only pairwise + Hadamard, ϕ̂k , K-only pairwise + Hadamard, ϕ̂k , stream-balanced pairwise + Hadamard, ϕ∗k , K-only

pairwise-only, ϕ̂k , default pairwise-only, ϕ̂k , default pairwise-only, ϕ∗k , default pairwise + Hadamard, ϕ̂k , default pairwise + Hadamard, ϕ̂k , default pairwise + Hadamard, ϕ∗k , default

6 6 6 6 6 6

−0.1666 −0.0539 −0.1143 +0.0272 +0.0128 −0.0081

0.0309 0.0297 0.0316 0.0097 0.0112 0.0156

[−0.1921, −0.1412] [−0.0783, −0.0294] [−0.1403, −0.0884] [+0.0193, +0.0352] [+0.0036, +0.0220] [−0.0209, +0.0047]

comparisons with identity (no R3 ) pairwise-only, ϕ̂k , default pairwise-only, ϕ̂k , K-only pairwise-only, ϕ̂k , stream-balanced pairwise-only, ϕ∗k , default pairwise-only, ϕ∗k , K-only

identity identity identity identity identity

6 6 6 6 6

+0.0192 −0.1475 −0.0347 +0.0735 −0.0408

0.0238 0.0403 0.0260 0.0359 0.0454

[−0.0004, +0.0388] [−0.1806, −0.1144] [−0.0561, −0.0133] [+0.0440, +0.1030] [−0.0781, −0.0035]

90% CI

Table 27: Exploratory paired comparisons within the Llama-3.2-3B angle-estimator analysis, using six shared seeds, sample standard deviations, and unadjusted 90% paired t-intervals. The first block compares aggregation rules at fixed transformation structure; the second compares each pairwise-only configuration with identity. Under the default estimator, the interval for pairwise-only ϕ̂k minus identity includes zero. The K-only and stream-balanced estimates and their intervals are negative. These intervals are reported for effect estimation and are not multiplicity-adjusted. lie far inside the ±0.05-PPL margin, and at this magnitude the sign is not resolvable across environments.

M. Mixing-support interpolation (Llama-3.2-3B) The mixing-support analysis varies the block size of a pair-interleaved block-Hadamard transformation in the same Llama-3.2-3B, WikiText-2, dynamic W4A4KV4 setting as the primary short-context comparison. The transformation orientation is normalised to the SO(2) convention. Block sizes b ∈ {2, 4, 8, 16, 32, 64, 128} each use three paired perplexity seeds and three K-diagnostic seeds. Channel-contiguous layout and raworientation controls use one seed and are reported descriptively. Complete machine-readable values are in docs/data_db/clean/mixing_support.csv. Mean K range, relative K quantisation error, and the paired perplexity difference relative to full-head Hadamard decrease monotonically as block size increases. The b = 2 diagnostic agrees with the per-pair H2 endpoint to the reported precision. At b = 128, relative K quantisation error matches the full-head endpoint at the reported precision, mean K range differs by 0.0011, and the paired perplexity interval includes zero. Only b = 2 commutes with the RoPE frequency-pair blocks; larger blocks deliberately leave the centraliser to test the effect of broader mixing support. The interpolation therefore evaluates the support hypothesis rather than proposing an additional member of the commuting family. configuration

channel layout / orientation

RoPE-commuting

n

PPL mean (sd)

PPL difference vs full-head Hadamard [90% CI]

mean K range

max K range

∆

relative K quantisation error

b=2 b=4 b=8 b = 16 b = 32 b = 64 b = 128

pair/SO(2) pair/SO(2) pair/SO(2) pair/SO(2) pair/SO(2) pair/SO(2) pair/SO(2)

yes no no no no no no

3 3 3 3 3 3 3

10.4981 (.3234) 10.3085 (.3087) 10.2195 (.3026) 10.1573 (.2877) 10.1367 (.2833) 10.0976 (.2913) 10.0861 (.2847)

+.4158 [+.3720, +.4596] +.2262 [+.2069, +.2456] +.1373 [+.1242, +.1503] +.0751 [+.0505, +.0996] +.0544 [+.0266, +.0822] +.0153 [−.0286, +.0592] +.0039 [−.0236, +.0313]

14.815614 13.118872 11.576764 10.733735 10.447527 10.135678 9.643106

36.458333 38.479167 35.875000 31.791667 33.645833 30.375000 27.062500

.987708 .874591 .771784 .715582 .696502 .675712 .642874

.02305523 .01805045 .01385570 .01183181 .01122667 .01053918 .00953660

b = 2, raw orientation b = 8, channel-contiguous layout control b = 32, channel-contiguous layout control

pair/raw channel/SO(2) channel/SO(2)

yes no no

1 1 1

10.8119 10.4793 10.3926

+.4429 +.1103 +.0236

14.748907 11.125169 10.207206

34.125000 33.000000 28.875000

.983260 .741678 .680480

.02307303 .01284636 .01080546

full-head Hadamard identity per-pair H2

full head n/a RoPE pair

no no yes

3 3 3

10.0823 (.2976) 10.5395 (.3072) 10.5176 (.3113)

0 +.4573 [+.4368, +.4777] +.4354 [+.4058, +.4649]

9.641980 15.851159 14.815659

27.145833 36.145833 36.375000

.642799 1.056744 .987711

.00953662 .02634069 .02305625

Table 28: Complete mixing-support comparison. The primary interpolation uses three paired seeds at each block size; the orientation and channel-layout controls use one seed and therefore have no confidence interval. K diagnostics are measured after R3 with the same procedure for every configuration.

35

C

Proofs

cos ϕ − sin ϕ
∈ SO(2) denotes the Givens rotation in the orientation sin ϕ cos ϕ (k) fixed by Definition 1 (so that H2 = G(π/4) diag(1, −1), as used in the body), Rm = G(mθk ) is the LK (k) band-k RoPE block, and Rm = k=1 Rm with K = dh /2. We write the band-k pre-RoPE block as 2
σ ρσ σ e (k) for the pooled-covariance, position-averaged surrogate of Eq. (2). In Σ(k) = ρσ 1σ σ12 2 and reserve Σ 1 2 2 (k)

Throughout this appendix G(ϕ) :=

the empirical application, Σ is reconstructed after raw calibration moments have been aggregated across observation rows. The analysis treats this fixed matrix as the pre-RoPE covariance at every position; it does (k) not estimate position-conditional matrices Σm . The algebra below is exact for this fixed-matrix surrogate.

Proof of Lemma 2. We prove both inclusions. LK L (k) (a) Band-aligned ⇒ commutes. Let R3 = k=1 G(ϕk ) ∈ Bdh . Both R3 and Rm = k Rm are block-diagonal under the same partition of {1, . . . , dh } into the K RoPE coordinate pairs (2k−1, 2k). The product of two block-diagonal matrices sharing a block partition is the block-diagonal matrix of the blockwise products, so it suffices to commute within each 2 × 2 block. There both factors lie in SO(2): G(ϕk ) by (k) definition and Rm = G(mθk ). Since SO(2) is abelian with G(α)G(β) = G(α + β) = G(β)G(α), (k) G(ϕk ) R(k) m = G(ϕk + mθk ) = Rm G(ϕk )

(k = 1, . . . , K),

and assembling the blocks gives R3 Rm = Rm R3 for every m ∈ Z. Hence Bdh lies in the centraliser of {Rm }m∈Z inside O(dh ). (b) Commutes with all m ⇒ band-diagonal, under distinct frequencies. Let A ∈ O(dh ) satisfy ARm = (k) Rm A for all m ∈ Z. It suffices to use m = 1. Work over the complexification Cdh = Rdh ⊗C. The block R1 = 2 ±iθk G(θk ) has characteristic polynomial λ − 2 cos θk λ + 1, hence eigenvalues e with conjugate eigenvectors vk± = e2k−1 ∓ i e2k spanning the complexification of the band-k real 2-plane Pk := spanR {e2k−1 , e2k }. Under the lemma’s assumptions, θk ∈ (0, π) and the θk are pairwise distinct, so the 2K = dh numbers {e±iθk }K k=1 are all distinct: eiθk = eiθk′ forces θk = θk′ , while eiθk = e−iθk′ forces θk + θk′ ∈ 2πZ, impossible on (0, π). Thus R1 has dh distinct eigenvalues and each eigenspace is one-dimensional. Any A commuting with R1 preserves each eigenspace; being real, A maps the conjugate pair {vk+ , vk− } to itself (a real matrix sends v to Av), hence preserves its real span PkL . Therefore A leaves each band 2-plane Pk invariant and is blockdiagonal under the band partition, A = k Ak with Ak ∈ O(2) (orthogonality of A restricts to the invariant 2-plane). It remains to place each Ak inside SO(2). Restricted to Pk the commutation reads Ak G(θk ) = G(θk )Ak with θk ∈ / πZ, so G(θk ) is a non-trivial rotation. If det Ak = +1 then Ak = G(ψk ) ∈ SO(2), which commutes with G(θk ) automatically. If det Ak = −1 then Ak is a reflection, and a reflection conjugates any rotation to its inverse: Ak G(θ)A−1 k = G(−θ), equivalently Ak G(θ) = G(−θ)Ak . Combined with the commutation hypothesis this yields G(θk )Ak = G(−θk )Ak , so G(2θk ) = I and θk ∈ πZ,Lcontradicting θk ∈ (0, π). Hence the reflection case cannot occur and Ak ∈ SO(2) for every k, giving A = k G(ψk ) ∈ Bdh . This proves the reverse inclusion, so the centraliser equals Bdh . Multi-head scope. Lemma 2 is a single-head statement. If cross-head mixing is excluded, its H-head direct-product extension is ( H K ) MM G(ϕh,k ) , h=1 k=1

which permits head-dependent angles. The implementation evaluates the strict subset satisfying ϕh,k ≡ ϕk within each layer. If mixing among the H repeated copies of a frequency were permitted, the corresponding real 2H-dimensional centraliser would contain a U (H) factor. Thus the lemma identifies the commuting factor for one head, whereas the empirical conclusions apply only to the implemented head-shared subset. e (k) , reduce the problem to a 2 × 2 Jacobi rotation, solve for Proof of Theorem 3. We verify the entries of Σ the equalising angle, identify the minimax branch, and establish the analytic minimum.

36

e (k) . Fix band k, drop the superscript, and write c = cos mθk , s = sin mθk , so R(k) (0) Entries of Σ m =
c −s . Direct multiplication of R(k) Σ(k) (R(k) )⊤ gives m m s c [ · ]11 = c2 σ12 − 2cs ρσ1 σ2 + s2 σ22 , [ · ]22 = s2 σ12 + 2cs ρσ1 σ2 + c2 σ22 , [ · ]12 = cs(σ12 − σ22 ) + (c2 − s2 )ρσ1 σ2 . With c2 = 12 (1 + cos 2mθk ), s2 = 12 (1 − cos 2mθk ), 2cs = sin 2mθk , c2 − s2 = cos 2mθk , these become σ12 +σ22 σ 2 −σ 2 + 1 2 2 cos 2mθk − ρσ1 σ2 sin 2mθk , 2 σ 2 −σ 2 σ 2 +σ 2 [ · ]22 = 1 2 2 − 1 2 2 cos 2mθk + ρσ1 σ2 sin 2mθk , σ 2 −σ 2 [ · ]12 = 1 2 2 sin 2mθk + ρσ1 σ2 cos 2mθk .

[ · ]11 =

For the fixed-matrix surrogate, averaging over m = 0, . . . , L − 1 passes the constants through unchanged and e (k) replaces cos 2mθk 7→ Ck , sin 2mθk 7→ Sk (definitions (1)), reproducing L the displayed entries of Σ . (i) Reduction to a single Jacobi problem. By Lemma 2, R3 = k G(ϕk ) ∈ Bdh commutes with each Rm , and so does R3⊤ . Hence L−1

L−1

 1 X 1 X ⊤ ⊤ e ⊤ Rm R3 Σ R3⊤ R⊤ R ΣR = R m 3 m R3 = R3 Σ R3 , m L m=0 L m=0 the first equality moving R3 , R3⊤ out through the average by commutativity, the second by the definition e We do not assume that the full covariance Σ is band-block-diagonal: cross-band covariance blocks of Σ. may be nonzero. However, because every Rm and R3 is block-diagonal under the band partition, the kth e ⊤ is exactly G(ϕk ) Σ e (k) G(ϕk )⊤ . Thus the objective, which uses only these principal 2 × 2 block of R3 ΣR 3 channel variances, decouples band by band. This reduction applies to the fixed pooled-covariance surrogate; it does not replace Σ with position-conditional matrices.
e (k) = λ1 β with λ1 = Σ e (k) , λ2 = Σ e (k) , β = Σ e (k) . (ii) Equalising condition and closed form. Write Σ 11 22 12 β λ2 With G(ϕ) in the orientation above, the post-rotation diagonal entries are d+ (ϕ) = λ1 cos2 ϕ + λ2 sin2 ϕ − 2β cos ϕ sin ϕ, d− (ϕ) = λ1 sin2 ϕ + λ2 cos2 ϕ + 2β cos ϕ sin ϕ, and, via cos2 ϕ − sin2 ϕ = cos 2ϕ, 2 sin ϕ cos ϕ = sin 2ϕ, f (ϕ) := d+ (ϕ) − d− (ϕ) = (λ1 − λ2 ) cos 2ϕ − 2β sin 2ϕ.

(4)

Equalisation d+ = d− is therefore (λ1 − λ2 ) cos 2ϕ − 2β sin 2ϕ = 0

⇐⇒

tan 2ϕ =

λ1 − λ2 . 2β

(5)

The two-argument inverse tangent solves (5) over a full half-period and resolves the β = 0 degeneracy by quadrant:

e (k) . e (k) − Σ e (k) , 2Σ ϕ∗k = 21 atan2 λ1 − λ2 , 2β = 21 atan2 Σ (6) 12 11 22 Indeed atan2(y, x) returns the angle 2ϕ with cos 2ϕ ∝ x = 2β and sin 2ϕ ∝ y = λ1 − λ2 (a common positive factor), so (λ1 − λ2 ) cos 2ϕ − 2β sin 2ϕ ∝ (λ1 − λ2 )(2β) − 2β(λ1 − λ2 ) = 0, verifying (5). After reduction modulo π/2, this is the body’s Eq. (3) under the standard Givens orientation of Definition 1. The reference implementation computes the same Jacobi formula on the pooled calibration covariance before applying the finite-phase correction, ϕ̂k = 12 atan2(σ12 − σ22 , 2ρσ1 σ2 ), i.e. (6) evaluated at (Ck , Sk ) = (1, 0). The two are related exactly, ϕ∗k = ϕ̂k − 21 atan2(Sk , Ck ) (mod π/2) (coinciding on bands with Sk = 0), and the excess of ϕ̂k over the minimum λ̄k in the position-averaged minimax metric of the statement is 37

p (σ12 − σ22 )2 + (2ρσ1 σ2 )2 , vanishing on isotropic bands and potentially largest in the low-frequency tail. Its numerical value must be computed from the checkpoint’s deployed inverse frequencies, including any RoPE scaling. When (λ1 − λ2 , β) = (0, 0) the band is already isotropic, every ϕ equalises, and the convention atan2(0, 0) = 0 returns the identity. (iii) Minimax vs. maximin branch. Since orthogonal conjugation preserves the trace, d+ (ϕ) + d− (ϕ) = λ1 + λ2 is constant in ϕ, so
2 + 12 |f (ϕ)|, max d+ (ϕ), d− (ϕ) = λ1 +λ 2 p with f the amplitude-r sinusoid (4), r = (λ1 − λ2 )2 + 4β 2 > 0 in the non-degenerate case (λ1 − λ2 , β) ̸= (0, 0). Thus minimising the maximum band variance is equivalent to minimising |f |. Its zeros occur at 2ϕ = atan2(λ1 −λ2 , 2β)+nπ, i.e. at ϕ = ϕ∗k +nπ/2. The two equalizing representatives differ by π/2 in ϕ and merely swap the two channel labels. Reducing modulo π/2 into [−π/4, π/4) selects one representative. By contrast the branch with 2ϕ = atan2(2β, λ2 −λ1 )+nπ maximises |f | and gives max(d+ , d− ) = 21 (λ1 +λ2 )+ 12 r. e (k) = (iv) Minimum maximal variance and O(2) optimality. Summing the diagonal entries of (0), tr Σ (k) (k) e e λ1 + λ2 = σ12 + σ22 (the Ck , Sk terms cancel between Σ 11 and Σ22 ). Hence for any ϕ, 1 2 |Sk |

d+ (ϕ) + d− (ϕ) σ 2 + σ22 max d+ (ϕ), d− (ϕ) ≥ = 1 = λ̄k , 2 2 with equality iff d+ (ϕ) = d− (ϕ); the minimax ϕ∗k attains it. This used only that the transform is a 2 × 2 rotation, but the bound extends to all of O(2) by trace invariance alone: for any U ∈ O(2) the conjugation e (k) U ⊤ preserves the trace, tr(U Σ e (k) U ⊤ ) = tr Σ e (k) = λ1 + λ2 , so the larger of its two diagonal entries UΣ 1 e (k) U ⊤ ) = λ̄k , a bound attained inside SO(2) by G(ϕ∗ ). Therefore λ̄k is the minimum of is at least 2 tr(U Σ k max(Var(ỹ2k−1 ), Var(ỹ2k )) over all of O(2), and the SO(2) optimum equals the O(2) optimum. Proposition 5 (Calibration-only and deterministic). The dh /2 angles {ϕ∗k } are computable in O(N dh ) time and O(dh ) memory from one calibration pass over N tokens. For fixed inputs, accumulation order, and numerical implementation, no optimisation randomness is involved. Proof of Proposition 5. For each band P k the data-dependent P entries of the pre-RoPE P 2 are the P P 2 quantities a2k−1 a2k , recover the two a2k , and covariance Σ(k) . Five streaming sums, a2k−1 , a2k , a2k−1 , variances and covariance after centering. This requires O(1) work per token and band, hence O(N dh ) time and O(dh ) auxiliary memory overall. The constants Ck , Sk depend only on the model frequencies and e (k) and evaluating one atan2 per band then costs calibration length and can be precomputed. Constructing Σ O(dh ). There is no iterative optimiser, learning-rate schedule, momentum, or per-iteration sampling. For fixed calibration tensors, reduction order, and numerical implementation, the computation is deterministic; this statement does not assert bitwise identity across hardware or numerical libraries. Proposition 6 (Hadamard comparison under the surrogate band metric). For the pooled-covariance, position-averaged per-band variance metric, G(ϕ∗k ) weakly dominates the 2 × 2 Hadamard proxy and strictly e (k) ̸= 0. dominates it whenever Σ 12 Proof of Proposition 6. For the per-channel minimax metric, H2 and G(π/4) yield the same unordered pair
1 , of output variances, although the channel order is reversed. Direct multiplication with H2 = √12 11 −1 √ (k) (k) (k) e e e λ1 = Σ , λ2 = Σ , and β = Σ = ρe λ1 λ2 gives 11

22

12

e (k) H⊤ = 1 H2 Σ 2 2

 λ1 + λ2 + 2β λ1 − λ2

 λ 1 − λ2 , λ1 + λ2 − 2β

so the two output band-channel variances are λ′1 (π/4) =

p λ1 + λ 2 + ρe λ1 λ2 , 2

λ′2 (π/4) =

p λ1 + λ2 − ρe λ1 λ2 , 2

the displayed formulas, with difference p λ′1 (π/4) − λ′2 (π/4) = 2β = 2e ρ λ1 λ2 . 38

As λ1 , λ2 > 0, √ this vanishes iff ρe = 0; whenever ρe ̸= 0, H2 fails to equalise and yields max(λ′1 , λ′2 ) = 1 ρ| λ1 λ2 > λ̄k . 2 (λ1 + λ2 ) + |e For strict domination, recall from the proof of Theorem 3(iii)–(iv) that any 2 × 2 orthogonal U scores max(d+ (U ), d− (U )) = λ̄k + 12 |fU | in the minimax band metric, where fU = d+ (U ) − d− (U ) and d+ + d− = √ λ1 + λ2 is fixed. The closed-form G(ϕ∗k ) achieves f = 0 and therefore reaches λ̄k , whereas fH2 = 2e ρ λ1 λ2 = e (k) . Thus G(ϕ∗ ) strictly dominates H2 exactly when Σ e (k) ̸= 0. If Σ e (k) = 0 and Σ e (k) ̸= Σ e (k) , Eq. (6) 2Σ 12 12 12 11 22 k ∗ yields ϕk = ±π/4, so the two transforms attain the same diagonal variances. Consequently, G(ϕ∗k ) weakly e (k) ̸= 0. dominates H2 on every band under the surrogate metric and strictly dominates it whenever Σ 12 Proof of Proposition 4. Both R3Q , R3K ∈ Bdh , so by Lemma 2 each commutes with every Rm . Using R⊤ m = R−m and M QK := (R3Q )⊤ R3K , Q ⊤ K ⊤ ⊤ QK (R3Q Rm q)⊤ (R3K Rn k) = q⊤ R⊤ Rn k. m (R3 ) R3 Rn k = q Rm M

L L Q ⊤ K K ⊤ Now M QK = k G(ϕQ k G(ϕk − ϕk ), using G(α) = G(−α) and SO(2) additivity. Since k ) G(ϕk ) = QK ⊤ M , Rm , Rn are all band-diagonal under the same partition, the bilinear form splits over bands: QK q ⊤ R⊤ Rn k = mM

K X

Q ⊤ K (k) (k) q(k)⊤ (R(k) , m ) G(ϕk − ϕk ) Rn k

k=1

with q(k) , k(k) ∈ R2 the band-k components. Within band k every factor lies in the abelian group SO(2), so the rotations compose by adding angles: (k)

Q Q Q Q ⊤ K (k) K K K (R(k) m ) G(ϕk −ϕk ) Rn = G(−mθk ) G(ϕk −ϕk ) G(nθk ) = G((n−m)θk ) G(ϕk −ϕk ) = Rn−m G(ϕk −ϕk ).

Substituting back yields the displayed identity (R3Q Rm q)⊤ (R3K Rn k) =

K X

(k)

Q (k) q(k)⊤ Rn−m G(ϕK . k − ϕk ) k

k=1 Q If ϕQ = ϕK (mod 2π) for all k, then G(ϕK k − ϕk ) = G(0) = I on every band and the sum collapses to P k (k)⊤ k(k) (k) Rn−m k = q⊤ Rn−m k, the original RoPE relative-position score, which depends on (m, n) only kq Q K K through n − m. If ϕQ k ̸≡ ϕk (mod 2π) for some k, the band-k term carries the extra factor G(ϕk − ϕk ), a fixed nonidentity rotation. Being position-independent, it cannot be absorbed into the relative rotation (k) Q Rn−m = G((n − m)θk ) for all (m, n) simultaneously (that would require ϕK k − ϕk ≡ 0 (mod 2π)); it does not cancel against RoPE and perturbs the band-k Q/K inner product on which attention relies. Hence a K single shared online R3 (ϕQ k = ϕk (mod 2π) for all k) is required to preserve the attention geometry.

39

Related documents

• Monoidal Quantaloids#128376
• Noncommutative QFT and Relative Entropy on Axisymmetric Bifurcate Killing Horizons#1143708
• Coordinating Pebble Motion on Graphs, The Diameter of Permutation Groups, and Applications#472203
• Axiomatic TQFT, Axiomatic DQFT, and Exotic 4-Manifolds#615385
• Duality between polyhedral approximation of value functions and optimal quantization of measures#1011003