arXiv CS open access

Ask Self, Ask Others: Relation Is All You Need

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

A SK S ELF, A SK OTHERS : R ELATION I S A LL YOU N EED Mingkai Nie3,†

Yang Pengju2,†

Yuting Ge1,∗ 1

Department of Electrical Engineering City University of Hong Kong

Jilin University

yangda1223@outlook.com

National University of Singapore mingkai.nie@u.nus.edu

yutingge3-c@my.cityu.edu.hk

arXiv:2608.20172v1 [cs.LG] 20 Aug 2026

†

These authors contributed equally.

∗

Corresponding author.

A BSTRACT Attention directly derives normalized information flow from pairwise scores. We introduce Relation, an alternative token-mixing primitive that first organizes pairwise evidence into explicit Self and Exchange relations and derives information flow afterward. This relational organization gives rise to Full Relation, FlashRelation, Linear Relation, Hybrid Relation, and a KV-style Relation Cache. Across matched decoder-only models at approximately 10M, 30M, and 100M parameters, Full Relation achieves lower final validation NLL than MHA at all three scales. In a fixed-context reference benchmark, FlashRelation is 3.60–4.41× faster than the materialized Full Relation implementation. Across scale-matched production workloads, it reaches 76.4–84.9% of PyTorch FlashAttention throughput while executing the Full Relation operator. Hybrid Relation uses 75% Linear Relation layers and achieves strong language-modeling quality. These results support a relation-first view of token mixing: ask Self, ask Others, then let Flow follow Relation.

I NTRODUCTION

In scaled dot-product Attention, each head maps the interaction between two projected token states to a scalar score (Vaswani et al., 2017), qi⊤ kj Uij = √ , dh

Fi = Softmax(Ui ).

Token interactions in language are heterogeneous. But in Attention, one scalar carries too much. Attention uses the same Uij to encode whatever relational structure matters and to directly determine information flow. Relation formation and flow allocation are therefore compressed into a single U → F step. Attention itself was not always primary. It first appeared as an auxiliary mechanism within recurrent encoder–decoder models before becoming the organizing primitive of the Transformer (Bahdanau et al., 2015; Vaswani et al., 2017). Relation-aware Attention places relation in a similarly subordinate position. Shaw et al. (2018) augment self-attention with pairwise relation representations, but relation is still inside Attention. Attention remains primary: it incorporates relation into its own computation and still determines the flow. We reverse this hierarchy: Flow follows Relation. We begin by decomposing Relation into two minimal structural roles: Self and Exchange. Self describes a token’s relation with itself, while Exchange describes its relation with other tokens. Pairwise evidence is mapped into these two roles before normalization, and the resulting entries are assembled into a causal Relation matrix R. Normalized information flow F is derived only afterward. This defines the basic Self–Exchange Relation (SER) operator, which extends naturally to Multi-Head Relation (MHR). This explicit organization also exposes further forms of Relation in practice: Full Relation retains explicit token-wise history, FlashRelation factorizes it for tiled execution, Linear Relation compresses historical Others into a recurrent state, Hybrid Relation combines Full and Linear layers, and Relation Cache supports autoregressive decoding. 1

Pairwise Evidence 𝑼𝑼

Self | Exchange

𝐒𝐒 𝐑𝐑 = 𝐄𝐄 𝐄𝐄

Relation Strength

Attention Relation

Full Relation

Current Token

Exchange

Historical Tokens

Self

done normalize

produce Y

−∞ −∞ 𝐒𝐒

Information

Flow Allocation Exchange

Self

Linear Relation

Hybrid Relation

causal tile scan masked

Online scan

Self Exchange

ΔNLL vs. MHA 10M −0.0412 30M −0.0151 100M −0.0310

−∞ 𝐒𝐒 𝐄𝐄

FlashRelation Exchange tiles construct entries

masked S E

Flow 𝑭𝑭

Softmax

done

current

No full R or F materialization

production throughput and speedup Ref. → Flash

3.60–4.41×

vs. FlashAttention

76.4–84.9%

1 Ask Self

𝑺𝑺𝒕𝒕 = 𝝈𝝈

𝒑𝒑⊤ 𝟏𝟏,𝒕𝒕 𝒑𝒑𝟐𝟐,𝒕𝒕 𝝉𝝉𝑺𝑺 𝒅𝒅𝒉𝒉

𝑪𝑪𝒕𝒕−𝟏𝟏 𝜶𝜶𝒕𝒕

𝑪𝑪− 𝒕𝒕 = 𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫 𝜶𝜶𝒕𝒕 𝑪𝑪𝒕𝒕−𝟏𝟏

2 Ask Others

3 Answer

𝑪𝑪− ˆ 𝟏𝟏,𝒕𝒕 𝒕𝒕 𝑺𝑺𝒕𝒕 𝒑𝒑

𝑰𝑰�𝒕𝒕

⊤ˆ 𝒑𝒑𝟏𝟏,𝒕𝒕 𝑬𝑬𝒕𝒕 = 𝑺𝑺𝒕𝒕 𝑪𝑪− 𝒕𝒕

Linear in 𝑻𝑻 𝑶𝑶 𝑻𝑻𝑻𝑻²/𝑯𝑯

𝑬𝑬𝒕𝒕

+ 𝒀𝒀𝒕𝒕 = 𝑰𝑰�𝒕𝒕 + 𝑬𝑬𝒕𝒕 Decode state

4 Become Others

p̂2,t

Ĩt

St 75% Linear, 25% Full

𝑪𝑪𝒕𝒕−

+ Ct

ˆ 𝟐𝟐,𝒕𝒕 𝑰𝑰�⊤ 𝑪𝑪𝒕𝒕 = 𝑪𝑪− 𝒕𝒕 + 𝑺𝑺𝒕𝒕 𝒑𝒑 𝒕𝒕 𝑶𝑶 𝒅𝒅²/𝑯𝑯

Linear Relation Full Relation 𝐍𝐍𝐍𝐍𝐍𝐍 𝟏𝟏. 𝟐𝟐𝟐𝟐𝟐𝟐𝟐𝟐 ± 𝟎𝟎. 𝟎𝟎𝟎𝟎𝟎𝟎𝟎𝟎

Figure 1: Overview of Relation. We evaluate Relation at approximately 10M, 30M, and 100M parameters under matched decoderonly settings. Full Relation achieves lower final-validation NLL than MHA at all three scales, while structural ablations examine the roles of its main relational components. We further evaluate practical realizations of Relation: FlashRelation provides exact tiled execution of Full Relation, Linear Relation replaces explicit token-wise history with a recurrent relational state, Hybrid Relation composes Full and Linear layers, and Relation Cache supports autoregressive decoding. Together, these results support Relation as a viable and practical alternative token-mixing primitive (Figure 1). Our main contributions are: • We introduce Self–Exchange Relation (SER) and Multi-Head Relation (MHR) as Relation operators for token mixing. • We characterize Self–Exchange flow and derive its exact factorization into Self mass, aggregate Exchange mass, and allocation within historical Exchange. • Building on Relation’s distinct relational organization, we develop Relation-specific practical forms, including FlashRelation, Linear Relation, and Hybrid Relation, and mathematically derive an exact KV-style Relation Cache for autoregressive decoding.

BACKGROUND

Scaled dot-product Attention represents pairwise token interactions through compatibility scores that are normalized into token-to-token mixing weights (Vaswani et al., 2017). Subsequent work has enriched how these interactions are represented and transformed. Relative-position Attention incorporates pairwise relation representations into the scoring process (Shaw et al., 2018), while learned or structured positional biases reshape pre-normalization scores (Raffel et al., 2020; Press et al., 2022; Chi et al., 2022). Sigmoid self-attention modifies the normalization itself (Ramapuram et al., 2025). Talking-Heads Attention introduces projections across the attention-head dimension around Softmax (Shazeer et al., 2020), while the Gated Attention Unit combines single-head attention with gating (Hua et al., 2022). Beyond Attention, alternative token-mixing operators explore different ways to organize information exchange across a sequence. FNet replaces self-attention with Fourier mixing (Lee-Thorp et al., 2022), PoNet uses multi-granularity pooling (Tan et al., 2022), and HyperMixer dynamically constructs token-mixing MLPs (Mai et al., 2023). A separate line of work focuses on how token mixing is executed and how historical information is represented. FlashAttention and FlashAttention-2 evaluate exact softmax Attention through tiled, IO-aware execution (Dao et al., 2022; Dao, 2024). Linear and recurrent approaches replace explicit 2

quadratic token-token interaction with associative or recurrent states (Katharopoulos et al., 2020; Sun et al., 2023; Yang et al., 2024; 2025). Kimi Delta Attention (KDA) further develops recurrentstate token mixing, while Kimi Linear combines KDA with full-attention MLA layers in a hybrid architecture (Kimi Team et al., 2025). These lines place Relation alongside three complementary directions in token mixing: interaction representation, efficient execution, and recurrent history. Relation connects these directions through a common relational organization. The following sections develop this framework from its basic Self–Exchange Relation (SER) operator to its practical forms.

R ELATION

We first introduce Self–Exchange Relation (SER), the basic Relation operator. We then extend SER to Multi-Head Relation (MHR), as illustrated in Figure 2. Multi-Head Relation

Self–Exchange Relation

𝐘𝐘𝐨𝐨𝐨𝐨𝐨𝐨 = 𝐘𝐘𝐖𝐖𝐎𝐎

𝒀𝒀𝒊𝒊 = � 𝑭𝑭𝒊𝒊𝒊𝒊 𝑰𝑰𝒋𝒋 𝒋𝒋≤𝒊𝒊

𝒀𝒀 = 𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪 𝒀𝒀 𝟏𝟏 , … , 𝒀𝒀 𝑯𝑯

𝑭𝑭𝒊𝒊 = 𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺 𝑹𝑹𝒊𝒊

𝑹𝑹 =

𝑭𝑭 𝒉𝒉 = 𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺 𝑹𝑹 𝒉𝒉

𝑺𝑺𝟏𝟏 −∞ −∞ ⋯ 𝑬𝑬𝟐𝟐𝟐𝟐 − 𝝀𝝀ℓ 𝒍𝒍𝒍𝒍𝒍𝒍 𝟐𝟐 𝑺𝑺𝟐𝟐 −∞ ⋯ 𝑬𝑬𝟑𝟑𝟑𝟑 − 𝝀𝝀ℓ 𝒍𝒍𝒍𝒍𝒍𝒍 𝟑𝟑 𝑬𝑬𝟑𝟑𝟑𝟑 − 𝝀𝝀ℓ 𝒍𝒍𝒍𝒍𝒍𝒍 𝟑𝟑 𝑺𝑺𝟑𝟑 ⋯ ⋮ ⋮ ⋮ ⋱ 𝑺𝑺𝒊𝒊 = 𝝈𝝈

𝑼𝑼𝒊𝒊𝒊𝒊 𝝉𝝉𝑺𝑺

𝑷𝑷𝟏𝟏 = 𝑿𝑿𝑾𝑾𝟏𝟏

Relation Construction 𝒉𝒉 𝑹𝑹𝒊𝒊𝒊𝒊

𝑬𝑬𝒊𝒊𝒊𝒊 = 𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺 𝑼𝑼𝒊𝒊𝒊𝒊 𝑼𝑼𝒊𝒊𝒊𝒊 =

𝒉𝒉

𝑷𝑷𝟏𝟏

⊤ 𝒑𝒑𝟏𝟏,𝒊𝒊 𝒑𝒑𝟐𝟐,𝒋𝒋

𝒅𝒅𝒉𝒉

𝑷𝑷𝟐𝟐 = 𝑿𝑿𝑾𝑾𝟐𝟐

𝒀𝒀 𝒉𝒉 = 𝑭𝑭 𝒉𝒉 𝑰𝑰� 𝒉𝒉

𝒉𝒉

𝑺𝑺𝒊𝒊 , 𝒋𝒋 = 𝒊𝒊, = 𝑬𝑬 𝒉𝒉 − 𝝀𝝀ℓ 𝒍𝒍𝒍𝒍𝒍𝒍 𝒊𝒊 , 𝒋𝒋 < 𝒊𝒊, 𝒊𝒊𝒊𝒊 −∞, 𝒋𝒋 > 𝒊𝒊.

𝑰𝑰 = 𝑿𝑿𝑾𝑾𝑰𝑰

× 𝑯𝑯

𝑰𝑰� 𝒂𝒂

𝑰𝑰� 𝒃𝒃

Givens Mixing 𝑰𝑰� 𝒂𝒂 = 𝒄𝒄𝒄𝒄𝒄𝒄 𝜽𝜽ℓ,𝒂𝒂𝒂𝒂 − 𝒔𝒔𝒔𝒔𝒔𝒔 𝜽𝜽ℓ,𝒂𝒂𝒂𝒂 𝑰𝑰 𝒂𝒂 𝒃𝒃 𝒔𝒔𝒔𝒔𝒔𝒔 𝜽𝜽ℓ,𝒂𝒂𝒂𝒂 𝒄𝒄𝒄𝒄𝒄𝒄 𝜽𝜽ℓ,𝒂𝒂𝒂𝒂 𝑰𝑰 𝒃𝒃 𝑰𝑰�

𝒉𝒉

𝑷𝑷𝟐𝟐

𝑰𝑰 𝒂𝒂

Alternating Adjacent-Head Pairing 𝓜𝓜𝑨𝑨 = 𝟏𝟏, 𝟐𝟐 , 𝟑𝟑, 𝟒𝟒 , … , 𝑯𝑯 − 𝟏𝟏, 𝑯𝑯 𝓜𝓜𝑩𝑩 = 𝟐𝟐, 𝟑𝟑 , 𝟒𝟒, 𝟓𝟓 , … , 𝑯𝑯, 𝟏𝟏 𝓜𝓜𝑨𝑨 → 𝓜𝓜𝑩𝑩 → 𝓜𝓜𝑨𝑨 → ⋯

𝑰𝑰 𝒃𝒃

Figure 2: Self–Exchange Relation and Multi-Head Relation. 3.1

S ELF –E XCHANGE R ELATION

Given an input sequence X, we project it into two relation spaces and an information space, P1 = XW1 ,

P2 = XW2 ,

I = XWI .

p⊤ 1,i p2,j Uij = √ . dh

(1)

The head-wise P1 and P2 entering U use RoPE, while I remains unrotated. Relation distinguishes a token’s interaction with itself from its interactions with other tokens. We define the Self and Exchange relation mappings as Uii Si = σ , Eij = SiLU (Uij ) , τS  j = i, Si , Rij = Eij − λℓ log i, j < i,  −∞, j > i.

(2)

The Self temperature τS > 0 controls the scale and smoothness of the bounded Self mapping. We set τS = 2 in all formal experiments and this fixed value is not treated as theoretically unique. As the number of visible predecessors increases, aggregate Exchange evidence receives a candidatecount contribution. We therefore use a learnable layer-wise coefficient λℓ and apply the logarithmic 3

correction −λℓ log i, following the broader use of logarithmic length corrections in attention mechanisms (Ramapuram et al., 2025). Normalized information flow is derived only after the Relation has been constructed and is applied to the information states, Fi = Softmax (Ri ) , X Fij Ij , Yi =

(3)

j≤i

Xℓ+1 = Xℓ + Y WO . 3.2

M ULTI -H EAD R ELATION

We extend Relation to H heads. The projected relation spaces and information space are partitioned into H subspaces, each of width dh = d/H. Each head independently applies the SER construction from Sec. 3.1, yielding its own R(h) and F (h) . Before applying the flow, we mix the information states across adjacent head pairs using learnable Givens rotations (Jing et al., 2017). The pairing pattern alternates between successive layers, (a) cos θℓ,ab Ie = sin θℓ,ab Ie(b)

− sin θℓ,ab cos θℓ,ab

(a) I . I (b)

(4)

Each head then transports its mixed information according to its own flow, and the head outputs are concatenated and projected back to the residual stream, Y (h) = F (h) Ie(h) , (h) Xℓ+1 = Xℓ + ConcatH WO . h=1 Y

(5)

W HY R ELATION

Why introduce an explicit Relation matrix between pairwise evidence and information flow? The purpose of R is to change how token mixing is organized before normalization. 4.1

R ELATION I S C ONSTRUCTED B EFORE F LOW

In canonical Attention, pairwise compatibility scores directly form the logits from which normalized information flow is derived. Relation instead constructs an explicit relational structure before normalization. As defined in Sec. 3.1, R is assembled from role-specific Self and Exchange entries, with Self on the diagonal and Exchange over the causal history. The two roles are therefore defined before they compete for information flow. Treating R as an explicit object in the operator definition does not require materializing the full matrix during execution. Because R is defined entrywise from pairwise evidence through the Self– Exchange construction, the required evidence and Exchange entries can be formed and consumed as needed. R specifies how token mixing is organized, while its computational representation can remain implicit. 4.2

H OW R ELATION F ORMS S ELF –E XCHANGE F LOW

Because Self and Exchange are already explicit in R, the normalized flow admits an exact decomposition aligned with these roles. For token i, collect the historical Exchange entries into For token i > 1, let Si denote the constructed Self relation from Eq. (2). We define the historical Exchange log-normalizer LE i , its count-corrected aggregate Ai , and the full row normalizer Zi as 4

LSEj<i (aj ) ≡ log

eaj ,

j<i

LE i = LSEj<i (Eij ),

(6)

Ai = LE i − λℓ log i = LSEj<i (Rij ), Zi = eSi + eAi .

The normalized flow first divides its mass between Self and aggregate Exchange,

Fii =

eSi eSi + eAi

= σ(Si − Ai ),

gi ≡

Fij =

j<i

eAi eSi + eAi

= σ(Ai − Si ),

Fii + gi = 1. (7)

Thus, the normalized flow is partitioned exactly between the Self mass Fii and the aggregate Exchange mass gi . Conditional on Exchange, the mass gi is distributed across historical tokens according to E πij ≡

Fij eRij eEij P =P = , Rik Eik gi k<i e k<i e

j < i,

eRij eAi eRij eRij E P = g Fij = = = gi πij . i Rik Zi Zi eAi k<i e

(8)

The communicated information is therefore Yi =

Fij Iej = Fii Iei + gi

Ee πij Ij .

(9)

j<i

j≤i

The normalized flow therefore has two levels. It first determines how much mass remains with Self and how much enters Exchange. Conditional on Exchange, it then determines how that mass is distributed across historical tokens. Intuitively, Relation first asks whether a token should rely on itself or draw from its history, and if it draws from history, it then decides where to look. 4.3

I NSIDE R ELATION : H OW A T OKEN ’ S F LOW I S D ETERMINED

We now examine what determines these two stages for an individual token i. The balance between Self and aggregate Exchange is given exactly by log

gi = Ai − Si = LSEj<i (Eij ) − Si − λℓ log i. Fii

(10)

Equation (10) shows that the Self–Exchange balance is governed exactly by the gap Ai − Si . A larger Ai − Si shifts flow toward Exchange, whereas a smaller Ai − Si shifts flow toward Self. Once flow enters Exchange, the question changes from whether to use history to where in history to look. For any two historical positions j, k < i, Rij − Rik = Eij − Eik .

(11)

Because the logarithmic correction is shared by all historical positions in the row, it cancels in pairwise differences. The relative allocation within Exchange is therefore determined entirely by the Exchange relations Eij . Thus, for token i, Self versus aggregate Exchange is governed by Ai − Si , while allocation within history is governed by relative Exchange relations. 5

4.4

W HAT R ELATION C HANGES IN T OKEN M IXING

In Relation, Self and Exchange are explicitly constructed before normalization. Their distinct roles survive in the resulting information flow, even after normalization. Canonical Attention has no such relational stage: its pairwise scores are normalized directly into flow. Relation therefore makes relation formation a distinct stage of token mixing, with information flow derived afterward. We next turn from this relational organization to its practical realizations and evaluation in Sections 5 and 6.

R ELATION IN P RACTICE

The same Self–Exchange organization gives rise to a family of practical Relation forms. We call the explicit token-to-token operator defined in Sec. 3 Full Relation. FlashRelation is its exact fused execution, while Linear Relation represents historical Relation through a recurrent state. Hybrid Relation composes Full and Linear layers within a single decoder, and Relation Cache provides the autoregressive state used by Full Relation. 5.1

FACTORIZED S ELF –E XCHANGE : F LASH R ELATION

The Relation factorization derived in Sec. 4.2 separates historical Exchange aggregation from the final allocation between Self and Exchange. This factorization leads us to develop FlashRelation, a tiled realization of Full Relation. For token i, historical Exchange entries are processed through a causal tiled scan. For each tile B, we maintain a running maximum m, normalizer l, and information accumulator z:

′

m = max m, max Eij , j∈B X X ′ ′ ′ m−m′ Eij −m′ ′ l =e l+ e , z = em−m z + eEij −m Iej . j∈B

(12)

j∈B

After each tile, (m, l, z) ← (m′ , l′ , z ′ ). Let (mi , li , zi ) denote the terminal accumulators after the historical Exchange scan. They yield LE i = mi + log li ,

zi I¯iE = , li

(13)

¯E where LE i is the aggregate historical Exchange log-normalizer and Ii is the normalized historical information summary. The row is then completed by Ai = LE i − λℓ log i,

gi = σ(Ai − Si ),

Yi = (1 − gi )Iei + gi I¯iE .

(14)

Following the online-softmax tiling principle of FlashAttention (Dao et al., 2022), the historical Exchange scan is evaluated tile by tile without materializing the full R or F . 5.2

C HANGE E XCHANGE : L INEAR AND H YBRID R ELATION

Full Relation represents Exchange explicitly between the current token and individual historical tokens. Exchange can instead operate over a recurrent relational state that compresses individual historical Others, while Self remains tied to the current token. This leads to Linear Relation, with (h)

∈ Rdh ×dh .

Using the same projections introduced in Sec. 3.1, the current Self relation and the two normalized relation coordinates are 6

St = σ

p⊤ 1,t p2,t √ τS dh

p1,t p̂1,t = , ∥p1,t ∥2

! , (15)

p2,t p̂2,t = . ∥p2,t ∥2

Historical Relation is maintained with an input-dependent channel-wise retention. Linear Relation applies the same RoPE to p1,t and p2,t before computing St and the normalized relation coordinates. Following the retention parameterization of KDA (Kimi Team et al., 2025), rt = Wα↑ Wα↓ x̄t ,

gt = − exp(Alog ) ⊙ softplus(rt + dtbias ),

αt = exp(gt ) ∈ (0, 1]dh . (16)

With historical Others compressed into a recurrent state, the current token determines how that history is used through its Self relation. Linear Relation therefore follows the sequence Ask Self → Ask Others → Answer → Become Others . The recurrent Relation step is Ct− = Diag(αt )Ct−1 , Et = St (Ct− )⊤ p̂1,t , Yt = Iet + Et ,

(17)

Ct = Ct− + St p̂2,t Iet⊤ . Here, St determines the current Self relation and modulates both the read from historical Others and the write that makes the current token part of future Others. The read occurs before the current write, so Et has strictly historical support. Each token asks Self, consults Others, forms its answer, and then becomes Others for future tokens. The resulting token-mixing complexity and decode-state requirements are summarized in Table 1. Table 1: Complexity refers to the token-mixing core. Linear sequential operations refer to the direct recurrence. Layer Type

Token-Mixing Sequential Ops. Decode History Complexity State

MHA / FlashAttention O(T 2 d) Full / FlashRelation O(T 2 d) Linear Relation O(Td2 /H)

O(1) O(1) O(T )

O(T d) O(T d) O(d2 /H)

Full and Linear Relation provide complementary regimes. Linear Relation scales linearly with sequence length and uses a fixed-size decode state, while Full Relation retains O(1) sequential operations. Hybrid Relation combines the two by interleaving Linear and Full Relation layers. Following the hybrid layerwise design used in Kimi Linear (Kimi Team et al., 2025), our evaluated configuration uses nine Linear Relation layers and three Full Relation layers. 5.3

C ACHE H ISTORY: R ELATION C ACHE

During autoregressive decoding, Attention commonly caches the key and value states of previous tokens to avoid recomputing the full prefix (Shazeer, 2019; Pope et al., 2023). Relation admits an analogous projected cache. At each step, only the new Relation row is formed. From the pairwiseevidence definition in Sec. 3.1, each future row uses its own current P1 coordinate against historical P2 coordinates. Once a token’s row has been formed, its P1 state is therefore no longer reused, whereas its P2 state continues to participate in Relation construction for future tokens. Historical information states are likewise reused for transport. Therefore, 7

Ct = {P2,1:t , Ie1:t }.

(18)

At the next decoding step, only the new Relation row is constructed from the current P1 and the cached historical states.

E XPERIMENTS

We compare MHA and Full Relation under matched decoder geometry, data order, and training budget. Experiments cover approximately 10M, 30M, and 100M parameters and use paired seeds 42, 43, and 44. Scale-specific architectures, datasets, context lengths, and token budgets are summarized in Table 2. Our primary metric is final-checkpoint NLL on the full validation set, and structural ablations use the 10M setting with the same three seeds. All models use full-head RoPE with base 104 and no positional scaling. Systems benchmarks are conducted separately on an NVIDIA RTX 5090 in BF16 and report tuned steady-state throughput. Full optimization, evaluation, per-seed, and secondary benchmark details are provided in Appendices B, C, and E. 6.1

L ANGUAGE M ODELING AND S TRUCTURAL A BLATIONS

Across the three model scales, Full Relation achieves lower mean final-validation NLL than MHA, with improvements of 0.0412, 0.0151, and 0.0310 at 10M, 30M, and 100M parameters, respectively (Table 2). Table 2: Main validation results. NLL is mean ± sample standard deviation over three seeds. Backbone reports layers / dmodel / heads ×dh / dff . Scale Backbone

Data/Vocab

Ctx Tokens

MHA NLL ↓

Relation NLL ↓

∆

10M 6L/384/8×48/768 TinyStories/4K 1024 150M 1.6853 ± 0.0042 1.6441 ± 0.0124 −0.0412 30M 10L/512/8×64/1024 TinyStories/4K 2048 450M 1.3001 ± 0.0044 1.2850 ± 0.0136 −0.0151 100M 20L/640/8×80/1280 SmolLM/32K 4096 1.071B 2.9373 ± 0.0093 2.9063 ± 0.0061 −0.0310

We next examine several structural choices in Full Relation on the 10M setting (Table 3). Exchangeonly transport and Raw-X communication increase final-validation NLL, while removing count calibration or collapsing Relation to a single head produces larger degradations. We also evaluate No-Givens mixing (Ie = I). The exact ablation definitions and per-seed results are provided in Appendix D. Table 3: Structural ablations on the 10M setting. NLL is mean ± sample standard deviation over three seeds. Metric

Full Relation Exchange-only transport P Yi = Fij Iej j<i

Raw-X communication I=X

No count calibration λℓ = 0

No Givens mixing Ie = I

Single head H=1

NLL ↓ 1.6441 ± 0.0124 1.6761 ± 0.0097 1.6807 ± 0.0074 1.6947 ± 0.0203 1.6474 ± 0.0176 1.6948 ± 0.0221 ∆ vs Full — +0.0320 +0.0366 +0.0506 +0.0032 +0.0507

6.2

E FFICIENT I MPLEMENTATIONS

In a fixed-context T = 1024 reference benchmark, FlashRelation is 3.60–4.41× faster than the materialized Full Relation reference implementation. Across the three scale-matched production workloads, FlashRelation reaches 76.4–84.9% of PyTorch FlashAttention throughput while executing the exact Full Relation operator (Table 4). Same-micro throughput comparisons and Linear Relation throughput results are reported in Appendices H and I.

Table 4: FlashRelation systems results on an RTX 5090 in BF16. (a) Fixed-context (T = 1024) comparison with the materialized Full Relation reference. (b) Scale-matched production throughput against PyTorch FlashAttention. (a) Fixed-context T = 1024 Scale Reference Full FlashRelation 10M 30M 100M

128,515 74,376 30,631

(b) Scale-matched production Scale Context FlashAttention 10M 30M 100M

1024 2048 4096

Speedup

560,909 327,927 110,311

4.36× 4.41× 3.60×

FlashRelation

FR / FA

672,838 307,146 82,333

0.799× 0.849× 0.764×

841,834 361,793 107,783

Hybrid Relation combines nine Linear and three Full Relation layers in a (LLLF )3 layout. Under the 30M-class training setting, it achieves 1.2780 ± 0.0050 final-validation NLL across three seeds (Table 5). This demonstrates that Linear and Full Relation layers can be composed in a decoder in which nine of twelve token-mixing layers use Linear Relation. Table 5: Hybrid Relation under the 30M-class training setting. Model

Backbone

Layout

Params 3

Hybrid Relation 12L/480/8×60/960 (LLLF )

NLL ↓

31.97M 1.2780 ± 0.0050

L IMITATIONS AND C ONCLUSION

Limitations. Our experiments cover decoder-only language models up to approximately 100M parameters and training budgets up to 1.071B tokens. Linear and Hybrid Relation are evaluated in selected configurations, while Relation Cache is established mathematically. Behavior at substantially larger scales and in multimodal or post-training settings remains open. Conclusion. We introduced Relation as an alternative token-mixing primitive in which pairwise evidence is first organized into explicit Self and Exchange relations, and information flow is derived afterward. Within this framework, Full Relation retains explicit token-wise history, while FlashRelation provides exact factorized execution. Linear Relation compresses historical Others into a recurrent state, and Hybrid Relation combines Full and Linear layers. A KV-style Relation Cache follows for autoregressive decoding. Across three model scales, Full Relation achieves lower final validation NLL than matched MHA baselines, while FlashRelation provides practical execution and Hybrid Relation retains strong language-modeling quality with 75% Linear Relation layers. Together, these results support a broader view of token mixing. Token mixing need not be organized directly around information flow: it can begin with Relation.

R EFERENCES Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015. URL https://arxiv.org/abs/1409.0473. Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. In Advances in Neural Information Processing Systems, volume 35, pp. 8386–8399, 2022. doi: 10.52202/068431-0610. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 37a413841a614b5414b333585e7613b8-Abstract-Conference.html. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, volume 2024, pp. 35549–35562, 9

2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/ 98ed250b203d1ac6b24bbcf263e3d4a7-Paper-Conference.pdf. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pp. 16344–16359, 2022. doi: 10.52202/068431-1189. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html. Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 9099–9117. PMLR, 2022. URL https://proceedings.mlr.press/v162/hua22a.html. Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljačić. Tunable efficient unitary neural networks (EUNN) and their application to RNNs. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1733–1741. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/jing17a. html. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 5156–5165. PMLR, 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html. Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, and Yulun Du. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025. URL https://arxiv.org/abs/2510.26692. James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón. Fnet: Mixing tokens with fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4296–4313. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.naacl-main.319. URL https://aclanthology.org/2022.naacl-main.319/. Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, and James Henderson. Hypermixer: An mlp-based low cost alternative to transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15632–15654. Association for Computational Linguistics, 2023. doi: 10.18653/v1/ 2023.acl-long.871. URL https://aclanthology.org/2023.acl-long.871/. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, volume 5, pp. 606– 624, 2023. URL https://proceedings.mlsys.org/paper_files/paper/2023/ hash/c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html. Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=R8sQPpGCv0. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-totext transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL https: //jmlr.org/papers/v21/20-074.html. 10

Jason Ramapuram, Federico Danieli, Eeshan Gunesh Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, and Russell Webb. Theory, analysis, and best practices for sigmoid self-attention. In International Conference on Learning Representations, volume 2025, pp. 65464–65519, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/ a43b459e9fab3a703148ba0c83b8a442-Paper-Conference.pdf. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464–468, 2018. doi: 10.18653/v1/N18-2074. URL https://aclanthology.org/ N18-2074/. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. URL https://arxiv.org/abs/1911.02150. Noam Shazeer, Zhenzhong Lan, Youlong Cheng, Nan Ding, and Le Hou. Talking-heads attention. arXiv preprint arXiv:2003.02436, 2020. URL https://arxiv.org/abs/2003.02436. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023. URL https://arxiv.org/abs/2307.08621. Chao-Hong Tan, Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, and Zhen-Hua Ling. Ponet: Pooling network for efficient token mixing in long sequences. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id= 9jInD9JjicF. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pp. 5998–6008, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 56501–56523. PMLR, 2024. URL https: //proceedings.mlr.press/v235/yang24ab.html. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, pp. 29687–29707, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 4904fad153f6434a7bcf04465d4be2cc-Abstract-Conference.html.

D ETAILED O PERATOR D ERIVATIONS

This appendix records the operator definitions used throughout the paper. Token indices in the count correction are one-based. A head has width dh , and the information state after input projection and e head mixing is denoted by I. A.1

S ELF –E XCHANGE R ELATION

A.1.1

R ELATION CONSTRUCTION

We first write the construction for one head. If the model-wide projection has width d, the following quantities are understood after selecting one head subspace of width dh = d/H. Thus P1 , P2 , I ∈ RT ×dh ,

P1 = XW1 ,

P2 = XW2 ,

I = XWI .

The head-wise P1 and P2 entering U use full-head RoPE, whereas I remains unrotated. Writing p1,i and p2,j for rows of these projected states, p⊤ 1,i p2,j Uij = √ . dh The Self and Exchange entries are Uii Si = σ , τS

Eij = SiLU(Uij ),

τS > 0,

τS = 2.

Here τS is the Self temperature used to set the scale and smoothness of the bounded Self mapping. The count-calibration parameter λℓ is one unconstrained FP32 scalar per layer, shared across heads and initialized to 0.5. Causality is imposed when these role-specific entries are assembled:  j = i, Si , Rij = Eij − λℓ log i, j < i,  −∞, j > i. A.1.2

E XACT ROW NORMALIZER

The normalized row is Fi = Softmax(Ri ). For i > 1, its normalizer is X Zi = eRik k≤i

= eSi +

eEij −λℓ log i

j<i Si

+ e−λℓ log i

eEij

j<i

= eSi + i−λℓ

eEij .

j<i

Define the historical Exchange log-normalizer and its count-corrected version by X eEij , Ai = LE LE i = LSEj<i (Eij ) = log i − λℓ log i. j<i

Then eAi = e−λℓ log i

eEij ,

j<i

Zi = eSi + eAi .

A.1.3

S ELF MASS AND AGGREGATE E XCHANGE MASS

The diagonal Self probability is eSi eSi = Si Zi e + e Ai 1 = σ(Si − Ai ) . = 1 + eAi −Si The total historical Exchange mass is X eAi gi = Fij = Si e + eAi j<i Fii =

1 = σ(Ai − Si ) . 1 + eSi −Ai

Consequently, Fii + gi = 1 . The Self–Exchange odds are eAi gi = Si = eAi −Si , Fii e and therefore log A.1.4

gi = Ai − Si = LE i − Si − λℓ log i. Fii

C ONDITIONAL E XCHANGE ALLOCATION

For a historical position j < i, define its conditional Exchange allocation as Fij E πij = . gi Substituting the normalized row and the aggregate Exchange mass gives eEij −λℓ log i /Zi E πij = eAi /Zi eEij −λℓ log i =X eEik −λℓ log i k<i Eij

e =X

eEik

k<i

The row-level count correction cancels because it is identical for every historical position. Hence X E E πij = 1, πij = Softmaxj<i (Eij ) , j<i

and the original historical flow entries factor as E Fij = gi πij .

A.1.5

C OUNT CORRECTION

For any two historical positions j, k < i, Rij − Rik = (Eij − λℓ log i) − (Eik − λℓ log i) = Eij − Eik . Equivalently, E πij = eEij −Eik . E πik Thus −λℓ log i is a row-level Exchange translation. It changes the Self–Exchange mass competition through Ai − Si , but it does not change conditional Exchange allocation.

A.1.6

E XACT INFORMATION TRANSPORT

Define the normalized historical information summary X Ee I¯iE = πij Ij . j<i

The original Full Relation transport is Yi =

Fij Iej

j≤i

= Fii Iei +

Fij Iej

j<i

= Fii Iei + gi

Ee πij Ij

j<i

= (1 − gi )Iei + gi I¯iE . This is an exact algebraic factorization of the original Full Relation transport, not a new operator. The residual output remains Xℓ+1 = Xℓ + Y WO . A.1.7

B OUNDARY CASE

For the first token, the historical set is empty. Define LSEj<1 = −∞,

A1 = −∞.

It follows directly that g1 = 0, A.2 A.2.1

Y1 = Ie1 .

F11 = 1,

M ULTI –H EAD R ELATION H EADWISE CONSTRUCTION

Let dh = d/H. After projection and head splitting, (h)

(h)

P1 , P2 , I (h) ∈ RT ×dh ,

h = 1, . . . , H.

Each head independently applies the Self–Exchange construction of Appendix A.1, yielding R(h) ,

F (h) ,

Y (h) .

Relation construction is head-local and cross-head interaction is applied only to the information branch. A.2.2

G IVENS BLOCK

For an adjacent head pair (a, b), define cos θ G(θ) = sin θ

− sin θ cos θ .

The information states are mixed by (a) (a) Ie I = G(θℓ,ab ) (b) . I Ie(b) Since

G(θ)⊤ G(θ) = I2 , the transformation is orthogonal. Applied to every information channel of the pair, Ie(a)

2 2

+ Ie(b)

= I (a)

2 2

+ I (b)

. 2

The same scalar angle θℓ,ab acts on the dh information channels associated with the pair. 14

A.2.3

A LTERNATING PAIRING

The formal construction uses an even number of heads, and in all formal experiments, H = 8. With one-based head indices, the two pairings are MA = {(1, 2), (3, 4), . . . , (H − 1, H)},

MB = {(2, 3), (4, 5), . . . , (H, 1)}.

They alternate across layers: MA −→ MB −→ MA −→ · · · . For H = 8, MA = {(1, 2), (3, 4), (5, 6), (7, 8)},

MB = {(2, 3), (4, 5), (6, 7), (8, 1)}.

There are H/2 = 4 learnable angles per layer. They are stored in FP32 and initialized to zero. A.2.4

M ULTI - HEAD TRANSPORT

After the role-specific flow is formed for each head, Y (h) = F (h) Ie(h) . The head outputs are concatenated and projected: Y = Concat Y (1) , . . . , Y (H) ,

Xℓ+1 = Xℓ + Y WO .

Thus the Relation construction remains head-local, Givens rotations act on the information branch, and WO combines the head outputs. A.3 A.3.1

F LASH R ELATION H ISTORICAL E XCHANGE INVARIANT

FlashRelation evaluates the exact factorization of Appendix A.1 without materializing the full T ×T matrices. For a fixed query row i, let J ⊆ {1, . . . , i − 1} be the historical indices already scanned. The historical Exchange state is defined by the invariant X X m = max Eij , l= eEij −m , z= eEij −m Iej . j∈J

j∈J

This scan contains Exchange entries only and Self is not included in (m, l, z). For the empty set, m = −∞, A.3.2

l = 0,

z = 0.

A DDING ONE TILE

Let B be the next causal tile and set m′ = max(m, mB ).

mB = max Eij , j∈B

The previously scanned terms are rescaled into the new maximum: X ′ ′ X eEij −m = em−m eEij −m j∈J

j∈J m−m′

Consequently, the updated normalizer is ′

l′ = em−m l +

′

eEij −m .

j∈B

The same rescaling gives the information accumulator X ′ ′ z ′ = em−m z + eEij −m Iej . j∈B

Together with m′ = max(m, mB ), these updates preserve the invariant with J ← J ∪ B. 15

A.3.3

T ERMINAL ACCUMULATORS

After all historical positions j < i have been scanned, X mi = max Eij , li = eEij −mi , j<i

zi =

eEij −mi Iej .

j<i

The terminal log-normalizer is mi + log li = mi + log

eEij −mi

j<i



 = log emi

eEij −mi 

j<i

= log

eEij

j<i

= A.3.4

LE i .

H ISTORICAL INFORMATION SUMMARY

The terminal accumulator gives X

eEij −mi Iej

zi j<i = X li eEik −mi k<i

eEij Iej

j<i

= X

eEik

k<i

Ee πij Ij .

j<i

The common factor e

−mi

cancels. Therefore zi = I¯iE . li

A.3.5

R ECOVERING THE F ULL R ELATION ROW

The row-level count correction is applied after the historical scan: Ai = mi + log li − λℓ log i, The final transport is

gi = σ(Ai − Si ).

zi Yi = (1 − gi )Iei + gi . li

Using the identities in Appendix A.1, YiFlashRelation = YiFullRelation . So FlashRelation changes the execution of the exact operator. A.3.6

M ATERIALIZATION AND BOUNDARY

FlashRelation does not materialize full T × T U , R, or F matrices. The required pairwise evidence and Exchange entries are formed and consumed tile by tile. The historical state is maintained as running statistics. 16

A.4 A.4.1

L INEAR R ELATION R ELATION COORDINATES AND RETENTION

In the equations below, p1,t and p2,t denote the projected relation coordinates after the formal RoPE step. The positional order is P1 /P2 −→ RoPE −→ St −→ L2Norm −→ recurrence. Thus ! p⊤ p1,t p2,t 1,t p2,t √ St = σ , p̂1,t = , p̂2,t = . ∥p1,t ∥2 ∥p2,t ∥2 τS d h The retention parameterization is rt = Wα↑ Wα↓ x̄t , gt = − exp(Alog ) ⊙ softplus(rt + dtbias ), αt = exp(gt ). Because exp(Alog ) > 0 elementwise and softplus(·) > 0, 0 < αt,c ≤ 1 .

gt < 0,

Here St is the Self Relation, whereas αt is channel-wise historical retention. A.4.2

R ECURRENT STEP

Let Dt = Diag(αt ),

C0 = 0 .

The recurrent step is Ct− = Dt Ct−1 , Et = St (Ct− )⊤ p̂1,t , Yt = Iet + Et , Ct = Ct− + St p̂2,t Iet⊤ . The execution order is Ask Self −→ Ask Others −→ Answer −→ Become Others. A.4.3

S TRICT HISTORICAL SUPPORT

The read uses Ct− = Dt Ct−1 , while the current write St p̂2,t Iet⊤ is added only after Et has been formed. Hence Et depends only on positions 1, . . . , t − 1, and the current token cannot enter its own historical read through the current write. A.4.4

U NROLLING THE RECURRENT STATE

Starting from Ct = Dt Ct−1 + St p̂2,t Iet⊤ , one expansion gives ⊤ Ct = Dt Dt−1 Ct−2 + St−1 p̂2,t−1 Iet−1 + St p̂2,t Iet⊤ ⊤ = Dt Dt−1 Ct−2 + Dt St−1 p̂2,t−1 Iet−1 + St p̂2,t Iet⊤ .

With the ordered product t Y

Du ≡ Dt Dt−1 · · · Dj+1 ,

u=j+1

and the empty product equal to the identity,   t X Y  Ct = Du  Sj p̂2,j Iej⊤ . j≤t

u=j+1

Thus each token contributes Sj p̂2,j Iej⊤ and is subsequently transformed by Dj+1 , . . . , Dt . 17

A.4.5

E XPANDED HISTORICAL E XCHANGE

The pre-write state is  Ct− =



t Y

Du  Sj p̂2,j Iej⊤ .

 j<t

u=j+1

Substituting this into the read gives Et = St (Ct− )⊤ p̂1,t     t X Y  = St Sj p̂⊤ Du  p̂1,t  Iej . 2,j j<t

u=j+1

Because each Du is diagonal, the transpose leaves the ordered product unchanged. A.4.6

I NFORMATION WRITE AND COMPLEXITY

The output is Yt = Iet + Et , but the state writes

St p̂2,t Iet⊤ , not St p̂2,t Yt⊤ . Each new state contribution therefore corresponds to the current token’s own information rather than recursively storing the information already read from history. (h)

For one head, Ct

∈ Rdh ×dh . The principal recurrent operations per token are Dt Ct−1 , (Ct− )⊤ p̂1,t , p̂2,t Ie⊤ , t

with leading cost O(d2h ) per token per head. Across H heads, 2 ! 2 d Td 2 O(T Hdh ) = O T H = O . H H The recurrent state size is 2 d d2 O(d2 /H) = , H H for the decode state. These complexities refer to the recurrent Relation token-mixing core and exclude common FFN and embedding costs. Hd2h = H

A.5 A.5.1

R ELATION C ACHE W HAT THE NEXT ROW NEEDS

At autoregressive position t + 1, the new pairwise evidence row is p⊤ 1,t+1 p2,j √ , j ≤ t + 1. dh For historical positions j ≤ t, the new row needs the cached p2,j states. Historical p1,j states do not participate in future row construction, so P1,1:t need not be retained. Ut+1,j =

A.5.2

W HAT TRANSPORT NEEDS

The new position transports Yt+1 =

Ft+1,j Iej .

j≤t+1

Thus the historical information states Ie1:t must also remain available. The exact projected cache is n o Ct = P2,1:t , Ie1:t . 18

A.5.3

C ACHE UPDATE

After position t + 1 has been processed, its projected historical coordinate and information state are appended: n o Ct+1 = Ct ∪ P2,t+1 , Iet+1 . At the following step, only the new current states P1,t+2 , P2,t+2 , and Iet+2 are formed, and the historical states P2,1:t+1 , Ie1:t+1 are read from the cache. A.5.4

O BJECTS NOT CACHED AND CACHE SIZE

The exact autoregressive dependency does not require retaining P1,1:t ,

U1:t,1:t ,

R1:t,1:t ,

F1:t,1:t .

Only P2 and Ie are needed as historical projected states for the next row and its transport. The projected cache stores two d-wide states per historical token, so its size grows as O(T d) .

E XPERIMENTAL S ETUP AND E VALUATION

B.1

M ODEL C ONFIGURATIONS

All models use full-head RoPE with base 104 . The three main scales use paired seeds 42, 43, 44. Table 6: Formal decoder-only model configurations. Global tokens per optimizer update are 131,072 for all three scales.

B.2

Scale

Params MHA/Rel.

10M 30M 100M

10,425,216/10,425,246 28,322,304/28,322,354 102,917,760/102,917,860

L/d/H × dh /dff

Vocab

6/384/8×48/768 10/512/8×64/1024 20/640/8×80/1280

4K 4K 32K

Data TinyStories TinyStories SmolLM corpus

Train

Micro/GA

1024 2048 4096

150M 450M 1.071B

32/4 16/4 MHA 2/16; Rel. 4/8

O PTIMIZATION AND C HECKPOINTS

Training uses BF16 and AdamW with betas (0.9, 0.95), weight decay 0.1, gradient clipping at 1.0, and a WSD80 schedule. Learning rates are 10−3 , 8 × 10−4 , and 6 × 10−4 for 10M, 30M, and 100M. Warmup tokens are 1.5M, 4.5M, and 10.7114M. Formal NLL is evaluated on the full validation set at the final checkpoint. B.3

P OSITION E NCODING

RoPE covers the full head dimension. The base is 104 . For MHA, RoPE is applied to Q and K after projection and head splitting. For Full Relation and FlashRelation, RoPE is applied to P1 and P2 before pairwise evidence. The information projection and Givens output remain unrotated. Linear Relation applies RoPE to its two relation coordinates before Self and normalized coordinate formation. Hybrid layers use the corresponding Full or Linear path. B.4

E VALUATION P ROTOCOL

The primary metric is final-checkpoint NLL. Formal BLiMP uses 67 categories and 67,000 examples per seed. Secondary evaluations use the final checkpoints and the same three seeds. Systems measurements report complete optimizer-step throughput. One-time initialization and JIT, warmup, and terminal partial steps are excluded when present. 19

P ER -S EED L ANGUAGE -M ODEL R ESULTS

C.1

F INAL VALIDATION NLL

Table 7 reports final validation NLL for all paired seeds. Aggregate rows use the sample standard deviation over three seeds. Table 7: Final validation NLL for the three paired seeds.

C.2

Scale

Seed

10M 10M 10M 10M 30M 30M 30M 30M 100M 100M 100M 100M

42 1.688823 1.631738 43 1.686405 1.643978 44 1.680626 1.656633 Mean ± SD 1.685285 ± 0.004212 1.644117 ± 0.012448 42 1.295263 1.300730 43 1.303885 1.277177 44 1.301160 1.277225 Mean ± SD 1.300103 ± 0.004407 1.285044 ± 0.013584 42 2.945050 2.912234 43 2.939826 2.899965 44 2.926899 2.906657 Mean ± SD 2.937258 ± 0.009344 2.906285 ± 0.006143

MHA NLL

Full Relation NLL Full–MHA -0.057085 -0.042427 -0.023993 -0.041168 +0.005467 -0.026708 -0.023935 -0.015059 -0.032816 -0.039861 -0.020242 -0.030973

AGGREGATED L ANGUAGE -M ODEL R ESULTS

Full Relation wins all three paired comparisons at 10M, two of three at 30M, and all three at 100M, for eight wins out of nine paired comparisons.

S TRUCTURAL A BLATION D ETAILS

D.1

A BLATION D EFINITIONS

All structural ablations use the 10M setting, the same prepared data, and seeds 42, 43, and 44. Each ablation changes only the named component of Full Relation. • Exchange-only transport computes the complete Full Relation flow and transports only P historical entries: Yi = j<i Fij Iej . Historical weights are not renormalized. • Raw-X communication replaces the independent information projection by I = X. Head splitting, Givens mixing, Flow, and WO remain unchanged. • No count calibration sets λℓ = 0. • No Givens mixing bypasses adjacent-head rotation, so Ie = I, while the Relation and transport path remains unchanged. • Single head uses H = 1. D.2

P ER -S EED A BLATION R ESULTS Table 8: Per-seed final validation NLL for the structural ablations.

Ablation

Seed 42

Seed 43

Seed 44

Mean ± SD

∆ vs Full

Full Relation Exchange-only transport Raw-X communication No count calibration No Givens mixing Single head

1.631738 1.665658 1.672332 1.671373 1.630321 1.669349

1.643978 1.677815 1.686600 1.704531 1.646350 1.707688

1.656633 1.684822 1.683090 1.708256 1.665386 1.707422

1.644117 ± 0.012448 1.676099 ± 0.009697 1.680674 ± 0.007434 1.694720 ± 0.020305 1.647352 ± 0.017554 1.694820 ± 0.022059

0.000000 +0.031982 +0.036557 +0.050604 +0.003236 +0.050703

S ECONDARY B ENCHMARKS

The secondary evaluations below use final checkpoints and the same three seeds as the languagemodel experiments. Accuracy is a fraction of examples. For LAMBADA, perplexity and mean target NLL are lower-is-better metrics. 20

E.1

AGGREGATED R ESULTS Table 9: Formal BLiMP results over 67 categories and 67,000 examples per seed. Scale

Seed

MHA BLiMP

Full Relation BLiMP Full–MHA

10M 10M 10M 10M 30M 30M 30M 30M 100M 100M 100M 100M

42 0.572567 0.588746 43 0.572299 0.581418 44 0.571746 0.579463 Mean ± SD 0.572204 ± 0.000419 0.583209 ± 0.004894 42 0.598075 0.614806 43 0.621657 0.618134 44 0.622328 0.625806 Mean ± SD 0.614020 ± 0.013813 0.619582 ± 0.005641 42 0.750433 0.745955 43 0.770299 0.766030 44 0.765090 0.749403 Mean ± SD 0.761940 ± 0.010300 0.753796 ± 0.010734

+0.016179 +0.009119 +0.007716 +0.011005 +0.016731 -0.003522 +0.003478 +0.005562 -0.004478 -0.004269 -0.015687 -0.008144

Table 10: Aggregated results for overall-accuracy task families. Task

Scale

MHA mean ± SD

MHR mean ± SD

MHR–MHA

blimp supplement blimp supplement blimp supplement comps comps comps entity tracking entity tracking entity tracking

10M 30M 100M 10M 30M 100M 10M 30M 100M

0.570190 ± 0.013869 0.553051 ± 0.016250 0.578655 ± 0.014846 0.530813 ± 0.012635 0.558654 ± 0.009019 0.628584 ± 0.004908 0.210176 ± 0.058427 0.363665 ± 0.050151 0.291394 ± 0.030456

0.569460 ± 0.012034 0.572745 ± 0.014946 0.594045 ± 0.009545 0.539954 ± 0.002803 0.556384 ± 0.008536 0.615241 ± 0.019606 0.229008 ± 0.045517 0.335445 ± 0.080940 0.328333 ± 0.009184

-0.000730 0.019693 0.015390 0.009141 -0.002270 -0.013343 0.018832 -0.028219 0.036939

Wins

2 3 3 2 0 0 2 2 3

3 3 3 3 3 3 3 3 3

Overall-accuracy task families. Table 11: Aggregated B-group metrics across the three seeds. Task

Scale

Split

Metric

arc challenge arc challenge arc challenge arc challenge arc challenge arc challenge openbookqa openbookqa openbookqa openbookqa openbookqa openbookqa lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard lambada standard

10M 10M 30M 30M 100M 100M 10M 10M 30M 30M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M 100M 100M 100M 100M 100M 100M

– – – – – – – – – – – – test test test validation validation validation test test test validation validation validation test test test validation validation validation

acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm accuracy perplexity mean target nll accuracy perplexity mean target nll accuracy perplexity mean target nll accuracy perplexity mean target nll accuracy perplexity mean target nll accuracy perplexity mean target nll

MHA mean ± SD

MHR mean ± SD

MHR–MHA

Wins

0.208473 ± 0.012662 0.261984 ± 0.020435 0.210702 ± 0.010033 0.247492 ± 0.008849 0.212932 ± 0.013517 0.241918 ± 0.006962 0.131333 ± 0.007572 0.169333 ± 0.011719 0.120000 ± 0.005292 0.174000 ± 0.015620 0.162667 ± 0.005033 0.286000 ± 0.011136 0.038812 ± 0.003498 539.084666 ± 194.121401 6.247946 ± 0.351631 0.038817 ± 0.004040 575.877730 ± 215.622529 6.311539 ± 0.359425 0.070315 ± 0.001954 133.042252 ± 22.776937 4.881056 ± 0.169080 0.074279 ± 0.003618 135.497543 ± 21.161153 4.900933 ± 0.154586 0.147358 ± 0.008834 21.944875 ± 0.824615 3.088069 ± 0.037225 0.151434 ± 0.010014 20.610276 ± 0.910404 3.025144 ± 0.043930

0.201784 ± 0.011745 0.254181 ± 0.017697 0.198439 ± 0.006962 0.221851 ± 0.010218 0.222965 ± 0.007724 0.237458 ± 0.005793 0.127333 ± 0.013013 0.164000 ± 0.007211 0.112000 ± 0.010583 0.170000 ± 0.012000 0.157333 ± 0.004619 0.286000 ± 0.003464 0.045669 ± 0.003888 488.769003 ± 25.590314 6.190993 ± 0.051624 0.048470 ± 0.003651 525.337625 ± 38.883730 6.262198 ± 0.074529 0.066951 ± 0.000846 202.971570 ± 68.930202 5.266584 ± 0.390220 0.071062 ± 0.002763 210.832091 ± 71.159280 5.305134 ± 0.387909 0.150463 ± 0.006235 21.292699 ± 0.434628 3.058226 ± 0.020297 0.161087 ± 0.005270 19.749080 ± 0.547300 2.982852 ± 0.027619

-0.006689 -0.007804 -0.012263 -0.025641 0.010033 -0.004459 -0.004000 -0.005333 -0.008000 -0.004000 -0.005333 -0.000000 0.006857 -50.315663 -0.056953 0.009653 -50.540104 -0.049341 -0.003364 69.929318 0.385528 -0.003218 75.334548 0.404201 0.003105 -0.652177 -0.029842 0.009653 -0.861196 -0.042292

1 0 1 0 3 0 1 1 1 1 1 1 3 2 2 3 1 1 0 0 0 0 0 0 2 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

B-group metrics. Hard-benchmark metrics.

Table 12: Aggregated hard-benchmark metrics across the three seeds.

E.2

Task

Scale

Metric

hellaswag hellaswag hellaswag hellaswag hellaswag hellaswag piqa piqa piqa piqa piqa piqa arc easy arc easy arc easy arc easy arc easy arc easy

10M 10M 30M 30M 100M 100M 10M 10M 30M 30M 100M 100M 10M 10M 30M 30M 100M 100M

acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm acc acc norm

MHA mean ± SD

MHR mean ± SD

MHR–MHA

0.256556 ± 0.001880 0.262929 ± 0.002150 0.266813 ± 0.001253 0.272024 ± 0.001322 0.277136 ± 0.002072 0.291343 ± 0.002961 0.535183 ± 0.004155 0.521763 ± 0.004318 0.542437 ± 0.002372 0.522488 ± 0.006889 0.593761 ± 0.009978 0.585963 ± 0.008233 0.269006 ± 0.015328 0.239766 ± 0.007304 0.276023 ± 0.023559 0.256725 ± 0.017308 0.440936 ± 0.002680 0.360234 ± 0.007090

0.257684 ± 0.000997 0.264722 ± 0.000663 0.266614 ± 0.001092 0.272124 ± 0.002409 0.279725 ± 0.000434 0.291044 ± 0.001597 0.529742 ± 0.005477 0.510519 ± 0.009174 0.550417 ± 0.009584 0.534458 ± 0.011635 0.595575 ± 0.007545 0.588683 ± 0.007903 0.267836 ± 0.008654 0.256140 ± 0.001754 0.271345 ± 0.003652 0.260819 ± 0.021936 0.437427 ± 0.004415 0.363158 ± 0.006077

0.001129 0.001792 -0.000199 0.000100 0.002589 -0.000299 -0.005441 -0.011244 0.007980 0.011970 0.001814 0.002720 -0.001170 0.016374 -0.004678 0.004094 -0.003509 0.002924

Wins

3 2 0 1 3 2 0 0 2 2 1 2 1 3 1 2 1 2

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

P ER -S EED S ECONDARY R ESULTS

The following tables report the per-seed results for all nine task families. Each row gives the final checkpoint result for one model, scale, and seed. Table 13: Per-seed arc challenge accuracy results. Scale

Model Seed

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

acc acc norm 0.197324 0.220736 0.220736 0.214047 0.227425 0.227425 0.193980 0.217391 0.214047 0.214047 0.200669 0.190635 0.200669 0.220736 0.210702 0.204013 0.190635 0.200669

0.244147 0.247492 0.234114 0.234114 0.244147 0.234114 0.244147 0.284281 0.257525 0.240803 0.274247 0.247492 0.240803 0.244147 0.257525 0.210702 0.230769 0.224080

Table 14: Per-seed arc easy accuracy results. Scale

Model Seed

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

acc acc norm 0.443860 0.438596 0.440351 0.433333 0.436842 0.442105 0.256140 0.285965 0.264912 0.271930 0.273684 0.257895 0.292982 0.249123 0.285965 0.275439 0.270175 0.268421

0.361404 0.352632 0.366667 0.370175 0.359649 0.359649 0.231579 0.245614 0.242105 0.257895 0.256140 0.254386 0.264912 0.236842 0.268421 0.285965 0.245614 0.250877

Table 15: Per-seed blimp supplement overall-accuracy results. Scale

Model Seed Overall accuracy Examples

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

0.570344 0.595795 0.569827 0.587775 0.605030 0.589331 0.582886 0.555388 0.572296 0.583166 0.560625 0.564588 0.546929 0.540752 0.571473 0.555547 0.582587 0.580101

5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218 5218

Table 16: Per-seed comps overall-accuracy results. Scale

Model Seed Overall accuracy Examples

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

0.623166 0.629854 0.632732 0.592602 0.626409 0.626713 0.544447 0.528496 0.519497 0.540799 0.536826 0.542238 0.567714 0.558573 0.549676 0.564775 0.556668 0.547710

49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340 49340

Table 17: Per-seed entity tracking overall-accuracy results. Scale

Model Seed Overall accuracy Examples

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

0.288467 0.262507 0.323208 0.324787 0.321449 0.338761 0.241183 0.246563 0.142782 0.179152 0.268344 0.239527 0.378111 0.307876 0.405007 0.407819 0.350471 0.248045

9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483 9483

Table 18: Per-seed hellaswag accuracy results. Scale

Model Seed

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

acc acc norm 0.278431 0.274746 0.278231 0.279526 0.279426 0.280223 0.258614 0.254929 0.256124 0.258813 0.256921 0.257319 0.267975 0.266979 0.265485 0.267477 0.266979 0.265385

0.294762 0.289584 0.289683 0.289385 0.292571 0.291177 0.260904 0.262697 0.265186 0.264390 0.265485 0.264290 0.271261 0.271261 0.273551 0.271261 0.270265 0.274846

Table 19: Per-seed LAMBADA Standard results. Scale

Model

Seed

Test acc

Test PPL

Test NLL

Val acc

Val PPL

Val NLL

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

0.147681 0.138366 0.156026 0.152727 0.143412 0.155249 0.035901 0.037842 0.042694 0.050068 0.042694 0.044246 0.072385 0.068504 0.070056 0.067339 0.067533 0.065981

21.5936 22.8870 21.3541 21.0142 21.7935 21.0704 375.7672 753.7046 487.7822 472.5370 518.2685 475.5015 112.6453 157.6199 128.8616 123.5239 246.8694 238.5214

3.0724 3.1306 3.0612 3.0452 3.0816 3.0479 5.9290 6.6250 6.1899 6.1581 6.2505 6.1644 4.7242 5.0602 4.8587 4.8164 5.5089 5.4745

0.158965 0.140070 0.155268 0.165537 0.155268 0.162456 0.034504 0.039433 0.042514 0.051345 0.049702 0.044362 0.078456 0.072294 0.072089 0.074143 0.070240 0.068803

19.8012 21.5961 20.4335 19.2588 20.3395 19.6489 405.5814 818.3232 503.7286 484.8567 562.3982 528.7579 116.3114 158.1946 131.9867 128.6658 252.3256 251.5048

2.9857 3.0725 3.0172 2.9580 3.0126 2.9780 6.0053 6.7073 6.2220 6.1839 6.3322 6.2705 4.7563 5.0638 4.8827 4.8572 5.5307 5.5275

Table 20: Per-seed openbookqa accuracy results. Scale

Model Seed

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

acc acc norm 0.162000 0.168000 0.158000 0.160000 0.152000 0.160000 0.128000 0.126000 0.140000 0.140000 0.114000 0.128000 0.116000 0.126000 0.118000 0.104000 0.108000 0.124000

0.298000 0.284000 0.276000 0.290000 0.284000 0.284000 0.156000 0.178000 0.174000 0.162000 0.172000 0.158000 0.182000 0.184000 0.156000 0.182000 0.158000 0.170000

Table 21: Per-seed piqa accuracy results.

H YBRID R ELATION

F.1

C ONFIGURATION

Scale

Model Seed

100M 100M 100M 100M 100M 100M 10M 10M 10M 10M 10M 10M 30M 30M 30M 30M 30M 30M

MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR MHA MHA MHA MHR MHR MHR

42 43 44 42 43 44 42 43 44 42 43 44 42 43 44 42 43 44

acc acc norm 0.605005 0.585963 0.590316 0.593580 0.603917 0.589227 0.539717 0.531556 0.534276 0.534820 0.530468 0.523939 0.540805 0.541349 0.545158 0.544614 0.561480 0.545158

0.595212 0.583243 0.579434 0.581066 0.596844 0.588139 0.520131 0.518498 0.526659 0.514690 0.516866 0.500000 0.529924 0.516322 0.521219 0.521763 0.544614 0.536997

Hybrid Relation uses a 12-layer, 480-dimensional decoder with eight heads of width 60 and dff = 960. Its layout is (LLLF )3 , with nine Linear Relation layers and three Full Relation layers. The vocabulary has 4,096 entries, context length is 2,048, and training budget is 450M tokens. The model has 31,967,571 parameters and uses micro/GA 16/4, BF16, AdamW, and 131,072 global tokens per update. This is a 30M-class configuration and is not geometry-identical to the 30M Full Relation model. F.2

P ER -S EED R ESULTS Table 22: Per-seed Hybrid Relation results. Seed

Final NLL

Final BLiMP

42 1.274777 0.620134 43 1.283791 0.637343 44 1.275435 0.631582 Mean ± SD 1.278001 ± 0.005025 0.629687 ± 0.008760

F.3

S ECONDARY R ESULTS

The per-seed Hybrid BLiMP values are shown in Table 22. No additional Hybrid benchmark is used here.

S YSTEMS B ENCHMARK P ROTOCOL

G.1

H ARDWARE AND S OFTWARE

Systems measurements use an NVIDIA RTX 5090 GPU in BF16. The recorded software environment is CUDA 13.0, PyTorch 2.12.1+cu130, and Triton 3.7.1. Each campaign uses its formal prepared data and frozen model geometry. G.2

T IMING AND ACCOUNTING

One-time initialization and JIT compilation are outside the reported interval. Warmup optimizer steps are excluded. Throughput uses complete optimizer steps, with the terminal partial token step excluded where the campaign reaches a non-integral final update. The denominator is the number of tokens consumed by the complete measured optimizer steps. 25

G.3

C AMPAIGN S ETTINGS

The fixed-context Reference–FlashRelation campaign uses T = 1024 for all scales. The scalematched FlashRelation–FlashAttention campaign uses T = 1024, 2048, 4096 for 10M, 30M, and 100M. The same-micro campaign uses T = 1024 for every scale, shared micro/GA at each point, and explicitly forced FlashAttention. Linear long-context scaling uses micro = 1, 131,072 effective tokens per update, five warmup steps, and ten measured complete steps.

F LASH R ELATION S YSTEMS R ESULTS

H.1

S CALE -M ATCHED F LASH R ELATION AND F LASH ATTENTION

FlashAttention is explicitly forced in these scale-matched production records. FlashRelation uses the final exact production dispatch for each head dimension. Table 23: Scale-matched production FlashAttention and FlashRelation throughput; memory is allocated/reserved GiB.

H.2

Scale

FlashAttention tok/s

FlashRelation tok/s

FR/FA

10M 30M 100M

1024 2048 4096

841,833.930 361,793.115 107,783.242

672,838.258 307,146.176 82,332.578

0.799253× 0.848955× 0.763872×

FR micro/GA

FA micro/GA

FR alloc./res.

FA alloc./res.

64/2 16/4 4/8

32/4 16/4 4/8

10.26/11.31 9.72/10.15 17.32/18.24

5.38/5.88 10.03/10.45 17.71/18.63

C OMPLETE S AME -M ICRO S CALING

The tables below retain every micro point from the matched-micro campaign. FlashAttention is explicitly forced for all scales. Both methods use the same micro and gradient accumulation at each point, with T = 1024 and 131,072 effective tokens per update. OOM rows are retained as measured outcomes. Table 24: Complete same-micro scaling for 10M; FlashAttention is forced. Method

Micro/GA

FlashAttention 1/128 FlashRelation 1/128 FlashAttention 2/64 FlashRelation 2/64 FlashAttention 4/32 FlashRelation 4/32 FlashAttention 8/16 FlashRelation 8/16 FlashAttention 16/8 FlashRelation 16/8 FlashAttention 32/4 FlashRelation 32/4 FlashAttention 64/2 FlashRelation 64/2 FlashAttention 128/1 FlashRelation 128/1

tok/s

FR/FA

∆ Alloc./res. GiB Status

30,706.567 0.658086 -0.341914 20,207.558 0.658086 -0.341914 63,454.977 0.997759 -0.002241 63,312.789 0.997759 -0.002241 121,247.542 0.693854 -0.306146 84,128.151 0.693854 -0.306146 242,675.755 0.672295 -0.327705 163,149.784 0.672295 -0.327705 460,243.223 0.790958 -0.209042 364,033.195 0.790958 -0.209042 835,322.853 0.725787 -0.274213 606,266.514 0.725787 -0.274213 769,418.196 0.877894 -0.122106 675,467.374 0.877894 -0.122106 754,729.251 0.877799 -0.122201 662,500.761 0.877799 -0.122201

0.41/0.44 PASS 0.41/0.45 PASS 0.57/0.63 PASS 0.58/0.64 PASS 0.89/1.00 PASS 0.88/1.00 PASS 1.54/1.76 PASS 1.50/1.62 PASS 2.82/3.02 PASS 2.75/2.97 PASS 5.38/5.87 PASS 5.25/5.76 PASS 10.52/11.61 PASS 10.26/11.35 PASS 20.76/21.06 PASS 20.23/20.52 PASS

Table 25: Complete same-micro scaling for 30M; FlashAttention is forced. Method

Micro/GA

tok/s

FR/FA

∆ Alloc./res. GiB Status

21,715.706 0.739590 -0.260410 16,060.716 0.739590 -0.260410 42,208.004 0.759708 -0.240292 32,065.762 0.759708 -0.240292 82,115.368 0.770310 -0.229690 63,254.298 0.770310 -0.229690 161,224.613 0.745938 -0.254062 120,263.553 0.745938 -0.254062 322,442.432 0.764497 -0.235503 246,506.254 0.764497 -0.235503 388,949.491 0.866383 -0.133617 336,979.375 0.866383 -0.133617 352,213.572 0.888005 -0.111995 312,767.346 0.888005 -0.111995 – – – – – –

0.85/0.87 PASS 0.83/0.85 PASS 1.15/1.21 PASS 1.11/1.21 PASS 1.74/1.83 PASS 1.69/1.79 PASS 2.92/3.18 PASS 2.83/3.11 PASS 5.29/5.60 PASS 5.13/5.24 PASS 10.03/10.39 PASS 9.72/10.13 PASS 19.51/20.47 PASS 18.90/19.91 PASS – OOM – OOM

Table 26: Complete same-micro scaling for 100M; FlashAttention is forced. Method

Micro/GA

tok/s

∆ Alloc./res. GiB Status

FR/FA

15,954.945 0.759780 -0.240220 12,122.251 0.759780 -0.240220 30,462.198 0.756342 -0.243658 23,039.846 0.756342 -0.243658 58,932.391 0.777275 -0.222725 45,806.677 0.777275 -0.222725 115,222.124 0.770535 -0.229465 88,782.650 0.770535 -0.229465 133,917.125 0.841217 -0.158783 112,653.302 0.841217 -0.158783 – – – – – – – – – – – – – – – – – –

2.83/2.96 PASS 2.82/2.99 PASS 3.83/4.08 PASS 3.80/4.06 PASS 5.85/6.30 PASS 5.74/6.22 PASS 9.80/10.77 PASS 9.61/10.60 PASS 17.69/18.73 PASS 17.30/18.38 PASS – OOM – OOM – OOM – OOM – OOM – OOM

L INEAR R ELATION S YSTEMS AND L ONG -C ONTEXT S CALING

I.1

S TEADY-S TATE T HROUGHPUT Table 27: Frozen Linear Relation steady-state throughput. T

Scale

10M 1024 30M 1024 100M 1024

I.2

Micro/GA Steady-state tok/s Raw mean tok/s 64/2 32/4 16/8

517,602 256,034 90,592

282,578 199,875 87,865

L ONG -C ONTEXT S CALING

Long-context runs use the production KL60 schedule with chunk 128, block 64, four warps, one stage, and row tile 1. Micro is fixed to 1 and the effective update size is 131,072 tokens. The T = 65536 point is a real OOM. Table 28: Linear Relation long-context scaling. T 1024 2048 4096 8192 16384 32768 65536

Micro GA Status 1 128 1 64 1 32 1 16 1 8 1 4 1 2

PASS PASS PASS PASS PASS PASS OOM

tok/s Allocated GiB 10,129.47 20,301.39 39,990.24 59,367.95 52,894.96 30,003.56 –

1.140 1.640 2.648 4.819 8.827 16.815 –

Ask Self, Ask Others: Relation Is All You Need

Analyze & explore related research in Conceptio

Related documents