Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decoding

Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decoding

arXiv:2608.15473v2 [cs.DC] 19 Aug 2026

WenJie Fan Yotta Labs fanwj@mail.ustc.edu.cn fanwenjie@yottalabs.ai https://github.com/fan-wenjie/qfirst arXiv:2608.15473

Abstract The problem. Disaggregated decoding puts the KV-cache sweep on memoryoptimised hardware and the projections and feed-forward on compute-optimised hardware, then inherits the decoder block’s own ordering: attention runs first and the feed-forward consumes its output, so within one sequence each device waits on the other. The usual repair, more sequences in flight, costs one resident KV cache each — what separating the devices was meant to avoid. The fix. At a context of j positions, j − 1 are already cached, so all but one of attention’s score-and-combine terms follow from the query alone. Run the feedforward first and that query exists while the compute side still has work; the two then run concurrently, the current key and value following as a write nothing waits on. What it costs. Nothing structural: the decode is stated as a protocol, runs on the attention kernel a stock framework ships, and verifies end to end on a trained checkpoint to 4.2×10−3 relative, with no new operator, no changed tensor shape, no new hardware, and no change to parameter count or arithmetic. Trained 8 ways at 3 seeds, no read point in the family the protocol needs differs from the one that moves nothing by more than 0.0011 bits per byte — an order of magnitude inside the 0.0117 those seeds resolve, while the same runs resolve a sub-layer exchange 56 times as large. A checkpoint in service moves to this read point by fine-tuning 9.5% of its weights, at a residual +0.0142 at Qwen3-0.6B and +0.0088 at Qwen3-4B. Why this read point. Such a stack is the ordinary stack with its layer boundaries drawn half a sub-layer over — an identity, verified constructively — so the freedom is exactly where each of attention’s three arguments is read, and only the query’s position buys the overlap. It is bounded: projecting every layer’s query from the network’s input costs +0.1003, refuting a pre-registered threshold at every seed.

1

Introduction

A decoder layer reads two kinds of tensor: static weights, reused for every token and arithmetic-dense, and the KV cache, which is state rather than weight — touched once per byte and memory-bound. Serving systems separate them onto hardware suited to each [Chen et al., 2024, Liang et al., 2025] and inherit from the decoder block the ordering that makes one device wait on the other within a single sequence. This paper asks what the cache-side device actually needs in order to start, and what it costs to give it that: the first answer is exact, the second measured. Contributions. Preprint.

• A decode protocol that gives the cache-side device the query and nothing else, so it may sweep while the compute side still has its feed-forward to run. Exact, no custom operator, on the attention kernel a stock framework ships; verified end to end on a trained checkpoint to 4.2×10−3 relative (Section 2). What a device returns is a mergeable aggregate, so one cache may be split across any number of devices and merged in any order (Section 5), and two ordinary accelerators can be split by which kind of weight they multiply by (Section 3). • The measurement that makes this deployable: making the query exist early does not cost quality. In a block whose feed-forward runs first, moving the query’s read point moves the model by -0.0006 bits per byte against the block that moves none, the key’s by -0.0004, both by -0.0004 — each inside a floor of 0.010 and an order of magnitude below what 3 seeds resolve, 0.0117. Re-drawing the boundaries half a sub-layer over moves it -0.0090, also inside the floor. Trained from scratch at Qwen3-0.6B at identical parameter counts, forward arithmetic, data order and seeds (Section 7). A trained checkpoint reaches the same read point by fine-tuning Wq alone, 9.5% of the model, leaving at most +0.0142 (Section 6). • The minimality argument the method is named for. The device sweeps keys already cached and the current key is a write nothing waits on, so exactly one of the attention’s three read points has to move. Every alternative moves more — two read points, or the graph itself, as the parallel block [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023] does in taking an L-layer stack from 2L sub-layers of serial depth to L (Section 8). What this paper does not claim. No speedup is measured. The decomposition is exact and the arrangement it admits is described in full, but what it buys needs two devices and a decode loop; on one card the stream serialises everything regardless of what depends on what. The training results are at 3 seeds and one scale at three per cent of compute-optimal, where a lead measures a disturbance to training rather than a quality reached. The conversion result is one corpus at one budget, and bounds the residual cost rather than measuring it.

2

The device needs only the query

Throughout, the device means the memory-optimised side that holds the KV cache, and the accelerator the compute-optimised side that holds every weight. At decode step j in one layer: Write uj for the block’s input at that step, N1 for the attention’s normalisation, and qj = Wq N1 (uj ), kj = Wk N1 (uj ), vj = Wv N1 (uj ) for the query, key and value it projects. In the rewired block the query still reads uj while the key and value read the post-feed-forward stream, kj = Wk N1 (zj ) and vj = Wv N1 (zj ) with zj = uj + M (N2 uj ) (Section 8); the protocol below is stated for either wiring. o<j and lse<j are the attention vector over the cached positions and their log-sum-exp.

accelerator device accelerator

sends

computes

qj o<j and lse<j kj , vj (a cache write, unblocking)

the feed-forward, then kj and vj attention over the j − 1 cached positions the combine below

The device attends over the cache alone and returns the attention vector among those positions with their log-sum-exp. The accelerator, which computed qj and kj itself, finishes with the standard combination of two partial softmax states,
qj · k j sjj = √ , out = lerp vj , o<j , σ(lse<j − sjj ) , lerp(x, y, t) = (1 − t) x + t y. d Overflow, where it actually is. Not in the combine: written as 1/(1 + e−x ) the weight is correct at every magnitude in IEEE arithmetic, agreeing with a library sigmoid bit-for-bit at two million points across [−120, 120] and finite out to ±800. The exponential form it replaces returns NaN past 89 in single precision. Three places do need care — the running log partition, how a fused kernel’s padded log-sum-exp is indexed, and the sign of sjj − lse<j , which is not one-directional on a trained checkpoint. Each is stated with its failure in the released code. The device’s input is the query alone, which holds of any decoder: the sweep is over positions already in the cache and needs neither the current key nor value. What differs between blocks is when 2

the query exists. The current key and value arrive afterwards as a cache write nothing waits on, so the sweep runs concurrently with the feed-forward producing vj and what stays serial is a combination of two vectors. No custom operator is needed: the memory-efficient attention of Dao et al. [2022] returns the log-sum-exp exactly, and we verify the step against fused attention to 4.2×10−3 relative on a trained checkpoint. Two caveats: that backend has no grouped-query kernel [Ainslie et al., 2023], so keys and values must be expanded first; and PyTorch’s flash entry point returned a second output matching no hand-computed log-sum-exp under either layout. The intervention is one send, moved earlier. Nothing above asks for a new operator, a new kernel, or a tensor of a different shape. The sweep is O(j d) per head and needs the query alone; what remains after it is a rank-one combination, O(d), one j-th of the layer’s attention work at a context of j: one 65,536th at 65,536 positions, and proportionally less at one million. A deployment obtains the overlap by computing qj from the block’s input and putting it on the wire at that moment rather than after the layer’s other arithmetic; the key and value follow whenever they are ready. What has to change to make that possible is the block’s ordering, not its parts. In an ordinary block the attention runs first, so its query is available immediately — and that is exactly the problem, because the feed-forward then waits on the attention’s output and the two devices alternate rather than overlap. Putting the feed-forward first, with the query still reading the block’s input, leaves every component, shape and parameter where it was and leaves both sides with work to do at once; Section 8 shows the re-bracketing is exact.

3

A two-accelerator arrangement

The protocol says what may be computed when, not where, and the arrangement it admits is worth stating because a reader will otherwise assume near-memory hardware that does not yet exist. Two ordinary accelerators suffice, split by what kind of weight they multiply by rather than by layer or head: the MLP side holds the static weights and runs every projection and feed-forward, the attention side holds the KV cache and does nothing but sweep it. This is the split model-attention disaggregation already builds [Chen et al., 2024, Zhu et al., 2025, StepFun, 2025]; what changes is only what the cache side is given and when. One layer, one decode step j, reading down: MLP side

attention side

qj → computes MLP, then kj , vj

receives qj sweeps the cache for o<j , lse<j parks the pair; stages layer L+1’s KV from its own VRAM

reads the pair, combines with vj kj , v j →

shared, both can reach

o<j , lse<j

appends to layer L’s cache

Three properties of this arrangement are not incidental. The result is parked, not pushed. The attention side writes o<j and lse<j to memory the MLP side can read and moves on rather than waiting for a handshake, so the two are coupled by data and not by arrival order. Across layers, not only within one. Read one layer at a time, the sweep runs beside the feedforward of the same layer; drawing the stack one half-layer over makes it span layers instead, which is the form to hand an implementer: [ Attn1 ] [ MLP1 ∥ Attn2 ] [ MLP2 ∥ Attn3 ] · · · [ MLPN −1 ∥ AttnN ] [ MLPN ] Each middle stage holds one feed-forward and the next layer’s attention, both reading the stage’s input, so layer i + 1’s query exists before layer i’s feed-forward runs. Only two of the N + 1 stages lose the concurrency: the first has no feed-forward before its attention, the last no attention after its feed-forward. The re-bracketing is exact and verified bit-identical (Section 8). The compute side is a preemptible pool. Sweep and feed-forward may run at once, neither being downstream of the other, and the pool holds no request state so it can be taken and released at will (Section 4). We do not claim either conceals the other: which finishes first depends on the context. 3

At batch one both sides are bandwidth-bound, so the comparison is bytes per layer per step: the feed-forward reads its weights whatever the context, the sweep reads 4.1 KB per cached position. One row per model, every figure this section quotes: model

feed-forward

sweep holds the pool

crossover

pooled

19 MB 149 MB

1% at one million, 4% at 128K 4% and 23%

4,608 36,480

3,584 28,800

Qwen3-0.6B Qwen3-4B

Crossover is where the two are equal; pooling the key and value projections with the feed-forward moves it to the last column. The remaining 99% is a device holding weights it cannot put down, which against a preemptible pool is another request’s time. None of this is measured: it is a scheduling consequence of the contract in Section 4, and what decides whether it pays is the mix of contexts a deployment sees.

4

What each side holds

A deployment has to break the symmetry: one side owns the request. Which side is settled by where the state is pinned and by what a shared pool may hold, not by which component is stateless — a database holds more state than the service in front of it and is still the callee. held by

what

why it cannot sit on the other side

cache side

the KV cache

cache side

Wq

cache side

Wo

migration is not an option, so a request is pinned for its lifetime to the device holding it [Kwon et al., 2023]. a query computed in the pool would arrive with the reply, no earlier than the feed-forward it was meant to overlap. Wq applies to u′ = z + Wo a, so Wo in the pool puts a round trip in front of every sweep — the ordering this arrangement exists to remove, one layer down.

pool

the feed-forward

pool

Wk , Wv

at batch one it reads its whole weight for one token’s arithmetic, and only batching across different requests amortises that [Yu et al., 2022]. A pool is called, and cannot own one request’s control flow without giving up the batching it exists for. inlined into the same call so the pool returns cache entries rather than a hidden state the caller must send back. Nothing writes into another device’s memory.

KV cache model

held per layer

on the wire

KB/token

GB at 1M

cache side

pool

KB/stage

KB/token

112 144

107 137

8.4 41.9

23.1 159.9

2/6 5/9

224 504

Qwen3-0.6B Qwen3-4B

One call per stage, with the state on the wire: u

z=u+M (N2 u); k,v from N1 (z)

sweep with its own q

cache −−→ pool −−−−−−−−−−−−−−−−−−−→ cache −−−−−−−−−−−→ u′ = z + Wo a. A pool holding no request state is preemptible: a stalled caller stops calling and nothing of the request is inside to protect.

5

The cache may be split anywhere

The two values the device returns are not an ad hoc pair. For a set S of cached positions with scores si and values vi , write X X Z(S) = esi , W (S) = esi vi , O(S) = W (S)/Z(S), i∈S

i∈S

4

and (W, Z) adds componentwise over disjoint sets: the online-softmax recurrence [Milakov and Gimelshein, 2018] behind single-pass fused attention [Dao et al., 2022] and sequence-parallel merging [Liu et al., 2023]. The contribution is the coordinates, not the algebra. A device returns (O, log Z) — the same pair normalised by its total, the total kept in logs — and the merge there is the combine of Section 2:
O(A ∪ B) = lerp O(B), O(A), σ(log Z(A) − log Z(B)) ,
log Z(A ∪ B) = logaddexp log Z(A), log Z(B) . A cache split across any number of devices, in any way, swept independently and merged in any order therefore returns what one sweep over the whole would: 200 random four-way splits agree to 1.4×10−7 . No split has to be balanced, contiguous or known in advance, and no device needs to know what another holds. One asymmetry is worth an implementer’s attention: the identity is degenerate. An empty shard is log Z = −∞ and a vector never read, so a device with nothing to contribute may return whatever is in its buffer, provided its log partition says so.

6

Converting a trained checkpoint

Everything above describes a decode a deployment would have to build a model for. Trained from scratch the read point costs nothing this campaign can measure (Section 7), but that obliges a reader to retrain before any of it is usable. This section asks the question an existing deployment actually has: what it costs to move a checkpoint already in service to the early-query read point. Qwen3-0.6B is rewired by hooks — the block’s own arithmetic is untouched, layer ℓ’s query reads hℓ−1 instead of xℓ , its key and value do not move, and layer 0 is exempt because its attention reads the embedding under either wiring. Every comparison in this paper is at that read point and that coverage — one half-layer back, every layer the wiring can reach — except the depth sweep, which varies it, and the two named boundary arms, which is what they are for. Only Wq is then trained, 56.6M of the model or 9.5%; everything else, including the tied embedding and therefore the readout, stays frozen. The reference must receive the same fine-tune. Scored against the checkpoint evaluated zero-shot, the converted-and-repaired model beats what it was converted from, because the fine-tune adapts it to a corpus the checkpoint never saw and that adaptation is worth several times the conversion. So the quantity is d = (converted + SFT) − (unconverted + the same SFT), at identical repair set, schedule, data order and seed.

Qwen3-0.6B Qwen3-4B

trainable

untrained

after 3,000 steps

d

9.5% 9.1%

+0.0746 +0.0364

0.9166 0.7428

+0.0142 +0.0088

Rewiring costs +0.0746 bits per byte before any training at Qwen3-0.6B; fine-tuning Wq alone removes most of it and leaves +0.0142, reproduced across 2 seeds to 0.0003. The residue is expected: the converted query must reproduce Wq x = Wq h + Wq M (h) from h alone, and the second term is not linear in h, so no retrained linear Wq is exact. Two things this does not say. It is not a quality comparison between arrangements — the parallel block trains perfectly well from scratch, and nothing here measures that. And equal steps is not equal difficulty: the converted arm spends part of its budget undoing the conversion while the reference spends all of its adapting, so +0.0142 is an upper bound on the residual cost, not a measurement of it. The recipe is therefore short: rewire the read point, retrain one projection per layer, leave everything else including the readout frozen. Adding the output projection — the other weight the cache side already holds, so still no new parameter and no new inference-time matmul — takes d to +0.0114 at Qwen3-0.6B and +0.0073 at Qwen3-4B, for 19.0% and 18.2% trainable. Larger repair sets, alternative wirings for the query path, and the forward-only diagnostics bounding them were also run; none beats this, and none is reported here. 5

How far back the query may be read. The arrangement above overlaps one feed-forward, and one half-layer opens it, so in a dense stack nothing deeper buys overlap. A pattern that alternates several linear-attention layers with one softmax-attention layer offers a longer shadow, and there deeper reads do open more of it — but the residual after repair rises with them, +0.0123 at one half-layer against +0.0556 at seven, and the residual is what a deployment carries. Half a layer is therefore the conservative choice under both block patterns, and it is where this paper measures. half-layers

of the shadow

untrained

d

removed

1 3 5 7

12.5% 37.5% 62.5% 87.5%

+0.0658 +0.1750 +0.3511 +0.4821

+0.0123 +0.0281 +0.0414 +0.0556

81% 84% 88% 88%

Untrained is the displacement before any repair, d what survives it, and the last column their ratio: the repair removes more at depth and still leaves more.

7

What it costs to make the query exist early

The protocol needs the query before the feed-forward; whether a block can be trained that way is a separate question, answered here. Every comparison holds the computation graph, the components, their shapes, the parameter count, the forward arithmetic, the data order and the seed fixed, and moves the read point alone. The arms fall into two families by whether a block’s feed-forward consumes that block’s own attention: in family A it does not, the feed-forward running first or the two running from the same input; in family B it does, the ordinary arrangement. Every run is at 3 seeds, all printed, because a mean hides which seed the spread came from. Family A: the feed-forward does not consume this block’s attention. Exchange the two sublayers and the query and key may each read the residual stream before the feed-forward or after it; the value always reads after, since reading it early too is the parallel arrangement. The control is Swapped, the corner that moves no read point. arm

reads early

seed 0

seed 1

seed 2

mean

vs Swapped

Swapped Early-K Early-QK Early-Q Parallel

nothing the key query and key the query a different graph

1.2388 1.2352 1.2369 1.2369 1.2335

1.2381 1.2365 1.2377 1.2403 1.2400

1.2322 1.2361 1.2332 1.2302 1.2323

1.2364 1.2359 1.2359 1.2358 1.2353

— -0.0004 -0.0004 -0.0006 -0.0011

The 5 arms span 0.0011 against a floor of 0.010, and none differs from the control by more than 0.0011, and the same holds throughout training (Figure 1). Nothing here is distinguishable from anything else, which is the only claim the table makes and the one the protocol needs: where the feed-forward does not wait on its attention, moving the read point is a change this campaign cannot find. Family B: it does. The ordinary ordering, with two arms that change where the query comes from without exchanging anything: the shifted stack re-draws the boundaries half a sub-layer over as Section 8 does, taking each query from the input of the preceding feed-forward; the shared-query arm projects every layer’s query from the network’s input. arm

the query is read from

vs Standard

Shifted-Q Single-Q

the preceding feed-forward’s input the network’s input

-0.0090 +0.1003

This family spans 0.1093, and the two treatments differ from the control in opposite directions: the shifted stack by -0.0090 and the shared source by +0.1003 (+0.0979, +0.0970 and +0.1061 by seed). The second was pre-registered with a refutation threshold of +0.05 and exceeds it at every seed, so a query may be read one feed-forward early and may not be read from the network’s input. That is where the method’s reach ends, and knowing the boundary is what the arm was run for. 6

family A, against Swapped

family B, against Standard

minus the family's control

0.03 0.02

Standard, its own 3 seeds Shifted (query early) Shared query (from the network input)

0.3

0.01

0.2

0.00

−0.01

0.1

Swapped, its own 3 seeds Early-K Early-Q 0.0 Early-QK Parallel

−0.02 −0.03 0

2000

4000

6000

8000

optimiser steps

10000

0

2000

4000

6000

8000

optimiser steps

10000

Figure 1: Each family against its own control, every seed: family A against Swapped, family B against Standard. Grey is the control’s own spread across seeds, the width inside which nothing is a difference; the panels do not share an axis, their ranges differing ninefold. Left: no read point leaves the band by much or for long and all four are inside it over the last third. Right: Shifted-Q sits just below the control at -0.0090, the shared query source far above it at +0.1003. 3 seeds do not settle the shifted arm: its mean is inside the floor, but its seeds disagree by 0.0149 — more than the effect, and the widest disagreement here. Two gave it a lead and the third reversed the sign. We report it and build nothing on it. At this budget a lead is a disturbance, not a verdict. These runs are at three per cent of computeoptimal, where a difference in bits per byte is a difference in optimisation trajectory rather than in reachable quality. A gap against the control measures how much the change disturbed training, so the quantity is how far each arm moved, not which leads. The 5 arms of family A move at most 0.0011 — an order of magnitude inside the 0.0117 that 3 seeds resolve, and smaller than an arm’s own spread across seeds, 0.0115. That is not a way of turning a null into a result: the same measurement resolves exchanging the sub-layers at 56 times any read point’s movement, and projecting the query from the network’s input at 91. The instrument has range and family A produces no signal in it. The flatness is not exact throughout — the read points separate above the seed band from roughly step 1,000 to 5,000 by at most half again, and the separation is gone by the last third, so a campaign stopped at its widest point would have reported an effect a longer one does not. The families are not compared across. Every family A arm sits below Standard, by -0.0623 to -0.0627. We list it and claim nothing from it, for three measured reasons: the two stacks are the same alternating chain differing only in the half-layers at their ends, and the shifted arm — family B by construction, family A in its interior — lands 7 times short of the gap; the tied embedding that is also the readout is frozen at its initialisation, so one ordering hands that representation to a feed-forward first and the other to attention; and the gap is a difference in convergence rate, -0.1036 at the first evaluation, widening to -0.1722, narrowing monotonically to -0.0623 by the end without having stopped. An arm with the embedding unfrozen in both orderings would settle it; that changes a control this campaign fixed. Nothing claimed here rests on the gap. Parameters and arithmetic. Every arm in both tables carries 596.05M parameters and identical forward arithmetic, measured rather than assumed, so no comparison here needs a budget argument and none is offered. The parallel arrangement [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023] is listed in family A because it shares the defining property, with the caveat of Section 8 that it is a different graph. What 3 seeds can and cannot say. Every seed is printed. The tables are read in paired differences, where the seed’s effect on initialisation and data order is common to an arm and its control and cancels; read that way the seeds scatter with a pooled standard deviation of 0.0047, putting a 95% band of 0.0117 on a mean of 3. That measured resolution agrees with the floor of 0.010 fixed before these runs, and is what makes family A a measurement rather than an absence: its arms sit within

7

0.0011 of their control, an order of magnitude inside the band, while the two effects this paper rests on sit 56 and 91 times outside it.

8

Where a query may be read from, as algebra

Why the query, and why a read point rather than a different graph, are settled here rather than assumed. Write the stream’s two updates as operators, each carrying its own normalisation: A for attention, M for the feed-forward. An ordinary block composes them; a block reading the query earlier splits the attention’s arguments, scoring from x and loading values from z: S = (I + M ) ◦ (I + A), Q = (I + Ax;z ) ◦ (I + M ), z = x + M (x). Over a stack the two are conjugate,  L QL = (I + A) ◦ (I + M ) = (I + M )−1 ◦ SL ◦ (I + M ), so a stack of such blocks is the ordinary stack with its layer boundaries drawn half a sub-layer over: the interiors agree exactly and only the two ends differ. We verify this constructively, building the shifted stack as L − 1 fused blocks between two half blocks and checking it is bit-identical to the ordinary one. The conjugation leaves each of attention’s three arguments free to be read before the feed-forward or after it, and the device selects among them: the sweep begins when the query arrives, so q from the earlier point buys the overlap, and q alone. Where the key is read changes nothing — the device works over cached keys and the current one is a write nothing waits on, holding to 3.6×10−3 relative for a block whose key is read late — and a query read from the later point is the ordinary block with its sub-layers exchanged, which gets no overlap. Reading q early is necessary and sufficient, and all a deployment changes. The arrangement that sums rather than composes, P = I + A + M [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023], buys the same overlap by changing the graph instead; a forward-pass test separates the three families, and that change is left to separate work.

9

Related work

Disaggregated serving splits a decode across machines by phase, prefill from decode [Zhong et al., 2024, Patel et al., 2024], or by tensor kind, the cache sweep from the static weights [Chen et al., 2024, Liang et al., 2025, Pan et al., 2024]. A production line carries the second split to its conclusion and disaggregates attention from the feed-forward outright [Zhu et al., 2025, StepFun, 2025] — the split of Section 3 — overlapping the two sides across requests by micro-batching. Within one sequence the two still alternate, and that is the dependency this paper removes; the two overlaps are orthogonal and compose. Closest in framing, Ma et al. [2026] route the query itself between instances, as DistAttention does for dense attention, shipping a query row with its softmax statistics [Lin et al., 2024]. What those inherit, and this paper removes, is the block’s own ordering. A separate line shrinks what the sweep must read rather than when it may start — one key-value head [Shazeer, 2019] or a few [Ainslie et al., 2023], or a low-rank code in place of the stored pair [DeepSeek-AI, 2024]; those change the cache and compose with this, which changes neither cache nor kernel. The parallel block [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023] buys the same overlap by changing the graph, and is treated in Section 8.

10

What was measured, and on what

Each of the 8 arms is Qwen3-0.6B trained from scratch at 3 seeds, every layer converted, on 372M tokens of WikiText-103 over three epochs: 11,355 optimiser steps at a peak learning rate of 4×10−3 under a cosine schedule, sequence length 1,024, 32,768 tokens per step, AdamW, compiled. The tied embedding, which is also the readout, is frozen at initialisation so the comparison is between decoder stacks rather than vocabularies. Two settings are pinned rather than left to a default, each because it would otherwise dominate what the study resolves: normalisation weights are held in float32, since in pure bfloat16 they begin at exactly 1.0 where the spacing is 0.0078 and the optimiser’s step is three orders of magnitude smaller, so they never move; and the attention backend is pinned to flash and recorded per run: the alternative kernels disagree with it by up to 1.6×10−2 on this model’s attention shapes, which does not reach the loss here but is not a promise the backend heuristic makes. Each run takes 2.0 to 2.7 hours on one accelerator; the 24 runs here are 51 GPU-hours. 8

Which split these numbers are. Every figure here is validation, and both the checkpoint and the learning rate were selected on it. We therefore read the same runs on a held-out test split that nothing selected on: against its own family’s control, no gap moves between splits by more than 0.0094 across 18 arm-seeds, and none in family A by more than 0.0020 — both inside 0.010. Selection did not manufacture the flatness, and the refutation in the second table is larger on the split it was not selected on. The validation split also shares windows with training. Each run scores a masked copy with those windows removed: the effect is -0.0071, varying by 0.0014 across 24 runs. A constant shift an order of magnitude below 0.010 in its variation cancels in every comparison made here. The protocol is verified against fused attention to 4.2×10−3 relative, on synthetic tensors and end to end on a trained checkpoint, at the shapes and layers the release names.

11

Conclusion

The cache-side device of a disaggregated decoder needs the query and nothing else. That is an identity, it runs on stock kernels, and what it returns is a mergeable aggregate, so a cache may be split across any number of devices and merged in any order. An existing checkpoint can be moved to this read point by fine-tuning Wq alone, 9.5% of the model, at a residual cost bounded by +0.0142 bits per byte at 3,000 steps (Section 6). Reproducibility. Every number here is generated from run records by a script in the repository; none is typed into the source, and an unmeasured number raises a build error rather than appearing as a placeholder. The block definitions, trainer, protocol tests and analysis are included with the exact command for each arm. The pre-registrations are under protocols/, one file per gate, each stating its verdict thresholds and the author’s prior before the arm ran; gate_SQ.md is the one the +0.05 refutation threshold comes from. The table each campaign was launched from is under run/experiments/, one row per run.

12

Limitations limitation

what would settle it

One model family, one scale, one corpus, 3 seeds, More scales and corpora. What 3 seeds do resolve at three per cent of compute-optimal — where a is measured, not assumed: 0.0117. difference in convergence and one in reachable quality are hardest to separate, and every separation is still contracting at the last evaluation. The arrangement is verified, not built. Correctness A deployment. Whether it pays is a ratio between an is an identity, checked end to end on a trained interconnect’s latency and a feed-forward’s duration checkpoint with the freshness condition stated in — a property of the machine, not of this decomposifull; no speedup is measured. tion. The family gap, -0.0623, is reported and not ex- Both orderings with the embedding unfrozen, which plained (Section 7). changes a control this campaign fixed. The conversion result is one budget on one cor- The same arms at a larger budget, and on a second pus, and equal steps is not equal difficulty: the corpus. converted arm spends part of its budget undoing the conversion, so +0.0142 bounds the residual cost from above rather than measuring it.

References Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2305.13245. arXiv:2305.13245. Shaoyuan Chen, Wencong Xiao, Yutong Lin, Mingxing Zhang, Yingdi Shan, Jinlei Jiang, Kang Chen, and Yongwei Wu. Efficient heterogeneous large language model decoding with 9

model-attention disaggregation. arXiv preprint arXiv:2405.01814, 2024. URL https://arxiv.org/abs/2405.01814. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. URL https://www.jmlr.org/papers/v24/22-1144.html. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2205.14135. arXiv:2205.14135. DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. URL https://arxiv.org/abs/2405.04434. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021. URL https://arxiv.org/abs/2012.14913. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In 29th ACM Symposium on Operating Systems Principles (SOSP), 2023. URL https://arxiv.org/abs/2309.06180. Yunkai Liang, Zhangyu Chen, Pengfei Zuo, Zhi Zhou, Xu Chen, and Zhou Yu. Injecting adrenaline into LLM serving: Boosting resource utilization and throughput via attention disaggregation. arXiv preprint arXiv:2503.20552, 2025. URL https://arxiv.org/abs/2503.20552. Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. Infinite-LLM: Efficient LLM service for long context with DistAttention and distributed KVCache. arXiv preprint arXiv:2401.02669, 2024. URL https://arxiv.org/abs/2401.02669. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023. URL https://arxiv.org/abs/2310.01889. Bole Ma, Jan Eitzinger, Harald Köstler, and Gerhard Wellein. Move the query, not the cache: Characterizing cross-instance latent attention redistribution across GPU fabrics. arXiv preprint arXiv:2606.01502, 2026. URL https://arxiv.org/abs/2606.01502. Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018. URL https://arxiv.org/abs/1805.02867. Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. Instinfer: In-storage attention offloading for cost-effective long-context LLM inference. arXiv preprint arXiv:2409.04992, 2024. URL https://arxiv.org/abs/2409.04992. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 51st ACM/IEEE Annual International Symposium on Computer Architecture (ISCA), 2024. URL https://arxiv.org/abs/2311.18677. 10

Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. URL https://arxiv.org/abs/1911.02150. StepFun. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint arXiv:2507.19427, 2025. URL https://arxiv.org/abs/2507.19427. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021. Gyeong-In Yu, Joo Seong Jeong, Geon Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022. URL https://www.usenix.org/conference/osdi22/presentation/yu. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. URL https://arxiv.org/abs/2401.09670. Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. MegaScale-Infer: Serving mixture-of-experts at scale with disaggregated expert parallelism. arXiv preprint arXiv:2504.02263, 2025. URL https://arxiv.org/abs/2504.02263.

11

A

the combine, and what the disaggregated stage costs

The body states the join as a lerp weighted by a sigmoid, and states the stage’s cost in one line. This part carries both: where that form comes from, why the logarithm is not a convenience, what the stage puts on the wire, and how wide the window it opens really is. Background: what the combine is. Combining two partial softmax states is a sigmoid of the difference of their log partition functions, and the form says what the step means. For a fixed query the whole history behaves as one pseudo-token of score lse<j and value o<j , so appending a token is a two-way softmax — a logistic. Thus lse<j − sjj is the log-odds that the step attends to what it already has rather than to what just arrived. Decoding is a fold over that pair,
o<j+1 = lerp vj , o<j , σ(lse<j − sjj ) , lse<j+1 = logaddexp(lse<j , sjj ), which is the sense in which the device’s two return values are sufficient: they are the state of the fold. Why the logarithm is not a convenience. Why it matters: in the undisguised coordinates a shard of 400 positions at a score spread of 30 already carries Z about 8.2×1036 and ∥W ∥ about 5.8×1037 , an order of magnitude below what single precision holds, so a wider spread or a longer shard cannot be exchanged at all. Logged, O is a convex combination and log Z a number near 85. What it costs. Statelessness puts the request’s state on the wire and the layer loop on the network. Per stage the cache side sends 2 KB and receives 6 KB — 224 KB per token over 28 round trips at Qwen3-0.6B, 504 KB at Qwen3-4B. The bytes are immaterial; the serial round trips are the cost. What pays for them is that the sweep begins the moment the request is dispatched, so a round trip is hidden whenever it is shorter than a sweep and the margin grows with context. Across a wide-area link it does not, and that is where the arrangement stops. The window is one feed-forward wide. A sweep runs concurrently with one feed-forward and no more. Reading every layer’s query from the network’s input would widen that window to the whole stack, since every query would exist at the step boundary — but it costs +0.1003 bits per byte at every seed (Section 7). The preemptible pool is what makes the limit survivable: a request away from the pool is not a pool standing idle. How small the change is. Letting a block hand over its query early is a smaller change than it appears: the feed-forward moves in front of the attention and the query keeps reading the block’s input, leaving every component, shape, parameter count, forward arithmetic and serial depth where they were. Over 3 seeds at Qwen3-0.6B, no read point in that family separates from any other by more than 0.0011 bits per byte, against a resolution of 0.0117, while the same runs resolve a sub-layer exchange 56 times as large and a shared query source 91 times as large. The reading is a magnitude, not a ranking: at this budget an arm leading by a wide margin would show the change perturbs training, not that it is good. The window is one feed-forward wide and no wider — a query projected from the network’s input moves +0.1003. What one card cannot answer. Whether the arrangement pays is a ratio between an interconnect’s latency and a feed-forward’s duration, which one card cannot answer and building it would. The code is released for that.

12

query across a layer boundary

query within one block

⋮

⋮

Attnℓ − 1

Attnℓ − 1

+

+

q

q

one block

+

one block

MLPℓ − 1

k, v

Attnℓ

MLPℓ

+

k, v

Attnℓ

+

+

MLPℓ

MLPℓ + 1

+ ⋮ attention ℓ beside feed-forward ℓ

+ ⋮

−1

attention ℓ beside feed-forward ℓ

Figure 2: The same wiring, bracketed two ways. Left: the query is tapped at the entrance of feedforward ℓ − 1 and crosses into attention ℓ. Right: the feed-forward comes first and the identical tap falls inside the block. The wires are identical; the bracket differs, and with it which pretrained feed-forward sits beside which pretrained attention, plus a half block at each end. Neither panel reads a staler query — one feed-forward separates the query’s tap from the key’s and value’s in both — and assuming otherwise attributes a pairing effect to staleness.

13

B

the conversion and the depth sweep in detail

The second scale. At Qwen3-4B the same procedure leaves +0.0088, and the untrained displacement is +0.0364 against +0.0746. The share of the model Wq occupies is essentially unchanged between the two (9.5% and 9.1%), so the improvement is not a budget effect; what produces it is not established here, and two scales are two points. How the four depths were run. The same conversion and the same repair were run at four depths to establish that. Depths are in HALF LAYERS: a layer is an attention and a feed-forward, and a query read one layer index back moves half of one, so every reachable depth is odd. Layers 1–3 are held out of the conversion at every depth — and layer 0 is exempt, as before — leaving 24 converted in all four arms — a layer with fewer layers beneath it than the depth asks for cannot read that far and clamps to the bottom of the stack, and without the hold-out the deeper arms would convert the same layers on paper while some of them read somewhere else. The shadow column is geometry, not measurement: consecutive softmax-attention layers in a threelinear-one-attention pattern are four layers apart, so a depth of h half-layers opens h/8 of that interval, and 7 half-layers — the deepest odd depth inside it — spans all but the last eighth. The two costs beside it move in opposite directions because a deeper arm starts further from its reference: it has more to remove, removes a larger share of it, and still ends further away. On a hybrid checkpoint the depths cost less to begin with. Qwen3.8-27B, 64 layers of which 16 are softmax attention, is displaced +0.0181 bits per byte at one half-layer and +0.2081 at seven, against a baseline of 0.6806 — against +0.4821 for the dense model at the same depth. That is forward-only: the checkpoint is rewired and evaluated, with nothing trained.

14

C

the design is forced, and each step is checked

The design is forced, and each step is checked. Collected, the case that this is the read point to move rests on four claims, each with the theory that predicts it and the measurement that tests it: claim

how it is established

The shifted stack is the ordinary stack rebracketed, not a different model. Only the query has to move early.

The conjugation QL = (I + M )−1 ◦ SL ◦ (I + M ), verified constructively bit-identical. The sweep runs over cached positions and the current key is a write nothing waits on, at 3.6×10−3 relative for a block reading its key late. Moving the key or value instead buys no overlap. 5 arms within 0.0011 of their control against a resolution of 0.0117, while the same runs resolve effects 56 and 91 times larger (Section 7). A query projected from the network’s input costs +0.1003, exceeding a pre-registered refutation threshold of +0.05 at every seed (Section 7). Converted the same way and repaired by the weights whose input it moves, the parallel block needs 44.6% of Qwen3-4B retrained against 9.1% for the query, is displaced +2.2290 against +0.0364, and retains +0.0427 against +0.0088 (Section 6).

Moving it is free at this budget. The freedom ends one feed-forward out. A checkpoint already in service reaches it far more cheaply than it reaches the alternative.

The last three are what separates this from an arrangement chosen for convenience: the campaign was run to find the boundary as much as the freedom, and it found one. The final row is about reaching the two arrangements from a trained checkpoint and not about the arrangements themselves — the parallel block is one of the family A arms above that this campaign cannot distinguish from any other, and it trains from scratch without difficulty.

15

D

the wiring as a schedule, worked

The algebra says which tensor is read where; what that does to a running server is the next question. One instance, small enough to check by hand: 12 requests over 3 layers, arriving in three groups one time unit apart, each with its own attention time. The feed-forward is a pool per layer — one unit per departure, one at a time, and it will not depart with fewer than 2 waiting. Communication is free. Group G1 arrives at t = 0 carrying R1(4), R2(3), R3(2), R4(1); G2 at t = 1 with R5(3), R6(2), R7(2), R8(1); G3 at t = 2 with R9(2), R10(2), R11(1), R12(1), the bracketed number being that request’s attention time at every layer. What the host may issue, and when. Layer ℓ’s query is projected from hℓ−1 , the residual after layer ℓ − 1’s attention and before its feed-forward; the key and value still come from xℓ , which is after it. So the moment hℓ−1 exists, two things are issuable at once — the sweep of layer ℓ, which needs only qℓ and the cached keys and values, and the feed-forward of layer ℓ − 1, which needs only hℓ−1 . Layer ℓ is ready when both return, the second because xℓ = hℓ−1 + MLPℓ−1 is where its own key and value come from: e1 = arrival + a,

layer 1’s query reads the embedding; nothing to pair with

eℓ = max eℓ−1 + a, ffnℓ−1 ,

ℓ = 2...3

finish = ffn3 ,

the last feed-forward has no sweep left to pair with.

In a composed block that edge sits one half-layer later: the feed-forward of layer ℓ reads hℓ , so it cannot be issued until layer ℓ’s attention has finished, and layer ℓ + 1 cannot begin until it returns.

Q-First: the query is read half a layer back 3

L1 pool L2 pool L3 pool R4 sweeps R1 sweeps

L1

L2

5 2

4 3 2

L3

L1

2 2

3

2 2

3

2

L2

R1 never departs

L3

composed block: the same host, pools and rule, one edge later 3

L1 pool L2 pool L3 pool R4 sweeps R1 sweeps

L1

0

5

2

3

L2

L1

1

4 2

3

2 2

L3

4

5

3 2

2 3

2

2 R1 never departs

L2

6

L3

7

8

9

time

10

11

12

13

14

Figure 3: One simulation, both wirings. Blue: each layer’s pool, labelled with how many requests are aboard; orange: the sweeps of the fastest request and the slowest. Read the bottom row. Under Q-First the slowest request’s sweeps abut — [0, 4], [4, 8], [8, 12] — because its next query exists before the previous feed-forward runs; under the composed block a feed-forward separates them at every boundary. The departures are the same batches in the same order in both panels, only shifted. What the picture shows, and what it does not. The last request to finish does so at t = 11 against t = 14, but that number is a property of these twelve service times and is not a speedup this paper claims. What generalises is the shape: the sweep chain is contiguous under one wiring and interrupted under the other by exactly one feed-forward at every layer boundary. The pool itself is untouched — same batches, same departures — which is the practical content of the claim that nothing but the query moves. One hazard the instance exposes. The departure rule as stated has no timeout, and 1 request never leaves: R1 is the slowest and the last, so when its final feed-forward request arrives there is nothing left to ride with. It happens identically under both wirings, so it belongs to the rule and not to the block, and to a finite example draining out rather than to a server in steady state. An implementation needs a maximum wait, and that wait enters the latency. 16

E

two checks before trusting the arithmetic

The arrangement that sums rather than composes, P = I + A + M [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023], delivers the same overlap and is not the same kind of change. S and Q differ in where three arguments are read; P differs in the graph. Its branches never meet inside the block, so an L-layer stack has L sub-layers on its longest dependency chain where S and Q have 2L. Which of the three a block belongs to is decidable. For a block B taking its two sub-layers scaled, the mixed second difference     ∆ = B(αA, µM ) − B(αA, M ) − B(A, µM ) − B(A, M ) vanishes identically exactly when the two branches never meet. Measured on this model, it is zero for P and nonzero for S and for every Q. We raise the distinction rather than pursue it because only one of the two changes is free. Halving a stack’s serial depth changes what the model can compose; moving a read point does not. The serial-depth change is left to separate work. Three numerical caveats.

The claims in Section 2 rest on these holding.

where

why

the running log partition

log(ea + eb ) overflows at a = b = 100, an ordinary magnitude at long context; a stabilised logaddexp is required. it is padded along the sequence axis with non-finite entries, so it must be read at the query’s position, not reduced over it. A reduction poisons the fold with an infinity from no score. lse<j spans only the cache, so sjj may exceed it — on a trained checkpoint, in 93 of 448 of head-positions at cache length one, by up to 9.7 nats. Short caches are the first tokens of every sequence, so a one-directional form handles none.

indexing the fused kernel’s logsum-exp the sign of sjj − lse<j

17

one stage of a decode step, Early-Q cache side : barrier

cache side : request

batches Wo then Wq

loop

[layer i = 1 .. N] U Q

0

pool : any instance

holds Wk, Wv, the MLP; no request state

holds the KV cache

,

a z

= Z + AWo⊤

= WqN1(U ) q, u 0

0

u

0

asynchronous; nothing waits

par

sweep own cache with q = u + M(N2u ) k , v from N1(z ) z,k,v z

0

0

0

0

0

0

0

0

0

combine; write k , v to own cache 0

0

Figure 4: One stage of an Early-Q decode, as an interaction. Each lifeline’s subtitle says what that participant holds. The barrier and the request are the same device, drawn apart because the projections are batched across whichever requests are ready while the sweeps run free and rejoin later; the par fragment is the dispatch the sweep does not wait for.

F

tuning the split

None of the following bears on a claim in the body. It is what a deployment has to decide once it has accepted the contract of Section 4, collected here so that the body can stay on the one thing it measures. The cache side must batch. Served alone, the two projections it owns are matrix-vector products — one operation per byte read — and cost 2.3 ms per token at Qwen3-0.6B and 15.1 ms at the Qwen3-4B shape, against 0.29 ms and 1.89 ms over a batch of eight. Batching is therefore a precondition of this placement and not an optimisation on top of it; a deployment unwilling to batch should pool the pair instead. One barrier per layer suffices, because Wo closes stage i and Wq opens stage i + 1 with only a residual add and a normalisation between them (Figure 4). The barrier holds those two products and not the sweeps: contexts differ by orders of magnitude, and a wave that locked the sweeps together would run at the pace of its longest member. When to pool the pair instead. Both projections may go to the pool, which then returns the projected query as a first reply and the feed-forward’s results as a second. The cache side holds no weights at all: a pure sweep engine, replicating nothing across a cache tier and pinned to no model, so one fleet may serve several. It costs two replies, twice the wire, a round trip before every sweep, and 27% more arithmetic on the pool. Which is right is a question about concurrency, and the two effects run opposite ways: the round trip is a stall another request fills, while the pool’s extra arithmetic scales with concurrency and never amortises. Below roughly 3 co-resident requests at Qwen3-0.6B (7 at Qwen3-4B) pool the pair; above it, keep it local. Keeping Wo also unloads the resource that saturates. The two sides do not saturate alike. A weight read amortises over the requests sharing a device; a cache sweep does not, because each request sweeps a cache of its own. So Wo on the cache side costs its bandwidth divided by the number of co-resident requests, and is paid for once past 3 of them at Qwen3-0.6B and 7 at the Qwen3-4B shape, even when the cache side is an order of magnitude slower than the pool. What it buys does not amortise away: 13% of a layer’s arithmetic at Qwen3-0.6B and 10% at Qwen3-4B, removed from the pool, which is the shared resource and therefore the one that runs out first under concurrency. It also deletes the watermark. Section 3 has the compute side writing into the cache side’s memory, so it needs a per-layer write watermark to stop a prefetch reading a cache one position short. A pure function returns its results instead: no shared mutable state is left, and the check that guarded it has nothing to guard.

18

G

what a query carries, and whether it compresses

Nothing here is needed for the protocol or for the measurement in Section 7, which establishes what moving a read point costs by training the arms rather than by reasoning about the projections. These are a second and independent line on the same question — how much a query’s staleness can matter — kept out of the body so that the body stays on the one claim it measures. Every figure is forward-only, on released weights, with only the query touched. A query is not a coordinate. It was put to us that a query’s job is to say WHICH — which token, which layer — and not to carry what a token means. Rotary embedding supplies the token coordinate whatever Wq produces, so what remains testable is whether Wq N (x) is close to a per-layer constant. Replace each layer’s query with its own mean over tokens, which preserves the layer exactly and removes every trace of content: model

baseline

query replaced by its layer mean

between-layer share of variance

Qwen3-0.6B Qwen3-4B

0.9022 —

1.8698 (+0.8080) 2.0895 (+1.1664)

67% 73%

The coordinate is entirely intact and the model is destroyed, by an order of magnitude more than the whole conversion costs. The large between-layer share does not rescue the proposal either: a layer’s query meets only that layer’s keys, so a tag saying which layer this is has no reader downstream. What the displacement should be measured against. The conversion moves a query by about a third of its norm. That figure cannot separate two opposite situations, because a query is a common component plus a token-dependent one and only the second is legible to a score. The denominator that matters is the spread across tokens within a layer: model

displacement / norm

displacement / spread

common share of the query

0.384 0.329 0.214

0.642 0.616 0.519

57% 61% 66%

Qwen3-0.6B Qwen3-4B Qwen3.8-27B

Both fall with scale and they fall at different rates, because the common share rises: the larger model’s query sits more in a part no score can distinguish. Neither series tracks the conversion cost closely enough to be offered as its explanation — that cost falls by a factor of four over the same three checkpoints while the ratio falls by about a fifth — and it is reported here as a measurement rather than as a mechanism. Tolerance to a query error, and what carries it. Perturbing a query by the displacement a conversion makes, in a random direction, costs a fraction of what the conversion itself costs; the ratio is within one model and does not depend on where its baseline sits. model Qwen3-0.6B, Qwen3-4B Qwen3-30B-A3B Youtu-LLM-2B Moonlight-16B-A3B

latent KV

sparse routing

conversion / random

no no yes yes

no 128 experts no 64 experts

0.19, 0.20 0.12 0.89 0.99

The split follows the latent cache and not the routing: a mixture of 128 experts sits in the dense band, and a dense model with a compressed cache does not. A ratio near one says the conversion is doing nothing a random perturbation of the same size does not — those checkpoints are intolerant of query error however it arrives, and a deployment converting one should expect the training-free cost to be large before any repair.

19

H

Why an early query is the cheap thing to want

A protocol that starts on the query is worth having only P if a query is cheap to produce early. One half of that is a bound, and needs no data. Writing o = j Aj vj with A = softmax(s), ∂o = Aj , ∂vj

∂o ∂sj = Aj (vj − o).

P

j Aj = 1,

The value path has total gain exactly one. The query path is second order and vanishes at both ends: sharp attention drives Aj to zero away from its peak, and diffuse attention drives (vj − o) to zero. Softmax is also invariant to a constant added to every score, so whatever part of a query’s change is uniform across keys does nothing at all. That bounds the sensitivity but says nothing about the size of the perturbation, which is a question about trained weights and is taken up in Section I, after the cost of the change itself has been measured. The bound predicts something checkable, and it holds. If a stale query hurt because the information it is missing has to be made up from somewhere, the damage would fall as context accumulates: at the first position there is nothing to make it up from, and by the thousandth there are a thousand completed blocks behind it. Measured on Qwen3-0.6B, the cost as a share of that position’s own loss does not decay with context: it rises early and then holds flat: positions

2 to 6

76 to 181

181 to 431

431 to 1024

cost, as a share of the loss at that position

1.7%

7.8%

7.5%

7.3%

Position zero is excluded: with one entry in the cache, attention puts all its mass there whatever the query says, so the cost is zero by construction rather than by measurement. What the flat tail rules out is the reading that earlier positions’ values compensate a stale query — if they did, the share would fall with context, and over the measured range — positions 2 to 1024 — it does not. The cost of an early query is set by how far back it is read and not by how much has been read before it.

20

I

What training puts in the projections

What follows leaves the deployment argument behind and asks what a trained model puts in a query — which the protocol does not need answered, and which is the natural question of a paper that has just moved a query’s read point. Section H bounded the sensitivity; what remains is the size of the perturbation a feed-forward makes [Geva et al., 2021], measured on the official Qwen3-0.6B release — the only fully trained model here and the one a deployment loads — against a randomly initialised model of the same shape. The null matters more than the measurement: several of these statistics have values at initialisation that a reader would take for structure. All three projections collapse; how far the value collapses depends on the budget. We report the stable rank, ∥W ∥2F /∥W ∥22 , rather than a count of singular values above a threshold, because a threshold is a tunable knob:

random initialisation official release a 372M-token model of the same shape

Wq

Wk

Wv

350.7 47.6 43.3

258.9 44.1 29.5

257.6 145.5 55.9

Out of d = 1024, training takes all three a long way down. In the official release the value keeps roughly three times what the query and key do; in a model of the same architecture trained on 372M tokens — three per cent of compute-optimal — it does not, and its stable rank sits beside theirs at 55.9. We report the second row because the first alone would read as a claim about transformers in general, whereas it is a claim about trained transformers. The separation between the value and the other two is something a model acquires with budget, not something the architecture imposes, and a reader measuring this on a small model of their own would otherwise find our number missing and conclude that their measurement was wrong. The query binds to the key more than to the value. We take each projection’s dominant subspace and ask how much of another projection’s energy lies inside it, against the chance level that a subspace of that dimension gives:

Wk inside Wq ’s subspace Wv inside Wq ’s subspace

official release

372M-token model

2.22 1.15

4.48 3.33

as ratios to chance, in every one of 28 layers and at both budgets. The two sides of a score share geometry, as an inner product needs them to, and more than the value does. The ordering survives the change of budget; the magnitudes do not, which is why both columns are here. This is worth stating because the family of head-sharing designs — one key-value head for many query heads [Shazeer, 2019], or a few [Ainslie et al., 2023] — treats the key and the value as one object to be shared, whereas the geometry says they are learned as different kinds of objects. The low rank does not license a factorisation. A stable rank of 47.6 out of 1024 invites the obvious inference: factorise the query and key projections through a rank-r bottleneck and save the difference. Compressing the attention’s projections into a low-rank latent is a design in use [DeepSeek-AI, 2024], so the question is not whether it can be done but whether these matrices in this model permit it. They do not. Truncating Wq and Wk to rank 48 on the official checkpoint costs +1.92 bits per byte; truncating Wv to the same rank costs +4.62; and at rank 512, half the width, the query and key truncations still cost +0.15 bits per byte. Stable rank is a concentration statistic and not a rank: the same matrices need 918 directions to hold 99% of their energy, so the spectrum is concentrated and the function is in its tail. The two are not in tension: a low rank says a generic perturbation reaches the query weakly, and the truncations say the directions it does reach are ones the model cannot be deprived of. 21

Related documents

• The Sample Complexity of Quantum Entanglement Allocation#997054
• Coupling between statistical learning and interference suppression.#662603
• Coupling between statistical learning and interference suppression#486069
• Coupling between statistical learning and interference suppression#358929
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing#259462