Q-First: Attention and Feed-Forward Concurrency at the Smallest Change to the Block

Q-First: Attention and Feed-Forward Concurrency at the Smallest Change to the Block

WenJie Fan Yotta Labs fanwj@mail.ustc.edu.cn fanwenjie@yottalabs.ai https://github.com/fan-wenjie/qfirst

arXiv:2608.15473v1 [cs.DC] 16 Aug 2026

Abstract Disaggregated LLM serving puts the KV-cache sweep on memory-optimised hardware and the projections and feed-forward on compute-optimised hardware [Chen et al., 2024, Liang et al., 2025], then inherits from the decoder block a dependency neither device wants: attention runs first and the feed-forward consumes its output, so within one sequence each side idles while the other works. The usual repair costs one resident KV cache per extra sequence in flight, which is what motivated separating the devices at all. We remove the dependency instead. The sweep needs only the query, and exchanging the two sub-layers makes that query available while the compute side still has work to do; the current key and value follow as a cache write nothing waits on. We state the resulting decode as a protocol, show that it runs on stock kernels, and verify it end to end on a trained checkpoint to a relative error of 3.2×10−3 — with no new operator, no changed shape and no new hardware. We then train the block 8 ways at two seeds each, varying only where the attention reads and holding everything else fixed. At three per cent of compute-optimal a lead in bits per byte measures how much a change disturbed training rather than what it reaches, so we read magnitudes and not rankings. Among the 5 blocks whose feed-forward does not consume their own attention, no read point differs from the one that moves nothing by more than 0.0026 bits per byte — smaller than the gap between an arm and itself at a second seed, 0.0066 — while the same runs resolve a sub-layer exchange 25 times as large. Moving the query early is a change the measurement cannot find, which is what the protocol needs. The reach is bounded: projecting every layer’s query from the network’s input costs +0.0974, refuting a pre-registered threshold at both seeds, so a query may be read one feed-forward early and no further back.

1

Introduction

A decoder layer multiplies by two kinds of weight. The static ones are reused for every token and are arithmetic-dense; the KV cache is touched once per byte and is memory-bound. Serving systems already separate them, placing the cache sweep on memory-optimised hardware and the projections and feed-forward on compute-optimised hardware [Chen et al., 2024, Liang et al., 2025]. What those deployments inherit from the block is its ordering: attention runs first and the feed-forward consumes its output, so within one sequence each device idles while the other works. The usual repair is to keep several sequences in flight, which fills the gap at the cost of one resident KV cache per sequence — and cache residency is what motivated separating the devices in the first place. This paper asks what the cache-side device actually needs to start, and what it costs to give it that. The answer to the first question is exact and the answer to the second is measured. Preprint.

Contributions. • A decode protocol in which the cache-side device is given the query and nothing else, so it may sweep while the compute side still has the feed-forward to run. The split is exact, needs no custom operator, runs on the attention kernel a stock framework already ships, and is verified end to end on a trained checkpoint to 3.2×10−3 relative (Section 3). • An arrangement for two ordinary accelerators, split by which kind of weight they multiply by rather than by layer or head, including the per-layer write watermark without which a prefetched cache block is silently one position short and the arrangement is wrong (Section 4). • The observation that what a device returns is a mergeable aggregate, so one cache may be split across any number of devices, in any way, swept independently and merged in any order (Section 5). • The measurement that makes the above deployable: making the query exist early does not cost quality. In a block whose feed-forward runs first, moving the query’s read point moves the model by +0.0001 bits per byte against the block that moves none, the key’s by -0.0026, and both together by -0.0011 — every one inside a run-to-run floor of 0.010, and every one smaller than the gap between an arm and itself at a second seed. At this budget that is the quantity to read: a lead would say the change perturbs training, not that it is good. In the ordinary attention-first block, re-drawing the boundaries half a sub-layer over moves the model by -0.0139, the only one of these read-point movements to fall outside the floor. Trained from scratch at Qwen3-0.6B with identical parameter counts, identical forward arithmetic, identical data order and identical seeds (Section 8). • The minimality argument that gives the method its name. The device needs the query and only the query: it sweeps keys already in the cache, and the current key is a write nothing waits on. So exactly one of the attention’s three read points has to move. Every alternative moves more — two read points, or the computation graph itself, which is what the parallel block of Wang and Komatsuzaki [2021] and Chowdhery et al. [2023] does when it takes an L-layer stack from 2L sub-layers of serial depth to L (Section 6). What this paper does not claim. No speedup is measured. The decomposition is exact and the arrangement it admits is described including the check without which it is silently wrong, but what it buys needs two devices and a decode loop; on one card the stream serialises everything regardless of what depends on what. The training results are at two seeds and one scale at three per cent of compute-optimal, where a lead measures a disturbance to training rather than a quality reached; every separation from the control is still shrinking at the end of training.

2

Related work

Disaggregated serving. Model-attention disaggregation places the cache sweep and the static weights on different hardware [Chen et al., 2024], and attention disaggregation is used to raise utilisation on otherwise idle resources [Liang et al., 2025]; in-storage offloading takes the same split further, onto computational storage [Pan et al., 2024]. Closest in framing, Ma et al. [2026] route the query itself between instances rather than moving the cache, finding that under latent and sparse attention a query row is smaller than the blocks it selects. That is the same asymmetry this paper exploits, applied across a fabric rather than within a layer: they take the block as given and ask where the query should be sent, and we ask how early in the block it can exist. All of this work takes the decoder block as given. This paper varies the block and leaves the systems argument to them. What the cache holds. Head sharing reduces the cache by giving many query heads one key-value head [Shazeer, 2019] or a few [Ainslie et al., 2023], and latent compression replaces the stored pair with a low-rank code [DeepSeek-AI, 2024]. These approaches change what is written; the question here is when it may be read, which is orthogonal to all of them — every arm below keeps the grouped-query layout it inherits. Parallel blocks. Running the attention and the feed-forward from one input and summing their outputs is a design in use [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023]. It delivers the 2

same overlap and is included here as a baseline, but it is a different computation graph rather than a different reading order, and Section 6 keeps the two apart. Exact attention kernels. The protocol needs the softmax normaliser that a fused kernel already computes [Dao et al., 2022], and the merge it uses is the one such kernels use internally to combine tiles. What is new here is not the merge but what it licenses: the device may be handed the query alone, before the rest of the layer exists.

3

The device needs only the query

At decode step j in one layer, with the cache on a memory-optimised device and the rest of the model on an accelerator:

accelerator device accelerator

sends

computes

qj o<j and lse<j kj , vj (a cache write, unblocking)

the feed-forward, then kj and vj attention over the j − 1 cached positions the combine below

The device attends over the cache alone and returns the attention vector among those positions together with their log-sum-exp. The accelerator, which computed qj and kj itself, finishes with the standard combination of two partial softmax states, writing
qj · k j sjj = √ , out = lerp vj , o<j , σ(lse<j − sjj ) , lerp(x, y, t) = (1 − t) x + t y. d Combining two partial softmax states is a sigmoid of the difference of their log partition functions, and the form says what the step means. For a fixed query, the whole history behaves as one pseudo-token whose score is lse<j and whose value is o<j ; appending a token is then a two-way softmax between that pseudo-token and the new one, and a two-way softmax is a logistic. Thus, lse<j − sjj is the log-odds that the step attends to what it already has rather than to what just arrived, and σ of it is the share the history keeps. Decoding is a fold over that pair,
o<j+1 = lerp vj , o<j , σ(lse<j − sjj ) , lse<j+1 = logaddexp(lse<j , sjj ), which is the sense in which the device’s two return values are sufficient: they are the state of the fold. Overflow, where it actually is. Not in the combine. Written naively as 1/(1 + e−x ), the weight is correct at every magnitude in IEEE arithmetic — a large negative x sends e−x to infinity and the reciprocal to zero, and a large positive x sends the reciprocal to one — and we verify agreement with a library sigmoid out to ±800; on the accelerator used here the library function and that expression agree bit-for-bit at every one of two million points across [−120, 120], and neither returns a non-finite value anywhere. The exponential forms this replaces are not so forgiving: esjj −lse returns NaN past about 89 in single precision. Two places do need care, and neither is the one an implementer expects. The running log partition must use a stabilised logaddexp: computed as log(ea + eb ), it overflows at a = b = 100, which is an ordinary magnitude for a long context, not an extreme one. In addition, the log-sum-exp returned by a fused kernel is padded along the sequence axis with non-finite entries, so it must be indexed at the query’s position rather than reduced over that axis; a reduction silently poisons the state of the fold with an infinity that did not come from any score. The sign of that difference is not a formality. Because lse<j spans only the cache, sjj may exceed it — on a trained checkpoint it does in 41 of 112 of head-positions at a cache of length one, by up to 4.1 nats, a factor of 57 in the weight. Short caches are the first tokens of every sequence, so both directions occur in every sequence, and a form that handles only one of them handles none. The device’s input is the query alone, and this much is true of any decoder: the sweep is over positions already in the cache, so nothing about it needs the current key or value. What differs between blocks is when that query exists. In an ordinary block the attention runs first, so the query is available immediately and the feed-forward is what waits; the two devices alternate. Section 6 gives the reordering that makes both available at once. 3

The current key and value reach the device afterwards as a cache write that nothing waits on, so the expensive sweep and the feed-forward that produces vj proceed concurrently, and what remains serial is a combination of two vectors. The protocol needs no custom operator: the memory-efficient attention of Dao et al. [2022] returns the log-sum-exp exactly, and we verify the whole step against fused attention with a relative error of 3.2×10−3 on a trained checkpoint’s own layers. Two caveats that an implementer should know rather than discover: that backend has no grouped-query kernel [Ainslie et al., 2023], so keys and values must be expanded first; and PyTorch’s flash entry point returned a second output that did not match a hand-computed log-sum-exp under either layout. The intervention is one send, moved earlier. Nothing above asks for a new operator, a new kernel, or a tensor of a different shape. The sweep over the j − 1 cached positions is O(j d) per head and needs the query alone; what remains after it is a rank-one combination, O(d), and at a context of one million that remainder is one 65,536th of the layer’s attention work. A deployment therefore obtains the overlap by computing qj from the block’s input and putting it on the wire at that moment, rather than after the layer’s other arithmetic has finished. The key and the value follow whenever they are ready; the device is not waiting on them. What has to change to make that possible is the block’s ordering, not its parts. In an ordinary block the attention already runs first, so its query is available immediately — and that is exactly the problem, because the feed-forward then waits on the attention’s output and the two devices alternate rather than overlap. Putting the feed-forward first and letting the query still read the block’s input leaves every component, every shape, and every parameter where it was, and leaves the two sides with work to do at the same time.

4

A two-accelerator arrangement

The protocol above says what may be computed when. It does not by itself say where, and the arrangement it admits is worth stating because a reader will otherwise assume that near-memory hardware, which does not yet exist, is required. Two ordinary accelerators suffice, split by what kind of weight they multiply by rather than by layer or by head: one holds the static weights and runs every projection and feed-forward, while the other holds the KV cache and does nothing but sweep it. This is the split that model-attention disaggregation already builds [Chen et al., 2024]; what changes is only what the cache side is given and when. We call them the MLP side and the attention side. One layer, one decode step j, reading down: MLP side

attention side

qj → computes MLP, then kj , vj

receives qj sweeps the cache for o<j , lse<j parks the pair; stages layer L+1’s KV from its own VRAM

reads the pair, combines with vj kj , v j →

shared, both can reach

o<j , lse<j

appends to layer L’s cache

Three properties of this arrangement are not incidental. The result is parked, not pushed. The attention side writes o<j and lse<j to memory the MLP side can read and moves on, rather than waiting for a handshake. The two sides are then coupled only by data, not by arrival order, and the sweep of one layer overlaps the feed-forward of the same layer — which is the whole point of sending the query first. Across layers, not only within one. The table above is one layer of one step, and read that way the device’s sweep overlaps the feed-forward of the same layer. Drawing the stack one half-layer over makes the overlap span layers instead, which is the form worth handing to an implementer: [ Attn1 ] [ MLP1 ∥ Attn2 ] [ MLP2 ∥ Attn3 ] · · · [ MLPN −1 ∥ AttnN ] [ MLPN ] Each middle stage holds one feed-forward and the next layer’s attention, and both read the stage’s input. Thus, layer i + 1’s query exists before layer i’s feed-forward runs, and the device sweeps layer i + 1’s history — returning that layer’s attention vector and log partition — during the entire time

4

the accelerator spends on MLPi . The join needs only the value, which the feed-forward has by then produced. Two stages of the N + 1 lose the overlap and no more: the first has no feed-forward in front of its attention, and the last no attention behind its feed-forward. The re-bracketing is exact — it holds the same N attentions and N feed-forwards, and we verify it produces bit-identical outputs — so this is a way of drawing the same computation, not a different one. The wait is spent staging the next layer, and it is a short wait. Having parked its result, the attention side has nothing to do until the next query arrives, and what it will need then is the next layer’s KV. The window is one feed-forward long, which fixes where that data may come from: it is a movement within the attention device’s own memory — VRAM into whatever the sweep kernel reads — and not a fetch across a host interconnect, which would not complete in time. The cache resides on this device already; that is what makes it the memory-optimised side. What the staging buys is that the sweep kernel does not begin by waiting on its own working set, and it is the one piece of work known in advance, since the layers are traversed in a fixed order and the prefetch has no branch to predict. A prefetched cache can be stale, and must be checked. The attention side is fetching layer L+1’s cache while the MLP side is still writing layer L+1’s entry for position j − 1 — the two are issued by different sides of the same step, and nothing orders them. A prefetch that starts too early is short by one position, and the resulting attention is over j − 2 positions rather than j − 1: numerically plausible, silently wrong, and invisible in any output check. The fix is a per-layer write watermark. The MLP side advances wL when its write of (ki , vi ) into layer L has landed; the attention side accepts a prefetched block for layer L at step j only when wL ≥ j − 1, and otherwise fetches the missing tail, which is one position. The check is a comparison of two integers per layer per step, and the arrangement is not correct without it. Where the overlap is worth having, and where it is not. The arrangement hides the sweep behind the feed-forward, so it pays exactly while there is a feed-forward long enough to hide it behind. At batch one both sides are bandwidth-bound and the comparison is bytes moved per layer per decode step: the feed-forward reads its weights once, 19 MB, whatever the context; the sweep reads 4.1 KB per cached position. They are equal at 4,608 tokens for Qwen3-0.6B, and at 36,480 tokens for the Qwen3-4B shape, whose larger feed-forward moves the point out rather than in. Below that, the feed-forward is the longer of the two and the sweep hides inside it. Above it the sweep dominates, and at a context of one million tokens it is 228 times the feed-forward: there is no longer anything to hide it behind, the cache side is the bottleneck rather than the idle party, and starting it earlier buys a diminishing fraction of a step. The premise of the arrangement — that the memory side would otherwise wait — inverts. We state the boundary rather than the best case because a reader deploying this needs to know which side of it they are on. It also says which of this paper’s two results applies where: at long context the question stops being when the sweep may start and becomes how many devices it may be spread across, and that is Section 5, where the pair a device returns is a mergeable aggregate and a cache may be split any way at all. What is not claimed. No speedup is measured here. That needs two devices and a decode loop, and on one card the stream serialises everything regardless of what depends on what. What is established is the decomposition: that the sweep needs the query alone, that the combine is exact, and that a trained model decodes through it with a relative error of 3.2×10−3 . Whether the arrangement pays is a question about interconnect latency against feed-forward time, and it should be answered by building it rather than by arithmetic in a paper. 4.1

Which side owns the request, and which side runs the projections

The arrangement above is symmetric in the telling, and a deployment has to break that symmetry: one side owns the request. The natural answer is that the attention side does, and it is the right answer, but not for the reason usually given for it — that the MLP side holds no context and need only be asked for a result. Holding state does not make a component a caller. A database holds far more state than the service in front of it and is nonetheless the callee, because what determines who

5

drives is not who stores the most but who owns the sequential dependency: here, the residual stream x1 → x2 → · · · → xN . Stated as “the stateless side is subordinate”, the argument equally supports its own converse, in which the attention side is a stateful service that returns a weighted sum when asked. Two properties do decide it. The MLP side must be shared to be efficient. At batch one its feed-forward reads 19 MB of weights to produce a single token’s worth of arithmetic. Only batching across different requests amortises that read, so the MLP side wants to be a pool serving many requests at once — and a pool is called. It cannot own the control flow of any one request without giving up the batching that is its reason for existing. The cache cannot move. At Qwen3-0.6B the cache is 112 KB per token across the stack, or 107 GB at a context of one million; at the Qwen3-4B shape, 144 KB and 137 GB. Migration is not among the options, so a request is pinned for its lifetime to the device holding its cache. Together these place the request on the attention side and make the MLP side a stateless pool, which is the conclusion above with the load-bearing part supplied. The pool is a pure function, and two weights cannot join it. That the cache sits on the attention side does not settle which device multiplies by Wq , Wk , Wv , Wo . Those are static weights like any other, and a pool serving many requests wants every one it can have, because a service holding no request state cannot be blocked by a caller that stalls: a stalled caller simply stops calling. So the pool is given every weight it can take, and the question is which it cannot. One call per stage, with the state on the wire because the pool keeps none: the cache side sends the pool replies the cache side then

the stage input u z = u + M (N2 u), and k, v read from N1 (z) sweeps with its own q, combines, and forms u′ = z + Wo a

The key and value projections are inlined into the same call, so the pool returns cache entries rather than a hidden state the cache side would have to send back to obtain them. The cache side writes them into its own cache, and nothing writes into anyone else’s memory. The query and output projections stay together, on the cache side. Wq must be there because a query computed in the pool would arrive with the reply, that is, no earlier than the feed-forward it was meant to overlap. But Wq alone is not enough: the value it is applied to is u′ = z + Wo a, the previous stage’s join. With Wo in the pool, the cache side must wait for that join to come back before it can form its next query, which puts a round trip in front of every sweep — the very ordering the arrangement exists to avoid, reintroduced one layer lower down. The two projections are a pair, and the pair belongs with the cache. The cache side therefore holds 8.4 MB per layer and 235 MB for the stack at Qwen3-0.6B (41.9 MB and 1,510 MB at the Qwen3-4B shape), against 23.1 MB per layer in the pool. Everything else is pooled. Keeping Wo also unloads the resource that saturates. The two sides do not saturate alike. A weight read amortises over the requests sharing a device; a cache sweep does not, because each request sweeps a cache of its own. So Wo on the cache side costs its bandwidth divided by the number of co-resident requests, and is paid for once past 3 of them at Qwen3-0.6B and 7 at the Qwen3-4B shape, even when the cache side is an order of magnitude slower than the pool. What it buys does not amortise away: 13% of a layer’s arithmetic at Qwen3-0.6B and 10% at Qwen3-4B, removed from the pool, which is the shared resource and therefore the one that runs out first under concurrency. It also deletes the watermark. The arrangement of Section 4 has the compute side writing into the cache side’s memory, which is why it needs a per-layer write watermark to stop a prefetch reading a cache short by one position. A pure function returns its results rather than writing them: there is no shared mutable state left, and the check that guarded it has nothing to guard. What it costs. Statelessness puts the request’s state on the wire and the layer loop on the network. Per stage the cache side sends 2 KB and receives 6 KB, which is 224 KB per token over 28 round trips at Qwen3-0.6B and 504 KB at the Qwen3-4B shape. The bytes are immaterial; the round trips 6

are the cost and they are serial. What pays for them is that the sweep begins the moment the request is dispatched — the query is local and needs nothing from the pool — so a round trip is hidden whenever it is shorter than a sweep, and the margin grows with context rather than shrinking. Across a wide-area link it does not, and that is where the arrangement stops. Pooling the feed-forward and the key and value projections moves the crossover of Section 4 to 3,584 tokens at Qwen3-0.6B and 28,800 at the Qwen3-4B shape. The cache side must batch, and once per layer suffices. The two projections it owns are matrixvector products when a request is served alone: one arithmetic operation per byte read, so the 8.4 MB is paid to produce a single token’s worth of work. Unbatched that is 2.3 ms per token at Qwen3-0.6B and 15.1 ms at the Qwen3-4B shape, which is not a cost worth discussing — it simply defeats the arrangement. Over a batch of eight it is 0.29 ms and 1.89 ms. Batching is therefore not an optimisation on top of this design; it is a precondition of it. A deployment unwilling to batch on the cache side should put the pair in the pool and accept the 18 microseconds of round trip (52 at Qwen3-4B) that this buys back, which is the alternative discussed below. Keeping the pair locally and serving requests one at a time is the worst of the three and should not be built. One barrier per layer, not two. The two products look like separate synchronisation points and are not: Wo closes stage i and Wq opens stage i + 1, with only a residual add and a normalisation between them (Figure 1). A single batching point per layer covers both. The barrier holds the products, not the sweeps. Contexts differ by orders of magnitude, so a wave that locked the sweeps together would run at the pace of its longest member. The sweeps are released to run independently and rejoin at the next barrier; a long-context request simply takes part in fewer of them. This is ordinary continuous batching and needs no machinery a serving system does not already have. What it does need, and what should be written down rather than buried, is the window over which a barrier accumulates its batch: requests free-running across N layers are scattered, and an effective batch of eight without a collection window would want 224 resident requests at Qwen3-0.6B. Fusing the two products was considered and rejected. The normalisation between them is not an obstacle in principle: the scalar that RMSNorm divides by cancels against the per-head query normalisation this model applies after projection, so q ′ = qnorm(Wq Λz + (Wq ΛWo )a) for the diagonal Λ of the norm’s gains, and the two matrices concatenate into one product on [z; a]. It does not pay: u′ must be sent to the pool regardless, so Wo survives the fusion, and the total becomes 16.8 MB per layer against 8.4 (75.5 against 41.9 at Qwen3-4B). When to put the pair in the pool instead. Both projections may be pooled, with the pool returning the projected query as a first reply and the feed-forward’s results as a second. The overlap survives, since the pair costs far less than the feed-forward behind it, and the cache side then holds no weights at all: a pure sweep engine, replicating nothing across a cache tier and pinned to no particular model, so one cache fleet may serve several. It costs two replies instead of one, twice the wire, a round trip before every sweep, and 27% more arithmetic on the pool. Which is right is a question about concurrency rather than about hardware, and the two effects run opposite ways. The round trip is a stall, so at high concurrency another request fills it and it costs nothing in throughput; the pool’s extra arithmetic scales with concurrency and never amortises. Below roughly 3 co-resident requests at Qwen3-0.6B (7 at Qwen3-4B) the pair belongs in the pool; above it, on the cache side. A deployment may hold it on both sides — it is 235 MB — and choose per wave, at the cost of a pool that must offer both entry points. Where a shared query source would have changed this, and why it does not. We measured such an arm and it is refuted (Section 8); the argument for running it is recorded here because it is why the run was worth its cost, and because it names the limit the arrangement keeps. The arrangement above hides each sweep behind one feed-forward, and that is a bounded quantity: past the crossover the sweep no longer fits, and the overlap saturates. A query read from the network’s input rather than from its own layer’s would not saturate, because every layer’s query exists at the step boundary, where every layer’s cache is complete through the previous position. All 28 sweeps could then be issued at once and would have the whole stack’s feed-forwards to hide in rather than one.

7

one stage of a decode step, Early-Q cache side : barrier

cache side : request

batches Wo then Wq

loop

[layer i = 1 .. N] U Q

0

pool : any instance

holds Wk, Wv, the MLP; no request state

holds the KV cache

,

a z

= Z + AWo⊤

= WqN1(U ) q, u 0

0

u

0

asynchronous; nothing waits

par

sweep own cache with q = u + M(N2u ) k , v from N1(z ) z,k,v z

0

0

0

0

0

0

0

0

0

combine; write k , v to own cache 0

0

Figure 1: One stage of an Early-Q decode, as an interaction. Each lifeline’s subtitle says what that participant holds: the cache side keeps the query and output projections, which are a pair – the query is projected from the value the output projection produces — and the cache; the pool holds every other weight and no request state, so a caller that stalls blocks nothing. The barrier and the request are the same device, drawn apart because they synchronise differently: the two projections are batched across whichever requests are ready, while the sweeps run free and rejoin later. The dispatch to the pool is asynchronous and the sweep does not wait for it, which is what the par fragment says; everything above that fragment exists to make the two branches inside it able to run at once. What such an arm would not change is the number of round trips. The residual chain is untouched, so the pool is still called once per layer in order; and the attention at a layer still needs that layer’s entry for the current position, which exists only when the walk reaches it. What is lifted out of the critical path is the sweep over history, which is the expensive part; what remains sequential is a merge of two aggregates, which is not. None of that is available: projecting every layer’s query from the network’s input costs +0.0974 bits per byte at both seeds, so the saturation described here is a limit the method keeps rather than one it escapes. Owning the request is not the same as calling synchronously. The contract above is a call per layer, and read carelessly that is N blocking round trips on the critical path, which would hand back exactly the concurrency this paper is about. It is not blocking: the cache side dispatches and then immediately projects its own query and sweeps, and the reply it is waiting for arrives during that sweep rather than in place of it. Ownership means holding the state and the ordering. It does not mean waiting. This division suits a heterogeneous host particularly well, and that is worth saying because it is the case a reader is most likely to have. Where the cache sits beside a large but slower memory, the cache side is also the weaker side, and every weight left there is arithmetic asked of the device that is already the constraint. Leaving it the two projections it cannot give up and nothing else — 8.4 MB per layer against 23.1 MB in the pool — puts most of the arithmetic on the faster device and all of the growing bytes on the one built to hold them.

8

5

The cache may be split anywhere

The two values the device returns are not an ad hoc pair. For a set S of cached positions with scores si and values vi , write X X Z(S) = esi , W (S) = esi vi , O(S) = W (S)/Z(S), i∈S

i∈S

and (W, Z) adds componentwise over disjoint sets. That is all the algebra there is: the merge is vector addition, its associativity and commutativity are addition’s, and calling the result a monoid names a property of Rd × R rather than one of attention. We say so plainly because the useful part is elsewhere. What a device returns is (O, log Z) — the same pair normalised by its total, with the total kept in logs — and the merge in those coordinates is the combine of Section 3:
O(A ∪ B) = lerp O(B), O(A), σ(log Z(A) − log Z(B)) ,
log Z(A ∪ B) = logaddexp log Z(A), log Z(B) . The reparameterisation is the contribution, not the algebra. In the undisguised coordinates a shard of 400 positions at a score spread of 30 already carries Z of about 8×1036 and ∥W ∥ of about 1.6×1037 , one order of magnitude below what single precision holds; a wider spread or a longer shard overflows, and the sums cannot be exchanged between devices at all. In the logged coordinates O is a convex combination and log Z is a number near 85. The consequence is a deployment-level one. A cache split across any number of devices, in any way, swept independently and merged in any order, returns what one sweep over the whole would: we check 200 random four-way splits and find the merged result within 1.4×10−14 of it. Nothing about the split has to be balanced, contiguous, or known in advance, and no device needs to know what any other holds. The pair a device already returns is exactly the state that makes this so. One asymmetry is worth an implementer’s attention: the identity is degenerate. An empty shard is log Z = −∞ and a vector that is never read, so a device with nothing to contribute may return whatever is in its buffer, provided its log partition says so.

6

Where a query may be read from, as algebra

The protocol above starts on the query, so it is worth saying precisely what freedom a block has in where that query comes from. Write the residual stream’s two updates as operators, each including its own normalisation: A for attention, M for the feed-forward. An ordinary decoder block is their composition, S = (I + M ) ◦ (I + A), and the attention takes its query, key, and value from one point of the stream. A block that reads the query earlier does not change this composition. It splits the attention’s arguments: Q = (I + Ax;z ) ◦ (I + M ),

z = x + M (x),

where Ax;z scores from x and loads values from z. Over a stack the two are conjugate,  L QL = (I + A) ◦ (I + M ) = (I + M )−1 ◦ SL ◦ (I + M ), so a stack of such blocks is the ordinary stack with its layer boundaries drawn half a sub-layer over: the interiors agree exactly and only the two ends differ. We verify this constructively rather than by inspection, building the shifted stack as L − 1 fused blocks between two half blocks and checking that it is bit-identical to the ordinary one. The single place the two differ is the subscript. This matters for a protocol rather than for a taxonomy, and it gives the family an exact boundary. The conjugation leaves each of the attention’s three arguments free to be taken from before the feed-forward or after it, and the device’s requirement selects among them: the sweep begins when the query arrives, so it is q from the earlier point that buys the overlap, and q alone. Reading the key early as well changes nothing about when the device may start; leaving the key late changes nothing either, because the device works over keys already in the cache and the current one is a write nothing 9

query across a layer boundary

query within one block

⋮

⋮

Attnℓ − 1

Attnℓ − 1

+

+

q

q

one block

+

one block

MLPℓ − 1

k, v

Attnℓ

MLPℓ

+

k, v

Attnℓ

+

+

MLPℓ

MLPℓ + 1

+

+ ⋮

⋮ attention ℓ beside feed-forward ℓ

−1

attention ℓ beside feed-forward ℓ

Figure 2: The same wiring, bracketed two ways. Left: the query is tapped at the entrance of feedforward ℓ − 1 and crosses into attention ℓ, so the block pairs an attention with the feed-forward above it. Right: the block takes its feed-forward first, and the identical tap falls inside it. The two panels are drawn identically because the wires are identical: in both, exactly one feed-forward separates the query’s tap from the key’s and value’s, which is the conjugation above. What differs is the bracket — and therefore which pretrained feed-forward sits beside which pretrained attention, plus a half block at each end. We check this by decomposing both stacks into the same unit and comparing outputs, which agree to 0 in float32. The distinction is easy to lose: the left panel does not read a staler query than the right, and an implementation that assumes it does will attribute a pairing effect to staleness. waits on — which we verify rather than argue, the split holding to 5.4×10−3 relative for a block whose key is read after the feed-forward. A block that takes its query from the later point is simply the ordinary block with its two sub-layers exchanged, and gets no overlap at all: its device waits on the same feed-forward the accelerator is running. Thus, reading q from the earlier point is necessary and sufficient, and it is all that a deployment has to change. Whether it costs anything in quality is an architectural question, and Section 8 measures it directly. What is not a reading order. A reader reaching for the arrangement that sums rather than composes, P = I + A + M [Wang and Komatsuzaki, 2021, Chowdhery et al., 2023], will find it delivers the same overlap and should know it is not the same kind of change. S and Q differ in where three arguments are read; P differs in the graph. Its two branches never meet inside the block, so an L-layer stack has L sub-layers on its longest dependency chain where S and Q have 2L, and its feed-forward no longer consumes the attention’s output at any layer. Which of the three a block belongs to is decidable rather than a matter of description. For a block B taking its two sub-layers scaled, the mixed second difference     ∆ = B(αA, µM ) − B(αA, M ) − B(A, µM ) − B(A, M ) vanishes identically exactly when the two branches never meet. Measured on this model, it is zero for P and nonzero for S and for every Q. We raise the distinction rather than pursue it because the two changes have different consequences and only one of them is free. Halving a stack’s serial depth is a change to what the model can compose; moving a read point is not, and leaves the parameter count, the forward arithmetic, and the composition where they were. Which of them costs quality, and how much, is measured for the read points in Section 8; the serial-depth change is left to separate work.

10

7

Why an early query is the cheap thing to want

A protocol that starts on the query is worth having only P if a query is cheap to produce early. One half of that is a bound, and needs no data. Writing o = j Aj vj with A = softmax(s), ∂o = Aj , ∂vj

∂o ∂sj = Aj (vj − o).

P

j Aj = 1,

The value path has total gain exactly one. The query path is second order and vanishes at both ends: sharp attention drives Aj to zero away from its peak, and diffuse attention drives (vj − o) to zero. Softmax is also invariant to a constant added to every score, so whatever part of a query’s change is uniform across keys does nothing at all. That bounds the sensitivity but says nothing about the size of the perturbation, which is a question about trained weights and is taken up in Section 10, after the cost of the change itself has been measured.

8

What it costs to make the query exist early

The protocol needs the query before the feed-forward. Whether a block can be trained that way is a separate question, and this section answers it twice: once in each of the two topologies that admit the change. Each comparison holds the computation graph, the components, their shapes, the parameter count, the forward arithmetic, the data order and the seed fixed, and moves the read point alone. The arms fall into two families, by whether a block’s feed-forward consumes that block’s own attention. In family A it does not: the feed-forward runs first, or the two run from the same input. In family B it does, which is the ordinary arrangement. Every run below is at two seeds, and both are printed rather than a mean alone, because a mean of two hides which of them the spread came from. Family A: the feed-forward does not consume this block’s attention. Exchange the two sublayers and the query and key may each read the residual stream before the feed-forward or after it; the value always reads after, since reading it early as well is the parallel arrangement, which changes the graph rather than the reading order. The control is Swapped, the corner that moves no read point, and every difference in this table is against it. arm

reads early

seed 0

seed 1

mean

vs Swapped

Swapped Early-K Early-QK Early-Q Parallel

nothing the key query and key the query a different graph

1.2388 1.2352 1.2369 1.2369 1.2335

1.2381 1.2365 1.2377 1.2403 1.2400

1.2384 1.2359 1.2373 1.2386 1.2367

— -0.0026 -0.0011 +0.0001 -0.0017

The 5 arms span 0.0027 bits per byte, against a run-to-run floor of 0.010, and no arm differs from the control by more than 0.0026. Nothing in this family is distinguishable from anything else in it. That is the result the protocol needs and the only claim this table makes: within a block whose feed-forward does not wait on its attention, moving the read point is a change this campaign cannot find, so the query may be read early and handed to a device. The paragraph after the second table says why “cannot find” is the right form of that sentence and “costs nothing” is not. Family B: it does. The ordinary ordering. Two arms change where the query comes from without exchanging anything: the shifted stack re-draws the boundaries half a sub-layer over, as Section 6 does, and takes each query from the input of the preceding feed-forward rather than its output; the shared-query arm projects every layer’s query from the network’s input instead of its own layer’s. arm

the query is read from

seed 0

seed 1

mean

vs Standard

Standard Shifted-Q Single-Q

its own layer the preceding feed-forward’s input the network’s input

1.2998 1.2862 1.3976

1.3038 1.2897 1.4008

1.3018 1.2879 1.3992

— -0.0139 +0.0974

11

This family spans 0.1113, eleven times the floor, and the two treatments differ from the control in opposite directions. Against Standard the shifted stack is -0.0139 and the shared source is +0.0974 (+0.0979 and +0.0970 by seed). The second was pre-registered with a threshold of +0.05 for refutation and exceeds it at both seeds: a query may be read one feed-forward early at no cost, and may not be read from the network’s input. That is where the method’s reach ends, and knowing the boundary is what the arm was run for. The first, -0.0139, is outside the floor. We list it and build nothing on it, for the reason the next paragraph gives about a larger figure of the same kind. At this budget a lead is a disturbance, not a verdict. These runs are at three per cent of computeoptimal, and far from convergence a difference in bits per byte is a difference in optimisation trajectory rather than in reachable quality. The point is familiar in the other direction: a smaller model leads a larger one through much of early training and the ordering reverses later, so the lead was never a statement about which model was better. It is the same here. What a gap against the control measures at this budget is how much the change disturbed training, and the useful question is not which arm leads but how far each moved. Read that way the two tables say something the phrase “costs nothing” understates. The 5 arms of family A differ from the control by at most 0.0026 bits per byte, which is smaller than the gap between one arm and itself at a second seed, 0.0066. They are not merely close; they are closer to each other than a run is to its own repetition. That is the strongest form the title’s claim can take at this budget: the change is small in effect, and not only small to describe. This is not a way of turning any null into a result. A claim that a disturbance is small is worth nothing unless the same measurement demonstrably resolves a large one, and these runs do: exchanging the sub-layers moves 25 times as far as any read point does, and projecting the query from the network’s input moves 38 times as far. The instrument has range; family A produces no signal in it. And the flatness is not exact throughout. The read points do separate in the middle of training — above the seed band from roughly the first thousand steps to the five thousandth, by at most half again — and the separation is gone by the last third. We report the transient rather than describing the family as identical, and note that a campaign stopped at its widest point would have reported a read-point effect that a longer one does not. The families are not to be compared across, and the gap is listed rather than claimed. Every arm of family A sits below Standard, by -0.0634 to -0.0659, and it would be easy to read that as an architectural gain. We list the numbers and make no such claim. Three reasons, each measured rather than supposed: • The two stacks are the same alternating chain of sub-layers and differ only in the two half-layers at the ends, so the quantity is not a comparison between orderings in any clean sense. The shifted arm shows this directly: it is family B by construction and the same chain as family A in its interior, and it lands at -0.0139, a fifth of the way. A gap that a re-bracketing of the same operations moves by that much is not a property of either ordering. • The tied embedding, which is also the readout, is frozen at its random initialisation throughout, so one ordering hands that representation to a position-wise feed-forward first and the other to attention first. Which sub-layer sees a frozen random embedding first is a property of this setup, not of the architecture. • It is a difference in convergence rate rather than in reached quality. It is -0.1036 at the first evaluation, widens to -0.1722, and then narrows monotonically to -0.0634 by the end of training without having stopped. An arm trained with the embedding unfrozen, in both orderings, would settle it. That is a change to the control this campaign fixed, so it is left to separate work rather than folded in here. Nothing this paper claims rests on the gap: the argument needs family A to be flat, which is a statement inside one family, and needs the query to be readable early at no cost, which is the same statement. Parameters and arithmetic. Every arm in both tables carries 596.05M parameters and identical forward arithmetic, measured rather than assumed, so no comparison here needs a budget argument 12

and none is offered. The parallel arrangement of Wang and Komatsuzaki [2021] and Chowdhery et al. [2023] is listed in family A because it shares the property the family is defined by, with the caveat that it is a different graph: its branches never meet inside a block, and an L-layer stack has L sub-layers on its longest dependency chain where the others have 2L. What two seeds cannot say. Both seeds are printed for every arm, and they agree: no arm moves by more than 0.0066 between them, against a floor of 0.010. Two seeds are enough to say that a claim of smallness was not an accident of one initialisation, which is what the first table needed, and they were the easiest way for that claim to have been overturned. They are not enough for a standard error, and none is quoted. The seed drives both initialisation and data order, so the common factor cancels in a paired difference and it is paired differences that are reported; a spread across seeds within an arm is shown rather than summarised for the same reason.

9

What was measured, and on what

Each of the 8 arms is Qwen3-0.6B trained from scratch at two seeds, with every layer converted, on 372M tokens of WikiText-103 over three epochs: 11,355 optimiser steps at a peak learning rate of 4×10−3 under a cosine schedule, sequence length 1,024, 32,768 tokens per step, AdamW, and the tied embedding — which is also the readout — frozen at initialisation so that the comparison is between decoder stacks rather than between vocabularies. Normalisation weights are held in float32; in pure bfloat16 they do not move at all, since they begin at exactly 1.0 where the spacing is 0.0078 and the optimiser’s step is three orders of magnitude smaller. Attention runs on the flash backend, pinned rather than left to a heuristic and recorded in each run, because the alternatives differ from it by up to 1.6×10−2 on these shapes — larger than the run-to-run floor of 0.010, and 6 times the spread of family A, so the choice of backend would dominate what this study resolves if it were left free. Every run is compiled. Each takes 2.1 to 2.8 hours on one accelerator and the 16 runs reported here are 34 GPU-hours; we give wall clock because it says what the work cost, and not as a measure of how fast an arm is, for which it is the wrong denominator. The protocol is verified against fused attention on synthetic tensors at four shapes up to 2048 positions, and end to end on a trained Qwen3-0.6B checkpoint — three layers of 28, real weights, the rotary rotation applied on both paths — with a relative error of 3.2×10−3 . The projection geometry is measured on the official Qwen3-0.6B release, which is the only fully trained model here and the one a deployment loads, against a randomly initialised model of the same shape as the null; the truncation series is the same release, evaluated on 24 batches of held-out text. None of it requires training, and the whole of it runs in under 0.5 GPU-hours on one accelerator. Wall clock says nothing about how fast any of this would run in a deployment, which is not measured here and cannot be: that needs two devices and a decode loop, and on one card the stream serialises everything regardless of what depends on what.

10

What training puts in the projections

The two sections that follow leave the deployment argument behind and ask what a trained model puts in a query — a question the protocol does not need answered, and the natural one to ask of a paper that has just moved a query’s read point. Section 7 bounded the sensitivity; what remains is the size of the perturbation a feed-forward makes, which is a question about trained weights and about what a feed-forward writes [Geva et al., 2021]. We measure it on the official Qwen3-0.6B release — the only fully trained model here and the one a deployment loads — against a randomly initialised model of the same shape. The null matters more than the measurement: several of these statistics have values at initialisation that a reader would take for structure. All three projections collapse; how far the value collapses depends on the budget. We report the stable rank, ∥W ∥2F /∥W ∥22 , rather than a count of singular values above a threshold, because a threshold is a tunable knob:

13

random initialisation official release a 372M-token model of the same shape

Wq

Wk

Wv

350.7 47.6 43.3

258.9 44.1 29.5

257.6 145.5 55.9

Out of d = 1024, training takes all three a long way down. In the official release the value keeps roughly three times what the query and key do; in a model of the same architecture trained on 372M tokens — three per cent of compute-optimal — it does not, and its stable rank sits beside theirs at 55.9. We report the second row because the first alone would read as a claim about transformers in general, whereas it is a claim about trained transformers. The separation between the value and the other two is something a model acquires with budget, not something the architecture imposes, and a reader measuring this on a small model of their own would otherwise find our number missing and conclude that their measurement was wrong. The query binds to the key more than to the value. We take each projection’s dominant subspace and ask how much of another projection’s energy lies inside it, against the chance level that a subspace of that dimension gives:

Wk inside Wq ’s subspace Wv inside Wq ’s subspace

official release

372M-token model

2.22 1.15

4.48 3.33

as ratios to chance, in every one of 28 layers, and at both budgets. The two sides of a score share geometry, as an inner product needs them to, and more than the value does. The ordering survives the change of budget; the magnitudes do not, which is the reason both columns are here rather than only the one that suits the argument. This is worth stating because the family of head-sharing designs — one key-value head for many query heads [Shazeer, 2019], or a few [Ainslie et al., 2023] — treats the key and the value as one object to be shared, whereas the geometry says they are learned as different kinds of objects.

11

The low rank does not license a factorisation

A stable rank of 47.6 out of 1024 invites the obvious inference: factorise the query and key projections through a rank-r bottleneck and save the difference. Compressing the attention’s projections into a low-rank latent is a design in use [DeepSeek-AI, 2024], so the question is not whether it can be done but whether these matrices in this model permit it. They do not. Truncating Wq and Wk to rank 48 on the official checkpoint costs +1.92 bits per byte; truncating Wv to the same rank costs +4.62; and at rank 512, half the width, the query and key truncations still cost +0.15 bits per byte. Stable rank is a concentration statistic and not a rank: the same matrices need 918 directions to hold 99% of their energy. The spectrum is concentrated and the function is in its tail. The two results are not in tension. What the low rank says is that a generic perturbation — a feed-forward’s output, which is not aligned with anything in particular — reaches the query weakly. What the truncation says is that the directions it reaches weakly are not directions the model can be deprived of.

12

Conclusion

The cache-side device of a disaggregated decoder needs the query and nothing else. That is an identity, it runs on stock kernels, and the state it returns is a mergeable aggregate, so the cache may be split across as many devices as one likes and merged in any order. Making a block able to hand over its query early is a smaller change than it appears: the feed-forward moves in front of the attention and the query keeps reading the block’s input, which leaves every 14

family A, against Swapped minus the family's control

0.03

family B, against Standard

Swapped, its own two seeds Early-K Early-Q 0.3 Early-QK Parallel

0.02 0.01

Standard, its own two seeds Shifted (query early) Shared query (from the network input)

0.2

0.00

−0.01

0.1

−0.02 0.0

−0.03 0

2000

4000

6000

8000

optimiser steps

10000

0

2000

4000

6000

8000

optimiser steps

10000

Figure 3: Each family against its own control, at both seeds: family A against Swapped, the corner that moves no read point, and family B against Standard. The grey region is the control’s own seed-to-seed gap — the width inside which nothing is a difference — and the two panels do not share an axis, their ranges differing by a factor of nine. Left: no read point leaves the band by much or for long, and all four are inside it over the last third; the excursion around a thousand steps is the transient discussed above, and it is the only place the arms exceed the band at all. Right: the same picture for the arms that keep the standard order, where Shifted-Q sits just below the control at -0.0139 and the shared query source sits far above it at +0.0974, with a spike near three thousand steps from which it recovers to no better than where it was. An earlier version of this figure plotted family A against Standard: a cross-family quantity this paper declines to interpret, and one about twenty-five times larger than the differences the panel exists to show, so the arms lay on one another at a scale where anything would have. component, every shape, the parameter count, the forward arithmetic, and the serial depth where they were. Measured at Qwen3-0.6B over two seeds, no read point in that family separates from any other by more than 0.0026 bits per byte — less than the gap between one arm and itself at a second seed — while the same runs resolve a sub-layer exchange 25 times as large and a shared query source 38 times as large. The right reading of that is a magnitude and not a ranking, and this is worth stating plainly because the study began by looking for a ranking. At three per cent of compute-optimal a lead in bits per byte is a statement about an optimisation trajectory, so “which arm wins” is not a question this budget can answer about any architecture, and an arm that led by a wide margin would be evidence that the change perturbs training rather than that it is good. The question the budget does answer is how far a change moves the model, and a change the instrument cannot find — in runs where it finds two others at 25 and 38 times the scale – is the outcome a protocol built on that change requires. Where the freedom ends is measured rather than assumed: a query projected from the network’s input rather than its own layer’s moves +0.0974, so it is one feed-forward wide and no wider. What remains is the part a single card cannot answer. Whether the arrangement pays is a ratio between an interconnect’s latency and a feed-forward’s duration, and it should be settled by building it. The code is released for that purpose. Reproducibility. Every number in this paper is generated from run records by a script in the repository; none is typed into the source, and a number that has not been measured raises a build error rather than appearing as a placeholder. The block definitions, the trainer, the protocol tests, and the analysis are all included, along with the exact command for each arm. The pre-registrations are in the supplementary material under protocols/, one file per gate, each stating its verdict thresholds and the author’s prior before the arm ran; gate_SQ.md is the one the +0.05 refutation threshold comes from. The table each campaign was launched from is there too, under run/experiments/, one row per run.

15

13

Limitations

One family, one scale, one corpus. The geometry is measured on a single model family at a single size, and the protocol is family-independent while the numbers are not. The training budget is three per cent of compute-optimal, which is the regime in which a difference in convergence and a difference in reachable quality are hardest to tell apart — and the reason Section 8 reads magnitudes rather than rankings. Every separation from the control is still contracting at the last evaluation. Two seeds, not more. Every arm is run twice. That supports the claim the first table makes, which is that a set of differences is small and was not small by accident of one initialisation; it does not support a standard error, and none is quoted. Two seeds were also the cheapest way for that claim to have been overturned, and it was not. The arrangement is described and verified, not built. Its correctness is established as an identity and end to end on a trained checkpoint, with the freshness check without which it is silently wrong stated in full. No speedup is measured, and none can be here: that needs two devices and a decode loop, and on one card the stream serialises everything regardless of what depends on what. Whether it pays is a ratio between an interconnect’s latency and a feed-forward’s duration, which is a property of a machine rather than of this decomposition. The family gap is reported and not explained. Family A sits below the standard block by about -0.0634, and this paper declines to read that as an architectural gain for the three measured reasons given in Section 8. The experiment that would settle it — both orderings with the embedding unfrozen — changes a control this campaign held fixed, and is left to separate work. One shifted arm is unexplained. The shifted stack is the same alternating chain as family A in its interior and differs only in the half-layers at its ends, yet it lands at -0.0139 rather than with family A. We report the discrepancy rather than resolve it; it bounds how far the re-bracketing argument of Section 6 may be pushed as an account of the numbers.

References Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. arXiv:2305.13245. Shaoyuan Chen, Wencong Xiao, Yutong Lin, Mingxing Zhang, Yingdi Shan, Jinlei Jiang, Kang Chen, and Yongwei Wu. Efficient heterogeneous large language model decoding with model-attention disaggregation. arXiv preprint arXiv:2405.01814, 2024. URL https://arxiv.org/abs/2405.01814. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. arXiv:2205.14135. DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 16

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021. Yunkai Liang, Zhangyu Chen, Pengfei Zuo, Zhi Zhou, Xu Chen, and Zhou Yu. Injecting adrenaline into LLM serving: Boosting resource utilization and throughput via attention disaggregation. arXiv preprint arXiv:2503.20552, 2025. URL https://arxiv.org/abs/2503.20552. Bole Ma, Jan Eitzinger, Harald Köstler, and Gerhard Wellein. Move the query, not the cache: Characterizing cross-instance latent attention redistribution across GPU fabrics. arXiv preprint arXiv:2606.01502, 2026. Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. Instinfer: In-storage attention offloading for cost-effective long-context LLM inference. arXiv preprint arXiv:2409.04992, 2024. URL https://arxiv.org/abs/2409.04992. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.

17

Related documents

• The Pause Invariant: Governing Decision Beyond Thresholds#20951
• The Pause Invariant: Governing Decision Beyond Thresholds#20950
• Public Quantum Records Without a Global Bookkeeper: Provenance-Admissible Standing for Redundant Quantum Records#694420
• QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices#1104791
• Theories of Truly Concurrent Processes#635869