Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models Li Wang∗† , Yi Su∗ , Xiabao Wu† , Chiran You∗ , Yongchao Liu† , Zhan Qiu† , Juelu Zhang† , Jiajun Zheng† , Fangxin Liu‡ , Jie Zhang§ , Chen Tian∗ , and Chengying Huan∗ ∗ State Key Laboratory for Novel Software Technology, Nanjing University ‡ Shanghai Jiao Tong University

† Ant Group

§ Peking University

arXiv:2608.01651v1 [cs.DC] 3 Aug 2026

{lwang.cs,suyi}@smail.nju.edu.cn, wuxiabao.wxb@antgroup.com, 231840050@smail.nju.edu.cn {yongchao.ly,qiuzhan.qz,zhangjuelu.zjl}@antgroup.com, zhengjiajun@gmail.com, liufangxin@sjtu.edu.cn, jiez@pku.edu.cn, {tianchen,huanchengying}@nju.edu.cn

Abstract—Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memorybound. Tree speculative decoding offers an attractive acceleration path, but existing tree-speculation systems are designed around the key–value caches of full-attention models. On hybrid models, they traverse recurrent layers branch by branch and materialize a full state for every proposal node, causing verification latency and transient memory to scale poorly with tree and batch sizes. We present Bole, a kernel–runtime co-design that enables efficient tree speculation for hybrid-attention LLMs. Bole transforms the linear-attention recurrence into a tree-structured closed form and realizes it with a resource-efficient GPU kernel, verifying all proposal nodes in parallel and accelerating linear-attention tree verification by 3.4–7.7×. It losslessly encodes speculative state updates as token-level factors and reconstructs only the state selected after sampling, reducing transient state memory by 82–99× and freeing GPU capacity for KV caches. Its integration into SGLang, a widely deployed production LLM serving engine, couples efficient state management with a batch-wide verification budget calibrated to the complete hybrid forward. Across four models, two GPU platforms, and diverse datasets, Bole delivers up to 4.72× the offline decode throughput of autoregressive decoding and up to 2.03× that of the strongest tree-speculative baseline. Under online agent workloads, it reduces TTFT and TPOT by up to 67.6% and 49.9%, respectively, over the strongest tree-speculative baseline.

I. I NTRODUCTION Large language models (LLMs) increasingly serve longcontext [45] and agentic workloads [23], [24], where decoding long responses is a major serving cost [1], [54]. Full attention computes against and retains a key–value (KV) pair for every preceding token [33], so its computation, data movement, and memory footprint grow with context length [25], [49]. Models such as Qwen3.5 [34] and Kimi Linear [39] instead interleave full attention with recurrent linear attention [46]. Linearattention layers summarize the prefix in a fixed-size state [31], while full-attention layers preserve global retrieval. Hybrid attention thereby reduces the per-token serving cost [12], but generation still advances through one target-model invocation per output token. Speculative decoding [5], [16] reduces these serial invocations by exploiting unused compute capacity during memoryCorresponding authors: Chengying Huan and Yongchao Liu.

bound decoding [20], [22]. Among speculative decoding methods, tree speculation is particularly effective due to its high average acceptance length: EAGLE-2 reports 4–5.5 accepted tokens per drafting–verification cycle and up to 5× speedup [19]. A lightweight drafter proposes a tree of candidate tokens, and the target model verifies all nodes together in one invocation. For full-attention models, an ancestor mask allows each proposal node to attend to the committed prefix and its own ancestors, so the target model can score the entire tree in one parallel forward [26], [47] before sampling commits an accepted path. Accepted candidates amortize model-weight loading and invocation overheads across multiple tokens while preserving the target model’s output distribution [16]. Tree speculation is therefore a promising approach to accelerating hybrid-attention LLMs. However, realizing this potential requires both an efficient GPU kernel for executing proposal trees over recurrent linear attention and a serving runtime that manages speculative recurrent states and tree work across requests. Existing systems provide neither capability for hybrid-attention LLMs. We target this kernel–runtime gap. Existing tree-speculation kernels cannot efficiently parallelize modern hybrid-attention verification. SpecInfer [26] and DeFT [47] flatten a proposal tree into an ancestor-masked sequence and score all nodes in one full-attention forward. These full-attention methods do not transfer directly to recurrent linear attention because its recurrent formulation exposes no attention matrix to mask. Instead, it encodes tree dependencies through parent-to-child state propagation. STree [44] composes diagonal state-space model (SSM) transitions over proposal trees, but this composition does not apply to modern gated delta recurrences [34], [46]. Existing tree-speculation runtimes inefficiently manage hybrid recurrent states and execution costs. Existing systems optimize draft policies and tree construction [11], tree-aware serving pipelines [9], [26], and service-level objective (SLO)aware budgeting [21]. In particular, SGLang [36], [53] provides a production tree-speculation runtime, while AdaServe [21] adapts tree budgets to heterogeneous SLOs. Both are designed around the append-only KV caches of full-attention models. Hybrid runtimes must manage the large, transient recurrent states generated for candidate branches during tree verifica-

tion [36] and size tree budgets across heterogeneous target operators [12], [21]. Existing systems do neither efficiently. In practice, these kernel and runtime limitations manifest as three system bottlenecks, quantified in Fig. 1 and detailed in § II-C. (1) Serial recurrence. Parent-to-child execution grows with the proposal tree, increasing linear attention’s share of the target forward from 4% to 27% as the tree expands from 8 to 64 nodes. (2) State explosion. Full-state snapshots grow with both tree and batch sizes, consuming 36 GiB for eight 32node requests on Qwen3.5-122B-A10B. This transient storage displaces KV-cache capacity and limits batching, ultimately reducing serving throughput. (3) Variable verification capacity. The number of proposal tokens that can be verified efficiently changes with the GPU and the current batch. Overcoming these limitations presents three challenges. (1) Tree-coupled recurrent dependencies. Each proposal node consumes its parent’s state, imposing a topological critical path that an ancestor mask cannot remove. Parallel execution must eliminate this order while preserving the exact recurrence and mapping efficiently onto the GPU. (2) State divergence before branch selection. Candidate branches produce distinct large states before the runtime knows which path will be accepted. Exact recovery thus appears to require materializing a complete state for every node. (3) Coupled, configurationdependent cost. Full-attention, recurrent linear-attention, and dense operators scale with different request and batch properties. Because they share one target forward, their joint efficient operating point depends on the GPU, KV lengths, and tree composition, precluding a fixed per-request capacity. Together, these challenges span the linear-attention algorithm, GPU kernel, and serving runtime. We present Bole, a kernel–runtime co-design that addresses these challenges through three mechanisms. (1) Parallel tree solver. A tree-structured factorization separates the committed state from interactions among proposal tokens, yielding a structured linear system that a resource-efficient GPU kernel solves for all nodes in parallel from one pre-tree state. (2) Factorized state lifecycle. Bole retains one committed state and represents candidate updates with token-scale factors. After sampling, it reconstructs and commits only the accepted state, while rejected branches require neither full recurrent states nor rollback. (3) Hardware-aware verification budget. Bole profiles the efficient verification capacity of each execution configuration and allocates this batch-wide budget across request trees. We implement the complete design in SGLang [53], covering tree construction, target-model execution, sampling, and state commit. In summary, our work makes the following contributions: • We derive an exact tree-structured factorization of the linear-

attention recurrence and develop a resource-efficient GPU kernel that verifies all proposal nodes in parallel, accelerating linear-attention tree verification by up to 7.7×. • We develop a factorized state lifecycle that losslessly represents candidate updates with token-scale factors and reconstructs only the accepted state, reducing transient state

memory by 82–99×. • We develop a hardware-aware, batch-wide verification bud-

get and integrate Bole into SGLang to support continuous batching, CUDA Graphs, and GPU-resident state commit. • Across four models and two GPU platforms, Bole achieves up to 4.72× the offline decode throughput of autoregressive decoding and 2.03× that of the strongest tree-speculative baseline. On online agent workloads, Bole reduces time to first token (TTFT) and time per output token (TPOT) by up to 67.6% and 49.9%, respectively, over the strongest tree-speculative baseline. II. BACKGROUND AND M OTIVATION A. Hybrid Attention Models and Gated-Delta Linear Attention Hybrid LLMs interleave full-attention and recurrent linearattention layers, combining long-range retrieval with a bounded recurrent state [14]. Qwen3.5, for example, repeats three Gated DeltaNet (GDN) layers followed by one gated full-attention layer [34]. Fig. 2 contrasts their history representations: full attention retains a KV pair per prefix token and grows with context length, whereas GDN summarizes the prefix in a fixedsize matrix. Next token: method

MLP / MoE O(L) 2 KB * L

Full Attention

L: Prompt Length MLP / MoE O(1) 4 MB

Linear Attention

LinearAttention is

a

Fig. 2. State representations in hybrid-attention models.

A GDN layer applies a short causal depthwise convolution and input-dependent projections and gates to form its recurrent operands. Its recurrent core has H independent state heads, each maintaining a dk × dv matrix. For one state head, GDN updates [46] St ∈ Rdk ×dv as Set = γt St−1 , ut = βt (vt − Set⊤ kt ), (1) St = Set + kt u⊤ ot = St⊤ qt , t , where qt , kt ∈ Rdk , vt , ut , ot ∈ Rdv , and γt > 0 and βt are decay and write gates. The in-place state St summarizes the prefix through token t. Because St depends on St−1 , ordinary decoding evaluates the recurrence in token order. B. Tree Speculative Decoding Autoregressive decoding ordinarily invokes the target model once per output token [32]. At small batches, each invocation is memory-bound: model-weight transfers dominate while

SGLang (Sequential Solve) Bole (Parallel Solve)

25

5

KV + state

factors

SGLang B = 16

13%

10

snapshots

SGLang B=8

20 15

weights

27%

Bole B=8

7% 4%

Bole B = 16

0 8

16

32

64

0

200

tree tokens per request T

(a) Linear-attention verification time grows rapidly with tree size.

240

280

MLP GEMM time / layer (ms)

linear-layer share of forward GPU time (%)

30

A100 GB10

100

320

compute-bound: pay per token

weight-bound: tokens ≈ free 1

4

16

64

256

1024

input rows n ( = B × T verify tokens)

HBM (GiB)

(b) Per-node snapshots consume memory and reduce serving capacity.

(c) The efficient token range is hardware-specific.

Fig. 1. Motivation for hybrid-aware tree verification: serial recurrence, snapshot memory overhead, and hardware-dependent capacity. Measurements use Qwen3.5-9B on A100 in (a), Qwen3.5-122B-A10B on 4×A100 in (b), and Qwen3.5-9B on A100 and GB10 in (c).

much of the GPU’s arithmetic throughput remains idle [10]. creating T sequential recurrent steps even though all proposal Tree speculative decoding exploits this headroom with a tokens are available before the target forward. Fig. 1(a) shows lightweight drafter, such as multi-token prediction (MTP) [8] the resulting latency growth: increasing the tree from 8 to 64 or EAGLE [19], that constructs a proposal tree, after which nodes raises the linear layers’ share of the target forward from the target scores all nodes together [26], [48]. Reusing each 4% to 27%. weight load across more token computations makes verification L2: Per-node state snapshots impose prohibitive memory latency grow much more slowly than the number of verified overhead. Before sampling, the accepted path is unknown. tokens, so extra tokens are nearly free within the hardware’s Existing engines materialize a complete post-update GDN efficient range [17], [51]. Standard acceptance and correction state for every tree node because one in-place state cannot sampling can then commit multiple tokens without changing represent diverging branches. Sampling commits only the the target model’s output distribution [5]. terminal accepted state and discards every rejected snapshot. Beyond HBM bandwidth, these snapshots displace KV-cache Prompt: Linear Attention is a and active-request capacity [15], [38], growing linearly with a batch size B and tree size T . Fig. 1(b) quantifies this batchDrafter method novel amplified loss when serving Qwen3.5-122B-A10B on four MTP / Eagle3 / ... A100-80GB GPUs. With T = 32, snapshots consume 36 GiB for used technique Proposal Tree at B = 8 and 72 GiB at B = 16, while Bole’s factors remain efficient in for Verifier below 1 GiB in both cases. For one request with a 100-node Target Model Root token Accepted token Miss token serving tree, Table I reports 4.7–14.1 GB for snapshots but only 57– 151 MB for Bole’s token-scale factors. Next method for efficient inference Draft tokens: 9 Accepted drafts: 3 Correction token: 1 Tokens

TABLE I T RANSIENT STATE MEMORY PER REQUEST.

Fig. 3. Tree speculative decoding and accepted-path commit.

Fig. 3 shows one round. Each proposal node is a draft token, each root-to-node path is a candidate continuation, and branches reuse their shared prefix. An ancestor mask restricts each node to the committed prefix and its own proposal ancestors, preventing leakage from sibling branches. Sampling commits one accepted path and a target-sampled correction token, then discards the other branches. The accepted-path length determines per-round progress, and we report its mean as mean accepted tokens (MAT). C. Limitations of Existing Tree Verification for Hybrid Attention Existing work cannot efficiently support tree speculative decoding for hybrid-attention models. We next identify three limitations, using Fig. 1 as empirical evidence. L1: Linear-attention verification time grows rapidly with tree size. Each proposal node requires the linear-attention state produced by its parent. Existing engines [36] must therefore traverse a T -node tree and apply Eq. (1) one node at a time,

Linear Model

state

SGLang

Bole

T =100 T =200 T =100 T =200

Qwen3.5-4B 48 MB 4.7 GB 9.4 GB 57 MB 113 MB Qwen3.5-9B 48 MB 4.7 GB 9.4 GB 57 MB 113 MB Qwen3.5-27B 144 MB 14.1 GB 28.1 GB 151 MB 302 MB Qwen3.5-122B-A10B 144 MB 14.1 GB 28.1 GB 142 MB 283 MB

Both representations scale linearly with T , but their coefficients differ: Table I shows that Bole reduces the per-node transient footprint by 82–99× relative to full-state snapshots. L3: Additional tree tokens are no longer near-free beyond a hardware-dependent threshold. The roofline model [43], [50] explains this latency threshold. Consider the MLP portion of target-model execution. At small row counts, its GEMMs are weight-bandwidth-bound, so adding verification rows incurs little additional latency. Beyond a hardware-dependent threshold, they become compute-bound and latency grows approximately linearly with row count [18], [52]. Fig. 1(c)

Scheduler Model Execution Engine

Request Pool

Batch Prepare

§V-A Hardware-Aware Batch-Shared Verification Budget

Draft Model Forward

Target Model Tree Forward

Parallel Linear-Attention Tree Verification §IV-A Closed Form for Parallel Verification §IV-B Value-Tiled Parallel Verification Kernel §IV-C Fused Tree Verification Pipeline

Next iteration

Speculative Sampling §IV-D Factorized Linear State Storage and Commit

§V-B Production-Engine Integration

Fig. 4. Bole architecture and section roadmap.

places this transition near 128 rows on A100, compared with 256 rows on GB10. This component-level transition illustrates why additional tree tokens are not indefinitely free. For a complete hybrid forward, the efficient capacity further depends on each request’s KV length [13] and tree shape [6], [21]. Fixed per-request or platform-agnostic capacities are unreliable. Requirement. Together, these limitations call for a tree verifier that is parallel, incurs low memory overhead, and automatically controls verification cost. Meeting this requirement raises the three challenges outlined in § I: tree-coupled recurrent dependencies, state divergence before branch selection, and coupled, configuration-dependent cost. These challenges motivate our joint algorithm, kernel, and runtime design, which we describe in the next section.

transformations. Together, these designs provide exact parallel tree verification while preserving model outputs. Design 2: Factorized speculative state (solves L2). § IV-D keeps one immutable canonical linear-attention state and represents candidate branches with token-scale factors. Once sampling identifies an accepted path, one batched matrix multiplication reconstructs its exact state. Rejected branches use neither state-cache slots nor rollback. Design 3: Batch-shared hardware-aware verification budget (solves L3). § V-A profiles the complete hybrid target forward for each execution bucket and records its batch-level capacity. At runtime, cumulative draft probability ranks candidates across requests within that capacity; score monotonicity keeps every request’s selected tree prefix-connected. Production integration. Bole is integrated into SGLang [53], a widely deployed production LLM serving engine. § V-B materializes selected request trees as a device-resident flatragged forest, performs sampling and accepted-path commit on the GPU, and reuses captured CUDA Graphs [28] across tree topologies within each capacity bucket. IV. PARALLEL T REE V ERIFICATION FOR R ECURRENT L INEAR ATTENTION A. Deriving Bole’s Closed Form for Parallel Verification

We first state Bole’s exact parallel closed form for the complete proposal tree. Theorem 1 computes the outputs of all proposal nodes jointly from one immutable pre-tree state. III. T HE B OLE A RCHITECTURE It uses parallel all-node computation and one shared pre-tree Fig. 4 gives an overview of Bole, a kernel–runtime co-design state in place of node-wise recurrent execution and per-node for parallel, memory-efficient tree verification in hybrid models. state snapshots. We derive it through three structural steps: Its hardware-aware scheduler and target-model execution engine Lemma 1 factorizes every path state, Lemma 2 stacks the jointly control verification cost, execute the selected request resulting corrections into an ancestor-masked linear system, trees, and commit the accepted states. and Lemma 3 expresses its inverse as a finite depth-bounded Overall workflow. Bole separates one-time calibration from polynomial. Together, these lemmas establish the theorem. the recurring serving loop. Before serving, a one-time profiler Table II collects their notation. records a hardware-efficient batch-level verification capacity. In Theorem 1 (Bole’s closed-form tree verification). Sequential every target iteration, (1) the scheduler batches active requests tree execution of the gated delta rule produces and reads this pre-profiled capacity. (2) The drafter generates a
candidate tree for each request, and the scheduler prunes these O = DP QSpre + C(I + G)−1 Dβ V − DP KSpre . (2) trees so that their total number of selected tokens does not exceed the batch budget. (3) The target verifies the selected The identity is algebraically exact and equivalent to sequential request trees, using tree masks in full-attention layers and execution. It requires neither per-node linear-attention states Bole’s parallel path in recurrent linear-attention layers. (4) nor a node-wise recurrent traversal. After sampling, the engine retains each request’s accepted KV Here DP QSpre is the decayed readout of the shared state, entries and commits its matching linear-attention state. R forms the local deltas, (I + G)−1 propagates them along Design 1: Closed-form formulation for parallel tree linear- ancestors, and C performs all readouts. Because nodes are attention verification and a value-tiled GPU kernel (solves indexed parent before child and Gij can be nonzero only when L1). § IV-A derives Bole’s exactly equivalent all-node closed j is a strict ancestor of i, every nonzero entry lies below the form for linear attention over a proposal tree. § IV-B maps diagonal. Hence G is strictly lower triangular. Meanwhile, M + it to a value-tiled GPU kernel that reuses tree factors across adds the local update required by each output. The identity value tiles, keeps finite-Neumann propagation on chip, and therefore exposes a small tree interaction shared by many valuefuses output construction. § IV-C integrates the kernel with channel right-hand sides. The following lemmas give the three Gated DeltaNet’s convolutional front end and surrounding data structural proof steps.

TABLE II N OTATION FOR CLOSED - FORM TREE VERIFICATION . Symbol

Definition

T , T, π(i)

Proposal tree, node count, and parent of node i. π(i) = 0 for a root child and S0 = Spre . Strict draft ancestors of i and the maximum number of strict ancestors of any node. Number of independent state heads, per-head key/value dimensions, and the committed per-head pre-tree state Spre ∈ Rdk ×dv . Stacked token operands in RT ×dk , RT ×dk , RT ×dv , with rows qi⊤ , ki⊤ , vi⊤ . Write and decay gates in RT , with γi > 0. ei = γi Sπ(i) , ui = βi (vi − S e⊤ ki ), Sequential rule: S i ei + ki u⊤ , and oi = S ⊤ qi . Si = S i i Q Pi = r∈A(i)∪{i} γr , P = (P1 , . . . , PT )⊤ , DP = diag(P ), and Dβ = diag(β). Both diagonal matrices are in RT ×T . Identity matrix I ∈ RT ×T , indicator function, and element-wise multiplication. − Ancestor masks in {0, 1}T ×T : Mij = 1[j ∈ A(i)] and M + = M − + I. G, C ∈ RT ×T and R ∈ RT ×dv , where G = −1 Dβ DP [(KK ⊤ ) ⊙ M − ]DP , C = DP [(QK ⊤ ) ⊙ −1 M + ]DP , and R = Dβ (V − DP KSpre ). Stacked correction and output rows in RT ×dv : row i is ⊤ u⊤ i or oi .

A(i), d H, dk , dv , Spre Q, K, V β, γ ei , ui , Si , oi S Pi , P, DP , Dβ

I, 1[·], ⊙ M −, M + G, C, R

U, O

Lemma 1 (Path-factorized state). For every node i, the preand post-update states are X Pi Sei = Pi Spre + kj u⊤ j , Pj j∈A(i) (3) X Pi Si = Pi Spre + kj u⊤ . j Pj

nilpotent if one of its finite powers is zero. Under the parentbefore-child ordering, G is depth-bounded nilpotent: d+1

G

= 0,

(I + G)

−1

=

d X

(−G)m .

(6)

m=0

Proof sketch. A nonzero entry of Gm represents an m-edge ancestor path. No such path exceeds d. It follows that Gd+1 = 0, and multiplication by I + G makes the series telescope. The same structure also guarantees uniqueness. Since G is strictly lower triangular, I + G is unit lower triangular and has determinant one for every valid proposal tree. Nilpotency further shows that its inverse is exactly a finite polynomial of degree at most d. Proof of Theorem 1. Lemma 2 supplies U = (I +G)−1 R, and Lemma 3 makes the inverse finite. Substitution into the readout following Lemma 1 proves Theorem 1. Importantly, the theorem specifies outputs independently of how the structured system is solved. It identifies all available parallelism while leaving the hardware schedule to the implementation. Bole uses the finite polynomial from Lemma 3 as the internal solve used by the value-tiled verification kernel in § IV-B. The maximum supported depth fixes its sequence of propagation steps before execution. B. Value-Tiled Parallel Verification Kernel

§ IV-A transforms the sequential recurrent updates into an execution-order-independent closed form, but leaves its hardware mapping unspecified. We realize this form as a value-tiled GPU kernel that shares the tree-dependent factors G, C across value tiles, assigns each tile to one CTA, and evaluates the finite polynomial with fixed-shape tensor-core j∈A(i)∪{i} operations while retaining transient intermediates on chip. We Proof sketch. Expanding Sπ(i) and multiplying by γi intro- next describe the tile decomposition, the execution of one CTA, duces the gate product Pi /Pj . Topological induction gives the and its operand lifetime. first line, and the local rank-one update gives the second. Value-domain decomposition. For each of the H state heads, As a direct consequence of Lemma 1, reading the post- Eq. (2) jointly produces O, U ∈ RT ×dv for the T proposal nodes, where dv is the per-head value dimension. Mapping all update state with qi gives dv columns to one cooperative thread array (CTA), or CUDA X Pi ⊤ ⊤ o⊤ (qi kj )u⊤ (4) thread block, would require Θ(Te2 + Tedv + dk dv ) on-chip i = Pi qi Spre + j . Pj j∈A(i)∪{i} storage and expose only one CTA per head. Here Te is the The coefficient of uj is precisely Cij . Stacking Eq. (4) yields fixed kernel extent that covers the complete tree, including its T proposal nodes. O = DP QSpre + CU . The factors G, C ∈ RT ×T describe propagation and readout Lemma 2 (Ancestor-masked correction system). Let U ∈ interactions among proposal nodes. They are determined by RT ×dv stack the correction vectors u⊤ i . Then the tree topology, Q/K, and gates, and are independent of the value channel. Bole therefore partitions the dv columns into (I + G)U = R. (5) Nv = ⌈dv /bv ⌉ sets Jr of at most bv columns. Let X (r) = Proof sketch. Substitution into ui gives one term Gij uj per X[:, Jr ] for X ∈ {Spre , V, B0 , R, U, O}. Restricting Eq. (2) strict ancestor, where Gij = βi (Pi /Pj )ki⊤ kj . Stacking the to one value tile gives   rows yields the system. The ancestor mask excludes siblings, (r) (r) (r) (r) (r) B = D (QS ), R = D V − D (KS ) , P β P pre pre 0 descendants, and the local node itself, so Eq. (5) preserves d exactly the dependency set of sequential tree execution. X (r) U (r) = (−G)m R(r) , O(r) = B0 + CU (r) . Lemma 3 (Depth-bounded nilpotency). Let d be the maximum m=0 (7) number of strict ancestors of any draft node. A matrix is

Concatenating the tile results reconstructs U and O, so the are resident. A larger bk provides coarser MMA work and decomposition changes only the hardware schedule. Each CTA fewer partial accumulations, whereas a smaller bk lowers the processes one value tile while covering the full Te node extent. transient shared-memory footprint but requires more chunks. The tree is not partitioned across CTAs. Together, the two tile widths set of  bound the resident working  2 e e e One-CTA execution. Before the value-tile CTAs run, a factor one CTA by MCTA = Θ T + T bv + bk (T + bv ) , while bv producer forms KK ⊤ and QK ⊤ , then applies the ancestor creates HNv independently schedulable CTAs per request. Bole masks, decay ratios, and β to construct G and C. It writes selects (bk , bv ) from compile-time configurations, balancing them once to a layer-local workspace, amortizing the O(Te2 dk ) the resource and parallelism benefits of smaller tiles against Gram construction across all Nv CTAs. the reuse and execution efficiency of larger tiles. (r) As Fig. 5 shows, one CTA reads Spre ∈ Rdk ×bv and V (r) ∈ C. Fused Tree-Verification Pipeline e (r) (r) RT ×bv , then forms QSpre and KSpre . These products are Sections IV-A–IV-B parallelize the recurrent core in Eq. (1). traversed along dk in chunks of bk and decomposed into tensorA complete Gated DeltaNet layer also executes the short causal core matrix multiply-accumulate (MMA) fragments. Although convolution introduced in § II-A and surrounding gate and Eq. (7) uses DP and Dβ , the implementation stores only the layout operations. Bole organizes the complete tree execution length-T vectors P and β and applies them row-wise to register into one round-level topology step followed by three per-layer tiles. It never materializes either T × T diagonal matrix. On-chip finite-Neumann computation. The CTA next initial- GPU stages: (1) tree-parallel convolution and gate preparation, izes Z (0) = R(r) and U (r) = R(r) , then executes the following (2) factor construction, and (3) value-tiled solve and readout. fixed sequence on-chip Round-level topology preprocessing. The parent map, M − , + (m+1) (m) (r) (m+1) Z = −GZ , U += Z , 0 ≤ m < d. (8) M , and maximum depth depend only on the tree. Bole builds them once per request and round and reuses them across layers Lemma 3 guarantees that the result is exactly U (r) = (I + and heads. Fixed buffers avoid repeated transitive closure and G)−1 R(r) . Each GZ (m) is decomposed along the node reducsupport captured graphs. tion dimension into MMA fragments, making every propagation round a regular tensor-core operation that jointly processes Stage 1: tree-parallel convolution and gate preparation. On all tree positions. Z is updated in place and accumulated into a linear sequence, the convolution reads a fixed number of U (r) . All d rounds execute on chip within the same CTA, preceding tokens. For a proposal-tree node, these inputs lie on retaining tree-wide parallelism without a node-wise traversal or its root-to-node path, not at adjacent positions in the flattened intermediate HBM traffic. After the d rounds, the CTA similarly tree. Each (node, channel tile) program follows parent links and fuses convolution, bias, SiLU, and gate preparation. All (r) micro-tiles CU (r) and forms O(r) = B0 + CU (r) . This fixed nodes execute concurrently; the fixed kernel width makes perfinite polynomial is exact and introduces no approximation node work independent of proposal-tree depth. relative to sequential execution. Stage 2: factor construction. For each layer and head, one (r) Operand lifetime and on-chip reuse. The CTA releases Spre producer combines the masks with Q, K, P , and β to construct after the two state products and no longer needs P or β after G, C once in layer-local scratch; all Nv tile CTAs reuse them. (r) (r) forming B0 and R . G remains live only during propagation, Stage 3: value-tiled solve and readout. The CTAs compute R while C is loaded for the final readout. The register tiles for and the finite-Neumann recurrence to produce O and compact (r) (r) (r) (r) B0 and U survive until output construction. B0 , R , U commit factors. Normalization, path decay, and layout (m) (r) and Z are never written to HBM. Only the final O tile conversion are folded into these stages. The kernel reads (r) and the U tile required by state commit leave the CTA. the canonical head-major state, with request indices in its Two-dimensional resource tiling. High CTA concurrency launch grid. Only layer outputs and commit factors survive the provides sufficient schedulable work to utilize the GPU SMs layer; G, C remain short-lived scratch. Keeping the factor/solve efficiently, but it cannot be increased simply by choosing the boundary avoids recomputing G, C in each of the N tile CTAs. v smallest tiles. The value width bv determines both the size and the number of value-tile CTAs. A larger bv amortizes tree-factor D. Factorized Linear-State Storage and Commit reads and per-CTA setup over more value channels, but enlarges Fig. 6 follows the state data across verification, sampling, and the persistent R/Z/U and accumulator tiles and creates fewer commit. Unlike an append-only KV cache, a linear-attention independent CTAs. A smaller bv reduces the register and shared- state is one large mutable summary. Forking it for every memory footprint of each CTA and exposes more parallelism, proposal node would require a full dk × dv snapshot per head but eventually duplicates factor reads and fixed CTA overhead and branch. Bole instead keeps one committed state per active across too many narrow tiles. Bole therefore uses bv to balance request and layer in an HBM slot. This Spre remains immutable per-CTA efficiency against CTA concurrency. throughout verification, so no branch owns a private state or Value tiling alone does not bound the temporary operands requires rollback. Slots use the serving runtime’s canonical of the state products: each value tile must still contract with [H, dv , dk ] layout. Verification reads the existing slot, and the full key dimension. Bole further traverses this dimension commit overwrites it only after the accepted path is known. The in bk -wide chunks and uses a single-stage software pipeline, verification kernel’s U output directly feeds the state lifecycle (r) so only the current Q/K chunk and the matching slice of Spre shown in Fig. 6.

Input HBM/ L2 Cache

CTAs

load once

SM-resident on-chip working set

Finite Neumann

GPU

Implicit scaling

↻

Output HBM

Fig. 5. Value-tiled CTA for parallel tree verification (DK = dk , BN = Te, and BDV = bv ).

Step 1 Target-Model Tree Forward

Step 2 Speculative Sampling Target / Draft Probs

Linear Attention

Linear Attention

...

Linear Attention

Step 3 State Commit

HBM

Cross-Layer Batched Commit

...

Linear State

1 Launch

Commit Factor

Snew = Pa Spre + (Da Ka )⊤ Ua .

...

Fig. 6. Factorized speculative-state lifecycle.

Factorized speculative states. Materializing T candidate states has space complexity Θ(T Hdk dv ). By Lemma 1, Bole retains only the node factors P, K, U in addition to the single committed state, reducing the speculative-state storage complexity to Θ(Hdk dv ) + Θ(T H(dk + dv + 1)) . | {z } | {z }

committed state

draft nodes. Accepted-path commit. For one head, suppose verification selects terminal node a. Let P(a) = A(a) ∪ {a} be its accepted path, let Ka ∈ R|P(a)|×dk and Ua ∈ R|P(a)|×dv stack the path factors in topological order, and define Da = diag(Pa /Pj )j∈P(a) . Lemma 1 reconstructs the only state that must persist:

(9)

tree factors

For grouped key heads, each shared K factor is counted once. When dk = dv = ds , a full state costs Θ(Hd2s ) elements per branch, whereas one factorized node costs only Θ(2Hds ). The advantage therefore grows with the state dimension rather than relying on a particular proposal-tree shape. The G, C buffers are short-lived verification scratch and are reused from layer to layer. They are not persistent sequence state. Rejected branches therefore consume no state-cache slots and disappear when their factors are recycled. Bole distinguishes three lifetimes. Canonical states persist with the request. The factors P, K, U persist only until the round’s acceptance decision. The terms G, C and the Neumann workspaces live only during one layer. This separation permits fixed-capacity scratch reuse across layers and rounds while state-cache capacity remains independent of the number of

(10)

Bole compacts the accepted path once per round and evaluates the second term as one tensor-core matrix multiplication. The selected path is shared across all linear layers, allowing their commits to be folded into one device launch that writes each layer’s canonical cache slot. The old state remains readable until this launch completes, so no additional state snapshot or copy-on-write slot is needed. If no draft token is accepted, the slot retains Spre unchanged. V. S YSTEM O PTIMIZATION Beyond parallel tree execution, serving must decide how much work to admit across concurrent requests and map the resulting irregular forest to a graph-captured GPU iteration. Bole combines hardware-aware batch-level scheduling with a device-resident production runtime. A. Hardware-Aware Batch-Level Verification Scheduling Separating capacity from proposal utility. Given one candidate tree per request, the scheduler answers two distinct questions: how many nodes the hybrid target can verify in its hardware-efficient regime, and which nodes should consume that capacity. A node count alone is not a stable cost model. For request i, let ni , Li , and di denote its selected P node count, committed KV length, and tree depth; let N = i ni . The target latency can be viewed as Tver (n, L, d) = Tfull (n, L) + Tlinear (n, d) + Tdense (N ) + Tother (N ).

(11)

Full-attention layers read the committed prefixes, so their device-side materializer, which topologically packs the variabledominant KV work depends on both ni and Li . Recurrent layers sized trees into one flat-ragged forest. Request offsets, parent avoid this prefix scan, but their tree interaction and propagation indices, and compact ancestor metadata describe the forest schedules depend on ni and di . The shared projections and without changing its total capacity. A request-aware block MLPs batch all N nodes and cross from weight-bandwidth- map then emits only verification tiles internal to each request, bound to compute-bound execution at a platform-specific so compacting the storage also reduces the work presented operating point. Thus equal node counts can expose different to the recurrent kernels. The same packed topology drives hybrid-forward costs, and the capacity must be calibrated for full-attention layers through SGLang’s tree-attention path and the complete target path. recurrent layers through Bole’s parallel verifier. Consequently, Offline calibration of hardware-efficient capacity. A profil- capacity-bucketed CUDA Graphs retain fixed addresses and ing configuration c specifies the model, GPU and parallelism kernel shapes while the node allocation and tree topology vary setup, batch-size and KV-length buckets, and a supported tree- from round to round. depth/template bucket. For each c, the offline profiler sweeps the total number of selected nodes over the statically supported Unified cross-layer state commit. Each recurrent layer writes graph capacities G and records the latency Tver (N | c) of the its compact P, K, U factors directly into a fixed region of a complete target forward. Let Tdec (c) be the latency of ordinary preallocated contiguous GPU buffer. Target sampling produces one-token-per-request decoding under the same configuration. one accepted path descriptor per request, and device-side Bole selects compaction maps these paths to the corresponding packed Bver (c) = max {N ∈ G : Tver (N | c) ≤ (1 + ϵ)Tdec (c)} , (12) rows. A single batched launch consumes the descriptors across where ϵ bounds the additional target-forward latency admitted recurrent layers to evaluate Eq. (10), while SGLang commits for speculative verification. Measuring the complete target the matching KV entries for full-attention layers. The buffer is forward captures recurrent propagation, full attention, MLP reused by the next round, providing a common data path from execution, kernel fusion, tensor-parallel communication, and verification to both hybrid-model state representations. CUDA Graph bucket effects in one capacity. At runtime, Bubble-free GPU execution. Selection, forest materialization, existing batching metadata identifies c and retrieves its batch- target execution, sampling, path compaction, and KV/GDN shared capacity Bver (c). state commit remain on the GPU as one captured execution flow. Batch-wide utility allocation. The calibrated capacity fixes This eliminates host-induced synchronization gaps and allows the total verification work; Bole then distributes it according to the CPU scheduler to prepare subsequent work while the current the drafter’s estimate of node utility. For node v, let P(v) be its target iteration executes. The GPU pipeline therefore remains continuously supplied across speculative rounds, sustaining root-to-v path
node u. We score node v Q and π(u) the parent of as ρ(v) = u∈P(v) pdraft u | π(u) , the probability assigned high utilization. by the drafter to reaching v along its complete path. After retaining the root of each active tree, a global selection fills the VI. E VALUATION remaining capacity with the highest-scoring nodes across the After describing the experimental setup (§ VI-A), we batch. Because a descendant’s cumulative probability cannot evaluate end-to-end decode throughput (§ VI-B), generality exceed its parent’s, the selected nodes of each request are prefixacross workloads (§ VI-C), and real-world online agent serving connected without a separate repair pass. A segmented count (§ VI-D). We then isolate parallel-verification kernel efficiency of the selected nodes produces a variable budget qi for each (§ VI-E), validate the hardware-aware verification budget request while preserving the fixed batch total. This allocation through sensitivity analysis (§ VI-F), and quantify each design’s maximizes the retained cumulative probability mass under the contribution through an ablation study (§ VI-G). calibrated capacity, shifting work toward the most promising continuations. The selector is implemented with fixed-shape device top-k and scatter operations, allowing allocation to A. Evaluation Setup remain inside the captured GPU iteration. Platforms and models. We use two platforms with distinct B. Production-Engine Integration We integrate Bole into SGLang, one of the most widely deployed production LLM inference engines, with approximately 6.2 kLoC of Python and Triton [40] code. The implementation comprises parallel verification kernels that can be readily ported to other inference engines, together with deep optimizations to SGLang’s speculative-decoding and execution runtime. CUDA Graph-native packed-forest execution. The integration connects Bole’s verifier to SGLang’s GDN backend, EAGLE loop, CUDA Graph runners, and linear-state pool. The allocator’s selected node IDs and per-request budgets feed a

compute-to-memory ratios: • A100. One server with four NVLink-connected NVIDIA A100 80 GB SXM4 GPUs, each with 312 FP16/BF16 TFLOP/s and 2.04 TB/s HBM2e bandwidth. • GB10. One NVIDIA DGX Spark system powered by the GB10 Grace Blackwell Superchip, with 256 FP16/BF16 TFLOP/s and 128 GB of coherent LPDDR5x at 273 GB/s. Table IV maps four Qwen3.5 [34] models, from 4B to 122B parameters, to these platforms. We use each model’s native multi-token-prediction (MTP) head as the drafter, execute unquantized weights, and use NVLink-connected tensor parallelism (TP) [27] only for Qwen3.5-122B-A10B.

1600

SGLang-Tree

AdaServe

Qwen3.5-9B

Bole Qwen3.5-27B

Qwen3.5-122B 600

1000 320

1200

750

800

500

400

250

0

1

2 4 Batch size

300

160 80

0

8

450

240

1

2 4 Batch size

0

8

150

OOM

Decode throughput (tok/s)

SGLang-AR Qwen3.5-4B

1

2 4 Batch size

0

8

1

2 4 Batch size

8

Fig. 7. A100 MBPP throughput across Qwen3.5 models and batch sizes. SGLang-AR

SGLang-Tree

AdaServe

Bole

Qwen3.5-9B

TABLE III M EAN ACCEPTED TOKENS (MAT) ON MBPP.

Qwen3.5-27B 100

400

240 75

300

180

200

120

100

60

25

0

0

0

1

2 4 Batch size

8

50

1

2 4 Batch size

8

Model OOM

Decode throughput (tok/s)

Qwen3.5-4B

1

2 4 Batch size

8

Qwen3.5-4B Qwen3.5-9B Qwen3.5-27B Qwen3.5-122B-A10B

SGLangTree AdaServe Bole 6.29 6.76 6.56 6.31

6.35 6.38 6.82 6.88 6.61 6.62 6.39 6.40

Fig. 8. GB10 MBPP throughput across models and batch sizes; OOM indicates out of memory.

TABLE IV M ODEL DEPLOYMENT ACROSS EVALUATION PLATFORMS .

Model

A100 (4×80 GB SXM)

GB10 (128 GB LPDDR5x)

Qwen3.5-4B Qwen3.5-9B Qwen3.5-27B Qwen3.5-122B-A10B

1 GPU (TP1) 1 GPU (TP1) 1 GPU (TP1) 4 GPUs (TP4)

1 GPU (TP1) 1 GPU (TP1) 1 GPU (TP1) –

Baselines. We compare against three baselines in SGLang [36], one of the most widely deployed production LLM serving engines. SGLang-AR performs standard autoregressive decoding. SGLang-Tree uses SGLang’s native tree-speculative decoding path. AdaServe [21] is a state-of-the-art tree-speculative serving system. Because its open-source implementation does not support the evaluated hybrid-attention models, we port and optimize it in SGLang. All tree methods use the same native MTP drafter, model weights, sampling configuration, and total proposal-token budget. We set top-k = 4 and the maximum depth to 8. Workloads. Offline experiments span code (MBPP), dialogue (ShareGPT), mathematical reasoning (GSM8K), and summarization (CNN/DailyMail) [3], [7], [35], [37]. For online experiments, we replay a real-world coding-agent workload: OpenHands [42] sessions from NVIDIA Open-SWE-Traces [2]. Metrics and methodology. For offline serving, we hold the active batch size fixed and report decode throughput in accepted output tokens per second. For online serving, we report time to first token (TTFT), including queueing and prefill, and time per output token (TPOT) after the first token.

B. End-to-End Decode Throughput We evaluate end-to-end offline decode throughput on MBPP using all four systems. Figs. 7 and 8 report results for all 28 configurations. Bole consistently improves decode throughput. Across all configurations, it achieves a geometric-mean speedup of 2.74× over SGLang-AR and 1.26× over the strongest speculative baseline. On A100, its peak speedups over SGLang-AR, SGLang-Tree, and AdaServe are 3.62×, 1.41×, and 1.39×, respectively; on GB10, the corresponding peak speedups reach 4.72×, 2.03×, and 2.06×. Bole delivers the highest throughput across the single-GPU deployments and in the 122B TP4 deployment, demonstrating that its benefits extend to tensorparallel execution. Table III further shows that our pruning method consistently attains higher MAT than both speculative baselines across all four models. The parallel verification kernel substantially accelerates tree verification. On platforms with a higher compute-to-memory ratio, memory-bound decoding leaves more compute headroom for a larger proposal budget. At these larger budgets, Bole’s parallel verifier delivers even greater speedups by efficiently absorbing the increased verification workload. On GB10, its speedup over the strongest speculative baseline increases from 1.11–1.14× at batch size 1 to 1.70–2.03× at batch size 8. Factorized state management expands the feasible batching capacity while preserving exact, lossless verification semantics. Table VI reports the additional memory used by tree speculation after subtracting AR and MTP-drafter memory. The results match the transient-state analysis in Table I: Bole reduces this overhead by 74.0–88.5% relative to AdaServe, freeing 6.26– 18.58 GiB for serving caches. The 122B row aggregates the four GPUs used for TP4. This additional capacity allows Bole

SGLang-AR

SGLang-Tree

AdaServe

Bole

Throughput (tok/s)

A100

TABLE V MAT ACROSS GENERATION WORKLOADS .

GB10

320

60

240

45

Dataset

160

30

80

15

MBPP GSM8K ShareGPT CNN/DailyMail

0

MBPP

GSM8K

ShareGPT

0

CNN/DailyMail

MBPP

GSM8K

ShareGPT

CNN/DailyMail

SGLangTree

AdaServe

Bole

6.57 7.07 5.31 4.71

6.68 7.08 5.50 4.95

6.70 7.08 5.51 4.98

Fig. 9. Qwen3.5-27B throughput across workloads on A100 and GB10 at batch size 4. SGLang-AR A100 TTFT

SGLang-Tree

mean

p50

16

90 60 30

8 0

p99

450 TPOT (ms)

1.5

24

TTFT (s)

TPOT (ms)

TTFT (s)

3.0

GB10 TPOT

120

32 4.5

Bole

GB10 TTFT

6.0

0.0

AdaServe

A100 TPOT

mean

p50

p99

0

mean

p50

p99

300 150 0

mean

p50

p99

Fig. 10. Mean, p50, and p99 online latency on Open-SWE-Traces, using Qwen3.5-122B-A10B on A100 and Qwen3.5-27B on GB10; lower is better.

TABLE VI M EASURED NON - CACHE SPECULATION OVERHEAD ON A100 AT BATCH SIZE 4 WITH 128 PROPOSAL NODES (G I B); LOWER IS BETTER .

Model Qwen3.5-4B Qwen3.5-9B Qwen3.5-27B Qwen3.5-122B-A10B

SGLangTree AdaServe 8.72 8.74 24.00 28.04

7.79 7.81 21.00 25.04

Bole 1.53 1.55 2.42 6.52

to run Qwen3.5-27B at batch size 8 without OOM on either platform, whereas both speculative baselines OOM. C. Generality Across Workloads Following the setup in § VI-B, we evaluate the other three workloads at batch size 4. The workloads vary substantially in prediction difficulty, yielding MATs from 4.98 on CNN/DailyMail to 7.08 on GSM8K (Fig. 9 and Table V). Relative to SGLang-AR, Bole achieves 1.98×–3.36× speedups on A100 and 2.85×– 4.12× on GB10. Relative to the faster speculative baseline, the corresponding speedups are 1.15×–1.20× and 1.38×– 1.45×. Overall, Bole delivers strong speedups across diverse workloads. D. Real-World Online Agent Serving We replay multi-turn Open-SWE-Traces sessions using the largest model supported by each platform: Qwen3.5-122BA10B on A100 and Qwen3.5-27B on GB10. Following standard LLM-serving methodology [1], session start times follow a Poisson process, and every system receives the same arrival trace on each platform. Fig. 10 reports mean, p50, and p99 TTFT and TPOT.

Bole reduces mean TTFT and TPOT against every baseline. Compared with SGLang-AR, it lowers mean TTFT by 15.8%– 64.3% and mean TPOT by 37.6%–67.6% across the two platforms. Against SGLang-Tree, the corresponding reductions are 63.5%–67.6% and 33.8%–50.3%, while against AdaServe they are 61.3%–73.3% and 28.9%–49.9%. Bole’s factorized state representation leaves substantially more memory for the KV cache, reducing repeated prefills and thus improving TTFT. Each agent turn extends a long prefix shared with its preceding turns, so evicting a cached prefix forces a repeated prefill and directly increases TTFT. Existing tree systems reserve a recurrent-state snapshot for every proposal, incurring 82–99× the transient-memory footprint of Bole (Table I) and leaving less capacity for reusable prefixes. By retaining only factorized updates, Bole attains 90.8% and 92.6% prefix-cache hit rates on A100 and GB10. SGLang-Tree reaches 69.7% and 58.3%, while AdaServe reaches 69.3% and 59.8%. Bole thus recovers 21.1 to 34.3 percentage points of cache hits and nearly matches SGLang-AR’s 89.8% and 93.3%, despite verifying a proposal tree. Under continuous batching, TTFT and TPOT interact through the shared execution stream, and Bole improves both. TTFT consists of queueing and prefill: compared with SGLang-AR, Bole’s lower TPOT releases batch slots sooner and shortens queueing, while its factorized state preserves a comparable prefix-cache hit rate and avoids additional prefills. Conversely, fewer prefills also reduce interference with active decoding, reinforcing the TPOT improvement. E. Parallel Verification Kernel Efficiency To isolate parallel recurrent verification, we compare the value-tiled verifier from § IV-B with SGLang-Tree’s serial delta-rule verifier on Qwen3.5-9B under the same proposal

TABLE VII P ER - LAYER G ATED D ELTA N ET TREE - VERIFICATION EFFICIENCY ON Q WEN 3.5-9B.

SGLang-Tree + Hardware-aware budget

+ Parallel tree solver + Factorized state lifecycle

GB10 Latency (µs)

Occupancy

L1/shared

Plat.

B

Serial

Bole Speedup Serial

Bole

Serial

Bole

A100 A100 GB10 GB10

1 16 1 16

174 822 333 3420

50.6 110 66.6 443

3.4× 7.5× 5.0× 7.7×

15.0% 41.4% 43.1% 58.9%

14.7% 51.3% 10.1% 12.5%

60.8% 93.5% 34.6% 36.8%

1.9% 11.4% 6.6% 16.4%

A100 0.0

0.5

1.0

1.5

2.0

Decode throughput (norm. to SGLang-Tree)

1300 1200 1100 1000 900

A100 GB10

290

across GPU architectures. This result corroborates Fig. 1(c) and validates the design in § V-A.

280

G. Component Ablation

GB10 (tok/s)

A100 (tok/s)

Fig. 12. Cumulative online ablation of Bole’s three designs over SGLang-Tree.

Fig. 12 cumulatively adds Bole’s three designs to SGLangTree under the online agent workload. 64 128 192 256 320 384 The hardware-aware budget first improves throughput by Total tree tokens per step (budget × batch) 5.3% on A100 and 10.7% on GB10. This step changes neither the serial verifier nor its state representation. It only stops Fig. 11. Budget sensitivity on Qwen3.5-9B/MBPP at batch size 8. expanding the batch after additional proposals cost more than their expected accepted tokens, isolating the benefit of matching the verification budget to the platform. tree and drafter output. Table VII reports complete-core latency Replacing serial recurrent verification raises the cumulative and solve-kernel metrics, both collected using nsys [30] and gains to 1.21× on A100 and 1.30× on GB10. This increment ncu [29]; B denotes batch size. is larger than budgeting alone because the selected proposal Across both platforms, Bole accelerates the complete verifibudget still contains many mutually independent branches. The cation core by 3.4×–7.7×. The speedup rises from 3.4× and solver removes their parent-to-child execution chain, allowing 5.0× at B = 1 to 7.5× and 7.7× at B = 16 on A100 and the engine to exploit the cross-request and intra-tree parallelism GB10, respectively. This scaling follows from removing the already present in the online batch. parent-to-child critical path within every proposal tree, which Finally, factorized state management raises the cumulative exposes node-level parallelism both within and across requests. speedup to 1.59× on A100 and 2.23× on GB10. At A100’s The closed-form solver exposes node-level parallelism, 128-token budget, it cuts Qwen3.5-122B-A10B’s transient state converting proposal nodes into independently schedulable from 18.0 to 0.18 GiB, freeing 17.8 GiB for the KV cache. The GPU work and increasing achieved occupancy by 3.6×–7.9× larger GB10 gain reflects its tighter memory capacity and cacheacross the four settings. By keeping the main verification hit improvement (§ VI-D). Together, budgeting selects useful work on chip rather than materializing full recurrent states proposals, parallel verification executes them efficiently, and in HBM, the factorized, value-tiled path increases L1/sharedfactorization preserves reusable agent state. All three contribute memory throughput by 1.8×–4.1×. At B = 16, it writes to the full online gain. 9.8 MB per layer instead of 824 MB, an 84× reduction. State materialization consumes 38% and 86% of serial verification VII. R ELATED W ORK time on A100 and GB10, respectively, explaining the larger Tree Speculative Decoding. Speculative decoding accelerates benefit on GB10’s lower-bandwidth unified memory. Factorization adds one batched commit over only the autoregressive inference by verifying a sequence of drafted accepted path. The commit runs once per target-model forward tokens in parallel [5], [8], [16], [17], [22], [41], [51]. Among these methods, tree speculation stands out for longer accepted and accounts for less than 0.5% of its latency. paths and greater speedups [4], [6], [19], [20]. Medusa [4] F. Hardware-Aware Verification-Budget Sensitivity and EAGLE-3 [20] generate branches with learned predictors, while Sequoia [6], EAGLE-2 [19], Yggdrasil [9], and GTO [11] Fig. 11 varies the total number of target-model tokens verified optimize static or probability-aware tree topologies. SpecInper decoding step while holding tree width, maximum depth, fer [26] and DeFT [47] reduce target-side verification cost. and batch size fixed. Initially, a larger budget exposes more These works do not transfer to recurrent linear attention, which likely branches and raises MAT faster than it increases targetexposes no attention matrix [26], [47]. AdaServe [21] allocates model cost. Beyond the near-free verification region, additional proposal budgets across requests to meet SLO requirements. work dominates the acceptance benefit and throughput declines. The peak occurs at 128 total tree tokens on A100 but at 256 Hybrid-Attention Model Serving. vLLM [15], SGLang [53], tokens on GB10. A single fixed budget is therefore suboptimal and related systems optimize batching, caching, and GPU

270

execution [1], [13], [32], [33], [48], [54]. For hybrid models, Marconi [31] caches recurrent prefixes, while HLX [12] and Pimba [14] specialize Transformer–SSM execution. STree [44] composes diagonal SSM transitions over proposal trees, but this algebra does not extend to the token-dependent, nondiagonal recurrence of gated linear attention. These systems do not jointly support parallel gated-linear tree verification and transient branch-state management [39], [44], [46]. Bole provides both within a production engine. VIII. C ONCLUSION We present Bole, a kernel–runtime co-design for efficient tree speculative decoding of hybrid-attention LLMs. Its closedform solver parallelizes recurrent tree verification, while factorized state management commits only the accepted branch. A hardware-aware batch-wide budget turns these gains into serving throughput. Integrated into SGLang, Bole accelerates linear-attention verification by up to 7.7×, reduces transient state memory by 82–99×, and outperforms the strongest treeserving baseline by up to 2.03×. On online agent workloads, it reduces TTFT and TPOT by up to 67.6% and 49.9%, respectively, over the strongest tree-speculative baseline. ACKNOWLEDGMENTS This work was supported by Ant Group Research Intern Program. R EFERENCES [1] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/agrawal [2] W. U. Ahmad, N. Ludwig, S. Majumdar, and B. Ginsburg, “Open-swe-traces: Advancing dual-mode multilingual distillation for software engineering agents,” 2026. [Online]. Available: https: //arxiv.org/abs/2606.16038 [3] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 [4] T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple LLM inference acceleration framework with multiple decoding heads,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. A. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. PMLR / OpenReview.net, 2024, pp. 5209–5235. [Online]. Available: https://proceedings.mlr.press/v235/cai24b.html [5] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” 2023. [Online]. Available: https://arxiv.org/abs/2302.01318 [6] Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable, robust, and hardware-aware speculative decoding,” arXiv preprint arXiv:2402.12374, 2024. [7] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 2021. [Online]. Available: https://arxiv.org/abs/2110.14168

[8] F. Gloeckle, B. Youbi Idrissi, B. Roziere, D. Lopez-Paz, and G. Synnaeve, “Better & faster large language models via multi-token prediction,” in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. PMLR, 21–27 Jul 2024, pp. 15 706–15 734. [Online]. Available: https://proceedings.mlr.press/v235/gloeckle24a.html [9] Y. Guan, C. Yu, S. Fang, W. Hu, Z. Pan, Z. Wang, Z. Liu, Y. Zhou, Y. Ding, M. Guo, and J. Leng, “Yggdrasil: Bridging dynamic speculation and static runtime for latency-optimal treebased LLM decoding,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=4E3I17pNEl [10] Y. He, H. Mao, C. Giannoula, M. Sadrosadati, J. Gómez-Luna, H. Li, X. Li, Y. Wang, and O. Mutlu, “Papi: Exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 766–782. [Online]. Available: https://doi.org/10.1145/3676641.3716009 [11] S. Hu, J. Li, Z. Lu, and P. Zhou, “Bridging draft policy misalignment: Group tree optimization for speculative decoding,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=dwPdYFqVWO [12] I.-J. Jung, G. Yang, J. Min, and J.-Y. Kim, “Hlx: A unified pipelined architecture for optimized performance of hybrid transformer-mamba language models,” in Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 461–475. [Online]. Available: https://doi.org/10.1145/3725843.3756115 [13] A. K. Kamath, R. Prabhu, J. Mohan, S. Peter, R. Ramjee, and A. Panwar, “Pod-attention: Unlocking full prefill-decode overlap for faster llm inference,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 897–912. [Online]. Available: https://doi.org/10.1145/3676641.3715996 [14] W. Kim, Y. Lee, Y. Kim, J. Hwang, S. Oh, J. Jung, A. Huseynov, W. G. Park, C. H. Park, D. Mahajan, and J. Park, “Pimba: A processing-in-memory acceleration for post-transformer large language model serving,” in Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 292–307. [Online]. Available: https://doi.org/10.1145/3725843.3756121 [15] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165 [16] Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 23–29 Jul 2023, pp. 19 274–19 286. [Online]. Available: https: //proceedings.mlr.press/v202/leviathan23a.html [17] C. Li, Z. Zhou, S. Zheng, J. Zhang, Y. Liang, and G. Sun, “Specpim: Accelerating speculative inference on pim-enabled system via architecture-dataflow co-exploration,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 950–965. [Online]. Available: https://doi.org/10.1145/3620666.3651352 [18] S. Li, Y. Chen, C. Li, Y. Fu, Z. Wang, Z. Yu, H. You, Z. Ye, W. Zhou, Y. Zhang, and Y. C. Lin, “Orches: Orchestrated test-time-compute-based llm reasoning on collaborative gpu-pim heterogeneous system,” in Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 476–489. [Online]. Available: https://doi.org/10.1145/3725843.3756039 [19] Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE-2: Faster inference of language models with dynamic draft trees,” in Proceedings of the 2024

Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 7421–7432. [Online]. Available: https://aclanthology.org/2024.emnlp-main.422/ [20] Y. Li, F. Wei, C. Zhang, and H. Zhang, “Eagle-3: Scaling up inference acceleration of large language models via training-time test,” in Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025, pp. 136 737–136 756. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2025/file/c7b5a35ea98b62512a869c19ea7b03cb-Paper-Conference.pdf [21] Z. Li, Z. Chen, R. Delacourt, G. Oliaro, Z. Wang, Q. Chen, S. Lin, A. Yang, Z. Zhang, Z. Chen, Y.-H. Lai, X. Cheng, X. Miao, and Z. Jia, “Adaserve: Accelerating multi-slo llm serving with slo-customized speculative decoding,” in Proceedings of the 21st European Conference on Computer Systems, ser. EUROSYS ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 36–54. [Online]. Available: https://doi.org/10.1145/3767295.3769315 [22] X. Liu, J. Yu, J. Park, I. Stoica, and A. Cheung, “Speculative decoding: Performance or illusion?” in Ninth Conference on Machine Learning and Systems, 2026. [Online]. Available: https: //openreview.net/forum?id=fzkqtezFEi [23] Y. Liu, Y. Gu, L. Zhang, C. Wu, G. Xue, J. Li, M. Guo, J. Hu, and J. Meng, “CacheSlide: Unlocking cross Position-Aware KV cache reuse for accelerating LLM serving,” in 24th USENIX Conference on File and Storage Technologies (FAST 26). Santa Clara, CA: USENIX Association, Feb. 2026, pp. 83–99. [Online]. Available: https://www.usenix.org/conference/fast26/presentation/liu-yang [24] M. Luo, X. Shi, C. Cai, T. Zhang, J. Wong, Y. Wang, C. Wang, Y. Huang, Z. Chen, J. E. Gonzalez, and I. Stoica, “Agentix: An efficient serving engine for LLM agents as general programs,” in 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton, WA: USENIX Association, May 2026, pp. 2443–2459. [Online]. Available: https://www.usenix.org/conference/nsdi26/presentation/luo [25] S. Luo and H. Shen, “No buffer, no bottleneck: Efficient Zero-Copy KV cache offloading for Long-Context LLMs,” in 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). Seattle, WA: USENIX Association, Jul. 2026, pp. 39–54. [Online]. Available: https://www.usenix.org/conference/osdi26/presentation/luo [26] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia, “Specinfer: Accelerating large language model serving with tree-based speculative inference and verification,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 932–949. [Online]. Available: https://doi.org/10.1145/3620666.3651335 [27] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3458817.3476209 [28] NVIDIA Corporation, “CUDA Programming Guide: CUDA Graphs,” https://docs.nvidia.com/cuda/cuda-programming-guide/04-specialtopics/cuda-graphs.html, 2026, accessed: July 2026. [29] NVIDIA Corporation, “NVIDIA Nsight Compute Documentation,” https://docs.nvidia.com/nsight-compute/NsightCompute/index.html, 2026, [Accessed 29-07-2026]. [30] NVIDIA Corporation, “NVIDIA Nsight Systems User Guide,” https: //docs.nvidia.com/nsight-systems/UserGuide/index.html, 2026, [Accessed 29-07-2026]. [31] R. Pan, Z. Wang, Z. Jia, C. Karakus, L. Zancato, T. Dao, Y. Wang, and R. Netravali, “Marconi: Prefix caching for the era of hybrid llms,” in Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin, Eds., vol. 7. MLSys, 2025. [Online]. Available: https://proceedings.mlsys.org/paper files/paper/2025/ file/7c180af017258d239bac6248d1eb26ac-Paper-Conference.pdf [32] P. Patel, E. Choukse, C. Zhang, A. Shah, I. n. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in Proceedings of the 51st Annual International Symposium on

Computer Architecture, ser. ISCA ’24. IEEE Press, 2025, p. 118–132. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00019 [33] R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y. Wu, W. Zheng, and X. Xu, “Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,” in 23rd USENIX Conference on File and Storage Technologies (FAST 25). Santa Clara, CA: USENIX Association, Feb. 2025, pp. 155–170. [Online]. Available: https://www.usenix.org/conference/fast25/presentation/qin [34] Qwen Team, “Qwen3.5: Towards native multimodal agents,” https://qwen. ai/blog?id=qwen3.5, 2026, [Accessed 26-07-2026]. [35] A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y. Kan, Eds. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1073–1083. [Online]. Available: https://aclanthology.org/P17-1099/ [36] sglang, “GitHub - sgl-project/sglang at release/v0.5.12 — github.com,” https://github.com/sgl-project/sglang/tree/release/v0.5.12, [Accessed 2807-2026]. [37] sharegpt, “anon8231489123/ShareGPT Vicuna unfiltered · Datasets at Hugging Face — huggingface.co,” https://huggingface.co/datasets/ anon8231489123/ShareGPT Vicuna unfiltered, [Accessed 29-07-2026]. [38] B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 173–191. [Online]. Available: https: //www.usenix.org/conference/osdi24/presentation/sun-biao [39] K. Team, Y. Zhang, Z. Lin, X. Yao, J. Hu, F. Meng, C. Liu, X. Men, S. Yang, Z. Li, W. Li, E. Lu, W. Liu, Y. Chen, W. Xu, L. Yu, Y. Wang, Y. Fan, L. Zhong, E. Yuan, D. Zhang, Y. Zhang, T. Y. Liu, H. Wang, S. Fang, W. He, S. Liu, Y. Li, J. Su, J. Qiu, B. Pang, J. Yan, Z. Jiang, W. Huang, B. Yin, J. You, C. Wei, Z. Wang, C. Hong, Y. Chen, G. Chen, Y. Wang, H. Zheng, F. Wang, Y. Liu, M. Dong, Z. Zhang, S. Pan, W. Wu, Y. Wu, L. Guan, J. Tao, G. Fu, X. Xu, Y. Wang, G. Lai, Y. Wu, X. Zhou, Z. Yang, and Y. Du, “Kimi linear: An expressive, efficient attention architecture,” 2025. [Online]. Available: https://arxiv.org/abs/2510.26692 [40] P. Tillet, H. T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, ser. MAPL 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 10–19. [Online]. Available: https://doi.org/10.1145/3315508.3329973 [41] R. Wang, Q. Wang, H. Liu, L. Zheng, X. Liao, H. Jin, and J. Xue, “Adaptive draft sequence length: Enhancing speculative decoding throughput on pim-enabled systems,” in 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2026, pp. 1–15. [42] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, D. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “Openhands: An open platform for ai software developers as generalist agents,” in International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., vol. 2025, 2025, pp. 65 882–65 919. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2025/ file/a4b6ad6b48850c0c331d1259fc66a69c-Paper-Conference.pdf [43] S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Commun. ACM, vol. 52, no. 4, p. 65–76, Apr. 2009. [Online]. Available: https://doi.org/10.1145/1498765.1498785 [44] Y. Wu, Z. Qin, A. Wong, and S. Soatto, “Stree: Speculative tree decoding for hybrid state space models,” in Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025, pp. 115 588–115 608. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2025/file/a7efd7734bb2e79997f8d1137ece3f09-Paper-Conference.pdf [45] Z. Xie, Z. Xu, M. Zhao, Y. An, V. S. Mailthody, S. Mahlke, M. Garland, and C. Kozyrakis, “Strata: Hierarchical context caching for long context language model serving,” in 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). Seattle,

WA: USENIX Association, Jul. 2026, pp. 1–16. [Online]. Available: https://www.usenix.org/conference/osdi26/presentation/xie-zhiqiang [46] S. Yang, J. Kautz, and A. Hatamizadeh, “Gated delta networks: Improving mamba2 with delta rule,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 29 687–29 707. [47] J. Yao, K. Chen, K. Zhang, J. You, B. Yuan, Z. Wang, and T. Lin, “Deft: Decoding with flash tree-attention for efficient tree-structured llm inference,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=2c7pfOqu9k [48] Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, and L. Ceze, “Flashinfer: Efficient and customizable attention engine for llm inference serving,” in Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin, Eds., vol. 7. MLSys, 2025. [Online]. Available: https://proceedings.mlsys.org/paper files/paper/2025/ file/dbf02b21d77409a2db30e56866a8ab3a-Paper-Conference.pdf [49] J. Yi, Z. Zhao, Y. Hu, K. Yan, W. Sun, H. Wang, L. Zhao, Y. Zhang, W. Li, and K. Li, “Pat: Accelerating llm decoding via prefix-aware attention with resource efficient multi-tile kernel,” in Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 1396–1412. [Online]. Available: https://doi.org/10.1145/3779212.3790200 [50] H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” in Proceedings of the 51st Annual International Symposium on Computer Architecture, ser. ISCA ’24. IEEE Press, 2025, p. 1080–1096. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00082 [51] Z. Zhang, Z. Jiang, C. Jiang, M. Yu, S. Zheng, H. Lin, X. Liu, and H. Hoffmann, “Swiftspec: Disaggregated speculative decoding and fused kernels for low-latency llm inference,” in Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 2197–2211. [Online]. Available: https://doi.org/10.1145/3779212.3790246 [52] Y. Zhao, C.-Y. Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,” Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024. [53] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “Sglang: Efficient execution of structured language model programs,” in Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 62 557–62 583. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf [54] K. Zhu, Y. Gao, Y. Zhao, L. Zhao, G. Zuo, Y. Gu, D. Xie, Z. Ye, K. Kamahori, C.-Y. Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci, “NanoFlow: Towards optimal large language model serving throughput,” in 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). Boston, MA: USENIX Association, Jul. 2025, pp. 749–765. [Online]. Available: https://www.usenix.org/conference/osdi25/presentation/zhu-kan

Related documents

• HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation#1083071
• X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding#986939
• What Kind of Reasoning (if any) is an LLM actually doing? On the Stochastic Nature and Abductive Appearance of Large Language Models#701151
• Bridging the Data Gap: LLM-Driven Synthetic Data Generation for Low-Resource Natural Language Inference#1120844
• Large Language Bayes Is Not Reparameterisation-Invariant#1154888