arXiv CS open access

Request-Adaptive Execution for Efficient Hybrid LLM Inference

Fahao Chen et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

Hybrid large language models (LLMs) have emerged as a promising way to combine the modeling capability of the Attention mechanism with the efficiency of state space models (SSMs). However, most existing hybrid LLMs still inherit a static design philosophy: once the Attention-to-SSM layout is chosen,

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

arXiv:2610.37432v1 [cs.DC] 27 Oct 2026

Request-Adaptive Execution for Efficient Hybrid LLM Inference Fahao Chen

Xuzhe Yan

Qi Zhang

chenfh@ieee.org Shandong University China

Xi’an Jiaotong University China Zedekiah5571@outlook.com

Xi’an Jiaotong University China zq0204@stu.xjtu.edu.cn

Peng Li

Dongxiao Yu

Song Guo

Xi’an Jiaotong University China pengli@xjtu.edu.cn

Shandong University China dxyu@sdu.edu.cn

Hong Kong University of Science and Technology Hong Kong SAR songguo@cse.ust.hk

Abstract Hybrid large language models (LLMs) have emerged as a promising way to combine the modeling capability of the Attention mechanism with the efficiency of state space models (SSMs). However, most existing hybrid LLMs still inherit a static design philosophy: once the Attention-to-SSM layout is chosen, every request must follow the same hybrid execution. This one-layout-fits-all design is inefficient for real-world serving workloads, where different requests rely on contextual information in different ways. This paper presents Flex, a serving system that co-designs the model architecture and runtime for request-adaptive hybrid LLM inference. Its model architecture pairs Attention and SSM layers within each block and uses prefill-oriented gating to select a request-specific layout that remains fixed throughout decoding. Its serving runtime preserves batching efficiency under divergent layer choices through selective batching and enables correct prefix reuse through hybridaware context caching. Experiments on six datasets show that Flex achieves a better quality-efficiency trade-off than existing static hybrid baselines. System-level evaluation further shows that Flex improves the serving throughput by up to 3.87× over baselines while substantially reducing inference latency.

Attention Layer

SSM Layer

Attention Layer

SSM Layer

Attention Layer

SSM Layer

Attention Layer

SSM Layer

Attention Layer

Reasoning

(a) Static hybrid layout

Reasoning

Summarization

(b) Proposed dynamic hybrid layout

Figure 1. (a) Hybrid LLMs use the static Attention-to-SSM layout for all requests. (b) Request-adaptive hybrid layout allows different requests to activate different Attention and SSM execution. Colored lines indicate layer selections.

prior tokens into compact recurrent states and update them in linear time. However, this compact recurrent state may limit the ability to recall context-dependent information, making SSMs less effective on tasks that require precise longrange context modeling [46]. To combine the complementary strengths of attention mechanisms and SSMs, recent works have proposed hybrid LLMs1 , which interleave Attention layers and SSM layers within the same model to balance model quality and inference efficiency. Representative works such as Jamba [22], MambaKeywords: Large language models, State space models, EfLLaMA [38], and Nemotron-H [3] have demonstrated a betficient serving ter quality-efficiency trade-off than Attention-only or SSMonly architectures. A core design factor behind the perfor1 Introduction mance of hybrid LLMs is the Attention-to-SSM layout, which specifies not only how many Attention and SSM layers are Large language models (LLMs) have become the foundation of modern AI services, powering applications such as search [31, used, but also where these layers are placed in the model. Existing hybrid LLMs typically rely on hand-crafted ar40], coding assistants [20], document analysis [18, 49], and chitecture designs, where the Attention-to-SSM layout is seautonomous agents [24, 34]. Most deployed LLMs are based lected based on empirical studies. In addition, this layout reon the Transformer architecture, where the attention mechmains static for all requests once the model is constructed. anism provides strong global-context modeling but incurs For example, Jamba adopts an 𝑎/𝑚 rotation strategy, insertquadratic computation and growing KV-cache memory during 𝑎 Attention layers for every 𝑚 SSM layers. As illustrated ing inference [43, 50, 56]. To reduce this cost, state space 1 models (SSMs), such as Mamba [9, 16], have been introduced We use the term “hybrid LLMs” to broadly refer to LLMs that combine as efficient sequence modeling primitives. SSMs summarize Attention layers with SSM layers within the same model. 1

in Figure 1(a), such static designs assume requests with different characteristics, e.g., summarization, QA, and reasoning, benefit from the same hybrid structure. However, our preliminary experiments in §2.2 show that this assumption does not always hold in real-world inference workloads. A layout that achieves the best accuracy on a reasoning task such as ARC-Challenge can lead to suboptimal quality on a summarization task such as PubMed. This inconsistency arises because different workloads rely on contextual information in different ways. For example, reasoning tasks may only require selected Attention layers for option comparison, while long-document summarization often depends more heavily on fine-grained global context modeling to identify and preserve salient information. As a result, a static Attentionto-SSM layout can either overuse Attention for simple requests, wasting computation, or underuse Attention for contextsensitive requests, sacrificing output quality. Key Idea. Our key idea is to make the hybrid LLM requestadaptive: rather than fixing the Attention-to-SSM layout, the model selects an appropriate hybrid layout for each incoming request. As illustrated in Figure 1(b), different requests can activate different Attention and SSM layers according to their context-modeling demands. We realize this idea through algorithm-system co-design in Flex, an inference system that integrates a request-adaptive hybrid model with runtime optimizations for efficient serving. The hybrid model consists of multiple adaptive hybrid blocks, each containing a lightweight gate, paired Attention and SSM branches, and shared components such as the FFN. For each request, the gates select one branch per block, forming a requestspecific execution layout that balances modeling capability and inference cost. A key design choice is the gating granularity. A natural token-level gating design would allow each token to independently choose whether to execute the Attention or SSM layer in each hybrid block. However, this design is ill-suited for autoregressive decoding, where newly generated tokens rely on the cached results of previous tokens, i.e., Attention KVs and SSM states, to avoid recomputation. If tokens from the same request choose different layers within the same block, later tokens may not be able to directly reuse the Attention KVs or SSM states produced by earlier tokens, forcing the runtime to recompute previous tokens under the new layer choice. Flex therefore adopts prefill-oriented gating mechanism (§4): during prefill, the gate uses all input tokens to make one request-level Attention and SSM choice for each hybrid block, and these choices are fixed throughout decoding. This allows the Attention KVs or SSM states built by previously generated tokens to be reused consistently. At the system level, request-adaptive layouts introduce two challenges: requests within the same batch may select different branches, and requests sharing a textual prefix may require incompatible layer-wise caches. The serving runtime 2

in Flex addresses these challenges through the following two designs to preserve batching efficiency and ensure correct context reuse. Selective Batching Engine. To maintain high inference throughput, Flex must efficiently batch multiple requests for parallel execution. However, requests in the same batch may select different branches at each hybrid block, which breaks the traditional batching assumption of conventional serving engines, where all requests in a batch follow the same operator sequence [12, 21, 32]. A naive solution would either serialize or execute block computations separately. The former loses parallelism, while the latter fragments the original batch and prevents request-independent operations, such as normalization and FFN computation, from being executed as a single compact batch [55]. Flex addresses this with selective batching engine (§5). The key idea is to keep branch-independent operations on the original packed batch, and split requests only for the branch-dependent Attention and SSM computation. At each block, it partitions requests into compact Attention and SSM sub-batches, executes them with their corresponding kernels, and then merges outputs back into the original batch layout. Since two branches may have different execution costs, Flex further applies load-aware branch coordination by assigning more resources to heavier branch, reducing synchronization stalls before merging. Hybrid-Aware Context Caching. Real-world LLM serving workloads often contain repeated long prefixes, such as system prompts, document contexts, and multi-turn conversation histories. Recomputing these prefixes for every request wastes substantial prefill computation, especially in long-context scenarios. Context caching [19, 41, 45] addresses this issue by identifying matched textual prefixes and reusing their previously generated caches. Existing techniques typically assume that a matched textual prefix uniquely determines all reusable caches. However, this assumption no longer holds for Flex: even when two requests share the same prefix, they may select different Attention and SSM layers across hybrid blocks, producing different layer-wise caches. Moreover, conventional cache eviction operates at the context level, where evicting a prefix removes all layer-wise caches associated with it. This coarse-grained eviction is inefficient for request-adaptive hybrid execution, because only a subset of layer caches under a matched context may be frequently reused. Evicting the whole context can reduce future cache hit ratios, while retaining it may keep many rarely used layer caches in memory. To address these limitations, Flex introduces the hybridaware context caching (§6). Its core mechanism, a two-level radix tree structure, decouples textual-prefix matching from layer-wise cache matching. After identifying a shared textual prefix, it further determines which layer-wise caches are compatible with the current hybrid layout. Built on this structure, Flex further performs priority-based eviction at

xN Feed-Forward Network

Feed-Forward Network

Attention Layer

SSM Layer

(a) Transformer-based LLM xN

Hybrid LLM Blocks

Feed-Forward Network

SSM Layer

Attention Layer

(b) SSM-based LLM

1:3 1:6

SSM-based LLMs. State space model (SSM), such as Mamba model [9, 16], provide an efficient alternative by summarizing historical information into compact recurrent states:

1:1

Hybrid LLMs

Attention-based LLM

1:12

SSM-based LLM

Better Performance

ℎ𝑡 = 𝐴̄𝑡 ℎ𝑡−1 + 𝐵𝑡̄ 𝑥𝑡 ,

𝑦𝑡 = 𝐶𝑡⊤ ℎ𝑡 + 𝐷𝑥𝑡 ,

(2)

(d) Pareto frontier of model performance.

where ℎ𝑡 is the SSM state at time 𝑡 , 𝐴̄𝑡 ∈ ℝ(𝑁 ,𝑁 ) and 𝐵𝑡̄ ∈ ℝ(𝑁 ,1) define the input-dependent state update, 𝐶𝑡 ∈ ℝ(𝑁 ,1) maps the state to the output, and 𝐷 is a skip connection. Unlike conventional linear time-invariant SSMs, Mamba makes the state update input-dependent, enabling selective preservation of information from the input sequence [16]. Since an SSM layer compresses historical tokens into a bounded-size state, it avoids the growing Attention KV cache required by Attention-based LLMs and supports efficient long-sequence the layer-wise cache level, accounting for context sharing inference. Mamba-2 further improves the efficiency of selecpotential, layer-wise reuse likelihood, and recomputation tive SSM computation through state-space duality [9]. Howcost. As a result, Flex preserves useful shared prefixes while ever, this compact recurrent state may limit the ability to evicting low-value layer caches, enabling correct and effirecall context-dependent information, making SSMs less efcient context reuse for dynamic hybrid LLM serving. fective on tasks that require precise long-range context modResults: We evaluate Flex on the NVIDIA H100 GPU useling [46]. ing six datasets spanning reasoning, QA, and summarizaHybrid LLMs. Hybrid LLMs combine Attention layers and tion tasks. Experiments show that the proposed hybrid model SSM layers within the same model to balance modeling caachieves a better quality-efficiency trade-off than homogepability and inference efficiency, as shown in Figure 2. A neous and static hybrid baselines, obtaining the best results key design factor is the Attention-to-SSM layout, which deon 13 out of 18 quality metrics while reducing estimated termines how frequently Attention and SSM layers are used. TFLOPs/token by up to 67.2% relative to LLaMA. At the sysIn Figure 2(d), we evaluate layouts of 1∶1, 1∶3, 1∶6 and 1∶12, tem level, Flex improves up to 3.87× the serving throughput where 1∶𝑘 layout denotes one Attention layer for every 𝑘 of vLLM while substantially reducing time-to-first-token (TTFT) SSM layers. The SSM-only configuration achieves the lowest and time-per-output-token (TPOT). computation cost but suffers substantial accuracy degradation, whereas the Attention-only configuration incurs much 2 Background and Motivation higher cost. Several hybrid layouts achieve higher accuracy than both endpoints with moderate computation cost, demon2.1 Background strating the advantage of combining Attention and SSM layAttention-based LLMs. Attention-based LLMs rely on multiers. Moreover, the performance varies noticeably across layhead self-attention to capture global token relationships [36]. Given the hidden representations 𝑋 , the Attention layer projects outs, indicating that the hybrid layout is a critical factor in determining the quality-efficiency trade-off. them into queries (𝑄 ), keys (𝐾 ), and values (𝑉 ) tensors and Figure 2. Overview and quality-efficiency trade-off of conventional hybrid LLMs. (a)-(c) Illustrations of Attentionbased, SSM-based, and conventional hybrid architectures. (d) PIQA accuracy versus model efficiency under different Attention-to-SSM layouts. All models have 3B parameters.

computes: 2.2 Attn(𝑄, 𝐾 , 𝑉 ) = softmax (

𝑄𝐾 ⊤ √𝑑ℎ

)𝑉,

Motivation

The quality-efficiency trade-off of a hybrid LLM is largely determined by its Attention-to-SSM layouts. Existing hybrid LLMs determine this configuration based on empirical studies and keep it static during model serving. For example, Jamba [22] periodically inserts Attention layers into an SSMdominated backbone, while Nemotron-H [3] fixes a predefined distribution of Attention and SSM layers. Once constructed, all requests are forced to execute the same layout, regardless of their input characteristics or contextual requirements. We find that such static layouts are insufficient for diverse inference workloads: different tasks, and even different requests from the same task, can prefer different uses of Attention and SSM computation.

(1)

where 𝑑ℎ denotes the head dimension. This dense interaction allows each token to directly aggregate information from previous tokens, providing strong capability for reasoning and long-context understanding. However, the computation of an Attention layer grows quadratically with the prompt length during prefill [43, 50, 54, 56, 57]. During decoding, it needs to maintain KV caches for preceding tokens, introducing increasing memory and computation costs as the context grows. 3

Attn:SSM

1:1 1:3

1:6 1:12

60 40 20 ARC-c (Accuracy)

PIQA (Accuracy)

PubMed (ROUGE-L)

100

60 40 20

Serving Runtime

87.11 76.33

Incoming Requests

Request-Adaptive Hybrid Model

Request Scheduler

MORPH

Batching requests for processing in parallel

51.09

Selective Batching Engine (§5) 23.83

1:1

1:3

1:6

Split & Merge

1:12

Attention Processing

Hybrid-Aware Context Caching (§6)

Figure 3. Quality and throughput of static hybrid LLMs with different Attention-to-SSM layouts.

CTX2 CTX1 CTX3

ARC-c Accuracy (%)

Attention/SSM tokens

1:1

Attn.:SSM 1:3 1:6

Block N

SSM Processing Reuse layer-wise KVs and states for hit prefix

Output Tokens

…

Throughput (tokens/s)

Inference Quality

100

CTX4 ✅ ✅ ✅

✅ ✅ ❌ ❌

Feed-Forward Network

SSM Layer

Attention Layer

Block 2 Block 1

Prefill-Oriented Gate (§4)

Input Tokens

Activate SSM or Attention

Attention KV/SSM state

Figure 5. System overview of Flex. For clarity, standard components in the model, e.g., normalization, are omitted.

1:12

46 44 40 36

Set 1

Set 2

Set 3

Set 4

Set 5

Set 6

each subset. As shown in Figure 4, the best layout changes across subsets. For example, Set 1 and Set 3 favor more Attentionheavy settings, while Set 4 and Set 6 achieve the best results under the 1∶6 hybrid layout. Set 5 instead performs best with the 1∶12 layout. Even within a single reasoning benchmark, there is no universally optimal static hybrid structure. This result suggests that the preferred Attention-toSSM layout depends not only on the task category, but also on the specific requests within the same dataset. Therefore, a static hybrid layout is fundamentally limited, motivating a dynamic hybrid architecture that can adapt its structure to each request.

Figure 4. Accuracy of different static Attention-to-SSM layouts on six disjoint ARC-Challenge subsets. Observation 1: No single Attention-to-SSM layout can achieve the best performance across different workloads. We first study whether a single static hybrid layout can achieve a consistently good quality-efficiency trade-off across different workloads. We use a set of static hybrid models from [37], which combine Attention layers with SSM layers under different Attention-to-SSM layouts ranging from 1∶1 to 1∶12. We evaluate these layouts on a single NVIDIA H100 GPU using three representative datasets: ARC-Challenge for reasoning, PIQA for commonsense QA, and PubMed for longdocument summarization. We report task quality together with generation throughput. As shown in Figure 3, different workloads favor different hybrid layouts. On ARC-Challenge, the 1∶6 hybrid layout achieves the highest accuracy, suggesting that reasoning benefits from sufficient SSM efficiency while retaining selected Attention computation. On PIQA and PubMed, the 1∶1 hybrid layout performs best, reflecting its stronger demand for fine-grained long-context modeling. Meanwhile, increasing the proportion of SSM layers significantly improves throughput, from 23.83 tokens/s under 1∶1 to 87.11 tokens/s under 1∶12. These results show that the Attentionto-SSM layout directly affects both task quality and serving efficiency, but no single static layout is consistently optimal across diverse workloads.

System Overview

Figure 5 presents an overview of Flex, an inference system that combines a request-adaptive hybrid model with an optimized serving runtime through algorithm-system codesign. Request-adaptive hybrid model. The right part of Figure 5 shows the proposed hybrid model, which consists of a stack of adaptive hybrid blocks. Each block contains a lightweight gate, paired Attention and SSM branches, and shared components such as FFNs. For each request, the gates select one branch per block, forming a request-specific hybrid layout that balances modeling capability and inference cost. Serving runtime. The left part of Figure 5 presents the runtime, which addresses two challenges of dynamic hybrid execution: requests within a batch may select different branches, and requests sharing a textual prefix may require incompatible caches. The runtime incorporates two modules. The selective batching engine executes shared operations on the original packed batch and splits requests into Attention and SSM sub-batches for branch-specific computation. Hybrid-aware context caching matches both textual prefixes and layer-selection histories to identify reusable Attention KV caches and SSM states. Workflow. Given incoming requests, the runtime forms serving batches and identifies candidate cached prefixes. During prefill, the gate at each hybrid block selects one branch per request, and the runtime executes the selected branches

Observation 2: Even within the same workload, different requests prefer different Attention-toSSM layouts. Even within the same task type, different requests can still prefer different hybrid structures. To show this, we further study ARC-Challenge by partitioning the evaluation set into six disjoint subsets, each containing 100 requests, and evaluate the same four static Attention-to-SSM layouts on 4

Hybrid Blocks Feed-Forward Network Prefill-Oriented Gate

SSM Layer Attention Layer

Feed-Forward Network Prefill-Oriented Gate

inference and newly generated tokens can directly reuse the Attention KV caches or SSM state caches produced by previous tokens, avoiding the recomputation overhead. From the model perspective, the input tokens processed during prefill already encode the key characteristics of the request, such as task type, context length, and dependency requirements [30, 52]. Our experimental results further show that such prefill information is sufficient to guide the Attention and SSM layer choices for the whole request, enabling the hybrid model to adapt its computation without repeatedly invoking the gate during decoding. In each hybrid block, the gate first aggregates the prefill hidden states into a compact request representation, and then feeds it into a lightweight gate network to obtain the probabilities of selecting the Attention and SSM layers. We use top-1 gating and activate the layer with the larger probability. The formal definition of the gating process is provided in the supplementary manuscript.

Hybrid Blocks

SSM Layer Attention Layer

Feed-Forward Network Prefill-Oriented Gate

SSM Layer Attention Layer

Feed-Forward Network Prefill-Oriented Gate

SSM Layer Attention Layer

Autoregressively Prefill Phase

Decoding Phase

Figure 6. Illustration of the prefill-oriented gating mechanism. The activated layers are shown with filled colors, while inactive layers are left unfilled. while reusing caches compatible with the layer-selection history up to that block. The runtime records these choices and keeps them fixed throughout decoding, grouping requests accordingly and updating their Attention KV caches or SSM states until generation completes.

4.2

Prefill-Oriented Gating

This section presents the prefill-oriented gating mechanism of Flex. The goal is to enable request-adaptive Attention and SSM layer selection while preserving efficient autoregressive decoding. 4.1

Gating Design

A natural way to enable dynamic hybrid execution is to make layer choices for each token. However, such tokenlevel gating is ill-suited for autoregressive hybrid LLM serving. During decoding, each newly generated token relies on the Attention KV caches or SSM state caches produced by previous tokens to avoid redundant computation. If different tokens from the same request choose different layers within the same block, later tokens may no longer be able to directly reuse the caches produced by earlier tokens. As a result, the runtime must recompute previous tokens under the new layer choice, introducing substantial overhead and complicating decoding. To avoid these problems, Flex adopts a coarse-grained prefill-oriented gating mechanism in the request-adaptive hybrid model. As shown in Figure 6, this mechanism has two key properties. First, the gate is evaluated only during prefill, where each hybrid block makes one request-level choice between the Attention layer and the SSM layer based on all input tokens. Second, the selected layer choices are fixed after prefill and directly reused throughout decoding, so generated tokens do not trigger additional gating decisions. This design is reasonable for both system efficiency and hybrid adaptation. From the system perspective, prefill-oriented gating preserves a consistent computation throughout model 5

Hybrid Model Training

The hybrid model in Flex is initialized from pretrained Attention and SSM components [11, 23]. We use structured pruning to align their hidden dimensions [26]. When their depths differ, we select source layers using KL-divergencebased importance scores and pair the selected Attention and SSM layers to form hybrid blocks. We jointly fine-tune the gates and paired branches while freezing the shared embedding, FFN, and normalization modules initialized from the Attention backbone. To ensure fine-tuning quality, we use a soft routing mode to keep optimization differentiable. For the ℓ-th hybrid block, let 𝐻 ℓ−1 denote the hidden states from the previous block, and let Attnℓ (⋅) and SSMℓ (⋅) denote the candidate Attention and SSM layers in this block, respectively. The gate outputs the probabilities of selecting the Attention and SSM layers. Instead of activating only one layer, the block computes a weighted combination of the two branch outputs: ℓ Attnℓ (𝐻 ℓ−1 ) + 𝑝 ℓ SSMℓ (𝐻 ℓ−1 ), ̃ ℓ = 𝑝attn 𝐻 ssm

(3)

ℓ and 𝑝 ℓ are the gate probabilities for the Attenwhere 𝑝attn ssm ̃ ℓ is the tion and SSM layers at the ℓ-th hybrid block, and 𝐻 output before the following feed-forward network. This soft routing formulation allows gradients to update the gate as well as both candidate layers. During inference, the model switches to hard routing, where each hybrid block activates only the layer with the larger gate probability. The training objective balances generation quality and inference efficiency. For generation quality, we use the standard supervised fine-tuning loss. To encourage efficient layer selection, we add a lightweight cost-aware regularization

Attention 𝑄, 𝐾, 𝑉 = softmax 𝑄𝐾 ! / 𝑑" 𝑉

[3, H]

Split [6, H]

Gate Linear

Attention

ℎ# = 𝐴̅# ℎ#$% + 𝐵)# 𝑥# ,

Merge

[3, H]

𝑥!" 𝑥""

𝑥!# 𝑥"#

[6, H] Attention KV Cache SSM State Cache

Next Block

Figure 7. Illustration of the proposed split-and-merge mechanism in selective batching engine. We ignore Layer Norm, Add, and other operations for simplicity.

𝑁

1 ℓ 𝑐̄ ), ∑ (𝑝 ℓ 𝑐 ̄ + 𝑝ssm ssm 𝑁 ℓ=1 attn attn

(4)

where ℒsft denotes the standard supervised fine-tuning loss for next-token prediction, 𝑁 is the number of hybrid blocks, and 𝑐attn ̄ and 𝑐ssm ̄ are normalized layer-level cost weights for selecting the Attention and SSM layers, respectively. The coefficient 𝜆 controls the trade-off between generation quality and inference efficiency. This regularization is used only to bias the gate toward efficient layer choices during training.

5.2

𝒪(𝐿𝑑 2 + 𝐿2 𝑑) 𝒪(𝐿𝑑 2 + 𝐿𝑑𝑠 )

𝒪(𝑑 2 + 𝑟𝑑) 𝒪(𝑑 2 + 𝑑𝑠 )

Load-Aware Branch Coordination

Although selective batching enables the Attention and SSM layers to run independently, simply launching them on normal CUDA streams is insufficient. The two branches have different execution characteristics: the Attention branch is sensitive to context length and KV access, while the SSM branch is mainly dominated by recurrent state updates. Moreover, the number of tokens assigned to each branch changes across blocks and batches. As a result, one branch may finish earlier than the other, forcing the merge operation to wait for the slower branch. To reduce this synchronization stall, Flex adopts loadaware branch coordination. The key idea is to estimate the relative workload of the Attention and SSM branches and assign more SM resources to the heavier branch. Guided by the computation complexity in Table 1, Flex uses lightweight profiled cost models to predict branch execution costs from simple runtime features, including the number of requests in each branch, input lengths during prefill, and current KV-cache lengths during decoding. Since branch coordination only requires relative costs, the predictor does not need exact latency estimation. In our implementation, its prediction error is within 6.32% for the Attention branch and 5.18% for the SSM branch. The profiling is a one-time effort for each model-GPU pair and can be completed efficiently. Detailed cost models are provided in the supplementary manuscript. Based on the predicted costs, Flex partitions the schedulable GPU resources between the Attention and SSM subbatches at the SM granularity. In our implementation, we

Selective Batching Engine

This section introduces the design of selective batching engine in Flex, which supports efficient batching processing of the proposed request-adaptive hybrid model. 5.1

Decoding

therefore preserve normal batching efficiency. During prefill, the gate of each hybrid block is executed once to determine whether each request uses the Attention or SSM layer, and the selected layer processes all input tokens of that request. Meanwhile, the corresponding Attention KV or SSM state is cached. After prefill, each request records its selected layer choices across all hybrid blocks. During decoding, the gate is disabled, and Flex directly reuses the recorded layer choices to form Attention and SSM sub-batches at each block. The Attention layer reads and appends the Attention KV cache, while the SSM layer reads and updates the SSM cache.

term: ℒ = ℒsft + 𝜆

Prefill

Table 1. Computation complexities of Attention and SSM layers. Here, 𝐿 denotes the input length in prefill, 𝑟 denotes the current cached context length during decoding, 𝑑 denotes hidden dimension, and 𝑑𝑠 denotes SSM state dimension.

FFN Linear

[6, H]

𝑥!! 𝑥"!

Attention Layer SSM Layer

[6, H]

𝑦# = 𝐶#! ℎ# + 𝐷𝑥#

SSM

[3, H]

Split-and-Merge Execution

Given the request-adaptive layer choices produced by dynamic hybrid execution, the key question is how to execute a mixed batch without sacrificing normal batching efficiency. Flex addresses this through a split-and-merge execution mechanism. As shown in Figure 7, the input tokens from different requests are first packed into a single batch 𝑗 tensor. For example, 𝑥𝑖 denotes the 𝑗 -th token of request 𝑖, and the two requests in the figure form a packed tensor with shape [6, 𝐻 ]. Flex keeps this packed layout for shared operations, such as gate network and FFN computation, and splits the batch only before the layer-specific Attention or SSM computation. Requests selecting the Attention layer are compacted into an Attention sub-batch, while requests selecting the SSM layer are compacted into an SSM subbatch. The two sub-batches are then executed by their corresponding kernels. Afterward, Flex merges their outputs back into the original packed batch layout, so subsequent shared operations can still be processed as a regular batch. This design confines dynamic layer selection to the Attention and SSM computation inside each hybrid block. Other operations still use the standard packed tensor interface and 6

Root

CTX1: You are a helpful assistant. KV

State

CTX2: Keep your answers concise. State

State

only the unmatched suffix. However, conventional RadixAttention cannot be directly applied to Flex. Its key assumption is that a matched textual prefix uniquely determines all layer-wise caches that can be reused. This assumption holds for static models because the layer type at each position is fixed by the model architecture. For example, in an Attention-only model, each layer always produces a KV cache, while in an SSM-only model, each layer always produces an SSM cache. In Flex, requests sharing a textual prefix may select different layer layouts depending on their full contexts. As shown in Figure 8, the incoming request matches the cached prefix “You are a helpful assistant.”, but requires Attention KV caches at the third and fourth blocks, where the cached entries are SSM states. Conventional methods treat the matched entry as a whole: it is either reused or rejected. Reusing the cached states despite the layout mismatch can introduce inconsistent intermediate representations and degrade inference quality. Alternatively, rejecting the incompatible entry results in a cache miss and requires recomputing the shared prefix, losing the opportunity to exploit the textual overlap. To support correct and efficient context reuse under dynamic hybrid execution, Flex implements hybrid-aware context caching using a two-level radix-tree structure. The outercontext radix tree indexes textual prefixes, while an innercache radix tree attached to each context node organizes layer-selection paths and their associated Attention KV caches and SSM states. Cache lookup proceeds in two steps. First, Flex locates the longest matching textual prefix in the outercontext radix tree. It then examines the inner-cache radix tree of the matched node and its ancestors to identify compatible cache paths. Compatibility requires matching the sequence of layer choices from the first block through the block being reused. Among these candidates, Flex selects the prefix and cache path with the lowest estimated recomputation cost, considering both prefix length and the number of reusable leading blocks. As illustrated in Figure 9, the inner-cache radix tree associated with CTX1 contains multiple execution paths. These paths share caches for the first two blocks, then branch according to the branch choices at the third and fourth blocks. For cache, Flex retains input hidden states and terminal SSM states, reconstructing Attention KV caches on demand. For an incoming request matching CTX1, Flex follows the inner-tree path consistent with its branch-selection history and reuses the corresponding caches. At the first route mismatch, the cached hidden states remain valid because all preceding branch choices match. Flex uses these states to recompute the prefix at that block and all subsequent blocks, then extends the inner-cache radix tree with the newly computed caches while retaining existing branches. This organization supports partial cache reuse and preserves alternative execution paths for future requests sharing same textual prefix.

CTX3: Include a brief example. KV

State

(a) Traditional radix tree indexing of KV caches and SSM states.

You are a helpful assistant. Use simple words.

Inference Request

CTX1: You are a helpful assistant.

Context Matching

State

Cache miss

Cache Mismatching

(b) Cache incompatibility despite a matching text prefix.

Figure 8. Motivation for hybrid-aware context caching. (a) A conventional radix tree indexes Attention KV caches and SSM states by textual prefixes. (b) The incoming request matches context 1 (CTX1) but selects different layer types, resulting in a cache miss.

use NVIDIA H100 SXM GPUs and empirically set the allocation granularity to 𝐺 = 16 SMs, following recent intra-GPU multiplexing practice [4, 13, 55]. Of the 132 SMs available on each GPU, 128 are included in the partitioned resource pool, yielding a compact set of feasible allocations in multiples of 16 SMs. Flex rounds the target allocation to the nearest feasible partition and assigns the remaining schedulable SMs to the other sub-batch. If only one sub-batch is nonempty, it skips partitioning and assigns the entire resource pool to that sub-batch. This compact allocation space enables lightweight online selection while providing sufficient flexibility to balance the two computations. The detailed SM allocation rule is provided in the supplementary manuscript. The selected SM partition is implemented using NVIDIA GreenContext [8]. Flex creates separate Green Contexts and CUDA streams for the Attention and SSM sub-batches, binding each stream to its assigned SM partition. Since GreenContext only constrains the SM scheduling pool, existing FlashAttention and mamba_ssm kernels can run without modification. The two sub-batches are launched concurrently and synchronized with CUDA events before the merge operation and the next hybrid block. This implementation assigns more compute resources to the heavier sub-batch and reduces the completion-time imbalance between them.

Hybrid-Aware Context Caching

This section presents the hybrid-aware context caching mechanism in Flex, which enables correct prefix reuse when the same textual context may produce different layer-wise Attention KV and SSM state caches. 6.1

Context Caching Mechanism

Context caching is widely used to improve LLM serving efficiency by reusing previously computed caches (e.g., Attention KVs and SSM states) for repeated prefixes [19, 41, 45]. For example, RadixAttention [51] organizes historical contexts with a radix tree, where each node represents a textual prefix. When a new request arrives, the runtime searches the radix tree to find the longest matched prefix, reuses the caches associated with the matched prefix, and computes 7

Outer-Context Radix Tree Root

CTX1: You are a helpful assistant.

Inner-Cache Radix Tree

State

CTX2: Keep your answers concise.

State

SSM states, supporting Attention KV reconstruction and recomputation from the first mismatched layer. For concurrent Attention and SSM execution, Flex uses NVIDIA GreenContext to partition SM resources and launches the two subbatches on separate CUDA streams. It also adopts a unified paging-based memory manager for heterogeneous Attention KV caches and SSM caches, following Jenga [47]. The runtime currently uses a conventional continuous batching scheduler [44], and is compatible with other scheduling policies such as chunked prefill, prefill-decode disaggregation, and latency-aware scheduling [1, 12, 27, 30, 53]. Detailed implementation is provided in the supplementary manuscript.

CTX3: Include a brief example.

State

Figure 9. Illustration of the proposed two-level radix-tree structure. The outer-context radix tree indexes contextual prefixes, while each inner-cache radix tree organizes alternative layer-selection paths and their associated Attention KV and SSM state caches, enabling cache reuse along compatible execution paths for a matched prefix.

6.2

Priority-based Eviction Policy

Hybrid-aware context caching enables correct cache reuse under dynamic hybrid execution, but also increases the number of cache entries. With a fixed layer layout, conventional RadixAttention associates each textual prefix with one set of layer-wise caches. In Flex, the same context node can maintain multiple cache paths corresponding to different Attention and SSM choices. Evicting an entire context node therefore discards all these paths, potentially removing frequently reused caches together with rarely accessed ones. Flex instead performs eviction at the granularity of individual layer-wise cache entries. Under GPU memory pressure, it evicts entries from the inner-cache radix trees. An outer-context node is removed only when all its associated caches have been evicted and the node is no longer needed to represent shared textual prefixes. To determine which entries to retain, Flex assigns each cache 𝑐 a retention priority:

𝑃(𝑐) =

𝐻 (𝑐) ⋅ 𝑅(𝑐) ⋅ 𝐷(𝑐), 𝑀(𝑐)

Evaluation

8.1

Evaluation Setup

Platform. We evaluate Flex on a server equipped with one NVIDIA H100 SXM GPU (80GB memory) and Intel Xeon Platinum 8458P CPUs. The server runs Ubuntu 22.04.5 LTS with NVIDIA driver 580.82.07 and CUDA runtime 12.4. We use Python 3.12.3, PyTorch 2.4.1, Transformers 4.46.0 and mamba_ssm 2.2.4. Models and baselines. We evaluate the proposed hybrid model against both homogeneous and static hybrid LLMs. The homogeneous baselines include Llama-3 [15] as an Attentiononly model and Mamba-2 [9] as an SSM-only model. The static hybrid baselines include Zamba [14], Jamba [22], and Nemotron-H [3], whose Attention-to-SSM layouts are fixed after model construction. For serving-system evaluation, we compare Flex with vLLM [21] and SGLang [51], two widely used LLM serving systems, as well as Jenga [47], which further provides memory management support for Attention KV and SSM states. All systems use the same model, workload, and generation settings, with unrelated optimizations such as speculative decoding disabled. Datasets and workloads. We fine-tune each model on a 10K-example subset of ShareGPT-90K and reuse the resulting checkpoint across all evaluation tasks without task-specific fine-tuning. We evaluate model quality on six datasets covering reasoning, question answering (QA), and summarization. For reasoning, we use ARC-Easy and ARC-Challenge [6]. For QA, we use PIQA [2] and TruthfulQA [25]. For summarization, we use CNN/DailyMail [35] and PubMed [7]. Metrics. For model quality, we report accuracy for ARCEasy, ARC-Challenge, and PIQA, MC1 accuracy for TruthfulQA, and ROUGE-L for CNN/DM and PubMed. For model efficiency, we estimate decoding TFLOPs per token:

(5)

where 𝐻 (𝑐) is its recent access frequency, 𝑅(𝑐) is the recomputation cost saved by reusing it, 𝑀(𝑐) is its memory footprint, and 𝐷(𝑐) is a depth factor that favors caches at earlier blocks. Flex evicts entries with the lowest priority first. This policy favors caches that provide greater expected computation savings per unit of memory. The depth factor further prioritizes early-layer caches, which can form shared ancestors of multiple execution paths in an inner-cache radix tree.

Implementation

We implement a prototype of Flex with approximately 6.5K lines of code using Python, PyTorch, HuggingFace Transformers, and CUDA/C++. The construction and fine-tuning of the request-adaptive hybrid model are implemented in PyTorch, while the serving runtime extends a vLLM-style execution loop with selective batching and hybrid-aware context caching. It caches layer hidden states and terminal

TFLOPs/token = 8

SSM lin +4𝐿 ∑ 𝐹𝑟,𝑡 𝑟,𝑡 ℓ∈𝒜𝑟 𝑛𝑞,ℓ 𝑑ℎ,ℓ +∑ℓ∈𝒮𝑟 𝐶ℓ

1012

(6)

lin counts executed linear operations, including prowhere 𝐹𝑟,𝑡 placement cannot adapt to the varying dependency patterns jections, FFNs, and the output head. 𝒜𝑟 and 𝒮𝑟 are the Atof different requests. tention and SSM layers activated by request 𝑟 . 𝐿𝑟,𝑡 is the atQA tasks. On PIQA and TruthfulQA, Flex also shows strong tended KV length at decoding step 𝑡 , and 𝑛𝑞,ℓ and 𝑑ℎ,ℓ are the performance. At the medium scale, it achieves 83.1% on PIQA and 68.9% on TruthfulQA, outperforming all baselines. At query-head count and head dimension. For Flex, only the the large scale, it reaches 85.3% on PIQA and 75.2% on Truthselected branch at each block is counted, together with the fulQA. These results indicate that QA workloads do not alshared operations. 𝐶ℓSSM counts convolution, state update, ways require Attention in every layer, but still need Attenand readout operations beyond the linear projections [9, 16]. tion in selected layers to preserve factual and contextual We count each multiply-add as two FLOPs [5, 17], omit norsignals. Flex captures this behavior by adapting the Attenmalization, activations, and softmax, and average over evaltion and SSM choices to each request instead of relying on uated decoding steps. For serving performance, we report request throughput, time-to-first-token (TTFT), time-per-output- a static hybrid layout. Summarization tasks. Summarization workloads have a token (TPOT), and end-to-end latency. stronger dependence on global context modeling. AttentionOther configurations. Fine-tuning uses Adam with a learnheavy models such as Llama remain competitive, especially ing rate of 2 × 10−4 , an effective batch size of 64, and a maxat the large scale. Nevertheless, Flex achieves the best sumimum sequence length of 4,096 tokens. We train for 1,000 marization results at the medium scale, reaching 23.4 ROUGEoptimization steps using BF16 precision and fully sharded L on CNN/DailyMail and 21.5 ROUGE-L on PubMed. At the data parallelism (FSDP), with 𝜆 = 0.2 across all model scales small scale, it also achieves the best PubMed score and reand benchmarks. mains close to the best CNN/DailyMail result. These results show that summarization benefits from more frequent At8.2 Overall Model Performance tention activation, but does not require all layers to be AtQuality-efficiency trade-off. Table 2 compares Flex with tention layers. Flex can allocate Attention to the layers and requests that need global context, while using SSM layers homogeneous and static hybrid LLMs across three model scales. Overall, Flex achieves the best results on 13 out of elsewhere to reduce computation. 18 quality metrics while maintaining competitive computaRouting Effectiveness. We compare three routing stratetion cost. At the small scale, Flex obtains the best results on gies using the same fine-tuned checkpoint at each model ARC-Easy, TruthfulQA, and PubMed, and remains close to scale. We collect candidate layouts from adaptive routing the best baseline on ARC-Challenge and CNN/DailyMail. At on held-out validation data and evaluate each as a fixed layout under matched inference-cost budgets. Within this canthe medium scale, it consistently outperforms all baselines on all six datasets. At the large scale, it achieves the best didate set, Global Optimal (G-Optimal) selects one layout results on reasoning and QA tasks, while remaining comthat maximizes the macro-average of normalized validation petitive on summarization tasks. More importantly, Flex scores across datasets, while Dataset Optimal (D-Optimal) achieves these quality gains with substantially lower estiselects the best layout separately for each dataset. Both stratemated TFLOPs/token: compared with Attention-only Llama gies keep their selected layouts fixed during evaluation, whereas Adaptive selects a layout for each request. As shown in Tamodels, it reduces computation cost by 40.5%, 33.2%, and 67.2% at the three evaluated scales, respectively. Compared ble 3, our adaptive method outperforms both static stratewith Nemotron-H, it reduces TFLOPs/token by 28.4%, 28.9%, gies across all 18 model-dataset combinations. Compared and 58.7%. Although Mamba and some static hybrid models with D-Optimal, it achieves average gains of 4.86 percentage can be cheaper in certain settings, their quality is often less points in accuracy and 2.55 points in ROUGE-L. Although Dstable across tasks. These results show that request-adaptive Optimal consistently improves over G-Optimal, its remainAttention and SSM selection provides a better quality-efficiency ing gap to Adaptive demonstrates the benefit of adapting trade-off than static hybrid layouts. We further analyze evallayer choices to individual requests within the same dataset. uation results across different tasks. Reasoning tasks. On ARC-Easy and ARC-Challenge, Flex 8.3 Overall System Performance performs particularly well at medium and large scales. For

We evaluate the system-level performance of Flex at three model scales on a single H100 GPU, using FP8 quantization for the 61B model. We compare its serving runtime with vLLM, SGLang, and Jenga, with all methods serving the same model checkpoint at each scale. We also include two ablated variants: Flex w/o Bat. disables selective batching (§5), while Flex w/o Con. disables hybrid-aware context reuse (§6). We construct mixed workloads from all datasets

example, at the medium scale, Flex achieves 77.8% accuracy on ARC-Easy and 57.7% on ARC-Challenge, outperforming all static hybrid baselines. At the large scale, it further improves the results to 85.6% and 85.1%, respectively. These results suggest that reasoning tasks benefit from selectively activating Attention in important layers, while still using efficient SSM computation elsewhere. Static hybrid models improve over pure SSM models, but their static attention 9

ARC-E

ARC-C

PIQA

TruthfulQA

CNN/DM

PubMed

Model Efficiency

𝑃

𝑃act

Base Config

𝑁attn

𝑁ssm

𝐻hidden

Accuracy ↑

Accuracy (MC1) ↑

ROUGE-L ↑

TFLOPs/Token

Mamba Llama Zamba Jamba Nemotron-H OuRs

5.3B 3.2B 2.7B 3.0B 4.5B 4.4B

5.3B 3.2B 2.7B 3.0B 4.5B 3.2B

– 28 9 2 4 28

64 – 45 26 24 28

2560 3072 2560 2560 3072 2560

67.4 ± 0.11 73.2 ± 0.08 69.4 ± 0.24 69.7 ± 0.15 74.0 ± 0.24 74.7 ± 0.17

35.8 ± 0.13 42.4 ± 0.12 41.9 ± 0.25 47.3 ± 0.26 44.6 ± 0.17 45.5 ± 0.26

74.9 ± 0.14 75.5 ± 0.15 78.7 ± 0.31 70.5 ± 0.32 71.1 ± 0.41 76.6 ± 0.31

33.3 ± 0.07 42.1 ± 0.33 41.8 ± 0.17 48.8 ± 0.16 49.9 ± 0.33 51.4 ± 0.19

10.9 ± 0.14 17.1 ± 0.27 20.5 ± 0.22 18.9 ± 0.33 16.8 ± 0.19 20.1 ± 0.25

14.3 ± 0.15 15.9 ± 0.18 16.0 ± 0.19 18.1 ± 0.19 16.5 ± 0.20 19.4 ± 0.24

1.01 × 10−2 1.08 × 10−2 7.59 × 10−3 6.02 × 10−3 8.98 × 10−3 6.43 × 10−3

Mamba Llama Zamba Jamba Nemotron-H OuRs

8.2B 13.1B 7.4B 13.0B 12.3B 14.5B

8.2B 13.1B 7.4B 13.0B 12.3B 6.5B

– 55 13 1 6 32

56 – 68 7 28 32

4096 4096 3584 4096 5120 4096

72.5 ± 0.23 76.7 ± 0.17 70.3 ± 0.30 72.8 ± 0.24 77.3 ± 0.18 77.8 ± 0.17

50.9 ± 0.19 54.3 ± 0.17 45.4 ± 0.17 52.5 ± 0.22 56.1 ± 0.17 57.7 ± 0.36

76.6 ± 0.33 79.5 ± 0.19 81.1 ± 0.22 80.3 ± 0.38 82.2 ± 0.26 83.1 ± 0.28

38.8 ± 0.32 59.8 ± 0.24 65.6 ± 0.24 66.2 ± 0.41 68.7 ± 0.41 68.9 ± 0.39

16.7 ± 0.24 19.5 ± 0.10 20.1 ± 0.32 19.4 ± 0.15 18.3 ± 0.39 23.4 ± 0.14

15.3 ± 0.25 16.4 ± 0.22 16.2 ± 0.15 18.3 ± 0.18 16.8 ± 0.25 21.5 ± 0.13

1.71 × 10−2 2.62 × 10−2 1.98 × 10−2 2.68 × 10−2 2.46 × 10−2 1.75 × 10−2

Mamba Llama Zamba Jamba Nemotron-H OuRs

– 70B – 52B 56B 61B

– 70B – 12B 56B 23B

– 80 – 4 10 32

– – – 28 54 32

– 8192 – 4096 8192 8192

– 84.3 ± 0.16 – 73.7 ± 0.22 85.2 ± 0.23 85.6 ± 0.21

– 82.7 ± 0.24 – 63.6 ± 0.19 84.8 ± 0.17 85.1 ± 0.17

– 84.3 ± 0.33 – 83.1 ± 0.31 84.4 ± 0.12 85.3 ± 0.25

– 63.7 ± 0.18 – 72.1 ± 0.17 73.4 ± 0.26 75.2 ± 0.31

– 25.7 ± 0.09 – 20.5 ± 0.16 20.1 ± 0.19 24.5 ± 0.27

– 23.6 ± 0.11 – 19.8 ± 0.11 17.9 ± 0.28 22.3 ± 0.21

– 1.41 × 10−1 – 2.40 × 10−2 1.12 × 10−1 4.63 × 10−2

Models

Table 2. Comparison of model quality and efficiency across homogeneous, static hybrid, and our proposed request-adaptive hybrid model. 𝑃 and 𝑃𝑎𝑐𝑡 denote the total model parameters and the activated parameters for one request, respectively. 𝑁attn and 𝑁ssm denote the numbers of Attention and SSM layers. For Flex, 𝑃 counts all paired candidate branches, while 𝑃act reports the average activated parameters under request-adaptive layer selection. All multiple-choice tasks use free-form generation with regex-based letter extraction. Chat-template-aware models receive a “letter only” system prompt and reasoning models use no_think mode.

2 8

0.03

TTFT (s) 4

Batch Si(e

0.03

0.00 0.09

0.06

10 0

0.06

0.00 0.09

TPOT (s)

TTFT (s) 4

Mor h

0.06

Mor h w/o Con. 0.09

Mor h w/o Bat. TPOT (s)

Jenga 30

TPOT (s)

SGLang TTFT (s)

Throughput (req/s)

Through ut (req/s)

vLLM 6

0.03

Batch Si(e

0.00

Batch Si(e

Figure 10. End-to-end serving performance of Flex under varying batch sizes on a single H100 GPU. Rows correspond to the 4.4B, 14.5B, and 61B models, respectively. Columns report request throughput, TTFT, and TPOT. The 61B model uses FP8 quantization to avoid out-of-memory.

Scale

Method

ARC-E

ARC-C

PIQA

TruthfulQA

CNN/DM

PubMed

4.4B

G-Optimal D-Optimal Adaptive

68.2 71.3 74.5

37.2 40.4 45.6

70.5 73.6 76.4

45.4 47.3 51.3

16.6 18.3 20.2

15.2 16.5 19.6

14.5B

G-Optimal D-Optimal Adaptive

71.8 73.4 77.9

49.5 52.6 57.7

73.7 77.2 83.2

60.3 63.5 68.8

17.6 19.2 23.4

18.8 20.6 21.6

61B

G-Optimal D-Optimal Adaptive

74.3 79.6 85.4

72.1 76.4 85.0

75.4 80.3 85.3

69.7 72.6 75.4

20.3 21.5 24.6

19.6 20.1 22.1

8.4

Table 3. Comparison of routing methods across model scales.

in which requests share article prefixes and append different questions. We vary the number of requests processed per scheduling iteration, using batch sizes of 8, 16, 32, and 64 for the 4.4B model and 4, 8, 16, and 32 for the other two scales. We report request throughput, TTFT, and TPOT. Figure 10 shows that Flex consistently outperforms all three baselines in throughput, TTFT, and TPOT across the evaluated batch sizes and model scales. It achieves up to 3.79×, 3.87×, and 3.44× the throughput of vLLM for the 4.4B, 14.5B, and 61B models, respectively. Its throughput advantage over vLLM grows with batch size. Larger batches offer more opportunities to group requests taking the same branch, which Flex exploits through selective batching. Both ablated variants also achieve lower throughput than the full system, showing that selective batching and context reuse contribute to serving efficiency. Hybrid-aware context reuse substantially reduces TTFT on the mixed workload by avoiding repeated computation over shared article prefixes. Under dynamic layer choices, matching prefix tokens alone does not establish cache compatibility: the cached states also depend on the execution paths that produced them. Flex uses the two-level radix tree to check both textual prefixes and execution-path compatibility before reusing layer-wise Attention KV caches and SSM states. At the largest tested batch size for each scale, Flex reduces TTFT by an average of 80.9%, 78.9%, and 75.2% compared with vLLM, SGLang, and Jenga, respectively. It also reduces TTFT by 59.1% on average compared with Flex w/o Con., demonstrating the benefit of context reuse. The TPOT results show the benefit of selective batching during decoding. Each request retains its layer choices after prefill, allowing Flex to group requests taking the same branch into compact Attention and SSM sub-batches. At the same configurations, Flex reduces TPOT by an average of 71.6%, 63.1%, and 56.5% compared with vLLM, SGLang, and Jenga, respectively. Compared with Flex w/o Bat., it reduces TPOT by 43.6% on average, demonstrating that selective batching improves decoding efficiency under request-specific layer choices. 11

Ablation Study

Effectiveness of selective batching. We further evaluate the selective batching engine by varying the batch size from 2 to 16 on the PubMed dataset. We compare Flex with vLLM and an ablated variant, denoted as Flex w/o Coord., which retains split-and-merge batching but disables load-aware branch coordination for SM allocation between the Attention and SSM sub-batches. As shown in Figure 11, Flex consistently achieves lower end-to-end (E2E) latency than vLLM across all tested batch sizes and model scales. Its selective batching engine groups requests according to their layer choices at each block, forming compact Attention and SSM sub-batches for execution with the corresponding kernels. Averaged over all batch sizes, Flex reduces E2E latency compared with vLLM by 54.6%, 66.0%, and 59.9% for the 4.4B, 14.5B, and 61B models, respectively. At batch size 16, the corresponding reductions reach 61.9%, 73.2%, and 70.9%, showing a larger latency advantage at the upper end of the tested batch sizes. Comparing Flex with Flex w/o Coord. isolates the additional benefit of load-aware branch coordination. Although split-and-merge batching groups requests by their selected layers, the Attention and SSM sub-batches can still have imbalanced workloads, and the merge operation must wait for both branches to complete. Flex coordinates SM allocation according to branch workloads to reduce this imbalance. Compared with Flex w/o Coord., the full system reduces E2E latency by 21.7%, 28.2%, and 26.2% on average for the three model scales, respectively. For the 61B model at batch size 16, vLLM requires 33.53s, while Flex w/o Coord. and the full Flex require 13.77s and 9.75s, respectively. Branch coordination thus provides a further 29.2% latency reduction beyond split-and-merge batching alone in this configuration. Effectiveness of hybrid-aware context caching. We evaluate the proposed hybrid-aware context caching mechanism using a PubMed-based shared-context workload. Each article serves as a shared textual prefix, and multiple requests append different questions after the same article [47]. We vary the number of articles from 2 to 20 under a fixed cache capacity, increasing cache competition while preserving prefix reuse opportunities within each article. For conventional context caching [51], a context is counted as reusable only when all layer-wise caches associated with the matched prefix can be reused. If any Attention KV cache or SSM cache is incompatible with the current request’s layer choices, the whole context is treated as a miss. In contrast, the proposed hybrid-aware context caching indexes layer-wise caches under each matched context, so it can reuse compatible caches even when later layers diverge. We compare the two mechanisms using effective cache hit rate and TTFT. As shown in Figure 12, the hit rates of both methods decrease as the number of articles increases, because more

10 5 0

Batch Size

20 10 0

vLLM Morph /o Coord. Morph

E2E Latency (s)

vLLM Morph /o Coord. Morph

E2E Latency (s)

Batch Size

vLLM Morph /o Coord. Morph

30 15 0

Batch Size

Figure 11. Ablation study of the selective batching engine and branch coordination under varying batch sizes. From left to right, the three subfigures report end-to-end latency for the 4.4B, 14.5B, and 61B hybrid models in Flex, respectively. 30

Hybrid-Aware Co text Cachi g

50 25 0

Number of Articles

ARC-Challenge

Context Cachi g

Hybrid-Aware Co text Cachi g

TruthfulQA

75% 0.6

SSM 1

0.2

Attn.

Number of Articles

50%

0.4

CNN/DM

100%

0.8

Attn.

12 0

1.0

Attn. SSM

CDF

Context Cachi g

TTFT (s)

Hit Rate (%)

100

25% ARC-Challenge TruthfulQA CNN/DM

SSM 1

Block ID

0.0

Top-k Hybrid Layouts

Figure 13. Routing patterns of the 28-block hybrid model across three datasets. Left: per-block Attention and SSM selection ratios. Right: cumulative request coverage by the 𝑘 most frequent hybrid layouts within each dataset.

Figure 12. Effectiveness of hybrid-aware context caching under varying numbers of shared PubMed contexts.

shared contexts compete for the fixed memory capacity. Nevertheless, hybrid-aware context caching consistently maintains a higher effective hit rate. Across all settings, it achieves an average hit rate of 43.4%, compared with 18.7% for the baseline, improving the hit rate by 24.7 percentage points. When the number of articles reaches 20, conventional caching method retains only a 7% hit rate, whereas hybrid-aware context caching still achieves 24%. This confirms that layerwise cache indexing captures valid reuse opportunities that conventional prefix-level caching would discard once any layer becomes incompatible. The higher hit rate directly reduces prefill latency. As the number of articles increases, TTFT grows for both methods due to more frequent cache misses and recomputation. However, TTFT grows more slowly with hybrid-aware context caching because it reuses compatible Attention KV caches and SSM state caches instead of recomputing the entire shared context. On average, hybrid-aware context caching reduces TTFT from 10.68s to 5.92s, corresponding to a 44.6% reduction. At 20 articles, it reduces TTFT from 16.18s to 9.36s, a reduction of 42.2%. These results demonstrate that hybridaware context caching improves prefill efficiency by enabling fine-grained and correct context reuse under dynamic hybrid execution. Layer-wise selection. Figure 13 characterizes routing in the 28-block hybrid model in Flex across three datasets. The heatmaps show the fractions of requests selecting Attention

and SSM at each block. On ARC-Challenge, Attention selection gradually increases toward deeper blocks. On TruthfulQA, it peaks in the early-to-middle blocks and decreases thereafter. On CNN/DailyMail, Attention selection is highest in the middle blocks and lower toward both ends. These patterns show that the allocation of Attention and SSM computation varies across both block depth and workloads. The right panel illustrates cumulative request coverage by the top-𝑘 full-model layouts, ranked by frequency within each dataset. This view complements the per-block statistics by characterizing complete routing decisions across requests. For ARC-Challenge, 500 requests activate 35 distinct layouts, with the six most frequent covering over 90% of requests. Request-level routing therefore combines a small set of frequently selected layouts with a long tail of less common choices.

Related Work

Hybrid LLMs. Recent works reduce the cost of long-context LLM inference by replacing part of the attention computation with efficient sequence modeling modules [10, 28, 33]. SSM-based models, such as Mamba [16] and Mamba-2 [9] maintain compact recurrent states instead of full Attention KV caches. Hybrid LLMs further combine Attention and SSM layers to balance global-context modeling and inference efficiency, with representative examples including Jamba [22], 12

Zamba [14], Hymba [11], Mamba-in-Llama [38], and Nemotron- [4] Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. H [3]. While prior hybrid LLMs demonstrate the promise of 2026. Towards High-Goodput LLM Serving with Prefill-decode MulAttention and SSM hybridization, they rely on static layouts, tiplexing. In Proceedings of the 31st ACM International Conference on whereas Flex enables request-adaptive Attention and SSM Architectural Support for Programming Languages and Operating Syslayer selection within the same model. tems, Volume 2. 2030–2047. Efficient batching. Efficient batching is central to high[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won throughput LLM serving. Orca [44] introduces iteration-level Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scalscheduling, while vLLM [21] improves batching capacity with ing language modeling with pathways. Journal of machine learning PagedAttention. SGLang [51] supports structured LLM proresearch 24, 240 (2023), 1–113. gram execution, and systems such as Sarathi-Serve [1], Split[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have wise [27], DistServe [53], LoongServe [39], NanoFlow [55], solved question answering? try arc, the ai2 reasoning challenge. arXiv and MuxWise [4] optimize prefill/decode scheduling or GPU preprint arXiv:1803.05457 (2018). resource coordination. Unlike these systems, Flex supports [7] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, batches with divergent Attention and SSM layer choices and Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discoursepreserves efficiency through selective hybrid batching. aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Context caching. Context caching reduces redundant preChapter of the Association for Computational Linguistics: Human Lanfill computation by reusing caches for repeated prefixes [19, guage Technologies, Volume 2 (Short Papers). 615–621. 41, 42, 45, 48]. SGLang [51] proposes RadixAttention for [8] NVIDIA Corporation. 2025. CUDA Driver API: Green Conprefix-based cache reuse, and works such as Parrot [24], CacheBlend [41], texts. https://docs.nvidia.com/cuda/cuda-driver-api/group_CUDA_ and Mooncake [29] exploit reuse opportunities in structured, GREEN_CONTEXTS.html RAG, multi-turn, or disaggregated serving scenarios. Unlike [9] Tri Dao and Albert Gu. 2024. Transformers are SSMs: generalized models and efficient algorithms through structured state space duprior techniques that bind each matched prefix to a single ality. In Proceedings of the 41st International Conference on Machine reusable cache, Flex uses the two-level radix tree to disLearning. 10041–10071. tinguish different layer-wise Attention KV and SSM state [10] Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, caches for the same textual prefix. George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427 (2024). [11] Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Mahabaleshwarkar, Shih-Yang Liu, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, et al. 2025. Hymba: A hybrid-head architecture for small language models. In International Conference on Learning Representations, Vol. 2025. 97729–97755. [12] Kuntai Du, Bowen Wang, Chen Zhang, Yiming Cheng, Qing Lan, Hejian Sang, Yihua Cheng, Jiayi Yao, Xiaoxuan Liu, Yifan Qiao, et al. 2025. Prefillonly: An inference engine for prefill-only workloads in large language model applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 399–414. [13] Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, and Murali Annavaram. 2025. DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing. arXiv preprint arXiv:2511.04791 (2025). [14] Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, and Beren Millidge. 2024. The zamba2 suite: Technical report. arXiv preprint arXiv:2411.15242 (2024). [15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). [16] Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023). [17] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems. 30016–30030.

10 Conclusion This paper presents Flex, an inference system that combines a request-adaptive hybrid model with an optimized serving runtime through algorithm-system co-design. The model uses prefill-oriented gating to select Attention or SSM branches for each request and keeps the resulting layout fixed throughout decoding. The runtime integrates selective batching and hybrid-aware context caching to support efficient mixedbranch execution and correct prefix reuse. Experiments show that the hybrid model achieves a better quality-efficiency trade-off than static hybrid baselines, while the runtime improves serving throughput and reduces inference latency.

References [1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). 117–134. [2] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 7432–7439. [3] Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. 2025. Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models. arXiv preprint arXiv:2504.03624 (2025). 13

[18] Tomoki Ito and Shun Nakagawa. 2024. Tender document analyzer with the combination of supervised learning and llm-based improver. In Companion Proceedings of the ACM Web Conference 2024. 995–998. [19] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. 2025. Ragcache: Efficient knowledge caching for retrieval-augmented generation. ACM Transactions on Computer Systems 44, 1 (2025), 1–27. [20] Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In Proceedings of the 2024 chi conference on human factors in computing systems. 1–20. [21] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [22] Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. 2025. Jamba: Hybrid transformer-mamba language models. In International Conference on Learning Representations, Vol. 2025. 67959–67984. [23] Yixing Li, Ruobing Xie, Zhen Yang, Xingwu Sun, Shuaipeng Li, Weidong Han, Zhanhui Kang, Yu Cheng, Chengzhong Xu, Di Wang, et al. 2025. Transmamba: Flexibly switching between transformer and mamba. arXiv preprint arXiv:2503.24067 (2025). [24] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient serving of LLMbased applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 929–945. [25] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers). 3214–3252. [26] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. Advances in Neural Information Processing Systems 37 (2024), 41076–41102. [27] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132. [28] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. 2023. Rwkv: Reinventing rnns for the transformer era. In Findings of the association for computational linguistics: EMNLP 2023. 14048–14077. [29] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX conference on file and storage technologies (FAST 25). 155–170. [30] Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T Kalbarczyk, Tamer Basar, and Ravishankar K Iyer. 2024. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction. In International Conference on Architectural Support for Programming Languages and Operating Systems.

[31] Abdur-Rahman Ibrahim Sayyid-Ali, Daanish Uddin Khan, and Naveed Anwar Bhatti. 2026. Are LLM Web Search Engines Sustainable? A Web-Measurement Study of Real-Time Fetching. In Proceedings of the ACM Web Conference 2026. 2083–2093. [32] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). 173–191. [33] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621 (2023). [34] Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards endto-end optimization of llm-based applications with ayo. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1302–1316. [35] Hugging Face Team. 2024. Hugging Face Dataset cnn dailymail (CNN/DM). https://huggingface.co/datasets/cnn_dailymail [36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [37] Dustin Wang, Rui-Jie Zhu, Steven Abreu, Yong Shan, Taylor Kergan, Yuqi Pan, Yuhong Chou, Zheng Li, Ge Zhang, Wenhao Huang, et al. 2025. A systematic analysis of hybrid linear attention. arXiv preprint arXiv:2507.06457 (2025). [38] Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. 2024. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems 37 (2024), 62432–62457. [39] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 640–654. [40] Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2024. Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks. In Proceedings of the ACM Web Conference 2024. 1362–1373. [41] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the twentieth European conference on computer systems. 94–109. [42] Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, et al. 2026. Kvcomm: Online cross-context kv-cache communication for efficient llm-based multi-agent systems. Advances in Neural Information Processing Systems 38 (2026), 17882–17928. [43] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems 7 (2025). [44] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22). 521–538. [45] Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Hank Levy, and David Culler. 2025. Ic-cache: Efficient large language model serving via in-context caching. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 375–398. 14

[46] Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, and Jian Tang. 2026. Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention. Advances in Neural Information Processing Systems 38 (2026), 43752–43779. [47] Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, et al. 2025. JENGA: Effective memory management for serving LLM with heterogeneity. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 446–461. [48] Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2025. Pqcache: Product quantization-based kvcache for long context llm inference. Proceedings of the ACM on Management of Data 3, 3 (2025), 1–30. [49] Yichi Zhang. 2024. Detecting code comment inconsistencies using llm and program analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 683–685. [50] Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Zhangyang Wang. 2024. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems 6 (2024), 381–394. [51] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583.

[52] Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. 2023. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline. Advances in Neural Information Processing Systems 36 (2023), 65517–65530. [53] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210. [54] Yuan Zhou, Shaojie Xiang, Lingfan Yu, Zhenyu Song, Charith Mendis, and Yida Wang. 2026. SAS: Sparse Attention Synthesizer for Efficient Language Model Inference. In Proceedings of the 21st European Conference on Computer Systems. 1707–1721. [55] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. 2025. NanoFlow: Towards optimal large language model serving throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 749–765. [56] Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Xiao Chuanfu, Dahua Lin, and Chao Yang. 2025. Sampleattention: Near-lossless acceleration of long context llm inference with adaptive structured sparse attention. Proceedings of Machine Learning and Systems 7 (2025). [57] Jiaxiang Zou, Yonghao Chen, Xingyu Chen, Chenxi Xu, and Xinyu Chen. 2025. AxCore: A Quantization-Aware Approximate GEMM Unit for LLM Inference. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 839–853.

Request-Adaptive Execution for Efficient Hybrid LLM Inference

Analyze & explore related research in Conceptio

Related documents