EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

E PI L O RA: F USING M ANY L O RA A DAPTERS INTO L OW-P RECISION GEMM S FOR S ERVING AND RL

arXiv:2611.03899v1 [cs.DC] 2 Nov 2026

Hieu Tran 1 Hong Yu 1 2

A BSTRACT Low-rank adapters (LoRA) let one base model serve many fine-tuned variants: multi-tenant engines keep a single copy of the base and batch the requests of many tenants, each with its own adapter. As base models move to 8-bit (FP8) and 4-bit formats, their general matrix multiplications (GEMMs) become cheaper, but the adapters do not: engines run them as separate small GPU kernels beside the GEMM, whose latency takes a growing share of every decoding step. An adapter projects its input down to a low rank and back up, and our key observation is that the GEMM needs the down-projection only in its epilogue, the final stage in which it writes its output. EpiLoRA, a plugin for the vLLM engine, therefore launches the down-projection first and the low-precision GEMM as a dependent kernel that streams its weights meanwhile and adds every tenant’s up-projection in its epilogue. On NVIDIA Hopper GPUs, EpiLoRA decodes for a single adapter with up to 24% lower time per token than vLLM’s fastest configuration, and faster than SGLang and TensorRT-LLM; with many tenants, its adapter kernels deliver more throughput than vLLM’s on most GPU–model pairs we tested. The same low-precision base also trains: a LoRA reinforcement-learning trainer that computes on the engine’s FP8 weights needs no 16-bit copy, so a 27B model trains on one 48 GB GPU, and an engine that then keeps activations in 16 bits removes most of the remaining mismatch between the model generating training data and the one trained. 1

I NTRODUCTION

Low-rank adapters (LoRA) (Hu et al., 2022) have become the standard way to specialize a large language model without copying it. An adapter adds a low-rank update BA to each frozen weight matrix W of the model, where A projects the layer’s input down to a small rank r (typically 16–128) and B projects it back up; together they hold a fraction of a percent of the model’s parameters. One base model can therefore carry thousands of fine-tunes. Multi-tenant serving engines exploit this by keeping a single copy of the base in GPU memory and batching the requests of many tenants, the users or applications that each bring their own adapter (Chen et al., 2024; Sheng et al., 2024). Adapters are also what LoRA reinforcement learning (RL) trains: LoRA matches full fine-tuning at a fraction of the memory (Schulman & Thinking Machines Lab, 2025), and several services now train adapters for many users on shared bases (Thinking Machines Lab, 2025; Mind Lab, 2026; Yu et al., 2026a). At the same time, base models are moving to low precision. NVIDIA’s Ada (e.g., L40S) and Hopper (H100 and H200) GPUs compute in 8-bit floating point (FP8) at twice their 1

University of Massachusetts Amherst, Amherst, MA, USA University of Massachusetts Lowell, Lowell, MA, USA. Correspondence to: Hieu Tran <hieutran@umass.edu>. 2

rate in the 16-bit bf16 format (Micikevicius et al., 2022), and Blackwell GPUs compute 4-bit formats such as NVFP4 at four times that rate (Rouhani et al., 2023; NVIDIA, 2025). Serving systems therefore quantize the base model’s general matrix multiplications (GEMMs) to these formats (Lin et al., 2024; 2025; Frantar et al., 2025), and RL frameworks increasingly generate their training samples, called rollouts, with FP8 inference engines (Liu et al., 2025; Xi et al., 2026; Chen et al., 2026a). Lower-precision bases make inference cheaper, but the cost of serving LoRA adapters remains largely unchanged in current engines. The reason is that the adapter path stays outside the lowprecision GEMM. Merging an adapter into a quantized base rounds the update onto the base’s coarse grid and can erase it (Miahi & Belilovsky, 2026; Li et al., 2026), so engines keep adapters separate and compute them in their own GPU kernels beside the base GEMM (Kwon et al., 2023; Chen et al., 2024). During decoding, when the engine generates one token per request at each step, this adapter path is a chain of small dependent kernels whose latency does not shrink with the base. The cheaper the base, the larger the adapter’s share of each step (Figure 2). Existing systems address other parts of the problem. Punica and S-LoRA made batching many adapters practical (Chen et al., 2024; Sheng et al., 2024), and later systems decide where adapters are stored and which tenants share a

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL (a) one stream: the adapter’s kernels follow the GEMM stream q

FP8 GEMM: stream W

shrink expand

the adapter’s latency (b) two streams: the LoRA branch beside the GEMM, added after it stream 1 q

FP8 GEMM: stream W

stream 2 shrink expand

add

join

join + add (c) EpiLoRA’s decode path: h is needed only in the epilogue h ready kernel 1 q shrink kernel 2

K2: stream W , prefetch Bℓ +hB the adapter’s cost

Figure 1. The GEMM needs the adapter’s down-projection only in its epilogue. One projection during decoding; a CUDA stream runs its kernels (GPU functions) in order, and two streams can run at once. (a) vLLM runs the down-projection (shrink, h = xA⊤ ) and the up-projection (expand, y += hB ⊤ ) after the FP8 GEMM; (b) its dual-stream mode and TensorRT-LLM run them on a second stream and add the result after the GEMM; (c) EpiLoRA launches the shrink first and the GEMM as a dependent kernel, which starts while the shrink runs and adds hB ⊤ in its epilogue.

et al., 2025). Because the FP8 engine then samples from a slightly different model than the trainer optimizes, FP8 rollout pipelines reweight each sample with an importance weight (Liu et al., 2025). EpiLoRA’s trainer instead computes on the engine’s own FP8 weights, which saves enough memory to train a 27B model on one 48 GB L40S, where its bf16 weights alone do not fit, and lets one copy serve both the engine and the trainer. A matched engine also keeps its activations in bf16, as the trainer does, which removes most of the mismatch that remains once the weights agree. We evaluate EpiLoRA on models from 8B to 72B parameters on Ada, Hopper and Blackwell GPUs, against the strongest configuration of each baseline we could build. Serving a single request (batch size 1), EpiLoRA takes 0.76–0.88× the time per output token of vLLM’s fastest configuration for 8B–9B models on Hopper GPUs, is faster than SGLang and TensorRT-LLM, and keeps its lead on a 72B model, on one GPU and across two. On the L40S it matches vLLM’s fastest configuration. With 32 tenants, whose batches are too large for the decode path, it delivers 1.05–1.09× vLLM’s throughput on 7 of 9 GPU–model pairs, from its adapter kernels on Hopper GPUs and fused adapters on the L40S. In summary, our contributions are:

batch (Wu et al., 2024; Li et al., 2024a; Chen et al., 2026b). SVDQuant fuses one low-rank branch, shared by every token, into a 4-bit GEMM (Li et al., 2025), and LoRAFusion fuses several adapters into a bf16 GEMM for finetuning (Zhu et al., 2026). However, none computes many tenants’ adapters in a low-precision GEMM for serving. We present EpiLoRA, a plugin for the vLLM engine that computes every tenant’s adapter together with the low-precision base GEMM. A GEMM kernel streams the weights through its main loop and applies scales and writes its output in its epilogue. Our key observation is that the GEMM needs an adapter’s down-projection only in that epilogue, after the main loop has finished (Figure 1). During decoding, EpiLoRA therefore launches the down-projection first and its own FP8 or NVFP4 GEMM as a dependent kernel: the GEMM streams its weights while the downprojection runs, and then adds each tenant’s up-projection in its epilogue. The down-projection’s latency is then mostly hidden: on full-size layers with rank-64 adapters, at most about 2 µs of it stays exposed. On Ada GPUs, whose standard FP8 GEMM is slow during decoding, EpiLoRA instead fuses all resident adapters into two kernels of its own. The kernels quantize activations exactly as vLLM does, so the base GEMM receives the same FP8 inputs as in vLLM. The low-precision base also benefits RL training, where frameworks that place the inference engine and the trainer on the same GPUs keep a separate bf16 copy of the base for the trainer (Hu et al., 2024; Sheng et al., 2025; Mei

• The decode path, which removes most of an adapter’s latency by adding the up-projection in the GEMM’s epilogue and overlapping the down-projection with its main loop, and a simple condition that bounds when the overlap hides the down-projection (Sections 3.1 and 4.1). • Multi-adapter kernels for FP8 and NVFP4 bases in a drop-in vLLM plugin that works with CUDA graphs and tensor parallelism (Sections 3.2, 4.1 and 4.2). • A LoRA RL system on the same low-precision base, with a decomposition of the engine–trainer mismatch showing what each design choice removes (Sections 3.3 and 4.3).

2

BACKGROUND AND MOTIVATION

Why engines do not merge adapters. Merging an adapter into the base (W → W + BA) produces one model per adapter, so the engine can no longer batch tenants. On a lowprecision base, merging is also lossy: rounding W + BA to its grid changes only entries whose update exceeds half a quantization step, erasing small fine-tunes. The cost of the separate adapter path. Each decoding step runs every linear layer (projection) of the model over the tokens of the current batch. Consider a projection over M tokens, each belonging to a request that uses one of L resident adapters. On an FP8 base, vLLM 0.28 runs four kernels for it (Chen et al., 2024): it quantizes the activations x to

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL (a) Existing systems W8A8

engine tenants

base GEMM

LoRA kernels bf16

̸= rollouts adapter

bf16

RL trainer

importance weights correct the gradient

FP8 base

bf16 base

the engine’s

the trainer’s own copy

(b) EpiLoRA W8A8 or W8A16

engine

base GEMM

Figure 2. Adapters cost relatively more on cheaper bases. Latency added at batch size 1 and throughput lost with vLLM’s LoRA path and default kernels, relative to the engine without LoRA: Qwen3-8B with bf16, FP8 and NVFP4 bases on a partition of a Blackwell GPU, one adapter of rank 16–128 or 32 of rank 64.

FP8, runs the FP8 GEMM, computes the down-projection h = xA⊤ ℓ for each adapter’s tokens (the shrink), and adds the up-projection hBℓ⊤ to the output (the expand). The activations are read twice and the output is written, read and written again, and the adapter weights are read by kernels that cannot share the GEMM’s work. None of these costs shrinks with the base. Tuning the LoRA kernels’ launch configurations, which we do for every vLLM baseline, reduces them, but with tuned kernels or vLLM’s dual-stream mode the adapter still adds 42–55% to a layer’s time at batch size 1 on Hopper GPUs (Table 1). On a Blackwell partition, one rank-64 adapter adds 16% to Qwen3-8B’s latency at batch size 1 with a bf16 base, 26% with an FP8 base and 43% with an NVFP4 base with vLLM’s default kernels (Figure 2). Programmatic dependent launch. A GPU kernel runs as many thread blocks, which the GPU schedules onto its streaming multiprocessors (SMs). On Hopper and Blackwell GPUs, programmatic dependent launch (PDL) lets a kernel’s successor start before it finishes: the dependent’s thread blocks start as soon as the producer signals that they may, and wait for its results only at an explicit wait instruction. Inference engines use PDL to hide the launch latency between consecutive kernels, placing the wait at the start of the dependent kernel, so that only its setup overlaps the producer. EpiLoRA moves the wait to the end of the GEMM, just before its epilogue, so that the GEMM’s entire main loop overlaps the adapter’s down-projection. LoRA RL on a low-precision engine. RL post-training alternates between generating rollouts with an inference engine and updating the model with a trainer, and it spends much of its time generating (Sheng et al., 2025; Fu et al., 2025). An FP8 inference engine and a bf16 trainer, however, produce slightly different token distributions because they compute at different precisions. Existing quantized rollout pipelines correct this mismatch with importance weights, the ratio of the trainer’s to the engine’s probability of each

tenants

W8A16

RL trainer

rollouts

fused or overlapped

adapter

fwd + bwd on FP8 codes; TIS, or none if matched

same FP8 codes, via one base FP8 or NVFP4 codes CUDA IPC or a copy

Figure 3. One low-precision base for serving and for RL. (a) Existing systems run adapters in kernels beside the FP8 GEMM, and an RL trainer keeps its own bf16 copy of the base, with importance weights correcting for the engine’s rounding. (b) EpiLoRA computes the adapters with the base GEMM, and its trainer computes on the engine’s FP8 weights, with rollouts from an FP8 engine with importance sampling or from a matched engine that needs no correction (W8A8: 8-bit weights and activations; W8A16: 16-bit activations); serving tenants from the copy RL trains works but does not pay (Appendix A).

sampled token (Liu et al., 2025; Qiu et al., 2026; Zhou et al., 2026). Placing engine and trainer on one GPU also duplicates the base in memory: two bf16 copies of a 27B model exceed an 80 GB GPU, and moving the trainer’s copy to host memory (offloading) copies it every step and pins it there. As LoRA RL updates only the adapter, Section 3.3 addresses both problems at their source.

3

D ESIGN

EpiLoRA has three parts: kernels that compute every adapter together with the low-precision base GEMM, a vLLM plugin that replaces vLLM’s LoRA layers, and an RL system on the plugin; Figure 3 compares it with existing systems. 3.1

Multi-adapter kernels

Fusing the adapters of many tenants into a low-precision GEMM is harder than fusing one low-rank branch that every token shares (Li et al., 2025), for three reasons. First, a GEMM divides its output into tiles, each computed by one thread block, and tokens of different adapters share a tile, so the epilogue must apply a different up-projection to each group of rows. Second, the kernels must quantize the activations exactly as the engine’s own GEMM does, so the base GEMM sees the same FP8 inputs. Third, during

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

decoding the adapter path is a chain of small dependent kernels whose latency, rather than its memory traffic, sets its cost. We address the first two with two kernels, K1 and K2, that fuse the adapters’ down- and up-projections into the base GEMM, and the third with the decode path, which overlaps the down-projection with the GEMM’s main loop.

result is added, the two streams must join after the GEMM, and a separate kernel (vLLM’s up-projection, TensorRTLLM’s add) then reads and writes the output again. Launching the GEMM as the down-projection’s dependent avoids this join: the GEMM itself adds the up-projection in its epilogue, so the output is written once and nothing follows.

K1 and K2. K1 (Figure 4b) processes the tokens grouped by adapter, using the batching metadata vLLM already maintains. It quantizes the layer’s activations once for the base GEMM, with the same per-token scale and rounding as vLLM, and computes each adapter’s down-projection hℓ = xA⊤ ℓ from the bf16 activations. Because a decode batch fills only a few dozen thread blocks, K1 splits the sum over the input dimension across several blocks (split-K) to use more of the GPU. K2 then runs the low-precision GEMM over the tokens in adapter order, so that each output tile meets only a few contiguous groups of adapters. Its epilogue applies the scales and, for each adapter group in the tile, adds hℓ Bℓ⊤ , reading only the tile’s slice of Bℓ . Adapters may also be stored in FP8 or NVFP4 and are dequantized in registers, which halves (FP8) or nearly quarters (NVFP4) the adapter bytes the epilogue reads, though 6 of 24 accuracy checks of stored adapters failed.

Portability. The decode path needs only PDL and a GEMM epilogue that can add a low-rank product for groups of rows, both available outside Triton, the GPU language of our kernels, so a CUTLASS-based engine could adopt it for any base format whose GEMM’s main loop streams its weights.

When fusion pays. The fused path removes vLLM’s separate adapter kernels but replaces the vendor’s GEMM with ours. This pays on Ada GPUs against their default FP8 GEMM, slow during decoding; on Hopper GPUs our GEMM only matches vLLM’s CUTLASS GEMM (Table 1), and the fused path runs the down-projection first. The decode path. During decoding, the base GEMM streams megabytes of weights, while the adapter is a chain of dependent steps: K2’s epilogue cannot add hℓ Bℓ⊤ before the down-projection has finished. On the H100 this chain costs about 13 µs per projection with vLLM’s tuned kernels (Table 1), although reading Aℓ and Bℓ takes only about a microsecond. For decode batches of up to 32 tokens, EpiLoRA therefore launches the down-projection first as a small kernel of its own, without PDL so that it starts only after x is complete, and launches K2 as its dependent with PDL (Section 2; Figure 1c). A few extra thread blocks of this kernel, or for some shapes K2 itself, quantize the activations exactly as vLLM does. K2 starts as soon as the down-projection has begun, prefetches Bℓ , streams its weights once the activations are quantized, and waits for h only before its epilogue. Reductions are summed in a fixed order, so results are deterministic for a given batch. The same design runs on Blackwell with NVFP4 bases. Why not a second stream? vLLM’s dual-stream mode and TensorRT-LLM also overlap the adapter with the GEMM, by running the adapter on a second CUDA stream, a kernel queue that the GPU can run concurrently (Figure 1b). Because their GEMM writes its output before the adapter’s

When the overlap hides the adapter. With the decode path, a projection’s adapter cost is approximately tepi + max(0, tshrink − tmain ): the epilogue’s extra work plus whatever part of the down-projection outlasts K2’s main loop. At decode, tmain is the time to stream the projection’s weights, which falls with the GPU’s memory bandwidth and when tensor parallelism splits each layer’s weights across GPUs, while tshrink , the latency of a small kernel, hardly changes, so the condition is easiest to meet for large projections on GPUs of modest bandwidth; Figure 5 measures where it holds. Where the down-projection lasts most of the main loop, the two kernels also compete for SMs, and a few microseconds stay exposed (Section 4.1). The condition therefore bounds the exposure from below: when the GEMM launches more thread blocks than the GPU has SMs, only its first blocks start before the down-projection ends, so less than the whole main loop can hide it. 3.2

The vLLM plugin

EpiLoRA installs as a vLLM plugin that replaces vLLM’s LoRA linear layers; any layer it cannot serve falls back to vLLM. The kernels read vLLM’s own adapter buffers and batching metadata, so adapters are never copied, and the fused call is a registered custom operator that vLLM can capture in its CUDA graphs, which record a step’s kernel launches once and replay them to avoid launch overhead. Timed dispatch. When the model loads, the plugin times up to three paths for each projection shape and batch size and keeps the fastest: the fused kernels, run as the decode path on Hopper GPUs; the hybrid path, the vendor GEMM followed by our adapter kernels; and the vendor GEMM followed by vLLM’s own adapter kernels. On Hopper GPUs it selects the decode path for decode batches of 8B–9B models. For larger batches, including prefill, which processes a request’s prompt, it selects the vendor GEMM with EpiLoRA’s or vLLM’s adapter kernels, whichever is faster. On the L40S it selects the fused path (Table 1). Memory. vLLM sizes its KV cache, which stores every request’s attention keys and values, from the memory left after the model loads, so the plugin shares scratch buffers across

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Wq FP8

(a) vLLM: four kernels per projection 1

quantize, per token: x → xq , sx

3

LoRA shrink: h = xA⊤ ℓ

xq , s x

2

FP8 GEMM: y = (xq Wq⊤ ) sx sw

4

LoRA expand: y += hBℓ⊤

x bf16

h

y written, read, bf16 written again

read twice

Aℓ

Bℓ

bf16 adapters, read by kernels outside the GEMM

Wq FP8 or NVFP4

(b) EpiLoRA, fused path: two kernels per projection all tokens: quantize x → xq , sx x bf16

mainloop over adapter-sorted rows: xq Wq⊤

xq , s x

K1

K2

each adapter’s tokens: hℓ = xA⊤ ℓ

h

epilogue: × sx sw , then + hℓ Bℓ⊤ per segment

y bf16

written once

read by K1

Aℓ

adapters:

bf16, or FP8 / NVFP4 decoded in registers

Bℓ

only the tile’s slice, read in the epilogue

Figure 4. Two kernels per projection instead of four. Boxes are tensors in GPU memory, arrows reads and writes. (a) vLLM 0.28 on an FP8 base, in launch order. (b) EpiLoRA’s fused path: K1 quantizes the activations once, exactly as vLLM does, and computes each adapter’s down-projection; K2 runs the FP8 or NVFP4 GEMM on tokens sorted by adapter and adds each up-projection in its epilogue.

layers, reuses vLLM’s weight codes on NVFP4 checkpoints and stores low-precision adapters in smaller slots; with bf16 adapters, its load-time warm-up fragments memory and leaves 0.85× vLLM’s cache on the L40S (Figure 12). Implementation and correctness. EpiLoRA is about 6,500 lines of Python and Triton, including 48 Triton kernels and the RL trainer’s kernels, and needs no changes to vLLM. Under tensor parallelism it shards the adapters as vLLM does: projections whose outputs are split across GPUs split the up-projection’s output, and projections whose inputs are split split the down-projection’s input. We test correctness at three levels: EpiLoRA’s FP8 and NVFP4 quantization reproduces vLLM’s codes and scales bit for bit, its kernels match vLLM’s LoRA path and are bitwise reproducible, and served adapters keep vLLM’s accuracy (Figure 8). 3.3

LoRA RL on the same base

The RL system places a trainer and a vLLM rollout engine on one GPU and trains with GRPO, a policy-gradient algorithm that compares groups of sampled responses to the same prompt (Shao et al., 2024; Sheng et al., 2025). After each optimizer step the trainer publishes only its adapter, which the engine loads into an adapter slot. A trainer on the engine’s FP8 weights. A bf16 trainer differentiates a slightly different model than the FP8 engine samples from. Our FP8 trainer instead computes on the engine’s FP8 weights with bf16 activations, which removes the weight part of this mismatch and shrinks the frozen base weights from 16.7 to 10.2 GiB for a 9B model (Table 36). With one copy, the trainer maps the engine’s weights directly through CUDA’s inter-process memory sharing instead of loading its own, and the engine releases its KV cache while the trainer runs, so no weights move between phases. The

trainer’s projections run a training GEMM that reads the FP8 weights in place and adds the LoRA update in its epilogue, as K2 does, so no 16-bit copy of the weights is ever made. Matched-precision rollouts. To first order, the engine’s and the trainer’s log-probabilities of a sampled token y differ by three terms, from rounding the weights, rounding the activations, and differences between the two sides’ kernels: log πeng (y) − log πtr (y) ≈ δW + δA + δK .

(1)

The FP8 trainer removes δW , but not δA , the engine’s rounding of each GEMM’s input activations. Importance corrections compensate for this difference in the gradient (Liu et al., 2025); we remove it in the arithmetic instead. In the matched engine, every FP8 layer computes with 8-bit weights and 16-bit activations (W8A16) instead of rounding both to 8 bits (W8A8) as vLLM’s FP8 engine does, using the trainer’s arithmetic on the same weights: FP8 weights widened to bf16 in registers, bf16 activations and fp32 accumulation. vLLM’s weight-only FP8 path, Marlin (Frantar et al., 2025), also leaves activations unrounded but repacks the weights into a layout the trainer cannot share.

4

E VALUATION

We ask what the kernels save (Section 4.1), what that buys end to end (Section 4.2), and whether the same base trains as well as bf16 with less memory (Section 4.3). Setup. We run vLLM 0.28 with CUDA graphs and PyTorch’s compiler, kernels cached before we measure. We serve Qwen3-8B (Qwen Team, 2025), Llama-3.1-8B (Llama Team, AI @ Meta, 2024), Qwen3.5-9B (Qwen Team, 2026a), which replaces most attention layers with linear attention, and Qwen2.5-72B, with FP8 bases that vLLM

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 1. Where one adapter’s batch-1 cost goes. One Qwen3-8B decoder layer (four projections) at one decode token, in microseconds; adapter rows give each system’s cost over its own base GEMM, against vLLM’s fastest path on each GPU (Marlin on the L40S, CUTLASS elsewhere). L2-flushed kernel times only, so they do not add up to Table 2’s time per token. µs ↓

L40S

H100

H200 Blackwell

Base GEMM, vLLM 413 126 104 Base GEMM, EpiLoRA 416 121 102 Adapter, vLLM 97 53 57 Adapter, EpiLoRA 46 12 10 EpiLoRA’s path fused overlap overlap Layer, EpiLoRA / vLLM ↓ 0.91× 0.75× 0.70×

286 276 54 17 overlap 0.86×

quantizes when the model loads, and our NVFP4 checkpoint of Qwen3-8B; RL trains Qwen3.5-9B and Qwen3.827B (Qwen Team, 2026b). The GPUs are the L40S (Ada), the H100, H200 and GH200 (Hopper), and a 48 GB partition of an RTX PRO 6000 (Blackwell), which we call the Blackwell slice. For serving, the baselines are vLLM’s LoRA path on its fastest base for each GPU, with LoRA kernels we tuned for each model, SGLang (Zheng et al., 2024) and TensorRT-LLM (NVIDIA, 2026). For RL, they are FlashRL’s FP8 engine beside a bf16 trainer with truncated importance sampling (TIS), which multiplies each token’s gradient by its importance weight capped at 2 (Liu et al., 2025), bf16 training with the trainer’s weights offloaded during rollouts (Sheng et al., 2025), and one bf16 copy shared by engine and trainer, as in Unsloth (Unsloth AI, 2025b) but keeping its KV cache. We measure latency as the time per output token (TPOT) of a single request, and throughput with 256 requests of 512 prompt and 256 output tokens, using random adapters of rank 16–128, since speed depends only on the adapters’ shapes. Each engine is started four times and we report the mean of the last three unless a table says otherwise; the appendices give every measurement. Kernel timings use CUDA graphs with the L2 cache flushed; unless noted, adapters have rank 64, and bold in tables marks the best value, with ties within 2%. 4.1

Figure 5. What stays exposed of the down-projection under the decode path. One point per projection: Qwen3-8B layers at 1– 16 decode tokens with ranks 16–128, their tensor-parallel shards and two smaller models, on four GPUs. The x axis subtracts the GEMM’s first-wave main loop, a window fitted after the data, from the down-projection’s time; dashed: the condition of Section 3.1.

on Blackwell the decode path cuts it from 54 to 17 µs. What stays exposed follows the overlap condition (Figure 5). The measured exposure tracks the condition with a window we fitted after the data: the down-projection stays hidden while it finishes within the main loop of the GEMM’s first wave of thread blocks, when the first epilogues need it. With a rank-64 adapter on full-size layers, the downprojection ends within the GEMM’s main loop on every projection we measured, and at most 2.1 µs of it remains exposed. At ranks 16 and 128 and on tensor-parallel shards, up to 9 µs per projection stays exposed, mostly where the GEMM launches more blocks than the GPU has SMs. The lead holds from 1 to 16 decode tokens (Table 4). With the decode path, one adapter adds 4–9% to the layer on Hopper GPUs, against 35–55% with vLLM’s tuned kernels or dual-stream mode, because the down-projection finishes within the GEMM’s main loop (Figure 5); run serially, the layer takes 1.13–1.31× as long, depending on the downprojection kernel (Tables 11, 12), so at one decode token the overlap accounts for a third to two thirds of the saving.

Kernels

The adapter’s cost falls by 2–6× on every GPU (Table 1). On Hopper GPUs, one rank-64 adapter costs 53–57 µs per layer with vLLM’s tuned kernels and 10–12 µs with the decode path, so the layer takes 0.70–0.75× vLLM’s time. On the H100, nine tenths of the saving is in the adapter and a tenth in EpiLoRA’s own GEMM. The overlap pays because the GEMM needs the down-projection only in its epilogue, so it streams its weights while the adapter is computed; vLLM’s dual-stream mode overlaps the adapter only with the GEMM as a whole (Figure 1b). On the L40S, fusing adapters into EpiLoRA’s own GEMM halves their cost, and

4.2

End-to-end serving

At batch 1, EpiLoRA is faster than every system we ran on Hopper GPUs (Tables 2 and 3). For 8B–9B models it takes 0.76–0.88× the time per token of vLLM’s fastest configuration, including Llama-3.1-8B with both systems’ kernels tuned for its shapes. SGLang and TensorRT-LLM take 1.4–1.8× EpiLoRA’s time on Qwen3-8B. Profiles show why (Table 3): per token on the H100, their adapter kernels add 2.9–3.0 ms of GPU time, vLLM’s dual-stream mode 1.9 and EpiLoRA’s 0.4, leaving EpiLoRA 9–14% slower than without LoRA (Table 18).

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 2. EpiLoRA against vLLM’s fastest configuration. Time per output token with one adapter at batch 1, and throughput with 32 tenants; vLLM runs its fastest base (NVFP4 on the Blackwell slice, FP8 elsewhere) with LoRA kernels tuned for each model (Qwen3.5-9B’s on the H100 for all its Hopper rows). On 2×H100, both systems use tiles searched for the shards; the H200 72B row comes from an earlier job than Table 29, which gives 0.94×. GPU

Model

vLLM EpiLoRA

One adapter at batch 1, ms per token ↓ H100 Qwen3.5-9B 6.26 5.14 H100 Qwen3-8B 6.02 4.81 H100 Llama-3.1-8B 5.59 4.76 H200 Qwen3.5-9B 6.11 4.81 H200 Qwen3-8B 5.94 4.51 GH200 Qwen3.5-9B 6.70 5.91 H200 Qwen2.5-72B 28.00 26.80 2×H100 Qwen2.5-72B 20.31 17.75 L40S Qwen3.5-9B 15.39 14.97 L40S Qwen3-8B 14.05 13.95 Blackwell Qwen3-8B 10.27 9.46 32 tenants, thousand tokens per second ↑ H100 Qwen3.5-9B 5.30 5.64 H100 Qwen3-8B 5.89 6.22 H100 Llama-3.1-8B 6.26 6.32 H200 Qwen3.5-9B 5.29 5.78 H200 Qwen3-8B 5.83 6.16 GH200 Qwen3.5-9B 5.57 6.00 L40S Qwen3.5-9B 1.47 1.59 L40S Qwen3-8B 1.63 1.76 Blackwell Qwen3-8B 2.08 2.11

Ratio 0.82× 0.80× 0.85× 0.79× 0.76× 0.88× 0.96× 0.87× 0.97× 0.99× 0.92× 1.06× 1.05× 1.01× 1.09× 1.06× 1.08× 1.08× 1.08× 1.02×

The lead is largest on the H200, whose faster memory shortens the base GEMM so the adapter is a larger share of each step, and smallest on the GH200, where both systems run slower than on the H100. On the L40S, where vLLM is fastest with Marlin’s weight-only GEMM, EpiLoRA’s weight-only mode (bf16 activations) matches it; on the Blackwell slice’s NVFP4 base, it takes 0.92× vLLM’s time. At batch 1, the lead extends to a 72B model (Table 2). A larger model leaves less to save, as its GEMMs grow with the square of the hidden size and the adapter linearly: Qwen2.5-72B takes 0.96× the time per token of tuned vLLM on one H200 and 0.87× across two H100s. With many tenants, EpiLoRA keeps more of the base’s throughput (Table 2). With 32 rank-64 tenants it delivers 1.05–1.09× the throughput of vLLM’s best configuration on 7 of the 9 GPU–model pairs. These batches exceed the 32 tokens the decode path covers, so the gain comes from EpiLoRA’s adapter kernels after the vendor GEMM: on the H200, swapping in vLLM’s turns 1.10× into 0.97× in one job (Table 14). They beat vLLM’s on projections with three or four outputs of unequal widths, such as attention’s query, key and value, where vLLM’s up-projection costs 1.7× as much per output column (Table 15); at 32 tokens,

Table 3. EpiLoRA against three serving systems on Qwen3-8B, each pair measured in one job. Time per output token at batch 1 and throughput with 32 tenants, relative to EpiLoRA’s, each system in its fastest LoRA configuration (TRT-LLM: TensorRTLLM’s PyTorch backend). a At its fastest KV-cache share, 0.6. b Without LoRA, 1.09–1.14× vLLM’s batch-1 time and 0.95× its throughput; H200: one start. c From NVIDIA’s Nsight Systems profiler (Table 21); vLLM in dual-stream mode with its default kernels. EpiLoRA vLLM SGLangb TRT-LLM Batch 1: time per token relative to EpiLoRA’s ↓ H100 1.00× 1.25× 1.65× 1.43× H200 1.00× 1.32× 1.81× 1.52× Batch 1: GPU time the adapter adds per token (ms)c ↓ H100 0.40 1.87 3.01 2.87 32 tenants: throughput relative to EpiLoRA’s ↑ H100 1.00× 0.95× 0.94× 0.97×a H200 1.00× 0.95× 0.93× 1.05×

Table 4. From 1 to 16 decode tokens, the decode path keeps one adapter’s cost to a few percent of the layer. One Qwen38B decoder layer, in microseconds; cost: the time over the layer without an adapter (vLLM: the better of its tuned kernels and dual-stream mode). µs ↓ No vLLM EpiLoRA Cost (%) ↓ Tokens adapter tuned dual serial overlap vLLM EpiLoRA H100, FP8 base 1 126 4 128 16 128 H200, FP8 base 1 104 4 106 16 106

179 180 179 172 180 174

163 162 167

134 133 136

42 35 36

6 4 7

161 167 162 157 162 159

144 147 150

112 112 115

55 48 49

8 6 9

the lead per layer shrinks as a batch mixes more distinct adapters (Figure 11). On the L40S the gain comes from fusing adapters into EpiLoRA’s GEMM; TensorRT-LLM beats EpiLoRA on the H200 (Table 3). Online, EpiLoRA lowers the time per output token at most loads (Figure 6). When CUDA graphs cover the prefill sizes, its median time per output token with 32 tenants is 0.77–0.95× vLLM’s on the H100 and 0.92–0.99× on the L40S, and its 99th-percentile (p99) time to first token (TTFT) stays within 1.1× of vLLM’s at 6 of these 8 GPU– rate points. On a WildChat trace, EpiLoRA takes 0.70– 0.86× vLLM’s time per output token on the H200 and 0.77–0.98× on the H100 at vLLM’s defaults, and serves at least vLLM’s goodput at every rate (Table 5). Its lead is largest at low load, where batches fit the decode path. At vLLM’s default graph sizes (dotted), prefills longer than 512 tokens run outside CUDA graphs, where every kernel

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

TPOT ratio

1.1

vLLM, tuned (same graphs)

EpiLoRA, default graphs

vLLM, default vs. 2,048 graphs

EpiLoRA, graphs to 2,048

(a) L40S

(b) H100 1.2

1.0

1.0 0.8

p99 TTFT ratio

0.9 3

3

1

1

0.3

0.3

0.1

0.1 1

2

4

6

2

4

8

16

requests per second (32 rank-64 tenants)

Figure 6. Online, EpiLoRA lowers the time per output token at most loads. Median time per output token (top) and p99 time to first token (bottom, log scale) with 32 tenants, relative to vLLM with tuned kernels. Solid: CUDA graphs captured up to 2,048 tokens; dotted: vLLM’s default capture, up to 512. Qwen3.5-9B, Poisson arrivals, 512 input and 256 output tokens; bars: range over three starts.

Table 5. On a real trace, EpiLoRA lowers the time per output token at every load. Per start, 1,000 of 3,383 first turns of WildChat conversations (short prompts: median 23 tokens) with gamma arrivals (burstiness 0.5), 32 tenants, Qwen3.5-9B at vLLM’s default CUDA-graph settings; goodput counts requests meeting a 2 s TTFT and a 50 ms TPOT. a One of vLLM’s three starts took 3.7 s (the others 0.19–0.20 s), so neither value is bold. Median TPOT (ms) ↓ p99 TTFT (s) ↓ Goodput (req/s) ↑ Req/s vLLM H200 2 4 8 16 H100 2 4 8 16

EpiLoRA vLLM EpiLoRA vLLM EpiLoRA

9.1 12.8 18.0 30.6

7.0 8.9 14.6 26.4

0.09 0.10 0.12 0.16

0.09 1.98 0.09 3.88 0.11 7.26 0.16 12.08

1.99 3.92 7.43 12.70

9.9 14.3 21.8 33.1

7.6 0.16 11.1 0.18 19.0 1.37a 32.4 0.24

0.18 1.98 0.19 3.87 0.22 7.15 0.27 12.00

1.99 3.91 7.35 12.38

launch costs host time: EpiLoRA then trails vLLM slightly on the L40S at 4–6 requests/s and by 1.18× on the H100 at 16 requests/s, and its p99 TTFT is 1.1–1.2× vLLM’s on the H100; Tables 24 and 25 add the H200 and long prompts. 4.3

LoRA RL on the same base

GRPO (Shao et al., 2024) trains a rank-64 adapter on competition math (DAPO-Math (Yu et al., 2025)) with engine

Figure 7. With the weights matched, rounded activations cause most of the remaining mismatch. Per-token KL from engine to trainer relative to a bf16 engine and trainer, five 9B adapters on the H100 (bars: geometric mean; whiskers: range); the trainer computes on the engine’s FP8 weights unless marked bf16, and hatched bars change one component of the bar above. Earlier, smaller decompositions measured up to 2.1× for the unrounded engines.

and trainer on one GPU: Qwen3.5-9B for 200 steps of 16 problems and Qwen3.8-27B for 100 steps of 8 problems, 8 samples each, with samples and answers capped at 2,048 tokens. We report accuracy on OlympiadBench (He et al., 2024) and Minerva (Lewkowycz et al., 2022), with 3–9 seeds and 95% intervals (Tables 30, 31); RL lifts the 9B from 50.5% to 60–61% (Figure 8b). Matching the weights removes the largest source of mismatch, and leaving activations unrounded removes most of the rest (Figure 7). We measure the mismatch as the KL divergence between the engine’s and the trainer’s nexttoken distributions. Rounding the weights raises it the most, to 12–21× that of a bf16 engine and trainer, and the FP8 trainer removes this part by computing on the engine’s own weights. Almost all of the rest comes from rounding activations: engines that round them retain 6–14×, while our matched engine and vLLM’s Marlin retain 1.0–1.4×, and the order holds on a dense second model (Table 33). On the 9B, corrected or matched FP8 RL ends where bf16 training does (Figure 8b). We call two configurations equivalent when the 95% interval of their difference lies within ±5 points, about half of RL’s gain. With nine seeds, W8A8 rollouts with TIS, FlashRL’s recipe and the matched engine without any correction end within 0.6 points of bf16 training, every interval within ±2.6 points (Table 32), while uncorrected W8A8 rollouts end 4.9 points below it; on an L40S, the matched engine with one copy that keeps its KV cache ended 3.4 points below bf16 training ([−5.7, −1.1]), but against two copies one copy moved it by −2.3 and W8A8 rollouts with TIS by +2.4 (Welch intervals spanning zero, paired ones excluding it in opposite directions), so the

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL accuracy (%)

(a) Serving: the same two adapters, served by vLLM or by EpiLoRA 100 75 50 25 0

90.2

90.3

vLLM's LoRA path 60.1

MATH-500

61.0

EpiLoRA

55.0

OlympiadBench

55.1

Minerva

(b) RL: accuracy after RL with 95% intervals (runs in parentheses) Qwen3.5-9B on one H100

before RL

bf16 training (9) FlashRL (FP8 rollouts, TIS) (9) EpiLoRA, FP8 trainer + TIS (9) EpiLoRA, matched, no correction (9) FP8 rollouts, no correction (6)

60.7 61.3 60.4 60.8 55.8

Qwen3.8-27B on one H100

before RL

bf16, base offloaded (3) FlashRL, bf16 offloaded (3) EpiLoRA, FP8, offloaded (3) EpiLoRA, one copy, KV released (3)

Qwen3.8-27B on one L40S

71.4 71.4 72.1 72.0

before RL

bf16 training, two A100s (3) EpiLoRA, FP8, offloaded (3)

50

61.7 60.8

60

Table 6. With long rollouts, one FP8 copy runs close to the fastest placement that fits, with a fraction of offloading’s host memory. Seconds per RL step and peak host memory (GiB) by placement of the trainer’s base, with rollouts of up to 8,192 tokens (12-step probes; details in Table 39); FlashRL’s placement runs in our trainer and engine; the bf16 copy keeps its KV cache, which Unsloth and EpiLoRA’s FP8 copy release while training; OOM: out of GPU memory. a Engine at 55% of the GPU; OOM at 70%. 9B, one L40S 27B, one H100 Placement of the trainer’s base One bf16 copy, KV kepta Two FP8 copies bf16, trainer offloaded FlashRL, trainer offloaded EpiLoRA, FP8 trainer offloaded EpiLoRA, one FP8 copy

s/step ↓ GiB ↓ s/step ↓ GiB ↓ 2,216 8 OOM 1,314 48 1,076 41 1,025 32 1,012 9

OOM OOM 659 160 421 126 418 94 422 9

70

accuracy after RL (%)

Figure 8. EpiLoRA keeps the accuracy of vLLM’s LoRA path in serving and, with nine seeds on the 9B, of bf16 training in RL. (a) Two RL adapters of Qwen3.5-9B served by vLLM and by EpiLoRA (dots: each adapter; differences −1.0 to +1.6 points, McNemar p ≥ 0.27). (b) Accuracy after RL on OlympiadBench and Minerva (the 27B on the H100 also MATH-500), with 95% intervals over runs; bf16 training of the 27B does not fit one L40S, so its reference uses two A100s.

one-copy effect is inconclusive (Appendix A). The matched engine needs no correction because it leaves the mismatch at the bf16 pair’s level (Figure 7) and keeps it there throughout training (Table 34). On the 27B, EpiLoRA’s configurations end at 72.0–72.1% (with MATH-500) against 71.4% for bf16 training and FlashRL, every interval within ±4 points with three seeds (Table 31). An FP8 trainer fits the 27B on one 48 GB GPU. Training on the engine’s FP8 weights saves 6.5 GiB for the 9B and 22.7 GiB for the 27B (Table 36). So the 27B trains on one 48 GB L40S, which its bf16 weights (50 GiB) exceed: with the trainer’s copy offloaded, EpiLoRA reaches 60.8%, against 61.7% for bf16 training on two A100s (Figure 8b). With long rollouts, one FP8 copy comes close to the fastest placement with a fraction of its host memory (Tables 6 and 7). With rollouts of up to 8,192 tokens, two FP8 copies fit neither model, and one bf16 copy that keeps its KV cache fits only the 9B, at 2.2× the step time (1.9× that of an FP8 copy that also keeps it, Table 38); Unsloth releases that cache while training, a variant we did not run (Appendix A). One FP8 copy then takes 0.94–1.00× the time of FlashRL’s offloaded recipe and is 1.3–1.6× as fast as bf16 training with offloading, while pinning 9 GiB of host memory instead of 32–160 GiB; in full runs of the 27B

Table 7. In full runs, one FP8 copy trains the 27B in 0.69× bf16 offloading’s GPU-hours with a fraction of the host memory. Single 100-step RL runs (two for one copy, which releases its KV cache) with rollouts of up to 8,192 tokens on one H100; per M tok.: GPU-hours per million trained tokens; Gen.: generation seconds per step; Host: peak host memory (GiB); Tokens: trained tokens per step. These runs compare cost: at our evaluation they change accuracy little (Table 38). GPU-h ↓ Placement

run per M tok. Gen. s ↓ Host ↓ Tokens

bf16, offloaded 14.67 FlashRL, offloaded 9.84 EpiLoRA, FP8 offloaded 9.56 EpiLoRA, one FP8 copy 10.05

0.91 0.60 0.62 0.61

434 247 245 257

160 127 95 10

162k 164k 155k 165k

it takes 1.02–1.05× the GPU-hours of the offloaded FP8 recipes, or 0.98–1.02× per trained token (Table 7). The difference is in generation: FP8 weights take half the memory of bf16 weights and leave the engine a KV cache almost three times as large (Figure 9), and with long rollouts the cache sets how many sequences decode at once; FlashRL’s FP8 engine gets the same cache. Where EpiLoRA does not pay (Appendix A): where two copies fit, FlashRL’s faster bf16 trainer makes its recipe slightly cheaper; at 2,048 tokens a bf16 copy that keeps its KV cache ties one FP8 copy on the 9B; a trainer rounding activations like the engine trains 1.2× faster at a larger mismatch; and the matched engine takes 1.35× (vLLM’s Marlin, also unrounded, 1.26×) the GPU-hours of corrected W8A8 rollouts on the H100.

5

R ELATED WORK

Multi-adapter serving. Other systems accelerate perrequest and per-token adapters (Wen & Chaudhuri, 2024; Kong et al., 2024), tile adapters of mixed ranks (Mi et al., 2025), multiply each group of rows by its own matrix (Ye

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL During rollouts FlashRL, two copies

E

T (bf16)

Two FP8 copies

E

T

One bf16 copy, KV kept

W (bf16)

bf16, offloaded

E (bf16)

FlashRL, offloaded

E

FP8, offloaded

E

One FP8 copy, KV kept

W

One FP8 copy, KV released

W

During training

Host –

does not fit E

T

A

– OOM

KV

W (bf16)

KV A

– OOM

KV

T (bf16)

A

159 GiB

T (bf16)

A

125 GiB

KV KV KV KV 80 GB

T

A

W

KV

W

A

93 GiB 10 GiB

A

8 GiB 80 GB

Figure 9. Releasing its KV cache, one FP8 copy keeps offloading’s cache without pinning the trainer’s copy in host memory. GPU memory of one Qwen3.8-27B RL step on one 80 GB H100 with rollouts of up to 8,192 tokens, to scale: E, T and W are the engine’s, the trainer’s and a shared copy of the base (FP8 unless marked bf16), KV the engine’s cache and A the trainer’s activations; right: median host memory pinned while training.

et al., 2025), or cache, pool, compress and co-quantize adapters or share KV caches across them (Xia et al., 2024; Iliakopoulou et al., 2025; Yao et al., 2025; Zhang et al., 2025; Brüel-Gabrielsson et al., 2025; Wang et al., 2026a; Yu et al., 2026b). All of them compute adapters beside the base GEMM (Table 9); EpiLoRA can be their kernel layer. Low-precision serving. AWQ and QServe quantize weights to 4 bits with 16- or 8-bit activations (Lin et al., 2024; 2025), Marlin provides fast mixed-precision GEMMs (Frantar et al., 2025), and Hopper and Blackwell GPUs support FP8 and block-scaled 4-bit formats natively (Micikevicius et al., 2022; Rouhani et al., 2023; NVIDIA, 2025); EpiLoRA keeps each engine’s quantization bit for bit and changes only how adapters are computed around the GEMM. Fusing low-rank branches into GEMMs. SVDQuant’s Nunchaku fuses one low-rank branch, shared by every token, into the 4-bit GEMMs of diffusion models (Li et al., 2025). For fine-tuning, LoRAFusion fuses the up-projection into the bf16 base GEMM and routes the tiles of a mixedadapter batch to their adapters (Zhu et al., 2026). K2 does the same for serving over FP8 and NVFP4 GEMMs, gathering rows sorted by adapter as mixture-of-experts GEMMs gather rows sorted by expert (Gale et al., 2023). The decode path applies a known principle, overlapping a producer with its consumer’s main loop, as systems that overlap a GEMM with communication between GPUs do tile by tile (Wang et al., 2023; Pati et al., 2024; Chang et al., 2024) and NanoFlow does across a batch’s operations (Zhu et al., 2024). Unlike weight prefetches issued before a dependent kernel’s wait, our producer is a side branch the GEMM needs only in its epilogue, so its whole main loop overlaps it; LoRAFusion needs the down-projection before its GEMM starts, and two-stream modes add the adapter’s output after the GEMM ends. Compressing adapters (Yadav et al., 2023; Ping et al., 2024; Liu et al., 2024; Mirzaei et al., 2025)

and training them for low-precision merges (Dettmers et al., 2023; Li et al., 2024b; Guo et al., 2024; Xu et al., 2024; Li et al., 2026) are complementary: they change the adapters. Low-precision RL. FlashRL, our RL baseline, pairs FP8 rollouts with truncated importance sampling beside a bf16 trainer (Liu et al., 2025). Quantized rollouts also speed up full-weight RL, which must re-quantize the weights at every step (Hu et al., 2026; Miahi & Belilovsky, 2026); other systems train and generate at one low precision, rounding activations on both sides, which leaves a mismatch unless both sides’ kernels agree bit for bit (Xi et al., 2026; Qiu et al., 2026; Chen et al., 2026a; Gu et al., 2026), correct the rounding of 4-bit rollouts (Zhuge et al., 2026; Wang et al., 2026b), or sample from 4-bit weight-only kernels (Huang et al., 2025). Adaptive importance sampling tunes the correction (Zhou et al., 2026) and score centering removes the drift a mismatched sampler adds to the policy gradient (Marek & Ryabinin, 2026); the matched engine instead removes most of the mismatch itself. FlexLLM co-serves inference and fine-tuning on one bf16 base (Miao et al., 2024), batch-invariant kernels make engine and trainer agree bit for bit (He & Thinking Machines Lab, 2025; vLLM Team, 2025), Unsloth shares one 4-bit or FP8 copy of the base between engine and trainer and releases its KV cache while training (Unsloth AI, 2025b;a;c), and a TRL patch shares FP8 weights with vLLM (ingambe, 2026). QLoRA first trained adapters on a frozen quantized base (Dettmers et al., 2023). EpiLoRA trains on the engine’s FP8 codes with bf16 activations, and its matched engine keeps them unrounded, as Marlin and QeRL do, in the trainer’s layout.

6

C ONCLUSION

Adapters have stayed outside the low-precision GEMM, so a cheaper base makes them relatively costlier to serve. EpiLoRA, a drop-in vLLM plugin, brings them back into the GEMM: on Ada GPUs it fuses every tenant’s adapter into its own low-precision GEMM, and on Hopper and Blackwell GPUs it launches the down-projection first and lets the GEMM stream its weights while the adapter is computed, waiting for it only in the epilogue. This change makes one adapter at batch 1 faster than vLLM, SGLang and TensorRTLLM on Hopper GPUs, and with many tenants its adapter kernels keep more of the base’s throughput than vLLM’s on most GPU–model pairs we tested. The same low-precision weights also serve RL: a trainer that computes on them needs no bf16 copy of the model, so a 27B model trains on one 48 GB GPU, and an engine that leaves activations unrounded needs no importance correction. Served adapters keep the accuracy of vLLM’s LoRA path, and on the 9B, corrected or matched FP8 RL ends where bf16 training does. Sharing one FP8 copy between engine and trainer saves most of a second copy’s memory, which matters most

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

for models that do not fit twice on a GPU; with long rollouts it nears the fastest placement with a fraction of its host memory. More broadly, a kernel can wait for its producer where it first needs the data, not where it starts.

Gu, H., Wang, H., Liu, J., Li, L., Zhu, Q., Liu, B., Xu, B., Wang, L., Yang, X., Lin, S., Han, S., and Guo, Y. QaRL: Rollout-aligned quantization-aware RL for fast and stable training under training–inference mismatch. arXiv preprint arXiv:2604.07853, 2026.

R EFERENCES

Guo, H., Greengard, P., Xing, E. P., and Kim, Y. LQLoRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In International Conference on Learning Representations (ICLR), 2024.

Brüel-Gabrielsson, R., Zhu, J., Bhardwaj, O., Choshen, L., Greenewald, K., Yurochkin, M., and Solomon, J. Compress then serve: Serving thousands of LoRA adapters with little overhead. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pp. 18062–18095, 2025. arXiv:2407.00066. Chang, L.-W., Bao, W., Hou, Q., Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, Z., Yao, C., Jiang, Z., Lin, H., Jin, X., and Liu, X. FLUX: Fast software-based communication overlap on GPUs through kernel fusion. arXiv preprint arXiv:2406.06858, 2024.

He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In ACL, pp. 3828–3850, 2024. He, H. and Thinking Machines Lab. Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism, 2025.

Chen, F., Jiang, Z., Wei, Z., Zhong, Z., Li, D., Zhang, C., Lin, H., and Venkataraman, S. Towards full pipeline FP8 reinforcement learning for LLMs. arXiv preprint arXiv:2609.22870, 2026a.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.

Chen, H., Ruan, L., Xu, Z., Li, Y., Chen, X., Leng, J., He, B., Guo, M., and Sun, S. InfiniLoRA: Disaggregated multi-LoRA serving for large language models. arXiv preprint arXiv:2604.07173, 2026b.

Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405.11143, 2024.

Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., and Krishnamurthy, A. Punica: Multi-tenant LoRA serving. In Proceedings of Machine Learning and Systems (MLSys), pp. 1–13, 2024. arXiv:2310.18547.

Hu, L., Zhao, R., Zhu, I., Zhang, Z., Zhang, H., Yin, H., and Zhao, J. SparseRL-Sync: Lossless weight synchronization with ∼100x less communication. arXiv preprint arXiv:2605.07330, 2026.

Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, pp. 10088–10115, 2023.

Huang, W., Ge, Y., Yang, S., Xiao, Y., Mao, H., Lin, Y., Ye, H., Liu, S., Cheung, K. C., Yin, H., Lu, Y., Qi, X., Han, S., and Chen, Y. QeRL: Beyond efficiency – quantization-enhanced reinforcement learning for LLMs. arXiv preprint arXiv:2510.11696, 2025.

Frantar, E., Castro, R. L., Chen, J., Hoefler, T., and Alistarh, D. MARLIN: Mixed-precision auto-regressive parallel inference on large language models. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP), pp. 239– 251, 2025. Fu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., Yang, T., Yuan, B., and Wu, Y. AReaL: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025. Gale, T., Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient sparse training with mixture-ofexperts. In Proceedings of Machine Learning and Systems (MLSys), pp. 288–304, 2023.

Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., and Torrellas, J. Chameleon: Adaptive caching and scheduling for many-adapter LLM inference environments. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 217–231, 2025. ingambe. FP8 training for GRPO: replay harness, FP8 recipes, LoRA over an FP8 base and FP8 weights shared with vLLM. GitHub pull request #119 to a fork of TRL, https://github.com/ingambe/trl/ pull/119, 2026. Kong, R., Li, Q., Fang, X., Feng, Q., He, Q., Dong, Y., Wang, W., Li, Y., Kong, L., and Liu, Y. LoRASwitch: Boosting the efficiency of dynamic LLM

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

adapters via system-algorithm co-design. arXiv preprint arXiv:2405.17741, 2024.

Llama Team, AI @ Meta. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023.

Marek, M. and Ryabinin, M. Score centering stabilizes off-policy reinforcement learning. arXiv preprint arXiv:2609.20807, 2026.

Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., GurAri, G., and Misra, V. Solving quantitative reasoning problems with language models. In NeurIPS, pp. 3843– 3857, 2022. Li, M., Lin, Y., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. SVDQuant: Absorbing outliers by low-rank components for 4-bit diffusion models. In International Conference on Learning Representations (ICLR), 2025. arXiv:2411.05007. Li, S., Lu, H., Wu, T., Yu, M., Weng, Q., Chen, X., Shan, Y., Yuan, B., and Wang, W. CaraServe: CPU-assisted and rank-aware LoRA serving for generative LLM inference. arXiv preprint arXiv:2401.11240, 2024a. Li, T.-L., Huang, J., Wang, L.-C., and Gotei, J. R. ScaleQLoRA: Code-invariant adapter merging for native 4-bit microscaling LLMs. arXiv preprint arXiv:2609.04526, 2026. Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. LoftQ: LoRA-fine-tuning-aware quantization for large language models. In International Conference on Learning Representations (ICLR), 2024b. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, pp. 87–100, 2024. Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Proceedings of Machine Learning and Systems, 2025. Liu, J., Xiao, G., Li, K., Lee, J. D., Han, S., Dao, T., and Cai, T. BitDelta: Your fine-tune may only be worth one bit. In Advances in Neural Information Processing Systems, pp. 13579–13600, 2024. Liu, L., Yao, F., Zhang, D., Dong, C., Shang, J., and Gao, J. FlashRL: 8-bit rollouts, full power RL. https:// fengyao.notion.site/flash-rl, 2025.

Mei, Z., Fu, W., Li, K., Wang, G., Zhang, H., and Wu, Y. ReaL: Efficient RLHF training of large language models with parameter reallocation. In Proceedings of Machine Learning and Systems (MLSys), 2025. Mi, L., Wang, W., Tu, W., He, Q., Kong, R., Fang, X., Dong, Y., Zhang, Y., Li, Y., Li, M., Dai, H., Chen, G., and Liu, Y. Empower vision applications with LoRA LMM. In Proceedings of the European Conference on Computer Systems (EuroSys), pp. 261–277, 2025. arXiv:2411.00915. Miahi, E. and Belilovsky, E. Understanding and exploiting weight update sparsity for communication-efficient distributed RL. arXiv preprint arXiv:2602.03839, 2026. Miao, X., Oliaro, G., Cheng, X., Wu, M., Unger, C., and Jia, Z. FlexLLM: A system for co-serving large language model inference and parameter-efficient finetuning. arXiv preprint arXiv:2402.18789, 2024. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S., Shoeybi, M., Siu, M., and Wu, H. FP8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022. Mind Lab. MinT: Managed infrastructure for training and serving millions of LLMs. arXiv preprint arXiv:2605.13779, 2026. Mirzaei, A. R., Wen, Y., Cao, Y., and Mou, L. LoRAQuant: Mixed-precision quantization of LoRA to ultra-low bits. arXiv preprint arXiv:2510.26690, 2025. NVIDIA. Pretraining large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025. NVIDIA. TensorRT-LLM. https://github.com/ NVIDIA/TensorRT-LLM, 2026. Pati, S., Aga, S., Islam, M., Jayasena, N., and Sinclair, M. D. T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 1146–1164, 2024. Ping, B., Wang, S., Wang, H., Han, X., Xu, Y., Yan, Y., Chen, Y., Chang, B., Liu, Z., and Sun, M. Delta-CoMe: Training-free delta-compression with mixed-precision for large language models. In Advances in Neural Information Processing Systems, pp. 31056–31077, 2024.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

Qiu, Z., Yu, S., Zhang, J., Zhang, S., Huang, X., Yang, J., and Lai, J. FP8-RL: A practical and stable low-precision stack for LLM reinforcement learning. arXiv preprint arXiv:2601.18150, 2026.

reinforcement-learning-rl-guide, 2025b. Accessed 2026-10-26.

Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

Unsloth AI. Memory efficient RL: Unsloth standby. https://docs.unsloth.ai/new/ memory-efficient-rl, 2025c. Accessed 2026-1024.

Qwen Team. Qwen3.5. https://huggingface.co/ Qwen/Qwen3.5-9B, 2026a.

vLLM Team. Bitwise-consistent on-policy reinforcement learning with vLLM and TorchTitan. vLLM blog, 2025.

Qwen Team. Qwen3.8. https://huggingface.co/ Qwen/Qwen3.8-27B, 2026b. Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Dusan, S., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodriguez, A., Schulte, M., Shafipour, R., Shao, L., Siu, M., Dubey, P., Micikevicius, P., Naumov, M., Verrilli, C., Wittig, R., Burger, D., and Chung, E. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023. Schulman, J. and Thinking Machines Lab. LoRA without regret. Thinking Machines Lab: Connectionism, 2025. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the European Conference on Computer Systems (EuroSys), pp. 1279–1297, 2025. Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J. E., and Stoica, I. SLoRA: Scalable serving of thousands of LoRA adapters. In Proceedings of Machine Learning and Systems (MLSys), pp. 296–311, 2024. Thinking Machines Lab. Tinker: a training API for fine-tuning open-weight models. https:// thinkingmachines.ai/tinker/, 2025. Unsloth AI. FP8 reinforcement learning. https://unsloth.ai/docs/get-started/ reinforcement-learning-rl-guide/ fp8-reinforcement-learning, 2025a. Unsloth AI. Reinforcement learning (RL) guide. Unsloth documentation, https://unsloth.ai/docs/basics/

Wang, S., Wei, J., Sabne, A., Davis, A., Ilbeyi, B., Hechtman, B., Chen, D., Murthy, K. S., Maggioni, M., Zhang, Q., Kumar, S., Guo, T., Xu, Y., and Zhou, Z. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 93–106, 2023. Wang, S., Ren, R., and Gui, L. ForkKV: Scaling multiLoRA agent serving via copy-on-write disaggregated KV cache. arXiv preprint arXiv:2604.06370, 2026a. Wang, X., Yu, H., Zhuge, Z., Mao, B., Li, Z., Feng, J., Luo, Y., Zhang, Y., Cao, Y., Zhang, M., Liu, D., and Zhang, J. TRACE: Rollout-guided quantization-aware training for FP4 reinforcement learning of MoE language models. arXiv preprint arXiv:2610.07767, 2026b. Wen, Y. and Chaudhuri, S. Batched low-rank adaptation of foundation models. In International Conference on Learning Representations (ICLR), 2024. Wu, B., Zhu, R., Zhang, Z., Sun, P., Liu, X., and Jin, X. dLoRA: Dynamically orchestrating requests and adapters for LoRA LLM serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 911–927, 2024. Xi, H., Ruan, C., Liao, P., Lin, Y., Cai, H., Zhao, Y., Yang, S., Keutzer, K., Han, S., and Zhu, L. Jet-RL: Enabling on-policy FP8 reinforcement learning with unified training and rollout precision flow. arXiv preprint arXiv:2601.14243, 2026. Xia, Y., Fu, F., Zhang, W., Jiang, J., and Cui, B. Efficient multi-task LLM quantization and serving for multiple LoRA adapters. In Advances in Neural Information Processing Systems (NeurIPS), pp. 63686–63714, 2024. Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., and Tian, Q. QA-LoRA: Quantization-aware low-rank adaptation of large language models. In International Conference on Learning Representations, 2024.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

Yadav, P., Choshen, L., Raffel, C., and Bansal, M. ComPEFT: Compression for communicating parameter efficient updates via sparsification and quantization. arXiv preprint arXiv:2311.13171, 2023.

Zhu, Z., Su, Q., Ding, Y., Song, K., Wang, S., and Pekhimenko, G. LoRAFusion: Efficient LoRA fine-tuning for LLMs. In Proceedings of the European Conference on Computer Systems (EuroSys), pp. 586–604, 2026.

Yao, X., Hu, Q., and Klimovic, A. DeltaZip: Efficient serving of multiple full-model-tuned LLMs. In Proceedings of the European Conference on Computer Systems (EuroSys), pp. 110–127, 2025.

Zhuge, Z., Yu, H., Wang, X., Li, Z., Cao, Y., Liu, D., and Zhang, J. QUADS: Stabilizing NVFP4 reinforcement learning for MoE via QUantization-error alignment across dual sides. arXiv preprint arXiv:2607.15810, 2026.

Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., and Ceze, L. FlashInfer: Efficient and customizable attention engine for LLM inference serving. In Proceedings of Machine Learning and Systems (MLSys), 2025. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., Liu, X., Lin, H., Lin, Z., Ma, B., Sheng, G., Tong, Y., Zhang, C., Zhang, M., Zhang, W., Zhu, H., Zhu, J., Chen, J., Chen, J., Wang, C., Yu, H., Song, Y., Wei, X., Zhou, H., Liu, J., Ma, W.-Y., Zhang, Y.-Q., Yan, L., Qiao, M., Wu, Y., and Wang, M. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Yu, T. T. L., Singh, G., Shi, G., Sadri, H., Zhang, Y., and Fan, Z. MARLaaS: Multi-tenant asynchronous reinforcement learning as a service. arXiv preprint arXiv:2605.08527, 2026a. Yu, Z., Venkatachalam, O. R., Wang, Z., Li, Y., Lin, Y., Yu, Z., Wang, Y., Liu, L., Tang, X., Pei, S., Kang, Y., and Ding, Y. PLoRA: An NDP-enhanced pooled-memory system for cost-efficient multi-LoRA serving. arXiv preprint arXiv:2608.05483, 2026b. Zhang, Y., Du, H., Cao, C., and Xu, J. Loquetier: A virtualized multi-LoRA framework for unified LLM fine-tuning and serving. arXiv preprint arXiv:2511.00101, 2025. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, pp. 62557–62583, 2024. Zhou, J., Shao, W., Zheng, L., Fan, Y., and Wong, N. AIS: Adaptive importance sampling for quantized RL. arXiv preprint arXiv:2605.13907, 2026. Zhu, K., Gao, Y., Zhao, Y., Zhao, L., Zuo, G., Gu, Y., Xie, D., Tang, T., Xu, Q., Ye, Z., Kamahori, K., Lin, C.-Y., Wang, Z., Wang, S., Krishnamurthy, A., and Kasikci, B. NanoFlow: Towards optimal large language model serving throughput. arXiv preprint arXiv:2408.12757, 2024.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

These appendices hold the measurements behind the paper’s claims. Appendix A discusses where EpiLoRA helps and its limitations, and Appendix B what each part of the RL system buys; Appendices C to F give the closest systems and the kernel, serving and RL measurements; Appendix G gives the code and the software.

A

D ISCUSSION AND LIMITATIONS

Where a single low-precision base helps. EpiLoRA helps where the adapter’s cost or the base’s memory is the bottleneck. At small decode batches, vLLM’s LoRA kernels add 35–55% to a Hopper layer, and the decode path removes most of it. With many tenants, EpiLoRA’s adapter kernels keep more of the base’s throughput, and adapters stored in low precision let more of them stay resident. In RL, an FP8 trainer removes the bf16 copy of the base, which lets larger models train on smaller GPUs. Other engines could adopt the decode path by adding the up-projection in a CUTLASS epilogue that waits for a down-projection launched first. Computing the down-projection inside the normalization kernel that already reads the input would hide it for the query/key/value and gate/up projections, whose inputs come from a normalization, but not for the output and down projections. Its gain is bounded by what stays exposed today: at one decode token, the exposed down-projection and the second kernel’s launch together take 1.4–2.2 µs of the 10–12 µs a rank-64 adapter still adds to a Hopper layer (Table 13); the rest is the epilogue’s extra work, so a leaner epilogue is the next step. Where it does not. Where neither memory nor latency is the bottleneck, simpler designs are as good or better. Where two copies fit, FlashRL’s recipe is the cheapest: on the 9B at 2,048-token rollouts on an H100 it takes 2.96 GPU-hours per run, against 3.10 for our FP8 trainer with TIS (the difference is in training) and 3.22 for bf16 training. At 2,048 tokens one copy saves memory, not time, on the 9B: one bf16 copy that keeps its KV cache trains as fast as one FP8 copy on the L40S (174 against 173 s per step), while on the 27B the FP8 copy is 1.33× faster (Table 37). With rollouts of up to 8,192 tokens, the bf16 copy that keeps its KV cache takes 2.2× EpiLoRA’s step time on the 9B, or 1.9× that of an FP8 copy that also keeps it (Table 38); Unsloth’s bf16 mode, which releases the cache while training and which we did not run, holds bf16 offloading’s memory without its weight transfers, so it would take at most bf16 offloading’s 1.3×. The matched engine samples more slowly than a W8A8 engine: it takes 1.35× the GPU-hours of corrected W8A8 rollouts on the H100, almost all in generation, and 0.95–1.03× on the L40S (Tables 35 and 36). vLLM’s Marlin engine, which also keeps activations in 16 bits, reaches the same accuracy at 3.9 GPU-hours, so the matched engine’s advantage, a layout the trainer can share, is unproven;

on the L40S one copy ended 3.4 points below bf16 training ([−5.7, −1.1]). With long rollouts on the 27B, FlashRL’s steps trained 19% more tokens than ours in the same time in the 12-step probes (Table 39), and in the full 100-step runs its recipe needed 2–3% fewer GPU-hours per trained token than ours (Table 7), since its bf16 trainer is faster; at 16,384 tokens the bf16 trainers run out of memory, and FlashRL’s recipe fits only with chunked log-probabilities and then takes 0.86× our step time while training fewer tokens per step (Table 38). A trainer that rounds its activations as the engine does (our proxy for Unsloth’s FP8 trainer; S8RA in Table 30) trains 1.2× as fast as ours on the L40S (35–39 against 45–46 s of training per step, from the runs’ logs), at 1.4–1.6× our trainer’s mismatch with the engine. With 32 tenants, TensorRT-LLM delivers 1.05× EpiLoRA’s throughput on the H200 in its own job, SGLang 0.5.9 delivers 1,803 tokens/s on Qwen3-8B on the L40S against EpiLoRA’s 1,761 in Table 2’s job (1.02×, in separate jobs; SGLang’s engine without LoRA delivers 3,179), and vLLM’s kernels tuned for Llama-3.1-8B match ours. Limitations. (i) Hardware. Datacenter Blackwell GPUs (sm 100) are untested; our Blackwell results come from a MIG slice of an RTX PRO 6000 (sm 120). sm 100’s shorter main loops would leave more of the down-projection exposed. The first-wave window used in Figure 5 is an empirical fit chosen after the data, and it overestimates the exposure beyond two waves. Our Triton GEMM trails CUTLASS on the H100 above 32 tokens, and at one token by up to 28% on narrow tensor-parallel shards and on small models (Table 16). (ii) Scope of serving. The tensor-parallel path is opt-in and was tested only at two-way parallelism on one 72B model, where its batch-1 lead requires tiles searched for the shards and where it ties vLLM with 8 tenants. Mixture-of-experts LoRA and block-scaled FP8 bases are untested, and speculative decoding was tested only with n-gram drafts on one GPU, where the decode path serves verification steps in 0.81–0.82× vLLM’s time per token except in runs whose drafts stopped matching after a first token that run-to-run nondeterminism flipped, as in some of vLLM’s own runs. TensorRTLLM ran on one model, with its PyTorch backend and a single-process worker. (iii) Workloads. Serving speed uses random adapters and, except for one WildChat trace, synthetic requests; storage accuracy uses only RL-trained adapters; and every model except Llama-3.1-8B is a Qwen model. (iv) Default CUDA-graph sizes. At vLLM’s default capture sizes, EpiLoRA’s operator runs eagerly for prefill steps beyond 512 tokens. It still leads online at every rate on the H200, on the 72B with tensor parallelism and on the

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

H100 with long prompts, but with 512-token prompts it trails vLLM on the H100 at 16 requests/s (1.18× its time per output token, p99 TTFT 1.11–1.20×) and trails it slightly on the L40S at 4–6 requests/s (1.02×, p99 TTFT up to 1.32×; Table 24). The kernel-level account (Table 27): the timed dispatch, which times its candidates on CUDA-graph replays, mixes vLLM’s adapter kernels into some projections and so pays both systems’ launch costs on the H100’s host CPU, and EpiLoRA’s own down-projection took twice vLLM’s GPU time at these batch sizes. Handing every batch above 32 tokens to vLLM’s kernels widens the gap (1.23× and 1.26× at 8 and 16 requests/s). Two optin settings chosen after these diagnostics, EpiLoRA’s own kernels above the capture bound and tiles searched for large batches, narrow it to 1.07× at 16 requests/s with p99 TTFT within 1.1×. Timing the dispatch’s candidates as they run outside the graphs, host time included, would let it choose these settings where they help. (v) Co-serving. Serving tenants from the copy that RL trains on works but does not pay off: on one L40S it saves only 5–8% of the GPU-hours of two separate deployments (0.92–0.95×), RL’s steps take 1.75× as long, and the tenants’ p99 time to first token rises from 0.26 to 2.87 s. (vi) Batch-dependent numerics. A request’s tokens are computed by the decode path in decode batches of up to 32 tokens and by the vendor GEMM with adapter kernels above, so greedy outputs can change with the batch’s composition, as they do between vLLM’s own kernel configurations: on 16 prompts, every alternative configuration, vLLM’s tuned and dual-stream kernels included, diverges from vLLM’s default path on 5–9 in the output check of our H100 end-to-end test, where EpiLoRA’s decode path diverges on at most one more; without adapters, the two engines’ outputs are identical. Deployments that need batch-invariant outputs need batch-invariant kernels (He & Thinking Machines Lab, 2025).

B

W HAT EACH PART OF THE RL SYSTEM BUYS

Table 8 summarizes, for each part of the RL system of Section 3.3, what it buys and what it costs, measured on Qwen3.5-9B on one L40S unless noted; Table 36 gives the full per-configuration accounting. Running the matched engine, one copy and the KV release together in a 30-step probe takes 153 s per step, against 155 s with the KV cache kept, with twice the KV cache (23 against 12 GiB).

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 8. What each part of the RL system buys and costs. Qwen3.5-9B on one L40S unless noted; GPU-hours per 200-step run; accuracy differences with 95% confidence intervals. Part

Benefit

Cost

FP8 trainer weights

6.5 GiB less GPU memory for the 9B and 22 GiB for the 27B; the 27B trains on one 48 GB L40S Pinned host memory falls from 48–159 GiB to 7–11 GiB A 21 GiB instead of a 10 GiB KV cache during rollouts

None measurable: 7.2 GPU-hours, the same as FlashRL’s bf16 trainer None measurable: 7.1 against 7.2 GPU-hours with offloading None measurable at 2,048 tokens; at 8,192 tokens the 27B’s KV-releasing copy took 1.05× FP8 offloading’s GPU-hours On the H100, 4.2 GPU-hours (Marlin 3.9) against 3.1 for W8A8 with TIS; on the L40S, 8.8 against 9.3 resident and 7.2 offloaded; with one copy at engine share 0.55, −3.4 points [−5.7, −1.1] against bf16

One shared copy KV-cache release

Unrounded (W8A16) rollouts, ours or Marlin’s

C

No correction needed: uncorrected runs end where W8A8 with TIS does

T HE CLOSEST SYSTEMS

Table 9 compares EpiLoRA’s capabilities with those of the closest serving and RL systems. Services that train and serve many users’ adapters on shared bases (Tinker, MinT, MARLaaS) do not document their base precision or kernels, so they are not in the table.

D

Table 9. EpiLoRA brings many tenants’ adapters and an RL trainer onto one low-precision base. Capabilities of the closest systems, not their speed. Fused: adapters computed inside the low-precision base GEMM. One copy: the trainer computes on the engine’s copy of the base. Matched: engine and trainer compute on the same weights with activations unrounded, so RL needs no off-policy correction.

K ERNEL DETAILS

One output tile of K2 (Figure 10). K2 runs the lowprecision GEMM over tokens sorted by adapter, so each output tile meets a few contiguous groups of rows, and its epilogue adds each group’s up-projection. Table 1’s L40S column comes from the weight-only kernel benchmark: Marlin’s W8A16 GEMM with vLLM’s tuned LoRA kernels against EpiLoRA’s fused weight-only kernel, one Qwen3-8B layer at one decode token with one rank-64 adapter, CUDA graphs with the L2 cache flushed; its other columns come from the rows of Table 10. The decode path across decode batch sizes (Table 10). From one to sixteen tokens, one rank-64 adapter adds 4–9% to a Qwen3-8B layer on Hopper GPUs and 3–7% on the Blackwell slice with the decode path, against 34–61% and 13–20% for vLLM’s tuned or dual-stream kernels. Controls for the overlap (Tables 11 and 12). Serializing a library-fast down-projection (cuBLAS’s GEMM or GEMV) before the same GEMM leaves the layer 1.17–1.25× slower than the decode path, and our own down-projection at its fastest split-K 1.13–1.17×; only a serialized kernel that does no work comes within 2%. Where the adapter’s remaining cost goes (Table 13). Under the decode path, most of what the adapter still adds to a layer is the epilogue’s extra work; the exposed downprojection and its launch take 1–3 µs per layer, and EpiLoRA’s GEMM without an adapter is up to 6 µs faster than CUTLASS with vLLM’s separate quantization.

Serving system

Many tenants

Fused

Low-precision adapters

Punica, S-LoRA vLLM, SGLang TensorRT-LLM SVDQuant LoRAFusiona FlexLLMb EpiLoRA

✓ ✓ ✓ × ✓ ✓ ✓

× × × ✓ ✓ × ✓

× × ✓ × × × ✓

RL system

Low-precision rollouts

One copy

Matched

FlashRL Jet-RLc QeRL Unslothd Unsloth, FP8c EpiLoRA

✓ ✓ ✓ ✓ ✓ ✓

× × × ✓ ✓ ✓

× × ✓ ✓ × ✓

a

Fine-tuning: mixed-adapter batches in a bf16 GEMM. b Co-serves inference and supervised fine-tuning, not RL. c Rounds activations alike on both sides, which leaves a mismatch (Section 3.3). d A bf16 or 4-bit copy, weight-only on both sides.

Which path the dispatch picks (Table 14). At engine start, the plugin times every candidate path per projection shape and batch-size bucket, on CUDA-graph replays with one adapter, and keeps the fastest. On the H100 with 32 rank-64 adapter slots it picks the decode path for every projection up to 32 tokens; above, it picks EpiLoRA’s adapter kernels after the vendor GEMM for the projections that combine three or four outputs, where the projection, vendor GEMM included, takes 0.53–0.85× its time with vLLM’s kernels, and vLLM’s own adapter kernels for the others. Why EpiLoRA’s adapter kernels win on these projec-

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 10. On the H100, the H200 and the Blackwell slice, the decode path brings one adapter’s cost to a few percent of the layer. One Qwen3-8B decoder layer (the four projections summed; FP8 base on the Hopper GPUs, our NVFP4 checkpoint on the Blackwell slice), one rank-64 adapter, CUDA graphs with the L2 cache flushed, microseconds (lower is better); each path at its best tile configuration. Previous: the plugin’s best path before the decode path (Hopper: hybrid or swap-AB; Blackwell: fused or hybrid). Bold: the fastest path with the adapter. Last column: the PDL path’s time over CUTLASS alone. µs ↓ Tokens

vLLM’s LoRA path CUTLASS alone

H100, FP8 base 1 126 4 128 8 128 16 128 H200, FP8 base 1 104 4 106 8 106 16 106 Blackwell slice, NVFP4 base 1 286 4 285 8 288 16 291

Ours

PDL over

default

tuned

dual-stream

previous

no PDL

PDL

CUTLASS (%)

205 195 195 196

179 179 179 180

180 172 172 174

188 192 191 181

163 162 157 167

134 133 133 136

+6 +4 +4 +7

190 180 180 180

161 162 162 162

167 157 156 159

171 174 175 164

144 147 144 150

112 112 113 115

+8 +6 +7 +9

388 360 359 359

342 339 343 348

340 325 326 334

316 317 320 324

330 339 342 345

293 298 298 312

+3 +5 +3 +7

Table 11. The overlap beats serializing even a library-fast down-projection, and comes within 2% of a serialized kernel that does no work (0.99–1.02×). One Qwen3-8B decoder layer’s four projections at one decode token with one adapter, microseconds summed over the projections (CUDA graphs, L2 cache flushed), on one H200 and one H100. Serialized: a kernel run before the same GEMM without overlap, which does no work (empty kernel), uses cuBLAS’s GEMM or GEMV, or is our own down-projection. GEMM, h given: the GEMM with the down-projection’s result already in memory, a lower bound for any overlap. Shrink alone: the down-projection by itself. µs ↓ Serialized GEMM, Shrink alone Empty GPU

Rank Decode path empty kernel cuBLAS GEMM cuBLAS GEMV 108.1 112.4 117.9 129.3 133.4 139.2

H200 16 64 128 H100 16 64 128

110.2 114.0 120.7 132.0 134.8 138.2

135.0 140.1 146.1 154.9 158.0 163.5

134.6 139.6 146.3 154.5 158.5 163.0

ours 138.9 144.6 150.8 156.8 163.0 164.8

h given cuBLAS ours 106.0 110.4 117.3 128.7 130.9 135.3

37.6 38.6 40.5 42.0 42.1 44.6

48.6 50.3 51.7 49.4 53.4 52.2

kernel 12.8 13.3 12.9 20.3 19.6 19.9

Table 12. Our own down-projection at its fastest split-K, run before the same GEMM without overlap, still trails the decode path. Table 11’s cases: one Qwen3-8B layer’s four projections at one decode token with one adapter, the down-projection at every split-K from 1 to 32, serialized before the GEMM; per projection the fastest split, microseconds summed over the projections (CUDA graphs). Ratio: serialized over the decode path, for the layer and per projection. Shrink alone: the down-projection at the decode path’s split and at the fastest. GPU rank decode path serialized ratio per projection shrink alone H200 H200 H200 H100 H100 H100

16 64 128 16 64 128

µs ↓

µs ↓

109.6 112.1 117.9 128.8 133.1 137.9

118.3 130.8 146.1 140.3 149.9 162.0

1.08× 1.17× 1.24× 1.09× 1.13× 1.17×

1.05–1.12× 1.11–1.24× 1.17–1.33× 1.06–1.15× 1.09–1.19× 1.11–1.26×

its split

fastest

11.1–14.6 9.0–17.8 9.7–18.2 9.3–15.2 10.1–18.1 10.6–18.2

6.1–8.6 6.6–9.0 6.9–9.7 7.4–9.7 7.5–9.9 7.9–10.5

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL acc xq rows (sorted)

h (rank r)

BM ×BN tile

r×BN Bℓ1 slice

ℓ1 mainloop acc += xq Wq⊤ FP8 or NVFP4 MMA over K

ℓ2

Bℓ2 slice

Bℓ3 slice

ℓ3

epilogue acc · sx sw per segment ℓ in the tile: acc[ℓ] += hℓ Bℓ⊤ (masked rank-r dot) scatter rows to y

Figure 10. One output tile of K2. Rows are tokens sorted by adapter, forming contiguous groups (here three). After the low-precision main loop, the epilogue adds each group’s downprojection h multiplied by its adapter’s slice of B, then writes the rows back in token order. Table 13. Under the decode path, the epilogue, not the downprojection, is most of the adapter’s remaining cost. One Qwen38B decoder layer (its four projections) with one rank-64 adapter at 1–16 decode tokens; CUDA graphs, L2 cache flushed. GEMM vs. CUTLASS: EpiLoRA’s GEMM without an adapter minus vLLM’s quantization and CUTLASS GEMM; epilogue: the GEMM with the adapter, the down-projection’s result given, minus the GEMM without it; exposed + launch: the decode path minus the former; adapter total: the decode path over EpiLoRA’s own GEMM. Microseconds per layer ↓ GPU Tokens GEMM vs. CUTLASS Epilogue Exposed + launch Adapter total H100

−5.0 −5.9 −5.9 −2.3 −1.9 −3.3 −2.4 +0.1

10.2 9.6 9.0 7.7 8.8 7.6 7.8 6.7

2.2 1.8 1.8 3.1 1.4 1.6 1.5 2.5

When K2 is slower than CUTLASS (Table 16). Without an adapter, K2 matches CUTLASS on the 8B’s and the 72B’s shapes and trails it on small and narrow ones, which lengthens its main loop and so widens the window that hides the down-projection.

12.5 11.4 10.8 10.8 10.2 9.1 9.3 9.2

The exposure model (Table 17). On every measured projection, the exposed time against the down-projection’s time and the GEMM’s main loop. The window of the GEMM’s first wave of thread blocks, assuming one block per SM, was fitted after the data; with the resident blocks per SM that the CUDA occupancy API reports (two in most Hopper cases), the same window tracks the exposure less well.

tions (Table 15). In the profiled step of Table 27, EpiLoRA’s GPU time on the H100 is 5.0 ms below vLLM’s: 3.4 ms in the up-projection (expand) and 1.9 ms in activation quantization and norms, which EpiLoRA’s base path computes with vLLM’s CUDA quantization kernel rather than the kernels vLLM’s compiler generates (as much with EpiLoRA’s own adapter kernels throughout, so not from them), while its down-projection takes 0.2 ms more. We attribute the up-projection’s difference to how the kernels index a projection’s outputs. vLLM’s expand kernel treats them as slices and launches one program per adapter, slice and tile of the widest slice, so the programs of narrower slices past their width exit without work. With several slices it reads each slice’s starting column in the output from memory at run time, and when their widths differ, as for attention’s query, key and value under grouped-query attention (8,192, 1,024 and 1,024 columns in Qwen3.5-9B) and for the linearattention input (2,048, 2,048, 4,096 and 4,096), also each slice’s strides, which the compiler then cannot specialize. EpiLoRA’s expand-add tiles the concatenated output instead,

Many distinct adapters (Figure 11). With one adapter in a 32-token batch, EpiLoRA adds 11% to the H100 layer where vLLM’s tuned kernels add 38%; with 32 distinct adapters the adapters’ own weights dominate the step, and storing them in FP8 or NVFP4 nearly halves the overhead on the L40S.

H200

1 4 8 16 1 4 8 16

each tile inside one slice, and receives the slice boundaries as kernel arguments. The costs per output column are consistent with this account: with tiles tuned for these shapes, vLLM’s expand takes 7.9 ns on the projections with one output, 13.4 ns on gate/up’s two equal ones and 21.6 ns on the three or four unequal ones, while EpiLoRA’s takes 12.1– 13.1 ns on every kind, so on a single output vLLM’s kernel is the faster and the dispatch keeps it there. On the H200, vLLM’s expand takes 17.4 ns per output column on the projections with two to four outputs against EpiLoRA’s 11.7, and 8.8 against 13.1 ns on those with one. We did not time vLLM’s kernel launched once per slice or with the slice offsets as constants, either of which might close the gap, and these costs come from one model’s 2,048-token step outside CUDA graphs; at the 64–512-token batches of the many-tenant runs, Table 14 shows the same ordering by kind of projection. The gain also depends on the model: on Llama-3.1-8B, whose query, key and value have Qwen38B’s widths, vLLM’s tuned kernels tie EpiLoRA with 32 tenants (Table 2). EpiLoRA’s kernels read vLLM’s own adapter buffers, so vLLM could adopt them.

E

E ND - TO - END SERVING

Batch 1 against throughput (Figure 12). Every vLLM configuration and every EpiLoRA path on each GPU, one point per configuration. SGLang and TensorRT-LLM (Tables 18 to 20). Each system ran in its own job beside EpiLoRA, which gives the same-job ratios of Table 3; Table 21 gives the GPU profiles behind its Nsight row. Served adapters’ accuracy (Table 22). The test behind Figure 8.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 14. Above 32 tokens, EpiLoRA’s adapter kernels win on the projections that combine three or four outputs, and the dispatch keeps vLLM’s kernels for the rest. The timed dispatch’s choice per projection of Qwen3.5-9B on the H100 with 32 rank-64 adapter slots (FP8 base, CUDA-graph replays, one adapter), and the projection’s time with vLLM’s adapter kernels over its time with EpiLoRA’s at the given batch size, the same vendor GEMM included in both (above 1: EpiLoRA’s are faster; our call of vLLM’s kernels zero-fills their buffer, at most 0.5 MB here, once more than vLLM does). The default-capture serving configuration’s dispatch picks the same paths, and so does the dispatch on an H200, where this ratio is 1.19–2.00 on the projections with three or four outputs and the gate/up projection switches to them from 256 tokens. With 32 rank-64 tenants on the H200, EpiLoRA delivers 5,813 tokens/s against vLLM’s 5,286 (1.10×); with vLLM’s adapter kernels above 32 tokens instead of its own, it delivers 5,148 (0.97×), so the gain comes from EpiLoRA’s kernels above 32 tokens (one job, three passes each). Path picked at Projection

1–32 tokens 64–2,048 tokens

linear attention, input (4 outputs) attention, query/key/value (3 outputs) MLP, gate/up (2 outputs) attention output MLP, down

decode path decode path decode path decode path decode path

EpiLoRA’s kernels EpiLoRA’s kernels vLLM’s kernels; EpiLoRA’s kernels at 256 vLLM’s kernels vLLM’s kernels

vLLM

376,832 786,432 262,144

vLLM tuned

EpiLoRA

21.6 13.4 7.9

EpiLoRA, FP8

(a) L40S layer time / no LoRA

4.71 9.52 3.42

8.14 10.54 2.08

12.5 12.1 13.1

EpiLoRA, NVFP4

(b) H100

7× 5× 3× 2× 1.5× 1× 1

4

8

16

32

128

256

512

1.17 1.25 0.89 0.72 0.74

1.62 1.78 1.00 0.84 0.89

1.57 1.89 0.99 0.70 0.78

L40S

EpiLoRA, decode path EpiLoRA, other paths EpiLoRA, weight-only EpiLoRA, NVFP4 adapters

6

H100

1.75 5 1.50 4 1.25 14

16

5

H200

6

7

Blackwell slice, NVFP4

6 2.00

ns per column ↓

Outputs (projections) Columns vLLM EpiLoRA vLLM EpiLoRA 3–4 unequal (32) gate/up, 2 equal (32) one output (64)

64 1.31 1.45 0.87 0.81 0.84

vLLM, default kernels vLLM, tuned, CUTLASS base vLLM, tuned, cuBLASLt base vLLM, tuned, Marlin base vLLM, tuned, two streams

32 tenants (k tokens/s)

Table 15. vLLM’s up-projection costs more per output column as a projection’s outputs multiply; EpiLoRA’s costs about the same on every kind. The up-projection’s GPU time per kind of LoRA projection of Qwen3.5-9B in the profiled H100 step of Table 27, with vLLM’s kernel tuned for these shapes and with EpiLoRA’s expand-add. Each kind’s time is a difference between profiles whose timed dispatches send it to different kernels: vLLM sends all 152 projections to its kernel (20.91 ms); EpiLoRA sends the 32 with unequal outputs to its own (4.71 ms, with 12.77 left on vLLM’s), with the new tiles gate/up as well (14.23; 2.23 left; a profile of the same step that Table 27 does not show), and with its own kernels all but the linear attention’s 24 two-column projections (17.65; 0.15 left). ms per step ↓

vLLM’s kernels’ time over EpiLoRA’s at (↑)

1

8

32

distinct adapters in a 32-token decode batch

Figure 11. EpiLoRA’s advantage is largest with few distinct adapters. Time of one Qwen3-8B layer with rank-64 adapters at 32 decode tokens, relative to vLLM’s FP8 GEMM without adapters, as the number of distinct adapters in the batch grows. (a) L40S, fused path, with adapters in bf16, FP8 or NVFP4; (b) H100, decode path. vLLM tuned: LoRA kernels tuned for the GPU.

4

1.75 4

7 10 6 5 batch-1 time per output token, one rank-64 adapter (ms)

12

Figure 12. EpiLoRA is faster at batch 1 than every vLLM configuration and leads or ties with 32 tenants. Each point is one configuration of one job: batch-1 time per output token with one rank-64 adapter (lower is better) against throughput with 32 rank64 tenants (higher is better), for Qwen3.5-9B (Qwen3-8B with an NVFP4 base on the Blackwell slice). Dotted lines mark the same engine without LoRA, which is also the speed of a merged adapter. On the L40S, EpiLoRA with adapters stored in NVFP4 delivers 1,767 tokens/s with 32 tenants, against 1,475 for vLLM’s best configuration in the same job (1.20×). The KV cache with 32 rank-64 slots on the L40S holds 422k tokens with vLLM, 357k with EpiLoRA’s bf16 adapters and 630k with NVFP4-stored ones; with bf16 adapters, the plugin’s load-time warm-up fragments the memory vLLM later allocates for its adapter buffers, and an opt-in deferred warm-up restores 410k against vLLM’s 422k in a job of its own. Table 2’s Qwen3-8B row on the L40S comes from one job of its own: at batch 1, 14.05 ms for vLLM’s Marlin path and 13.95 ms for EpiLoRA’s weight-only mode; with 32 tenants, 1,628 tokens/s for vLLM’s tuned path and 1,761 for EpiLoRA.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL ±5-point margin

Table 16. Without LoRA, the decode path’s GEMM matches CUTLASS on the 8B’s and the 72B’s shapes, and trails it on small and narrow ones. K2 without an adapter (with its own per-token quantization) over vLLM’s no-adapter path (pertoken quantization, then CUTLASS) per projection on the H100 (M tokens, n cases; CUDA graphs; CUTLASS with vLLM’s quantization), from the kernel benchmark’s rows (Tables 10 and 17; the 72B’s shards from their own run, whose tiles the job of Table 28 uses); K2’s tiles fully searched except for Qwen3-8B’s tensorparallel shards (a coarse sweep). Where K2 is slower, its longer main loop also widens the window that hides the shrink. Shapes

M

n K2 / CUTLASS ↓ min median max

Qwen3-8B 1–16 16 0.93 Qwen3-1.7B 1 8 0.92 Qwen3-0.6B 1 8 1.15 Qwen3-8B, TP2 1 4 0.98 Qwen3-8B, TP4 1 4 0.97 Qwen3-8B, TP8 1 4 0.99 Qwen2.5-72B, TP2 1–32 20 0.93

0.96 1.00 1.08 1.12 1.16 1.24 1.15 1.28 1.06 1.23 1.19 1.22 0.99 1.07

With CUDA graphs over the prefill sizes (Table 23). Every point of Figure 6’s solid series, and the same test with rank16 tenants. The WildChat trace (Table 5). Of 4,000 first turns, vLLM’s benchmark keeps the 3,383 with at most 1,024 prompt and 2,048 total tokens: prompts have a median of 23 tokens (90th percentile 247), replies 253 (759), and each request generates its recorded reply length. Each start draws 1,000 of them, a different sample per start and the same for both systems, with 85–102 prompt and 334–345 reply tokens on average. vLLM’s p99 TTFT of 1.37 s on the H100 at 8 requests/s comes from one of its three starts (3.7 s; the other two 0.19–0.20 s); EpiLoRA’s p99 TTFT there is 1.06–1.13× vLLM’s at the other three rates. At vLLM’s default CUDA-graph sizes (Table 24). Steps that prefill more than 512 tokens run outside CUDA graphs; with 512-token prompts and 32 tenants, EpiLoRA lowers the time per output token at every rate on the H200 and at low rates on the H100 and the L40S. Long prompts, a 72B model and host time (Tables 25 and 26). At vLLM’s default CUDA-graph sizes, with direct launches, EpiLoRA lowers the time per output token with 2,048-token prompts on both Hopper GPUs and on a 72B model; Table 26 gives each path’s host and GPU time per layer for the steps that run outside CUDA graphs. Where a step outside CUDA graphs spends its time (Table 27). In a step that prefills 2,048 tokens, EpiLoRA’s default configuration takes less GPU time than vLLM but more host time, because its timed dispatch picks vLLM’s adapter kernels for some projections and pays both systems’ launch costs, which the H100’s host CPU makes larger.

Qwen3.5-9B, H100 (bf16: 60.7%) W8A8 + TIS (9) FlashRL's recipe (9) Matched W8A16 (9) Marlin W8A16 (vLLM) (4)† Qwen3.5-9B, L40S, base offloaded (bf16: 61.8%) W8A8 + TIS (7) Qwen3.5-9B, L40S, base kept on the GPU (vs. bf16 above) W8A8 + TIS, two copies (3)† W8A8 + TIS, one copy (3)† W8A8 + TIS, one copy, KV released (3)† Matched W8A16, two copies (3)† Matched W8A16, one copy (5)† Qwen3.8-27B, H100 (bf16, base offloaded: 71.4%) FP8, base offloaded (3) FP8, two copies (6)† FP8, one copy (7)† FP8, one copy, KV released (3)† Qwen3.8-27B, L40S (bf16 on two A100s: 61.7%) FP8, base offloaded (3) W8A8 engine (vLLM FP8 with EpiLoRA) −10 0 −5 +5 Matched W8A16 engine (EpiLoRA) final accuracy minus bf16 (points) Marlin W8A16 (vLLM) before RL

Figure 13. FP8 rollouts end close to bf16 training when they are corrected or matched. Final accuracy minus that of bf16 training on the same GPU (in each heading), with 95% confidence intervals and the number of runs; shaded: ±5 points; dotted: before RL. Gray: other systems’ recipes; KV freed: the engine releases its KV cache during training; † compared against this reference after the data; the 27B’s two- and one-copy rows pool every completed run, as in Table 36. Accuracy on OlympiadBench and Minerva pooled, except the 27B on the H100, which also includes MATH500 (Table 31).

Restricted to its own kernels, EpiLoRA matches vLLM’s host time but its down-projection takes about twice vLLM’s GPU time at this batch size. Tiles searched for large batches remove most of that difference, bringing the step to 1.01× vLLM’s time on the H100 and 0.98× on the H200. Online on the H100, the two settings together narrow the gap at 16 requests/s from 1.18× to 1.07× and bring the p99 time to first token within 1.1× at every rate (Table 24). On the H200, whose host CPU is faster, EpiLoRA’s default dispatch, which mixes in vLLM’s kernels, is faster still (0.94× vLLM’s step time), so the settings stay opt-in. The 72B model (Tables 28 and 29). Across two H100s with tensor parallelism, against vLLM’s LoRA kernels tuned for the shard shapes, and on one H200 with the timed dispatch and with the decode path forced on every projection.

F

L O RA RL

Every configuration (Figure 13, Tables 30 and 31). Final accuracy of each RL configuration against bf16 training on the same GPU, with its cost; Table 32 tests each comparison for equivalence within ±5 points, also paired by seed and on one common engine.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 17. What stays exposed of the down-projection, on every measured projection (Figure 5). Per GPU and shape family, with one adapter at 1–16 decode tokens (tensor-parallel shards: Qwen3-8B’s per-GPU shapes at one token): the down-projection alone (tshrink ), the GEMM’s main loop (tmain ), the exposed time (the decode path’s time minus the same GEMM with the down-projection already computed), and the median error of the bound max(0, tshrink − twindow ) with the whole main loop as the window and with the main loop of the GEMM’s first wave of thread blocks, assuming one block per SM (a window fitted after the data). Where the CUDA occupancy API reports two resident blocks per SM, as it does in most Hopper cases, the first-wave window tracks the exposure less well (Pearson 0.50 against 0.71 for the one-block window on the same 115 projections; 0.70 over all projections, as in Figure 5). Exposed (µs) ↓ |error|, median (µs) ↓ GPU

Shapes

H100 rank 64 H100 rank 16 H100 rank 128 H100 TP 2 shards H100 TP 4 shards H100 TP 8 shards H200 rank 64 H200 rank 16 H200 rank 128 H200 TP 2 shards H200 TP 4 shards H200 TP 8 shards GH200 TP 8 shards GH200 Qwen3-0.6B, -1.7B Blackwell slice rank 64

Cases tshrink (µs) tmain (µs) median 24 8 8 4 4 4 16 8 8 4 4 4 3 16 16

10.4–18.3 16.7–50.5 11.8–15.5 18.7–50.8 14.0–18.5 18.7–50.8 10.0–14.5 15.0–33.3 10.7–14.2 14.4–20.6 10.4–14.3 12.6–19.3 9.2–17.8 14.9–41.5 8.4–14.7 15.3–41.4 9.8–18.2 15.3–43.0 9.5–13.7 14.2–28.2 9.9–13.7 13.1–17.6 9.7–13.5 11.7–17.4 9.4–13.8 11.4–17.0 7.8–11.5 12.3–18.0 15.0–29.7 31.6–126.0

max whole loop first wave

0.6 2.6 5.0 3.7 2.5 3.1 0.4 0.2 1.5 3.2 1.5 3.2 4.3 0.0 0.7

1.3 6.1 9.3 5.1 4.3 5.4 1.2 1.0 5.7 4.8 4.8 5.7 4.9 0.5 2.1

0.6 2.6 5.0 3.7 2.5 2.5 0.4 0.2 1.5 3.2 1.5 2.7 2.5 0.1 0.9

0.6 0.7 2.3 2.1 1.1 1.9 0.4 0.2 1.5 1.7 1.1 1.9 2.5 0.1 0.9

Table 18. On the H100, EpiLoRA has 0.61× SGLang’s batch-1 latency and 1.06× its throughput with 32 tenants. Qwen3-8B with an FP8 base, rank-64 adapters; each engine’s W1 (time per output token at batch 1) and W2 (throughput) without LoRA, and each LoRA path’s change against its own engine; means of three starts. SGLang ran with CUDA graphs up to batch 256 and as many LoRA slots as adapters; its LoRA path adds about the same time per step with one adapter or 32 (3.2 ms at batch 1). Without LoRA Engine, LoRA path SGLang 0.5.9, csgmv SGLang 0.5.9, triton vLLM 0.28, default kernels vLLM 0.28, dual-stream ours, decode path

1 adapter (%)

W1 (ms) ↓ W2 (tok/s) ↑ W1 ↓ 4.91 4.91 4.52 4.52 4.52

10,125 10,125 10,663 10,663 10,663

+64.9 +67.4 +52.1 +39.3 +8.6

32 adapters (%)

W2 ↑ W1 ↓

W2 ↑

−43.8 −77.2 −27.5 −30.6 −28.1

−43.7 −77.2 −54.3 −54.4 −43.4

+64.8 +66.6 +83.5 +67.8 +8.1

Table 19. On the H200, EpiLoRA has 0.55× SGLang’s batch-1 latency and 1.07× its throughput with 32 tenants. The protocol of Table 18 on one H200 NVL; one start per configuration. SGLang’s LoRA path again adds about the same time per step with one adapter or 32 (3.5 ms at batch 1). Without LoRA 1 adapter (%) 32 adapters (%) Engine, LoRA path SGLang 0.5.9, csgmv SGLang 0.5.9, triton vLLM 0.28, default kernels vLLM 0.28, dual-stream ours, decode path

W1 (ms) ↓ W2 (tok/s) ↑ 4.69 4.69 4.11 4.11 4.11

10,192 10,192 10,742 10,742 10,742

W1 ↓

W2 ↑

W1 ↓

W2 ↑

+74.9 +78.4 +67.9 +52.1 +10.4

−45.3 +74.9 −79.1 +78.1 −28.8 +106.9 −30.8 +90.7 −28.6 +12.5

−45.3 −79.0 −56.6 −55.9 −44.4

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 20. EpiLoRA against TensorRT-LLM, each pair measured in one job. Qwen3-8B with an FP8 base and rank-64 adapters, CUDA graphs; for each metric, TensorRT-LLM’s fastest configuration we found: a its block-FP8 checkpoint with the LoRA path overlapped with the base; b the ModelOpt FP8 checkpoint, LoRA overlapped; c the same at a KV-cache share of 0.6, found after the first run, with EpiLoRA measured again in that job. Without LoRA, EpiLoRA is vLLM’s engine. H100

H200

Table 23. Online serving with CUDA graphs captured up to 2,048 tokens for both systems. Qwen3.5-9B with an FP8 base, 32 tenants, Poisson arrivals at the given rates, 512-token prompts and 256 output tokens; EpiLoRA’s median time per output token and p99 time to first token over those of vLLM with tuned LoRA kernels, means of three server starts. On the H100 at 4 requests/s, one of EpiLoRA’s three starts has an outlying p99 TTFT (335 against 150–160 ms); at 6 requests/s the L40S is past saturation, where vLLM’s p99 TTFT grows to seconds. Median TPOT / vLLM’s ↓ p99 TTFT / vLLM’s ↓

TRT-LLM EpiLoRA TRT-LLM EpiLoRA Time per token, no LoRA (ms) ↓ Time per token, one adapter (ms) ↓ Throughput, one adapter (tok/s) ↑ Throughput, 32 adapters (tok/s) ↑

4.59b 6.93a 6,858b 5,901c

4.52 4.85 7,703 6,072c

4.08b 6.89a 7,256b 6,269b

4.11 4.53 7,654 5,985

Table 21. Where the batch-1 time goes. Time per output token for Qwen3-8B with an FP8 base on one H100, without and with one rank-64 adapter, and the GPU time the adapter adds per token, measured with NVIDIA’s Nsight Systems profiler. Each system uses CUDA graphs; vLLM runs its dual-stream mode with its default LoRA kernels (6.02 ms with tuned ones, Table 2) and TensorRT-LLM its overlapped mode. System

Time per token (ms) ↓

Added GPU

no adapter one adapter

time (ms) ↓

vLLM, two streams SGLang TensorRT-LLM EpiLoRA

4.39 5.10 4.60 4.39

6.52 8.23 7.50 4.92

1.87 3.01 2.87 0.40

H100, requests/s

2

4

8

16

One

Difference

Benchmark vLLM EpiLoRA

only

(95%)

bf16-trained adapter MATH-500 91.4 Olympiad 61.8 Minerva 54.4 W8A8-trained adapter MATH-500 89.0 Olympiad 58.4 Minerva 55.5

8

16

rank 64 0.92 0.95 0.99 rank 64, NVFP4-stored 0.91 0.88 0.86 rank 16 0.94 0.94 0.95

0.97 1.00 0.98 0.99 1.21 0.86 1.00 0.98 0.92 0.10 0.87 1.02 0.96 0.90 0.83

Table 24. Online serving at vLLM’s default CUDA-graph sizes. Qwen3.5-9B with an FP8 base, 32 rank-64 tenants, Poisson arrivals at the given rate (requests per second), 512-token prompts and 256 output tokens, 1,000 requests per rate; vLLM with LoRA kernels tuned for this model (on the H200, the H100’s tuning) against EpiLoRA (with direct launches on the H100), both at vLLM’s default capture up to 512 tokens. New tiles: down-projection tiles searched for batches above 512 tokens; own kernels: no vLLM adapter kernels in steps above the capture bound; both opt-in, each in a job of its own with vLLM measured again. Means of three server starts; ratios are EpiLoRA’s over vLLM’s. Median TPOT (ms) ↓

Accuracy (%) ↑

4

0.95 0.98 1.37 1.08 1.03 0.93 1.07 1.06 1.13 1.13 6 1 2 4 6

GPU

Table 22. EpiLoRA serves adapters at the accuracy of vLLM’s LoRA path. Two RL adapters of Qwen3.5-9B on an FP8 base, greedy decoding, on MATH-500 (500 problems), OlympiadBench (675) and Minerva (272): accuracy with each system, the problems only one of them solves (vLLM/EpiLoRA), the difference (EpiLoRA’s accuracy minus vLLM’s, in points) ± the half-width of its paired 95% interval, and McNemar’s exact p.

2

0.77 0.77 0.86 0.90 0.95 1.01 1 2 4

rank 64 rank 16 L40S, requests/s

H100

H100, new tiles

H100, new tiles, own kernels

p

H200

90.4 14/9 −1.0 ± 1.9 0.40 63.4 36/47 +1.6 ± 2.6 0.27 54.4 7/7 +0.0 ± 2.7 1.00

L40S

p99 TTFT (ms) ↓

Rate vLLM EpiLoRA Ratio vLLM EpiLoRA Ratio 2 4 8 16 2 4 8 16 2 4 8 16 2 4 8 16 1 2 4 6

9.9 15.5 28.9 54.8 9.9 15.5 29.0 54.9 9.9 15.5 29.0 54.9 8.8 12.0 21.2 46.6 23.7 32.0 51.4 86.9

7.9 13.0 29.0 64.7 7.7 12.5 27.7 61.1 7.5 11.8 24.9 58.6 6.5 8.9 17.6 43.1 21.9 31.1 52.2 88.5

0.79 0.84 1.00 1.18 0.78 0.81 0.95 1.11 0.76 0.76 0.86 1.07 0.74 0.74 0.83 0.93 0.93 0.97 1.02 1.02

213 238 268 360 214 243 269 361 214 243 269 361 96 120 166 270 235 255 332 3,062

254 274 305 400 242 262 289 386 213 240 266 386 112 138 164 243 241 262 339 4,054

1.20 1.15 1.14 1.11 1.13 1.08 1.08 1.07 1.00 0.99 0.99 1.07 1.17 1.14 0.99 0.90 1.03 1.03 1.02 1.32

90.2 13/19 +1.2 ± 2.2 0.38 58.5 40/41 +0.1 ± 2.6 1.00 55.9 8/9 +0.4 ± 3.0 1.00

The decomposition on a second model (Table 33). On Qwen3-4B, a dense model, with five of its math RL adapters, the ordering of Figure 7 holds: rounding activations leaves 6.3–7.4× the bf16 pair’s KL, most of it the activations’ rounding itself (the fake-quant control), and the unrounded engines retain 1.1–1.6×.

The mismatch over training (Table 34). Two hundred steps cannot show a collapse that comes later, but the mismatch’s trend can: with the matched engine and no correction it stays at the level of a bf16 engine and trainer, while uncorrected W8A8 rollouts more than double theirs. Where the GPU-hours go on the H100 (Table 35). Generation takes most of each step, and the matched engine’s extra cost on the H100 is almost all in generation.

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 25. At vLLM’s default CUDA-graph sizes, EpiLoRA lowers the time per output token with 2,048-token prompts and on a 72B model. Top: Qwen3.5-9B with an FP8 base, 32 rank-64 tenants, Poisson arrivals, 2,048-token prompts and 256 output tokens. Middle: Qwen2.5-72B-Instruct on one H200 with 8 tenants and 512-token prompts. Bottom: the 72B across two H100s with tensor parallelism and 8 tenants, against vLLM’s kernels tuned for the shards. vLLM with tuned LoRA kernels against EpiLoRA with direct launches; means of three server starts, whose time-per-token ratios agree within 0.03. Median TPOT (ms) ↓ GPU Requests/s vLLM

ours

p99 TTFT (ms) ↓

ratio vLLM ours

Qwen3.5-9B, 32 tenants, 2,048-token prompts H200 1 8.00 5.88 0.73 175 2 9.79 7.08 0.72 235 4 16.70 11.09 0.66 338 H100 1 8.08 6.41 0.79 242 2 10.02 7.84 0.78 295 4 16.89 13.04 0.77 402

ratio

173 210 277 250 297 384

0.99 0.90 0.82 1.03 1.01 0.95

Qwen2.5-72B, 8 tenants, 512-token prompts H200 0.5 35.71 34.75 0.97 422 416 1 41.84 39.70 0.95 541 505 2 59.30 54.57 0.92 703 694

0.99 0.93 0.99

Qwen2.5-72B at TP2 on two H100s, vLLM tuned for the shards H100 0.5 23.91 20.78 0.87 219 249 1.14 1 26.14 23.26 0.89 272 280 1.03 2 31.66 28.57 0.90 641 368 0.57

Table 26. Host and GPU time per layer for the steps that run outside CUDA graphs. One layer of Qwen3.5-9B (its four projections) with 32 adapter slots at the batch sizes of vLLM’s eager steps, microseconds: the host time to launch each path (calls queued behind a GPU sleep, so the GPU never throttles them) and its GPU time (CUDA-graph replays). The H100’s host CPU (Intel Xeon Gold 6448Y) pays about twice the host time per call of the H200’s (AMD EPYC 9335). Direct launches halve EpiLoRA’s host time and give bit-identical outputs. H100

H200

L40S

Path, µs ↓

512 tokens 2,048 512 tokens 2,048 512 tokens 2,048 host GPU host GPU host GPU host GPU host GPU host GPU

vLLM’s quantize, GEMM and LoRA kernels the vendor GEMM alone ours, hybrid ours, hybrid, direct launches ours, fused ours, fused, direct launches

648 916 136 208 541 870 245 873 350 1699 106 1700

639 2211 138 732 532 1769 243 1768 345 3921 106 3919

299 982 68 215 260 875 125 876 174 1804 59 1802

287 2332 69 730 258 1756 125 1756 171 4164 59 4142

460 1579 101 713 405 1689 180 1686 265 1554 80 1559

454 4320 100 2952 402 4211 181 4354 259 3490 81 3474

Table 27. Where the time of a step outside CUDA graphs goes. One four-tenant step of Qwen3.5-9B that prefills 2,048 tokens (FP8 base, 32 rank-64 adapter slots), above vLLM’s default CUDA-graph sizes, so it runs outside the graphs; median step time over 20 steps, and the host (CPU) time and GPU time of one profiled step by kernel class, milliseconds. EpiLoRA: its default configuration, whose timed dispatch may pick vLLM’s adapter kernels inside its operator; own kernels: EpiLoRA restricted to its own adapter kernels above 512 tokens; new tiles: the same with the down-projection’s tiles searched for large batches. ms ↓ H100 H200 vLLM EpiLoRA own kernels own, new tiles vLLM EpiLoRA own kernels own, new tiles Step time (ms) Host time, profiled (ms) GPU time (ms) CUTLASS FP8 GEMM down-projection (shrink) up-projection (expand) activation quantization and norms everything else

66.7 75.3 65.6 22.6 5.0 20.9 11.5 5.6

71.0 88.2 60.6 22.7 5.2 17.5 9.6 5.7

71.7 74.3 67.3 22.7 11.5 17.8 9.8 5.5

67.4 71.9 62.9 22.7 6.8 18.1 9.9 5.5

65.5 65.7 68.0 22.4 5.6 22.7 11.9 5.3

61.4 59.2 59.0 22.3 6.8 16.0 8.6 5.3

68.6 66.2 66.5 22.4 13.0 17.2 8.7 5.2

64.5 62.8 61.1 22.4 7.3 17.4 8.8 5.2

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 28. Across two H100s, EpiLoRA leads vLLM’s kernels tuned for the tensor-parallel shards at batch 1. Qwen2.5-72B-Instruct with vLLM’s online FP8 at two-way tensor parallelism, one random rank-64 adapter; W1: time per output token at batch 1 (mean of three starts and their range) and its increase over the engine without LoRA (16.39 ms); W2: throughput with one adapter (without LoRA 2,877 tokens/s). EpiLoRA runs the decode path on tiles searched for the shards; with 8 tenants vLLM delivers 1,567 tokens/s against EpiLoRA’s 1,549. Path

W1 (ms) ↓ over no LoRA ↓ W2 (tok/s) ↑

vLLM, tuned 20.40 [20.31, 20.54] vLLM, dual, tuned 20.31 [20.25, 20.34] EpiLoRA, searched 17.75 [17.70, 17.77]

+24% +24% +8%

2,141 2,017 2,124

Table 29. On one H200, EpiLoRA’s timed dispatch serves the 72B fastest at batch 1. Qwen2.5-72B-Instruct with vLLM’s online FP8, random rank-64 adapters, vLLM’s LoRA kernels tuned for this model on this GPU; W1 as in Table 28 (without LoRA 23.35 ms), W2 with one adapter and with 8 (without LoRA 1,320 tokens/s). Forcing the decode path on every projection is 1.03× slower than the timed dispatch, which serves a third of the projections with vLLM’s kernels. An earlier job with the same configurations measured 28.00 against 26.80 ms at batch 1 (0.96×, the value in Table 2) and, with 8 tenants, 985 against 951 tokens/s. Path

W1 (ms) ↓ over no LoRA ↓ W2, one (tok/s) ↑ W2, 8 (tok/s) ↑

vLLM, tuned for the 72B 28.20 [28.06, 28.27] vLLM, tuned, two streams 29.18 [29.17, 29.19] EpiLoRA, timed dispatch (searched tiles) 26.50 [26.48, 26.51] EpiLoRA, decode path forced 27.32 [27.31, 27.32]

What each part costs and buys (Tables 36 and 37; Figure 9 shows the memory of each placement).

+21% +25% +13% +17%

1,042 994 1,038 1,041

980 926 974 975

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 30. Every LoRA-RL configuration on Qwen3.5-9B: its configuration, cost and accuracy against its reference (continued in Table 31). Arms grouped by setting, each group named with its reference. Accuracy: OlympiadBench and Minerva pooled, greedy, each policy on its own engine. Engines: W8A8, vLLM’s FP8 engine with EpiLoRA; W8A16, vLLM’s weight-only FP8 engine (Marlin) or EpiLoRA’s matched engine; asleep, sleep mode during training. Trainers: bf16, or on the engine’s FP8 weights, dequantized on every call (deq.) or through our kernels (ours, with flash-linear-attention, FLA). ∆: final accuracy minus the reference’s, Welch 95% interval; ≤ pre-RL: runs that ended at or below their accuracy before RL. Code

Engine

Trainer, correction

GPU

Runs GPU-h ↓

Accuracy (%) pre-RL

∆ (points) ↑

final [95% CI]

Qwen3.5-9B math; against S0G S0G bf16 bf16 H100a 5 6.9 49.8–50.5 57.4–61.8 reference S0T bf16, asleep bf16 H100 3 5.6 50.5 61.0–61.7 +1.5 [−0.4, +3.5] S8G W8A8 FP8 (deq.) H100 5 5.6 50.6–51.5 2.3–60.7 −12.9 [−43.9, +18.0] S4F W8A8 bf16 H100 3 5.3 49.7–51.0 4.3–52.4 −32.7 [−92.4, +27.0] S8T W8A8 FP8 (deq.), TIS H100 4 5.7 50.6–51.0 57.6–61.0 −0.4 [−2.8, +2.1] S0TF bf16, asleep bf16 + FLA H100 9 3.2 49.8–50.5 58.2–63.5 +0.9 [−1.2, +2.9] S8C W8A8 FP8 (ours), centering H100 4 4.0 49.9–50.6 57.1–62.8 +0.7 [−2.9, +4.4] S8W W8A16, Marlin FP8 (ours) H100 4 3.9 49.5–50.4 58.9–62.1 +0.3 [−2.2, +2.7] S8TL W8A8 FP8 (deq.), TIS L40S 3 13.4 49.9–50.5 60.5–60.9 +1.0 [−1.0, +2.9] S8ML W8A16, matched FP8 (ours) L40S 3 8.8 50.2–50.6 60.0–62.0 +0.9 [−1.5, +3.3] S8WA W8A16, Marlin FP8 (deq.) + FLA A100 8 6.1 49.2–50.6 58.8–63.4 +0.8 [−1.3, +2.8] Qwen3.5-9B math, three jobs on one engine and trainer base (S9P pipelined, S9B batched); against S0G S9P W8A8 FP8 (deq.) H100 3 5.7 50.7–51.5 56.2–58.5 −2.3 [−4.8, +0.1] S9B W8A8 FP8 (deq.) H100 3 5.8 49.6–50.2 47.7–59.7 −6.3 [−20.4, +7.8] Qwen3.5-9B math, fast trainers; against S0TF S8TF W8A8 FP8 (ours), TIS H100 9 3.1 49.4–51.0 57.7–62.1 −0.3 [−2.0, +1.3] S4TF W8A8 bf16 + FLA, TIS H100 9 3.0 50.2–51.0 60.1–62.8 +0.6 [−0.9, +2.0] S8MH W8A16, matched FP8 (ours) H100 9 4.2 50.7–50.8 58.2–64.4 +0.1 [−1.6, +1.9] S8UF W8A8 FP8 (ours) H100 6 3.1 49.9–51.5 54.2–57.2 −4.9 [−6.5, −3.3] Qwen3.5-9B math, fast trainer on the L40S; against S8TL S8TLF W8A8 FP8 (ours), TIS L40S 3 9.3 49.6–50.5 57.7–60.2 −2.0 [−5.1, +1.1] Qwen3.5-9B math with offload; against S0OL S0OL bf16 bf16 + FLA L40S 7 8.4 49.6–51.1 59.9–64.8 reference S8OL W8A8 FP8 (ours), TIS L40S 7 7.2 49.5–50.7 58.1–62.4 −1.1 [−3.1, +0.9] S4OL W8A8 bf16 + FLA, TIS L40S 3 7.2 50.2–50.8 59.2–63.6 −0.3 [−4.6, +4.1] Qwen3.5-9B math on the L40S without correction (S8RA: activations rounded alike); against S0OL S8RA W8A8 FP8 W8A8 (ours) L40S 3 8.2 49.6–51.3 51.1–60.5 −6.5 [−17.5, +4.5] S8ULF W8A8 FP8 (ours) L40S 3 8.5 49.5–50.8 52.5–60.1 −6.3 [−15.4, +2.8] a

Seeds 3–4 ran on A100s; GPU-hours over the H100 seeds.

≤ pre-RL 0/5 0/3 1/5 2/3 0/4 0/9 0/4 0/4 0/3 0/3 0/8 0/3 1/3 0/9 0/9 0/9 0/6 0/3 0/7 0/7 0/3 1/3 0/3

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 31. Every LoRA-RL configuration (continued from Table 30): the 27B, one copy shared by the engine and the trainer, and Qwen3-4B. Columns as in Table 30; each block’s header names its benchmarks and, for the 27B, the accuracy before RL. The 27B is scored on the bf16 engine (on the L40S also on OlympiadBench and Minerva only, as the 9B), except where a header says otherwise. One FP8 copy of the 27B with its KV cache kept ran out of memory on the L40S. Code

Engine

Trainer, correction

GPU

Runs GPU-h Accuracy (%) ∆ (points) ↑ ↓

final [95% CI]

≤ pre-RL

Qwen3.8-27B math, MATH-500, OlympiadBench and Minerva (before RL 61.6–62.8%); against Q27 Q27 bf16 bf16 2×A100 3 15.7 70.8–74.6 reference 0/3 S8T1g W8A16, Marlin FP8 (deq.), TIS A100 3 15.3 71.9–73.3 −0.4 [−4.4, +3.6] 0/3 S8T27H W8A8 FP8 (ours), TIS H100 2 + 1b 4.4 72.4–73.5 +0.2 [−3.9, +4.2] 0/2 S0T27A bf16, offloaded bf16 + FLA A100 3 7.3 71.2–72.0 −1.2 [−5.6, +3.1]c 0/3 S8T27A W8A16, Marlin, offloaded FP8 (deq.) + FLA, TIS A100 3 6.5 70.6–72.0 −1.3 [−5.3, +2.7]c 0/3 S8S27H W8A8 FP8 (ours), one copy, TIS H100 5 3.3 68.1–72.6 −2.5 [−6.2, +1.2] 0/5 Qwen3.8-27B math on the H100, the same benchmarks; against S0T27HO S0T27HO bf16, offloaded bf16 + FLA H100 3 4.0 70.4–72.6 reference 0/3 S8T27HO W8A8, offloaded FP8 (ours), TIS H100 3 3.4 70.6–73.7 +0.7 [−2.5, +3.9] 0/3 S4T27HO W8A8, offloaded bf16 + FLA, TIS H100 3 3.2 70.7–72.4 −0.1 [−2.4, +2.3] 0/3 Qwen3.8-27B math on the H100, the same benchmarks, one copy, KV cache released for training; against S8T27HO S8K27H W8A8 FP8 (ours), one copy, TIS H100 3 3.3 70.9–73.6 −0.1 [−3.5, +3.3] 0/3 Qwen3.8-27B math on one L40S, OlympiadBench and Minerva (before RL 49.7–51.1%); against Q27 on the same benchmarks Q27 bf16 bf16 2×A100 3 15.7 59.7–63.7 reference 0/3 S8T27L W8A8, offloaded FP8 (ours), TIS L40S 3 11.5 58.7–63.0 −0.9 [−5.7, +3.8] 0/3 Qwen3.8-27B math on one L40S, the same benchmarks on each run’s own FP8 engine, one copy, KV cache released; against S8T27L S8K27L W8A8 FP8 (ours), one copy, TIS L40S 3 11.2 57.8–61.1 +0.0 [−3.9, +3.9] 0/3 Qwen3.5-9B math on the L40S, one copy; against S8TLF (two copies) S8S9L W8A8 FP8 (ours), one copy, TIS L40S 3 7.1 60.4–61.7 +2.4 [−0.2, +5.1] 0/3 Qwen3.5-9B math on the L40S, one copy, matched engine; against S8ML (two copies) S8MS9L W8A16, matched FP8 (ours), one copy L40S 5 7.3 56.1–60.1 −2.3 [−4.7, +0.1] 0/5 Qwen3.5-9B math on the L40S, one copy, KV cache released for training; against S8OL S8K9L W8A8 FP8 (ours), one copy, TIS L40S 3 7.1 56.6–61.8 −0.3 [−5.8, +5.1] 0/3 Qwen3-4B math; against S0L S0L bf16 bf16 L40S 3 4.5 49.7–50.2 reference 0/3 S8L W8A8 FP8 (deq.) L40S 3 3.6 49.7–51.4 +0.7 [−1.3, +2.7] 0/3 b

A run that stopped early and was not resumed (seed 2: out of memory at step 88).

c

No reference run.

Table 32. Equivalence of the RL configurations with their references, three ways. Final accuracy (OlympiadBench and Minerva pooled) of each configuration minus its reference: on each policy’s own engine, the Welch 95% interval over all runs and the t interval of the per-seed differences over the seeds both ran (a seed fixes the problem order); and the Welch interval with every policy re-scored on one common bf16 engine (avg@8). p̸=0 : Welch’s test of a zero difference; p±5 : the two one-sided tests of the ±5-point margin; both Holm-corrected over these twelve comparisons (a family chosen after the data). Runs: the configuration’s and the reference’s. Each policy’s own engine Comparison

GPU Runs Welch 95% interval

W8A8 + TIS against tuned bf16 H100 FlashRL’s recipe against tuned bf16 H100 matched engine against tuned bf16 H100 W8A8 + TIS, offloaded, against bf16 offloaded L40S FlashRL’s recipe, offloaded, against bf16 offloaded L40S one copy against two copies (W8A8 + TIS) L40S one copy against two copies (matched) L40S one copy, KV released, against offloading (W8A8 + TIS) L40S two copies (W8A8 + TIS) against bf16 offloaded L40S one copy (W8A8 + TIS) against bf16 offloaded L40S two copies (matched) against bf16 offloaded L40S one copy (matched) against bf16 offloaded L40S

p̸=0

Paired by seed p±5 Seeds interval

9/9 −0.3 [−2.0, +1.3] 1.000 <0.001 9/9 +0.6 [−0.9, +2.0] 1.000 <0.001 9/9 +0.1 [−1.6, +1.9] 1.000 <0.001 7/7 −1.1 [−3.1, +0.9] 1.000 0.005 3/7 −0.3 [−4.6, +4.1] 1.000 0.106 3/3 +2.4 [−0.2, +5.1] 0.601 0.108 5/3 −2.3 [−4.7, +0.1] 0.601 0.106 3/7 −1.1 [−6.8, +4.6] 1.000 0.151 3/7 −3.0 [−5.5, −0.5] 0.305 0.151 3/7 −0.6 [−2.4, +1.3] 1.000 0.002 3/7 −1.1 [−3.3, +1.2] 1.000 0.017 5/7 −3.4 [−5.7, −1.1] 0.108 0.151

One bf16 engine, avg@8 Welch 95% interval p̸=0

p±5

9 −0.3 [−2.6, +1.9] −0.3 [−1.6, +1.1] 1.000 <0.001 9 +0.6 [−1.3, +2.4] +0.5 [−0.6, +1.6] 1.000 <0.001 9 +0.1 [−1.3, +1.5] +0.1 [−1.4, +1.7] 1.000 <0.001 7 −1.1 [−3.5, +1.3] −0.3 [−2.0, +1.5] 1.000 <0.001 3 −1.2 [−8.6, +6.3] −0.1 [−3.8, +3.6] 1.000 0.046 3 +2.4 [+0.3, +4.5] +2.1 [−0.3, +4.4] 0.748 0.046 3 −2.8 [−5.3, −0.3] −2.3 [−5.3, +0.7] 0.902 0.046 3 −0.3 [−4.2, +3.6] −1.6 [−4.5, +1.3] 1.000 0.046 3 −3.9 [−11.3, +3.5] −1.6 [−3.8, +0.6] 1.000 0.026 3 −1.5 [−9.1, +6.1] +0.5 [−1.1, +2.0] 1.000 <0.001 3 −2.0 [−6.5, +2.5] +0.4 [−2.4, +3.1] 1.000 0.026 5 −3.6 [−5.8, −1.5] −1.9 [−3.5, −0.3] 0.304 0.006

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 33. On a second, dense model the decomposition’s ordering holds. Qwen3-4B, five math RL adapters (bf16- or W8A8-trained), 128 DAPO-Math prompts × 4 samples on one H100: each engine’s per-token KL to the weight-matched trainer over that of a bf16 engine and trainer; the fake-quant control’s share of the W8A8 engine’s excess; and a trainer that rounds its activations as the engine does, over the W8A16 trainer. KL over the bf16 pair’s ↓ Fake-quant Rounded alike Adapter

W8A8 matched W8A16 Marlin W8A16

bf16-trained, seed 0 bf16-trained, seed 1 bf16-trained, seed 2 W8A8-trained, seed 0 W8A8-trained, seed 1

6.47× 6.80× 7.41× 6.31× 7.16×

1.34× 1.49× 1.60× 1.30× 1.30×

1.35× 1.20× 1.51× 1.14× 1.38×

share over W16 trainer 0.76 0.74 0.86 0.79 0.76

1.35× 1.36× 1.50× 1.49× 1.37×

Table 34. Without correction, the matched engine’s mismatch stays at the bf16 level over training; W8A8 rollouts’ grows. Qwen3.59B on the H100, the nine-seed configurations of Figure 13 and uncorrected W8A8 rollouts; mean absolute difference between the trainer’s and the engine’s log-probabilities of the sampled tokens, averaged over each window of steps and then over runs (descriptive, from the runs’ step logs). TIS clipped 0.08% of the tokens’ importance weights in the first ten steps and 0.16% in the last twenty with FlashRL’s recipe, and 0.01% and 0.06% with the FP8 trainer (at most 0.31% in any step). Mismatch, mean |∆ log p|, at steps (↓) Configuration

Runs

bf16 engine and trainer FlashRL’s recipe (W8A8, bf16 trainer, TIS) W8A8, FP8 trainer, TIS matched engine, no correction W8A8, FP8 trainer, no correction

181–200 Late/early ↓

1–10 91–110

9 0.0055 9 0.0272 9 0.0173 9 0.0070 6 0.0174

0.0061 0.0308 0.0192 0.0068 0.0244

0.0064 0.0348 0.0220 0.0075 0.0394

1.18× 1.28× 1.27× 1.07× 2.26×

Table 35. On the H100, the matched engine’s extra GPU time is almost all in generation. GPU-hours per 200-step run of the nine-seed H100 arms of Figure 13, split into generation and the rest of each step (training and the adapter’s sync), from the runs’ step logs. Configuration bf16 engine and trainer FlashRL’s recipe W8A8 with TIS, FP8 trainer matched engine, FP8 trainer

Runs Generation ↓ Training ↓ Total ↓ Generation share 9 9 9 9

2.00 1.77 1.74 2.81

1.22 1.19 1.36 1.39

3.22 2.96 3.10 4.20

62% 60% 56% 67%

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 36. Colocated LoRA RL on one GPU, one part at a time: FP8 weights let two copies fit, and one shared copy returns memory to the KV cache. Means over runs: where the trainer’s base waits during rollouts, the engine’s memory share and KV cache, seconds per step generating and training, GPU-hours per run, and final accuracy minus the block’s bf16 run with the Welch 95% half-width. a The engine releases its KV cache while training; b one copy that keeps its KV cache. The 27B’s two-copy (6 runs) and one-copy (7 runs) rows pool every completed run; Table 31 lists the runs. Trainer’s base Engine

Trainer

Engine

∆ acc.

Seconds per step

Corr. Where GiB ↓ Share KV (GiB) ↑ Rollout ↓ Train ↓ GPU-h ↓ Runs (points) ↑

Qwen3.5-9B on one L40S (48 GB), 200 steps bf16 bf16 none GPU 16.7 out of memory at every engine share tried bf16 bf16 none host 16.7 0.80 15.3 105 46 W8A8 bf16 TIS host 16.7 0.80 22.7 84 47 W8A8 FP8 (ours) TIS host 10.2 0.80 21.3 84 47 W8A8 FP8 (ours) TIS GPU 10.2 0.45 5.8 123 45 W8A8b FP8 (ours) TIS shared 0.0 0.55 10.3 84 44 W8A8 FP8 (ours) TIS shareda 0.0 0.80 21.4 84 45 W8A16 (ours) FP8 (ours) none GPU 10.2 0.45 7.7 114 44 W8A16 (ours)b FP8 (ours) none shared 0.0 0.55 12.1 86 45 Qwen3.8-27B on one H100 (80 GB), 100 steps bf16 bf16 none GPU 50.1 does not fit: 101 GiB of weights bf16 bf16 none host 50.1 0.85 12.6 110 33 W8A8 FP8 (ours) TIS host 27.4 0.85 34.8 83 39 W8A8 FP8 (ours) TIS GPU 27.4 0.48 5.5 123 35 W8A8b FP8 (ours) TIS shared 0.0 0.65 19.5 82 35 W8A8 FP8 (ours) TIS shareda 0.0 0.85 35.3 82 35

8.4 7.2 7.2 9.3 7.1 7.1 8.8 7.3

7 61.8 % 3 −0.3 ± 4.3 7 −1.1 ± 2.0 3 −3.0 ± 2.5 3 −0.6 ± 1.8 3 −2.2 ± 5.5 3 −1.1 ± 2.3 5 −3.4 ± 2.3

4.0 3.4 4.4 3.3 3.3

3 71.4 % 3 +0.7 ± 3.2 6 +0.0 ± 2.2 7 −0.3 ± 2.4 3 +0.6 ± 2.9

Table 37. With 2,048-token rollouts, one bf16 copy that keeps its KV cache trains as fast as one FP8 copy on the 9B; on the 27B the FP8 copy is 1.33× faster. 12 steps of the RL recipe with one bf16 copy shared by engine and trainer that keeps its KV cache while training (Unsloth releases it) against the same steps of full runs with one FP8 copy and with bf16 offloading; seconds per step over steps 3–12 and the run’s peak host memory. Seconds per step Model, GPU

Copy

Share KV (GiB) ↑ Rollout ↓ All ↓ Host peak (GiB) ↓

Qwen3.5-9B, L40S one bf16 Qwen3.5-9B, L40S one bf16 Qwen3.8-27B, H100 one bf16

0.55 0.70 0.80

5.5 12.2 9.1

187 120 136

245 174 165

8.6 10.1 14.2

Qwen3.5-9B, L40S one FP8 Qwen3.5-9B, L40S bf16, offloaded Qwen3.8-27B, H100 one FP8 Qwen3.8-27B, H100 bf16, offloaded

0.55 0.80 0.65 0.85

10.3 15.3 19.5 12.6

122 127 94 129

173 182 124 161

7.4 48.2 10.6 159.5

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL Table 38. Of the placements we ran, only FP8 trainers fit the 27B on a 48 GB GPU, and with long rollouts one FP8 copy runs close to offloading’s speed without its host memory. Seconds per RL step (bold: fastest): the 27B’s 100-step runs on a 48 GB L40S at 2,048-token rollouts (mean over three seeds) and 12-step probes in thinking mode with rollouts of up to 8,192 or 16,384 tokens (steps 3–12). OOM: out of GPU memory (at 16,384 tokens the bf16 trainers’ runs stopped before their first or, for FlashRL’s offloaded recipe, their second step; both FP8 trainers ran all 12 steps, their mean completion length reaching 10.7–10.8k tokens). d Rerun with chunked log-probs, the LM head computed in recomputed chunks so that the full logits never exist: FlashRL’s offloaded recipe then ran all 12 steps, at 73 GiB of GPU and 127 GiB of host memory; its responses trained fewer tokens per step (176k against 265–287k), so its step time is not like for like; bf16 offloading with chunked log-probs still ran out of GPU memory in its third step. All in our trainer, which keeps activation checkpoints and a micro-batch’s logits on the GPU. Host, training: the 27B run’s median host memory over its steps 2–4 on the H100 (Table 6 gives each placement’s peak). One bf16 copy runs out of memory at engine share 0.70 as shown; at 0.55 it fits the 9B at 8,192 tokens, at 2,216 s per step (Table 6). FlashRL: vLLM’s FP8 engine beside a bf16 trainer, with TIS. Unsloth’s FP8 standby shares one FP8 copy as the last row does, with a trainer that rounds activations alike; its bf16 standby needs at least bf16 offloading’s GPU memory while it trains. a Not run: it needs more GPU memory than a placement that ran out of it. b From the probe’s own trace. c Not run: its copies of the base alone exceed the GPU’s memory. n/r: not run. Over full 100-step runs at 8,192 tokens on an H100 (Table 7) the final accuracies (OlympiadBench and Minerva, non-thinking, 2,048 tokens) are: one FP8 copy 50.6 and 52.1 (two seeds), FlashRL’s offloaded recipe 51.0, FP8 offloading 51.6, bf16 offloading 51.1, against 51.1 for the 27B before RL: these runs change accuracy little at this evaluation, so they compare cost, not learning. Host, training Placement of the trainer’s base FlashRL: FP8 engine, bf16 trainer, two copies FP8 engine and trainer, two copies bf16, one shared copy, KV cache kept bf16, trainer’s base offloaded (veRL-style) FlashRL, trainer’s base offloaded FP8, trainer’s base offloaded FP8, one shared copy FP8, one copy, KV cache released (EpiLoRA)

48 GB L40S

80 GB H100, the 27B

27B (GiB) ↓

27B, 2,048 ↓

9B, 8,192 ↓

8,192 ↓

16,384 ↓

– 9 6b 159 125b 93 10 8

c

a

a

×a ×a OOM OOM 606d 673 n/r 701

× ×c ×c ×c ×c 415 OOM 404

× OOM OOM 1314 1076 1025 1167 1012

× OOM OOM 659 421 418 476 422

Table 39. The long-rollout probes behind Table 6. Per placement: seconds per step and generation seconds per step (steps 3–12; a steps 3–11, engine at 55% of the GPU), mean response length, trained tokens per step (responses in groups with zero reward variance are not trained) and the engine’s KV cache. Bold: best within each model, and any within 2% of it. Placement 9B, one L40S One bf16 copy, KV kepta bf16, trainer offloaded FlashRL, trainer offloaded EpiLoRA, FP8 trainer offloaded EpiLoRA, one FP8 copy 27B, one H100 bf16, trainer offloaded FlashRL, trainer offloaded EpiLoRA, FP8 trainer offloaded EpiLoRA, one FP8 copy

s/step ↓ generation (s) ↓ response (tokens) trained tokens KV (GiB) ↑ 2,216 1,314 1,076 1,025 1,012

1,886 963 732 725 722

7,992 7,902 7,906 7,888 7,795

527k 558k 549k 544k 531k

4.2 15.3 21.4 21.4 21.4

659 421 418 422

521 286 291 290

5,575 5,549 5,549 5,545

244k 248k 206k 207k

12.6 34.8 34.8 34.8

EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

G

S OFTWARE AND HARDWARE

Tuning vLLM’s LoRA kernels. For each model and GPU, we record the shape of every shrink and expand call of a serving run and time, for each shape, rank (64, for some models also 16), slot count (1 and 32; 8 for the 72B) and token count (1 to 8,192), vLLM’s default configuration and the 12 to 47 others that run (block sizes, split-K, warps) with CUDA graphs and the L2 cache flushed; we keep the fastest whose output matches the default’s, a median 1.2–1.4× faster than the default over 140 to 400 keys per model and GPU. vLLM looks the files up by GPU name: for Qwen3.5-9B, the H200 and GH200 use the H100’s, as we did not tune on those GPUs (on Qwen3-8B, tuning on the H200 itself kept the H100’s configuration for 112 of 160 keys). Software: vLLM 0.28.0, PyTorch 2.13.0, Triton 3.7.1, FlashInfer 0.6.16, Transformers 5.16.1, SGLang 0.5.9 and

TensorRT-LLM in their own environments; CUDA 12.8 and 13.1. Hardware: NVIDIA L40S (48 GB, Ada); H100 SXM (80 GB), H200 NVL (141 GB) and GH200 (96 GB, Hopper); A100 (80 GB, RL only); a 2g.48gb MIG slice of an RTX PRO 6000 (Blackwell).

Related documents

• Scute: Exact Batched Serving for PEFT Adapters Beyond LoRA#1536898
• How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions#986271
• Spreading Factor Assignment Strategy for Coverage and Capacity Flexible Tradeoff#1061068
• Spreading Factor Assignment Strategy for Coverage and Capacity Flexible Tradeoff#1043057
• Load Balancing Under Adaptive Bin Deletions / Haim Kaplan, Shay Sapir, Uri Stemmer#1188572