SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads

To appear in 2027 IEEE International Symposium on High-Performance Computer Architecture (HPCA)

arXiv:2611.14776v1 [cs.AR] 10 Nov 2026

SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads Anish Saxena*

Hritvik Taneja

Christos Kozyrakis

Aamer Jaleel

Georgia Tech Atlanta, GA, USA

Georgia Tech Atlanta, GA, USA

NVIDIA Research & Stanford University Santa Clara, CA, USA

NVIDIA Research Westford, MA, USA

Memory vendors are developing new technologies that scale both memory bandwidth and capacity, such as HighBandwidth Flash (HBF). HBF replaces the DRAM dies in an HBM stack with NAND flash to provide up to 16× the capacity of HBM at comparable bandwidth [12], [13]. This additional capacity comes with four important limitations: high read latency, low write bandwidth, high power consumption, and limited write endurance. The write-bandwidth and endurance constraints can be avoided by using HBF for readonly data. LLM weights are therefore particularly well suited for placement in HBF in proposed HBM-HBF systems [13], [14]. Doing so also frees up HBM capacity to store more KV state and improve decoding throughput, which can improve energy efficiency (e.g., lower EDP or higher throughput per watt). Thus, HBF offers the potential to provide substantially greater GPU memory capacity and improve LLM inference. Read latency, however, remains a fundamental obstacle to using HBF in GPUs. An HBF round-trip access can incur 20× the latency of HBM, 20µs compared to about 1µs on current GPUs [13]. By Little’s Law, sustaining a fixed bandwidth across a 20× longer latency requires 20× more traffic in flight, which current GPUs cannot maintain. In our simulations, replacing HBM with iso-bandwidth HBF on A100 GPU slows inference of Llama-8B by 7.0× and Llama-70B by 11.1×. Increasing in-flight traffic requires a corresponding increase in miss-tracking state (Fig. 1(a)). Typical miss-status holding registers (MSHRs) perform associative lookup, making a proportional increase expensive in both area and complexity. Moreover, increasing tracking state alone does not solve the HBF latency problem: the GPU must also generate enough independent memory requests to keep the enlarged memory pipeline full. Existing SMs do not expose this degree of demand-side memory-level parallelism (e.g., due to limited shared memory), and high-end GPUs already approach reticlescale die limits, leaving little opportunity to increase SMs and MSHRs merely to accommodate flash latency. A long-standing approach to tolerating memory latency is to prefetch data before the corresponding demand requests arrive. Unfortunately, conventional GPU prefetchers issue requests at cache-line granularity and do not scale to the substantially long HBF latency. Extending these prefetchers to HBF would still require proportionally larger MSHRs and request queues for tracking, along with more buffering for returned data. Recent HBF proposals instead prefetch entire tensors by streaming their pages sequentially into a staging buffer [13].

Abstract—Large language model (LLM) inference increasingly requires memory capacity for larger models and batch sizes. Emerging High-Bandwidth Flash (HBF) technology replaces HBM’s DRAM dies with NAND flash, providing up to 16× the capacity at comparable read bandwidth. However, HBF incurs a nearly 20 µs read latency, 20× that of HBM, causing on-demand HBF accesses to slow inference by up to 12.8×. Conventional cache-line prefetchers require 20× more in-flight tracking state to cover HBF latency, while tensor-streaming designs such as H 3 follow virtual-address rather than kernel-demand order and therefore need hundreds of megabytes of buffering to perform similarly to an idealized HBF system with HBM-like latency. We propose SlipStream, a software–hardware co-designed prefetcher that turns prefetching into a demand-paced producer– consumer staging pipeline. SlipStream’s software component profiles the repeatable per-CTA weight-page demand order of data-independent LLM kernels and inserts a staging warp to replay this order with non-blocking STAGE instructions. SlipStream’s hardware component provides architectural support for the staging warp, treating it as the producer and the work warps as consumers. Our key insight is to retire each STAGE tracking entry only when demand first accesses the page, rather than when the page returns from HBF, so demand itself paces staging, bounding how far staging can lead consumption. SlipStream tracks 4 KB pages instead of cache lines, and allocates staging buffer slots only when pages return, thereby decoupling its size from outstanding traffic. Across seven dense models spanning 16–800 GB, SlipStream with a 40 MB staging buffer achieves 92.4% of ideal-HBF performance on average, whereas H 3 retains only 43.1% performance of ideal-HBF (i.e., 2.3× slowdown), even with a 320 MB staging buffer. SlipStream’s buffer sizing is independent of HBF latency and workload characteristics, and SlipStream with a small 16MB staging buffer—20× lower SRAM than H 3 —performs within 4% of SlipStream with 40MB buffer. Finally, SlipStream supports Mixture-of-Experts models and reaches 91.2% of ideal-HBF with a 40 MB buffer.

I. I NTRODUCTION Large language model (LLM) inference increasingly demands memory capacity without a corresponding demand for additional computation. Model weights alone can occupy hundreds of gigabytes; for example, Llama 3.1 405B requires approximately 405GB even with one-byte weights [1]. Longcontext and high-concurrency serving further increase the memory footprint through key-value (KV) caches [2]–[5]. Systems today obtain the required capacity by distributing inference across many GPUs [6]–[9]. During bandwidth-bound decoding, however, much of the compute provided by these additional GPUs remains underutilized [10], [11]. * Anish can be reached at asaxena317@gatech.edu.

1

Little’s Law ~ B x L (B=2TB/s) HBM, L = 1 µs

2 MB

Prefetch stream (VA order) p1

p2 p3

4x

Runtime demand order HBF, L = 20 µs

40 MB

p3

p2

p4

Tensor prefetching

8x

p4

p1

One-time

Software

Slowdown

Profile, Bind

LLM

Access Stream

Architecture

GOAL

1x

Tracking, demand gen. ↑↑

Inaccurate

Untimely

1x 4x 8x Buffer SRAM (BxL=40MB)

(a) HBF’s high in-flight traffic

(b) Tensor prefetch problem

(c) Goal: Low overheads

STAGE instruction

SM Staging Warp Work warps

SSHR

4KB pages Staging buffer

(d) Our solution: SlipStream

Fig. 1. (a) Increasing load-to-use latency from 1 µs with HBM to 20 µs with HBF increases the required outstanding data by 20×. (b) Prefetching tensor pages by virtual address does not reproduce the order in which each kernel requests them, resulting in inaccurate and untimely prefetching. (c) Goal: minimize slowdown, tracking SRAM, and buffer SRAM. (d) SlipStream uses software guidance and hardware flow control to bound tracking and buffer SRAM.

This approach requires either a buffer large enough to retain the tensor or an accurate prefetch stream that matches the page-demand order. In practice, software tiling causes GPU kernels to consume tensor pages in an order that can differ substantially from their in-memory layout order (e.g., virtualaddress order), as Fig. 1(b) shows. With a smaller buffer than tensor’s footprint, demand might outrun the prefetch stream, resulting in exposed HBF latency. In our studies, such a design with 40 MB buffer leaves Llama-70B at 10.2× the execution time of an ideal-HBF system (same latency as HBM), and even a 320 MB buffer still incurs 2.3× slowdown.

Conventional prefetchers estimate a prefetch distance and tolerate occasional mistakes, but an uncovered HBF access can expose up to 20 µs of latency. SlipStream instead treats the staging warp as a producer and the work warps as consumers. Each STAGE allocates a page-granular Stage Status Holding Register (SSHR) entry, and a full SSHR backpressures only the staging warp. Unlike a conventional Miss Status Holding Register (MSHR), which releases its entry when the requested data returns, SlipStream retains each SSHR entry until the page’s first demand access. When a staged page serves its first demand, SlipStream releases the SSHR entry, enabling the staging warp to make progress. SSHR occupancy therefore measures the hardware-bounded producer–consumer gap and ties staging progress to consumer progress. Third, a page-granular software–hardware interface makes this long-latency staging pipeline inexpensive. At the evaluated 2 TB/s design point, a 20 µs lookahead spans approximately 40 MB of data—about 9,800 4 KB pages, or more than 300,000 128-byte cache lines. Tracking this window at cacheline granularity with MSHRs would be impractical. Fortunately, as HBF transfers 4 KB pages, one STAGE instruction’s tracking entry can represent an entire page, reducing required tracking entries by 32×. Moreover, we only use a 4KB slot in the staging SRAM buffer when the page actually arrives from HBF, and in-flight pages consume only compact tracking metadata. We make buffer eviction policy consumption-aware to shrink its size relative to volume of in-flight data. Across seven dense LLMs running on Ampere, Hopper, and Blackwell GPUs, replaying SlipStream’s profiled streams adds 1–3% runtime overhead on GPU silicon. In cycle-level simulation with these models (16–800GB in size) and a 40 MB staging buffer, SlipStream retains 92.4% performance of idealHBF, whereas the H 3 tensor prefetcher retains just 43.1% even with a 320 MB buffer. SlipStream also supports Mixtureof-Experts models. On two Qwen3 MoEs (60–470GB), it reaches 91.2% of ideal-HBF. Finally, demand-paced staging and consumption-aware buffering further allow SlipStream to work with a small 16MB buffer at just 12.6% slowdown. Overall, this paper makes the following contributions:

Our goal is to exploit the capacity and bandwidth of HBF for LLM inference at near-HBM performance with low tracking and SRAM overhead (Fig. 1(c)) while minimizing changes to GPU programming model. To this end, we propose SlipStream, a software–hardware co-designed prefetcher in which a software-profiled access stream runs ahead of demand warps, while demand-based feedback bounds the gap between producer and consumer. Software determines what pages to stage and in what order; hardware determines how far ahead staging may run and manages the returned data. We develop SlipStream, shown in Fig. 1(d), based on three insights. First, software constructs an accurate staging stream. The staging order must follow the kernel’s page-demand order, which can differ substantially from the tensor’s in-memory layout. We find that this demand order is repeatable for data-independent LLM kernels: they request weight pages in an identical per-CTA order across inputs and decode steps. Software can therefore profile this order once and cache it as a compact access stream. At runtime, a staging warp added to each CTA replays the stream as non-blocking STAGE instructions retrieving 4 KB pages from HBF. Because the staging warp resides in the same CTA as the consumer work warps, it is mapped to the same SM. Its nonblocking, page-granular STAGE instructions allow it to advance rapidly through the exact sequence that the work warps subsequently demand. Second, hardware keeps staging of access stream timely by pacing it with consumer progress. A page staged too late exposes HBF latency, whereas one staged too early occupies buffer space before its consumer arrives and may displace useful data; either outcome can stall an SM for microseconds.

•

2

We find that iso-bandwidth HBF requires 20× more outstanding data than HBM, beyond what GPUs can sustain.

We develop SlipStream’s software that profiles LLMs to produce access streams for runtime replay via staging warps. • We devise an efficient caching scheme so streams remain reusable across different LLMs and GPU architectures. • We architect SlipStream’s hardware, in which first-demand feedback paces the staging warp’s lead over demand. • We leverage demand-paced staging and buffer eviction to further reduce buffer size with negligible runtime overheads.

TABLE I M EMORY DEVICE PROPERTIES .

•

Property Capacity per stack Aggregate read bandwidth SM load-to-use latency Read granularity Outstanding data† Power consumption Aggregate write bandwidth Write endurance

II. BACKGROUND AND M OTIVATION A. LLM Inference and the GPU Memory Wall

†

LLM inference demands more memory capacity and bandwidth to increase model size and batch more requests. During inference, compute-bound prefill phase processes an input sequence in parallel, while the memory-bound decode phase generates one token at a time; modern deployments disaggregate the two phases [6], [10], [11], [15]. Each decode iteration of a dense model reads all the model weights irrespective of the batch size (§VII-B discusses Mixture-of-Experts [16]). So, the weight footprint and read traffic grow with model size [1], and require high-capacity, high-bandwidth memory.

HBM

HBF

16GB 2 TB/s ∼1 µs 32B line 2 MB ∼40W 2 TB/s ≥ 1015 accesses

256GB 2 TB/s ∼20 µs 4 KB page 40 M B ∼160W ∼1 GB/s sustained‡ 104 –105 P/E cycles

Little’s Law ( §II-C). ‡ Assuming three-year service life and continuous writes.

more outstanding data than HBM—40 MB total (Table I). Today’s GPUs are not provisioned for this scale: MSHRs are sized for HBM’s ≈1 µs load-to-use latency [18], [23], [24], and even the aggregate shared memory (18 MB/29 MB on an A100/H100) falls short. Our simulations of an A100class GPU show that serving weights from HBF slows LLM inference by 10.8× (geometric mean), and by up to 12.8×, across seven LLMs, relative to an ideal-HBF system with same read latency as HBM. The slowdown grows with latency across our 5–20 µs sweep (Fig. 2(a)). Scaling request-side structures (e.g., MSHRs) alone is insufficient—more MSHRs need more demand-generating SMs to fill them, and modern GPUs are already reticle-limited [23]. In our simulations, 20× more MSHRs and queues improve on-demand HBF to only 14.6% of ideal-HBF performance, due to insufficient demandside memory level parallelism. A larger L2 cache also fails to help, because reuse distance of weights (across iterations) far exceeds cache capacity. Using bulk-copy engines such as Tensor Memory Accelerator [24] still requires reserving more shared memory than what current GPUs provision. Hiding HBF latency therefore requires a more efficient solution.

B. Emerging High-Bandwidth Flash (HBF) Technology Recently, memory vendors have proposed High-Bandwidth Flash (HBF), where DRAM dies in an HBM stack are replaced with NAND flash while retaining an HBM-like base die [12], [13]. This organization provides 16× the capacity of HBM at comparable aggregate read bandwidth. Table I summarizes HBF (and HBM) properties in a GPU system. However, the NAND substrate introduces four key challenges. #1 Write endurance. HBF’s limited write endurance precludes storing frequently updated data: NAND cells endure only 104 –105 program/erase cycles [17]. Thus HBF would have 100–1000× lower write bandwidth than HBM (Table I). LLM weights are written once during model loading and read at every decode step, so we use HBF exclusively for weights. #2 Power consumption. HBF can consume 4× more power than iso-bandwidth HBM [13], but memory is a small fraction of a GPU’s kW-scale power budget [18]. Moving weights to HBF frees HBM capacity for KV state and can significantly improve inference throughput at a lower EDP (§VII-C). #3 Read granularity. HBF returns a 4 KB page per read. The SM should read the entire page shortly after it arrives to avoid wasting bandwidth. SMs operate on compute tiles, so a tilemajor layout, where a page corresponds to a single tile, would ensure full page use, and is already used by some LLM kernel libraries [19]–[21]. We assume consume-once behavior: each page read is fully consumed shortly after arrival. #4 Read latency. HBF’s NAND sensing, controller, and errorcorrection overheads can yield read latency of up to 20µs (SM load-to-use latency), 20× that of HBM [13], [22], making it a fundamental obstacle to using HBF for GPU systems.

D. Limitations of Existing Prefetching Techniques for HBF

C. Problem: Long HBF Read Latency

1) Prefetching cachelines: GPUs prefetch cache lines either transparently in hardware [25]–[27] or explicitly through software prefetch instructions [28]. However, line-granular prefetching inherits the bookkeeping of the demand path, so covering a 20× longer access requires 20× the tracking state (§II-C). Moreover, one tardy or inaccurate prefetch stalls the SM for microseconds, incurring drastic slowdown. §VIII provides quantitative comparison for various prior techniques. 2) Prefetching tensors: A recent work, H 3 , proposes prefetching at tensor-granularity, because “LLM inference has a deterministic and sequential access pattern” [13]. A software hint names a tensor, and H 3 streams its pages into a latencyhiding staging buffer. Unfortunately, H 3 does not describe its architecture, so we construct a capable and optimistic design with lightweight flow control and sufficient MSHRs. It prefetches a page only when a buffer slot is free (paces aggressiveness) and evicts only accessed pages (avoids thrashing)1 . Tensor prefetching hides HBF latency if the prefetch stream

By Little’s Law, sustaining bandwidth B across latency L requires B × L bytes in flight, so HBF requires 20×

1 H 3 advocates double-buffering, which permits cheaper single-ported SRAM at twice the capacity for similar runtime overhead.

3

0

QKV-proj

0

5 10 20 HBF access latency (μs)

(a) HBF on-demand fetch slowdown

0

LM-head

1 0

1

12

Norm. execution time

4

Norm. execution time

8

102

accesses

1

Tensor offset (norm.)

Norm. execution time

12

101

100

12.8x

Order stall Capacity stall Access stall Ideal-HBF

40 MB buffer

10.2x

8

5.9x

4.8x

4 1.2x

0 8B

Stream position (norm.)

70B

VA-order

8B

70B

Demand-order

(c) Limitations of H 3

(b) LLM kernel access order

10

H3 H 3 with demand order

8 6 4 2 0

goal

40 80 160 320 Staging Buffer (MB)

(d) Goal: low overheads for HBF

Fig. 2. (a) Execution time normalized to ideal-HBF of LLM on-demand fetch from HBF for different HBF access latencies. 20µs access incurs 10.8x average slowdown, and up to 12.8x (seven LLMs, §III). (b) Profiled accesses of two LLM kernels. LLM kernel accesses do not follow tensor’s VA-order (x=y line). (c) Performance of H 3 for Llama-8B and Llama-70B, when prefetcher streams in VA-order (high slowdown due to order stalls) and in demand-order (provided by SlipStream in §IV, capacity stalls dominate). (d) Seven-model performance vs buffer SRAM for H 3 . Our goal is ideal-HBF parity within a small buffer.

matches the kernel’s access order and the buffer retains every fetched page until demand arrives. We find the first condition fails outright and second requires impractical SRAM.

consumes previously staged pages. Without such flow control, tensor prefetching must overprovision the buffer to absorb cross-SM consumption skew. Even with a 320 MB buffer, the execution time is 1.48× (Fig. 2(d)), and up to 4.7×.

Challenge #1: matching kernel’s access order. H 3 prefetches pages in tensor’s virtual-address (VA) order. However, LLM kernels partition each tensor among its concurrent CTAs and swizzle the mapping for locality [21], [29]. Fig. 2(b) shows the demand order of two Llama-8B kernels: QKV-proj touches VAs across the entire tensor at nearly every stream position, LM-head walks blocks of rows, and neither kernel follows the VA diagonal. Thus, demand often reaches a page before the stream does, so the load fetches the page itself and waits the full 20 µs; we call this an order stall. With 40 MB buffer, order stalls leave Llama-8B at 4.8× and Llama-70B at 10.2× slowdown (Fig. 2(c)-left). An oracle stream that fetches pages in the demand order eliminates order stalls and recovers Llama-8B to 1.2× due to access stalls, in which demand waits for pages that the stream has fetched but that are still in flight (mostly cold start). Llama-70B, however, remains at 5.9×.

Key Takeaway and Goal: A practical HBF prefetcher must supply the demand order of each kernel and decouple buffer capacity from the tensor footprint. Our goal is to serve HBF accesses at near-HBM performance with low storage overhead and minimal architectural and programming changes. III. E XPERIMENTAL M ETHODOLOGY A. System organization Figure 3 shows our baseline system organization, which follows H 3 , the first system-architecture proposal to employ HBF [13]: read-only weights reside in HBF, mutable activations and KV state remain in HBM, and a staging buffer helps hide the access latency. Each HBF stack connects to the GPU through the base die of an HBM stack, and the staging buffer resides on that base die. Our system pairs five HBM stacks with five HBF stacks, so each base die holds an 8 MB slice of the 40 MB buffer2 . HBF data is not staged at HBM.

Challenge #2: retaining pages in the buffer. Even with kernel’s access order, prefetched pages must remain in the staging buffer until demand consumes them. Since the staging buffer is allocated on arrival from HBF and retained until consumption by demands, unconsumed pages can fill the staging buffer and prevent subsequent prefetched pages from being installed. When kernels exhibit cross-SM load imbalance (e.g., the gate and up kernels of Llama-70B launch 224 equalwork CTAs on 108 SMs, so eight SMs run three CTAs and access 1.5× more pages than the other 100), pages awaiting lagging SMs can occupy the majority of the shared buffer, leaving insufficient space for pages needed by faster SMs. The prefetcher then stalls until buffer slots become free, and the faster SMs eventually demand pages whose prefetch requests have not yet been issued, causing capacity stalls (e.g., 5.9× loss for Llama-70B in Fig. 2(c)). This imbalance comes from grid quantization, so CTA-to-SM scheduling cannot remove it. It appears in the gate and up kernels of three of our seven models (with 55–60% of their weight traffic). Thus, the staging buffer requires demand-proportional allocation: an SM should acquire additional capacity only as its demand

= 32B line

= 4 KB page

GPU SM

SM

R/W KV-cache

read-only weights

HBM

SM 2TB/s

L2

HBM stack Staging Buffer

HBF 2TB/s

HBF stack Flash cntlr

Fig. 3. H3 -like system organization [13]: HBF stores read-only weights, HBM stores mutable data, and a staging buffer hides HBF latency from SMs.

B. Simulation Framework We use Accel-Sim v1.3 [33], a cycle-level GPU simulator, interfaced with Ramulator2 [34], a memory-system simulator 2 HBF pages reside within a stack at 4 KB granularity. The address translation layer routes the prefetch and demand loads of HBF pages to the corresponding stack. Page placement is static and managed by the OS.

4

TABLE II W ORKLOAD CHARACTERIZATION ( PER - LAYER , BATCH SIZE 64, FP16). CTA COUNTS INCLUDE SPLIT-K SLICES ( GRID z). T IME IS IN µS . QKV-Proj

O-Proj

MLP-Gate

MLP-Up

MLP-Down

CTAs

Time

Wt. pages

CTAs

Time

Wt. pages

CTAs

Time

Wt. pages

CTAs

Time

Wt. pages

CTAs

Time

Wt. pages

CTAs

256 256 200 272 272 224 272

34.7 41.0 37.2 117.3 117.3 118.3 378.9

8K 10K 13K 33K 33K 33K 131K

64 80 80 64 64 64 64

91.5 219.5 160.5 242.4 420.5 419.0 1001.5

29K 82K 69K 88K 115K 118K 426K

56 256 108 86 224 231 416

89.8 218.9 163.2 245.6 424.0 418.2 1003.1

29K 82K 69K 88K 115K 118K 426K

56 256 108 86 224 231 416

86.7 234.2 193.0 238.1 308.2 310.1 1226.3

29K 82K 69K 88K 115K 118K 426K

96 100 100 96 96 96 64

353.0 773.8 627.0 964.5 1391.0 1387.9 4034.0

107K 271K 238K 338K 418K 429K 1557K

528 948 596 604 880 846 1232

Model

Time

Wt. pages

Llama-8B [1] Mistral-24B [30] Qwen2.5-32B [31] DeepSeek-67B [32] Llama-70B [1] Qwen2.5-72B [31] Llama-405B [1]

50.3 60.2 73.1 121.1 121.0 122.3 424.2

12K 15K 18K 41K 41K 41K 148K

to serve HBM requests. We extend memory system of Ramulator with an HBF model and route read-only weight pages to HBF and all other traffic to HBM. Each 4 KB page fetch incurs an unloaded request-to-return latency of 20 µs, and additional queuing delays. HBF page fetches wait for HBF channel bandwidth and for an install slot in the staging buffer (16banks), and cacheline requests wait for lookup and fill ports of the buffer. The Accel-Sim setup matches A100 silicon with a 0.7% MAPE on bandwidth-intensive streaming kernels. The modeled GPU provides 80 GB of HBM in the baseline, and 1,280 GB of HBF (16×) at equal aggregate read bandwidth in the HBF system. §VI-F further evaluates sensitivity to flash architecture-induced latency variance [35]–[37]. Simulations report execution time normalized to an idealized HBF system with the same latency as HBM. IdealHBF runs 2.9% faster than a hypothetical HBM-based system iso-capacity to HBF, though ideal-HBF is not a strict upper bound. Real-GPU runs report runtime and resource overheads. Table III provides the system configuration.

layer reads 15× more weight pages than Llama-8B, but the QKV projection kernel only launches 6% more CTAs. D. Execution Model of SlipStream SlipStream operates with conventional GPU kernels. Each workload CTA contains its original work warps and one staging warp. At kernel entry, the staging warp begins replaying that CTA’s profiled page stream while the work warps execute the operator. Because STAGE is nonblocking and requests one 4 KB page, the staging warp rapidly establishes sufficient lookahead within the kernel. Neither SlipStream nor H3 cross kernel boundaries. Consequently, each weight-reading kernel incurs a startup interval before the staging stream reaches its steady-state lead, and all reported results include this cold-start latency. §VII-A discusses how to remove this cold start. IV. S LIP S TREAM : HW/SW C O -D ESIGNED P REFETCHING We now present SlipStream3 , a software–hardware codesigned approach to hide HBF latency. This section describes the software component and §V covers architectural support.

TABLE III C ONFIGURATION OF THE BASELINE HBM-HBF SYSTEM . Parameter SM count GPU core / memory clock GPU, HBM Cacheline size L1 / shared memory per SM L2 capacity Memory channels HBM, HBF peak read bandwidth Aggregate HBM, HBF capacity HBF page size / request-to-return latency

Total

Value

A. Profiling and Kernel Modifications

108 1.410 / 1.512 GHz 128B, 32B 192 KB 40 MB 40 2TB/s, 2TB/s 80 GB, 1280 GB 4 KB / 20 µs

We observe that LLMs repeatedly execute the same kernels with the same launch shapes across decoder layers/steps. For data-independent kernels, this repetition produces the same per-CTA page-access order. Consequently, when the serving system first hosts a given model and kernel-library configuration, a Stream Profiler traces one representative execution and stores each CTA’s tensor-page sequence—its access stream— in a compact and reusable Stream Cache. SlipStream then augments each workload CTA with a staging warp that replays the access stream at runtime, staging HBF pages into the staging buffer ahead of their demand accesses. Fig. 4 provides an overview of the SlipStream-SW framework. We focus on data-independent kernels in §IV–§VI and §VII-B describes the lightweight changes to provide support for data-dependent kernels (e.g., expert kernels in MoE models).

C. Simulated Workloads We trace seven dense FP16 models at batch size 64: Llama8B, Mistral-24B, Qwen2.5-32B, DeepSeek-67B, Llama-70B, Qwen2.5-72B, and Llama-405B (and two MoEs in §VII-B), using NVBit 1.7.6 [38]. We simulate each operator using its traced grid, CTA resource allocation, occupancy, and instructions [39]; only additional staging-warp resources described in §IV-A3 are added. Table II characterizes the weight-reading kernels (other operators like LayerNorm are fused [40]–[42]). Execution time scales with model size but CTAs launched within each kernel remain similar. For example, a Llama-405B

3 SlipStream takes its name from the reduced-drag region behind a lead racer that helps a close follower move efficiently. Similarly, SlipStream stages data ahead of demand, reducing the effective latency of demand accesses.

5

Serving software Stream Cache

Stream Profiler

LLM

across GPU drivers and even GPU architectures, provided kernel parameters and tensor sizes are unchanged at runtime. 3) Staging Warp: The GPU assigns CTAs to SMs only at runtime. SlipStream therefore adds one staging warp to each CTA. The staging warp reads its assigned stream from HBM, converts each tensor-relative stream entry into a virtual address, and issues a STAGE instruction, our new ISA extension. Each STAGE instruction specifies a 4 KB-aligned virtual address and issues a non-blocking request for the corresponding 4 KB weight page, which HBF returns into the staging buffer ahead of demand. We integrate the staging warp once per kernel family, independent of model or GPU: an entry macro in the serving stack’s kernel-library template redirects the added warp to a generic replay routine, and each CTA’s launch configuration grows by one warp. This costs ∼40 lines per kernel family, after which any model using that family runs on SlipStream unmodified. We integrated Marlin [21] once with Llama-8B, and the remaining six evaluated models required no changes. The staging warp competes with the work warps for SM resources: a warp slot, registers, and instruction fetch and issue bandwidth. We model this contention explicitly: in simulation, the staging warp is an additional resident warp with its own register allocation, warp-slot occupancy, and fetch/issue traffic; on real GPUs, we emulate it as an added warp that issues L2 prefetches. Staging warp execution is low-cost. It issues one STAGE per 4 KB page, occupies under 0.5% of issue slots, and runs at least 20µs ahead of work warps. The warp scheduler runs unmodified (i.e., staging warps are not given any priority). Warp-specialized kernels with full register occupancy (e.g., CUTLASS 3.x) still accommodate the replay routine: we fold it into an idle warp of an existing warpgroup (e.g., a spare TMA producer warp), adding no SM resources.

Frequency

CTA Warp Integration

Work warps

…

Staging Warp

One-time per model One-time per kernel family

Fig. 4. Overview of SlipStream profiler. The stream profiler collects access streams and stores them in a reusable stream cache database. We integrate a staging warp per CTA of the workload to use the access streams at runtime.

1) Stream Profiler: The Stream Profiler comprises a tracer, which records workload CTAs’ page accesses, and a validator, which confirms that the resulting access streams remain safe to reuse during subsequent executions at runtime. Tracer We use binary instrumentation (e.g., NVBit) to record the virtual address (VA) of every global load, together with the kernel identity, launch configuration, tensor sizes, and issuing CTA. Because VAs may change across runs, the tracer maps each weight address to a tensor-relative page index and emits a (tensor, page-index) tuple. Loads outside the input weight tensors, such as activations and KV-cache accesses, are excluded. The tracer records each kernel invocation as a separate set of per-CTA page sequences, with kernel-launch boundaries delimiting the profiled streams. Validator SlipStream runs the tracer across multiple inputs, and the validator admits a stream only if the per-CTA access orders match exactly, indicating data-independence. The validator also assumes consume-once layout (§II-B) and may repack weights in tile-major layout when necessary. If a tensor still fails the data-independence check, the profiler directs the serving system to retain that tensor in HBM. This ensures SlipStream has accurate access streams for all validated kernels. In practice, for dense models, only the token-embedding table—a data-dependent gather—and small normalization and bias tensors fail this check: on Qwen2.5-32B, they account for 8% of weight bytes but 0.1% of weight decode traffic. 2) Stream Cache: Once generated, the Stream Cache serves as a reusable database of profiled LLM kernel families and their access streams. The cache stores one stream per kernel class rather than one per LLM layer, which substantially reduces its footprint. We observe that demand order of LLM kernels is determined by their launch shape and tile schedule. We therefore key each stream by kernel name, launch parameters (e.g., grid and block dimensions), and tensor sizes (thus, no virtual addresses). Layers sharing a kernel class differ only in the weight tensors they access and the binder (§IV-B) resolves these identities at model load time for dense LLMs. For example, five stream classes totaling 1.6 MB cover the 257 weight-operator bindings of Llama-8B for a given input shape (0.03% of the model weights). Because entries are keyed by launch parameters, serving the same model at multiple batch sizes adds one entry per distinct launch shape, at negligible storage cost. The Stream Cache resides on disk and is reusable

B. Bootstrapping Access Streams to Warps The Stream Cache contains tensor-relative page indices, whereas a STAGE instruction requires a virtual address that becomes known only after the serving system loads the model. Furthermore, because the workload kernel has already been compiled and captured in the decode CUDA graph, the staging warp cannot receive a profile-specific stream pointer as an additional launch argument. SlipStream bridges this gap using a binder. The binder exploits the weight tensor’s base address, which is available both in the captured graph node and as a runtime kernel argument, as a common key between the serving software and the staging warp. Fig. 5 shows the overview of SlipStream’s bootstrapping. At model load, the binder walks the weight-kernel nodes in the captured CUDA graphs and matches each node to an entry in the Stream Cache using its kernel parameters. It then creates a Bind Map entry keyed by the kernel’s weight-tensor base address. Each entry stores the base addresses of all tensors accessed by the operator, such as weights and scales, along with a pointer to the corresponding access stream. At runtime, the staging warp uses the weight pointer already passed to the kernel to locate this entry, reconstruct the virtual addresses

6

VAs eg. 0x5, 0xD

CUDA Graph Model Load

name, params

Weight registry Size of tensors B,C Stream Cache

Staging Warp B’s VA

B=0x5, C=0xD Binder Access stream

5%

Runtime

Access stream

Decode overhead

kernel=name(params: tensor B’s VA, C’s VA) { out = B*input //weight tensor out = C*out //other tensor eg. scaler }

Stage (page)

Bind Map Key: wt. tensor (B) VA Value: 1. Tensor VA and sizes 2. Access stream ptr

A100

H100

Blackwell

4% 3% 2% 1% 0% Llama8B

Mistral24B

Qwen2.5- DeepSeek32B 67B

Llama70B

Qwen2.572B

Llama405B

GM

Fig. 6. Real-GPU decode overhead of replaying SlipStream’s access streams. Fig. 5. SlipStream bootstrapping. The binder gets kernel parameters from CUDA graph capture. It then uses the Stream Cache and serving system’s weight registry to build the bind map which staging warps use at runtime.

with generated tokens remaining bit-identical on all three architectures. Four stock W8A8 layout revisits each page 28.6– 31×. After SlipStream validator’s repacking, all four evaluated W8A8 classes satisfy the consume-once requirement. 2) Weight coverage: Validated streams cover nearly all measured weight traffic, with 99.1–99.95% traffic coverage (geometric mean 99.6%); the remaining traffic comprises normalization parameters, biases, and group indices, which account for at most 0.9% of weight traffic and SlipStream retains in HBM. Weight-storage coverage is 81.0–97.2% (geometric mean 91.9%). The embedding table—a data-dependent gather—is the only HBM-resident weight, and it scales with vocabulary rather than model depth: the highest loss of coverage is for Llama-8B at 19% of model weights, while Llama405B loses just 2.8% coverage. 3) Profiling overheads: Offline profiling takes 40 minutes to 2.6 hours per model, while model-load binding takes approximately one minute. Moreover, a stream acquired on one GPU generation can be reused across all three evaluated generations, so SlipStream requires neither manual kernel analysis nor repeated deployment-time profiling.

from the tensor-relative page indices, and select the portion of the stream associated with its CTA. For Llama-8B, the binder reads 1.6 MB of access streams and creates 14 KB of Bind Map entries. Because addresses are rebound at each model load, the same stream survives process restarts. C. SlipStream Software’s Evaluation Methodology We evaluate SlipStream’s software stack and staging-warp overhead on A100 (108 SMs), H100 (132 SMs), and Blackwell RTX PRO 6000 (188 SMs) GPUs. All seven models of Table II run GPTQ-INT4 with Marlin kernels [21]. We use GPTQINT4 because the larger FP16 models (up to 800 GB) exceed the memory of our GPUs even when profiling few layers. We also evaluate W8A8 Llama-8B and the FP16 LM-head (both CUTLASS [20]). Overall, we profile 84 kernel classes— 74 Marlin, 8 W8A8, and 2 FP16—through vLLM [2] with CUDA-graph decoding at batch size 8 (for low KV pressure). Because STAGE is unavailable on current GPUs, we measure replay overhead by adding a resident staging warp that issues L2 prefetches in the recorded page order. For the largest models we profile a subset of layers: 20/80 for Llama-70B and Qwen2.5-72B, 24/95 for DeepSeek-67B, and 8/126 for Llama-405B. We report full-model equivalents (e.g., Llama70B’s embedding-table overhead shrinks 4× at 80 layers).

V. A RCHITECTURAL S UPPORT FOR S LIP S TREAM A. SlipStream Architecture Overview SlipStream’s software pipeline inserts STAGE instructions that move pages from HBF ahead of demand; we now describe the architectural support that executes them. Each STAGE prefetches a 4 KB page into the staging buffer, from which subsequent demand loads are served at near-HBM latency. While we employ a dedicated buffer, in principle on-chip GPU caches can also be repurposed as the staging buffer (especially for SlipStream 16MB owing to its small buffer) [43]. Because STAGE instructions generate long-latency memory requests, SlipStream adds a per-SM Stage Status Holding Register (SSHR) to track their progress. Fig. 7 presents these architectural components, which we describe next.

D. Results: Overhead of Staging Warps The staging warp incurs low overhead on GPUs. Fig. 6 reports 0.76–2.96% end-to-end decode overhead across the seven models (geometric means 2.1%, 2.4%, and 1.2% on A100, H100, and Blackwell). At kernel launch on A100, we measure that the Bind Map lookup adds a nominal 0.15 µs latency. Moreover, 128B cachelines of the access stream allows staging 128KB of data for 0.1% extra traffic. Our realsilicon and simulated results account for these costs (§VI-A). The warp uses no dynamic shared memory and adds 2–4 registers per thread, but the added warp grows a CTA from 256 to 288 threads and 3.6 K registers, 5.5% of the 64 K register file, while occupancy remains unchanged (§IV-A3).

B. Stage Status Holding Register (SSHR) The per-SM SSHR tracks STAGE requests and bounds how far the staging stream can run ahead of demand. Each STAGE instruction allocates one SSHR entry for a 4 KB page. Demand requests consult the SSHR before memory system access. If a demand matches an in-flight page entry, the requesting warp waits on that entry, analogous to a miss waiting on an MSHR.

E. Results: Stream Cache Coverage Analysis 1) Kernel class coverage: The offline validation reproduces the exact per-CTA page order for all evaluated kernel classes,

7

GPU System Work warps

CTA …

Staging Warp

MSHR

4KB pages Staging Buffer

SM ALU/ LDST

CTA count per SM (computed by the hardware). Thus, SMs with more CTAs/ page-accesses receive proportionally more staging budget than other SMs. This keeps the in-flight staging window saturated while staging remains demand-proportional.

32B lines

HBF

C. Staging Buffer

SSHR

The staging buffer serves both STAGE and demand requests (see Fig. 3). A STAGE instruction initiates 4 KB transfer from HBF, whereas demand loads are line-granular. Demands normally arrive after the page is installed. If a demand arrives while the page is in flight, it waits on the SM’s SSHR entry; a request from another SM to the same page coalesces with the shared outstanding request at the buffer. Upon completion, the buffer installs the 4 KB page and returns an ACK to the requesting SM, allowing waiting demands to proceed. Little’s Law requires 40MB of requests in flight to sustain peak HBF bandwidth. This requirement, however, does not imply that the staging buffer must reserve a data slot for every in-flight page. SlipStream therefore tracks pages while they are in flight but allocates data SRAM only when they return. Provisioning the staging buffer below 40 MB introduces two challenges: bounding how many pages return before demand reaches them and retaining resident pages until their final use. Demand-paced admission. Demand consumption rate is slow at kernel launch (e.g., due to prologue operations). As such, staging warps could populate the entire SSHR window before the new kernel issues any demand requests, causing more pages to return than a smaller-than-40MB buffer can hold. SlipStream therefore begins each kernel with an initial staging allowance, S0 , whose aggregate value does not exceed the staging buffer capacity. Each SM tracks U , the number of staged pages that have not yet received their first demand, including both in-flight and resident pages, and F , the cumulative number of first demands in the current kernel. The SSHR admits a STAGE instruction if U < min(SSHRMax, S0 + F ). Each first demand decreases U and increments F , allowing the staging window to grow toward SSHRMax as demand advances. The counters are reset at each kernel launch, and S0 is recomputed from the buffer capacity and active-SM count. Consumption-aware eviction. A smaller buffer may evict a page before demand loads have consumed all of its cache lines. The consume-once layout guarantees that every staged page is read in full, allowing a per-page completion counter to identify its final demand. The counter counts the consumeonce read; later consumers either hit in L2 or re-STAGE the page, which coalesces if the page is in flight or hits if resident. SlipStream marks a page evictable only after this final access and appends it to a consumed-page queue. A prefetched page installs into a free slot or replaces the queue head, ensuring residence of partially consumed pages. Moreover, to maximize shared pages’ residence, the buffer prioritizes unshared pages for eviction. Pages with very high reuse distance can still be refetched via subsequent STAGE instructions. If a page returns when no buffer slot is available, the buffer discards the payload, marks the request retry-pending using a single bit in its tag (a discarded page retains its tag entry), and

STAGE instruction

Fig. 7. Overview of SlipStream’s architecture. Staging warps issue STAGE instructions, the per-SM SSHR tracks each staged page until its first demand access, staging buffer serves demands at near-HBM bandwidth/latency.

SlipStream employs a cooperative page-access protocol: once the page is installed in the staging buffer, the buffer returns an ACK to requesting SM, and SSHR marks the corresponding entry as ready. Any warps waiting on the corresponding entry issue their cache-line demand requests to the memory system. A demand load that finds no SSHR entry issues to the memory system unchanged: staging is a performance hint, not a correctness requirement. Like ordinary loads, STAGE requests carry virtual addresses and translate through the GPU TLB hierarchy. STAGEs and ACKs (8B flits) use the existing on-chip network (0.8% extra HBF traffic). The network routes each STAGE by page address to the memory partition that owns it. Demand loads proceed unmodified. For timely staging of data, we want knowledge of demand’s consumption rate (§II-D). A conventional MSHR releases its entry when the requested data returns, and so MSHRs carry no information about consumption. Our key insight is to retain each SSHR entry until the staged page receives its first demand access. When all SSHR entries are occupied, the staging warp is backpressured; each first demand response releases one entry and admits the next STAGE instruction. SSHR occupancy therefore counts the pages staged but not yet demanded, so demand itself paces the staging stream and a slow consumer cannot overrun the staging buffer. As each entry tracks 4 KB page rather than 128-byte cache line, SlipStream reduces tracking state by 32× compared to MSHRs. We size the SSHR to provide the bandwidth–latency product of staging requests (as required by Little’s Law). In our system, 2 TB/s × 20 µs corresponds to approximately 40 MB, or 9,800 4 KB pages. Distributing these pages across all 108 SMs requires 90 entries per SM. Some LLM operators run on fewer SMs—the sparsest measured operators launch just 56– 64 CTAs on the 108-SM A100 (Table II). We provision 256 SSHR entries per SM, so that about one third of the SMs can saturate HBF bandwidth when necessary. Each entry stores a 37-bit page tag, a valid bit, and a ready bit, padded to 5 B, so the SSHR of each SM occupies only 1.25 KB (Table IV). A per-SM SSHRMax register exposes only the appropriate number of entries during each kernel based on the active-SM count (the value is stored at the prologue of access stream). Moreover, grid quantization gives some SMs more CTAs than others (§II-D). SSHRMax is therefore weighted by the page count of each CTA (computed by the stream profiler) and the

8

H 3 40 MB

H 3 160 MB

H 3 320 MB

SlipStream 40 MB

SlipStream 16 MB

Norm. Performance

Norm. Performance

HBF-naive

1.2 1.0 0.8 0.6 0.4 0.2 0.0

H3

1.2

SlipStream

1.0

40 MB

0.8

16 MB

0.6 0.4 0.2 0.0

Llama8B

Mistral24B

Qwen2.532B

DeepSeek67B

Llama70B

Qwen2.572B

Llama405B

Geomean

4

8

16

32

64

128

256

512

Staging SRAM Buffer (MB)

Fig. 8. Performance of on-demand fetch, H 3 , and SlipStream (40 MB and 16 MB buffers), normalized to an ideal-HBF system with HBM-like latency (left), and geometric-mean performance against buffer SRAM (right). SlipStream performs within 8% (12%) of ideal-HBF at just 40MB (16MB) of buffer SRAM.

reissues the HBF access. Newly reclaimed slots are reserved for retries before younger arrivals. If a retry still cannot obtain a slot, it preempts the youngest resident page that has not yet received demand, returning that page to the staging pipeline. Together, they guarantee progress without evicting partially consumed data. Thus, first demand regulates kernel-launch staging, while final demand regulates buffer reclamation. We evaluate SlipStream variants with 40MB and 16MB buffer sizes. For the 40MB and 16MB variants, S0 value is 9800/108 = 90 and 4000/108 = 37, respectively.

Fig. 8 (right) sweeps SlipStream from 8 MB to 40 MB: performance degrades smoothly from 92.4% at 40MB to 81.4% at 8 MB, making SlipStream a scalable and low-cost solution. Even at 8MB buffer, only Qwen2.5-32B discards pages (0.3% extra HBF traffic). No other model/ buffer-size incur discards. B. Analysis of HBF-Induced Stalls Fig. 9 attributes runtime to wait-free demand loads and stall types (§II-D). In SlipStream, loads have 100% buffer hits (median AMAT of 0.5–1.4 µs; §VI-D), so its only stalls are access stalls, primarily cold starts (15–22% of time on the small models at 40 MB). At 40 MB, order stalls take 74– 91% of H 3 ’s time. Oracle demand order (second bar) removes order stalls but exposes capacity stalls: three models recover to 1.13–1.21× yet still incur the cold start (11–17% of time), while the large models (and 24B, owing to SM load imbalance) lose 82–85% of their time to capacity stalls and run at 5.9– 6.8×. Demand-order H 3 runs small models slightly faster than SlipStream because our H 3 model is optimistic. Its demand loads read pages directly from fill queues of the buffer, and it provisions 2.8× more MSHRs. Under identical hardware, the two designs match for Llama-8B (within 1%); SlipStream’s flow control is essential at larger models/ smaller buffers.

VI. E VALUATION R ESULTS A. SlipStream Performance SlipStream performs close to ideal-HBF for all LLMs. Fig. 8 shows the performance of HBF-naive, H 3 , and SlipStream relative to an ideal-HBF system. HBF-naive reaches only 9.2% of ideal-HBF. With a small, model-independent 40 MB buffer, SlipStream reaches 92.4% overall and 78.0–107.2% per model. Llama-70B and Qwen2.5-72B exceed ideal-HBF. Both systems serve every demand load from buffer SRAM, so they differ only in when weight traffic enters the memory hierarchy. Ideal-HBF releases each sub-kernel’s loads as an unmetered burst at the barrier, saturating the L2 MSHR pool and stalling later requests at the L2. SlipStream’s pacing avoids this, reducing weight AMAT by 28% and L2 MSHR reservation retries by 2.45×, for up to 7.2% speedup. Small models (8B, 24B, 32B) lose performance because the cold-start HBF latency at each weight-kernel boundary is model-independent: it consumes 18-22% of 353–773 µs layer, but only 4% of Llama-70B’s 1391 µs layer. Mistral-24B has load imbalance in some kernels (1.27× of mean pages for some SMs), which SSHRMax mitigates. Optimizing access stream reads can also reduce costs: Incurring no stream read cost improves SlipStream by 3.2% (Llama-8B). At 16 MB, SlipStream reaches 88.8% overall and 74.1– 102.7% per model, a 4.0% loss relative to 40 MB. The gap comes from more conservative staging at kernel launch: the 16 MB variant issues just 37 pages per SM, 40% of the bandwidth–delay product, so kernels start more slowly. H 3 ’s performance is much worse than SlipStream. At 40 MB it reaches only 10.6% of ideal-HBF performance— 4.8–11.5× slowdowns, barely better than on-demand fetching. Even with an impractical 320 MB buffer, H 3 reaches 43.1%.

Stall-free

Access stall

Order stall

Capacity stall

Execution time

100% 75% 50% 25% 0% L-8B

M-24B

Q-32B

DS-67B

L-70B

Q-72B

L-405B

Fig. 9. Stall attribution at runtime. Bars per model, left to right: H 3 -40MB (VA order), H 3 -40MB with the oracle demand order, SlipStream 40 MB, and SlipStream 16 MB. SlipStream eliminates order and capacity stalls.

C. Analysis of HBF Bandwidth Utilization We compare each design against ideal-HBF system and report HBF’s read bandwidth utilization in Fig. 10. SlipStream40MB sustains 49.2–81.2% of peak: within 2.4–3.3% of peak of the reference on DeepSeek-67B and Llama-405B, above it

9

filed demand order to H 3 but reaches only 32.8% of idealHBF performance as demand can outrun prefetch, leaving large models capacity-stalled (§VI-B). SM-side SSHR staging provides the largest gain, reaching 88.1% with 40MB buffer. The demand-paced admission at kernel launch raises performance to 92.4% and makes 16MB buffer viable at 73.7% performance of ideal-HBF. While consumption-aware eviction has little effect for 40MB buffer owing to its size, it is critical in improving performance of 16MB buffer, taking it to 88.8%. Thus, at 16 MB, all three mechanisms become necessary.

100%

Ideal-HBF reference

H 3 40 MB

SlipStream 40 MB

SlipStream 16 MB

75%

1.2

Norm. Performance

50% 25%

L8

M24

Q32

D67

L70

Q72

L405

0.2 H 3-40MB + SlipStream-SW

+SSHR staging (FIFO buffer)

+demand-paced +consume-aware admit evict (SlipStream)

Fig. 12. Contribution breakdown normalized to ideal-HBF’s performance. Each optimization is necessary (especially for 16MB buffer) and additive.

SlipStream’s buffer spends most of its capacity on retention of in-use data. Fig. 11 contrasts useful lifetime (installation to final demand) with residence (installation to eviction) at 16 MB. Demand consumes a page in 1.1–1.7 µs on average (95th percentile below 4 µs), but the buffer retains it for 9.8– 24.7 µs, leaving 8.5–23 µs of margin which aids in maintaining performance even in presence of SM load imbalance that can occur due to runtime congestion. SlipStream does not evict pages before final use and we also measure zero discards of pages returned by HBF. The 16 MB point is tightly provisioned: its maximum install-to-first-demand lead (6.5–13.4 µs) approaches the ≈12 µs measured average buffer residence. On average, though, only 8.9% of buffer’s page are staged but unconsumed, so demand tightly follows staging. Yet load-touse demand latency does not increase: median weight-load latency is 0.5–1.4 µs and the 99th percentile is 1.8–2.7 µs, comparable to HBM’s 0.9–1.8 µs AMAT.

F. Sensitivity to HBF Read Latency Unloaded Latency. We evaluate HBF request-to-return latencies of 5, 10, and 20 µs (Fig. 13). Both buffer configurations stay at their design capacity, while the SSHR window scales with the bandwidth–delay product. SlipStream improves as latency decreases, retaining 92.4%, 100.7%, and 103.7% at 40 MB and 88.8%, 101.2%, and 103.7% at 16 MB for 20, 10, and 5 µs. Below 20 µs it exceeds ideal-HBF on average, with four of seven models individually above it, due to the pacing reason of §VI-A. The cost of a smaller 16MB buffer also reduces: the two capacities differ by 4.0% performance at 20µs, by only 0.5% at 10 µs, and produce identical cycle counts on all seven models at 5 µs, since a shorter latency leaves fewer pages in flight and 16 MB already covers that window. The SSHR therefore serves the entire 5–20 µs range.

p95 Max

Norm. Performance

10

1

SlipStream 40 MB

1.2

SlipStream 16 MB

1.0 0.8 0.6 0.4 0.2

5 μs

Fig. 11. Per-page lifetime in the 16 MB buffer (log scale): the time SlipStream needs each page versus the time it holds it. The gap is post-consumption retention; measured to be 8–23µs (so sufficient margin against jitters).

10 μs

M

L8

M

L8

G

L405

24 Q 32 D 67 L7 0 Q 72 L4 05

Q72

M

L70

G

D67

24 Q 32 D 67 L7 0 Q 72 L4 05

Q32

M

M24

M

L8

M

0.0 L8

Page lifetime (μs)

0.4

H 3-40MB

D. Analysis of Buffer Residency and Load-to-use Latency

100

0.6

0.0

Fig. 10. Sustained HBF read bandwidth (norm. to 2TB/s) for different designs. SlipStream retains most bandwidth utilization in all evaluated models.

Min. req. residence (install→last use) Page lifetime (install→evict)

0.8

G

0%

40 MB 16 MB

1.0

24 Q 32 D 67 L7 0 Q 72 L4 05

HBF bandwidth util.

on Llama-70B and Qwen2.5-72B (≈107%; see §VI-A), and 11.3–18.5% below it on the three smaller models, consistent with their performance losses. The 16 MB configuration stays within 5.0% of peak of 40 MB on every model and is 2.8% higher on Llama-405B. In contrast, HBF-naive sustains only 6.0–7.1%, and H 3 7.3–11.4% at the same 40 MB, because insufficient buffer capacity limits in-flight traffic and thereby HBF bandwidth.

20 μs

Fig. 13. Sensitivity to HBF request-to-return latency, normalized to idealHBF. SlipStream retains low average overheads at various HBF latencies.

Latency Variance. Flash adds two sources of latency variance. First, flash organizes each stack into channels and subarrays, and each read occupies a subarray for its sensing time (tR ), leading to subarray contention. Second, when a

E. Performance Contribution Breakdown Fig. 12 shows SlipStream’s ablated mechanisms at both buffer capacities. At 40 MB, SlipStream-SW supplies the pro-

10

read has bit errors (at probability pe ), the controller rereads the page at other reference voltages and decodes it again, repeating across decoding levels as needed4 . Vendors have not disclosed these parameters for HBF, and recent HBF studies assume a fixed read latency [13], [44]–[46]. We add both effects to the 20 µs unloaded latency. We model the subarray organization of [14] with tR of 3 and 5 µs [22], [47], and set pe to 10−4 [35]; each further decoding level is 100× less likely and adds 1.75 tR . Subarray contention accounts for most of the loss, mainly at kernel launch, when the initial burst of reads contends on subarrays. Read-retry costs at most one point, because retries are rare. Together, the two effects lower SlipStream from 92.4% to 89.2% and 86.9% of ideal-HBF at 40 MB for tR of 3 and 5 µs, and from 88.8% to 87.9% and 84.8% at 16 MB. Neither effect causes page discards or buffer misses.

TABLE IV P HYSICAL COST OF S LIP S TREAM ’ S ADDED STRUCTURES . T OP - CELL VALUE IS FOR S LIP S TREAM -40MB, BOTTOM IS 16MB VARIANT.

G. Storage, Area, and Power Overheads

Structure

Storage

Area (mm2 )

Access (ns)

Leak. (W)

Dyn. (W)

GPU die (per SM) SSHR CAM SSHRMax register

1.25 KB 4B

0.0036 —

0.10 —

0.001 —

0.001 —

Total (108 SMs)

135 KB

0.39

—

0.153

0.09

HBM base die (per stack) Staging buffer 8 MB 3.2 MB

7.62 3.39

2.5 1.6

2.56 1.13

4.65 2.95

Tag + metadata

32 KB 25.6 KB

0.026 0.022

<0.2 <0.2

0.010 0.008

0.002 ∼0

Control + queues

1.7 KB

0.028

<0.2

0.001

∼0

Total (5 stacks)

40.2 MB 16.1 MB

38.35 17.21

— —

12.86 5.69

23.26 14.77

such a megakernel, cross-operation staging raises SlipStream from 92.4% to 96.1% of ideal-HBF at 40 MB and from 88.8% to 99.5% at 16 MB, owing to even better pacing and less L2 congestion at kernel starts with 16MB buffer. The worst-case model’s slowdown falls from 28–35% to just 13–16%.

Table IV reports CACTI [48] estimates at 22 nm for the structures that SlipStream adds. On the GPU die, SlipStream adds only the SSHRs and the SSHRMax registers. Each SM searches its 1.25 KB SSHR in one cycle, and all 108 SSHRs occupy 0.39 mm2 , or 0.02% of the 826 mm2 GA100 die when scaled to 7 nm. The staging buffer and its tags reside on the five HBM base dies (§III-A). Each 8 MB slice uses 16 banks to sustain 0.4 TB/s of installs from flash alongside 0.4 TB/s of demand reads. The slice occupies 6.3% of a 121 mm2 base die at 22 nm, or 1.9% at the 3 nm bitcell that H 3 assumes [13]. H 3 -320MB occupies 42.7% and 12.7%, respectively (10.7% by H 3 ’s bitcell-only formula). The tag array scales with bandwidth–delay product rather than with buffer capacity. The 2.5 ns buffer access is negligible against HBF’s 20 µs latency. The five base dies leak 12.9 W and draw at most 23.3 W of dynamic power. At the measured 49.2–81.2% HBF utilization, SlipStream therefore adds 24.5–32.0 W. The HBF stacks draw 4× the energy per bit of HBM, which raises the A100 envelope from 400 W to 654 W. SlipStream adds 3.7–4.9% to this envelope at 40 MB and 1.9–2.8% at 16 MB, while H 3 (320MB) adds a significant 18–22% to system power.

B. Support for Other Inference Deployment Targets Prefill. Prefill kernels are data-independent, so SlipStream profiles and replays them similar to decode kernels. Serving engines bucket kernel shapes by prompt tokens, and the Stream Cache holds one stream per launch shape. Compute-bound prefill also shares each weight page across multiple CTA tiles, which SlipStream handles natively (§V-C). Mixture-of-Experts models. In an MoE layer [16], [50]–[52], a router assigns each token to a subset of the experts, so expert weights that a kernel reads are data-dependent. We examine popular MoE kernels [2], [53] and make two observations. First, routing completes before the MoE kernel launches, and the kernel receives the routing result as an input, so experts and tiles of each CTA are known at launch. Second, all experts share one shape and tiling, so the page order within a tile is identical across experts. The access stream of each CTA is therefore known at kernel entry, without prediction. We extend SlipStream-SW to support MoEs. First, the Stream Profiler records one expert-relative stream per tile, along with the tile scheduler of the kernel. The validator admits the kernel only if, across routings, every tile follows its stream and every CTA follows the scheduler5 . Second, the Bind Map entry of each MoE layer adds the expert stride, from which the staging warp computes the base address of each expert. Third, at kernel entry, the staging warp runs the tile scheduler on the routing input, as the work warps do, to find the tiles of its CTA. It sets SSHRMax from relative load of its CTA and replays stream of each tile from base address of its expert. We trace and simulate persistent MoE kernels of TensorRTLLM [53] for Qwen3-30B and Qwen3-235B [54] (the w13

VII. D ISCUSSION A. Cross-kernel Staging We can avoid the 20µs cold-start at kernel launch by letting the staging warp cross kernel boundaries. Weights do not depend on the output of the previous kernel, so staging them before launch is safe. The obstacle is attribution, because the staging warp must know which SM consumes each page. Persistent megakernels [49] remove this obstacle by fixing the operation sequence of each CTA at compile time. One staging warp per SM then persists across all operations. It begins the next operation once it has issued its current stream, and each SM limits its next-operation pages at its share of the buffer (no new barrier is needed). In our simulations of

5 Kernels that assign tiles dynamically (e.g., work stealing) require each CTA to claim its tiles at least one HBF latency (20 µs) before computing them, so that the staging warp stages them in time.

4 Write interference, a third common source, is virtually absent because

weights are read-only; only periodic read-disturb refresh rewrites them.

11

and w2 GEMMs are fused). At batch sizes of 8 and 64, 31 and 87 of 128 experts activate on average. Fig. 14 shows that SlipStream reaches 91.2% of ideal-HBF at 40 MB and 87.3% at 16 MB (geometric mean), with no discarded pages or buffer misses. The bandwidth utilization with ideal-HBF is very high at 96–98%, so SlipStream’s pacing does not provide additional improvement. At batch 64, 6.7% of tiles share weight pages with another tile; SlipStream-HW coalesces their STAGEs and serves the shared pages without refetches. Performance improves with batch/ model size (HBF’s target regime), from 83.7% on Qwen3-30B at batch 8 to 97.1% on Qwen3-235B at batch 64. The fixed 20 µs cold start per layer amortizes over kernels that grow from 173 µs to 1.82 ms. Even at batch=1, SlipStream retains 82.2% of ideal-HBF on Qwen3-235B. SlipStream 40MB

Qwen3-30B-A3B

HBM only

Norm. to HBM only

Norm. Performance

H 3 (unsupported)

throughput, energy/token, and EDP improvements. Llama70B fits in four HBM GPUs with sufficient KV space; SlipStream raises throughput by 1.17×. Llama-405B needs sixteen HBM GPUs, whereas eight HBF GPUs hold a full replica. Sixteen HBF GPUs with SlipStream therefore deliver 2.27× the throughput, at 0.52× energy/token and 0.23× EDP. H 3 , even in demand order, serves Llama-405B at a TPOT of 602 ms, 10× that of HBM-only serving, and just 0.55× the throughput. Thus, HBF can improve serving throughput and energy efficiency when prefetcher hides HBF latency.

SlipStream 16MB

Qwen3-235B-A22B

1.0 0.8 0.6

Llama-70B, 4 GPUs

0.2 Batch = 8

Batch = 64

Batch = 8

SlipStream (40MB)

Llama-405B, 16 GPUs

8 4 2

2.27 1.17

1

0.93

0.5

0.52

0.79

0.23

0.25 Throughput Energy/token (↑ better) (↓ better)

0.4 0.0

16

H 3 (40MB, demand order)

EDP (↓ better)

Throughput Energy/token (↑ better) (↓ better)

EDP (↓ better)

Fig. 15. Serving throughput, energy per token, and EDP, normalized to HBMonly system. HBF with SlipStream increases lowers EDP by 1.27–4.35x.

Batch = 64

Fig. 14. Performance of MoE kernels [53], normalized to ideal-HBF. SlipStream-40MB reaches 91.2% of ideal-HBF (87.3% at 16MB).

VIII. R ELATED W ORK A. GPU prefetching

Shared-prefix KV cache. Supporting this follows the same approach as MoEs. Long-lived prefix KV is written once and read many times, which suits HBF. Paged-attention kernels locate KV pages through a block table that the serving engine writes before launch [2]. The staging warp therefore reads the table, as the work warps do, and stages the KV pages in walk order. Requests that share a prefix read the same pages, which SlipStream handles natively as shared pages (§V-C). Limitations. SlipStream does not directly support kernels whose data dependence resolves inside the kernel and that run for less than one HBF latency (20 µs). Such tensors (e.g., datadependent gathers) remain in HBM. Kernels with arbitrary tileto-CTA mappings may also elude the profiler and validator, and require the staging warp to emulate the accesses explicitly.

Prior GPU prefetchers [25]–[27], [56] target HBM-scale latency and track each prefetched line, so a 20 µs lookahead exhausts their tracking state. CABA runs assist warps on idle SM resources to relieve bottlenecks such as memory bandwidth [57], similar to SlipStream’s staging warp. At HBF latency, however, a mispredicted or mistimed page stalls an SM for microseconds. The staging warp must therefore replay exact page streams under page-granular flow control. Fig. 16 compares four prior works for HBF within the GPUdie area of SlipStream (geomean over seven models). Each design fetches 4 KB pages into the staging buffer, replays exact per-SM page streams of SlipStream-SW, and keeps its in-flight requests and flow control. The designs fail for two reasons. First, three designs keep too few requests in flight. Warped Pre-execution (Warped PE) [58] runs warps ahead on spare registers, which bound its lead to about 6 pages per SM, far short of the 90 pages that hide 20 µs. It reaches 19.3% of ideal-HBF. CAPS [26] and Snake [27] track every prefetched line, so a 256-entry budget, equal to that of an SSHR, covers only 8 pages per SM, so they incur 4x slowdown. Second, designs with enough lead lack demand pacing. CPU-assisted GPGPU [59] precomputes the page streams on the CPU and issues them as one global stream. It reaches only 66.8% at 40 MB and 14.9% at 16 MB, because pages for lagging SMs hold buffer slots while other SMs starve. A 90-page lead for Warped PE needs twice the L1 capacity and register file (8% more die area). This lead reaches 80.3% at 40 MB but only 18.8% at 16 MB, because without pacing it overruns the smaller buffer.

C. End-to-end Serving Throughput Serving engines shard large models across GPUs. With each rank staging its own shard, SlipStream retains 87.0–104.3% of ideal-HBF on Llama-405B at 2–8 GPUs with a 40 MB buffer, whereas H 3 in address order reaches only 10.2–13.8%. The retained fraction declines at higher degrees because kernels shorten while the 20 µs cold start per kernel remains fixed. Moving the weights to HBF frees HBM for KV cache, which allows for a larger batch. However, reading the weights from HBF can degrade time per output token (TPOT) if its latency is exposed. We derive energy per token from a 336 W compute die, HBM at 3.97 pJ/bit [55], and HBF at 4× that energy [13]. For each system, we report the tensor-parallel layout with the highest max-batch throughput. Fig. 15 shows

12

ACKNOWLEDGMENTS

Norm. Performance

Finally, for SlipStream, using page-granular tracking alone is insufficient. Releasing each page entry on fill (like MSHRs) rather than at first demand lets the stream outpace consumption and evict pages before use. For example, on Llama-405B, 13% of pages then require refetching, and performance drops by 9%. Thus, our key insight of using first-demand release is essential to maximize performance (89–92% of ideal-HBF), 40 MB buffer

1.0 0.8 0.6 0.4 0.2 0.0

0.92 0.89

0.15 CPU-assisted GPGPU

0.19 0.19

0.25 0.25

Warped PE Ideal line pref. 6-page lead (CAPS, Snake)

R EFERENCES

16 MB buffer

Iso GPU-die area with SlipStream 0.67

We thank the anonymous reviewers of HPCA-2027 for their valuable feedback We thank Dr. Po-An Tsai (NVIDIA Research) and Prof. Moinuddin Qureshi (Georgia Tech) for insightful discussions. Anish was partly funded by the NVIDIA Graduate Fellowship.

Extra GPU-die area

[1] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [2] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [3] L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez et al., “Sglang: Efficient execution of structured language model programs,” Advances in neural information processing systems, vol. 37, pp. 62 557–62 583, 2024. [4] Google, “Our next-generation model: Gemini 1.5,” 2024. [Online]. Available: https://blog.google/technology/ai/google-gemini-next-gener ation-model-february-2024/#sundar-note [5] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. [6] L. Org, “Deploying deepseek with pd disaggregation and large-scale expert parallelism on 96 h100 gpus,” 2025. [Online]. Available: https://lmsys.org/blog/2025-05-05-large-scale-ep/ [7] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019. [8] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” Proceedings of machine learning and systems, vol. 5, pp. 606–624, 2023. [9] N. Corporation, “Nvidia blackwell architecture technical overview.” [Online]. Available: https://resources.nvidia.com/en-us-blackwell-archi tecture [10] P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132. [11] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 193–210. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin [12] S. Corporation, “Sandisk unveils the future of memory architecture for ai introducing: High bandwidth flash,” 2025. [Online]. Available: https://documents.sandisk.com/content/dam/asset-library/en us/assets/p ublic/sandisk/collateral/company/Sandisk-HBF-Fact-Sheet.pdf [13] M. Ha, E. Kim, and H. Kim, “H 3: Hybrid architecture using high bandwidth memory and high bandwidth flash for cost-efficient llm inference,” IEEE Computer Architecture Letters, 2026. [14] P.-K. Hsu, W. Xu, Q. Liu, T. Rosing, and S. Yu, “Haven: Highbandwidth flash augmented vector engine for large-scale approximate nearest-neighbor search acceleration,” arXiv preprint arXiv:2603.01175, 2026. [15] N. Corporation, “Blackwell-datasheet-3384703.pdf.” [Online]. Available: https://nvdam.widen.net/s/wwnsxrhm2w/blackwell-datasheet-338 4703 [16] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. [17] Y. Luo, “Architectural techniques for improving nand flash memory reliability,” Ph.D. dissertation, Carnegie Mellon University, 2018.

0.80

0.19

SlipStream

Warped PE 90-page (8% extra GPU-die area)

Fig. 16. Prior prefetchers adapted to HBF, normalized to ideal-HBF. All designs that fit within the GPU-die area of SlipStream but incur 1.5-5.2x slowdown, while Warped PE with 90-page lead requires 8% extra die area.

B. Memory Expansion Techniques Several systems move tensors among GPU, CPU, and storage [60]–[64], prefetch experts over CPU–GPU links [65]– [67], or prefetch tensors or pages into HBM from host memory or flash [68]–[71]. These techniques stage each tensor in HBM before its kernel starts (tensor fits in HBM). SlipStream instead stages its pages in a 16–40 MB buffer that is orders of magnitude smaller than HBM. Such a buffer requires exact access order and fine-grained flow control to avoid misses. C. Data Orchestration CUTLASS and TMA pipelines stream tiles through shared memory in demand order [20], [72], and Buffets provide decoupled staging [73]. Both reserve buffer storage for each request at issue and release it after consumption, so storage grows with the bandwidth–delay product, which is impractical at 20 µs. SlipStream instead tracks in-flight pages with compact SSHR entries and allocates storage only when a page arrives. Staging is a non-binding hint, first demand paces admission, and final demand reclaims storage. IX. C ONCLUSION High-Bandwidth Flash (HBF) offers higher capacity at HBM-like bandwidth, but its 20 µs read latency requires 20× more outstanding data than current GPUs support. SlipStream, a software–hardware co-designed prefetcher, replays the profiled page-demand order of LLM kernels ahead of demand. Its hardware retains each page-granular tracking entry until first demand, so demand paces staging, and allocates buffer storage only when pages arrive. With a 40 MB buffer, SlipStream achieves 92.4% of ideal-HBF on seven dense models (88.8% with 16 MB) and 91.2% on two MoE models. In contrast, H 3 incurs a 2.3× slowdown even with a 320 MB buffer. SlipStream thus makes HBF practical for GPUs.

13

[18] A. Jarmusch and S. Chandrasekaran, “Microbenchmarking nvidia’s blackwell architecture: An in-depth architectural analysis,” 2026. [Online]. Available: https://arxiv.org/abs/2512.02189 [19] N. Corporation, “Cute layouts,” Mar 2026. [Online]. Available: https: //docs.nvidia.com/cutlass/latest/media/docs/cpp/cute/01 layout.html [20] “Gemm in cuda: Nvidia cutlass documentation,” Mar 2026. [Online]. Available: https://docs.nvidia.com/cutlass/latest/media/docs/cpp/efficie nt gemm.html [21] E. Frantar, R. L. Castro, J. Chen, T. Hoefler, and D. Alistarh, “Marlin: Mixed-precision auto-regressive parallel inference on large language models,” in Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, 2025, pp. 239–251. [22] W. Cheong, C. Yoon, S. Woo, K. Han, D. Kim, C. Lee, Y. Choi, S. Kim, D. Kang, G. Yu et al., “A flash memory controller for 15µs ultra-lowlatency ssd using high-speed 3d nand flash with 3µs read time,” in 2018 IEEE International Solid-State Circuits Conference-(ISSCC). IEEE, 2018, pp. 338–340. [23] N. Corporation. [Online]. Available: https://www.nvidia.com/content/d am/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepape r.pdf [24] ——. [Online]. Available: https://resources.nvidia.com/en-us-hopper-a rchitecture/nvidia-h100-tensor-c [25] J. Lee, N. B. Lakshminarayana, H. Kim, and R. W. Vuduc, “Manythread aware prefetching mechanisms for GPGPU applications,” in Proceedings of the 43rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE Computer Society, 2010, pp. 213–224. [26] G. Koo, H. Jeon, Z. Liu, N. S. Kim, and M. Annavaram, “CTA-aware prefetching and scheduling for GPU,” in Proceedings of the 32nd IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2018, pp. 137–148. [27] S. Mostofi, H. Falahati, N. Mahani, P. Lotfi-Kamran, and H. SarbaziAzad, “Snake: A variable-length chain-based prefetching for GPUs,” in Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). ACM, 2023, pp. 728–741. [28] NVIDIA Corporation, Parallel Thread Execution ISA, Version 9.3, NVIDIA Corporation, 2026. [Online]. Available: https://docs.nvidia.co m/cuda/parallel-thread-execution/ [29] M. Osama, D. Merrill, C. Cecka, M. Garland, and J. D. Owens, “Stream-k: Work-centric parallel decomposition for dense matrix-matrix multiplication on the gpu,” in Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, 2023, pp. 429–431. [30] M. AI, “Mistral small 3.” [Online]. Available: https://mistral.ai/news/ mistral-small-3/ [31] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan, “Qwen2 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2407.10671 [32] X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu et al., “Deepseek llm: Scaling open-source language models with longtermism,” arXiv preprint arXiv:2401.02954, 2024. [33] M. Khairy, Z. Shen, T. M. Aamodt, and T. G. Rogers, “Accel-sim: An extensible simulation framework for validated gpu modeling,” in 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2020, pp. 473–486. [34] H. Luo, Y. C. Tuğrul, F. N. Bostancı, A. Olgun, A. G. Yağlıkçı, and O. Mutlu, “Ramulator 2.0: A modern, modular, and extensible dram simulator,” IEEE Computer Architecture Letters, vol. 23, no. 1, pp. 112– 116, 2023. [35] Y. Cai, S. Ghose, E. F. Haratsch, Y. Luo, and O. Mutlu, “Error characterization, mitigation, and recovery in flash-memory-based solidstate drives,” Proceedings of the IEEE, vol. 105, no. 9, pp. 1666–1704, 2017. [36] K. Zhao, W. Zhao, H. Sun, X. Zhang, N. Zheng, and T. Zhang, “LDPCin-SSD: Making advanced error correction codes work effectively in solid state drives,” in 11th USENIX Conference on File and Storage Technologies (FAST 13). San Jose, CA: USENIX Association, Feb.

2013, pp. 243–256. [Online]. Available: https://www.usenix.org/confere nce/fast13/technical-sessions/presentation/zhao [37] J. Park, M. Kim, M. Chun, L. Orosa, J. Kim, and O. Mutlu, “Reducing solid-state drive read latency by optimizing read-retry,” in Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ser. ASPLOS ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 702–716. [Online]. Available: https://doi.org/10.1145/3445814.3446719 [38] O. Villa, M. Stephenson, D. Nellans, and S. W. Keckler, “Nvbit: A dynamic binary instrumentation framework for nvidia gpus,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 2019, pp. 372–383. [39] N. Corporation. [Online]. Available: https://docs.nvidia.com/cuda/pdf/C UBLAS Library.pdf [40] D. Zhuang, Z. Zheng, H. Xia, X. Qiu, J. Bai, W. Lin, and S. L. Song, “{MonoNN}: Enabling a new monolithic optimization space for neural network inference tasks on modern {GPU-Centric} architectures,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 989–1005. [41] N. Nayak, X. Wu, T. O. Odemuyiwa, M. Pellauer, J. S. Emer, and C. W. Fletcher, “Fusemax: Leveraging extended einsums to optimize attention accelerator design,” in 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2024, pp. 1458–1473. [42] L. Qin, J. Cui, W. Cai, and J. Huang, “Chimera: Communication fusion for hybrid parallelism in large language models,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture, ser. ISCA ’25. New York, NY, USA: Association for Computing Machinery, 2025, pp. 498–513. [Online]. Available: https://doi.org/10.1 145/3695053.3731025 [43] V. Selfa, J. Sahuquillo, L. Eeckhout, S. Petit, and M. E. Gómez, “Application clustering policies to address system fairness with intel’s cache allocation technology,” in 2017 26th international conference on parallel architectures and compilation techniques (pact). IEEE, 2017, pp. 194–205. [44] A. Ilkbahar, R. Ramanujan, and T. Coughlin, “High bandwidth flash: A new memory for the next generation of ai computing,” Computer, vol. 59, no. 5, pp. 134–138, 2026. [45] D. Son, Y. Park, H. Cho, H. Ham, O. Mutlu, S. Lee, G. Kim, and J. Park, “Exploring high-bandwidth flash for modern llm inference: Opportunities and challenges,” IEEE Computer Architecture Letters, vol. 25, no. 2, pp. 251–254, 2026. [46] X. Wang, Y. Xue, X. Sun, X. Zhang, X. Zhang, C. Dou, X. Li, and X. Chen, “Flashaccel: Leveraging high-bandwidth flash (hbf) for high-throughput llm inference,” 2026. [Online]. Available: https://arxiv.org/abs/2607.10186 [47] T. Kouchi, N. Kumazaki, M. Yamaoka, S. Bushnaq, T. Kodama, Y. Ishizaki, Y. Deguchi, A. Sugahara, A. Imamoto, N. Asaoka, R. Isomura, T. Handa, J. Sato, H. Komai, A. Okuyama, N. Kanagawa, Y. Kajiyama, Y. Terada, H. Ohnishi, H. Yabe, C. Hsu, M. Kakoi, and M. Yoshihara, “13.5 a 128gb 1b/cell 96-word-line-layer 3d flash memory to improve random read latency with tprog=75µs and tr=4µs,” in 2020 IEEE International Solid-State Circuits Conference - (ISSCC), 2020, pp. 226–228. [48] S. Thoziyoor, N. Muralimanohar, J. H. Ahn, and N. P. Jouppi, “Cacti 5.1,” Technical Report HPL-2008-20, HP Labs, Tech. Rep., 2008. [49] X. Cheng, Z. Zhang, Y. Zhou, J. Ji, J. Jiang, Z. Zhao, Z. Xiao, Z. Ye, Y. Huang, R. Lai, H. Jin, B. Hou, M. Wu, Y. Dong, A. Yip, Z. Ye, S. Wang, W. Yang, X. Miao, T. Chen, and Z. Jia, “MPK: A compiler and runtime for Mega-Kernelizing tensor programs,” in 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). Seattle, WA: USENIX Association, Jul. 2026, pp. 1909–1926. [Online]. Available: https: //www.usenix.org/conference/osdi26/presentation/cheng [50] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022. [Online]. Available: http://jmlr.org/papers/v23/21-0998.html [51] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024. [52] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024.

14

[53] Nvidia, “Tensorrt-llm: A tensorrt toolbox for optimized large language model inference,” 2025. [Online]. Available: https://github.com/NVIDI A/TensorRT-LLM [54] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [55] M. O’Connor, N. Chatterjee, D. Lee, J. Wilson, A. Agrawal, S. W. Keckler, and W. J. Dally, “Fine-grained dram: energy-efficient dram for extreme bandwidth systems,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO-50 ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 41–54. [Online]. Available: https: //doi.org/10.1145/3123939.3124545 [56] D. Ganguly, Z. Zhang, J. Yang, and R. G. Melhem, “Interplay between hardware prefetcher and page eviction policy in CPU-GPU unified virtual memory,” in Proceedings of the 46th International Symposium on Computer Architecture (ISCA). ACM, 2019, pp. 224–235. [57] N. Vijaykumar, G. Pekhimenko, A. Jog, A. Bhowmick, R. Ausavarungnirun, C. Das, M. Kandemir, T. C. Mowry, and O. Mutlu, “A case for core-assisted bottleneck acceleration in gpus: enabling flexible data compression with assist warps,” in Proceedings of the 42nd Annual International Symposium on Computer Architecture, ser. ISCA ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 41–53. [Online]. Available: https://doi.org/10.1145/2749469.2750399 [58] K. Kim, S. Lee, M. K. Yoon, G. Koo, W. W. Ro, and M. Annavaram, “Warped-preexecution: A gpu pre-execution approach for improving latency hiding,” in 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2016, pp. 163–175. [59] Y. Yang, P. Xiang, M. Mantor, and H. Zhou, “Cpu-assisted gpgpu on fused cpu-gpu architectures,” in IEEE International Symposium on HighPerformance Comp Architecture, 2012, pp. 1–12. [60] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Machine Learning. PMLR, 2023, pp. 31 094–31 116. [61] K. Alizadeh, S. I. Mirzadeh, D. Belenko, S. Khatamifard, M. Cho, C. C. Del Mundo, M. Rastegari, and M. Farajtabar, “Llm in a flash: Efficient large language model inference with limited memory,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 12 562–12 584. [62] L. Fusco, M. Khalilov, M. Chrapek, G. Chukkapalli, T. Schulthess, and T. Hoefler, “Understanding data movement in tightly coupled heterogeneous systems: A case study with the grace hopper superchip,” arXiv preprint arXiv:2408.11556, 2024. [63] A. Eliseev and D. Mazur, “Fast inference of mixture-of-experts language models with offloading,” 2023. [64] L. Xue, Y. Fu, Z. Lu, L. Mai, and M. Marina, “Moe-infinity: Activationaware expert offloading for efficient moe serving,” 2024. [65] S. Cao, S. Liu, T. Griggs, P. Schafhalter, X. Liu, Y. Sheng, J. E. Gonzalez, M. Zaharia, and I. Stoica, “Moe-lightning: High-throughput moe inference on memory-constrained gpus,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, 2025, pp. 715–730. [66] A. C. Yüzügüler, J. Zhuang, and L. Cavigelli, “Preserve: Prefetching model weights and kv-cache in distributed llm serving,” arXiv preprint arXiv:2501.08192, 2025. [67] R. Hwang, J. Wei, S. Cao, C. Hwang, X. Tang, T. Cao, and M. Yang, “Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference,” 2024. [68] M. Rhu, N. Gimelshein, J. Clemons, A. Zulfiqar, and S. W. Keckler, “vdnn: virtualized deep neural networks for scalable, memory-efficient neural network design,” in The 49th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO-49. IEEE Press, 2016. [69] X. Peng, X. Shi, H. Dai, H. Jin, W. Ma, Q. Xiong, F. Yang, and X. Qian, “Capuchin: Tensor-based gpu memory management for deep learning,” in Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ser. ASPLOS ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 891–905. [Online]. Available: https://doi.org/10.1145/3373376.3378505 [70] H. Zhang, Y. Zhou, Y. Xue, Y. Liu, and J. Huang, “G10: Enabling an efficient unified gpu memory and storage architecture with smart tensor migrations,” in Proceedings of the 56th Annual IEEE/ACM

International Symposium on Microarchitecture, ser. MICRO ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 395–410. [Online]. Available: https://doi.org/10.1145/3613424.3614309 [71] J. Jung, J. Kim, and J. Lee, “Deepum: Tensor migration and prefetching in unified memory,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 207–221. [Online]. Available: https://doi.org/10.1145/3575693.3575736 [72] M. Bauer, H. Cook, and B. Khailany, “Cudadma: optimizing gpu memory bandwidth via warp specialization,” in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’11. New York, NY, USA: Association for Computing Machinery, 2011. [Online]. Available: https://doi.org/10.1145/2063384.2063400 [73] M. Pellauer, Y. S. Shao, J. Clemons, N. Crago, K. Hegde, R. Venkatesan, S. W. Keckler, C. W. Fletcher, and J. Emer, “Buffets: An efficient and composable storage idiom for explicit decoupled data orchestration,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 2019, pp. 137–151.

AI U SE The authors have used generative AI tools, Claude and ChatGPT, in this paper to improve readability, reduce text length, and format figures and tables.

15

Related documents

• SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads#1648806
• ZenFlow: Enabling Stall-Free Offloading for LLM Training#1355898
• CascadeLUT: Information-Ordered Streaming Inference for Bandwidth-Constrained FPGAs#1129348
• CVE-2026-125217 — MEDIUM 4.8#1627350
• Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU#1199367