arXiv CS open access

FlashBoot: Sub-Second Weight Loading for Large Models at Rack Scale

2026 · arxiv_cs

arXiv CS · Papers · 2026

cloud distributed-computing parallel-computing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

FlashBoot

Sub-second rack-scale weight loading

FlashBoot: Sub-Second Weight Loading for Large Models at Rack Scale Isaac Zhu, Hscos Zhang, Keith Jiang, Jack Li, Hugh Yin, Jason Zhao∗ Scitix.AI

Flagship Mixture-of-Experts (MoE) models are growing along two axes at once—total parameter count and the number of experts—with DeepSeek-V4-Pro reaching 1.6 T parameters across 384 experts. In elastic deployment scenarios such as cold start, autoscaling, and fault recovery, many GPUs across many nodes must become serving-ready quickly, and this growth makes weight loading a noticeable part of the latency budget. Even on NVIDIA’s GB300 NVL72, whose chip-to-chip (C2C) link and NVLink Switch fabric have ample raw bandwidth, today’s state-of-the-art loaders (Hugging Face SafeTensors, InstantTensor, and SGLang’s NCCL-based Tensor R-Fork) leave most of that bandwidth unused. The losses are structural: (C1) weight memory is fragmented into tens of thousands of per-tensor objects, so transfers run far below link bandwidth; (C2) cross-node replication is gated by NCCL communicator setup, which costs 10–110 s before a single weight byte moves; and (C3) the existing cross-node GPU→GPU clone path is serial and scales poorly to concurrent multi-node bring-up. We present FlashBoot, a hardware-friendly, framework–workflow co-designed weightloading subsystem built on SGLang. At its core is FabricArena, a contiguous, exportable devicememory layout that lets a model’s discrete weight tensors occupy one large 1-D image while remaining inter-node addressable. On top of it, FlashLoad loads from CPU (host→GPU) as a single bulk, zero-copy transfer, and FlashClone replicates a resident model GPU→GPU by a remote-mapping mechanism that removes NCCL setup. In experiments on NVL72 with DeepSeek-V4-Pro (1.6 T) and DeepSeek-V4-Flash (284 B), FlashClone maps remote weight memory in ∼10 ms (versus 10–110 s for NCCL) and sustains ≥700 GB/s per clone. Against the state of the art, FlashBoot accelerates single-node weight loading by up to 50× (from 20.1 s to 0.4 s) and concurrent rack-level weight loading by > 270× (from 87 s to 0.32 s). (a) Single-node load, Flash SafeTensors FlashBoot

R-Fork

17.7 14.8

800

10 0

87 s

100

20.1

0.4 cold

600 400 200

0.4 warm

NCCL

FlashBoot

GB/s

FlashBoot

30.8 time (s)

(b) Bring-up 𝑁 replicas, Pro

InstantTensor

time (s)

arXiv:2608.08482v1 [cs.DC] 9 Aug 2026

Abstract

0.32 s 1

replicas 𝑁

Figure 1. Results at a glance. (a) Single-node DeepSeek-V4-Flash load time: FlashBoot collapses the engine-visible load to ∼0.4 s versus 15–31 s for SafeTensors and InstantTensor. (b) Time to bring up 𝑁 DeepSeek-V4-Pro replicas: FlashBoot fills all clones concurrently along a chain in ∼0.32 s, whereas SGLang’s R-Fork stands up an NCCL group and copies serially, so its wall-clock grows with 𝑁 (it serializes the per-clone standup and transfer; §5.4). (c) Chainbroadcast bandwidth per clone stays near the link rate as 𝑁 grows and exceeds NCCL broadcast on the same fabric.

∗

Corresponding author.

FlashBoot

Sub-second rack-scale weight loading

1 Introduction and Background 1.1 Where fast weight loading matters Outside steady-state serving, an inference service loads its weights more than once, and sometimes under a deadline, in several elastic-deployment scenarios: • Initial deployment / cold start. A fresh service brings a model onto a pool of GPUs from storage; serving cannot begin until the last byte is resident. • Serverless / scale-to-zero. A model that is paged out to reclaim capacity must be paged back in on the next request, so load latency lands directly on the request’s tail. • Autoscaling. When load rises, the scheduler adds replicas; the faster a new replica becomes servingready, the tighter the system can track demand instead of over-provisioning for it. • Fault recovery. When a node or a process dies, the service must restart it or stand up a replacement and re-fill its weights to restore capacity and redundancy. These scenarios share one path: getting a (often very large) weight image onto one or many GPUs, frequently across several nodes at once, quickly. As models grow this path lengthens, and in latency-sensitive cases it can slow how quickly a deployment reacts. FlashBoot targets this path in these scenarios; it does not change steady-state serving. 1.2 Two axes of model growth Open-weight flagship models are growing along two axes simultaneously (Figure 2). The first is total parameter count, which has climbed from the hundred-billion class to beyond a trillion in roughly a year. The second, specific to the Mixture-of-Experts (MoE) designs that now dominate the frontier, is the number of experts: each layer fans out into many expert weight tensors, of which only a few are active per token. The forthcoming DeepSeek-V4-Pro reaches 1.6 T parameters across 384 experts. The two axes compound: the expert dimension is precisely what multiplies a checkpoint into tens of thousands of individual weight tensors, and the parameter axis is what makes the aggregate byte volume large. Both directly inflate weight loading: one by sheer volume, the other by sheer object count. 1.3 The thesis The hardware industry has answered the bandwidth side of this problem with a structural shift: the rack-scale system is becoming the unit of large-model deployment. A rack is no longer a loose set of servers behind a commodity network but a co-designed machine in which tens of accelerators share a high-bandwidth, nearall-to-all “scale-up” fabric, with a separate scale-out network between racks. NVIDIA’s GB300 NVL72 binds 72 Blackwell-Ultra GPUs over an NVLink Switch fabric [1], and its successor, the Vera Rubin POD, carries the same idea across five co-designed rack types [2]; the pattern is industry-wide, spanning AMD’s Helios rack (UALink-over-Ethernet scale-up) [3], Huawei’s CloudMatrix384 supernode (384 Ascend NPUs on an allto-all Unified Bus) [4], and Google’s TPU pods (thousands of chips on a 3-D-torus ICI fabric) [5]; across them the deployment unit spans from tens to thousands of high-bandwidth-coupled accelerators (Figure 3). Two forces drive this convergence: the two axes of model growth above, and the token economics of inference, the explosion in tokens consumed by reasoning and agentic workloads, which together reward packing more model into one high-bandwidth domain and standing serving replicas up fast enough to track demand. Fast, elastic weight loading is precisely the operation that turns this dense fabric into served capacity on a deadline. On a rack-scale GB300 NVL72, every GPU reaches host memory over a fast Grace↔GPU C2C link and reaches every other GPU in the rack over an all-to-all NVLink Switch fabric (§2.1). On paper, that is more than enough bandwidth to fill a model in well under a second. Yet, as we show in §2 with direct measurements, today’s state-of-the-art loaders realize only a small fraction of it: a Pro rank that the C2C bus could move in ∼1.2 s still takes ∼45–117 s to load, and replicating a resident model to even one other node first pays a 10 s+ NCCL tax before any weight byte moves. The gap is not bandwidth; it is how the bytes are laid out and moved. This report makes that case quantitatively and then presents FlashBoot, a hardware-friendly framework– workflow co-design that closes the gap by changing the weight memory layout first and rebuilding the load and replication paths on top of it. We study the GB300 NVL72 as the most widely available member of this class, but the design rests only on properties these systems share, a contiguous, exportable memory region and a fast intra-rack fabric, so its ideas carry to the others (§6.3). 2

FlashBoot

Sub-second rack-scale weight loading

(b) experts per MoE layer

(a) total parameters (B)

896

2,400

768 experts

parameters (B)

2,800

1,600

1,600 1,000

800 120

284

397

400

3.5 en Qw

754

550

512

384

o Pr

256

384 128

128

160

0 s os

TGP

sh F la

Lla

m Ne

n3 tro

1 -5.

6 -2.

o Pr

o TGP

1 4 3.5 -5. ma en M Lla GL Qw

Fla

-2 mi

n ro

t mo Ne

3 Ki

-K mi

Figure 2. Two axes of growth in flagship open-weight MoE models. (a) total parameters (ordered low to high) and (b) experts per MoE layer (ordered low to high; Kimi-K3 has the most at 896) both trend upward. Values are approximate, from public model cards and reports [20, 16, 15, 17, 8, 18, 19, 14]; the two DeepSeek-V4 models used in this report (Flash, 284 B / 256 experts; Pro, 1.6 T / 384) sit on this trend, and the recently announced Kimi-K3 (2.8 T / 896 experts) now tops both axes. More experts is what turns a checkpoint into tens of thousands of small per-tensor objects; more parameters is what makes the aggregate byte volume large.

NVIDIA GB300 NVL72

72 ⋅ NVLink Switch

AMD Helios (IF128)

128 ⋅ UALink

Huawei CloudMatrix384

384 ⋅ Unified Bus

NVIDIA Vera Rubin POD

1,152 ⋅ NVLink 6

Google TPU v7x superpod

9,216 ⋅ ICI + OCS 10

100

1,000

10,000

accelerators per rack-scale system (log scale) Figure 3. Rack-scale systems are the emerging unit of large-model deployment. Tightly-interconnected accelerators per rack-scale system across vendors, ordered smallest to largest (log scale): NVIDIA GB300 NVL72 (72 GPUs) [1], AMD Helios IF128 (128 GPUs) [3], Huawei CloudMatrix384 (384 Ascend NPUs) [4], NVIDIA Vera Rubin POD (1,152 Rubin GPUs) [2], and a Google TPU v7x (Ironwood) superpod (up to 9,216 chips) [5]. The reported unit differs, single NVLink/UALink/Unified-Bus scale-up domains versus multi-rack pods, but the trend is uniform: the deployment unit has grown from tens to thousands of high-bandwidth-coupled accelerators. We evaluate FlashBoot on the GB300 NVL72, the smallest and most widely available member of the class.

2 Opportunity and Challenges 2.1 The rack-scale hardware opportunity The rack-scale systems surveyed in §1 share one structural property that fast loading can exploit: a highbandwidth, near-all-to-all intra-rack scale-up fabric that is far faster than each accelerator’s path to host memory. Unless stated otherwise, experiments in this report run on a GB300 NVL72 system (NVIDIA Grace– Blackwell Ultra), which we take as the most widely available exemplar of this class; a study of other rack-scale platforms (e.g. H100 islands joined by RDMA) is left to future work and sketched in §6.3. Table 1 summarizes the platform and Table 2 its interconnect bandwidths, giving both the vendor specification and the values we measure on the machine. Two links matter for loading: the Grace↔GPU C2C link (the host→GPU path) and the GPU↔GPU NVLink fabric (the GPU-to-GPU path). The structural fact we lean on throughout is that per-GPU NVLink bandwidth is ∼4× the per-GPU C2C bandwidth; this is the hardware reason that, once a model is resident somewhere in the rack, replicating it GPU→GPU is fundamentally cheaper than reloading it from host memory. Models under test. We use the two checkpoints in Table 3: Flash (DeepSeek-V4-Flash) and the much larger 3

FlashBoot

Sub-second rack-scale weight loading

Table 1. Platform and software environment. Item

Value

System GPUs per node Rack Nodes used NVIDIA driver CUDA / runtime

GB300 NVL72 (NVIDIA Grace–Blackwell Ultra) 4 (sm_103); 2 Grace CPUs per node one NVLink-Switch fabric domain (single IMEX partition) up to 9 (1 seed + 8 clones) of a ∼17-node rack 580.105.08 13.0 (nvcc 13.0.88), torch 2.9.1+cu130, NCCL 2.27.7

Table 2. GB300 NVL72 interconnect bandwidths (vendor spec vs. measured). Path NVLink, per GPU seed-GPU egress NVLink Switch fabric Grace–GPU C2C, per GPU H2D (effective) HBM3e, per GPU LPDDR5X, per Grace

Spec (per dir.)

Measured

1.8 TB/s (≈900) — 130 TB/s 450 GB/s (≈225) — 8 TB/s —

825–826 GB/s (1 copy) 837–839 GB/s — 223.5 GB/s ∼185 GB/s — ∼500 GB/s

Table 3. Evaluation models (DeepSeek-V4 family) at TP4. Model

Params

Experts

Layers

Checkpoint

Per-rank (expert + non-expert)

Flash Pro

284 B 1.6 T

256 384

43 61

149 GB 805 GB

41.4 GB (36.8 + 4.64) 217.0 GB (205.51 + 11.44)

Pro (DeepSeek-V4-Pro). Both are evaluated at tensor-parallel degree TP=4 (one shard per GPU on a node); we use TP as the running example but the design supports the parallelism modes inference engines use in general (§3). Per-rank sizes are byte-exact: a Pro rank is 217.0 GB, namely 205.51 GB of expert weights plus 11.44 GB of non-expert weights. The per-rank total times four exceeds the on-disk checkpoint because non-expert weights are replicated across TP ranks and the pre-packed image carries the kernel-ready, post-quantization layout rather than the compact on-disk form. 2.2 The baseline reality We measured three representative loaders on this hardware: Hugging Face SafeTensors (the de-facto standard loader), the academic InstantTensor, and SGLang’s production Tensor R-Fork (load_from_remote_instance, an NCCL-based GPU→GPU replication path). Figure 4 shows single-node load time for SafeTensors and InstantTensor from cold disk and from warm RAM (/dev/shm). Even from warm RAM (with disk I/O entirely removed), a Pro rank still takes 45–67 s and a Flash rank ∼15–18 s. Since the C2C bus can move a Pro rank’s 217 GB in ∼1.2 s at the measured ∼185 GB/s, the loaders are spending > 97% of their time not moving bytes over the bus. Something other than bandwidth dominates. We now isolate the three structural causes; each is stated as a claim, derived, and backed with a measured number, and each maps to a specific element of FlashBoot’s design (§3). 2.3 C1: fragmented weight memory starves the interconnect The standard loader treats a checkpoint as what it is on disk: a large collection of independent tensors. Each parameter, and for an MoE model, each expert, is allocated and transferred as its own object, with re-layout and staging copies along the way. A Pro rank has on the order of tens of thousands of such objects (e.g. 384 experts across 61 layers). The dominant cost is then the count of operations, not the byte volume, which is exactly why the load stays slow even when the bytes already sit in RAM (Figure 4). The bandwidth penalty of small transfers is intrinsic, not an artifact of the loader. A single cross-node copy-engine DMA only approaches link bandwidth when it is large: Figure 5 sweeps one transfer from 1 MB to 32 GB and shows the achieved bandwidth climbing from 47 GB/s at 1 MB to a ∼826 GB/s plateau only past 4

FlashBoot

Sub-second rack-scale weight loading

(a) Flash 149 GB, TP4 116.8

30.8

20.1

17.7

engine load time (s)

InstantTensor

SafeTensors

(b) Pro 805 GB, TP4

14.8

100 72.9

67.4 45.8

0 cold (disk)

warm (RAM)

cold (disk)

warm (RAM)

achieved bandwidth (GB/s)

Figure 4. State-of-the-art single-node load time (engine-visible weight load, TP4). Even from warm RAM (disk removed), a Pro rank takes 45–67 s, yet its 217 GB would cross the C2C bus in ∼1.2 s at 185 GB/s. The loaders are busidle > 97% of the time; the cost is per-tensor overhead, not bandwidth (C1). link plateau ≈826 GB/s 800

per-expert tensor regime (∼1–50 MB)

600 400 200 0

16M

64M

256

B B 16G 32G

transfer size

Figure 5. Small transfers starve the interconnect (C1). Achieved bandwidth of a single cross-node DMA versus payload size (GB300 NVL72). Peak (∼826 GB/s) is reached only past ∼2 GB; in the per-expert-tensor regime (∼1–50 MB) even an ideal transport sustains a fraction of the link. Loading or replicating a model tensor-by-tensor therefore cannot use the fabric, regardless of how fast the fabric is.

∼2 GB. Per-expert tensors live in the ∼1–50 MB regime (shaded), where even a perfect transport reaches only ∼50–500 GB/s, a fraction of the link. Fragmentation thus caps every movement of the weights, on both the host→GPU and the GPU→GPU paths. The implication for design is direct: make the weights contiguous so they can move in one big transfer (§3.3). 2.4 C2: NCCL communicator setup dominates cross-node replication Replicating an already-resident model to another node is the natural way to exploit the rack: read once from storage, then copy GPU→GPU over the fast fabric. SGLang’s production path (R-Fork) does exactly this, but routes the copy through a NCCL process group, which must be stood up before any weight byte moves: a rendezvous, ncclCommInit (P2P buffer allocation, per-pair connection setup, ring/tree topology search), a barrier, and an SM-kernel warm-up. We measure this standup at 10–110 s (e.g. init_process_group over 20 ranks alone takes 11.9 s; Figure 6). To see how badly this inverts the cost, do the arithmetic on the transfer it gates. A Pro rank is 217 GB; at 80% of the ∼900 GB/s per-GPU NVLink direction, the bytes themselves cross in 𝑡xfer =

217 GB ≈ 0.30 s. 0.8 × 900 GB/s

Setup is thus 30–370× the transfer it enables. For a library used continuously this amortizes; for weight loading, a one-shot or low-frequency event, it is pure overhead on the critical path. A general-purpose collective library is simply the wrong tool for fast startup. The design implication: replace the communicator with a handle exchange that maps remote memory directly, in milliseconds, with no group to build (§3.7).

FlashBoot

Sub-second rack-scale weight loading

NCCL init (20 ranks)

11.9 s

NCCL group (1-to-N)

354 ms

NCCL group (pairwise)

112 ms

FlashBoot map

10 ms 10

100

1,000

10,000

100,000

one-time setup before first weight byte (ms, log scale) Figure 6. Cross-node setup cost before the first weight byte (C2), log scale. NCCL must stand up a communicator; the components shown reach 11.9 s for init_process_group over 20 ranks alone, and the full standup is 10–110 s in practice. FlashBoot instead imports a 64-byte fabric handle and maps remote memory in ∼10 ms, a ∼103 –104 × reduction, then moves the ∼0.30 s of bytes.

2.5 C3: the production path is serial and does not scale The third problem appears precisely in the scenario that matters most for elasticity: bringing up many replicas at once. R-Fork is, by construction, a serial seed→clone design, with a fresh communicator standup and transfer per clone, so the wall-clock to fill 𝑁 replicas grows roughly linearly in 𝑁 . Even setting NCCL aside, the obvious “parallel” alternative of having all 𝑁 clones pull from one seed at once (a 1→N star) does not help, because a single seed GPU’s NVLink egress is a fixed ceiling (837–839 GB/s, measured) that the 𝑁 readers must share. Per-clone bandwidth then collapses as ∼egress/𝑁 for 𝑁 ≥ 2: we measure 825 GB/s at 𝑁 =1 (a single copy, limited by the link rather than egress) but only ∼104 GB/s at 𝑁 =8 (≈ 838/8; §5.4, Figure 17). Either way, whether serial standup or shared egress, the time to make a rack of replicas serving-ready scales the wrong way with the rack size. The design implication is to remove the single-seed bottleneck entirely with a chained, pipelined broadcast in which every node reads from a distinct predecessor over an independent link (§3.8). Taken together, C1–C3 say the problem is not the GB300’s bandwidth but the layout and movement discipline layered on top of it. FlashBoot addresses all three with one coherent design, to which we now turn.

3 Design FlashBoot is a weight-loading subsystem for NVLink/RDMA-interconnected GPU racks. It optimizes both halves of the loading lifecycle: loading from CPU (FlashLoad, host→GPU) and replicating from a resident peer (FlashClone, GPU→GPU). It is designed for rack-scale platforms such as GB300 NVL72. While our prototype targets the NVLink Switch fabric, the design’s central ideas apply equally to NVLink-only systems and, in principle, to other accelerators with an exportable-memory primitive. Throughout we use tensor parallelism (TP) as the running example for concreteness, but nothing in the design is TP-specific: the weight image is treated as an opaque byte range, so the same machinery applies to the data-, expert-, and pipeline-parallel layouts an inference engine produces (the coverage our prototype realizes today is detailed in §4). 3.1 Design principles The challenges of §2 translate into five principles that the rest of this section instantiates: P1. Fix the layout first. Contiguity is the enabler for everything else: a single large image can be moved in one bandwidth-saturating transfer, whereas tens of thousands of per-tensor objects cannot (answers C1). P2. One image, served in place. The weights live exactly once and the model’s parameters are views onto that image, with no per-parameter storage and no redundant same-device copies (1× memory). P3. Map memory, don’t build a communicator. Cross-node access is a one-time exchange of a small handle plus a direct memory mapping, not a collective-library group (answers C2). P4. No single-point bottleneck. Every transfer is arranged so that each GPU link carries exactly one reader, so aggregate bandwidth is independent of how many nodes participate (answers C3).

FlashBoot

Sub-second rack-scale weight loading

Workflow Offline weight pre-pack

Preloader daemon (disk→shm)

FlashLoad (CPU→GPU) Bulk-load pipeline

FlashClone (GPU→GPU)

Zero-copy views

Weight layout

Out-of-band rendezvous

Remote arena map (NCCL-free)

Shard + all-gather

Chain / ring pipelined DMA

FabricArena: one contiguous, compact, inter-node–exportable 1-D arena per GPU

Hardware & inter-connection (NVL72, H100, B300, etc.) GPU node 1 G0

intra-node: C2C / PCIe

GPU node 2 G3

GPU node 3 G3

GPU node 𝑁 G3

NVLink Switch fabric / RDMA (inter-node)

Figure 7. FlashBoot architecture. From the bottom: rack hardware; the FabricArena weight-memory layout; the FlashLoad (CPU→GPU) and FlashClone (GPU→GPU) logical modules; and an orchestration layer. FabricArena is the waist: both modules move weights as one contiguous byte range.

P5. Pay slow work once, and off the critical path. Sharding, quantization, and re-layout are done once, offline; the slow cold disk read is overlapped with engine startup so the boot critical path is pure data movement. 3.2 Architecture overview Figure 7 shows how the pieces compose. At the bottom is the rack: GPU nodes connected intra-node by Grace↔GPU C2C (and PCIe) and inter-node by the NVLink Switch fabric (or RDMA). Directly above the hardware sits the one structural change FlashBoot introduces: FabricArena, a contiguous, compact, and internode–exportable weight memory layout (§3.3). Two logical modules build on it: FlashLoad fills an arena from CPU (bulk-load pipeline, zero-copy views, and a shard + all-gather variant for concurrent multi-node CPU loads), and FlashClone replicates a resident arena across nodes (NCCL-free remote mapping plus a chain/ring pipelined DMA). An orchestration layer on top supplies the offline pre-pack, the preloader daemon, and the out-of-band rendezvous that ferries handles and provides the start barrier. The arena is the waist of the design: both modules see weights as one byte range, so neither contains any per-tensor or expert/non-expert special casing. 3.3 FabricArena: a contiguous, inter-node–addressable layout Motivation. C1 (§2.3) says the root problem is that weights are scattered across many small allocations. The fix is to make them contiguous without disturbing the engine’s compute and runtime logic, which expects to find each parameter at its own tensor handle. Mechanism. FabricArena resolves this tension with the CUDA virtual-memory management (VMM) API. We reserve one large, contiguous virtual address range per GPU and back it with a single physical allocation (cuMemCreate → cuMemAddressReserve → cuMemMap → cuMemSetAccess). Every weight tensor then lives at a distinct offset inside this one arena. The model’s parameters keep their familiar tensor interface, but their storage is now a slice of a single 1-D buffer (Figure 8). Crucially, the allocation is created with a fabric handle type (CU_MEM_HANDLE_TYPE_FABRIC), which is what lets a 64-byte handle for the whole arena be exported and imported by a GPU on another node within the same fabric/IMEX domain, making the layout not just contiguous but inter-node addressable (used by FlashClone, §3.7). The arena capacity is rounded up to the VMM allocation granularity, and seed and clone apply the same rounding, so both sides always agree on the exact byte count. Layout. We lay the arena out as two contiguous regions, [ experts ∣ non-experts ], with the separator offset equal to the total expert byte size (Figure 8). This mirrors how MoE engines already group weights and keeps the two natural populations (the many large expert tensors and the comparatively few atten7

FlashBoot

Sub-second rack-scale weight loading

model parameters (discrete tensor objects): L0 w13

L0 w2

⋯

L0 scales

separator = expert_bytes

expert segment: per-layer blocks L0

⋯

attn q

norm / embed

non-expert

L𝐾 -1

capacity

w13

w13 scale

one layer’s expert tensors, packed back-to-back (each a view)

w2 scale

param.data = view(shape, stride, offset)

Figure 8. FabricArena layout and zero-copy views. Discrete model parameters (top) are not given private storage; each parameter’s data is an as_strided(shape, stride, offset) view onto a byte range of one contiguous arena (bottom), laid out [ experts ∣ non-experts ]. The weights exist exactly once and are served in place; both FlashLoad (host→GPU) and FlashClone (GPU→GPU) only ever fill the flat buffer. Table 4. Transport comparison for weight movement. FlashBoot uses copy-engine DMA, which is zero-copy into the final layout, occupies no SMs, moves the image as one bulk transfer, works both intra- and inter-node, and reaches the highest measured peak. Marks: 3 supported, partial, 7 not supported.

SafeTensors InstantTensor Tensor R-Fork (NCCL) FlashBoot (DMA)

Zero-copy (1× mem)

Single bulk transfer

SM-free (copy engine)

Cross-node (NVLink/RDMA)

Peak BW (measured)

7 partial 7 3

7 7 7 3

— — 3 3

low low–mid ∼651 GB/s ∼826 GB/s

tion/embedding/norm tensors) each in one run. The on-disk pre-packed image (§3.9) is byte-identical to this in-GPU order, so filling the arena is a pure copy with no parse-and-reshape on the critical path. Because the byte layout already matches what the serving kernels expect, the only special case the engine needs is the boundary between the two regions; everything else is offset arithmetic. 3.4 Zero-copy serving and a uniform transport Principle. Once the weights are one contiguous image, we never want a second copy of them anywhere. We bind each parameter’s data pointer to an as_strided view into the arena at the offset, shape, and stride recorded for it (Figure 8). Reading a weight reads arena bytes in place: there is exactly one copy in GPU memory (1× occupancy), and the per-tensor views are re-derived from the offset table on each side rather than materialized. Why it unifies the transport. Because the destination is always “the arena at offset 𝑜,” the source can be anything that speaks bytes: a host pinned buffer (the CPU-load path) or a remote GPU’s mapped arena (the cross-node path). Both reduce to filling one contiguous 1-D range, so a single mover, a copy-engine DMA, serves every case. The copy engine is the right primitive here for three reasons that Table 4 makes precise: it is genuinely zero-copy into the final layout, it uses no streaming multiprocessors (leaving the SMs free, unlike NCCL’s kernel-based collectives), and it reaches the highest peak bandwidth we measure on the fabric (∼826 GB/s, versus ∼651 GB/s for NCCL on the same link, §5.1). 3.5 Bulk-load pipeline From contiguity to bandwidth. With the arena in place, the host→GPU load is a single contiguous H2D copy, which already saturates the C2C link at the measured ∼185 GB/s. The remaining cost on a cold boot is the slow disk read that feeds that copy. We remove it from the critical path with a preloader daemon (Figure 9): a separate process reads each rank’s pre-packed files O_DIRECT from disk into NUMA-pinned shared memory ahead of and concurrently with engine startup. When the engine reaches the fill point it finds the pages 8

FlashBoot

Sub-second rack-scale weight loading

daemon

disk (NVMe)

O_DIRECT read

NUMApinned shm .ready (io_wait≈0)

engine

engine startup

H2D → FabricArena

PIN (register)

serve (views)

critical path = PIN + H2D only

Figure 9. Bulk-load pipeline. The preloader daemon stages disk→pinned shm concurrently with engine startup; the engine then only PINs and H2Ds the already-staged image into the arena. The slow cold read is hidden (io_wait≈ 0), so the engine’s critical path is pure data movement. full weight image

(byte-sharded 𝑁 ways)

node 0 (reads s0)

node 3 (reads s3)

ring all-gather each node pulls the other (𝑁 −1)/𝑁 from its predecessor

node 1 (reads s1)

node 2 (reads s2)

Figure 10. Shard + fabric all-gather for concurrent multi-node CPU load. 𝑁 nodes split the cold disk read 𝑁 ways (∼ 1/𝑁 ), then reassemble the full image with one NCCL-free ring all-gather over NVLink.

already staged (signalled by a .ready flag, so the wait io_wait≈ 0), registers them for DMA (PIN), and H2Ds them straight into the arena. The engine-visible critical path is thus only PIN+H2D; the cold read, which dominates wall-clock, overlaps work the engine had to do anyway. Pinned buffers are NUMA-bound to the Grace socket nearest each GPU so the H2D does not cross sockets, and the H2D itself runs as a few large chunks over multiple streams to keep the copy engine busy. 3.6 Concurrent multi-node CPU load: shard and all-gather Motivation. The single-node path still has one node read the entire checkpoint, and that cold read dominates its cost. When the same model comes up on 𝑁 nodes at once (e.g., 𝑁 data-parallel serving replicas), we can do better by using the rack’s aggregate disk and CPU bandwidth. Mechanism. The checkpoint is sharded by bytes across the 𝑁 nodes: each reads only its 1/𝑁 slice from its own NVMe (in parallel with the others) and H2Ds it into the correct offset of its full-size arena. Each node now holds only a 1/𝑁 fragment, so the nodes all-gather the missing pieces over the NVLink fabric, reusing the same exportable arena handles, with no NCCL group (Figure 10). We run the gather as a ring: in each round a node reads only from its predecessor over an independent link, so every GPU has exactly one reader and per-link bandwidth stays high and uniform however large 𝑁 grows (principle P4). The dominant cold read therefore scales as ∼ 1/𝑁 , while the gather adds only a fixed, bandwidth-bound pass over the missing (𝑁 −1)/𝑁 at the ∼740 GB/s fabric rate. 3.7 Remote memory mapping: replacing the communicator Motivation. C2 (§2.4) showed that NCCL’s communicator standup, not the transfer, dominates cross-node replication. The fabric’s per-GPU bandwidth is so high that the communicator is the bottleneck. Patching NCCL to remove the standup is conceivable but invasive and hard to maintain; instead we sidestep collectives entirely. Mechanism. We treat the rack as one shared address space (a partitioned global address space, or PGAS): 9

FlashBoot

Sub-second rack-scale weight loading

Seed

Clone

weights → FabricArena

fetch manifest (out-of-band)

manifest (few hundred B, once)

export 64-byte handle

grant self read access

publish manifest Passive (holds arena)

import handle + map remote (∼10 ms, IMEX)

NVLink DMA (the only data movement)

DMA read → rebind views

Figure 11. Remote arena mapping (NCCL-free). The seed exports one 64-byte fabric handle and goes passive; the clone imports it, maps the seed’s arena, grants itself read access (∼10 ms), and pulls the bytes with a DMA over NVLink, with no process group, no runtime rendezvous. Only the tiny manifest crosses nodes out of band.

because FabricArena memory is created with a fabric handle (§3.3), a seed can export a 64-byte handle to its whole arena, and any other node can import that handle and map the seed’s physical memory into its own virtual address space, granting itself read access (gated by the IMEX security service). The seed is never in the authorization loop; it exports once and goes passive. After mapping, a clone reads the seed’s weights with a plain DMA over NVLink, straight into its own arena (Figure 11). The only per-event cost is the export/import, measured at ∼2–15 ms per handle and independent of payload size; the handful of handles are ferried by an out-of-band TCP rendezvous (§3.9) that can overlap the rest of startup. The result is the ∼10 ms setup that replaces NCCL’s 10–110 s (Figure 6). Transports: NVLink (IMEX) and RDMA. The fabric-handle export/import we use here is the NVLink path: it relies on IMEX, the internode memory-sharing service specific to multi-node NVLink (MNNVL) platforms such as NVL72 and DGX/HGX GB200/GB300 [13]. The mechanism is otherwise transport-agnostic, needing only a primitive that grants one node direct read access to another’s memory plus a small out-ofband channel for the access token; on clusters without MNNVL that role falls to RDMA memory registration and one-sided reads, which we discuss in §6.3. We evaluate the IMEX/NVLink implementation here and leave the RDMA backend to future work. 3.8 Star vs. chain: scaling the broadcast Remote mapping makes one clone fast; bringing up many requires care (C3, §2.5). We designed and compared two topologies (Figure 12). In the star (pull) topology every clone maps and reads the same seed arena; it is the simplest scheme, one copy per clone, but all 𝑁 readers share the seed GPU’s single NVLink egress, so per-clone bandwidth falls as ∼838/𝑁 . In the chain topology the clones form a line, seed → 𝑐0 → 𝑐1 → ⋯, each reading from its predecessor rather than the seed. Every hop is a distinct NVLink, so the transfers run on independent links in parallel and per-clone bandwidth stays near the single-link ceiling regardless of 𝑁 (principle P4). To keep every link busy, the arena streams down the line in large fixed-size chunks, so a node receives chunk 𝑗 while its predecessor pulls chunk 𝑗+1 (software pipelining over the copy engine, synchronized by lightweight in-fabric flags, with no NCCL or SM kernels). The only cost is a one-time pipeline-fill latency, a small head-to-tail gradient, amortized over the chunks; chunk size is the lone knob, trading per-chunk synchronization against fill-tail length (we analyze it in §6). Chain is therefore the default whenever more than one clone is filled; all subsequent multi-node measurements use it. 3.9 Workflow: pay the heavy work once The last principle (P5) is realized by the orchestration layer (Figure 13). All distribution-dependent work (tensor-parallel sharding, fp8/mxfp4 quantization, and any kernel-required re-layout) is done once, offline, by a pre-pack step: the source checkpoint is loaded once into the engine under the exact target serving configuration, and each rank’s now-resident weights are serialized to disk as contiguous byte ranges (per-layer expert blobs followed by a single non-expert blob) plus a small metadata.json that records each parameter’s name → (offset, shape, stride, dtype). Because this on-disk order is byte-identical to the in-GPU arena layout (§3.3), every subsequent boot is pure data movement: there is no per-parameter dequantization, permutation, or reshape on the critical path. The remaining orchestration is deliberately thin: the preloader daemon (§3.5)

FlashBoot

Sub-second rack-scale weight loading

S S c0

chain (pipelined): 1 reader / link ≈ 715 GB/s, flat in 𝑁

star (pull): shared seed egress ≈ 838/𝑁 per clone

Figure 12. Broadcast topologies. Star: every clone pulls from one seed, sharing its egress (∼838/𝑁 ). Chain: each clone reads its predecessor over an independent link, chunk-pipelined, so per-clone bandwidth is flat in 𝑁 . The seed is passive in both.

source ckpt (HF read once safetensors)

load once into pre-packed imengine under targetdump contiguous age: 1-D blobs + config (shard, metadata.json quantize, re-layout) paid once, offline memcpy, no reshape

boot: FlashLoad (H2D) / FlashClone (DMA)

paid every boot (pure data movement)

Figure 13. Workflow. A one-off offline pre-pack puts the checkpoint in its final, distribution-baked, byte-for-byte arena order. Every boot thereafter, whether a CPU load or cross-node clone, is a pure copy with as_strided views and no per-parameter reshape.

and a small out-of-band TCP rendezvous that exchanges fabric handles and provides a start barrier for the chain/all-gather, a single sub-millisecond round-trip in place of any communicator.

4 Implementation We implemented FlashBoot as a lightweight, self-contained native library plus a thin integration into the SGLang inference engine (a DeepSeek-V4–capable development build; CUDA 13.0, torch 2.9.1+cu130, NCCL 2.27.7). The library is deliberately credential-free: it manipulates fabric memory and reads/writes local manifest files only; all cluster orchestration (node selection, manifest delivery, the rendezvous server) lives in separate scripts. The engine-side integration is a single patch of roughly 5,800 added and 280 removed lines across 19 files; Table 5 summarizes the integration surface. Arena and the engine’s tensors. The engine’s loader already meta-initializes the model (parameters with no backing storage) and then materializes weights. We hook the materialization step: instead of allocating per-parameter storage, the loader allocates one unified FabricArena per GPU and rebinds every param.data to an as_strided view into it (§3.4); the rest of the engine is unchanged and never learns there is a single allocation underneath. In practice we distinguish two weight populations because they are produced differently: expert weights (MoE w13/w2 and their quantization scales) are written in their post-quantization, fused/shuffled form so the load can scatter them directly, while non-expert weights (attention, embedding, norms) keep the checkpoint’s native layout and flow through the engine’s ordinary TP-sharding path. The arena’s [ experts ∣ non-experts ] split (§3.3) falls out of exactly this distinction; the separator is the total expert byte size, which for both evaluation models is 4096-byte aligned, so the non-expert views inherit a valid dtype alignment without padding. Pre-pack, model, and parallelism coverage. The offline pre-pack (§3.9) reuses the engine’s own weight_loader and process_weights_after_loading, so the produced bytes are identical to what the engine

would hold at serving time; there is no separate, drift-prone serializer. Because the engine’s distributed sharding is a local operation keyed off the parallel rank/size captured at module init (no collective runs on 11

FlashBoot

Sub-second rack-scale weight loading

Table 5. Integration surface. The FlashBoot library is standalone; the engine changes are confined to the loader and weight-handling paths. Component

Contents

FlashBoot native core

arena (VMM), importer, pluggable allocator, ring/chain movers, copy backends (copy-engine / TMA / SM), IMEX preflight, manifest, pinned-shm stager, host mover; one stable C ABI seed / clone / DP loaders, transport, rendezvous, chunk policy, preflight (no torch needed for orchestration helpers) ∼5.8k added / ∼0.3k removed lines across 19 files, concentrated in the model loader, MoE/linear layers, the mxfp4 quantizer, weight utilities, and the remote-instance connector

FlashBoot Python SGLang patch

the load path), we produce all target ranks on a single physical GPU by faking the (rank, size) context per rank, which makes pre-pack a single-GPU offline job even when a full per-rank shard is large. Two layers should be kept separate when reasoning about coverage. The FabricArena layout, the zerocopy views, and the entire FlashClone replication path treat the weights as one opaque contiguous byte image plus an offset table, so they are model- and parallelism-agnostic by construction: any image a seed produces can be mapped and cloned with no per-tensor or per-architecture logic. Coverage is scoped only by the producer of that image, the FlashLoad fast path. Our prototype gates the fast loader to DeepseekV4ForCausalLM and bakes in the DeepSeek-V4 MoE conventions (gate/up → w13 fusion, the w2 down-projection, fp8 block scales, and the mxfp4 expert quantizer); the generic per-shard prefetch we add to SGLang’s linear and fusedMoE layers, by contrast, benefits any model. Extending the fast path to the other MoE families SGLang serves (e.g. Qwen3-30B-A3B, GLM, Nemotron) is a matter of registering each family’s expert-fusion and quantization convention in the pre-pack and scatter, not a redesign; the contiguous-arena machinery beneath is unchanged. On the parallelism axis the fast expert scatter handles tensor parallelism (a per-shard narrow) and expert parallelism (a global→local expert remap keyed off moe_ep_rank/moe_ep_size); data-parallel and DP-attention layouts are captured verbatim by the pre-pack, since they only change which bytes a rank holds; non-trivial expert-placement maps (redundant-expert EPLB) fall back to the engine’s standard loader; and pipeline parallelism, whose per-rank layer subset the current pre-pack does not yet emit, is left to future work. NUMA, pinning, and the host mover. The host path is tuned for the Grace–Blackwell topology. Staging buffers are mmap’d and bound to the Grace socket nearest the target GPU (mbind), and registered with cudaHostRegister, which, unlike cudaMallocHost, keeps the pages on local LPDDR rather than migrating them, so the H2D reads from the near socket. The mover reads files with many threads under O_DIRECT (falling back to buffered I/O if a container/overlay filesystem rejects direct reads at runtime) and issues the H2D as large multi-stream chunks. Peak host memory is held to arena + one staging buffer rather than a second full copy of the weights, which matters because a Pro arena is already 217 GB per GPU; in daemon mode the engine attaches the daemon’s already-staged shm (the same physical pages, no copy) and registers it in place. Three runtime details. Three details proved important in practice. (1) A clone must not use the engine’s native remote-instance load format, which would make the engine handshake the seed’s NCCL send-weights group and block the seed’s own serving; the fabric path dispatches on its own switch and needs only the published manifest. (2) The seed must keep its arena alive for the service lifetime (we hold a strong reference on the served model); if the seed exits, exported handles go stale and clones must re-fetch. (3) All participants must run as the same user sharing an IMEX channel (the fabric security model is per-user), so a one-line preflight check is run on each node, and the system falls back to the existing NCCL path if IMEX cannot be configured.

5 Evaluation We evaluate on the GB300 NVL72 platform of Table 1, with the two models of Table 3 at TP4. All transfer timings measure data movement only (CUDA-event timed; for microbenchmarks we report the minimum over 12

FlashBoot

Sub-second rack-scale weight loading

(a) 1→1 broadcast FlashBoot

(b) 1→N broadcast (5 nodes)

NCCL

800

GB/s

600 400 200 0

B 16M

256

B B 32G 1M

16M

256

B B 32G 1M

16M

256

Figure 14. Fabric copy-engine vs. NCCL, cross-node bandwidth (5 GPUs, 1 per node; per-link for broadcast, busBW for all-gather; critical path = slowest participant; broadcasts swept to 32 GB, all-gather to its 8 GB per-node shard). FlashBoot leads at every size and every collective. The gap is largest for small (latency-bound) messages and for allgather. Table 6. Fabric/NCCL bandwidth speedup. Small messages are latency-bound (fabric copy_d2d ∼22 𝜇s/op vs. NCCL 69–176 𝜇s/op); large messages are bandwidth-bound, where FlashBoot’s copy engine saturates the link higher than NCCL’s SM kernels. Setup is separate: NCCL needs a 10–110 s communicator standup; FlashBoot needs none. Collective

1 MB

32 MB

plateau

plateau BW (fabric vs. NCCL)

1→1 broadcast 1→N broadcast all-gather

3.33× 4.65× 2.07×

1.58× 1.72× 1.60×

1.27× 1.16× 1.49×

826 vs. 651 GB/s (+27%) 740 vs. 637 GB/s (+16%) 740 vs. 497 GB/s (+49%)

30 runs); the one-time fabric map and the start barrier are reported separately and excluded from bandwidth, as they are setup, not per-transfer cost. All reported runs validated correct: chain replication matched the seed 8/8 byte-exact, and the DP loads passed byte-exact arena verification. We answer four questions: how the fabric transport compares to NCCL at the primitive level (§5.1); single-node FlashLoad (§5.2); concurrent multi-node FlashLoad (§5.3); and cross-node FlashClone (§5.4). Where a quantity was measured at several dates we use the most recent. 5.1 Microbenchmark: fabric transport vs. NCCL We first isolate the transport from the loader. On a single node, the contiguous H2D over C2C sustains ∼185 GB/s per GPU, near the 223.5 GB/s C2C ceiling and limited only by it, not by software (it does not inflate when ranks are desynchronized, confirming it is the true contended rate). The more interesting comparison is cross-node, where FlashBoot’s copy-engine DMA replaces NCCL collectives. Figure 14 sweeps payload from 1 MB to 32 GB (to 8 GB for all-gather) over 5 cross-node GPUs for three collectives; Table 6 summarizes the speedups. The plateaus tell the bandwidth story: FlashBoot reaches 826 GB/s on a single link versus NCCL’s 651 (+27%), and on all-gather, where NCCL’s SM-kernel bus bandwidth tops out well below the physical link, 740 versus 497 (+49%). The small-message regime tells the latency story: at 1 MB the fabric copy is 3–5× faster because it carries no kernel-launch or protocol overhead. And neither number includes NCCL’s 10–110 s communicator standup, which FlashBoot eliminates outright (§2.4). 5.2 Single-node FlashLoad With the cold disk read hidden by the preloader daemon (io_wait= 0 on all ranks, §3.5), the engine-visible load collapses to PIN+H2D. Figure 15 shows the per-rank breakdown: every Flash rank is resident in ∼0.4 s and every Pro rank in ∼2.1 s, with H2D running at 181–187 GB/s (the C2C-bound rate). The PIN (cudaHostRegister) cost is not negligible (it is comparable to the H2D), but both are small and fixed. Table 7 places these against the baselines: FlashBoot is up to ∼50× faster than the strongest prior loader (InstantTensor), and up to 77× faster than SafeTensors on Flash from cold disk, where the baseline pays the full disk-read-on-the-criticalpath cost that FlashBoot hides. 13

FlashBoot

Sub-second rack-scale weight loading

(b) Pro, per-rank load (s)

(a) Flash, per-rank load (s) PIN

H2D

0.6 0.4

time (s)

0.2 0

0 rank0

rank1

rank2

rank3

rank0

rank1

rank2

rank3

Figure 15. Single-node FlashLoad critical path (TP4, daemon-staged, io_wait= 0). The engine-visible load is PIN+H2D only: ∼0.4 s per Flash rank, ∼2.1 s per Pro rank; H2D holds 181–187 GB/s. Table 7. Single-node load time (s) and FlashBoot speedup. Baselines are end-to-end weight load; FlashBoot is the daemon-staged PIN+H2D critical path (cold read hidden). Model Flash Pro

Source

SafeTensors

InstantTensor

FlashBoot

30.8 17.7 116.8 67.4

20.1 14.8 72.9 45.8

0.4 0.4 2.1 2.1

cold disk warm RAM cold disk warm RAM

H2D

50× – 77× 37× – 44× 35× – 56× 22× – 32×

(b) all-gather BW (GB/s)

(a) Pro critical path (s) PIN

speedup (vs. IT – ST)

Pro

all-gather

Flash

800 600 GB/s

time (s)

400 200

replicas / DP nodes 𝑁

Figure 16. Concurrent multi-node FlashLoad (TP4). (a) Pro critical path shrinks ∼2.1 s → 0.57 s as PIN/H2D scale ∼ 1/𝑁 and the ring all-gather stays modest. (b) all-gather bandwidth is flat in 𝑁 and in model size (∼732–739 GB/s); each GPU reads one predecessor, so there is no egress contention.

5.3 Concurrent multi-node FlashLoad (shard + all-gather) When 𝑁 replicas come up together, the data-parallel path (§3.6) splits the cold read 𝑁 ways and reassembles over the fabric. Figure 16(a) shows the Pro critical path shrinking from ∼2.1 s (𝑁 =1, no gather) to ∼0.57 s (𝑁 =8): the local PIN and H2D legs each scale as ∼ 1/𝑁 (smaller shard per node), while the ring all-gather adds only a modest, near-constant pass. The off-critical-path cold disk read scales roughly as ∼ 1/𝑁 (in fact a little better, since splitting the read also relieves per-node NVMe contention), from ∼60 s for a whole copy to ∼25 s (𝑁 =2), ∼15 s (𝑁 =4), and ∼5.7 s (𝑁 =8, slowest node), which is the whole point of the scheme: use the aggregate disk and CPU bandwidth of all nodes. Figure 16(b) shows the all-gather bandwidth is decoupled from 𝑁 and from model size: ∼732–739 GB/s for both Flash and Pro at 𝑁 ∈ {2, 4, 8}, because each GPU always reads exactly one predecessor. Adding nodes therefore continues to shrink the critical path, with no bandwidth degradation as 𝑁 grows.

FlashBoot

Sub-second rack-scale weight loading

(a) per-clone bandwidth vs. 𝑁 chain, Pro

chain, Flash

(b) chain-fill BW by position (Pro, 𝑁 =8)

pull (star), Pro

740

800

GB/s

600 400

720

200 700 0

number of clones 𝑁

clone rank (chain position)

Figure 17. Cross-node FlashClone. (a) Per-clone bandwidth vs. clone count: pull degrades as ∼838/𝑁 (seed-egress contention); chain stays flat (∼715 GB/s Pro, ∼705 Flash). (b) The only chain cost is a ∼4% head-to-tail pipeline-fill gradient over the 809 chunks (256 MiB each) of a Pro transfer. Table 8. FlashClone end-to-end per-clone cost (𝑁 =8): a one-time map plus the transfer. Pull’s transfer balloons under shared egress; chain holds near the single-link rate. (Pro 217 GB/rank, Flash 41.4 GB/rank.) Model Pro Flash

Topology pull chain pull chain

map (ms)

transfer (ms)

per-clone BW

7–13 7–13 2–6 3–7

2075 303 395 59

104.6 GB/s 715.4 GB/s 104.9 GB/s 704.7 GB/s

5.4 Cross-node FlashClone Finally we replicate a resident model seed→clones over the fabric. The fabric map is 2–15 ms regardless of payload, replacing NCCL’s 10–110 s. Figure 17(a) is the central scaling result: under the star (pull) topology, per-clone bandwidth collapses as ∼838/𝑁 (Pro: 825 → 418 → 209 → 105 GB/s for 𝑁 = 1, 2, 4, 8), because all clones share one seed GPU’s egress; under chain, it stays essentially flat (806 → 715 GB/s for Pro; 805 → 705 for Flash), because each clone reads a distinct predecessor over an independent link. The two coincide only at 𝑁 =1. Chain’s advantage over pull therefore grows with scale: 1.7× at 𝑁 =2, 3.4× at 𝑁 =4, and 6.8× at 𝑁 =8. Figure 17(b) shows the only cost chain pays: a small head-to-tail pipeline-fill gradient (732 → 705 GB/s along the 8-deep chain, ∼4%), amortized over the 809 chunks (256 MiB each) of a Pro transfer. The end-to-end consequence (Table 8): a full Pro replica (217 GB/rank) is resident in ∼0.32 s by chain regardless of how many clones are filled, versus ∼2.1 s per clone by pull at 𝑁 =8. Set against the production R-Fork path, which serializes a 10 s+ NCCL standup and transfer per clone, bringing up 8 replicas drops from a serialized ≳ 8 × 10.9 ≈ 87 s to a concurrent ∼0.32 s, a > 270× reduction in time-to-serving for the rack.

6 Discussion and Ablations 6.1 Chunk size: the chain’s one knob The pipelined chain (§3.8) has a single tunable, the chunk size, that trades two effects. Larger chunks let each copy-engine DMA run closer to peak, so the head of the chain speeds up monotonically; but they also deepen the fixed pipeline-fill latency, which lengthens the tail. Figure 18 sweeps a Flash chain transfer: the head bandwidth rises steadily from 695 to 748 GB/s as the chunk grows 64 → 2048 MB, but the end-to-end tail (the figure that actually decides “how long until the last clone is ready”) is minimized at 256 MB (60.1 ms). Below it, per-chunk synchronization overhead grows; above it, the fill tail dominates. We therefore default to 256 MB, which also matches the large-transfer optimum seen independently for Pro. 6.2 All-gather scheduling: predecessor-only ring vs. staggered pulls The data-parallel all-gather (§3.6) can be scheduled two ways. A staggered schedule has each node directly pull the other 𝑁 −1 owned ranges in a rotating permutation; a ring has each node read only from its pre-

FlashBoot

Sub-second rack-scale weight loading

(a) chain-fill bandwidth (GB/s) head (rank0)

(b) end-to-end tail time (ms)

tail (rank7)

GB/s

700

600

70 min

60 64

8 12

6 25

2 51

2 10

chunk size (MB)

Figure 18. Chunk-size sweep (Flash chain). (a) the head bandwidth rises with chunk size, but the tail peaks near 256 MB. (b) the end-to-end tail time is minimized at 256 MB: smaller chunks add per-chunk overhead, larger chunks deepen the fill tail.

ring (predecessor-only)

676.9

staggered pulls

582.1

whole-chunk (no pipeline)

575.5 0

200

400

600

mean all-gather bandwidth (GB/s)

800

Figure 19. All-gather scheduling ablation (Pro, 𝑁 =8). Predecessor-only ring beats staggered pulls (more uniform: ±31 vs. ±118 GB/s spread; import 6–9 vs. 30–55 ms), and sub-chunk pipelining beats a whole-chunk store-and-forward ring.

decessor and forward onward, overlapped via flag-pipelined sub-chunks. The ring wins on three counts (Figure 19): higher mean bandwidth, far more uniform per-node bandwidth (because no node is ever read by more than one peer), and an order-of-magnitude cheaper handle import (each node imports one predecessor, not 𝑁 −1 peers: 6–9 ms vs. 30–55 ms). The figure also shows why sub-chunk pipelining matters: a wholechunk ring (one ∼27 GB copy per hop, no pipelining) is store-and-forward and leaves links idle, costing both mean bandwidth and uniformity.1 6.3 Beyond NVL72: NVLink islands and RDMA FlashBoot’s only hard requirement is a primitive for one node to grant another direct read access to a memory region, plus a small out-of-band channel for the access token. On NVL72 we realize this with the CUDA VMM fabric handle (CU_MEM_HANDLE_TYPE_FABRIC) and an IMEX channel, the internode memory-sharing service of multi-node NVLink platforms (§3.7). It helps to separate which parts of the design are platform-independent from the two that our implementation ties to that fabric. The contiguous arena is platform-independent: laying weights out as one large 1-D image addressed by offsets pays off on any interconnect, because a single bulk transfer saturates a link that tens of thousands of per-tensor objects cannot (C1), whether the mover is a copy engine, a GPUDirect P2P read, or an RDMA verb. The fabric-specific pieces are exactly the two that cross nodes: the fabric-handle export/import, which goes through IMEX and so does not span an RDMA boundary, and the cudaMemcpy device-to-device mover, which reaches a remote GPU only because IMEX has already mapped that GPU’s memory into the local address space. Neither survives a move to a platform whose nodes are joined by RDMA rather than NVLink. Table 9 makes this concrete for an H100-class deployment whose nodes are internally NVSwitch1

The absolute bandwidths in Figure 19 are from the scheduling ablation runs (256 MB sub-chunks); the production byte-equal ring of §5.3 reaches ∼737 GB/s with a whole-shard chunk, the configuration we ship. The relative ordering (ring > staggered > wholechunk) is the point here.

FlashBoot

Sub-second rack-scale weight loading

Table 9. FlashBoot on an H100-class platform (intra-node NVSwitch, inter-node RDMA), component by component. Marks: 3 works as implemented; partial needs a small change (e.g. a POSIX-FD share handle); 7 not supported by the current implementation. The contiguous layout is platform-independent; only the cross-node movers are NVLink/IMEX-specific. Component FabricArena contiguous layout + zero-copy views FlashLoad bulk H2D load (CPU→GPU) Fabric handle export/import (FABRIC + IMEX) cudaMemcpy remote map & D2D clone Chain/ring pipelined broadcast (as built) Shard + all-gather DP load (as built) Chain/all-gather over an RDMA backend

Intra-node (NVSwitch)

Inter-node (RDMA)

3 3 partial 3 3 3 —

3 3 7 7 3 3 3

connected and externally joined by RDMA. Within a node the design runs as built: the arena, the bulk H2D load, zero-copy serving, and the cudaMemcpy chain/ring movers all operate over intra-node NVLink P2P (the one adjustment is the cross-process share handle, which on a node without an IMEX/MNNVL fabric becomes a POSIX file-descriptor IPC handle rather than a fabric handle). Across nodes the fabric map and the cudaMemcpy mover do not apply, so cross-node FlashClone and the multi-node FlashLoad all-gather, as implemented, do not run over RDMA. Closing that gap is a backend, not a redesign: register each arena once, exchange the memory key out of band, and issue one-sided RDMA reads in place of cudaMemcpy, so the contiguous layout, NCCL-free mapping, and chunk-pipelined chain carry over with RDMA bandwidth in place of NVLink’s. An RDMA backend does, however, need the connection machinery NVLink let us skip: a warm, reusable communicator or queue-pair pool so the one-time standup is cached across scale-out events instead of re-paid each time (NCCL’s lazy connection establishment and ncclCommSplit reuse, plus its scalable bootstrap, are the relevant primitives [21]), and a GPUDirect/RDMA pipelined broadcast to play the role our chain does. That broadcast is well precedented: pipelined and binomial broadcasts over GPUDirect RDMA are standard in MPI [22], and recent fast-scaling systems for LLM inference move weights this way, e.g. 𝜆Scale’s block-pipelined one-sided-read broadcast [23] and TransferEngine’s pipelined one-sided writes [24]. A cross-platform study with such an RDMA backend on B300/H100 islands is left for future work; we expect the relative advantages to hold and the absolute numbers to track each platform’s link bandwidth. 6.4 Limitations A few caveats bound the results. The seed must keep its arena resident for the service lifetime; if it exits, handles go stale and clones re-fetch. All participants must share an IMEX channel as the same user (a per-user security model), with a graceful fall-back to NCCL when IMEX is unavailable. The contiguous-arena scheme assumes the expert/non-expert separator stays aligned; both evaluation models satisfy this, but a future model that violates it would need a padded separator (and a second H2D for the padded region). Finally, the results are scoped to what we have built out, not what the design admits, along two axes. In software, the FlashLoad fast path targets DeepseekV4ForCausalLM under TP and EP (with DP and DP-attention layouts captured by the pre-pack), so other MoE families, pipeline parallelism, and the cross-node RDMA backend are the extensions identified in §4 and §6.3. In hardware, every number reported here is measured on the GB300 NVL72; the portability to the wider rack-scale class (Vera Rubin POD, AMD Helios, Huawei CloudMatrix384, TPU pods) rests on the platform-independent contiguous arena and is argued structurally (§6.3), not yet measured.

7 Conclusion In elastic deployment of large MoE models, weight loading is a cost worth optimizing, and we showed that on a rack-scale GB300 NVL72 that cost is structural, not a bandwidth shortfall: fragmented per-tensor memory starves the interconnect, NCCL communicator setup dwarfs the transfer it gates, and the production replication path is serial. FlashBoot addresses all three with one coherent co-design: make the weights one contiguous, inter-node–exportable arena and serve them in place, then load from CPU as a single bulk zerocopy transfer with the cold read hidden (FlashLoad), and replicate GPU→GPU by mapping remote memory 17

FlashBoot

Sub-second rack-scale weight loading

in ∼10 ms, with no NCCL, and broadcasting along a chain that scales flat in the number of clones (FlashClone). The result is up to ∼50× faster single-node weight loading, ≥700 GB/s per-clone replication, a Pro replica resident in ∼0.32 s, and > 270× faster concurrent multi-node weight loading than the state of the art. As the rack-scale system becomes the industry’s unit of large-model deployment, we expect these principles, layout first, map don’t communicate, and no single-point bottleneck, to carry across the emerging platforms (§6.3) and to remain a foundation for fast model loading on any memory-mappable accelerator fabric.

References [1] NVIDIA Corporation. GB300 NVL72. Product page, 2024. https://www.nvidia.com/en- us/data- cente r/gb300- nvl72/ . [2] NVIDIA Corporation. NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer. NVIDIA Technical Blog, 2026. https://developer.nvidia.com/blog/nvidia- vera- rubin- pod- sev en- chips- five- rack- scale- systems- one- ai- supercomputer/ . [3] AMD. AMD Helios: An AI Rack Built on Meta’s 2025 OCP Design. AMD Blog, 2025. https://www.amd.com/ en/blogs/2025/amd- helios- ai- rack- built- on- metas- 2025- ocp- design.html . [4] P. Zuo, et al. Serving Large Language Models on Huawei CloudMatrix384. arXiv:2506.12708, 2025. https://ar xiv.org/abs/2506.12708 . [5] Google Cloud. TPU7x (Ironwood). Google Cloud documentation, 2025. https://docs.cloud.google.com/ tpu/docs/tpu7x . [6] LMSYS Org. Tensor R-Fork: Fast Weight Replication for SGLang Remote Instances. Blog post, 2025. https:// www.lmsys.org/blog/2025- 12- 10- rfork/ . [7] Scitix.AI. InstantTensor. GitHub repository, 2025. https://github.com/scitix/InstantTensor . [8] NVIDIA Corporation. NVIDIA Nemotron-3-Ultra Technical Report. Technical report, 2026. https://research .nvidia.com/labs/nemotron/files/NVIDIA- Nemotron- 3- Ultra- Technical- Report.pdf . [9] Hugging Face. Safetensors. GitHub repository, 2023. https://github.com/huggingface/safetensors . [10] L. Zheng, L. Yin, Z. Xie, et al. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104, 2024. https://github.com/sgl- project/sglang . [11] NVIDIA Corporation. NCCL: NVIDIA Collective Communications Library. Software documentation, 2024. http s://developer.nvidia.com/nccl . [12] NVIDIA Corporation. CUDA Driver API: Virtual Memory Management. CUDA Toolkit documentation, 2024. ht tps://docs.nvidia.com/cuda/cuda- driver- api/group__CUDA__VA.html . [13] NVIDIA Corporation. Multi-Node NVLink Systems: IMEX (Internode Memory Exchange) Guide. Product documentation, 2024. https://docs.nvidia.com/multi- node- nvlink- systems/imex- guide/overview. html . [14] DeepSeek-AI. DeepSeek-V4-Pro. Model card, 2026. https://huggingface.co/deepseek- ai/DeepSeek - V4- Pro . [15] Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025. https://github.com/QwenLM/Qwen3 . [16] Meta AI. The Llama 4 Herd. Blog post, 2025. https://ai.meta.com/blog/llama- 4- multimodal- intel ligence/ . [17] Zhipu AI. GLM-4.5 Technical Report. arXiv:2508.06471, 2025. https://github.com/zai- org/GLM- 4.5 . [18] Moonshot AI. Kimi K2 Technical Report. arXiv:2507.20534, 2025. https://github.com/MoonshotAI/Kimi - K2 . [19] Moonshot AI. Kimi-K3. Model card, 2026. https://huggingface.co/moonshotai/Kimi- K3 . [20] OpenAI. gpt-oss-120b and gpt-oss-20b Model Card. Technical report, 2025. https://openai.com/index/int roducing- gpt- oss/ . [21] NVIDIA Corporation. Memory Efficiency, Faster Initialization, and Cost Estimation with NCCL 2.22. NVIDIA Technical Blog, 2024. https://developer.nvidia.com/blog/memory- efficiency- faster- initial ization- and- cost- estimation- with- nvidia- collective- communications- library- 2- 22/ . 18

FlashBoot

Sub-second rack-scale weight loading

[22] C.-H. Chu, K. Hamidouche, A. Venkatesh, et al. Exploiting Hardware Multicast and GPUDirect RDMA for Efficient Broadcast. IEEE Trans. Parallel Distrib. Syst., 2019. [23] M. Yu, et al. 𝜆Scale: Enabling Fast Scaling for Serverless Large Language Model Inference. arXiv:2502.09922, 2025. https://arxiv.org/abs/2502.09922 . [24] TransferEngine: Portable RDMA for LLM Systems. arXiv:2510.27656, 2025. https://arxiv.org/abs/2510. 27656 .

FlashBoot: Sub-Second Weight Loading for Large Models at Rack Scale

Analyze & explore related research in Conceptio

Related documents