The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

arXiv:2608.12114v1 [cs.OS] 12 Aug 2026

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks Yuan Si

Yufeng Lin

University of Waterloo Waterloo, Canada

Independent Researcher Macau, China

Daming Li

Jialu Zhang∗

Independent Researcher Mountain View, USA

University of Waterloo Waterloo, Canada

Abstract

1

Open-weight models can occupy a middle capacity regime: active weights fit in DRAM as cached file pages, but a second framework-owned representation does not fit or must be refilled as layers run, so low-batch decode rereads the weights every token. On integrated and coherent-memory systems the checkpoint’s file pages may already be GPUreadable, yet ordinary accelerator-loading paths copy them into framework-owned allocations before use. We call this the ingestion tax: the operating system holds the bytes as clean, evictable file pages, the memory architecture makes them GPU-readable, and only the framework’s ownership model stands between the two. We present file-backed weight adoption: a frameworkindependent producer maps each tensor with map_shared, wraps the pages as a no-copy GPU buffer, and exports a DLPack capsule that PyTorch or MLX imports as ordinary storage. Zero-copy import alone is not enough: a randomized same-harness factorial establishes a three-part execution contract—read the mapping, keep activations accelerator-resident, order on the GPU; a bolt-on extension that forfeits the last two runs 2.3× slower than stock. Under the contract, adoption removes the tax at no rate cost: the public route reaches 516 GB/s where the frameworks’ default constructors reach 53–82, at parity with the identical kernel over resident storage, from the microbenchmark ([-0.66%, +0.48%]) to a matched Qwen2.5-72B (7.14 versus 7.23 tok/s). Parity buys ownership: the weights stay one set of clean, shared, evictable file pages, so 𝑁 processes decode one mapped copy where resident loading pays 𝑁 —at capacity, 5.5 versus 0.08 tok/s—and a 65 GB checkpoint reaches its first token 6.4× sooner. In Kimi K3, the spine stage falls from 2.62 to 0.35 s per token, 3.8× of it from the storage change alone. The same mechanism improves llama.cpp by 1.21× at roughly half the footprint on an AMD Ryzen 7 9700X APU, ties overlapped streaming on a capacity-exceeding NVIDIA GH200 workload, and loses 39× across PCIe: memory topology, not the API, decides the byte path. The deployment rule treats the page cache as a first-class, reclaimable accelerator-storage tier.

A model whose active weights exceed accelerator-owned capacity can still fit in host DRAM and the page cache. Lowbatch decode then rereads each active matrix per token. Integrated GPUs and coherent CPU–GPU links change what that rereading must cost: the checkpoint’s file pages already occupy a memory domain the GPU can read, so a kernel need not move them to distinct physical memory. Tensor-framework loaders nevertheless make that move in software: constructor or transfer paths allocate frameworkowned storage and copy file-backed bytes into it, and a streaming loader refills it as layers run. We call the transfer the ingestion tax. Unlike offload from storage or a copy into discrete-GPU memory, it creates a second software-owned representation in the same GPU-readable domain: bandwidth and capacity spent without a faster placement. Standalone runtimes such as llama.cpp avoid the copy on Metal because they own both storage and command submission [15]. A tensor framework must also preserve activation placement, ordering, aliasing, lifetime, and mutation semantics; DLPack and MLX’s zero-copy Metal import [25] do not by themselves make a checkpoint mapping a fast, correctly ordered framework tensor. Our first implementation removed ingestion this way yet ran 2.3× slower than stock: synchronization against the framework’s in-flight work cost more than the copy it removed (§2.1). This paper develops file-backed weight adoption: map a tensor with map_shared, wrap the pages as a no-copy GPU buffer, export a DLPack capsule, and import it as ordinary framework storage. The problem and the mechanism live on two boundaries. On the operating-system side, mapping semantics, page-cache residency, and clean-page reclaim decide what storage is: adoption makes VM-managed file pages first-class accelerator storage, shared and evictable under existing kernel policy. On the architecture side, memory topology (unified, coherent-link, or discrete) decides whether reading that storage in place is profitable at all. The contributions track those two boundaries: an execution contract and causal decomposition at the first, a topology-driven placement rule at the second, and a public mechanism that connects them.

∗ Corresponding author.

1

Introduction

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

1. A cross-layer execution contract and causal decomposition. We separate three requirements: adopt the mapped pages (C1), retain activations in the accelerator-visible domain (C2), and establish dependencies on the GPU (C3). A randomized same-harness factorial removes each condition while holding the kernel, tensor shapes, allocator, and host path fixed. Removing C1, C2, or C3 adds +12.3, +5.6, and +3.4 ms per pass, and the full 23 model resolves one interaction: ordering only costs where activations are resident. A campaign across 8 processes prices the ordering rungs (in-stream 0.33 ms, foreign-queue GPU event 0.44, host drain 5.80), so the boundary is host synchronization, not queue ownership alone. 2. A public adoption path with end-to-end impact. A framework-independent producer turns aligned file mappings into versioned Metal DLPack tensors. The public PyTorch path reaches 516 GB/s under the matched protocol, and MLX reaches 517.5 GB/s through the same capsule. In Kimi K3, a packed kernel over adopted storage takes the dense stage from 2.62 to 0.35 s (a 3.8× storage-only step) and full-token latency from 4.14 to 1.87 s. A matched Qwen2.5-72B evaluation reaches 7.14 tok/s through adoption versus 0.94 tok/s through per-use ingestion; the adopted run matches a load-once resident control (7.23 tok/s) while the weights stay reclaimable. Rebinding the parameters of stock transformers modules preserves their model code and probe logits bitwise, and a capacity-scale Qwen2.5-32B reaches its first token 6.4× sooner. 3. A topology- and capacity-aware deployment rule. The same loader decision improves llama.cpp by 1.21× and removes its duplicate model footprint on an AMD Ryzen 7 9700X APU, falls within the 5% policy band of overlapped streaming for a capacity-exceeding NVIDIA GH200 workload, and is rejected on a NVIDIA RTX 5070 Ti because every mapped read crosses PCIe. Full-working-set endpoint probes, applied retrospectively, match all four observed outcomes, and a fifth held-out Apple machine exercises the capacity gate, the tie band, and map-versus-copy across two sessions.

2

Regime and execution contract

2.1

The motivating deployment

1.446 TB pool (the routed subset cached on local SSD), plus a dense int8 “spine” of 1253 matrices totaling 52.8 GiB that fits the page cache. Decode was slow, and the SSD is the obvious suspect. Measurement says otherwise. An identity ablation, which preserves byte reads but removes layer arithmetic, attributes 86% of decode time to the byte path and 14% to arithmetic. The byte path’s largest share belongs not to the disk but to the spine, whose bytes already sit in DRAM: on every token the framework re-ingested them into its own allocation, widening the packed int8 checkpoint to fp32 as it copied, and that stage alone took 2.62 s of the 4.14 s token. The fix is not the obvious one. Our first implementation mapped the weights and dispatched from its own queue outside the framework; it removed the copy and ran 2.3× slower, because the synchronization it incurred against the framework’s in-flight work cost more than the copy had (§4). Rebuilt instead as a packed-int8 kernel over weights adopted into the framework’s own execution path, the stage runs in 0.35 s and the token in 1.87 s. Packing alone reaches 1.34 s; adoption takes the same kernel the rest of the way, a 3.8× storage-only step (§6). The 2.27 s recovered per token exceeds the 1.52 s that remain of the token: attention, sampling, and all 24.0 GiB of expert streaming combined. The bottleneck was software: a re-representation of bytes the GPU could already read, costing more per token than the machine’s real storage traffic. The routed experts stay disk-bound and are unchanged by this work. 2.2

Where the tax binds

The regime is bounded on both sides. If the active weights fit framework-owned storage, one copy amortizes over many tokens; if the active set does not fit the page cache, storage traffic dominates and no in-memory byte path helps. Between those endpoints, the active files are GPU-readable but a second allocation either does not fit or must be refilled as layers run. The relevant capacity is not installed memory alone: the copy competes with the mapped checkpoint, activations, the OS, and dtype materialization. Adoption sidesteps that competition, leaving the tensor’s storage as clean file pages the OS may share or evict. 2.3

Three distinct costs

The stock path can combine three costs that must not be conflated. Ingestion copies file-backed bytes into frameworkowned storage. Representation expansion converts a packed checkpoint into a wider kernel format. Execution handoff moves activations or waits on the host when a streamed kernel runs outside the framework’s dependency graph. Filebacked adoption removes ingestion, but it does not by itself provide a packed kernel or correct ordering. Figure 1 maps the two byte paths onto these costs; Table 1 states the required execution contract as three conditions.

Running open-weight models locally is now routine, and the largest models get there through a substitution: what does not fit memory streams from storage on demand. Kimi K3, a 2.8-trillion-parameter mixture of experts (93 layers, 896 routed experts, 16 selected in each of 92 routed layers), runs on one Apple M5 Max with 128 GB of unified memory only this way: each token reads 24.0 GiB of experts out of a 2

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

Weights on this machine

copy

One file per tensor, page-aligned. The page cache holds them as clean file pages: shared, evictable, never written back. The bytes are already in DRAM. The rest is bookkeeping.

map

Per-use ingestion

Framework pool

mx.array(...) · .to("mps") constructors or staged copy allocate framework storage

a second representation: anonymous, exclusive, reclaimed under VM pressure

Mapped-file producer

Framework adopts it

mmap(…, MAP_SHARED) newBufferWithBytesNoCopy(…) → kDLMetal DLPack capsule

torch.from_dlpack(capsule) → an ordinary MPS tensor; kernels join its own stream

copy-limited two representations

read-ceiling

C1 adopt the mapping

C2 retain activations

one representation

C3 order on the GPU

Figure 1. Two byte paths from the same clean file pages: per-use ingestion builds a second, framework-owned representation; adoption republishes the pages as ordinary framework storage. Condition

Requirement

How weights reach the kernel

C1: adopt

The kernel reads the mapped file pages; no per-use framework copy is created. C2: retain Inputs and outputs remain acceleratorvisible across layer calls; activations do not round-trip through host arrays. C3: order on GPU Dependencies are established in the framework stream or with an on-GPU event; the host does not drain outstanding work.

GB/s live copies

MLX constructor 53 PyTorch .to("mps") 61–82 pipelined, double-buffered copy 134

2 2 2

public DLPack adoption

1

516

Table 2. How weights reach the kernel: one task, one machine.

Table 1. The execution contract. so the constructor gap opens before GEMV execution. Placement forensics agree: constructor paths grow anonymous framework-owned storage, while adoption leaves the clean mapped pages as the only weight representation.

A standalone runtime normally owns all three conditions. In a tensor framework, the mapped pages begin outside the allocator and scheduler. The core design problem is therefore to make those pages ordinary framework storage, after which the framework can supply C2 and C3 itself.

3

Characterizing the ingestion tax

3.1

Matched protocol

3.2

The latent route

MLX documents mx.array as copying and its Metal DLPack import as zero-copy [16, 25]; PyTorch likewise exposes DLPack import and MPS custom kernels. Neither stack supplies a producer that turns a file mapping into a Metal capsule; ours fills that gap. The same capsule reaches 516 GB/s through torch.from_dlpack and 517.5 GB/s through mx.from_dlpack, versus 506.1 GB/s for the same MLX kernel over its resident allocation; the paired mappedminus-resident difference is +0.7% with interval [-1.6%, +9.3%]. Mutation-visibility checks establish storage identity; file checksums and kernel output oracles confirm that the adopted tensor reads the mapped bytes. A rotated paired campaign measures adoption against the identical kernel over resident weights. Across 10 independent processes, the medians are 516.2, 517.8, and 522.2 GB/s for the mapped, resident, and pure-read arms. The mappedminus-resident difference is +0.16% with interval [-0.66%, +0.48%]; mapped-minus-pure-read is [-1.41%, -0.23%] (all intervals in this paper are 95% bootstrap intervals over process medians). The data bound any mapped-versus-resident loss

Every Apple-silicon rate in this section uses one task: a singletoken int8 GEMV over 12288×7168 weights (88.1 MB) with fp16 row scales. A pass visits 12 page-cache-resident files and brackets the batch with one synchronization; the protocol discards 12 warm-up passes and reports the median. Headline cells replicate in 15 processes with arm order randomized, and dispersion is the IQR across process medians. The system is an Apple M5 Max (128 GB unified memory, macOS 26.6, PyTorch 2.13, MLX 0.29.3; MLX 0.32.0 for its DLPack row). Rates count weight bytes once, the traffic common to the compared kernels. Table 2 places the routes against the protocol’s 517 GB/s pure-read ceiling: constructors reach 53–82 GB/s, the strongest staged copy 134, and adoption 516. The resident kernels are not the bottleneck: with weights already in each framework’s pool, MLX reaches 497 GB/s and our packed-int8 kernel 523, both near the pure-read ceiling, 3

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

to roughly one percent while resolving GEMV’s small extra traffic against a read-only kernel. No single instrument establishes placement, aliasing, correctness, and throughput at once; the full instrument chain (matched-kernel wall time, residency probes, mutation visibility, checksum and output oracles, and the paired campaign) is tabulated in Appendix A. Standalone Metal runtimes already exploit mapped pages because they own storage and ordering [15]. A framework must instead make the mapping ordinary storage, so that activation placement and dependency tracking stay inside it; the next section prices those conditions.

(a) idle queues, 15 processes

4

(b) + in-flight framework load, 8 processes

A same-harness causal decomposition

GB/s

IQR

map, device, stream map, device, own map, round trip, own map, round trip, stream copy, device, stream copy, round trip, stream copy, device, own copy, round trip, own resident, same kernel stock framework path

489 526 265 169 76 60 57 54 523 61

[484, 491] [519, 528] [265, 269] [166, 169] [74, 79] [58, 60] [56, 58] [54, 55] [520, 526] [55, 61]

excess ms/pass

IQR

Cell

We vary the three conditions of Table 1 in one Python host path, one allocator, one compile_shader kernel, and one framework stream. C1 toggles adopting versus re-ingesting the mapped pages; C2, accelerator-resident activations versus a host round trip; C3, GPU-side order versus a host-visible drain. Cell order is randomized independently in every process. Each toggle is an implementation substitution (the alternative a builder would write), so the campaign licenses claims about this execution model, not a necessity theorem over all frameworks and backends. 4.1

Cell (weights, activations, queue)

C1+C2+C3 remove C3 remove C2 remove C1 remove all

0.04 [-0.23, 0.23] 3.46 [3.28, 3.73] 5.62 [5.18, 5.68] 12.38 [12.15, 12.65] 17.90 [17.53, 18.22]

Table 3. The decomposition: (a) idle queues, (b) against inflight framework work.

The isolated cube

4.2

Before the causal campaign, we enumerate the implementation space in an idle queue. The cube varies mapped versus copied weights, device-resident versus round-tripped activations, and framework-stream versus own-queue dispatch. The Metal source and tensor shapes are identical; only storage and handoff change. The idle view gives the wrong deployment answer for a principled reason. Every copy cell pins at 54–76 GB/s, and own-queue mapping comes out on top because it batches dispatches and waits once, as a standalone runtime does, exploiting the full read path. Inside eager framework execution, the same storage must instead order against work already in flight, and a host-visible exchange drains it. The queue win of Table 3(a) becomes a synchronization loss because the benchmark omitted the state the storage must be ordered against. Against an in-flight framework load (Table 3(b)), the full cell leaves no resolved excess, and removing any one condition incurs a cost well outside its IQR. The full 23 model over process-level paired contrasts (Appendix F) puts the main effects at +12.49, +3.72, and +1.86 ms, with every interaction spanning zero except C2×C3 (1.79 ms, [1.67, 2.20]): the only stall that hides another is the host round trip masking the ordering cost, which is why C3’s single-condition ablation (+3.4 ms relative to the full cell) exceeds its averaged main effect. Adoption’s effect is additive with both.

Ordering is about host visibility

C3 is not synonymous with “the framework’s stream.” A second randomized campaign splits it into stream order, a plain on-GPU event, and a host drain. With C1 and C2 enabled, the three cells cost 0.26, 0.54, and 2.27 ms. The paired eventminus-stream interval is [-0.14, +0.95]; the experiment does not resolve those two GPU-side mechanisms. The host drain is resolved as slower, with paired interval [+1.41, +2.63]. The operational requirement is therefore to keep dependency establishment on the GPU, while in-stream encoding remains the fastest measured implementation. This loaded-queue setting also explains why a perdispatch rate can improve when more framework work is present: fine-grained weight reads fill encoder gaps, and a dense queue raises the operating clock. GPU timestamps show serialized encoder intervals, not simultaneous kernels (Appendix A): queue occupancy, not overlap, is the state variable the idle cube lacks. A separate campaign prices four concrete rungs for the same pass across 8 independent processes, with the foreignqueue cells driving a standalone Metal dispatcher against the framework stream: in-stream 0.33 ms, plain MTLEvent on the foreign queue 0.44, MTLSharedEvent 3.46, and host drain 5.80 (medians of process medians; intervals in Appendix G). The foreign-queue event costs +0.17 ms against in-stream (paired interval [+0.02, +0.21]), an order of magnitude below any host-visible mechanism: cost begins where a signal becomes host-visible, not where the queue changes owner. 4

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

Algorithm 1 The file-backed tensor producer.

With C2 off, the per-call round trip has already serialized the pass and the orderings collapse together (Appendix F); C3 only has a price where C2 is being paid (the C2×C3 term above). 4.3

Require: tensor file; shape (𝑀, 𝐾); dtype 𝜏 1: 𝑛 ← size(𝑝𝑎𝑡ℎ) 2: if 𝑛 ≠ 𝑀·𝐾 ·sizeof (𝜏) or 𝑛 mod pagesize ≠ 0 then 3: reject 4: end if 5: 𝑝 ← mmap(𝑝𝑎𝑡ℎ, 𝑛, prot_read, map_shared) 6: 𝑏 ← NewBufferWithBytesNoCopy(𝑝, 𝑛) 7: close the descriptor 8: 𝑐𝑡𝑥 ← ⟨𝑏, 𝑝, 𝑛⟩ 9: 𝑡 ← ⟨𝑑𝑎𝑡𝑎=𝑏, kDLMetal, 𝜏, 𝑠ℎ𝑎𝑝𝑒=(𝑀, 𝐾)⟩ 10: 𝑡 .𝑠𝑡𝑟𝑖𝑑𝑒𝑠 ← (𝐾, 1); 𝑓 𝑙𝑎𝑔𝑠 ← read_only 11: 𝑑𝑒𝑙𝑒𝑡𝑒𝑟 ← 𝜆 : Release(𝑏); munmap(𝑝, 𝑛) 12: return versioned capsule ⟨𝑡, 𝑐𝑡𝑥, 𝑓 𝑙𝑎𝑔𝑠, 𝑑𝑒𝑙𝑒𝑡𝑒𝑟 ⟩

Why the isolated microbenchmark misleads

With idle queues, a standalone mapped path reaches 526 GB/s and can look better than the framework-stream cell at 489 GB/s. The live system reverses that conclusion: mapping on a foreign queue while round-tripping activations takes 5.96 s, versus 2.62 s for stock ingestion. The negative configuration has C1 but lacks C2 and C3, so the synchronization it adds costs more than the copy it removes. The production-spine replay gives the same ordering over 1253 real matrices. The full configuration takes 0.115 s; removing C3, C2, or C1 multiplies that time by 2.7, 4.5, and 17.0, respectively, while stock fp32 takes 32.7× the full cell. This is the central execution lesson: mapping is useful only after the mapped storage participates in the framework’s activation and ordering rules. C2 and C3 describe what ordinary framework code already does. A tensor produced by a framework operator stays in the accelerator-visible domain, and work encoded into the current stream is ordered by the framework’s own dependency tracking. An implementation forfeits both by leaving the framework, and mapped storage is the occasion to leave: a private queue and a host-observable completion are the natural shape of a bolt-on extension, and they are what the negative configuration above measures. The contract therefore states where adoption belongs. Inside the execution path, the last two conditions hold by construction, which is why the all-public route of §5 satisfies the contract with no extension symbol in the process.

5

Implementation

5.1

A framework-independent producer

follows the consumer. The capsule owns exactly one mapping and one buffer; nothing is cached across calls. Its deleter, invoked when the adopted tensor dies or the capsule is never consumed, releases the buffer and unmaps; it touches no Python state and may run from any thread. Tests enforce content identity and file-write aliasing, misalignment rejection, tensor lifetime beyond producer references, and 200 consumed plus 200 unconsumed cycles with a flat footprint. Under the identical public kernel, both producers read at the same rate (526.1 GB/s vs. 521.8 GB/s, paired +0.83%), with bitwise-identical GEMV outputs. The shared mapping is semantically important. A private mapping is converted to anonymous pages when Metal wires it, recreating the copy at model scale: a 60 GB private mapping produced 58 GB of anonymous memory, 10× slower wiring, and a 3× throughput loss (Appendix B). The trap is macOS-specific; Linux and Windows keep private file mappings file-backed. Page alignment is also a real format constraint: container checkpoints need aligned tensor offsets or a one-time relayout into per-tensor files. Our converter performs that relayout once; adoption then maps and wires the 2506-file spine in 1.37 s at 137 GB/s. DLPack’s read-only flag conveys producer intent, not enforcement: both tested consumers import the versioned capsule and still admit in-place stores. The substrate then determines the failure mode. In our probes, a GPU store to the read-only Metal mapping is silently discarded, the CUDA path faults, and the tested Windows Vulkan drivers reject the read-only view during import. A framework-level mappedstorage interface should propagate immutability and reject mutating operations before dispatch; DLPack import alone cannot guarantee that behavior.

The public path is a 226-line C extension that compiles against Metal, POSIX, and CPython only: no PyTorch or MLX headers, and no linkage against either framework. Each call opens one page-aligned tensor file, maps it read-only with map_shared, wraps the range using newBufferWithBytesNoCopy on the system Metal device, and returns a versioned kDLMetal capsule with the read-only flag set and explicit compact strides. torch.from_dlpack or mx.from_dlpack then adopts the capsule as ordinary framework storage; the MLX import runs without PyTorch installed. Algorithm 1 states the producer; every guard corresponds to a measured failure mode (§5.3, Appendix B), not a defensive default. Line 2 rejects partial pages because Metal wires whole pages and a partial page would alias a neighboring tensor; line 9 uses explicit strides because DLPack ≥1.2 forbids null strides and one consumer faults on them. Lifetime

5.2

Framework execution

For read-only downstream operations, the imported tensor participates as ordinary MPS or MLX storage. PyTorch’s custom-kernel surface encodes against it in the framework stream, supplying C2 and C3 without private scheduler 5

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

hooks. This all-public path (standalone producer, public import, public kernel surface) is the primary implementation. Every matched-campaign adoption rate on Apple silicon is measured through it, with two labeled exceptions: the Kimi K3 system integrates at the extension layer below, and the held-out machine’s first session ran the extension’s producer core (rate parity above). An internal-symbol extension remains for the multi-token kernel, the ordering mechanism study, and the Kimi K3 production integration; a spine-specialized sibling supports the replay harness. It caches mappings by path for the life of the process (an instrument choice, not the deployable lifetime discipline) and encodes into PyTorch’s current Metal encoder on the stream’s serial dispatch queue. It does not end the encoder, allowing the framework to coalesce surrounding work. Its packed-int8 GEMV serves up to 8 activation columns so speculative batches reuse each weight read, and reaches 523 GB/s on resident weights; adoption changes storage ownership, not the arithmetic. Mechanism studies that exercise this extension’s own execution paths (the factorial’s foreign-queue and ordering cells, the batch sweep) use it by construction; every adoption-route claim rests on the standalone producer.

RQ3: how does adopted storage behave under memory pressure? RQ4: do the mechanism and the endpoint procedure of §8 transfer across architectures (§7) and to a machine neither was built on? Shared protocol controls are collected in §6.2.

5.3

Protocol controls are part of the experimental design (Table 5). Apple GPU frequency ramps over several passes, unequally across mechanisms, so every Apple campaign discards 12 passes and rotates arm order; timing the ramp can understate the mapped arm several-fold. The trap is platformscoped: an A10 audit finds every CUDA cell at steady state by pass 2, and applying the Apple discard moves no CUDA number by more than 0.1% (Appendix A); data-center parts under load do not idle their clocks the way a fanless laptop does. In the matrix-size sweep the mapped path rises from 28.9 to 550 GB/s while the copy path plateaus near 87; dispatch cost dominates below roughly 23 MB, and even at the smallest point mapping leads by 4.4×. The endpoint itself must also match deployment: the first APU probe recycled a resident buffer smaller than the model, measured 53.5 GB/s where the full-set arm measures 30.2, and predicted the wrong policy (Algorithm 2). All three individual measurements were numerically stable; the error was semantic, not noisy.

6.1

The adopted stage falls from 2.62 to 0.35 s; complete-token latency falls from 4.14 to 1.87 s (2.2×), the remainder being routed-expert storage traffic this work leaves unchanged (Table 4). Fixed routing and oracle-token regressions produce byte-identical outputs across the compared byte paths. A rotated same-process session (mapped 518.9, resident 520.3, pure read 522.6 GB/s) agrees with the independentprocess result of §3, ruling out session drift. At scale the mapped path rises to 578 GB/s across 4 working-set points spanning 2–16 GB (Fig. 2a), above the survey’s dispatchbound reference, while dispatch granularity governs the approach (Fig. 2b). Memory pressure marks the regime boundary: §6.7 applies the residency check before selecting a byte path, and the RQ3 results below quantify it. 6.2

Constraints that determine correctness

Four implementation details are externally visible at model scale. First, the mapping must remain shared; the failure is subtle because small tests return correct values and the hidden anonymous copy appears only when the GPU wires a large range. Second, the imported base and length must satisfy Metal’s page constraints: per-tensor files make this explicit, and an aligned container format would provide the same property without the inode count. Third, command encoding must follow the framework’s serialization rule: the internal path enters PyTorch’s stream dispatch queue before touching the current encoder and leaves the encoder open. Encoding from an arbitrary host thread can race the framework and terminate the process; ending the encoder per weight destroys the batching that makes C3 effective. The public DLPack path sidesteps the constraint because the framework is the sole encoder. Fourth, lazy execution must be forced at the measurement boundary: an unevaluated MLX result reports rates above physical bandwidth while measuring dead-code elimination. Every MLX arm evaluates its outputs before synchronization, and the paired campaigns rotate arm order within a pass so clock ramp, allocator state, and page residency are shared.

6

RQ1: the motivating system and the mapped ceiling

Protocol controls and falsification

Spine path stock fp32 ingestion same kernel, copied int8 mapped + host round trip adopted, in-stream

Evaluation

The evaluation asks four research questions. RQ1: does adoption remove the tax end to end without costing throughput against resident storage? RQ2: what does adoption require of model code, and what does it change at capacity and startup?

stage (s) token (s) 2.62 1.34 5.96 0.35

contract

4.14 none — C2+C3 — C1 only 1.87 C1+C2+C3

Table 4. The Kimi K3 spine, live (3 runs).

6

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks (b) dispatch granularity

(a) mapped working set

GB/s

GB/s

600 550 500 0

5

10

200 0

15

working set (GB)

mapped copy

400

100

101

102

matrix size (MB)

Figure 2. The public mapped path at scale: (a) the working-set sweep, (b) dispatch granularity. Threat

Control (claim it protects)

clock ramp cache-sized endpoint kernel confound format denominator page-cache eviction dead lazy graph

discard 12 passes, rotate arms (rates) hold the full deployed set (policy) one mapped/resident GEMV (attribution) count tensors read per token (rates) residency queried before/after (regime) force output evaluation (MLX rates)

differs in what remains afterwards, 71.48 GB of reclaimable file pages versus a second owned copy. The internal-symbol route reaches 6.13 tok/s with a wider [4.30, 6.81] interval from per-layer host work. Finding (RQ1): Adoption removes the tax at no kernel cost: 516 GB/s on the public route, mapped-minus-resident bounded in [-0.66%, +0.48%]. On Kimi K3 the redesigned spine stage is 7.5× faster (3.8× from storage alone); on a matched Qwen2.5-72B, adoption is 7.6× faster than per-use ingestion.

Table 5. Controls that falsify alternative explanations.

6.4 Qwen2.5-72B int8

Storage adoption does not require a bespoke model implementation. Rebinding each two-dimensional parameter of a stock transformers model through load_state_dict with assign=True preserves its modules, F.linear, and generate. On Qwen2.5-7B, across 5 process rounds per arm with arm order alternating, the last-position logits over a fixed probe sequence are bitwise identical between the mapped and stock arms and across every run, and decode rates match (23.0 versus 22.9 tok/s). The process’s peak resident set is 0.5 GB against 17.9 GB: the weight bytes still occupy DRAM, but as shared, reclaimable page-cache pages rather than process-owned allocations. At capacity, an unmodified Qwen2.5-32B checkpoint occupies 65 GB of the 128 GB machine. A 8-round campaign runs three arms, one process each, order rotated: the stock loader; rebinding over page-aligned per-tensor files; and a copy control sharing the rebind arm’s files and construction but materializing resident copies. All three decode at 6.9 tok/s with bitwise-identical probe logits; they differ before the first token. Stock spends 32.59 s parsing shards and constructing modules and 4.51 s copying: first token at 37.95 s. Rebinding reaches it at 6.97 s; the median per-round paired TTFT ratio is 6.4×, interval [5.4, 11.2]. The map itself takes 0.21 s, and the first token absorbs the page-fault wiring (6.75 s of first-token latency, not load time). The copy control lands at 8.42 s: most of the stock gap is the relayout removing shard parsing, the storage change contributes 1.6× on identical files, and what persists afterwards is still reclaimable file pages. A cold-state variant (3 rounds, the page cache evicted before every arm) preserves the picture: 32.1/6.9/7.2 s, per-round ratio 5.6×.

tok/s logical GB/s storage

public adoption 7.14 load-once resident 7.23 internal extension 6.13 stock per-use ingestion 0.94

510 page cache 517 owned copy 438 page cache 67 transient copy

Table 6. Matched Qwen2.5-72B runs, 8 processes per arm (resident 𝑛=4).

Correctness is checked at three levels: mutation visibility establishes storage aliasing; checksums verify the kernel reads the intended mapping and that eviction preserves bytes; end-to-end runs compare logits, greedy tokens, or fixed-oracle outputs. Together they prevent a fast result from resting on a stale tensor, dead graph, or skipped layer. Matched comparisons share prompt, token count, format, kernel, process count, and arm rotation; native MLX and llama.cpp runs use different formats or protocols and serve as engineering references, not parity baselines. The GH200 rebind model establishes placement and compatibility, the APU patch the production-loader integration; each result is stated at the level its instrument identifies. 6.3

RQ2: stock modules and capacity

RQ1: a matched dense model

The public route reaches 7.14 tok/s with interval [5.76, 7.20], versus 0.94 tok/s with interval [0.85, 0.97] (Table 6; logical rate is 71.48 GB per token times tok/s). A load-once resident control (the identical kernel over a one-time owned copy of the same int8 bytes) reaches 7.23 tok/s: adoption gives up no decode rate against the strongest feasible resident arm, and 7

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

6.5

RQ2: startup and repeated process cost

6.7

The production spine maps and wires its 2506 files in 1.37 s; later tokens reuse the mappings. Every stock launch reingests the checkpoint, while adoption rebinds module storage to the file pages, which is why decode rates match while time to first token differs by 6.4×. For local interactive inference, first-token latency is user-visible, not an amortized provisioning event.

The 8 GB held-out machine has run the procedure twice, in different memory states, exercising different branches. Session one (probe 56.0/56.9/8.4 GB/s, coexistence 4/5, extension producer): the capacity gate removes resident, and map beats per-use copy. Session two, over the standalone producer with more headroom (59.3/60.7/7.7, coexistence 5/5): the leaders fall inside the 5% band, which resolves to map. Both predictions precede the end-to-end runs and agree: 25.4 and 30.6 tok/s through adoption versus 0.90 and 0.99 through stock ingestion, identical tokens in both sessions. The larger 2.74 GB working set fails the residency precondition: after the framework copy is created, only 0% of the mapping stays cached and every path approaches storage throughput. The machine does not test a case where resident storage fits reliably and beats mapping.

Finding (RQ2): Adoption needs no bespoke model code: rebinding stock transformers modules leaves probe logits bitwise identical while a 65 GB checkpoint reaches its first token 6.4× sooner (1.6× from storage alone under identical files), and the weights stay reclaimable.

6.6

RQ4: the held-out machine

RQ3: page-cache citizenship under pressure

Adoption changes more than the steady-state rate: clean mapped pages are reclaimable without write-back, while an anonymous framework copy must be compressed, swapped, or retained. We track the distinction with mincore bitmaps, VM counters, and GPU checksums after eviction. Under cold anonymous pressure the OS removes the idle competitor and preserves the mapping touched on every pass; only the first recovery pass pays storage faults. Under a hot competitor the two sets evict one another once their sum passes roughly 75–80% of installed memory in our sweeps (a descriptive threshold for this machine and pressure pattern, not a fitted constant), and alternating passes fall to storage speed. Checksums after complete eviction reproduce the file values: reclaim is transparent, not a silent data-loss path. The same ownership difference multiplies across processes. 4 concurrent Qwen2.5-7B decoders over one adopted checkpoint add 1.1 GB of anonymous memory and no file-backed bytes (every process reads the same shared clean pages), while four resident loaders add 32.6 GB of duplicates and decode slower besides. At capacity the contrast is terminal: two Qwen2.5-32B processes share one 65.5 GB mapped copy and decode at 5.5 and 3.6 tok/s, while two resident copies (+61.2 GB anonymous) push the machine into compression, evict 30.9 GB of the shared cache, and collapse to 0.08 tok/s each (Appendix J).

Finding (RQ4): The endpoint procedure transfers: two sessions on a machine it was not built on exercise the capacity-gate and tie-band branches, and both predict the end-to-end winner: 25.4/30.6 versus 0.90/0.99 tok/s, token-identical, gaps of 28–31×.

7

Architecture boundaries

File-backed adoption is a memory-topology optimization, not a universal loader replacement. We exercise the same choice on an integrated Vulkan GPU, a coherent CPU–GPU link, and a discrete PCIe GPU. 7.1

Integrated Vulkan: the tax and the fix

On an AMD Ryzen 7 9700X APU, llama.cpp’s Vulkan backend reports unified memory yet its stock loader retains both the mapped model and a Vulkan allocation in one DRAM pool. We connect the loader’s existing buffer_from_host_ptr hook to VK_EXT_external_memory_host, advertised only on unified-memory devices; loader and inference loop are unchanged. The driver caps a single imported allocation, so the 4.36 GiB model is split into 3 files with llama.cpp’s own tool. Across 4 paired processes, adoption raises decode from 2.82 to 3.42 tok/s, a 1.21× improvement, while peak working set falls from 8.13 to 4.20 GiB (Table 7). Relative to the model file, those footprints are 1.86× and 0.96×. Greedy generation is byte-identical. The platform also exposes a cumulative importable-host-memory ceiling of 12.5 GiB; sharding fixes the per-allocation limit but not that capacity boundary. The footprint difference becomes latency under pressure: with 10.3 GiB free and a 8.4 GiB model, stock peaks at 14.7 GiB and completes in 58.8 s; adoption peaks at 8.1 GiB and completes in 22.4 s, the cost of carrying a duplicate inside one DRAM pool. A deployment-scale endpoint probe explains the result: mapping reaches 34.6 GB/s, a whole-working-set resident buffer 30.4 GB/s, and per-use copy 6.9 GB/s, so mapping leads the resident endpoint by 1.14×. An earlier small-buffer probe

Finding (RQ3): Adopted weights are page-cache citizens: under cold anonymous pressure the OS pages out the competitor and keeps the touched mapping resident; eviction is transparent and checksums survive re-fault; and 𝑁 processes share one mapped copy where resident loading pays 𝑁 (at capacity, 5.5 versus 0.08 tok/s).

This is why “fits in RAM” is not a capacity test: checkpoint, framework copy, activations, kernel caches, and the OS compete for one pool. Adoption removes a claimant, and the benefit is largest near capacity, where the default loader repeatedly creates the representation most likely to trigger compression or paging. 8

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

paired decode (tok/s) peak working set (GiB) pressure run wall time (s) pressure-run peak set (GiB) deployment endpoint (GB/s)

default copy / bus

stock / resident adopted / mapped 2.82 8.13 58.8 14.7 30.4

GB/s of weight

APU metric

3.42 4.20 22.4 8.1 34.6

101 100 discrete (5070 Ti)

Figure 3. The same three byte strategies on both memory architectures (log scale). On discrete hardware, mapping is not competitive (48 vs. 815: the interconnect decides) and staging into VRAM is the right policy. On unified memory, mapping reaches resident-class bandwidth, while the frameworks’ default copy path (61 GB/s) restages almost exactly the discrete bus rate (57) on hardware that has no bus.

overstated the resident arm by 1.8× because it repeatedly read a cache-sized working set instead of holding the deployed model. Holding the byte total (4096 MiB) and dispatch shape fixed, one import and 3 imports produce identical map-to-resident ratios of 1.174 and 1.174. Sharding changes feasibility; the full-working-set endpoint probe determines performance.

7.3

Discrete PCIe: reject adoption

On a NVIDIA RTX 5070 Ti, a mapped host read reaches 48 GB/s against a 815 GB/s resident kernel; forcing the same llama.cpp adoption path across PCIe lowers decode from 146.9 to 3.79 tok/s (39×). Here the copy buys a real placement change and should be retained: same patch, same format, same loop as the APU; topology changes the verdict. Figure 3 sets the two architectures side by side, and the comparison explains where the default came from. The bus imposes 57 GB/s on the discrete part; the unified-memory machine’s default constructors deliver 53–82 GB/s, and its best staged copy 134, on hardware that has no bus and whose memory serves the same kernels at 516 GB/s through a mapped buffer. The frameworks treat unified memory as if a PCIe link of roughly this speed were still there. The offloading literature’s copy-scheduling machinery is right for the discrete case and pure overhead for this one: the abstraction is not wrong, it is transplanted. The correct interface is therefore a capability plus a measured placement policy, not an unconditional zero-copy switch.

Coherent link: capacity without a local copy

NVIDIA GH200 combines 96 GB of HBM with 480 GB of CPU memory over a coherent link. Table 8 separates three placement classes (local HBM, direct C2C reads, fault-driven migration): file-backed mappings and pinned Grace allocations occupy the direct-read class; ordinary CPU-first-touch allocation does not. For a 119 GB working set that exceeds HBM, mapping sustains 363–381 GB/s and overlapped streaming reaches 355 GB/s, both link-bound and inside the 5% band; mapping avoids a second staging footprint. The capacity result also appears end to end: a stock transformers tree rebound to DLPack views of Graceresident file pages runs a 145 GB fp16 Qwen2.5-72B at 2.48 tok/s despite exceeding HBM, while anonymous CPU-first-touch allocations enter a migration path moving only 0.25–0.92 GB/s. Migration’s one success mode is a set that fits HBM and settles there; a set larger than HBM never finishes migrating in our runs, so in the streaming regime, which exists precisely because the set exceeds fast memory, that success mode is structurally unavailable (Appendix X). The distinction is placement class, not “CPU versus GPU memory.” GH200 endpoint

resident

102

unified (M5 Max)

Table 7. The integrated-GPU result at loader and system level.

7.2

mapped in place

103

7.4

Why the same mechanism changes sign

The three platforms separate software ownership from physical placement. On Apple silicon and the APU, mapped and framework-owned storage share one DRAM domain: a second allocation duplicates bytes without a faster tier. On GH200, once the model exceeds HBM, the live comparison is mapping against link-bound streaming, both on the coherent link. On the discrete GPU every mapped read traverses PCIe: the one-time copy buys a materially faster placement. The APU integration confirms this is not a Metal-only artifact. The Vulkan backend already exposes the loader-level adoption hook its Metal path uses, but did not advertise it for the tested device. Importing a loader tensor also means widening the host range to the driver’s alignment while keeping the byte offset; rejecting unaligned bases would make every ordinary checkpoint miss the fast path. With

GB/s physical route

resident GEMV 3063 local HBM bulk copy 376 C2C link overlapped streaming 355 C2C link file-backed mapping 350 direct Grace read pinned host allocation 346 direct Grace read plain host allocation 15–20 migration path

Table 8. Protocol-local GH200 endpoints.

9

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

Algorithm 2 Endpoint selection at deployment.

that bookkeeping done, the unmodified loader, graph, and decode loop use the file pages directly, and the paired output and footprint measurements tie the rate change to the disappearance of the Vulkan duplicate. GH200 supplies a different control: bulk C2C copy, pinned host allocation, and file-backed mapping approach one link class while anonymous CPU-first-touch allocation enters migration; the mapped path shows no settling phase through the largest tested set. A cold-cache variant reaches storage rather than link speed, so adoption does not bypass the pagecache precondition. Across the rented CUDA fleet, mapped reads track each machine’s host interconnect and resident reads its device memory: topology, not API name, predicts the plausible endpoint. The API surface can also mislead outright. On an A10, cudaHostRegister rejects a file-backed mapping under every flag, read-only or read-write alike: the objection is fileness, not protection. cudaHostRegisterReadOnly, the flag built to promise the device will not write, returns operation not supported for every memory we tried. The same pointer handed to a kernel without registration reads in place at bus rate (§7.2). That failure is quiet: a developer who reaches for registration gets invalid argument, concludes the hardware cannot do it, and writes the copy, precisely the reasoning this paper interrupts. The gatekeeping API and the hardware disagree about exactly the memory this regime is made of: the paper’s argument restated one layer down (Appendix T).

8

Deployment and framework support

8.1

Choose among three endpoint paths

Require: working set 𝑊 : its files, sharding, kernel; band 𝜃 = 5% 1: map 𝑊 at deployed import granularity; prefault 2: if the mapped set is not resident then 3: report storage-bound; stop 4: end if 5: 𝐵𝑚𝑎𝑝 ← rate over the full mapped set 6: build a resident arm holding all of 𝑊 7: if allocation fails or the mapping is evicted after one pass of every arm then 8: drop the resident endpoint 9: end if 10: 𝐵𝑑𝑒𝑣 , 𝐵𝑐𝑜𝑝𝑦 ← rates with the same kernel and set 11: drop endpoints that violate import or capacity limits 12: rank by 𝑊 /𝐵; if leaders lie within 𝜃 , choose map 13: return the surviving minimum rechecked after work runs; and a resident arm smaller than 𝑊 measures cache residency, which inflated one platform’s endpoint by 1.8×. The held-out M3 machine (§6.7) has tested the rule prospectively twice, in different memory states: in session one the mapped and resident sets fail to coexist and the capacity gate removes resident; in session two coexistence holds, the leaders land inside the band, and the tie resolves to map. Both predictions precede token generation and match the observed winners. Between them the sessions exercise the capacity gate, the tie band, and map-versus-copy, but not a resident-faster branch.

At deployment, a weight byte has three relevant endpoints: map, which reads file pages in place; resident, which copies once into device-local storage and amortizes that copy; and stream-copy, which stages the bytes per use. For workingset size 𝑊 and achieved endpoint rate 𝐵, the measured path time is 𝑊 /𝐵. Resident is excluded when the working set does not fit. When two point estimates differ by at most 5%, we prefer mapping because it retains one evictable file-backed representation rather than a second owned copy. The band is an operational reporting threshold, not an equivalence test, and it is not load-bearing: sweeping the band 𝜃 from 2% to 20% changes no verdict in Table 9; only below the GH200 pair’s own 1.4% gap does that tie resolve to stream-copy, and no 𝜃 promotes a slower mapping past a faster resident. Algorithm 2 states the procedure used to produce Table 9. It is a deployment heuristic applied retrospectively to the four measured platforms and prospectively to one held-out machine, not a model validated over a platform population. The storage-bound stop of line 3 marks the regime boundary; the drop of line 8 encodes that coexistence, not allocation success, is the capacity test; the tie rule prefers the one evictable representation. Both guards are corrections of measured false verdicts: accelerator allocation is lazy, so coexistence must be

8.2

What frameworks should package

The kernels and interchange mechanisms already exist; the missing surface is a mapped-file constructor that handles alignment, lifetime, mutability, and placement policy: a readonly mapped storage object that propagates immutability and selects adoption only when the pages inhabit a GPUreadable domain. Packed kernels remain a separate concern: adoption removes the copy, not dtype expansion in a framework that lacks the checkpoint’s arithmetic format. 𝑊 /𝐵 (ms) platform

map

decision

res. copy rule

M5 Max spine 109.9 109.5 Ryzen APU 135.3 154.0 GH200, 119 GB 340 — RTX 5070 Ti 97.6 5.7

run

900 tie adopted 679 map adopted 335 tie tied 82.2 resident rejected

Table 9. Algorithm 2 applied to the measured deployments.

10

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

Ordering requires no user intervention: once a capsule is ordinary framework storage, downstream operations inherit the framework’s stream and dependency rules, as the PyTorch, MLX, and llama.cpp paths demonstrate. The remaining work is packaging and capability detection, not a new scheduler or IR. 8.3

reports direct CPU-memory access and exploits it for KVcache movement [12, 35]. Our cross-architecture result separates file-backed direct-read placement from anonymous migration and shows when the same mechanism should be enabled or rejected. Memory mapping and interchange. The value of mmap for fast storage depends on workload and eviction policy [7, 26, 27]. PyTorch exposes memory-mapped CPU tensors through torch.from_file [28], but moving one to an accelerator remains a separate placement operation; DLPack standardizes producer-owned exchange and provides the Metal device type used here [10]. Our producer makes the file mapping itself accelerator storage; the remaining contributions are placement policy and ordering evidence.

Cross-layer implications

Three implications extend beyond this loader. First, allocation ownership is a proxy for accelerator accessibility, and external file pages break it: frameworks need a first-class external-storage abstraction carrying placement, lifetime, aliasing, and mutability that ordinary operators can consume. Second, on integrated and coherent-memory systems the page cache is a reclaimable, shared accelerator-storage tier: clean mapped pages reclaim, refault, and serve any number of processes under existing VM policy; an anonymous copy forfeits all three. Capacity checks should measure hot-set residency and representation count, not model size against installed memory. Third, “zero copy” is a mechanism, not a placement policy: the same imported mapping can reach local-DRAM, coherent-link, or PCIe-host bandwidth, so the deployable decision pairs a capability check with a full-working-set endpoint probe (§7).

9

10

Scope and limitations

Platform coverage is asymmetric. Full framework integrations exist for Metal on Apple silicon and Vulkan on one AMD APU; GH200 is evaluated with placement probes and storage rebinding rather than a production loader; the discrete-GPU experiment is a control. The APU integration comprises 4 paired processes and GH200 4 suite executions on 3 instances: enough to establish the reported outcomes, not a parametric model for all integrated or coherent-memory devices. The workload is low-batch decode. Holding the kernel and both byte paths fixed from 𝑇 =1 to 16, adoption stays ahead at every point (5.31× narrowing to 1.25×) as the aggregate logical weight rate rises from 425 to 609 GB/s. The ratio is 1 + 𝐶 (𝑇 )/𝐴(𝑇 ), and a faster kernel under the same storage path can only raise it, so the measured ratios are lower bounds for that substitution; they do not extend to batched GEMM, prefill, or concurrent serving, where layout and overlap change 𝐶 (Appendix N). Adoption has two preconditions: page-aligned tensor ranges (an aligned container or a one-time relayout) and an active mapped set that stays in the page cache. Once residency fails, storage traffic dominates and the in-memory endpoint comparison no longer applies.

Related work

Mapped-weight runtimes. llama.cpp/ggml established nocopy mapped weights on Metal [15], and related runtimes extend the design [1, 29]; all own the full command pipeline and satisfy ordering internally. Mapped weights and zerocopy DLPack import [25] are standard practice; neither is claimed as new here. The missing step is a producer turning a file mapping into ordinary tensor-framework storage, plus the contract that decides when the import is worth using. The closest prior MLX prototype modified its allocator to map files and independently reached one-file-per-tensor layout [23, 24]; its reported page-cache collapse is the regime boundary measured here. Offload, sparsity, and serving. LLM in a Flash, PowerInfer, Deja Vu, and MoE offload systems reduce or schedule the bytes moved [3, 9, 19, 22, 31–33]. FlexGen and DeepSpeedInference schedule unavoidable discrete-GPU transfers [5, 30]. ServerlessLLM accelerates checkpoint loading [11]; continuous batching and paged attention improve resident serving [2, 20, 34]. Adoption is complementary: it changes whether cached bytes acquire a second representation before a kernel reads them. Integrated and coherent memory. Zero-copy shared memory on integrated GPUs is longstanding [6, 8, 17, 18, 37]. CUDA unified-memory work focuses on migration, prefetch, and oversubscription on discrete devices [4, 13, 14, 21, 36]. Grace–Hopper adds a coherent CPU–GPU link; prior work

11

Conclusion

File-backed weights can already sit in memory the GPU can read, yet a framework loader may copy them into a second allocation before every use. A mapped-file DLPack producer removes that ingestion tax under a three-condition execution contract (adopt the pages, keep activations on the accelerator, order on the GPU), reaching resident-class bandwidth on Apple silicon, improving an integrated Vulkan runtime, tying overlapped streaming on a coherent link, and losing on a discrete GPU. The deployment rule is topology- and workingset-aware: adopt cached pages only when they already sit in a GPU-readable domain. 11

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

References

[20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of SOSP. [21] Raphael Landaverde, Tiansheng Zhang, Ayse K. Coskun, and Martin Herbordt. 2014. An investigation of unified memory access performance in CUDA. In IEEE High Performance Extreme Computing Conference (HPEC). [22] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. 2023. Deja vu: Contextual sparsity for efficient LLMs at inference time. In Proceedings of ICML. [23] ml-explore/mlx contributors. 2024. Discussion #615: Loading models with mmap. https://github.com/ml-explore/mlx/discussions/615. [24] ml-explore/mlx contributors. 2025. Issue #2878: On-disk (out-of-core) streaming of model weights for low-RAM inference. https://github.c om/ml-explore/mlx/issues/2878. [25] ml-explore/mlx contributors. 2026. MLX documentation: Conversion to NumPy and other frameworks (DLPack interchange). https://mlexplore.github.io/mlx/build/html/usage/numpy.html. [26] Anastasios Papagiannis, Giorgos Xanthakis, Giorgos Saloustros, Manolis Marazakis, and Angelos Bilas. 2020. Optimizing Memory-mapped I/O for Fast Storage Devices. In USENIX Annual Technical Conference. [27] Anastasios Papagiannis, Giorgos Xanthakis, Giorgos Saloustros, Manolis Marazakis, and Angelos Bilas. 2021. FastMap: Revisiting the Memory-mapped I/O Path for Fast Storage Devices. In USENIX Annual Technical Conference. [28] PyTorch contributors. 2026. PyTorch documentation: torch.from_file. https://docs.pytorch.org/docs/stable/gen erated/torch.from_file.html. [29] Prabod Rathnayaka, Fabian Waschkowski, and Lukas Wesemann. 2026. BaseRT: Best-in-class LLM inference on Apple silicon via native Metal. arXiv:2607.00501. [30] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of ICML. [31] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast large language model serving with a consumer-grade GPU. In Proceedings of SOSP. [32] Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2024. MoE-Infinity: Activation-aware expert offloading for efficient MoE serving. arXiv:2401.14361. [33] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast large language model inference on a smartphone. arXiv:2406.06282. [34] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). [35] Jiahuan Yu, Mingtao Hu, Zichao Lin, and Minjia Zhang. 2026. SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips. In Proceedings of Machine Learning and Systems (MLSys). [36] Qi Yu, Bruce Childers, Libo Huang, Cheng Qian, and Zhiying Wang. 2020. A Quantitative Evaluation of Unified Memory in GPUs. In The Journal of Supercomputing. [37] Feng Zhang, Jidong Zhai, Bingsheng He, Shuhao Zhang, and Wenguang Chen. 2017. Understanding Co-Running Behaviors on Integrated CPU/GPU Architectures. IEEE Transactions on Parallel and Distributed Systems (2017).

[1] Abacus Noir. 2026. Zero-copy GPU inference from WebAssembly on Apple silicon. https://abacusnoir.com/2026/04/18/zero-copy-gpuinference-from-webassembly-on-apple-silicon/. [2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). [3] Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a flash: Efficient large language model inference with limited memory. In Proceedings of ACL. [4] Tyler Allen and Rong Ge. 2021. In-Depth Analyses of Unified Virtual Memory System for GPU Accelerated Computing. Proceedings of SC (2021). [5] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale. In Proceedings of SC. [6] Bennett Cooper, Thomas R. W. Scogland, and Rong Ge. 2024. Shared Virtual Memory: Its design and performance implications for diverse applications. In Proceedings of ICS. 26–37. doi:10.1145/3650200.3656608 [7] Andrew Crotty, Viktor Leis, and Andrew Pavlo. 2022. Are You Sure You Want to Use MMAP in Your Database Management System?. In Conference on Innovative Data Systems Research (CIDR). [8] Mohammad Dashti and Alexandra Fedorova. 2017. Analyzing memory management methods on integrated CPU-GPU systems. In Proceedings of ISMM. [9] DeepSpeed team. 2022. ZeRO-Inference: Democratizing massive model inference. https://www.deepspeed.ai/2022/09/09/zeroinference.html. [10] DLPack contributors. 2023. DLPack: an open in-memory tensor structure. https://dmlc.github.io/dlpack/latest/. [11] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low-latency serverless inference for large language models. In Proceedings of OSDI. [12] Luigi Fusco, Mikhail Khalilov, Marcin Chrapek, Giridhar Chukkapalli, Thomas Schulthess, and Torsten Hoefler. 2024. Understanding data movement in tightly coupled heterogeneous systems: A case study with the Grace Hopper superchip. arXiv:2408.11556. [13] Debashis Ganguly, Ziyu Zhang, Jun Yang, and Rami Melhem. 2019. Interplay Between Hardware Prefetcher and Page Eviction Policy in CPU-GPU Unified Virtual Memory. In International Symposium on Computer Architecture (ISCA). [14] Debashis Ganguly, Ziyu Zhang, Jun Yang, and Rami Melhem. 2020. Adaptive Page Migration for Irregular Data-intensive Applications under GPU Memory Oversubscription. In IEEE International Parallel and Distributed Processing Symposium (IPDPS). [15] ggml-org/llama.cpp contributors. 2023. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama.cpp. [16] Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. 2023. MLX: Efficient and flexible machine learning on Apple silicon. https://github.com/ml-explore/mlx. [17] Jiong He, Mian Lu, and Bingsheng He. 2013. Revisiting co-processing for hash joins on the coupled CPU-GPU architecture. Proceedings of the VLDB Endowment (2013). [18] Intel. 2014. Getting the most from OpenCL 1.2: increasing performance by minimizing buffer copies on Intel processor graphics. Intel Developer Zone whitepaper. [19] Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. 2024. Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models. arXiv:2402.07033. 12

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

The appendices are grouped: measurement discipline (A– B); the Apple-silicon campaigns in full (C–K); end-to-end models (L–P); and the cross-architecture record (Q–Y).

A

Finding a protocol error on one platform obliges us to scope it, so we audited the others. On CUDA it does not reproduce. Tracing twenty consecutive passes with no discard across 7 independent processes on an A10, every cell is at steady state by pass 2—resident 477 GB/s (across-process range 475–480), mapped file in place 25.2, system allocator 413—and applying the Apple correction moves every one of them by less than 0.1%. Data-center parts under a compute load do not idle their clocks the way a fanless laptop does. The CUDA and GH200 rates in this paper were taken under the single-warm-up protocol and stand as published; only the Apple-silicon figures needed regenerating. The same traces confirm the system allocator’s mechanism directly: the malloc cell consumed 2.15 GB of VRAM to read 2.11 GB of matrices, which is the implicit ingestion the fleet table names, and the mapped-file cell consumed none.

Graph capture, and measurement traps

Does graph capture change the story? The free ride of the granularity study lives in encode bubbles, so a captured/compiled op stream might close it. Measured: torch.compile on this fine profile speeds the chunk 1.02× (the chain is serially dependent; there is little to fuse) and leaves the in-stream excess at zero (0.09 → −0.02 ms, noise) while the round-trip cells still pay 5.4–5.5 ms. Compilation is orthogonal: at worst it shrinks the bubble and the in-stream path pays its own ∼2 ms of GPU time (the granularity study, coarse row); it never pays the drain. Measurement traps (all cost us a wrong conclusion first): freshly written weight files measure 1.5× slow until their pages are cleaned (fsync+sync before timing); PyTorch’s w @ x matvec form hits a known-issue fallback at our shape (60 GB/s vs. 258 GB/s for F.linear on identical tensors— the harness must give stock its best form); idle-queue microbenchmarks invert the queue axis and run at the 338 MHz DVFS floor (§4); single-run ablations are meaningless under 12–15% run-to-run spread—we report medians with IQR across independent processes (headline cell: fifteen independent processes); short-sample rates mislead (an 8-token MLX generation timed 7.9 tok/s where the 96-token steady state is 6.9—the former implies bandwidth above the machine’s physical ceiling); and, the one that survived longest in our own harness, a single warm-up pass does not warm the clock. The warm-up trap. In the five-process campaign the mapped path’s first timed pass lands anywhere from 2% to 84% of its steady rate: the GPU clock is ramping from its idle floor, and one warm-up call does not finish the ramp. Discarding a single pass and taking the median of seven therefore mixes ramp with steady state—at 16.8 MB matrices that estimator lands in 134–167 GB/s against a 328 GB/s twelve-discard median, and in an earlier extension-kernel revision of this sweep the same mistake produced a spurious peak-then-fall curve that put the 134 MB point below the 88 MB point. Twelve discarded passes make the fiveprocess aggregate monotone (Fig. 2b), but they are not a cure-all: the largest sizes reach 90% of steady within 2–3 passes, while dispatch-bound cells below ∼34 MB ride the clock through the whole window in some processes—the 16.8 MB medians still span 287–354 GB/s across processes. That residual volatility is why every rate in this paper carries cross-process dispersion. The largest size settles to 550 GB/s with an 11.5 GB/s cross-run spread. The copy path shows no such ramp—it never leaves the copy engine—which is why the trap inflates the gap rather than both arms equally. Every rate measured after this discovery uses the twelve-pass discard, and the affected figures were regenerated.

Instrument

Establishes

matched-kernel wall time byte-path throughput, arithmetic fixed residency probes whether a second representation exists mutation visibility imported tensor aliases producer storage checksum + output oracle the kernel reads the intended file bytes paired campaign mapped–resident gap under shared state

Table 10. The claim is a chain, not a timing-only inference. No single instrument establishes placement, aliasing, correctness, and throughput.

B

The constraints catalog

Nine measured constraints govern the implementation. None appears in framework documentation; each was found by violating it. Metal command encoding. A process holds one in-flight command buffer, and encoding is serialized on the stream’s dispatch queue: the extension encodes via dispatch_sync, and a synchronize() issued from a second thread aborts the process. Calling endEncoding between operations defeats encoder coalescing—the framework’s encoder must be left open and ended only to signal events. Mapping semantics. No-copy buffers require a pagealigned base and length, which one file per tensor makes structural; the earlier MLX prototype converged on the same layout [23]. map_private is privatized into anonymous pages when the GPU wires the mapping (at 60 GB it created 58 GB of anonymous memory, 10× slower wiring, and a 3× throughput loss), while map_shared stays file-backed. The trap is macOS-specific: Linux and Windows keep private file mappings file-backed. First touch is cheap and one-time: 0.36 ms per file to map and 98–137 GB/s to wire, so the production spine maps in 0.96 s, wires in 0.41 s, and needs no CPU prefault. 13

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

Algorithm 3 One decode step over adopted weights.

Cell (A, B, C)

GB/s

Require: adopted weights 𝑊1 ..𝑊𝐿 (Alg. 1); token 𝑥 1: ℎ ← Embed(𝑥) {ℎ is created on the device and stays there (C2): round-tripping ℎ per layer costs +5.6 ms/pass} 2: for ℓ = 1 to 𝐿 do 3: ℎ ← Kernel(𝑊ℓ , ℎ) {𝑊ℓ is the mapping itself (C1): reingesting it per use costs +12.3 ms/pass} no synchronise, no host round trip {the dispatch joins the open 4: encoder (C3): draining per dispatch costs +3.4 ms/pass} 5: end for 6: return Sample(LmHead(ℎ))

map, dev, stream (this work) map, dev, own (llama.cpp shape) map, RT, own map, RT, stream copy, dev, stream copy, RT, stream copy, dev, own copy, RT, own

489 [484, 491] 526 [519, 528] 265 [265, 269] 169 [166, 169] 76 [74, 79] 60 [58, 60] 57 [56, 58] 54 [54, 55]

ref: resident, our kernel ref: resident, F.linear fp16 ref: stock map→.to(mps) ref: stock + fp32 materialise

523 [520, 526] 264 [261, 266] 61 [55, 61] 25 [24, 25]

Writes and immutability. A GPU store to a prot_read mapping is silently dropped, so adopted weights are immutable in effect and a faulty kernel is masked rather than trapped. The DLPack read-only flag does not restore the trap: PyTorch 2.13 and MLX 0.32 both accept a versioned capsule with the bit set and still admit in-place stores, so a producer cannot rely on consumers to enforce immutability. Carrying the bit at all requires the versioned capsule (DLPack ≥1.0), and DLPack 1.2+ additionally requires non-null strides for ndim ≠ 0; the producer emits dltensor_versioned with explicit (𝐾, 1) strides, keeping the legacy form only for older consumers. Capacity. The hot mapped set must fit free RAM, and degradation has a signature: a rising step curve as alternating passes fault (§6). A shared mapping needs no free-RAM headroom of its own—the earlier “≥55 GB free” rule for the spine was an artifact of the privatized copy, not of the mapping.

C

Table 11. The cube in isolation (idle queues); medians with IQR across 15 independent processes (median of 5 passes within each). Twelve warm-up passes are discarded per cell and cell order is randomized per process; both matter (appendix). The copy axis dominates—every copy cell is pinned at the copy engine’s 54–76 GB/s—and the queue axis is invisible: the own-queue cell is fastest in all fifteen processes, by a median of +35 GB/s. Both facts reverse under load.

E

Per-cell implementation inventory

The cube’s cells share kernel source, dispatch geometry, and logical weights; this table lists what else differs. cell

host path

alloc

CBs/pass waits

map,·,stream extension none PT’s adopt (public) from_dlpack none PT’s copy,·,stream compile_sh. per-use PT’s map,dev,own standal. none 1 map,RT,own standal. none 12 copy,·,own standal. prealloc 1–12 map,dev,own+event standal. none 12

One decode step under C1–C3

Algorithm 3 is the execution model the paper measures, with each condition annotated by the price of dropping it alone— the paired excess of the binary factorial of Appendix F (eight independent processes, 1.06 GB per pass; full cell 0.04 ms with IQR [-0.23, 0.23]).

D

IQR

1 1 1 1 12 1–12 0 (GPU)

Table 12. What varies across cells besides the three axes. The PSO is compiled from identical source in all three hosts; dispatch geometry and threadgroup size are identical. The ordering study of §4 controls the axis these confounds touch most.

The cube in isolation, in full

Table 11 repeats the eight implementation cells of Table 3(a) and adds the two reference rows the body omits: F.linear over resident fp16, and stock ingestion with fp32 materialisation.

F

The two same-harness factorials

The experiments below use one Python host path, one compile_shader kernel, one stream, and one allocator. Every cell interleaves its 1.06 GB of weight reads with the same in-flight framework workload; values are excess milliseconds over that workload alone. Cell order is randomized independently in each process. Binary necessity campaign (2×2×2, 𝑛=8 processes).

14

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

C1

C2

C3

excess ms

✓ ✓ ✓ ✓ — — — —

✓ ✓ — — ✓ ✓ — —

✓ — ✓ — ✓ — ✓ —

0.04 3.46 5.62 5.22 12.38 16.11 17.67 17.90

G

IQR [ −0.23, 0.23] [3.28, 3.73] [5.18, 5.68] [4.83, 5.42] [12.15, 12.65] [15.85, 16.47] [17.52, 18.13] [17.53, 18.22]

C1 (adopt) C2 (residency) C3 (ordering) C1×C2 C1×C3 C2×C3 C1×C2×C3

MTLEvent

95% CI

12.49 3.72 1.86 0.08 -0.13 1.79 0.02

[12.38, 12.77] [3.33, 3.89] [1.56, 1.91] [-0.15, 0.17] [-0.33, 0.18] [1.67, 2.20] [-0.31, 0.24]

C2

ordering excess ms

dev dev dev RT RT RT dev dev dev RT RT RT

stream event drain stream event drain stream event drain stream event drain

0.26 0.54 2.27 3.89 4.12 4.65 10.32 10.59 15.31 16.53 16.85 16.06

queues overlap

0.44

signal leaves the GPU—overlap lost 3.46

MTLSharedEvent

every in-flight op retires 5.80

host drain one 1.06 GB pass; time runs right

Figure 4. The ordering hierarchy; each excess is the median of process medians across 8 independent processes, ten timed passes each (process-level 95% intervals: in-stream [0.24, 0.47], event [0.40, 0.56], shared [3.14, 3.67], drain [5.66, 6.02]). In-stream work largely fills an encode bubble that existed anyway. A plain MTLEvent keeps the dependency on-GPU and the queues overlap. MTLSharedEvent’s host-observable semantics coincide with lost overlap; the experiment does not identify which internal component causes the cost. A host drain retires every in-flight operation first, and its cost tracks queue depth rather than bytes moved.

Every interaction spans zero except C2×C3: the host round trip and the host drain hide one another, which is the subadditivity the body reports. Storage adoption’s effect is additive with both. Ordering refinement (2×2×3, 𝑛=10 processes). map map map map map map copy copy copy copy copy copy

excess (ms) 0.33

in-stream

effect (ms)

C1

weight reads

framework’s own work

Here C3 is binary: in-stream ordering versus a host drain. Removing C1, C2, or C3 alone costs +12.3, +5.6, or +3.4 ms relative to the full cell. A later refinement split C3 into three levels. The full 23 model. Effects are process-level paired contrasts (the cost in ms of dropping the condition or condition product), summarised across the 8 processes with 95% bootstrap intervals. term

The ordering timeline

H

IQR

The depth sweep

As framework work in flight grows 8×, the in-stream cell’s excess stays flat; round-trip cells pay a fixed ∼0.33 ms per exchange—the drain itself, not the 77 KB moved; and the copy cell’s tax partially hides behind unrelated GPU work (17.4→9.5 ms). That last effect is why systems that copy see less than the isolated 8× penalty in practice, and why the isolated cube overstates how bad copying looks to a practitioner while understating what ordering costs.

[ −0.35, 0.56] [0.33, 0.86] [1.78, 2.58] [3.80, 4.48] [3.78, 4.46] [3.52, 5.98] [9.71, 10.46] [10.19, 10.81] [14.42, 15.76] [15.51, 17.24] [16.51, 18.28] [15.61, 16.99]

With C1 and C2 on, event minus stream has paired 95% CI [0.14, +0.95]; the experiment does not resolve a difference, but the interval is too wide to establish equivalence. Drain minus stream has CI [+1.41, +2.63] and is resolved as slower. These values are distinct from the one-process four-mechanism study in Fig. 4 and from the production-spine replay in Table 4.

15

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

I

The working-set sweep

pay one anonymous copy per process until, at capacity, the duplicates evict the shared cache itself and decode collapses by two orders of magnitude. Outputs stay byte-identical in every cell.

GB/s

600 550

K 500 0

5

10

15

mapped working set (GB)

Figure 5. The four measured working-set points use the standalone producer and the public kernel. Throughput rises from 548 to 578 GB/s across 2–16 GB, above the survey protocol’s 517 GB/s pure-read reference (dashed): a 16 GB pass amortizes the dispatch and synchronization boundaries that bound the 12-dispatch survey, so the reference is a protocol ceiling, not a machine limit.

J

Citizenship at page granularity

mincore bitmaps over the mapped files and vm_stat deltas turn the contrast into counters. Squeezing a 25 GiB mapped set with an incompressible balloon grown to 105 GB on a 128 GB machine, the mapped set stays 100% resident throughout while the balloon pays the entire bill: the compressor absorbs it and Pageouts stays exactly zero—clean mapped pages are never written back, and here they are not even dropped. Under the private mapping the same sweep evicted the mapped set down to 71% at the same pressure, because a privatized anonymous copy was competing for the same memory; the shared mapping removes the competitor. When eviction does occur (deeper pressure), GPU-checksumming fully-evicted 1 GiB files against ground truth returns bitcorrect sums in 73 ms with 7 page-ins: reclaim is transparent, reversible, and verified. The copy path’s anonymous pool has no such exit. Sharing across processes. We launch 𝑁 concurrent decoders of one checkpoint and sample vm_stat against a settled baseline. Adopted arms share one set of file-backed pages; resident arms materialise one anonymous copy per process. model, 𝑁

arm

7B, 1 7B, 1 7B, 2 7B, 2 7B, 4 7B, 4 32B, 2 32B, 2

mapped resident mapped resident mapped resident mapped resident

tok/s per process

Δfile (GB)

Δanon (GB)

22.9 23.0 18.1, 18.8 16.9, 17.1 15.3–20.1 12.5–16.8 5.5, 3.6 0.08, 0.07

+15.4 +15.2 0.0 0.0 0.0 0.0 +65.5 −30.9

+0.4 +11.0 +0.8 +11.8 +1.1 +32.6 +1.6 +61.2

The bandwidth ledger

Every M5 Max read rate quoted in this paper appears below. The protocol-local range runs from 489 to 578 GB/s. They are not interchangeable: the protocol, route, framework, shape, and statistical design differ. This ledger places the reference families that the body compares and names the factor separating adjacent values. What is counted. The 519.4 GB/s machine reference is a copy: it touches each byte twice, and counting reads and writes is the convention STREAM-style numbers follow. Every other row is a read-only gemv counting weight bytes once. Quoting 519.4 as “the” read ceiling therefore overstates what a weight-streaming kernel can reach by about 4%. Shape and route. The survey protocol streams 12×88 MB per pass. The matrix-size sweep reaches 550 GB/s at its largest point because fixed dispatch and mapping overhead amortise over more bytes; the production spine’s 1253 unequal matrices sit at 508–524 GB/s. The internal extension performs host work on every call (489), while DLPack adoption does not (516); the 72B experiment shows the same route distinction end to end. Queue occupancy. The bottom of the span, 489, is the extension route replayed in isolation—the same route as 489, measured with nothing else in flight, which leaves the GPU at its DVFS floor. That is the isolated-queue trap of §3 appearing inside our own ledger, and it is why every rate here is reported with the load it was taken under. Protocol correction. An earlier draft carried a resident reference of 470 GB/s with IQR [453, 501] and explained its disagreement with another campaign as session variance. That explanation was wrong: timing began before the GPU clock finished ramping. Under the corrected twelve-pass warm-up the cube resident cell is 523 with IQR [520, 526], consistent with the paired campaign’s 517.8. Mapped versus resident is settled only by the paired design, which rotates both arms inside one process, not by differencing campaign medians.

The mapped arms add file-backed bytes only when the checkpoint first enters the cache (𝑁 =1; the 32B cold start) and essentially no anonymous memory afterwards: every further process reads the same shared clean pages. The resident arms 16

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

reference

GB/s distinguishing factor

machine reference fp16 copy

519.4 counts reads and writes

M

matched survey, 12×88 MB pure-read ceiling 517–523 protocol-local ceiling band mapped, torch.from_dlpack 516 the paired campaign’s mapped arm resident, MLX best matvec 497 different framework paired campaign, arms rotated—the only rows differenced mapped 516.2 shared process and state resident 517.8 mapped−resident CI [-0.66%, +0.48%] pure read 522.2 mapped−pure CI [-1.41%, -0.23%]

The K3 spine stage, in full Weights

Activations, ordering

live replay stage harness

copied, fp32 copied, int8 mapped mapped

device-resident, in-stream 2.62 s device-resident, in-stream 1.34 s host round trip, own queue 5.96 s device-resident, in-stream 0.35 s

4.109 s 1.219 s 1.262 s 0.121 s

Table 15. One decode token’s spine, four byte paths. Live is wall time for the stage inside the running system (Kimi K3 on one M5 Max) using proprietary weights. Replay applies the corrected cube campaign, 15 processes same four paths to the same 1253 matrices in a separate mapped, extension route 489 per-call host work harness; it reproduces the ordering but not the absolute mapped, own queue 526 standalone batching, ordering times because the live stage interleaves attention, norms resident, our kernel 523 IQR [520, 526] after warm-up fix and sampling that the replay omits. We give no rate column: MLX rotating paired design mapped, mx.from_dlpack 517.5 paired diff CI [-1.6%, +9.3%] the stock path fuses dtype materialisation into the multiply resident, same weights 506.1 same process and passes and so has no separable gemv to divide by, and one column isolated replay, nothing else in flight for two bases is not a rate. Reading either column, removing mapped, extension route 489 idle queue: the DVFS floor dtype inflation is worth 2.0× live and 3.4× replayed, and shape and production workload removing the copy a further 3.8× live and 10× replayed. The largest matrix-size point 550 dispatch cost amortised third row is the one to look at twice: mapping with neither working-set sweep, 16 GB 578 one pass amortises the whole protocol other condition is worse than copying in both columns. K3 production spine, live 508–524 1253 unequal matrices cells run on the internal-symbol extension; the all-public Table 13. M5 Max read-rate reference families used in the route is exercised end to end on the 72B and 32B models of paper. Other platforms report their own protocol-local ref§6. erences in their sections. Only explicitly paired rows are differenced statistically.

N L

The tax against batch

The body targets low-batch decode. This controlled sweep asks how the byte path changes as one kernel receives more activation columns. Both arms use a kernel that loads each weight row once and reuses it across 𝑇 columns; the only difference is adopted storage versus per-use .to("mps") ingestion. The protocol uses five independent processes, nine timed passes, twelve discarded warm-up passes, and passlevel arm rotation.

The per-token byte budget

adopted / stock

Table 14 itemises what one decode token must read. The spine is page-cache-resident—where the ingestion tax binds and what this paper removes; the expert pool is disk-bound, so no byte path can help it. Arithmetic accounts for ∼14% of decode time; the remaining 86% is this table. Spine accounting, stated once and used throughout. The streamed int8 matrices are 52.8 GiB (= 56.7 GB) in 1253 files— what the gemv chain reads and what every rate column divides by. The full mapped set, adding one fp16 scale vector per matrix, is 2506 files and 0.02 GiB larger; mapping and first-touch costs cover that set. Binary units throughout, since these are file sizes.

6 5 4 3 2 1

1

2

4

8

16

activation columns 𝑇 Per token, Kimi K3 dense spine (int8) routed experts, 16×92 KV and state

bytes 52.8 GiB 24.0 GiB 𝑂 (100 MiB)

held in

Figure 6. Controlled same-kernel sweep. Adoption’s advantage narrows from 𝑇 =1, passes through 3.66×, 1.88×, and 1.50×, and remains 1.25× at 𝑇 =16. The dashed line is parity.

page cache on disk (16.5 GB/s) device memory

Table 14. One decode token’s weight-byte budget. Let 𝐴(𝑇 ) denote the adopted arm’s arithmetic and dispatch time and 𝐶 (𝑇 ) the stock arm’s additional ingestion time. The ratio is 1 + 𝐶 (𝑇 )/𝐴(𝑇 ). In this implementation, the copy is approximately a fixed per-pass cost while 𝐴(𝑇 ) grows, producing the measured decline. The kernel is a decode kernel 17

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

P

handed multiple columns, not a tuned batched GEMM, so its 𝐴(𝑇 ) is comparatively large. Holding the storage path and 𝐶 (𝑇 ) fixed, replacing only the arithmetic with a faster kernel would increase the ratio; under that controlled substitution, the measured ratios are lower bounds. This statement does not extend to an arbitrary batched implementation that changes layout, fuses work, or overlaps the copy, because those changes can also change 𝐶 (𝑇 ). Batching still increases useful work per weight read: the adopted arm’s aggregate logical weight rate 𝑇 𝐵 𝑤 rises from 425 to 609 GB/s across the sweep. It reduces the relative cost of ingestion but does not remove the per-pass copy in the measured path. The first version of the kernel placed the 𝑡 loop outside the weight-index loop and reread every weight row for every activation column. That code did𝑇 times the intended weight traffic in the adopted arm and measured a different workload. The erroneous-kernel result serves only as a negative control and is excluded from every paper value.

O

The held-out machine

An Apple M3 with 8 GB was not used to construct the deployment comparison in Section 8. The precondition is tested first. A 2.74 GB working set is fully cached before the framework creates its device copy and 0% cached afterward according to mincore; all arms then run at storage speed. Sizing the experiment to available memory gives Qwen2.5-1.5B, 1.29 GB in 140 matrices. The deployment probe uses five processes, nine timed passes, and twelve discarded warm-up passes. Its map, resident, and per-use-copy rates are 56.0, 56.9, and 8.4 GB/s. Resident storage coexists with the mapping in 4/5 processes; the capacity gate removes it because coexistence is not reliable. The resulting prediction is map rather than per-use copy. Token generation, which the rule did not consume, reaches 25.4 tok/s through public adoption, 15.8 through the internal map route, and 0.90 through per-use ingestion, with identical tokens. This is a held-out check of the capacity gate and mapversus-copy branch. The mapped and resident endpoint rates are nearly equal in the processes where both representations coexist, so the run does not test a resident-fits case with a material resident advantage. A second session, run over the standalone producer with the machine holding more free memory, exercises the branch the first could not: the probe gives 59.3/60.7/7.7 GB/s, coexistence holds in 5/5 processes, and the surviving map and resident endpoints fall inside the 5% band, which the rule resolves to map. Token generation again agrees (30.6 tok/s through adoption versus 0.99 through per-use ingestion, byte-identical outputs). Between them the two sessions cover the capacity gate, the tie band, and map-versus-copy.

The small-memory Apple case in full

We repeated the suite on a fanless M3 MacBook Air with 8 GB of unified memory (suite cells 𝑛=5, medians with ranges; the paired run uses its own 𝑛=25 protocol). The measured copy bandwidth is 80.6 GB/s under macOS 26.4 and MLX 0.32.0. The paired ceiling remains stable: mapped 88.0 [86.8, 90.9], resident 88.6 [85.1, 90.2], and pure read 88.9 [84.2, 90.8] GB/s. Median paired differences are +0.3 and +0.5 GB/s, below 1% and without a consistent direction. The extension’s per-call cell reaches 68.8 [65.7, 71.8] GB/s, or 85% of machine copy bandwidth. The public-API route (run here over the extension’s producer core) outruns it by 1.3× because it performs no per-call host work. Stock ingestion reaches 0.6 GB/s (range 0.5–1.0), versus 20.6 [18.0, 26.8] GB/s for the own-queue copy cell over preallocated buffers. The copy engine remains usable; repeated framework allocation and the duplicate representation trigger eviction churn. The mapped path retains one representation and avoids that failure mode. MLX 0.32.0 ingests at 13.4 GB/s through its default constructor against a 68.3 GB/s dense-kernel ceiling, while its DLPack import supplies the no-copy route. The ordering hierarchy remains in-stream 38 ms < plain event 49 < shared event 55 < host drain 124 per 1.06 GB (single-warm-up protocol; the ranking, not the levels, is the comparable quantity). On this smaller GPU, reduced queue depth narrows the encode bubbles, but host-visible draining remains the slowest mechanism. The case demonstrates how a duplicate weight representation can turn ingestion overhead into memorypressure overhead.

Q

The AMD APU probe in full

Is the copy necessary there? No: Vulkan has carried the adoption primitive since 2017. A 400-line probe imports the mapped weight file with VK_EXT_external_memory_host and reads it in place from a compute shader: 41.3 GB/s (IQR [40.7, 41.8], 𝑛=10) through a copy-on-write file view, with the process working set unchanged across import (1072 MiB before and after), consistent with zero-copy (driver-side allocations outside process accounting are not excluded), and checksums matching the copy path. That run’s companion rates should not be used to choose a byte path: it read a 1 GiB working set and its resident arm staged through a 256 MiB buffer, so the two companion rates it reports (57.7 resident, 13.8 copy-then-use) are measured against a working set the deployment does not have; the resident one in particular is a cache-residency rate. §6 re-measures all three at the deployed model’s 4.68 GB with a device-local buffer that holds the whole set (34.6/30.4/6.9 GB/s). An anonymous host allocation imports at 53.1 (92% of resident). The one quirk inverts Metal: the two Windows drivers we tested refuse the read-only file 18

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

S

view and accept the copy-on-write spelling, where on Metal the read-only PROT_READ mapping is exactly what works. The same program on this machine’s RTX 5070 Ti reads the same mapped file at 53.1 GB/s—7% of that card’s 801.7 resident rate—the bus again, and a third independent stack confirming the discrete control’s ceiling (§7.3). On unified memory the ingestion copy is a default, not a necessity, on a second vendor’s substrate; integrating the mechanism into the affected backend is packaging work this probe does not do.

R

Integrating adoption into a second framework

Two constraints surfaced that a probe could not have found. The read-only file view that prot_read makes natural on Metal is rejected by both Windows drivers at vkAllocateMemory; llama.cpp maps its model PAGE_READONLY, so the integration required moving it to the copy-on-write spelling, which keeps the pages file-backed and shared but is the only form the drivers import. And the AMD driver reports a 2 GiB maximum buffer size, so one imported buffer cannot cover a larger model. The evaluated configuration stays in spec by splitting the model file with llama.cpp’s own gguf-split, so the loader—which calls the adoption hook once per file—makes only right-sized imports; an earlier configuration that instead forced one oversized buffer through llama.cpp’s override served as a negative control. Neither constraint is visible from a standalone probe, and both are the kind of thing that decides whether a mechanism is deployable. The first is a one-line change with a subtle justification: copy-on-write sounds like a copy and is not one—pages stay file-backed and shared until something writes, and a weight-streaming reader never writes—but it is the only spelling the drivers will import, which inverts the Metal contract where prot_read is exactly what works. The second is solved at the file level but not yet at the buffer level: the Metal backend splits one mapping across several buffers internally, and the Vulkan backend should eventually do the same so that unsharded models also stay in spec.

Import granularity, and the heap ceiling

The APU integration took one more step to reach, and the step is worth stating because it is not about this platform. The AMD driver caps a single imported allocation, and a model given to the loader as one file needs one import. Split the same model with llama.cpp’s own gguf-split into 3 shards and the loader—which already calls the adoption hook once per file—makes 3 right-sized imports instead, no override, entirely in spec. Forced through one oversized buffer instead, adoption measured 14% slower; sharded, it is 21% faster. That reversal is the driver’s per-allocation cap being hit, not a bandwidth property of import count: with dispatch shape fixed and every import under the cap, one import and 3 imports produce identical map-to-resident ratios (§7). At the deployed working set, mapping leads the whole-set resident endpoint by 1.14× here, and reads at parity on Apple silicon—in both cases adoption buys footprint without paying throughput. Squeezing the machine separates the two further. With a resident balloon holding free memory at 10.3 GiB and an 8.4 GiB model, the copy path’s peak resident set is 14.7 GiB against adoption’s 8.1, and the whole benchmark finishes in 22.4 s instead of 58.8—because adoption never makes the copy in the first place. Notably the copy path does not fail there even though its footprint exceeds free memory: Windows compresses rather than refusing, so on this OS the second copy is paid in paging time rather than as a fit-or-not boundary. That is a milder failure mode than the one Apple silicon shows under the same pressure (§6), and worth stating because it changes what the tax looks like to a user: slower, not broken. The ceiling that remains is the platform’s, and we measured it: cumulative imported host memory on this part stops at 12.5 GiB, so an 18.5 GB model cannot be adopted here however it is split. The gate is not caution, it is the whole difference. Forcing the same adoption on this machine’s discrete RTX 5070 Ti— one flag, same binary, same model—takes decode from 146.9 to 3.79 tok/s, a 39× slowdown, because every weight read now crosses PCIe instead of sitting in VRAM. The footprint halves there too, and it buys nothing. Same mechanism, same code, opposite verdict: what decides is the memory architecture, which is what the capability is gated on.

T

Read-only host memory, three substrates

The three substrates disagree about read-only host memory in three different ways, and none of them is the way an application would want. Metal accepts the mapping and drops device writes without a diagnostic. The Windows Vulkan drivers refuse the read-only file view outright and accept only the copy-on-write spelling. CUDA refuses the file mapping at registration (§7): on an A10, cudaHostRegister rejects a file-backed mapping under every flag, and identically whether the mapping is prot_read or prot_read | prot_write, while accepting anonymous memory and malloc in the same process; cudaHostRegisterReadOnly returns operation not supported for every kind of memory we tried; and the GPU reads the refused file mapping in place at bus rate anyway, with no registration at all (§7.2).

U

The discrete-GPU control, in full

On a discrete GPU it is not the copy that is unavoidable but the interconnect traversal: weights must cross PCIe into VRAM, and no software choice lifts the cap. Running the same-shape protocol on an RTX 5070 Ti (PCIe 5.0 ×16, torch 19

Yuan Si, Yufeng Lin, Daming Li, and Jialu Zhang

X

resident bus mmap in place malloc

The migration machine, in full

Table 16 extends the discrete-GPU control across four rented CUDA parts. File-backed in-place reads are interconnectbound everywhere: 65–104% of each platform’s bulk-copy link rate, never within reach of resident bandwidth. The malloc column is the volatile one, because convenient allocation changes mechanism by platform—migration into VRAM on the PCIe parts (implicit ingestion), a crawl on GH200, and link rate in prior controlled-placement work. Deliberate placement is predictable; convenient placement is not.

The system-allocator path is a migration machine. CPU-firsttouched pages start on the Grace node; GPU access triggers driver-internal migration to HBM that the kernel’s own accounting never sees (pgmigrate_success stays zero). The machinery moves 0.25–0.92 GB/s across the sizes we ran: 1 GB crosses in 1.9–4 s of 10–180 GB/s transient and then reads at 2,950–2,985; 24 GB settles at 26–38 s depending on driver knobs; and a 109 GB set—larger than the HBM—never settles, crawling at 70–140 GB/s while ∼104 of 1.8M pages migrate per pass. Every earlier cell falls out: the 15–20 GB/s suite numbers were five-pass measurements inside the transient, GPU-first-touch and membind skip it, and prior work’s ∼420 GB/s [12] is another point on the same trajectory. Knob levers close the attribution: disabling the prefetcher slows settling 47%, disabling the thrashing detector changes nothing— the fault-service path is the mover. The class boundary is migratability, not file-ness: hugetlb allocations (2 MB and 512 MB alike) are never migrated across 90 s of reads, steady at 357–368. Larger pages do not migrate faster; they leave the migratable class. File-backed pages were never migrated in our sessions. The mapped route therefore shows no migration transient or settling condition through the 119 GB point tested, sustaining 363–381 GB/s where the same-size convenient path is stuck at a tenth of that. In the streaming regime, which exists precisely because the set exceeds fast memory, the convenient path’s one success mode is structurally unavailable.

W

Y

A10 A100 SXM4 H100 PCIe H100 SXM5 GH200 (C2C)

470 25 1268 26 1822 55 2859 54 3063 376

25 27 36 35 350

400 846 1779 2709 15–20

Table 16. GB/s across the rented fleet, five-run medians per cell.

2.11+cu128), streamed weights top out at 51–57 GB/s; mapping the host buffer is worse (48); and residency beats streaming 14× (815 GB/s) however the bytes arrive. Resident crossvalidates across three stacks: torch 815.4, cupy 814.6, Vulkan 801.7. The two-architecture comparison this control anchors appears as Figure 3 in §7.3.

V

The rented CUDA fleet

The GH200 reference rates

Placement is programmable and file semantics survive it: the same file on a tmpfs bound to the HBM node reads at 3.0– 3.1 TB/s, Grace-bound tmpfs at 358–363, ext4 variants at 338–342. The drop-in pattern ports to CUDA, at target scale: an unmodified transformers model (1.5B) rebound to DLPack views of the Grace-mapped checkpoint generates tokenidentically with forward-pass parity (26.9 vs. 26.8 ms) and 0.07 vs. 2.91 GB of HBM; the same rebinding runs Qwen2.572B fp16—145 GB, 1.5× the HBM—at 2.48 tok/s (360 GB/s of weight traffic) on 0.04 GB of HBM, after a 2 s map and a 14 s first token that wires all 145 GB. No resident configuration exists at this scale here; the model decodes anyway.

Table 17 anchors the coherent-link sections: every GH200 rate quoted in the body appears here against the machine’s own references. The in-place file-backed gemv runs at 93% of the measured C2C link—mapping delivers the interconnect, and the interconnect is the ceiling. The one cell that does not sit on this ladder is plain malloc, whose 15–20 GB/s is not a bandwidth but a symptom; §X traces it to the driver’s page-migration machinery. GH200 480GB (median of 5, runs ±2%) d2d copy (HBM reference) resident fp16 GEMV (HBM) C2C link (H2D pinned / pageable) streamed GEMV, overlapped in-place GEMV, mmap’d file in-place GEMV, cudaHostAlloc in-place GEMV, plain malloc CPU memcpy (1 thread)

GH200 placement, in full

GB/s 3461 3063 376 / 368 355 350 346 15–20 29

Table 17. The coherent-link case. In-place file-backed reads run at 93% of the link.

20

Related documents

• Precision Bootstrap Studies of Fermions and Stress Tensors in Conformal Field Theories#220291
• New perspectives on the relevance of gravitation for the covariant description of electromagnetically polarizable media#955083
• 44 CFR Part 329 — Use of Priorities and Allocation Authority for Federal Supply Classification (FSC) Common Use Items (DMO-12)#510250
• The ontology of General Relativity#763893
• Duality invariant quantum field theories of charges and monopoles#499103