UMLoad: Fast On-Demand Model Loading on Unified Memory Architecture

UMLoad: Fast On-Demand Model Loading on Unified Memory Architecture Zimeng Wang, Jinrong Yang, Rong Chen Institute of Parallel and Distributed Systems, School of Computer Science, Shanghai Jiao Tong University

arXiv:2611.04253v1 [cs.OS] 3 Nov 2026

Abstract On-demand model loading dominates response latency for interactive on-device AI, yet unified memory architectures (UMAs)—now common on edge devices—should make it cheap by placing weights in memory the GPU already sees. However, frameworks still use the loading flow built for discrete GPUs, leaving significant inefficiencies. This paper presents UMLoad, a system-level model-loading accelerator that decouples a framework’s logical allocations from their physical page backing. UMLoad uses on-demand reconstruction to eliminate the weight copy while preserving copy independence, and GPU-assisted page zeroing to accelerate runtime allocation while preserving its security guarantee. Implemented on Intel, AMD, and NVIDIA UMA platforms and evaluated with llama.cpp, PyTorch, and vLLM across six models, UMLoad reduces average loading time by 36.6%– 71.9% over vanilla and 19.9%–62.7% over pipelined loaders, and cuts mean LLM-routing time to first token (TTFT) by 51.3%–59.3% and model-backed tool completion latency by 10.2%–31.6%.

1

Figure 1. (a) Model-loading contribution to response latency: end-toend completion time for non-streaming tasks and TTFT for streaming tasks. Percentages indicate the fraction spent on model loading. Weight-file sizes are 3.7 GB (Chat), 1.7 GB (Vision), 4.5 GB (TTS), and 6.5 GB (ImgGen). (b) Step-by-step illustration of model loading on UMA.

Introduction Figure 2. Breakdown of Qwen3-1.7B model loading, with 3.7 GB of weights and a 6 GB KV-cache reservation, using vanilla PyTorch, PyTorch+Pipeline, and PyTorch+UMLoad. Tests use an Intel Arc XeLPG integrated GPU, with a main-memory bandwidth of 89.6 GB/s and an SSD bandwidth of 15.75 GB/s.

On-device AI is moving from isolated tasks to applications that combine many models, pairing general-purpose large language models (LLMs) with specialized ones for speech, vision, retrieval-augmented generation (RAG), and sensing. AI use at the edge is therefore on-demand, in both which model runs and when. Model choice is dynamic, based on task requirements [62] and cost–quality trade-offs [50], making it hard to know which models to keep in memory. Invocation timing varies with bursty user interactions [10] and reactive agent execution [73], making advance loading difficult. On memory-constrained devices, competing applications can evict model weights and inference state, requiring restoration when the model is invoked again [78]. When the selected model is not resident, loading enters the request’s critical path. Model loading can dominate cold-start response latency on mobile devices [75], and measurements across commercial smartphones show substantial model-preparation delays [74]. We measure response latency from user input to the first uservisible result: end-to-end completion time for non-streaming tasks and time to first token (TTFT) for streamed text responses [3]. Loading costs are especially exposed when inference consists of a short subtask; long-context prefill and lengthy generation can narrow the gap. In the four evaluated on-demand workloads, model loading accounts for 29.0%–

80.9% of response latency (Figure 1(a), detailed in §7.4). An architectural trend offers an opening to attack this cost. Edge devices increasingly adopt a unified memory architecture (UMA), where the CPU and GPU share one physical memory pool, as in Apple M-series, AMD APUs, Intel Core Ultra, and NVIDIA GB10 [2, 5, 22, 49]. UMA suits these workloads: it lets memory-limited devices hold larger models and longer contexts, and in principle removes CPU–GPU transfer from loading, as the GPU already sees weights in memory. In practice, frameworks retain the three-step loading flow designed for discrete GPUs (Figure 1(b)): file I/O reads weights into the page cache; weight transfer allocates GPU buffers and performs host-to-device (H2D) copies of weights into them; and runtime allocation reserves memory for the key–value (KV) cache and compute buffers. On UMA, this creates two inefficiencies (Figure 2). First, weight transfer is redundant: it preserves buffer independence and loader portability, but duplicates weights already in shared mem1

ory, adding copy latency and doubling their peak memory (3.7 GB extra for Qwen3-1.7B). Second, runtime allocation forces synchronous page zeroing: GPU allocations draw from system memory, which the OS must zero to prevent cross-process leaks. Zeroing the 6 GB KV cache takes 0.63 s, as the CPU achieves under a third of the 89.6 GB/s memory bandwidth. Existing solutions do not close this gap. NVIDIA GDS [46] streams weight files from SSD directly into GPU memory on supported configurations. On DGX Spark (GB10), however, GDS supports only compatibility mode, preventing use of its native storage-to-GPU DMA path [47]. Asynchronous loading [14, 16, 35, 60] overlaps file I/O with weight transfer, but the copy remains exposed in our UMA measurements—file I/O is only 61% of the 0.97 s transfer in Figure 2 (“Pipeline”)— and staging buffers add to peak memory use (§2.3). Zero-copy interfaces [44] offer the right abstraction, but deliver inconsistent performance across frameworks and platforms and require intrusive, per-framework changes to the loader. Besides weight transfer, synchronous page zeroing remains a runtime-allocation bottleneck on the tested platforms (§2.2). UMA thus calls for a loading mechanism that (G1) eliminates weight transfer rather than overlapping it, thereby also avoiding the staging memory that pipelined loaders need; (G2) accelerates synchronous page zeroing on the allocation path while preserving its security semantics; and (G3) generalizes across UMA platforms and integrates with existing inference frameworks with little or no modification.

and cache maintenance remain substantial. UMLoad combines two techniques to turn fast GPU zeroing into fast allocation. First, it splits each allocation request into page batches and pipelines CPU work with GPU zeroing across batches, overlapping CPU and GPU execution. However, CPU work can still outlast GPU execution and leave the GPU waiting between batches. Second, it shortens CPU stages by preparing GPU mapping resources ahead of time to reduce per-batch mapping work and parallelizing cache maintenance. Together, these techniques allow allocation throughput to approach the GPU’s zeroing throughput. Throughout, pages are still zeroed before use, preserving the security guarantee (§5). Implementation and results. We implement UMLoad on Intel and AMD without framework changes, and extend it to NVIDIA Jetson AGX Orin with minor changes to the framework (due to the closed-source CUDA runtime). In the overall evaluation with llama.cpp, PyTorch, and vLLM across six models, per-platform average loading-time reductions are 36.6%–71.9% over vanilla and 19.9%–62.7% over pipelined loaders. UMLoad’s peak memory during loading is 1.01–1.02× the weight-file size, compared with 1.1–2.0× for pipelined and vanilla loading. In application case studies, UMLoad improves user-perceivable response latency by reducing mean LLM-routing TTFT by 51.3%–59.3% and model-backed tool completion latency by 10.2%–31.6%. Contributions. We summarize our contributions as follows: • A characterization of on-demand workloads on UMA that identifies two inefficiencies and explains why existing solutions fail to address them (§2). • Two techniques for transparent model loading on UMA: on-demand reconstruction, which eliminates weight transfer while preserving copy independence (§4), and GPUassisted page zeroing, which accelerates runtime memory allocation while preserving its security semantics (§5). • An implementation across three UMA platforms (§6) and an evaluation that demonstrates the advantage and efficacy of UMLoad (§7).

Our approach. We present UMLoad, a system-level modelloading accelerator for UMA devices. UMLoad reuses source pages on CPU to eliminates weight transfer (G1) and uses GPU-assisted zeroing to accelerate runtime allocation (G2). It integrates with existing inference frameworks across UMA platforms with little or no modification (G3). Eliminating weight transfer transparently. Reusing source pages must preserve independent CPU and GPU buffers. Direct sharing allows later CPU writes to affect GPU contents. Unconditional ownership transfer is also insufficient: CPU contents must remain accessible, but sources without file backing cannot be reconstructed after handoff. UMLoad uses on-demand reconstruction: it transfers exclusive ownership of file-backed pages in an aligned, huge-page layout to the GPU and reconstructs the CPU source in separate pages only upon later CPU access. This preserves copy independence while supporting ordinary GPU reads and writes. The aligned, contiguous huge-page mapping preserves efficient GPU address translation. During model loading, the CPU rarely accesses these pages afterward, so reconstruction is almost never triggered, preserving copy independence with virtually no additional memory overhead (§4).

2

Background and Motivation

2.1

On-demand AI workload on UMA platforms

On-device AI spans general-purpose LLMs and specialized models for audio, vision, and content generation [1, 11, 18, 23, 29, 77]. This diverse ecosystem naturally gives rise to on-demand workloads. For example, LLM routing dispatches each query at runtime to the best-fit model among candidates of differing capabilities [50]. Multi-model applications can also fuse outputs from multiple LLMs [25, 63, 70]. Similarly, agentic tool-calling allows an orchestrator LLM to invoke these specialized models as tools and collect their outputs to compose a response [62]. Generating calls to model APIs is also studied by Gorilla [51]. Cloud models can decompose requests into simpler subtasks executed by on-device models [42]. On-device inference systems such as Ghidorah

Accelerating synchronous page zeroing. Offloading page zeroing to the GPU shifts the bottleneck from zeroing bandwidth to CPU orchestration: page acquisition, GPU mapping, 2

Torch [52], vLLM [27], and OpenVINO [20] loaders. Despite differences in scheduling and allocation granularity, they share three stages. ❶ Weight-file I/O accesses weights through file mappings, represented by mmap [19, 21]. Accessing mapped ranges brings their contents into the system page cache and supplies the host addresses used for weight transfer. ❷ Weight transfer allocates GPU memory through DeviceAlloc and initializes it through H2DCopy, using mapped weights directly or through host staging buffers. Although allocation may be bulk or incremental, each weight tensor typically occupies a contiguous GPU virtual-address range, determined by PlanDeviceAddr. The first use of this range after allocation is as an H2D destination, before GPU computation consumes its contents. Thus, the framework needs the destination address early, but physical backing can be deferred until H2D reveals the source pages. On UMA, eligible source pages can then directly back the destination range, avoiding separate weight-page allocation and copying. ❸ Runtime memory allocation then prepares memory for inference, primarily the KV cache and compute buffers. Their sizes depend on execution configuration and framework planning, rather than weight-file contents alone. PlanRuntimeBuffers determines these requests, which invoke the same DeviceAlloc interfaces. The driver learns the actual allocation sizes only when these requests arrive, limiting advance preparation based solely on preceeding stages. Figure 3 further quantifies these costs across three LLMs and three frameworks on a representative UMA platform. Across all nine cases, weight transfer and runtime memory allocation dominate model loading. Device allocation alone, including weight allocation and runtime memory allocation, consumes 28.4–48.2% of model-loading latency. Together, they account for 1.8–3.7× the file-I/O time. The gap is larger for smaller models because runtime memory allocation is relatively stable as model size grows.

Figure 3. Model-loading breakdown across frameworks and models. Runtime memory reservation is fixed at 4 GiB.

Algorithm 1 A common model-loading procedure. 1: function L OAD M ODEL(model) [ FRAMEWORK ] 2: ▷ Weight-file I/O 3: fileMap = mmap(model.file, flags) 4: ▷ Weight transfer 5: gpuMem = DeviceAlloc(. . . ) 6: foreach weight ranges (src, len) in fileMap do 7: dst = P LAN D EVICE A DDR(gpuMem, len) 8: H2DCopy(dst, src, len) 9: ▷ Runtime memory allocation 10: foreach sz in P LAN RUNTIME B UFFERS(model) do 11: DeviceAlloc(sz)

target single-request execution [72], while specialized agent subtasks motivate the use of smaller language models [6]. Edge devices have limited memory capacity and bandwidth, with other applications further reducing the space available for resident models and loading buffers. To use this memory efficiently, many edge platforms adopt a unified memory architecture (UMA), where the CPU and GPU share one physical DRAM pool [72]. For example, Jetson Orin NX integrates both processors with 8–16 GB of shared LPDDR5 at 102.4 GB/s [48]. Besides making more memory available to the GPU, UMA can possibly eliminate CPU–GPU copies in various workloads. On the platforms we study, driver-managed buffer objects (BOs) record GPU buffers’ sizes and physical backing, which is mapped to GPU virtual addresses (GVAs). A BO can be backed by either private, driver-owned pages allocated for GPU use or pages imported from an existing CPU buffer. These pages are managed separately and often prepared synchronously at allocation time despite UMA’s shared main memory. Conventional AI workloads allocate BOs with synchronously prepared physical backing and receive their associated GVAs, which the framework uses in subsequent data transfers and GPU operations. 2.2

API overhead analysis. To explain the slow model loading, we break down the overheads of the two GPU memorypreparation APIs involved in stages ❷ and ❸. First, H2D copy incurs work beyond GPU execution. Pageable sources can require pinning, mapping, or staging before the GPU can access them. Our Intel microbenchmark shows that synchronous overheads before GPU copy execution account for 33.9% of the API overhead for a 512 MiB H2D transfer, therefore merely using asynchronous H2D copy APIs still blocks model loading. Second, device allocation incurs substantial CPU zeroing. Figure 4 breaks down native allocation latency for a 1 GiB request on three UMA platforms. CPU zeroing accounts for 41–96% of allocation time in these cases. GPU zeroing (estimated using a memory-fill operation), however, is 2.5–6.4× faster than CPU zeroing. The measured Intel and NVIDIA allocation paths also perform CPU cache maintenance after zeroing to manually handle coherency. The remainder includes physical-page acquisition, GPU mapping,

Model Loading on UMA

Loading stage breakdown. Algorithm 1 summarizes the common loading path of the evaluated llama.cpp [16], Py3

Page acquisition CPU zeroing

GPU mapping GPU zeroing

Intel

AMD

NVIDIA

Device GPU alloc. zeroing

Device GPU alloc. zeroing

Device GPU alloc. zeroing

Time (ms)

120

Cache maintenance Other

80 40 0

Figure 4. Device allocation breakdown and GPU zeroing time for 1 GiB on three UMA platforms. Allocation uses zeMemAllocDevice, hipMalloc, and cudaMalloc, respectively. GPU zeroing uses userspace drivers’ native memory-fill APIs on warmed allocations.

Figure 5. Weight-loading costs of Qwen3-1.7B on three UMA platforms. Each pair shows vanilla loading (left) and pipelined loading (right). The pipeline remainder is pipelined wall time minus file-I/O time, showing work not hidden by I/O.

and other driver/runtime work. NVIDIA’s remainder is additionally expensive, because it handles page mappings and metadata at base-page granularity on our platform.

(Figure 3), allocation and copying can exceed file I/O, leaving weight transfer on the critical path and preventing the pipeline from reaching the file-I/O bound. We examine two pipelined model loaders: llama.cpp’s asynchronous loader [16] and PyTorch’s Run:ai Model Streamer [60], on three UMA platforms (detailed in Table 1) with Qwen3-1.7B, and their pipeline remainders are shown in Figure 5. On Intel, llama.cpp and PyTorch leave remainders of 0.70 s and 0.53 s, respectively. The overlap is naturally limited because weight transfer already exceeds file I/O in both frameworks (averaging 0.83 s versus 0.40 s). Across AMD and NVIDIA, transfer takes only 0.31–0.33 s against 0.61 s of reads. However, llama.cpp and PyTorch still leave average remainders of 0.33 s and 0.31 s, respectively. The two loaders expose different costs. llama.cpp allocates weight memory and prepares upload buffers before the pipeline starts, leaving 0.13 s on AMD and 0.41 s on NVIDIA not overlapped. It also issues H2D copies on the same thread as file I/O, leaving API submission work on the critical path and adding 0.085 s on AMD. PyTorch uses multithreading to overlap both weight allocation and H2D copies with file I/O, but staging-buffer preparation and remaining exposed work still cost 0.15 s on AMD and 0.47 s on NVIDIA. Using a pipeline to reach this bound is not free, either. The staging buffer that lets Run:ai reach high pipeline efficiency scales with the model, growing from 3.7 GB at 1.7B to 10.4 GB at 7B. The llama.cpp asynchronous loader requires fewer staging buffers, but the minimum buffer size is still determined by the model’s largest tensor (about 10% of the total weight size). Finally, neither evaluated pipeline accelerates synchronous zeroing of fresh runtime pages.

2.3

Current Solutions and Their Limitations

Reducing model loading. Quantization reduces the weight representation size [13, 28, 32, 75], while contextual sparsity enables selective weight loading [4]. Pruning reduces the number of nonzero weights [12]; reducing file I/O additionally requires a representation and loading path that omit those zeros. Other approaches retain resources in pools and caches [45, 81] or optimize prefetching and eviction during loading [34]. These techniques reduce transferred data or exploit reuse and overlap. They can complement UMLoad’s elimination of redundant weight copies and acceleration of fresh-page zeroing. Bypassing weight transfer. Drivers offer APIs that help to achieve zero-copy or even skip weight transfers between GPU and CPU on UMA platforms. Hostregistration APIs expose existing host buffers to the GPU (e.g., cudaHostRegister), while shared-allocation APIs provide memory accessible to both processors (e.g., cudaMallocManaged) [44]. Finally, NVIDIA GPUDirect Storage (GDS) reads files directly into GPU buffers on supported configurations, bypassing host staging [46]. However, adopting these mechanisms requires changes to framework loading paths. In addition, shared allocation still incurs slow zeroing on all tested UMA platforms. For writable private source mappings, host registration can incur synchronous copy-on-write during registration (§7.2). Finally, on DGX Spark, GDS supports only compatibility mode, preventing use of its native storage-to-GPU DMA path [47].

3

Overlapping. Existing systems hide loading latency by overlapping it with independent computation or communication [24, 30, 31, 33, 71, 75] Others overlap stages within loading itself, including file I/O, weight allocation, and H2D copies [14, 16, 35, 60]. However, in our measured UMA cases

UMLoad Design

UMLoad is a driver-level system that transparently accelerates model loading on UMA platforms by eliminating redundant weight transfer (GPU weight allocation and H2D copying) and accelerating runtime memory allocation. UMLoad’s trans4

The UMD interposer interposes driver API calls issued by the framework. It patches the corresponding UMD logic and invokes UMLoad’s ABIs in the KMD in the following ways. ❶ As model loading starts, the framework issues device allocation API calls for model tensors. The interposer turns each call into a CreatePBO call and sends the allocation size to the Page Zeroer to begin preparing private pages (§5). ❷ On an H2D copy into an open PBO, the interposer issues ReclaimPage to resolve and reclaim the source pages. Successful reclamation installs them in the destination range and suppresses the original copy. Otherwise, UMLoad finalizes the PBO as needed and follows the ordinary copy path. ❸ A PBO that remains open is finalized before its first consuming operation other than an H2D copy. UMLoad interposes on all driver APIs with kernel arguments to check cases in which frameworks use pointers directly. Supporting the remaining uncommon indirect usage requires replayable GPU page faults.

Figure 6. UMLoad architecture and interposing workflow.

parency goal is to require no framework-side code changes and have no effect on GPU inference. Figure 6 shows its components across the user-mode driver (UMD) and kernel-mode driver (KMD).

4

On-demand Reconstruction

4.1

Eliminating Weight Allocation and H2D Copy

Weight transfer comprises two operations: allocating GPU buffers for weights and populating them with host-to-device (H2D) copies. On UMA, the source pages already reside in shared physical memory. The straightforward approach is therefore to let the GPU directly access the pages backing the H2D copy source, eliminating redundant physical-page allocation and the H2D copy. First, UMLoad replaces intercepted allocation calls with CreatePBO, which reserves GPU addresses while postponing physical-page allocation. On the loading paths represented by Algorithm 1, H2D copies initialize weight buffers before GPU computation uses them. The PBO’s physical backing can therefore remain open until H2D calls supply the source addresses and byte ranges. Second, when an H2D call targets an open PBO, UMLoad invokes ReclaimPage to obtain the physical pages backing the source range, faulting them in if necessary, and installs them as the corresponding destination range’s backing. Before the PBO’s first consuming operation other than an H2D copy, UMLoad finalizes it and establishes GPU page-table entries mapping these pages at the framework-selected destination addresses, allowing the GPU to directly access their contents. The completed PBO then supports ordinary GPU operations. Direct page mapping requires the source and destination to have the same in-page offset, whereas H2D copy arguments may have arbitrary byte offsets. To match these offsets, during a one-time preparation run, UMLoad records each weight tensor’s framework-selected GPU VA. It then rewrites the weight file using PADDING: given the current absolute file offset c and a recorded destination d, it inserts (d − c) mod P padding bytes, where P is the base-page size. This matches their in-page offsets without changing the framework’s GPU

Architecture. Figure 6 provides an overview of UMLoad. It spans the user-mode driver (UMD) and the kernel-mode driver (KMD), and requires no changes to framework code on supported paths. The Page Reclaimer (§4) implements Ondemand Reconstruction. Its ReclaimPage API identifies eligible page-cache pages through H2D source ranges and reclaims them with exclusive ownership as the destination buffer’s physical memory. These reclaimed pages replace weight transfer. Later CPU accesses reconstruct the source contents from the backing file in separate physical pages. The Page Zeroer (§5) accelerates runtime memory allocation by offloading page zeroing to the GPU. It accelerates the allocation of private pages, zeroed system pages managed by the driver, for buffer ranges not covered by page reclamation. UMLoad uses a provisional buffer object (PBO) to decouple a framework-visible GPU allocation from its physical backing. A PBO exposes the handle, size, and GPU virtual address (GVA) of an ordinary GPU buffer object (BO), but leaves its backing open for construction from either reclaimed pages or private pages. Internally, a PBO maintains an array of page slots and an ownership bitmap that distinguishes the two: an empty slot is U NBOUND, and a populated one is R E CLAIMED or P RIVATE . Both classes reside in UMA system memory and differ only in origin and release path. One PBO can therefore combine several reclaimed and private ranges under a single framework-visible GVA and lifetime. Before forwarding a PBO for GPU execution, UMLoad completes its physical backing by filling the remaining U NBOUND slots with P RIVATE pages supplied by the Page Zeroer. The PBO then behaves as an ordinary GPU buffer object in any subsequent GPU operation. 5

Algorithm 2 Pseudocode for On-demand Reconstruction.

GPU contents and vice versa. One possible approach is to give the GPU exclusive ownership of every source page and revoke CPU access. However, later CPU accesses must still find valid source contents after the GPU has gained exclusive ownership of the pages. UMLoad uses On-demand Reconstruction (ODR): it gives the GPU exclusive ownership of the source pages for ordinary GPU reads and writes, and reconstructs the CPU source in separate physical pages upon later CPU access. If the GPU later writes to these pages (the framework may reuse the memory for other purposes), no write-triggered copying is needed. The design contains four steps.

▷ Prepare weights with R EORDER and PADDING, ▷ using huge-page (G) alignment; enable system THP. 1: function I NTERPOSED H2D(dst, src, len) [ UMD ] 2: pbo = F IND O PEN PBO(dst, len) 3: if pbo && R ECLAIM PAGE(pbo, dst, src, len) then 4: return ▷ original H2D copy suppressed 5: if pbo then 6: F INALIZE(pbo) 7: return O RIGINAL H2DC OPY(dst, src, len) ▷ fallback 8: function R ECLAIM PAGE(pbo, dst, src, len) [ KMD ]

▷ 1. Identify reconstructible source pages s = E LIGIBLE S OURCE(pbo, dst, src, len) if !s then return FAIL if !L OCK A ND C HECK(s) then return A BORT R ECLAIM(s) ▷ 2. Acquire exclusive GPU ownership 14: U NMAP CPUPTE S(s) ▷ retain source VMA 15: if !W RITE BACK WAIT(s) then 16: return A BORT R ECLAIM(s) 17: if !D ETACH PAGE C ACHE(s) then ▷ refcnt drops to 1 18: return A BORT R ECLAIM(s) 19: I NSTALL I N PBO(pbo, dst, s.pages) 20: U NLOCK(s) 21: return O K

Identifying reconstructible sources. Whether the CPU contents can be reconstructed depends on the source’s backing. An H2D source may map page-cache pages or anonymous pages. For page-cache pages, the backing file provides the contents needed for later reconstruction. Anonymous pages, however, may contain private contents absent from the file. Their contents cannot be reconstructed from the file after the pages are handed to the GPU. UMLoad therefore reclaims only source ranges backed by the file page cache, excluding anonymous pages such as stacks, heaps, and private CoW pages. UMLoad checks eligibility when interposing H2D copy calls (lines 9–11 in Algorithm 2). If validation fails, UMLoad finalizes the PBO as needed and falls back to copying from the original source. Since eligibility depends on the source’s backing, eligible transfers of non-weight file contents can also use reclamation.

9: 10: 11: 12: 13:

▷ 3. Recover huge-page GPU mappings 22: function F INALIZE(pbo) [ KMD ] 23: F ILL U NBOUND S LOTS(pbo) ▷ private pages 24: foreach r in M APPING R ANGES(pbo, G) do 25: if H UGE M APPABLE(r, G) then 26: M AP GPU(r, G) 27: else 28: M AP GPU(r, P) 29: pbo.open = false

Reclaiming pages with exclusive ownership. To give the GPU exclusive ownership, UMLoad follows Linux’s pagecache eviction mechanism. It revokes CPU page-table mappings, completes necessary writeback, and detaches the source pages from the page cache before installing them as the GPU destination’s backing (lines 14–19 in Algorithm 2). Reclamation changes page-cache residency and can disrupt other processes using the same file. To limit this disruption, UMLoad rejects page-cache ranges already mapped into multiple processes. If this check or ownership acquisition fails, UMLoad follows the ordinary H2D copy path, finalizing the PBO as needed while preserving previously initialized ranges.

▷ 4. Reconstruct on CPU access (Linux page-fault path) 30: function DO _ FAULT(vmf) [ LINUX , SIMPLIFIED ] 31: file = vmf.vma.vm_file; idx = vmf.pgoff 32: page = L OOKUP PAGE C ACHE(file, idx) 33: if !page then 34: page = R ECONSTRUCT PAGE C ACHE(file, idx) 35: return M AP CPUF ILE PAGE(vmf, page)

Recovering huge-page GPU mappings. Preserving inference performance also requires efficient address translation. A tensor assembled from a source buffer can require more GPU page-table entries than a normal GPU allocation, increasing translation overhead. UMLoad recovers huge-page GPU mappings by combining weight-file layout adjustments with contiguous page-cache backing. Specifically, UMLoad extends the earlier PADDING from modulo P to modulo the large-mapping granularity G for tensors larger than G (2 MiB on our evaluated Intel and AMD platforms). Corresponding file and GPU ranges now reach large-page boundaries together, enabling large mappings within individual tensors.

layout. Subsequent loads validate the offset relationship and follow the ordinary copy path as a fallback. 4.2

Preserving Copy Semantics

Directly mapping the source pages into the GPU destination leaves the CPU and GPU accessing the same physical pages. An ordinary H2D copy, however, allows the CPU source and GPU destination to evolve independently. The driver cannot assume that every intercepted H2D call transfers immutable weights. With shared pages, later CPU writes can modify the 6

Huge-page mappings also require contiguous physical backing. UMLoad uses the OS’s transparent huge-page (THP) support for the page cache when available and installs huge GPU page-table entries at finalization (lines 24–28 in Algorithm 2). To further recover physical contiguity across tensors with consecutive GPU address ranges, UMLoad applies R E ORDER, placing these tensors consecutively in the weight file. Their shared boundary pages are checked and reclaimed to preserve physical contiguity across tensors. Padding is thus needed only between such groups. The resulting layout can support a huge-page mapping for every complete, G-aligned block within the group.

Figure 7. GPU-assisted zeroing of a 4 GiB request on NVIDIA Jetson AGX Orin. The top row shows a sequential method, where block labels show stage time for the whole request. The middle and bottom rows show a pipeline-only method and UMLoad, respectively, where block labels show individual 64 MiB batches.

Reconstructing the CPU source on demand. The CPU source’s virtual memory area (VMA) remains valid after reclamation. Since the reclaimed pages were backed by the file page cache, Linux’s normal file-fault path restores the source contents from the backing file into separate physical pages upon subsequent CPU access (lines 31–35 in Algorithm 2). The GPU retains the reclaimed pages throughout. Thus, subsequent writes to either buffer cannot affect the other, preserving ordinary copy behavior without eagerly allocating and copying a second set of pages or requiring GPU accesses to trigger faults and copies. Section 7.2 evaluates ODR’s transfer latency, inference performance, and overheads.

5

mapping, and cache maintenance all grow with the request size. On our measured NVIDIA platform, CPU-side work for a sequential 4 GiB request takes about 34.1 ms, accounting for 57% of end-to-end latency. About 49% of the submission time is spent on page acquisition and 41% on GPU mapping. On Intel, CPU-side latency accounts for 56% of the 179.41 ms request time, and 83% of the submission time is spent on cache maintenance, as the Intel iGPU’s fast noncoherent zeroing task still requires explicit CPU cache flushing. GPU offloading therefore leaves substantial CPU-side memory management on the allocation path.

GPU-Assisted Page Zeroing

UMLoad’s Page Zeroer offloads page zeroing to the GPU to accelerate runtime memory allocation on UMA. It pipelines CPU submission, GPU execution, and CPU cleanup, then reduces CPU work to keep the pipeline supplied with zeroing tasks.

Pipelining zeroing tasks. To overlap this CPU work with GPU execution, UMLoad splits each request into smaller page batches, each processed as one zeroing task. UMLoad chooses the per-task size to be the smallest that does not incur a visible GPU bandwidth drop (1%), because extra per-task submission costs can be overlapped with GPU execution, and smaller batches mitigate bubbles at the start and end of the pipeline. The three stages then overlap across batches (Figure 7): the CPU prepares and submits task i + 1 while the GPU executes task i, and a separate cleanup worker processes earlier completed tasks. Task order is preserved, and completion fences are used to track per-task status. At Finalize, UMLoad checks the completion fences and waits until the pages are ready. For the same 4 GiB request, pipelining reduces latency from 59.76 to 29.64 ms. However, CPU submission takes 28.8 ms, exceeding the 25.6 ms GPU execution time. Preparing the next task can therefore limit the pipeline’s throughput.

GPU zeroing. CPU zeroing accounts for much of runtime memory allocation (§2.2). A straightforward approach is to offload it to the GPU. The KMD creates an execution context and keeps the zeroing program available across requests. It processes each request through three sequential stages before starting the next: 1. Task submission. The CPU acquires raw physical pages, performs required cache maintenance to make them accessible to the device, installs GPU VA mappings, and prepares task descriptors, command buffers, and completion fences before submitting the task. 2. Task execution. The GPU writes zeros to the mapped pages. Following open-source driver implementations, the KMD builds command buffers that issue hardware fill commands or launch a precompiled zeroing kernel (detailed in §6). 3. Task cleanup. After GPU completion, the CPU performs required visibility operations, releases GPU mappings and task resources, and returns pages to the driver’s allocation path.

Reducing CPU overhead. First, the Page Zeroer creates a private GPU address space, reserves GPU virtual addresses, and installs the page-table hierarchy during initialization. Each task updates existing page-table entries for newly acquired pages and patches arguments in the command templates. This avoids repeated page-table and VA allocation and reduces mapping and task cleanup. Second, cache maintenance requires sequentially issuing flush operations across the entire memory range. UMLoad uses multiple CPU workers to ac-

CPU work after GPU offloading. The CPU performs task submission and cleanup. Offloading zero stores leaves page and resource management on the CPU. Page acquisition, GPU 7

Table 1. Implementation bases and added source lines across evaluated UMA platforms. Platform

Evaluated stack

Userspace interposer (UMD)

Page Reclaimer

Page Zeroer

Intel Core Ultra 9 185H AMD Ryzen AI 9 HX 370 NVIDIA Jetson AGX Orin

Arc Xe-LPG iGPU; Linux 6.16.9; DDR5-5600: 89.6 GB/s; NVMe: 15.75 GB/s Radeon 890M iGPU (gfx1150); Linux 6.16.9; LPDDR5: 120 GB/s; dual NVMe: 5.45 GB/s GA10B iGPU (sm_87); Jetson Linux R39.2 (6.8.12); LPDDR5: 204.8 GB/s; NVMe: 4.27 GB/s

Built on Intel Compute Runtime 25.12; ∼2.7 K LoC Built on ROCm Systems 7.0.2; ∼3.1 K LoC CUDA 13.2 API interposer; CUDA UMD untouched; ∼2.85 K LoC

Built in Intel i915; ∼1.31 K LoC Built in Linux AMDGPU/KFD; ∼1.21 K LoC Built in NVIDIA Jetson Linux driver; 709 LoC

Built in Intel i915; ∼3.4 K LoC Built in Linux AMDGPU/TTM; ∼2.6 K LoC Built in NVIDIA Jetson Linux driver; ∼2.0 K LoC

quire raw pages and process flush operations on independent ranges concurrently, thus increasing cache-maintenance throughput and shortening the total wall time. These optimizations reduce CPU submission time below GPU execution time on both Intel and NVIDIA, reducing gaps between GPU zeroing tasks. Figure 7 illustrates this effect on NVIDIA, where the 4 GiB request latency decreases from 29.64 to 25.96 ms. Detailed measurements and implementation details appear in §7.3 and §6, respectively.

6

waiting for zeroing. The zeroer prepares private pages in the background, accounting for ready pages and work in flight. Successful ReclaimPage calls reduce demand by the reclaimed page count. Finalize waits for and installs private pages in the remaining slots, reducing demand accordingly. When demand decreases, UMLoad waits for preparation of any excess pages to finish and releases them. UMLoad uses the copy engine on AMD and Intel and the compute engine on NVIDIA; all three backends are GPUarchitecture-specific. AMD’s command buffer follows the KMD’s off-the-shelf code for memset and is submitted directly to the SDMA ring. Intel’s command buffer uses XY_FAST_COLOR_BLT commands following the UMD specification for userspace memset, and is submitted as i915_requests on a KMD-owned blit context. NVIDIA’s compute kernel specializes the Mesa driver’s generic compute-fill design for 64 MiB page batches, using coalesced 128-bit stores and a 64-block grid-stride launch sized to Orin’s 16 SMs and precompiled for sm_87. It achieves a speed comparable to cuMemset at sizes larger than 32 MiB and is submitted through a constructed QMD/GPFIFO on a persistent KMD-owned nvgpu TSG/channel in the kernel. UMLoad uses 64 MiB page batches for requests larger than 64 MiB and 16 MiB slots otherwise. Following the benchmark in §7.3, we set a threshold of 8 MiB for the minimum request size of a zeroing task; requests smaller than this threshold are handled by CPU zeroing.

Implementation

We implement UMLoad using open-source drivers on Intel and AMD UMA platforms, and extend the implementation to NVIDIA with a closed-source UMD. Table 1 lists the modified components and their lines of code (LoC) on each platform. UMD interposer. On AMD and Intel, the interposer is integrated into the userspace driver; the modified APIs are limited to memory management and submission paths that resolve buffer-object metadata. On NVIDIA it is a loadable library that interposes the same API classes and uses cuMemAddressReserve to reserve a private range of GPU virtual addresses for UMLoad to manage exclusively. UMLoad provides two primitive APIs on NVIDIA GPUs for frameworks to mark their model-loading phases, during which UMLoad interposes on memory operations. This explicit tagging is necessary to restrict UMLoad’s independently managed memory range to weight tensors. Interposing runtime memory allocations can lead to failures when the CUDA runtime calls APIs that UMLoad has not implemented through manual implementations.

7

Evaluation

We evaluate UMLoad’s model-loading latency and memory use on three representative UMA platforms, with microbenchmarks and ablation studies to explain the performance benefits. Finally, we apply UMLoad to real application traces to evaluate its performance in practical scenarios.

Page Reclaimer. The Page Reclaimer similarily implements ReclaimPage across all three platforms. UMLoad revokes CPU page-table entries (PTEs) using unmap_mapping_range, removes each page-cache entry and establishes exclusive ownership using remove_mapping, and then obtains FOLL_PIN accounting using try_grab_folio manually. Because remove_mapping and try_grab_folio are not exported to modules, UMLoad resolves them through kallsyms lookup.

Workloads. We evaluate across three mainstream inference frameworks: PyTorch (v2.8.0), llama.cpp (tag b8192 for AMD and NVIDIA, b5074 for Intel), and vLLM (v0.11.2). Our workloads comprise three LLMs: Qwen3-1.7B [55] (3.2 GiB), Qwen2.5-3B [53] (5.8 GiB), and Qwen2.5-7B [54] (13.2–14.2 GiB, depending on serialization format), and three conventional models: ConvNeXt-L [39] (749 MiB), WideResNet-101 [67] (486 MiB), and RMBG-2 [9] (845 MiB). Runtime memory reservation is fixed at 6 GiB for all overall model-loading tests.

Page Zeroer. To coordinate PBOs’ delayed physical backing, the Page Zeroer maintains a demand counter tracking U NBOUND slots across open PBOs. CreatePBO increases demand by the requested page count and returns without

Baselines and configuration. We compare UMLoad with 8

Figure 8. Overall model-loading time on Intel, AMD, and NVIDIA platforms for (a) PyTorch LLMs, (b) PyTorch conventional models, (c) llama.cpp LLMs, and (d) vLLM. Crosses denote unavailable measurements. Lower is better.

the following baselines: (1) Vanilla baseline: the vanilla inference frameworks running on the unmodified kernel and userspace drivers with the default model loader option. (2) Pipelined baseline: a framework-level pipelined loader. As in §2.3, for llama.cpp, we use the off-the-shelf pipelined model loader with direct I/O. For PyTorch and vLLM on AMD and NVIDIA, we use Run:ai Model Streamer [60] to enable asynchronous file I/O and H2D copies. vLLM on Intel uses the OpenVINO backend and already utilizes a pipeline that overlaps file I/O and H2D copying.

pipelined paths. The benefit varies across platforms and workloads. Intel generally shows the largest improvement, and its absolute savings increase with model size. In PyTorch, the saving over vanilla grows from 2.74 s at 1.7B to 5.38 s at 7B, while the saving over pipelined loading grows from 1.90 s to 3.09 s. The component measurements explain this trend: file I/O takes only 64%–65% of the combined weight-allocation and copy time, and the pipeline time beyond file I/O grows from 0.26 s to 1.13 s. Consequently, substantial memory-preparation work remains exposed even with pipelining, increasing the opportunity for UMLoad as model size grows. On AMD and NVIDIA, the advantage over pipelined loading narrows for larger models. Averaged over PyTorch and llama.cpp, the reduction falls from 26.8% at 3B to 12.8% at 7B on AMD, and from 44.0% to 22.1% on NVIDIA. In the I/O-normalized PyTorch pipeline, the read component accounts for 58.7%–59.9% of pipeline time at 3B, increasing to 97.8%–98.0% at 7B. Pipelining therefore leaves less weighttransfer time exposed for UMLoad to eliminate. The time saved on runtime memory allocation is approximately constant across model sizes. Its contribution to relative speedup consequently decreases as loading time grows. vLLM additionally retains more model-construction and warm-up work in model loading, reducing UMLoad’s relative benefit. For conventional models, UMLoad outperforms both baselines on Intel and AMD. NVIDIA’s gains over Pipeline are smaller because its slower disk bandwidth hides more weight transfer than the other two platforms. For example, ConvNeXt-L takes 412 ms with UMLoad versus 417 ms with

Metrics. For LLMs, we measure latency from the first framework loading operation until prefill begins. For conventional models, we measure from the same starting point to the completion of weight transfer. Runtime initialization times are excluded. 7.1

Overall Model-Loading Performance

Overall latency and memory. Figure 8 compares UMLoad with vanilla and pipelined model loading across the three platforms. We report UMLoad with only On-demand Reconstruction (ODR) enabled and with both optimizations enabled. Averaged over the available model–framework configurations, UMLoad reduces loading time over vanilla by 71.9%, 44.7%, and 36.6% on Intel, AMD, and NVIDIA, respectively. The corresponding reductions over pipelined loading are 62.7%, 19.9%, and 22.3%. The vanilla Qwen2.5-7B configurations on AMD and NVIDIA are unavailable because their transient peak memory exceeds the platforms’ memory budgets. UMLoad supports the model and remains faster than the available 9

Table 2. Peak memory during Intel PyTorch model loading (GiB). Values in parentheses show peak memory relative to the weight-file size.

Method

Qwen3-1.7B Qwen2.5-3B Qwen2.5-7B

Baseline Run:ai UMLoad

6.46 (2.02×) 3.91 (1.21×) 3.29 (1.02×)

11.72 (2.01×) 6.63 (1.16×) 5.84 (1.02×)

28.60 (2.01×) 15.50 (1.10×) 14.27 (1.01×)

Pipeline, a 1.2% reduction; RMBG-2 improves by 4.3%. We then measure the transient memory footprint of each loading path. Table 2 reports peak memory measured with the cgroup memory controller during Intel PyTorch loading. We subtract growth in the model-file page cache because it is evictable. The vanilla baseline peaks at approximately twice the weight-file size, while Run:ai reduces the footprint to 1.10×–1.21× by reusing a bounded staging pool. UMLoad remains within 1.01×–1.02× of the weight-file size. The vanilla baseline’s peak memory consists of the weight file and the copy-on-write memory produced during weight transfer. Run:ai’s consists of the weight file and its reusable staging buffer, while UMLoad’s consists of the weight file and framework-managed memory; no transient memory amplification is introduced during weight transfer.

Figure 9. Weight-transfer API latency on (a) Intel, (b) AMD, and (c) NVIDIA. Baseline allocates device memory and copies weights from the host. Latency excludes file I/O.

Figure 10. Effect of huge-page mapping recovery on steady-state decode TBT with On-demand Reconstruction (ODR) on Intel and AMD. Values show the change relative to vanilla inference.

Ablation. We compare UMLoad with ODR alone and with both optimizations enabled to quantify their contributions to loading performance. ODR alone reduces loading time by 10.7%–55.9% relative to vanilla across supported LLM configurations. It covers 99.9956%–100% of weight-transfer bytes; the remaining bytes are in boundary pages handled by copying. As a result, weight-transfer latency falls to 3.13%, 7.46%, and 15.76% of its vanilla value on Intel, AMD, and NVIDIA, respectively. Adding GPU-assisted allocation further reduces loading time by an average of 1.64 s on Intel, 228 ms on AMD, and 500 ms on NVIDIA. The differences in absolute savings reflect each platform’s baseline runtime-memory allocation cost. With both optimizations enabled, weight transfer and runtime-memory allocation together account for only 9.46%, 2.93%, and 13.2% of total loading time on Intel, AMD, and NVIDIA, respectively. 7.2

tency. At 1 GiB, ReclaimPage is at least 6× faster than the baseline and shared memory on all three platforms. AMD host registration is the only exception. Host registration only maps the existing source pages, whereas ReclaimPage takes ownership of the pages and initializes the corresponding PBO’s metadata. On AMD, ReclaimPage spends 4.04 ms committing ownership and another 1.91 ms attaching the pages, while host registration only changes the GPU page table, attaching the pages in 2.36 ms. This explains why host registration is faster in its best case (changing the source buffer’s mapping to a writable private mapping will trigger synchronous copy-on-write and severely degrade its performance). In return, ReclaimPage owns the backing storage, preserving memory independence and providing more stable and transparent acceleration. Huge-page mapping recovery. Figure 10 evaluates whether huge pages and alignment recover decode performance with ODR. Without rewriting the weight file or enabling THP, reclaimed pages fall back to 4 KiB entries. This increases time between tokens (TBT) by 1.3%–4.5% on Intel and 3.4%– 10.3% on AMD across the three models. UMLoad maintains the proportion of weight bytes covered by huge-page mappings at 96.41%–99.94% (vanilla: 100%). The residuals are BOs packed with tensors at noncontiguous GPU VAs, for which UMLoad cannot use Reorder to restore the contiguity disrupted by earlier Padding. This recovery also removes the inference regression: TBT with aligned ODR

Effect of On-demand Reconstruction

ReclaimPage Latency. Figure 9 compares ReclaimPage API with two zero-copy APIs discussed in §2.3 to show UMLoad’s performance relative to alternative weight-transfer APIs. Note that using the two zero-copy APIs requires framework code to be modified. The baseline (device allocation followed by an H2D copy) is added as the conventional reference. The final metric reports the sum of critical steps for each path. Weight-file I/O is excluded from timing for all methods. At small sizes, all paths have similar sub-millisecond la10

CPU baseline

GPU baseline

Submission GPU Execution

UMLoad

(a) Allocation latency

Intel

AMD

750

300

NVIDIA

100

200 200

500 100

100

250 0

(b) Optimization breakdown (4 GiB)

Time (ms)

Time (ms)

1000

.25 .5

1

2

4

Size (GiB)

8

0

.25 .5

1

2

4

Size (GiB)

8

Cleanup UMLoad

0

.25 .5

1

2

4

8

75 50 25 0

Intel

AMD

NVIDIA

Size (GiB)

Figure 11. End-to-end allocation latency across three UMA platforms. (a) Vanilla allocation using CPU zeroing, baseline allocation using GPU zeroing, and UMLoad using reduced CPU work and pipelined GPU zeroing. (b) UMLoad’s optimization breakdown for 4 GiB. Within each platform, bars from left to right show baseline CPU-side latency, UMLoad CPU-side latency, UMLoad’s GPU execution latency, and overall allocation latency using the pipeline (UMLoad).

stays within 0.1% of the copy-backed baseline on Intel and within 0.6% on AMD. Relative to unaligned ODR, alignment reduces TBT by 1.3%–4.4% and 3.0%–9.9% on Intel and AMD, respectively. The NVIDIA driver uses base-page GPU mappings and does not exhibit the alignment-induced decode slowdown even without huge pages or 2 MiB realignment (less than 0.5% change in TBT).

stages sequentially once. At 4 GiB, UMLoad is 2.29×–6.40× faster than CPU zeroing and 1.21×–2.40× faster than the GPU baseline. Gains between the CPU baseline and GPU baseline come from GPU-zeroing acceleration. UMLoad’s additional advantage over the GPU baseline comes from pipelining CPU work with GPU execution and shortening CPU-side work.

Introduced overheads. Aligning weights to 2 MiB increases file size by 1.07%–5.13% across the three evaluated Qwen models (ratios are larger on small models). The file rewrite is performed offline. The larger file slightly increases the number of bytes read during subsequent model loads, but the padding bytes are not transferred to the GPU as weights. Later CPU access can also incur reconstruction I/O, faulting source pages back in from the file while the GPU retains the reclaimed pages. For dirty pages, ReclaimPage must complete writeback before acquiring ownership. In the configured microbenchmark, median reclamation latency for 1 GiB rises from 41.4 ms to 3.79 s on Intel, from 6.16 ms to 390 ms on AMD, and from 12.7 ms to 330 ms on NVIDIA. The severe slowdown comes from synchronous disk operations during writeback. After reclamation, a CPU access to the source range can fault pages in from the backing file while the GPU retains the reclaimed pages. Faulting a cold 5.75 GiB file mapping takes a median of 717, 1091, and 2091 ms on Intel, AMD, and NVIDIA, respectively, matching each platform’s measured storage bandwidth. Nevertheless, dirty-page writeback and CPU-side reconstruction are uncommon for file-backed source buffers in normal loading and were not observed in the workloads we evaluated.

CPU work reduction and pipelining. Pipelining CPU submission, GPU zeroing, and cleanup across chunks reduces 4 GiB allocation latency from 179.41 to 93.92 ms on Intel and from 59.76 to 29.64 ms on NVIDIA. However, pipelining alone leaves CPU submission as the bottleneck on these platforms. At 4 GiB, UMLoad further reduces submission latency from 92.80 to 56.77 ms on Intel and from 28.65 to 17.45 ms on NVIDIA, bringing both below their GPU completion times of 74.03 and 25.61 ms, respectively. On Intel, multithreading reduces cache-maintenance wall time from 78.95 ms to at most 56.77 ms. On NVIDIA, reusing mapping resources reduces GPU mapping from 10.73 to 1.17 ms and task construction from 1.48 to 0.036 ms. Together, these optimizations improve pipeline efficiency, reducing allocation latency further by 20.5% on Intel and 12.4% on NVIDIA compared with pipelining alone. Figure 11(b) shows the resulting CPU-side latency, including submission and cleanup, against the serial GPU baseline. AMD’s CPU work is already small relative to its GPU phase (Figure 4). Although CPU-side time increases from 5.19 to 8.57 ms, UMLoad reduces allocation latency from 63.30 to 52.17 ms through its optimized pipeline. Across all three platforms, the pipeline approaches GPU completion time: at 4 GiB, end-to-end latency is only 0.9%, 2.4%, and 1.1% above GPU execution time on Intel, AMD, and NVIDIA, respectively.

7.3

Effect of GPU-Assisted Allocation

Overall allocation latency. Figure 11(a) shows that UMLoad outperforms both baseline allocation with CPU zeroing and allocation with naive GPU-assisted zeroing across all three platforms over 256 MiB–8 GiB. The GPU baseline uses the same in-kernel context and zeroing kernel and runs the three

7.4

Case Studies

We evaluate how UMLoad reduces response latency in the LLM-routing and tool-calling workloads introduced in §2.1. In both cases, runtimes remain warm while models are cold11

Baseline

UMLoad

Loading

generation with Llama 3.2 1B [38], visual question answering with ViLT [26], and ASR with Wav2Vec2 Base 960h [41]. UMLoad reduces tool response latency by 25.1% on average, ranging from 10.2% for DETR to 31.6% for ViT (Figure 12(b)). These gains come from faster loading; inference remains unchanged. The slight increase in the remaining latency is due to longer disk reads caused by the larger weight files produced by UMLoad’s rewrite. ViT has 200 tensors with a median size of 9 MiB, whereas 681 of DETR’s 785 tensors are smaller than 1 MiB. DETR’s many small tensors require more frequent reclamation calls, and framework-side construction accounts for more of its loading time, limiting its gains.

Remainder

(a) LLM Routing 30.4%

20

Response latency (s)

15 10

51.3%

59.3%

24.0%

15.2%

19.3%

TTS

ASR

ImgGen

30.7%

28.6%

VQA

ASR

5 0

Code

General

Vision

(b) Tool calling 22.0%

27.6%

3 2 1 0

10.2%

31.6%

Img. cls.

Img. gen.

Obj. det.

Text gen.

8

Related Work

UMA characterization. UMA allows CPUs and GPUs to share physical memory, making duplicate allocations and data copies avoidable [64]. However, exploiting this capability requires appropriate software support. Wahlgren et al. [68] characterize allocation, page faults, address translation, and memory access on MI300A, and propose porting strategies that address both performance and correctness when replacing separate CPU/GPU buffers with shared storage. Yang et al. [76] characterize CPU–GPU memory-sharing correctness and performance across UMA platforms and introduce libumsh, including a PyTorch case study that eliminates copies by loading weights directly into shared memory. Complementary hardware research improves coherence for CPU–GPU sharing [43].

Figure 12. Response latency for (a) LLM routing and (b) tool calling. Code/General measure TTFT; multimodal LLMs and tools measure completion time. Percentages give latency reductions.

loaded for each request or tool invocation. We measure TTFT for coding and general requests, and completion time for multimodal requests and tools. LLM routing. We route among six models that cannot all fit in memory: Qwen3-30B-Int4 for coding [59], GLM-4.7Flash-Int4 for general requests [79], Qwen3-VL-2B-Instruct (Q8_0) for vision-language tasks [56], Qwen3-TTS-1.7B (F16) for TTS [58], Qwen3-ASR-0.6B (BF16) for ASR [57], and Flux2 Klein-4B (Q4_0) for image generation [8]. Request traces interleave samples from LMSYS-Chat1M [80] and ToolMMBench [69]. Flux2 and Qwen3-TTS use stable-diffusion.cpp and qwen3-tts.cpp, respectively; the other models use llama-server, requiring a 6 GiB reservation for runtime memory. The baseline uses pipelined loading where supported by llama-server and vanilla loading otherwise; UMLoad enables all available optimizations. Figure 12(a) shows that UMLoad improves both types of response latency: TTFT for normal LLMs and end-to-end completion time for multimodal LLMs. UMLoad reduces TTFT by 55.3% on average across coding and general requests, and completion latency by 22.2% across the multimodal workloads. For the latter, model-loading time falls by 49.8% on average, while the remainder (mostly inference) accounts for 73.9% of the optimized response time, showing that UMLoad removes much of the startup cost, making the remaining inference and framework overheads the dominant remaining cost.

Application-level UMA adoption. Application-level efforts demonstrate UMA’s benefits in HPC: OpenFOAM adopts OpenMP’s unified-memory model for GPU offloading [66], iPIC3D uses unified memory for large-scale plasma simulations [37], and Quicksilver/CloverLeaf benefit from allocatoraware inter-APU communication [61]. These works guide application adaptation to shared memory. For mobile LLMs, mzCache [78] modifies llama.cpp to manage weights and KV state in shared UMA buffers and overlap restoration of evicted state with inference. UMLoad targets model-loader integration, preserving independent CPU/GPU buffer semantics while eliminating weight copies, and accelerates zeroing of fresh runtime pages. Runtime support for unified memory. Runtime support also enables unified-memory adoption in other settings. Fumero et al. [15] place Java heaps in unified memory and coordinate garbage collection with GPU execution to support shared object access. Bertolli et al. [7] provide implicit zero-copy execution for existing OpenMP applications and eager GPU page-table mapping to reduce first-access overhead.

Tool calling. We replay tool invocations from MNMs benchmark [36] using their plans, arguments and inputs. The benchmark’s PyTorch scripts serve six local tools: image classification with ViT Base [17], image generation with SDTurbo [65], object detection with DETR ResNet-101 [40], text

9

Conclusion

This paper presents UMLoad, a system that accelerates ondemand model loading on UMA devices by eliminating re12

dundant weight copies and accelerating runtime allocation. Our evaluation across platforms, frameworks, and workloads demonstrates lower loading latency and memory use, improving response latency for on-device AI.

Chakrabarti, Lawrence Meadows, and Ron Lieberman. Performance analysis of runtime handling of zero-copy for OpenMP programs on MI300A APUs. In SC24W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1420–1429, Piscataway, NJ, USA, 2024. IEEE.

References [1] Sivanand Achanta, Albert Antony, Ladan Golipour, Jiangchuan Li, Tuomo Raitio, Ramya Rasipuram, Francesco Rossi, Jennifer Shi, Jaimin Upadhyay, David Winarsky, and Hepeng Zhang. On-device neural speech synthesis. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1155– 1161, Piscataway, NJ, USA, 2021. IEEE.

[8] Black Forest Labs and leejet. FLUX.2-klein-4BGGUF. https://huggingface.co/leejet/ FLUX.2-klein-4B-GGUF. [9] BRIA AI. RMBG-2.0. https://huggingface. co/briaai/RMBG-2.0.

[2] Advanced Micro Devices, Inc. AMD announces expanded consumer and commercial AI PC portfolio at CES. https://ir.amd.com/newsevents/press-releases/detail/1232/ amd-announces-expanded-consumer-andcommercial-ai-pc-portfolio-at-ces, 2025.

[10] Hossein Falaki, Ratul Mahajan, Srikanth Kandula, Dimitrios Lymberopoulos, Ramesh Govindan, and Deborah Estrin. Diversity in smartphone usage. In Proceedings of the 8th international conference on Mobile systems, applications, and services, pages 179–194, New York, NY, USA, 2010. Association for Computing Machinery. [11] Biyi Fang, Xiao Zeng, and Mi Zhang. NestDNN: Resource-aware multi-tenant on-device deep learning for continuous mobile vision. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pages 115–127, New York, NY, USA, 2018. Association for Computing Machinery.

[3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, Santa Clara, CA, July 2024. USENIX Association. https://www.usenix.org/conference/ osdi24/presentation/agrawal.

[12] Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in oneshot. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10323– 10337. PMLR, 2023. https://proceedings. mlr.press/v202/frantar23a.html.

[4] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12562–12584, Bangkok, Thailand, August 2024. Association for Computational Linguistics.

[13] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023. https://arxiv.org/abs/2210.17323. [14] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Low-latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 135–153, Santa Clara, CA, July 2024. USENIX Association. https://www.usenix.org/conference/ osdi24/presentation/fu.

[5] Apple Inc. Apple introduces M3 Ultra. https://www.apple.com/newsroom/2025/ 03/apple-reveals-m3-ultra-takingapple-silicon-to-a-new-extreme/, 2025. [6] Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic AI. arXiv:2506.02153, 2025.

[15] Juan Fumero, Florin Blanaru, Athanasios Stratikopoulos, Steve Dohrmann, Sandhya Viswanathan, and Christos Kotselidis. Unified shared memory: Friend or foe? understanding the implications of unified memory on

[7] Carlo Bertolli, Thorsten Blass, Lynd Stringer, Nicole Aschenbrenner, Jan-Patrick Lehr, Doru Bercea, Dhruva 13

[16] ggml-org. llama.cpp. https://github.com/ ggml-org/llama.cpp.

[25] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLMBlender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada, July 2023. Association for Computational Linguistics.

[17] Google. ViT Base (Patch16-224). https: //huggingface.co/google/vit-basepatch16-224.

[26] Wonjae Kim. ViLT B32 Fine-tuned on VQAv2. https://huggingface.co/dandelin/ vilt-b32-finetuned-vqa.

[18] Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo-yiin Chang, Kanishka Rao, and Alexander Gruenstein. Streaming end-to-end speech recognition for mobile devices. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6381–6385, Piscataway, NJ, USA, 2019. IEEE.

[27] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pages 611–626, New York, NY, USA, 2023. Association for Computing Machinery.

managed heaps. In Proceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, MPLR 2023, pages 143– 157, New York, NY, USA, 2023. Association for Computing Machinery.

[28] Jiedong Lang, Zhehao Guo, and Shuyu Huang. A Comprehensive Study on Quantization Techniques for Large Language Models. arXiv:2411.02530, October 2024.

[19] Hugging Face. Safetensors: Memory-mapped PyTorch storage. https://github.com/huggingface/ safetensors/blob/v0.6.2/bindings/ python/src/lib.rs. Version 0.6.2.

[29] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. SnapFusion: Text-to-image diffusion model on mobile devices within two seconds. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 20662–20678, Red Hook, NY, USA, 2023. Curran Associates, Inc.

[20] Intel Corporation. OpenVINO documentation. https: //docs.openvino.ai. [21] Intel Corporation. Optimizing memory usage in OpenVINO. https://docs.openvino.ai/ nightly/openvino-workflow/runninginference/optimize-inference/ optimizing-memory-usage.html.

[30] Zonghang Li, Wenjiao Feng, Mohsen Guizani, and Hongfang Yu. TPI-LLM: Serving 70B-Scale LLMs Efficiently on Low-Resource Mobile Devices. IEEE Transactions on Services Computing, 18(5):3321–3333, September 2025.

[22] Intel Corporation. Intel Core Ultra Processors. https://www.intel.com/content/www/ us/en/products/details/processors/ core-ultra.html, 2025.

[31] Zonghang Li, Tao Li, Wenjiao Feng, Rongxing Xiao, Jianshu She, Hong Huang, Mohsen Guizani, Hongfang Yu, Qirong Ho, Wei Xiang, and Xue Liu. Prima.cpp: Fast 30–70b LLM inference on heterogeneous and low-resource home clusters. In The Fourteenth International Conference on Learning Representations, 2026. https://iclr.cc/virtual/ 2026/poster/10008093.

[23] Joo Seong Jeong, Jingyu Lee, Donghyun Kim, Changmin Jeon, Changjin Jeong, Youngki Lee, and ByungGon Chun. Band: Coordinated multi-DNN inference on heterogeneous mobile processors. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, MobiSys ’22, pages 235–247, New York, NY, USA, 2022. Association for Computing Machinery.

[32] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, WeiMing Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activationaware Weight Quantization for On-Device LLM Compression and Acceleration. In P. Gibbons, G. Pekhimenko, and C. De Sa, editors, Proceedings of Machine

[24] Fucheng Jia, Zewen Wu, Shiqi Jiang, Huiqiang Jiang, Qianxi Zhang, Yuqing Yang, Yunxin Liu, Ju Ren, Deyu Zhang, and Ting Cao. Scaling up on-device LLMs via active-weight swapping between DRAM and flash. arXiv:2504.08378, 2025.

14

[40] Meta AI. DETR ResNet-101. https: //huggingface.co/facebook/detrresnet-101.

Learning and Systems, volume 6, pages 87–100. MLSys, 2024. https://proceedings.mlsys. org/paper_files/paper/2024/file/ 42a452cbafa9dd64e9ba4aa95cc1ef21Paper-Conference.pdf.

Wav2Vec2 Base 960h. https: [41] Meta AI. //huggingface.co/facebook/wav2vec2base-960h.

[33] Xuran Liu, Nan Xue, Rui Bao, Yaping Sun, Zhiyong Chen, Meixia Tao, Xiaodong Xu, and Shuguang Cui. CSGO: Generalized optimization for cold start in wireless collaborative edge LLM systems. arXiv:2508.11287, 2025.

[42] Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Minions: Cost-efficient collaboration between on-device and cloud language models. arXiv:2502.15964, 2025.

[34] Yubo Liu, Hongbo Li, Xiaojia Huang, Yongfeng Wang, Hanjun Guo, Hui Chen, Yuxin Ren, and Ning Jia. Accelerating model loading in LLM inference by programmable page cache. In 24th USENIX Conference on File and Storage Technologies (FAST 26), pages 117–132, Santa Clara, CA, February 2026. USENIX Association. https://www.usenix.org/conference/ fast26/presentation/liu-yubo.

[43] Anoop Mysore Nataraja, Ricardo Fernández-Pascual, and Alberto Ros. Enhanced system-level coherence for heterogeneous unified memory architectures. In 2024 IEEE International Symposium on Workload Characterization (IISWC), pages 273–283, Piscataway, NJ, USA, 2024. IEEE. [44] NVIDIA Corporation. CUDA Runtime API: Memory management. https://docs.nvidia.com/ cuda/cuda-runtime-api/group__CUDART_ _MEMORY.html.

[35] Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. HydraServe: Minimizing cold start latency for serverless LLM serving in public clouds. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pages 415–430, Renton, WA, May 2026. USENIX Association. https://www.usenix.org/ conference/nsdi26/presentation/lou.

[45] NVIDIA Corporation. CUDA Runtime API: Stream ordered memory allocator. https://docs. nvidia.com/cuda/cuda-runtime-api/ group__CUDART__MEMORY__POOLS.html. [46] NVIDIA Corporation. GPUDirect Storage documentation. https://docs.nvidia.com/ gpudirect-storage/.

[36] Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks. In Computer Vision – ECCV 2024, volume 15068 of Lecture Notes in Computer Science, pages 18–34, Cham, Switzerland, 2024. Springer Nature Switzerland.

[47] NVIDIA Corporation. NVIDIA GPUDirect Storage release notes. https://docs.nvidia. com/gpudirect-storage/releasenotes/index.html. Accessed September 25, 2026.

[37] Stefano Markidis, Andong Hu, Ivy Peng, Luca Pennati, Ian Lumsden, Dewi Yokelson, Stephanie Brink, Olga Pearce, Thomas R. W. Scogland, Bronis R. de Supinski, Gian Luca Delzanno, and Michela Taufer. Exascale implicit kinetic plasma simulations on El Capitan for solving the micro-macro coupling in magnetospheric physics. arXiv:2507.20719, 2025.

[48] NVIDIA Corporation. NVIDIA Jetson Orin: Orin NX specifications. https://www.nvidia.com/enus/autonomous-machines/embeddedsystems/jetson-orin/. Accessed September 25, 2026. [49] NVIDIA Corporation. NVIDIA DGX Spark. https://www.nvidia.com/enus/products/workstations/dgx-spark/, 2025.

[38] Meta. Llama 3.2 1B Instruct. https: //huggingface.co/meta-llama/Llama3.2-1B-Instruct. [39] Meta AI. ConvNeXt Large (224). https: //huggingface.co/facebook/convnextlarge-224.

[50] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data. arXiv:2406.18665, 2024.

15

[63] Yingtian Shi, Jinda Yang, Yuhan Wang, Yiwen Yin, Haoyu Li, Kunyu Gao, and Chun Yu. LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition. arXiv:2510.19252, October 2025.

[51] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 126544–126565, Red Hook, NY, USA, 2024. Curran Associates, Inc.

[64] Alan Smith, Gabriel H. Loh, Michael J. Schulte, Mike Ignatowski, Samuel Naffziger, Mike Mantor, Mark Fowler, Nathan Kalyanasundharam, Vamsi Alla, Nicholas Malaya, Joseph L. Greathouse, Eric Chapman, and Raja Swaminathan. Realizing the AMD exascale heterogeneous processor vision: Industry product. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 876–889, Piscataway, NJ, USA, 2024. IEEE.

[52] PyTorch Foundation. PyTorch. https://pytorch. org. [53] Qwen Team. Qwen2.5-3B-Instruct. https: //huggingface.co/Qwen/Qwen2.5-3BInstruct. [54] Qwen Team. Qwen2.5-7B-Instruct. https: //huggingface.co/Qwen/Qwen2.5-7BInstruct.

[65] Stability AI. SD-Turbo. https://huggingface. co/stabilityai/sd-turbo.

Qwen3-1.7B. https: [55] Qwen Team. //huggingface.co/Qwen/Qwen3-1.7B.

[66] Suyash Tandon, Leopold Grinberg, Gheorghe-Teodor Bercea, Carlo Bertolli, Mark Olesen, Simone Bna, and Nicholas Malaya. Porting HPC applications to AMD Instinct MI300A using unified memory and OpenMP. In ISC High Performance 2024 Research Paper Proceedings (39th International Conference), pages 1–9, Piscataway, NJ, USA, 2024. Prometeus GmbH, IEEE.

[56] Qwen Team. Qwen3-VL-2B-Instruct-GGUF. https://huggingface.co/Qwen/Qwen3VL-2B-Instruct-GGUF. [57] Qwen Team and ggml-org. Qwen3-ASR-0.6BGGUF. https://huggingface.co/ggmlorg/Qwen3-ASR-0.6B-GGUF.

[67] timm Maintainers. Wide ResNet-101-2 (Torchvision V2, ImageNet-1k). https://huggingface.co/ timm/wide_resnet101_2.tv2_in1k.

[58] Qwen Team and librellama. Qwen3-TTS GGUF Models. https://huggingface.co/librellama/ qwen3-tts-GGUF.

[68] Jacob Wahlgren, Gabin Schieffer, Ruimin Shi, Edgar A. León, Roger Pearce, Maya Gokhale, and Ivy Peng. Dissecting CPU-GPU Unified Physical Memory on AMD MI300A APUs. In 2025 IEEE International Symposium on Workload Characterization (IISWC), pages 368–380, Piscataway, NJ, USA, 2025. IEEE.

[59] Qwen Team and Unsloth AI. Qwen3-Coder-30BA3B-Instruct-GGUF. https://huggingface. co/unsloth/Qwen3-Coder-30B-A3BInstruct-GGUF. [60] Run:ai. Run:ai Model Streamer. https://github. com/run-ai/runai-model-streamer, 2025.

[69] Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Xiaohua (Michael) Xuan, Zhengxin Li, Lin Ma, and Shenghua Gao. Tool-LMM: A large multi-modal model for tool agent learning. arXiv:2401.10727v1, January 2024.

[61] Gabin Schieffer, Jacob Wahlgren, Ruimin Shi, Edgar A. León, Roger Pearce, Maya Gokhale, and Ivy Peng. InterAPU communication on AMD MI300A systems via Infinity Fabric: A deep dive. In Proceedings of the International Symposium on Memory Systems, pages 238–250, New York, NY, USA, 2025. Association for Computing Machinery.

[70] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents enhances large language model capabilities. arXiv:2406.04692, 2024. [71] Xunjie Wang, Jiacheng Shi, Zihan Zhao, Yang Yu, Zhichao Hua, and Jinyu Gu. TZ-LLM: Protecting OnDevice Large Language Models with Arm TrustZone. In Proceedings of the 21st European Conference on Computer Systems, EUROSYS ’26, pages 657–674, New York, NY, USA, April 2026. Association for Computing Machinery.

[62] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 38154– 38180, Red Hook, NY, USA, 2023. Curran Associates, Inc.

16

[72] Jinhui Wei, Ye Huang, Yuhui Zhou, Jiazhi Jiang, Jiangsu Du, and Yutong Lu. Ghidorah: Fast LLM inference on edge with speculative decoding and hetero-core parallelism. arXiv:2505.23219, 2025.

[77] Wangsong Yin, Mengwei Xu, Yuanchun Li, and Xuanzhe Liu. LLM as a system service on mobile devices. arXiv:2403.11805, 2024. [78] Hongseung Yu, Minsung Kim, Jongseok Park, and Kyunghan Lee. mzCache: On-device LLM memory management under multitasking. arXiv:2609.01338, September 2026.

[73] Xinming Wei, Jiahao Zhang, Haoran Li, Jiayu Chen, Haoning Guan, Rui Qu, Maoliang Li, Xiang Chen, and Guojie Luo. Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC. arXiv:2506.24045, January 2026.

[79] Z.ai and Unsloth AI. GLM-4.7-Flash-GGUF. https://huggingface.co/unsloth/GLM4.7-Flash-GGUF.

[74] Jie Xiao, Qianyi Huang, Xu Chen, and Chen Tian. Understanding large language models in your pockets: Performance study on COTS mobile devices. arXiv:2410.03613, 2024. [75] Yongsheng Yan, Jiacheng Shen, Xuchuan Luo, and Yangfan Zhou. EdgeFlow: Fast cold starts for LLMs on mobile devices. arXiv:2604.09083, 2026.

[80] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. arXiv:2309.11998, March 2024.

[76] Jinrong Yang, Zimeng Wang, Rong Chen, and Haibo Chen. Characterizing CPU-GPU memory sharing over Unified Memory Architecture. https://arxiv. org/abs/2610.13125, October 2026.

[81] Wenbin Zhu, Zhaoyan Shen, Zili Shao, Hongjun Dai, and Feng Chen. Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity. arXiv:2512.01357, December 2025.

17

Related documents

• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption#798900
• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption#797486
• LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference#925446
• LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference#921210
• T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator#1216473