arXiv CS open access

AutoUVM: Automated Prefetching Framework for LLMs under UVM Oversubscription

· arxiv_cs

arXiv CS · Papers

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

AUTO UVM: Automated Prefetching Framework for LLMs under UVM Oversubscription Mao Lin∗ , Hui Feng∗ , Xianzhong Ding∗ , Guilherme Cox† , Qian Wang∗ , Hyeran Jeon∗

arXiv:2609.06172v1 [cs.OS] 5 Sep 2026

∗ University of California, Merced

Abstract—Large language models (LLMs) increasingly exceed the memory capacity of commodity GPUs, making memory oversubscription common in practical deployments. NVIDIA Unified Virtual Memory (UVM) provides transparent access to host memory, but its page-fault–driven migrations introduce severe performance overhead. While UVM exposes primitives (e.g., prefetching and placement hints) to mitigate these costs, they require low-level CUDA modifications, limiting their applicability for most LLM users. Meanwhile, existing UVM optimizations operate at coarse managed-object granularity and fail to capture deep learning frameworks’ internal tensor-level memory behavior, leading to excessive data movement and CPU–GPU interconnect bottlenecks. We propose AUTO UVM, an automated, framework-aware UVM prefetching system for efficient LLM execution under memory oversubscription. AUTO UVM bridges the semantic gap between deep learning frameworks and UVM by exposing tensorlevel access information and enabling policy-driven prefetching at fine granularity. Implemented as a transparent extension, AUTO UVM requires no changes to model code and dynamically adapts to runtime memory pressure. We instantiate AUTO UVM with a roofline-inspired policy to identify performance-critical data transfers. Across ten LLMs, AUTO UVM achieves an average 3.1× speedup over baseline UVM and consistently surpasses the best-performing prior UVM prefetcher by 1.9×, with improvements of up to 4.7× over object-level prefetchers, while significantly reducing page faults. Index Terms—Memory Oversubscription, Prefetching, UVM, LLMs, GPU Memory

I. I NTRODUCTION Memory demand continues to grow rapidly in modern data-intensive workloads, especially for large language model (LLM) training and serving, making GPU memory capacity a critical bottleneck. To mitigate memory pressure, prior work has explored multi-GPU parallelization [1], host-memory offloading [2], activation recomputation [3], and memory compression or quantization [4]. However, these approaches often require workload-specific tuning, specialized hardware support, or remain fundamentally limited by GPU memory capacity. To alleviate limited GPU memory capacity, NVIDIA’s Unified Virtual Memory (UVM) can be used. UVM unifies CPU and GPU address spaces and enables on-demand data migration between host and device memory. With UVM, applications can allocate memory beyond the GPU’s physical capacity, enabling memory oversubscription with minimal programmer intervention. With prefetch APIs, such as cudaMemPrefetchAsync and cudaMemAdvise, devel-

† NVIDIA

opers can explicitly control and optimize the migration and placement of UVM-managed data objects. However, manual UVM management requires substantial engineering effort, including source-code modifications and workload-specific expertise. In contrast, UVM’s autonomous memory-management mechanisms, including its tree-based prefetch engine and follow-up variants [5]–[7], exploit coarse-grained locality to prefetch and evict data without programmer intervention. Existing programmer-agnostic UVM prefetching approaches operate at the granularity of UVM-managed objects by leveraging the UVM driver’s object-level access monitoring capability [2], [5]–[7]. However, modern deep learning (DL) frameworks introduce a semantic gap between framework-level tensor management and the UVM driver. Frameworks such as PyTorch and TensorFlow employ internal allocators (e.g., the PyTorch Caching Allocator (PCA)) [8], [9], that reserve large monolithic memory pools and sub-allocate them to many tensors. These sub-allocations are invisible to the UVM driver; from its perspective, tensor accesses appear as reuse within a few large managed objects. Consequently, the UVM driver cannot observe tensorlevel memory activities, preventing accurate tensor-level prefetching and limiting the effectiveness of autonomous UVM optimizations. This mismatch fundamentally limits the effectiveness of prior UVM optimizations, not only prefetching, but also page placement, proactive eviction, and offloading. To address these limitations, a practical UVM solution must (1) recover tensor-level access information hidden by DL frameworks, (2) identify which tensor migrations are performance-critical, and (3) dynamically adapt decisions as memory pressure and eviction behavior evolve during execution. In this work, we propose AUTO UVM, an automated, framework-semantic-aware UVM prefetching framework for oversubscribed LLM workloads. Because tensor-level memory activities are hidden behind framework-managed memory pools, the UVM driver cannot observe which tensors are accessed by individual kernels at runtime. To recover this missing information without modifying model code, AUTO UVM performs a single offline profiling pass to recover kernel–tensor access relationships. While profile-guided prefetching with runtime monitoring has been extensively explored [10]–[13], AUTO UVM is the first to bring it to the UVM path and enable automated tensor-level prefetching

without modifying model code. During execution, AUTO UVM automatically issues policy-driven, fine-grained tensor-level prefetches. Since LLM kernels exhibit heterogeneous sensitivity to CPU–GPU migration latency, blindly prefetching all tensors can waste CPU–GPU interconnect (CGI) bandwidth and increase GPU-memory pressure. To selectively target migrations that are likely to improve performance, AUTO UVM decouples framework-semantic extraction from prefetch-policy design, enabling different policies to be deployed across workloads and hardware platforms. In this paper, we instantiate AUTO UVM with a roofline-inspired policy that distinguishes CGI-bound, GPU-memory-bound, and compute-bound kernels. Furthermore, because oversubscription-induced evictions can alter runtime behavior, AUTO UVM combines offline profiles with a lightweight online executor that continuously adapts prefetching decisions as execution conditions evolve. The key insight behind AUTO UVM is that effective UVM management requires both tensor-level visibility and selective prefetching. Prior approaches operate without tensor semantics, which is fundamentally misaligned with tensorlevel memory management in DL frameworks and often results in excessive data migration. By combining tensor-aware profiling with roofline-inspired analysis, AUTO UVM reduces unnecessary migrations, mitigates CGI bottlenecks, reduces page faults, and enables predictable, performance-aware UVM behavior for memory-intensive workloads. Our contributions are as follows: • We reveal the limitations of monolithic object-level UVM prefetching in modern DL frameworks and introduce AUTO UVM, the first automated, policy-driven UVM prefetching system with tensor-level visibility. • We develop a profile-driven analysis framework that performs a single offline pass to trace kernel–tensor accesses, generate stable tensor identifiers, and infer tensor affinity, enabling framework-aware prefetch planning without modifying model code. We also design a lightweight Online Executor that autonomously issues and adapts tensor-level prefetching decisions using a lightweight runtime monitor. • We instantiate AUTO UVM with a roofline-inspired policy to identify performance-critical data transfers. Across ten LLMs, AUTO UVM achieves an average 3.1× speedup over baseline UVM, outperforms the best-performing prior prefetcher (DeepUM) by 1.9×, and improves over objectlevel prefetchers by up to 4.7×, while significantly reducing page faults. II. BACKGROUND A. Unified Virtual Memory Modern heterogeneous systems with discrete CPUs and GPUs maintain separate physical memories connected through CPU-GPU interconnect (e.g., PCIe, NVLink), traditionally requiring explicit allocation and copy operations. To simplify programming, NVIDIA introduced Unified Virtual Memory (UVM) in CUDA 6.0 [14], enabling CPUs and GPUs to share data through a unified pointer. Pascal architecture [15]

Tensor 1

…

Tensor 2

…

Unused

Pool 1

… Allocation n

Allocation n

Tensor 3 Used

Allocation 1

n=2

Pool n

Fig. 1: Memory Management in the PyTorch Framework extended UVM with on-demand page migration and memory oversubscription. With cudaMallocManaged, data pages migrate to GPU memory on first access; if a nonresident page is referenced, a page fault triggers the UVM driver and GPU Memory Management Unit (GMMU) to allocate device space and copy the page from host to device. Although UVM removes explicit data transfers, page-fault handling introduces substantial latency. NVIDIA provides advisory APIs such as cudaMemPrefetchAsync for proactive migration and cudaMemAdvise for preferred placement. UVM offers a flat virtual address space shared by CPU and GPU and supports memory oversubscription, where CPU memory backs allocations exceeding GPU capacity. When GPU memory is full, the UVM driver evicts least recently used (LRU) pages to host memory. Oversubscription severity is expressed by the oversubscription factor, the ratio of allocated UVM memory to GPU capacity [16]. A factor ≤ 1.0 indicates no oversubscription, while > 1.0 requires frequent evictions; for instance, a factor of 1.5 on an 80 GB GPU corresponds to 120 GB of managed memory, causing continuous page movement between CPU and GPU. B. Memory Management of Modern DL Frameworks DL workloads would suffer substantial overhead if each tensor is individually managed via costly cudaMalloc and cudaFree APIs, given the increasing DL model sizes. To avoid this, PyTorch [17] employs the PyTorch Caching Allocator (PCA) [8], which uses a pooling strategy to amortize allocation costs. Instead of allocating memory for each tensor, PCA obtains large contiguous blocks via cudaMalloc and subdivides them for tensor use. Unused segments are retained in pools for reuse, and freed tensors return their space to the pool rather than triggering cudaFree. After a brief warm-up phase, most tensor allocations are served from cached blocks. Figure 1 illustrates this design. PyTorch maintains separate large and small pools, each consisting of multiple preallocated blocks. Within each block, many tensors coexist and are tracked by a framework-level metadata structure (e.g., a doubly linked list) that records allocation boundaries and reuse opportunities. These behaviors occur entirely inside the framework and remain invisible to the GPU runtime, which sees only a small number of large allocations. Consequently, low-level allocators and profilers have limited visibility into the fine-grained tensor reuse and pooling dynamics managed by PCA.

UVM-enabled 561.8 19.7

16.9 Memory capacity (12 GB)

10 4.1

6.0

100

10.1

7.8

150

10k

12k

Memory Size (GB)

UVM-disabled

Execution time (s)

Memory size (GB)

Memory usage

103

Fig. 3: Breakdown of used and unused prefetch sizes for object-level UVM prefetching in PyTorch, when the prefetcher is unaware of PyTorch’s internal memory management (log scale).

III. M OTIVATION

GPU Peak FLOPS

A. High Memory Demand Calls for UVM

B. Limitations of Existing UVM Prefetching Existing UVM prefetching techniques lack visibility into framework-level tensor-management semantics, making prefetching decisions using only UVM-level information [5]– [7], [20]. Object-level approaches [5], [6] further operate at the granularity of cudaMallocManaged allocations, assuming that all data within a managed object exhibits similar access behavior. However, this assumption breaks down for large LLM workloads running on modern deep learning frameworks such as PyTorch. A single managed allocation often contains many tensors with heterogeneous access patterns and lifetimes. Without visibility into tensor-level behavior, existing prefetchers fail to capture the fine-grained memory needs of individual kernels, resulting in inefficient migration. To quantify this inefficiency, we implement a naive objectlevel strategy that prefetches entire managed objects immediately before each kernel launch. Figure 3 shows the breakdown of used versus unused prefetched data across several LLM models (detailed in Table II). On average, useless prefetch

GPU Mem Bdw

104

20 19

Performance (GFLOPS)

With its support for memory oversubscription, UVM is the only developer-agnostic mechanism on NVIDIA GPUs that enables large workloads to run on devices with limited memory capacity, despite its inherent overheads. As the backend engine of emerging memory-management interfaces, such as Heterogeneous Memory Management (HMM) [18] and zerocopy transfer in Grace–Hopper architectures [19], UVM is one of the most essential technologies to provide flexible and transparent CPU–GPU data movement. Figure 2 shows this benefit for Qwen1.5 inference on an RTX 3060. Without UVM, PyTorch fails once sequence length exceeds 10K, resulting in an out-of-memory (OOM) error that would normally require multiple GPUs. With UVM enabled, the same GPU can execute significantly longer sequences, even when memory usage exceeds its 12 GB device capacity. As LLM sizes continue to grow and high-end GPUs remain expensive, UVM offers a practical path for enabling largemodel workloads on commodity hardware. However, UVM performance can degrade sharply under oversubscription due to the substantial cost of page-faultdriven on-demand migrations. Designing an effective prefetching strategy is therefore essential to fully unlocking UVM’s potential.

. k S M -L 2.5 n1.5 see lama lama H-1 -LLM Geo M- SLM en e p -L E L SL Qw Qw Dee A

M-

Sequence length

Fig. 2: Qwen1.5 performance with and without UVM and memory usage across varying sequence lengths.

Used Size Unused Size

104

16 2 11 9

8 14

101 100

10−1 10−2

100

5 15 17 6

CPU-GPU Interconnect (CGI) Bdw ID: Kernel (Call Frequency) 1: scalar_elementwise_mul (29049) 2: gemv_var5 (19272) 3: vec_add_float (19152) 4: vec_elementwise_mul (14551) 5: reduce_mean (9800) 6: vec_add_long (9800) 7: vec_elementwise_pow (9800) 8: cat_batch_copy (9648) 9: elementwise_copy (9624) 10: elementwise_neg (9600) 11: gemv_var8 (9552) 12: gemv_var6 (9384) 13: elementwise_add (4848) 14: softmax_warp_forward (2208) 15: vec_fill_long (399) 16: gemv2T_val (199) 17: vec_compare_long (199) 18: ampere_sgemm_32 (169) 19: ampere_sgemm_128tn (24) 20: ampere_sgemm_128nn (24)

101 104 102 103 Arithmetic Intensity (FLOPs / Byte)

105

Fig. 4: Roofline visualization of Qwen1.5 running on NVIDIA RTX 3060. The figure highlights the top 20 kernels, which dominate performance and together contribute to more than 90% of overall runtime. is 6× (up to 23×) more than useful prefetch. Such useless prefetches waste both CGI bandwidth and scarce GPU memory capacity, illustrating that coarse-grained, applicationagnostic prefetching is ill-suited to the fine-grained memory access patterns of LLM workloads. C. Heterogeneous Migration Sensitivity Across Kernels Different kernels exhibit different sensitivity to CPU–GPU migration latency. Memory-bound kernels rely heavily on fast data access, whereas compute-bound kernels are typically more tolerant. LLM workloads contain many such kernel types, often invoked thousands of times during inference. Figure 4 shows this heterogeneity using a multi-level roofline analysis of Qwen1.5 on an NVIDIA RTX 3060. The plot includes the GPU compute roof, GPU memory-bandwidth roof, and the significantly lower CPU–GPU interconnect (CGI) roof that reflects transfer limits over PCIe. Elementwise kernels cluster in the low-intensity memory-bound region, while GEMV/GEMM kernels approach the compute roof. Several operators fall near or below the CGI roofline, indicating substantial throughput degradation once tensor pages migrate between CPU and GPU memory via UVM. This observation highlights that blindly prefetching all tensors is inefficient; prefetching should instead prioritize kernels most sensitive to interconnect bottlenecks. Note that this finding applies even to the emerging GPUs that have significantly higher CGI bandwidth, because GPU memory bandwidth (e.g., 4 TBps in

Offline Profiler

Profiling Run

Execution Tracer • Tensor Stats • Kernel Stats • UVM Stats

• Memory Affinity • Reuse/Dependency • Prefetch Policy

Profiles Generation

•••

Execution Profiles

Prefetch Engine • Automated Prefetch • Policy-Driven Prefetch • Fine-grained Prefetch

LLM Models

•••

UVM Analyzer

Online Executor

Real Run

Guide Prefetch Update Stats

Runtime Monitor • Performance Monitor • Runtime Adaptation

Fig. 5: Design of AUTO UVM.

Hopper GPU) is typically significantly higher than the CGI’s (e.g., 900 GBps in NVlink). IV. D ESIGN OF AUTO UVM A. Design Goals Motivated by the observations in Section III, AUTO UVM aims to bridge the semantic gap between DL frameworks and the UVM driver to enable efficient execution under memory oversubscription. The design of AUTO UVM is guided by four goals: (1) Tensor awareness. Prefetching should operate at tensor granularity rather than coarse cudaMallocManaged objects. (2) Policy-driven selectivity. Prefetching should target kernels most sensitive to CPU–GPU migration overhead. (3) Runtime adaptivity. Prefetch decisions should adapt to changing memory pressure and eviction behavior. (4) Framework transparency. The system should require no model-code modifications and integrate seamlessly with DL frameworks such as PyTorch. B. Key Ideas The core insight behind AUTO UVM is that effective UVM prefetching requires both tensor-level visibility and selective migration. Rather than assuming all kernels benefit equally from proactive migration, AUTO UVM applies policy-driven prefetching based on workload and runtime characteristics. In this work, we instantiate AUTO UVM with a rooflineinspired policy that prioritizes kernels bottlenecked by CPU– GPU interconnect (CGI) transfers while avoiding unnecessary migration for compute-bound and GPU-memory-bound kernels, as discussed in Section III-C. Prefetching at the cudaMallocManaged object level is too coarse because framework allocators pack many unrelated tensors into large shared regions, causing conventional prefetching to migrate substantial unused data. AUTO UVM addresses this by identifying the exact tensors each kernel accesses and migrating only those. To ensure these profiles remain valid across runs, AUTO UVM assigns each tensor a stable identity derived from its creation-site call stack and a global allocation counter, making it robust to address randomization by the OS or framework. These ideas collectively form the foundation of AUTO UVM’s design. C. System Overview Figure 5 presents the architecture of AUTO UVM, which consists of an Offline Profiler, an Online Executor, and an

Execution Profile Database. The Offline Profiler observes a single profiling run to capture kernel–tensor access patterns, collect runtime statistics, and infer tensor affinity across consecutive kernels. The resulting execution profiles encode kernel behaviors, kernel–tensor mappings, and metadata required to identify tensors across executions. During inference, the Online Executor loads the execution profiles and issues policy-driven prefetch operations before kernel launches. As oversubscription pressure changes during execution, the Online Executor continuously monitors runtime behavior and dynamically adjusts prefetching decisions. This adaptive mechanism ensures that prefetching is applied only when beneficial, even when runtime evictions cause execution behavior to deviate from the offline profile. D. Offline Profiler 1) Execution Tracer: Profiling-based memory optimization for LLMs faces two challenges that prevent naive reuse of profiling results: 1) Dynamic computation graphs. Frameworks like PyTorch construct graphs eagerly, causing operator sequences to differ across runs, making static prefetch schedules unreliable. 2) Address randomization. Tensor addresses change between profiling and execution, invalidating pointer-based profiles. To address these issues, we employ two techniques: Kernel–Tensor Access Tracing. Instead of relying on the computation graph, the Execution Tracer records kernel–tensor access pairs. During execution, the Online Executor simply prefetches the tensors needed by the next kernel. Two-Level Tensor Identifier. To track tensors across runs, AUTO UVM assigns each tensor a stable identity derived from its creation-site call stack and a monotonic allocation counter. PyTorch’s caching allocator is instrumented with a lightweight hook to record this information. If a mismatch occurs at runtime, AUTO UVM disables prefetch for the affected tensors to ensure correctness, though such cases are rare for fixedarchitecture LLMs. 2) UVM Analyzer: The UVM Analyzer analyzes kernel behavior and runtime statistics to guide policy-driven prefetching. In this work, we instantiate the analyzer with a rooflineinspired policy that classifies kernels into GPU-computebound, GPU-memory-bound, and CGI-bound categories. Since kernels launched from the same call stack and configuration typically exhibit similar behavior, AUTO UVM applies a unified prefetch strategy across such instances. In addition, the Analyzer performs look-ahead tensor affinity analysis to identify tensors reused across consecutive kernels. Tensors with high short-term reuse are marked for GPU residency using cudaMemAdvise, reducing migration overhead during oversubscription. 3) Execution Profiles: The Offline Profiler consolidates collected information into compact execution profiles that map kernels to accessed tensors. Kernels are identified using launch-site call-stack hashes and launch counters, while tensors use creation-site call-stack hashes and allocation counters. The profiles additionally store runtime metadata, policy

Profiling Run LLM Model Execution

Extract Info 1

TABLE I: Hardware and Software Environment.

Offline Profiler Execution Tracer

Analyze 2

UVM Analyzer

Actual Run

Online Executor Guide Prefetch Prefetch Engine 2

LLM Model Execution

Update 4 Roofline 3

Monitor Perf.

Roofline Monitor

Store Profiles

Execution Profiles

Read Profiles x

Offline Steps

Online Steps

Fig. 6: Workflow of AUTO UVM. information, and tensor-affinity hints used to guide subsequent prefetching and pinning decisions during real execution. E. Online Executor 1) Prefetch Engine: Unlike prior UVM techniques that operate at the granularity of entire managed objects, the Prefetch Engine performs fine-grained tensor-level prefetching, migrating only the tensors required by upcoming kernels. This selective approach reduces unnecessary data movement and improves GPU memory utilization under oversubscription. Prefetch decisions are guided by the selected policy. In our roofline-inspired policy, kernels bottlenecked by CPU– GPU interconnect (CGI) transfers trigger proactive prefetching to hide migration latency, while compute-bound and GPUmemory-bound kernels avoid unnecessary migration to conserve bandwidth and reduce memory pressure. For tensors reused across consecutive kernels, the Prefetch Engine uses cudaMemAdvise to pin them on the GPU during their reuse window. The runtime mechanism relies only on lightweight hooks on kernel launches and tensor allocations, keeping runtime overhead minimal. 2) Runtime Monitor: Under oversubscription, runtime evictions may alter kernel behavior and reduce the effectiveness of static prefetch decisions. To remain adaptive, the Runtime Monitor continuously tracks kernel performance during execution and updates policy decisions accordingly. In our roofline-inspired policy, the monitor dynamically re-evaluates kernel classifications based on updated runtime measurements (e.g., kernel time). When a classification shift is detected, the Prefetch Engine immediately adjusts its migration decisions to match current memory pressure. The Runtime Monitor introduces only lightweight timestamping overhead, while policy updates execute in parallel with subsequent kernels, resulting in negligible runtime cost. F. Workflow Figure 6 shows the two-phase workflow of AUTO UVM. During the profiling phase, the Execution Tracer extracts kernel–tensor access relationships from an LLM execution ( 1 ), and the UVM Analyzer analyzes runtime statistics to generate execution profiles ( 2 ). The generated profiles are then stored in the Profile Database for reuse across executions ( 3 ). During online execution, the Online Executor loads the execution profiles at startup ( 1 ) and uses them to guide

System Memory

GPU Driver

GPU Toolkit

Linux 6.11

32 GB

560.28.03

CUDA 12.1

Linux 6.16

128 GB

580.65.06

CUDA 12.1

Linux 5.15

560 GB

560.35.05

CUDA 12.4

CPU

GPU

System

AMD Ryzen 7 5800X Intel Xeon Gold 5320 AMD PYC 9454

NVIDIA GeForce RTX 3060 NVIDIA A100 NVIDIA H100 NVL

TABLE II: Evaluated LLM models. LLM Models

Transformer Layers

Sequence Length

Memory Footprint

Abbr.

SmolLM2-S [24] SmolLM2-M [24] SmolLM2-L [24] Qwen2.5 [25] Qwen1.5 [26] Deepseek-coder [27] AMD-Llama [28] Llama [29] Helium-1 [30] EuroLLM [31]

30 32 24 24 24 24 12 12 28 24

4.0k 4.0k 4.0k 4.0k 4.0k 4.0k 1.9k 1.9k 3.9k 3.9k

3.6GB 4.8GB 11.4GB 6.9GB 11.2GB 10.2GB 1.4GB 1.5GB 10.8GB 11GB

SLM-S SLM-M SLM-L Qwen2.5 Qwen1.5 Deepseek A-Llama Llama H-1 E-LLM

tensor prefetching before kernel launches ( 2 ). As execution progresses, the Runtime Monitor continuously tracks kernel performance ( 3 ) and updates runtime roofline information under changing memory pressure. The Prefetch Engine then dynamically adjusts prefetching decisions based on the updated runtime behavior ( 4 ). G. System Implementation Our implementation consists of a UVM profiler and a prefetch engine. AUTO UVM is integrated into PyTorch as a transparent extension that requires no changes to model code. We use tensor-creation hooks [21] in PyTorch’s CUDA caching allocator to record call stacks, tensor sizes, and allocation counters. NVIDIA NVBit [22] is used to count FLOP instructions and memory loads. NVIDIA Compute Sanitizer APIs [23] are used to inject prefetch and affinity primitives with minimal overhead. To enable UVM semantics, we modify PyTorch’s caching allocator to use cudaMallocManaged in place of cudaMalloc, ensuring full UVM compatibility. We evaluate the power and energy impact of AUTO UVM using a lightweight Python measurement tool built on NVML that samples GPU power sensors and accumulates total energy during execution. V. E VALUATION A. Methodology We evaluated AUTO UVM on three GPU platforms, as summarized in Table I. We evaluate AUTO UVM across 10 widely used LLMs from Hugging Face, listed in Table II. To precisely control the degree of UVM oversubscription, we constrain the effective GPU memory capacity by allocating a fixed amount of device memory in a separate helper process, following standard practice in prior work [9], [32]. We measure the number of page faults by adding a counter to the NVIDIA UVM driver module [33]. 1) Compared solutions: We compare AUTO UVM against a baseline and three UVM prefetching approaches: SUV and Forest, which operate at the managed-object level, and DeepUM, which operates at the UM-block level.

SUV

Forest

DeepUM

AutoUVM

2 1 0

M SL

-S

M SL

-M

-L

M SL

k 2.5 n1.5 see lama lama H-1 e p L L AQw Dee

n we

L E-L

Number of Faults

Fig. 7: Execution time normalized to Baseline under memory oversubscription of 2.0 (lower is better). Baseline

SUV

Forest

DeepUM

AutoUVM

106 105

. L k S ma ma H-1 -LLM Geo 2.5 1.5 ee M- LM-M LMen wen eps -Lla Lla E S S A Qw Q De

Fig. 8: Number of page faults under memory oversubscription of 2.0 (lower is better). • Baseline uses UVM without explicit prefetching. When a kernel accesses data not resident on the GPU, a page fault triggers on-demand migration, reflecting default UVM behavior. • SUV [6] optimizes UVM at the object level by estimating each managed object’s access density (GPU accesses per byte) via static analysis and pinning the object to host or device memory accordingly. We compute access density from memory-access traces collected by AUTO UVM’s profiler. • Forest [5] uses hardware counters to detect per-object access patterns and adjusts prefetch granularity. We emulate Forest using offline access-pattern analysis and vary granularity from 256 KB to 4 MB, matching its original setup. Because our pattern detection is offline, it avoids Forest’s online detection overhead. • DeepUM [20] uses correlation-based prefetching to predict future page accesses from historical fault patterns. Though effective for stable, repetitive access behavior, it operates at the coarse UM-block level, lacks framework-level semantics, and aggressively prefetches all correlated pages regardless of whether the movement is performance-critical. B. Overall Performance Figure 7 reports execution time normalized to the Baseline under an oversubscription factor of 2.0 (lower is better). AUTO UVM achieves average speedups of 3.1×, 4.7×, 4.1×, and 1.9× over Baseline, SUV, Forest, and DeepUM, respectively. Both object-level approaches (SUV and Forest) often perform worse than the Baseline, showing that tensor-unaware prefetching is ineffective for LLM workloads on modern deep-learning frameworks. As discussed in Section III-B, object-level prefetching migrates entire UVM objects—including many unused tensors—which wastes CGI bandwidth, increases GPU-memory pressure, and triggers severe page thrashing. Between the two, Forest outperforms SUV by reducing prefetch granularity for irregular access

Execution Time

Execution time

Baseline

Factor = 1.5

Factor = 1.75

Factor = 2.0

0.5

0.0

M-

. k ma ma H-1 -LLM Geo 2.5 1.5 ee en wen eps -Lla Lla E A e Q D

M-

Fig. 9: Execution time of AUTO UVM normalized to Baseline under different oversubscription factor. patterns, limiting unnecessary migration. DeepUM is the bestperforming baseline, as its finer UM-block granularity and UVM-aware correlation prefetching consistently outperform the Baseline. However, it lacks framework-level semantics and prefetches correlated blocks regardless of whether the transfer is performance-critical, resulting in more data movement than AUTO UVM. By prefetching only the tensors each kernel actually accesses, AUTO UVM migrates far fewer bytes and achieves the best results. C. Page Fault We measure the total number of page faults, as shown in Figure 8. A strong correlation emerges between execution time and the number of page faults across all models and methods. AUTO UVM issues the fewest page faults among the evaluated solutions, due to its policy-driven, tensor-level prefetching strategy, which allocates limited CGI bandwidth and GPU memory only to tensors that actually need them. By avoiding unnecessary migrations, AUTO UVM minimizes thrashing and significantly improves runtime performance. D. Sensitivity 1) Sensitivity of Oversubscription Levels: We further evaluate the impact of different oversubscription factors. As shown in Figure 9, AUTO UVM achieves average speedups of 1.4×, 2.2×, and 3.1× under oversubscription factors of 1.5, 1.75, and 2.0, respectively. The gains increase as GPU memory becomes more constrained, indicating that AUTO UVM is most effective when UVM paging is most severe. This trend underscores AUTO UVM’s value as LLM memory demands grow and oversubscription becomes increasingly common. 2) Sensitivity to GPU Platforms: Figure 10 evaluates AUTO UVM across RTX 3060, A100, and H100 GPUs. AUTO UVM consistently achieves substantial speedups of 3.1×, 2.9×, and 2.7×, respectively. Although the gains decrease slightly on newer GPUs with higher interconnect bandwidth and larger memory capacity, AUTO UVM remains effective across diverse GPU architectures. E. Ablation Study To evaluate the contribution of each component, we compare three variants of AUTO UVM against the baseline: T, which applies tensor-level prefetching; T+A, which adds lookahead tensor affinity; and T+A+P, the full design with policydriven selectivity. As shown in Figure 11, T delivers a 2.0× speedup by avoiding unnecessary object-level migration.

A100

Baseline

H100

Energy (J) (log scale)

Execution Time

RTX 3060 0.50 0.25 0.00

-S

M SL

. k ma ma H-1 -LLM Geo 2.5 1.5 ee en wen eps -Lla Lla E A Qw Q De

102

Fig. 10: Execution time of AUTO UVM normalized to Baseline under different GPU platform.

Forest

AutoUVM

L k S 2.5 1.5 see lama lama H-1 M- LM-M LMen wen S L S ep A-L Qw Q De

-L

M SL

SUV

. LM Geo

E-L

Fig. 12: Total energy consumption across models (log scale). 4

Adding affinity (T+A) increases the average speedup to 2.5× by keeping highly reused tensors on the GPU. The full system (T+A+P) achieves the best results at 3.1×, showing that combining tensor-level visibility, affinity analysis, and policydriven prefetching yields the highest performance. F. Energy and Power Figure 12 shows total energy consumption across all models (log scale). AUTO UVM leaves average GPU power essentially unchanged (a 3–4W increase), as it does not alter kernels, GPU frequency, or computation. Yet it cuts total energy by 31% on average over baseline UVM and up to 74% over object-level prefetchers, as shorter execution time due to fewer page faults and unnecessary transfers outweighs the small power increase. G. Profiling Overhead Figure 13 reports the cost of AUTO UVM’s offline profiling pass, normalized to a single inference iteration (41.5× on average). This is a one-time, offline cost: the profile is generated once and reused across all subsequent inference runs, so it is fully amortized and does not affect steady-state performance. H. Generalization to Fused Kernels To verify that AUTO UVM generalizes beyond eager-mode execution, we evaluate it with kernel fusion enabled, where multiple operators are consolidated into single kernels with coarser tensor-access patterns. We apply torch.compile to all evaluated models, letting the PyTorch Inductor backend fuse elementwise, normalization, and attention operations, and regenerate AUTO UVM’s offline profiles against the fused execution. As shown in Figure 14, AUTO UVM achieves an average 2.3× speedup over baseline UVM under an oversubscription factor of 2.0, confirming that its tensor-level prefetching remains effective under the more complex access patterns introduced by kernel fusion.

Geo.

H-1

E-LLM

Llama

A-Llama

Qwen1.5

Deepseek

Geo.

H-1

E-LLM

Llama

A-Llama

Qwen1.5

Deepseek

Fig. 11: Ablation study of AUTO UVM. T: tensor-level prefetching; T+A: adding look-ahead tensor affinity; T+A+P: full design with policy-driven selectivity (higher is better).

SLM-L

. k 2.5 n1.5 see lama lama H-1 -LLM Geo e p E L L AQw Dee

Qwen2.5

M-

SLM-M

M-

SLM-S

M-

SLM-L

0.0

Qwen2.5

1.0

SLM-M

2.5

Speedup

5.0

SLM-S

T T+A T+A+P

Overhead

Speedup

Fig. 13: Profiling overhead Fig. 14: Performance with across models. fused kernels. VI. R ELATED W ORK Profile-Guided and Runtime-Adaptive Prefetching. DL workloads execute iteratively, enabling offline profiling to capture stable access patterns for later execution. Capuchin [10] profiles one iteration to guide prefetching, eviction, and recomputation, while TERAIO [11] profiles tensor lifetimes to plan fine-grained offloading. Neither targets UVM prefetching. On the UVM path, Forest [5] combines compile-time pattern detection with runtime access classification but relies on hardware access counters. These systems require source instrumentation or hardware support, and their runtime feedback adjusts prefetch aggressiveness rather than transfer necessity. In contrast, AUTO UVM is software-only, decoupling offline kernel–tensor profiling from a lightweight online monitor and enabling tensor-level prefetching through transparent allocator hooks without model-code changes. UVM Prefetching. Prior work improves UVM through prefetching and data placement. SUV [6] and Forest [5] operate at the granularity of cudaMallocManaged allocations, while DeepUM [20] uses correlation-based prediction to prefetch future pages from historical fault patterns. However, these approaches lack framework-level tensor-management semantics exposed by modern DL frameworks. SUV and Forest further assume uniform access behavior within each managed object, using static [6] or hardware-counter–driven [5] accesspattern analysis. None consider the interaction between UVM and framework-level memory management. Ganguly et al. [34] improve the UVM built-in prefetcher using access locality and intensity, while EarlyAdaptor [35] adapts prefetching based on page-fault history. These methods target general workloads and are orthogonal to software UVM primitives such as cudaMemPrefetchAsync and cudaMemAdvise. To the best of our knowledge, AUTO UVM is the first automated prefetching system that incorporates modern DL-framework semantics to enable programmer-agnostic software prefetching.

VII. C ONCLUSION We presented AUTO UVM, an automated, framework-aware UVM prefetching system for oversubscribed LLM execution. AUTO UVM exposes fine-grained tensor behavior to UVM and migrates only performance-critical data, guided by a single offline profile and lightweight online runtime monitoring, without modifying model code. Across 10 LLMs, AUTO UVM achieves an average 3.1× speedup over baseline UVM, outperforms the best-performing prior prefetcher (DeepUM) by 1.9×, and improves over object-level prefetchers by up to 4.7×, while reducing page faults and energy. AUTO UVM offers a practical, programmer-agnostic path for memoryintensive LLM workloads on commodity GPUs. ACKNOWLEDGEMENTS This work was supported by NSF grants, 2541979, CCF2452081, and CAREER-2341039. R EFERENCES [1] Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri et al., “Pytorch fsdp: experiences on scaling fully sharded data parallel,” arXiv preprint arXiv:2304.11277, 2023. [2] M. Lin, K. Zhou, and P. Su, “Drgpum: Guiding memory optimization for gpu-accelerated applications,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2023, pp. 164–178. [3] S. G. Patil, P. Jain, P. Dutta, I. Stoica, and J. Gonzalez, “Poet: Training neural networks on tiny devices with integrated rematerialization and paging,” in International Conference on Machine Learning. PMLR, 2022, pp. 17 573–17 583. [4] V. Young, S. Kariyappa, and M. K. Qureshi, “Enabling transparent memory-compression for commodity memory systems,” in 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2019, pp. 570–581. [5] M. Lin, Y. Feng, G. Cox, and H. Jeon, “Forest: Access-aware gpu uvm management,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 137–152. [6] B. Pratheek, G. Cox, J. Vesely, and A. Basu, “Suv: Static analysis guided unified virtual memory,” in 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2024, pp. 293–308. [7] D. Ganguly, Z. Zhang, J. Yang, and R. Melhem, “Interplay between hardware prefetcher and page eviction policy in cpu-gpu unified virtual memory,” in Proceedings of the 46th International Symposium on Computer Architecture, 2019, pp. 224–235. [8] P. O.-S. Community., “Pytorch cuda caching memory allocator,” Accessed February 2025, https://github.com/pytorch/pytorch/blob/main/ c10/cuda/CUDACachingAllocator.cpp. [9] M. Lin and H. Jeon, “Understanding oversubscribed memory management for deep learning training,” in Proceedings of the 5th Workshop on Machine Learning and Systems, 2025, pp. 46–55. [10] X. Peng, X. Shi, H. Dai, H. Jin, W. Ma, Q. Xiong, F. Yang, and X. Qian, “Capuchin: Tensor-based gpu memory management for deep learning,” in Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, 2020, pp. 891–905. [11] Z. Yuan, H. Zhang, Y. Zhou, A. Mohan, I. Chung, S. Seelam, J. Huang et al., “Cost-efficient llm training with lifetime-aware tensor offloading via gpudirect storage,” Advances in Neural Information Processing Systems, vol. 38, pp. 49 421–49 444, 2026. [12] M. Lin, H. Jeon, and K. Zhou, “Pasta: A modular program analysis tool framework for accelerators,” in 2026 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). IEEE, 2026, pp. 520–534. [13] M. Lin, X. Wang, G. Cox, D. Li, and H. Jeon, “Hybridgen: Efficient llm generative inference via cpu-gpu hybrid computing,” arXiv preprint arXiv:2604.18529, 2026. [14] N. Corporation, “Unified memory in cuda 6,” Accessed February 2025, https://developer.nvidia.com/blog/unified-memory-in-cuda-6/.

[15] N. Corporation, “Nvidia tesla p100 whitepaper,” Accessed February 2025, https://images.nvidia.com/content/pdf/tesla/whitepaper/ pascal-architecture-whitepaper.pdf. [16] N. Corporation, “Improving gpu memory oversubscription performance,” Accessed February 2025, https://developer.nvidia.com/blog/ improving-gpu-memory-oversubscription-performance/. [17] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019. [18] T. linux kernel development community., “Heterogeneous memory management (hmm),” Accessed February 2025, https://www.kernel.org/doc/ html/latest/mm/hmm.html. [19] N. Corporation, “Nvidia grace hopper superchip architecture whitepaper,” Accessed February 2025, https://resources.nvidia.com/ en-us-grace-cpu/nvidia-grace-hopper. [20] J. Jung, J. Kim, and J. Lee, “Deepum: Tensor migration and prefetching in unified memory,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2023, pp. 207–221. [21] P. Community, “Pytorch tensor creation hook,” Accessed November 2025, https://github.com/pytorch/pytorch/blob/release/2.9/c10/core/ Allocator.h#L302-L307. [22] O. Villa, M. Stephenson, D. Nellans, and S. W. Keckler, “Nvbit: A dynamic binary instrumentation framework for nvidia gpus,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 2019, pp. 372–383. [23] N. Corporation, “Compute sanitizer api reference manual,” https://docs. nvidia.com/compute-sanitizer/SanitizerApiGuide/index.html, Accessed: April 2025. [24] L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafioti, H. Kydlı́ček, A. P. Lajarı́n, V. Srivastav et al., “Smollm2: When smol goes big–data-centric training of a small language model,” 2025. [25] Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/ [26] Q. Team, “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023. [27] D. Team, “Deepseek-coder-1.3b model,” https://huggingface.co/ deepseek-ai/deepseek-coder-1.3b-base, accessed: 2024. [28] A. Corporation, “Amd-llama-135m model,” https://huggingface.co/amd/ AMD-Llama-135m, accessed: 2023. [29] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, R. Y. Y. Wong, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia, “Specinfer: Accelerating generative llm serving with speculative inference and token tree verification,” p. 4, 2023. [30] K. A. R. Lab, “Helium-1-2b model,” https://huggingface.co/kyutai/ helium-1-2b, accessed: 2025. [31] P. H. Martins, P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski et al., “Eurollm: Multilingual language models for europe,” Procedia Computer Science, vol. 255, pp. 53–62, 2025. [32] D. Ganguly, Z. Zhang, J. Yang, and R. Melhem, “Adaptive page migration for irregular data-intensive applications under gpu memory oversubscription,” in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2020, pp. 451–461. [33] N. Corporation, “Nvidia linux open gpu kernel module source,” Accessed June 2024, https://github.com/NVIDIA/ open-gpu-kernel-modules. [34] D. Ganguly, Z. Zhang, J. Yang, and R. Melhem, “Interplay between hardware prefetcher and page eviction policy in cpu-gpu unified virtual memory,” in Proceedings of the 46th International Symposium on Computer Architecture, 2019, pp. 224–235. [35] S. Go, H. Lee, J. Kim, J. Lee, M. K. Yoon, and W. W. Ro, “Earlyadaptor: An adaptive framework forproactive uvm memory management,” in 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2023, pp. 248–258.

AutoUVM: Automated Prefetching Framework for LLMs under UVM Oversubscription

Analyze & explore related research in Conceptio

Related documents