Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices Yangyijian Liu
Hongyi Ye
School of Computer Science, Nanjing University Nanjing, China [email protected]
School of Computer Science, Nanjing University Nanjing, China [email protected]
Mingyang Li
Wu-Jun Li
School of Computer Science, Nanjing University Nanjing, China [email protected]
School of Computer Science, Nanjing University Nanjing, China [email protected]
arXiv:2607.10183v1 [cs.DC] 11 Jul 2026
Abstract Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94× and decode throughput by up to 3.29×, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.
1
Introduction
Large language models (LLMs) have rapidly become a foundation for language understanding, code generation, question answering, and interactive assistants [1, 2, 22, 37]. As demand grows for privacy, offline availability, and predictable latency, deployment is increasingly moving from cloud servers to consumer devices such as laptops and desktops. This shift changes the optimization target. Unlike server-side LLM serving, which often relies on many concurrent requests and large batches to maximize throughput, local deployment usually runs at low concurrency, making per-request responsiveness the most important performance metric. Deploying modern LLMs on consumer devices remains difficult because model weights often exceed GPU memory
GPU Compute Layer
PCIe Transfer
CPU Compute Layer
GPU-centric Offloading
0
1
2
Hybrid CPU-GPU Inference
0
1
2
3 3
4
4
Timeline
Figure 1. Execution timelines for GPU-centric offloading and hybrid CPU-GPU inference on consumer devices. Hybrid inference replaces long PCIe transfers of model weights with shorter transfers of activations, but CPU execution remains substantially slower than GPU execution and becomes the new bottleneck.
capacity even after quantization. Offloading [31] is therefore unavoidable for practically useful models. A common approach is GPU-centric offloading, in which the GPU remains the primary compute device and model weights are moved across a storage hierarchy on demand [13, 20, 29, 31, 38]. However, this design is fundamentally constrained by the bandwidth mismatch between GPU memory and PCIe: consumer GPUs typically provide on-device memory bandwidth in the hundreds to over a thousand GB/s, whereas PCIe 4.0 provides at most 32 GB/s of host-device bandwidth. Once weights must be fetched repeatedly from host memory, PCIe transfer becomes the bottleneck and the GPU is difficult to keep fully utilized [33]. This bottleneck has motivated hybrid CPU-GPU inference, in which the CPU participates directly in model computation instead of treating it only as a source of transferred weights [4–6, 8, 24, 32, 33, 36, 39–41]. Hybrid execution reduces the need to move every tensor to the GPU and expands the design space beyond transfer-computation overlap alone. However, it introduces a new bottleneck: CPU execution can dominate end-to-end latency, especially on consumer devices where CPU and GPU performance is highly asymmetric
on massively parallel operators such as matrix multiplication [6, 33]. Once a substantial fraction of the model remains on the CPU, long CPU execution leaves the GPU idle for extended periods and sharply reduces the benefit of offloading [8, 24]. Figure 1 illustrates how the primary bottleneck shifts from PCIe transfer to CPU execution. To relieve CPU-side bottlenecks under tight GPU memory constraints, we focus on two limitations of existing hybrid inference systems.
GPU
VRAM
DRAM SMs Copy
Pinned
Engine
• Coarse-grained placement. Existing hybrid systems typically place weights at layer granularity [8, 24, 29, 31], or at expert granularity [5, 6, 19, 32, 36, 39–41] for Mixture-of-Experts (MoE) models [2, 3, 17, 22, 34, 37]. This granularity leaves substantial intra-layer heterogeneity unexplored. • Load-unaware scheduling. Existing runtime policies are either fixed [6, 24, 29, 31] or adapt only to input-dependent signals such as sequence length or expert activation [5, 32, 36, 40]. However, they do not respond to changing device conditions during execution.
Memory
PCIe
Figure 2. Two host-to-device transfer paths on NVIDIA GPUs. Besides the hardware Copy Engine path, mapped and pinned host memory also allows an SM-driven copy kernel to fetch data through Zero-Copy.
2
Background
2.1 CPU and GPU Architecture for Hybrid Inference
To address these challenges, we present ATSInfer, a tensorgranularity hybrid CPU-GPU inference system with loadaware dynamic transfer. ATSInfer combines three complementary mechanisms: asynchronous CPU-GPU scheduling to coordinate storage, data movement, and computation across heterogeneous backends; static tensor placement to determine GPU residency under memory and switching-cost constraints; and load-aware dynamic transfer to adjust runtime tensor movement according to inference phase and backend load. Together, these three components improve hybrid CPUGPU inference for local low-concurrency workloads, where trade-offs among residency, transfer, and execution directly determine latency. In summary, this paper makes the following contributions:
Hybrid inference executes different parts of an LLM across the CPU and GPU. Its efficiency depends on several architectural properties of the two processors. CPU Instruction Set Optimizations. Modern x86 CPUs provide Single Instruction, Multiple Data (SIMD) extensions for accelerating matrix multiplication, the core operation in LLM inference. AVX2 and AVX-512 [15] support wide-vector execution on mainstream CPUs, while Advanced Matrix Extensions (AMX) offer higher throughput but are largely limited to server-class processors such as Intel Xeon [14]. This distinction matters in practice: llama.cpp’s ggml backend emphasizes broad SIMD support, whereas systems such as KTransformers rely on AMX-optimized kernels that are typically unavailable on consumer CPUs [16]. GPU Concurrent Execution and Data Transfer. Overlapping data transfers with GPU computation is a standard way to improve concurrency. NVIDIA GPUs support CUDA streams, where operations within a stream execute in order but operations across streams may overlap [26]. This allows systems to pipeline host-device transfers with GPU kernels. However, concurrent execution of multiple transfers is more constrained on consumer devices. On NVIDIA GPUs, host-device transfers are handled primarily by Copy Engines, which move data over PCIe without consuming Streaming Multiprocessor (SM) cycles. Consumergrade RTX GPUs typically expose only one Copy Engine, so even transfer requests issued from different streams execute sequentially on that engine. To supplement this hardware path, systems can use a copy kernel. When host memory is mapped and pinned, NVIDIA’s Zero-Copy feature allows GPU threads to access it directly, enabling SM-driven transfers to overlap with Copy Engine-driven transfers while
• We empirically analyze hybrid CPU-GPU inference on consumer devices and show why coarse layer- or expert-level placement often fails to minimize latency. • We design ATSInfer around three coordinated mechanisms: asynchronous CPU-GPU scheduling, static tensor placement, and load-aware dynamic transfer, which together substantially alleviate the CPU bottleneck in hybrid inference and improve end-to-end execution efficiency. • We implement ATSInfer by extending llama.cpp with about 15,000 lines of C++ code to support tensor-level offloading for both dense and MoE models. • Compared with llama.cpp under the same GPU VRAM budget, ATSInfer improves prefill throughput by up to 1.94×, improves decode throughput by up to 3.29×, and increases average GPU utilization during decode by about 70%. 2
sharing PCIe bandwidth, as illustrated in Figure 2. This distinction is important for hybrid inference as it prevents highpriority transfers from being blocked by long-running transfer tasks.
Execution Time (ms)
2.2
LLM Inference on Consumer Devices
LLM inference on consumer devices differs from datacenter deployment in three key respects. First, local deployment faces tighter resource constraints. Server deployments can often keep the full model on one or more GPUs, whereas consumer devices typically cannot fit even quantized models entirely in VRAM [20, 38]. As a result, offloading becomes a common choice for LLM inference on consumer devices. Second, the role of the key-value cache (KV-cache) differs from that in server deployments. Servers optimize for throughput under high concurrency and therefore maintain large aggregate KV-caches across many requests, making mechanisms such as paging [20], prefix sharing [20, 38], and large KV-cache pools [28] important for reducing serving cost and sustaining throughput. In local interactive inference, the KV-cache is typically far smaller than the model weights, often occupying only hundreds of MB rather than tens of GB. Thus, it is usually kept with the backend executing the corresponding attention operator to avoid extra transfers in practice. Third, local deployment must handle both prefill and decode [20] on the same machine. Prefill processes the full prompt and is more compute-intensive, whereas decode generates one token per step and is more memory-intensive [27]. Server systems often separate these phases by prefill-decode disaggregation technique [27, 28] and optimize them independently, but local systems must handle both under the same hardware constraints.
3
1.2
Gate
GPU
0.20
Up
1.0
Qwen3-30B-A3B Down_exp
0.15
Up_exp Gate_exp
0.8 0.10
0.6
Q
0.4 0.2 0.0
K V
0
Q
Out
Down
0.05
Out
K V
20
40
60
0.00
0
Weight Tensor Size (MiB)
5
10
Figure 3. Sizes and backend execution times of major weight tensors within one layer for Qwen3-14B and Qwen3-30BA3B on an RTX 3060 + Intel i7-11800H laptop. For MoE operators, the reported weight size includes only the activated experts rather than all experts.
to the GPU may yield a much larger speedup than moving another tensor of comparable size. This heterogeneity exposes an optimization space that coarse-grained policies cannot exploit. Figure 3 illustrates this point by comparing the sizes and backend execution times of major weight tensors within one layer during decode. Under tight VRAM budgets, offloading decisions should instead prioritize tensors by the latency reduction they deliver per byte of GPU memory. 3.2
Limits of Load-unaware Scheduling
Many systems rely on policies determined before execution and keep them unchanged at runtime. For example, llama.cpp [8] offloads selected compute-heavy operators to the GPU when batch size or sequence length exceeds a hardcoded threshold, while KTransformers uses Expert Deferral to overlap delayed CPU-side expert computation with later GPU work during decode [6]. Some MoE systems do adapt at runtime, but mainly to input-dependent behavior rather than device conditions. Specifically, they adjust expert retention and replacement based on activation patterns to improve GPU hit rates and reduce CPU load [5, 32, 36, 39, 40]. Even so, these mechanisms still respond primarily to model structure, requestlevel demand, and expert activation, rather than to transient hardware bottlenecks. Crucially, these approaches overlook device-level performance variability on consumer hardware. Unlike server deployments, which usually run in stable and dedicated environments, local inference often co-runs with browsers, IDEs, media applications, and other background workloads. These applications dynamically change the CPU, GPU, and memory resources available to inference. In addition, sustained
Motivation
This section provides empirical evidence for the two design pressures highlighted in the introduction: coarse offloading granularity and load-unaware scheduling on consumer hardware. 3.1
CPU
Qwen3-14B
Limits of Coarse Offloading Granularity
Existing offloading systems typically make placement decisions at a coarse granularity. For dense Transformer models, the common unit is the layer, while for MoE models some systems refine the unit to experts. However, layer- or expert-level placement remains too coarse for hybrid inference on consumer devices. A layer or expert comprises multiple weight tensors, and the operators consuming these tensors differ substantially in compute intensity, memory access patterns, and kernel implementation efficiency. As a result, the latency benefit of GPU residency is not uniform: on consumer hardware, moving one tensor 3
50
Ideal stable TPOT TPOT affected by thermal wall
TPOT (ms)
45
Dense
LLM Models
User Interface
MoE
Fluctuating with temperature
40
ATSInfer
35
High system load
30
Load-Aware Dynamic Transfer
Reach thermal wall Asynchronous CPU-GPU Coordination
25 20
Static Tensor Placement
0
5
Time (minute)
10
GGML Tensor Library
15
Heterogeneous Hardware
Figure 4. TPOT over time under dynamic local conditions. The figure highlights two realistic sources of performance variation on consumer devices: interference from background tasks and later degradation caused by a thermal wall.
NVIDIA RTX GPU
Consumer CPU
6~32GB VRAM
8~64GB VRAM PCIe 4.0/5.0
Figure 5. System architecture of ATSInfer.
mechanism. Together, these components directly orchestrate CPU and GPU storage, computation, and data movement across heterogeneous hardware, while exposing userfriendly interfaces and preserving llama.cpp’s broad support for dense and MoE models. Figure 6 summarizes how these components interact during execution. The workflow consists of an offline initialization phase and an online execution phase. In the offline phase, ATSInfer collects hardware characteristics and profiles tensor-level execution behavior on the target backends. Using these measurements, it derives empirical performance estimates for CPU execution, GPU execution, and transfer, then computes a default tensor placement plan under the available memory budget. It also initializes the memory layout required for hybrid inference, including the GPU resident region, CPU pinned-memory storage, and the temporary GPU buffers used for dynamic transfer. In the online phase, ATSInfer serves each request through repeated scheduling and execution rounds. Before prefill and each decode step, it uses recent measurements to estimate current CPU speed, GPU speed, transfer cost, and overlap opportunity. It then selects a subset of CPU-resident tensors for temporary promotion to the GPU. Next, it allocates space for these tensors in the GPU temporary buffers, constructs an execution plan with backend assignments, transfer events, and synchronization points, and executes the round while recording the performance information needed for subsequent scheduling decisions.
high utilization on laptops and mobile devices can trigger thermal throttling or power limits, reducing processor frequencies over time. As a result, the relative benefits of CPU execution, GPU execution, and host-device transfer can shift substantially during execution, making both static policies and input-driven runtime policies suboptimal. Figure 4 illustrates this mismatch by showing how Time per output token (TPOT) evolves over time as the system first experiences interference from background tasks and then hits a thermal wall. Therefore, practical local inference requires a runtime scheduler that reacts to transient bottlenecks. If CPU throughput degrades due to contention or thermal throttling, the system should shift profitable computation to the GPU. Conversely, if GPU performance or PCIe transfer efficiency becomes the limiting factor, it should reduce transfers and rely more on CPU execution. The key implication is that runtime transfer decisions must be loadaware rather than fixed or merely input-driven. Taken together, these observations motivate the design of ATSInfer, a high-performance local inference system for consumer devices that supports tensor-level fine-grained offloading and load-aware dynamic transfer.
4
ATSInfer Design
4.1
Overview of ATSInfer
Figure 5 shows the overall architecture of ATSInfer. Built on top of llama.cpp [8], ATSInfer adds a tensor-granularity execution path organized around three components: an asynchronous CPU-GPU coordination substrate, a static tensor placement mechanism, and a load-aware dynamic transfer
4.2
CPU-GPU Coordination
To enable overlap between the two kinds of data movement and computation, ATSInfer first organizes CPU memory 4
Static
Dynamic
GPU Tensor
CPU Tensor
Tensor for transfer
LLM Model L0 L1
...
Ln
Performance
Static Tensor
Profile
Placement
Load Model
Requests
User
Initializaiton Execution
Hardware Load
Info Load-aware
Build
Dynamic GPU
PCIe
CPU
DRAM
Runtime
Transfer
Asynchronous CPU-GPU Coordination Performance Recorder
ATSInfer
Figure 6. Workflow overview of ATSInfer.
Compute Buffer
Pinned-memory storage
pinned memory improves the stability and efficiency of CPUto-GPU transfer and makes asynchronous movement with copy kernels practical at runtime. The third is a set of GPU temporary buffers for tensors that are promoted from CPU memory to the GPU on demand. Rather than allocating a separate GPU region for every transferred tensor, ATSInfer organizes temporary storage by tensor lifetime and reuses the same buffer space across tensors whose live ranges do not overlap. In practice, tensors that play the same role in different layers can often share the same temporary space because they are consumed sequentially. This reuse strategy reduces peak GPU memory consumption without introducing additional synchronization complexity. This layout cleanly separates long-lived residency from short-lived runtime promotion. In addition to these three regions, both the CPU and the GPU maintain their own compute buffers for operator execution; these buffers are preallocated and efficiently managed by the underlying ggml backend.
Compute Buffer
Resident region
CPU GPU
Temporary buffers
Figure 7. Memory layout of ATSInfer.
and GPU memory carefully. On top of this memory layout, it builds a pipelined CPU-GPU coordination mechanism designed to maximize concurrency while minimizing unnecessary synchronization.
4.2.2 Asynchronous CPU-GPU Scheduling. Before prefill and each decode round, ATSInfer constructs an execution plan for the current round and reassigns GPU temporarybuffer space accordingly. To control synchronization precisely, it partitions the computation graph at two kinds of positions: backend-switch boundaries and operators that consume tensors stored in temporary GPU buffers. The resulting plan is a sequence of ordered splits with explicit transfer and synchronization boundaries. Figure 8 illustrates a representative CPU-GPU execution pipeline. ATSInfer uses two streams to drive execution. The compute stream serializes operator execution together with the necessary transfer of activations across backends, while the
4.2.1 Memory Layout. As shown in Figure 7, ATSInfer organizes model weights into three logical memory regions to support fine-grained hybrid execution. The first is the GPU resident region, which stores tensors selected by static placement. These tensors are loaded into GPU memory before inference and kept resident whenever possible, so frequently used high-value operators can execute without repeated transfer overhead. The second is CPU pinned-memory storage, which holds tensors that cannot remain permanently on the GPU because of memory limits. Keeping these tensors in mapped and 5
CPU Split
Transferred Split
Empirical performance density ( s / MiB)
GPU Split
Activation Transfer
Device Synchronize CPU
3
6
9
Copy Kernel Compute Stream GPU Copy Engine
0-2
4
4
5
7
8
8
10 - 13
14
14
Transfer Stream
Stream Synchronize
Figure 8. Asynchronous CPU-GPU execution pipeline in ATSInfer.
25
CPU GPU
20
Qwen3-30B-A3B-Q4 20 15
15
10
10 5
5 0 Gate Up K V Out Q Down
0 Out Q K V Gate Up Down
Figure 9. Empirical performance density of representative tensors during decode, sorted by the density difference between CPU and GPU execution. For MoE expert tensors, the size is computed as the total size of all expert weights of the operator, because the actual routed experts are unknown before model loading and all experts within the same operator follow a unified placement decision.
transfer stream handles asynchronous CPU-to-GPU movement of dynamically selected weights. Separating the two streams is not sufficient by itself: the system also maps them to different execution engines. Activation transfer associated with computation is performed by copy kernels running on the GPU SMs, whereas weight movement is handled by the dedicated CUDA copy engine. Although the two transfer types still share PCIe bandwidth, this design prevents small activation transfers from being blocked behind large weight transfers and therefore preserves downstream computation overlap. To maximize overlap, ATSInfer launches weight transfers as early as possible without violating data dependencies. In dense models, weight transfer can typically start as soon as the required temporary-buffer slot becomes available. In MoE models, expert-weight transfer must wait until routing has been resolved, so that only the experts selected by the current token are moved to the GPU. Although MoE models may offer only a limited overlap window between computation and transfer, decode typically activates only a small number of experts, which keeps the corresponding transfer time short. Correctness relies on explicit synchronization among the CPU thread, the compute stream, and the transfer stream. When a GPU split finishes and triggers activation transfer, both operations remain ordered within the compute stream. When execution moves from GPU to CPU, the runtime synchronizes with the compute stream to ensure that the required data movement has completed. When asynchronous weight transfer starts or completes, lightweight GPU events coordinate cross-stream dependencies without forcing unnecessary global synchronization. Under these rules, ATSInfer overlaps transfer and computation while preserving execution consistency and avoiding races or premature temporary-buffer reuse. 4.3
Qwen3-14B-Q4
Section 3, the benefit of offloading varies substantially across tensors, making tensor-granularity placement essential. 4.3.1 Empirical Performance Density. To decide which tensors deserve permanent GPU residency, ATSInfer relies on measured execution behavior rather than a purely analytic model. In practice, tensor performance depends on multiple hardware- and implementation-specific factors, so theoretical metrics alone are often insufficient for guiding placement. For tensor 𝑖, let 𝑠𝑖 denote its memory footprint, 𝑔 and let 𝑡𝑖𝑐 and 𝑡𝑖 denote the measured execution time of the corresponding operator on CPU and GPU. We define the empirical performance density on backend 𝑏 ∈ {𝑐, 𝑔} as 𝑡𝑏 𝑘𝑖𝑏 = 𝑖 . 𝑠𝑖 This metric measures the execution cost associated with each unit of model memory, enabling comparison across tensors of different sizes. Compared with traditional compute-intensity metrics, empirical performance density has three advantages: • It reflects observed runtime behavior, because 𝑡𝑖 is derived from direct measurement and therefore captures both hardware characteristics and kernel implementation efficiency. • It jointly captures compute and memory effects, because the measured latency already incorporates both computational bottlenecks and memory-access overheads. • It is directly actionable for system optimization, because it maps naturally to the trade-off between latency benefit and GPU memory consumption.
Static Tensor Placement
Static placement determines which tensors should remain resident on the GPU before inference starts. As discussed in 6
Algorithm 1 Static Tensor Placement
As discussed in Section 3, tensor-level heterogeneity is substantial enough that coarse layer- or expert-level placement fails to capture the full performance opportunity. Figure 9 shows that empirical performance density varies substantially across tensors and backends. This observation motivates tensor-granularity placement and suggests that uniform treatment of tensors within a layer or expert can lead to suboptimal resource utilization.
1: Input: 2: model M; 3: GPU memory budget 𝑀 4: Output: 5: placement decision 𝑏 6: 𝑠 ← ReadTensorSizes(M); 7: 𝐺 ← BuildComputationGraph(M); 8: (𝑡 𝑐 , 𝑡 𝑔 , 𝑐) ← Profile(𝐺); 9: 𝑟 ← 𝑡 𝑐 − 𝑡 𝑔 ;
4.3.2 Problem Formulation. Using the tensor-level quan𝑔 tities defined above, including 𝑠𝑖 , 𝑡𝑖𝑐 , 𝑡𝑖 , and 𝑟𝑖 , we now formulate static placement. Let 𝑏𝑖 ∈ {CPU, GPU} denote the backend assigned to tensor 𝑖, and let 𝑀 denote the GPU memory budget available for static residency. When two adjacent tensors are assigned to different backends, the system incurs an additional switching cost due to activation transfer. Let 𝑐𝑖 denote the switching cost at the boundary between tensors 𝑖 − 1 and 𝑖. If the measured PCIe bandwidth is 𝐵 pcie and the total size of the transferred inputs at this boundary is 𝑆 in,𝑖 , we estimate𝑐𝑖 = 𝑆 in,𝑖 /𝐵 pcie . Static placement therefore solves the following optimization problem: ! 𝑛 𝑛 ∑︁ ∑︁ max 𝑟𝑖 1{𝑏𝑖 = GPU} − 𝑐𝑖 1{𝑏𝑖 ≠ 𝑏𝑖 −1 } 𝑖=1
10: if M is dense then 11: 𝑏 ← SolveKnapsackDP(𝐺, 𝑠, 𝑟, 𝑐, 𝑀); 12: else 13: (𝑇exp,𝑇nonexp ) ← PartitionTensors(𝐺); 14: 𝑠 nonexp ← TotalSize(𝑇nonexp, 𝑠); 15: if 𝑀 ≥ 𝑠 nonexp then
𝑏 nonexp ← GPU; 𝑏 exp ← SolveKnapsackDP(𝑇exp, 𝑠, 𝑟, 𝑐, 𝑀 − 𝑠 nonexp ); 18: else 19: 𝑏 exp ← CPU; 20: 𝑏 nonexp ← SolveKnapsackDP(𝑇nonexp, 𝑠, 𝑟, 𝑐, 𝑀); 21: end if 22: end if 23: Return 𝑏 16: 17:
𝑖=2
subject to 𝑛 ∑︁
𝑠𝑖 1{𝑏𝑖 = GPU} ≤ 𝑀,
𝑏𝑖 ∈ {CPU, GPU}.
by the dynamic program. In practice, ATSInfer quantizes both 𝑠𝑖 and 𝑀 at MB granularity. This discretization preserves sufficient accuracy for placement while substantially shrinking the search space and reducing runtime and memory overhead. Because static placement is performed offline or during initialization, its cost is amortized across subsequent inference requests.
𝑖=1
The first term captures the latency reduction from GPUresident tensors, while the second penalizes backend fragmentation. Overall, the problem is a knapsack-style optimization with an additional penalty on adjacent backend switches. 4.3.3 Tensor Placement Solver. Before solving the optimization, ATSInfer first profiles the target platform to obtain all required inputs. It measures per-operator latency 𝑔 on CPU and GPU to estimate 𝑡𝑖𝑐 , 𝑡𝑖 , and thus 𝑟𝑖 ; records tensor sizes to obtain 𝑠𝑖 ; and estimates switching costs 𝑐𝑖 from activation sizes and measured PCIe bandwidth. The resulting static placement is therefore specialized to the target model, runtime, and hardware. For dense models, ATSInfer directly solves the knapsackstyle optimization above with a standard dynamic program and then recovers the placement by backtracking. For MoE models, ATSInfer integrates the solver with an expert/nonexpert partitioning policy to avoid overestimating the value of expert residency. If the GPU memory budget 𝑀 is discretized as the dynamicprogramming memory dimension, the dense-model solver runs in 𝑂 (𝑛𝑀) time and uses 𝑂 (𝑛𝑀) space. For MoE models, tensor partitioning and capacity checks add only linear preprocessing overhead, so overall complexity is still dominated
4.4
Load-Aware Dynamic Transfer
Static placement provides an effective default configuration, but it is insufficient once inference begins. To further relieve the CPU bottleneck, ATSInfer selectively transfers some CPU-resident tensors to the GPU for execution and overlaps their weight movement with preceding computation, so that only the exposed portion of transfer time remains on the critical path. This design exploits otherwise idle PCIe bandwidth to shift part of the workload away from the CPU, thereby combining GPU-centric offloading with hybrid CPU-GPU inference and balancing pressure across CPU, GPU, and PCIe. Because its benefit depends on current CPU speed, GPU speed, transfer bandwidth, and overlap opportunity, ATSInfer makes these transfer decisions online in a load-aware manner rather than fixing them entirely through offline profiling. 4.4.1 Problem Formulation. Let 𝑏 = (𝑏 1, . . . , 𝑏𝑛 ) denote the default backend placement produced by static placement, 7
Algorithm 2 Dynamic Transfer Scheduling
Under these definitions, total latency can be written as
1: Input: 2: default placement 𝑏; 3: GPU/CPU execution times 𝑡 𝑔 , 𝑡 𝑐 ;
𝑇 (𝑟𝑏 1, . . . , 𝑟𝑏𝑛 ) =
𝑖=1
4: activation-transfer costs 𝑐; 5: weight-transfer times 𝑤 6: Output: 7: runtime backend assignment 𝑟𝑏 8: 𝑠𝑒𝑔[ 𝑗, 𝑖] ←
Í𝑖 −1
Í𝑖 −1 𝑏𝑘 𝑘=𝑗+1 𝑡𝑘 + 𝑘=𝑗+1 𝑐 𝑘 1{𝑏𝑘 −1 ≠ 𝑏𝑘 };
𝑗 <𝑖 𝑏 𝑗 =CPU 𝑔
17: 𝑑𝑝 [𝑖, GPU] ← 𝑏𝑒𝑠𝑡 + 𝑐𝑖 1{𝑏𝑖 −1 = GPU} + 𝑡𝑖 ; 18: end if 19: end for 20: (𝑇 ★, 𝑟𝑏 ★) ← arg min𝑟𝑏 ∈ {GPU,CPU} 𝑑𝑝 [𝑛, 𝑟𝑏]; 21: Recover 𝑟𝑏 1, . . . , 𝑟𝑏𝑛 by backtracking; 22: Return 𝑟𝑏
and let 𝑟𝑏𝑖 ∈ {CPU, GPU} denote the runtime backend assignment in the current round. Tensors with 𝑏𝑖 = GPU remain on the GPU, whereas tensors with 𝑏𝑖 = CPU may be temporarily transferred to the GPU and executed there. 𝑔 Let 𝑡𝑖𝑐 , 𝑡𝑖 , and 𝑐𝑖 denote the current CPU execution time, GPU execution time, and activation-transfer cost, respectively, and let 𝑤𝑖 denote the weight-transfer time of tensor 𝑖. Unlike static placement, these quantities are refreshed from recent runtime measurements before each scheduling round. For MoE expert weights in a decode round, 𝑤𝑖 is defined as the total transfer time of the activated experts rather than that of all experts in the operator. The objective is to minimize the end-to-end latency of the current round. For a tensor that is CPU-resident by default but promoted to the GPU, only the non-overlapped fraction of transfer time contributes to the critical path. Let 𝑖 −1 ∑︁ 𝑘=𝑗+1
𝑡𝑘𝑏𝑘 +
𝑖 −1 ∑︁
𝑛 ∑︁ 𝑖=2
𝑐𝑖 1{𝑟𝑏𝑖 −1 ≠ 𝑟𝑏𝑖 } +
∑︁
𝛿𝑖 ,
𝑖∈G
4.4.2 Dynamic Transfer Algorithm. Algorithm 2 uses dynamic programming to minimize the latency of the current round. It traverses tensors in execution order while distinguishing whether the current partial schedule ends on the GPU or on the CPU. For tensors that are GPU-resident by default, the algorithm only needs to update the backend-switch cost. For tensors that are CPU-resident by default, besides the option of keeping them on the CPU, it also enumerates the start point of the most recent CPU→GPU transfer and evaluates the corresponding exposed transfer cost. In this way, the dynamic program jointly captures execution cost, backend-switch overhead, and transfer/computation overlap within a unified optimization. Empirically, the optimal transfer pattern varies by stage. During prefill, heavier computation and a larger overlap window make GPU execution more attractive, so more CPUresident tensors tend to be promoted. During decode, the available overlap window is smaller, and the scheduler becomes correspondingly more selective, often yielding an intermittent promotion pattern. These results suggest that dynamic transfer should adapt to stage-specific execution characteristics rather than rely on a fixed policy. In the worst case, the algorithm enumerates a legal transfer start point 𝑗 for each tensor that is CPU-resident by default, which yields 𝑂 (𝑛 2 ) time complexity. The interval values 𝑠𝑒𝑔( 𝑗, 𝑖) can be precomputed in 𝑂 (𝑛 2 ) time, so the overall runtime remains 𝑂 (𝑛 2 ). Explicitly storing all interval values requires 𝑂 (𝑛 2 ) space; if only dynamic-programming states and the information needed for backtracking are retained, auxiliary state can be reduced to 𝑂 (𝑛). In practice, the overhead is usually lower because dynamic promotion is considered only for the subset of tensors that are CPU-resident under the static placement.
Mark 𝑑𝑝 [𝑖, CPU] as infeasible; else Update 𝑑𝑝 [𝑖, CPU] by extending 𝑑𝑝 [𝑖 − 1]; 𝑏𝑒𝑠𝑡 ← min 𝑑𝑝 [ 𝑗, CPU] + max 𝑤𝑖 , 𝑠𝑒𝑔[ 𝑗, 𝑖] ;
𝑠𝑒𝑔( 𝑗, 𝑖) =
𝑡𝑖𝑟𝑏𝑖 +
where G = {𝑖 | 𝑏𝑖 = CPU, 𝑟𝑏𝑖 = GPU} is the set of tensors dynamically promoted to the GPU.
9: Initialize dynamic-programming states; 10: for 𝑖 = 1 to 𝑛 do 11: if 𝑏𝑖 = GPU then 12: Update 𝑑𝑝 [𝑖, GPU] by extending 𝑑𝑝 [𝑖 − 1]; 13: 14: 15: 16:
𝑛 ∑︁
4.4.3 Load-Aware Re-scheduling. ATSInfer records the observed transfer and computation times of the current plan and compares them with the measurements used by the most recent scheduling decision. As summarized in Algorithm 3, rather than rerunning scheduling whenever load changes, ATSInfer triggers re-scheduling only when the measured deviation exceeds a threshold 𝜖 and sufficient time has elapsed since the previous re-scheduling event. This design is necessary because reconfiguration overhead is nonnegligible relative to decode latency. For Qwen3-14B on an RTX 3060, TPOT is approximately 40 ms, whereas rebuilding the computation graph takes about 7 ms and rerunning
𝑐𝑘 1{𝑏𝑘 −1 ≠ 𝑏𝑘 }
𝑘=𝑗+1
denote the amount of computation and activation-transfer time along the default path between the previous CPU-side endpoint 𝑗 and the start of tensor 𝑖, i.e., the overlap window available to hide weight transfer. The exposed transfer time is therefore 𝛿𝑖 = max 𝑤𝑖 − 𝑠𝑒𝑔( 𝑗, 𝑖), 0 . 8
Algorithm 3 Load-aware Re-scheduling
the RTX 4090 platform, we test Llama3.1-70B [11], Qwen3Next-80B-A3B, Qwen3.5-122B-A10B [34], and GPT-OSS-120B. GLM-Z1-9B is evaluated in FP16, GPT-OSS models use the native MXFP4 format, and all remaining models use INT4 quantization. Baselines. We compare ATSInfer against three state-ofthe-art baselines: llama.cpp [8], vLLM [20], and KTransformers [6]. llama.cpp serves as the primary baseline and implementation foundation of ATSInfer, and represents a high-performance C++ runtime for practical heterogeneous inference. vLLM is widely used in server-oriented deployments and also provides support for single-GPU offloading. KTransformers targets MoE inference and already integrates SGLang [38] for GPU-side execution. Importantly, vLLM and KTransformers do not yet support the full model set considered in this paper. In particular, vLLM (v0.18.0) does not currently support offloading for some recent models with customized attention implementations, and KTransformers (v0.5.2) supports only a subset of MoE models because its custom CPU kernels do not support the MXFP4 format. Workloads and Metrics. We evaluate prefill throughput with prompt lengths ranging from 512 to 4096 tokens, and decode throughput with a prompt length of 2048 tokens and a generation length of 128 tokens. To study sustainedgeneration stability on the laptop platform, we further vary the output length from 32 to 2048 tokens and examine how decode throughput evolves over the course of generation. Beyond throughput, we also measure TPOT under simulated hardware pressure and compare GPU SM utilization and effective PCIe bandwidth to better understand the source of performance differences. All experiments use batch size 1, which matches the most common local-deployment setting on consumer devices. For fairness, all systems use the same chunk size for chunked prefill (512 on the RTX 3060 platform and 2048 on the RTX 4090 platform), the same offloading policy that fills GPU memory as much as possible, and the same KV-cache size.
1: Input: 2: previous measurement snapshot 𝑚𝑝𝑟𝑒𝑣 ; 3: current measurement snapshot 𝑚𝑐𝑢𝑟 ;
deviation threshold 𝜖; minimum re-scheduling interval 𝜏; 6: time of last re-scheduling 𝑡 𝑙𝑎𝑠𝑡 ; 7: default placement 𝑏 8: Output: 9: runtime backend assignment 𝑟𝑏 10: 𝑑 ← ComputeDeviation(𝑚𝑝𝑟𝑒𝑣 , 𝑚𝑐𝑢𝑟 ); 11: if 𝑑 < 𝜖 or 𝑡 𝑛𝑜𝑤 − 𝑡 𝑙𝑎𝑠𝑡 < 𝜏 then 12: Return previous assignment 13: else 14: (𝑡 𝑔 , 𝑡 𝑐 , 𝑐, 𝑤) ← 𝑚𝑐𝑢𝑟 ; 15: 𝑟𝑏 ← DynamicTransferScheduling(𝑏, 𝑡 𝑔 , 𝑡 𝑐 , 𝑐, 𝑤); 16: BuildRuntime(𝑟𝑏); 17: Update 𝑚𝑝𝑟𝑒𝑣 ← 𝑚𝑐𝑢𝑟 and 𝑡 𝑙𝑎𝑠𝑡 ← 𝑡 𝑛𝑜𝑤 ; 18: Return 𝑟𝑏 19: end if 4: 5:
Table 1. Hardware platforms in the evaluation.
CPU GPU RAM PCIe
Laptop
Desktop
Intel i7-11800H RTX 3060 (6GB) 32GB DDR4 Gen4 x16
Intel i7-11700 RTX 4090 (24GB) 64GB DDR4 Gen4 x16
the dynamic program takes about 9 ms. In our implementation, we therefore set 𝜖 to 15% and enforce a minimum re-scheduling interval equal to five times the recent TPOT, thereby avoiding repeated re-optimization under short-term performance fluctuations. This threshold-based, rate-limited policy preserves load awareness while keeping reconfiguration overhead under control.
5.2
5
Evaluation
5.1
Experimental Setup
End-to-End Performance
Figures 10 and 11 compare the throughput of ATSInfer against the baselines in the prefill and decode phases, respectively. In prefill, KTransformers delivers low throughput on consumer platforms because its optimized AMX-based CPU kernels are unavailable on consumer CPUs, resulting in substantial CPU-side overhead. vLLM also becomes less competitive as the activated parameter size increases, indicating that a fully GPU-centric strategy remains constrained by the PCIe bottleneck. Meanwhile, the ggml backend of llama.cpp relies on hard-coded thresholds during prefill to identify computeintensive matrix-multiplication operators and place them on the GPU. By contrast, ATSInfer combines static tensor placement with dynamic transfer to schedule tensors
Hardware. We evaluate on two representative classes of consumer devices: a high-performance desktop platform and a mid-range laptop platform. Table 1 summarizes the CPU, GPU, memory, and PCIe configurations of both systems. The laptop platform is constrained by thermal and power limits, making it difficult to sustain stable decode latency over long runs, whereas the desktop platform provides more stable sustained performance. Models. We evaluate ATSInfer across a range of state-ofthe-art dense and MoE models with different parameter scales. On the RTX 3060 platform, we test GLM-Z1-9B [9], Qwen3-14B, Qwen3-30B-A3B [37], and GPT-OSS-20B [3]. On 9
RTX 4090 RTX 3060 Prefill Throughput (tok/s) Prefill Throughput (tok/s)
ATSInfer
GLM-Z1-9B-FP16
800
llama.cpp
vllm
Qwen3-14B-INT4
800
600
600
400
400
400
200
200
200
600
0
1250
512
1024
2048
0
4096
Llama3.1-70B-INT4
1024
2048
4096
Qwen3-Next-80B-A3B-INT4
1000
1500
750
1000
500
512
1024
2048
Prompt Length
0
4096
0 800
512
1024
2048
4096
1024
2048
4096
0
512
1024
2048
4096
GPT-OSS-120B-MXFP4 1000 500
200
Prompt Length
1000 800 600 400 200 0
1500
400
512
GPT-OSS-20B-MXFP4
Qwen3.5-122B-A10B-INT4
600
500
250 0
512
KTransformers
Qwen3-30B-A3B-INT4
512
1024
2048
Prompt Length
4096
0
512
1024
2048
Prompt Length
4096
Figure 10. Prefill throughput comparison between ATSInfer and other baselines on laptop and desktop platforms. Please note that vLLM (v0.18.0) does not currently support offloading for Qwen3-Next, Qwen3.5, or GPT-OSS. KTransformers (v0.5.2) supports the MoE models except GPT-OSS, because its custom CPU kernels do not yet support the MXFP4 format.
30
llama.cpp
vllm
RTX 3060
25 20
KTransformers
ATSInfer
RTX 4090 TPOT (ms/tok)
Decode Throughput (tok/s)
ATSInfer
50 40
15
30
10
20
5
10
0 B B B B GLM9 QW14 QW30 GPT20
0 B B B B LM70 QW80 QW122 GPT120
60 55 50 45 40 35
GPT-OSS-20B-MXFP4
32
128
llama.cpp
80 GPT-OSS-120B-MXFP4 70 60 50 40 30 512 1024 2048 32 128 512 1024 2048
Gen Length
Gen Length
Figure 12. Average decode TPOT as generation length increases on the laptop and desktop platforms.
Figure 11. Decode throughput comparison between ATSInfer and other baselines on laptop and desktop platforms.
set of weights with relatively small average empirical performance density gaps between CPU and GPU, thereby incurring a smaller penalty from CPU execution. vLLM remains limited by the PCIe bottleneck inherent in GPU-centric offloading, and consequently performs well only for models with relatively small per-step weight transfer, such as Qwen330B-A3B (about 1.0 GB), compared with Qwen3-14B (about 4.9 GB) and GLM-Z1-9B (about 13.5 GB). Guided by empirical performance density, ATSInfer makes placement decisions at tensor granularity and uses dynamic transfer to exploit otherwise idle PCIe bandwidth to relieve CPU pressure, thereby achieving the best decode performance overall. On the laptop platform, the maximum decode speedups of ATSInfer are 3.29× over llama.cpp, 4.35× over vLLM, and 3.15× over KTransformers. On the high-end desktop
automatically, and further uses CPU-GPU coordination to overlap transfer with computation more effectively than the serialized execution path in llama.cpp. As a result, ATSInfer improves prefill throughput over llama.cpp from 1.28× to 1.88× on the laptop platform and from 1.15× to 1.94× on the high-end desktop platform. In decode, the computation is less intensive than in prefill, which allows some low-parallelism operators to be handled effectively by the CPU and makes KTransformers substantially stronger in decode than in prefill. On the models it supports, KTransformers consistently outperforms llama.cpp because expert-granularity offloading effectively selects a 10
25
50
Time (s)
75
SM Utilization (%) 100
Figure 13. Decode TPOT of static policy and load-aware policy under time-varying system load. CPU pressure is applied first, followed by GPU+PCIe pressure.
1
2
25 20 15 10 5 0
Decode Phase
0
2
4
6
0
2
4
6
6 4
4
2 0
2 0
Prefill Throughput (tok/s)
Load Adaptation
Motivated by the load variation discussed in Section 3, we next evaluate whether ATSInfer can adapt to realistic changes in CPU, GPU, and PCIe pressure, as illustrated in Figure 13. We compare two policies: a static policy, which is manually tuned from offline profiling data, and a load-aware policy, which is generated online by the Dynamic Transfer Scheduling algorithm in Algorithm 2. When CPU, GPU, and PCIe resources are perturbed by co-running applications, the static policy quickly becomes suboptimal, whereas the load-aware policy adjusts tensor transfer decisions according to current runtime conditions and thereby reduces performance degradation. As a result, compared with the static policy, the load-aware policy reduces the TPOT increase by 13% under 60% external CPU load and by 38% under 60% external GPU+PCIe load. 5.4
0
ATSIfer
1
2
0
Time (s)
Figure 14. Timeline of GPU SM utilization and effective PCIe bandwidth for ATSInfer and llama.cpp across the prefill and decode phases.
platform, the corresponding maximum speedups are 3.12×, 2.03×, and 1.33×. Figure 12 shows how average decode TPOT changes with generation length on the laptop and desktop platforms. On the laptop platform, both llama.cpp and ATSInfer exhibit rising TPOT as generation continues, because thermal throttling prevents the device from sustaining its peak operating frequency over long runs. By contrast, the desktop platform provides more stable TPOT across different generation lengths. 5.3
100 80 60 40 20 0
Prefill Phase
Base Base+TP
Base+TP+AC Base+TP+AC+DT
Decode Throughput (tok/s)
0
PCIe Bandwidth (GB/s)
TPOT (ms)
llama.cpp
Static Policy Load-aware Policy
90 80 70 60 50 40
700
9
600
8
500
7
400
6 Prefill
Decode
Figure 15. Performance breakdown of ATSInfer on Qwen314B. The four configurations are base (llama.cpp), base+TP, base+TP+AC, and base+TP+AC+DT (ATSInfer), where TP denotes tensor placement, AC denotes async coordination, and DT denotes dynamic transfer.
on the GPU, while also overlapping transfer and execution more effectively. As a result, ATSInfer sustains markedly higher GPU SM utilization during prefill, indicating that the GPU spends less time waiting on serialized transfer-compute interactions. During decode, ATSInfer not only transfers intermediate activations, but also continuously moves selected CPUresident weights into GPU memory according to runtime scheduling decisions. This design leads to more sustained PCIe bandwidth usage over time than in llama.cpp. In turn, the additional data movement increases PCIe bandwidth utilization and improves average GPU SM utilization by about 70% relative to llama.cpp.
Hardware Utilization
Figure 14 compares GPU SM utilization and effective PCIe bandwidth over time for ATSInfer and llama.cpp across different inference phases, using traces collected with NVIDIA Nsight Systems [25]. During prefill, ATSInfer and llama.cpp keep a comparable amount of weights resident in GPU memory and transfer a similar total volume of data from CPU memory. The key difference is that ATSInfer benefits from its static tensor placement policy, which places more compute-intensive weights 11
5.5
a single fixed partition or on GPU-centric transfer overlap alone.
Ablation Study
To better understand the contribution of each component in ATSInfer, we perform a detailed performance breakdown on Qwen3-14B with prompt length 1024 and generation length 128. Starting from llama.cpp, we incrementally add three mechanisms: Static Tensor Placement, asynchronous CPUGPU Coordination, and Load-Aware Dynamic Transfer. Figure 15 shows that the gains in prefill and decode arise from different sources. In prefill, the dominant improvements come from Static Tensor Placement and asynchronous CPUGPU Coordination, with most of the benefit already realized before dynamic transfer is enabled. During decode, static placement alone provides only a modest gain, whereas the larger improvements come from asynchronous CPU-GPU Coordination and Load-Aware Dynamic Transfer. This trend is consistent with the utilization analysis above and indicates that runtime coordination and adaptation are more important in the latency-sensitive decode phase.
6
Related Work
6.1
GPU-Centric and Hybrid LLM Inference
6.2
MoE-Aware Hybrid Inference
Mixture-of-Experts models present an additional opportunity for hybrid inference because expert sparsity reduces the amount of model state and computation required for each token. Recent systems [5, 6, 32, 36, 39–41] exploit this structure to distribute storage and execution across CPU and GPU. Their results show that routing-aware placement and execution can make large MoE models practical on resourceconstrained devices. Some of these systems also support online scheduling by adapting expert residency and swapping decisions to input-dependent changes in the activatedexpert distribution [5, 32, 36, 39, 40]. However, they do not explicitly account for time-varying hardware load, which materially affects performance on consumer devices. Their coarse scheduling granularity and lack of load-aware adaptation therefore limit performance in local deployment. Importantly, ATSInfer applies the same optimizations to both dense and MoE models and does not yet incorporate MoE-specific optimization mechanisms. As a result, expertresidency and swapping strategies are complementary to ATSInfer and could potentially be combined with its tensorgranularity scheduling within experts to achieve further speedups.
Recent systems have made LLM inference increasingly practical under limited GPU memory, but they adopt different assumptions about the role of CPU in the execution path. GPU-centric systems such as FlexGen, DeepSpeed Offload, and Hugging Face Accelerate directly target memory-limited execution by moving weights or runtime state across a storage hierarchy while keeping GPU as the main compute backend [13, 29, 31]. Systems such as vLLM and SGLang are primarily designed for high-throughput server-side serving, but their memory-management and offloading mechanisms are also relevant to single-GPU deployment when model size exceeds device memory [20, 38]. Across this line of work, GPU remains the dominant compute backend, and performance is fundamentally shaped by host-to-device bandwidth and by how effectively transfer latency can be hidden. Hybrid systems move beyond this design by allowing CPU to participate directly in inference. llama.cpp shows that local hybrid execution can make LLM inference practical across a wide range of commodity devices [8]. FlexInfer explores CPU-GPU cooperation to reduce transfer bottlenecks, while PowerInfer and LLM in a Flash exploit sparsity or hot-neuron behavior to retain part of the computation on the GPU while relying on CPU-side storage for the rest [4, 24, 33]. These systems demonstrate that hybrid execution is viable, but most still rely on layer-level partitioning, fixed placements, or coarse heuristics. ATSInfer differs from this line of work in two key respects. First, it treats tensors rather than layers as the main scheduling unit, following the fine-grained opportunity highlighted in Section 3. Second, it combines offline static placement with runtime load-aware transfer, rather than relying on
6.3
Compression and Model Reduction
Model compression is a fundamental way to reduce the deployment cost of LLMs on consumer devices. Its primary objective is to reduce parameter size, memory traffic, and runtime storage demand while preserving model quality as much as possible. Among existing approaches, low-bit quantization is the most widely used [7, 8, 21, 30, 35]. By representing weights at lower precision, it substantially reduces both model size and transfer volume. In practice, 4-bit weight quantization is a common operating point for consumer-device deployment because it often offers a favorable balance between model accuracy and system efficiency [7, 21]. Beyond quantization, sparsity and pruning [4, 18, 23, 33] reduce the number of parameters or activations involved in inference, thereby lowering computation, storage, and memory-access overheads. Knowledge distillation [10, 12] reduces deployment cost from the model side by transferring the capability of a large model to a smaller one, which is typically easier to deploy under constrained memory and compute budgets. 12
7
Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531 [stat.ML] https: //arxiv.org/abs/1503.02531 [13] Hugging Face. 2022. Accelerate: Training and Inference at Scale Made Easy. https://github.com/huggingface/accelerate. [14] Intel. 2026. Intel Advanced Matrix Extensions (AMX) Overview. https: //www.intel.com/content/www/us/en/products/docs/acceleratorengines/advanced-matrix-extensions/overview.html. Retrieved April 1, 2026. [15] Intel. 2026. Intel AVX-512 Overview. https://www.intel.com/content/ www/us/en/architecture-and-technology/avx-512-overview.html. Retrieved April 1, 2026. [16] Intel. 2026. Intel Processor Claims. https://intel.com/processorclaims. Retrieved April 1, 2026. [17] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of Experts. arXiv:2401.04088 [cs.LG] https://arxiv.org/abs/2401.04088 [18] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems 37 (2024), 52481–52515. [19] Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. 2024. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models. arXiv:2402.07033 [cs.LG] https://arxiv.org/abs/2402.07033 [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626. [21] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, WeiChen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems 6 (2024), 87–100. [22] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437 [23] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. 2023. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. In Proceedings of the 40th International Conference on Machine Learning. 22137–22176. [24] Seonjin Na, Geonhwa Jeong, Byung Hoon Ahn, Aaron Jezghani, Jeffrey Young, Christopher J. Hughes, Tushar Krishna, and Hyesoon Kim. 2025. FlexInfer: Flexible LLM Inference with CPU Computations. In Proceedings of Machine Learning and Systems, Vol. 7. [25] NVIDIA. 2018. NVIDIA Nsight Systems. https://developer.nvidia.com/ nsight-systems. Retrieved March 31, 2026. [26] NVIDIA. 2026. CUDA C++ Best Practices Guide. https://docs.nvidia. com/cuda/cuda-c-best-practices-guide/. Retrieved April 1, 2026. [27] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118– 132. [28] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving {LLM} chatbot. In 23rd USENIX Conference
Conclusion
In this paper, we presented ATSInfer, a system for efficient hybrid CPU-GPU inference of large language models on consumer devices. By combining tensor-granularity static placement, load-aware dynamic transfer, and asynchronous CPUGPU coordination, ATSInfer achieves substantially higher throughput than existing baselines, with up to 1.94× higher prefill throughput and 3.29× higher decode throughput than llama.cpp. These optimizations mitigate key bottlenecks in coarse-grained offloading and load-unaware scheduling under tight memory and bandwidth constraints. Overall, ATSInfer helps make local LLM deployment on resource-limited consumer devices more practical and responsive.
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774 [2] Aaron Adcock, Aayushi Srivastava, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pande, Abhinav Pandey, Abhinav Sharma, Abhishek Kadian, Abhishek Kumawat, Adam Kelsey, et al. 2026. The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes. arXiv:2601.11659 [cs.SE] https://arxiv.org/abs/2601.11659 [3] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL] https://arxiv.org/abs/2508.10925 [4] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 12562– 12584. [5] Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E Gonzalez, Matei Zaharia, and Ion Stoica. 2025. Moelightning: High-throughput moe inference on memory-constrained gpus. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 715–730. [6] Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, et al. 2025. Ktransformers: Unleashing the full potential of cpu/gpu hybrid inference for moe models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 1014–1029. [7] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/ abs/2210.17323 [8] Georgi Gerganov. 2023. llama.cpp: LLM inference in C/C++. https: //github.com/ggerganov/llama.cpp [9] Team Glm, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793 [cs.CL] https://arxiv.org/abs/2406. 12793 [10] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge Distillation: A Survey. International Journal of Computer Vision 129 (2021), 1789–1819. [11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil 13
on File and Storage Technologies (FAST 25). 155–170. [29] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3505–3506. [30] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. arXiv:2308.13137 [cs.LG] https://arxiv.org/abs/ 2308.13137 [31] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning. 31094–31116. [32] Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. 2025. ProMoE: Fast MoE-based LLM Serving using Proactive Caching. arXiv:2410.22134 [cs.DC] https://arxiv.org/abs/2410.22134 [33] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 590–606. [34] Qwen Team. 2026. Qwen3.5: Accelerating Productivity with Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5 [35] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference
on Machine Learning. 38087–38099. [36] Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2025. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache. arXiv:2401.14361 [cs.LG] https://arxiv. org/abs/2401.14361 [37] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https://arxiv.org/ abs/2505.09388 [38] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems. 62557–62583. [39] Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. 2025. AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design. 1–9. [40] Shuzhang Zhong, Yanfan Sun, Ling Liang, Runsheng Wang, Ru Huang, and Meng Li. 2025. Hybrimoe: Hybrid cpu-gpu scheduling and cache management for efficient moe inference. In 2025 62nd ACM/IEEE Design Automation Conference. 1–7. [41] Zeyu Zhu, Gang Li, Peisong Wang, Zitao Mo, Minnan Pei, Zhuoran Song, Xiaoyao Liang, and Jian Cheng. 2026. DALI: A WorkloadAware Offloading Framework for Efficient MoE Inference on Local PCs. arXiv:2602.03495 [cs.DC] https://arxiv.org/abs/2602.03495
14