Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs Zhihao Xu1 , Hao Zhong2 , Zeting Zhou1 , Yuhanag Xu1 , Haoyu Tong1 , Wei Wang1 , Jinshan Chen3 , Keqiang He1 , Chong Zhu3 , Shengzhong Liu∗1∗, Fan Wu1 , Guihai Chen1 , 1: Shanghai Jiao Tong University, 2: Lenovo Information Products (Shenzhen) Ltd, 3: Lenovo (Beijing) Ltd Email: {persistentstriverng, portcat123, xuyuhangtmx, thy030702, w10493, shengzhong}@sjtu.edu.cn, {kqhe, fwu, gchen}@cs.sjtu.edu.cn {zhonghao4, chenjs1, zhuchong}@lenovo.com
arXiv:2607.23115v1 [cs.DC] 25 Jul 2026
Abstract
tants [44, 48, 81] has significantly amplified the demand for edge computing, which typically operates in latency-sensitive interactive settings, favoring local execution for privacy, responsiveness, and cost considerations. As a result, even personal devices (e.g., laptops) are expected to accommodate heavy GPU workloads. However, such devices often lack sufficient capacity for computation-intensive AI tasks, while many distributed workstations remain underutilized. This mismatch highlights an urgent need for efficient and controllable GPU resource sharing among local devices to better accommodate demand surges and fully exploit available compute resources. We study task offloading within LANs, where nearby devices can efficiently share GPU resources. Unlike homogeneous workloads that are clearly partitioned and assigned in large-scale data centers, LAN environments host highly heterogeneous AI tasks, making a generic and lightweight offloading mechanism especially valuable. To this end, we adopt API remoting, where client-side CUDA GPU APIs are intercepted, forwarded through LAN, and executed transparently on remote servers, only requiring the GPU drivers installed on the server side. This design supports diverse tasks while avoiding the heavy setup of standard deployments, which often require application configuration and complex dependency installation that may take up to hours. By forwarding only GPU-related API calls, rather than the entire environment, API remoting enables fast, flexible, and efficient GPU sharing, as shown in Figure 1, where underutilized servers seamlessly serve multiple client requests. Moreover, we target API remoting connections between clients and servers for resource sharing, distinguished from load-balance-driven inter-server API remoting in serverless computing [22, 79]. However, unlike homogeneous server constitutions in cloud data centers, LAN servers are heterogeneous (e.g., due to progressive installations) and lack high-performance interconnects such as remote direct memory access (RDMA) [49] or NVSwitch [56], leaving API remoting constrained by limited bandwidth and high latency. Compared with task-specific offloading, API remoting generates far more transmissions. For
This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices. We achieve distributed task offloading via CUDA API remoting. However, beyond raw computation, network constraints emerge as the primary bottleneck: limited bandwidth, high-frequency API invocations, and cross-task contention significantly hinder performance. To address these challenges, we propose Gleam, a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions. First, we reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution. Second, we design a runtime task scheduler that dynamically determines API remoting pairs between LAN clients and servers, explicitly accounting for both network conditions and GPU resource contention under parallel workloads. Finally, we introduce dedicated mechanisms to ensure CUDA context consistency across distributed executions. Extensive experiments on heterogeneous NVIDIA GPUs and diverse AI workloads show Gleam consistently outperforms state-of-the-art baselines, achieving 1.4×–24.2× improvements in API remoting efficiency and up to 1.79× higher system throughput.
1
Introduction
With the rapid advancement of GPU manufacturing, a wide range of tasks have achieved remarkable performance gains, including machine learning [14,55,67], large language models (LLM) [18,30,40,47,58], and scientific computing [52,53,66]. Enterprises, research institutions, and universities have invested heavily in GPUs, making such resources increasingly available within local area networks (LANs). Meanwhile, the recent emergence of AI agents and personal assis∗ Shengzhong Liu is the corresponding author.
1
Client 1
Client 1
Server 1
Server 1
10% RTX 4090
AIGC
Client 2
Server 2 20%
PhysicsNeMo
LLM
RTX 4070
Client 3
Server 3 30%
Scientific computing
80%
Gleam GPU API Remoting
AIGC
RTX 4090
Client 2
Server 2
co-optimize
Computation
LLM
RTX 4070
Client 3
Server 3
Scientific computing
RTX 4060
Communication
RTX 4060
PhysicsNeMo
90%
70%
Figure 1: Scenario of GPU sharing among local devices. example, a sampling step of stable diffusion involves around 18,000 API calls. The transmission overhead of such shortlived but frequent requests often outweighs the actual GPU execution time, blocking subsequent calls and slowing down the progress. Moreover, GPU APIs expose limited upper-level task semantics, hindering the design of resource management strategies.
we design a systematic CUDA API remoting path manager that integrates asynchronous execution with client simulation and batch prefetch, reducing send-receive round trips and improving end-to-end latency. Contention-Aware Task Scheduling: Based on the optimized CUDA API remoting workflow, we further design an adaptive task scheduler for Gleam that dynamically dispatches AI tasks between clients and server GPUs at runtime, explicitly considering the previously overlooked resource contention [54, 76] during transmission and GPU computation between parallel tasks. Specifically, using latency predicted by a congestion-based online predictor as input, the scheduler jointly accounts for contention in both network communication and GPU computation, improving system throughput. CUDA Context Consistency Guardian: Existing solutions [70] focus on exclusive, short-term API remoting and overlook the risk of CUDA context crashes. In contrast, Gleam incorporates a reconciliation and protection module to maintain the consistency of long-lived CUDA contexts under network failures and multi-task cross-stream multiplexing, thereby enabling robust, long-lived API remoting. We implement Gleam based on gRPC and protobuf, and intercept more than 1,000 CUDA APIs from 6 dynamic libraries. Extensive evaluations on a prototype system of heterogeneous Nvidia GPUs with mainstream AI tasks show that Gleam consistently outperforms the state-of-the-art (SOTA) baselines, achieving 1.4×-24.2× speedup in API remoting efficiency, while improving the throughput by up to 1.79×. Our main contributions can be summarized as follows: • We present the Gleam framework for API-level GPU task offloading across LAN devices, enabling flexible resource sharing without complex runtime deployment.
Another key challenge in API remoting for task offloading is the tight coupling of communication and computation, especially under LAN constraints. Model loading requires transmitting large weight files, which can quickly saturate limited bandwidth and interfere with ongoing transfers. When multiple tasks execute concurrently, contention also arises at the shared GPU computation, making it difficult to sustain efficiency across all clients. These runtime dynamics call for adaptive scheduling that can balance model transmission with execution demands, mitigate interference, and maximize overall throughput. Existing approaches [22, 79] optimize API-level communications but lack mechanisms to jointly manage model loading and multi-task contention, leading to sub-optimal performance in our scenario. Besides, in long-lived API remoting systems with multitask execution, accidental network failures or cross-CUDAstream multiplexing can easily cause crashes in the shared CUDA context on each server. To tackle these challenges, we propose Gleam, a unified framework for efficient API remoting-based GPU task offloading between heterogeneous devices in LANs. It explicitly considers both network bandwidth and latency constraints by performing joint optimizations on the CUDA API remoting pipeline and contentionaware task scheduling between clients and servers, while ensuring long-lived API remoting through a guarding mechanism that preserves CUDA context consistency.
• We propose dedicated CUDA API remoting optimization strategies during model loading and inference stages to minimize the communication overhead.
CUDA API Remoting Optimization: Unlike existing works [22, 79] that solely reduce API call frequency, we address the distinct communication challenges in the model loading and inference stages with tailored optimizations. For model loading, we introduce a weight manager that leverages hashing-based indexing, caching, and reuse to avoid redundant weight transmissions across tasks. For model inference,
• We propose a contention-aware task scheduling algorithm explicitly considering both transmission and computation contention between parallel tasks. • Extensive experiments show that Gleam improves 1.4×24.2× in API remoting efficiency and improves throughput 2
Table 1: Latency comparison between client local CPU execution, server GPU local execution, and API remoting. Task
Client CPU (s)
Server GPU (s)
API Remoting
llama-3B-ggml sd-compvis
34.75 182.79
4.04 2.41
9.10 4.31
APP Client
Loading Phase
Inference Phase
x1 cudaGetDevice cudaMemcpy x1 cudaStreamSynchronize x1 cudaMemcpy x1 cudaStreamSynchronize x1
xM cudaGetDevice cudaLaunchKernel xN cudaGetDevice xM cudaLaunchKernel xN cudaStreamSynchronize x1
CPU CPU
Client 1
Client 2
Connection 1
Router
Switch
Connection 2
Server
Server 1
Figure 3: Typical GPU CUDA API invocation patterns and the intuition of local interception + remote forwarding.
Server 2
API calls at the client side and forwarding the corresponding requests to a remote machine with sufficient GPU capability, which is defined as API remoting. As shown in Table 1, for tasks such as llama-3B-ggml and sd-compvis, API remoting leads to unavoidable overhead compared to running directly on the server locally, but it is acceptable compared to running on the client without any GPU. API remoting approach yields several general advantages over approaches that require application program modifications or offloading entire program images. First, API remoting is non-intrusive, requiring no modifications to the application source code. Second, the approach offers remarkable flexibility by operating at the API level rather than requiring entire program offloading, making it universally applicable to diverse GPU tasks. Finally, it provides fine-grained control at the API level, allowing selective optimization of data transfers and computation tasks. Taken together, these properties make API remoting a promising mechanism for sharing GPU capacity among local devices while still enabling cross-task optimizations across a wide range of workloads.
Figure 2: Connection topology of the toy prototype. by up to 1.79× over SOTA baselines.
2
Background and Motivations
To quantify challenges in real-world edge GPU sharing scenarios, we build a prototype system for analysis of our APIremoting-based approach. As shown in Figure 2, two client machines are connected to a LAN via a wireless router, which links to a switch. The switch, in turn, connects to two GPU servers. This heterogeneous environment, with its inherent network limitations (1000Mbps) and resource diversity, serves as the foundation for our motivational experiments below.
2.1
GPU
CUDA API Remoting
Contemporary AI applications primarily run on GPUs. Applications interact with GPUs by invoking a series of APIs to transfer data, submit tasks (kernels), and synchronize execution. Since mainstream NVIDIA GPUs use CUDA APIs [1], we mainly target CUDA-based GPU tasks, and categorize common CUDA APIs into four functional groups in the rest of the paper for clarity. • State Maintenance APIs: Query or set GPU state (e.g., cudaGetDevice, cudaSetDevice).
2.2 API Remoting Communication Deficiency API remoting approach over LANs, e.g., Wi-Fi and Ethernet, encounters two critical bottlenecks: 1) high frequency in API calls and 2) limited bandwidth for bulk data transfers. These issues severely degrade task performance and must be addressed to make remote GPU sharing practical.
• Data Movement APIs: Copy data between CPU and GPU or two GPUs on the same machine (e.g., cudaMemcpy).
2.2.1
• Computation APIs: Submit a computation request (i.e., kernel) to GPU (e.g., cudaLaunchKernel).
High Communication Frequency
Many GPU tasks issue a high frequency of API calls. In a distributed setting, each call is handled as a synchronous remote procedure call (RPC), incurring network round-trip time. This overhead becomes prohibitively high for frequently invoked APIs. To illustrate, we trace the execution of sd-compvis on our prototype system. As shown on the left of Figure 4, for cudaGetDevice, one of the most frequent APIs, the actual server execution time is negligible, accounting for less than 0.26% of the total call duration, while the vast majority of time is spent on network communication. This observation reveals a critical deficiency: The strict synchronous execution model is expensive for all APIs. For instance, if cudaGetDevice can be executed asynchronously
• Control APIs: Block the GPU until data transfers or kernel executions complete (e.g., cudaStreamSynchronize). API Trace: As shown in Figure 3, GPU task interactions can be represented as a sequence of API calls, known as an API trace, which typically consists of two stages: model loading and inference. The loading stage is dominated by data movement APIs, such as copying model weights and input data from host (CPU) memory into device (GPU). By contrast, the inference stage is dominated by computation APIs. API Remoting: When the onboard GPU is insufficient, we can leverage nearby GPUs within the LAN by intercepting 3
Table 3: Multi-task communication contention.
Table 2: Loading cost of server local executing, forwarding API to remote without and with weight cache, combined with memory cost. llama-8B-ggml1
llava7B-PyTorch
Total memory (GB) Weight memory ratio
14.55 96%
15.45 91%
Local execute latency Forward latency w/o cache Forward latency with cache
2.19s 151.97s 9.32s
2.21s 162.68s 8.78s
cublasSet Stream Get device
50 40
Overall latency (s)
Server Exec. Send & Recv. Transmission
30
22.8% 21.4%
Memcpy -htod 0%
10
25% 50% 75% 100%
(a) API Latency Composition
0
Connection 1 Connection 2
12.4%
Inference Latency
Single
Concurrent
Single
Concurrent
41.50s 38.42s
58.34s 54.29s
34.57s 10.28s
34.69s 10.49s
Compute Pattern
Server 2 Local
Connection-2
2x Connection-2
GPU Util. Cost Per Infer. (s)
96% 9.32
63% 13.76
92% 18.28
2.3
8.1%
Multi-Task Contention
In a multi-tenant GPU sharing scenario, multiple offloaded tasks inevitably execute concurrently, leading to contention on both network communication and GPU computation, which makes task performance unpredictable and complicates corresponding task scheduling.
API-0 API-1 API-2 API-3 API-4
(b) Top 5 APIs Cost
Figure 4: API remoting analysis, here ‘htod’ refers to memcpy from client (host) to server (device). without experiencing the round-trip paths, the throughput of this API will increase significantly. As shown in the right of Figure 4, this simple optimization can save up to 19.32s, a 15.7% reduction in total time. Prior works [22, 70, 83] show similar findings, but they either consider only limited API types or rely on RDMA connections. This drives the necessity for systematic communication optimization for API remoting that effectively reduces the communication frequency without affecting the execution correctness. 2.2.2
Loading Latency
Table 4: Multi-task computation contention.
cudaGetLastError cudaMemcpy-htod cudaGetDevice cudaLaunchKernel cublasSetStream
15.7%
20
Communication
2.3.1
Communication Contention
When multiple clients transmit data to servers simultaneously, they compete for shared network resources. To investigate this, we run two sd-compvis tasks concurrently on the built prototype system (Figure 2), with Client 1 sending requests to Server 1 and Client 2 sending to Server 2. The results in Table 3 show that when running in isolation, the tasks’ model loading stages take 41.50s and 38.42s, respectively. However, when run concurrently, their latency increases by nearly 50%. This is because the loading stage is dominated by bandwidth-intensive cudaMemcpy operations, which would introduce severe network contention. In contrast, the inference stage, which involves less data transfer, is almost unaffected. This observation motivates the necessity for a network-aware task dispatcher that explicitly considers the potential network contentions between parallel connections and heterogeneous computation-communication characteristics between different stages.
Excessive Data Transfer Volume
Modern AI tasks, especially large language models (LLMs), require transferring huge amounts of data, such as model weights, from the host to the target GPU during the loading stage. When forwarded over a bandwidth-limited LAN, this stage becomes a major bottleneck. To address this, we measured the memory footprint for a multimodal language model (MLM), llava-7B-PyTorch, and a large language model (LLM), llama-8B-ggml. As shown in Table 2, the model weights occupy the vast majority of their memory consumption, accounting for 91% and 96%, respectively. The performance impact of transferring model weight is significant. Loading the weights into a local GPU takes only about 2s. However, when they are transferred from a client over our 1000 Mbps network, the loading time rises drastically to approximately 150s, a nearly 75x slowdown, caused by the bandwidth difference between PCIe and LAN connections. This degradation highlights the infeasibility of naively transferring all task data. If the server caches these weight chunks, the loading time for subsequent task executions is reduced to just 9s. This motivates the necessity for an efficient and generalized caching mechanism that can identify and reuse stationary data blocks to mitigate bandwidth constraints.
2.3.2
Computation Contention
Similarly, when multiple tasks are scheduled on the same GPU, their kernels inevitably contend for compute and memory resources. Compared to local execution, remote execution slows down individual tasks and reduces GPU utilization, since transmission latency increases GPU idle time. However, when multiple remote tasks are launched concurrently, GPU utilization recovers, and the overall throughput improves, despite each task experiencing higher latency due to contention. As shown in Table 4, running llama-8B-ggml locally on Server 2 takes 9.32s with 96% utilization. Forwarding a single client raises latency to 13.76s and lowers utilization to 63% due to transmission-induced API blocking. With two 4
Task1
Task2
PhysicsNeMo
Task3 4
API Remoting Model Weight Manager
GPU2
§5 Contention-Aware Task Scheduling
Task Type Target Server
Client Requests Buffer 2
3
Cross-Device Scheduler Per Task Target Generation Communication
API Remoting Path Optimization GPU1
1
§6 CUDA Context Consistency Guardian
Latency Predictor
Task Progress
Server Status Buffer
Reconciliation
CUDA Stream Protection
3
Framework Overview API call Optimize
cudaMemcpy 2
×
cudaMemcpy 2
√
cudaMalloc 3
cudaMalloc 4
cudaMemcpy 1
×
cudaMemcpy 1
√
cudaMemcpy 2
×
cudaMemcpy 2
√
4
Weights
Server
Task #1, Type A Unshareable cudaMallocs vMalloc 2
vMalloc 4
Malloc4
Malloc2 Hash 1
Hash 2
Hash 3
Hash 4
HtoD 1
HtoD 2
HtoD 3
HtoD 4
CUDA API Remoting Optimization
To address the communication deficiency in both loading and inference stages, we design a model weight cache manager to avoid duplicated stationary data transmission in Section 4.1 and an API remoting path manager to reduce the API-level communication frequency in Section 4.2.
Task2
Communicatio n Intensity
√
Figure 6: The tree of chunk transmission requests.
concurrent tasks, contention increases, but utilization climbs to 92%. This trade-off suggests that intelligent co-locating and scheduling can mitigate the resource idling caused by API Task Register remoting, motivating a cross-device scheduler that balances Cacheable Chunks Resources Usage Characteristics resource sharing and concurrent execution. Task3
cudaMemcpy 1
Client
Figure 5: Overview of Gleam framework.
Task1
cudaMalloc 2
√
Computation
Daemon GPU3
cudaMalloc 1 cudaMemcpy 1
Shareable
§4 CUDA API Remoting Optimization
4.1 Computation Intensity
Model Weight Manager
Since most transmitted data consists of static model weights, a natural idea is to cache these weights on server GPUs and reuse them across tasks, thereby reducing LAN bandwidth consumption and server GPU memory usage. In API remoting scenarios, the same model weights may be transmitted multiple times to the same GPU for homogeneous tasks. However, the system observes only low-level CUDA calls, such as cudaMalloc and cudaMemcpy, without high-level semantics to distinguish long-lived model weights from short-lived inputs, which makes weight-cache management non-trivial. Tasks interact with GPU memory through data movement APIs such as cudaMemcpy, whose parameters indicate the copy direction, for example, host-to-device (i.e., HtoD). We observe that a common pattern during model loading is one cudaMalloc call followed by multiple HtoD cudaMemcpy calls. Therefore, we need to identify the exact weight chunks from the parameters of these HtoD cudaMemcpy calls. After that, we can construct a shareable memory pool for homogeneous tasks. To address this challenge, we design a model weight manager with three key mechanisms: 1) weight chunk identification, 2) hash-based weight chunk retrieval, and 3) weight chunk sharing.
Memory: cache chunk recognition
Motivated by Section 2, we build an efficient API remoting Profiling system for multi-task sharing in a LAN Offline environment, as illustrated in Figure 5. To optimize communication over a single API-remoting we design a comprehensive optiGPU2connection, GPU3 GPU1 mization module, described in Section 4, which consists of two components: • Model Weight Manager (§ 4.1): Reduce data transmission by large weight identification and hash-based retrieval, while enabling flexible runtime weights sharing. • API Remoting Path Manager (§ 4.2): Reduce the overhead of frequent round-trip communications for API through exploiting synchronous execution. Furthermore, to enable efficient multi-task sharing of computation and communication, we design a contention-aware scheduling module in Section 5, which includes two parts: • Contention-Aware Latency Predictor (§ 5.1): Facilitate scheduling by online predicting the latency with task characteristics and real-time server status. • Cross-Device Scheduler (§ 5.2): Determine the offloaded server target for each task, considering both computation and communication. Besides, we include a CUDA context consistency guardian in Section 6 to ensure robust, long-lived API remoting by reconciliation after network failure and protection of crossCUDA-stream multiplexing. Gleam Workflow: First, clients specify task types to the scheduler, while servers report task progress and resource usage through heartbeats. The scheduler then predicts task execution latency based on server status and assigns a target server to each client. Subsequently, when a client begins invoking APIs, Gleam intercepts the calls and forwards them to the selected remote server with API-remoting optimizations.
4.1.1
Weight Block Identification
Intuition: Here, a block refers to the contiguous GPU memory region returned by a single cudaMalloc, while a chunk refers to a payload transferred by an HtoD cudaMemcpy. We identify a block as containing weights only if all values written by its HtoD chunks remain unchanged throughout the block’s lifetime (i.e., from the initial HtoD write until reclamation by cudaFree). For any address segment that is overwritten multiple times, we consider only the first HtoD write. To realize this design at runtime, two events must be considered after a block is allocated: 5
Memory Usage (GB)
#Cache chunk
300 Cache Chunk Sizes 225 150 75 0 25 210 215 220 225 Size of Chunk (Bytes) Figure 7: Cache chunk size distribution of compvis
Network
24 Task-wise Memory Usage sd3 18 llava-7B Total Used 12 sd3 6 evicted 0 0 5 10 15 Time (Sample Index)
Client Server
CPU
CPU
GPU
GPU (a) w/o async
(b) w/ async
the same malloc block may find a shareable copy in different malloc blocks (e.g., PyTorch tasks). Thus, Gleam enhanced this by constructing a virtual memory layer, mapping clientside memory addresses to actual shareable positions. Moreover, to support long-lived multi-task API remoting, Gleam adopts an LRU-based eviction strategy for shareable blocks. As shown in Figure 8, llava-8B gradually evicts the unreleased blocks of sd3 around sample index 6, thereby extending the model weight manager into a closed-loop module.
Figure 8: sd3 GPU Cache eviction by llava-7B
• On cudaFree interception: Before a block is reclaimed, we perform a one-time consistency check by comparing each chunk recorded from HtoD events with the current data at the corresponding address. As shown in Figure 6, a block is identified as containing weights only if all HtoD chunks match perfectly (e.g., Blocks 2 and 4).
4.2
API Remoting Path Manager
Intuition: Conventional API remoting incurs a send–receive round trip for each call, resulting in high overhead and limited throughput in LAN environments, especially over WiFi. Moreover, inherent dependencies among CUDA APIs often require strict ordering, making naive batching ineffective. Gleam addresses this issue by breaking unnecessary client-server dependencies and enabling asynchronous execution while still preserving correctness. As illustrated in Figure 9, unlike inherently asynchronous CUDA APIs (e.g., cudaLaunchKernel), the asynchronism here is introduced between the client CPU and the server CPU to eliminate accumulated network latency, rather than between the server CPU and the GPU. Among the four API categories defined in Section 2.1, we primarily focus on State Maintenance and Computation APIs because they are invoked frequently and are inherently short-lived. Gleam identifies three cases for introducing asynchronism, covering most APIs in these two categories: • API returning only error code → Basic Async: These APIs return either a success status when execution completes correctly or an error code when the client application uses them improperly. Since the API remoting middleware neither introduces nor masks such errors, Gleam can optimistically continue without blocking and report any error asynchronously if it occurs. Note: A small number of APIs whose normal execution results are encoded in return codes (e.g., cudaStreamQuery) are excluded from this case.
Hash-based Weight Chunk Retrieval
The weight blocks of a given task type are stored under a parent directory for that task type as separate, flattened HtoD chunks. Each chunk is indexed by a hash-based identifier. When a connection is established, the server sends the client the set of hash identifiers for all resident weight chunks associated with the client’s task type. Before forwarding each HtoD chunk to the server, the client first computes a hash of the source data. If the hash appears in the received hash set, the client sends only reuse metadata, including the hash and chunk size, and the server retrieves the corresponding data locally from disk. Otherwise, the client transmits the full chunk. If this source data is later identified as model weight, it can be reused in future connections. Runtime Example: To verify the feasibility of our design, we run a demo task, sd-compvis, and collect the cached chunk profile shown in Figure 7. The results show that chunk sizes range from KB to MB, and 3GB of chunk transmission can be eliminated using <1K lightweight identifiers. 4.1.3
Async API call
CPU
Figure 9: Illustration of asynchronous API execution.
• On cudaMemcpy interception: For each HtoD event whose destination falls within a monitored block, we record the chunk bytes, along with the corresponding address and offset, on disk, while forwarding the call as usual.
4.1.2
Sync API call
CPU
Weight Chunk Sharing
Beyond eliminating repeated transfers within a single execution, the cache also enables online weight sharing across multiple invocations of the same application. Once a weight block has been identified and cached on the server, its GPU memory is placed into a global shareable pool instead of being released immediately, thereby facilitating subsequent executions of the same task type. However, directly reusing the shareable block and skipping the cudaMalloc and cudaMemcpy traces will incur risks. As shown in the right part of Figure 6, two HtoD chunks within
• API maintaining trivial states → Local Simulation: These APIs have simple semantics that can be reproduced using lightweight client-side state, such as stack/queue operations (e.g., cudaPopCallConfiguration and cudaPushCallConfiguration) and cacheable queries (e.g., cudaGetDevice and cudaSetDevice). The client simulates the corresponding state updates locally while issuing the remote call to preserve consistency. As a result, 6
default result
Client Client App
API Intercept
Server Check Status
Async？
Basic Async
Local Simulation
Async？
Forward
API result
API params
Wi-Fi TCP/IP
Wi-Fi TCP/IP
Server GPU
capacity on server j. Let Ψ = {⟨ j, r, g, u⟩|r > 0, 0 < g < Ḡ j , u ∈ [0, 1]} be current active tasks on all servers, where r stands for estimated remaining time without contention and g stands for GPU memory requirement. u represents the resource occupancy of this task, detailed in Section 5.1.1. In addition, let Liinfer and Liload stand for latency predictors for inference and loading stages with current states Ψ and incoming task pattern X. Finally, we use L̄iinfer and L̄iload , the latency without contention, to normalize the predicted latency, representing throughput independent of task type: 1 M Liinfer (Ψ, X) Liload (Ψ, X) + (1) min ∑ X M L̄iinfer L̄iload i=1
Batched Prefetch
API Execute
Receive
Figure 10: Workflow of API remoting path manager. it can directly produce the correct return values from local state, thereby enabling asynchronous execution. • APIs applying resource handles → Batched Prefetch: These APIs return resource handles that are often consumed immediately by subsequent calls, thereby creating blocking dependencies. Gleam removes them through batched pre-creation and local caching: the server creates multiple resources of the same type in a single request, and the client serves subsequent requests from the local handle cache by wrapping cached handles in API responses until exhausted. The API requests that draw from the handle cache must still be forwarded to the server, because some parameterized handles (e.g., cublasLtMatrixLayoutCreate) need post-modification (e.g., cublasLtMatrixLayoutSetAttribute). Accordingly, we design an API remoting path manager, whose workflow is summarized in Figure 10. When an API is intercepted on the client, Gleam determines whether it can be transformed into one of the three asynchronously executable forms. If so, the client immediately generates a local success response Meanwhile, the original request is still forwarded to the server to preserve consistency with the remote execution state. Finally, the server executes the APIs and returns responses either asynchronously or synchronously, as appropriate. Detailed information about APIs-cases relationships is provided in Appendix A.4.
5
s.t. ∀ j, G j (Ψ, X) ≤ Ḡ j
(2)
N
∀i, ∑ xi, j = 1, xi, j ∈ {0, 1}
(3)
j=1
Equation (2) gives the maximum available GPU memory on each server during scheduling. Equation (3) means that each task can only be executed on one specific server.
5.1
Contention-Aware Latency Predictor
5.1.1
Resource Contention Model
We model the latency increase due to multi-task contention using a congestion model built on two core components: the resource occupancy and the latency increase factor. • Resource Occupancy (ui ): It quantifies the extent to which a single task monopolizes a resource and can be obtained through offline, standalone single-task measurements. For communication, it is defined as the ratio between the bandwidth consumed by API request transmission and the total available bandwidth capacity. For computation, it is directly defined by GPU utilization. • Latency Increase Factor (α): It adjusts a task’s predicted latency under concurrent execution. As Figure 11 shows, for tasks with resource occupancies u1 , . . . , un , Gleam assumes latency remains stable without severe contention as long as total occupancy is below full saturation, while scaling up as the sum of ui increases, yielding the Equation (4), where γ ≥ 1 stands for the penalty factor. ( )
Contention-Aware Task Scheduling
In this section, we first formalize the contention-aware task scheduling problem. To solve it and enable efficient task execution with multiple clients and servers, we then propose a lightweight latency predictor for computation and communication in Section 5.1. Finally, we introduce the Gleam scheduling algorithm in Section 5.2. Objective: For a distributed local-area system consisting of multiple clients and servers, we aim to improve overall throughput while maintaining acceptable task latency. Problem Formulation: Suppose that we have N servers and have received M GPU tasks in the recent interval T , e.g., 5s. Let φi denotes task i, (1 ≤ i ≤ M), Φ = {φi }M be the task queue of the M tasks. Let X ∈ {0, 1}M×N be the task pattern to be solved, whose element xi, j (1 ≤ j ≤ N) indicating whether task i is processed on server j. Let G j be a function that computes the memory cost on server j, while Ḡ j be the memory
n
α = max 1, γ ∑ ui
(4)
i=1
Moreover, as shown in Table 5, the communication and computation occupancy patterns2 vary across model loading and inference stages, and should be tackled differently: • Communication: The primary transmission occurs during the model loading stage, where the occupancy ranges from 68.3% to 80.4%, while inference imposes only modest 2 Reported occupancies were obtained from standalone single-task mea-
surements in Table 17.
7
Table 5: Occupancy characteristics of communication and computation for model loading and inference stages.
Task 1 alone
Task 1
Task 2 alone
Task 2
Waiting
With Contention
Task 1
model-loading inference
Task 2
Communication Occu.
Computation Occu.
68% to 80% around 8%
<3%, omit 13% to 90%
Figure 11: Resource contention illustration. Bandwidth (Mbps)
1024 768 512 256 00
Bandwidth over Time
25
50
75
100 Time (s)
125
llava-7B wo cache llava-7B with cache Inference Start
150
175
200
Online Latency Predictor
Based on the resource contention model, we build a lightweight online latency predictor. We estimate task overlap and adjust the overlapping portions by multiplying them by a latency increase factor. Each task φi , i > 0 under the loading or inference stage can be described by a tuple (ui , ri ), where ui denotes occupancy and ri denotes remaining time without contention. After sorting φi in descending order, latency is predicted as shown in Equation (5):
i=1
u2
Task 1
u1
r3
r2
r1
Real
Predict
2.3 2.2 2.1 100
110
120
cudaSynchronize Count
Figure 14: Remain time prediction: running sd-compvis.
regarded as a monitor of inference progress. We track the number of sync calls, Cinvoked , and apply a linear approximation on r with it. Figure 14 shows that though sync calls may not be evenly distributed, their patterns tend to be periodic3 , enabling error-bounded prediction of r. To aid runtime latency prediction, we measure standalone latency for the two stages, i.e., Lload , Linfer , combined with Qtotal and the total number of cudaStreamSynchronize calls during inference, Ctotal . By substituting the estimated proportion and latency without contention, wecan get the remaining sent × Lload and the compucommunication time rload 1 − QQtotal tation time rinfer = 1 − CCinvoked × Linfer . total
• Computation: GPU utilization remains negligibly low during model loading because the GPU is often stalled by remote memory-copy. During inference, computation becomes dominant and varies substantially across different tasks due to intrinsic characteristics.
|Φ|
Task 2
u : Resource occupancies r : Remaining time
Figure 13: Example of runtime latency prediction.
communication demand for non-weights API forwarding, typically with an average occupancy around 8%. Figure 12 exemplify this using llava-7b.
5.2
Cross-Device Task Scheduler
As the complexity of brute-force search for M tasks and N servers is O(N M ), Gleam adopts an efficient two-phase scheduling algorithm, illustrated in Figure 15. We also provide the complete pseudocode in the Appendix, as shown in Algorithm 1. As input, the task configuration specifies each task’s GPU memory requirement, along with its computation and communication occupancy, while the server status indicates the set of tasks currently executing on that server. Let Φ be the queues of tasks and Ψ be the set of servers. The procedures of the inference stage and the loading stage are as follows: • The first phase focuses on tasks with cache on some servers, which can directly starts inference stage, determining their dispatch based on predicted computation latency, while grouping tasks of the same type to enhance weight reuse.
i
Latency = ∑ α( ∑ u j ) × (ri − ri−1 ), r0 = 0,
u3
current
Figure 12: llava-7B bandwidth consumption w/wo cache.
5.1.2
Task 3
Remain time (ms)
(b) Dense trans.
(a) Sparse trans.
(5)
j=1
where α can be obtained by Equation (4). Figure 13 shows a toy example with 3 active stages, φ1 , φ2 , φ3 whose remaining times satisfy r1 > r2 > r3 . Then we apply Equation (5) and predict the latency as r3 · α(u1 + u2 + u3 ) + (r2 − r3 ) · α(u1 + u2 ) + (r1 − r2 ) · α(u1 ). The prediction relies on knowledge of the occupancy u and the remaining time r for each active stage. While the method for obtaining u was described earlier, we now focus on determining r, which can be estimated with standalone cost Lload , Linfer , and 2 easily obtained runtime states. • Loading: The ratio of total bytes to be transmitted Qtotal to the sent bytes Qsent can be used for estimation. In particular, the total bytes to be sent for a model remain fixed during the model-loading stage.
• The second phase handles tasks without cache or skipped by the first stage due to computation contention, making dispatch decisions according to the network contention. Specifically, in Phase 1, tasks are considered in descending order of their GPU memory requirements. For each task
• Inference: GPU programs typically repeatedly invoke cudaStreamSynchronize during execution, which can be
3 Periodic API patterns are common in modern AI tasks, e.g., , LLM decoding and Stable Diffusion sampling.
8
Task #1 #1 Task Server #1
Phase 1
Phase 2
Task Type
Task Type
2x Type1→ S1
Type2→ S1
Predict cost: 12 s
No cache | mem limit
3x Type1→ S2
Type2→ S2
Predict cost: 15 s
S2 GPU saturated
Skip Type2
Server ID
Task#1 #1 Task #1 Task
Server ID
Input
Type2→ S3 Has cache
Type2→ S4 20 s infer, 100s load
Output
Server side: sent + client receive ? API Call Start
Task Server Pairs Task incomplete
A
Response Buffer
Network
B
Remove one
Next request SEQ
Resend all
Reconnect Event
Server Process
C
Client side: sent + server receive ? Request Buffer
Figure 15: Illustration of the two-phase scheduling algorithm, where 3 tasks with type 1 are scheduled to server 2 in Phase 1, and 1 task with type 2 will offload to server 4 in Phase 2.
∑ Pcf ≤ 1,
task f ∈s
(g + 1) × Pc +
B
Server Receive
C
Server Send
D
Client Receive
D
API Call Complete
Reconnect Event
until they are acknowledged by the receiver, as illustrated in Figure 16. Concretely, each sender maintains a message buffer: it appends each outgoing request to the buffer and removes it once the corresponding ACK is received. Details: • Bounded State Reservation: To avoid unbounded storage overhead, the client reserves only a bounded number of inflight states, and delays further transmission until #in-flight requests falls below a predefined threshold.
∑ Pcf > 1, (6)
task f ∈s
We add as many homogeneous tasks as possible to server s, while preventing accumulated occupancy from growing without limit. After that, the corresponding latency is estimated. If g > 0, the resulting task group is assigned to the server with the minimum predicted latency, and the server status is updated accordingly. Otherwise, the task will be temporarily skipped, but still left in Φ. In Phase 2, tasks are processed in descending order of GPU memory demand similarly. For each task, the algorithm checks whether the current bandwidth utilization permits additional occupancy. If servers with sufficient memory and have no cache are available, assign the task to the one with the minimum predicted computation latency, and both bandwidth utilization and server status are updated; otherwise, add the task to Φwait . Finally, the algorithm outputs the dispatched task-server pairs D and the suspended task set Φwait . For the next scheduling, we obtain Φ from incoming tasks and Φwait . In summary, Phase 1 groups homogeneous tasks to improve GPU utilization and reduce redundant memory operations, while Phase 2 mitigates network bottlenecks through contention-aware dispatch of model-loading tasks. Together, these strategies significantly lower end-to-end latency and boost throughput under multi-tenant workloads.
6
ACK Response
Client Send
Figure 16: Illustration for CUDA API remoting reconciliation.
with computation occupancy Pc , the algorithm evaluates the servers that satisfy the memory constraint and has cache for the task, determining the grouping size g for tasks of the same type such that: g × Pc +
Remove one Resend all
Network
A
• Bidirectional Acknowledgment: Since a server response inherently acknowledges the corresponding client request, Gleam further treats the receipt of the i+N-th request as the acknowledgment of the buffered i-th response due to the bounded reservation. ▶ Protection of Cross-CUDA-stream Multiplexing: Several modern CUDA stream features impose strict constraints on CUDA context usage across multiple streams. For example, during CUDA graph capture, context-level APIs (e.g., memory allocation and synchronization; see Table 14) are prohibited when they are issued from other CUDA streams. To address this issue, Gleam employs a global gating to block CUDA context-level APIs during capture, thereby ensuring safe graph construction. This overhead is acceptable because graph capture is typically shorter than overall execution.
7 7.1
Experiments Experimental Setups
▶ Hardware Platform: For servers, we use 4 physical machines, equipped with 1 GPU with various specifications (e.g., RTX A4500, GeForce RTX 4090), respectively. They are interconnected through an Ethernet switch. For clients, we use a machine equipped with a 20-core Intel i7-14700 CPU for emulation. The machine is connected to the server through an advanced router with Wi-Fi 6. The maximum upload bandwidth is 1000 Mbps. The detailed configurations of our servers are listed in Table 16, Appendix A.2. ▶ CUDA Tasks: We use 7 widely used GPUaccelerated real-world tasks. Among them, llama-8B, llama-3B [4], sd-compvis, sd3-medium-t5 [8], and whisper-large-v3 [9] are directly built by CUDA API through ggml, while llava-7B [5] and Fourier Neural Operator [6] (i.e., fno) are built by PyTorch. These tasks include language models, image generation, and scientific
CUDA Context Consistency Guardian
To enable robust, long-lived, mixed-task sharing in Gleam via API remoting, i.e., maintaining CUDA context consistency between client and server over time, we design two mechanisms to handle network failures and prevent interference across CUDA streams ▶ Reconciliation after Network Failure: Gleam supports asynchronous API remoting and uses a duplex TCP connection for client-server communication, which may leave multiple APIs incomplete when the connection breaks. Therefore, the key idea behind Gleam reconciliation is that both senders, i.e., , the client and the server, retain all transmitted messages 9
llama3b-ggml
Latency (s)
103 102
103 102
3.1x
1.6x
101
100 Ethernet
WIFI6
Server Local Exec. GVirtuS cricket+SR DGSF Gleam llama8b-ggml 3 10diffusion-compvis-ggml 103 diffusion-sd3-ggml 103 llava7b-pytorch 1.5x
2.3x
101 100 Ethernet
102 101
102
100 Ethernet
WIFI6
1.4x
1.2x
101
2 3.0x 10
1.8x
101
100 Ethernet
WIFI6
2 1.5x 10
1.2x
WIFI6
fno-pytorch
103
101
100 Ethernet
WIFI6
1.0x
100 Ethernet
2.3x
WIFI6
Figure 17: Comparison on efficiency of API remoting with SOTA baselines.
Gleam
#Servers
Chose GPU IDs
#Client
#Task Type
1-GPU 2-GPU 3-GPU 4-GPU
1 2 3 4
(0) (0, 1) (0, 1, 2) (0, 1, 2, 3)
3 7 10 15
3 4 6 6
FGD
Latency (s)
Case
500 400 300 200 100 1
computations. We measured the resource occupancy (computation/communication), cost, and cacheable chunk sizes of these tasks with different GPUs without contention, which are shown in Table 17, Figure 27 and 28 in Appendix A.2. ▶ Configurations of Clients and Servers: We designed 4 sets of experiments in which the number of GPUs was gradually increased from 1 to 4, while the number of concurrent simulated clients scaled from 3 to 15. The diversity of task types was also expanded from 3 to 6. In each experiment, every client continuously executed multiple instances of a task chosen from the corresponding type range. The details are shown in Table 6. ▶ Baselines: We first use 3 baselines for comparison on effect of API remoting: • GVirtuS [25]: Pioneer work for GPGPU API remoting.
25 20 15 10 5 01
Mudi
Throughput
Table 6: Configurations of 4 experiment setups.
2
3
# GPUs
4
2
3
# GPUs
4
Figure 18: Average end-to-end latency and request throughput as the number of GPUs varies. • Queuing delay: The waiting time each client incurs before being assigned to a server. • Makespan: The completion time of the last client, indicating how well the scheduling policy balances the tasks. • Bandwidth: The actual occupied bandwidth resources measured on each server at different time stamps.
7.2
API Remoting Efficiency
Firstly, we perform several single-task measurements to show the efficiency of API remoting optimization in Figure 17, under both Wi-Fi and Ethernet connections. ▶ Performance Gain: Across all six applications, Gleam consistently achieves 1.4× to 24.2× speedup against SOTA baselines. Specifically under Wi-Fi, for sd-medium-t5-ggml, Gleam attains nearly 54× speedup over GVirtuS, and still outperforms cricket+SR and DGSF by about 8.4× and 3.9×, respectively. This advantage comes from the fact that GVirtuS adopts a naive API remoting design and therefore incurs frequent API communication, whereas cricket+SR and DGSF lack model caching support and still introduce redundant overhead in the model loading stage. ▶ Connection Type: Both Ethernet and Wi-Fi introduce additional latency compared to local execution. For the same sd3 program without optimization (GVirtuS), Wi-Fi and Ethernet incur approximately 81× and 31× additional overhead, respectively, with Wi-Fi being more severely affected due to its higher delay and instability. However, as communication optimization techniques are applied, the performance gap among local, Ethernet, and Wi-Fi environments gradually narrows, partially mitigating the disadvantage of Wi-Fi. When all the optimization methods are enabled, the additional overhead is reduced to about 53% for Wi-Fi and 19% for Ethernet.
• cricket+SR [70]: Enables asynchronous execution for resource create API through shadow resource (SR). • Disaggregated gpus for serverless function(DGSF) [22]: Eliminates #remoting APIs by resource handles prefetching and GPU states local maintenance. Then we use 2 SOTA baselines enhanced with our API remoting optimizations for overall comparison: • Fragmentation Gradient Descent (FGD) [72]: Scheduling approach for cloud computing, which places tasks to target that minimizes the system resource fragmentation. • Mudi [16]: Solution for edge resource multiplexing, which assigns tasks by avoiding distributed resource contention. ▶ Evaluation Metrics: To assess the performance of our system, we adopt the following metrics: • Throughput: #requests processed per minute, normalized by task executing latency without contention to eliminate task type correlation. • Latency: Average end-to-end latency, representing the processing speed experienced by a client request. 10
2
3
# GPUs
4
# Concurrent Clients
Figure 19: Overall queuing delay and makespan.
20
40
FGD
60
2 GPUs inference latency (s)
1.00 0.75 0.50 0.25 0.000
Figure 22: Overhead of cross-stream protection. Homogeneous setting only includes llama-3b while the Hybrid setting involves the other 3 types of tasks.
Mudi
Used Memory (GB)
20
40
60
4 GPUs inference latency (s)
Figure 20: Cumulative distribution (CDF) of per-task inference time for 2 GPUs and 4 GPUs cases
24 18 12 6 00
llama-3B
sd3
llava-7B
Total Used
Shareable Task-wise Memory Lifespan
50
4.0
25
25
50
75 100 125 150 Time (Sample Index)
175
100
200
Time (s)
300 0
Time (s)
Figure 23: Memory and GPU utilization for the 4-GPU case. 100 75 50 25 00
GPU0: RTX A4500
100
200
100 GPU1: GeForce RTX 4090 75 50 25 300 00 100 200 300
Time (s) Time (s) Figure 24: Bandwidth occupancy for the 4-GPU case.
200
Figure 21: Shareable GPU memory alternative eviction.
7.3
75
8.0 0.00
GPU Utilization (%)
24.0GPU 1: GeForce RTX 4090100 18.0 75 12.0 50 6.0 25 0.00 100 200 300 0
100
12.0
Bandwidth Occupancy (Mbps)
Memory Usage (GB)
compvis
GPU 0: RTX A4500
16.0
Used Memory (GB)
Distribution ratio
1.00 0.75 0.50 0.25 0.000
Gleam
Proportion of Overhead Hybrid Tasks 5 4 3 2 1 2 4 6 0 # Concurrent Clients
Overhead Ratio (%)
1
500 400 300 200 100 0
GPU Utilization (%)
4
Absolute Protection Overhead 500 Homogeneous Tasks 5 400 4 300 3 200 2 100 1 0 2 4 6 0
Overhead Ratio (%) Overhead (ms)
3
Mudi
GPU Utilization (%) Used Memory (GB)
2
# GPUs
600 400 200 0
Overhead (ms)
1
FGD Makespan (s)
Waiting Time (s)
Gleam
30 20 10 0
7.4
Micro Experiments
End-to-End Performance ▶ GPU Memory Eviction: Figure 21 shows the GPU memory trace for a 4-mixed-task pressure test on a single RTX 4090 GPU. The unreleased task-wise shareable memory size continuously increases or decreases as tasks start or complete, meeting the expectation in Section 4.1. For example, around sample index 50, the unused memory of compvis and llava-7B is successively evicted w.r.t. their last used time.
▶ Overall Performance: We first compare average end-toend latency and request throughput across different GPU scales, as shown in Figure 18. Across all GPU counts, Gleam consistently achieves the lowest latency by up to 1.56× reduction and the highest throughput by up to 1.79× improvement, demonstrating that the contention-aware scheduling and API remoting optimization effectively mitigate both communication and computation delays. ▶ Scheduling Efficiency: We further report the request queuing delay and total makespan in Figure 19. For the 4-GPU case, Gleam significantly reduces up to 2.88× queuing delays compared to FGD and Mudi, leading to up to 1.76× shorter makespan and more balanced completion times across clients. This reduction stems from an important factor: optimized handling of computation and communication contention during scheduling and API remoting. ▶ Inference Latency: We then analyze the inference latency distributions per task to assess latency stability and tail behavior, as illustrated in Figure 20. The cumulative distribution functions for both the 2-GPU case and the 4-GPU case show that Gleam completes a larger fraction of tasks with lower inference latency and exhibits a shorter tail compared to the baselines, indicating both faster median latency and improved predictability with resource contention.
▶ Cross-Stream Protection: Figure 22 shows the cost of homogeneous and hybrid configurations for the protection mechanism in Section 6. The overhead and ratio to end-toend latency increase as #concurrency gains, while the hybrid one issues less overhead due to less usage of features (i.e., CUDA graph) that need to be protected across streams. Meanwhile, Gleam would prevent unbounded overhead gain through contention-aware scheduling. ▶ Resource Utilization: In Figure 23 and 24, we show the evolution of GPU memory usage, utilization, and bandwidth occupancy over time when processing API remoting connections for GPU 0 and GPU 1 in the 4-GPU case. The dynamic patterns of all metrics reflect the runtime adjustments made by the scheduler. The memory usage increases as new tasks arrive, aligned with the rise in GPU utilization, while GPU 1 (GeForce RTX 4090) consumes more bandwidth than GPU 0 (RTX A4500) due to more resources and assigned tasks. 11
103
0:Naive 1:+Async 2:+Batch
3:+Simulate 4:+W. Cache
8
101
ut lay cy an Laten ThroughpQ. De Makesp
100
-7B -8B llava llama Figure 25: Ablation results Figure 26: Ablation results on on API path manager and different types of remoting optimization. task scheduler in Gleam. Table 7: Per-invocation overhead of Gleam scheduling. Configurations
1-GPU
2-GPU
3-GPU
4-GPU
Average overhead (µs)
77.94
74.68
84.32
81.28
Table 8: Overhead of Gleam reconciliation running llava-7B. Max pending batch
2
4
8
16
Reserving request size (KB) Performance degradation
0.98 9.13%
1.12 6.24%
1.22 4.16%
1.42 3.05%
7.5
Ablation Study
Figure 25 presents the performance comparison of Gleam when the complete API path manager and the contentionaware scheduler are individually removed on 4-GPU case. When only the basic async APIs are reserved, the average end-to-end latency increases by 7.9×, and throughput decreases by 66%. Replacing the contention-aware scheduler with a fairness-based one that is agnostic to contention results in less severe degradation, with queuing delay increasing by around 6.2×. Then, in Figure 26, we pick llama-8B-ggml and llava-8B-PyTorch, and gradually add API remoting optimization techniques from the naive version, achieving a consistent decreasing tendency. These results highlight that both modules are indispensable to Gleam’s efficiency. Moreover, the API path manager is designed to be independent, allowing seamless integration with any schedulers.
7.6
Related Work
Distributed Communication Optimization: Motivated by the limited bandwidth, research focuses on optimizing the volume of point-to-point data by compressing transmitted data [10, 17, 21, 29, 43, 46, 77], employing adaptive configurations [19, 62, 74, 80], or implementing system-level enhancements, including in-network computation [36, 38, 45], communication-computation overlap [12, 63, 68], and optimizations of mechanisms [11, 15, 23, 31]. Our work optimizes distributed multi-GPU resource sharing over LANs by minimizing data transmission via dynamic caching and modeling contention in concurrent multi-channel transfers. Multi-task GPU Serving: Previous studies have explored several approaches to improve system throughput, including GPU resource awareness [41, 50, 71], general kernel scheduling techniques [24, 27, 28, 39, 64, 73, 85], and task-specific optimizations [34,40,42,86]. API remoting enables fine-grained GPU sharing via local API interception [28, 35, 64, 73, 79] or RDMA-based forwarding [20, 22, 65, 70]. Our work introduces a novel approach that intercepts and forwards API calls over the LAN (especially on Wi-Fi), while integrating communication and computation to enable efficient serving. Coordination on Heterogeneous Resources: It investigates how to efficiently deploy and orchestrate GPU tasks across devices with diverse resources, with the goals of reducing monetary cost [13] and meeting Service Level Objectives (SLOs) [16, 33]. Numerous works primarily focus on task scheduling [26, 32,37, 51,59, 69,72, 78,82] to improve overall system efficiency, while others consider heterogeneity within one resource like memory [35, 57, 60, 61, 75]. Distinct from prior efforts that primarily focus on allocating tasks among heterogeneous GPUs, our work takes a broader end-to-end system perspective, explicitly incorporating network resources into the coordination framework.
102
1 0
Performance (s)
Normalized time
2
Gleam Gleam-w/o-API-opt Gleam-fair-schedule
9
Conclusion
We presented Gleam, a novel framework for efficient GPU sharing across heterogeneous devices in LAN environments. We addressed the often-overlooked network bottlenecks of CUDA API remoting by introducing fine-grained path management with model weight caching, and by designing a contention-aware runtime scheduler to balance network and GPU resources. Extensive evaluations on diverse GPUs and AI workloads demonstrated that Gleam achieves 1.4×24.2× speedup in API remoting efficiency and improves task throughput by up to 1.79×. The results confirm the practicality of distributed GPU sharing and highlight its potential to enable ubiquitous AI inference on personal edge devices. Our future work will extend to addressing the data security concerns during edge GPU sharing and developing general collaborative inference techniques with distributed edge GPUs.
Overhead Quantification
▶ Gleam Scheduling: The task scheduler is invoked periodically (e.g., 1 s) to check whether the queuing tasks can be scheduled. As shown in Table 7, the average time per scheduling invocation is negligible, ranging from 84.32 µs with the 3-GPU case to 74.68 µs with the 2-GPU case, justified the efficiency of Gleam among local LANs. ▶ Gleam Reconciliation: The CUDA context reconciliation module buffers at most unacknowledged #max_pending_batch API requests and responses for consistency recovery. As shown in Table 8, for llava-7B-PyTorch, setting max pending size as 8 would incur at most 4.16% degradation, while the reserved request size is negligible. 12
References
[15] Jingrong Chen, Yongji Wu, Shihan Lin, Yechen Xu, Xinhao Kong, Thomas Anderson, Matthew Lentz, Xiaowei Yang, and Danyang Zhuo. Remote procedure call as a managed system service. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 141–159, 2023.
[1] Cuda toolkit 12.4, 2024. [2] ggml: Tensor library for machine learning, 2025. [3] grpc: A high performance, open source universal rpc framework, 2025.
[16] Wenyan Chen, Chengzhi Lu, Huanle Xu, Kejiang Ye, and Chengzhong Xu. Multiplexing dynamic deep learning workloads with slo-awareness in gpu clusters. In Proceedings of the Twentieth European Conference on Computer Systems, pages 589–604, 2025.
[4] llama.cpp: Llm inference in c/c++, 2025. [5] llava-v1.6-mistral-7b-hf, 2025.
[17] Yihua Cheng, Ziyi Zhang, Hanchen Li, Anton Arapin, Yue Zhang, Qizheng Zhang, Yuhan Liu, Kuntai Du, Xu Zhang, Francis Y Yan, et al. {GRACE}:{LossResilient}{Real-Time} video through neural codecs. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 509–531, 2024.
[6] Nvidia physicsnemo: open-source deep-learning framework for building, training, fine-tuning, and inferring physics ai models, 2025. [7] Protocol buffers, 2025. [8] stable-diffusion.cpp: Diffusion model(sd,flux,wan,...) inference in pure c/c++, 2025.
[18] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344– 16359, 2022.
[9] whisper.cpp: High-performance inference of openai’s whisper automatic speech recognition (asr) model, 2025. [10] Hyunho Ahn, Munkyu Lee, Sihoon Seong, Gap-Joo Na, In-Geol Chun, Blesson Varghese, and Cheol-Ho Hong. Scissionlite: Accelerating distributed deep learning with lightweight data compression for iiot. IEEE Transactions on Industrial Informatics, 20(10):11950–11960, 2024.
[19] Kuntai Du, Yuhan Liu, Yitian Hao, Qizheng Zhang, Haodong Wang, Yuyang Huang, Ganesh Ananthanarayanan, and Junchen Jiang. Oneadapt: Fast adaptation for deep learning applications via backpropagation. In Proc. ACM SoCC 2023. [20] Niklas Eiling, Jonas Baude, Stefan Lankes, and Antonello Monti. Cricket: A virtualization layer for distributed execution of cuda applications with checkpoint/restart support. Concurrency and Computation: Practice and Experience, 34(14):e6474, 2022.
[11] Congkai An, Huanhuan Zhang, Shibo Wang, Jingyang Kang, Anfu Zhou, Liang Liu, Huadong Ma, Zili Meng, Delei Ma, Yusheng Dong, et al. Tooth: Toward optimal balance of video {QoE} and redundancy cost by {FineGrained}{FEC} in cloud gaming streaming. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 635–651, 2025.
[21] Shiwei Fang, Jin Huang, Colin Samplawski, Deepak Ganesan, Benjamin Marlin, Tarek Abdelzaher, and Maggie B. Wigness. Optimizing intelligent edge-clouds with partitioning, compression and speculative inference. In MILCOM 2021 - 2021 IEEE Military Communications Conference (MILCOM), pages 892–896, 2021.
[12] Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao, Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. Crux: Gpu-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024.
[22] Henrique Fingler, Zhiting Zhu, Esther Yoon, Zhipeng Jia, Emmett Witchel, and Christopher J Rossbach. Dgsf: Disaggregated gpus for serverless functions. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 739–750. IEEE, 2022.
[13] Tzu-Tao Chang and Shivaram Venkataraman. Eva: Costefficient cloud-based cluster scheduling. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1399–1416, 2025.
[23] Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. Rdma over ethernet for distributed training at meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 57–70, 2024.
[14] Benjamin Charlier, Jean Feydy, Joan Alexis Glaunes, François-David Collin, and Ghislain Durif. Kernel operations on the gpu, with autodiff, without memory overflows. Journal of Machine Learning Research, 22(74):1– 6, 2021. 13
[24] Guin Gilman and Robert J. Walls. Refine: A reactive and fine-grained scheduling framework for concurrency on general purpose gpus. In Proceedings of the 37th ACM Symposium on Parallelism in Algorithms and Architectures, SPAA ’25, page 193–209, New York, NY, USA, 2025. Association for Computing Machinery.
[33] Beomyeol Jeon, Chen Wang, Diana Arroyo, Alaa Youssef, and Indranil Gupta. A house united within itself: Slo-awareness for on-premises containerized ml inference clusters via faro. In Proceedings of the Twentieth European Conference on Computer Systems, pages 524–540, 2025.
[25] Giulio Giunta, Raffaele Montella, Giuseppe Agrillo, and Giuseppe Coviello. A gpgpu transparent virtualization component for high performance computing clouds. In European Conference on Parallel Processing, pages 379– 391. Springer, 2010.
[34] Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z. Morley Mao. Compute or load kv cache? why not both?, 2025. [35] Jaehoon Jung, Jinpyo Kim, and Jaejin Lee. Deepum: Tensor migration and prefetching in unified memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2023, page 207–221, New York, NY, USA, 2023. Association for Computing Machinery.
[26] Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A GPU cluster manager for distributed deep learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 485–500, Boston, MA, February 2019. USENIX Association.
[36] Nichoas Kaminski, Irene Macaluso, Emanuele Di Pascale, Avishek Nag, John Brady, Mark Kelly, Keith Nolan, Wael Guibene, and Linda Doyle. A neural-networkbased realization of in-network computation for the internet of things. In 2017 IEEE International Conference on Communications (ICC), pages 1–6, 2017.
[27] Bing-Shiun Han, Tathagata Paul, Zhenhua Liu, and Anshul Gandhi. Kace: Kernel-aware colocation for efficient gpu spatial sharing. In Proceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ’24, page 460–469, New York, NY, USA, 2024. Association for Computing Machinery.
[37] Alind Khare, Dhruv Garg, Sukrit Kalra, Snigdha Grandhi, Ion Stoica, and Alexey Tumanov. {SuperServe}:{Fine-Grained} inference serving for unpredictable workloads. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 739–758, 2025.
[28] Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, Carlsbad, CA, July 2022. USENIX Association.
[38] Somayeh Kianpisheh and Tarik Taleb. A survey on in-network computing: Programmable data plane and technology specific applications. IEEE Communications Surveys & Tutorials, 25(1):701–761, 2023.
[29] Zhiwei Hao, Guanyu Xu, Yong Luo, Han Hu, Jianping An, and Shiwen Mao. Multi-agent collaborative inference via dnn decoupling: Intermediate feature compression and edge learning. IEEE Transactions on Mobile Computing, 22(10):6041–6055, 2023.
[39] Jiho Kim, John Kim, and Yongjun Park. Navigator: Dynamic multi-kernel scheduling to improve gpu performance. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1–6, 2020.
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. [31] Changho Hwang, KyoungSoo Park, Ran Shu, Xinyuan Qu, Peng Cheng, and Yongqiang Xiong. {ARK}:{GPUdriven} code execution for distributed deep learning. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 87–101, 2023.
[40] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023.
[32] Akshay Jajoo, Y. Charlie Hu, Xiaojun Lin, and Nan Deng. A case for task sampling based learning for cluster job scheduling. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 19–33, Renton, WA, April 2022. USENIX Association.
[41] Seonho Lee, Amar Phanishayee, and Divya Mahajan. Forecasting gpu performance for deep learning training and inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 493–508, 2025. 14
[42] Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based on kv cache management, 2025.
Microsecond-Scale tasks. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 1–18, Renton, WA, April 2022. USENIX Association.
[43] Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. THC: Accelerating distributed deep learning using tensor homomorphic compression. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1191–1211, Santa Clara, CA, April 2024. USENIX Association.
[52] Richard Tran Mills, Mark F Adams, Satish Balay, Jed Brown, Alp Dener, Matthew Knepley, Scott E Kruger, Hannah Morgan, Todd Munson, Karl Rupp, et al. Toward performance-portable petsc for gpu-based exascale systems. Parallel Computing, 108:102831, 2021. [53] Andrew Myers, A Almgren, Ligia Diana Amorim, J Bell, Luca Fedeli, Lixin Ge, Kevin Gott, David P Grote, M Hogan, Axel Huebl, et al. Porting warpx to gpuaccelerated platforms. Parallel Computing, 108:102833, 2021.
[44] Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024.
[54] Vinod Nigade, Pablo Bauszat, Henri Bal, and Lin Wang. Jellyfish: Timely inference serving for dynamic edge networks. In 2022 IEEE Real-Time Systems Symposium (RTSS), pages 277–290, 2022.
[45] Ming Liu, Liang Luo, Jacob Nelson, Luis Ceze, Arvind Krishnamurthy, and Kishore Atreya. Incbricks: Toward in-network computation with an in-network cache. SIGARCH Comput. Archit. News, 45(1):795–809, April 2017.
[55] Corey J Nolet, Victor Lafargue, Edward Raff, Thejaswi Nanditale, Tim Oates, John Zedlewski, and Joshua Patterson. Bringing umap closer to the speed of light with gpu acceleration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 418– 426, 2021.
[46] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38–56, 2024.
[56] WP NVIDIA. Nvidia nvswitch: The world’s highestbandwidth on-node switch. 2018. [57] Arthi Padmanabhan, Neil Agarwal, Anand Iyer, Ananthanarayanan, and et.al. Gemel: Model merging for memory-efficient,real-time video analytics at the edge. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 973–994, 2023.
[47] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022. [48] Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023.
[58] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
[49] Shaonan Ma, Teng Ma, Kang Chen, and Yongwei Wu. A survey of storage systems in the rdma era. IEEE Transactions on Parallel and Distributed Systems, 33(12):4395– 4409, 2022.
[59] Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. {CASSINI}:{Network-Aware} job scheduling in machine learning clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1403–1420, 2024.
[50] Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica. Skyserve: Serving ai models across regions and clouds with spot instances. In Proceedings of the Twentieth European Conference on Computer Systems, pages 159–175, 2025.
[60] Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li. Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 598–611. IEEE, 2021.
[51] Sarah McClure, Amy Ousterhout, Scott Shenker, and Sylvia Ratnasamy. Efficient scheduling policies for 15
[61] Jie Ren, Dong Xu, Shuangyan Yang, Jiacheng Zhao, Zhicheng Li, Christian Navasca, Chenxi Wang, Harry Xu, and Dong Li. Enabling large dynamic neural network training with learning-based memory management. In 2024 IEEE International Symposium on HighPerformance Computer Architecture (HPCA), pages 788–802, 2024.
[70] Tianxia Wang, Zhuofu Chen, Wei, and et.al. Characterizing network requirements for gpu api remoting in ai applications. arXiv preprint arXiv:2401.13354, 2024. [71] Xiaoyang Wang, Yongkun Li, Fan Guo, Yinlong Xu, and John C. S. Lui. Dynamic gpu scheduling with multiresource awareness and live migration support. IEEE Transactions on Cloud Computing, 11(3):3153–3167, 2023.
[62] Muhammad Osama Shahid, Daniel Koch, Jayaram Raghuram, Bhuvana Krishnaswamy, Krishna Chintalapudi, and Suman Banerjee. {Cloud-LoRa}: Enabling cloud radio access {LoRa} networks using reinforcement learning based {Bandwidth-Adaptive} compression. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1959– 1976, 2024.
[72] Qizhen Weng, Lingyun Yang, Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. Beware of fragmentation: Scheduling {GPU-Sharing} workloads with fragmentation gradient descent. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 995–1008, 2023. [73] Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. Transparent {GPU} sharing in container clouds for deep learning workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 69–85, 2023.
[63] Shaohuai Shi, Xiaowen Chu, and Bo Li. Exploiting simultaneous communications to accelerate data parallel distributed deep learning. In Proc. IEEE INFOCOM 2021. [64] Foteini Strati, Xianzhe Ma, and Ana Klimovic. Orion: Interference-aware, fine-grained gpu sharing for ml applications. In Proceedings of the Nineteenth European Conference on Computer Systems, pages 1075–1092, 2024.
[74] Zhengxu Xia, Yajie Zhou, Francis Y. Yan, and Junchen Jiang. Genet: automatic curriculum generation for learning adaptation in networking. SIGCOMM ’22, page 397–413, New York, NY, USA, 2022. Association for Computing Machinery.
[65] Dongjie Tang, Linsheng Li, Jiacheng Ma, Xue Liu, Zhengwei Qi, and Haibing Guan. gremote: Cloud rendering on gpu resource pool based on api-forwarding. Journal of Systems Architecture, 116:102055, 2021.
[75] Dong Xu, Yuan Feng, Kwangsik Shin, Daewoo Kim, Hyeran Jeon, and Dong Li. Efficient tensor offloading for large deep-learning model training based on compute express link. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–18, 2024.
[66] Aidan P Thompson, H Metin Aktulga, Richard Berger, Dan S Bolintineanu, W Michael Brown, Paul S Crozier, Pieter J In’t Veld, Axel Kohlmeyer, Stan G Moore, Trung Dac Nguyen, et al. Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer physics communications, 271:108171, 2022.
[76] Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. Infless: a native serverless system for low-latency, highthroughput inference. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 768–781, 2022.
[67] Alexander Van Craen, Marcel Breyer, and Dirk Pflüger. Plssvm—parallel least squares support vector machine. Software Impacts, 14:100343, 2022.
[77] Shuochao Yao, Jinyang Li, Dongxin Liu, Tianshi Wang, Shengzhong Liu, Huajie Shao, and Tarek Abdelzaher. Deep compressive offloading: Speeding up neural network inference by trading edge computation for network latency. In Proceedings of the 18th conference on embedded networked sensor systems, pages 476–488, 2020.
[68] Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. Aqua: Network-accelerated memory offloading for llms in scale-up gpu domains. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 48–62, 2025.
[78] Parham Yassini, Khaled Diab, Saeed Zangeneh, and Mohamed Hefeeda. Horus: granular in-network task scheduler for cloud datacenters. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI’24, USA, 2024. USENIX Association.
[69] Xinchen Wan, Xinyu Yang, Kaiqiang Xu, Xudong Liao, Yilun Jin, Yijun Sun, Zhenghang Ren, Han Tian, and Kai Chen. Coflow scheduling for llm training. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 1232–1234, 2025. 16
Table 9: Summary of CUDA API coverage statistics.
[79] Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping. arXiv preprint arXiv:2306.03622, 2023. [80] Ben Zhang, Xin Jin, Sylvia Ratnasamy, John Wawrzynek, and Edward A. Lee. Awstream: adaptive wide-area streaming analytics. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, SIGCOMM ’18, page 236–252, New York, NY, USA, 2018. Association for Computing Machinery.
Library
#Supported
#Total
Cover Rate
CUDA Runtime (libcudart.so) CUDA Driver (libcuda.so) cuBLAS (libcublas.so) cuDNN (libcudnn.so) cuFFT (libcufft.so) NVML (libnvidia-ml.so)
283 244 385 280 49 123
396 429 508 294 52 131
71% 57% 75% 95% 94% 94%
A A.1
Appendix Implementation Details
We implement the Gleam framework with ∼6K lines of C++ code and more than 10K lines for API interception, which supports remoting 1,000 CUDA API in 6 CUDA dynamic libraries, including cuda_runtime, cuda_driver, cublas, and cudnn, details of which are summarized in Table 9. We compile the server and scheduler modules into executable files, while constructing a client library as a shared object, enabling API interception by LD_PRELOAD. ▶ Fundamental framework: We utilize the gRPC [3] framework, combined with protobuf [7] to build the communication framework of Gleam. It includes 3 main parts:
[81] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632– 19642, 2024. [82] Liangyu Zhao, Siddharth Pal, Tapan Chugh, Weiyang Wang, Jason Fantl, Prithwish Basu, Joud Khoury, and Arvind Krishnamurthy. Efficient {Direct-Connect} topologies for collective communications. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 705–737, 2025.
• We unify different parameter formats of APIs into a 2-level union-like message, apply a duplex synchronous stream in gRPC for efficient request exchange.
[83] Wei Zhao, Anand Jayarajan, and Gennady Pekhimenko. Tally: Non-intrusive performance isolation for concurrent deep learning workloads. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 1052–1068, 2025.
• On the client side, we intercept the CUDA API in the main thread and employ another parallel thread for API forwarding, enabling the API to be executed asynchronously. • On the server side, we manually maintain a thread-safe RPC dispatcher for each synchronous stream between client and server to support concurrent service.
[84] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024.
▶ Virtual Memory Map: We implement a handcrafted virtual memory mapping mechanism that allocates a 1 TB virtual address space for each task and manages mappings using a red-black-tree-based structure, rather than the CUDA VMM API (e.g., cuMemCreate), due to its minimum mapping granularity restrictions. For APIs whose signatures explicitly specify device pointers, we add tags to the corresponding fields in the protobuf requests listed in Table 10. Specifically, for cudaLaunchKernel, we compare the input parameters with and without memory mapping for each kernel, and record the kernel ID and argument offsets for future use. ▶ Communication optimization: We design a lightweight and universal workflow for async-based API optimization. We first use an automatically generated and static pre-complied array for fast optimization-type queries. Then, we use several local response messages for optimized API, which don’t need remote writeback. The basic async API is listed in Table 11. For the client-simulated APIs in Table 12, we construct templates for basic operations such as push/pop. For
[85] Jianlong Zhong and Bingsheng He. Kernelet: Highthroughput gpu kernel executions with dynamic slicing and scheduling. IEEE Transactions on Parallel and Distributed Systems, 25(6):1522–1532, 2014. [86] Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. Megascale-infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 592–608, 2025.
17
Table 10: APIs containing GPU memory pointer, requiring to maintain virtual memory mappings in Section 4.1.3.
Table 12: APIs maintaining trivial states, involving in case Local Simulation in Section 4.2.
MEMORY_MAP
CLIENT_SIMULATE
ID
API
#
ID
API
#
ID
API
Dependency
1 3
cublasCgemm3mEx cublasSgemmEx
x3 x3
2 4
cublasCgemm3mEx_64 cublasSgemmEx_64
x3 x3
5 7
cublasSetWorkspace cublasGemmEx_64
x1 x3
6 8
cublasGemmEx cublasCgemmEx
x3 x3
9 11
cublasCgemmEx_64 cublasGemmBatchedEx
x3 x3
10 12
cublasLtMatmul cublasGemmBatchedEx_64
x5 x3
13
cublasGemmStrided BatchedEx
x3
14
cublasGemm StridedBatchedEx_64
x3
1 2 3 4 5 6 7
cuCtxPopCurrent cuDevicePrimaryCtxGetState cublasGetMathMode cudaGetLastError cudaPopCallConfiguration cudaGetDevice cudaGetDeviceCount
cuCtxPushCurrent other cuDevicePrimaryCtx APIs cublasSetMathMode cudart APIs cudaPushCallConfiguration cudaSetDevice None
15 17
cublasSgemm cublasCgemm
x3 x3
16 18
cublasDgemm cublasZgemm
x3 x3
19
cublasLtMatmul DescSetAttribute
x1
20
cudnnBatchNormalization ForwrdInference
x4
21
cudnnmultiheadattn backwarddata
x6
22
cudnnnormalization backward
x2
23
cudnnbatchnormalization backward
x8
24
cudnnbatchnormalization x13 backwardex
25 27 29
cudnnConvolutionForward x10 cudnnsoftmaxbackward x3 cudaMemcpy_dtod x2
26 28 30
cudnnsoftmaxforward cudaMemsetAsync cudaMemcpy_dtoh
Table 13: APIs applying resource handles, involving in case Batch Prefetch in Section 4.2. BATCH_CREATE
x13 x1 x1
ID
API
Dependency
1 2 3 4 5
cudnnCreateTensorDescriptor cudnnBackendCreateDescriptor cublasLtMatrixLayoutCreate cublasLtMatmulDescCreate cublasLtMatmulPreferenceCreate
Late Call cudnnSetTensorNdDescriptor Sub Key by cudnnBackendDescriptor_t Late Call cublasLtMatmulDescSetAttribute Late Call cublasLtMatmulDescSetAttribute None
Table 11: APIs only returning error code, involving in case Basic Async in Section 4.2.
Table 14: APIs that might modify CUDA context state, involving in protection of cross-CUDA-stream multiplexing for context-restricted feature like CUDA graph in Section 6.
ASYNC
CONTEXT_LEVEL
ID
API
ID
API
1 3
cudaGraphLaunch cudaRegisterFunction
2 4
cudaRegisterVar cudaRegisterFatBinary
5 7
cudaUnregisterFatBinary cufftXtExec
6 8
cudaPushCallConfiguration cufftSetStream
9 11
cufftSetWorkArea cublasSetStream
10 12
cublasLtMatmul cublasCgemmStridedBatched
13 15
cublasGemmStridedBatched cublasLtMatmulDescDestroy
14 16
cublasGemmStridedBatchedEx cublasLtMatmulDescSetAttribute
17
cublasLtMatmul PreferenceDestroy
18
cublasLtMatmul PreferenceSetAttribute
19 21
cublasLtMatrixLayoutDestroy cudaLaunchKernel
20 22
cublasSetWorkspace cudaFuncSetAttribute
23 25
cublasSgemm cublasSetMathMode
24 26
cublasGemmEx cudaMemcpy_dtod
27 29
cudaMemcpy_htod cudnnSetStream
28 30
cudnnBackendFinalize cudnnSetTensorNdDescriptor
31 33
cudnnDestroyTensorDescriptor cudnnDestroyFilterDescriptor
32 34
cudnnSetFilterNdDescriptor cudnnBackendExecute
35 37
cudnnsetconvolutiongroupcount cudnnBackendSetAttribute
36 38
cudnnsetconvolutionmathtype cudnnConvolutionForward
39
cudnnBatchNormalization ForwardInference
40
cudnnSetConvolution NdDescriptor
41
cudnnDestroy ConvolutionDescriptor
42
cudnnBackendDestroyDescriptor
ID
API
ID
API
1 3 5 7 9 11 13
cudaMalloc cudaFree cudnnCreate cublasLtCreate cublasCreate cudnnDestroy cublasLtDestroy
2 4 6 8 10 12 14
cuCtxPopCurrent cuCtxPushCurrent cudaRegisterFunction cudaRegisterVar cublasDestroy cudaRegisterFatBinary cudaUnregisterFatBinary
Table 15: APIs considered in RAII implementation for robust long-lived API serving. HANDLE_RAII ID
API
Destroy
1 2 3 4-6 7-8
cublasCreate cudnnCreate cublasLtCreate cudaStreamCreate series (x3) cudaEventCreate series (x2)
cublasDestroy cudnnDestroy cublasLtDestroy cudaStreamDestroy cudaEventDestroy
data transfer between them bypasses the scheduler entirely. the batch-creation APIs in Table 13, we design rollback and reclamation mechanisms to handle batch-creation failures or over-allocation. ▶ Monitoring and scheduling: We design a daemon thread for each server program, which periodically sends server status to the scheduler, including GPU memory usage, GPU utilization, and task progress. It enables our runtime contentionaware scheduling in a lightweight manner. Our cross-device scheduler is deployed on the machine acting as the server. Notably, once the client and server establish a connection,
▶ Compatibility and stability: We introduce an access controller for APIs that may affect the CUDA context rather than individual CUDA streams, in order to support CUDA Graphs (e.g., in llama.cpp). The APIs in Table 14 may crash the entire context if they are called while another CUDA stream is being captured. We also apply an RAII-based design to the commonly used resource handles in Table 15 to avoid CUDA memory leaks after running multiple tasks consecutively. 18
Table 16: Configurations of edge server in Gleam evaluation. Note that the CPU core refers to logic core here. ID
CPU
Nvidia GPU
GPU FLOPS
GPU Mem.
0 1 2 3
24-core i7-13700K 16-core i7-13700K 96-core i7-14700 28-core AMD EPYC 7K62
RTX A4500 GeForce RTX 4090 GeForce RTX 4070 RTX A6000
23.7 TFLOPS 82.6 TFLOPS 40.1 TFLOPS 38.7 TFLOPS
20.0 GB 24.0 GB 12.0 GB 48.0 GB
Table 18: Summary of model weights and identified cacheable size. Task Type sd3medium-t5 llama3B-ggml llama8B-ggml sd-compvis whisper-large-v3 llava7B-PyTorch fno-PyTorch
Table 17: Summary of model loading cost on evaluated workloads. Transmission Cost (s) 146.03 63.85 152.59 29.68 30.25 178.04 1.47
Bandwidth Occupancy 80.45% 75.15% 73.52% 72.01% 76.95% 67.12% 68.30%
GeForce RTX 4070
63% 6.9s
×
90%
83% 78% 73% 90% RTX A4500 23.7s 7.9s 19.9s 3.0s
53% 55% 20% 7.9s 22.1s 11.7s
70%
84% 53% 63% 54% RTX A6000 22.5s 6.9s 18.7s 4.4s
49% 23% 14% 7.4s 41.6s 25.3s
×
41% 27% 6.1s 13.9s
GeForce 75% 66% 58% 26% 32% 24% 13% RTX 4090 14.7s 4.2s 14.4s 3.6s 5.6s 29.0s 10.9s ml rch ml rch -v3 vis -t5 ium -comp a8b-gg o-PyTo a3b-gg b-PyTo r-large d e sd llam fn llam ava7 ispe -m sd3 ll wh
100 75 50 25 0 25 120 90 60 30 0 25
GPU Utilization
12% 1.8s
×
600 450 300 150 0 25
#Cache chunk #Cache chunk #Cache chunk
Task Type sd3medium-t5 llama3B-ggml llama8B-ggml sd-compvis whisper-large-v3 llava7B-PyTorch fno-PyTorch
50% 30%
Weight Size (MB) 15030 6135 15324 4067 2951 14435 128
sd3
Identified Size Ratio 98.87% 100.00% 93.42% 66.91% 100.00% 100.00% 100.00%
llama-3B
210 215 220 225
100 75 50 25 0 25
210 215 220 225
210 215 220 225
800 600 400 200 0 25
210 215 220 225
llama-8B
llava-7B
210 215 220 225 Size of Chunk (Bytes)
20 15 10 5 0 25
wsp-large-v3
fno
210 215 220 225 Size of Chunk (Bytes)
Figure 28: Distribution of cache chunks of the remaining 6 tasks, where compvis is shown in Figure 7.
10%
butions shown in Figure 28. We successfully identify most weight chunks across seven heterogeneous tasks, demonstrating the generalizability of Gleam. Although chunk sizes vary across workloads, the weight chunks in PyTorch-based tasks (i.e., , fno and llava-7b) are more well-structured, whereas the remaining handcrafted ggml-based tasks exhibit a more scattered distribution. Meanwhile, thanks to its API-level design and application-agnostic nature, Gleam supports weight caching for all tasks. ▶ Framework dependency in API remoting: In Figure 17, task behavior also varies across frameworks: with all communication optimization techniques enabled, PyTorch-based applications typically suffer more than 3× latency degradation, whereas most ggml (cpp) applications usually keep the loss within 2×. This is because C++ programs involve fewer CUDA API types, allowing optimization techniques to cover them more effectively, whereas some APIs invoked by PyTorch cannot be optimized in a general way, so they require a complete round-trip path between the client and the server.
Figure 27: Heatmap of GPU utilization and inference time across different models and GPUs.
A.2 Experiment Supplement ▶ Hardware Configurations: We list the four heterogeneous edge servers used in Gleam ’s evaluation in Table 16. Their performance varies due to progressive hardware deployments. Gleam addresses this heterogeneity through careful offline profiling and online contention-aware scheduling. ▶ Standalone Single Task Measurement: For all tasks included in Section 7, we measure their communication and computation characteristics during the model loading and inference stages, respectively, as shown in Table 6 and Figure 27. For communication, differences in absolute cost mainly stem from variations in model size, while bandwidth utilization remains relatively stable, ranging from 67.12% to 80.45%. For computation, GPU utilization and execution cost vary significantly across tasks and GPUs, depending on the intrinsic characteristics of each task and the compute capacity of each GPU. In general, the more compute-intensive the kernels a task uses, or the more APIs covered by Gleam ’s optimizations, the higher the efficiency it can achieve and the closer its performance is to that of local server execution. ▶ Profile of Cacheable Size: Besides, we record the cacheable weight chunk sizes for all tasks and report their identified cacheable ratios in Table 18, with the size distri-
A.3
Analysis for Gleam Scheduling
▶ Verification for resource contention model: We provide a deeper analysis using the example in Table 3 to further validate the efficiency and accuracy of the resource contention model presented in Section 5.1.1. When only a single model loading stage is present, communication utilization reaches 72%, with a duration of 41.50s. When two model loading 19
1024 in our evaluation, because weights from different layers of the same model are inherently distinct. For smaller chunks, Gleam does not build cache entries for them, although it still records them in the cache table. ▶ Corner Cases for Weight Identification and Sharing: Gleam does not classify a cudaMalloc block as weights if any segment within that block changes its value when cudaFree is invoked (e.g., in-place activations or inputs), even when most of the block contains weight data. However, as shown in Table 18, Gleam successfully identifies more than 93% of weight chunks for all tasks except compvis, indicating that such cases are uncommon and can be safely skipped during the identification procedure. Moreover, memory-efficient frameworks [60] typically manage static, long-lived model weights separately from short-lived activations or inputs, which is consistent with our insight.
Algorithm 1: Gleam Task Scheduling Input: Task set Φ with configs Π; Server status Ψ. Output: Task-server pairs D; Suspended task set Φwait . /* Phase 1: Inference-stage task scheduling
*/
foreach task f ∈ Φ in descending order of GPU memory do 2 foreach server s ∈ Ψ satisfying GPU memory constraint & has cache for f do 3 Determine grouping size gs based on Equation (6); 4 Predict computation latency L( f , s, gs ); 5 Select s∗ = arg mins L( f , s, gs ); 6 Assign f of group size gs to s∗ and update D; 7 Update server status Ψ;
1
/* Phase 2: Model-loading task scheduling
*/
foreach task f ∈ Φ in descending order of GPU memory do 9 Dispatched ← False; 10 if bandwidth util. + comm. occupancy of f < 1 then 11 if ∃ s ∈ Ψ with available GPU memory & has no cahce for f then 12 Select s∗ with min pred. computation latency; 13 Assign f to s∗ and update D; 14 Update bandwidth util. and server status; 15 Dispatched ← True; 16 if not dispatched then 17 Add f to Φwait ; wait 18 return D, Φ 8
A.4.2
▶ University vs. Specificity: Gleam implements a unified abstraction for CUDA API optimization, in which we summarize common optimization patterns while avoiding dependence on specific APIs. As illustrated by the three cases in Section 4.2, if a new API satisfies one of these patterns, it can be optimized simply by adding its name to a configuration file. However, some APIs cannot be incorporated in this way and instead require API-specific modifications. Such extensions are reasonable when certain heavily used tasks frequently invoke those APIs. However, Gleam makes no assumptions about the distribution of task types. Therefore, these API-specific optimizations are beyond the scope of this paper. ▶ Relationship with Distributed Frameworks: SOTA distributed inference frameworks, such as vLLM [40] and SGLang [84], mainly focus on optimizing inference for specific task domains (e.g., LLMs), and thus achieve substantial performance gains. In contrast, API-remoting frameworks like Gleam emphasize support for generic and heterogeneous tasks, aiming to provide universal optimization strategies across diverse workloads. Therefore, a direct comparison between Gleam and these frameworks would be inappropriate.
stages run concurrently, the expected duration increases to 72% × 2 × 41.50 = 59.76s, which is close to the measured latency of 58.32s, thereby validating the proposed formula in Equation. (5). The predicted increase in concurrent computation time follows a similar trend. ▶ Complexity of Gleam scheduling: We provide the pesedocode of the two-phase cross-device scheduler in Section 5.2 in Algorithm 1. It replaces the exponential O(N M ) search with greedy matching of M tasks to N servers based on predicted communication or computation latency, achieving O(MN) complexity, allowing periodic runtime invocation without noticeable delays, even at large scales.
A.4
Discussion
A.4.1
Model Weights Manager
CUDA API Remoting Optimization
▶ Hash Cost and Collision for Weight Chunks: Gleam uses a lightweight hash-based method for weight chunk management and retrieval. Specifically, it computes an MD5 hash string from the first 1024 bits of each weight chunk transferred via cudaMemcpy HtoD. Taking llama-3b as an example, hashing all HtoD chunks takes only a negligible 5ms, while retrieving chunks from the server disk takes about 2s, which is comparable to normal weight loading during local execution. Besides, the first-run consistency check takes about 4,s, after which Gleam builds a cache table for cacheable chunks. Therefore, this overhead is acceptable and becomes unnecessary once the table has been constructed. Moreover, we do not observe hash collisions for chunk sizes larger than 20