arXiv:2604.17861v1 [cs.DC] 20 Apr 2026
GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion Yiwei Yang
Xiangyu Gao
Yuan Zhou
UC Santa Cruz Santa Cruz, USA [email protected]
University of Washington Seatle, USA [email protected]
UC Berkeley Berkeley, USA [email protected]
Yuhang Gan
Yusheng Zheng
Andi Quinn
UC Santa Cruz Santa Cruz, USA [email protected]
UC Santa Cruz Santa Cruz, USA [email protected]
UC Santa Cruz Santa Cruz, USA [email protected]
Abstract
maintaining full compatibility with the PyTorch ecosystem. GPUOS provides a practical pattern for building GPU runtime systems that bridge the gap between operator flexibility and execution efficiency.
Modern deep learning workloads increasingly involve numerous small tensor operations—particularly in inference scenarios, attention mechanisms, and micro-batched training—where producing non-negligible kernel launch overhead. Traditional GPU computing models launch separate kernels for each operation, incurring significant CPU-GPU synchronization costs that can even exceed the actual computation time by orders of magnitude. We present GPUOS, a GPU runtime JIT system that reduces kernel launch overhead through a persistent kernel architecture combined with runtime operator injection. GPUOS deploys a single long-lived GPU kernel that continuously processes tasks from a host-managed work queue, avoiding repeated kernel launches entirely. To obtain operational flexibility by effectively supporting multiple types of operations, we leverage NVIDIA’s NVRTC to just-in-time compile new operators at runtime and dynamically inject them into the running kernel via device function pointer tables. This approach enables hot-swapping of GPU operators without kernel restarts or system recompilation. Our system introduces several key innovations: (1) a persistent worker kernel with atomic-synchronized task queues that eliminates per-operation launch overhead, (2) a runtime operator injection mechanism using NVRTC and relocatable device code that maintains an updatable jump table of device function pointers, (3) a dual-slot aliasing scheme enabling safe operator updates without suspending concurrent tasks, and (4) transparent PyTorch integration via TorchDispatch that automatically aggregates micro-operations into batched submissions. The system can support arbitrary tensor shapes, strides, data types, and broadcasting semantics through a generic tensor abstraction layer. Experimental results demonstrate that GPUOS achieves 15.3x speedup over standard PyTorch for workloads dominated by small operations, with particularly strong performance on micro-batched inference (up to 8.7x utilization) and attention computation patterns. Our transparent scheduler integrates well with the existing PyTorch code base while
ACM Reference Format: Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, and Andi Quinn. 2026. GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion. In . ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
1.1
From Big Batches to Tiny, Fast Ones
Over the past decade, extensive efforts have been made to make GPUs faster in computation and larger in memory capacity. Vendor libraries saturate tensor cores with nearperfect efficiency. Compiler stacks like TVM and the PyTorch 2.0 toolchain fuse long chains of operations into heavyweight launches that minimize data movement. Memory hierarchies are carefully orchestrated to keep reuse patterns close to compute units. These optimizations remain crucial for offline training and dense inference, where batch sizes are large and execution patterns are regular. Yet, they do not fit well for current scenarios where Production machine learning systems increasingly serve latencysensitive, micro-batch workloads where the very properties that once made the host-to-device handshake negligible now make it dominant. When a single user interaction requires hundreds of microsecond-scale kernel executions— each performing a small computation on limited data—a five-microsecond submission path is no longer an ignorable constant. It becomes a tax, applied one hundred times per request, that can dominate the total response time budget [18, 19]. Consider the timeline traces from modern profilers. The GPU excels at mathematical operations, computing matrix multiplications and activations in blazing bursts. But between these bursts lies whitespace: the device sits idle while the host marshals the next kernel launch. The host, in turn, 1
Conference’17, July 2017, Washington, DC, USA
Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, and Andi Quinn
spends a surprising fraction of its time navigating the submission path—crossing into kernel space, updating driver structures, synchronizing state. For instance, if each of one hundred operations per token incurs five microseconds of overhead, it means half a millisecond of pure coordination cost before we’ve even considered the actual computation. Code is open sourced at https://github.com/Multi-V-VM/GPUOS/ 1.2
We set out not to replace compilers or graphs, but to give them a complementary option that keeps the GPU fed when dynamism prevents capture, when shapes refuse to sit still, and when the system just needs to say: don’t launch—call. Our measured results—15.3× on micro-batched elementwise operations, 8.7× on attention decoding, 23.1× on mixed pipelines, with 20–22% energy savings—aren’t products of exotic compiler stunts. They’re the arithmetic of removing a fixed cost applied too frequently. The system is complementary, integrates well with existing frameworks, and coexists with CUDA Graphs, MPS, and MIG, each mechanism pulling its weight where it excels.
Our Approach
To remedy the problem, we claim that the right unit to optimize in these scenarios is not an individual kernel but the boundary that incurs the launch itself. Therefore, instead of investing ever more ingenuity in making each launch cheaper—or trying to statically fuse away the multiplicity of small operations, which dynamism often defeats—GPUOS sidesteps the issue by launching the kernel exactly once and doing everything else inside. Concretely, we regard the persistent kernel as the medium; the message is a fundamental change in where scheduling resides. If the GPU is to stay busy, the source of work should not be a series of host calls separated by system calls and driver crossings. Instead, it should be a ring buffer visible to device threads that never sleep. These threads poll for work, grab task descriptors, dispatch to the appropriate operator through a function pointer table, and immediately return for the next task. Because models evolve and applications grow new features, this persistent runtime should be extensible without downtime. GPUOS accomplishes this through runtime compilation with NVRTC [17] and a device function-pointer table that can be updated safely under load via the CUDA Driver API [14]. When a new operator is needed (e.g., a custom attention variant or a novel activation function), the operations team does not have to roll the cluster or coordinate maintenance windows. They compile a small template to PTX (Parallel Thread Execution), load the module, resolve the function symbol, and publish a pointer into an inactive slot of the operator table. A version counter flip with storerelease semantics makes the operator instantly callable by all device threads. 1.3
2
Motivation
2.1
One Painpoint in Production
We illustrate our motivation through a representative realworld case. Consider a real-time text generation service serving thousands of concurrent sessions. Each session decodes token-by-token, weaving together a pattern of attention kernels, vector additions, activation functions, small reductions, and key-value cache updates. The workload is inherently dynamic: prompt lengths vary wildly, control flow branches based on generated content, and new model features are deployed continuously to stay competitive. The engineering team upgraded to recent CUDA toolkits, enabled mixed precision everywhere possible, fused operations aggressively where the framework permitted, and experimented extensively with CUDA Graphs. In synthetic benchmarks—where prompts were short, predictable, and shape-stable—graphs delivered impressive results. But production reality proved unruly. Users paste entire paragraphs as prompts. Streaming inputs hiccup and stutter. Different code paths activate different operators. The shape of work per token is polymorphic in ways that complicate capture and replay. The timeline in the profiler shows unsatisfiable results. It looks like the teeth of a comb: the GPU computes intensely for tens of microseconds, then waits. The host issues the next launch and waits. The launch path is short in absolute time (e.g., 3–7 microseconds on modern systems) but the repetition makes it the dominant factor in the response-time budget. If a single token requires one hundred such microoperations and each launch with associated synchronization costs five microseconds, causing around 500 microseconds of overhead per token, regardless of how efficiently the actual kernels execute. It might be useful for the developers to capture multiple graph variants to cover common shapes, but maintaining a stable of graphs proved error-prone. Recaptures were necessary when models changed. Graph replay paths sometimes
The Mechanical Consequence
The consequence of this relocation is promising. Launch overhead disappears as a factor within the steady-state execution loop. Device function calls—measured in nanoseconds— replace host submissions measured in microseconds. The whitespace between kernels in timeline traces shrinks until it is indistinguishable from instruction scheduling gaps within the persistent kernel itself. Tail latency improves because queuing and wakeup paths are deterministic and short. And because there is no need to predict which sequence of operations will occur next, GPUOS retains the semantics of eager execution while reaping the performance gains of persistent scheduling. 2
GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion
fell back to eager execution in corner cases, quietly reintroducing the very overhead they sought to avoid. The conditional logic to select the right graph became its own source of complexity and latency.
Conference’17, July 2017, Washington, DC, USA
3
Background and Context
3.1
Source of Launch Overhead
When host code calls cudaLaunchKernel, the CUDA runtime marshals arguments, establishes a grid configuration, and issues a request through the runtime or driver API. This 2.2 The Alternative Vision call crosses from user space into kernel mode, where the driDifferent from the previously mentioned approach, an alver performs bookkeeping: updating stream queues, checkternative is to flip the entire arrangement. Specifically, at ing for dependencies, allocating resources, and programming process startup, the service brings up GPUOS and the perhardware schedulers. Finally, the GPU hardware enqueues sistent kernel takes residence on the GPU. This is a one-time the work onto an execution stream. setup cost: allocate the ring buffer in device memory, launch Under steady, coarse-grained loads, the latency of this the persistent kernel with one block per streaming multipropath is easily amortized. A kernel that runs for milliseconds cessor, and let those warps spin up. From this point forward, can tolerate even tens of microseconds of overhead as mere they never exit. Then, we can simply poll the ring buffer, “noise" [19]. The ratio is favorable: if 10 microseconds of waiting for work. launch overhead precedes 1000 microseconds of execution, When the first user request arrives, the PyTorch dispatch that’s a 1% tax—negligible and acceptable in most scenarios. layer—instrumented with GPUOS integration—recognizes However, in eager execution with abundant small operthat the upcoming sequence of operations consists of small, ations, the launch overhead has big impact on the whole launch-overhead-dominated kernels. Instead of calling cudaLaunchKernel process. Even a null kernel—one that does no useful work— repeatedly, it writes task descriptors into the ring buffer. clocks in around 3–7 microseconds in public measurements Each descriptor is compact: an operator ID, pointers to input and forum discussions [18]. For a kernel that executes in 10 and output tensors, dimension parameters, and a few control microseconds, this represents 30–70% overhead. For operaflags. The submission is completed with a single store-release tions that complete in single-digit microseconds, the overon a commit field, making the task visible to device threads. head can exceed the computation time itself. Device threads, already awake and polling, see the commit. The problem compounds when operations are serialized. They grab the work atomically, look up the operator ID in the A single request might involve a hundred distinct operafunction-pointer table, and dispatch. The operator executes tions: attention computations, vector additions, activations, as a device function call—not a kernel launch—completing in normalizations, reductions, cache updates. If each operation nanoseconds of scheduling overhead rather than microseclaunches independently, the cumulative overhead can reach onds. When it finishes, the thread immediately polls for the 300–700 microseconds—dominating the execution time for next task. many workloads and directly impacting user-perceived laWhen new operators are introduced by the machine learntency. ing developers (e.g., a custom rotary positional embedding with a parameterization) to better fit their specific use case, instead of recompiling PyTorch extensions, rebuilding Docker 3.2 CUDA Graphs: Strengths and Limitations images, coordinating a rollout, and hoping nothing breaks, the process in GPUOS becomes different. CUDA Graphs address this problem directly by moving prepaTo be specific, a small CUDA template—parameterized ration out of the hot path. If a program can capture a directed with the embedding’s specifics—is passed to NVRTC for justacyclic graph (DAG) of work and replay it as a single submisin-time compilation. The PTX output is loaded as a module sion, the CPU cost per operation is replaced by a smaller CPU via the CUDA Driver API. The function symbol is resolved, cost per graph [6]. Recent enhancements, including constantyielding a device function pointer. This pointer is written time launch techniques for certain graph shapes [13], have into an inactive slot of the operator table. A version counter further reduced submission overhead. is incremented with store-release semantics. Warps in the Graphs are elegant when applicable. They shine in scepersistent kernel observe the version change at well-defined narios with regular, repeatable execution patterns: training points in their polling loop and atomically switch to the loops with fixed batch sizes, inference pipelines with stable updated table. The operator becomes callable immediately, shapes, and benchmark suites with controlled inputs. The with zero downtime. captured graph becomes a reusable execution plan, amortizThe shape of work remains dynamic, evolving contining the capture cost over many replays. uously with product features and model updates. But the However, graphs impose a promise of stability and repeatalaunch boundary—the old chokepoint—has been absorbed bility that dynamic, control-flow-heavy inference workloads into the device. The cost of coordination has collapsed from often cannot keep. Real-world challenges arise from multiple microseconds per operation to a single-digit number of nanosec- sources. First of all, variable shapes. Their different prompt onds for a function call. lengths, varying sequence positions, and dynamic attention 3
Conference’17, July 2017, Washington, DC, USA
Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, and Andi Quinn
window sizes create polymorphic execution patterns that resist single-graph capture. The second challenge comes from control flow, where conditional layers, early exit strategies, and adaptive computation paths break graph capture unless every variant is pre-captured, leading to graph proliferation and management complexity that can become unwieldy in production. Late-binding operators further amplify these difficulties. New model features and custom operators injected at runtime cannot be incorporated into pre-captured graphs without expensive recapture operations that may interrupt service. Graph maintenance itself becomes a burden as updates to models or frameworks may invalidate captured graphs, requiring re-capture at inopportune times. Managing a stable of variant graphs to cover common cases introduces fragmentation and fallback paths that quietly reintroduce the very launch overhead the graphs were meant to avoid. In practice, production systems often maintain multiple graph variants to cover common shapes, with fallback to eager execution for uncommon cases. This hybrid approach helps but does not eliminate the fundamental tension: graphs want predictability, but modern ML systems evolve continuously.
for multi-tenant scenarios, MPS does not reduce per-kernel submission cost within a single tenant’s pipeline. Besides, MIG (Multi-Instance GPU) takes a different approach by partitioning a GPU into isolated slices with enforced resource limits and memory protection [15]. This mechanism is critical for strong multi-tenant guarantees in cloud environments, but remains orthogonal to withintenant launch optimization. Dynamic Parallelism enables device-launched kernels, allowing a running kernel to spawn additional work [12]. Though powerful, it does not eliminate launch overhead but merely relocates the initiator to the device, while also introducing complexity in resource accounting and debugging. GPUOS can coexist with all these mechanisms. It can run as an MPS client, operate within a MIG slice, and complement dynamic parallelism for hierarchical work patterns. Each mechanism pulls its weight in the places where it excels.
3.3
Ring Buffer Communication Channel. A lock-free, single-producer-single-consumer ring buffer resides in devicemapped memory, allowing the host to enqueue task descriptors and device threads to dequeue them with minimal synchronization overhead. Each descriptor is compact—typically 64–128 bytes—containing an operator ID, tensor pointers, dimension parameters, and control flags. The ring buffer uses atomic operations for synchronization: the host advances a write cursor with store-release semantics after writing a complete descriptor, and device threads poll a read cursor with load-acquire semantics. This ordering ensures that when a device thread observes a new task, all associated data is visible.
Design of GPUOS
4.1
Architecture Overview
GPUOS consists of three primary components working in concert (shown in Figure 1):
Persistent Threads: The Alternative Paradigm
Persistent threads offer a different approach. Instead of launching many short-lived kernels, launch one long-lived kernel that pulls work from a queue. This idea predates modern machine learning but has enjoyed a renaissance in micro-batch inference because it directly targets the overhead we now care about [9]. The core concept is to maintain resident threads on the device that loop indefinitely, polling a shared work queue for tasks. When a task appears, a thread claims it, executes the work, and returns to polling. This eliminates per-operation launch overhead at the cost of keeping some threads always active. What distinguishes GPUOS from typical persistent-thread sketches is its ability to evolve the operator set without pausing the service. Traditional persistent kernels are compiled with a fixed set of operators, limiting flexibility. GPUOS extends the paradigm through dynamic operator injection: new operators can be compiled at runtime, loaded safely, and made callable without restarting the persistent kernel or interrupting in-flight work. 3.4
4
Persistent Kernel Executor. A single persistent kernel launches at process startup and never exits. By default, it occupies one thread block per streaming multiprocessor (SM), keeping resource footprint modest to coexist with large conventional kernels. Each warp in the persistent kernel independently polls the ring buffer, claims available work atomically, dispatches to the appropriate operator, and returns for the next task. The polling loop is carefully tuned to balance responsiveness against power consumption. In high-throughput scenarios, pure spin-polling minimizes latency. In lower-load scenarios, brief exponential backoff with pauses can reduce power draw without significantly impacting tail latency.
Adjacent Mechanisms
There are several mechanisms trying to improve the performance and GPUOS is complementary to them. For instance, MPS (Multi-Process Service) raises utilization by allowing multiple client processes to submit work cooperatively, avoiding the serialization that would otherwise occur with exclusive GPU access [16]. While essential
Dynamic Operator Table. The operator table is a deviceresident array of function pointers, indexed by operator ID. 4
GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion
Conference’17, July 2017, Washington, DC, USA
PyTorch Dispatch Decide GPUOS submission
Host (CPU)
CUDA Templates (param → code)
custom op? Device (GPU) CUDA Driver API cuModuleLoadData, cuModuleGetFunction
NVRTC (JIT) C++ → PTX
enqueue & commit poll
SPSC Ring Buffer (device-visible) TaskDesc: op id, ptrs, shape/stride, flags Per-SM Warps dispatch Persistent Kernel Executor 1 block/SM; spin/backoff; low footprint → poll ring → lookup fnptr → call → release write inactive slot + publish version
fnptr lookup Versioned Operator Table (device function pointers) slots: add / gelu / sdpa / rotary / . . . version: N
Figure 1. GPUOS Architecture
while ( t r u e ) { TaskDescriptor desc ; i f ( r i n g _ b u f f e r . p o l l (& d e s c ) ) { / / l o a d − a c q u i r e o p e r a t o r _ t a b l e [ desc . op_id ] ( desc ) ; ring_buffer . release_slot ( ) ; } }
Each entry points to a device function implementing a specific operation: element-wise addition, matrix multiplication kernels, attention mechanisms, activations, and so on. Crucially, the table can be updated at runtime through a carefully orchestrated protocol. The process begins with compiling the new operator to PTX using NVRTC, followed by loading the PTX module via the CUDA Driver API. The function symbol is then resolved to obtain a device function pointer, which is written to an inactive slot in the operator table. An atomic flip of a version counter with store-release semantics completes the update, allowing device threads to observe the version change and switch to the updated table. This protocol ensures that no thread ever sees a partiallyupdated table. Operator injection completes in milliseconds, making new functionality available without service interruption.
This design achieves submission latencies under 100 nanoseconds in favorable conditions, compared to 3–7 microseconds for traditional kernel launches—a 30–70× reduction in coordination overhead. 4.3
Memory Management and Safety
GPUOS operates within PyTorch’s memory management framework. Tensors are allocated through PyTorch’s caching allocator, and GPUOS merely receives pointers to already4.2 Task Submission Protocol allocated memory. This integration ensures compatibility with existing code and avoids conflicts with PyTorch’s interThe host-side submission path is streamlined for minimal nal memory bookkeeping. overhead: For operator injection, GPUOS maintains a compiled mod/ / Prepare descriptor ule cache indexed by operator signature. When a new operTaskDescriptor desc ; ator is requested, the system checks the cache before comd e s c . o p _ i d = OP_ADD ; piling. Compiled modules are retained in memory, and their function pointers remain valid for the process lifetime, avoiddesc . input_a = tensor_a . data ( ) ; ing repeated compilation overhead. desc . input_b = tensor_b . data ( ) ; Safety is enforced through multiple complementary layers. desc . output = tensor_c . data ( ) ; Template-based compilation forms the first line of defense: desc . s i z e = tensor_a . s i z e ( ) ; operators are compiled from curated templates with parameter substitution rather than arbitrary code, reducing the / / Enqueue a t o m i c a l l y risk of malicious injection. Version-gated access provides uint32_t slot = ring_buffer . acquire_slot ( ) ; the second layer, ensuring that device threads always read r i n g_ b u f f er . write ( slot , desc ) ; the operator table version atomically before indexing the r i n g _ b u f f e r . commit ( s l o t ) ; / / s t o r e − r e l e a s e table, guaranteeing they see a consistent snapshot. Bounds checking adds another safeguard by validating operator IDs On the device side, warps poll and dispatch: before indexing the table, with out-of-range IDs triggering 5
Conference’17, July 2017, Washington, DC, USA
Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, and Andi Quinn
fallback to CPU execution and error reporting. Finally, audit logging records all operator injections with timestamps, source templates, and parameter values for post-hoc security review and forensic analysis.
5
buffer in device memory, sampled asynchronously by a host thread for minimal overhead. Performance counters. Track ring buffer utilization, task throughput, operator dispatch frequencies, and stall events. Counters are exported through a simple query interface for monitoring dashboards.
Implementation
We implement GPUOS using 3793 lines of code in C++ and Python. By writing an extention of pytorch codebase, it’s fully compatible to be a plugin design to all pytorch projects. 5.1
Kill switches. Each operator table entry can be replaced with a stub function that immediately fails any task targeting that operator, returning an error code to the host. This allows quick disabling of misbehaving operators without full service restart.
Integration with PyTorch
GPUOS integrates with PyTorch through dispatch interposition at the autograd engine level. When PyTorch prepares to execute an operation, a dispatcher hook evaluates whether the operation is a good candidate for GPUOS submission (we call it filtering). To build such a filter, GPUOS evaluates multiple factors: the operation type (favoring element-wise operations, small reductions, and cache updates), tensor size (operations on small tensors benefit most from overhead reduction), execution context (prioritizing high-frequency operations in inner loops), and current load (falling back to conventional kernels if the ring buffer fills). Eligible operations are redirected to the GPUOS submission path. Ineligible operations proceed through PyTorch’s normal kernel launch path. This hybrid approach ensures that GPUOS accelerates where it helps most without introducing regression on workloads that don’t benefit. 5.2
Visual profilers. GPUOS emits markers compatible with NVIDIA Nsight and other profiling tools, allowing timeline visualization of task submission, execution, and completion events. 5.4
Operator Library
The initial operator library covers common micro-batch inference primitives across several categories. Element-wise operations include addition, multiplication, ReLU, GELU, softmax, and layer normalization. Small matrix operations encompass vector-matrix products and small matrix multiplications that do not warrant full CUBLAS dispatch. Attention mechanisms cover scaled dot-product attention, rotary embeddings, and attention masking operations. Cache operations handle key-value cache updates, prefix matching, and cache compression, which are particularly frequent in autoregressive generation. Finally, reductions perform sum, max, and min operations over small dimensions. Each operator is implemented as a device function template accepting task descriptor parameters. Templates are instantiated and compiled on-demand as new shapes and data types are encountered during execution. 5.3
GPUOS Syscall API Reference
Table 1 summarizes the host-side API exposed by GPUOS for runtime management and integration with deep learning frameworks. The interface follows a minimal design, exposing only the controls necessary for initialization, task fusion, scheduling, and shutdown. The init() call creates the GPUOS runtime, allocates the device-visible ring buffer, and launches the persistent worker kernel. Once initialized, fuse() allows multiple small operations to be combined into a single aggregated submission, reducing per-operation overhead. The set_yield_every() function controls how frequently worker threads yield to avoid monopolizing GPU resources—useful in shared or MPS environments. The peek_queue() API provides introspection into the runtime queue state, reporting head, tail, and processed counts for monitoring and debugging. worker_alive() checks whether the persistent kernel remains active, while shutdown() performs an orderly termination by signaling worker exit and releasing GPU resources.
6
Evaluation
6.1
Experimental Setup
Our evaluation uses single NVIDIA H100 GPUs (96GB) on supermicro X14-SBF with single socket Intel Xeon 6787P, single NVIDIA RTX 5090 GPU (32GB) on Z890 with Intel 285k installed and NVIDIA Digit Spark with GB10 GPUs (128GB) with CUDA 13.0. We compare GPUOS against three baselines Eager PyTorch[22]: Unmodified PyTorch 2.2 with eager execution. TorchScript[21]: Traced execution with PyTorch’s JIT compiler. CUDA Graphs[3]: Hand-captured graphs for regular segments of execution. Workloads include synthetic micro-benchmarks and productionrepresentative inference scenarios. Micro-benchmarks have sequences of 100 element-wise operations on small tensors
Debugging and Observability
Long-lived kernels can become opaque without good tooling. GPUOS addresses this through comprehensive instrumentation: Tracepoints. Record task identifiers, enqueue timestamps, dequeue timestamps, execution times, and operator table versions for each operation. Tracepoints write to a circular 6
GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion
Conference’17, July 2017, Washington, DC, USA
Table 1. GPUOS host-side runtime API for persistent-kernel management and dynamic task fusion. Function
Description
init(capacity, threads_per_block) Initialize GPUOS runtime, allocate queue fuse() Fuse the runtime operation set_yield_every(every) Control task yield policy (0 = never yield) peek_queue() Query queue state (head, tail, processed count) worker_alive() Check if persistent worker kernel is running shutdown() Signal worker exit, free GPU resources
(1K–16K elements), repeated 1000 times to measure steadystate throughput and latency. Attention decoding is tokenby-token generation with attention over varying context lengths (128–2048 tokens), measuring per-token latency and throughput. Mixed pipelines are realistic inference combining attention, FFN layers, normalizations, and activations, with dynamic control flow based on generated tokens. 6.2
Token-by-Token Generation Throughput
70000
PyTorch Naive FlashAttention GPUOS Scheduled
Tokens Generated / Second
60000 50000 40000 30000 20000 10000
Performance Results
0
The element-wise benchmark—where launch overhead is most pronounced—shows dramatic improvements. GPUOS reduces per-operation latency from ∼8 microseconds (5 𝜇s launch + 3 𝜇s execution) to ∼3.1 microseconds (100 ns dispatch + 3 𝜇s execution), translating to 15.3× speedup on H100 11.3× on 5090 and 2.8× on GB10. The better performance of baseline on GB10 is caused by the CPU-GPU fabric that in hardware optimize the launch kernel performance. So we only gain minor speed up over the baseline. Attention decoding benefits from eliminating repeated small kernel launches for query-key computations, softmax, and value aggregation. Per-token latency drops from ∼140 microseconds to ∼16 microseconds, an 8.7× improvement. The mixed pipeline scenario—most representative of real inference—achieves 23.1× speedup by accelerating the long tail of small operations between large matrix multiplications. This workload also demonstrates GPUOS’s ability to coexist with conventional kernels: large GEMMs still launch traditionally while surrounding micro-ops route through GPUOS. Energy savings of 20–22% result from reduced idle time and more efficient GPU utilization. The persistent kernel’s power footprint is modest (one block per SM), while eliminating thousands of launch-idle-launch cycles per request significantly reduces wasted energy. As shown in Figure 2, GPUOS delivers substantial gains in attention throughput, outperforming both FlashAttention [8] and PyTorch compiled mode. By maintaining a persistent kernel that continuously consumes queued operations, GPUOS eliminates the per-token launch latency characteristic of conventional GPU execution. This enables sustained high utilization across streaming attention workloads, particularly in small and mid-size tensor regimes where launch overheads dominate. The observed performance trend is
150
200
250
300
350
400
Sequence Length
450
500
Figure 2. Attention Throughput compared to FlashAttention and Pytorch Compiled Multi-Process GPU Sharing Performance Concurrent (No MPS)
0.0175
Speedup vs Sequential
0.0150 0.0125 0.0100 0.0075 0.0050 0.0025 0.0000
2
4
Number of Processes
8
Figure 3. Multi-Process GPU Sharing Performance
consistent with prior findings in micro-batched inference systems [1, 11], demonstrating that minimizing host-device synchronization can yield order-of-magnitude throughput improvements even without modifying the model architecture. Figure 3 further highlights how GPUOS coexists with NVIDIA’s Multi-Process Service (MPS) [16], achieving strong scaling under concurrent workloads. When multiple processes issue requests simultaneously, the persistent kernel maintains steady per-process performance by relying on a low-footprint, one-block-per-SM scheduling design. This 7
Conference’17, July 2017, Washington, DC, USA
Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, and Andi Quinn
Table 2. End-to-end speedup vs. eager PyTorch baseline Workload
H100
5090
GB10
Energy Saving
Element-wise ops Attention decoding Mixed pipeline
15.3× 8.7× 23.1×
11.3× 2.5× 15.3×
2.8× 2.1× 3.1×
22% 21% 20%
MIG Simulated Partition Performance
2500
GPUOS achieves 8.7× without graph management complexity.
7.14× slower
Time (ms)
2000
1500
6.4
Scalability and Throughput
3.57× slower
1000
GPUOS scales effectively with increasing concurrency. With one persistent kernel per GPU, throughput saturates at ∼800K operations/second on GB10. This represents a 12× improvement over eager execution’s ∼67K ops/sec, limited by launch serialization. Ring buffer contention is minimal until extreme concurrency (>64 concurrent host threads). At high load, the ring buffer size (default 4096 entries) provides adequate buffering to smooth transient bursts without stalls.
2.63× slower
500
1.00× slower
0
P Full G
0%)
U (10
Ms)
2% S
b (~4
g.40g
MIG 3
Ms)
8% S
b (~2
g.20g
MIG 2
Configuration
Ms)
4% S
b (~1
g.10g
MIG 1
Figure 4. MIG Simulated Partition Performance
avoids contention within the CUDA stream scheduler and allows efficient hardware sharing. The results show that GPUOS preserves high occupancy and predictable latency under concurrent inference sessions, aligning with earlier studies of GPU multi-tenancy and runtime partitioning [26, 31]. Figure 4 examines GPUOS behavior under simulated MIG [15] (Multi-Instance GPU) partitions. Even when the GPU is subdivided into smaller logical slices with isolated resources, GPUOS maintains proportional performance scaling and consistent speedups (up to 3.4×), demonstrating that its design generalizes across resource-isolated environments. Unlike approaches that rely on global launch control or driver-level graph replay, GPUOS’s lightweight device-side runtime remains effective even under reduced SM availability. This property is essential for next-generation multi-tenant cloud inference clusters, where MIG slicing and containerized deployment are standard practice [28, 29]. 6.3
Comparison with CUDA Graphs
7
Efficiency and Reliability Analysis
7.1
Resource Usage
Running an always-on kernel demands careful etiquette on shared devices. Even when a process owns a full GPU, it may schedule large kernels that shouldn’t be starved by an overeager persistent executor. GPUOS addresses this through conservative resource allocation: by default, it occupies only one block per SM, representing approximately 2–4% of total threads. It maintains modest register usage to preserve high occupancy for coexisting kernels and makes no attempt to saturate shared memory or other limited resources. Rather than displacing large conventional kernels, GPUOS fills the valleys between large launches, acting as a complement rather than a competitor. In strict multi-tenant environments, GPUOS can constrain itself to a MIG slice, ensuring complete isolation from neighbors [15].
7.2
CUDA Graphs achieve 6.2× speedup on element-wise benchmarks when shapes are stable, but performance degrades to 2.1× when shapes vary due to recapture overhead. GPUOS maintains consistent performance across shape variation because it doesn’t rely on pre-captured patterns. In attention decoding with dynamic sequence lengths, CUDA Graphs require maintaining separate graph variants for different length ranges. Graph selection overhead and occasional fallbacks to eager execution limit speedup to 4.3×.
Debugging and Failure Handling
A long-lived kernel that continuously calls device functions could become an opaque tangle without good tooling. GPUOS invests heavily in observability through multiple mechanisms. Tracepoints record task IDs, timing information, and operator versions for each operation, providing a detailed execution history. Kill switches enable instant disabling of misbehaving operators by replacing their table entries with stubs that fail quickly and surface errors on the host. Audit logs track all operator injections with timestamps and 8
GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion
source provenance for security review and debugging. Finally, GPUOS maintains compatibility with standard profilers like Nsight, allowing developers to leverage familiar tooling for performance analysis. When a newly injected operator misbehaves, a kill switch replaces its table entry with a stub that fails quickly and surfaces errors on the host. This fail-fast behavior prevents cascading failures and aids debugging. 7.3
as unstructured grid solvers and ray tracing [2, 9, 24], the model has re-emerged in the context of modern ML inference pipelines where launch overhead dominates fine-grained execution. Systems such as Softshell [24], Whippletree [23], and RtGPU [32] demonstrated persistent scheduling for irregular graphics workloads, while Gunrock [27] applied similar principles to graph analytics. GPUOS extends this lineage by introducing dynamic operator injection, allowing operators to be compiled and patched into a running persistent kernel without relaunching or halting service, and by providing production-grade instrumentation and PyTorch integration for transparent deployment in ML inference systems. CUDA Graphs provide a complementary approach to reducing CPU-side overhead through DAG capture and replay [6, 13, 28, 30]. Graphs are effective when workloads are regular and repeatable—typical in offline training and static inference—because they allow a batch of operations to be recorded once and replayed efficiently. However, their rigidity makes them less suitable for dynamic, control-flow-heavy workloads that evolve at runtime. GPUOS and CUDA Graphs can coexist: graphs handle the regular segments of computation, while GPUOS accelerates the dynamic fragments that defy static capture. Moreover, while CUDA Graphs optimize the launch process, their execution remains confined to a single process context [29], limiting applicability in multitenant or composable runtime environments. CUDA Dynamic Parallelism (CDP) [10, 12] allows kernels to launch other kernels directly from the device, reducing host involvement but not eliminating the underlying launch cost. CDP also introduces additional challenges in synchronization and resource accounting, often leading to lower effective occupancy. GPUOS addresses these limitations by replacing device-initiated launches with in-kernel function dispatch through a versioned function-pointer table, avoiding redundant scheduler invocations entirely. Compiler and graph-level optimization frameworks such as PyTorch 2.0’s torch.compile [20], TensorFlow XLA [25], and Apache TVM [4, 5] improve performance by fusing operations and generating optimized execution graphs. These techniques are powerful when graphs are extractable, but their benefits diminish when workloads exhibit input-dependent control flow, dynamic tensor shapes, or operator polymorphism. GPUOS complements such compilers by operating below the graph level, transparently aggregating micro-operations that remain after graph fusion, without requiring model-level changes. At the system level, NVIDIA’s Multi-Process Service (MPS) [16] and Multi-Instance GPU (MIG) [15] mechanisms improve GPU utilization and isolation in multi-tenant environments. MPS allows multiple clients to share the GPU context efficiently, while MIG partitions the GPU into isolated slices with dedicated resources. These mechanisms, however, do
Security Considerations
Runtime code loading demands sober security treatment. In production deployments, GPUOS should follow several key principles. First, compilation should be restricted to curated templates with parameterization rather than free-form code, limiting the attack surface. Second, compiled artifacts should be stored in a signed cache with integrity checks to prevent tampering. Third, comprehensive audit logs of injected functions with timestamps and source provenance must be maintained for forensic analysis. Fourth, injection rights should be restricted to authorized processes only, preventing unauthorized code execution. Finally, optional sandboxing through MIG can provide additional isolation in high-security environments. While single-tenant regimes reduce cross-tenant risk, prudent mitigation remains essential. The default should be to fall back to baseline paths when an operator isn’t recognized, failing safely. 7.4
When Not to Use GPUOS
GPUOS outperforms in specific scenarios. It targets a specific performance pathology: excessive launch overhead from abundant small operations. Where programs are naturally coarse-grained with large, efficient kernels, traditional execution paths suffice. Where execution is regular enough for CUDA Graphs to capture reliably, graphs may offer simpler deployment. It excels when operations are fine-grained, executing in microseconds to tens of microseconds. It proves valuable when execution patterns are dynamic and resist graph capture due to control flow or shape variation. It becomes essential when models evolve frequently, requiring flexible operator injection without service interruption. Finally, it delivers the most benefit when latency matters more than peak throughput, as in user-facing inference services. The larger architectural point is that GPUOS complements, rather than replaces, existing optimization strategies. It fills a gap where compilers and graphs struggle.
8
Conference’17, July 2017, Washington, DC, USA
Related Work
Persistent threads are a long-established technique for amortizing kernel launch overheads by maintaining resident GPU threads that continuously fetch work from device memory. Originally proposed for irregular scientific workloads such 9
Conference’17, July 2017, Washington, DC, USA
Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, and Andi Quinn
not address the per-operation coordination cost within a single tenant. GPUOS operates orthogonally, mitigating launch overhead even within an MPS client or MIG slice, ensuring that small-kernel workloads remain efficient in both shared and isolated configurations. Several recent research systems explore similar goals. Zelos [32] and Cocktailer [30] and LithOS [7] investigate device-side task scheduling and cooperative multitasking for low-latency workloads. eGPU [28] and hetGPU [29] propose heterogeneous GPU virtualization layers for multi-process scheduling and dynamic workload migration. GPUOS is complementary to these efforts, providing a primitive that can be embedded within higher-level scheduling frameworks to reduce the launch and dispatch latency for small or transient tasks. In summary, GPUOS unifies two historically distinct threads of work: (1) operating-system-like persistent execution models from the graphics and HPC communities, and (2) compiler and runtime graph optimization frameworks from the ML community. By merging persistent kernels, dynamic function injection, and transparent framework integration, GPUOS offers a general mechanism for bridging the gap between flexible operator composition and efficient device execution.
9
[6] Jack Choquette. Cuda graphs for work submission. NVIDIA Developer Blog, 2019. https://developer.nvidia.com/blog/cuda-graphs/. [7] Patrick H Coppock, Brian Zhang, Eliot H Solomon, Vasilis Kypriotis, Leon Yang, Bikash Sharma, Dan Schatzberg, Todd C Mowry, and Dimitrios Skarlatos. Lithos: An operating system for efficient machine learning on gpus. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages 1–17, 2025. [8] Tri Dao, Daniel Fu, et al. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022. [9] Kshitij Gupta, Jeff A. Stuart, and John D. Owens. A study of persistent threads style gpu programming for gpgpu workloads. In Proceedings of the 2012 Innovative Parallel Computing (InPar), pages 1–14, San Jose, CA, USA, 2012. IEEE. [10] Seung Wook Lee et al. Cuda dynamic parallelism api and performance. NVIDIA Technical Report, 2014. [11] Deepak Narayanan, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. Efficient large-scale language model training on gpu clusters using megatron-lm. arXiv preprint arXiv:2104.04473, 2021. [12] NVIDIA Corporation. CUDA Dynamic Parallelism Technical Brief. NVIDIA Corporation, 2023. CUDA Programming Guide. [13] NVIDIA Corporation. Constant-time graph launch techniques. Technical brief, NVIDIA Corporation, 2024. CUDA 12.3 Release Documentation. [14] NVIDIA Corporation. CUDA Driver API Reference. NVIDIA Corporation, 2024. CUDA Toolkit Documentation. [15] NVIDIA Corporation. Multi-Instance GPU User Guide. NVIDIA Corporation, 2024. NVIDIA Data Center GPU Documentation. [16] NVIDIA Corporation. Multi-Process Service User Guide. NVIDIA Corporation, 2024. CUDA Toolkit Documentation. [17] NVIDIA Corporation. NVRTC: CUDA Runtime Compilation. NVIDIA Corporation, 2024. CUDA Toolkit Documentation. [18] NVIDIA Developer Forums. Kernel launch overhead discussions. Online forum discussion, 2023. NVIDIA Developer Forums. [19] Oak Ridge National Laboratory. Cuda graphs performance analysis. Technical report, Oak Ridge National Laboratory, 2022. [20] PyTorch Team. Pytorch 2.0: The journey to compilation. PyTorch Blog, 2023. https://pytorch.org/blog/pytorch-2.0-release/. [21] PyTorch Team. Pytorch torchscript. https://docs.pytorch.org/docs/ stable/jit.html, 2024. Accessed: 2025-10-30. [22] PyTorch Team. Pytorch/xla eager mode (r2.4). https://docs.pytorch. org/xla/release/r2.4/eager_mode.html, 2024. Accessed: 2025-10-30. [23] Markus Steinberger et al. Whippletree: Task-based scheduling of dynamic workloads on the gpu. In ACM SIGGRAPH, 2014. [24] Markus Steinberger, Michael Kenzel, et al. Softshell: Dynamic scheduling on gpus. In ACM SIGGRAPH Asia, 2012. [25] Google Brain Team. Xla: Tensorflow, compiled. TensorFlow Developer Blog, 2017. [26] Tianyu Wang et al. Improving gpu multi-tenancy through dynamic multi-instance gpu reconfiguration. In Arxiv, 2024. [27] Yangzihao Wang et al. Gunrock: Gpu graph analytics. In ACM Transactions on Parallel Computing, 2017. [28] Yiwei Yang, Tong Yu, Yusheng Zheng, and Andrew Quinn. egpu: Extending ebpf programmability and observability to gpus. In Proceedings of the 4th Workshop on Heterogeneous Composable and Disaggregated Systems, pages 73–79, 2025. [29] Yiwei Yang, Yusheng Zheng, Tong Yu, and Andi Quinn. Hetgpu: The pursuit of making binary compatibility towards gpus. arXiv preprint arXiv:2506.15993, 2025. [30] Chen Zhang, Lingxiao Ma, Jilong Xue, Yining Shi, Ziming Miao, Fan Yang, Jidong Zhai, Zhi Yang, and Mao Yang. Cocktailer: Analyzing and optimizing dynamic control flow in deep learning. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 681–699, 2023.
Conclusion
The era of micro-batch machine learning brings challenges caused by coordination costs. The GPU remains fast in computation, but the cadence of repeated host submissions for small kernels has become the bottleneck. We develop GPUOS that addresses this challenge by making fewer launches— ideally one—and a disciplined device-side runtime that consumes tasks as quickly as the host produces them. Dynamic operator injection preserves flexibility. As models evolve and features arrive, the persistent kernel adapts without restart or relink. It needs only a new function pointer at the right table index. The practical consequence on real hardware is unmistakable: whitespace between kernels vanishes, throughput rises, tail latencies compress, and energy consumption drops proportionally to time saved.
References [1] Amey Agrawal et al. Taming throughput-latency tradeoff in llm inference with sarathi-serve. OSDI, 2024. [2] Timo Aila and Samuli Laine. Understanding the efficiency of ray traversal on gpus. Proc. High Performance Graphics, 2009. [3] Alan Gray. Getting started with cuda graphs. https://developer.nvidia. com/blog/cuda-graphs/, 2019. Accessed: 2025-10-30. [4] Tianqi Chen et al. Tvm: An automated end-to-end optimizing compiler for deep learning. In OSDI, 2018. [5] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Tvm: An automated endto-end optimizing compiler for deep learning. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 578–594, Carlsbad, CA, USA, 2018. USENIX Association. 10
GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion
[31] Shulai Zhang et al. Efficient performance-aware gpu sharing with compatibility and isolation through kernel space interception. In USENIX ATC, 2023.
Conference’17, July 2017, Washington, DC, USA
[32] An Zou et al. Rtgpu: Real-time gpu scheduling of hard deadline parallel tasks with fine-grain utilization. In Arxiv, 2021.
11