Conceptio › Archive › arXiv CS

arXiv CSopen access

Performance Isolation and Semantic Determinism in Efficient GPU Spatial Sharing

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

kernel operatingsystems virtualization

operating systems, kernel, virtualization

arXiv:2603.15042v3 [cs.DC] 3 Apr 2026

Performance Isolation and Semantic Determinism in Efficient GPU Spatial Sharing Zhenyuan Yang∗

Wenxin Zheng∗

Key Laboratory of System Software (Chinese Academy of Sciences) Institute of Software, Chinese Academy of Sciences University of Chinese Academy of Sciences

Shanghai Jiao Tong University

Mingyu Li

Haibo Chen

Key Laboratory of System Software (Chinese Academy of Sciences) Institute of Software, Chinese Academy of Sciences

Key Laboratory of System Software (Chinese Academy of Sciences) Institute of Software, Chinese Academy of Sciences Shanghai Jiao Tong University

Abstract Existing GPU spatial sharing systems face a three-way tradeoff: resource utilization, performance isolation, and semantic determinism. Hardware partitioning suffers from hardware under-utilization. Hardware multiplexing fails to avoid performance interference. Recently proposed software-based GPU kernel slicing reshapes floating-point reduction orders, destroying semantic determinism and inducing catastrophic token drift in generative models. We present DetShare, a transparent spatial sharing system that resolves this trilemma. DetShare introduces GPU coroutine, a novel abstraction that enables logical-to-physical resource decoupling. By dynamically mapping immutable virtual contexts to mutable physical resource via lightweight cooperative migration, DetShare enables extensible, workloadaware scheduling without altering kernel semantics. Evaluations demonstrate DetShare simultaneously achieves high utilization, strong isolation, and absolute semantic determinism (guaranteeing zero token mismatch). In multi-tenant co-location, it improves training throughput by up to 79.2% over temporal sharing and reduces P99 inference tail latency by 15.1%. Its pluggable architecture supports custom policies; compared to the default policy, a TPOT-FIRST policy further reduces SLO violations by 21.2% under dynamic traffic.

Introduction

Artificial intelligence (AI) workloads, such as chatbots [45, 60], autonomous agents [56, 78], and model training [8, 62], have become the dominant consumers of modern cloud GPU clusters. Unlike traditional applications, these AI workloads exhibit highly dynamic resource demands characterized by pronounced temporal variability and rapid phaseshifting. However, existing cloud infrastructure struggles to accommodate this fine-grained dynamism [34, 38, 54]. To satisfy stringent Service-Level Objectives (SLOs), operators are forced to over-provision resources for worst-case peaks [9], ∗ Equal contribution

leaving massive computational capacity trapped and underutilized during off-peak cycles. Such staggering inefficiency directly translates into billions of dollars in stranded capital expenditure and wasted energy [67]. To reclaim this idle capacity and improve hardware utilization, GPU sharing has emerged as a compelling solution, enabling co-located workloads to consume residual compute resources during off-peak periods. Existing GPU sharing mechanisms can be broadly classified into two paradigms: temporal sharing [68–70] and spatial sharing [18, 27, 58]. Temporal sharing, the default sharing mechanism in modern clusters, serializes GPU access through time-slicing. Under resource contention, this forced serialization introduces nontrivial queuing delays and incurs expensive context-switch overhead, causing a dramatic increase in tail latency. In contrast, fine-grained spatial sharing facilitates true parallel execution by multiplexing GPU resources (e.g., Streaming Multiprocessors (SMs)) across co-located workloads, thereby offering a theoretically superior performance envelope. Motivated by this potential, this paper focuses on realizing the full benefits of spatial sharing in practice. To unlock this capability, a practical GPU spatial sharing system must simultaneously satisfy three stringent design objectives: • O1: Resource Utilization. Idle capacity must be promptly reclaimed and utilized to maximize overall hardware efficiency. • O2: Performance Isolation. High-priority workloads must consistently meet their SLOs, regardless of stand-alone or co-located execution. • O3: Semantic Determinism. Identical GPU kernels must consistently yield identical numerical outputs, regardless of stand-alone or co-located execution. Unfortunately, we observe a fundamental trilemma in existing spatial sharing approaches, forcing operators into a bitter compromise. As summarized in Figure 1, existing approaches systematically fail at least one of these critical design objectives:

Hardware Partitioning

Hardware Multiplexing

Software Kernel Slicing

workload-aware scheduler to fluidly migrate contexts across physical resources to match instantaneous priorities. Reduce Order Kernel GPU Die (Multiple Instances) Task A Original Jobs Instance 1 This decoupled architecture achieves all three design obInstance 2 Task B SMs SMs 1 2 3 6 Task C jectives. First, by enabling fluid, fine-grained multiplexing Memory Memory 5 4 of virtual contexts across available physical resources, DetJob 1 Job 2 Job 3 Queue Running SMs SMs SMs H L Value unequal Share seamlessly reclaims idle cycles, maximizing overall Memory Mem Mem Sliced Jobs Job 1 Instance 1 resource utilization (achieving O1). Second, by dynamically Task A 1.1 6.2 1.2 3.2 5.1 Task B Job 2 Instance 2 2.1 3.3 5.2 orchestrating the vCtx-to-pCtx mapping to provision rightTask C Job 3 2.2 4 6.1 Tasks have unpredictable Job 3 has resource, 3.1 sized SM quotas, it effectively mitigates hardware contention completion time but cannot be scheduled Low Resource Utilization Performance Interference Semantic Non-determinism and provides strong performance isolation for high-priority workloads (achieving O2). Third, because this system-level Figure 1. A comparison of existing GPU sharing approaches. indirection is completely transparent, it eliminates the need for GPU kernel slicing or invasive modifications. DetShare • Hardware Partitioning (e.g., MIG [15]): Static and coarsepreserves the original thread-block scheduling order, ensurgrained resource boundaries trap compute capacity within ing absolute semantic determinism and functional correctisolated silos, leaving massive residual resources underness with zero application-level code changes (achieving utilized (failing O1). O3). • Hardware Multiplexing (e.g., NVIDIA MPS [14]): A To capture the complexity of multi-tenant environments, lack of fine-grained control to dynamically prioritize we benchmark DetShare across diverse workloads spanlatency-critical requests leads to unpredictable perforning DNN training, large language model (LLM) inference, mance degradation and severe SLO violations under reand co-located scenarios. Our results demonstrate that Detsource contention (failing O2). Share improves training throughput by up to 79.2% com• Software Kernel Slicing (e.g., kernelet [81], 𝜇Share [33]): pared to temporal sharing, and reduces the P99 tail latency Altering the intra-kernel dispatch order (e.g., thread block of inference by 15.1% under co-location without degrading scheduling) inherently introduces semantic non-determinism. background training throughput. Furthermore, to rigorously Due to floating-point non-associativity, this causes nuverify semantic determinism, we construct a microbenchmerical divergence and destroys functional correctness mark isolating the LM Head and Softmax layers, ubiquitous (failing O3). components across all modern LLM architectures. While Our key observation is that this trilemma fundamentally baseline spatial sharing approaches exhibit a 4% token misstems from the rigid architectural coupling between an applimatch rate across 100 trials due to kernel slicing artifacts, cation’s logical execution view and the underlying physical DetShare strictly preserves identical numerical outputs. It hardware. This strict, direct mapping between logical conguarantees 100% token consistency with standalone executexts and physical resources inherently obstructs dynamic tion, completely immune to interference from co-located multiplexing. Consequently, any attempt at spatial sharing workloads. Finally, demonstrating the extensibility of Detdevolves into a forced compromise. However, breaking this Share under production traces, our workload-aware schedrigid coupling is non-trivial. It requires navigating three uling reduces average inference latency by 69.1%, with the formidable hurdles: (1) enforcing highly efficient cooperaTPOT-First policy further cutting SLO violations by 21.2%. tive preemption; (2) mitigating the latency of context state Contributions. We make the following contributions: and memory migration; and (3) enacting workload-aware • We systematically analyze the fundamental limitations of prioritization for highly dynamic traffic. existing GPU spatial sharing paradigms. We demonstrate To overcome this trilemma and address these core chalthat the rigid architectural coupling between logical exelenges, we present DetShare, a transparent GPU spatial cution and physical hardware inherently forces a spatial sharing system designed to simultaneously achieve high sharing trilemma among resource utilization (O1), perutilization, strong performance isolation, and absolute seformance isolation (O2), and semantic determinism (O3). mantic determinism. DetShare achieves this by completely • We propose GPU coroutines, a novel system-level abstracdecoupling the application’s logical execution view from tion that completely decouples logical virtual contexts the physical hardware resources through a novel abstraction (vCtx) from physical resources (pCtx). This establishes called GPU coroutines. Specifically, logical execution contexts a transparent, lightweight scheduling substrate that envisible to applications (vCtx) are completely detached from ables dynamic, fine-grained multiplexing without alterphysical GPU resources managed by the CUDA runtime ing the monolithic execution semantics. (pCtx). By dynamically binding vCtxs to pCtxs provisioned with varying SM quotas, DetShare establishes a lightweight dispatch and migration substrate. This separates the scheduling mechanism from the policy, enabling a centralized, (e.g., MIG, Partitioned MPS)

(e.g., MPS)

(e.g., 𝜇Share)

≠

• We design and implement DetShare, a workload-aware GPU sharing system based on this abstraction. Extensive evaluations demonstrate that DetShare successfully breaks the trilemma: it improves overall hardware utilization (O1), enforces strong performance isolation for latency-sensitive tasks (O2), and guarantees absolute semantic determinism (O3) with zero application code changes.

MPS [14]) enable fine-grained multiplexing by allowing concurrent GPU kernel dispatch to the same GPU. However, they fundamentally lack robust fault and performance isolation [12, 75]. Without fine-grained control to dynamically prioritize requests, these context-multiplexing approaches offer zero defense against interference under multi-tenant co-location. Consequently, a high-priority, latency-critical inference job can suffer severe head-of-line (HoL) blocking and easily be starved by resource-intensive background training tasks. This unmanaged contention leads to unpredictable execution delays and severe SLO violations. Conversely, to guarantee predictable performance, hardware partitioning approaches (e.g., NVIDIA MIG [15]) achieve strict hardware-level isolation by statically dividing physical resources. While this effectively protects latency-sensitive workloads, this rigid isolation relies on heavyweight reconfiguration mechanisms that necessitate device resets and inevitably lead to significant resource fragmentation [47]. The system cannot fluidly reallocate residual resources to accommodate transient workload spikes, leaving massive compute capacity trapped within isolated silos and failing to maximize efficiency.

Background: The Road of GPU Sharing

In this section, we first introduce the highly dynamic nature of AI workloads that necessitates fine-grained GPU sharing. We then analyze the limitations of existing GPU spatial sharing paradigms, demonstrating how current hardware and software mechanisms force a bitter compromise among resource utilization, performance isolation, and semantic determinism 2.1

Why GPU Spatial Sharing Matters?

AI Workload Characteristics. AI applications have emerged as the dominant consumers of modern cloud computing resources [21, 37, 55]. A defining characteristic of these workloads is their highly dynamic and phase-shifting hardware resource demands. In AI model training, execution constantly alternates between compute-intensive passes and communication-bound gradient synchronization, leaving compute cores severely underutilized during network waits [29, 49]. Similarly, LLM inference exhibits drastic phase disparities between compute-bound prefill and memory-bound autoregressive decoding [3, 36, 82]. Because execution bottlenecks continuously shift, dedicating exclusive GPUs forces operators to over-provision, resulting in massive stranded capacity and inefficiency.

2.3

Temporal Sharing. To reclaim this idle capacity, GPU sharing is essential. Historically, temporal sharing served as the default paradigm [25, 68–70], allowing multiple applications to share a GPU by slicing execution time. However, this forced serialization incurs expensive context-switch overheads[46] and keeps overall utilization strictly bounded by a single active workload’s instantaneous demand [4, 6]. To overcome these temporal bottlenecks, fine-grained spatial sharing has emerged, facilitating true parallel execution by concurrently dispatching and multiplexing physical GPU resources across co-located workloads. 2.2

Resource Utilization vs. Performance Isolation

While spatial sharing offers a theoretically superior performance envelope, relying solely on hardware-backed sharing mechanisms forces operators into a bitter compromise between maximizing overall hardware efficiency and achieving strong performance isolation. In pursuit of maximum resource utilization, lighte-weight multiplexing mechanisms (e.g., CUDA Streams [44] and NVIDIA 3

The Pitfalls of Software-Level Interventions

To alleviate the tension between utilization and isolation, recent software-based sharing frameworks forcibly intervene in kernel execution, primarily through intrusive modifications or dynamic slicing, both of which introduce prohibitive costs. First, intrusive modification of the GPU kernel [36, 58], such as manually fusing compute- and memory-intensive tasks, improve utilization, but lack scalability. Mandating application-specific code rewrites imposes a severe verification tax in production environments [2] and remains too rigid to accommodate unpredictable multi-tenant workloads. Second, to achieve flexibility without manual rewriting, software kernel slicing [13, 33, 81] breaks heavy kernels into granular sub-kernels to avoid HoL blocking. However, this introduces a catastrophic trade-off for generative AI: the destruction of semantic determinism. By dynamically altering intra-kernel dispatch order to fit real-time hardware constraints, slicing inherently injects numerical deviations due to floating-point non-associativity. As we will detail in § 3.2, these micro-divergences accumulate over the course of execution, distorting the final log probability distribution and ultimately inducing incorrect token selection. Consequently, existing spatial sharing paradigms remain trapped in this fundamental trilemma, forcing an inevitable compromise among resource utilization, performance isolation, and semantic determinism.

(a) Tail Latency Spike 6496.9 (72.2x)

104

Isolated Co-located

1959.3 (37.8x)

103 102 101

90.0

TTFT (Prefill)

51.9

TPOT (Decode)

(b) Numerical Deviations in Custom Op Absolute Numerical Deviation

P90 End-to-End Latency (ms)

105

FP16 BF16

Mean Abs. Dev. Max Abs. Dev.

We love operating

Reference Execution (e.g., Standard Blocksize)

20 15

We love operating

Shared Execution (e.g., Altered Blocksize)

5 0

128

512

Vocab

Logit distribution

Logit distribution “principles”

“system”

Vocab

low entropy

“We love operating system design”

“We love operating system principles”

high entropy

auto-regressive decoding (Greedy Sampling).

the SMs are saturated by opaque, non-preemptible background kernels, destroying performance isolation [53]. • Software is Constrained (SLO-Aware): Conversely, while software frameworks can perceive application SLOs, they are bottlenecked by the rigid coupling between logical execution and physical resources. Once a kernel is dispatched, its execution state is firmly anchored to the hardware. Although software can forcibly intervene [7, 28, 59], it faces a lose-lose scenario under contention: either passively endure HoL blocking, or trigger coarse-grained mechanisms where preemption incurs prohibitive overheads or abortion discards computational progress. Furthermore, existing software schedulers [10] typically optimize for one-dimensional objectives, such as peak throughput or tail latency, and thus struggle to adapt to the complex, highly dynamic execution patterns in multi-tenant environments.

spikes experienced by a latency-critical LLM inference stream (e.g., Qwen3-32B) when co-located with a throughput-oriented DNN training job on a Hopper GPU. (b) Semantic Non-Determinism: Absolute numerical deviation of FP16 and BF16 summations compared to the standalone execution. Dynamically splitting the computation grid alters the reduction tree, exposing floating-point nonassociativity.

Deconstructing the Trilemma of Sharing

To understand the root causes of this trilemma, we deconstruct the inherent conflicts within the GPU software-hardware stack. We first analyze the architectural mismatch that pits aggregate throughput against latency isolation (§ 3.1). We then investigate how software-level interventions, in their attempt to bypass this hardware bottleneck, inadvertently destroy semantic determinism (§ 3.2). Based on these insights, we establish the core supply-side imperatives required to transcend this architectural compromise (§ 3.3). 3.1

Logit distribution “design”

Figure 3. The cascading effect of numerical deviations in LLM

2048

Kernel Slicing Granularity (Grid Splits)

Figure 2. (a) Performance Interference: Severe tail-latency

Logit distribution “system”

3.2

The Mirage of Semantic Determinism

To overcome the architectural mismatch outlined in § 3.1, existing software frameworks [13, 18, 36, 58, 73, 81] tend to manipulate the resource demands of GPU kernels. By employing techniques like kernel slicing to dynamically reduce the execution granularity of individual GPU kernel dispatches, these systems attempt to circumvent the scheduling conflicts caused by oblivious hardware. However, dynamically altering the kernel launch configuration introduces a catastrophic trade-off for generative AI: the destruction of semantic determinism.

The Hardware-Software Objective Mismatch

Achieving strong performance isolation while simultaneously maximizing GPU throughput under GPU spatial colocation is notoriously difficult [41]. As illustrated in Figure 2(a), naively multiplexing a latency-critical LLM inference stream (e.g., Qwen3-32B [61]) with a throughput-oriented DNN training job causes severe latency spikes, degrading TPOT by nearly 38×. This tension is not merely an engineering shortcoming, but the result of a fundamental objective mismatch between the hardware and software layers: • Hardware is Blind (Throughput-Oriented): Commodity GPU hardware and their thread-block schedulers are designed predominantly for greedy resource saturation [11, 22]. Their primary built-in objective is to maximize aggregate throughput by keeping all SMs fully occupied. Because the hardware operates without visibility into upper-layer application semantics or SLO requirements, it employs oblivious dispatch policies (e.g., round-robin). Consequently, the hardware cannot distinguish between a highly sensitive millisecond-scale inference phase and a massive background gradient update. High-priority interactive requests suffer from immediate HoL blocking once

The Origin of Semantic Non-Determinism. At the lowest level, the root cause is the IEEE 754 standard for floatingpoint arithmetic [24]. Due to finite precision, floating-point addition is fundamentally non-associative: (𝐴 + 𝐵) + 𝐶 ≠ 𝐴 + (𝐵 + 𝐶). Any variation in the sequence of arithmetic reductions inevitably yields divergent rounding errors. When spatial sharing systems dynamically adjust kernel configurations (e.g., grid and block sizes), they fundamentally reshape the topology of the underlying reduction trees used for parallel computations [30]. This infrastructure-level intervention forces a different reduction order for the floatingpoint operations, inherently triggering the non-associativity issue. Formally, this semantic violation breaks the computational contract. Standard GPU kernels operate under a 4

strict deterministic premise: 𝑅𝑒𝑠𝑢𝑙𝑡 = 𝐹 (𝐷, 𝑆), where the output is solely a function of the input data (𝐷) and model state (𝑆). However, spatial sharing systems shatter this abstraction by implicitly coupling the mathematical reduction order to the cluster’s real-time physical constraints. As the instantaneous resource availability 𝑅(𝑡) forces the system to alter the launch configuration 𝐶 (𝑡), the topology of the reduction tree, denoted as 𝑇 𝑟𝑒𝑒 (𝐶 (𝑡)), is reshaped accordingly. Consequently, the computational output degrades into a stochastic artifact of the co-location environment: 𝑅𝑒𝑠𝑢𝑙𝑡 ′ = 𝐹𝑇 𝑟𝑒𝑒 (𝐶 (𝑡 ) ) (𝐷, 𝑆), rendering strict semantic determinism impossible.

and goals: the hardware blindly pursues aggregate throughput, while the software should consider SLO requirements and expects predictable execution. Forcing these conflicting objectives to tightly couple directly induces the spatial sharing trilemma. The Fallacy of Demand-Side Adaptation. To circumvent this mismatch, existing frameworks intervene on the demand side. They attempt to force the application to adapt to fluctuating hardware availability by intrusively mutating its execution geometry (e.g., dynamically altering grid or block sizes). However, as established in § 3.2, these demandside manipulations inadvertently reshape the mathematical reduction order. Consequently, the system avoids hardware blocking only by sacrificing functional correctness, irrevocably destroying semantic determinism.

Impact on Generative AI and RLHF. This environmentdependent non-determinism extends beyond mere theoretical artifacts; it has severe consequences for modern generative AI paradigms. For LLMs, which rely heavily on lowprecision data types (e.g., FP16 or BF16) for inference, these micro-divergences are significantly amplified, as illuminated in Figure 2(b). Such accumulated deviations skew the final logits, leading to incorrect token selection during the autoregressive decoding phase. As shown in Figure 3, this vulnerability is particularly pronounced during high-entropy reasoning steps, where even minor probability shifts can trigger a cascading divergence from the model’s offline-validated policy. Crucially, this non-determinism severely disrupts advanced training paradigms such as Reinforcement Learning from Human Feedback (RLHF). Micro-divergences introduced during inference trigger a policy shift [50]—the model’s behavior during online rollouts deviates from the policy evaluated during offline training, substantially stalling convergence. While recent application-level frameworks [72, 80] introduce strict “deterministic modes,” they implicitly assume exclusive, unpartitioned access to the entire GPU. When workloads are subjected to dynamic spatial multiplexing at the infrastructure layer, the semantic determinism meticulously preserved by the application is effectively invalidated.

Shifting to Supply-Side Orchestration. To overcome this trilemma, we must revisit the root of the problem: entities with mismatched objectives must be decoupled. A practical sharing system must separate the application’s logical execution view from the underlying hardware constraints. This paradigm shift transforms the problem from demandside adaptation to supply-side orchestration, resting on two principles: • Semantic-Preserving Abstraction: The system must provide an abstraction layer that hides the underlying hardware fragmentation from the application. By maintaining a stable logical execution context for the kernel, the system ensures that numerical results remain invariant, regardless of the physical co-tenancy state. • Supply-Side Orchestration: With the logical context secured and decoupled, the infrastructure can now dynamically orchestrate the physical resource supply. By dynamically adjusting the physical resource supply to match logical demand at runtime, the system enforces strong performance isolation and reclaims idle cycles, matching the software’s demands without ever mutating its core logic.

The Principle for Semantic Determinism. Based on these observations of semantic drift, we argue that a practical spatial sharing system must adhere to a strict negative constraint: it must never mandate dynamic alterations to kernel launch configurations. To guarantee true semantic determinism, the infrastructure must transparently adapt its physical resource provisioning to the workload, rather than forcing the workload’s immutable arithmetic logic to conform to fragmented hardware states.

3.3

DetShare Design

As observed in § 3.3, relying on demand-side interventions forces an inevitable compromise. To escape this spatial sharing trilemma, this section presents DetShare, a transparent system that shifts the paradigm from demand-side manipulation to supply-side orchestration. The linchpin of DetShare’s design is the strict decoupling of logical execution from physical resource provisioning. By establishing an architecture where an application’s execution geometry remains strictly immutable, DetShare guarantees absolute semantic determinism. Simultaneously, by fluidly adapting the underlying physical resource supply to match realtime workload priorities, it reclaims idle cycles and enforces strong performance isolation.

The System Imperative: Decoupling

The severe performance degradation under contention (§ 3.1) are not merely scheduling inefficiencies; they are the inevitable result of a fundamental objective mismatch. Hardware and software possess inherently different capabilities 5

Logical View

Client Application 2

Client Application 1

One-to-One Map vCtx 2

vCtx 1

Kernels Launch

Pluggable Policies CoGPU Scheduler

Interception Shim Layer Mapping

Allocate pCtx1 Kernel 1

4.2 Update

RCK

Kernel 1

RCK

SMs

GPU Memory

pCtx4

pCtx5

… (pCtxs unallocated)

Mapping Table

Physical View

Figure 4. The overall architecture of DetShare.

The pCtx Pool. A naive approach that dynamically instantiates hardware contexts on-demand incurs prohibitive driver latency, destroying microsecond-scale responsiveness. Instead, DetShare maintains a pre-provisioned pool of pCtx instances at initialization, configured with discrete SM quota tiers (e.g., 10%, 25%, 75%). Crucially, a pCtx serves purely as a lightweight execution container rather than a static resource reservation. An idle pCtx in the pool consumes negligible device memory for driver metadata and zero SM compute cycles, as the hardware is only occupied during active GPU kernel dispatch. Therefore, this pooling strategy avoids resource fragmentation or waste, while completely removing control-plane overhead from the critical path to enable zero-allocation-latency binding and dispatching.

The remainder of this section unpacks how DetShare turns this philosophy into a practical system. We first introduce the GPU Coroutine abstraction, the foundation for our logical-physical separation (§4.1). Next, we detail the runtime mechanisms for dynamic binding and cooperative preemption (§4.2). To prevent migration overheads from eroding performance, we then present a set of memory state optimizations (§4.3). Finally, we describe an extensible scheduling framework that leverages these primitives to manage highly dynamic, multi-tenant AI traffic (§4.4). 4.1

Mechanism: GPU Coroutines

To translate the GPU Coroutine abstraction into a practical scheduling substrate, the DetShare runtime dynamically orchestrates these logical entities across physical resources. Rather than relying on static assignments, the runtime actively manages the lifecycle through three critical phases: dynamic binding for initial dispatch, asynchronous signaling for runtime-to-device coordination, and cooperative preemption to resolve real-time resource contention.

pCtx2 K3

SMs

pCtx3

with an adjusted resource quota. Consequently, the fundamental computational contract is perfectly preserved as 𝑅𝑒𝑠𝑢𝑙𝑡 = 𝐹𝑇 𝑟𝑒𝑒 (𝐶 ) (𝐷, 𝑆) ≡ 𝐹 (𝐷, 𝑆), completely independent of the instantaneous physical state 𝑅(𝑡).

Overview

As illustrated in Figure 4, DetShare introduces a dynamic indirection layer between the application and the GPU hardware. The cornerstone of this architecture is the GPU Coroutine, an abstraction that dynamically enforces this decoupling by isolating the application’s logical view of the device from underlying hardware constraints. DetShare formalizes this transparent architecture through two primary entities: • Virtual Context (vCtx): A application-facing handle that encapsulates the logical execution state. DetShare exposes vCtx as a fully transparent abstraction, functionally equivalent to a standard CUDA context from the client’s perspective. • Physical Context (pCtx): A hardware-backed resource container. Each pCtx is a strictly isolated execution unit provisioned with a hard limit on SMs.

Dynamic Binding and Dispatch. DetShare maintains a Binding Table to track the mapping between active vCtx instances and physical resources. Upon intercepting a launch API (e.g., cuLaunchKernel), the runtime consults the global scheduler to determine the optimal placement. The scheduler employs lightweight heuristics—evaluating the kernel’s grid size, block size, and historical latency profiles against the active policy—to estimate resource demands. Depending on the current mapping and resource state, the dispatch process proceeds under two distinct scenarios: • Direct Dispatch: If a pCtx whose SM quota satisfies the kernel’s predicted resource demand is readily available (whether already bound or idle in the pool), the intercepted API is transparently passed to the CUDA driver with negligible overhead. • Context Remapping: If the scheduler detects a priority inversion or resource fragmentation, it triggers a Remapping Event prior to execution. The vCtx is unbound from its current container and undergoes a lightweight migration to a target pCtx with the appropriate quota.

Semantic Determinism via Abstraction. By separating the logical definition of a task from its physical execution capacity, DetShare guarantees semantic determinism. When resource availability 𝑅(𝑡) fluctuates, DetShare does not perform GPU kernel slicing, which would degrade the launch configuration into a time-dependent 𝐶 (𝑡) and alter the reduction tree 𝑇 𝑟𝑒𝑒 (𝐶 (𝑡)). Instead, it employs a migrate-to-yield mechanism: the vCtx (with its immutable configuration 𝐶) is transparently rebound to a different pCtx

Cooperative Preemption via Signaling. To overcome the preemption dilemmas discussed in § 3.1, DetShare introduces a low-latency, cooperative preemption mechanism. As shown in Figure 5, rather than relying on high-overhead host 6

High-Priority Program vCtx2

Low-Priority Program vCtx1

①

④

…

Introspection-Based Input Analysis. To further prune the migration footprint, DetShare performs lightweight data-flow analysis on GPU kernel launch parameters. Prior to dispatch, the runtime intercepts execution to examine parameter structures and their associated device pointers, deriving the minimal closure of memory objects required for correctness. Especially for workloads exhibiting sparse access patterns over large memory allocations, indiscriminately transferring the global heap is highly inefficient. By isolating the kernel-reachable working set instead, DetShare restricts data movement strictly to the touched regions, thereby reducing migration latency.

pCtx1 vCtx2 pCtx2 vCtx1 …

High-Priority First Policy

Preemption Trigger

③ pCtx1 (Large SM Quota) pCtx Migration Kernel 1 Kernel 2 Kernel 3

Update

pCtx2

CoGPU Scheduler ②

High-Priority Kernel ⑤ Mapping pCtx Pool

pCtx1 vCtx1

RCK Interrupt! Kernel Reassign SMs

GPU Memory

pCtx2 (Small SM Quota) Kernel 1 Kernel 2 Kernel 3

RCK SMs

GPU Memory

Lazy Copying pCtx3

pCtx4

Demand-Driven Lazy Copying. Finally, DetShare leverages a lazy migration policy to amortize data transfer costs. During a migration event, only the critical execution state– comprising GPU kernel launch parameters and their immediate dependencies—is eagerly transferred to the target pCtx. The remainder of the working set is migrated on demand: if the resumed kernel accesses a non-resident region, DetShare triggers an asynchronous, incremental transfer in the background. This strategy eliminates synchronous blocking, effectively overlaps communication with computation, and ensures that migration remains transparent to the application’s control flow.

pCtx5 ...

Figure 5. An example of Remapping Event and pCtx migration. interrupts, DetShare transparently injects a lightweight Resident Control Kernel (RCK) into each active pCtx. Operating as a device-side signal handler, the RCK is a persistent, singlethreaded kernel that consumes negligible resources (<0.1% SM occupancy). When the global scheduler dictates preemption, it asserts a flag in a shared memory region mapped to the target pCtx. Upon detecting this signal, the RCK orchestrates a cooperative yield. Unlike traditional mechanisms that abruptly terminate running kernels, DetShare leverages CUDA stream dependencies to pause the execution flow after preemption occurs at the inter-kernel or thread-block boundaries. This safe suspension allows DetShare to synchronize state and initiate the context migration process, effectively yielding physical resources while preserving the logical progress of the preempted task. 4.3

4.4

Extensible Scheduling Framework

As established in § 3.1, relying on static, one-dimensional scheduling objectives inevitably fails to capture the complex, phase-shifting nature of multi-tenant AI workloads. Recognizing that a universally optimal policy for such dynamic environments simply does not exist, the design philosophy of DetShare explicitly rejects the pursuit of a singular scheduling algorithm. Instead, it adopts a decoupled architecture that strictly separates the scheduling mechanism from the policy. This design exposes an extensible framework that allows operators to define pluggable custom logic tailored to specific application SLOs without modifying the underlying runtime.

State Management and Migration

While cooperative preemption safely halts execution, relocating a suspended context introduces a severe bottleneck: memory migration. Because full-heap copies across pCtx instances would dominate context-switching latency and negate scheduling benefits, DetShare replaces monolithic copies with precise memory tracking. It employs three complementary optimizations to migrate only the indispensable state:

The Pluggable Policy Interface. DetShare abstracts the scheduling decision process into a Policy Interface. Users or cluster administrators can implement this interface to define how DetShare reacts to lifecycle events. The framework exposes key hooks such as OnLaunch, OnCompletion, and OnCongestion. By implementing these methods, custom policies can inspect the global state (including the active pCtx pool status, queuing delays, and historical GPU kernel execution profiles) to make granular decisions. For example, a user can implement a "Cost-Aware" policy that prioritizes throughput during off-peak hours but switches to strict latency prioritization when a specific flag is set. DetShare treats these policies as pluggable modules, enabling the runtime to switch strategies dynamically without recompilation.

Fine-Grained Usage Tracing. A naive migration strategy that snapshots the entire GPU memory address space incurs prohibitive overheads, particularly for GPU kernels that access only a fraction of their allocated buffers. Inspired by the memory tracking mechanisms introduced in PhenixOS [66], DetShare leverages fine-grained tracing to monitor the "dirty" state and access patterns of memory regions. By maintaining a precise view of the active memory footprint for each vCtx, the runtime identifies the exact subset of data requiring coherence in the destination pCtx, thereby filtering out stale or idle pages from the migration set. 7

Table 1. Colocated AI Model Training Setup. Workloads are categorized by their dominant resource bottleneck.

Reference Policies. While the framework is extensible, DetShare provides two representative reference policies: • SLO-Aware Policy (Default): This policy targets mixed inference-training workloads. It utilizes a lightweight predictor (based on historical grid sizes and duration) to estimate the "HoL Blocking" time. If an incoming highpriority kernel is predicted to miss its deadline due to a running background task, the policy returns a preempt decision, triggering the mechanism described in §4.2. • TPOT-First Policy: Designed specifically for generative AI, this policy (evaluated in §6.3) introspects the semantic phase of LLM inference. It prioritizes the decoding phase over the prefill phase, ensuring predictable inter-token latency even under heavy contention.

Colocated Workloads Category

Job 1

QoS Enforcement Mechanisms. Regardless of the chosen policy, DetShare operationalizes scheduling decisions through a robust QoS layer. This layer acts as a translation engine, mapping high-level logical directives into concrete physical resource partitions. By leveraging hardwareassisted isolation primitives for both spatial partitioning and temporal throttling, DetShare ensures that the user-defined policies are strictly bounded by hardware-enforced guarantees, preventing low-priority workloads from eroding the performance of critical tasks.

Job 2

A High Compute B Contention C D

ResNet-50 [31] ResNet-101 BERT-Large [17] BERT-Large ResNet-50

Low Compute Contention

1 2

RNN [51] DeepFM [26] ResNet-101

LLM Training

GLM-4-9B [23] (LoRA [32])

• Uncompromised Performance: For mixed training workloads, DetShare improves training throughput by up to 79.2% compared to temporal sharing. For inferencetraining co-location, it reduces the average inference latency by up to 69.1% without degrading background training throughput. • Strict Semantic Determinism: Across 100 end-to-end LLM inference trials using greedy decoding (temperature = 0), DetShare exhibited zero argmax inversions, completely eliminating the incorrect token selections. • Extensible Scheduling Framework: The pluggable TPOT-First policy reduces the TPOT SLO violation rate by an additional 21.2% relative to the default policy under highly dynamic traffic.

Implementation

The DetShare prototype is implemented in approximately 6K lines of C++ and 1K lines of CUDA. To realize the decoupled architecture transparently, DetShare is structured around two primary components: a lightweight interception shim layer and a standalone global scheduler daemon. To interpose on workload submissions without modifying user codes, the shim layer is transparently injected via LD_PRELOAD to intercept CUDA Driver and Runtime APIs (e.g., cuLaunchKernel, cudaMalloc). These API wrappers capture launch metadata and seamlessly enqueue requests into lock-free, shared-memory ring buffers. By leveraging shared memory for Inter-Process Communication (IPC) with the global scheduler, DetShare minimizes submission latency and keeps the direct dispatch overhead negligible. At the execution level, DetShare leverages native hardware primitives to enforce strong physical resource isolation. The global scheduler instantiates physical GPU contexts (the pCtx pool) utilizing NVIDIA Green Contexts [42], which achieve fine-grained spatial partitioning by enforcing hard quotas on SMs. To safely multiplex these hardware units while strictly preserving the execution semantics of GPU kernels, DetShare orchestrates the contexts over NVIDIA’s MPS to guarantee robust address space isolation.

Config

6.1

End-two-End Performance Evaluation

Experimental Setup. We evaluate DetShare across two distinct hardware platforms, an NVIDIA A800 cluster and NVIDIA Hopper architecture GPUs (e.g., H20), to capture diverse resource bottlenecks. Baselines. We compare DetShare against industry-standard sharing mechanisms (NVIDIA MPS and MIG), as well as state-of-the-art software-defined schedulers, including TickTock [64], Orion [58], GSlice [18] and Salus [75]. Since the source code for LithOS [13] is not publicly available, we implemented a faithful reproduction of its scheduling logic based on its design principles to serve as a comparative baseline. Workloads. Our evaluation spans two primary settings: (1) Homogeneous and Heterogeneous Model Training (Table 1), which pairs diverse throughput-oriented jobs; (2) Mixed Workload Colocation (Table 2), which co-schedules heavy background training tasks alongside latency-critical LLM inference streams.

Evaluation

6.1.1 Colocated AI Model Training. Figure 6 reports the normalized throughput across all configurations. DetShare

Our key takeaways from the evaluation are: 8

Figure 6. Normalized throughput of colocated model training tasks (higher is better). DetShare consistently outperforms baselines, particularly in scenarios with high resource contention or varying interference patterns.

consistently achieves superior aggregate throughput by effectively balancing SM utilization and minimizing inter-job interference.

LoRA). DetShare (95%) outperforms hardware-level partitioning (MIG, 93%) by enabling flexible resource redistribution rather than static allocation. Notably, static spatial schedulers like GSlice struggle in this scenario (75%–79%), failing to manage the massive kernel launches and dynamic memory footprints associated with LLMs. DetShare sustains high execution efficiency even under these intense resource demands, highlighting its robust applicability to modern GenAI workloads.

High Compute Contention (Configs A–D). These configurations represent arithmetic-bound scenarios involving compute-intensive models where the GPU’s SMs are saturated. Traditional paradigms degrade performance here either through severe context-switching overheads (temporal sharing) or rigid resource fragmentation (spatial partitioning). Taking Config C (dual BERT-Large) as an example, DetShare attains a normalized throughput of 58% per job. In contrast, rigid partitioning schemes like MIG are limited to 53%, while temporal sharing (Orion) drops to 50% due to high context-switching overheads. By leveraging pCtx pooling and dynamic vCtx-pCtx dispatch without stalling the pipeline, DetShare significantly reduces inter-job interference, maintaining high efficiency even under heavy arithmetic contention and outperforming static heuristics such as GSlice.

Table 2. Mixed Workload Colocation Setup. We pair a throughput-oriented batch job (Job 1) with a latency-critical inference job (Job 2).

Low Compute Contention (Configs 1–2). Workloads such as RNNs and DeepFM exhibit intermittent GPU utilization, often bounded by memory bandwidth or host-device I/O. DetShare effectively harvests these ephemeral idle cycles (“execution bubbles”) by rapidly migrating vCtxs to underutilized pCtxs. In Config 1 (dual RNN jobs), DetShare achieves near-optimal isolation with normalized speeds of 99% per job, effectively matching dedicated execution performance. While baselines like Tick-Tock and Salus plateau around 85%–90%, DetShare exhibits superior scheduling granularity in identifying and filling millisecond-level idle slots.

Colocated Workloads Job 1 (Train) Job 2 (Inference)

Category

Config

CV + LLM Inference

A B

ResNet ResNet

Llama 3 [19] GPT-J [63]

NLP + LLM Inference

1 2

BERT BERT

Llama 3 GPT-J

6.1.2 DNN Training with LLM Inference. To evaluate DetShare in mixed-criticality scenarios, we pair throughputoriented training jobs with latency-sensitive LLM inference requests. The specific workload combinations (Table 2) cover Computer Vision (CV) and NLP models as background tasks, colocated with Llama 3 and GPT-J inference streams. Figure 7 presents the performance trade-offs between batch training throughput and inference tail latency. DetShare demonstrates a superior Pareto frontier compared to all baselines, successfully maintaining strict SLOs for interactive workloads while maximizing the utilization of residual GPU cycles for background training.

LLM Training (Config E). We further evaluate DetShare on large-scale foundation model fine-tuning (GLM-4-9B with 9

×10 2

Stage 1: LM Head MatMul Logit Drift

×10 3

Stage 2: Softmax Probability Drift

Max Number Drift

0 5 2 0 2

Trial Number (Multi-Tenant GPU Sharing Environment)

Baseline (Argmax Drifts: 4)

CoGPU (Argmax Drifts: 0)

100

Absolute Determinism

Figure 8. Semantic Determinism Evaluation. DetShare perfectly preserves numerical outputs and guarantees zero token drift under multi-tenant interference.

6.2

Semantic Determinism Evaluation

Experimental Setup. To rigorously evaluate whether DetShare can preserve the exact outputs of LLM inference in multi-tenant environments, we construct a microbenchmark isolating the LM Head (GEMM) and Softmax layers. These two operators are ubiquitous components across all modern LLM architectures. To isolate the cause of environmentdependent non-determinism from application-level dynamics like continuous batching [30, 74], we implement operators that are mathematically insensitive to upper-layer batch dimension variations. Consequently, under a standalone, exclusive GPU execution environment (our Absolute Determinism Baseline), the computational results remain strictly identical. We then inject co-location interference to simulate a multitenant environment. The baseline approach applies dynamic GPU kernel slicing strategies to accommodate the fluctuating resource availability caused by co-located workloads.

Figure 7. Performance comparison on mixed workloads. (a) DNN training throughput (higher is better) and (b) LLM inference p99 latency (lower is better) under 4 configurations. DetShare balances both efficiency and SLOs.

Guaranteeing Inference SLOs. For latency-critical LLM workloads, DetShare achieves the lowest 99th percentile (p99) latency across all configurations. In Config A, DetShare reduces the p99 latency to 4.5 (normalized units), representing a 2.6× and 2.2× reduction compared to Orion (12.0) and MIG (10.0), respectively. Similarly, in the more memory-intensive Config B (ResNet + GPT-J), DetShare maintains a p99 latency of 5.9, significantly outperforming Orion (15.0) and MPS (10.2). Hardware partitioning methods like MIG enforce inflexible spatial boundaries; they lack the dynamic prioritization required to accommodate bursty inference traffic. Conversely, existing software schedulers like Orion fail to provide strong performance isolation. Under coarse-grained task co-location, critical inference GPU kernels are easily starved by heavy training workloads, resulting in severe interference. DetShare, by contrast, couples compute resource isolation via Green Contexts with cooperative preemption driven by injected RCKs. This joint mechanism ensures strong performance isolation and immediate physical resource reclamation for latency-sensitive inference tasks, without violating semantic determinism.

Results and Analysis. As shown in Figure 8, this dynamic slicing alters the reduction order of floating-point operations, introducing severe numerical artifacts in both the output logits and the normalized probabilities. Across 100 trials, the baseline spatial sharing approach exhibits a 4% argmax drift rate. This means the model alters its predicted output—producing a different word—simply due to kernel slicing artifacts triggered by underlying resource contention. In contrast, DetShare fundamentally avoids this pitfall by refusing to alter the kernel launch configuration under interference, aligning with the core principles outlined in our design. Our evaluation demonstrates that DetShare strictly preserves identical numerical outputs and guarantees 100% token consistency with standalone execution. By decoupling the logical execution logic from physical resource allocation, DetShare proves to be immune to multi-tenant interference, preserving semantic determinism.

Maximizing Training Throughput. While prioritizing inference, DetShare does not compromise the progress of background training jobs (Job 1). For example, in Config 1, DetShare achieves a normalized throughput of 60%, outperforming LithOS (58%), MPS (55%), and Orion (53%). Because inference workloads exhibit bursty arrival patterns, static partitioning (MIG/MPS) leaves compute units underutilized during idle gaps. DetShare’s dynamic vCtx-pCtx remapping allows the background training job to instantly reclaim these idle cycles, maximizing aggregate GPU utilization.

6.3

Scheduling Extensibility Evaluation

To evaluate the extensibility of DetShare’s decoupled scheduling framework, we present a case study demonstrating how 10

Figure 9. End-to-end evaluation on Azure, LongBench, and BurstGPT traces: Throughput (higher is better), Latency Breakdown and SLO Violations (lower is better). DetShare reduces average and tail latency while maintaining throughput. Adopting the TPOT-First strategy further decreases decoding tail latency.

custom scheduling policies can be seamlessly integrated to meet diverse production demands.

6.3.2 Performance Analysis. Figure 9 depicts the throughput, latency breakdown, and SLO violation rates. Overall, DetShare-backed SGLang significantly outperforms the baseline. For instance, on the Azure trace, DetShare reduces the average TTFT by 69.1% (0.55s to 0.17s) while maintaining comparable throughput. Furthermore, DetShare demonstrates superior stability by reducing the P99 TPOT on LongBench by 4.7× (1723.9ms vs. 366.2ms). Given that strict tail-latency guarantees during token generation are paramount for a seamless interactive user experience, this dramatic reduction highlights DetShare’s efficacy even in severely memory-constrained scenarios.

6.3.1 Experimental Setup. To demonstrate DetShare’s transparency and effectiveness at the infrastructure layer, we evaluate an unmodified instance of the state-of-the-art serving system, SGLang [80], running entirely on top of DetShare to serve the Llama-3.1 model. We execute these workloads on a Hopper architecture GPU (e.g., H20) and compare DetShare against a baseline of SGLang executing on a default, exclusively provisioned GPU. We run across three traces representing distinct production scenarios: • Azure LLM Trace 2024 [57]: Highly stochastic request arrival times, testing general adaptability. • LongBench [5]: Long-context tasks imposing significant pressure on GPU memory capacity and bandwidth. • BurstGPT [65]: Extreme traffic spikes challenging the scheduler’s queue management. We analyze DetShare under two pluggable configurations: the Default Policy (balancing resource fairness) and the TPOT-First Policy (explicitly prioritizing the decoding phase).

Impact of TPOT-First Policy. The TPOT-First Policy exhibits enhanced robustness in minimizing generation latency violations, particularly under fluctuating loads. On the Azure trace, the TPOT-First policy reduces TPOT SLO violations by 21.2% compared to the Default policy. The BurstGPT benchmark, characterized by extreme traffic spikes, exposes the true efficacy of the TPOT-First policy. As shown in Figure 9, this policy reduces TPOT SLO violations by 46.1% (17.77% to 9.58%) and dramatically lowers the P90 TPOT from 847.7ms to 262.9ms. However, we observe a concomitant increase in TTFT SLO violations, which rise from 4.33% to 6.37%. 11

Normalized Overhead

Exclusive GPU Baseline

1.0

1.00 1.04

CoGPU

1.00

Multi-GPU Spatial Sharing. Modern generative AI workloads typically rely on distributed execution across multiple GPUs [40, 79]. While DetShare currently focuses on single GPU spatial sharing, its principle—decoupling virtual contexts from physical hardware—provides a conceptual framework for distributed environments. Extending to multi-GPU settings often necessitates heavyweight cross-device state migration via interconnects (e.g., NVLink), which precludes fast preemption. GPU coroutines offer a potential mitigation: by encapsulating execution state within the vCtx, distributed preemption could be formulated as gang-scheduled context remapping in the control plane. However, realizing this requires addressing challenges in distributed synchronization, network latency, and collective communication state management. We leave it as future work.

1.12

0.5 0.0

Switching Overhead

Preemption Overhead

Figure 10. Overhead analysis. DetShare incurs only 4% and 12% overhead for context switching and preemption, respectively, normalized to the exclusive execution. Analysis. We attribute this to a necessary scheduling tradeoff. Under extreme burstiness, the TPOT-First policy proactively delays the scheduling of new prefill requests (increasing TTFT) to reserve compute and memory bandwidth for ongoing decoding phases. This strategy effectively insulates active requests from interference caused by sudden spikes in arrival rates, ensuring a smooth generation experience at the cost of slightly higher initial queuing delays. 6.4

GPU Spatial Sharing. Spatial sharing partitions hardware to enable concurrent execution. Native mechanisms include intra-context CUDA Streams and static partitioning tools like NVIDIA MPS and MIG [1, 14, 15]. To address the inherent fate-sharing vulnerabilities of MPS, orthogonal software fault isolation (SFI) frameworks like Guardian [48] have been proposed to prevent rogue kernels from triggering devicewide crashes. Systems such as GSlice [18], GPUlets [12], and COLTI [39] build upon these primitives to construct higherlevel spatio-temporal abstractions. Recent literature pushes for finer-grained multiplexing [13, 33, 81], such as LithOS, which slice kernels to pack execution fragments. However, these invasive techniques necessitate user-kernel modifications, risking behavioral inequivalence and complicating verification.

Overhead Analysis

Figure 10 quantifies the performance overhead introduced by DetShare, normalized to the exclusive GPU baseline. Overall, DetShare incurs a marginal overhead of 4% for context switching and 12% for preemption. Analysis. The context switching overhead is kept minimal because the pCtx leverages Context Memory Tracing to avoid redundant data migration between device and host memory. The 12% preemption overhead is primarily dominated by the internal synchronization costs when the RCK interrupts the GPU execution flow. Given the substantial reductions in tail latency and SLO violations (e.g., 4.7× lower P99 TPOT), we argue that this moderate overhead is a highly favorable trade-off for multi-tenant production environments.

Related Work

Semantic Determinism. Semantic determinism is conventionally maintained at the hardware layer through strict adherence to IEEE 754 arithmetic standards, while being sensitive to architectural shifts across GPU vendors [71, 72, 76]. At the application layer, frameworks ensure reproducibility via strict deterministic modes or carefully managed algorithmic optimizations (e.g., fixed tile sizing) [30, 52]. To our knowledge, DetShare is the first work to identify that spatial sharing optimizations at the system infrastructure layer (e.g., dynamic kernel slicing) invalidate the determinism guaranties established by the application layer. By maintaining immutable GPU kernel configurations, DetShare eliminates infrastructure-induced non-determinism.

Discussion

Scope of Determinism Guarantees. Non-determinism in modern AI pipelines stems from multiple sources, including hardware architectural shifts [71, 76] (e.g., migrating across disparate GPU vendors) and algorithmic optimizations [16, 43]. DetShare does not attempt to resolve these out-of-scope stochasticities. Instead, our design advocates for a strict separation of concerns. By guaranteeing that infrastructure-level spatial multiplexing injects zero environmental stochasticity, DetShare provides a deterministic execution substrate. This clean system abstraction ensures that any numerical deviations are strictly bounded to the application’s own algorithmic choices, empowering developers to reason about and control determinism without opaque interference from the underlying resource scheduler.

GPU Memory Management and Migration. To accommodate the massive memory footprints of modern AI workloads, existing systems typically mitigate the memory capacity wall via hierarchical offloading or efficient state transfer. To alleviate capacity constraints, systems like DeepUM [35] enable transparent tensor migration, while frameworks such as ServerlessLLM [20] and BlitzScale [77] leverage multi-tier 12

storage architectures (e.g., DDR and SSDs) for fast, hierarchical model loading and weight swapping. For execution state management, PhoenixOS [66] introduces fine-grained memory tracking to minimize transfer overhead during OSlevel checkpoint and restore. In contrast, DetShare adapts and highly optimizes memory trace mechanisms specifically for spatial sharing of GPU. These hierarchical loading and capacity-expansion techniques are entirely orthogonal to DetShare’s compute-centric scheduling; integrating them presents a natural avenue for future work to maximize both execution efficiency and multi-tenant memory availability.

learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages. [9] Guoyu Chen, Srinivasan Subramaniyan, and Xiaorui Wang. 2024. Latency-Guaranteed Co-Location of Inference and Training for Reducing Data Center Expenses. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). 473–484. doi:10.1109/ ICDCS60910.2024.00051 [10] Quan Chen, Hailong Yang, Jason Mars, and Lingjia Tang. 2016. Baymax: QoS Awareness and Increased Utilization for Non-Preemptive Accelerators in Warehouse Scale Computers. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems (Atlanta, Georgia, USA) (ASPLOS ’16). Association for Computing Machinery, New York, NY, USA, 681–696. doi:10.1145/2872362.2872368 [11] KyungWoon Cho and Hyokyung Bahn. 2020. Performance Analysis of Thread Block Schedulers in GPGPU and Its Implications. Applied Sciences 10, 24 (2020). doi:10.3390/app10249121 [12] Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving heterogeneous machine learning models on {Multi-GPU} servers with {Spatio-Temporal} sharing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 199– 216. [13] Patrick H Coppock, Brian Zhang, Eliot H Solomon, Vasilis Kypriotis, Leon Yang, Bikash Sharma, Dan Schatzberg, Todd C Mowry, and Dimitrios Skarlatos. 2025. LithOS: An operating system for efficient machine learning on GPUs. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 1–17. [14] NVIDIA Corporation. 2025. CUDA Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/ Describes the MPS client-server model that multiplexes multiple processes into a single CUDA context to reduce context-switch overhead and enable concurrent kernel execution.. [15] NVIDIA Corporation. 2025. NVIDIA Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ Describes GPU partitioning into multiple isolated GPU instances with dedicated compute, cache, and memory resources, enabling spatial sharing with strong isolation.. [16] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FLASHATTENTION: fast and memory-efficient exact attention with IO-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 1189, 16 pages. [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. [18] Aditya Dhakal, Sameer G Kulkarni, and K. K. Ramakrishnan. 2020. GSLICE: controlled spatial sharing of GPUs for a scalable inference platform. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC ’20). Association for Computing Machinery, New York, NY, USA, 492–506. doi:10.1145/3419111.3421284 [19] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kulkarni, Gaurav Goel, Kanshul Nguyen, Punit Kulkarni, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783 (2024). [20] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: LowLatency Serverless Inference for Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 135–153. https: //www.usenix.org/conference/osdi24/presentation/fu

Conclusion

Modern GPU spatial sharing has long been trapped in a fundamental trilemma among resource utilization, performance isolation, and semantic determinism. DetShare resolves this trilemma through a transparent spatial sharing system that decouples logical execution contexts from physical hardware resources via a novel GPU coroutine abstraction.

References [1] 2025. CUDA Runtime API :: CUDA Toolkit Documentation. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART_ _STREAM.html. (Accessed on 01/12/2025). [2] A World-Wide Leading AI Company Infrastructure Team. 2026. Private Communication regarding Production GPU Sharing Constraints. Personal Communication. Unpublished industry insights. [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117–134. [4] Rachata Ausavarungnirun, Vance Miller, Joshua Landgraf, Saugata Ghose, Jayneel Gandhi, Adwait Jog, Christopher J Rossbach, and Onur Mutlu. 2018. Mask: Redesigning the gpu memory hierarchy to support multi-application concurrency. ACM SIGPLAN Notices 53, 2 (2018), 503–518. [5] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). [6] Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020. {PipeSwitch}: Fast pipelined context switching for deep learning applications. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 499–514. [7] Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020. PipeSwitch: fast pipelined context switching for deep learning applications. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). USENIX Association, USA, Article 28, 16 pages. [8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot 13

[35] Jaehoon Jung, Jinpyo Kim, and Jaejin Lee. 2023. Deepum: Tensor migration and prefetching in unified memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 207–221. [36] Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. Pod-attention: Unlocking full prefill-decode overlap for faster llm inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 897–912. [37] Yueying Li, Zhanqiu Hu, Esha Choukse, Rodrigo Fonseca, G Edward Suh, and Udit Gupta. 2025. Ecoserve: Designing carbon-aware ai inference systems. arXiv preprint arXiv:2502.05043 (2025). [38] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 663–679. https://www.usenix.org/conference/ osdi23/presentation/li-zhouhan [39] Jaiaid Mobin, Avinash Maurya, and M Mustafa Rafique. 2023. COLTI: Towards Concurrent and Co-located DNN Training and Inference. In Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing. 309–310. [40] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 58, 15 pages. doi:10.1145/3458817.3476209 [41] Kelvin K. W. Ng, Henri Maxime Demoulin, and Vincent Liu. 2023. Paella: Low-latency Model Serving with Software-defined GPU Scheduling. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 595–610. doi:10.1145/3600006.3613163 [42] NVIDIA. 2025. CUDA driver API – Green Contexts. https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA_ _GREEN__CONTEXTS.html Accessed: 2025-12-10. [43] NVIDIA Corporation. 2024. cuBLAS Library Documentation. https: //docs.nvidia.com/cuda/cublas/index.html Accessed: 2026-03-26. [44] NVIDIA Corporation. 2024. CUDA C++ Programming Guide. https: //docs.nvidia.com/cuda/cuda-c-programming-guide/index.html Accessed: 2024. [45] OpenAI. 2023. ChatGPT. https://chat.openai.com. Accessed: 2025-1128. [46] Jason Jong Kyu Park, Yongjun Park, and Scott Mahlke. 2015. Chimera: Collaborative Preemption for Multitasking on a Shared GPU. In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (Istanbul, Turkey) (ASPLOS ’15). Association for Computing Machinery, New York, NY, USA, 593–606. doi:10.1145/2694344.2694346 [47] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2025. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (Buenos Aires, Argentina) (ISCA ’24). IEEE Press, 118–132. doi:10. 1109/ISCA59077.2024.00019 [48] Manos Pavlidakis, Giorgos Vasiliadis, Stelios Mavridis, Anargyros Argyros, Antony Chazapis, and Angelos Bilas. 2024. Guardian: Safe GPU Sharing in Multi-Tenant Environments. In Proceedings of the 25th International Middleware Conference (Hong Kong, Hong Kong) (Middleware ’24). Association for Computing Machinery, New York,

[21] Gartner. 2025. Gartner Says AI-Optimized IaaS Is Poised to Become the Next Growth Engine for AI Infrastructure. https://www.gartner.com/en/newsroom/press-releases/2025-1015-gartner-says-artificial-intelligence-optimized-iaas-is-poisedto-become-the-next-growth-engine-for-artificial-intelligenceinfrastructure Accessed: 2025-11-28. [22] Guin Gilman, Samuel S Ogden, Tian Guo, and Robert J Walls. 2021. Demystifying the placement policies of the NVIDIA GPU thread block scheduler for concurrent kernels. ACM SIGMETRICS Performance Evaluation Review 48, 3 (2021), 81–88. [23] GLM-4 Team and Zhipu AI. 2024. GLM-4: Towards Open Source Language Models for Academic Research. arXiv preprint arXiv:2406.12793 (2024). [24] David Goldberg. 1991. What every computer scientist should know about floating-point arithmetic. ACM computing surveys (CSUR) 23, 1 (1991), 5–48. [25] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like clockwork: performance predictability from the bottom up. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). USENIX Association, USA, Article 25, 20 pages. [26] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI). 1725–1731. [27] Bing-Shiun Han, Tathagata Paul, Zhenhua Liu, and Anshul Gandhi. 2024. KACE: Kernel-Aware Colocation for Efficient GPU Spatial Sharing. In Proceedings of the 2024 ACM Symposium on Cloud Computing (Redmond, WA, USA) (SoCC ’24). Association for Computing Machinery, New York, NY, USA, 460–469. doi:10.1145/3698038.3698555 [28] Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 539– 558. https://www.usenix.org/conference/osdi22/presentation/han [29] Sayed Hadi Hashemi, Sangeetha Abdu Jyothi, and Roy Campbell. 2019. TicTac: Accelerating Distributed Deep Learning with Communication Scheduling. In Proceedings of Machine Learning and Systems, A. Talwalkar, V. Smith, and M. Zaharia (Eds.), Vol. 1. 418–430. https://proceedings.mlsys.org/paper_files/paper/2019/file/ 94cb28874a503f34b3c4a41bddcea2bd-Paper.pdf [30] Horace He and Thinking Machines Lab. 2025. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism (2025). doi:10.64434/tml.20250910 https://thinkingmachines.ai/blog/defeatingnondeterminism-in-llm-inference/. [31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 770–778. [32] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR). [33] Wenhao Huang, Zhaolin Duan, Laiping Zhao, Yuhao Zhang, Yanjie Wang, Yiming Li, Yihan Wang, Yichi Chen, Zhihang Tang, Kang Chen, Deze Zeng, Wenxin Li, and Keqiu Li. 2026. µShare: Non-Intrusive Kernel Co-Locating on NVIDIA GPUs. 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA) (2026), 1–14. https://api.semanticscholar.org/CorpusID:286243747 [34] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of large-scale multitenant GPU clusters for DNN training workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 947–960.

NY, USA, 313–326. doi:10.1145/3652892.3700768 [49] Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo. 2019. ByteScheduler: A generic communication scheduler for distributed DNN training acceleration. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). 516–529. [50] Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, and Philipp Krähenbühl. 2026. Entropy-Preserving Reinforcement Learning. In International Conference on Learning Representations (ICLR). [51] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. Nature 323, 6088 (1986), 533–536. [52] SGLang Contributors. 2025. Deterministic Inference — SGLang Documentation. https://docs.sglang.io/advanced_features/deterministic_ inference.html. Accessed: 2026-03-26. [53] Weihang Shen, Mingcong Han, Jialong Liu, Rong Chen, and Haibo Chen. 2025. XSched: preemptive scheduling for diverse XPUs. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 37, 22 pages. [54] Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, and Mark Russinovich. 2022. Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads. arXiv preprint. https://www.microsoft.com/en-us/research/publication/singularityplanet-scale-preemptive-and-elastic-scheduling-of-ai-workloads/ [55] Tom Sorensen and Bob Sorensen. 2024. Cloud-based AI Activity for HPC: Widespread but Primarily Exploratory. Technical Report HR4.0492.09.20.2024. Hyperion Research. https: //hyperionresearch.com/wp-content/uploads/2024/09/HyperionResearch-Special-Report-AI-in-the-Cloud-September-2024.pdf Accessed: 2025-11-28. [56] Peter Steinberger and OpenClaw Contributors. 2025. OpenClaw: Personal AI Assistant. [57] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In Proceedings of the 31st IEEE International Symposium on High-Performance Computer Architecture (HPCA). Best Paper Award. [58] Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, fine-grained gpu sharing for ml applications. In Proceedings of the Nineteenth European Conference on Computer Systems. 1075–1092. [59] Ivan Tanasic, Isaac Gelado, Javier Cabezas, Alex Ramirez, Nacho Navarro, and Mateo Valero. 2014. Enabling preemptive multiprogramming on GPUs. In Proceeding of the 41st Annual International Symposium on Computer Architecuture (Minneapolis, Minnesota, USA) (ISCA ’14). IEEE Press, 193–204. [60] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). [61] Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388 [62] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, MarieAnne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] https://arxiv.

org/abs/2302.13971 [63] Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. EleutherAI. [64] Guanhua Wang, Kehan Wang, Kenan Jiang, Xiangjun Li, and Ion Stoica. 2021. Wavelet: Efficient DNN training with tick-tock scheduling. Proceedings of Machine Learning and Systems 3 (2021), 696–710. [65] Yuxin Wang, Yibo Chen, Zhaozhu Li, Xinyu Kang, Yinan Fang, Yangtian Zhou, Yujie Zheng, Zhennan Tang, Xiuming He, Rong Guo, Xin Wang, Qiang Wang, Aoying Zhou, and Xiaowen Chu. 2025. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). [66] Xingda Wei, Zhuobin Huang, Tianle Sun, Yingyi Hao, Rong Chen, Mingcong Han, Jinyu Gu, and Haibo Chen. 2025. PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 996–1013. doi:10.1145/3731569.3764813 [67] Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA, 945–960. https://www.usenix.org/conference/nsdi22/ presentation/weng [68] Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Transparent {GPU} sharing in container clouds for deep learning workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 69–85. [69] Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, et al. 2018. Gandiva: Introspective cluster scheduling for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 595–610. [70] Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. 2020. {AntMan}: Dynamic scaling on {GPU} clusters for deep learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 533–548. [71] Peichen Xie, Yang Wang, Fan Yang, and Mao Yang. 2025. MMASim: Bit-Accurate Reference Model of Tensor Cores and Matrix Cores. arXiv:2511.10909 [cs.AR] https://arxiv.org/abs/2511.10909 [72] Peichen Xie, Xian Zhang, and Shuo Chen. 2025. RepDL: Bit-level Reproducible Deep Learning Training and Inference. arXiv:2510.09180 [cs.LG] https://arxiv.org/abs/2510.09180 [73] Qiumin Xu, Hyeran Jeon, Keunsoo Kim, Won Woo Ro, and Murali Annavaram. 2016. Warped-slicer: efficient intra-SM slicing through dynamic resource partitioning for GPU multiprogramming. In Proceedings of the 43rd International Symposium on Computer Architecture (Seoul, Republic of Korea) (ISCA ’16). IEEE Press, 230–242. doi:10.1109/ISCA.2016.29 [74] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538. [75] Peifeng Yu and Mosharaf Chowdhury. 2020. Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications. In Proceedings of the 3rd MLSys Conference (MLSys). Austin, TX, USA. [76] Anwar Hossain Zahid, Ignacio Laguna, and Wei Le. 2025. Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs. In Proceedings of the SC ’24 Workshops of the International Conference on High Performance Computing, Network, Storage, and 15

Analysis (Atlanta, GA, USA) (SC-W ’24). IEEE Press, 547–557. doi:10. 1109/SCW63240.2024.00077 [77] Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. BLITZSCALE: fast and live large model autoscaling with O(1) host caching. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 16, 19 pages. [78] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM Agents Are Experiential Learners. Proceedings of the AAAI Conference on Artificial Intelligence 38, 17 (Mar. 2024), 19632–19642. doi:10.1609/aaai.v38i17.29936 [79] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Yin, Jianyu Huang, et al. 2022. Alpa: Automating inter-and Intra-Operator tensor-distributed pipeline parallel compute graph. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI

22). 359–378. [80] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS). [81] Jianlong Zhong and Bingsheng He. 2014. Kernelet: High-Throughput GPU Kernel Executions with Dynamic Slicing and Scheduling. IEEE Transactions on Parallel and Distributed Systems 25, 6 (2014), 1522–1532. doi:10.1109/TPDS.2013.257 [82] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. 2025. {NanoFlow}: Towards optimal large language model serving throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 749–765.

Record · ID 2722 · SHA-256 468fb9b6ef20372d
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.