Conceptio › Archive › arXiv CS

arXiv CSopen access

Strait: Perceiving Priority and Interference in ML Inference Serving

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Strait: Perceiving Priority and Interference in ML Inference Serving Haidong Zhao

Nikolaos Georgantas

Inria & Sorbonne University Paris, France

Inria Paris, France

intrusions in workplaces or production premises may take precedence over other workloads [86, 103]. Second, the growing adoption of on-premises servers [66, 69, 88] for timely, reliable processing [16, 95] makes resource optimization more critical than in cloud environments [19, 64, 76]. Serving systems typically employ batching to consolidate multiple requests for the same model, thereby improving GPU utilization by amortizing off-chip memory accesses. Temporal sharing [29, 30, 42, 79] processes batches sequentially, providing relatively predictable inference latency. However, it may be susceptible to Head-ofLine (HOL) blocking, which can lead to GPU underutilization. Conversely, spatial sharing leverages concurrency to enhance GPU utilization [26, 34, 40, 74, 91, 99]; however, it introduces interference during concurrent kernel execution and data transfers, impeding latency estimation and potentially compromising deadline satisfaction. The economic impact of deadline violations has been quantitatively studied in online services; e.g., Amazon reports that every 100 ms of additional latency results in a 1% loss in sales [38, 73]. In on-premises scenarios, the consequences can also be severe. For instance, in high-throughput manufacturing, defective items may escape certain quality inspections, as interruptions in the manufacturing process would incur noticeable production losses, e.g., $22,000 per minute in the automotive sector [32, 84]. However, allowing defective items to reach customers may ultimately undermine both economic outcomes and brand reputation. Given that a GPU can serve hundreds to thousands of requests per second, even a 1 percentage-point (pp) drop in deadline satisfaction can cause deadline-violated requests to scale with the number of GPUs and the duration of serving. Nevertheless, existing techniques for supporting task prioritization under high GPU utilization are limited at both the hardware and software levels. Regarding internal GPU scheduling, CUDA stream priority [3] provides a hardware-level hint to favor operations in higher-priority streams. However, its efficacy as a standalone solution remains limited; concurrent kernels can still contend for shared GPU resources, and depending on workload characteristics, high-priority (HP) tasks may experience even more severe interference than low-priority (LP) tasks (Section 2.1). Regarding task scheduling, priority scheduling can prioritize HP tasks [11, 13, 102]; however, its efficacy may be undermined by interference from currently executing or subsequently scheduled tasks,

arXiv:2604.28175v1 [cs.LG] 30 Apr 2026

Abstract Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present Strait, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on lowpriority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.

Introduction

Machine learning (ML) techniques and already-trained deep neural network (DNN) models are deployed to power intelligent applications. Since ML inferences are both resourceintensive and latency-sensitive workloads, accelerators such as GPUs1 , which offer massive parallel processing capabilities [31], are employed to meet these demands. ML inference serving systems can multiplex system resources across deployed DNN models to enhance cost efficiency while delivering satisfactory performance. Nonetheless, existing systems may exhibit suboptimal performance across many onpremises scenarios. First, scheduling inference requests by priority levels may be insufficiently supported, as existing inference serving systems are primarily tailored to online services. Consequently, these systems typically aim to optimize overall performance in terms of target metrics [26, 29, 30, 42, 46, 79, 94]. However, certain scenarios can exhibit varying priority levels [43, 66, 80], urgency, and sensitivity to deadline violations. For example, in industrial monitoring, tasks such as quality control [47, 48, 51, 87, 95]—inspecting components for defects—and equipment health monitoring [20, 27, 33, 82] are deemed more critical than workloads such as indoor temperature control. Additionally, detecting unsafe operations or 1 This paper adopts terminology from NVIDIA GPUs

Haidong Zhao and Nikolaos Georgantas

which can still cause HP tasks to miss their deadlines. Consequently, a critical concern remains: whether scheduling a new batch would violate the deadlines of ongoing tasks or fail to meet its own deadline. Worse, commodity GPUs may not support true task-level preemption [43, 65, 68, 80, 83], and inference is often executed as a compiled computation graph [12, 14]. These factors virtually eliminate any opportunity for remediation once scheduling decisions are dispatched. Accordingly, effective task prioritization and deadline satisfaction are inherently constrained. To this end, we introduce Strait, an ML inference serving system designed to enhance deadline satisfaction for dualpriority inference traffic under high GPU utilization. Strait accounts for potential interference, task priority levels, and runtime overheads when making scheduling decisions. First, we model interference in data transfer based on the observation that PCIe data transfer with pinned memory follows a FIFO process. Subsequently, we propose a prediction model for kernel execution interference. This model is grounded in GPU characteristics and posits that interference exhibits an exponential growth pattern with increasing resource pressure. Moreover, this prediction model can be dynamically updated using inference feedback, as static models may suffer from performance degradation when workload characteristics drift or when directly applied to different GPU types. Finally, we design a priority-aware scheduling algorithm that integrates interference prediction to guide its decisions. Its objective is to meet the deadline of the scheduled batch while reducing deadline violations for running batches to the extent possible. To summarize, we make the following contributions: In Section 2, we demonstrate that task prioritization in existing serving systems is limited, as are interference prediction approaches for inference workloads. In Section 3.2, we model data transfer interference and propose an adaptive prediction model to estimate kernel execution interference. In Section 3.3, we design a priority-aware scheduling method that explicitly manages interference at scheduling time. In Section 5, we present evaluation results showing that Strait reduces deadline violations for HP tasks by 1.02 to 11.18 pp, while incurring an acceptable performance trade-off for LP tasks. We further evaluate the accuracy, limitations, and adaptability of the interference prediction approach, assessing its sustainability using realistic production traces [78].

batches to saturate GPU resources, incurs stalls from sequential operations (e.g., data transfer and kernel execution), and can lead to HOL blocking. However, spatial sharing introduces inevitable interference, as concurrent batches can contend for shared resources. Additionally, developers lack fine-grained control once kernels are dispatched to nonpreemptive GPUs [43, 68, 80, 83], and internal GPU scheduling mechanisms remain proprietary [18, 39, 97]. Furthermore, inference workloads are often compiled into computation graphs [12, 14], where execution proceeds continuously from the first kernel to the last without interruption. We next examine the limitations of task prioritization techniques in existing serving systems, as well as interference prediction approaches. The experiments are conducted on an NVIDIA L4 GPU with the Ada Lovelace architecture, unless stated otherwise. 2.1

Scheduling Inference with Priorities

When scheduling inference requests on GPUs, existing inference serving systems may exhibit suboptimal task prioritization. For instance, TensorFlow Serving (TFS) includes a priority queue mechanism for task scheduling in its codebase [13]; however, to our knowledge, this feature has not been officially released. NVIDIA’s Triton Inference Server [11] explicitly supports configuring the CUDA stream priority [3] mechanism to influence internal GPU scheduling. To evaluate these mechanisms, we adopt the TFS design and employ task priority scheduling and CUDA stream priority. TFS scans each model’s task queue in a round-robin manner and submits a batch once it meets scheduling criteria [13]; we replace this with First-Come, First-Served (FCFS), as our empirical results indicate that it reduces tail latency and improves deadline satisfaction. We co-locate an HP task with an LP task in each scenario and restrict the maximum batch concurrency to two, allowing co-location only between batches from different models. Figure 1 presents the results, where a ResNet-50 [44] is sequentially co-located with a relatively lightweight YOLOv8n [52], a compute-intensive VGG-19 [81], and a memoryintensive RoBERTa-B [61]. Across the scenarios, the request arrival rates follow a 3:2:1 ratio, with the load evenly balanced between the co-located models at each priority level. Although the request volume for ResNet-50 decreases across the scenarios, its deadline compliance rate progressively declines (Figure 1a). Additionally, despite using prioritization techniques to favor ResNet-50, its tail kernel execution interference (quantified as the slowdown relative to its profiled p95 isolated kernel execution latency, normalized to 1) can exceed that of the co-located model. This effect is especially pronounced when paired with heavier models (Figure 1b). We utilize Nsight Systems [10] to profile the interactions among concurrent kernels from distinct batches. We investigate a case in which a batch-1 (batch size of 1) ResNet-50 experiences severe interference when co-located

Background and Motivation

At runtime, executing a batch is equivalent to executing a sequence of kernels, which are functions specifically designed to exploit the GPU’s parallel resources to accelerate DNN operations. Spatial sharing executes batches concurrently, further improving utilization relative to the sequential execution of temporal sharing. This is because temporal sharing is limited by timing constraints when forming larger ii

95.4 99.1

94.3 92.6

high-priority 82.1 82.2

75 50 25 0

YOLO-v8n

VGG-19 RoBERTa-B Model: ResNet-50

(a) Deadline satisfaction

low-priority

1.00

high-priority 3.45 2.91

3 2

2.07

0.75

2.31 1.45

CDF

low-priority 100

P99 Interference (slowdown)

Deadline Compliance (%)

Strait: Perceiving Priority and Interference in ML Inference Serving

1.39

1 0

Validation set Diff. request rates Diff. deployed models Heterogeneous models Diff. GPU architecture

0.50 0.25

YOLO-v8n

VGG-19 RoBERTa-B Model: ResNet-50

0.00 0

(b) Kernel execution interference

40 60 80 100 Absolute Relative Error (%)

120

140

Figure 1. In each scenario, ResNet-50 (HP task) is co-located with a distinct model (LP task).

Figure 2. CDF of the prediction error under varying workload characteristics or on another GPU architecture.

with a batch-4 RoBERTa-B. Compared with isolated execution, the total kernel execution latency slowdown exceeds 3.4×, the total inter-kernel intervals exceed 6.6×, and consequently the overall slowdown exceeds 3.6×. In contrast, when this batch-1 ResNet-50 is co-located with a batch-2 YOLO-v8n, these values drop to 1.7×, 2.73×, and 1.79×, respectively. This discrepancy arises because the resource demands of co-located kernels from RoBERTa-B are considerably higher than those from YOLO-v8n, requiring 2.2× more avg. registers per thread, 2.7× more avg. shared memory, and 3.5× more total threads, among other resources. Regarding the increase in kernel execution latency, co-location with a resource-intensive kernel or one with similar resource demands can lead to severe interference [83]. Additionally, increasing intervals arise between consecutive kernels, owing to the fact that a kernel’s thread blocks must be provisioned with sufficient resources—such as registers and shared memory—before they can be scheduled [18, 39, 92, 97]. Although larger batch sizes improve GPU utilization, they naturally incur longer execution times, thereby severely constraining the time budget available to tolerate potential severe and variable interference. Consequently, managing these interference effects becomes critical to improve task prioritization and deadline satisfaction under high GPU utilization.

of performance degradation than the case where a computebound kernel co-locates with a memory-bound one [83]. Moreover, when using a runtime library [12, 14] to perform inference, the kernels within a batch execute without interruption, making it difficult to determine which kernels are likely to be co-located. As a result, estimating interference at the batch level rather than at the kernel level may offer a tractable solution. Prior work [26, 54, 55, 63, 98, 105] has trained an ML model to estimate kernel execution interference. However, several limitations remain. First, they are often restricted to pairwise or rigid co-location, neglecting scenarios involving multiple concurrent batches. Moreover, they are coarse-grained and disregard temporal effects and co-location dynamics. To estimate interference, prediction models typically use the resource throughput of co-located batches as input features. However, since a batch may colocate with varying batches during execution, identifying peers solely at the time of prediction overlooks critical past context. For instance, a batch might have been co-located with a previously completed task for an extended period, while only briefly overlapping with the batch present during prediction. Given that batches exhibit varying resource demands, such prediction models may rely on inputs that are insufficiently precise to capture the actual resource pressure induced by co-located batches. Furthermore, they are typically static, which may lead to degraded performance under changing workload characteristics or GPU types [17]. To examine this potential degradation, we conduct the following experiment. We evaluate with the gpulets [26] approach, which employs a linear regression model to estimate interference based on the resource throughputs of both the interfered batch and the co-located batch. The selected metrics include profiled L2 and DRAM throughputs; compute activity is ignored as gpulets employs Multi-Process Service (MPS) [5] to partition compute resources. Therefore, we additionally include the throughput of Streaming Multiprocessors (SMs), the basic compute units, to approximate the overall compute activity. We first deploy model set 1 (ResNet-50 [44], ViT-B-16 [35], ConvNeXt-B [62]) and collect serving data, using 80% of it for training and 20% for validation. Figure 2 presents the cumulative distribution function (CDF) of the absolute relative

2.2

Estimating Interference Effects

As shown above, interference-induced kernel execution slowdown can exceed 3.45× (Figure 1b). Notably, these experiments were conducted with the maximum batch concurrency limited to 2. In production environments, deployed models can be more heterogeneous, and supported concurrency levels could be higher. Our empirical results indicate that interference can further intensify in such scenarios, and the corresponding distribution exhibits a long tail. Directly modeling kernel execution [24, 25] to estimate interference effects is complicated, as executing a batch corresponds to running a sequence of kernels with varying resource demands. This approach would lead to several issues. First, kernels with complementary resource demands that co-locate may exhibit different interference effects compared to those with mutual resource demands. For instance, two memory-bound kernels may experience a greater degree iii

Haidong Zhao and Nikolaos Georgantas Model 1: high-priority

GPU Instance pool Async. Control Submit Inference

Model 1 instances

Model 2: low-priority

Figure 4. The figure visualizes contention during data transfer (green blocks) and kernel execution (blue blocks). When using pinned memory, concurrent batch submission results in FIFO-ordered data transfers.

Model 2 instances

Running Task List Interference Prediction Priority-aware

Batch Size: 3

M2-B3 M1-B2

Deadline

Remove completed tasks

Figure 3. System design overview.

delays. A global prediction model is used to serve all GPUs within the node to estimate kernel execution interference. This model continuously adapts to dynamic workloads to sustain prediction accuracy (Section 3.2). This estimated latency is then utilized in a priority-aware algorithm to guide scheduling decisions (Section 3.3). Each GPU maintains a running task list containing GPU-specific information and details of ongoing batches for interference prediction.

prediction error across different scenarios. When only the request arrival rate changes, the prediction model maintains accuracy comparable to that of the validation set. However, when the deployed models are replaced with model set 2 (VGG-19 [81], YOLO-v8n [52], RoBERTa-B [61]), the accuracy declines noticeably, and the error distribution exhibits a longer tail. This observation suggests that model architecture has a measurable impact on resource-contention behavior. In a heterogeneous setting where all models are deployed, the accuracy declines, though not as significantly as in the previous case. This occurs since the prediction model has learned the characteristics of model set 1 during the training phase, yet remains unfamiliar with those of model set 2. The most significant accuracy drop occurs when the prediction model is directly deployed on a different GPU architecture (NVIDIA A10G, Ampere), indicating that different GPU architectures may exhibit varying sensitivities to resource throughput. Several factors likely contribute to this discrepancy, including microarchitectural differences and architecture-specific compiler optimizations [59]. To maintain a certain degree of accuracy, the prediction model may need to adapt to the given workloads and GPU types.

3.1

Priority Level

For each model, Strait supports two priority levels: highpriority and low-priority. Strait employs CUDA stream priorities [3] and priority scheduling to favor HP tasks, treating LP tasks on a best-effort basis where their performance may be sacrificed if necessary. While this dual-priority setup aligns with previous work [43, 83, 92], it could be extended to support multiple priority levels, bounded by the GPU-specific CUDA stream priority range [3]. 3.2

Interference Prediction

When a batch is submitted to a discrete GPU for execution, it typically undergoes the following processes: (1) Host-toDevice (upstream) data transfer: Input data is written to pinned host memory and subsequently transferred to GPU memory. (2) Kernel execution: The GPU executes a sequence of dependent kernels and stores the resulting computations in memory. (3) Device-to-Host (downstream) data transfer: The inference results are copied back to pinned host memory. Accordingly, two types of resource contention may exist: one during data transfer and the other during kernel execution. The inference latency of a batch can be estimated from its profiled isolated latency and the potential delays caused by interference and queuing. We formalize Equation 1, where 𝐵𝑖, 𝑗 represents model 𝑖 with batch size 𝑗. 𝑇inf (𝐵𝑖, 𝑗 ) is the esisol (𝐵 ) is the profiled p95 isotimated inference latency; 𝑇inf 𝑖, 𝑗 lated overall inference latency; 𝑇data accounts for the delays caused by data transfer interference; 𝑇kernel (𝐵𝑖, 𝑗 ) accounts for the delays caused by kernel execution interference; and 𝑇queue denotes the queuing delay, defined as the time elapsed since the front request of the given batch was enqueued, measured at runtime.

System Design

We propose Strait, an ML inference serving system that natively supports task prioritization and deadline-aware scheduling under high GPU utilization. Similar to production inference serving systems for standard DNNs [11, 15, 67], Strait operates locally, enabling fine-grained control over runtime scheduling decisions to manage interference; it is thus orthogonal to cluster-level approaches and generative model solutions (Section 6). Figure 3 illustrates the system workflow. Incoming inference requests are initially enqueued in the front-end task queue associated with the target DNN model. When accessing a queue, the scheduler makes a scheduling decision and, if appropriate, dispatches the resulting batch to a backend instance pool on a designated GPU for execution. Strait loads the DNN models and concurrently initializes their instances across all GPUs, with each model assigned a priority level (Section 3.1). The inference latency for a batch is estimated by accounting for both interference and queuing

isol 𝑇inf (𝐵𝑖, 𝑗 ) = 𝑇inf (𝐵𝑖, 𝑗 ) + 𝑇data + 𝑇kernel (𝐵𝑖, 𝑗 ) + 𝑇𝑞𝑢𝑒𝑢𝑒

(1)

Strait: Perceiving Priority and Interference in ML Inference Serving

Interference in data transfer. This contributes to the delay term 𝑇data . Figure 4 illustrates upstream data transfer operations contending for bandwidth over PCIe, a duplex interface that handles both host-to-device and device-to-host transfers. Given that ResNet-50 [44] processes image inputs of dimension 3 × 224 × 224, data transfer latency can constitute more than 10% of the total inference latency, depending on the batch size, GPU type, and interconnect bandwidth. Therefore, interference in upstream data transfer should be taken into account. Conversely, interference in downstream data transfer may be disregarded, as many models have negligible output data sizes, and the benefits of modeling such effects may not justify the associated overhead. Since the PCIe link handles data transfer in a FIFO manner, 𝑇data can be estimated based on the completion time of the previous batch. Equation 2 expresses the potential delays in upstream data transfer, where 𝑡 current represents the current time and 𝑡 available denotes the time at which the PCIe link becomes available. Equation 3 defines how 𝑡 available is updated, representing the timestamp when the PCIe link becomes available for the next batch; if 𝐵𝑖, 𝑗 is scheduled, the PCIe will be free again after the sum of its scheduled start time and its 𝑖, 𝑗 profiled p95 isolated upstream data transfer latency, 𝑡 HtoD . 𝑇data = max(0, 𝑡 available − 𝑡 current )

(2)

𝑖, 𝑗 𝑡 available ← max(𝑡 current, 𝑡 available ) + 𝑡 HtoD

(3)

Equation 4 estimates the interference effect, denoted as 𝑘𝑒𝑟𝑛𝑒𝑙𝑒 𝑓 𝑓 . It is parameterized by a scaling factor 𝑘, a base 𝑏, and an additive constant 𝐶. The interference effect is modeled as an exponential function of increased resource pressure, driven by the following intuition. When a batch executes in isolation, no interference occurs. At lower resource pressure, interference among kernels tends to increase at a relatively slow rate. This is because contention for resources is effectively mitigated, as GPUs possess massive parallel resources and a throughput-oriented architecture [18, 39, 89, 104]. Furthermore, GPUs can effectively hide latency [89] by interleaving the execution of concurrent warps (groups of threads). This mechanism allows ready warps to execute while others are stalled—e.g., during memory operations—thereby keeping the compute pipelines highly utilized. However, as the resource pressure continues to increase, concurrency benefits diminish rapidly, since contention increasingly exceeds the GPU’s ability to mask delays, leading to rapid, superlinear slowdowns. The exponent approximates resource pressure through a linear regression (LR) model, computed as a cmp weighted sum of contributions from 𝑚 1avg to 𝑚𝑛avg , 𝑚 self and 𝑚 mem . Here, 𝑚𝑖avg represents the aggregate throughput of self co-located batches for a specific metric used to approximate cmp resource pressure. Additionally, 𝑚 self and 𝑚 mem represent self the compute and memory throughputs of the interfered batch itself, respectively. This reflects that batches with varying resource demands exhibit different sensitivities to resource pressure. For 𝑚𝑖avg , we incorporate multiple resource metrics because GPUs comprise heterogeneous execution units and a complex memory hierarchy [7, 37]. Specifically, to capture the overall resource pressure more accurately, we consider metrics from both the memory hierarchy (L1 cache, L2 cache, and DRAM) and the compute pipelines (tensor cores, and cmp CUDA cores with fused multiply–add operations). For 𝑚 self mem and 𝑚 self , we empirically selected metrics for the tensor cores (which specialize in dense matrix multiplications) and the L2 cache (a shared global resource for compute units) to represent these resource demands. Recall that a batch’s set of co-located batches may change through execution; consequently, 𝑚𝑖avg is expected to fluctuate accordingly. To capture this temporal dynamic—where different segments of a batch may experience varying levels of resource pressure—each time a batch arrives or departs, Strait updates 𝑚𝑖avg and records the corresponding timestamp. When calculating 𝑚𝑖avg , we employ a time-weighted average of all observed throughput values to approximate this dynamic. ˆ , Equation 5 defines the estimated interference degree, Intf representing the slowdown factor, which is 1 when there is no interference. It also captures the potential effects of CUDA stream priority through the learnable priority coefficient, coeff 𝑝 , which is uniquely assigned to each task priority level to reflect that HP tasks tend to experience less interference under identical resource pressure. Equation 6 estimates the

Online calibration. 𝑇data is estimated at scheduling time; however, it can be more precisely updated by timing the actual data transfer once it is completed. This calibration improves the accuracy of determining whether subsequent scheduling decisions will violate the deadlines of currently executing batches (Section 3.3). Additionally, it compensates for potential measurement skew caused by NUMA effects or differences in the PCIe Root Complex. Interference in kernel execution. This contributes to the delay term 𝑇kernel (𝐵𝑖, 𝑗 ). Recall that estimating interference effects at the batch level may offer a more tractable solution. To characterize the resource demand of a batch, we extract resource throughput metrics for its constituent kernels using Nsight Compute [9]. Assuming these kernels have resource throughputs 𝑣 1 to 𝑣𝑛 and corresponding execution durations 𝑑 1 to 𝑑𝑛 , the resource demand of the batch is approximated as the time-weighted average throughput Í𝑛 𝑣 ·𝑑 𝑖=1 Í𝑛 𝑖 𝑖 . The delay is calculated as follows: 𝑑𝑖 𝑖=1

cmp

mem

𝑘𝑒𝑟𝑛𝑒𝑙𝑒 𝑓 𝑓 = 𝑘 · 𝑏 LR (𝑚avg ... 𝑚avg | 𝑚self , 𝑚self ) + 𝐶

(4)

ˆ = 1 + 𝑘𝑒𝑟𝑛𝑒𝑙𝑒 𝑓 𝑓 · coeff Intf 𝑝

(5)

ˆ − 1 · 𝑇 isol (𝐵𝑖,𝑗 ) 𝑇kernel (𝐵𝑖, 𝑗 ) = Intf kernel

(6)

𝑛

Haidong Zhao and Nikolaos Georgantas

delay induced by kernel execution interference by applying the slowdown to the profiled p95 isolated kernel execution isol (𝐵 ). latency, 𝑇kernel 𝑖,𝑗 Model parameters. The parameters can be initialized conservatively and subsequently updated via the online learning methods detailed below. Alternatively, their initial values can also be warmed up using simulated workload or restored from a checkpoint of the previous serving session. Self-adaptive model. To accommodate dynamic workloads and different GPU types (Section 2.2), we dynamically update the prediction model. Upon the completion of each batch, we compare the estimated interference degree with the actual interference degree, which is quantified as the ratio of the measured kernel execution latency to the profiled p95 isolated value. By continuously feeding back this discrepancy, the prediction model dynamically recalibrates itself, adjusting for over- or underestimations of interference to reflect recent resource contention behavior. To rapidly adapt to dynamic workloads and maintain stability under concept drift [17], this regression model requires a robust optimization strategy. Consequently, we collected and replayed evolving workload traces (e.g., varying request arrival rates and diverse model deployments) in an online fashion to evaluate candidate methods. We restrict our selection to first-order methods, as they are computationally efficient and exhibit greater numerical stability to noise than second-order methods in online learning. Among candidates, we select Adam [56], an adaptive method that automatically adjusts the effective learning rate for each parameter based on past gradients. Compared to SGD [58] and SGD with momentum [70], adaptive optimization methods exhibit greater robustness during updates; we hypothesize that this is because global learning rates are susceptible to large variances in gradient magnitudes, leading to instability. While Adagrad [36] and AMSGrad [71] perform well empirically, their monotonically decaying learning rates reduce adaptability over time. Since we update the prediction model upon receiving inference feedback from each completed batch, we prefer Adam over RMSProp [45], as its formulation facilitates smoother updates. Hyperparameter tuning. We empirically tune the Adam hyperparameters to determine the optimal calibration step size based on the prediction error [41, 75]. Specifically, to enhance responsiveness to recent gradients, we adjust 𝛽 1 to 0.7 and 𝛽 2 to 0.9. To balance stable updates with rapid adaptation, we empirically set the learning rate 𝛼 to 0.0075. Furthermore, we employ the Huber loss function [23] to promote stable updates, as outliers induced by severe prediction errors could otherwise trigger disproportionately large gradients that destabilize the online learning process. We set this threshold to 0.50, corresponding to the p95 absolute prediction error observed when fitting the model to the captured serving traces.

Algorithm 1: Scheduling Policy Input: 𝑇inf (): estimated inference latency (Section 3.2) Input: 𝐶𝑙𝑜𝑤 : capping throughput for LP tasks 1 running_tasks ← [ ] 2 for task queues in descending priority do 3 Removing requests with early dropping 4 for 𝑘 ← Binary Search(1 to 𝑗) do 5 for 𝑚 ← GPU(1 to 𝑎𝑙𝑙) do 6 if ¬𝑣𝑖𝑜𝑙𝑎𝑡𝑒 (𝐶𝑙𝑜𝑤 ) ∧ 𝑚𝑒𝑒𝑡 (𝐵𝑖, 𝑘 ) then 7 𝑏𝑠 ← 𝑘 // largest batch size 8 latency[𝑏𝑠, 𝑚] ← 𝑇inf (𝐵𝑖, 𝑘 ) 9 end 10 end 11 end 12 if 𝑏𝑠 is set then 13 𝑛 ← argmin(latency[𝑏𝑠]) 14 Submit 𝐵𝑖, 𝑏𝑠 to GPU 𝑛 15 Log this batch to running_tasks 16 end 17 end // Running in background execution threads 18 while 𝑡𝑟𝑢𝑒 do 19 Remove the completed batch from running_tasks 20 Update the prediction model 21 end

3.3

Priority-aware Scheduling

Algorithm 1 presents the scheduling policy. The scheduled batch should potentially meet its own deadline without violating the deadlines of ongoing tasks with equal or higher priority. The scheduler first sorts the task queues awaiting scheduling according to the priority levels of their associated models. We employ the early-dropping mechanism [46, 79] to discard requests whose remaining time is insufficient for completion even in isolation. This mitigates cascading deadline violations and prevents memory exhaustion under overload. When the scheduler reaches the task queue for model 𝑖, assuming 𝑗 requests have been buffered. While larger batch sizes improve GPU utilization, they tend to impose stricter time budgets (longer inference latency) and may cause more severe interference with other batches; therefore, we employ binary search to determine the maximum feasible batch size that satisfies the following constraints. • violate() evaluates whether the schedule 1) causes the aggregate resource throughput of LP tasks on a GPU to exceed the capping value 𝐶𝑙𝑜𝑤 , or 2) violates the deadlines of ongoing batches with equal or higher priority. Regarding condition 1), we restrict the aggregate peak resource throughput [1] of LP tasks on the GPU. This prevents them from aggressively acquiring resources, as GPUs may not vi

Strait: Perceiving Priority and Interference in ML Inference Serving

Table 1. List of DNN models used in the evaluation.

provide true task-level preemption [43, 65, 80, 83]. To adapt to the workload, 𝐶𝑙𝑜𝑤 is adjusted using an additive-increase multiplicative-decrease (AIMD) policy. We perform a sensitivity study in Appendix A.1 to select the control parameters under an evolving workload. To balance aggressive resource acquisition of LP tasks against suboptimal GPU utilization, 𝐶𝑙𝑜𝑤 ranges from 75% to 100%, with a conservative increase rate of 0.25% every 100 ms. Upon an HP task deadline violation, 𝐶𝑙𝑜𝑤 resets to the initial value. Regarding condition 2), we check whether the schedule violates the deadlines of ongoing batches. Notably, an HP task can still be scheduled if it causes deadline violations only for LP tasks. For the ongoing batch, we employ a time-weighted average to approximate the co-located resource throughput (Section 3.2); under the interference effects, we then estimate the completed portion based on the elapsed time. We verify deadline feasibility by assuming that the remaining portion executes under the updated co-located throughput if the new batch is scheduled. • meet() evaluates whether the scheduled batch can potentially meet its own deadline. However, this batch’s prospective average co-located resource throughput remains uncertain at scheduling time. Consequently, we conservatively require that the scheduled batch, if assigned to a GPU, can potentially meet its deadline under half of that GPU’s runtime throughput. Under this assumption, we record the estimated latency in a 𝑙𝑎𝑡𝑒𝑛𝑐𝑦 list to select the GPU yielding the lowest latency in a multi-GPU node. If a batch is eligible for scheduling, its details are recorded in the running task list. Otherwise, this indicates that system resources are insufficient for the model, and scheduling is deferred. The system then proceeds to the next task queue, if one exists. When a batch completes, its execution thread removes the corresponding record from the running task list, and the measured kernel execution latency together with its estimated value is used to update the prediction model.

Model

Architecture

Task Classification

GFLOPs

Priority

Deadline

ResNet-50 ViT-B-16 ConvNeXt-B VGG-19 YOLO-v8n RoBERTa-B

CNN Transformer CNN CNN CNN Transformer

Image Classification Image Classification Image Classification Image Classification Object Detection Machine Comprehension

4.09 16.85 15.35 19.63 9.57 10.87

high high low low low low

8 ms 15 ms 25 ms 25 ms 20 ms 45 ms

Evaluation

We evaluate the prototype from the following perspectives: (i) comparison against existing scheduling policies and softwaredefined preemption approaches; (ii) inaccuracy of prediction, performance gains of an adaptive model relative to a static baseline, sustainability over time, and sensitivity to profiling drift; and (iii) ablation study of the prioritization mechanisms and an analysis of scheduling overhead. 5.1

Methodology

Experimental Setup. We evaluate Strait primarily using two types of cloud instances: a 4-GPU node (4 NVIDIA L424GB GPUs, 32 vCPUs, and 192 GB DRAM) and a single-GPU node (1 NVIDIA L4-24GB GPU, 8 vCPUs, and 48 GB DRAM). To ensure consistency, we fix the GPU frequency at 1650 MHz during both profiling and serving. The system operates on Ubuntu 22.04, with computational acceleration enabled by CUDA 12 and TensorRT 10. Workloads. We consider representative pre-trained DNN models that are already widely used for downstream tasks. As listed in Table 1, ResNet-50 [44], VGG-19 [81], ConvNeXtB [62], and YOLO-v8n [52] are classified as CNN architectures, while ViT-B-16 [35] and RoBERTa-B [61] are categorized as Transformer architectures. We assign a specific priority level to each model; however, Strait permits the deployment of the same model with a different priority by launching new instances with distinct trigger names. We set a uniform maximum batch size of 8 per model and a batch concurrency limit of 4 per GPU. The deadline for each model is preset and configured based on its isolated inference latency and urgency. Each model is assigned a batch formation timeout proportional to its deadline; it holds the initial request briefly to await subsequent arrivals, thereby opportunistically forming a larger batch. As in previous work [83], we evaluate Strait using stochastic workloads modeled as a Poisson process [22, 72], uniform workloads [6], and production serverless traces [78]. Each workload represents a heterogeneous scenario in which multiple models, each with distinct configurations and priority levels, may be triggered at varying arrival rates. We adopt the open model setting [77], in which each request is generated independently of prior inference completions. Baselines. We reimplemented the scheduling policies of other serving systems, where the scheduling unit is at the request level. All policies incorporate CUDA stream priorities,

Implementation

The prototype includes ~5473 lines of C++ code in the critical paths of serving. To utilize the profiling results, online serving must run at the same GPU frequency as offline profiling. We implemented Strait using the TensorRT runtime [14]. For each model, we developed a thread-pool-based instance pool where each instance maintains independent pinned memory for data transfer and dedicated resources for inference execution. The scheduler manages a task queue for each DNN model, and each model notifies the scheduler when preset scheduling criteria are met (e.g., batch formation timeout). A thread-safe mechanism records information about ongoing batches to enable interference prediction and latency estimation. vii

Haidong Zhao and Nikolaos Georgantas

task priority scheduling, and an early-dropping mechanism. (1) Temporal sharing: This policy exclusively executes batches sequentially [29, 30, 42, 79]. (2) Static spatial sharing: This policy caps the maximum number of concurrent batches per GPU at 3 [11, 67]. (3) Reactive spatial sharing: The policy reactively adjusts system resources based on inference feedback [34, 74]. Since INFaaS [74] mitigates interference via cluster-level scaling and GSLICE [34] leverages MPS [5]—a different GPU sharing mechanism—to reallocate resources among co-located models, a direct comparison may be infeasible. Drawing inspiration from these approaches, we dynamically limit the batch concurrency allowance for LP tasks when a model experiences severe interference, thereby mitigating contention and facilitating task prioritization. We set the global batch concurrency limit per GPU to 4 and assign a default upper bound of 3 to each task priority. If an HP task misses its deadline, the concurrency allowance for LP tasks is reduced by 1. To prevent LP tasks from being completely halted, we enforce a strict lower bound of 1. Every 200 ms, this allowance is unconditionally reset to the default upper bound of 3, providing LP tasks an opportunity to reclaim throughput. Furthermore, we compare with software-defined preemption approaches [80, 83], which intercept kernel sequences issued by applications to prevent all kernels from being submitted to the GPU at once. Consequently, their scheduling unit operates at the kernel level. We report this comparison in Section 5.3. (4) XSched [80] implements abstract queues for kernels from diverse applications and enables varying degrees of preemption tailored to vendor-specific hardware capabilities. For inference serving, it employs fixed-priority scheduling [60] on recent NVIDIA GPUs. Evaluation Metrics. The latency of a request is measured from the moment it is enqueued in the task queue to the point when its inference result is written back to the host memory. Requests omitted by the early dropping mechanism are also counted as deadline-violation requests. We use goodput to denote the number of requests meeting deadlines in a time window. 5.2

Temporal sharing exhibits a high deadline violation rate. It suffers from severe HOL blocking, especially for LP tasks; most requests are dropped early rather than incurring wasted computation (executions that fail to meet deadlines), resulting in low GPU utilization. Nevertheless, it avoids interference and can immediately serve HP tasks once the GPU becomes available, as evidenced by the lower and tighter HP latency box plots (e.g., Figure 5c). This allows temporal sharing to outperform spatial sharing baselines in terms of task prioritization in certain scenarios. Static spatial sharing improves GPU utilization but exhibits poor prioritization, primarily due to wasted computation. Because it completely ignores interference and colocation dynamics during scheduling, its performance is highly sensitive to workload characteristics. For instance, it incurs a significant increase in deadline violations for HP tasks under uniform workloads. Nevertheless, it may achieve the best performance for LP tasks by mitigating severe HOL blocking and employing fewer prioritization techniques than both the reactive approach and Strait. While reactive spatial sharing improves task prioritization over the static approach, its effectiveness remains highly sensitive to workload characteristics. When HP requests dominate, the improvement for HP tasks is marginal (0.4 pp), whereas other scenarios yield gains of up to 10.02 pp. This discrepancy arises because the reactive approach facilitates prioritization primarily by throttling LP tasks. Consequently, its effectiveness is limited when LP requests are infrequent (Figure 5a). Moreover, since it still lacks explicit interference management, it remains unable to effectively mitigate wasted computation. Notably, this prioritization may overly penalize LP tasks by blocking them, resulting in the lowest deadline satisfaction among spatial sharing approaches. Across the evaluated scenarios, Strait reduces the deadline violations for HP tasks by 1.02 pp to 11.18 pp; however, it underperforms compared to the strongest baseline for LP tasks by 0.61 pp in the worst case. Several factors contribute to these outcomes. First, Strait employs interference prediction to facilitate deadline-aware scheduling; as evidenced by a tighter latency distribution compared to other spatialsharing approaches, this effectively reduces wasted computation. Nevertheless, its scheduling policies may compromise LP task performance. Specifically, Strait schedules HP tasks even if they cause ongoing LP tasks to violate their deadlines. Conversely, LP tasks are blocked if their execution risks violating the deadlines of ongoing HP tasks. Additionally, it throttles LP tasks using an adaptive cap. An analysis of the causes of LP deadline violations reveals that approximately two-thirds of the violating requests result from early dropping rather than wasted computation. Since LP tasks are treated as best-effort, the observed degradation of less than 0.61 pp may represent an acceptable performance trade-off.

Comparison with Request-Level Scheduling

We synthesize both stochastic and uniform workloads to evaluate these policies. Motivated by the skewed request arrival distributions observed in production traces [78], we configure the stochastic workloads to frequently trigger either HP or LP tasks. Consistent with uniform workloads used in monitoring scenarios [6], we balance the requests across models. To emulate more heterogeneous scenarios, we duplicate each evaluated model listed in Table 1, yielding a total of 12 tasks. To evaluate performance under high GPU utilization, we use intense workloads and intentionally induce deadline violations for quantitative comparison. Figure 5 illustrates the comparative performance. viii

Strait: Perceiving Priority and Interference in ML Inference Serving

1.51% 0.20%

0.34%

0.18%

2.78%

5.64%

Static

Reactive

Strait

9.43% 2.97% 0.96%

1.99% 1.57%

0.57%

29.94%

Violation (%)

3.04% 3.27% 2.87%

Latency (ms)

0 30 20 10 high-priority

Latency (ms)

Violation (%)

Temporal 7.44%

20 10 high-priority

low-priority

(a) Frequent triggers on HP tasks

low-priority

(b) Frequent triggers on LP tasks

11.98%

6.96% 7.98% 1.96%

1.82%

3.79%

0.80%

0 40 20

high-priority

low-priority

1.00

0.75

0.75 CDF

CDF

Figure 5. Deadline violation rates and latency distributions for different scheduling policies under a 4-GPU node.

0.50 0.25

0.00 1

10 Latency (ms)

100

(a) Pressure on HP tasks

0.00 1

on task deadlines, such as opportunistically scheduling large batches or dropping requests that would violate deadlines, thereby achieving more equitable performance.

XSched (HP) XSched (LP) Strait (HP) Strait (LP) 10 Latency (ms)

100

5.4

Validation of Interference Prediction

Recall that we employ an adaptive prediction model to continuously recalibrate for the over- and under-estimation of interference. We quantify interference-induced overhead in ker-

(b) Pressure on LP tasks

Figure 6. CDF of inference latency for XSched and Strait.

nel execution as 5.3

actl (𝐵 ) −𝑇 isol (𝐵 ) | |𝑇kernel 𝑖,𝑗 kernel 𝑖,𝑗 isol (𝐵 ) 𝑇kernel 𝑖,𝑗

actl (𝐵 ) de, where 𝑇kernel 𝑖,𝑗

isol (𝐵 ) notes the measured kernel execution latency, and 𝑇kernel 𝑖,𝑗 denotes the corresponding profiled p95 value. Inaccuracy under intense workloads. Figure 7a presents the results for the workloads used in Section 5.2, illustrating the prediction errors for both kernel execution and overall inference latency, alongside the distribution of kernel execution overhead. The distribution of kernel execution overhead exhibits a long tail—reaching approximately 682% with a p99 value of 245%—which indicates that severe interference occurs during heterogeneous model deployment under GPU oversubscription. This skewness in kernel execution overhead complicates latency prediction. The predictions for kernel execution latency exhibit absolute relative errors of 29.6% and 46.6% at p95 and p99, respectively. Since the prediction model relies on continuous inference feedback for online updating, it is constrained by the underlying data distribution. Due to data sparsity, the prediction model is ill-equipped to handle severe long-tail interference, primarily resulting in interference underestimation. Moreover, this severe interference would encourage the model to calibrate for these outliers, leading to overestimation in other scenarios; e.g., by analyzing data, we find that predictions under relatively small kernel execution overhead (<60%) tend to overestimate interference. In contrast, we find that the model performs relatively well at an overhead of near 80%; this is because sufficient training data is available at that range, as reflected by the steep ascent of the green curve. We discuss potential ways to mitigate the long tails in Section 6. Other factors may also limit prediction accuracy, including workload characteristics that the adaptive model does not adequately capture initially. Additionally, we employ time-weighted averages to approximate resource

Comparison with Kernel-Level Scheduling

We use XSched [80] to support fixed-priority scheduling [60] for Triton inference server [11]. We adopt XSched’s original configuration, where batching is not employed and kernels are directly submitted to its abstract queues. For comparison, we enable batching in Strait but eliminate the batch formation timeout. Both systems utilize the TensorRT [14] runtime. To account for Triton-introduced overhead (e.g., gRPC), we measure the difference in isolated inference latency between Triton and Strait for each model. We then subtract this difference from the measured latency of XSched to ensure a fair comparison. Figure 6 illustrates the inference latency distributions. When pressure is on HP tasks (case in Figure 6a), Strait incurs 1.43% and 1.20% deadline violations for HP and LP tasks, respectively, compared to 2.07% and 29.89% for XSched. When pressure is on LP tasks (case in Figure 6b), Strait incurs 0.69% and 4.27% violations for HP and LP tasks, respectively, whereas XSched results in 0.043% and 11.41%. Overall, Strait maintains comparable deadline satisfaction for HP tasks while yielding measurable improvements for LP tasks. Several factors contribute to these results. First, XSched’s prioritization allows for fine-grained control by stalling kernels from LP task launches when HP kernels arrive. While this mechanism effectively protects HP tasks when LP requests dominate, its efficacy degrades when HP requests dominate. This is because equal-priority kernels can still contend for shared GPU resources, and XSched lacks explicit interference management. Additionally, XSched may suffer from HOL blocking for LP tasks, as HP kernels delay LP kernel launches, suppressing concurrency and underutilizing GPU resources. In contrast, Strait can make more informed decisions based ix

1.0 0.8

0.6

CDF

0.8

1) Bursty traffic (Adaptive) 1) Bursty traffic (Static) 2) Diff. model sets (Adaptive) 2) Diff. model sets (Static) 3) Ampere Arch. (Adaptive) 3) Ampere Arch. (Static) 4) Hopper Arch. (Adaptive) 4) Hopper Arch. (Static)

0.6

0.4 Overall latency prediction Kernel execution latency prediction kernel execution overhead

0.2 0.0

10 100 Absolute relative error / Overhead (%)

0.4 0.2 5

20 Absolute Relative Error (%)

(a) CDF of prediction errors and kernel exe- (b) CDF of prediction errors with adaptive cution overhead under intense loads. and static model under different scenarios.

×10

Load

4.0

Goodput

3.5 HP

2.5 0.0

120

160

200

240

280

Time (Minutes) Relative error (%)

1.0

Violations (%) Requests/GPU

Haidong Zhao and Nikolaos Georgantas

P1-P99 band

P5-P95 band

Median

0 −25 0

Iteration (bin = 10000)

7 ×10

Figure 7. Evaluation results for inaccuracy, adaptability, and sustainability. pressure; however, this approach may be limited if a batch’s resource demands vary significantly during execution. Fortunately, the overall latency prediction can mitigate these prediction errors, and this value is ultimately used for scheduling. This is because kernel execution latency is only one element of the overall latency, and we employ methods such as offline profiling and online calibration to enhance the estimation of the other elements. As a result, the overall latency prediction errors are reduced to 18.95% at p95 and 27.9% at p99 in the extreme case. Adaptive over static model. We evaluate whether an adaptive prediction model outperforms a static model under dynamic workloads or when deployed directly on other NVIDIA GPU types. We begin by deploying a CNN-based model set (ResNet-50, ConvNeXt-B, and YOLO-v8n) and updating the prediction model under a moderate load to initialize the model parameters. Subsequently, we transition to the following scenarios for 45 seconds: 1) bursty traffic; 2) an alternative model set dominated by Transformer architectures (VGG-19, ViT-B-16, and RoBERTa-B); 3) an NVIDIA A10G GPU featuring the Ampere architecture; and 4) a highperformance NVIDIA H100 GPU featuring the Hopper architecture, where loads are scaled up to introduce interference. Figure 7b presents the results. For scenario 1), the adaptive method demonstrates comparable, albeit slightly inferior, performance to the static model for small prediction errors (under 20%). This occurs because the data distribution shifts toward regimes with more severe interference. Consequently, the adaptive method learns this trend, achieving around a 3 pp improvement in p99 prediction error over the static approach, though it suffers from fluctuations in smaller errors. For scenario 2), the adaptive model exhibits noticeable improvements for both small and tail errors (e.g., a 7 pp improvement in p95 errors), suggesting that varying the model sets may alter resource contention behavior. For scenarios 3) and 4), the adaptive method also yields consistent improvements across both small and tail errors, achieving reductions of over 7.2 pp and 8 pp in p99 prediction errors, respectively. This indicates that the throughput-interference mapping may vary across GPU architectures. Accordingly, the performance gains of adaptive methods are scenariodependent, with more noticeable improvements observed

when transitioning across different model sets or GPU architectures. Serving a period of time. Following prior work [42, 83], we evaluate Strait on production serverless traces [78] to assess its ability to maintain performance and prediction accuracy. Since the traces record per-minute aggregate requests for each function, we generate discrete arrivals using a Poisson process parameterized by these rates. To pressure Strait, we map the arrival rates of the six most frequent functions during the busiest 5-hour window of a single day to the evaluated models, scaling them according to GPU capacity. Figure 7 presents the results. Goodput generally overlaps with the incoming load, and Strait can absorb moderate load spikes (e.g., shortly after 40 mins and 280 mins), as evidenced by the minor fluctuations in deadline violation rates. During peak burst periods (e.g., around 90 mins), the load may exceed system capacity, leading to temporary spikes in deadline violations; however, Strait can recover shortly thereafter. To evaluate prediction accuracy, we analyze a 10,000-batch update window to align with the throughput per GPU. The median error remains consistently near 0%, and the two error bands exhibit steady variance. This suggests that Strait continuously calibrates for both overestimations and underestimations of interference. Although severe interference can occur when GPU resources are oversubscribed, mechanisms such as the Adam optimizer and Huber loss can prevent catastrophic updates from destabilizing the system. Profiling drift. Since inference latency can be rapidly reprofiled whereas resource throughput profiling is significantly more time-consuming, we evaluate the model’s sensitivity to throughput profiling drift. We apply random perturbations of ±𝑥% to each profiled value. When this perturbation magnitude is below 15%, the p99 prediction drift remains below 1.5 pp, suggesting that the adaptive model can tolerate moderate profiling drift in throughput (Appendix A.2). Ablation Study. Appendix A.3 investigates the contributions of each task prioritization mechanism, suggesting that throttling LP tasks and checking whether the scheduled batch would violate the deadlines of ongoing tasks are the most critical components.

Strait: Perceiving Priority and Interference in ML Inference Serving

5.5

Overheads

Algorithm 1 employs binary search to determine the feasible batch size for a model. Given constant-time interference prediction, the search over candidate batch sizes 𝑗 and GPU number 𝑛 has complexity 𝑛 · 𝑂 (log 𝑗). In our setup, scheduling overhead is typically bounded by tens of microseconds. In practice, we can configure each serving system instance to manage a subset of GPUs within a node for efficient scheduling. This also mitigates potential NUMA effects and lock contention.

Related Work

Serving systems such as Clipper [30], InferLine [29], Clockwork [42], and Nexus [79] optimize inference through batch size control and framework support but uniformly rely on temporal sharing. Serving systems that employ static spatial sharing, such as TFS [67], Triton [11], and Scrooge [46], require preset concurrency or model instances, limiting adaptability to dynamic workloads. Reactive spatial-sharing systems, such as INFaaS [74] and GSLICE [34], passively mitigate interference by reallocating resources, switching model variants, or scaling instances only after observing performance degradation. With proactive spatial-sharing, serving systems dynamically make scheduling decisions. gpulets [26] employs a linear regression model to estimate interference during resource provisioning; however, its interference prediction approach is coarse-grained and may be susceptible to evolving workloads (Section 2.2). GPU sharing techniques [28, 43, 80, 83, 104] enable finegrained scheduling for task prioritization. REEF [43] enables preemption of best-effort kernels for real-time tasks, but it is limited to AMD GPUs. Orion [83] schedules kernels to avoid co-locating those with similar resource intensities while prioritizing HP tasks. XSched [80] supports varying preemption levels depending on the accelerators’ hardware capabilities; we compare it against Strait when integrated with the inference server. LithOS [28], a recent work designed for inter-process applications, argues that scheduling granularity can be further refined to the thread-block level to facilitate task prioritization. Orthogonal to these works, Strait employs a retrofitted interference prediction approach for inference workloads [26, 54, 55, 63, 105] and integrates it into scheduling to enhance task prioritization.

Discussion

Interference prediction. Extreme interference may arise in bursty workloads, where severe resource contention and stall-induced delays can inflate kernel execution time. To alleviate this issue, limiting GPU utilization by imposing a maximum aggregate throughput or a maximum allowed batch concurrency are natural choices. However, our empirical results indicate that naively applying these limits could reduce deadline satisfaction. We suggest that more fine-grained overload control mechanisms should be explored. Priority and Scalability. Strait schedules inference requests on available non-preemptive GPUs, making its approach orthogonal to priority-handling mechanisms in CPU resource management [2, 4, 21, 85] or priority-aware autoscaling [50]. It manages interference at the node level; future extensions may include priority-aware admission control and load balancing at the cluster level. Generalize to edge devices. Strait is evaluated on servergrade discrete GPUs. However, edge devices also serve inference workloads, and they are typically equipped with integrated GPUs that share physical memory with the CPU [49, 53, 90, 93, 96, 101]. Although there is no explicit data transfer between host and device memory, memory contention could become more unpredictable owing to concurrent CPU activities. Additionally, the compute capability of integrated GPUs is more constrained and is further influenced by factors such as power and thermal budgets. Moreover, as integrated GPUs incorporate more heterogeneous processing units, extending the input features of the prediction model could be further explored. Large Language Models (LLMs). Recent serving frameworks target autoregressive generation in LLMs, focusing on continuous batching, KV-cache management, and disaggregated prefill and decoding phases [8, 57, 100, 106]. In contrast, Strait focuses on on-premises deployments [33, 51, 66, 86, 87] of standard DNNs with fixed input/output shapes. In addition, on-premises scenarios may need to co-locate heterogeneous models on the same hardware, whereas LLM serving infrastructures focus on scaling one or several LLMs across multiple servers [106].

Conclusion

ML inference serving systems are often required to support task prioritization and to deliver timely predictions under high GPU utilization. Strait responds to this challenge by explicitly managing interference during its priority-aware scheduling. Notably, the interference prediction approach is adaptive to mitigate the potential degradation in prediction accuracy under changing workload characteristics and GPU types. The evaluation results demonstrate consistent performance gains in task prioritization and deliver more equitable performance compared to software-defined preemption approaches. The interference prediction model serves as a key building block; we anticipate that further refinements to this model could further improve deadline satisfaction.

References [1] 2020. Terminology used in Nsight Compute. https: //stackoverflow.com/questions/63403203/terminology-usedin-nsight-compute?rq=1 [2] 2026. Apache Hadoop YARN. https://hadoop.apache.org/docs/ current/hadoop-yarn/hadoop-yarn-site/YARN.html xi

Haidong Zhao and Nikolaos Georgantas [3] 2026. CUDA C++ Programming Guide: v13.1. https://docs.nvidia. com/cuda/pdf/CUDA_C_Programming_Guide.pdf [4] 2026. Kubernetes. https://kubernetes.io/ [5] 2026. MULTI-PROCESS SERVICE: vR590. https://docs.nvidia.com/ deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf [6] 2026. nuScenes. https://www.nuscenes.org/nuscenes#datacollection NVIDIA ADA LOVELACE PROFESSIONAL GPU AR[7] 2026. CHITECTURE. https://images.nvidia.com/aem-dam/enzz/Solutions/technologies/NVIDIA-ADA-GPU-PROVIZArchitecture-Whitepaper_1.1.pdf [8] 2026. NVIDIA Dynamo Platform. https://developer.nvidia.com/ dynamo [9] 2026. NVIDIA Nsight Compute. https://developer.nvidia.com/nsightcompute [10] 2026. NVIDIA Nsight Systems. https://developer.nvidia.com/nsightsystems [11] 2026. NVIDIA Triton Inference Server. https://developer.nvidia.com/ triton-inference-server [12] 2026. ONNX Runtime. https://onnxruntime.ai/ [13] 2026. Tensorflow Serving shared batch scheduler. https: //github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ kernels/batching_util/shared_batch_scheduler.h [14] 2026. TensorRT Documentation. https://docs.nvidia.com/ deeplearning/tensorrt/ [15] 2026. TorchServe. https://pytorch.org/serve/ [16] Vivek Adarsh, Michael Nekrasov, Udit Paul, Tarun Mangla, Arpit Gupta, Morgan Vigil-Hayes, Ellen Zegura, and Elizabeth Belding. 2021. Coverage is Not Binary: Quantifying Mobile Broadband Quality in Urban, Rural, and Tribal Contexts. In 2021 International Conference on Computer Communications and Networks (ICCCN). 1–9. doi:10. 1109/ICCCN52240.2021.9522152 [17] Evidently AI. 2025. What is Concept Drift in ML, and How to Detect and Address It. https://www.evidentlyai.com/ml-in-production/ concept-drift [18] Tanya Amert, Nathan Otterness, Ming Yang, James H. Anderson, and F. Donelson Smith. 2017. GPU Scheduling on the NVIDIA TX2: Hidden Details Revealed. In 2017 IEEE Real-Time Systems Symposium (RTSS). 104–115. doi:10.1109/RTSS.2017.00017 [19] Romil Bhardwaj, Kirthevasan Kandasamy, Asim Biswal, Wenshuo Guo, Benjamin Hindman, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2023. Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 623–643. https://www.usenix.org/conference/osdi23/presentation/bhardwaj [20] Sumon Kumar Bose, Bapi Kar, Mohendra Roy, Pradeep Kumar Gopalakrishnan, and Arindam Basu. 2019. ADEPOS: Anomaly detection based power saving for predictive maintenance using edge computing. In Proceedings of the 24th Asia and South Pacific Design Automation Conference (ASPDAC ’19). Association for Computing Machinery, 597–602. doi:10.1145/3287624.3287716 [21] Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, Jingren Zhou, Zhengping Qian, Ming Wu, and Lidong Zhou. 2014. Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). 285–300. https://www.usenix.org/conference/osdi14/ technical-sessions/presentation/boutin [22] Jin Cao, William S. Cleveland, Dong Lin, and Don X. Sun. 2001. On the nonstationarity of Internet traffic. In Proceedings of the 2001 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS ’01). 102–112. doi:10.1145/378420. 378440 [23] Bohsun Chen. 2024. Understanding Huber Loss function: Insights from Applications. https://medium.com/@devcharlie2698619/

understanding-huber-loss-function-insights-from-applications5c1c5145d2c4 [24] Quan Chen, Hailong Yang, Minyi Guo, Ram Srivatsa Kannan, Jason Mars, and Lingjia Tang. 2017. Prophet: Precise QoS Prediction on Non-Preemptive Accelerators to Improve Utilization in WarehouseScale Computers. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’17). Association for Computing Machinery, 17–32. doi:10.1145/3037697.3037700 [25] Quan Chen, Hailong Yang, Jason Mars, and Lingjia Tang. 2016. Baymax: QoS Awareness and Increased Utilization for Non-Preemptive Accelerators in Warehouse Scale Computers. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’16). Association for Computing Machinery, 681–696. doi:10.1145/2872362. 2872368 [26] Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 199–216. https://www.usenix.org/conference/atc22/presentation/ choi-seungbeom [27] Brad Cline, Radu Stefan Niculescu, Duane Huffman, and Bob Deckel. 2017. Predictive maintenance applications for machine learning. In 2017 Annual Reliability and Maintainability Symposium (RAMS). 1–7. doi:10.1109/RAM.2017.7889679 [28] Patrick H. Coppock, Brian Zhang, Eliot H. Solomon, Vasilis Kypriotis, Leon Yang, Bikash Sharma, Dan Schatzberg, Todd C. Mowry, and Dimitrios Skarlatos. 2025. LithOS: An Operating System for Efficient Machine Learning on GPUs. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25). 1–17. doi:10. 1145/3731569.3764818 [29] Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph Gonzalez, and Alexey Tumanov. 2020. InferLine: LatencyAware Provisioning and Scaling for Prediction Serving Pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing (SoCC ’20). 477–491. doi:10.1145/3419111.3421285 [30] Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 613– 627. https://www.usenix.org/conference/nsdi17/technical-sessions/ presentation/crankshaw [31] William J. Dally, Stephen W. Keckler, and David B. Kirk. 2021. Evolution of the Graphics Processing Unit (GPU). IEEE Micro 41, 6 (2021), 42–51. doi:10.1109/MM.2021.3113475 [32] Priyanka Das. 2024. Real-Time IoT-Based Predictive Maintenance System for Automotive Assembly Lines. Fuel Cells Bulletin (02 2024). doi:10.52710/fcb.224 [33] Narjes Davari, Bruno Veloso, Rita P. Ribeiro, Pedro Mota Pereira, and João Gama. 2021. Predictive maintenance based on anomaly detection using deep learning for air production unit in the railway industry. In 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA). 1–10. doi:10.1109/DSAA53316.2021.9564181 [34] Aditya Dhakal, Sameer G Kulkarni, and K. K. Ramakrishnan. 2020. GSLICE: controlled spatial sharing of GPUs for a scalable inference platform. In Proceedings of the 11th ACM Symposium on Cloud Computing (SoCC ’20). 492–506. doi:10.1145/3419111.3421284 [35] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. https://arxiv.org/abs/2010.11929 xii

Strait: Perceiving Priority and Interference in ML Inference Serving [36] John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research 12, 61 (2011), 2121–2159. http://jmlr.org/papers/v12/duchi11a.html [37] Paul Elvinger, Foteini Strati, Natalie Enright Jerger, and Ana Klimovic. 2025. Understanding GPU Resource Interference One Level Deeper. In Proceedings of the 2025 ACM Symposium on Cloud Computing (SoCC ’25). 687–694. doi:10.1145/3772052.3772270 [38] GigaSpaces. 2023. Amazon Found Every 100ms of Latency Cost them 1% in Sales. https://www.gigaspaces.com/blog/amazon-found-every100ms-of-latency-cost-them-1-in-sales [39] Guin Gilman, Samuel S. Ogden, Tian Guo, and Robert J. Walls. 2021. Demystifying the Placement Policies of the NVIDIA GPU Thread Block Scheduler for Concurrent Kernels. SIGMETRICS Perform. Eval. Rev. 48, 3 (March 2021), 81–88. doi:10.1145/3453953.3453972 [40] Guin Gilman and Robert J. Walls. 2021. Characterizing concurrency mechanisms for NVIDIA GPUs under deep learning workloads. Performance Evaluation 151 (2021), 102234. doi:10.1016/j.peva.2021. 102234 [41] Roger Grosse. 2017. Lecture 8: Optimization. https://www.cs.toronto. edu/~cmaddis/courses/sta314_f25/rgrosse_optimization_notes.pdf [42] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 443–462. https://www.usenix.org/conference/osdi20/ presentation/gujarati [43] Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 539–558. https://www.usenix. org/conference/osdi22/presentation/han [44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778. doi:10. 1109/CVPR.2016.90 [45] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Neural Networks for Machine Learning. https://www.cs.toronto.edu/ ~tijmen/csc321/slides/lecture_slides_lec6.pdf [46] Yitao Hu, Rajrup Ghosh, and Ramesh Govindan. 2021. Scrooge: A Cost-Effective Deep Learning Inference System. In Proceedings of the ACM Symposium on Cloud Computing (SoCC ’21). 624–638. doi:10.1145/3472883.3486993 [47] Szu-Hao Huang and Ying-Cheng Pan. 2015. Automated visual inspection in the semiconductor industry: A survey. Computers in Industry 66 (2015), 1–10. doi:10.1016/j.compind.2014.10.006 [48] Nebbiolo Technologies Inc. 2020. Audi’s Automated Factory Moves Closer to Industry 4.0 with Intel’s Edge Machine Learning and Nebbiolo Technologies’ Intelligent Edge Computing Software Platform. https://www.prweb.com/releases/audi-s-automatedfactory-moves-closer-to-industry-4-0-with-intel-s-edge-machinelearning-and-nebbiolo-technologies-intelligent-edge-computingsoftware-platform-805836575.html [49] Rakshith Jayanth, Neelesh Gupta, and Viktor Prasanna. 2024. Benchmarking Edge AI Platforms for High-Performance ML Inference. https://arxiv.org/abs/2409.14803 [50] Beomyeol Jeon, Chen Wang, Diana Arroyo, Alaa Youssef, and Indranil Gupta. 2025. A House United Within Itself: SLO-Awareness for On-Premises Containerized ML Inference Clusters via Faro. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). Association for Computing Machinery, 524–540. doi:10.1145/3689031.3696071 [51] Yizhou Jin, Yu Lu, Gang Zhou, Qingjie Liu, and Yunhong Wang. 2023. Glass Wool Defect Detection Using an Improved YOLOv5. In 2023

IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 4385–4394. doi:10.1109/CVPRW59228.2023. 00461 [52] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. Ultralytics YOLOv8. https://github.com/ultralytics/ultralytics [53] Leela S. Karumbunathan. July 2022. NVIDIA Jetson AGX Orin Series: Technical Brief. https://www.nvidia.com/content/dam/enzz/Solutions/gtcf21/jetson-orin/nvidia-jetson-agx-orin-technicalbrief.pdf [54] Sejin Kim and Yoonhee Kim. 2021. Interference-aware execution framework with Co-scheML on GPU clusters. Cluster Computing 26, 5 (May 2021), 2577–2589. doi:10.1007/s10586-021-03299-z [55] Yeonjae Kim, Igjae Kim, Kwanghoon Choi, Jeongseob Ahn, Jongse Park, and Jaehyuk Huh. 2024. Interference-Aware DNN Serving on Heterogeneous Processors in Edge Systems. In 2024 IEEE 42nd International Conference on Computer Design (ICCD). 199–206. doi:10. 1109/ICCD63220.2024.00038 [56] Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980 [57] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, 611–626. doi:10.1145/3600006.3613165 [58] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation 1, 4 (1989), 541–551. doi:10.1162/neco.1989.1.4.541 [59] Seonho Lee, Amar Phanishayee, and Divya Mahajan. 2025. Forecasting GPU Performance for Deep Learning Training and Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 493–508. doi:10.1145/3669940.3707265 [60] C. L. Liu and James W. Layland. 1973. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment. Journal of the ACM (JACM) 20, 1 (Jan. 1973), 46–61. doi:10.1145/321738.321743 [61] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. https://arxiv.org/abs/1907.11692 [62] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11966–11976. doi:10.1109/CVPR52688.2022.01167 [63] Daniel Mendoza, Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. Interference-Aware Scheduling for Inference Serving. In Proceedings of the 1st Workshop on Machine Learning and Systems (EuroMLSys ’21). Association for Computing Machinery, 80–88. doi:10.1145/3437984.3458837 [64] Victor Millnert and Johan Eker. 2020. HoloScale: horizontal and vertical scaling of cloud resources. In 2020 IEEE/ACM 13th International Conference on Utility and Cloud Computing (UCC). 196–205. doi:10.1109/UCC48980.2020.00038 [65] Kelvin K. W. Ng, Henri Maxime Demoulin, and Vincent Liu. 2023. Paella: Low-latency Model Serving with Software-defined GPU Scheduling. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, 595–610. doi:10.1145/3600006.3613163 [66] Shadi A. Noghabi, Landon Cox, Sharad Agarwal, and Ganesh Ananthanarayanan. 2020. The Emerging Landscape of Edge Computing. GetMobile: Mobile Comp. and Comm. 23, 4 (May 2020), 11–20. doi:10.1145/3400713.3400717 xiii

Haidong Zhao and Nikolaos Georgantas [67] Christopher Olston, Fangwei Li, Jeremiah Harmsen, Jordan Soyke, Kiril Gorovoy, Li Lao, Noah Fiedel, Sukriti Ramesh, and Vinu Rajashekhar. 2017. TensorFlow-Serving: Flexible, High-Performance ML Serving. In Workshop on ML Systems at NIPS 2017. [68] Nathan Otterness and James H. Anderson. 2020. AMD GPUs as an Alternative to NVIDIA for Supporting Real-Time Workloads. In 32nd Euromicro Conference on Real-Time Systems (ECRTS 2020), Marcus Völp (Ed.), Vol. 165. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 10:1–10:23. doi:10.4230/LIPIcs.ECRTS.2020.10 [69] Arthi Padmanabhan, Neil Agarwal, Anand Iyer, Ganesh Ananthanarayanan, Yuanchao Shu, Nikolaos Karianakis, Guoqing Harry Xu, and Ravi Netravali. 2023. Gemel: Model Merging for MemoryEfficient, Real-Time Video Analytics at the Edge. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 973–994. https://www.usenix.org/conference/nsdi23/presentation/ padmanabhan [70] Ning Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural Networks 12, 1 (1999), 145–151. doi:10.1016/S08936080(98)00116-6 [71] Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2019. On the Convergence of Adam and Beyond. https://arxiv.org/abs/1904.09237 [72] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. 2020. MLPerf inference benchmark. In Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA ’20). IEEE Press, 446–459. doi:10.1109/ISCA45697.2020.00045 Milliseconds Make Millions. [73] Deloitte Research. 2020. https://www.deloitte.com/ie/en/services/consulting/research/ milliseconds-make-millions.html [74] Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 397–411. https://www.usenix.org/conference/atc21/presentation/romero [75] Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. https://www.ruder.io/optimizing-gradient-descent [76] Krzysztof Rzadca, Pawel Findeisen, Jacek Swiderski, Przemyslaw Zych, Przemyslaw Broniek, Jarek Kusmierek, Pawel Nowak, Beata Strack, Piotr Witusowski, Steven Hand, and John Wilkes. 2020. Autopilot: workload autoscaling at Google. In Proceedings of the Fifteenth European Conference on Computer Systems (EuroSys ’20). Association for Computing Machinery. doi:10.1145/3342195.3387524 [77] Bianca Schroeder, Adam Wierman, and Mor Harchol-Balter. 2006. Open Versus Closed: A Cautionary Tale. In 3rd Symposium on Networked Systems Design & Implementation (NSDI 06). USENIX Association. https://www.usenix.org/conference/nsdi-06/open-versusclosed-cautionary-tale [78] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). 205–218. https://www.usenix.org/conference/ atc20/presentation/shahrad [79] Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019.

Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). 322–337. doi:10.1145/3341301.3359658 [80] Weihang Shen, Mingcong Han, Jialong Liu, Rong Chen, and Haibo Chen. 2025. XSched: Preemptive Scheduling for Diverse XPUs. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 671–692. [81] Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. https: //arxiv.org/abs/1409.1556 [82] Ruben Sipos, Dmitriy Fradkin, Fabian Moerchen, and Zhuang Wang. 2014. Log-Based Predictive Maintenance. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’14). 1867–1876. doi:10.1145/2623330.2623340 [83] Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24). Association for Computing Machinery, 1075– 1092. doi:10.1145/3627703.3629578 [84] VarTech Systems. 2025. Reducing Latency: Edge AI vs. Cloud Processing in Manufacturing. https://www.vartechsystems.com/articles/ reducing-latency-edge-ai-vs-cloud-processing-manufacturing [85] Muhammad Tirmazi, Adam Barker, Nan Deng, Md E. Haque, Zhijing Gene Qin, Steven Hand, Mor Harchol-Balter, and John Wilkes. 2020. Borg: the next generation. In Proceedings of the Fifteenth European Conference on Computer Systems (EuroSys ’20). Association for Computing Machinery. doi:10.1145/3342195.3387517 [86] Si Van-Tien Tran, Doyeop Lee, Quy Lan Bao, Taehan Yoo, Muhammad Khan, Junhyeon Jo, and Chansik Park. 2023. A Human Detection Approach for Intrusion in Hazardous Areas Using 4D-BIM-Based Spatial-Temporal Analysis and Computer Vision. Buildings 13, 9 (2023). doi:10.3390/buildings13092313 [87] Rishi Vaish and Michael C. Hollinger. 2023. Case Study: IBM – Automating Visual Inspection. In Springer Handbook of Automation. 1439–1450. https://doi.org/10.1007/978-3-030-96729-1_69 [88] Deepak Vasisht, Zerina Kapetanovic, Jongho Won, Xinxin Jin, Ranveer Chandra, Sudipta Sinha, Ashish Kapoor, Madhusudhan Sudarshan, and Sean Stratman. 2017. FarmBeats: An IoT Platform for DataDriven Agriculture. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 515–529. https://www.usenix. org/conference/nsdi17/technical-sessions/presentation/vasisht [89] Vasily Volkov. 2016. Understanding latency hiding on GPUs. University of California, Berkeley. [90] Leyuan Wang, Zhi Chen, Yizhi Liu, Yao Wang, Lianmin Zheng, Mu Li, and Yida Wang. 2019. A Unified Optimization Approach for CNN Model Inference on Integrated GPUs. In Proceedings of the 48th International Conference on Parallel Processing (ICPP ’19). Association for Computing Machinery. doi:10.1145/3337821.3337839 [91] Lukasz Wesolowski, Bilge Acun, Valentin Andrei, Adnan Aziz, Gisle Dankel, Christopher Gregg, Xiaoqiao Meng, Cyril Meurillon, Denis Sheahan, Lei Tian, Janet Yang, Peifeng Yu, and Kim Hazelwood. 2021. Datacenter-Scale Analysis and Optimization of GPU Machine Learning Workloads. IEEE Micro 41, 5 (2021), 101–112. doi:10.1109/ MM.2021.3097287 [92] Bo Wu, Xu Liu, Xiaobo Zhou, and Changjun Jiang. 2017. FLEP: Enabling Flexible and Efficient Preemption on GPUs. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’17). Association for Computing Machinery, 483–496. doi:10.1145/3037697. 3037742 [93] Carole-Jean Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim Hazelwood, Eldad Isaac, Yangqing Jia, Bill Jia, Tommer Leyvand, Hao Lu, Yang Lu, Lin Qiao, Brandon Reagen, Joe Spisak, Fei Sun, Andrew Tulloch, Peter Vajda, Xiaodong Wang, xiv

Strait: Perceiving Priority and Interference in ML Inference Serving Yanghan Wang, Bram Wasti, Yiming Wu, Ran Xian, Sungjoo Yoo, and Peizhao Zhang. 2019. Machine Learning at Facebook: Understanding Inference at the Edge. In 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA). 331–344. doi:10.1109/HPCA.2019.00048 [94] Xiaorui Wu, Hong Xu, and Yi Wang. 2020. Irina: Accelerating DNN Inference with Efficient Online Scheduling. In Proceedings of the 4th Asia-Pacific Workshop on Networking (APNet ’20). Association for Computing Machinery, 36–43. doi:10.1145/3411029.3411035 [95] Carol Xu, Mahmoud Famouri, Gautam Bathla, Mohammad Javad Shafiee, and Alexander Wong. 2023. High-Throughput, HighPerformance Deep Learning-Driven Light Guide Plate Surface Visual Quality Inspection Tailored for Real-World Manufacturing Environments. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’23/IAAI’23/EAAI’23). AAAI Press. doi:10.1609/aaai.v37i13.26869 [96] Minghao Yan, Hongyi Wang, and Shivaram Venkataraman. 2024. PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices. https://arxiv.org/abs/2310.19991 [97] Ming Yang, Nathan Otterness, Tanya Amert, Joshua Bakita, James H. Anderson, and F. Donelson Smith. 2018. Avoiding Pitfalls when Using NVIDIA GPUs for Real-Time Tasks in Autonomous Systems. In 30th Euromicro Conference on Real-Time Systems (ECRTS 2018) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 106), Sebastian Altmeyer (Ed.). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 20:1–20:21. doi:10.4230/LIPIcs.ECRTS.2018.20 [98] Gingfung Yeung, Damian Borowiec, Renyu Yang, Adrian Friday, Richard Harper, and Peter Garraghan. 2022. Horus: InterferenceAware and Prediction-Based Scheduling in Deep Learning Systems. IEEE Transactions on Parallel and Distributed Systems 1 (Jan. 2022), 88–100. doi:10.1109/TPDS.2021.3079202 [99] Fuxun Yu, Di Wang, Longfei Shangguan, Minjia Zhang, Chenchen Liu, and Xiang Chen. 2022. A Survey of Multi-Tenant Deep Learning Inference on GPU. https://arxiv.org/abs/2203.09040 [100] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538. https://www.usenix.org/conference/osdi22/presentation/yu [101] Chenyang Zhang, Feng Zhang, Kuangyu Chen, Mingjun Chen, Bingsheng He, and Xiaoyong Du. 2023. EdgeNN: Efficient Neural Network Inference for CPU-GPU Integrated Edge Devices. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). 1193–1207. doi:10.1109/ICDE55515.2023.00096 [102] Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 787– 808. https://www.usenix.org/conference/nsdi23/presentation/zhanghong [103] Tan Zhang, Aakanksha Chowdhery, Paramvir (Victor) Bahl, Kyle Jamieson, and Suman Banerjee. 2015. The Design and Implementation of a Wireless Video Surveillance System. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom ’15). Association for Computing Machinery, 426–438. doi:10.1145/2789168.2790123 [104] Yongkang Zhang, Haoxuan Yu, Chenxia Han, Cheng Wang, Baotong Lu, Yunzhe Li, Zhifeng Jiang, Yang Li, Xiaowen Chu, and Huaicheng Li. 2025. SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP ’25). 267–281. doi:10.1145/3710848. 3710863

[105] Haidong Zhao and Nikolaos Georgantas. 2025. ML Inference Scheduling with Predictable Latency. In Proceedings of the Middleware for Autonomous AIoT Systems in the Computing Continuum (MAIoT ’25). Association for Computing Machinery, 25–30. doi:10.1145/3774901. 3778066 [106] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210. https: //www.usenix.org/conference/osdi24/presentation/zhong-yinmin

low-priority high-priority

Deadline Violation (%)

Haidong Zhao and Nikolaos Georgantas

1.00

] ] 0.25 [50,0.5 [50,

[50,

] ] 0.25 [65,0.5

[65,

] ] 0.25 [75,0.5

[75,

2.18% 2.44% 2.18%

1 0.37% 0.40%

1.0 0.5

±5%

±7.5%

±10%

Drift in Profiled Throughput

±12.5%

±15%

Figure 9. Profiling drift in resource throughput relative to the baseline without drift.

Appendix

A.1

Adaptive Throttling

high-priority

0.49%

low-priority

A.3 Ablation on Prioritization Mechanisms Figure 10 presents the ablation study by gradually removing a task prioritization mechanism, evaluated under an evolving workload. Removing task priority scheduling yields no noticeable degradation, as it neither manages interference nor enables deadline-aware scheduling. Removing CUDA stream priority multiplies deadline violations by 2.68× for HP tasks, demonstrating that hardware-level mechanisms are still useful. Removing deadline feasibility checking multiplies deadline violations by 1.35× and 1.42× for HP and LP tasks, respectively, suggesting that leaving some headroom for potential interference and carefully selecting batch size helps improve deadline satisfaction. The most significant impact arises from constraint violation checking. Removing checking of deadline violations of ongoing tasks and adaptive throttling leads to an 18.3× degradation in HP tasks and a substantial improvement for LP tasks, suggesting this is the most critical component in Strait.

We employ an AIMD policy to throttle the aggregate resource throughput of concurrent LP tasks on the GPU (Section 3.3). We select the control parameters to prevent LP tasks from oversubscribing GPU resources without causing severe underutilization. Figure 8 shows the results of our sensitivity study. Although a lower initial value appears to improve task prioritization, it risks severely penalizing LP tasks, particularly when 𝑎 = 50. Conversely, a higher increase rate may improve deadline satisfaction for LP tasks but risks reducing task prioritization. To balance these trade-offs, we adopt a larger initial value combined with a conservative increase rate. Specifically, 𝐶𝑙𝑜𝑤 ranges from 75% to 100%, with an increase rate of 0.25% every 100 ms. When HP tasks miss their deadlines, 𝐶𝑙𝑜𝑤 returns to 75%. A.2

0.50%

sensitivity to profiling drift. Figure 9 shows that under a random perturbation rate of 15%, the changes in p99 prediction error remain below 2 pp, suggesting that Strait can adapt to moderate levels of profiling drift. The small impact on prediction accuracy is attributable to Strait’s online learning mechanism, which dynamically tunes the model parameters. Consequently, the prediction model continuously adapts to the discrepancy between the perturbed profiling data and the actual runtime behavior, thereby absorbing moderate profiling drift. To mitigate inference latency drift, we suppose that the latency can be quickly reprofiled when necessary, or that an online monitoring mechanism can be employed to recalibrate for potential variations, provided that the batch executes in isolation.

1.5

±2.5%

3.10%

0.99%

Figure 10. Impact of sequentially removing a task prioritization mechanism.

[75,

Figure 8. Deadline violation rates under various configurations. For a configuration [a, b], the first term a denotes the initial value, that is, the lowest allowed capping value for LP tasks, whereas b denotes the associated increase rate applied every 100 ms.

0.0

w/o checking for meeting deadlines w/o constraint violation checking

6.77%

0.10

P99 Prediction Drift (%)

Strait (full version) w/o task priority scheduling w/o CUDA stream priority

Sensitivity to Profiling Drift

Strait relies on profiling data to estimate interference. We evaluate the impact when profiling data drifts due to environmental changes (e.g., different runtime versions or hardware configurations), without re-profiling the models. To simulate this, we artificially perturb the stored throughput values. We carefully control the perturbation magnitudes to ensure that all perturbed throughput values remain within valid bounds (0%–100%). We use an evolving workload to evaluate Strait’s xvi

Record · ID 146911 · SHA-256 4778c43c960223ab
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.