Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference

R IPPLE : D EADLINE -AWARE CPU–GPU S HARING FOR E MBODIED I NFERENCE

arXiv:2611.01067v1 [cs.DC] 31 Oct 2026

Jiadong Zhang 1 Xiaosong Ma 1

A BSTRACT Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control. GPU memory traffic can overwhelm these tasks, yet platforms such as DGX Spark expose neither bandwidth counters nor direct bandwidth regulation. We present Ripple, a CPU–GPU co-scheduler that combines model-derived GPU memory volumes with CPU completion feedback to regulate interference. Ripple divides inference into pausable, layer-based units and translates each CPU job’s deadline slack into a GPU-byte budget, admitting work within the budgets of all overlapping jobs. Online updates track changing interference, while deadline-aware pacing and judicious SM footprint narrowing preserve inference progress. Implemented with SGLang engine hooks and a central scheduler, we evaluate Ripple on DGX Spark with four robotic CPU task mixes and LLM/VLM inference. It is the only evaluated approach that consistently meets the 2% CPU deadline-miss target and VLM deadlines across the tested deadline settings while retaining best-effort LLM throughput. 1

I NTRODUCTION

AI models increasingly run on edge devices (Apple, 2020; NVIDIA, 2026i), with privacy protection, low/consistent latency, low cost, and device portability. Many of these devices adopt a unified memory architecture (UMA), where the CPU and GPU share one physical memory on one package, as with Apple’s M-series processors and NVIDIA’s DGX Spark. UMA is cheaper, smaller, and easier to build and maintain, eliminating host-device memory copies and providing a decent capacity (128GB on DGX Spark) to hold reasonably large models (e.g., 32B at FP16). With the rapid growth of agentic and embodied AI, edge AI devices are increasingly versatile, often pairing model inference with CPU computation tasks for perception and control, which typically come with timing requirements. Consider a home robot working on clearing a dinner table while interacting with human users on travel planning. Every few 10s of milliseconds its CPU must work out how far away things are and where it can safely move, or the robot may walk into a person who has just stepped into its path; every few seconds a vision-language model (VLM) examines the table again; meanwhile a large language model (LLM) plans the next robot movements, retrieves family travel history/preference and calendar events using retrieval-augmented generation (RAG) (Lewis et al., 2020), and generates responses to the user. A missed obstacle-avoidance cycle can cause an acci1

dent, a late scene update stalls the next action, and a slower answer merely arrives later — all three run on one chip and share its memory bus (Macenski et al., 2020; Driess et al., 2023; Rana et al., 2023; Chang et al., 2025). Our study finds that memory bandwidth is the contention point, in a lopsided fight, dominated by the less timecritical inference tasks (user response generation in the above scenario). Figure 1 showes this with a workload mix on the GB10 processor of DGX Spark, with 4 routine robotic/agentic CPU tasks — costmap updates, nearestneighbor queries, stereo matching, and vector retrieval over the robot’s memory. The left chart reports the mutual slowdown when this CPU mix co-runs with a GPU-side LLM decode task, with growing CPU thread allocations (CPU-side performance saturating at 12). The more deadline-sensitive CPU tasks suffer much higher loss. The right chart provides a reference by running the same test on an H200 server, with separate HBM. The CPU tasks sees no impact of concurrent LLM inference there, as opposed to 1.7-2.5× on the Spark.

MBZUAI. Figure 1. CPU–GPU interference under LLM decode.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

Managing bandwidth on these devices is harder than on a conventional CPU, where we have homogeneous cores and a rich set of hardware counters. On edge AI devices with UMA, bandwidth management faces four major challenges: (1) the platform lacks bandwidth counters and exposes no interface that reports per-requester memory bandwidth, neither that regulates bandwidth consumption; (2) modern transformer-based AI models are bandwidth-intensive even at token granularity: on GB10 one LLM token has a 6974ms forward pass and one VLM image a 285ms one, longer than the 45 to 70ms deadlines for recurrent CPU robotic tasks; (3) the CPU side is not designed to take most of the unified memory bandwidth, consuming at most 60% of it even with all cores; (4) the power-aware design of edge devices perform aggressive and black-box, temperaturetriggered clock-throttling, further complicating performance monitoring and resource regulation. These constraints rule out most existing (and relatively mature) multi-tenant bandwidth management solutions (Yun et al., 2013; Ali & Yun, 2018; Sohal et al., 2022; Bakita & Anderson, 2023; 2025; Qiao et al., 2026; NVIDIA, 2026f), because the interfaces they rely on are missing. The one scheduler to our knowledge that work without, published recently, is S ERENO (Xin et al., 2026), which protects foreground mobile apps from background NPU inference. It targets a phone SoC, where the bandwidth relinquished by the NPU can be almost entirely absorbed by the CPU. Edge UMA devices are not zero-sum in this way. Also, S ERENO pairs one foreground app with one background model, where our setting has many concurrent workloads on both sides, each with its own timing requirement. This work is based on the observation that “what the CPU and GPU cannot do symmetrically, they can do complementarily.” GPU inference is opaque in time and transparent in volume: when a forward pass will finish depends on contention and thermal throttling, but its memory access volume is well known. A real-time CPU task is the exact opposite — transparent in time and opaque in volume: its release time and deadlines are set in stone, whether each deadline is met can be easily known, but access volume remains not observable without detailed source code instrumentation. Together, the two pieces of information give sufficient guidance for our scheduling, compensating for the lack of bandwidth counters (or even surpassing their capability). To this end, we propose Ripple, a CPU-GPU co-scheduling framework that performs fine-grained GPU inference task admission, to protect CPU-side deadline-sensitive tasks while retaining overall bandwidth utilization and inference speed as allowed by the deadlines. Ripple overcomes the lack of counters by leveraging the semantic knowledge on today’s mainstream transformer-based LLM/VLM workflow to decompose the inference forward pass into smaller,

layer-based work units with known memory access volumes, for GPU-side bandwidth regulation. It performs continuous modeling by observing the GPU-side memory traffic on CPU-side deadline-sensitive tasks, leading to a pricing model that releases metered volume budgets to admit GPU work units, dynamically adjusted to current clock throttling levels and CPU-side contentions. We implement Ripple as a standalone scheduler with engine hooks in SGLang and evaluate it on DGX Spark with four mixes of robotic CPU tasks plus LLM/VLM inference. It is shown to consistently meet CPU deadline-miss targets and VLM deadlines, while retaining substantial best-effort LLM throughput across workload mixes and deadline settings. Novelty To the best of our knowledge, we are the first to target the bandwidth asymmetry on DGX Spark. Also, Ripple is the first system that makes independent CPU tasks’ declared deadlines and miss tolerances explicit constraints for LLM/VLM co-serving on UMA devices. It introduces the first counter-free scheduler that translates CPU deadline slacks into GPU memory-byte budgets, with the first theoretical throughput bound for best-effort inference under these CPU timing constraints. Scope. This proof-of-concept study targets declared deadline miss tolerances and batch-one, pausable inference.

2

BACKGROUND AND M OTIVATION

2.1

Unified Memory at the Edge

Unified Memory Architecture (UMA) (Glaskowsky, 1999) began as a power- and cost-driven solution in integrated graphics (Intel, 2002; Branover et al., 2012), enabling a small GPU to “borrow” system memory instead of carrying its own. It has since become the common design for AIcapable edge devices: Apple’s M-series processors, AMD’s Ryzen AI Max, NVIDIA’s Jetson modules as well as DGX Spark all come with a single memory, of up to 128GB, directly accessed by both the GPU and CPU side. Table 1 summarizes these recent models and their major parameters (Mittal & Vetter, 2015; Apple, 2020; 2024; AMD, 2026a; NVIDIA, 2026d;e;i). All the above devices with UMA adopt LPDDR (Low-Power Double-Data-Rate memory) (JEDEC, 2023). Unlike the DDR modules of desktops and servers, LPDDR is designed for battery-powered devices: it runs at lower voltages, enters deep power-saving states when idle, and is soldered next to the processor rather than socketed. It offers large capacities at low power and price, with modest bandwidth, as reflected by the bandwidth/GB column and power/GB columns. Edge GPUs therefore raise their bandwidth by widening the LPDDR bus, to 256 bits on DGX Spark, or by raising its pin rate, as Jetson Thor does over Orin (JEDEC,

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference Table 1. Memory and power of representative GPUs. “–” denotes missing vendor specifications (Apple, 2024; NVIDIA, 2026h). Device

Memory

Edge, UMA M4 Max LPDDR5 Ryzen AI Max+ LPDDR5X Jetson Orin LPDDR5 Jetson Thor LPDDR5X DGX Spark LPDDR5X

Size BW BW Power Shared (GB) (GB/s) /GB /GB w/ CPU 128

546 4.3

–

yes

128 64 128 128

256 205 273 273

2.0 3.2 2.1 2.1

0.9 0.9 1.0 1.1

yes yes yes yes

Discrete RTX 5090 GDDR7

32

1792

56

18

no

Figure 2. GB10 package and memory topology

Datacenter H100 NVL HBM3 H200 NVL HBM3e MI300A HBM3 GB300a HBM3e

94 141 128 288

3900 41 4800 34 5300 41 8000 28

4.3 4.3 4.3 4.9

no no yes no

DRAM throughput out of the 273GB/s from NVIDIA specification, depending on the access pattern (Appendix A).

a

Per GPU of GB300 NVL72. Its Grace CPU has its own LPDDR5X (Aubrey & Stam, 2025; NVIDIA, 2026a).

2023; 2025; 2026). Such limited bandwidth is further shared between GPU and CPU. This stands in stark contrast to datacenter GPUs, also shown in the table, where the more familiar HBM (High Bandwidth Memory) delivers 13 to 20× the bandwidth/GB of DGX Spark, with rated chip/module power of 4.3–4.9 W per GB of memory. Among the datacenter models listed, only the AMD MI300A shares its memory between GPU and CPU, and its bandwidth/GB is around 20× that of DGX Spark. Discrete accelerators, such as the NVIDIA RTX 5090, adopt GDDR (Graphics DDR) memory, which drives each pin several times faster at higher power and delivers a bandwidth/GB even higher than the datacenter GPUs (Table 1; NVIDIA, 2025a; Wahlgren et al., 2025; Aubrey & Stam, 2025; NVIDIA, 2026b;c; AMD, 2026b;c). 2.2

Memory Bandwidth Contention on DGX Spark

For the rest of the paper, we focus our discussion on the NVIDIA DGX Spark architecture (“GB10” for short). Released in October 2025, it has rapidly grown into a product line sold by seven major PC makers and become a mainstream edge AI device (NVIDIA, 2025b). Neither the bandwidth contention problem on UMA devices (Table 1) nor our proposed approach, meanwhile, is specific to GB10. DGX architecture GB10, the DGX Spark processor, places a CPU and a GPU die on a single package (Figure 2). The CPU has 20 cores in two clusters, each containing 5 performance (P) and 5 efficiency (E) cores. The 48-SM GPU die reaches the 128GB shared LPDDR5X DRAM through NVLink-C2C and the memory controllers on the CPU die. Our benchmarking achieves 190-257GB/s combined

Sharing introduces contention, especially with the low overall bandwidth availability relative to capacity. GB10 provides no direct bandwidth allocation mechanism between its CPU and GPU sides: how bandwidth divides appears to depend on how many memory requests each keeps in flight. This gives the GPU side an easy advantage, where thousands of memory accesses can be issued with its massive parallelism, burying the small number of time-critical and latency-sensitive CPU requests. The above asymmetry is too strong to mitigate through high-level concurrency parameters, in other words, CPU core or GPU SM counts, as shown earlier in Figure 3.

Figure 3. (a) Slowdown brought by GPU inference (with different SM counts) on CPU time-sensitive tasks; (b) bandwidth with concurrent CPU+GPU memory streaming

Bandwidth Management Challenges In addition to such significant CPU-GPU imbalance, UMA devices like GB10 pose unique challenges for bandwidth management: No observation or regulation Unlike server CPUs, with mature per-workload memory bandwidth monitoring/allocation mechanisms such as Intel RDT or ARM MPAM (Sohal et al., 2022; Arm Limited, 2022), the GB10 firmware exposes no memory-controller counters and does not enable MPAM. Our tests found that NVIDIA DCGM’s DRAM-activity metric returns no reading. Limited GPU control An MPS SM limit is set when a process starts and remains throughout its execution. Lower-

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

ing the GPU clock reduces GPU application throughput while freeing little bandwidth. The recent CUDA green contexts (NVIDIA, 2026g) can confine a stream’s kernels to a subset of the SMs. Our measurement (Figure 3(a)), however, demonstrates that reducing LLM’s GPU SM count from the full 48 to 8 (minimum allowed) has very limited effect on mitigating CPU slowdown. Meanwhile, with the GB10 CPU clusters unable to saturate the memory bus, when both the GPU and CPU sides run memory streaming benchmarks, the bandwidth yielded by reducing the GPU SM count cannot be absorbed by the CPU, resulting in low overall bandwidth utilization, as shown in Figure 3(b). Run-to-run performance variability Being power- and heataware, ddge devices constantly adjust CPU/GPU clock rate to temperature, further complicating bandwidth control. 2.3

Major Edge Workloads

CPU-side Representative CPU tasks on edge AI devices, in the current pursuit of AGI, are diverse and often tightly timed. Embodied systems run their periodic perception, localization, and control loops on CPU: constructing LiDARbased occupancy grids, stereo disparity and image features, nearest-neighbour queries for scan matching, and modelpredictive control. These loops run with fixed frequency, typically at 10-50Hz with deadlines of tens of milliseconds. Agentic systems retrieve from their memory before responding: in a spoken conversation that retrieval must return within the few hundred milliseconds of a turn. Table 2. Common edge robotic CPU tasks (Bentley, 1975; Hirschmüller, 2008; Stivers et al., 2009; Rublee et al., 2011; Geiger et al., 2012; Macenski et al., 2020; Stellato et al., 2020; Park et al., 2023; Open Navigation LLC, 2026). Task

Description

Costmap Maps nearby obstacles from LiDAR scans. KDNN Finds nearby points to help align scans and track position. SGM Estimates object distances from two camera images. Control Computes quadrotor control commands using model-predictive control (OSQP). ORB Extracts visual features from camera images.

Table 2 lists representative CPU tasks in embodied AI workloads. Unlike GPU tasks, they tend to be complex yet resource-light. Each of the above tasks has its algorithm and access pattern, but moves at most 1.6GB of data per iteration, about 1/10 of what an LLM moves in generating one token (e.g., 15.2GB for Qwen3-8B). A task typically tolerates a small fraction of misses, as in the weakly-hard model (Bernat et al., 2001) of real-time systems. The tasks run with independent intervals and phases, with overlapping

deadline windows. GPU-side GPU workloads on edge AI devices, on the other hand, are computationally intensive, with rather elastic timing requirements. Examples include LLMs for planning and answering and VLMs (Vision-Language Models) for scene understanding, reasoning, and task planning in embodied robots. With all mainstream models, a forward pass goes through a regular sequence of layers, with per-layer model weights streamed from DRAM. Edge LLM/VLM inference processing is particularly bandwidth-hungry during decode, where a small batch (mostly 1) reads GBs per token. The small batch sizes, limited model sizes, and limited context lengths in edge AI scenarios also make KV cache access much lighter than in datacenter services: our 8192-token context used with Qwen3-32B-FP8 yields KV cache memory access at around 6.5% of model weight read volume. Between LLM and VLM, the latter requires timely updates to its understanding of the environment to support reasoning and task planning, though with an interval often much longer than CPU-side perception/control tasks. Also, they spend host time differently: a VLM decode step issues about a thousand kernels lasting 28 µs on average, making CUDA graph replay indispensable to avoid a host bottleneck; the LLM kernels are fewer and last much longer, receiving negligible performance gain from CUDA graphs. Finally, running the LLM and VLM engines on GB10 side by side has little GPU SM-level interference: DRAM bandwidth is the clear contention point (Appendix B).

3

A PPROACH OVERVIEW

Rationale In this work, we leverage several opportunities to overcome the lack of UMA bandwidth monitoring or regulating mechanisms. First, the time-critical CPU tasks do not require full dedication of CPU resources or DRAM bandwidth, so uniform GPU throttling would be an overkill. Second, the many layers of LLM/neural-network models provide a natural granularity of GPU-side task admission, with known data volume, to allow bandwidth regulation without wasting resources on aborted work. Finally, one does not need a detailed hardware counter for our purpose: the coarse observation based on the completion times of past jobs, plus the GPU-side DRAM access volume during each deadline window, provide sufficient information for task-level coordination. Note that a task’s completion time already reflects how much the job itself read and how fast the clocks ran, giving a complete CPU-side task profile. Based on these observations, we present Ripple, a CPUGPU task scheduler for UMA devices that protects timecritical CPU tasks while retaining the progress of bandwidthhungry GPU inference. Most importantly, it does so without requiring a bandwidth counter/arbitrator. Instead, it focuses

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

on informed GPU task admission, predicting the delay the task adds to every active CPU job. Note that GPU work could also be deadline-bound, such as VLM inference for the scene updates that guide the robot’s next action. However, due to the biased memory arbitration discussed in Section 2.2, only the CPU tasks need bandwidth protection: deadline-bound GPU tasks are safe as long as Ripple gives them scheduling priority.

scheduler then takes the global budget B(t), the minimum across active uncertified CPU jobs, selects the longest grant that fits it, and charges its volume to every active CPU job. When a CPU job completes, the scheduler updates that task’s model of its perceived interference from GPU volumes. When the same PT’s next job is released, Ripple adjusts that job’s allowance from the updated model.

System model and objective Following the above reasoning, we term the time-critical CPU workloads protected tasks (PTs). Each P Ti releases a sequence of jobs and is associated with an inter-arrival interval Ti : known and fixed for periodic tasks such as costmap updates and stereo matching, and a declared minimum for sporadic tasks such as memory retrieval for a spoken reply. P Ti also comes with a relative deadline Di ≤ Ti and a tolerated deadline miss rate (DMR) εi . Every job reports its completion; Ripple neither modifies nor throttles the task. Each GPU inference workload is either deadline-bound (with a per-request deadline), or best-effort. Ripple observes only the release/completion time of a CPU job and the DRAM access volume and duration of each GPU job. It learns about the CPU tasks online from their completions and requires little offline profiling. The more predictable GPU inference tasks are profiled once per device and model, recording each unit’s completion time, each engine’s overhead per grant, and the peak byte rate ρ. In priority order, its objectives are (1) that in every window of ni consecutive jobs of a protected task, at most a fraction εi miss Di , except with probability at most δ (a chance constraint (Charnes & Cooper, 1959)), where P Ti,j misses when its completion time CTi,j , measured from its release Ri,j , exceeds Di ; (2) that every deadline-bound GPU workload meets its deadline; and (3) that best-effort GPU work makes as much progress as possible. Ripple acts only on the GPU work it schedules: when each piece is dispatched and on how many SMs. Non-time-critical CPU tasks or the division of service among several best-effort workloads (CPU or GPU) is beyond the scope of this work. Main idea Ripple leverages the common software architecture for GPU model inference today for scheduling GPU work at an appropriate granularity. It cuts each engine’s forward pass into units, the units at which execution can pause with state information in memory, and authorizes a contiguous run of units, called a grant, at a time. Each CPU job receives an allowance Ai,j , the total GPU memory access volume (“volume” for short) assigned from its predicted deadline slack. Its budget Bi,j (t) is the volume available for additional GPU work at time t: the allowance released by then, less the volume already charged. When an engine finishes a grant, it reports its execution time. The

Figure 4. Ripple’s modules and their interactions, with solid lines indicating control flow while dashed ones feedback

Software architecture Ripple comprises three major modules, as illustrated in Figure 4. It adds engine hooks in SGLang (Zheng et al., 2024) that divide forward passes at unit boundaries, pause engines between grants while preserving state, apply SM allocations through CUDA green contexts, and report each grant’s CUDA time. Its scheduler runs as a separate process occupying one performance core; it maintains per-job byte accounts, orders GPU work, and admits grants within the CPU jobs’ budgets. It controls the SGLang engines (also running on CPU cores), one per inference service, via the Ripple hooks. The Ripple estimator runs within the scheduler, updating each CPU task’s model on job completion (dashed feedback edges) and computing allowances for subsequent jobs.

4

R IPPLE D ESIGN

4.1

Software Architecture and Scheduling Workflow

Ripple manages UMA bandwidth through GPU-side task admission, with no interference with the scheduling of CPU-side tasks. It deploys a central scheduler, along with its scheduling-relevant components through SGLang modification. Such a Ripple-customized SGLang scheduler performs bandwidth-aware LLM/VLM task decomposition/regrouping at layer boundaries, as well as the GPU task monitoring/reporting duties required by Ripple. The Ripple scheduler, which makes all admission decisions, occupies one GB10 performance core by default, due to its heavy processing and time-sensitive task nature; the SGLang processes of the inference engines share a separate set of cores, disjoint from those of the PTs.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

The chief goal of the Ripple scheduler is to maximize overall DRAM bandwidth utilization (and consequently the GPU inference throughput), while keeping each time-sensitive task’s deadline misses within its tolerance: PTs on CPU and deadline-bound GPU tasks. It makes admission decisions at scheduling points triggered by the following events: (1) a CPU PT job starts or completes; (2) a GPU grant completes; (3) an engine announces new work; or (4) the budget accrues enough to schedule the next unit. On the CPU side, as mentioned earlier, Ripple does not interfere with CPU tasks’ scheduling or execution, but observes the slowdown they experience from GPU memory bandwidth usage (major) and other CPU tasks or environment variances such as thermal throttling (minor). Ripple builds a per-PT interference model through linear regression and updates it constantly by new data points collected upon PT job completion. Based on the model, it assigns each PT job a total volume allowance Ai,j for its lifetime. At time t, P Ti,j ’s budget Bi,j (t) is the released part of Ai,j minus the GPU volume already charged to the job. Section 4.2 details the slowdown modeling and allowance/budget calculation. On the GPU side, Ripple breaks an LLM/VLM forward pass into units at the natural boundaries of modern transformerbased models: transformer layers, the output head, and the attention/MLP blocks of a vision encoder. At each scheduling point, if possible, Ripple admits the next grant, the longest stretch of one engine’s units, that fits the global budget B(t), the minimum over active uncertified PT jobs. Deadline-bound and best-effort units are ordered by the Ripple scheduler by their own deadlines: a deadline-bound request’s work gets due times spread evenly over a fraction γ of the interval to its deadline, and a best-effort grant runs before the next due time only if it ends by then and leaves enough budget for the request’s next grant.

duration estimate āi,j and slope estimate κui,j , P Ti,j has the nominal allowance   max(0, D − ā ) i i,j max , (2) , A Ai,j = min i κui,j converting the remaining deadline slack into allowed GPU volume. Here āi,j reserves time for the job’s own execution, CPU-side inter-task interference (very minor), thermal throttling, and run-to-run variances.1 Amax = ρDi caps i tasks with κ near zero (insensitive to bandwidth contention), where ρ is the peak GPU memory access rate. Ai,j is the total GPU volume allowed by job P Ti,j , released linearly over its predicted interference-free duration âi,j . At time t within this interval following P Ti,j ’s release time Ri,j , the job has a remaining budget Bi,j (t) =

Ai,j (t − Ri,j ) − Ci,j (t), âi,j

(3)

where Ci,j (t) is the volume already charged to the job. At a scheduling point (time t), Ripple identifies the next grant, the maximum sequence of units whose combined volume fits under the global budget B(t) = min Bi,j (t), for all active P Ti,j

(4)

so that the volume is allowed by the most sensitive PT. Once issued, the GPU grant is charged in full to every PT it overlaps with, as it simultaneously delay all of them.

As mentioned in Section 2.3, of the two engines, only the VLM gains from CUDA-graph replay, so Ripple keeps SGLang’s breakable graphs and pauses between their segments. The LLM, in contrast, runs eagerly and can leave and re-enter its forward pass at any unit boundary. A grant that an engine has started runs to its end, since work submitted to the GPU cannot be recalled. 4.2

Figure 5. Ripple’s control loop illustrated with sample budget calculation and grant issuing

Pricing GPU Work Against CPU Deadlines

Ripple models how GPU volume delays P Ti ’s jth job: CTi,j = ai + κi Ci,j + ei,j ,

(1)

where ai estimates the job’s interference-free duration, κi its delay by GPU volume, and ei,j the residual execution time not explained by the model. Upon the job’s completion, Ripple refits ai and κi by least squares with exponential forgetting (Ljung & Söderström, 1983), following changes in measured completion times. Using a conservative CPU

Figure 5(a) illustrates the allowance/budget calculation with 3 PTs and no active GPU workloads. The purple solid line marks the global budget B(t), which follows the lowest active per-PT budget line (dashed lines). Figure 5(b) simplifies with only one PT, but adding two LLM/VLM inference tasks, with grants admitted subject to B(t) (the illustrated 1 Details of Ripple’s pricing model and budget management are given in Appendix C.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

LLM and VLM grants require 0.45 GB and 1 GB, respectively). When P T1 completes, Ripple is able to release more LLM units in a single grant. Different height of the grant boxes indicate SM narrowing applied only to LLM decode. 4.3

Dynamic SM Regulation and Overhead Mitigation

Dynamic SM allocation Ripple is built with the recognition that common high-level concurrency tuning knobs, such as through the CPU core and GPU SM allocations, do not effectively protect CPU PTs, as explained in Section 2.2. However, within the framework of Ripple’s model-based pricing and fine-grained GPU job admission, we re-examined the potential of dynamic SM allocation, through the relatively recent NVIDIA green context mechanism. We observe that the effect of reducing the SM footprint (referred to as SM narrowing in our discussion) depends on the GPU workload. Intuitively, it slows compute-heavy operators substantially. A bandwidth-bound decode job, on the other hand, can retain much of its speed when narrowed to fewer SMs. Interestingly, it can maintain most of its throughput while reducing CPU delay per GB of its GPU traffic, in other words, a flatter interference slope κi . This is because that fewer SMs would reduce the intensity of concurrent GPU memory accesses, allowing more breathing room for CPU tasks. A lower interference slope κi can increase the allowance and let the budget Bi,j (t) accrue faster, allowing earlier GPU admission. Ripple therefore runs prefill at full width and retains fullwidth CUDA graphs for the VLM, including its computeheavy vision-encoder units. For continuous LLM decode jobs overlapping active CPU jobs, Ripple uses a fixed SM width (typically 8 or 16), based on its quick profiling during warm-up. CPU-free decode uses all SMs. Figure 8(a) in the next Section gives related evaluation results. This allows Ripple to expand its solution to a 2-D space of admission/scheduling and SM footprint management. While a more comprehensive investigation is planned for our future work, these initial results reveal that under careful task decomposition and admission, mechanisms such as green context could deliver a smoothing effect on GPU bandwidth consumption under edge CPU–GPU co-running, not previously observed at inference engines’ native scheduling granularity. Grant pipelining To hide the scheduler round trip, Ripple authorizes and delivers one successor grant while the current grant runs. At the next grant boundary, the engine checks the cached grant’s work identity, lease and CPU-release limit, and discards it if stale. A successor delivered ahead for a CPU release waits until that release before it can start.

5

E VALUATION

We implemented Ripple’s three modules (engine hooks, scheduler, and estimator) in Python, with approximately 10.5k, 3.7k, and 2.2k LoC, respectively. Our evaluation examines four properties: (1) Ripple’s capability to meet the CPU and VLM targets across representative CPU and GPU workload combinations (Section 5.2); (2) its capability to retain GPU-side inference throughput while meeting given DMR limits (Section 5.3); (3) its overhead and adaptation to changes (Section 5.3); and (4) the impact of individual design/optimization (Section 5.3). 5.1

Experimental Setup

Testbed We evaluate on an ASUS Ascent GX10, an NVIDIA DGX Spark system. Its GB10 SoC has a 20-core Arm CPU (10 Cortex-X925 and 10 Cortex-A725 cores) and a Blackwell GPU with 48 SMs, which share 128 GB of LPDDR5X at 273 GB/s. The machine runs Ubuntu 24.04.3 LTS with Linux 6.14, NVIDIA driver 580.95.05 and CUDA 13.0, and SGLang 0.5.12. Workloads and configurations Our evaluation runs three robotic perception/control tasks on the CPU (from those listed in Table 2), with an LLM and a VLM inference engine on the GPU. We create two CPU workload mixes, Robotic1 (Costmap, KDNN, and SGM), and Robotic2 (Control, ORB, and SGM). For LLM and VLM, we test two model configurations each: LLML with Qwen3-32B-FP8, LLMS with Qwen3-8B, V LML with Qwen3-VL-30B-A3B-FP8, and V LMS with Qwen3-VL-8B (Yang et al., 2025; Bai et al., 2025). We evaluate four workload mixes to examine how workload composition affects protection and throughput. CPU PTs use 100 ms periods, following KITTI’s 10 Hz sensor rate (Geiger et al., 2012), except Control, which runs every 20 ms, a 50 Hz cadence used in embedded quadrotor MPC (Salzmann et al., 2023). We set the deadlines of Costmap, KDNN, and SGM to k times their co-run p99, measured with the other CPU tasks running and both GPU engines resident but idle, and sweep k from 1.1 to 2.0 to cover different amounts of timing slack. For detailed experiments, we select k = 1.5, giving these tasks 50% slack over their co-run p99; results at other settings are labeled separately. Control and ORB use deadlines equal to their periods, requiring completion within the control cycle or before the next frame; the Robotic2 sweep therefore varies only SGM’s deadline. In our experiments, the VLM period and deadline are both 5 s for V LML and 10 s for V LMS ; the LLM runs continuously without a deadline. We use a DMR limit of 2% for each CPU task in the scored window and require every VLM frame to meet its deadline. We report mean throughput with one-standard-deviation error bars (often invisible).

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

Figure 6. End-to-end results. (a) Worst-task DMR of CPU PTs and LLM tokens/s; hollow markers indicate a CPU DMR limit or VLM deadline was missed. (b) Response times with deadlines at k = 1.5 on the (Robotic1 , LLMS , V LML ) mix; each inset magnifies where the curves cross the 98th percentile.

Baselines As the first framework designed for real-time CPU tasks that share a unified memory with GPU inference, we have a hard time identifying baseline systems. Though there are no direct competitors, we do a best-effort comparison with several existing systems or alternative strategies: (1) Sereno (Xin et al., 2026), designed for mobile SoCs that run background model inference beside foreground apps, pauses the LLM between decode steps to yield time shares to UI tasks (we run its “Balanced” and “UI-First” policies); (2) SiGAMMA (Capodieci et al., 2017) uses a high-priority blocking kernel to suppress GPU memory traffic while a PT job is active; (3) Pause pauses both engines while any PT job is active, using SGLang’s own pause command, which takes effect after the forward pass in flight; (4) Cap-x uses the same command and gives each PT job a budget fraction x of its solo median time to tolerate the engines’ co-run (x = 1/8, 1/4, 1/2), pausing the latter whenever a active PT job has spent its budget; Sereno and SiGAMMA run using their published configuration, untuned, on SGLang 0.5.12 with the same engine flags used by Ripple. Only Ripple react to the deadline. For reference, we also have the unmanaged strategy, where all GPU-side engine jobs are admitted at once while the PT workloads run unprotected.

5.2

Effectiveness in CPU PT Protection

Figure 6 (a) gives Ripple’s overall performance, with 4 CPUGPU workload mixes, at different deadline scaling factors (k along the x axis). Markers are filled only when all CPU tasks meet the 2% DMR limit and every VLM frame meets its deadline, otherwise hollow. The first row plots the worst deadline miss rate (DMR) across all three CPU PTs (y axis in log scale). Ripple is the only solution meeting the DMR requirement consistently. A distant second is SiGAMMA, getting below the 2% target with the most relaxed 2-3 deadline settings, due to its partial suppression of inference through time-slicing with its blocking kernel. All the other baselines not only miss the target in all or nearly all tests, but produce close-to-100% DMRs, with the worst CPU task missing most of its deadlines even at k = 1.5. This indicates existing approaches’ failure in regulating the GPU-side aggressive memory bandwidth usage. In particular, the Sereno strategies fail to respond to the remaining slack of individual CPU jobs, since their controller observes the LLM’s slowdown rather than CPU deadlines. Even the pause strategy, which is in theory the harshest on the GPU workloads, is unable to contain the inference engines’ memory traffic due to the pause request

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

taking effect only after the in-flight forward pass completes. The second row plots the GPU-side throughput, by LLM token generation speed. Here we add a theoretical throughput bound (diamonds), computed by replacing each GPU work unit with a fixed-rate memory-access stream with the same volume. Therefore it also removes Ripple overheads associated with discrete grants, VLM deadlines, and the scheduling protocol. As expected, Ripple’s result trails behind this theoretical throughput bound, due to the conservative nature of its budget-based admission, its safety margins, as well as Ripple’s overhead. Meanwhile, it clearly out performs any solutions that meet the DMR requirement (SiGAMMA, which delivers lower LLM throughput due to its 48-SM “blocking kernel” wasting GPU resources even with relatively safe CPU deadlines). With more relaxed deadlines (k=1.8 or higher), it delivers higher LLM throughput than a few approaches that yields much higher DMR, as it admits more GPU work as deadline slack increases, whereas these baselines retain deadline-independent throttling. Among the four CPU-GPU workload combinations, the LLML + V LML GPU combination is most aggressive in bandwidth use, rendering most baselines miss the DMR target significantly, even with the most forgiving 2× deadline setting. Robotic2 , meanwhile, is slightly less vulnerable than Robotic1 , due to its lower bandwidth demand (Control and ORB has a per-job data volume of 10s of MBs, while Costmap and KDNN are at 100MB-GB level.) Overall, the performance behavior of Ripple and the baselines are quite consistent across the board. Figure 6(b) shows PT response-time CDFs at k = 1.5 for Robotic1 , LLMS , and V LML ; the vertical dashed lines mark deadlines. Under Ripple, the 98th-percentile response times of Costmap, KDNN, and SGM fall 8.6, 3.4, and 4.9 ms below their respective deadlines, significantly outperforming the baselines. The shapes of the CDF curves here also clearly reflect the vulnerability of each CPU PT, with SGM most sensitive to GPU bandwidth occupation.

Figure 7. Sensitivity to ε on the (Robotic1 , LLMS , V LML ) mix. Labels show worst-task CPU DMR.

Figure 7 examines Ripple’s capability to work under varied DMR targets, with ε varied between 0.5 and 5%, under 1.5× and 2× deadlines, respectively. For reference, we provide the LLM throughput of three baselines, SiGAMMA,

Pause (SGLang), and Unmanaged, who deliver flat lines as they are oblivious to deadlines. Ripple, on the other hand, is able to relax its GPU work admission and produces higher inference performance while staying within the DMR tolerance. Consistent with Figure 6(a), Ripple allows better LLM throughput in general, compared to baselines that meet the DMR target. The dip of Ripple throughput from 4% to 5% DMR target with 1.5× deadline shows its sensitivity to misses, which cost bandwidth utilization with the admission decision adjusted more conservative.

Figure 8. Breakdown and (Robotic1 , LLMS , V LML ) mix

5.3

adaptivity

on

the

Breakdown Analysis

Impact of individual techniques Figure 8(a) gives the DMR and LLM throughput results by incrementally adding our proposed techniques: “pricing” (base-variation with pricing only), “grant” (issuing contiguous units), and “dynamic SM” (SM narrowing with green context on LLM decode jobs). For comparison, we have two versions of the “Pause” strategy, one native from SGLang (pausing entire forward passes), the other at our refined unit level. The SGLang native Pause significantly exceeds the 2% DMR limit (84.8%), although it stops the GPU whenever a CPU job runs. When pausing at unit level, it becomes extra-conservative. With each of its optimizations, the green Ripple bars safely stay under the 2% DMR limit to protect the CPU PTs, while allowing more efficient bandwidth utilization by the GPU tasks, finally reaching 90.4% of the SGLang Pause’s throughput. Adaptivity With its pricing model observing the CPU PTs’ performance, Ripple is able to adapt to variability due to CPU-side contention or thermal throttling. The latter factor is beyond our control while the former can be tested. Figure 8(b) evaluates Ripple by injecting two sporadic CPU tasks (memory readers) during a 200s run. The top plot shows how the KDNN’s allowance automatically lowers during the two perturbation intervals and recovers thereafter, thanks to the āi,j term in our pricing model. The bottom plot shows the DMR readings (binned at 3s), comparing Ripple with a variation that has its adaptation component disabled

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

(dashed line). The difference is dramatic: KDNN’s DMR averages at 0.6% and 1.0% in the two intervals, respectively, while the non-adaptive version reaches 60.4% and 67.0%.

Inference preemption and co-serving ConServe (Qiao et al., 2026) preempts offline inference at layer boundaries, while Valve (Liu et al., 2026) bounds preemption latency and rate to protect GPU requests. Sereno (Xin et al., 2026) yields background NPU inference according to draft-subgraph latency to protect foreground applications. Ripple admits work at fine boundaries against each independent CPU job’s declared deadline; inference latency alone does not reveal that job’s slack.

Figure 9. Breakdown of inter-grant gap time as percentages of scored time on the (Robotic1 , LLMS , V LML ) mix.

Overhead We then examine the management and scheduling overhead of Ripple, by measuring the percentage of time during the GPU workloads’ active execution that Ripple has no grant running. Our detailed profiling reveals the breakdown of such “gap time”, plotted in Figure 9. With a more stringent deadline (1.5×), the gap is dominated by “budget wait”, the time Ripple waits for sufficient budget to release the next unit, a price it has to pay to regulate GPU bandwidth usage and protect CPU PTs. This item becomes much smaller with a 2× deadline, as Ripple schedules units more aggressively. The other categories, such as Ripple decision wait and delivery times, are its overhead. They also shrink when the deadline is relaxed, as there is a higher chance more units are issued within each grant, reducing this cost. Together, such gap times contribute to the inference throughput loss against the “unmanaged” reference in Figure 6. At 2× deadline, only 5% of GPU time is not running inference. Other results After a 3 s GPU copy, allowances recovered within 20 s for KDNN and SGM and 24 s for Costmap in both runs. Without a CPU-task profile, Ripple misses no more deadlines in its first 10 s than when started with one. The same configuration meets the CPU DMR limits and VLM deadlines across all four workload mixes without perworkload tuning. In two one-hour runs at k = 1.5, every 100 s window met CPU DMR limits, with no VLM deadline misses.

6

R ELATED W ORK

Real-time memory regulation Memory regulators use budgets or coordinated phases. MemGuard (Yun et al., 2013) budgets per-core memory accesses, and RT-Gang (Ali & Yun, 2019) throttles best-effort cores while a real-time gang runs. BWLOCK++ (Ali & Yun, 2018) throttles CPU traffic to protect GPU kernels; SiGAMMA (Capodieci et al., 2017) blocks GPU accesses during PREM (Pellizzoni et al., 2011) CPU memory phases. These require counters and throttling or phase separation. Ripple learns from CPU completions, admitting overlapping inference within per-job allowances without bandwidth counters or CPU memoryphase annotations.

GPU spatial partitioning libsmctrl (Bakita & Anderson, 2023) and nvtaskset (Bakita & Anderson, 2025) provide compute partitioning; CUDA green contexts (NVIDIA, 2026g) restrict a stream’s kernels to selected SMs. DuetServe (Gao et al., 2025) adaptively divides SMs between prefill and decode to meet GPU latency targets. Ripple combines per-grant SM control through green contexts with temporal admission to protect CPU deadlines. Narrowing can lower CPU interference per GPU byte and increase the allowance; throughput improves only when enough additional admitted work offsets slower execution.

Prediction and feedback Clockwork (Gujarati et al., 2020) predicts inference duration, SOLA (Hong et al., 2025) updates latency estimates using measured-to-predicted ratios, and FC-EDF (Stankovic et al., 1999) adjusts service levels and admission from deadline-miss feedback. Ripple combines duration prediction with a model of CPU delay per GPU byte and adaptive conformal inference (Gibbs & Candès, 2021) for its CPU prediction margin (Appendix C); together they turn deadline slack into a GPU-byte allowance.

7

C ONCLUSION AND F UTURE W ORK

Ripple demonstrates that with UMA edge AI devices, the CPU can accommodate diverse time-critical robotic/agentic workloads while contemporary LLM/VLM inference runs on the GPU. We protect the less intensive, but highly sensitive CPU-side memory accesses by leveraging semantic knowledge as well as available resource management mechanisms on the GPU workloads to overcome the lack of bandwidth counters on these edge devices with unified memory. Two questions merit further study. First, narrowing the SM footprint can reduce CPU interference per GPU byte, but the throughput benefit depends on the workload and deadline slack. Jointly choosing the SM count and admission timing could exploit this trade-off more fully. Second, long prefill operations call for finer execution boundaries; the challenge is to shorten each uninterrupted interval without excessive weight rereads or scheduling overhead.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

R EFERENCES Aghilinasab, H., Ali, W., Yun, H., and Pellizzoni, R. Dynamic memory bandwidth allocation for real-time GPU-based SoC platforms. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(11):3348–3360, 2020. doi: 10.1109/TCAD.2020.3012210. Ali, W. and Yun, H. Protecting real-time GPU kernels on integrated CPU-GPU SoC platforms. In 30th Euromicro Conference on Real-Time Systems (ECRTS 2018), volume 106 of LIPIcs, pp. 19:1–19:22, 2018. doi: 10.4230/LIPIcs.ECRTS.2018.19. Ali, W. and Yun, H. RT-Gang: Real-time gang scheduling framework for safety-critical systems. In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 143–155, 2019. doi: 10.1109/RTAS.2019.00020. AMD. AMD Ryzen AI Max+ 395. Official product specifications, 2026a. URL https://www.amd.com/en/products/proc essors/laptop/ryzen/ai-300-series/am d-ryzen-ai-max-plus-395.html. AMD. AMD Instinct MI300A accelerators. Official product specifications, 2026b. URL https://www.amd.com/en/products/acce lerators/instinct/mi300/mi300a.html. Accessed October 23, 2026. AMD. AMD Instinct MI300A APU overview. AMD GPU Driver (amdgpu) documentation, 2026c. URL https://instinct.docs.amd.com/projec ts/amdgpu-docs/en/latest/gpu-partiti oning/mi300a/overview.html. Accessed October 23, 2026. Angelopoulos, A. N. and Bates, S. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023. doi: 10.1561/2200000101. Apple. Apple unleashes M1. Apple Newsroom, November 2020. URL https://www.apple.com/newsro om/2020/11/apple-unleashes-m1/. Accessed September 23, 2026. Apple. Apple introduces M4 Pro and M4 Max. Apple Newsroom, October 2024. URL https: //www.apple.com/newsroom/2024/10/app le-introduces-m4-pro-and-m4-max/. Arm Limited. Arm Architecture Reference Manual Supplement: Memory System Resource Partitioning and Monitoring (MPAM), for A-profile Architecture. Arm Limited, 2022.

Aubrey, K. and Stam, N. Inside NVIDIA Blackwell Ultra: The chip powering the AI factory era. NVIDIA Technical Blog, August 2025. URL https://developer.nvidia.com/blog/in side-nvidia-blackwell-ultra-the-chi p-powering-the-ai-factory-era/. Published August 22, 2025. Accessed October 23, 2026. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., and Zhu, K. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. URL https://arxiv.org/abs/2511.21631. Bakita, J. and Anderson, J. H. Hardware compute partitioning on NVIDIA GPUs. In Proceedings of the 29th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 54–66. IEEE, 2023. URL https: //www.cs.unc.edu/˜jbakita/rtas23.pdf. Bakita, J. and Anderson, J. H. Hardware compute partitioning on NVIDIA GPUs for composable systems. In Proceedings of the 37th Euromicro Conference on Real-Time Systems (ECRTS), volume 335 of LIPIcs, pp. 21:1–21:25. Schloss Dagstuhl — Leibniz-Zentrum für Informatik, 2025. doi: 10.4230/LIPIcs.ECRTS.2025.21. URL https: //www.cs.unc.edu/˜jbakita/ecrts25.pdf. Bentley, J. L. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18 (9):509–517, 1975. doi: 10.1145/361002.361007. Bernat, G., Burns, A., and Llamosı́, A. Weakly hard real-time systems. IEEE Transactions on Computers, 50 (4):308–321, 2001. doi: 10.1109/12.919277. Branover, A., Foley, D., and Steinman, M. AMD Fusion APU: Llano. IEEE Micro, 32(2):28–37, 2012. doi: 10.1109/MM.2012.2. Capodieci, N., Cavicchioli, R., Valente, P., and Bertogna, M. SiGAMMA: Server based integrated GPU arbitration mechanism for memory accesses. In Proceedings of the 25th International Conference on Real-Time Networks and Systems, pp. 48–57. Association for Computing Machinery, 2017. doi: 10.1145/3139258.3139270. URL

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

https://hipert.unimore.it/people/mar ko/papers/RTNS17.pdf. Casella, G. and Berger, R. L. Statistical Inference. Duxbury, 2nd edition, 2002. Chang, M., Chhablani, G., Clegg, A., Cote, M. D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., Patki, S., Prasad, I., Puig, X., Rai, A., Ramrakhya, R., Tran, D., Truong, J., Turner, J. M., Undersander, E., and Yang, T.-Y. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations (ICLR), 2025. Charnes, A. and Cooper, W. W. Chance-constrained programming. Management Science, 6(1):73–79, 1959. Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 8469–8488. PMLR, 2023. URL https://proceedi ngs.mlr.press/v202/driess23a.html. Gao, L., Jiang, C., Zarch, H. E., Wong, D., and Annavaram, M. DuetServe: Harmonizing prefill and decode for LLM serving via adaptive GPU multiplexing, 2025. URL https://arxiv.org/abs/2511.04791. Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012. Gibbs, I. and Candès, E. Adaptive conformal inference under distribution shift. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2021. Glaskowsky, P. N. Intel innovates with integrated graphics. Microprocessor Report, 13(6), May 1999. URL https://www.cecs.uci.edu/˜papers/mpr/ MPR/19990510/130604.pdf. Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving DNNs like Clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443–462. USENIX Association, 2020. URL https://www.usenix.org/conference/os di20/presentation/gujarati.

Hirschmüller, H. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2): 328–341, 2008. doi: 10.1109/TPAMI.2007.1166. Hong, K., Li, X., Chen, L., Mao, Q., Dai, G., Ning, X., Yan, S., Liang, Y., and Wang, Y. SOLA: Optimizing SLO attainment for large language model serving with state-aware scheduling. In Proceedings of Machine Learning and Systems (MLSys), volume 7, 2025. Intel. Intel 810 and 815 chipset family dynamic video memory technology. Technical Specification Revision 3.0, Intel Corporation, March 2002. URL https://www.intel.com/content/www/us /en/content-details/840352/intel-810 -and-815-chipset-family-dynamic-video -memory-technology-product-specifica tions.html. JEDEC. Low power double data rate (LPDDR) 5/5x. Standard JESD209-5C, JEDEC Solid State Technology Association, July 2023. URL https://www.jedec.org/standards-docum ents/docs/jesd209-5c. JEDEC. High bandwidth memory (HBM3) DRAM. Standard JESD238B.01, JEDEC Solid State Technology Association, April 2025. URL https://www.jedec.org/standards-docum ents/docs/jesd238b01. JEDEC. Graphics double data rate 7 SGRAM standard (GDDR7). Standard JESD239F, JEDEC Solid State Technology Association, August 2026. URL https://www.jedec.org/standards-docum ents/docs/jesd239f. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pp. 9459–9474, 2020. Liu, F., Liu, H., Lyu, X., Ai, S., Liang, H., Chen, L., Hu, Z., Zha, C., Jin, X., Luo, H., and Chen, P. Valve: Production online-offline inference colocation with jointly-bounded preemption latency and rate, 2026. URL https://arxiv.org/abs/2604.07874. Ljung, L. and Söderström, T. Theory and Practice of Recursive Identification. MIT Press, Cambridge, MA, 1983. Macenski, S., Martı́n, F., White, R., and Ginés Clavero, J. The marathon 2: A navigation system. In 2020

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020. doi: 10.1109/IROS45743.2020.9341207. URL https://arxiv.org/abs/2003.00368.

NVIDIA. CUDA driver API: Green contexts. Official CUDA documentation, 2026g. URL https: //docs.nvidia.com/cuda/cuda-driver-a pi/group__CUDA__GREEN__CONTEXTS.html.

Mittal, S. and Vetter, J. S. A survey of CPU-GPU heterogeneous computing techniques. ACM Computing Surveys, 47(4):1–35, 2015. doi: 10.1145/2788396. URL https://doi.org/10.1145/2788396.

NVIDIA. Hardware overview: NVIDIA DGX Spark user guide. Official documentation, 2026h. URL https://docs.nvidia.com/dgx/dgx-spark /hardware.html.

NVIDIA. NVIDIA RTX Blackwell GPU architecture. Official architecture whitepaper, 2025a. URL https://images.nvidia.com/aem-dam/Sol utions/geforce/blackwell/nvidia-rtx-b lackwell-gpu-architecture.pdf.

NVIDIA. System overview: NVIDIA DGX Spark porting guide. Official hardware documentation, 2026i. URL https://docs.nvidia.com/dgx/dgx-spark -porting-guide/latest/overview.html. Last updated October 21, 2026. Accessed October 23, 2026.

NVIDIA. NVIDIA DGX Spark arrives for world’s AI developers. NVIDIA Newsroom, October 2025b. URL https://nvidianews.nvidia.com/news/n vidia-dgx-spark-arrives-for-worlds-a i-developers. NVIDIA. NVIDIA GB300 NVL72. Official product page. Specifications list CPU memory as LPDDR5X and GPU memory as HBM3E, reported as rack totals (17 TB LPDDR5X; 20 TB HBM3E across 72 GPUs), 2026a. URL https://www.nvidia.com/en-us/data -center/gb300-nvl72/. NVIDIA. NVIDIA H100 Tensor Core GPU: Product specifications. Official product page, 2026b. URL https://www.nvidia.com/en-us/data-cen ter/h100/. Accessed October 23, 2026. NVIDIA. NVIDIA H200 GPU: Specifications. Official product specifications, 2026c. URL https://www. nvidia.com/en-in/data-center/h200/. Accessed October 23, 2026.

Open Navigation LLC. Nav2 Controller Server: Configuration Parameters. Nav2 documentation, 2026. URL https: //docs.nav2.org/rolling/configuratio n_and_development/configuration_guid e/core_servers/controller_server/. Accessed September 15, 2026. Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pp. 1–22, 2023. doi: 10.1145/3586183.3606763. Pellizzoni, R., Betti, E., Bak, S., Yao, G., Criswell, J., Caccamo, M., and Kegley, R. A predictable execution model for COTS-based embedded systems. In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 269–279, 2011. doi: 10.1109/RTAS.2011.33.

NVIDIA. Jetson AGX Orin for next-gen robotics. Official product page: 64 GB / 32 GB 256-bit LPDDR5, 204.8 GB/s, 2026d. URL https: //www.nvidia.com/en-us/autonomous-mac hines/embedded-systems/jetson-orin/. Accessed October 23, 2026.

Qiao, Y., Yu, S., Anzai, S., Ma, H., Yang, S., Wang, Y., Kim, M., Wu, Y., Zhou, Y., Xing, J., Gonzalez, J. E., Stoica, I., and Xu, H. ConServe: Fine-grained GPU harvesting for LLM online and offline co-serving. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https: //icml.cc/virtual/2026/poster/62686.

NVIDIA. NVIDIA Jetson Thor: Technical specifications. Official product specifications, 2026e. URL https: //www.nvidia.com/en-au/autonomous-mac hines/embedded-systems/jetson-thor/. Accessed October 23, 2026.

Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I., and Sünderhauf, N. SayPlan: Grounding large language models using 3D scene graphs for scalable robot task planning. In Proceedings of the 7th Conference on Robot Learning (CoRL), 2023.

NVIDIA. NVIDIA Multi-Instance GPU user guide. Official documentation, 2026f. URL https://docs.nvidia.com/datacenter/t esla/mig-user-guide/. Accessed September 21, 2026.

Rublee, E., Rabaud, V., Konolige, K., and Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2564–2571, 2011. doi: 10.1109/ICCV.2011.6126544.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

Salzmann, T., Kaufmann, E., Arrizabalaga, J., Pavone, M., Scaramuzza, D., and Ryll, M. Real-time neural MPC: Deep learning model predictive control for quadrotors and agile robotic platforms. IEEE Robotics and Automation Letters, 8(4), 2023. doi: 10.1109/LRA.2023.3246839. URL https://arxiv.org/abs/2203.07747. Sohal, P., Bechtel, M., Mancuso, R., Yun, H., and Krieger, O. A closer look at Intel Resource Director Technology (RDT). In Proceedings of the 30th International Conference on Real-Time Networks and Systems (RTNS), 2022. doi: 10.1145/3534879.3534882. Stankovic, J. A., Lu, C., Son, S. H., and Tao, G. The case for feedback control real-time scheduling. In Proceedings of the 11th Euromicro Conference on Real-Time Systems, pp. 11–20. IEEE, 1999. doi: 10.1109/EMRTS.1999.777445. URL https://www. cse.wustl.edu/˜lu/papers/ecrts99.pdf. Stellato, B., Banjac, G., Goulart, P., Bemporad, A., and Boyd, S. OSQP: An operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4):637–672, 2020. doi: 10.1007/s12532-020-00179-2. Stivers, T., Enfield, N. J., Brown, P., Englert, C., Hayashi, M., Heinemann, T., Hoymann, G., Rossano, F., de Ruiter, J. P., Yoon, K.-E., and Levinson, S. C. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106 (26):10587–10592, 2009. doi: 10.1073/pnas.0903616106. Wahlgren, J., Schieffer, G., Shi, R., León, E. A., Pearce, R., Gokhale, M., and Peng, I. Dissecting CPU-GPU unified physical memory on AMD MI300A APUs. arXiv preprint arXiv:2508.12743, 2025. URL https://arxiv.org/abs/2508.12743. Xin, T., Shi, X., Dong, M., and Mi, Z. Inference in the shadows: Taming memory bandwidth contention in mobile LLM inference with Sereno. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 2299–2318. USENIX Association, 2026. URL https://www.usenix.o rg/conference/osdi26/presentation/xin. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren,

X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388. Yun, H., Yao, G., Pellizzoni, R., Caccamo, M., and Sha, L. MemGuard: Memory bandwidth reservation system for efficient performance isolation in multi-core platforms. In 2013 IEEE 19th Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 55–64. IEEE, 2013. doi: 10.1109/RTAS.2013.6531079. URL https://www.ittc.ku.edu/˜heechul/pap ers/memguard-rtas13.pdf. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2312.07104.

AI U SAGE D ISCLOSURE We used OpenAI Codex and Anthropic Claude iteratively under author direction to explore design alternatives, discover and summarize literature, propose experimental contrasts, implement and debug runtime and experimentharness code, develop data-reduction and figure-generation scripts, and draft, revise, and format manuscript text. This assistance included executing author-authorized commands and helping interpret retained experiment outputs. Reported hardware measurements are drawn from experiment records rather than AI-generated numerical observations. The authors retain responsibility for the design decisions, experimental methodology, references, analysis, and all content of the paper.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

A

H ARDWARE AND I NTERFERENCE M EASUREMENTS

Shared-memory contention. The selected co-run measurements give about 257 GB/s for sequential CPU reads beside Qwen3-8B decode and 190 GB/s for read–write copying, against the nominal 273 GB/s. Table 3 summarizes how the rates are computed. GPU decode traffic is calculated from its output rate and per-token byte count; CPU traffic is calculated by the benchmark from its processed data and elapsed time. In separate measurements, Nsight Compute L2 read misses agree with checkpoint-based decode-byte counts within 0.7 % for Qwen3-8B and Qwen3-32B in BF16 and AWQ. Beside Qwen3-8B decode, a CPU thread that reads 30 GB/s alone obtains 6.6 GB/s, while the combined rate remains about 50 GB/s below the peak. Thus the peak is not a fixed pool either processor can fully reclaim when the other withdraws. In a separate experiment, an eight-thread CPU reader shares memory with a GPU reader reduced by duty cycling, SM count, or arithmetic intensity. The comparison uses three matched GPU rates. The CPU recovers 0.401, 0.572, and 0.570 GB/s per GB/s withdrawn, respectively. DRAM rather than cache interference. A CPU reader sustains 150–151 GB/s beside GPU data fitting in the GPU’s 24 MB L2, but only 85–86 GB/s when the GPU data spill to DRAM. Thrashing the 16 MB system-level cache does not shift this threshold. A dependent CPU load chain’s mean access latency rises from 126.1 to 283.7 ns under a GPU stream. Its last-level read miss ratio changes by only +0.5%, and memory accesses per instruction remain unchanged. A dependent integer chain without memory traffic retains its rate. The dependent-load experiment thus identifies longer memory service time rather than additional cache misses or a slower CPU. Available interfaces. Table 4 describes the tested firmware and software. The administrator-level inventory finds CPUcore, statistical-profiling, and SMMU counters, but no uncore or memory-controller PMU: firmware provides neither an APMT table for those units nor an MPAM table, although the kernel has both drivers.

Table 3. Inputs and calculations for the shared-memory bandwidth measurements. Quantity

Definition

16.4 GB/token

Qwen3-8B weight bytes per batch-1 decode token used in the co-run calculation Decode tokens/s × 16.4 GB/token Benchmark bytes processed divided by elapsed time Sum of GPU and CPU rates Nominal GB10 memory bandwidth

GPU rate CPU rate Combined rate 273 GB/s

Table 4. Memory and GPU control interfaces on the evaluated GB10 stack. Means

Effect on CPU

Shortcoming

Uncore / memory-controller PMU DCGM and Nsight DRAM metrics CPU core PMU, GPU L2 counters MPAM, MIG, OS regulation GPU clock 2.4→1.1 GHz CPU clock

–

Not exposed by the firmware

–

None for this GPU

–

One side’s cache traffic only Not enabled, or blind to the GPU Too small; locking halves decode Limit accepted, not applied Static, deadline-blind

SMs 48→24 CPU cores, engine priorities

– p50 −7% – p50 −9%, tokens −14% CPU side / GPU queue only

Cannot move the GPU’s share

CPU traffic and narrowing. With the GPU idle, core bus counters give about 1.1 GB per Costmap job, 370 MB for SGM, 250 MB for KDNN, 11 MB for ORB, and 2 MB for Control. The two smallest include cache hits and are upper bounds. With Costmap, KDNN, and SGM together, average CPU traffic is 18.7 GB/s and the 1 ms p99 is 64–65 GB/s; their p99 completion times rise 2–7% over running alone. In the synthetic GPU-stream experiment, narrowing FP8 work to 16 of 48 SMs reduces its byte rate by 8% and CPU delay per GB by 9–35%. BF16 still saturates memory at 16 SMs, with harm per byte unchanged within run-to-run spread.

B

GPU VOLUME ACCOUNTING AND E XECUTION

Operation granularity. Table 5 compares the duration of complete forwards with individual operations: Qwen38B decode takes 69–74 ms per forward, while an operation spans 1.68–8.40 ms. Table 6 identifies the operation boundaries used by the two engines: Ripple can admit one or more operations in a grant, with eager execution for the LLM and breakable CUDA graphs for VLM decode. Byte charges. A decode step’s weight and KV traffic grows with model size and context length; Figure 10 converts the counted bytes at a 16,384-token context to step time using a 220 GB/s streaming rate. A decoder layer at sequence length T is charged Wℓ + Aℓ + (T + 1)K: weights, activation traffic, and KV reads for T positions plus one write. Counts come from checkpoint tensors and the engine’s sequence length; the output head, final norm, and embedding row are charged separately. Qwen3-8B has Wℓ = 0.386 GB and K = 4096 bytes, so each thousand context tokens adds

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference Table 5. Diagnostic CUDA-event spans on GB10, including host enqueue gaps. Model / phase

Whole forward (ms)

Single operation (ms)

Qwen3-8B Prefill Qwen3-8B Decode Qwen3-VL Image extend Qwen3-VL Decode

76.61

1.83–7.37

69.00–74.04

1.68–8.40

285.23

0.05–4.72

19.48–21.10

0.02–4.69

Ranges show observed minima and maxima.

Figure 10. Time one batch-1 decode step takes at a 16,384-token context when its bytes stream at 220 GB/s. Appendix B gives the method.

about 1% to step bytes. A short-context step reads 15.19 GB in 69–74 ms, or 205–220 GB/s. MoE decode charges the router and selected experts; image prefill charges all experts. One-metric-per-run Nsight Compute checks match computed decode bytes to L2 read misses within 0.7% for Qwen3-8B/32B in BF16 and AWQ; a known-size stream agrees within 0.02%. An initial VLM check corrected its per-step charge by 17%. Subsequent Qwen3-32B-FP8 and Qwen3-VL-30B-A3B/8B checks agree within 0.5% for decode, 0.4% for routed/all-expert MoE charges, and 1% for a frame. These checks are offline; admission reads no counter. Prefill can reread weights across GEMM tiles: Qwen3-8B reads 1.6 times its weights at 1024 input tokens and 18.9 times at 16,384, versus 1.009 for the evaluation’s 37-token prompt. Execution boundaries. LLM execution is eager; VLM decode uses SGLang’s breakable CUDA graphs, with eager image processing and prefill. Figure 11 compares graph replay with eager execution in each engine alone. Replay speeds VLM decode by 30–38% and a frame by 16–21% in engine-only tests, versus 0–3.4% for the tested dense LLMs. Segmenting a graph without admission callbacks increases a fixed frame sequence’s duration by 0.4%. VLM

decode issues roughly 1000 kernels averaging 28 µs; tested LLM kernels average 166 µs. A CUDA event marks grant completion, so reports measure execution rather than submission. Stream dependencies preserve serial order when switching SM widths. Separate engine processes time-slice on the tested GB10 stack; restricting one does not reserve SMs for the other. In a single-process BF16 matrix-kernel microbenchmark, concurrent decode-like streams took 0.9– 1.6% longer than serial execution; a decode-like stream paired with a prefill-like stream took 5.4–5.7% longer.

C

P RICING AND E STIMATOR D ETAILS

Each task’s model uses recursive least squares with exponential forgetting: effective windows of W = 50 jobs for the intercept and Wκ = 200 for the slope. The held slope changes only when the new estimate is positive and at least two standard errors above zero. The adaptive margin uses the last Wq = 200 residuals, starts after 50 samples, and uses one adaptive level across VLM phases. Before then, solo profiles seed the fit and margin; the warm-up margin uses the larger of the phase and pooled quantiles over up to 50 residuals. The phase split separates releases in the first 10 % of a VLM period from the rest. The conservative CPU duration is āi,j = âi,j + qi,j : the fitted interference-free duration plus a residual margin. Margin. The margin is a rank among the ni,j ≤ Wq retained residuals whose level αi,j adapts after every job,   qi,j = ê(m) , m = (1 − αi,j )(ni,j + 1) , (5) αi,j+1 = αi,j + η (ε′i − erri,j ), with η = 0.01, where erri,j = 1 if the job exceeds its saved release-time pricing line âi,j + qi,j + si,j Ci,j ; an unfinished job exceeds any finite line. Each residual êi,j = CTi,j − âi,j − κ̂i,j Ci,j is computed before its job enters the fit. This adaptive conformal inference update (Gibbs & Candès, 2021; Angelopoulos & Bates, 2023) widens the margin by lowering the level when a job runs longer than its saved prediction plus margin, and otherwise raises it gradually. The feedback error and the deadline miss CTi,j > Di are scored separately; rank endpoints are specified in Appendix C. Window target. To leave room for variation within the contract’s window (Section 3), Ripple sets ε′i = p −6 max(10 , εi −zε εi (1 − εi )/ni ) with zε = Φ−1 (1−εi ) and ni = max(1, ⌊L/Ti ⌋) for L = 100 s. The subtraction uses a normal approximation to an independent binomial count (Casella & Berger, 2002) as the reference buffer; actual window misses are evaluated from job outcomes. With εi = δ = 2 %, ε′i is 1.09 % for the 100 ms tasks (ni = 1,000) and 1.59 % for the 20 ms Control task in Robotic2 (ni = 5,000).

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference Table 6. Selected single-request SGLang paths for Qwen3-8B and Qwen3-VL-30B-A3B-Instruct-FP8. Counts are registered operations per forward pass; one grant can admit several. Engine and phase Qwen3-8B Prefill Qwen3-8B Decode Qwen3-VL Image/prompt Qwen3-VL Decode

Operations Boundary groups

Execution mode

38 Input preparation; 36 decoder layers; output head 37 36 decoder layers (preparation included with the first); output head 108 Preparation; vision input, attention/MLP regions, and merger; language input, 48 layers, norm, and output 52 Preparation; language input, 48 layers, norm, and output

Eager Eager Eager Breakable CUDA graphs

time must finish 1 ms before it. Pacing. Releasing a job’s allowance linearly over âi,j (Section 4.2) is what budget-pacing regulators do (Aghilinasab et al., 2020). With floor f , the budget is    Ai,j (t − Ri,j ) Bi,j (t) = min Ai,j , max f, âi,j (6) − Ci,j (t),

Figure 11. CUDA-graph ablation, engine alone on GB10. (a) Throughput change of graph replay over eager execution, one contrast per bar; the hatched bar is 48 graph segments per decode step against one. (b) Host-dispatch idle per VLM decode token.

Pessimistic slope. Pricing reserves slack for residual variation and slope uncertainty. The margin covers the scatter of jobs around the fit of Eq. 1 at the GPU volumes they received, but not the error of the slope itself, which grows with the volume. The uncertainty estimate is −1 κui,j = max(0, κ̂LS (1 − εi ), usi,j ) + zε SEi,j with zε = Φ ing the current least-squares slope. Pricing uses si,j = max(κ̂i,j , κui,j ), where κ̂i,j is the held slope, or si,j = κ̂i,j /0.85 until a positive uncertainty estimate is available. Equivalently, βi,j = κ̂i,j /si,j is the share of residual slack used for GPU volume. The cap Amax = ρDi uses the proi filed peak rate ρ: the larger of the highest per-unit byte rate and the rate measured with both engines decoding concurrently. Issuing the allowance. The nominal allowance is ratelimited to a base that changes by at most ±25 % per release, using one unit as the growth reference below that size, and remains capped at Amax . Downward rate limiting can retain i bytes above the new nominal allowance. While slope uncertainty exceeds 10 % or is not yet estimable, one job in eight of the task with the smallest allowance receives 40 % fewer bytes. The one-unit probes below also affect the issued Ai,j . A best-effort grant before a deadline-bound request’s due

with âi,j floored at 1 ns. An allowance below one unit receives a one-unit probe on alternate releases, subject to the guard below. The probe size is the same for every task: the largest, over the bound engines, of each engine’s smallest nonzero decode unit, so that every engine can issue one unit of its steady-state work under it (385,978,880 bytes, one Qwen3-8B layer, on (Robotic1 , LLMS , V LML )); a job issued exactly Amax , because its raw allowance exceeds it, i has its whole allowance released at entry. Inherited grants remain charged separately, as described below. Charging. Bytes are charged at admission and never refunded once a grant has executed, so splitting work into smaller grants does not relax a job’s limit; a grant returned unexecuted is refunded. With no CPU job active, no budget limits GPU work. The crossing bound. The crossing check of Section 4.2 models the response time of a job P Ti,j released at Ri,j under a grant with predicted end tend and c bytes as (tend − Ri,j ) + max(âi,j + qi,j , pi ) + κ̂i,j max(0, Āi − c): after the grant the job can receive at most Āi − c further bytes, and max(·, pi ), with pi the task’s solo p99, keeps a low intercept from shrinking its busy time. Āi bounds the allowance the job can be issued whatever completes before Ri,j : it includes the 25 % growth rule and the one-unit probe, both capped at Amax . A job’s charge can still exceed Amax by i i the grants in flight that it inherits at its release. Capping the crossing at Ti keeps a grant from reaching the task’s next release. Rank endpoints and the tight-deadline guard. If the rank exceeds the retained sample count, the margin update uses

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference

qi,j = Di to represent an unbounded margin; a saved qi,j ≥ Di records erri,j = 0. At αi,j ≥ 1 it uses the smallest residual. The updates use integer parts per million. Jobs assigned this margin follow the same rate limit and probe rules, and their deadline outcomes are scored normally. At the tightest deadlines Ripple adds one rule: if the predicted delay from GPU volume f exceeds a job’s remaining slack, that is, ⌈κ̂i,j f ⌉ > Di − max(âi,j + qi,j , pi ), evaluated with the estimates that price it, then that job gets neither the probe nor the floor f nor a crossing into its release. Task exemptions. The model-based certification check requires κui,j ρ < 1 and hi,j + 2κui,j ρH ≤ min Di , âi,j + qi,j 1 − κui,j ρ
+ βi,j [Di − âi,j − qi,j ]+ , where hi,j = âi,j + qi,j + [κ̂i,j − κui,j ]+ C̄i reanchors the intercept at its fit’s weighted mean charge C̄i . After 50 consecutive passes the task is exempt. A failed check or an observed response above the saved line hi,j + κui,j C ends exemption; a deadline miss while exempt, or a new u lower slope estimate κ̂LS i,j − 2SEi,j above the previous κi,j , disables exemption for the rest of the run.

D

S UPPLEMENTARY E VALUATION

Protocol and baselines. The Ripple grid uses 140 s preparation; the external baselines use 30 s. Both use a 100 s score. A point is feasible only if every run meets every CPU task’s 2% DMR limit and every VLM deadline; all Ripple grid runs meet these criteria. Each workload mix covers seven deadline settings. The controller uses the same parameter settings across mixes, with the SM width chosen automatically during warm-up. Table 7 reports LLM throughput and the highest pooled CPU-task DMR for every deadline and workload. For example, at k = 1.5 on (Robotic1 , LLMS , V LML ), Ripple produces 7.00 tokens/s at 0.78% DMR, while SiGAMMA produces 6.16 tokens/s at 21.50% DMR. Each 10 Hz task contributes 1000 jobs per score, and Control contributes 5000. Costmap, KDNN, and SGM use cores 15– 18, 11–14, and 7–9; Control and ORB use 15 and 16. Both engines use cores 0–6. The LLM repeatedly serves a 37token prompt with 51 output tokens. VLM periods/deadlines are 5 s for V LML and 10 s for V LMS . CPU deadlines use the co-run p99 profile associated with each repetition, with both engines resident but idle. Published baselines use the same engine flags as Ripple, including eager LLM execution. Pause and Cap act at stock forward-pass boundaries; Ripple uses finer hooks and reacts to the declared CPU deadline. The VLM due-time fraction is γ = 0.95. Baseline configuration.

For Sereno, we tune PI

gains separately for each workload and host using its Ziegler–Nichols procedure: proportional-only gain rungs are (1, 2, 4, . . . , 64) × 106 in parts-per-million units; the ultimate gain and period come from sustained oscillation or the relay fallback, and Kp = 0.45Ku , Ki = 0.54Ku /Tu . UI-First uses a contention target of 0.07; Balanced uses 0.15 and a 12-token/s guardrail. The contention score compares streamed LLM decode-step intervals with their solo median; the controller pauses only the LLM between steps, without speculative decoding. For SiGAMMA, a separate process launches a highest-priority spin kernel with 512 threads per block and enough blocks to fill the thread slots on all 48 SMs. Each CPU job waits for the blocker to become resident before executing; the blocker remains active while any CPU job is active. For Cap-x, each job’s GPU exposure budget is x times its task’s solo median duration, with x ∈ {1/8, 1/4, 1/2}. Exposure accumulates while the job is active and the engines are commanded to run; when any active job exhausts its budget, the controller pauses both engines at their next forward-pass boundary. For the CPU-load comparison, both arms use 8 decode SMs, 48 prefill SMs, 140 s preparation, and a 200 s score. Both restore the same saved initial state and update for 30 s; the control then freezes CPU adaptation state. Two 128 MiB readers on cores 4 and 10 run during 30–80 s and 130–180 s, with 500 KDNN offers per interval. Each session learns separately before the control freezes its adaptation state. The following experiments use (Robotic1 , LLMS , V LML ) at k = 1.5 and 140 s preparation unless specified otherwise. Sensitivity. Table 8 retains the three settings discussed in the body. Each changes one parameter from H = 18 ms, a 20 ms lease, and W = 50. SM selection runs during preparation, so results include its selection. Every run meets CPU and VLM targets. Mean throughput changes by at most 1.7%; the changes are relative to the current Ripple control. Transient recovery. A 3 s GPU copy reaches 187.8– 194.4 GB/s. Costmap’s allowance first returns within 10% of its pre-copy value after 21.5–23.9 s, KDNN after 8.5–19.3 s, and SGM after 14.9–15.2 s. Sustained settling is observed for Costmap in a disturbed run within the post-copy interval. Pooled CPU misses over the 100 s scores are 64/73/73 out of 2000 per task (Costmap/KDNN/SGM), and VLM misses are 4/40. The disturbed runs fail the CPU and VLM requirements. Cold start. Table 9 compares the first-entry times and initial misses for starts with and without a CPU profile. Profiled and no-CPU-prior starts disable certification and its dependent drift handling in both arms, retaining other calibration. All starts have zero CPU misses in the first 10 s (100

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference Table 7. End-to-end results by workload and deadline. Each cell gives LLM throughput (tokens/s) / CPU DMR (%). DMR is the highest task miss rate after pooling jobs across runs. Cap-x selects the highest-throughput level meeting the CPU and VLM targets, or the lowest-DMR level if none meets them. 1.1×

1.2×

1.3×

1.4×

1.5×

1.8×

2.0×

Ripple Sereno UI-First Robotic1 Sereno Balanced LLMS SiGAMMA V LML Pause Cap-x Unmanaged

4.06/0.82 7.25/85.50 9.53/100.00 6.16/99.98 8.03/98.92 8.21/99.98 9.55/100.00

4.59/0.85 7.25/80.47 9.53/100.00 6.16/99.80 8.03/98.65 8.37/99.88 9.55/100.00

5.45/0.55 7.25/76.15 9.53/100.00 6.16/98.83 8.03/91.25 8.21/99.55 9.55/100.00

6.10/0.45 7.25/71.67 9.53/100.00 6.16/62.92 8.03/86.33 8.21/93.42 9.55/100.00

7.00/0.78 7.25/66.95 9.53/99.85 6.16/21.50 8.03/84.20 8.21/84.05 9.55/99.72

8.08/0.10 7.25/26.60 9.53/63.95 6.16/0.18 8.03/28.93 8.21/19.83 9.55/48.77

8.44/0.08 7.25/4.57 9.53/24.78 6.16/0.02 8.03/1.08 8.21/1.28 9.55/17.27

Ripple Sereno UI-First Robotic1 Sereno Balanced LLML SiGAMMA V LML Pause Cap-x Unmanaged

1.68/1.02 3.27/83.50 4.23/100.00 2.71/100.00 3.83/98.85 3.98/99.78 4.21/100.00

1.91/0.78 3.27/80.75 4.23/99.88 2.71/99.50 3.83/98.10 3.98/99.62 4.21/99.95

2.27/0.58 3.27/78.28 4.23/99.67 2.71/98.05 3.83/95.73 3.98/99.30 4.21/99.70

2.52/0.40 3.27/75.75 4.23/99.48 2.71/53.68 3.83/90.92 4.01/96.30 4.21/99.52

2.76/0.35 3.27/72.50 4.23/98.08 2.71/12.35 3.83/84.70 3.98/89.10 4.21/98.10

3.34/0.20 3.27/36.02 4.23/46.65 2.71/0.12 3.83/47.75 4.01/40.30 4.21/62.00

3.56/0.25 3.27/13.15 4.23/15.30 2.71/0.00 3.83/12.60 4.01/9.50 4.21/32.34

Ripple Sereno UI-First Robotic1 Sereno Balanced LLMS SiGAMMA V LMS Pause Cap-x Unmanaged

3.33/0.88 7.08/89.85 8.72/100.00 5.50/99.98 7.21/99.90 7.31/99.90 8.76/100.00

3.79/0.85 7.08/85.45 8.72/100.00 5.50/99.75 7.21/99.90 7.31/99.78 8.76/100.00

4.64/0.52 7.08/80.90 8.72/100.00 5.50/99.12 7.21/99.80 7.31/98.78 8.76/100.00

5.22/0.58 7.08/77.38 8.72/100.00 5.50/70.55 7.21/99.70 7.31/93.58 8.76/100.00

5.81/0.50 7.08/73.12 8.72/99.78 5.50/35.92 7.21/98.65 7.31/88.38 8.76/99.72

6.92/0.15 7.08/33.27 8.72/55.45 5.50/0.08 7.21/28.30 7.37/23.65 8.76/50.40

7.38/0.02 7.08/6.90 8.72/14.92 5.50/0.02 7.21/3.93 7.41/2.22 8.76/18.55

Ripple 5.11/0.85 6.33/1.05 Sereno UI-First 8.14/86.65 8.14/79.72 Robotic2 Sereno Balanced 10.55/99.42 10.55/98.92 LLMS SiGAMMA 6.55/96.43 6.55/83.70 V LML Pause 9.06/98.58 9.06/96.23 Cap-x 9.21/99.80 8.74/98.80 Unmanaged 10.50/100.00 10.50/99.88

6.86/0.50 8.14/71.65 10.55/96.78 6.55/45.93 9.06/94.05 8.74/94.45 10.50/98.65

7.80/0.85 8.14/62.87 10.55/88.60 6.55/12.35 9.06/89.88 8.74/83.93 10.50/91.25

8.08/0.45 8.14/50.80 10.55/69.32 6.55/0.82 9.06/79.88 8.74/66.88 10.50/78.10

8.83/0.42 8.14/12.20 10.55/21.15 6.55/0.00 9.06/10.18 8.74/4.60 10.50/27.15

9.44/0.20 8.14/0.65 10.55/1.15 6.55/0.00 9.06/0.00 9.21/0.00 10.50/2.40

Mix

Scheduler

At k = 1.5 for Robotic1 / LLMS / V LML , Ripple delivers 7.00 tokens/s at 0.78% DMR; SiGAMMA delivers 6.16 tokens/s at 21.50% DMR. The same table shows how throughput and miss rates change across workloads and deadlines.

Table 8. Sensitivity: mean tokens/s per setting. Setting Default H = 12 ms Lease 26 ms W = 100

Mean tokens/s

Change

7.130 7.130 7.010 7.070

— 0.0% −1.7% −0.8%

jobs per task per run); all 100 s scores meet CPU and VLM targets. In a no-prior start, the full pre-score interval has 41/1470 KDNN and 36/1470 SGM misses, including three dropped SGM jobs, both above 2%. Profiled/no-prior mean first-entry times are 57.4/8.4 s for Costmap, 97.3/60.5 s for KDNN, and 45.6/63.6 s for SGM. One-hour serving. The experiment uses a 3600 s score after 140 s preparation, selecting 8 decode SMs. Table 10 reports the whole-hour outcomes and the worst per-task DMR over consecutive 100 s windows. All 216 task-windows meet 2%, and all 1440 VLM frames meet their deadlines.

Table 9. Cold start at k = 1.5: first entry within 10% of the final allowance and CPU deadline misses in the first 10 s, for Costmap / KDNN / SGM. Start

First entry (s)

Misses, 10 s

Profiled No profile

57.4 / 97.3 / 45.6 8.4 / 60.5 / 63.6

0 0

Both starts have zero misses in the first 10 s. In a no-profile start, the full pre-score interval has 41/1470 KDNN misses and 36/1470 SGM misses.

No throttle flag is active in the 7195 GPU samples.

Ripple: Deadline-Aware CPU–GPU Sharing for Embodied Inference Table 10. One-hour serving at k = 1.5: CPU deadline miss rates for Costmap / KDNN / SGM and worst task DMR across consecutive 100 s windows. Run

Whole-hour DMR (%)

Worst window (%)

1 2

0.281 / 0.625 / 0.708 0.253 / 0.719 / 0.572

1.1 1.1

Related documents

• Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference#1536876
• Energy-Efficient Multimodal Inference Serving with Tri-serve#984186
• Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms#1355071
• KREX: Concurrent Kernel Benchmarking on Shared GPUs via Region-Granular Exclusivity#1108864
• KREX: Concurrent Kernel Benchmarking on Shared GPUs via Region-Granular Exclusivity#1078128