GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

arXiv:2610.38196v1 [cs.DC] 28 Oct 2026

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading Qi Qi

Yifan Sui

Hanfei Yu

University of Massachusetts Amherst Amherst, Massachusetts, USA qqi@umass.edu

Hong Kong University of Science and Technology Hong Kong, China suiyifan.sjtu@gmail.com

Stevens Institute of Technology Hoboken, New Jersey, USA hyu42@stevens.edu

Hong Zhang

Hao Wang

University of Waterloo Waterloo, Ontario, Canada hongzhangblaze@gmail.com

Stevens Institute of Technology Hoboken, New Jersey, USA hwang9@stevens.edu

Abstract Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAGaware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the endto-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1× and 10.0× over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2× strong scaling on four workers.

1

Prewarm Preload Inference-time load (a) SMIless

Warm wait Inference Keep-alive

Request arrival E2E

34.97s

OD HAP IR FR TRS (b) InstaInfer 13.02s

OD HAP IR FR TRS

(c) InstaInfer Chain 17.19s

OD HAP IR FR TRS (d) GearUp

12.15s

OD HAP IR FR TRS -10

0 10 20 30 40 Time Relative to Request Arrival (s)

Figure 1. Per-node execution timelines of AMBER Alert request #317 under different serving approaches.

Introduction

Serverless computing offers fine-grained scaling and pay-asyou-go billing, and is increasingly used for Machine Learning (ML) inference, where workflows are modeled as Directed Acyclic Graphs (DAGs) of interdependent functions. Platforms such as Kubeflow [24], MLRun [25], and Airflow [1] express inference pipelines in this form. The same structure underlies video analytics [16, 18], scientific workflows [19], market surveillance [7], and agentic applications [9, 15].

While attractive, serverless DAG-based inference faces a critical bottleneck from initialization overhead. Beyond container startup, each ML node often requires expensive initialization steps, including loading libraries, model weights, and CUDA kernels. Since these costs occur across dependent DAG stages, they accumulate along execution paths and significantly increase end-to-end latency, directly degrading user experience.

Qi Qi et al.

OD

Allocated Used

FR TRS

60 30

The cost of ignoring this structure is large. Figure 1 traces one representative AMBER Alert request across four serving strategies on identical hardware, and Figure 3 reports the GPU memory each strategy holds. SMIless [13], a DAGaware serving system, starts each function only after its dependencies finish, so ML loading stays on the request path and the request takes 34.97 s, most of it spent loading rather than performing inference. InstaInfer [23] instead pre-loads libraries and weights proactively; applied to the DAG as a single unit it reaches 13.02 s, but readies downstream nodes long before their dependencies finish and holds 1046 GB·s of GPU memory largely idle. InstaInfer Chain predicts each node’s request arrival independently without DAG dependency information, causing late pre-loads and cascading stalls that leave GPUs idle while waiting for downstream initialization, giving 17.19 s at only 40% GPU utilization. GearUp completes the same request in 12.15 s using 281 GB·s. Roughly two thirds of the reactive end-to-end latency is therefore initialization that a DAG-aware system can hide, and hiding it well costs a quarter of the memory eager pre-loading spends. No existing system closes this gap. Prior work does not jointly exploit DAG dependencies and fine-grained ML loading stages for pre-loading. Single-function pre-loaders [17, 23] optimize ML loading without knowing which nodes a request will reach. DAG serving systems [3, 10, 14] schedule and size workflows without treating ML loading as a schedulable stage. Table 1 and §2 compare these systems in detail. Our key insight is that DAG dependencies expose opportunities to overlap downstream ML pre-loading with upstream execution, hiding initialization from the end-to-end critical path. However, realizing this opportunity raises three challenges. C1. Pre-loading must be timely and nonintrusive. ML nodes have different initialization costs. Starting too late delays execution, while starting too early can interfere with active inference. C2. Pre-loading is costly within a request and redundant across requests. Eager preparation can leave substantial memory idle, while initializing the same models separately for each request repeats work. C3. Traffic and topology interact. Changing traffic and upstream progress make it difficult to decide which instances to retain and for how long. To bridge this gap, we propose GearUp, a DAG-aware framework for serverless ML inference serving. Only the critical path of a DAG determines its end-to-end latency, so

0 60 GPU Memory (GB)

Figure 2. DAG of AMBER Alert [16, 18], a typical DAGbased ML inference workload for image-based emergency alerts. Each vertex denotes an ML function instance.

InstaInfer 30 0 60 InstaInfer Chain 30 0 60 GearUp 30

Request arrival E2E E2E 34.97s Util. 0.58 Alloc. 1208 GB*s

SMIless

IR

Utilization Prep start

E2E 13.02s Util. 0.62 Alloc. 1046 GB*s

E2E 17.19s Util. 0.40 Alloc. 608 GB*s

E2E 12.15s Util. 0.74 Alloc. 281 GB*s

0

1.00 0.50 0.00 1.00 0.50 0.00 1.00

Utilization

HAP

0.50 0.00 1.00 0.50 0.00

-10

0 10 20 30 Time Relative to Request Arrival (s)

Figure 3. GPU memory allocation and utilization for AMBER Alert request #317. every off-path node carries slack—time it spends waiting for its dependencies—and that slack is initialization time the request has already paid for. GearUp turns this observation into three mechanisms answering C1–C3, applied both within and across requests. Our key contributions can be summarized as follows: • Slack-guided intra-request pre-loading (C1). We traverse the DAG in reversed topological order to find the critical path and each node’s slack 𝑇𝑠𝑙𝑎𝑐𝑘 , then schedule its loading stages inside that window, hiding initialization behind upstream execution without extending the critical path. • Execution-level instance reuse across requests (C2). We decouple instance-level execution from the request-level DAG configuration, so warmed instances serve later requests that share the same model nodes. • DAG-aware traffic-adaptive keep-alive (C3). A probabilistic cost model sets node lifetimes from DAG topology and observed arrivals, retaining reused nodes through bursts and propagating release decisions to dependents. • Implementation and evaluation. We implement GearUp across the controller, workers, function handlers, and DAG orchestrator, and evaluate it on the AMBER Alert and FINRA-ML workflows using NVIDIA L40S GPUs. GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, cuts integrated GPU and host memory allocation by up to 83.9% and 94.6% against eager pre-loading, and achieves 4.2× strong scaling on four workers.

GearUp : DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

2

Background and Motivation

2.1

Serverless ML Inference and Pre-loading

A cold ML invocation must import libraries, load model weights, transfer them to GPU memory, and initialize CUDA kernels. We call these four steps the node’s loading stages, and they are the unit GearUp schedules. For short inference tasks they dominate end-to-end latency, unlike generic serverless cold starts where container startup dominates. Table 1 summarizes where existing systems fall. One group is ML-aware but DAG-agnostic: InstaInfer [23] pre-loads libraries and model artifacts for individual functions, and PISeL [17] pipelines download, deserialization, and execution within a function. Both treat functions as independent units, so neither knows whether a pre-loaded node will be reached in the current request. The other group is DAG-aware but loading-agnostic: Kraken [3], Orion [14], and iRoute [10] optimize workflow scheduling, routing, and resource sizing, serverful systems such as Airflow [1] and InferLine [4] orchestrate DAG execution, and SMIless [13] co-optimizes resource configurations for ML DAGs but still launches each node reactively. No system in either group treats the loading stages as work that can be scheduled ahead of invocation using DAG dependencies. The consequence is a timing dilemma: loading too late exposes initialization on the execution path, while loading too early holds expensive GPU memory idle. 2.2

Reactive DAG Execution Exposes ML Loading Overhead DAG-aware scheduling alone does not hide ML loading when node initialization remains reactive. A downstream node waits for its dependencies before starting its own loading stages, causing initialization delays to accumulate along the request path. Figure 1 illustrates this dilemma on one representative AMBER Alert request, executed by all four approaches under the configuration of §7.2. The DAG contains five GPU-backed functions: Object Detection (OD), Human Attribute Prediction (HAP), Image Recognition (IR), Face Recognition (FR), and Transcription (TRS), with the dependencies in Figure 2. Figure 1(a) shows SMIless, which starts each function only after its dependencies become ready. This avoids pre-loading and its resource management complexity, but leaves the loading stages on the request path. The request spends most of its time loading rather than performing inference, yielding 34.97 s. Hiding that cost requires starting downstream initialization before its predecessors finish. 2.3

DAG-Unaware Pre-loading Can Be Wasteful or Late

Pre-loading the entire DAG as one unit can prepare downstream instances too early. In the Instainfer setup, we treat

Table 1. Comparison of ML and DAG serving approaches. GearUp uniquely combines ML-aware pre-loading with DAG dependencies to proactively determine when downstream ML instances should be loaded. DAGAware

MLAware

Proactive Pre-load

DAG-Guided Pre-load

✗ ✗ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✗ ✗ ✗ ✓ ✓

✗ ✓ ✗ ✗ ✗ ✗ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✓

Serverful DAG serving Airflow [1] ✓ InferLine [4] ✓

– ✓

– –

– –

Method PISeL [17] InstaInfer [23] iRoute [10] Kraken [3] Orion [14] SMIless [13] GearUp

the whole request as one function and let InstaInfer preload every ML instance on arrival. As a result, Figure 1(b) shows that this setup reduces latency to 13.02 s. However, downstream instances become ready long before their dependencies finish, holding containers and GPU memory through long warm-wait periods (Figure 3(b)). When several requests compete for limited GPUs, this over-provisioning contends with the instances actually executing. The second setup instead predicts each node’s pre-load time independently. In the Instainfer Chain setup (InstaInfer Chain), each node is treated as an independent function, with its pre-load time predicted without dependency information. Figure 1(c) shows the opposite inefficiency: a late HAP preload delays FR, the delay cascades to TRS, and the late TRS pre-load extends the critical path to 17.19 s. 2.4

Overlapping Pre-loading with Upstream Execution

The above observations highlight the need to align preloading with DAG progress. Our key insight is that DAG dependencies expose opportunities to overlap downstream ML pre-loading with upstream execution, hiding initialization from the end-to-end critical path. Using these dependencies and execution slack to place loading stages allows downstream instances to be prepared before invocation without keeping them ready throughout upstream execution. Figure 1(d) shows GearUp, which uses DAG dependencies and execution slack to place pre-loading. Downstream instances such as FR and TRS are prepared before they become executable, but without the warm-wait periods of Instainfer pre-loading, giving 12.15 s. Figure 3 quantifies the resource cost of reaching those latencies. SMIless and Instainfer allocate 1208 and 1046 GB·s of GPU memory, and Instainfer Chain reduces allocation to 608 GB·s but reaches only 40% utilization because its pre-loads are poorly timed. GearUp

Qi Qi et al.

uses 281 GB·s at 74% utilization, the lowest allocation and the highest utilization of the four. 2.5

Design Implications

Realizing this opportunity requires coordinating loading within a request and managing prepared instances across requests. This raises the three challenges introduced in §1. C1: pre-loading must be timely and non-intrusive. Unlike lightweight CPU function nodes, ML nodes have heterogeneous initialization costs, so some nodes must start loading several stages before execution. The schedule must use DAG dependencies to find this window while avoiding interference with active inference. C2: pre-loading is costly within a request and redundant across requests. Preparing every reachable node can hold substantial resources idle or waste preparation on unused branches. The same model nodes also recur across requests, making repeated initialization unnecessary. C3. Traffic and topology interact. DAG fan-out can amplify changes in root traffic into larger changes in downstream demand. Static schedules and per-node policies can therefore retain too many instances or release them too early.

3

An Overview of GearUp

3.1

Objectives and Challenges

GearUp pursues two objectives: 1) minimize end-to-end inference latency while preserving the DAG’s dependency constraints, and 2) avoid redundant initialization so that containers and GPU memory are not held for nodes that will not run soon. Meeting them requires addressing challenges C1–C3 of §2.5: placing multi-stage pre-loading inside DAG slack, controlling its resource cost within and across requests, and coordinating node lifecycles under dependency-induced traffic. 3.2

overlaps each node’s loading stages with upstream execution. • Inter-Request Batching and Reuse Scheduler (C2, §5) batches requests by DAG type and reuses unreleased instances across requests and DAGs that share model images. • Dynamic Keep-Alive Manager (C3, §6) sets pernode retention from a topology-aware cost model over observed arrivals.

GearUp’s System Architecture

GearUp rests on one observation: only the critical path of a DAG determines its end-to-end latency, so every off-path node carries slack that the request is already spending. Initialization placed inside a node’s slack window is therefore free, while the same work placed outside it either delays the request or holds resources before they are needed. Before any of this can be scheduled, GearUp makes initialization schedulable: each container exposes its loading stages— library import, weight load into host memory, GPU transfer, and CUDA-kernel warm-up—as separately invocable, idempotent steps, and a profiler records the latency of each stage per node (§4.1). Over an inference DAG 𝐺 (𝑉 , 𝐸) annotated with these profiles, GearUp then applies three components, one per challenge: • Intra-Request DAG-Aware Pre-Loader (C1, §4) computes slack from a reversed topological traversal and

3.3

GearUp’s Workflow

Figure 4 shows how these components interact, in four phases; the numbers match the callouts in the figure. Deployment-time profiling (0). Developers deploy a DAG job with its code and model configuration. Before serving, GearUp runs pilot executions over sampled batch sizes to build the per-node stage-latency profiles used by the preloading pipeline of §4.1. Admission (1–3). The front end groups arriving requests by DAG type within a time window, amortizing scheduling cost and exposing reuse opportunities across similar workloads (§5.2). For each batch it loads the DAG metadata and walks the nodes, separating root nodes from downstream nodes. This split decides which policy applies: a root node has no upstream execution to hide behind, so its initialization must be predicted from arrival history, whereas a downstream node can be hidden inside DAG slack. Policy generation (4–6). The workload extractor and online profiler supply per-node execution times, stage latencies, and resource usage (§4.1). For a root node, a histogram over inter-arrival times decides when to pre-warm and whether a live instance can be reused (§5.1, §5.2). For a downstream node, reversed-topology analysis yields the critical path and 𝑇slack , and the profiled stage latencies determine where inside that window each stage is issued (§4.2, §4.3). Execution and feedback (7–9). Policies reach the workers as pre-load, request, reuse, and keep-alive commands, and the container scheduler enforces them over a shared GPU container pool, where an idle instance is either reused by an arriving request or released on timeout (§6.1). The retention window 𝐷 ∗ is set per node by the cost model and conditioned on DAG parent activity, so active parents lengthen it and quiet parents shorten it (§6.2, §6.3, §6.4). Runtime observations then update the stage latencies, arrival history, and parent states that these policies read, so they track changing traffic instead of staying fixed.

4

Intra-Request DAG-Aware Pre-Loader

Existing systems either pre-load all nodes indiscriminately or rely on generic execution models that ignore ML-specific initialization pipelines. GearUp instead uses DAG topology and slack to defer each node’s initialization to the latest point that still hides it.

GearUp : DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

0 Deployment & profiling

Libraries → host model → GPU → kernels

1 Batch requests

2 Load DAG

3 Identify roots

4 Profile runtime

Root nodes

Downstream nodes

5 Inter-request reuse

6 DAG-aware pre-loading

Arrival history

Root pre-warm

DAG request A

DAG request B

Critical path Slack Upstream

Shared ML instance Retained · same image

7 Policy dispatch

Execute Pre-load Execute Dependencies ready

Downstream

Container scheduler

Shared GPU container pool

8 Adaptive keep-alive Parents active

Longer

Parents quiet

Shorter

Results

Traffic + parent state Retention window D*

9 Online feedback

Arrival

Reused

Timeout

Idle

Released

Stage latencies · arrival history · parent states

Figure 4. Overview of GearUp. Deployment-time profiling (0) precedes admission (1–3), per-node policy generation for root and downstream nodes (4–6), and execution with online feedback (7–9). 4.1

Per-Node Multi-Stage Pre-loading Pipeline and Profiler

Each function container exposes a unified /handle endpoint that supports a staged, idempotent initialization pipeline: 1. preload_libs: import framework libraries (e.g., PyTorch). 2. preload_model_container: load model weights into host memory. 3. preload_model_gpu: transfer the model to GPU. 4. preload_cuda_kernels: execute a dummy forward pass to trigger kernel initialization. Idempotence lets the scheduler issue the stages one at a time and stop after any of them without leaving the container in a partial state. The scheduler orchestrates these stages sequentially via the endpoint, and stage latencies are recorded and persisted as JSON profiles for runtime scheduling. 4.2

Critical Path Identification via Reversed Topology

As shown in Figure 5, upon receiving a request the scheduler uses profiled inference durations to compute earliest start times in topological order and latest start times in reversed topological order, identifying the inference-only critical path.

Critical

Non-critical

Slack window E2E = 9.22s

T_slack(v) = LST(v) - EST(v) O/ESE MCE

Slack=2.65s

E/AGE

Slack=4.51s

N/FTE CRQAD L/SD BEAD W/STD

Slack=1.08s

I/NLD RFR ANG 0

2

4 6 Time (seconds)

8

Figure 5. EST/LST schedule and slack windows for FINRAML. Bars show per-node invocation durations.

For each node 𝑣𝑖 , we compute its slack time: 𝑇𝑠𝑙𝑎𝑐𝑘 (𝑣𝑖 ) = 𝐿𝑆𝑇 (𝑣𝑖 ) − 𝐸𝑆𝑇 (𝑣𝑖 ),

Qi Qi et al.

4.3

Cold-start term Normalized expected cost

where 𝐸𝑆𝑇 (𝑣𝑖 ) and 𝐿𝑆𝑇 (𝑣𝑖 ) denote the earliest and latest inference start times relative to request arrival under this profiled schedule. Nodes on the critical path have 𝑇𝑠𝑙𝑎𝑐𝑘 = 0, while non-critical nodes expose additional temporal flexibility for deferred initialization. Because 𝐸𝑆𝑇 (𝑣𝑖 ) + 𝑇𝑠𝑙𝑎𝑐𝑘 (𝑣𝑖 ) = 𝐿𝑆𝑇 (𝑣𝑖 ), the latest start time is the deadline that preserves the profiled critical-path latency. GearUp targets this deadline when scheduling pre-loading, exploiting both upstream execution time and any additional slack.

𝑡𝑖 = max 0, 𝐸𝑆𝑇 (𝑣𝑖 ) + 𝑇𝑠𝑙𝑎𝑐𝑘 (𝑣𝑖 ) − 𝑇pre (𝑣𝑖 ) ,
𝑇end (𝑣𝑖 ) = max 𝑇dep (𝑣𝑖 ), 𝑡𝑖 + 𝑇pre (𝑣𝑖 ) + 𝑇inf (𝑣𝑖 ). Pre-loading therefore finishes by 𝐿𝑆𝑇 (𝑣𝑖 ) whenever feasible; the clamp at zero marks nodes whose loading cannot be fully hidden, and for those the completion estimate absorbs the resulting delay. Deferring initialization to this deadline, rather than issuing it at request arrival, is what separates GearUp from eager pre-loading: a container is held only over the interval it is needed, which is what lowers integrated GPU memory allocation. The cost is that a node keeps no margin of its own, so an underestimated 𝑇pre (𝑣𝑖 ) extends the critical path; the online profiler refreshes these estimates from observed executions to keep that error small. Pre-load and inference run in separate threads, synchronized by dependency completion and pre-load readiness, and a root node’s pre-load stage is skipped when it is already warm.

5

Inter-Request Batching and Reuse Scheduler

5.1

Inter-Request Preloading via Histogram Prediction

For recurring DAGs, GearUp predicts inter-arrival times using histogram-based estimation. From historical traces, we derive an empirical distribution and compute a percentilebased estimate 𝑡ˆ. The preload trigger for the root node, measured from the preceding root request’s arrival, is:
𝑇trigger = max 0, b 𝑡 − 𝑇pre (root) . At this time, the root is preloaded in anticipation of the next invocation.

Total cost

ℒ(D) = DCkeep + CcoldP(τ > D) P = 0.95Phist + 0.05Ppred (current window)

1.0

D* = 7.5 s 0.5

0.0

Slack-Guided Pre-loading

The orchestrator implements DAG-aware pre-loading from the identified critical path and the profiled latencies. For each node 𝑣𝑖 , 𝑇pre (𝑣𝑖 ) is the sum of its four sequential pre-load stage latencies, and 𝑇inf (𝑣𝑖 ) is its inference time. Nodes are processed in topological order. For node 𝑣𝑖 , let𝑇dep (𝑣𝑖 ) denote the maximum estimated completion time of its parents, or zero for a root, and 𝑡𝑖 its pre-load start time relative to request arrival:

Keep-alive term

0

5

10 15 Keep-alive duration D (s)

20

Figure 6. Probabilistic keep-alive optimization for AMBER Alert with 𝑤 = 0.95.

5.2

Request Batching and Instance Reuse Across Requests

Request Batching. A batching time window is employed to reduce per-request overhead. Upon the first request for the same DAG, a batching window (default 500 ms) is initiated, during which subsequent requests are aggregated. In parallel, the dispatcher prepares execution by either reusing an idle container or launching a new one. After the batching window expires, the aggregated requests are dispatched in a single call, which forwards them to the container for batched execution. Instance Reuse. The framework decouples reuse policy from execution mechanism. Two nodes are interchangeable when they declare the same container image and model configuration, so an instance already initialized for one is in the state the other requires. Such nodes recur both across successive requests to the same DAG and across different DAGs that embed the same model, and in either case an idle instance is assigned instead of initializing a new one. The controller makes this decision because it maintains the pool across workers (§6.1); the scheduler’s intra-DAG timing logic is unchanged, and a reused instance simply skips the pre-load stages of §4.1. Reuse and retention are therefore coupled: an instance can only be reused while it is still being kept alive, so the retention window 𝐷 ∗ of §6 determines how much reuse is available, and each reuse hit is a cold start and a model load the request does not pay for.

6

Dynamic Keep-Alive Manager

Static timeout-based keep-alive policies are insufficient under bursty or irregular workloads and ignore DAG topology. As shown in Figure 6, we instead formulate keep-alive as a probabilistic cost optimization problem. 6.1

Container Lifecycle

Containers are launched with GPU support and mapped to dynamically assigned host ports. Workers maintain container metadata (function, port, status, timestamps) in a

GearUp : DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

thread-safe structure. Upon request, idle containers are reused when available; otherwise, new containers are created. The controller coordinates reuse across workers and enforces global limits via eviction of the oldest idle containers. 6.2

Cost-Benefit Keep-Alive Optimization

For a candidate keep-alive duration 𝐷, measured from when a container becomes idle, we use the following surrogate cost objective: L (𝐷) = 𝐷 · 𝐶𝑘𝑒𝑒𝑝 + 𝐶𝑐𝑜𝑙𝑑 · P(𝜏 > 𝐷) , | {z } | {z } Keep-Alive Cost

Expected Cold-Start Cost

where 𝐶𝑘𝑒𝑒𝑝 is the unit cost per second of keeping an idle instance alive, 𝐶𝑐𝑜𝑙𝑑 is the cold-start penalty expressed in the same cost units, and 𝜏 is the waiting time from the start of the idle period to the next invocation. The retention term charges the full timeout 𝐷, approximating the idle-memory cost. We estimate P(𝜏 > 𝐷) using a histogram and a shortterm predictor. The optimal keep-alive duration under this objective is:
𝐷 ∗ = arg min 𝐷 · 𝐶𝑘𝑒𝑒𝑝 + 𝐶𝑐𝑜𝑙𝑑 · P(𝜏 > 𝐷) . (1) 𝐷 ≥0

Extending 𝐷 raises the retention term but lowers the expected cold-start term, since P(𝜏 > 𝐷) is non-increasing in 𝐷. 𝐷 ∗ is the point at which the two balance. 6.3

Idle-Wait Probability Estimation

To estimate P(𝜏 > 𝐷), we combine historical observations and short-term prediction: P(𝜏 > 𝐷) = 𝑤 · 𝑃ℎ𝑖𝑠𝑡 (𝜏 > 𝐷) + (1 − 𝑤) · 𝑃𝑝𝑟𝑒𝑑 (𝜏 > 𝐷). The weight 𝑤 adapts dynamically: • 𝑤 → 1 under stable traffic, • 𝑤 → 0 when sudden workload shifts are detected. Histogram-Based Component. 𝑃ℎ𝑖𝑠𝑡 is derived from recent observations of the waiting time from idle onset to the next invocation. Instead of relying on a single average, we maintain bucketed frequency counts and compute: Í 𝜏 𝑗 >𝐷 freq(𝜏 𝑗 ) 𝑃ℎ𝑖𝑠𝑡 (𝜏 > 𝐷) = Í . 𝜏 𝑗 freq(𝜏 𝑗 ) As an additional burst-handling heuristic, if dense arrivals are detected within a short interval, the system enforces a minimum non-zero 𝐷 to bridge burst gaps. Predictive Component. 𝑃𝑝𝑟𝑒𝑑 reflects short-term forecasting. It can be modeled as a step function where the probability is 1 if the predicted waiting time exceeds 𝐷, and 0 otherwise. This hybrid mechanism combines historical stability with short-term responsiveness.

6.4

DAG-Aware Adjustment

Unlike DAG-unaware approaches, we condition keep-alive decisions on DAG structure. The waiting time until node 𝑣𝑖 receives its next invocation depends on the state of its parent nodes. We refine the probability as: 𝑃 ′ (𝜏𝑖 > 𝐷) = 𝑃 (𝜏𝑖 > 𝐷 | S𝑝𝑎𝑟𝑒𝑛𝑡 ), where S𝑝𝑎𝑟𝑒𝑛𝑡 represents the runtime state of predecessor nodes. For each node 𝑣𝑖 , we substitute this conditional probability into Eq. (1) to compute its keep-alive duration 𝐷𝑖∗ . Parent activity updates the predicted distribution of the next invocation, and the cost model recomputes 𝐷𝑖∗ accordingly to balance retention cost against cold-start risk. Through this DAG-aware probabilistic control, the system adapts container retention to both intra-DAG execution signals and inter-request traffic patterns.

7

Implementation and Evaluation

7.1

Implementation

We implement GearUp as a layer that can be integrated into serverless inference serving platforms, spanning the controller, workers, function handlers, and DAG orchestrator. The system runs on GPU-enabled nodes, where each ML function is packaged as a Docker container exposing a unified HTTP interface. GearUp follows a centralized-controller, distributed-worker design. The controller is the entry point, exposing a REST API through Flask, holding global container state in a threadsafe map, and enforcing a configurable upper bound on GPU memory. DAG specifications are given as JSON adjacency lists, loaded at startup and broadcast to all workers. Each worker manages container lifecycles on its node through the Docker SDK, probing readiness over HTTP with exponential backoff and reporting status to the controller. A worker also runs the keep-alive optimizer of §6, with a background thread reclaiming containers past their duration 𝐷 ∗ . Each function is profiled offline over sampled batch sizes to measure its four pre-load stages and a representative inference latency; profiles, DAG structures, and system parameters are externalized as JSON and YAML. 7.2

Experiment Setup

Testbed. Our single-worker testbed uses one AWS EC2 g6e.4xlarge instance with 16 vCPUs, 128 GiB of host memory, and one NVIDIA L40S GPU with 48 GB of GPU memory. The instance hosts the client, controller, worker, and ML function containers. The worker maintains a shared container pool and creates, reuses, or releases containers according to controller decisions. The scalability experiments use two identical instances, placing one, two, or four workers across them. One instance also hosts the controller, which keeps a global view of worker

Qi Qi et al.

BEAD

ANG

I/NLD

0.50 0.00

CRQAD

P50 34.9s P50 12.4s

FaceNet / ImageNet ResNet50 / ImageNet T5 / SQuAD

Transformer Encoder / CAT Order Events Temporal Fusion Transformer / SIP and TAQ Market Data E/AGE Entity/Account Graph R-GCN / CAT Entity-Account Encoder Graph N/FTE News/Filing Text Encoder FinBERT / SEC EDGAR and Financial News L/SD Layering/Spoofing XGBoost / Labeled CAT Detector Surveillance Cases BEAD Best-Execution Anomaly LightGBM / CAT and NBBO Detector Execution Records W/STD Wash/Self-Trade Detector GraphSAGE / CAT Trade-Account Graph I/NLD Inside/News Leakage FinBERT + Transformer / CAT Detector Trades and News Events CRQAD CAT Reporting-Quality Isolation Forest / CAT Anomaly Detector Reporting Audit Logs RFR Risk-Fusion Ranker LambdaMART / Historical Surveillance Alerts ANG Alert-Narrative Generator T5 / Historical Alert Narratives

resources and assigns DAG nodes while preserving dependencies. Workloads. We evaluate GearUp on the AMBER Alert and FINRA-ML workflows. AMBER Alert is the five-function DAG of Figure 2, where OD, HAP, IR, and FR use a batch size of 15 and TRS uses 512. FINRA-ML is a market-surveillance workflow of eleven functions: four encoders feed five detectors, whose outputs a risk-fusion ranker aggregates before an alert-narrative

0.75

P50 2.4x

P50 3.1x

0.50 0.25

P50 1.1x

P50 3.1x

0.00

0

1.00 P50 21.0s P50 15.2s Cold 92.6s P50 9.2sP50 38.2s

0.75 0.50 0.25 0.00

50 100 E2E latency (s) (c) FINRA-ML — Latency

Model / Dataset YOLOv5 / COCO ResNet50 / ImageNet

1.00

1.00 CDF

Table 2. DAG nodes, functions, models, and datasets for the AMBER Alert and FINRA-ML workloads.

Cold-start baseline

50 100 1 2 4 8 16 32 E2E latency (s) Speedup over cold start (a) AMBER Alert — Latency (b) AMBER Alert — Speedup

Figure 7. DAG of FINRA [7, 10], a typical DAG-based ML inference workload for market surveillance. Each vertex denotes an ML function instance.

AMBER Alert OD Object Detection HAP Human Attribute Prediction FR Face Recognition IR Image Recognition TRS Transcription FINRA-ML O/ESE Order/Event Sequence Encoder MCE Market-Context Encoder

Cold 38.7s P50 15.8s P50 12.7s

0.75 0.25

N/FTE

Function

GearUp

0

CDF

RFR

CDF

W/STD

Abbr.

InstaInfer Chain

InstaInfer 1.00

MCE E/AGE

SMIless

CDF

L/SD O/ESE

0.75 0.50 0.25

P50 4.4x P50 6.1x P50 2.4x

P50 10.0x

0.00

1 2 4 8 16 32 Speedup over cold start (d) FINRA-ML — Speedup

Figure 8. End-to-end latency and speedup distributions for AMBER Alert and FINRA-ML. Dotted lines mark the coldexecution references (latency and 1× speedup).

generator. Figure 7 gives its structure and Table 2 the models and datasets for both workflows. Request trace. We sample a continuous 300-minute segment from the Azure Functions Trace 2019 dataset [21]. The segment contains 319 root requests, with an average rate of 1.06 requests per minute. The ablation and scalability experiments each use a 160-request subset of this trace. In Figure 10, we aggregate requests into 120-second windows. The maximum window contains eight root requests. We define a burst as a non-empty window containing at least three root requests, which corresponds to the 90th percentile of non-empty windows. The median inter-arrival time is 54.5 s, and the coefficient of variation is 0.516. The trace contains both busy and idle intervals, allowing us to evaluate instance reuse and adaptive keep-alive decisions under changing request pressure. Baseline systems. We compare GearUp with three baselines. SMIless follows DAG dependencies but initializes downstream functions reactively and does not preload their ML components. InstaInfer eagerly prewarms and preloads all reachable ML functions without considering DAG execution timing. InstaInfer Chain applies function-local pre-loading along the invocation chain, with each function predicting its preload time independently. The cold-execution reference 𝐿cold defined above is 38.7 s for AMBER Alert and 92.6 s for FINRA-ML. 7.3

Evaluation Metrics

We use end-to-end latency, speedup over cold execution, resource utilization, and integrated resource allocation as the main evaluation metrics.

GearUp : DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

End-to-end latency. End-to-end latency is measured from the submission of a DAG request until its sink node returns the final result. It includes container startup, library and model loading, GPU preparation, inference, communication, and dependency-induced waiting. Speedup over cold execution. For request 𝑖, speedup is defined as Speedup𝑖 = 𝐿𝐿cold , where 𝐿cold is the cold-execution 𝑖 latency of the corresponding workload and 𝐿𝑖 is the endto-end latency of request 𝑖. 𝐿cold is a common unoptimized reference shared by all methods within a workload, obtained from a cold DAG traversal in which every node incurs container prewarming, model loading, and inference with no pre-loading, reuse, or keep-alive. Dependencies and parallel branches are preserved, so 𝐿cold remains set by the critical path. A value greater than one indicates an improvement over cold execution. Resource utilization. For resource 𝑟 , request-window utilization is defined as ∫ 𝑅used,𝑟 (𝑡) 𝑑𝑡 , 𝑈𝑟 = ∫ W 𝑅 (𝑡) 𝑑𝑡 W alloc,𝑟 where W denotes request execution windows, 𝑅used,𝑟 (𝑡) is the utilized resource, and 𝑅alloc,𝑟 (𝑡) is the allocated resource. We report GPU memory and host memory utilization. We also report integrated allocation, ∫ 𝐴𝑟 = 𝑅alloc,𝑟 (𝑡) 𝑑𝑡,

Memory efficiency. Figure 9 shows GPU and host memory allocation and utilization. For AMBER Alert, GearUp cuts integrated GPU memory allocation from InstaInfer’s 304k to 56k GB s (81.6%) and host memory from 244k to 41k GB s (83.2%). GearUp retains the same average GPU utilization as InstaInfer, 0.64, while keeping resources allocated for a much shorter period. For FINRA-ML, the reductions are 527k to 85k GB s (83.9%) and 1,287k to 69k GB s (94.6%). These results show that GearUp reduces latency without keeping all DAG stages initialized throughout the workload. Traffic adaptation and instance reuse. Figure 10 shows the keep-alive behavior of GearUp under the FINRA-ML workload over the 300-minute trace. Although the number of root requests varies within a relatively narrow range outside the initial burst, each request fans out across the elevennode DAG. This amplifies changes in root arrivals into larger variations in function invocation pressure. The number of live containers changes with the resulting invocation demand, while a small idle-warm pool is retained across nearby windows. Reuse hits occur mainly during higher-load intervals, showing that recently initialized instances are reused when subsequent invocations arrive. During lower-load intervals, the live-container population decreases and few containers remain idle-warm. These results show that GearUp adjusts container retention according to the invocation pressure produced by both external arrivals and DAG fan-out.

in GB s. This metric captures both the amount of allocated 7.5 Ablation Study memory and how long the allocation is retained. Memory plots summarize average utilization and integrated allocaWe evaluate four GearUp variants on the AMBER Alert worktion, shown as Util./Alloc. annotations or utilization/allocation load. These runs use the 160-request subset of the trace, so pairs in row headings. their absolute allocations are lower than the full-trace figures reported above and should be compared only with each 7.4 Effectiveness of GearUp other. -Intra disables intra-request pre-loading, -Slack disEnd-to-end latency and speedup. Figure 8 compares the ables DAG slack-guided scheduling, -Reuse disables instance latency and speedup distributions of the four systems. For reuse across requests, and -KeepAlive disables adaptive keepAMBER Alert, SMIless has a median latency of 34.9 s because alive decisions. downstream functions still incur initialization overhead. EaLatency impact. Figure 11 shows the latency and speedup ger pre-loading reduces the median to 12.7 s with InstaInfer, distributions. Removing intra-request pre-loading causes the while function-local pre-loading achieves 15.8 s with InstaInlargest degradation, shifting latency close to cold execution fer Chain. GearUp achieves the lowest median latency of and confirming that overlapping downstream initialization 12.4 s, a 64.5% reduction relative to SMIless. Based on the with upstream execution provides the main latency bene38.7 s cold-execution baseline, the corresponding median fit. Removing instance reuse also increases latency because speedups are 1.1×, 3.1×, 2.4×, and 3.1×, respectively. recurring requests must initialize new instances. Removing The benefit is larger for FINRA-ML, whose deeper and slack-guided scheduling produces a wider latency distribuwider DAG provides more opportunities for coordinated pretion, showing that DAG timing is needed to keep pre-loading loading and reuse. SMIless, InstaInfer, and InstaInfer Chain work away from the active execution path. Disabling adaphave median latencies of 38.2 s, 15.2 s, and 21.0 s, respectively. tive keep-alive has a smaller latency effect for this trace beGearUp reduces the median to 9.2 s, which is 75.9% lower cause many reusable instances are selected directly through than SMIless and 39.5% lower than InstaInfer. Based on the inter-request reuse. 92.6 s cold-execution baseline, its median speedup reaches Resource impact. Figure 12 reports GPU and host memory 10.0×, compared with 2.4× for SMIless, 6.1× for InstaInfer, usage. Full GearUp uses 47k GB s of GPU memory allocation and 4.4× for InstaInfer Chain. and 21k GB s of host memory allocation. Without instance

Qi Qi et al.

Util. 0.39 Alloc. 119k

25 0 50

Util. 0.64 Alloc. 56k

25 0

0 50

0 1

Util. 0.70 Alloc. 53k 0.5

25 0 50

0 1 Util. 0.39 Alloc. 41k 0.5

25

0 6k 12k Cumulative active time (s)

0

0 0 6k 12k Cumulative active time (s)

0 50

Util. 0.91 Alloc. 527k

25 0 50

Util. 0.28 Alloc. 239k

25 0 50

Util. 0.38 Alloc. 85k

25 0

250 0 500

0 1 Util. 0.24 Alloc. 1287k 0.5

250 0 500

0 1

Util. 0.40 Alloc. 152k 0.5

250 0 500

0 1 Util. 0.24 Alloc. 69k 0.5

250

0 6.5k 13k Cumulative active time (s)

(a) AMBER Alert

1 Util. 0.23 Alloc. 496k 0.5

0

0 0 6.5k 13k Cumulative active time (s)

Utilization

25

500

Utilization

25

Util. 0.39 Alloc. 577k

Utilization

0 50

0 1 Util. 0.38 Alloc. 244k 0.5

50

Utilization

25

0 50

Utilization RAM (GB) RAM (GB) RAM (GB) RAM (GB)

Util. 0.64 Alloc. 304k

25

GPU (GB)

0 50

1 Util. 0.37 Alloc. 225k 0.5

GPU (GB)

25

50

GPU (GB)

Util. 0.60 Alloc. 301k

Utilized Res.

GPU (GB)

GPU (GB) GPU (GB) GPU (GB) GPU (GB)

50

RAM (GB) RAM (GB) RAM (GB) RAM (GB)

SMIless InstaInfer InstaInfer Chain GearUp

Allocated Res.

(b) FINRA-ML

Figure 9. GPU and host memory usage for (a) AMBER Alert and (b) FINRA-ML, excluding inter-request idle intervals. Live Running

Cold GearUp

Reuse Hit

5

50

0

100

0.5

50

5

0

50

100 150 200 Elapsed Time (min)

250

300

0.0 Reuse Hit Rate (%)

Containers

-Reuse -KeepAlive

0

10

0

-Intra -Slack

1.0

CDF

Func Invoc. Idle Warm

Func Invoc. (/120s)

Root Req. (/120s)

Root Req. Burst

0

Figure 10. Traffic-adaptive keep-alive behavior for FINRAML: request and invocation counts (top), and container counts and instance-reuse hit rate (bottom).

reuse, these values increase to 100k and 44k GB s, respectively, making reuse the largest contributor to resource savings. Disabling intra-request pre-loading increases GPU and host memory allocation to 64k and 29k GB s and reduces average GPU utilization from 0.39 to 0.29. Removing slack-guided scheduling results in 46k GB s of GPU allocation and 21k GB s of host memory allocation. Its main effect is therefore on latency rather than total allocation. Disabling adaptive keep-alive increases the allocations only slightly, to 48k and 23k GB s, under this workload. Together, the results show that intra-request pre-loading primarily reduces latency, while instance reuse provides the largest reduction in resource allocation.

10

20 30 E2E Latency (s)

40

(a) End-to-end latency CDF

1

2 3 Speedup (b) Speedup CDF

Figure 11. AMBER Alert ablation results: (a) end-to-end latency CDF and (b) speedup CDF.

7.6

Scalability

We evaluate strong and weak scaling with one, two, and four workers. Strong scaling keeps the total workload fixed as workers are added. Weak scaling increases the workload in proportion to the worker count. Strong scaling. As shown in Figure 13, going from one to four workers raises throughput from 0.07 to 0.27 requests per second and cuts completion time from about 1,250 s to 300 s, a 4.2× speedup at a scaling efficiency slightly above one. Two workers give the lowest per-request latency, with mean latency falling from 15.7 s to 12.4 s and P95 from 30.7 s to 12.8 s; at four workers both rise to 14.3 s and 20.5 s as coordination increases, still below the one-worker results. Weak scaling. Under weak scaling, throughput rises from 0.08 to 0.29 requests per second while mean latency stays between 12.1 s and 13.5 s and P95 grows only from 12.2 s to 14.0 s. Completion time grows from about 970 s to 1,100 s for a four-fold workload, a scaling efficiency near 0.9, so

GearUp : DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

0

0

0

25

0.5

0

0

25

RAM (GB)

GPU (GB)

(d) GearUp-Slack · GPU 0.33/46k · RAM 0.40/21k 50 50

0

1

25

0.5

0

0

RAM (GB)

GPU (GB)

0

25

0.5

0

0

25 0

RAM (GB)

GPU (GB)

(f) GearUp-KeepAlive · GPU 0.39/48k · RAM 0.36/23k 50 50 1

0 2.5k 5k Cumulative active time (s)

25 0

0.5

0 0 2.5k 5k Cumulative active time (s)

Figure 12. GPU and host memory usage in the AMBER Alert ablation study, excluding inter-request idle intervals.

GearUp holds per-request performance steady as the workload grows.

8

Related Work

Serverful pipeline serving. InferLine [4] profiles individual models and jointly tunes provisioning across pipeline stages, while Clipper [5], TFX [2], Airflow [1], Kubeflow [24], and MLRun [25] orchestrate ML DAGs in long-running clusters. Because their services stay active, cold-start cost is largely irrelevant and fine-grained initialization control is unnecessary. Serverless DAG execution. Kraken [3], Xanadu [6], DayDream [19], Owl [26], and Orion [14] schedule and provision serverless workflows, but treat functions as generic tasks. They do not model the large model artifacts, GPU residency, and multi-stage initialization that dominate ML inference startup.

20

1

0.3 0.2 0.1 1

(e) GearUp-Reuse · GPU 0.33/100k · RAM 0.40/44k 50 50 1 25

13

1

2 4 Worker Count (b) P95 E2E Latency

1.0 0.5 0.0

2 4 Worker Count (c) Throughput

1

2 4 Worker Count (d) Scaling Efficiency

4 Speedup vs 1 Worker

RAM (GB)

GPU (GB)

25

1

Throughput (req/s)

0.5

(c) GearUp-Intra · GPU 0.29/64k · RAM 0.39/29k 50 50

14

2 4 Worker Count (a) Mean/Median E2E Latency

Workload Wall Time (s)

0

25

Ideal

30 15

12

1

Utilization

25

RAM (GB)

GPU (GB)

(b) GearUp · GPU 0.39/47k · RAM 0.39/21k 50 50

Median P95 E2E Latency (s)

0

Weak

Scaling Efficiency

0

E2E Latency (s)

0.5

Utilization

25

Utilization

0

1

Utilization

RAM (GB)

GPU (GB)

(a) Cold Start · GPU 0.36/255k · RAM 0.38/114k 50 50 25

Strong

Utilization

Utilization

Utilized Res.

Utilization

Allocated Res.

1000

500 1

2 4 Worker Count (e) Workload Completion Time

2

0

1

2 4 Worker Count (f) Scaling Speedup

Figure 13. Strong and weak scaling with one, two, and four workers. Dotted lines indicate ideal scaling.

Cold-start mitigation. IceBreaker [20] keeps containers warm from workload prediction and Pagurus [11] shares containers across functions, but both treat a function as a black box with opaque startup cost. InstaInfer [23] and PISeL [17] target ML initialization directly, pre-loading libraries and weights or pipelining loading with inference; both assume a flat execution structure and cannot coordinate decisions across dependent stages. SMIless [13] serves DAG-based ML inference but schedules and sizes reactively, without topology-aware pre-loading. Serverless LLM and agentic serving. ServerlessLLM [8], HydraServe [12], and ServerlessLoRA [22] reduce cold-start cost for LLM endpoints through checkpoint loading, model partitioning, and backbone sharing. These optimize an individual model’s lifecycle, whereas GearUp targets the workflow layer above it. Agent frameworks compose model calls, retrievers, and tools into graph-structured executions [9, 15] whose latency-critical segments are bounded ML DAGs, and GearUp coordinates pre-loading and retention across those dependent functions rather than replacing model-level optimizations.

Qi Qi et al.

9

Conclusion

We presented GearUp, a DAG-aware serverless inference system that coordinates initialization, execution, reuse, and retention across dependent ML stages. GearUp derives each node’s slack from the DAG’s critical path and defers its loading stages to the latest point that still hides them, reuses initialized instances across requests that share a model, and sets container lifetimes from a cost model conditioned on parent activity. On the AMBER Alert and FINRA-ML workflows, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless and cuts integrated GPU and host memory allocation by up to 83.9% and 94.6% against eager pre-loading, while sustaining a 4.2× strong-scaling speedup on four workers.

References [1] Apache Software Foundation. 2024. Apache Airflow. Apache Software Foundation. https://airflow.apache.org/ A platform to programmatically author, schedule, and monitor workflows. [2] Denis Baylor, Eric Breck, Heng-Tze Cheng, Noah Fiedel, Chuan Yu Foo, Zakaria Haque, Salem Haykal, Mustafa Ispir, Vihan Jain, Levent Koc, et al. 2017. Tfx: A tensorflow-based production-scale machine learning platform. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 1387–1395. [3] Vivek M Bhasi, Jashwant Raj Gunasekaran, Prashanth Thinakaran, Cyan Subhra Mishra, Mahmut Taylan Kandemir, and Chita Das. 2021. Kraken: Adaptive container provisioning for deploying dynamic dags in serverless platforms. In Proceedings of the ACM Symposium on Cloud Computing. 153–167. [4] Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph Gonzalez, and Alexey Tumanov. 2020. Inferline: latencyaware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing. 477–491. [5] Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A {Low-Latency} online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 613–627. [6] Nilanjan Daw, Umesh Bellur, and Purushottam Kulkarni. 2020. Xanadu: Mitigating cascading cold starts in serverless function chain deployments. In Proceedings of the 21st International Middleware Conference. 356–370. [7] Financial Industry Regulatory Authority. [n. d.]. FINRA: Financial Industry Regulatory Authority. https://www.finra.org/. Accessed: 2026-08-11. [8] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: LowLatency Serverless Inference for Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 135–153. https://www.usenix.org/conference/osdi24/presentation/fu [9] LangChain. 2026. LangGraph: Agent Orchestration Framework for Building Stateful Agents. https://docs.langchain.com/oss/python/ langgraph/overview. Accessed: 2026-06-07. [10] Yiming Li, Laiping Zhao, Zhiyuan Su, Guowei Liu, Wenhao Huang, Kang Chen, Zhaolin Duan, Jingjie Zong, Wenxin Li, Deze Zeng, et al. 2026. iRoute: Local Routing Table-based Workflow Management in Serverless Computing. In Proceedings of the 21st European Conference on Computer Systems. 144–159. [11] Zijun Li, Quan Chen, and Minyi Guo. 2021. Pagurus: Eliminating cold startup in serverless computing with inter-action container sharing. arXiv preprint arXiv:2108.11240 (2021).

[12] Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. 2026. HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 415–430. https://www.usenix. org/conference/nsdi26/presentation/lou [13] Chengzhi Lu, Huanle Xu, Yudan Li, Wenyan Chen, Kejiang Ye, and Chengzhong Xu. 2024. SMIless: Serving DAG-based Inference with Dynamic Invocations under Serverless Computing. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–17. doi:10.1109/SC41406.2024.00044 [14] Ashraf Mahgoub, Edgardo Barsallo Yi, Karthick Shankar, Sameh Elnikety, Somali Chaterji, and Saurabh Bagchi. 2022. {ORION} and the three rights: Sizing, bundling, and prewarming for serverless {DAGs}. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 303–320. [15] OpenAI. 2026. OpenAI Agents SDK. https://openai.github.io/openaiagents-python/. Accessed: 2026-06-07. [16] Alex Poms, Will Crichton, Pat Hanrahan, and Kayvon Fatahalian. 2018. Scanner: Efficient video analysis at scale. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13. [17] Masoud Rahimi Jafari, Jianchang Su, Yifan Zhang, Oliver Wang, and Wei Zhang. 2024. PISeL: Pipelining DNN Inference for Serverless Computing. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 1951–1960. [18] Francisco Romero, Mark Zhao, Neeraja J Yadwadkar, and Christos Kozyrakis. 2021. Llama: A heterogeneous & serverless framework for auto-tuning video analytics pipelines. In Proceedings of the ACM symposium on cloud computing. 1–17. [19] Rohan Basu Roy, Tirthak Patel, and Devesh Tiwari. 2022. Daydream: Executing dynamic scientific workflows on serverless platforms with hot starts. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–18. [20] Rohan Basu Roy, Tirthak Patel, and Devesh Tiwari. 2022. Icebreaker: Warming serverless functions better with heterogeneity. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 753–767. [21] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2020 USENIX annual technical conference (USENIX ATC 20). 205–218. [22] Yifan Sui, Hao Wang, Hanfei Yu, Yitao Hu, Jianxun Li, and Hao Wang. 2025. ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs. doi:10.48550/arXiv.2505.14468 arXiv:2505.14468 [cs.LG] [23] Yifan Sui, Hanfei Yu, Yitao Hu, Jianxun Li, and Hao Wang. 2024. PreWarming is Not Enough: Accelerating Serverless Inference With Opportunistic Pre-Loading. In Proceedings of the 2024 ACM Symposium on Cloud Computing. 178–195. [24] The Kubeflow Community. 2024. Kubeflow: The Machine Learning Toolkit for Kubernetes. Cloud Native Computing Foundation. https: //www.kubeflow.org/ A composable, portable, and scalable opensource platform for machine learning workflows on Kubernetes. [25] The MLRun Community. 2026. MLRun: Open Source MLOps Orchestration. https://www.mlrun.org/ An open-source MLOps framework for managing and automating the machine learning lifecycle from experimentation to production. [26] Huangshi Tian, Suyi Li, Ao Wang, Wei Wang, Tianlong Wu, and Haoran Yang. 2022. Owl: Performance-aware scheduling for resourceefficient function-as-a-service cloud. In Proceedings of the 13th Symposium on Cloud Computing. 78–93.

Related documents

• GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading#1488988
• LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices#1061659
• A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs#10327
• Poseidon: DAG-Guided Parallelism Search for LLM Pre-Training on Heterogeneous Clusters#973693
• DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures#792673