Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads at LinkedIn

F IREPOOL : B ROKERING WARM GPU C APACITY FOR AGENTIC AND RL W ORKLOADS AT L INKED I N

arXiv:2611.10832v1 [cs.DC] 7 Nov 2026

Ankit Goyal 1 Santosh Jha 1 Zhuo Zhi 1 Youmin Han 1 Cong Gu 1 Chen Xie 1 Animesh Singh 1

A BSTRACT AI agents and reinforcement-learning (RL) trainers run thousands of short, isolated GPU jobs: tool calls, verifiers, and rollouts. A job often finishes in seconds, but starting a GPU pod for it takes 20–35 s even when the container image is already on the node, and minutes when it is not. A warm pod removes the wait only if it can serve the next job, and keeping pods warm must neither waste GPUs nor leak state between callers. Firepool, the sandbox and execution broker behind agentic and RL workloads at LinkedIn, meets both requirements. It makes pods interchangeable by building them from a shared base image per hardware profile and binding each job’s code, extra dependencies, and cluster topology when the job starts. It keeps them warm cheaply and safely by reusing a pod only for the caller that used it last, staging zero-VRAM standby pods in fractional GPU shares alongside active pods, and capping the GPUs that warm pools may hold via queue-driven budgets. Over a sampled peak period in production, Firepool serves 94.7% of 3,206 pod requests from warm or standby capacity. In paired benchmarks with fixed-duration jobs on production H200 nodes, agent and RL loops finish 13× and 10.8× faster than with a fresh pod per job; against a fresh reuse tag per job, warm reuse cuts p90 start latency from 59.1 s to 3.5 s. When a job cannot reuse the previous pod, standby promotion shortens the handover wait from 51 s to 0.94 s. In local and simulated runs, a keep-alive matched to the loop’s pauses keeps the latency benefit while holding about half the capacity of an always-on pool. 1

I NTRODUCTION

A coding agent edits a file, runs a test in a sandbox, reads the result, and tries again. An RL trainer fans out rollouts, scores them, and updates the policy. When a program replaces a person in this loop, steps arrive seconds apart instead of minutes, in bursts, around the clock. On a GPU each step often runs for seconds, but starting a pod for it takes much longer: 20–35 s on our H200 nodes with the image cached, and 1.3–7.5 minutes on average across our fleet when it is not. Over thousands of short jobs, these waits dominate the loop’s running time (Figure 1). Common fixes each cover part of the problem. Caching images on nodes removes the network pull but not the 20–35 s of scheduling, device binding, and platform initialization, and it does not scale when every experiment brings a new image. Persistent workers, such as a Ray cluster or a static rollout pool held for a whole run, remove the wait between steps but hold GPUs idle through every think-time pause and off-peak hour. Batch schedulers such as Kueue and Volcano manage queueing and quotas but do not keep reusable warm pods across independent jobs. 1

LinkedIn. Correspondence to: Ankit Goyal <angoyal@linkedin.com>. Preprint.

Firepool is the sandbox broker behind agentic and RL workloads at LinkedIn. Its goal is that no job waits for a pod to be created once capacity is free. We contribute: • Interchangeable pods through late binding (§4). Pods are built from a shared base image per hardware pool (SKU); each job’s code, incremental dependencies, and multi-pod topology are bound when it starts, so many distinct experiments share a few warm pools without fragmenting idle capacity. • Warmth that is cheap and safe (§5–§7). Firepool reuses a pod only for the caller that used it last and retires it after the first job of a new session; a zero-VRAM standby pod waits on a busy GPU in a fractional scheduling share to take over between jobs; and per-accelerator budgets with queue-driven scaling cap how many GPUs warm pools may hold. We also describe the alternatives we rejected. • Evaluation (§9). On production H200 nodes with fixedduration jobs, agent and RL loops finish 10.8–13× faster than with a fresh pod per job and standby hands a GPU across a reuse boundary in under a second; locally and in simulation, a keep-alive matched to the loop’s pauses holds about half the capacity of an always-on pool. Fault and scale tests cover the control plane. • Production experience (§8). Across our production clusters, Firepool served 94.7% of pod requests in a sampled

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads Fresh reuse tag per job (Reuse Off) done 185 s

agent 2 agent 1

Table 1. Anatomy of a cold start on a production H200 node with the 12.2 GB image cached on the node (88 pods). Phase percentiles do not sum to the total. Phase (pod creation → first instruction) Median p90

Firepool warm reuse (Reuse On) done 53 s

agent 2 agent 1 0

25

50

75

100

125

150

175

200

time since the repeat started (s) waiting: warm pod waiting: recycled pod

waiting: standby pod waiting: pod created

Scheduling and GPU device binding Platform init containers of which credential fetch Mounts, image check (around init) In-pod agent ready, job started

3.8 s 14.0 s 5.0 s 2.0 s 2.0 s

12.8 s 17.0 s 8.5 s 2.0 s 3.1 s

Total, image cached Create→Ready, 8 at once (1 pod: 19.8 s)

22.1 s 47.5 s

34.7 s –

Fleet mean with image pull, 1-GPU pods Fleet mean with image pull, 8-GPU pods

1.3–3.4 min 1.7–7.5 min

running

Figure 1. One repeat of a two-agent tool loop on a production H200 node under a two-GPU cap. Bars show each job’s wait, colored by how its pod was obtained, then its run. Top: a fresh reuse tag per job retires every pod, and jobs that wait for a new pod hold up the agent’s next steps; the repeat takes 185 s. Bottom: Firepool reuses each agent’s warm pod and the repeat finishes in 53 s. The medians over 10 paired repeats: 186.4 s and 51.8 s (§9.1).

peak period from warm or standby capacity, and under bursts jobs waited for quota rather than for pods. App. D gives operating guidance.

2

AGENTIC AND RL L OOPS AND T HEIR S ANDBOXES

2.1

The Workload

duction pod in our clusters runs, including a 5 s credential fetch; a framework’s own CUDA initialization comes on top. When eight pods start together on one node, we observed node-level init contention more than double create-to-Ready time, from about 20 s to nearly 50 s, with the init chain stalling after the credential fetch. Without the image, the network pull dominates: over a month on our production clusters, we observed average pod-creation times of 1.3–3.4 minutes for 1-GPU shapes and 1.7–7.5 minutes for 8-GPU shapes, with p90 up to 10 minutes. Faster image distribution (Wang et al., 2021), microVMs (Agache et al., 2020), checkpoint loaders (Fu et al., 2024; Zhang et al., 2025a) and GPU checkpoint/restore (NVIDIA Corporation, 2024a; Wei et al., 2025) each shorten one step. Reusing an initialized pod removes all of these steps from the critical path. Caching every custom image on nodes does not scale: each team, framework and pin set yields a distinct 10–30 GB image, and a node’s disk holds only a handful.

We target two loop shapes. An RL loop fans out parallel rollout jobs, then evaluation jobs, then a policy update, with a barrier between stages, so the slowest pod start in each stage delays the whole iteration. An agent loop alternates short tool executions in isolated sandboxes with LLM thinktime pauses. In both, jobs last from seconds to hours or days, need an isolated environment with known GPUs and mounts, and arrive repeatedly from the same caller.

2.3

The caller is a program, which shapes the API. A program tolerates queueing as long as its job starts promptly once a GPU is free, but it retries blindly on ambiguous errors. Firepool therefore rejects unknown request fields, separates a job (the intent) from its attempts, and reports every terminal failure with an error class and exit code (App. A.6).

3

2.2

Where a Cold Start Goes

With the image cached, 20–35 s remain (Table 1). The largest share is the platform init containers that every pro-

Three Goals in Tension

Fast starts, few idle GPUs and isolation between jobs pull against each other (Table 2): warm GPUs cost GPU-hours, reuse can carry state between jobs, and every boundary that retires a pod costs a cold replacement unless something hides it. These trade-offs matter only if a warm pod can serve the next job at all (§4).

D ESIGN OVERVIEW

Firepool runs as one control-plane service per Kubernetes cluster (Figure 2). A caller’s job names a hardware pool (SKU) and, optionally, a code package; the SKU fixes the GPU model, CPU, memory, base image and scaling policy. The broker records the job in a durable SQL store (MySQL in production). A dispatch loop takes queued jobs in priority order, claims warm pods from an in-memory tracker, and starts each job through an executor that talks to a small agent in the pod. Submission, pod release and pod readiness wake

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads Table 2. How common approaches handle bursts of short GPU jobs. Characterizations of other systems reflect their public documentation. Firepool’s pool policy reproduces the first three regimes for controlled comparisons (§7). Approach

Fast start

Few idle State isoGPUs lated between jobs

Yes New pod per job (K8s No Job) Persistent workers Yes No (Ray, static pool) Serverless keep-alive After first Partly job Warm-pool controllers For match- Per pool (agent-sandbox, Sage- ing jobs Maker) Images pre-cached on Removes Yes nodes pull only Firepool Yes, when Capped warm by budget

agents, RL loops, engineers

dispatcher ke

event-driven

start, reset

Yes No Per function Per claim Yes Per caller

warm pods on Kubernetes pod agent

REST API

pod tracker

pod agent

auth, validate

in memory, 1 lock

record

job store durable SQL

wa

claim

submit

supply

standby 0 VRAM

queue pool controller create, retire code packages scaler, budgets

Figure 2. Firepool architecture. Callers submit jobs to one broker per Kubernetes cluster, which records them in a durable job store. The dispatcher claims a warm pod from an in-memory tracker in one atomic step and starts the job through the pod’s agent, which fetches the job’s code package. A pool controller creates and retires pods from queue depth and budgets.

the loop; a 3 s tick remains as a fallback. A pool controller creates and retires pods to match queued demand. One lock, held for microseconds. The tracker indexes every pod by name, SKU, caller and job behind one read/write lock. Finding an eligible pod, validating the transition, checking per-caller limits and recording the lease happen in one step under the write lock, so that, by design, no two workers claim the same pod or overrun a quota. On a quiet host the lock is held 1.5–2.6 µs at the median (≤8 µs p99) with 100,000 tracked pods, sustaining 110k–220k claims/s, and the index costs about 1 KB per pod (§9.6). The durable commit that follows the claim, not the lock, bounds dispatch. No user code runs before this version-checked commit, and a failed commit returns the pods. A restarted broker rebuilds the tracker before dispatching, trusting the job store for jobs and pod annotations for ownership (App. A.1).

Rejected alternative: one lock per SKU. Several invariants span SKUs (per-caller GPU limits, per-accelerator budgets, standby exclusivity), so per-SKU locks would need a multilock order; one lock held for 2 µs is simpler and far cheaper than the commit that follows.

4

I NTERCHANGEABLE S ANDBOXES : B IND THE J OB L ATE

A warm pool helps only if arriving jobs can use its pods. If every image, requirement set or topology needed its own pool, warm capacity would splinter and most starts would be cold. Firepool keeps pods interchangeable within each SKU by binding as little as possible when a pod is created and as much as possible when a job is assigned. A shared base image per pool. Each SKU pool uses one curated image with the heavy, slow-changing layers, so the cluster caches a few base images per GPU generation instead of every experiment’s. Code and dependencies arrive with the job. User code travels as a content-addressed zip package, uploaded once and unpacked into the pod’s workspace at job start (about 0.4 s at the median for packages of a few megabytes). A bootstrap step installs extra Python dependencies on the caller’s first job on a pod and keeps them for that caller’s later jobs, which owner-only reuse (§5) sends back to the same pod. The install is keyed by a digest of the requirements file, and a pod remembers only the last set installed. Topology arrives with the job. Kubernetes training operators usually bake ranks into pod specs; Firepool pods boot as generic workers. When a job asks for N pods, the dispatcher claims all N in one step under the tracker lock, so no job holds a partial set that could deadlock against another. Without reservations, a large gang (a multi-pod job) could starve behind sustained small jobs (App. A.2). The executor then designates a coordinator, injects the torchrun rendezvous and rank variables (or starts Ray head and worker daemons), and watches for completion. When the job ends, the pods’ reset strips the ephemeral cluster state and they return to the pool as independent warm workers. Where the boundary falls. Native builds and framework pins belong in the image, where incompatibilities surface once at build time; prompts, rewards, verifiers and experiment logic belong in the package, where an edit takes effect on the next job. Costs and limits. Packages are unpacked on every job, so they are capped (512 MiB compressed, 2 GB extracted) and should stay small relative to job time (§9.2). Incremental dependencies are added or upgraded in place and never uninstalled until the pod retires; a caller who needs a clean

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

environment asks for a single-use pod (App. A.2). Callers who must bring their own image may do so from an allowlist: a mutable tag runs on a single-use pod, and a digest-pinned image may be reused by the same caller but never joins the shared SKU pool. Rejected alternative: exact shapes or per-image pools. Arbitrary shapes, or a separate pool for each image, fragment warm capacity and turn constant-time bucket lookups into best-fit searches under the lock; Firepool rounds requests up to the smallest fitting SKU (§9.2).

5

S AFE R EUSE AND I SOLATION B OUNDARIES

Reusing a pod blindly carries files, shared memory or credentials from one job to the next. Firepool bounds what crosses with four rules, decided at claim time so the release path stays fast. 1. Owner-only reuse and owner holds. In production, a used pod is reused only for the caller that used it last. To place a job, the broker searches the SKU bucket in order: an idle pod last used by the same caller (keeping its page cache and installed dependencies); a never-used idle pod; a zero-VRAM standby on an active GPU (§6); and only then a new pod. When a job finishes, the pod stays reserved for its caller for an owner hold (caller_hold_seconds, configurable per pool), so the caller’s next loop step reclaims it before anyone else can. 2. Retire at the session boundary. A caller marks the steps of one session with a reuse tag (lease_tag), and each pod records the tag and queue of its last job. Jobs from the same caller, queue and tag reuse the warm pod. When the caller starts a new session (a new tag) or switches queue, its next job may run once on the old pod so it does not wait, and the broker retires that pod when the job finishes, without consulting the database. A pod whose owner hold expires is also retired. State from one session can therefore reach at most one job of the same caller’s next session; by default, pods are never reused across callers. Operators can instead configure the tag as a soft routing hint that prefers the same pod without retiring it (§9.5); caller and queue boundaries still apply. 3. Verified reset between jobs. Between jobs, the in-pod agent kills the job’s entire process tree and verifies against /proc that it is gone; the exiting processes close every file descriptor and release the CUDA context and GPU memory. It then empties the package directory, /dev/shm, per-job GPU accounting files and any volume marked for wiping, while keeping read-only caches. If the agent cannot confirm a clean reset, for example a process that survives SIGKILL, the broker deletes the pod instead of reusing it.

(a) Without standby Pod A

Job 1

Job 2

retire waits for a new pod: 22–200 s

Pod E

Job 2

(b) With three standbys on the same GPU Pod A

Job 1, full VRAM

Pod B

standby, 0 VRAM

per-GPU gate promotes B in <1 s owner hold or retire Job 2, full VRAM

Pod C

standby, 0 VRAM

Pod D new

standby, 0 VRAM staged in background

Figure 3. Closing the gap between boundary-crossing jobs on one GPU. (a) When Pod A must be held for its owner or retired, Job 2 waits for a new pod. (b) Standby pods B–D share A’s GPU in fractional scheduler slots but run no CUDA context and hold no VRAM. When Job 1’s processes are gone, a per-GPU gate promotes B with the whole card’s memory while A is held or retired off the critical path.

4. Verified caller identity. Callers authenticate with cluster-issued or federated identity tokens (validation takes 6.6 µs when cached). The broker injects the verified identity into the job and filters user-supplied environment variables that could affect the agent or the job’s loader.

6

S TANDBY: H ANDING OVER A GPU W ITHOUT A N EW P OD

When the next job comes from a different caller, or crosses a reuse-tag boundary, the previous pod must be held or retired, and on a capacity-capped pool the job waits 22–200 s for a replacement (Figure 3a). Zero-VRAM standby. GPU sharing usually lets several jobs run at once (Yu & Chowdhury, 2020; Xiao et al., 2020); Firepool uses fractional scheduling instead to overlap the next pod’s provisioning with the current job (Figure 3b). On our H200 clusters the GPU-sharing scheduler (HAMi (Project HAMi, 2024) with Volcano (Volcano Project, 2024)) allocates fractional shares of a card. For SKUs with standby enabled, Firepool places one active pod and up to three standby pods on the same physical GPU (standbys_per_active is configurable), pinned to the active’s card by GPU UUID. The share is only a scheduler slot: a standby runs nothing but the idle pod agent, creates no CUDA context and allocates no GPU memory. When a job runs on the active pod, the agent raises the CUDA interception library’s per-GPU limit to the whole card in the job process’s environment, never in the pod spec, so the job can use all 141 GB of the H200 while standbys keep their fractional cap. Atomic promotion behind a per-GPU gate. A standby is promoted only for a dispatched job, which claims it under the tracker lock when no same-owner or unheld idle pod is

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

available. A per-GPU gate lets at most one pod per physical GPU run a job at a time, so the gate promotes the claimed standby once the previous job has exited. A pod reset for reuse is released only after its process tree is verified dead (§5), and the driver frees a context’s memory when its processes exit. The controller later stages a fresh standby in the open slot. Counting the physical GPU. Firepool counts each active– standby group as one physical GPU in every budget; counting its up to four pods separately would let held pods spill standbys onto extra GPUs. Before promoting, the broker checks that every share on the card belongs to that Firepool group; after a restart, promotions fail closed until a live sweep re-verifies exclusivity (App. A.4).

7

P RICING WARMTH : Q UEUE -D RIVEN S CALING AND B UDGETS

Warm pods cost GPU time, so every pool has explicit budgets counted in physical GPUs (or CPU cores), and 1-GPU and 8-GPU pools share one per-accelerator ledger. Scale on the queue, not on utilization. Warm pods are idle by design, so a utilization-driven scaler would shrink the pool just when it is ready to absorb a burst. Firepool instead scales each pool from queued demand and current supply, in whole pods, on every reconcile pass (30 s by default). It creates add pods, where add = min(need , ceiling − physical , step, headroom), need = max(floor − physical , demand − eligible, 0),

and floor and ceiling are pool knobs defined below. Demand counts queued, assigned and running pods; eligible counts starting, idle, assigned and running pods; and physical adds owner-held and stale pods to the eligible ones (App. A.5). All terms count GPUs rounded to whole pods, and an active– standby group counts once. Queued includes jobs waiting on their caller’s quota; the ceiling bounds the overshoot this causes. Stale pods (old templates, single-use pods) occupy capacity but cannot serve warm jobs. Step caps pod creations per pass, and headroom caps them by the global budget. Two accounting choices prevent oscillation and starvation. Starting pods are supply: GPU pods take 22–200 s to become ready, and without this the controller would create another wave on every pass. Owner-held pods count toward the floor and the ceiling but are not supply for other callers: a held pod is reserved for its previous caller, so counting it as supply would starve the rest of the queue, while leaving it out of the physical count would overshoot the floor and the ceiling. Held pods that match their owner’s own queued jobs count as idle. When a queued job is blocked at the ceiling, the broker breaks expired or excess holds on pods older than a minimum tenure (60 s) and retires those pods, sparing owners with queued jobs.

Table 3. Warm-or-standby service in production over a sampled 14-hour peak period, including a 3-hour burst of ∼2,500 singleGPU H200 jobs. Pool rows are broker counter rates; the standby row counts per-job records. Scope

Warm or standby

All pools (3,206 acquisitions) Single-GPU H200 pool Second single-GPU H200 pool Standby when idle pods ran out

94.7% 98.6% 97.7% 187/222 (84.2%)

Scale-down removes only idle pods whose keep-alive has expired, never below the floor, and retains up to the idle cap (App. A.5). Four knobs. Operators tune each pool’s latency–cost tradeoff with a floor (min_count, pods kept ready at zero demand), an idle cap (max_idle_count, unheld idle pods kept past their keep-alive; it retains but never pre-creates), a ceiling (max_count, a hard cap on GPUs held), and a keep-alive τ (idle_timeout, with the separate owner hold caller_hold_seconds). Floor = idle cap = 0 with keep-alive τ is serverless TTL retention; floor = peak with τ = ∞ is a static cluster; single-use pods are Kubernetes Jobs (§9.4). Hard limits and borrowed capacity. A deployment caps each GPU model across all pools, reserving GPUs in an in-memory ledger before calling the Kubernetes API, so concurrent reconcilers never overshoot. Firepool can also place warm or burst pods on unused reserved capacity, as advertised by the cluster’s capacity controller, and gives them back within about 30 s when the owner reclaims its quota (§9.6, App. A.7).

8

P RODUCTION E XPERIENCE

Firepool runs at LinkedIn across our production Kubernetes clusters, one broker per cluster, managing CPU pools and multiple generations of NVIDIA data-center GPUs. It powers coding-agent, RL training (fanning out 100+ parallel rollout jobs per step) and other agentic research workloads. 94.7% of pod requests are served warm. Over a sampled 14-hour peak period, Firepool served 94.7% of 3,206 pod acquisitions (98.6% on the single-GPU H200 pool) without creating a pod (Table 3). When a 3-hour burst exhausted that pool’s unheld idle pods, standby promotions served 187 of the 222 jobs that found none (84.2%); the other 35 got new pods. After the burst, a floor of two idle pods absorbed steady arrivals. Under a burst, jobs wait for quota, not for pods. Once a job is assigned a warm or standby pod, its code starts in 0.3–0.5 s at the median and ≤0.7 s at p90, quiet or burst

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads Table 4. Start latency by how the pod was obtained, single-GPU H200 pools, one week. Submit→first is what the caller sees, including queueing for quota; pod→first is setup after a pod is assigned. Quiet: an uncontended pool; burst: the burst above, where new base-image pods are not broken out. Load Pod obtained by

Submit→first Pod→first p50 / p90 (s) p50 / p90 (s)

Quiet Warm reuse 1.8 / 3.5 Standby promotion 2.4 / 50.4 New pod (base image) 65.2 / 114.4 Burst Warm reuse 53.0 / 267.6 Standby promotion 112.1 / 278.6 New pod (custom image) 69.7 / 234.3

0.5 / 0.7 0.5 / 0.7 0.6 / 0.7 0.4 / 0.6 0.3 / 0.5 54.1 / 171.6

(Table 4). During the burst, warm-reuse jobs waited 53 s at the median because callers had saturated their GPU quota and queued behind their own jobs; budgets and borrowing (§7) govern that wait. Only custom-image jobs wait after assignment (54.1 s), while their image is pulled; their lower burst wait reflects different callers and queues, not faster pods. The quiet standby p90 reflects retry polling from before dispatch became event-driven. What warmth costs in production. Idle pods account for a small share of the GPU time Firepool holds in production. Most of that is floors kept on low-traffic pools, which operators can lower; owner holds add little. The broker adds only a few requests per second to each cluster’s API server and did not trigger client throttling in our observations.

9

E VALUATION

Venues. Headline results come from controlled benchmarks in dedicated pools on production nodes: H200 uses one pinned 8×H200 node with 1-GPU pods, a cached 12.2 GB base image and a two-GPU pool cap, and CPU uses 4 vCPU/8 GiB pods with a cached 4.4 GiB image and a 16-pod cap. Mechanism costs come from a local cluster, and large parameter sweeps from a trace-calibrated simulator of Firepool’s control plane (App. B). Workloads and arms. W-RL runs 4 barrier-synchronized iterations of 8 rollouts, 2 evaluations and 1 policy update (fixed 1.5–4 s bodies). W-Agent runs 2 or 4 concurrent agents for 3 iterations of 2–4 calls each (fixed 0.5–1.6 s bodies, the last a verify step), with 0.5–2 s think pauses between calls and 2–5 s between iterations. We compare three arms. Reuse On gives each caller a stable reuse tag. Reuse Off gives every job a fresh tag, so a pod runs at most one job past its own tag and then retires, and a job may also land on an existing idle, recycled or standby pod; it ablates owner reuse. Fresh pod per job makes Firepool create a single-use pod for each job and delete it at release, the closest analogue of a Kubernetes Job. Arms are inter-

leaved in a Williams crossover with 6–10 paired repeats; we report pooled percentiles and paired differences with 95% bootstrap CIs. 9.1

Is Pod Creation Off the Critical Path?

Yes. With fixed-duration jobs on H200 and against a fresh pod per job, warm reuse finishes agent and RL loops 13.0× and 10.8× faster; against per-job reuse tags, it removes the tail (p90 59.1 s → 3.5 s). Fresh pod per job. Under a two-GPU cap, a single-use pod per job admits two jobs at a time and each waits for the previous pod’s deletion and a new pod, so a job queues about 90 s at the median and an end-to-end multi-step agent loop takes 22–32 minutes instead of 0.6–2.5 minutes (Table 5; paired differences in App. C.6). Every one of the 1,600 jobs succeeded, with zero image pulls. The warm RL p90 (80.8 s) comes from rollouts queueing behind the two busy GPUs. Reuse Off. This arm removes only owner reuse. The H200 pool keeps three zero-VRAM standby pods per active GPU, so standby or recycled pods served 137 of the arm’s 190 jobs, and its median start latency is close to Reuse On’s (the paired CI spans zero). The 20 jobs that waited for a new pod (82 s median), and those queued behind them, make the loop 3.6× slower (paired +130 s [74, 141]). On CPU, which has no standby, Reuse Off pays a pod start on most jobs (+43.7 s and +25.5 s at p50), and W-RL’s barriers turn these stragglers into a 4.5× slower loop (Figure 4a). 9.2

Does Late Binding Keep Pods Interchangeable?

Yes: binding a job costs about 0.4 s, and one shared pool stays flat as environments multiply. Across the production benchmarks, delivering the code package, unpacking it and starting the job took 0.40–0.54 s at the median (p99 ≤1.45 s on H200). Late binding’s larger benefit is the fragmentation it avoids (Figure 5). With 64 environments, a pool per environment that may grow to the budget sees its RL-rollout hit rate fall from 0.85 to 0.68 and holds 2.15× an always-on pool, against 1.06× for one shared pool; splitting the budget instead pushes p90 start latency to 400 s. The shared pool loses only if binding is slow: with a 30 s hand-off it holds as much as per-environment pools. 9.3

Is the Gap Between Boundary-Crossing Jobs Gone?

Yes: standby promotion cuts the handover wait from 51 s to 0.94 s and the loop 3.0×. One caller runs 30 sequential agent jobs, each with a fresh reuse tag (Table 6). A job may reuse the previous job’s pod once, after which that pod

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads Table 5. Warm reuse against a fresh pod per job (H200) and against per-job reuse tags (H200 and CPU), production nodes. Percentiles pool all jobs; the paired difference is the median of per-repeat p50s, Off/Fresh minus On, with 95% bootstrap CI. Makespan is the median over repeats. (N) counts jobs over both arms. Agent columns ran 4 agents, except H200 vs Reuse Off (2). The fresh-pod columns ran the event-driven dispatcher of §3; the Reuse Off columns ran an earlier, tick-driven one with the reuse tag as a retiring boundary (§5). Speedups use unrounded medians. H200 vs fresh pod per job W-Agent (720)

vs Reuse Off (fresh tag per job)

W-RL (880)

H200 Agent (380)

CPU RL (528)

CPU Agent (444)

fraction of jobs

Start p50 (s), On / Off 0.61 / 89.5 3.92 / 115.8 1.91 / 2.29 2.83 / 52.6 1.78 / 27.5 Start p90 (s), On / Off 3.51 / 354.6 80.8 / 315.2 3.52 / 59.1 29.9 / 62.0 52.8 / 58.0 Paired p50 diff. (s) +95.4 [82.4, 107.8] +119.1 [105.5, 134.4] +0.43 [−0.03, 0.84] +43.7 [36.2, 49.1] +25.5 [24.9, 26.1] Makespan (s), On / Off 119.3 / 1,554.2 163.0 / 1,761.9 51.8 / 186.4 116 / 516 186 / 403 Loop speedup 13.0× 10.8× 3.6× 4.5× 2.2×

1.0 0.9

(a) CPU, RL loop

(b) CPU, agent loop

(c) H200, agent loop

(d) H200, standby benchmark

0.5

0.0 1

10

100

1

10

100

submit → first instruction (s)

submit → first instruction (s)

Reuse On: p50 2.8, p90 29.9 s Reuse Off: p50 52.6, p90 62.0 s

Reuse On: p50 1.8, p90 52.8 s Reuse Off: p50 27.5, p90 58.0 s

1

10

100

submit → first instruction (s)

1

10

100

submit → first instruction (s)

Reuse On: p50 1.9, p90 3.5 s Reuse Off: p50 2.3, p90 59.1 s

standby on: p50 1.7, p90 27.5 s standby off: p50 28.2, p90 57.7 s

Figure 4. Start-latency distributions, all repeats pooled (circles p50, triangles p90). (a, b) On CPU, Reuse Off shifts most jobs right by a pod start. (c) On H200 the medians are close because Reuse Off jobs often land on standby or recycled pods; the gap is the tail of jobs that wait for a new pod under the two-GPU cap. (d) The standby benchmark (§9.3), a fresh reuse tag per job, all jobs.

Table 6. Standby on H200: 2-GPU pool, 10 paired repeats of 30 sequential jobs with a fresh reuse tag each (300 jobs per arm). A handover is a job that found no idle pod. Queue wait is over handovers; submit→first pools all jobs. The makespan difference is the median of paired differences. Standby on Standby off Handovers served by standby 101/117 (86%) Queue wait at handover, median 0.94 s Submit→first instr., p50 1.66 s Loop makespan, median (3.0×) 337 s

0/156 51.0 s 28.2 s 1,023 s

is retired, so jobs alternate between reusing the previous pod and a handover. With standby, 86% of handovers were promotions; the median queue wait over handovers was 0.94 s (0.81 s for promotions alone), after which hand-off and first instruction add 0.5 s. All 10 paired repeats favored standby (paired makespan +685 s [591, 766]). Figure 4d shows the distributions.

9.4

What Does Warmth Cost?

Locally and in simulation, a keep-alive matched to the loop’s pauses keeps the latency benefit while holding about half the capacity of an always-on pool; measured on H200, warm reuse costs 15% more GPU time per job for a 3.6× faster loop. We measure held capacity: GPU- or core-seconds held by a pool’s pods from binding to deletion, busy, idle or owner-held, relative to an always-on pool sized to peak concurrency. On the local cluster (Figure 6a), 60 s keep-alives held 0.24– 0.52× the always-on pool for +28–46 ms at the median. With production’s 22 s pod start (Figure 6b), a 60–120 s keep-alive holds 0.48–0.68× for +0.01 to +0.27 s at the median. In contrast, a long 900 s keep-alive with no idle cap holds 1.15× measured and 1.12–1.54× simulated, as it retains burst overshoot across the entire timeout window. On H200, warm reuse held 12.3 GPU-seconds per job against 10.7 for Reuse Off, mostly pods held for their caller between agent steps.

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads 1.0

warm-or-standby hit rate

held pod-time ÷ always-on pool

2.5

0.9

2.0

0.8

1.5

p90 start latency (s, log) 100 10

1.0

0.7 1

8

1

64

1

distinct environments E

8

64

1

distinct environments E

one shared base-image pool (late binding) pool per environment, each up to budget

pool per environment, budget split RL rollouts

8

64

distinct environments E

agent sessions

Figure 5. Environment fragmentation (simulator, Firepool’s control plane, CPU calibration with a 22 s pod start, 64-pod budget, Zipfdistributed environments, 10 paired seeds). One late-bound pool stays flat as the number of distinct environments E grows; a pool per environment roughly doubles held capacity when each may grow to the budget, and with the budget split it sharply raises tail latency while still holding more than the shared pool.

5

0.159×, +4.26 s

2 1

(b) Simulated: 22 s pod start, bursty agent sessions

always-on pool at peak 60 s keep-alive floor 1, idle cap 2, 60 s 900 s keep-alive fresh pod per job

1.15×, +57 ms

0.235×, +46 ms

0.5

0.523×, +28 ms

0.2

0.5

1

start p50 ÷ peak pool (log)

start latency p50 (s, log)

(a) Measured: local cluster, 5 paired repeats, 95% CI

2

floor × idle-cap × keep-alive grid keep-alive τ static pools 900 s keep-alive, no idle cap fresh pod per job

10 5

2

15 s

60 s

0.5

held processor-time ÷ always-on peak pool (log)

120 s

900 s

300 s

1 1

held processor-time ÷ always-on peak pool (log)

Figure 6. Held capacity against start latency, relative to an always-on pool sized to peak concurrency (lower-left is better). (a) Measured on the local cluster (2,300 jobs). (b) Simulated (CPU calibration) with the production 22 s pod start; the shaded band is a 60–120 s keep-alive. A 900 s keep-alive holds more than an always-on pool in both.

Warmth stops paying for loops when jobs are long: the perjob saving stays at 10–35 s on CPU, but the loop speedup falls below 1.1× beyond about 200 s (RL) or 100 s (agent) bodies, and vanishes when think time exceeds the keep-alive (App. C.1).

pods created per repeat fell from 28.9 to 2.5. The in-pod reset takes 9.3 ms for a clean job and 1.67 s in the worst case we built; App. C.2 also covers injected unrecoverable resets and cross-caller isolation. 9.6

9.5

What Does Isolation Cost?

The boundary policy dominates; the reset itself takes 9 ms. Locally, retiring a pod at each boundary added 1.53 s [0.97, 2.00] to start latency, and a fresh pod per job added 3.42 s [2.80, 4.00]. On production GPUs, a retirement costs a 22–200 s pod creation unless a standby absorbs it. Treating the reuse tag as a retiring boundary kept only about half of multi-step jobs warm; with the tag as a routing hint and the event-driven dispatcher, the warm share rose to 0.99 and

Does It Hold Under Failure and Scale?

Yes. Table 7 summarizes fault-injection and scale tests validating broker recovery, gang claims and capacity borrowing (detailed in App. C.3). Durable dispatch bounds throughput, at 483 jobs/s on MySQL, far above the peak arrival rates we have observed on a cluster.

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads Table 7. Control-plane failure and scale tests (local cluster, quiet hosts, kwok). Test Result Broker killed 30× under load Rollout, saturated pool Gang claims, N =2–8 Tracker, 100k pods Durable dispatch kwok, 20k pods Borrowing

10

0 lost, 0 duplicated jobs; 90/90 owner holds restored; dispatching 0.045 s and first claim 3.0 s after restart; API reachable 20 s after the kill Converged in 324 s; 12/12 callers served; 0 failed jobs; cap never exceeded; ∆p50 +0.03 s vs. no rollout 0 partial claims in 50 jobs; warm claim 0.27–0.45 s 110k–220k claims/s; lock 1.5–2.6 µs; ∼1 KB/pod One replica: MySQL 483, PostgreSQL 2,365 jobs/s No API throttling; Firepool’s own create rate binds first Borrow 11.8 s; return 30.0 s; owner reclaim 80/80

R ELATED W ORK

Cold starts and keep-alive. SOCK (Oakes et al., 2018), Catalyzer (Du et al., 2020), Firecracker (Agache et al., 2020), FaaSNet (Wang et al., 2021) and FaaSnap (Ao et al., 2022) cut serverless cold starts with lightweight sandboxes, snapshots and fast image provisioning; ServerlessLLM (Fu et al., 2024) and BlitzScale (Zhang et al., 2025a) load model checkpoints fast; Shahrad et al. (Shahrad et al., 2020) tune keep-alive windows against memory. GPU checkpoint/restore (NVIDIA Corporation, 2024a; Wei et al., 2025) restores an initialized CUDA process without a live pod, but not a container’s credentials, mounts and arbitrary process tree. These shorten a cold start; Firepool removes it from the critical path and composes with them for the starts it cannot avoid. Warm-pool controllers such as Kubernetes agent-sandbox (Kubernetes SIG Apps, 2026), SageMaker Managed Warm Pools (Amazon Web Services, 2026) and OpenKruise Agents (OpenKruise Contributors, 2026) keep pre-warmed sandboxes or instances for matching requests (our per-environment baselines in §9.2), and SpecBox (Zhang et al., 2026) pre-warms them speculatively. Firepool adds late binding, which lets diverse jobs match one pool, as well as zero-VRAM GPU standby, queuedriven GPU budgets, and claims made in memory rather than through the API server. GPU sharing and switching. MIG, MPS (NVIDIA Corporation, 2024b;c), Salus (Yu & Chowdhury, 2020), AntMan (Xiao et al., 2020) and HAMi (Project HAMi, 2024) co-run workloads on a GPU; PipeSwitch (Bai et al., 2020) switches contexts between processes in one server. Firepool uses fractional shares only to stage standbys, handing a GPU across isolated pods. Cluster schedulers and RL frameworks. Gandiva (Xiao

et al., 2018), Tiresias (Gu et al., 2019), Pollux (Qiao et al., 2021), MAST (Choudhury et al., 2024) and Rubick (Zhang et al., 2025b) schedule long-running training, where minutes of startup amortize over hours (characterized in (Jeon et al., 2019; Weng et al., 2022)). Kueue (Kubernetes SIG Scheduling, 2024), Volcano (Volcano Project, 2024) and SkyPilot (Yang et al., 2023) manage queueing, quotas and provisioning per job without reusable warm pods. Ray (Moritz et al., 2018), HybridFlow (Sheng et al., 2025), RLHFuse (Zhong et al., 2025) and RollArt (Gao et al., 2025) hold or disaggregate workers for a training job; Firepool brokers the short, isolated tool, verifier and rollout sandboxes around them.

11

C ONCLUSION

When agents and RL coordinators replace people in the ML loop, provisioning moves onto the critical path of every iteration. Firepool takes it off by making pods interchangeable through late binding and keeping them warm safely within GPU budgets. On H200, agent and RL loops finish 10.8–13× faster than with a fresh pod per job. Limits and next steps. Warmth does not help jobs that bring their own images, which dominate our 8-GPU pools; image-keyed pools or late binding for multi-GPU images are next. Standby is a buffer that refills at pod-creation speed; extending it to multi-GPU gangs needs gang-wide promotion, and large gangs need reservations to avoid starvation. Reuse scoped to a caller and session, with a verified reset between jobs, is designed to isolate callers from each other, though state from one session can reach the first job of the same caller’s next session. Firepool targets internal workloads; its pools compose with microVMs where kernellevel isolation is required. Recovery is tested against broker crashes, not network partitions, during which a re-queued attempt can overlap its predecessor (App. A.1).

AI U SAGE D ISCLOSURE The authors designed Firepool and its experiments. GitHub Copilot (coding agent and CLI, with Claude and GPT models) helped write parts of Firepool, its benchmark harnesses and its analysis and figure scripts; all code was reviewed by the authors. All reported numbers are computed by those scripts from recorded run artifacts; no data was generated by AI. For the paper, Copilot converted the authors’ draft to LaTeX and applied author-directed revisions and wording edits. AI reviewers (DeepReview, GPT-6 Astra and Claude Opus 5.5) critiqued drafts, and the authors decided which findings to apply. AI tools helped find related work and check citation metadata; every reference was verified by hand against its publisher or arXiv record. The authors take full responsibility for the content.

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

R EFERENCES Agache, A., Brooker, M., Iordache, A., Liguori, A., Neugebauer, R., Piwonka, P., and Popa, D.-M. Firecracker: Lightweight Virtualization for Serverless Applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pp. 419–434, Santa Clara, CA, 2020. USENIX Association. ISBN 978-1939133-13-7. Amazon Web Services. SageMaker AI managed warm pools. https://docs.aws.amazon.com/sage maker/latest/dg/train-warm-pools.html, 2026. Accessed October 2026. Ao, L., Porter, G., and Voelker, G. M. FaaSnap: FaaS Made Fast Using Snapshot-based VMs. In Proceedings of the Seventeenth European Conference on Computer Systems (EuroSys ’22), pp. 730–746, Rennes, France, 2022. ACM. doi: 10.1145/3492321.3524270. Bai, Z., Zhang, Z., Zhu, Y., and Jin, X. PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 499–514. USENIX Association, 2020. ISBN 978-1-939133-19-9. Choudhury, A., Wang, Y., Pelkonen, T., Srinivasan, K., Jain, A., Lin, S., David, D., Soleimanifard, S., Chen, M., Yadav, A., Tijoriwala, R., Samoylov, D., and Tang, C. MAST: Global Scheduling of ML Training across GeoDistributed Datacenters at Hyperscale. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 563–580. USENIX Association, 2024. ISBN 978-1-939133-40-3. Du, D., Yu, T., Xia, Y., Zang, B., Yan, G., Qin, C., Wu, Q., and Chen, H. Catalyzer: Sub-millisecond Startup for Serverless Computing with Initialization-less Booting. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’20), pp. 467–481, Lausanne, Switzerland, 2020. ACM. doi: 10.1145/3373376.3378512. Fu, Y., Xue, L., Huang, Y., Brabete, A.-O., Ustiugov, D., Patel, Y., and Mai, L. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 135–153. USENIX Association, 2024. ISBN 978-1-939133-40-3. Gao, W., Zhao, Y., Wu, T., Xiong, S., Wang, W., An, D., Cao, L., Muhtar, D., Liu, Z., Zhao, H., Huang, J., Yang, S., Li, Y., Su, W., Wang, J., Qu, L., Zheng, B., and Wang, W. RollArt: Disaggregated multi-task agentic RL training at scale. arXiv preprint arXiv:2512.22560, 2025.

Gu, J., Chowdhury, M., Shin, K. G., Zhu, Y., Jeon, M., Qian, J., Liu, H., and Guo, C. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pp. 485–500. USENIX Association, 2019. Jeon, M., Venkataraman, S., Phanishayee, A., Qian, J., Xiao, W., and Yang, F. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 947–960. USENIX Association, 2019. Kubernetes SIG Apps. Agent sandbox. https:// github.com/kubernetes- sigs/agent- s andbox, 2026. Accessed October 2026; provides SandboxWarmPool and SandboxClaim extension APIs for pre-warming and claiming pooled Sandboxes. Kubernetes SIG Scheduling. Kueue: Kubernetes-native job queueing. https://kueue.sigs.k8s.io, 2024. Accessed October 2026. Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., and Stoica, I. Ray: A Distributed Framework for Emerging AI Applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 561–577. USENIX Association, 2018. ISBN 978-1-939133-08-3. NVIDIA Corporation. cuda-checkpoint: A command-line utility for checkpointing and restoring CUDA state from Linux processes. https://github.com/NVIDIA/ cuda-checkpoint, 2024a. Accessed October 2026. NVIDIA Corporation. NVIDIA multi-instance GPU (MIG) user guide. https://docs.nvidia.com/data center/tesla/mig-user-guide/, 2024b. Accessed October 2026. NVIDIA Corporation. Multi-process service. https:// docs.nvidia.com/deploy/mps/index.html, 2024c. Accessed October 2026. Oakes, E., Yang, L., Zhou, D., Houck, K., Harter, T., ArpaciDusseau, A. C., and Arpaci-Dusseau, R. H. SOCK: Rapid Task Provisioning with Serverless-Optimized Containers. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pp. 57–70. USENIX Association, 2018. ISBN 978-1-931971-44-7. OpenKruise Contributors. OpenKruise Agents. https: //github.com/openkruise/agents, 2026. Accessed October 2026. Project HAMi. HAMi: Heterogeneous AI Computing Virtualization Middleware. https://github.com/P roject-HAMi/HAMi, 2024. Accessed October 2026;

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

CNCF Incubating project, formerly known as k8s-vGPUscheduler. Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., and Xing, E. P. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pp. 1–18. USENIX Association, 2021. ISBN 978-1-93913322-9.

Scheduling for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 595–610. USENIX Association, 2018. ISBN 978-1-939133-08-3. Xiao, W., Ren, S., Li, Y., Zhang, Y., Hou, P., Li, Z., Feng, Y., Lin, W., and Jia, Y. AntMan: Dynamic Scaling on GPU Clusters for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 533–548. USENIX Association, 2020. ISBN 978-1-939133-19-9.

Shahrad, M., Fonseca, R., Goiri, I., Chaudhry, G., Batum, P., Cooke, J., Laureano, E., Tresness, C., Russinovich, M., and Bianchini, R. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pp. 205–218. USENIX Association, 2020. ISBN 978-1-939133-14-4.

Yang, Z., Wu, Z., Luo, M., Chiang, W.-L., Bhardwaj, R., Kwon, W., Zhuang, S., Luan, F. S., Mittal, G., Shenker, S., and Stoica, I. SkyPilot: An Intercloud Broker for Sky Computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 437– 455. USENIX Association, 2023.

Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25), pp. 1279–1297. ACM, 2025. doi: 10.114 5/3689031.3696075.

Yu, P. and Chowdhury, M. Fine-Grained GPU Sharing Primitives for Deep Learning Applications. In Proceedings of Machine Learning and Systems 2 (MLSys 2020), 2020. Commonly cited as “Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications”; the “Salus” codename does not appear in the official MLSys proceedings title.

Volcano Project. Volcano: Cloud native batch scheduling system for compute-intensive workloads. https://vo lcano.sh, 2024. Accessed October 2026. Wang, A., Chang, S., Tian, H., Wang, H., Yang, H., Li, H., Du, R., and Cheng, Y. FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Compute. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 443– 457. USENIX Association, 2021. ISBN 978-1-93913323-6. Wei, X., Huang, Z., Sun, T., Hao, Y., Chen, R., Han, M., Gu, J., and Chen, H. PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation. In Proceedings of the 31st ACM SIGOPS Symposium on Operating Systems Principles (SOSP ’25), pp. 996–1013. ACM, 2025. doi: 10.1145/3731569.3764813. Weng, Q., Xiao, W., Yu, Y., Wang, W., Wang, C., He, J., Li, Y., Zhang, L., Lin, W., and Ding, Y. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pp. 945–960. USENIX Association, 2022. ISBN 978-1-939133-27-4. Xiao, W., Bhardwaj, R., Ramjee, R., Sivathanu, M., Kwatra, N., Han, Z., Patel, P., Peng, X., Zhao, H., Zhang, Q., Yang, F., and Zhou, L. Gandiva: Introspective Cluster

Zhang, D., Wang, H., Liu, Y., Wei, X., Shan, Y., Chen, R., and Chen, H. BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pp. 275–293. USENIX Association, 2025a. ISBN 978-1-939133-47-2. Zhang, X., Zhao, H., Xiao, W., Jia, X., Xu, F., Li, Y., Lin, W., and Liu, F. Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Scheduling. In Proceedings of Machine Learning and Systems 7 (MLSys 2025), 2025b. Zhang, Y., Wo, T., Wang, J., Sun, X., Zhang, M., Yuan, C., Li, L., Hu, C., Zomaya, A. Y., and Yang, R. SpecBox: Speculative sandbox scheduling for efficient LLM agent serving. arXiv preprint arXiv:2607.23933, 2026. Zhong, Y., Zhang, Z., Wu, B., Liu, S., Chen, Y., Wan, C., Hu, H., Xia, L., Ming, R., Zhu, Y., and Jin, X. Optimizing RLHF Training for Large Language Models with Stage Fusion. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pp. 489–503. USENIX Association, 2025. ISBN 978-1-939133-46-5. Commonly referred to as “RLHFuse”.

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

The appendices give implementation details behind §3–§7 (Appendix A), the evaluation methodology (Appendix B), additional results (Appendix C) and operating guidance (Appendix D). Every number here comes from a recorded benchmark run or, where stated, the simulator.

A

D ESIGN AND I MPLEMENTATION D ETAILS

A.1

Claim Path, Durability and Restart Recovery

Order of operations. For each queued job the dispatcher (1) claims the job’s full pod set in the in-memory tracker under its write lock, which validates the transition, checks per-caller GPU and pod limits and records the lease; (2) prepares the lease on the pods (and any per-job storage); (3) commits the job’s QUEUED→RUNNING transition with its pod bindings and acquisition path to the durable store, conditional on the job row’s version, so a stale or concurrent writer cannot move it; (4) marks the pods running in the pool, persisting owner and timing annotations in the pod PATCH the broker already issues; and (5) launches the job through the executor and pod agent. If (2) fails the pods are recycled and the job stays queued; if (3) fails the pods are released back to the tracker. Step (4) is best-effort; a launch failure after (3) is an attempt failure, retried or terminal. No user code runs before the durable transition commits. Attempts.

A job is the caller’s intent; each execution is an attempt identified to the pod agent as

<job-id>:<retry-count>, so a re-queued attempt can never be confused with an earlier one at the agent. Automatic retries are bounded by max_retries.

Restart. On start, before dispatching, the broker runs a synchronous reconcile that rebuilds the tracker from the Kubernetes pods Firepool owns (labels and the GPU scheduler’s card annotation give pod state, physical GPU and standby role; annotations give the last caller, release time and remaining owner hold), and from the job store. The job store is authoritative for job state; the pod annotations are authoritative for ownership. For each job recorded as RUNNING, recovery reads its pod bindings and probes the pod agents: a live attempt is re-attached; one that finished during the outage is finalized without a retry; and one whose pods or bindings are gone, that the agent does not know (never launched), or that stays unreachable after bounded retries is re-queued (RUNNING→QUEUED) and its pod released. The old attempt is not fenced before the retry is dispatched: the release cannot reset an unreachable pod, so the broker deletes it and Kubernetes terminates the attempt asynchronously, and distinct attempt IDs keep the agent from confusing the two. A network partition can therefore let both attempts run briefly; partitions are outside the tested scope. A multi-pod job is treated as a unit: if any rank is unreachable, the set stays assigned until every rank is accounted for. The scale-down cooldown is rebuilt from pod creation times, so a restart does not delete warm pods the previous process would have kept, and standby promotions fail closed until a live node sweep re-verifies GPU exclusivity (A.4). In the crash benchmark (30 SIGKILLs under load; C.3) no job was lost or ran twice and all 90 owner holds were restored. A.2

Packages, Dependencies and Multi-Pod Jobs

Packages. User code is a zip archive stored content-addressed, so identical uploads are deduplicated. At job start the broker either streams it to the pod agent or, with an object store configured, hands the agent a short-lived presigned URL, keeping the broker off the bulk data path. Limits: 512 MiB compressed (configurable), 2 GB written during extraction (a zip-bomb guard), a 5-minute push timeout, and rejection of symlinks and paths that escape the workspace. On release, the agent empties the package directory. Stored packages are retained under the object store’s lifecycle rules, which should outlast the longest queue time. Incremental dependencies. The client bootstrap installs a requirements file into the pod’s interpreter (with uv, falling back to pip) and writes a completion marker named by the SHA-256 of the file into site-packages, only after the install exits cleanly. Before every install it removes all markers, so at most one marker exists: that of the last pin set installed completely. Returning to an earlier pin set therefore reinstalls it rather than trusting a stale marker. Installs add and upgrade; a package dropped from the pins stays until the pod retires, so a caller who needs a clean environment asks for a single-use pod. Owner-only reuse confines this residue to one caller. Reusable digest-pinned images (§4) are fenced by caller, pool, queue, reuse tag and image digest, and count against the pool’s ceiling and the global GPU limits but not its idle cap. Multi-pod jobs. Rank and rendezvous variables injected by the executor cannot be overridden by user environment. A multi-pod claim is all-or-nothing over idle pods and promotable standbys on distinct cards; if package distribution fails on any pod the launch fails, and release resets every pod of the set. The dispatcher does not reserve pods for a job it cannot

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

place yet: it skips the job so smaller jobs behind it proceed, and the scaler counts the job’s N pods as demand. A.3

Reuse Boundaries and Reset Scope

What crosses reuse. On every release the pod agent kills the job’s whole process tree and verifies against /proc that it is gone (waiting a bounded time for tasks in uninterruptible sleep), then empties the package root, /dev/shm, volumes marked for reset and the per-attempt GPU accounting files. A reset that cannot confirm all of this reports unclean and the broker recycles the pod. Boundaries. Reuse is scoped by caller and by a caller-supplied reuse tag (lease_tag) that names a session. A used idle pod is claimable only by its last caller (owner-only reuse). Jobs sharing the same caller, queue and reuse tag reuse the warm pod; when a caller switches to a new session or a hold expires, the prior pod is retired, so state from one session reaches at most one job of the same caller’s next session. Operators can configure the tag as a soft routing hint that prefers affinity without retiring pods. Owner holds. After release a pod is held for its last caller for caller_hold_seconds (inheriting the idle timeout when unset). The caller and release time are persisted as pod annotations in requests the broker already makes, so holds survive restarts. A hold is broken early only when a queued job is blocked at the pool ceiling, and only on pods older than a 60 s minimum tenure, oldest hold first; holds whose owner has queued jobs are spared. A broken hold retires its pod; it is never handed to another caller. Isolation. Owner-only reuse, retirement at session boundaries, verified process and scratch-space resets, and environmentvariable filtering isolate callers from each other and bound what crosses between a caller’s sessions; they compose with VM-level sandboxes where kernel-level isolation is required. A.4

Standby Mechanics

Shares and memory cap. On H200 the GPU-sharing scheduler (HAMi with Volcano) places one active pod and standbys_per_active standbys on fractional shares of one card, pinned to it by GPU UUID, and caps each pod’s memory at its share through a CUDA interception library. A standby runs only the pod agent: no user code, no CUDA context, no allocation. When a job starts on the active pod, the agent sets CUDA_DEVICE_MEMORY_LIMIT_<i> to the card’s full framebuffer in the job process environment, which wins over the values injected at admission, and points CUDA_DEVICE_MEMORY_SHARED_CACHE at a fresh per-attempt file, so a reused pod does not inherit an earlier limit. The pod spec is never changed, so standbys keep their fractional cap. Gate and exclusivity. At most one Firepool pod per card serves at a time, so the gate promotes a standby once the previous job has exited. A pod reset for reuse is released only after its process tree is verified dead; the driver frees a context’s memory when its processes exit, so gating on process death is more reliable than polling device memory. A pod marked for retirement is deleted off the critical path while the promoted standby serves. A periodic sweep checks that every share on a card belongs to Firepool’s active–standby group: a card missing from the usage report counts as non-exclusive, a failed read changes nothing that tick, and after a restart promotions stay disabled until the sweep has verified the card. The group counts as one physical GPU in pool ceilings and global budgets. A.5

Scaler

Each reconcile pass counts processors (GPUs or cores) rounded to whole pods; an active–standby group counts once and standbys are never supply. queued sums the pods of every queued warm job, including jobs waiting on their caller’s quota, so such jobs can drive creation up to the ceiling. A held pod matched by its owner’s own queued jobs counts as idle for that pass, as supply for that owner only. With demand , eligible and physical as in §7 (idle excludes other held pods; stale covers old-template and single-use pods, which occupy capacity but cannot serve warm jobs), the pass computes
floorDeficit = max 0, floor − (physical − singleUse) ,

demandDeficit = max(0, demand − eligible),
add = min max(floorDeficit, demandDeficit), ceiling − physical , step, headroom .

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

Single-use pods occupy the ceiling but cannot satisfy the floor. When nothing is created, scale-down considers only idle pods whose keep-alive has expired:
idleCap = min maxIdle, ceiling − (physical − idle) ,
retain = max idleCap, floor − (held + assigned /running), demand − (starting + assigned /running) ,
remove = max 0, min(idle − retain, expiredIdle, physical − floor ) . Here assigned /running is assigned + running. Because retain keeps floor − (held + assigned /running) idle pods, scale-down never takes warm pods below the floor even when stale pods inflate physical . Global per-accelerator budgets are reserved in an in-memory ledger before the Kubernetes create call. Defaults: reconcile every 30 s, hold-break minimum tenure 60 s. A.6

Caller API and Errors

The REST API decodes requests with unknown fields rejected, so a typo’d option fails loudly instead of being ignored. Job states are PENDING, QUEUED, RUNNING, SUCCEEDED, FAILED and CANCELLED. A terminal job carries an error class (e.g. OOM, NonZeroExit), the exit code of the final attempt’s rank-0 process (absent, rather than 0, when no process ever exited), its retry count, and the acquisition path of its pods (warm, recycled, standby, new, custom image) with the reason a warmer path was not used. Callers authenticate with cluster-issued or federated identity tokens (cached validation: 6.6 µs). A.7

Borrowed Capacity

A capacity controller publishes, per GPU model, the reserved capacity that is currently unused. Firepool polls it (default 10 s) and may place warm or burst pods in that headroom up to the published ceiling. When an owner reclaims, the controller lowers the ceiling; Firepool records the reclaiming owner’s quota demand, as reported by the controller, as demand and preempts borrowed pods (default every 15 s), releasing one GPU per GPU the owner needs. On kwok with a stub controller (C.3), borrowing took 11.8 s, release to a lowered ceiling 30.0 s, and owners got 80 of 80 pods with exactly one borrowed GPU released per owner GPU.

B

E VALUATION M ETHODOLOGY

B.1

Venues

Headline benchmarks run in dedicated benchmark pools on production nodes. Production H200: one pinned 8×H200 node per campaign, 1-GPU pods, a cached 12.2 GB base image and a two-GPU cap (§9.1 H200 columns, §9.3, paper Table 1). Production CPU: 4 vCPU/8 GiB pods, a cached 4.4 GiB image and a 16-pod cap, beside steady tenant load (§9.1 CPU columns). §8 uses production fleet telemetry: broker metrics, per-job records and pool gauges. Mechanism costs (§9.4–9.6, C.2–C.5) run on a local cluster, single-host Kubernetes (k3d) on a shared 8-vCPU VM with runs serialized under a host lock and a quiet-load guard. Tracker, dispatch, catalog and authentication microbenchmarks (C.4) run on a quiet host, a 64-vCPU cloud VM (AMD EPYC 7763 / Xeon 8370C) with stock MySQL 8.0.33 and PostgreSQL 16.15. API load to 20k pods and borrowing (A.7) use kwok: a real API server, etcd and scheduler with simulated nodes and a MySQL store. Frontiers and sweeps (§9.2, §9.4, C.1) use the simulator, which runs Firepool’s control-plane code (pool, tracker, scaler, dispatcher) in virtual time with calibrated venue models. B.2

Workloads

Workloads follow §9 and are parameterized from production job shapes; job bodies are fixed-duration payloads, so differences between arms come from acquisition, not work. Per repeat and arm, W-RL submits 44 jobs; W-Agent submits 19 jobs with 2 agents (R3), 37 with 4 agents on CPU and 36 in the cap sweep; and the standby benchmark runs one caller’s 30 sequential W-Agent jobs, each with a fresh reuse tag. Makespan runs from a repeat’s first submission to its last completion. B.3

Arms

Reuse On: a stable reuse tag per caller, so each caller’s steps share its warm pod. Reuse Off: a fresh reuse tag per job with the tag as a retiring boundary: a job may run once on the previous job’s pod, which then retires, or land on an unheld idle, recycled or standby pod; this arm ablates owner reuse. Fresh pod per job: each job asks for the pool’s own image

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

with reuse disabled, so Firepool creates a single-use pod for each job and deletes it at release, the closest analogue of a Kubernetes Job. B.4

Statistics

Arms are interleaved per repeat in a Williams crossover design to balance carry-over (e.g. warm pods left by the previous arm). Percentiles pool all jobs of an arm with linear interpolation. Paired effects are computed per repeat and summarized with 95% bootstrap CIs over repeats (seed 2027, 2,000 resamples); the start-latency effect is the median of per-repeat p50 differences, and the makespan effect the median paired difference. Simulator calibration. The simulator’s venue constants (per-phase pod start, termination and dispatch costs) are fitted on train campaigns and validated on held-out campaigns registered in advance; a fit is adopted only if it does not worsen any hold-out set. Under the adopted configuration, 46 of 55 held-out metrics (84%) fall within band, 52 of 65 (80%) including the H200 run (R3), and 132 of 145 of a second hold-out set fall inside their 95% prediction intervals. Prediction intervals are ×0.69–1.44 for queue medians and ×0.76–1.31 for makespan and throughput. Rankings of arms are preserved in every hold-out set and all effect signs are correct. We use the simulator for shapes, orderings and crossovers, never for headline numbers.

C

A DDITIONAL R ESULTS

C.1

Held Capacity, Job Length and Think Time

Simulated frontier (production CPU pod start). Held processor-time relative to an always-on pool sized to peak, median of 3 seeds, with ∆p50 start against that pool (∆p50 not recorded at τ = 300 s):

Pattern

τ = 15 s

τ = 60 s

τ = 120 s

τ = 300 s

τ = 900 s

Fresh pod per job

Steady Bursty Diurnal

0.435 (+0.38 s) 0.371 (+0.64 s) 0.486 (+0.37 s)

0.571 (+0.08 s) 0.476 (+0.27 s) 0.606 (+0.01 s)

0.679 (+0.01 s) 0.615 (+0.13 s) 0.659 (+0.03 s)

0.916 0.849 0.860

1.543 (−0.02 s) 1.484 (+0.09 s) 1.124 (−0.01 s)

1.347 (+22.8 s) 0.646 (+23.0 s) 1.543 (+22.9 s)

With 22 s pod starts the scaler creates more pods than the peak concurrency during bursts (11–16 against 5–7), because queued calls trigger creates while released pods free up seconds later; a 900 s keep-alive keeps that overshoot through the whole tail. A fresh pod per job is not the cost floor at production pod-start times. Measured frontier (local cluster). 2,300 jobs, 5 paired repeats: a floor of 1, idle cap of 2 and 60 s keep-alive held 0.52× [0.52, 0.53] the always-on pool for +28 ms at the median; a bare 60 s keep-alive 0.24× for +46 ms; a 900 s keep-alive (no idle cap) 1.15× [1.09, 1.18] for +57 ms. Simulated frontier with a floor. With production’s 22 s pod start, a floor of 4 with an idle cap of 2 buys back the bursty tail: p90 3.39 s at 0.78× the always-on pool. H200 GPU time. From per-minute pool gauges during the R3 run: warm reuse held 2,340 GPU-s for 190 jobs (12.3 per job; 87% idle, almost all owner-held between agent steps); Reuse Off held 2,040 GPU-s (10.7 per job; 35% creating pods). Standbys add no physical GPUs. C.2

Isolation Costs (local cluster)

Pod-agent reset (130 trials on one pod, owner-only reuse on): clean 9.3 ms; 10,000 scratch files 187 ms; 8 GiB of /dev/shm 1.35 s; leftover processes about 110 ms (one re-check after SIGKILL); worst case (all three) 1.67 s. Injected

unremovable residue: the agent reported unclean, the pod was recycled 17 ms after the job finished, and the next job ran on a new pod (5/5, 0 failed jobs). Three concurrent callers: 0 cross-caller reuse, Jain goodput fairness 0.976.

makespan, Reuse Off ÷ Reuse On

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads (a) CPU RL loop

(b) CPU agent loop think 0 s think 30 s think 120 s think 600 s think 1800 s

4 3 2 1 1

10

100

job body length L (s, log)

1000

1

10

100

1000

job body length L (s, log)

Figure 7. Loop speedup of Reuse On over Reuse Off against job body length and think time (simulator, production CPU calibration; bands span 3 replicates). The gain falls below 1.1× at about 196 s (RL) and 104 s (agent) bodies with no think time, and vanishes when think time exceeds the 900 s keep-alive. At 1–2 s bodies the sweep gives 4.1–4.3× for RL (measured 4.5×) and 1.6–2.0× for agents (measured 2.2×).

C.3

Failure, Rollout and Borrowing

Crash. 10 repeats (5 agent with 3 callers, 5 RL), 300 s each, SIGKILL at 60, 150 and 240 s: 30 kills. 0 lost and 0 duplicate jobs; held 90/90, unheld idle 92/92 and busy 27/27 pods kept; process start to dispatching 0.045 s p50 (0.068 s p90); first claim 3.0 s after start; kill to first API response through the Service 20 s (kubelet back-off and readiness probe). Template rollout under saturated load. An 8-pod pool (floor 4, idle cap 6, ceiling 8, drain budget 1), 12 callers with no think time, a template edit: converged in 324 s [286, 359]; all 12 callers served in every run; longest wait 181–242 s; 0 overtakes of the longest waiter; 0 failed jobs; at most one pod retiring and never more than 8 pods; ∆p50 +0.03 s. Borrowing (kwok). 10 ceiling cycles and 10 owner cycles: time to borrow 11.8 s [10.5, 13.2]; release to a lowered ceiling 30.0 s; owners placed 80/80 pods within 240 s; 1.0 borrowed GPU released per owner GPU in all cycles; 0 over-placement. C.4

Control-Plane Scale

Warm start. With an idle H200 pod, a warm start took 1.85 s under the 3 s dispatch tick (1.27 s of it waiting for the tick) and takes 0.59 s with event-driven dispatch: accept 0.04, dispatch 0.01, hand-off 0.39 and first instruction 0.12 s. At 100,000 pods the tracker sustains 191–219k claims/s with one claimer and 109–120k with 32; a claim holds the write lock 1.5–2.6 µs at the median (≤8 µs p99); lock-wait p99 is 0.66 ms at 32 claimers; a reconcile pass takes at most 0.56 s of its 30 s interval; the index costs about 1 KB per pod. Durable MySQL dispatches 33 jobs/s with one goroutine and 483 with 64 (21 ms per commit); PostgreSQL 184 and 2,365 (3.7 ms); with the database on another host, 244 and 1,505 at 64 goroutines. The 5,025 jobs/s we measured on in-memory SQLite is an engine bound, not a production figure. Catalog load: 1,000 pool definitions in 347 ms. Authentication: cached service-account token 6.6 µs, uncached with local keys 47 µs. In production the broker sends only a few requests per second to each API server without throttling; on kwok to 20,000 pods across 3,282 nodes, API priority-and-fairness never throttled (p99 wait ≤0.13 s), and Firepool’s own serial pod creation and 100-QPS client limiter bound first. C.5

Late Binding Details

The simulator behind paper Figure 5 runs Firepool’s control plane with a 22.1 s cached cold start, a 64-pod budget, a 900 s keep-alive, Zipf (s = 1) environment popularity and 10 paired seeds (3,360 cells). With per-image cold starts of 300 s, per-environment pools hold 5.96× the always-on pool on RL rollouts while the shared pool stays at 1.06×; with a 30 s per-job hand-off, the shared pool holds as much as per-environment pools at E = 64 (2.08× vs 2.15×). Dependencies. First install 0.11 s (3 pins), 2.5 s (23 pins), 4.3 s (CPU PyTorch); repeated pins 0 (marker hit); one changed

Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads

pin 0.03–0.06 s. Ray. Submit to first task on a warm N -pod Ray cluster formed at job start: 8.6 s (N =2), 9.9 s (N =4), 13.1 s (N =8), against 14.4–19.6 s cold. Warm pods save the pod start but not Ray’s own bootstrap (driver start 2.0–2.6 s, ray.init 0.4 s, first task 1.1 s after the last node registers). Multi-pod warm claims for torchrun-style jobs take 0.27–0.45 s for N =2–8, with 0 partial claims in 50 jobs; an 8-pod job behind a 3-pod holder held no pods while waiting and claimed all 8 at once. C.6

H200: Fresh Pod per Job and Reuse Off

Fresh pod per job (cap sweep). Paired repeat-makespan differences, fresh pod minus warm reuse, under the two-GPU cap: +1,438 s [1,314, 1,769] (agent) and +1,572 s [1,423, 2,097] (RL), 10 paired repeats each. Acquisition paths (R3). Warm / standby / recycled / new pods: Reuse On 171 / 11 / 4 / 4; Reuse Off 33 / 66 / 71 / 20.

D

O PERATING G UIDANCE

We follow five operating rules: • Judge warm pools by tails, loops and held capacity. Our H200 medians differ by 0.4 s while p90 differs by 55 s and the loop by 3.6×; report p90, loop makespan and held GPU time alongside the median. • Size keep-alives to the loop’s pauses. A keep-alive just above think time bridges pauses; a long one keeps every burst’s overshoot. Owner holds partition idle pods by caller, so size idle caps by active callers. • Make each isolation boundary a per-tenant choice. A hard per-tag boundary kept half of jobs warm and a routing hint 99%; choose per tenant. • Build for a caller that is a program. Reject unknown fields and return typed errors to stop retry storms; record how each pod was obtained to tell cold-start regressions from quota exhaustion. • Test the control plane as hard as the data plane. Once pods are warm, the dispatch wake-up and the durable commit dominate a warm start (C.4); fault-inject broker restarts, template rollouts and capacity reclaims.

Related documents

• EmbodiedIntentBench: Benchmarking Intent-Driven Runtime Adaptation for Mobile Robotic Missions#1605227
• Reproducible LoRA Inference Through Deterministic Split-K Reduction#1605228
• Continual Instruction Tuning for LLM-ased Management of Continuum Resources#1605229
• FirstSwap: Pre-Consensus Trade Approvals for Decentralized Market Design#1605230
• PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM Agents#1605231