ReclaimNet: Reclaim-Aware Network Protocols for Voluntary GPU Sharing on Campus Wenyang Jia∗ , Jingjing Wang∗ , Xianneng Zou† , and Kai Lei∗‡ ∗ ICN Lab, Shenzhen Graduate School, Peking University † Tencent
arXiv:2605.28872v1 [cs.NI] 23 May 2026
‡ Corresponding author: [email protected]
Abstract—University campuses host abundant but fragmented GPU resources whose voluntary sharing is blocked by a mismatch between revocable, autonomous ownership and migration mechanisms that assume stationary failure hazards, homogeneous interconnects, and unbounded transfer windows. We present ReclaimNet, a network-layer migration protocol suite that treats provider reclaim as a first-class contract rather than a failure case, combining three mechanisms: (i) reclaim-aware checkpoint scheduling that jointly adapts to time-varying departure hazards and contended bandwidth across co-resident jobs; (ii) volatilityaware destination selection integrating topology, survival probability, and notice-window feasibility; and (iii) deadline-aware migration traffic control with edge enforcement and a submillisecond TC BPF kill-switch. A two-month deployment on a 54-node heterogeneous campus testbed reduces work loss by 66% over Slurm preempt-and-requeue and 38% over pipelineredundancy checkpointing, with 38% shorter downtime and under 3% degradation of background research traffic. The prototype is open-sourced at the anonymous repository https: //anonymous.4open.science/r/ICNP2026-ReclaimNet/. Index Terms—Network protocols, eBPF, reclaim-aware GPU migration
I. I NTRODUCTION The rapid expansion of AI research has made GPU clusters a critical resource in academic institutions. University campuses host a rich but fragmented collection of GPU resources, ranging from consumer-grade workstations to institutional HPC servers, that collectively represent substantial computing capacity. Yet this capacity remains severely underutilized: our measurements across 54 GPU nodes spanning four buildings over two months reveal an average utilization rate below 35%, driven by temporal mismatches between resource availability and demand (§IV). The campus GPU sharing opportunity. Resource pooling across departments can substantially raise utilization while reducing procurement costs and supporting green research. Unlike commercial cloud services, campus sharing occurs within a trusted institutional network where accountability mechanisms already exist and the marginal cost of electricity and maintenance is low. The very properties that make campus sharing attractive, namely mutual trust and lightweight coordination, also define its fundamental challenge. The voluntary participation challenge. Campus GPU resources are individually owned by faculty and research labs. Unlike data-center nodes governed by institutional mandates, campus providers are volunteers: they contribute idle resources
on their own terms and retain the unconditional right to reclaim hardware immediately, without prior notice, negotiation, or penalty. This property is not an edge case to be tolerated but a first-class design requirement. Existing systems are ill-suited for this model. Industrial cluster managers such as Kubernetes [1] and Slurm [2] assume persistent node availability under centralized control and treat departures as failures. Volunteer computing systems such as SETI@home [3] and Folding@home [4] support voluntary participation over wide-area networks but provide no stateful workload migration. Recent work on campus GPU sharing [5] demonstrates that provider-autonomy-first design with application-level checkpointing is feasible, but leaves the underlying network-layer protocols for migration unaddressed. Why existing protocol techniques do not transfer. Voluntary campus GPU sharing breaks three foundational assumptions that prior approaches rely on: • Challenge 1 (Checkpoint timing): Classical checkpoint models [6], [7] assume a stationary hardware failure hazard and a fixed write cost. In voluntary sharing, departure follows time-varying behavioral patterns (daily peaks at lunch and evening) and checkpoint cost itself depends on the bandwidth available at write time. Scheduled departures add a notice window τ in which the system must decide whether a final checkpoint can complete before reclaim, while emergency departures must fall back to the freshest completed snapshot. No prior model captures this joint time-varying hazard, network-limited cost, and reclaim-window feasibility. • Challenge 2 (Migration destination): Campus networks are three-tier hierarchies with heterogeneous bandwidth: intra-building paths are ≈3.4× faster than the crossbuilding backbone in our measurements, and a destination that itself departs soon compounds failure risk. Topologyoblivious selection (round-robin, least-loaded) and conventional topology-aware placement (minimizing instantaneous transfer time) both miss the joint bandwidthstability-deadline objective: a fast destination is useless if the checkpoint cannot land before the source’s notice expires, and a nearby one is undesirable if its own reclaim hazard is high. • Challenge 3 (Migration traffic): Voluntary reclaim imposes hard migration deadlines and instantaneous revocation events. Off-the-shelf rate control such as DSCP
TABLE I: Why standard mechanisms are insufficient under voluntary reclaim semantics. Existing idea
Standard tion
Checkpointing
Stationary failures; Owner reclaim fixed write cost varies; cost depends on network Destination is stable Destination may once selected be reclaimed soon; source has deadline Fairness or rate cap is Flows have reclaim enough deadlines and infeasible overload cases App action can en- Provider needs imforce reclaim mediate local autonomy
Topology-aware placement Traffic shaping
Kill-switch
assump- Broken by VVN
ReclaimNet increment Risk + bandwidth controller Path + survival + deadline score Deadline admission with edge enforcement Network-layer isolation semantics
queuing or generic token-bucket shaping can cap throughput but cannot decide which migration flows are feasible under their notice windows or how to degrade gracefully when aggregate demand exceeds capacity. Commodity enforcement paths (e.g., iptables) further add ∼100 ms latency, incompatible with sub-second reclaim events. Voluntary departure thus turns workload migration into a coupled control problem: the checkpoint interval determines how much state must be transferred; the destination choice determines how much bandwidth is available; and the traffic schedule determines whether that transfer completes before the deadline. Each dimension interacts with the others, and no existing protocol addresses their joint requirements. Our contribution. We present ReclaimNet, which addresses these three challenges through network-layer protocols grounded in analysis and validated at scale. (P1) Reclaim-Aware Checkpoint Scheduling. P1 turns checkpointing from a fixed-period user policy into an online control loop for voluntary reclaim. At each epoch it uses the current emergency reclaim hazard λe (t) and available bandwidth B(t) (measured via eBPF) to select a local interval: s 2C ∗ ∆t (t) = λe (t) · Beff (t) This local rule is only the per-job primitive. The protocol contribution is the shared-network controller in Theorem 1: when K jobs contend for a checkpoint bandwidth budget Bckpt , P1 minimizes aggregate reclaim loss plus checkpointPoverhead by allocating bandwidth as b∗i = Bckpt (λi Ci )1/3 / j (λj Cj )1/3 and then setting each job’s interval from its assigned bandwidth. This gives P1 a network-level coupling absent from independent per-job timers. P1 also enforces a loss budget, checks whether scheduled departures have enough notice for a final checkpoint, and coordinates the resulting rate demands with P3 (§V). A separate adaptivity-gap result (Theorem 2) explains when state-dependent checkpointing improves over a state-independent fixed-interval policy tuned to marginal averages: strict improvement occurs iff λe (t)Beff (t) is not almost surely constant, and the gap is a directly measurable L2 Cauchy–Schwarz defect of the joint (λe , B) trace. P1 reduces average work loss by 66% compared to industry Slurm
preempt+requeue and 38% versus Bamboo [8], and by 6.7% relative to the strongest state-independent online baseline (a time-varying fixed-interval schedule), the latter matching the ∼5.2% Cauchy–Schwarz gap predicted by Theorem 2 on the deployment trace (§IX). (P2) Volatility-Aware Destination Selection. P2 extends topology-aware scheduling with two constraints absent from conventional placement: the source’s notice deadline and the destination’s own reclaim hazard. It filters candidate destinations that cannot receive the checkpoint before the notice window closes, then scores feasible nodes by current transfer cost and expected future re-migration cost over the campus network graph. A locality filtering lemma explains why same-building candidates dominate under hierarchical campus bandwidths, while the full score prevents choosing unstable local nodes (§VI). P2 reduces median migration downtime by 38% versus random destination selection. (P3) Deadline-Aware Migration Traffic Scheduling. P3 separates the protocol from the mechanism: the protocol converts notice periods into per-flow bandwidth demands, performs admission/allocation under a reserved research traffic budget, and marks infeasible flows for degraded recovery; Linux TC BPF token buckets are the edge enforcement substrate. Under controlled-edge assumptions, P3 provides two guarantees: feasible planned migrations complete before their reclaim deadlines, and aggregate migration traffic is bounded so non-migration traffic keeps at least Bmin bandwidth (§VII). Kill-switch enforcement moves to the network layer, reducing latency from ∼100 ms to under 1 ms. Evaluation. We deploy ReclaimNet on a 54-node campus testbed with heterogeneous GPUs (RTX 3090, RTX 4090, A100, A6000) across four buildings (§IX). End-to-end results over two months show 28% higher GPU utilization, an 11.7 percentage-point increase in migration success, and under 3% research traffic degradation compared to fixed-interval and topology-oblivious baselines, with scheduling latency scaling sub-linearly up to 500 nodes in emulation. The remainder of this paper is organized as follows. §II formalizes the voluntary volatile network model. §III reviews related work. §IV presents our measurement study. §V–§VII describe the three protocols. §VIII covers implementation. §IX presents evaluation. §X discusses limitations and concludes. II. BACKGROUND AND M OTIVATION A. The Voluntary Volatile Network Model We formalize the campus GPU sharing environment as a Voluntary Volatile Network (VVN), characterized by the following properties. Definition 1 (Voluntary Volatile Network). A VVN is a distributed resource-sharing system in which: (i) each node v ∈ V is owned and operated by an independent provider who may exit at any time, and (ii) node departures are classified as scheduled (notice period τ > 0, drawn from an empirical distribution Fτ with τmin ≥ 10 s in our deployment) or emergency (τ = 0), with emergency departures modeled as a
TABLE II: Comparison of GPU/cluster management platforms against VVN requirements. System Kubernetes [1] Slurm [2] Gandiva [9] GPUnion [5] ReclaimNet
Voluntary Partic.
NetworkAdaptive
Migration Protocol
Formal Guarantees
✗ ✗ ✗ ✓ ✓
✗ ✗ ✗ ✗ ✓
✗ ✗ Partial ALC only P1+P2+P3
✗ ✗ ✗ ✗ ✓
non-homogeneous Poisson process with time-varying intensity λe (t). This model captures two key differences from data center environments: departure is normative, not a failure to be avoided, and hardware is heterogeneous, precluding low-level state migration across GPU architectures. It also induces a protocol contract not present in ordinary clusters: a provider must be able to reclaim locally without waiting for a central scheduler; emergency reclaim can only recover from the freshest completed checkpoint; scheduled reclaim succeeds only if the checkpoint transfer and restart fit within the notice window; and migration traffic must not consume the shared campus network simply because a provider is exercising this right. B. Why Existing Approaches Fall Short Table II summarizes how key systems compare against the VVN requirements, and Fig. 1 previews ReclaimNet’s coordinator plus provider-node decomposition that we elaborate in §V–§VII. III. R ELATED W ORK A. Campus and Volunteer Computing Condor [10], [11] and volunteer platforms such as SETI@home, Folding@home, and BOINC [3], [4], [12] show that idle, independently owned machines can be pooled, but they target coarse-grained or stateless tasks rather than longrunning stateful GPU training. Volatile-grid systems such as MPICH-V [13] address different MPI-era failure models. GPUnion [5] demonstrates that provider-autonomy-first campus GPU sharing is feasible with application-level checkpointing; ReclaimNet focuses on the missing network-layer migration protocols required for reliable and efficient operation under voluntary reclaim. B. Checkpoint and Migration Theory Young and Daly [6], [7] derive checkpoint intervals under stationary hazards and fixed write costs. Checkpoint libraries and user-level frameworks, including SCR, FTI, libckpt, BLCR, and DMTCP [14]–[18], optimize state capture and recovery under similar assumptions; HPC and warehousescale failure studies likewise treat hazards as quasi-stationary hardware properties [19]–[21]. P1 retains Young/Daly as a local primitive but adds time-varying reclaim hazard, networkdependent checkpoint cost, notice-window feasibility, and multi-workload bandwidth allocation.
VM live migration [22]–[24] and container checkpointing via CRIU reduce downtime in more controlled settings, but they rely on hardware, kernel, or CUDA compatibility that campus GPU pools cannot assume. ReclaimNet therefore uses application-level checkpoints and focuses on network-aware transfer timing, destination choice, and traffic admission. C. GPU Cluster Scheduling GPU schedulers such as Gandiva, Tiresias, Themis, HiveD, Optimus, Pollux, and AntMan [9], [25]–[30] improve utilization, fairness, or elasticity in centrally managed clusters; production traces reflect the same stable-tenancy assumption [31]. Pipeline and distributed training systems [32]– [34] and topology-aware co-design [35] similarly assume a non-volatile substrate. Spot-market schedulers address cloud preemption [36], [37], and general-purpose managers such as Mesos, YARN, and Borg [38]–[40] manage offers and quotas. None of these systems model voluntary provider reclaim or the network-coupled migration deadlines that dominate campus VVNs. D. eBPF and Programmable Network Functions eBPF provides programmable end-host telemetry and enforcement for packet processing and per-flow shaping [41]– [44]. Datacenter transports and schedulers such as DCTCP, pFabric, CONGA, PIAS, and Varys [45]–[49] optimize congestion or coflow completion in managed fabrics. P3 does not introduce a new congestion-control primitive; it adds reclaimdeadline admission/allocation and enforces those decisions at provider edges during GPU migration. IV. M EASUREMENT S TUDY A. Deployment Setup We deployed a ReclaimNet prototype across 54 GPU nodes in four buildings of a research-university campus over two months: 20 RTX 3090, 15 RTX 4090, 8 A100 (4 GPUs each), 10 A6000 (2 GPUs each), plus one coordinator. Nodes span access- and distribution-layer segments on a three-tier campus network (1 Gbps access, 1–10 Gbps distribution, 10 Gbps core). Provider agents logged departure events, checkpoint sizes, and per-flow bandwidth samples from eBPF TC BPF programs attached to each node’s egress interface. B. Findings F1: Departure patterns are bursty and predictable. Emergency (no-notice) departures account for 42% of all events, concentrated in two daily windows (11:30–13:00 and 17:30–20:00); λe (t) varies up to 6× between peak and offpeak hours (Fig. 2a). This time-of-day structure motivates P1’s non-homogeneous Poisson model: a fixed-interval strategy tuned for peak hours over-checkpoints during quiet periods by up to ∼4.1×. F2: Checkpoint sizes span three orders of magnitude. Payloads range from ∼120 MB (ResNet-50 dense state-dict) to 48 GB (full-precision LLM state-dicts with optimizer state), with a 3.2 GB median across the mixed
Reclaim lifecycle: ① select → ② admit → ③ telemetry → ④ pre-sync → ⑤ resume
Coordinator
Provider Node A
P2 Destination Selector
Peer Providers + Campus Topology
Training Job + GPU state (RTX 3090) (model · optim · activations)
Feasibility: vram≥r, Tmig ≤τ
⑤ Resume
Provider B (A100 ×2) P1 · eBPF · TC BPF · NIC
P1 Adaptive-Ckpt scheduler
Tmig
Survival Sd =exp(−∫0 hd )
① P2 select
̂ Tick: Δt * = √ 2C/(λ̂ B)
Provider C (RTX 4090) P1 · eBPF · TC BPF · NIC
Topology score (Thm 4.2)
ClipAndRedistribute (NIC · admin)
P3 Admit + BW Allocator Deadline-feasible admission Cube-root
bi ∝(Ci2 λi)1/3
NVMe ckpt cache (incremental · LRU)
(Thm 3.4)
Building-1 Switch (10 GbE, P2 prefers)
eBPF probes (kernel) P3 admit ③② Telemetry
Trace estimators: λ̂ e(t), B̂ eff(t)
Sniffer λe · BW B · queue depth
P3 TC BPF (egress)
④ Ckpt sync
Campus Backbone (10 G, hier.)
HTB · Bmin reserve · token bucket Kill-switch ( <1\,ms isolation)
Building-2 Switch + remote providers
NIC (1G / 10G)
Control (mTLS)
Telemetry (eBPF)
Admission/Alloc
Ckpt data
Resume signal
Intra-day Emergency Departure Rate
Checkpoint Payload Size CDF 1.00
±1 std dev λe(t)
0.6
0.75
Static Young/Daly mean (λ)̄
0.4
CDF
Departure rate λe (h−1)
Fig. 1: ReclaimNet architecture: coordinator (P2 selection + P3 admission), provider node (P1 Adaptive-Ckpt with eBPF/TC BPF enforcement), and topology-aware peer placement.
0.2
0.50
ResNet-152 BERT-Large GPT-2 Stable Diff. Median 3.2 GB
0.25
0.0 0
4
8
12
16
20
Hour of day
(a) λe (t) over two months.
24
0.00 102
103
104
Checkpoint size (MB)
(b) Checkpoint payload CDF.
Fig. 2: Measurement findings: (a) intra-day reclaim hazard and (b) checkpoint-size spread motivate adaptive intervals and topology-aware destination selection.
CVλ = 0.62, CV1/B = 0.31. This violates the independence assumption underlying classical periodic-checkpoint analyses [6], [7], which optimize against E[λ] and E[B] separately and under-estimate the cost of long intervals when λ and 1/B co-vary positively. P1 exploits this joint structure as its core algorithmic increment over fixed-interval schedules (§V-D, Thm. 2). Together, F1–F4 plus the observation of up to 7 simultaneous departures drive the three protocol designs: F1’s λe (t) dynamics motivate P1’s time-varying interval; F2’s payload spread motivates P2’s topology-aware selector; concurrent departures motivate P3’s concurrency control. V. P1: R ECLAIM -AWARE C HECKPOINT S CHEDULING
workload of small models, PEFT/LoRA fine-tuning, and fullstate training (Fig. 2b); 1 Gbps-link transmission ranges from <2 s to >6 min, showing that a topology-unaware selector routing across the 10 Gbps core can be slower than a samebuilding hop when the core is loaded. F3: Bandwidth is highly dynamic. eBPF measurements show intra-building Blocal averaging 820 Mbps versus corecrossing Bcore 240 Mbps at peak (r ≈ 3.4); per-link B(t) fluctuates by ±60% over 10-minute windows, requiring realtime measurement for accurate cost prediction in both P1 and P2. F4: Hazard and bandwidth are anti-correlated. Peak reclaim hours (lunch, evening) coincide with the worst campus bandwidth because human activity drives both: ϱλ,B = corr(λe , Beff ) = −0.43 (95% bootstrap CI [−0.51, −0.34], n=6048 buckets), with corr(λe , 1/Beff ) = +0.39 and CVs
A. Reclaim-Aware System Model P1 separates the two reclaim modes. Emergency reclaim (τ = 0) cannot rely on the source remaining available, so recovery uses the freshest checkpoint that has already completed. Scheduled reclaim (τ > 0) first tests final checkpoint feasibility: a zero-loss handoff is possible only if C/Beff (t) + Tr ≤ τ , where C is the checkpoint payload size, Tr is restart time, and Beff (t) = min(W, B(t)) is the effective transfer throughput from local write speed W and measured network bottleneck B(t). For emergency protection, P1 treats λe (t) and Beff (t) as piecewise-stationary over one control epoch. This is the operational assumption behind the closed form: the local interval is recomputed at each checkpoint, not a claim that a single interval is globally optimal for an arbitrary non-homogeneous
process. The per-unit-time expected slowdown cost of a checkpoint interval ∆t is: ∆t L(∆t, t) = λe (t) · + | {z 2} expected work loss
C Beff (t) · ∆t | {z }
(1)
checkpoint I/O overhead
The first term grows linearly in ∆t (more work lost per departure when intervals are long) and the second decays as 1/∆t (checkpointing overhead amortized over longer intervals). B. Local Checkpoint Interval Lemma 1 (Local Reclaim-Aware Optimal Interval). Within a control epoch where λe (t) and Beff (t) are treated as constant estimates, the checkpoint interval that minimizes L(∆t, t) in (1) is: s 2C (2) ∆t∗ (t) = λe (t) · Beff (t) p with optimal cost L∗ = 2λe C/Beff .
The minimized aggregate cost is
p P 2/Bckpt ( i (λi Ci )1/3 )3/2 .
Proof. See Appendix A. Theorem 1 is the P1 protocol rule that is absent from independent per-job timers: checkpoint bandwidth is not split equally, nor linearly by checkpoint size. The cube-root allocation gives more bandwidth to jobs with higher reclaim hazard and larger checkpoints, but with diminishing returns, so a single large or risky job cannot starve the rest. Lemma 2 (Box-Constrained Allocation). Let Wi denote a per-job or administrator cap on bi , and consider p P P write-speed min i 2λi Ci /bi s.t. i bi ≤ Bckpt , 0 < bi ≤ Wi . The following water-filling iteration terminates in at most K steps and returns the global optimum: (1) solve the unconstrained problem via (4); (2) for every job whose unconstrained b∗i > Wi , fix bi = Wi , subtract Wi from Bckpt , and remove i from the active set; (3) re-solve over the residual active set and residual budget; repeat until no cap is violated.
Proof. See Appendix A.
Proof sketch. See Appendix A.
Lemma 1 is the local control law inside P1. It generalizes the classical periodic-checkpoint formula [6] to the dynamic network-constrained setting: Beff (t) replaces the implicit constant write-speed, and λe (t) is re-estimated from observed reclaim behavior rather than assumed as a stationary hardware failure rate.
If a local write cap Wi or an administrator cap is binding, P1 invokes Lemma 2; the same constrained clipping used in Eq. (9) below also enforces loss budgets and scheduledreclaim deadlines.
C. Network-Coupled Checkpoint Bandwidth Allocation The local rule is insufficient when multiple jobs share a campus bottleneck: if K checkpoints start together, the bandwidth each job sees is an allocation decision rather than an exogenous constant. P1 therefore solves a per-epoch bandwidth-allocation problem before instantiating the intervals in Lemma 1.PLet bi be the checkpoint bandwidth assigned to job i, with i bi ≤ Bckpt , where Bckpt is the checkpoint bandwidth budget after P3’s research-traffic reservation. For job i with emergency reclaim hazard λi and checkpoint payload Ci , the local cost under assigned bandwidth bi is: λ i ∆i Ci + . (3) 2 bi ∆ i Theorem 1 (Network-Coupled Checkpoint Allocation). Within a control epoch, assume K active checkpointing jobs share a checkpoint bandwidth budget Bckpt and that no local write-speed cap is binding. The allocation and intervals that minimize Li (∆i , bi ) =
K X
Li (∆i , bi )
s.t.
i=1
are:
K X
bi ≤ Bckpt , bi > 0
i=1
(λi Ci )1/3 , b∗i = Bckpt PK 1/3 j=1 (λj Cj )
and
s ∆∗i =
2Ci . λi b∗i
(4)
(5)
D. Adaptivity Gap of State-Dependent Checkpointing Let (Λ, Θ) denote the joint stationary distribution of (λe (t), 1/Beff (t)) over an operational horizon, with λe , Beff > 0 a.s. and finite first moments. Define two policy classes: statedependent πdep samples the current (λe , B) at each epoch and applies Lemma 1’s closed form (in multi-workload epochs, P1 first applies Theorem 1 and treats the assigned bi as the job’s effective bandwidth); state-independent πind commits to a single ∆t0 chosen optimally from the marginals (λ̄e , 1/B), the best a stationary periodic-checkpoint analysis can achieve, even with full distributional knowledge, because its cost model factors through E[λ] and E[1/B] only. p Theorem 2 (Adaptivity Gap). Let W := λe /Beff . The expected per-unit-time cost ratio satisfies p E[λe ] E[1/Beff ] E[L∗ind ] p ≥ 1, (6) = ∗ E[Ldep ] E λe /Beff where the numerator is the square root of the product E[λe ] E[1/Beff ] and the denominator is the expectation of the p random variable W := λe /B ; the √eff p inequality is exactly the Cauchy–Schwarz bound on ( λe , 1/Beff ) in L2 (P). with equality iff λe Beff is a.s. constant. Equivalently, the absolute gap is √ ∆CS E[L∗ind ] − E[L∗dep ] = 2C · √ , (7) λ̄ Θ̄ + E[W ] where := E[1/Beff ] and the L2 (P) Cauchy–Schwarz defect √ Θ̄ p of ( λe , 1/Beff ) is 2 ∆CS := E[λe ] · E[1/Beff ] − E[W ] ≥ 0. (8)
Proof. See Appendix A.
Algorithm 1 A DAPTIVE -C KPT: network-coupled P1 controller
Interpretation. A stationary fixed-interval analysis treats λ and B as scalar nominal values, so ∆CS ≡ 0 by construction and the adaptivity gap is structurally invisible. Theorem 2 expresses the benefit of state-dependent interval selection as a directly measurable functional (λ̄, Θ̄, E[W ]) of the deployment trace, observable from any deployment that estimates λe and Beff online. Plugging the two-month calibration statistics into (6) predicts a ∼5.2% gap, which we verify empirically against baseline B7 in Table III. A detailed binding-regime analysis (B-binding vs. W-binding epochs) appears in Appendix C.
− Require: active jobs J with (Ci , Lmax,i , λ̂i ); link budget B̂ckpt (t); reclaim notice τi 1: procedure PT ICK(t) 1/3 2: S ← j∈J (λ̂+ ▷ cube-root coupling, Thm. 1 j (t) Cj ) 3: for all i ∈ J do − 1/3 4: bi ← B̂ckpt (t) · (λ̂+ /S i (t) Ci ) 5: bi ← C LIPA ND R EDISTRIBUTE(bi ) ▷ NIC & admin caps 6: if τi ̸= ⊥ ∧ Ci /bi + Tr,i ≤ τi then 7: emit final-ckpt(i); continue 8: end if q Ci 9: ∆i ← clip 2Ci /(λ̂+ (t) b ), , 2L i max,i i βbi
E. Constrained Solution and Online Algorithm Three reclaim-induced constraints bound the unconstrained optimum after Theorem 1 assigns checkpoint bandwidth bi : network feasibility (∆i ≥ Ci /(βbi ), β=0.2; see below), loss budget (∆i ≤ 2Lmax,i ), and scheduled-reclaim deadline (Ci /bi + Tr,i ≤ τi for notice τi ). The constrained per-job solution is s ! Ci 2Ci ∗ , , 2Lmax,i . (9) ∆i,c (t) = clip λi (t)bi βbi Operational meaning of β. The floor Ci /(βbi ) enforces a budget split on the allocated egress slice bi : at most a fraction β of bi is consumed by checkpoint writes, leaving (1 − β)bi for live gradient/activation traffic of co-located training jobs. We set β = 0.2 so that checkpoint I/O never steals more than 20% of the slice, keeping all-reduce tail latency within the SLO window observed in §IV (F2–F3). Smaller β throttles checkpoint frequency (raising recompute loss after preemption); larger β inflates communication tail and violates fairness. The choice is robust: a sensitivity sweep over β ∈ {0.1, 0.2, 0.3} in Appendix B changes end-to-end goodput by < 2%, with β = 0.2 giving the best loss–throughput tradeoff. The online controller A DAPTIVE -C KPT (Algorithm 1) estimates λe (t) via sliding-window MLE over the departure log and samples B(t) from eBPF egress counters at 100 ms intervals; deadline checks use a lower-confidence bound B̂ − . F. Multi-Workload Coordination When K jobs are co-resident, independent checkpoint timers can synchronize and collapse per-job bandwidth to roughly B(t)/K ′ (K ′ ≤ K active transferrers). Theorem 1 prevents this by converting the shared bottleneck into per-job rate demands before timers fire. P1 exports each checkpoint as a demand tuple (i, Ci , bi , ∆i , Lmax,i ) to P3, which staggers non-urgent checkpoints, reserves deadline bandwidth for scheduled reclaim events, and rejects infeasible overload sets into degraded recovery (§VII). This allocation-plus-admission path is the protocol difference from independent fixed-interval timers.
10: export (i, Ci , bi , ∆i , Lmax,i ) → P3 11: end for 12: end procedure 13: loop every control period: T ICK(t) 14: end loop
VI. P2: VOLATILITY-AWARE D ESTINATION S ELECTION A. Network Model We model the campus network as a weighted graph G = (V, E, w), where V includes both servers and switches, and w(e, t) is the available bandwidth on link e at time t, measured passively by eBPF TC BPF programs installed on each provider node. For a scheduled reclaim at source s with notice window τs , destination selection has a hard feasibility constraint in addition to a cost objective. The migration transfer time from source s to destination d is: C Tmig (s, d, t) = + Tr (10) bwbot (s, d, t) where bwbot (s, d, t) = mine∈P (s,d) w(e, t) is the path bottleneck and Tr is the container restart time. A destination is deadline-feasible only if Tmig (s, d, t) ≤ τs ; otherwise P2 excludes it for zero-loss handoff and marks the job for degraded recovery from the latest completed checkpoint. B. Joint Scoring Function Minimizing only Tmig is myopic: in a VVN, the destination is itself an autonomous provider. If d has high future reclaim hazard, the workload will re-migrate shortly, incurring additional overhead. P2 therefore combines path cost, deadline feasibility, and destination survival. Let Sd (Trem ) = R t+T exp(− t rem hd (u) du) be the probability that destination d survives for the job’s remaining runtime under estimated hazard hd . We minimize the one-step lookahead cost: Score(d) = Tmig (s, d, t) + α · (1 − Sd (Trem )) · T̄mig (d) (11) where Trem is the estimated remaining job runtime, T̄mig (d) is the average migration cost from d to other candidates, and α ∈ [0.5, 2.0] is a (dimensionless) stability penalty weight. We calibrate α once per deployment by minimizing the held-out work loss of the P1+P2+P3 stack on weeks 1– 2 of the trace over the grid {0.5, 1.0, 1.5, 2.0}; the empirical
Algorithm 2 T OPOAWARE -S ELECT: P2 destination scoring
minimum is α = 1.0 on our deployment. Re-calibration is only needed when the ratio Blocal /Bcore or the inter-building hazard heterogeneity changes by more than ∼30%, an event we did not observe in the held-out period (§IX-E8). The optimal destination is: d∗ = arg mind∈Dc (s) Score(d), where Dc (s) is the set of compatible candidates satisfying VRAM, CUDA capability, load, and deadline feasibility constraints. C. Locality Filtering Lemma Lemma 3 (Hierarchical Locality Filter). Assume a hierarchical campus window in which every same-building feasible − and every crosspath from s has bottleneck at least Blocal − + + . building path has bottleneck at most Bcore , with Blocal ≥ Bcore If a same-building candidate satisfies GPU and deadline constraints, then the best same-building candidate has transfer time no larger than the best cross-building candidate for the current migration. Proof sketch. See Appendix A. With a measured median ratio Blocal /Bcore ≈ 3.4 in §IV, the lemma justifies using locality as a candidate filter rather than a final decision rule. The technical increment over conventional topology-aware scheduling is the subsequent survival and deadline scoring: P2 will reject a local node that is unstable or cannot receive the checkpoint before τs . D. Algorithm: TopoAware-Select P2’s destination selector (Algorithm 2) first applies the feasibility filter (VRAM, capability, load, deadline), then promotes same-building candidates via Lemma 3 (falling back to the cross-building set when fewer than kmin local options exist), and finally scores each remaining destination by σd = C/bwd + Tr + α(1 − Sd ) T̄mig (d) where Sd is the survival probability over the remaining runtime. The total cost is O(|Dcand | · L) with L ≤ 3 (three-tier hop count)— negligible compared to the transfer itself. Bandwidth estimates are refreshed from the shared eBPF measurement table on each invocation.
Migration Downtime CDF (E2) 1.00 0.75
CDF
Require: src s; spec (C, r, cmin ); remaining runtime Trem ; notice τs ; penalty α Ensure: destination d∗ or FAIL 1: function S ELECT(s, C, r, cmin , Trem , τs ) 2: Dc ← {d ̸= s : vram(d) ≥ r, cap(d) ≥ cmin , load(d) ≤ θ, C/bw(s, d) + Tr ≤ τs } 3: if Dc = ∅ then return FAIL 4: end if 5: Dℓ ← {d ∈ Dc : same-bldg(s, d)} 6: D ← Dℓ if |Dℓ | ≥ kmin else Dc 7: for all d ∈ D do 8: bwd ← E BPF-B OTTLENECK(s, d) 9: Sd ← S URVIVAL(d, Trem ) 10: σd ← C/bwd + Tr + α(1 − Sd ) T̄mig (d) 11: end for 12: return arg mind∈D σd 13: end function
ReclaimNet Bamboo Rand-Dst Slurm-PR No-Mig.
0.50 0.25 0.00 0
50
100
150
200
250
300
Migration downtime (s)
Fig. 3: Per-migration downtime CDF: ReclaimNet cuts median by 38% and p99 by 52% versus Random-Dst.
VII. P3: D EADLINE -AWARE M IGRATION T RAFFIC S CHEDULING A. Traffic Classification P3 classifies all traffic into four priority classes using DSCP markings applied by TC BPF programs at the egress of each provider node: • Phi (Emergency migration): Emergency reclaim with τ = 0 cannot assume the source remains available long enough to send new state. The job resumes from the freshest completed checkpoint; only small metadata/notification traffic is marked DSCP EF (46). If a provider grants a short grace period, P3 treats it as an urgent planned migration with a small τ . • Pmed (Planned migration): Provider departure with notice period τ > 0; subject to deadline-aware admission and allocation, marked DSCP AF41 (34). • Plo (Pre-sync): Incremental background presynchronization from Algorithm 1; marked DSCP CS1 (8). • Pbg (Research traffic): All other flows (SSH, Jupyter, data downloads); marked Best Effort (0). B. Admission and Bandwidth Allocation Total available bandwidth B is split into a migration share Bmig = B − Bres and a reserved share for research traffic: Bres = max(βB, Bmin ), with β = 0.3 calibrated from measurements and Bmin set by the campus administrator. For each planned migration k, P3 converts the provider’s notice period into a minimum bandwidth demand: Ck . (12) τk − Tr,k P min The set is fully feasible when ≤ Bmig . In that k rk case, P3 admits all flows, assigns each rkmin , and distributes remaining bandwidth with max-min water filling. When the set is infeasible, P3 admits flows by earliest deadline first subject to the same research-traffic reservation, and marks the rest for degraded recovery from their freshest completed rkmin =
A TC BPF program attached to each node’s egress interface implements per-flow token-bucket rate limiting using BPF_MAP_TYPE_HASH. Each bucket entry stores the allocated rate (bytes/s), token count, and last-refill timestamp. On each packet, tokens are replenished proportionally to elapsed time; packets are forwarded if the token count exceeds packet length, otherwise dropped (TCP’s congestion control handles retransmission, achieving smooth rate enforcement without kernel scheduler changes). D. Theorems 3 and 4: Formal Guarantees Theorem 3 (Migration Completion Guarantee). For an admitted planned migration k with notice τk > Tr,k , if P3 assigns rate rk ≥ Ck /(τk −Tr,k ) and the controlled edge path sustains that rate, then the checkpoint transfer and restart complete before provider departure. Proof. See Appendix A. Theorem 4 (Research Traffic Isolation on Controlled Access Bottleneck). Consider a single access-layer bottleneck of capacity B whose upstream traffic originates entirely from provider nodes running the ReclaimNet agent. Assume all migration flows crossing this bottleneck are classified by P3 and enforced at participating provider egress points. Then P3 bounds aggregate migration traffic on the bottleneck to at most B − Bmin , leaving at least Bmin capacity for non-migration traffic on that bottleneck. Proof. See Appendix A. Scope of the guarantee. Theorem 4 applies per access bottleneck and only over flows whose source egresses are controlled by a ReclaimNet agent. It does not extend to distribution- or core-layer links shared with non-participating tenants, and partial deployments (a subset of providers without P3) weaken the guarantee proportionally to the uncontrolled migration share. In our 54-node deployment all providers run the agent; we discuss the partial-deployment regime in §X. E. Kill-Switch Network Enforcement When a provider activates the kill-switch, the provider agent sets a flag in a BPF array map. A TC BPF ingress program checks this flag on every inbound packet: new SYN segments are dropped immediately, isolating the node from new work; in-flight migration transfers (already past SYN) are allowed to complete only when the provider selected scheduled reclaim or a grace period, preserving checkpoint integrity without
Completion time (s)
C. eBPF Implementation
Migration Completion vs. K 0 (E3) 300
ReclaimNet (P3) No-TC-BPF Slurm-PR (B10)
200 100 0 1
2
3
4
5
6
7
8
9
10
Concurrent migrations K 0
Fig. 4: Migration completion time vs. concurrent migrations K ′ : P3 keeps admitted flows within their notice windows while baselines degrade linearly. Research throughput (Mbps)
checkpoints. This admission step is the protocol component; the token bucket only enforces the selected rates. Rates are written to a per-flow BPF hash map consumed by the TC BPF token-bucket program. Staggered scheduling offsets the start of planned flows by δ = Tnotice /K ′ each to eliminate synchronized TCP slow-start bursts.
Research Traffic vs. K 0 (E3) 800 600 400
ReclaimNet (P3) Slurm-PR (B10) No-TC-BPF Bmin (300 Mbps)
200 0 1
2
3
4
5
6
7
8
9
10
Concurrent migrations K 0
Fig. 5: Research traffic throughput vs. K ′ : P3 sustains the Bmin reserve while No-TC-BPF collapses to near zero. blocking the provider’s local reclaim decision. End-to-end killswitch latency is under 1 ms—two to three orders of magnitude lower than application-layer enforcement paths (∼100 ms) and iptables (∼10 ms). VIII. I MPLEMENTATION ReclaimNet is implemented in ∼5,400 LoC across three layers: node agents (∼1,500 LoC C++17) drive container lifecycle, checkpoint triggers, P2 selection, and BPF-map updates; eBPF programs (∼800 LoC C, Clang 17 + libbpf) realise the P3 TC BPF token bucket and the kill-switch; and the coordinator (∼2,000 LoC C++17) hosts P3 admission/allocation, the P2 topology database, and P1’s departurerate estimator. mTLS-1.3 with short-lived coordinator-issued JWTs authenticates all control traffic; A100/A6000 nodes use MIG hardware partitioning while RTX 3090/4090 nodes zero VRAM pages on container teardown. A full software-stack and security-mechanism description appears in Appendix D. IX. E VALUATION A. Testbed and Baselines We evaluate on the 54-node testbed of §IV, comparing ReclaimNet with seven online baselines and an offline lower
cum. work loss (GPU-h)
(a) Vision (ResNet-50)
(b) NLP (BERT-Large)
(c) LLM-7B
(d) LLM-13B
10
7.5 5.0
5
5.0
5
5
24
48
72
0
24
time (h)
48
72
0
24
time (h)
ReclaimNet
2.5
0
0
0 0
7.5
10
2.5 0.0
48
72
0.0 0
24
time (h)
Bamboo (B1)
(e) RL (PPO)
Slurm-PR (B1')
48
72
0
24
time (h)
Static (B3)
48
72
time (h)
Time-Varying YD
Oracle (B6)
(a) Cumulative work loss across five representative job classes (72-h slice of the two-month deployment): (a) Vision (ResNet-50, ∼120 MB dense state), (b) NLP (BERT-Large, ∼340 MB), (c) LLM-7B with LoRA fine-tuning (∼720 MB adapter+optimizer), (d) LLM-13B with LoRA (∼1.2 GB), (e) RL (PPO, ∼210 MB). Full-state LLM training (multi-GB to 48 GB payloads, cf. Fig. 2b) is shown in the Appendix B stress-test panel. ReclaimNet tracks the Oracle (B6) lower bound within 8–12% on every job class. (b) Morning ramp 08:00–12:00
downtime (s)
(a) Off-peak 03:00–07:00 200
(c) Midday peak 12:00–16:00
200
200
200
150 100
100
(e) Evening peak 18:00–22:00
(d) Afternoon 14:00–18:00
200
100
100
100
50 0
1
2
3
4
0
1
time (h)
2
3
4
0
1
time (h) ReclaimNet
2
3
4
0
1
time (h) Random-Dst (B2)
2
3
4
0
1
time (h)
No-Topo (B4)
2
3
4
time (h)
No-TC-BPF (B5)
(b) Per-migration downtime across five reclaim-intensity regimes drawn from the two-month trace: (a) Off-peak (03:00–07:00), (b) Morning ramp (08:00–12:00), (c) Midday peak (12:00–16:00), (d) Afternoon (14:00–18:00), (e) Evening peak (18:00–22:00). ReclaimNet keeps median downtime within ±10% of off-peak even at evening peak.
200
5
(d) traffic degrad.
drift
drift
20
150
%
seconds
GPU-h
10
(c) success rate 100
drift
80
10
100 60
2
4
6
week
8 10
drift
0
50 0
(e) hazard est. err
|λ̂ e − λe|/λe
(b) median downtime
drift
%
(a) cum. work loss
0
2
4
6
8 10
0
2
week ReclaimNet
4
6
8 10
week Static (B3)
Time-Varying YD
0.4 0.2 0.0
0
2
4
6
8 10
week
0
2
4
6
8 10
week
Oracle (B6)
(c) Multi-metric robustness across the two-month main deployment plus a held-out two-week drift period (dashed line marks drift onset): (a) cumulative work loss, (b) median downtime, (c) success rate, (d) research-traffic degradation, (e) hazard-rate estimation error |λ̂e −λe |/λe . ReclaimNet recovers within ∼18 min; baselines stay elevated.
Fig. 6: End-to-end evaluation panels organised as multi-scenario sweeps. Top: work-loss sensitivity to job class. Middle: downtime sensitivity to reclaim intensity. Bottom: multi-metric robustness to distribution drift. Shared method palette across all three rows.
bound: Bamboo [8] (B1), Slurm-PR with DMTCP 10-min checkpoints (B1′ ), No-Migration (B2), trace-wide Static-Ckpt (B3, fixed periodic interval), Random-Dst (B4), No-TC-BPF (B5), state-independent Time-Varying Fixed-Interval (B7), and Oracle (B6). B4 and B5 remove topology-aware destination selection and TC/eBPF isolation, respectively; B7 keeps P1’s estimators, the cube-root allocator (Thm. 1), and P2/P3, but
fixes one ∆t per epoch; B6 solves an offline CPLEX ILP over 24-h windows. Full parameters and the ILP formulation appear in Appendix C. The two-month run comprises 312 GPU-training jobs (BERT-Large, ResNet-152, GPT-2, Stable Diffusion), totaling 4,890 GPU-hours, and 847 departure events (421 natural reclaims and 426 empirical injections shared by all baselines).
System
Work Loss ↓ (GPU-h)
Downtime (s) ↓ (median)
Mig. Succ. % ↑
Traffic Degr. % ↓
GPU Util. % ↑
Slurm-PR (B1′ ) No-Mig. (B2) Static (B3) Bamboo (B1) Rand-Dst (B4) No-BPF (B5) TV-FI (B7)
8.2 18.4 7.6 4.5 3.8 3.1 3.0
195 — 127 75 98 71 64
64.5 0 71.3 85.4 84.2 88.9 90.4
26.4 0.0 18.4 8.2 7.3 31.0 2.9
38.1 31.2 46.8 58.7 59.1 65.3 67.5
ReclaimNet Oracle (B6)
2.8 2.6
61 53
91.3 94.1
2.8 0.0
68.7 71.4
Natural-only results closely match aggregate results—2.6 vs. 2.8 GPU-h work loss, 58 vs. 61 s median downtime, 2.7% vs. 2.8% traffic degradation, and a 6.4% vs. 6.7% B7 gap—with unchanged baseline ordering. ReclaimNet reaches within 7.7% of Oracle on work loss while providing the only combination of > 91% migration success and < 3% research traffic degradation; the per-class, perregime, and drift breakdowns in Fig. 6 (subfigures 6a, 6b, 6c) confirm these aggregates are not driven by a single workload or operating regime. B. E1: End-to-End Work Loss (Gap 1) ReclaimNet reduces average work loss by 66% over SlurmPR (B1′ ) and 38% over Bamboo (B1), while remaining within 8% of the Oracle lower bound (B6). Its 6.7% gain over Time-Varying Fixed-Interval (B7) isolates the benefit of statedependent interval selection and is consistent with the ∼5.2% adaptivity gap predicted from ∆CS (Thm. 2). The residual gap is explained by diurnal non-stationarity: P1 re-estimates online, whereas B7 commits to an epoch-wide interval and consequently shows higher downtime and lower migration success during peak hours. Ablation (Table IV) attributes 64% of the work-loss reduction to P1; P2+P3 recover the remainder by preventing failed migrations. P1 shortens checkpoint intervals to 4–6 min under high reclaim risk and relaxes them during low-risk periods to reduce network load. All differences are significant on the 847-event population (paired Wilcoxon, p < 0.01); the two-month utilization trace in Fig. 7 visualises the headline deployment gain. C. E2–E4: Downtime, Traffic Impact, Kill-Switch E2 Downtime (Gap 2). P2 reduces median migration downtime by 38% and p99 downtime by 52% relative to RandomDst (B4) (Fig. 3). The gain comes from preferring samebuilding targets when feasible (71% of migrations; 821 Mbps intra-building vs. 243 Mbps cross-building) and falling back only when locality violates volatility or notice constraints. E3 Traffic impact (Gap 3). Under peak concurrent migrations (up to seven events) with a co-running reference workload, P3 holds research-traffic degradation to 2.8%, compared with 31% for No-TC-BPF (B5) (Figs. 4 and 5). Its admission and rate
GPU Utilization Time Series (E1/E4)
GPU utilization (%)
TABLE III: End-to-end evaluation results (two-month experiment, 847 departure events). ↓ lower is better; ↑ higher is better.
80
ReclaimNet Bamboo (B1) No-Sharing Exam / high departure
60 40 20 0 0
10
20
30
40
50
60
Day
Fig. 7: Two-month GPU utilization: ReclaimNet sustains ∼68.7%, +10 pp over Bamboo and +37.5 pp over no-sharing.
enforcement reduce the unconstrained burst from ∼6.8 Gbps to ≤4.2 Gbps, preserving ≥30% headroom. E4 Kill-switch latency. TC BPF enforcement cuts reclaim blocking latency to sub-millisecond scale (median 0.6 ms, p99 0.9 ms), over 100× faster than application-layer kill paths; fewer than 500 B of new work can reach an evicting node after the signal. D. E5–E8: Scalability, Sensitivity, Ablation, Drift E5 Scalability. Mininet emulation to 500 nodes keeps coordinator runtime below 18 ms, far under the 10-second notice budget (Appendix B, Fig. 8). E6 Sensitivity. Sweeping βp1 , α, βp3 on the two-month trace changes work loss/downtime by at most ±6.4% and traffic degradation by ±1.1 pp; the defaults remain near the empirical minima (Fig. 9). E7 Ablation. The per-protocol study (Table IV) confirms orthogonal effects: P1 dominates work-loss reduction, P2 halves median downtime (142 s to 71 s), and P3 reduces traffic degradation from 31.0% to 2.8%. E8 Drift. Under a held-out two-week deadline-crunch period with a 1.7× hazard shift, P1 reconverges within 18 min; work loss stays within 7% of the in-distribution mean (95% bootstrap CI [2.7, 3.3] GPU-h, p = 0.21). X. D ISCUSSION AND C ONCLUSION Limitations and scope. ReclaimNet relies on applicationlevel checkpointing (PyTorch, JAX hooks) and does not cover compiled or legacy GPU workloads; CRIU-based CPU-state checkpointing with GPU hooks is future work. The VVN model assumes trusted providers; untrusted wide-area settings need additional authentication and sandboxing. Partial deployment. Theorem 4 holds only over the ReclaimNet-controlled migration share: with participation ϕ ∈ (0, 1] and uncontrolled burst ū, the guarantee degrades to B −Bmin −(1−ϕ)ū. Restoring it requires rate-capping uncontrolled tenants or provisioning B ≥ Bmin + Bmig + (1 − ϕ)ū. Coordinator failure is handled by leader election with at-mostonce checkpoint-completion semantics. Conclusion. ReclaimNet couples reclaim-aware checkpointing (P1), volatility/deadline-aware destination selection (P2),
and notice-driven migration admission with TC BPF enforcement (P3). A 54-node campus testbed shows that provider autonomy and system reliability coexist with lightweight coordination and no switch reconfiguration. Future work targets joint P1/P2 optimization, departure-forecast preemptive migration, and RDMA checkpoint transfer. R EFERENCES [1] Cloud Native Computing Foundation, “Kubernetes,” https://kubernetes. io, 2014, accessed: 2026-05-23. [2] A. B. Yoo, M. A. Jette, and M. Grondona, “SLURM: Simple linux utility for resource management,” in Workshop on Job Scheduling Strategies for Parallel Processing. Springer, 2003, pp. 44–60. [3] D. P. Anderson, J. Cobb, E. Korpela, M. Lebofsky, and D. Werthimer, “SETI@home: An experiment in public-resource computing,” Communications of the ACM, vol. 45, no. 11, pp. 56–61, 2002. [4] V. S. Pande, I. Baker, J. Chapman et al., “Atomistic protein folding simulations on the submillisecond time scale using worldwide distributed computing,” Biopolymers, vol. 68, no. 1, pp. 91–109, 2003. [5] Y. Li, Y. Zhang, H. Liao, D. Guo, and G. Tang, “GPUnion: Autonomous GPU sharing on campus,” in Proceedings of the 24th ACM Workshop on Hot Topics in Networks (HotNets), 2025. [6] J. W. Young, “A first order approximation to the optimum checkpoint interval,” Communications of the ACM, vol. 17, no. 9, pp. 530–531, 1974. [7] J. T. Daly, “A higher order estimate of the optimum checkpoint interval for restart dumps,” Future Generation Computer Systems, vol. 22, no. 3, pp. 303–312, 2006. [8] J. Thorpe, P. Zhao, J. Eyolfson, Y. Qiao, Z. Jia, M. Zhang, R. Netravali, and G. H. Xu, “Bamboo: Making preemptible instances resilient for affordable training of large DNNs,” in Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’23). USENIX Association, 2023, pp. 497–513. [9] W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, N. Kwatra, Z. Han, P. Patel, X. Peng, H. Zhao, Q. Zhang, F. Yang, and L. Zhou, “Gandiva: Introspective cluster scheduling for deep learning,” in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18). USENIX Association, 2018, pp. 595–610. [10] M. J. Litzkow, M. Livny, and M. W. Mutka, “Condor—a hunter of idle workstations,” in Proceedings of the 8th International Conference on Distributed Computing Systems (ICDCS). IEEE, 1988, pp. 104–111. [11] D. Thain, T. Tannenbaum, and M. Livny, “Distributed computing in practice: The Condor experience,” Concurrency and Computation: Practice and Experience, vol. 17, no. 2–4, pp. 323–356, 2005. [12] D. P. Anderson, “BOINC: A system for public-resource computing and storage,” in Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing (GRID ’04). IEEE, 2004, pp. 4–10. [13] G. Bosilca, A. Bouteiller, F. Cappello, S. Djilali, G. Fedak, C. Germain, T. Hérault, P. Lemarinier, O. Lodygensky, F. Magniette, V. Neri, and A. Selikhov, “MPICH-V: Toward a scalable fault tolerant MPI for volatile nodes,” in Proceedings of the ACM/IEEE Conference on Supercomputing (SC ’02), 2002, pp. 29:1–29:18. [14] A. Moody, G. Bronevetsky, K. Mohror, and B. R. de Supinski, “Design, modeling, and evaluation of a scalable multi-level checkpointing system,” in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’10). IEEE, 2010, pp. 1–11. [15] L. Bautista-Gomez, S. Tsuboi, D. Komatitsch, F. Cappello, N. Maruyama, and S. Matsuoka, “FTI: High performance fault tolerance interface for hybrid systems,” in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’11), 2011, pp. 32:1–32:32. [16] J. S. Plank, M. Beck, G. Kingsley, and K. Li, “Libckpt: Transparent checkpointing under Unix,” in Proceedings of the USENIX 1995 Technical Conference. USENIX Association, 1995, pp. 213–223. [17] P. H. Hargrove and J. C. Duell, “Berkeley lab checkpoint/restart (BLCR) for Linux clusters,” Journal of Physics: Conference Series, vol. 46, pp. 494–499, 2006. [18] J. Ansel, K. Arya, and G. Cooperman, “DMTCP: Transparent checkpointing for cluster computations and the desktop,” in Proceedings of the 23rd IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2009, pp. 1–12.
[19] B. Schroeder and G. A. Gibson, “Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?” in Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST ’07). USENIX Association, 2007, pp. 1–16. [20] E. Pinheiro, W.-D. Weber, and L. A. Barroso, “Failure trends in a large disk drive population,” in Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST ’07). USENIX Association, 2007, pp. 17–28. [21] B. Schroeder, E. Pinheiro, and W.-D. Weber, “DRAM errors in the wild: A large-scale field study,” in Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems (SIGMETRICS ’09). ACM, 2009, pp. 193–204. [22] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, “Live migration of virtual machines,” in Proceedings of the 2nd USENIX Symposium on Networked Systems Design and Implementation (NSDI ’05). USENIX Association, 2005, pp. 273–286. [23] M. R. Hines, U. Deshpande, and K. Gopalan, “Post-copy live migration of virtual machines,” ACM SIGOPS Operating Systems Review, vol. 43, no. 3, pp. 14–26, 2009. [24] T. Wood, K. K. Ramakrishnan, P. Shenoy, and J. van der Merwe, “CloudNet: Dynamic pooling of cloud resources by live WAN migration of virtual machines,” in Proceedings of the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE ’11). ACM, 2011, pp. 121–132. [25] J. Gu, M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. Liu, and C. Guo, “Tiresias: A GPU cluster manager for distributed deep learning,” in Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19). USENIX Association, 2019, pp. 485–500. [26] K. Mahajan, A. Balasubramanian, A. Singhvi, S. Venkataraman, A. Akella, A. Phanishayee, and S. Chawla, “Themis: Fair and efficient GPU cluster scheduling,” in Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20). USENIX Association, 2020, pp. 289–304. [27] H. Zhao, Z. Han, Z. Yang, Q. Zhang, F. Yang, L. Zhou, M. Yang, F. C. M. Lau, Y. Wang, Y. Xiong, and B. Wang, “HiveD: Sharing a GPU cluster for deep learning with guarantees,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’20). USENIX Association, 2020, pp. 515–532. [28] Y. Peng, Y. Bao, Y. Chen, C. Wu, and C. Guo, “Optimus: An efficient dynamic resource scheduler for deep learning clusters,” in Proceedings of the 13th European Conference on Computer Systems (EuroSys ’18). ACM, 2018, pp. 3:1–3:14. [29] A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing, “Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning,” in Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’21). USENIX Association, 2021, pp. 1–18. [30] W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, and Y. Jia, “AntMan: Dynamic scaling on GPU clusters for deep learning,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’20). USENIX Association, 2020, pp. 533–548. [31] M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, and F. Yang, “Analysis of large-scale multi-tenant GPU clusters for DNN training workloads,” in Proceedings of the 2019 USENIX Annual Technical Conference (ATC ’19). USENIX Association, 2019, pp. 947–960. [32] D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “PipeDream: Generalized pipeline parallelism for DNN training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). ACM, 2019, pp. 1–15. [33] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “GPipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems 32 (NeurIPS), 2019, pp. 103–112. [34] S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “PyTorch distributed: Experiences on accelerating data parallel training,” Proceedings of the VLDB Endowment, vol. 13, no. 12, pp. 3005–3018, 2020. [35] W. Wang, M. Khazraee, Z. Zhong, M. Ghobadi, Z. Jia, D. Mudigere, Y. Zhang, and A. Kewitsch, “TopoOpt: Co-optimizing network topology and parallelization strategy for distributed training jobs,” in Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’23). USENIX Association, 2023, pp. 739–767.
[36] P. Sharma, S. Lee, T. Guo, D. Irwin, and P. Shenoy, “SpotCheck: Designing a derivative IaaS cloud on the spot market,” in Proceedings of the 10th European Conference on Computer Systems (EuroSys ’15). ACM, 2015, pp. 16:1–16:15. [37] A. Harlap, A. Tumanov, A. Chung, G. R. Ganger, and P. B. Gibbons, “Proteus: Agile ML elasticity through tiered reliability in dynamic resource markets,” in Proceedings of the 12th European Conference on Computer Systems (EuroSys ’17). ACM, 2017, pp. 589–604. [38] B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica, “Mesos: A platform for fine-grained resource sharing in the data center,” in Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’11). USENIX Association, 2011, pp. 295–308. [39] V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O’Malley, S. Radia, B. Reed, and E. Baldeschwieler, “Apache Hadoop YARN: Yet another resource negotiator,” in Proceedings of the 4th Annual Symposium on Cloud Computing (SoCC ’13). ACM, 2013, pp. 5:1–5:16. [40] A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at Google with Borg,” in Proceedings of the 10th European Conference on Computer Systems (EuroSys ’15). ACM, 2015, pp. 18:1–18:17. [41] S. McCanne and V. Jacobson, “The BSD packet filter: A new architecture for user-level packet capture,” in Proceedings of the USENIX Winter 1993 Conference. USENIX Association, 1993, pp. 259–269. [42] T. Høiland-Jørgensen, J. D. Brouer, D. Borkmann, J. Fastabend, T. Herbert, D. Ahern, and D. Miller, “The eXpress Data Path: Fast programmable packet processing in the operating system kernel,” in Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (CoNEXT ’18). ACM, 2018, pp. 54–66. [43] S. Jouet and D. P. Pezaros, “BPFabric: Data plane programmability for software defined networks,” in Proceedings of the ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS ’17). IEEE, 2017, pp. 38–48. [44] A. Saeed, N. Dukkipati, V. Valancius, V. T. Lam, C. Contavalli, and A. Vahdat, “Carousel: Scalable traffic shaping at end hosts,” in Proceedings of the ACM SIGCOMM 2017 Conference. ACM, 2017, pp. 404–417. [45] M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan, “Data center TCP (DCTCP),” in Proceedings of the ACM SIGCOMM 2010 Conference. ACM, 2010, pp. 63–74. [46] M. Alizadeh, S. Yang, M. Sharif, S. Katti, N. McKeown, B. Prabhakar, and S. Shenker, “pFabric: Minimal near-optimal datacenter transport,” in Proceedings of the ACM SIGCOMM 2013 Conference. ACM, 2013, pp. 435–446. [47] M. Alizadeh, T. Edsall, S. Dharmapurikar, R. Vaidyanathan, K. Chu, A. Fingerhut, V. T. Lam, F. Matus, R. Pan, N. Yadav, and G. Varghese, “CONGA: Distributed congestion-aware load balancing for datacenters,” in Proceedings of the ACM SIGCOMM 2014 Conference. ACM, 2014, pp. 503–514. [48] W. Bai, L. Chen, K. Chen, D. Han, C. Tian, and H. Wang, “Informationagnostic flow scheduling for commodity data centers,” in Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15). USENIX Association, 2015, pp. 455–468. [49] M. Chowdhury, Y. Zhong, and I. Stoica, “Efficient coflow scheduling with Varys,” in Proceedings of the ACM SIGCOMM 2014 Conference. ACM, 2014, pp. 443–454.
A PPENDIX A P ROOFS OF T HEOREMS AND L EMMAS A. Proof of Lemma 1 (Local Optimal Interval) C ∗ Setting ∂L/∂(∆t) = 0: λ2e − Beff (∆t) = 2 = 0, giving ∆t p 2C 2C/(λe Beff ). The second derivative Beff (∆t)3 > 0 confirms a global minimum. B. Proof of Theorem 1 (Network-Coupled Allocation) ∗ pFor fixed bi , minimizing (3) over ∆i gives ∆i ∗(bi ) = p2Ci /(λi bi ) by Lemma 1, and the resulting cost is Li (bi ) = 2λi Ci /bi . The remaining problem is convex in bi > 0:
P P √ −1/2 s.t. i bi ≤ Bckpt . The bandwidth min{bi } i 2λi Ci bi constraint at optimum. The KKT stationarity condition √ is tight −3/2 1 is −P 2λ C b + µ = 0, so bi ∝ (λi Ci )1/3 . Normalizing i i i 2 by i bi = Bckpt gives (4); substitution gives (5). C. Proof of Lemma 2 (Box-Constrained Allocation) The objective is convex and separable, and the constraint set is a polytope. KKT stationarity together with complementary slackness gives: if a box constraint is inactive at the optimum, the cube-root rule holds among the remaining jobs sharing the residual budget; if a box constraint is active, the corresponding bi = Wi . At every iteration, the active set strictly shrinks: a job is only removed if its unconstrained allocation under the current residual budget exceeds its cap, in which case it is fixed at the cap. Since the residual budget is monotonically non-increasing and the number of jobs is finite, the iteration terminates in at most K rounds. If the procedure halted with some cap still violated, the residual problem would be the unconstrained one whose solution satisfies all remaining caps by construction, a contradiction. The returned point therefore satisfies KKT and, by strict convexity of the objective in each bi > 0, is the unique global minimizer. D. Proof of Theorem 2 (Adaptivity Gap) ∗
optimum at (λe , B) is L√(λe , B) = pThe state-dependent √ 2Cλe /B = 2C ·W (Lemma 1), so E[L∗dep ] = 2C E[W ]. For πind , since the cost rate under a fixed ∆t0 is linear in λe and 1/Beff , the expected cost rate is E[L(∆t0 )] = (∆t p0 /2)λ̄+ ∗ 2C Θ̄/λ̄ (C/∆t0 )Θ̄; minimizing over ∆t yields ∆t = 0 0 √ and E[L∗ind ] = 2C λ̄ Θ̄ (the square root covers the entire 2 product). inequality to p L (P) Cauchy–Schwarz √ Applying the X = λe and Y = 1/Beff gives (E[XY ])2 = (E[W ])2 ≤ E[X 2 ] E[Y 2 ] = λ̄Θ̄, which establishes (6) and (8). Equation √ (7) follows from a − b = (a2 − b2 )/(a + b) with a = λ̄Θ̄ and b = E[W ]. Equality holds iff Y = cX a.s., i.e. λe Beff is a.s. constant. E. Proof of Lemma 3 (Hierarchical Locality) For any same-building destination dl , Tmig (s, dl , t) ≤ − C/Blocal + Tr . For any cross-building destination dx , + Tmig (s, dx , t) ≥ C/Bcore + Tr . The bandwidth separation assumption implies the former is no larger than the latter. The full P2 score still compares survival risk among local candidates and falls back to the global candidate set when too few local nodes are feasible. F. Proof of Theorem 3 (Migration Completion) Transfer time at rate rk is Ck /rk ≤ τk −Tr,k . Adding restart time Tr,k gives total migration time ≤ τk . G. Proof of Theorem 4 (Traffic Isolation) P3 admits and allocates migration flows only within Bmig = B − Bres ≤ B − Bmin , and the TC BPF token buckets enforce the assigned rates at the controlled egress points. Thus the aggregate migration load on the protected bottleneck is no greater than B − Bmin , leaving at least Bmin residual capacity for non-migration traffic.
Work loss (GPU-h)
Each panel: 3 sweep values per hyper-parameter; asterisk = default (ReclaimNet in Table III). 3
2.9
2.8
2.9
3
2
2
1
1
0
2.8
2.8
0.2*
Median downtime (s)
61
2.0
0.2
0.3*
60
58
61
64
0.2
0.3*
0.4
0 0.5
1.0*
α 65
61
60
40
40
20
20
0 0.2*
0.4
βp3
60 40 20
0 0.1
0
0.3
0.5
1.0*
2.0
α
βp1
βp3
Fig. 9: Hyperparameter sensitivity rendered as a 2×3 sweep grid (rows: work loss / median downtime; columns: βp1 , α, βp3 ). The default (asterisk, red bar) reproduces ReclaimNet in Table III; all six panels remain flat within ±6.4%.
Scheduling latency (ms)
TABLE IV: Per-protocol contribution. The all-off row is a naive baseline; rows P1+P2, P1+P3, P1+P2+P3 reproduce baselines B5, B4, and ReclaimNet from Table III. P1
P2
P3
Loss (GPU-h)
Down. (s)
Degr. (%)
✗ ✓ ✗ ✗ ✓ ✓ ✗ ✓
✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓
✗ ✗ ✗ ✓ ✗ ✓ ✓ ✓
8.7 4.9 8.1 8.5 3.1 3.8 4.6 2.8
145 142 79 144 71 98 67 61
34.8 33.6 32.4 3.1 31.0 7.3 2.9 2.8
Coordinator Scalability (E5) 10
4
102 101 100 50
75
100
150 200
We sweep the three principal hyperparameters (βp1 ∈ {0.1, 0.2, 0.3} for P1’s network-feasibility floor, α ∈ {0.5, 1.0, 2.0} for P2’s stability penalty, βp3 ∈ {0.2, 0.3, 0.4} for P3’s research-traffic reservation) on the same two-month trace (Fig. 9). Defaults were calibrated on weeks 1–2 (held out from the evaluation in Tables III–IV, which use weeks 3–8); the swept points cover ±50% to ±100% around each default. All three parameters operate in stable regions: median work loss varies by at most ±5.7% across the swept ranges, downtime by ±6.4%, and research-traffic degradation by ±1.1 pp. The default values (βp1 = 0.2, α = 1.0, βp3 = 0.3) sit near the empirical minimum of each metric. C. Per-Protocol Ablation (E7) We run all seven {P1,P2,P3} subsets against the same trace (Table IV). The all-off row uses a deliberately naive baseline—fixed 30-min checkpoint interval, uniformly random destination, no traffic shaping. This row is strictly worse than Bamboo (B1) because Bamboo’s pipeline-redundancy recovery re-executes only affected micro-batches, while the all-off configuration accepts larger rollbacks. The three twoprotocol rows exactly match baselines B5 (P1+P2 = NoTC-BPF), B4 (P1+P3 = Random-Dst), and a P2+P3-only configuration, confirming that Tables III and IV are derived from the same runs. D. Robustness to Distribution Drift (E8)
ReclaimNet Linear (O(N)) Notice period (10 s)
103
near-linear scaling; Algorithm 1’s per-node computation adds < 0.1 ms per node (Fig. 8). B. Hyperparameter Sensitivity (E6)
1
0.3
60
3.1
2.8
2
βp1 64
3
2.7
0 0.1
60
3
300 400 500
Number of nodes
Fig. 8: Coordinator scheduling latency vs. cluster size (emulation). ReclaimNet scales sub-linearly from 3 ms (54 nodes) to 18 ms (500 nodes), well below the minimum 10-second provider notice period.
We re-evaluate ReclaimNet on a held-out two-week period spanning the end-of-semester deadline crunch (beyond the main two-month deployment), during which the empirical hazard rate λe shifts upward by 1.7× relative to the training window. P1’s sliding-window MLE re-converges within 18 min < 0.10 |/λtrue of the regime change (criterion |λ̂e − λtrue e e sustained for two windows); end-to-end work loss on the heldout period is 3.0 GPU-h (95% bootstrap CI [2.7, 3.3] over 1,000 resamples; within 7% of the in-distribution number, p = 0.21), confirming that the adaptive controller does not require trace-specific calibration. A PPENDIX C P ROTOCOL D ERIVATION D ETAILS A. Adaptivity-Gap Binding Regimes
A PPENDIX B E XTENDED E VALUATION : S CALABILITY, S ENSITIVITY, A BLATION , D RIFT A. Scalability via Emulation (E5) Using Mininet with 54 real measurements as ground truth, we emulate up to 500 nodes. Coordinator scheduling latency grows sub-linearly (from 3 ms at 54 nodes to 18 ms at 500 nodes), below the 10-second minimum notice period in all tested scenarios. Algorithm 2’s O(|Dcand |·L) complexity yields
Because Beff = min(W, B(t)), in epochs where the local write speed W is the binding term, Beff collapses to a constant and the corresponding contribution to ∆CS vanishes. On our two-month trace, intra-building NVMe sustains W ≈ 1.1 GB/s while B(t) averages 820 Mbps intra-building and 240 Mbps across core; B(t) ≪ W in nearly all loaded epochs, so B is the binding term in ∼97% of 10-minute epochs. Conditioning the gap estimator on B-binding epochs alone yields a predicted gap of ∼5.4%; the deployment-wide 5.2% figure is mixed with the W -binding minority. Deployments with faster access links
(e.g. 25 GbE) shift more epochs into the W -binding regime and see a smaller gap; those with slower SSDs at fixed access bandwidth see the gap remain close to the B-binding value. B. Calibration Procedure for the Gap We estimate the three sufficient statistics from the calibraˆ ˆ tion trace (weeks p1–2): the sample means λ̄, Θ̄, and the sample [ mean E[W ] of λe /Beff over the psame epochs. Plugging into ˆ Θ̄ ˆ /E[W [] ≈ 1.052, i.e. a (6) yields a predicted ratio of λ̄ ∼5.2% adaptivity gap on this trace. This ratio strictly above one is the direct evidence that λe Beff is not a.s. constant on the deployment trace; the anti-correlation and CV figures in §IV are consistent with that gap but are not, by themselves, the equality condition. C. Full Baseline Parameter Settings We fix all baseline hyperparameters from their original papers or published configurations. Static-Ckpt (B3) is parameterized analytically as C̄time = C̄size /B̄ = (3.2 GB · 8)/(0.530 Gbps) ≈ 48.3 s with the trace-wide hazard λ̄e ≈ 0.99/h (358 emergency events over 361.4 spot-priced GPUh), giving ∆t∗ ≈ 9.8 min; an empirical grid search over ∆t ∈ {5, 10, 15} min selects 10 min independently (7.6 GPUh, 71.3% success), which we adopt. Alternative denominators (total GPU-h → 36 min; natural-only emergencies → 13.6 min) yield strictly worse loss and are reported in our artefact. Bamboo (B1) reuses its public pipeline-redundancy implementation with provider departure mapped to a preemption event. Slurm-PR (B1′ ) runs SLURM with -requeue and a DMTCP 10-min checkpoint. Random-Dst (B4) reuses P1+P3 with destinations sampled uniformly among VRAMcompatible nodes. No-TC-BPF (B5) disables P3 admission and edge enforcement; flows use vanilla TCP CUBIC. TimeVarying Fixed-Interval (B7) re-estimates λ̄e , Θ̄ = 1/B each epoch with the same eBPF estimators and p the cube-root allocator (Thm. 1), but commits to ∆t = 2Csize Θ̄/λ̄e per epoch—the πind policy of Thm. 2; P2 and P3 are unchanged so the comparison isolates the adaptivity-gap contribution.
D. Oracle (B6) ILP Formulation (B6) is solved offline given the full two-month reclaim and bandwidth trace. With xj,t,d ∈ at P{0, P1} (job j checkpointed P time t toPdestination d): min j ej ∆rb (x) s.t. x ≤ j,t,d ej d 1 ∀j, t, j rj xj,t,d ≤ w(e, t) ∀e ∈ E, t, Tmig (j, t, d) ≤ τj,t for scheduled events. We solve in 24-h batches with CPLEX 22.1.1 (mean wall-clock 4.7 h per window on a 64-core server); infeasible online but a tight upper bound. A PPENDIX D I MPLEMENTATION D ETAILS a) Software stack.: ReclaimNet is implemented in ∼5,400 LoC. Node agents (∼1,500 LoC modern C++17, built with CMake and gRPC) manage container lifecycle, trigger checkpoints through PyTorch torch.save and Hugging Face save_pretrained bindings invoked via a thin pybind11 bridge, invoke P2’s destination selection, and update BPF maps through libbpf. eBPF programs (∼800 LoC C, Clang 17 + libbpf) implement P3’s TC BPF tokenbucket and kill-switch using CO-RE so a single object loads across the heterogeneous campus kernels. The coordinator (∼2,000 LoC C++17) runs the P3 admission/enforcement loop, the P2 topology database (refreshed by periodic eBPF probes), and P1’s departure-rate estimators plus cube-root allocator; persistent state lives in a local SQLite store and Prometheus metrics are exposed on /metrics. Containers run under Docker with the NVIDIA Container Toolkit; GPU isolation uses MIG on A100/A6000 and process-level VRAM reservation on RTX consumer cards. Node agents drive tcBPF attachment and map updates directly via libbpf, so loading and reconfiguration require no node reboots. b) Security considerations.: Inter-node communication is authenticated via mTLS (TLS 1.3) with short-lived JWTs issued at registration. Containers run under Docker Seccomp profiles restricted to CUDA and libbpf syscalls; MIG hardware partitioning protects A100/A6000 GPUs, while RTX 3090/4090 nodes zero VRAM pages on container teardown before returning GPUs to providers. The kill-switch TC BPF program is a protocol contribution (§VII) distinct from these engineering safeguards.