Ineffective Occupancy in Heterogeneous GPU Clusters: Prediction, Contribution Analysis, and Readiness-Aware Scheduling

Ineffective Occupancy in Heterogeneous GPU Clusters: Prediction, Contribution Analysis, and Readiness-Aware Scheduling Tao Zhang1

Bin Liao2,3,*

arXiv:2610.25066v1 [cs.DC] 18 Oct 2026

1. School of Information Engineering, Guizhou University of Traditional Chinese Medicine, Guiyang 550025, Guizhou, China 2. School of Big Data Statistics, Guizhou University of Finance and Economics, Guiyang 550025, Guizhou, China 3. School of Mathematics and Statistics, Guizhou University of Finance and Economics, Guiyang 550025, Guizhou, China (zhangtao031@gzy.edu.cn)

Abstract In heterogeneous GPU clusters, GPUs are costly and scarce, yet placing a job on a machine does not mean that the job becomes ready to run. A large amount of allocated capacity can remain occupied for long periods without producing useful computation, further aggravating the waste of compute resources. On the Alibaba GPU cluster trace we find that, even when GPU capacity is sufficient and placement is complete, 52.70% of execution records never became ready. We term this phenomenon ineffective occupancy. Building on this diagnosis, and keeping the placement policy fixed, we propose Readiness-Aware Scheduling (RAS): it uses only nine attributes observable at admission time to predict readiness probability and reorders admission accordingly. Under a five-fold split grouped by machine, the gradient-boosted model reaches an ROC-AUC of 0.8187. A feature-group contribution analysis shows that the model’s score is driven mainly by job profile, duration, and GPU model, with the request-size group contributing least, followed by the scheduling and concurrency group. In GPU terms, RAS reduces the ineffective-occupancy ratio (the share of cluster GPUs held under ineffective occupancy) from 43.64% to 27.15% and raises the effective coverage of large requests (eight GPUs or more) from 46.11% to 63.18%, without sacrificing capacity utilization. Relative to the upper bound obtained by ordering with ground-truth labels, RAS captures 65.1% of the attainable improvement, well above the interpretable rules built only from the contribution-analysis conclusions; the gain therefore comes mainly from predicting with all nine attributes jointly rather than from combining rules built on any single attribute group. These results indicate that readiness status is a measurable, predictable, and optimizable dimension of admission decisions in heterogeneous GPU clusters.

Keywords: heterogeneous GPU clusters; ineffective occupancy; readiness prediction; admission ordering; interpretable machine learning; scheduling optimization

1

Introduction

Heterogeneous GPU clusters have become the mainstream infrastructure for large-scale AI training and inference. Unlike homogeneous clusters, heterogeneous clusters deploy multiple GPU models at the same time, and jobs (such as large language model training and online inference services) differ in their requirements for GPU model, GPUs per node, and network topology. This difference changes the basic resource-allocation problem: a homogeneous cluster asks whether any GPU is still available, whereas a heterogeneous cluster must answer whether the available GPUs can satisfy a specific job’s requirements— that is, the match between job demand and GPU supply [1, 2]. Funding: Guizhou Provincial Basic Research Program (Natural Science) (No. MS[2025]226); Doctoral Startup Fund of Guizhou University of Traditional Chinese Medicine (No. [2024]07); Scientific Research Startup Project for Introduced Talents of Guizhou University of Finance and Economics (No. 2023YJ26). Corresponding author: Bin Liao (E-mail: liaobin665@163.com).

1

In practice, however, this match works poorly. A Cast AI survey of more than 23,000 Kubernetes clusters reports an average GPU utilization of only 5% in enterprise environments [3]; an ACM Computing Surveys review likewise reports that GPU utilization in production clusters at Alibaba, SenseTime, Microsoft, and other organizations typically ranges from 25% to 50% [4]. The problem is not limited to a mismatch between supply and demand. Even when GPUs have been allocated, a substantial fraction of jobs never become ready. The Cast AI report lists “allocated but persistently near-zero utilization” as one of the main types of GPU waste [3]; Alibaba’s production traces record the same phenomenon [1]. Taken together, these two lines of evidence show that successful allocation does not imply that a job becomes ready: the problem extends from “GPUs cannot be allocated” to “GPUs have been allocated but the job never becomes ready.” The coexistence of compute scarcity and compute waste has become a prominent obstacle to improving the efficiency of AI infrastructure. Characterization studies of production clusters further reveal how this contradiction manifests. A study of Alibaba production traces [1] points out that high GPU demand does not translate into high effective utilization: idle GPUs cannot be allocated promptly because of missing matched CPUs, networklocality constraints, user over-reservation, and whole GPUs idling across machines. Another characterization study of MLaaS clusters [2] likewise documents systematic mismatches between the form of GPU supply and job demand. These works, however, focus mainly on cluster-level allocation and fragmentation rates; we focus on a more basic question: can a placed job actually become ready? We answer this question at the job level. On our trace, placement is usually not the bottleneck: only one record requests more GPUs than the machine it was placed on can hold (Section 3.3). The real loss is not “cannot fit” but the substantial share of placed jobs that never become ready. We call this phenomenon ineffective occupancy: GPUs are occupied by a job, but the job never truly runs. Among the 40,129,894 labeled execution records we use, this phenomenon accounts for 52.70%. Its existence implies that scheduling optimization should not pursue placement counts alone; it should also consider whether a placed job can actually run. To address this problem, we propose Readiness-Aware Scheduling (RAS). Our contributions are threefold: (1) Identifying and measuring ineffective occupancy: We focus on the phenomenon that a substantial share of placed jobs never become ready, and we measure its scale and distribution at the job level. (2) Building a readiness predictor and analyzing feature contributions: Using only nine attributes observable at admission time, we build a readiness predictor and verify its ranking ability under a machinegrouped split. A contribution analysis of feature groups shows that the model’s score is driven mainly by job profile, duration, and GPU model rather than being uniformly distributed. (3) Proposing and validating a readiness-aware scheduling algorithm: With the placement rule fixed, we propose an admission-ordering algorithm based on predicted readiness probability and verify its effect in suppressing ineffective occupancy. The remainder of this paper is organized as follows. Section 2 reviews related work, Section 3 defines the problem and the data, Section 4 presents the diagnostic analysis, Section 5 develops and evaluates readiness-aware scheduling, and Section 6 discusses the results and concludes.

2

Related Work

This section reviews existing work along three lines: cluster characterization and effective utilization, placement optimization and admission ordering, and the mapping from prediction to scheduling. Section 2.4 further contrasts our work with these lines.

2.1

Cluster Characterization and Effective Utilization

Cluster characterization studies focus on GPU utilization, resource allocation, and fragmentation [1, 2, 5]. Early analyses of Alibaba production traces [5], together with measurements of heterogeneous MLaaS

2

clusters [2], document a persistent gap between requested size and actual usage. A study of Alibaba production traces [1] further attributes this gap mainly to supply-side factors: idle GPUs are not scarce in aggregate but are scattered across machines, lack matched CPUs, violate network-locality constraints, or are held by user reservations; the same study notes that fractional-GPU fragmentation has become relatively minor, while the idling of whole GPUs across machines has become the main problem. Industry reports and reviews estimate utilization at the 5% and 25%–50% levels, respectively [3, 4]; scheduling studies give broad estimates of 5%–30% [6] or below 50% [7]. These works take the cluster as the unit of analysis and focus on allocation rate, fragmentation, or overall utilization. For example, the defragmentation algorithm of the Alibaba trace study [1] reduces the number of machines with idle resources. But a higher cluster-level allocation rate does not necessarily imply higher effective output at the job level. Work on ghost occupancy [8] observes a related phenomenon: Kubernetes clusters contain GPUs that are allocated yet produce no useful work, and such allocations are detected and reclaimed. Overall, these works lay the foundation for understanding GPU utilization, but their unit of analysis remains the cluster and does not extend to the job level.

2.2

Placement Optimization and Admission Ordering

(1) Placement optimization: Placement optimization is one of the main lines of GPU cluster scheduling and has evolved from generic bin-packing rules to GPU-specific schedulers. Early one-dimensional bin-packing rules (First-Fit, Best-Fit, Worst-Fit) target packing efficiency and struggle to capture the heterogeneity of GPU jobs. FGD [9] moves the objective from “can the job be placed” to “is the remaining space still usable after placement”: the quality of a placement decision depends not only on whether the current job fits but also on whether the remaining space will remain usable in the future. Later GPU-specific schedulers incorporate job characteristics: Tiresias [10] orders jobs by least attained service; Gavel [11] solves a linear program over a speed matrix; Pollux [12] chooses the number of GPUs by goodput; Sia [13] mixes coarse- and fine-grained resource allocation; and Synergy [14] allocates according to resource sensitivity. These works cover the two core questions of placement: where to place a job and how many GPUs to allocate to it. Industrial schedulers typically abstract placement as a pluggable scoring module; for example, Borg (Google’s cluster management system) [15] and Kubernetes [16] implement placement rules as configurable scoring items. (2) Admission ordering: Admission ordering has recently attracted attention as a problem in its own right. Work in queueing theory [6] proves that under multi-dimensional resource demands the optimal admission-ordering problem is NP-hard, and points out that the greedy policy of Kueue (a Kubernetes job queueing controller) lacks theoretical guarantees in the waiting-time dimension. In contrast to this theoretical analysis, industry has made admission a separate controller: Runway (an AI video company) uses Kueue as its admission controller and raises GPU utilization by more than 20 percentage points [17]. In that architecture, Kueue decides whether a job starts, and kube-scheduler (the default Kubernetes scheduler) [16] decides where admitted pods run. Another line starts from scheduling as a whole: a multi-cluster compute scheduling algorithm based on deep deterministic policy gradient (DDPG) reinforcement learning [18] solves scheduling as an integrated optimization problem and does not treat admission ordering separately.

2.3

From Prediction to Scheduling

By prediction target and scheduling interface, existing work falls into three categories—duration prediction, resource prediction, and readiness/startup-delay prediction; learning-based scheduling can serve as a separate decision interface. (1) Duration prediction and learning-based scheduling: Duration prediction is the most mature direction: it answers “how long will the job run”, outputs a scalar duration, and serves placement or backfilling decisions [19]. Another line is learning-based scheduling: replacing hand-written heuristics with policy networks and embedding predictors into the decision policy [20]. GPARS [21] uses a graph

3

attention network to predict job duration and reduces waiting time by 10.29% and improves resource utilization by 7.47% on the Alibaba and Philly traces; its prediction target is still duration, and the prediction is used for GPU-type assignment. (2) Resource prediction: On GPU clusters, prediction-driven scheduling has made several advances in recent years. PRISM [22] uses an LSTM to predict job arrival patterns and resource demands, reducing mean job completion time (JCT) by 25%–40%. Wind [23] uses historical execution patterns to predict task demands and completion times and outperforms the default Kubernetes scheduler on a 139-machine cluster. GFS [24] uses a lightweight prediction model to anticipate GPU demand across tenants, reducing the eviction rate by 33.0% and queueing delay by 44.1%. (3) Readiness prediction and startup-delay estimation: A study of the Alibaba GPU trace [25] predicts instance readiness status, scheduling delay, and capacity risk for a recommendation-service scenario, using a large language model (LLM) augmentation approach. On container startup delay, ADAPT [26] tracks container cold-start duration with an online exponentially weighted moving average (EWMA) estimator and feeds it as a dynamic variable into a model-predictive controller to optimize the replica count and planning horizon; CLUEAI [27] combines machine learning with statistical methods to predict CPU allocation during container startup, reducing average application startup time by 19.9%.

2.4

Differences from Existing Work

Table 1 compares this paper with four representative lines of work. Table 1. Differences between this paper and representative work. Cluster characterization[1, 2]

Fragmentation scheduling[9]

GPU-specific schedulers[10–13]

Prediction-driven scheduling[19–25]

Placement decision Placement

Uses prediction

No

No

Placement decision Placement and GPU count Placement + GPU count No

Job/cluster Placement/allocation

Adjustment

Cluster Analysis (no decision change) None (analysis)

Prediction target

—

—

—

Optimization goal

Allocation rate, fragmentation

Fragmentation increment

Goodput, JCT

Yes (duration/demand) Duration, arrival pattern, resource demand JCT, utilization

Evaluation

Cluster level (production)

Fragmentation rate (production)

JCT, makespan

JCT, utilization

Aspect Research object Decision

Placement rule

Placement/allocation

This paper Job Admission order (placement fixed) Admission order Yes (readiness probability) Binary readiness probability Effective output (share of GPUs actually used) Job-level ready output

Note: the seven aspects describe, for each line, the unit of analysis, the decision content, the adjustment mechanism, whether prediction is used, the prediction target, the optimization goal, and the evaluation method; the “This paper” column is the reference. “Job-level ready output” means that historical readiness labels serve as the outcome attribute in replay (see Section 5.1). Cold-start prediction work [26, 27] is related, but its scenario does not map directly onto GPU admission, so it is not listed.

Taken together, the main differences between this paper and the four lines above can be summarized in two respects. (1) Different problem framing. Cluster characterization remains at the cluster level and focuses on allocation rate, fragmentation, or overall utilization; we move the unit of analysis down to individual jobs and reveal ineffective occupancy—“GPU capacity is sufficient but the job never becomes ready.” Accordingly, we propose RAS, which uses predicted readiness probability as the admission score and admits jobs in descending order, thereby reducing the extent to which GPUs are held by jobs that remain unready for long periods. (2) Different decision variables and prediction targets. Fragmentation scheduling and GPU-specific schedulers optimize the placement choice; prediction-driven scheduling predicts duration, arrival patterns, or resource demand and embeds the predictor in the placement algorithm. We fix the placement rule, adjust only the admission order, and predict the job-level binary readiness probability. Our predictor is therefore not embedded in the placement decision but used directly for admission ordering. The evaluation

4

target differs accordingly: existing work asks whether placement choices are sound and whether utilization improves, whereas we ask whether GPUs are actually put to use.

3

Problem Definition and Data

Resource decisions are a counting problem in homogeneous clusters and a matching problem in heterogeneous clusters: the former asks “is there any GPU left?”, whereas the latter asks “do the available GPUs satisfy the job’s demand?” This section turns ineffective occupancy into a measurable, interpretable object: we first give the definitions, then state three problems and fix the notation, and then introduce the data and the evaluation protocol.

3.1

Definition of Ineffective Occupancy

Definition 1 (Effective and ineffective occupancy). A job-matching instance consists of a triple (𝑞 𝑗 , 𝐶 𝑗 , 𝑦 𝑗 ): • 𝑞 𝑗 = (𝑠 𝑗 , 𝑔 𝑗 , 𝑡 𝑗 , 𝜇 𝑗 , 𝑝 𝑗 , 𝑎 𝑗 , 𝑑 𝑗 ) is the request of job 𝑗; the seven components are the accelerator model 𝑠 𝑗 , the number of requested GPUs 𝑔 𝑗 , the job type 𝑡 𝑗 , the model type 𝜇 𝑗 , the priority 𝑝 𝑗 , the generative-request flag 𝑎 𝑗 ∈ {0, 1}, and the declared duration 𝑑 𝑗 ; • 𝐶 𝑗 is the cluster state when job 𝑗 enters admission: the set of machines M, and for each machine 𝑖 ∈ M its model, GPU count 𝑐 𝑖 , CPU capacity, and rack membership; • 𝑦 𝑗 ∈ {0, 1} is the readiness label, with 𝑦 𝑗 = 1 indicating that the job becomes ready. Readiness means that the job has obtained a GPU allocation and has completed image pulling, container startup, and runtime preparation, so that it can begin effective computation. Suppose job 𝑗 is placed on machine 𝑖: if 𝑦 𝑗 = 1, we call the placement effective occupancy; if 𝑦 𝑗 = 0 although the capacity 𝑐 𝑖 of machine 𝑖 satisfies 𝑐 𝑖 ≥ 𝑔 𝑗 , we call the placement ineffective occupancy—the job was admitted and the machine on which it was placed had enough GPUs for the request, yet the job never became ready. 𝐶 𝑗 cannot be observed directly. The released data record the machine on which each job actually ran, so what we observe is “which machine this match landed on”, not “the state of the whole cluster at admission time”. Definition 1 divides placement outcomes into two classes—effective and ineffective occupancy—and we focus on the latter. Definition 1 characterizes the occupancy state of a single job; the following metric aggregates it to the cluster level. Definition 2 (Ineffective-occupancy ratio). Given the placed set 𝑃 of one admission and the total number Í of cluster GPUs Γ = 𝑖 ∈ M 𝑐 𝑖 , the ineffective-occupancy ratio is 𝑊 (𝑃) =

1Õ (1 − 𝑦 𝑗 ) 𝑔 𝑗 , Γ 𝑗∈𝑃

(1)

that is, the share of cluster GPUs held under ineffective occupancy. 𝑊 (𝑃) is the main metric of Section 5: it separates “GPUs are occupied” from “GPUs are used effectively” and aggregates Definition 1 to the policy level. We use readiness status as the observable indicator. The official definition in the released data is “whether the pod has ever been ready”. ASI also provides a “Standby” mechanism: high-priority users can voluntarily release allocated GPUs and CPUs while keeping the container, and restore them when needed. Voluntary yielding is recorded by the state_public field of the pod-hour table, which is not included in the execution-summary table we use; hence “not ready” in this paper does not include voluntary yielding. In addition, this convention does not exclude executions that exit shortly after becoming ready; such records count as effective occupancy. On these definitions, the paper addresses three questions. Let 𝑥 𝑗 denote the feature vector formed by the nine observable attributes (the seven request components 5

of Definition 1 plus two scheduling-side attributes, the scheduling delay and the number of concurrent requests on the same machine). Problem 1 (Readiness prediction). Given the feature vector 𝑥 𝑗 and the readiness label 𝑦 𝑗 , learn a probabilistic model 𝑓ˆ : 𝑥 𝑗 ↦→ Pr(𝑦 𝑗 = 1 | 𝑥 𝑗 ), and evaluate its ranking ability under a machine-grouped protocol. Problem 2 (Structural contribution analysis). Given a fitted 𝑓ˆ and the feature-group partition G defined in Section 4.1, measure how much each group drives the predicted value of 𝑓ˆ, and check the correspondence of this structure on the supply side and in the spatial distribution. Problem 3 (Readiness-aware scheduling). Given a set of jobs J , each job’s requested GPU count 𝑔 𝑗 and readiness probability 𝑓ˆ(𝑥 𝑗 ), and cluster capacity given by the idle GPU counts {𝑢 𝑖 }𝑖 ∈ M of the machines in M, determine the admission order under a fixed placement rule so as to minimize the ineffectiveoccupancy ratio 𝑊 (𝑃). Problems 1 and 2 are answered in Section 4 and Problem 3 in Section 5. The three connect in sequence: readiness probability provides the object to be explained by the contribution analysis, and the contribution analysis delimits the room for adjustment at admission. Table 2 lists the main notation. Table 2. Main notation. Symbol

Meaning

𝑗, J 𝑞𝑗 𝑥𝑗

Job and job set Job request (components in Definition 1) Nine observable attributes

𝑖, M 𝑐𝑖 𝑐max , Γ

𝑦𝑗 𝑓ˆ u, 𝑢𝑖 𝑃 𝜙 𝑘 ( 𝑓ˆ, 𝑥 𝑗 )

Readiness label, {0, 1} Readiness probability model Idle-GPU vector and its components Placed set of one admission Contribution of feature 𝑘

𝐶𝑗 𝜋𝑗 𝜌 𝑊 ( 𝑃) 𝜙¯𝐺 , 𝜎𝐺

Machine and machine set GPU capacity of machine 𝑖 Maximum GPUs per machine and total cluster GPUs Cluster state at admission Admission score, 𝜋 𝑗 = 𝑓ˆ ( 𝑥 𝑗 ) Load (requested GPUs over total GPUs) Ineffective-occupancy ratio (Definition 2) Contribution value and contribution share

Here Γ, 𝜋 𝑗 , u, 𝑢 𝑖 , 𝑃, and 𝑊 (𝑃) are used only in Section 5, and 𝜙 𝑘 , 𝜙¯𝐺 , and 𝜎𝐺 only in Section 4.3; the remaining notation is used throughout.

3.2

Dataset and Features

We use the cluster-trace-gpu-v2026 trace (Alibaba Serverless Infrastructure, ASI) publicly released by the Alibaba cluster trace programme [28]. It provides four fact tables: the pod-hour table, the serverhour table, the network-element-hour table, and the job execution-summary table. We use the executionsummary table for demand and the server-hour table for supply; the latter provides each machine’s model, GPU count, and CPU capacity, together with its rack (access-switch domain) identifier, and is the only source that links a job request to the machines it could use. Table 3 lists the demand-side data conventions and the nine features, Table 4 gives the machine structure we reconstruct from the supply side, and Appendix A reports the value ranges and descriptive statistics of the nine features. Table 3. Execution-summary table and the nine features used in this paper. Item

Value

Records Labeled records Records with an empty label Distinct machine identifiers Accelerator models Features (9)

40,522,321 40,129,894 392,427 (0.9684%) 37,701 13; job types: 6; priority classes: 3 gpu_spec_public, priority_class, job_type_public, model_type_public, is_genai_request, gpu_request, duration_hours, schedule_delay_sec, server_req_count ready_status schedule_status, ready_delay_sec

Label Excluded (observable only after the decision) Aggregation granularity

One record per job execution (only execution spans that request GPUs)

Note: the nine features are accelerator model, priority class, job type, model type, generative-request flag, requested GPUs, declared duration, scheduling delay, and the number of concurrent requests on the same machine.

6

(a) Label balance: Ready 47.30% / Not ready 52.70%

(b) Declared duration is heavy-tailed (mean = 5.15 h)

Jobs (log)

106

52.70% never ready

105 104 103

n = 40,522,321 labeled 40,129,894

102 10−4

10−3

10−2

10−1

100

101

102

104

103

Duration (hours, log)

GPU request (GPU)

(c) One request size dominates

(d) Accelerator types | 13 types

2.04 5.01 0.52 0.39 2.95

0.07 0.11 0.23 0.31 0.5 1 2 4 8 16

15.3%

H20

14.1%

XPU-A 11.1%

L20 78.42

10.3%

A10

6.84 1.03 1.39 0.72 0

28.5%

XPU-E

7.1%

XPU-C

5.1%

XPU-B 2.7%

A100 20

40

60

80

100

0

Share of all records (%)

5

10

15

20

25

30

35

Share of all records (%)

Figure 1: (a) Label balance; (b) declared-duration distribution; (c) requested-GPU distribution; (d) share of records by accelerator model.

The properties of the demand side bear directly on the analysis in Section 4. The labels are nearly balanced: 47.30% of labeled records are ready, so accuracy is uninformative and Section 4.2 therefore relies on threshold-independent ranking metrics. Request sizes are highly concentrated: 78.42% of records request exactly one accelerator, and only 0.73% request more than eight. Declared duration is heavy-tailed: the mean is 5.15 hours and the median is 0.42 hours, with the maximum shown in Figure 1. The jobidentifier field workload_id is empty for 89.0% of labeled records, so it cannot serve as an independent grouping variable. The structure of the supply side makes matching the core problem. The cluster expanded throughout the observation period: the number of machines grew from 28,036 to 37,707 and the number of GPUs from 111,321 to 155,410. Expansion did not change the distribution of machine structure, however: GPUs per machine is bimodal, the vast majority of machines have only one or two GPUs, and a job requesting eight GPUs can land on only 30.8% of machines (although those machines hold 74.3% of the GPUs), while a job requesting sixteen can land on only 7.5%. The CPU-to-GPU ratio is equally uneven: across model and GPU-count combinations, the number of CPU cores per GPU varies widely (Figure 2). Table 4. Supply-side structure reconstructed from the server-hour table (185-day union, 38,048 machines). Supply attribute

Observed value

Accelerator models GPUs per machine

13 Bimodal, with only a few distinct values; 30.8% of machines can host a job requesting 8 GPUs 12–192 2,377 90.5% 7 / 136 Day 109 27,573 (of 38,048) 347 254,980,926 / 587,703,816

CPU cores per GPU Racks Share of racks with a single model Machines per rack (median/max) First day the rack field is available Machines present throughout Machines never appearing in the job table Time integrals of demand and supply GPU·hours

Demand GPU·hours are about 43.4% of supply, indicating that capacity is sufficient in aggregate while local matching remains a problem. Two further points about the supply side. First, the rack field is available only in the later part of the observation period, so topology-related analysis covers only that period. Second, the set of machines is identical at different times within the same day—the server-hour table is effectively a daily snapshot—so we use it only at daily granularity.

7

(a)

Union census (185 days):37,707 servers

(b)

Few machines host this size, yet hold most GPUs (c) Share of feasible machines Share of GPUs held by those machines

16,000 13,836

14,000

12,000

191.9

XPU-B·1GPU 128.0

A30·1GPU

128.0

A10·1GPU up to 1 GPU

12,266

8,776 8,000

up to 4 GPU 43.5 percentage points

6,000

up to 8 GPU

4,000 2,829

96.0

XPU-C·2GPU

up to 2 GPU

10,000

Servers

Range 12.0–191.9 cores/GPU

up to 16 GPU

2,000

A10·2GPU

96.0

L20·2GPU

96.0

H20·8GPU

25.3

XPU-D·8GPU

24.0

XPU-B·8GPU

24.0

XPU-A·8GPU

24.0

L20·8GPU

24.0

H800·8GPU

24.0

H20-141GB·8GPU

24.0

A800·8GPU

16.0

A100·8GPU

16.0

XPU-E·16GPU

12.0

0 1

2

8

16

0

GPUs per server

25

50

75

100

0

Share of machines / of GPUs (%)

50

100

150

200

CPU cores per GPU

Figure 2: (a) Distribution of GPUs per machine; (b) share of machines that can host each request size and the share of GPUs they hold; (c) CPU cores per GPU.

Initial evidence of ineffective occupancy. Applying the ineffective-occupancy case of Definition 1 to the data: of the 40,129,894 labeled records, 52.70% never became ready, and the GPU capacity of the machines they were placed on is sufficient for the request (test in Section 3.3). This ratio is the core phenomenon of this paper. Section 4 shows that it is predictable and decomposable, and Section 5 gives an admission rule that exploits this prediction.

3.3

Evaluation Protocol and Leakage Check

If a field’s value is produced only after the admission decision is complete, we exclude it from the feature matrix. All models in Section 4 exclude these fields. The execution-summary table has no time column, so we use a single-point observation and do not analyze temporal drift or seasonality. Table 5. Leakage check: two fields observable only after the decision and the nine observable attributes. Column

Single-feature ROC-AUC

Treatment

schedule_status ready_delay_sec Nine observable attributes, single-feature ROC-AUC range

0.5055 0.9644 0.5056–0.5786

Excluded (produced after the decision) Excluded (produced after the decision; directly leaks the label) Included

The two fields in Table 5 are excluded because of when their values arise: both are produced only after the admission decision is complete and therefore cannot enter the feature matrix. schedule_status is the scheduler’s outcome flag for the execution. Its relation to the label is almost degenerate in one direction: its conditional probability of being 1 is 1.0000 on ready records and 0.9890 on non-ready records; but as a single predictor it carries no information, with a single-feature ROC-AUC of only 0.5055 after aligning its direction with the label. ready_delay_sec is the decisive case: it is measured only after a job becomes ready, its single-feature ROC-AUC is 0.9644, and a model that includes it would directly leak the label. By comparison, the single-feature ROC-AUC of the nine observable attributes ranges from 0.5056 to 0.5786. One further fact concerns the data convention. The execution-summary table contains only execution spans already placed on machines with sufficient GPU capacity: among the 40,129,894 labeled records, only one has gpu_request exceeding the GPU capacity of the machine it was placed on. Hence the 52.70% non-ready share is not due to insufficient capacity but to ineffective occupancy. This distinction separates our work from cluster-level utilization analysis; Section 4.4 analyzes its structure. We split by whole machine: if records from the same machine appeared in both the training and test sets, the model could memorize whether that machine’s jobs are usually ready, whereas a new machine at deployment provides no such information. Grouping by machine puts records from the same machine 8

in the same fold, so the evaluation measures generalization to new machines; a random split is optimistic, whereas a machine-grouped split is closer to deployment. We use the machine-grouped split as the primary protocol. Evaluation relies mainly on a threshold-independent ranking metric (ROC-AUC) because the classes are nearly balanced and accuracy is uninformative; intervals come from bootstrap resampling of held-out records.

4

Diagnosis: Readiness Is Predictable and Decomposable

This section answers the three questions. The method is given in Sections 4.1 and 4.2, the contribution analysis in Section 4.3, and the structural evidence in Section 4.4. The output is the readiness probability 𝑓ˆ(𝑥 𝑗 ) used for admission ordering. Section 4.4 narrows the actionable variable to the admission order and connects to Section 5.

4.1

Readiness Predictor

This section answers Problem 1: given the instance set of Definition 1, learn 𝑓ˆ : 𝑥 𝑗 ↦→ Pr(𝑦 𝑗 = 1 | 𝑥 𝑗 ), the probability that a job request enters the ready state. It outputs no scheduling action and is not embedded in a placement algorithm; its only use is to provide an admission score. This differs from the common practice in prediction-driven scheduling, where predictors output duration, arrival patterns, or resource demand and are embedded inside the placement decision; our predictor answers only “will this job become ready?”, and is used for admission ordering. The nine observable attributes are partitioned into five disjoint feature groups forming G = {𝐺 form , 𝐺 dur , 𝐺 spec , 𝐺 req , 𝐺 sched }: the job-profile group 𝐺 form (job type 𝑡 𝑗 , model type 𝜇 𝑗 , generativerequest flag 𝑎 𝑗 , priority 𝑝 𝑗 ), the duration group 𝐺 dur (declared duration 𝑑 𝑗 ), the GPU-model group 𝐺 spec (request-side model 𝑠 𝑗 , i.e., gpu_spec_public, excluding the placement-machine identifier), the request-size group 𝐺 req (requested GPUs 𝑔 𝑗 ), and the scheduling-and-concurrency group 𝐺 sched (scheduling delay ℓ 𝑗 and the number of concurrent requests on the same machine ℎ 𝑗 ). The two fields observable only after the decision—the scheduler’s decision flag and the ready delay—are always excluded from the feature matrix, because their values arise only after the admission decision and mixing them in would constitute data leakage. This partition is used consistently throughout this chapter. The data split follows Section 3.3. The six models compared cover the linear, single-tree, and ensemble families; two gradient-boosting implementations are included so that the main result does not depend on a single library [29, 30]. Evaluation focuses on ROC-AUC (a measure of ranking ability) because the classes are nearly balanced and accuracy is uninformative; PR-AUC and probability-calibration deviation serve as auxiliary metrics. Intervals come from bootstrap resampling of held-out records, and model differences are tested with a paired test on the same resampling indices.

4.2

Results

Under the machine-grouped five-fold split, the models outperform the majority-class and lookup baselines on readiness prediction. The reference model achieves an ROC-AUC of 0.8187 with interval [0.8130, 0.8244], versus 0.4960 for the majority-class baseline; on the same held-out predictions, the reference model performs comparably to XGBoost (Table 6). Compared with “looking up historical readiness rates”, the two lookup methods reach 0.6789 and 0.7520; the reference model’s gain over the stronger lookup method (0.7520) is 0.0666 in AUC. The lookup provides a lower bound under the same information; the model must discriminate further. The lookup is fitted within each training set, so it reads the same information as the model; the only difference is whether the nine attributes are used jointly. A positive gain indicates that the nine attributes jointly discriminate better than any single group, which is why Section 5 uses the joint prediction rather than a single attribute group for admission ordering.

9

(a) ROC

(b) PR

1.0

(c) Calibration (10 equal-width bins)

1.0

1.0

reference-model ECE = 0.0028

0.8

Precision

True positive rate

Observed ready frequency

0.9

0.8

0.6

0.4

0.6

LogReg 0.7028 DecisionTree 0.7705 RandomForest 0.8115 XGBoost 0.8189 LightGBM 0.8187

0.2

0.7

positive-rate baseline 0.4730

0.5

0.0

0.8

0.6

0.4

0.2

0.0 0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

False positive rate

0.4

0.6

0.8

1.0

0.0

Recall

0.2

0.4

0.6

0.8

1.0

Predicted ready prob. (bin mean)

Figure 3: (a) ROC curve; (b) PR curve, with the dashed line showing the positive-rate baseline; (c) calibration curve (ten equal-width bins).

Probability calibration is also usable: the reference model’s calibration deviation is 0.0028, so its probability values can be interpreted directly against a threshold; Figure 3 compares the ROC, PR, and calibration curves graphically. Table 6. Main results: machine-grouped five-fold held-out prediction. Model

ROC-AUC

95% CI

PR-AUC

Calibration deviation

Majority class Logistic regression Decision tree Random forest XGBoost LightGBM (reference)

0.4960 0.7028 0.7705 0.8115 0.8189 0.8187

[0.4880, 0.5032] [0.6962, 0.7096] [0.7640, 0.7772] [0.8056, 0.8175] [0.8131, 0.8245] [0.8130, 0.8244]

0.4703 0.7065 0.7857 0.8191 0.8268 0.8266

0.0000 0.0204 0.0008 0.0287 0.0016 0.0028

Note: all three metrics are computed from the predictions of the same fit on the same held-out records; intervals are bootstrap 95% percentile intervals over 20,000 held-out records; the calibration deviation uses ten equal-width bins.

The values in Tables 6, 8, and 9 come from different fits and are used only for within-table comparison.

4.3

Contribution of Feature Groups and Ablation Analysis

The two measures—contribution values and ablation analysis—give somewhat different internal orders but the same conclusion. This section examines the contribution of each of the five feature groups. The global contribution analysis uses path-dependent SHAP (SHapley Additive exPlanations) values, summed by group and averaged over 20,000 held-out records drawn without replacement from the grouped test split; it measures the influence of a group of attributes on the predicted value [31]. Let 𝜙 𝑘 ( 𝑓ˆ, 𝑥 𝑗 ) denote the contribution of feature 𝑘 on record 𝑥 𝑗 in sample set S; then the contribution value and contribution share of group 𝐺 ∈ G are 𝜙¯𝐺 =

1 Õ Õ 𝜙 𝑘 ( 𝑓ˆ, 𝑥 𝑗 ) , |S| 𝑘 ∈𝐺

𝜎𝐺 = Í

𝑥𝑗 ∈S

𝜙¯𝐺 𝐺 ′ ∈ G 𝜙𝐺 ′

¯

,

(2)

where 𝐺 ′ ranges over the five feature groups in G. Because G is a disjoint partition, the 𝜎𝐺 sum to one, which makes the contribution shares comparable across groups; Eq. (2) sums absolute values over all features in a group, so the number of features in a group affects the absolute magnitude, and comparisons should be read together with group size. Table 7 and Figure 4(a) give the contribution value and share of the five groups: job profile 38.3%, duration 25.6%, GPU model 18.3%, together 82.1%; request size 6.8%, ranked last; scheduling and concurrency 11.1%. Table 7. Contribution of the five feature groups. Feature group

Features in group

Contribution value

Share (%)

Job profile Duration GPU model

4 1 1

0.9874 0.6606 0.4710

38.3 25.6 18.3

10

(a)

Five groups partition; shares sum to 100% (n=20,000 held-out records)

(b)

9 features by mean |SHAP| (GPU request rank 5) 0.661

Duration 0.987 38.3%

Job form

0.661 25.6%

Duration

0.471 18.3%

Machine type

0.286 11.1%

Sched. & load

Request size

0.00

Job type

0.485

Machine type

0.471

Priority class

0.193

GPU request

0.175

Model type

0.166

Schedule wait

0.161

Is generative

0.143

Server request load

0.125

0.175 6.8%

0.25

0.50

0.75

1.00

1.25

1.50

0.0

0.1

0.2

mean |SHAP| (sum by group)

0.3

0.4

0.5

0.6

0.7

0.8

mean |SHAP| (per feature)

Figure 4: (a) Mean absolute Shapley value and contribution share of the five feature groups; (b) mean absolute Shapley value of the nine individual attributes. Feature group

Features in group

Contribution value

Share (%)

Scheduling and concurrency Request size

2 1

0.2857 0.1748

11.1 6.8

Note: contribution share is the share of the contribution value defined in Eq. (2) among the five groups, summing to one; rounded values may not sum to exactly 100%. Contribution values come from the same reference model on 20,000 held-out records; the group partition is unique throughout the paper.

Contribution value measures how strongly a group drives the prediction. If one instead uses the performance loss after removing the group, the internal order of the top three changes. This section reports that difference. Table 8 places the two side by side and Figure 5 gives the corresponding bar chart. Path-dependent Shapley values rank job profile first and duration second; ablation analysis ranks duration first. Both are clearly larger than request size and scheduling. The two measures differ because they are different statistics: contribution value is inflated by the number of features in a group, whereas ablation analysis measures whether the group’s information can be substituted. The part where the two agree is what the conclusion relies on: both rank request size last (removing it changes AUC by −0.0098) and both support the dominance of the three groups; when each group is used alone, duration reaches 0.6567, above job profile (0.6152) and GPU model (0.6222). This section reports the part where they agree: three groups dominate and request size ranks last. Table 8. Contribution ranking and ablation ranking side by side. Feature group

Contribution value

Performance change after ablation

Duration GPU model Job profile Scheduling and concurrency Request size

0.6606 0.4710 0.9874 0.2857 0.1748

-0.0669 -0.0409 -0.0328 -0.0133 -0.0098

Note: the contribution column is the mean absolute value over 20,000 held-out records; the ablation column is the change in test AUC relative to the all-feature model (0.8167); the two columns come from independent fits and are used only to compare rankings.

Figure 6 shows the individual contributions of the nine attributes at the record level; it follows the convention of Figure 4(b) but retains per-record variation.

4.4

Structural Origin and Concentration

This section analyzes the supply-side structure and spatial distribution corresponding to the three dominant groups (job profile, duration, GPU model). As noted in Section 3.2, the cluster expanded throughout the observation period but the distribution of machine forms did not change; two points matter here: GPUs per machine is bimodal, and GPU models are not freely interchangeable. Both indicate that “enough GPUs but the job is not ready” corresponds to the supply11 structure rather than to a transient fluctuation on the demand side. Ineffective occupancy is also spatially concentrated rather than being uniformly random noise across the cluster. Measured by the variance ratio 𝐷 in Eq. (3), the observed variance of per-machine readiness

Full-model AUC = 0.8167 (9 features, test set 155,808)

(a) Duration

Single-group: duration group still strongest

(b)

-0.0669

0.6567

Duration -0.0409

Machine type

-0.0328

Job form

-0.0133

Sched. & load

0.6222

Job form

0.6152 0.5860

Sched. & load

-0.0098

Request size

Machine type

0.5690

Request size

Full model 0.8167 -0.08

-0.06

-0.04

-0.02

0.00

0.50

0.55

AUC drop after removing group

0.60

0.65

0.70

0.75

0.80

0.85

AUC using only this group

Figure 5: (a) Difference in AUC from the all-feature model after removing the group; (b) AUC when trained with only that group. (a) SHAP beeswarm: each dot is one record

(b) Global importance 0.661

Duration (h)

Duration (h) Job type

1.0

Machine type

Job type

0.485

Machine type

0.471

Priority class 0.6 GPU request 0.4 Model type

Feature value

0.8 Priority class

0.193

GPU request

0.175

Model type

0.166

Schedule wait (s)

0.161

Is generative

0.143

Server request load

0.125

0.2

Schedule wait (s)

0.0

Is generative Server request load

-7.5

-5.0

-2.5

0.0

2.5

5.0

7.5

10.0

SHAP value (impact on model score)

0.0

0.2

0.4

0.6

0.8

mean |SHAP| value

Figure 6: (a) SHAP beeswarm for the nine attributes, colored by attribute value; (b) mean absolute SHAP value per attribute.

overall readiness rate. The numerator is the record-weighted observed variance and the denominator is the expected binomial variance at the same sample size. The difference between the two conventions comes from the machine-size distribution: the equal-weight quantile answers “how low is a typical machine”, whereas the record-weighted ratio answers “how concentrated is non-readiness in aggregate”; they are not the same quantity and should not substitute for each other. Taken together, these findings narrow the adjustable variables available to Section 5. The model’s predicted score is driven mainly by three groups—job profile, duration, and GPU model; the scheduler does not directly change these attributes; the request-size group is not the dominant axis, so policies such as “large requests first” do not touch the dominant factors. Combined with the fact noted in Section 3.3 that the execution-summary table contains only execution spans already placed on machines with sufficient GPU capacity (only one record has gpu_request exceeding the GPU capacity of the machine it was placed on), the 52.70% non-ready share does not stem from insufficient GPU capacity. Hence the capacity constraint on the placement side is not tight, job attributes cannot be changed, and the only actionable variable left is the admission order. Section 5 therefore connects the predictor to admission and uses the effective output defined in Section 5.1 as the evaluation quantity.

12

(a) Per-machine ready rate spans widely (variance ratio 55.13 vs random, controlling for 9 attributes) 1.0

14.0%

<20%

0.8

Ready rate

(b) Machines by ready-rate band

0.6

83.7%

20–80% 0.4 0.2

2.3%

>80%

0.0 0.0

0.2

0.4

0.6

0.8

1.0

0

20

Quantile of per-machine ready rate

40

60

80

100

Share of all machines (%)

(c) Ready rate x machine type x job type (%): darker = higher, n/a = no records 100 dev

72

48

n/a

57

48

n/a

100

66

94

40

50

67

57

offline_inference

33

36

16

47

44

30

39

19

55

36

29

34

52

online_inference

43

73

90

91

56

91

53

60

84

51

78

52

52

60

other

100

69

n/a

80

78

n/a

68

86

93

100

87

62

85

40

training

96

80

n/a

83

91

59

82

96

90

96

100

82

96

unknown

71

60

67

54

50

49

69

57

81

60

61

49

64

0 A1

00 A1

0 A3

00 A8

0 H2

GB 41 -1 0 H2

00 H8

0 L2

XP

A U-

XP

B U-

XP

C U-

XP

D U-

XP

E U-

Ready rate (%)

80

20

0

Figure 7: (a) Per-machine readiness rates sorted by quantile, with the dashed line showing the overall cluster readiness rate; (b) share of machines by readiness-rate bin; (c) readiness rate across GPU model and job type, with darker color indicating higher readiness and cells with no records marked n/a.

5

Readiness-Aware Scheduling

Section 4 narrows the adjustable variable to a single place: the admission order. This section answers Problem 3, turns this restriction into a deployable, comparable rule, and measures it with a different evaluation quantity. Section 5.1 explains why the evaluation quantity must change, Section 5.2 gives the algorithm, complexity, and policy set, Section 5.3 gives the experimental setup and main results, and Section 5.4 performs ablation and sensitivity analyses.

5.1

From Placement Metrics to Effective Output

We begin with a minimal example. A cluster has one idle machine and two jobs: one can become ready and one cannot, and the idle GPUs suffice for only one of them. First-come, first-served (FCFS) queues them by arrival; if the job that never becomes ready arrives first, the GPUs are held by a job that cannot become ready. Under a different order, the same GPUs can serve a job that actually runs. In both orders the number of machines and GPUs is unchanged, yet the GPUs actually put to use differ. This example shows that placement metrics (match rate, allocation rate, fragmentation) measure whether GPUs are occupied and do not answer whether they are used effectively. We focus on the latter: the measurements in Section 4 show that on our trace almost every request that a machine can hold is placed, so placement feasibility is not the bottleneck; the real loss is that a substantial fraction of placed jobs never become ready. The evaluation quantity in this chapter is therefore not how many jobs are placed but the effective output: the share of cluster GPUs occupied effectively, and the job-level metrics derived from it. Definition 2 in Section 3 already defines this quantity as 𝑊 (𝑃), and this chapter uses it directly as the primary objective; the remaining metrics are computed on the same placed set 𝑃 under fixed conventions. To make comparisons fair, this chapter fixes the placement rule of all policies to best-fit (the machine with the smallest idle GPU count that can still hold the job); policies differ only in admission order. The resulting differences therefore come only from admission order, not from the placement algorithm— which is also the only adjustable variable identified by the contribution analysis in Section 4. 13

Assumption 1 (Replay assumption). The admission replay experiment re-simulates admission on historical jobs in a new order, keeps the placement rule fixed, and changes only the admission order; readiness labels are therefore treated as attributes of a job under the current machine supply and enter the comparison. The boundary of this assumption is discussed in Section 6.2.

5.2

Method: Readiness-Aware Scheduling

The admission rule can be stated in one sentence: admit in descending order of predicted readiness probability. In deployment it has two stages: offline, a readiness predictor is trained on historically labeled samples; online, its output scores each job and orders admission accordingly. Algorithm 1 gives predictor training, and Algorithm 2 gives the admission procedure; both share the feature-group partition G and the nine observable attributes from Section 4, and the placement rule stays the same across the two stages. Algorithm 1 Readiness predictor training (RAS stage 1, offline) Input: Labeled samples {(𝑥 𝑗 , 𝑦 𝑗 )} 𝑚 𝑗=1 and grouping key 𝑖 𝑗 (the machine on which the job actually ran); feature-group partition G; number of folds 𝐾; number of bins 𝐵 𝑚 Output: Per-fold reference models { 𝑓ˆ𝑘 } 𝐾 𝑘=1 and held-out readiness probabilities { 𝑝ˆ 𝑗 } 𝑗=1 1: Build the feature matrix 𝑋 ∈ R𝑚×9 from the nine observable attributes, excluding the scheduler decision flag and the ready delay, which are observable only after the decision 2: Split the samples into 𝐾 folds {D 𝑘 } 𝐾 𝑘=1 by grouping key 𝑖 𝑗 , so that records from the same machine do not cross folds 3: for 𝑘 = 1 to 𝐾 do 4: Draw a stratified 10% of the training set D−𝑘 as validation set D 𝑘val 5: Fit a gradient-boosted model 𝑓ˆ𝑘 on D−𝑘 \ D 𝑘val , with early stopping on log loss over D 𝑘val 6: Take held-out predictions 𝑝ˆ 𝑗 ← 𝑓ˆ𝑘 (𝑥 𝑗 ) on the held-out set D 𝑘 7: end for 8: Evaluate the probability-calibration deviation over 𝐵 equal-width bins and the ranking ability (ROCAUC, PR-AUC) from the held-out predictions 9: return { 𝑓ˆ𝑘 } and { 𝑝ˆ 𝑗 } Algorithm 1 evaluates on held-out predictions: ranking ability by ROC-AUC and PR-AUC, and probability reliability by the expected calibration error (ECE), the arithmetic mean calibration deviation over 𝐵 equal-width bins, 𝐵 Õ |𝑆 𝑏 | acc(𝑆 𝑏 ) − conf (𝑆 𝑏 ) , (4) ECE = 𝑚 𝑏=1 where 𝑆 𝑏 is the 𝑏-th bin and acc(𝑆 𝑏 ) and conf(𝑆 𝑏 ) are the within-bin label mean and mean predicted probability. The held-out probabilities are not post-processed and are used directly as the readiness probability estimates for all samples. The ranking key 𝜋 𝑗 in Algorithm 2 takes one of three forms depending on the score source:   −𝜏 𝑗 ,     𝜋 𝑗 = 𝑟ˆ (𝑠 𝑗 , 𝑡 𝑗 ),     𝑝ˆ 𝑗 , 

baseline (arrival order), interpretable rule (historical readiness by model × job type),

(5)

this paper (held-out readiness from Algorithm 1).

The ranking keys of the remaining baseline policies are given in the “Basis of admission order” column of Table 9: shortest-job-first sorts by declared duration 𝑑 𝑗 ascending, smallest-request-first by 𝑔 𝑗 ascending, large-request-first by 𝑔 𝑗 descending, and random ordering uses a uniform random key. All policies share the same placement rule and differ only in the ranking key. Proposition 1 gives the computational cost of Algorithm 2; Algorithm 1 is offline training whose cost is dominated by 𝐾 model fits and does not enter the online admission path. 14

Algorithm 2 Readiness-aware admission (RAS stage 2, online) Input: Job set J (𝑛 = |J |); requested GPUs 𝑔 𝑗 , arrival order 𝜏 𝑗 , and admission score 𝜋 𝑗 ; machine set M and idle GPU counts 𝑢 𝑖 ; maximum GPUs per machine 𝑐 max Output: Placement 𝛼 𝑗 or an unmatched marker for each job, and the ineffective-occupancy ratio 𝑊 (𝑃) 1: Sort J by 𝜋 𝑗 descending as the primary key and arrival order 𝜏 𝑗 ascending as the secondary key, obtaining admission sequence 𝜎 2: Maintain bucket counts over the idle-GPU domain [0, 𝑐 max ] and a query structure for “the smallest idle count not less than a given value” 3: for all each job 𝑗 in admission sequence 𝜎 do 4: if 𝑔 𝑗 > 𝑐 max then 5: Mark 𝑗 as not accommodatable by any machine 6: continue 7: end if 8: 𝑖★ ← BestFit(𝑔 𝑗 ), the machine with the smallest idle GPU count that is ≥ 𝑔 𝑗 9: if no such 𝑖★ exists then 10: Mark 𝑗 as unmatched 11: continue 12: end if 13: 𝑢 𝑖★ ← 𝑢 𝑖★ − 𝑔 𝑗 , and update the query structure 14: Record placement 𝛼 𝑗 ← 𝑖★, classified as effective or ineffective occupancy according to 𝑦 𝑗 15: end for 1Í 16: 𝑊 (𝑃) ← (1 − 𝑦 𝑗 ) 𝑔 𝑗 Γ 𝑗∈𝑃 17: return {𝛼 𝑗 }, unmatched markers, and 𝑊 (𝑃) Proposition 1 (Complexity). Let 𝑛 = |J |. Algorithm 2 runs in 𝑂 (𝑛 log 𝑛 + 𝑛 log 𝑐 max ), where 𝑐 max is the maximum per-machine GPU capacity; in our cluster this value takes only a few distinct values, so the running time is dominated by sorting, 𝑂 (𝑛 log 𝑛). Proof. The sort on line 1 is 𝑂 (𝑛 log 𝑛). For placement queries, bucket the machines by idle GPU count: the number of buckets is at most 𝑐 max + 1 (idle counts are integers in [0, 𝑐 max ]), and maintain a segment tree over the GPU-count domain with 𝑂 (𝑐 max ) nodes. Querying “the machine with the smallest idle GPU count not below 𝑔 𝑗 ” and decrementing that machine’s idle count each require one root-to-leaf descent in the segment tree, 𝑂 (log 𝑐 max ) each; the loop body totals 𝑂 (𝑛 log 𝑐 max ). Adding the two terms gives the result. In our clusters 𝑐 max takes small values (GPUs per machine is bimodal with only a few distinct values), so log 𝑐 max is much smaller than log 𝑛 and the running time is dominated by the sorting term. □ Admission ordering relies on the scores from the Section 4 predictor; Figure 8 shows, on two heldout records, how the score is composed from the nine attributes, illustrating that the ordering basis is a decomposable score rather than a threshold on a single attribute. The score 𝜋 𝑗 has three sources, used to separate whether the gain comes from the model or from a simple rule; the policy set is organized into baselines, interpretable rules, this paper, and the upper bound. Table 9. Admission policies and score sources. Block

Policy

Basis of admission order

Baseline Baseline Baseline Baseline Baseline Interpretable rule

FCFS Shortest job first Smallest request first Large request first Random ordering Lookup by GPU model and job type

Interpretable rule

Lookup by job type and duration

This paper

RAS

Arrival order Declared duration ascending Requested GPUs ascending Requested GPUs descending Uniform random key Historical readiness by model × job type, descending Historical readiness by job type × duration bin, descending Readiness probability of the nine-feature gradient-boosted tree, descending

15

Held-out ROC-AUC — — — — — 0.6785 0.6934 0.8176

Block

Policy

Basis of admission order

Held-out ROC-AUC

Upper bound

Ordering by true labels

True readiness label descending

—

Note: all policies share the same placement rule (best-fit) and differ only in admission order; “held-out” means the score is estimated outside the training set to which the job belongs.

Table 9 is organized into four blocks. Random ordering is a lower-bound control carrying no information. The lookup by GPU model and job type has a held-out AUC of 0.6785, almost the same as the 0.6789 reported in Section 4; the lookup by job type and duration (0.6934) lies between it and the threekey joint lookup (0.7520). The last row, ordering by true labels, is used only to indicate the upper bound of the improvement space. Table 10 gives the evaluation metrics. The four main metrics quantify the same thing from the GPU and job perspectives, and the remaining metrics check whether a policy harms other parts of the system. Table 10. Definitions of the evaluation metrics. Metric

Definition

Direction

Role

Ineffective-occupancy ratio

GPUs under ineffective occupancy ÷ total cluster GPUs, i.e., 𝑊 ( 𝑃) of Definition 2 in Section 3 Jobs effectively occupied and placed ÷ jobs a machine can accommodate Jobs effectively occupied and placed ÷ placed jobs Share of requests of 8 GPUs or more that are placed and effectively occupied Jobs effectively occupied and placed ÷ placed GPUs Placed jobs ÷ jobs a machine can accommodate (placement convention) Placed GPUs ÷ total cluster GPUs; Jain index of readiness rates across seven request-size bins

lower

main

higher higher higher higher — higher

main main main secondary control control

Effective job ratio Effective share among placed jobs Large-request effective coverage Effective jobs per GPU Placeable job ratio Allocation rate / fairness across request-size bins

Note: all metrics refer to a single admission run over the same batch of jobs; “a machine can accommodate” in the denominators means that the requested GPUs do not exceed the maximum per-machine GPUs in the cluster.

5.3

Experimental Setup and Main Results

Data and protocol: The demand side comes from the execution-summary table of Section 3.2; the modeling sample over the nine features contains 2,006,496 records (a 5% row sample of the 40,129,894 labeled records), split into five machine-grouped folds with held-out prediction (the AUC in Table 9 is measured on this split), with an overall readiness rate of 0.4729. The supply side uses three clusters together: two parameterized synthetic heterogeneous clusters (reconstructed from the measured GPUs-per-machine shares and rack homogeneity of Section 4.4, with “empirical replica” and “heavy-tailed” model mixes, respectively) and a real-snapshot cluster (37,707 machines); demand is aggregated in two ways, record-equal weighting (each record counts once) and weighting by requested GPUs. Load 𝜌 is defined as the ratio of the total requested GPUs submitted in an admission to the total cluster GPUs, with five levels from 1.00 to 2.50; each policy runs 3 seeds per cell (random seeds fixed starting at 20261008), and all policies share the same batch of jobs in the same configuration, so comparisons are paired. All code and data-processing scripts are open source at https://github.com/liaobin665/gpu-ras. Table 11 gives the mean of each policy over all 90 experimental configurations. The four main metrics quantify the same thing from the GPU and job perspectives; we also report the cross-cell standard deviation, the paired difference of each policy against FCFS, and its 95% confidence interval and paired-test 𝑝 value. Figure 9 plots the same results in three panels.

16

Policy

Ineffectiveoccupancy Effective ratio job ratio (%) (%)

Effective share among placed jobs (%)

Largerequest effective coverage (%)

Fairness across requestAllocation size bins rate (%) (%)

Crosscell SD

Paired difference vs FCFS (95% CI, pp)

𝑝 value

Table 11. Main results of the admission policies (3 clusters × 2 aggregation conventions × five load levels × 3 seeds, 90 configurations; each configuration contains 9 policies, 810 runs in total). Ineffectiveoccupancy Effective ratio job ratio (%) (%)

Effective share among placed jobs (%)

Largerequest effective coverage (%)

Fairness across requestAllocation size bins rate (%) (%)

Crosscell SD

Paired difference vs FCFS (95% CI, pp)

𝑝 value

FCFS Shortest job first Smallest request first Large request first Random ordering Lookup by model and job type Lookup by job type and duration RAS

43.64 47.70 46.09 41.11 43.48 34.74

34.78 34.70 42.44 27.58 34.87 34.63

48.99 49.77 48.41 51.35 49.05 59.23

46.11 40.22 27.98 60.40 46.07 59.91

98.52 98.53 98.50 98.54 98.52 98.53

89.90 81.45 68.29 65.66 89.71 82.33

5.78 4.79 7.67 5.31 5.75 7.80

— 4.06 [2.75, 5.38] 2.46 [1.91, 3.00] -2.53 [-3.09, -1.97] -0.16 [-0.27, -0.05] -8.90 [-10.02, -7.77]

— 3 × 10−7 9 × 10−9 6 × 10−13 0.003 3 × 10−15

33.88

36.04

59.69

58.01

98.53

87.49

8.37

-9.75 [-11.06, -8.44]

3 × 10−15

27.15

39.48

65.85

63.18

98.53

87.26

11.02

3 × 10−15

True-label upper bound

18.31

46.21

75.31

63.41

98.52

91.81

16.68

-16.49 [-18.62, -14.36] -25.32 [-28.87, -21.78]

Policy

3 × 10−15

Note: all policies use best-fit as the placement rule and differ only in admission order; the first four metrics are the main metrics of Section 5.1 and the last three are controls; “Cross-cell SD” is the standard deviation (SD) of each metric across all configurations; the paired difference is “policy − FCFS” in percentage points, and the paired test uses 𝑛 = 90 batches under the same experimental configuration.

The main results are threefold. (1) The gain is direct and statistically significant: Giving admission priority to jobs predicted to become ready directly reduces the GPUs held. RAS lowers the ineffective-occupancy ratio from 43.64% under FCFS to 27.15%, a decrease of 16.49 percentage points (95% CI [-18.62, -14.36], 𝑝 = 3 × 10−15 ); at the same time, the effective share among placed jobs rises from 48.99% to 65.85%. That is, for the same set of occupied GPUs, the share actually in use rises markedly, while the total number of occupied GPUs is unchanged (both allocation rates are around 98.53%). This is not about admitting more jobs but about making placed jobs more likely to actually run. The decrease is statistically significant: the pairing unit is the job batch under the same experimental configuration, and the confidence interval does not cross zero. (2) The gain comes from the model, not from shuffling the order: The gain depends heavily on the model—simple rules capture only a small fraction of it—and it does not arise from reordering per se. Taking true-label ordering as the theoretical upper bound, RAS captures 65.1% of the improvement space, whereas the two interpretable rules that use only the contribution-analysis conclusions capture 38.5% (lookup by job type and duration) and 35.1% (lookup by GPU model and job type); random ordering yields an ineffective-occupancy ratio of 43.48%, essentially the same as FCFS’s 43.64%, so without readiness information, changing the order alone brings no gain. This is not in conflict with the contribution analysis: contribution share measures the group-level average contribution of attribute groups to the model’s prediction; admission ordering must distinguish, record by record, “this one will become ready and that one will not”, which relies on the joint discriminative power of the nine attributes. Thus “three groups dominate” explains which kinds of attributes the problem concentrates in, but turning it into a usable admission rule requires predicting with all nine attributes jointly, not just the top groups. (3) The gain from admission order grows with load and approaches zero when capacity is ample: When reading Figure 9(c), focus on the gaps between policies rather than the level of any single curve: the gap between FCFS and RAS grows monotonically from 2.40 percentage points at 𝜌 = 1.00 to 29.80 at 𝜌 = 2.50 (9.99, 16.02, and 24.24 at 𝜌 = 1.25, 1.50, and 2.00, respectively). This is consistent with “admission order matters only when something must be traded off”: when capacity is sufficient, everything fits regardless of order, so order cannot play a role. The remaining gap to the true-label upper bound is 17

(a) Decision: a READY record ready probability 0.53 → 1.00

(b) Decision: a NOT-READY record ready probability 0.53 → 0.00

Duration (h)

Schedule wait (s)

Server request load

Duration (h)

Model type

Job type

Machine type

GPU request

Priority class

Server request load

Is generative

Machine type

GPU request

Is generative

Job type

Model type

Schedule wait (s)

Priority class 2

4

6

8

10

-8

SHAP value (logit scale)

-6

-4

-2

0

SHAP value (logit scale)

Figure 8: (a) and (b) show the SHAP decision process for a ready record and a non-ready record, respectively; red contributions raise the model score and blue ones lower it; the horizontal axis is the SHAP value on the logit scale, and the title gives the predicted readiness of the record.

also concentrated at the high-load end: at 𝜌 = 1.00, RAS is only 0.92 percentage points from the upper bound, versus 18.31 and 14.56 at 𝜌 = 2.0 and 2.5; the remaining gap at 𝜌 = 2.5 is smaller than at 𝜌 = 2.0 because the true-label upper bound is itself limited by large-request coverage there, and the two policies are nearly tied on the large-request bins. Two points require explanation. First, the “effective job ratio” counts jobs only, and smallest-requestfirst is higher on it (42.44%) because it places the largest number of jobs into the cluster; the cost is that it sacrifices almost all large requests, with large-request effective coverage of only 27.98%, less than half of RAS, and fairness across request-size bins falling from 89.90 to 68.29. “Highest effective job ratio” is not the same as optimal; it merely shifts the cost to another class of jobs. RAS has a fairness of 87.26%, only slightly below FCFS’s 89.90%. Second, the readiness outcomes in this table come from the data; the admission replay experiment changes only the admission order, and the comparison above holds under Assumption 1 of Section 5.1.

5.4

Ablation and Sensitivity Analyses

The main results answer whether adjusting the admission order works. This section decomposes the question into four testable ones: does the gain really come from predictor quality; does the conclusion hold under a different placement rule; how does the effect vary with load; and does any request-size bin lose readiness. (1) Predictor-quality sensitivity: The four policy classes in Table 11 (no information, weak information, joint prediction, information upper bound) themselves constitute this ablation analysis. The ineffective-occupancy ratio decreases along the information order. Random ordering is essentially equivalent to FCFS, showing that the gain is not a side effect of “shuffling”; the jump from lookup to joint prediction shows that the gain comes from the predictor’s ability to use the nine attributes jointly. See Table 11 for the values. (2) Placement-rule sensitivity: All comparisons above are fixed to best-fit. Replacing the placement rule with first-fit or worst-fit and rerunning the admission schemes gives Table 12. Table 12. Placement-rule sensitivity: ineffective-occupancy ratio (%) under three placement rules. Placement rule

FCFS

RAS

True-label upper bound

Gain (pp)

Best-fit First-fit Worst-fit

43.64 43.63 40.51

27.15 27.09 29.13

18.31 18.32 23.61

16.49 16.54 11.38

18

(a) Lower is better

Big-request ready coverage (%, higher is better)

(c) Effect grows with load

Wasted card ratio (%)

18.3

Oracle RAS (ours) Rule: type x duration Rule: spec x job type Largest request first Random FCFS (FIFO) Smallest request first Shortest job first

27.1 33.9 34.7 41.1 43.5 43.6 46.1 47.7 0

10

20

30

40

40 30 20 10 0

50

60

1.0

1.5

2.0

2.5

Load ρ

Wasted card ratio (%) (b) Position of 9 admission policies

60

40

20

0

FCFS (FIFO) Shortest job first Smallest request first Largest request first Random 0

Rule: spec x job type Rule: type x duration RAS (ours) Oracle

10

20

30

40

50

Wasted card ratio (%, lower is better)

Figure 9: (a) Ineffective-occupancy ratio of the nine admission schemes; (b) their positions in the plane of ineffective-occupancy ratio and large-request effective coverage; (c) ineffective-occupancy ratio as a function of load 𝜌. Note: admission order is the same as in Table 11 and only the placement rule changes; “gain” is FCFS minus RAS under that placement rule; the three placement rules share the same batch of jobs under the same experimental configuration.

RAS’s gain does not depend on a specific placement rule: under all three rules the difference is positive and in the same direction (16.49, 16.54, and 11.38 percentage points for best-fit, first-fit, and worst-fit); best-fit and first-fit are nearly identical, and the gain shrinks but remains under worst-fit. Worst-fit itself also lowers the ineffective-occupancy ratio, but the gain from changing the admission order persists under all three rules. (3) Load sensitivity: The conclusion is the same as the third point of Section 5.3. (4) Request-size sensitivity: Table 13 gives readiness rates for seven request-size bins, to test whether RAS sacrifices any bin. Table 13. Request-size sensitivity: readiness rates (%) across seven request-size bins. Policy

≤ 0.5 GPU

(0.5, 1) GPU

=1 GPU

=2 GPU

=4 GPU

=8 GPU

≥ 16 GPU

FCFS RAS

36.51 52.34

30.10 26.70

27.95 31.16

31.55 39.59

50.37 70.82

52.36 70.56

28.65 43.27

Note: readiness rates are record-weighted within each bin; bins are defined by requested GPUs; the two policies share the same batch of jobs under the same experimental configuration.

Across the seven bins, RAS has a higher readiness rate than FCFS in six, with the largest improvements in the large-request bins (8-GPU: 52.36% to 70.56%; 16+ GPU: 28.65% to 43.27%), consistent with “large-request effective coverage from 46.11% to 63.18%”. The only exception is the narrow (0.5, 1) GPU bin (30.10% versus 26.70%); this bin has the fewest records and carries little weight in the total benefit. The allocation rate of placed GPUs and overall fairness across the seven bins are not harmed (Table 11).

6

Discussion and Conclusion

6.1

Summary

This paper formalizes ineffective occupancy as a measurable object, measures it at the job level, and proposes a readiness-aware scheduling algorithm on that basis. The main findings are as follows. Ineffective occupancy is predictable and has a clear structure. Under the machine-grouped five-fold protocol, readiness prediction reaches an ROC-AUC of 0.8187, better than the majority-class baseline (0.4960) and the 19

lookup baselines. The contribution analysis shows that the model’s score is driven mainly by job profile, duration, and GPU model, with request size contributing least. Ineffective occupancy is spatially concentrated: measured by the variance ratio 𝐷 of Eq. (3), the observed variance of per-machine readiness rates is far larger than the variance expected from a binomial distribution at the same sample size (values of 𝐷 in Figure 7). Readiness-aware scheduling substantially reduces ineffective occupancy. With placement fixed and only the admission order adjusted, RAS lowers the ineffective-occupancy ratio from 43.64% under FCFS to 27.15% and raises large-request effective coverage from 46.11% to 63.18%, without harming capacity utilization. The gain comes mainly from the predictor’s discriminative power rather than from a simple order perturbation: compared with the theoretical optimum when every job’s true readiness label is known, RAS achieves 65.1% of the improvement potential, whereas the two interpretable rules derived from the contribution analysis achieve only 38.5% and 35.1%. The work can be summarized in three respects. (1) Existing work focuses on cluster-level allocation and fragmentation rates: We move the unit of analysis down to individual jobs, reveal the phenomenon of “GPU capacity is sufficient but the job never becomes ready”, and measure its scale and distribution at the job level. (2) Using only the nine attributes observable at admission time: We build a readiness predictor and achieve good ranking performance under a machine-grouped five-fold split. The contribution analysis shows that ineffective occupancy is not uniformly distributed random noise but a structural phenomenon organized along job-side attributes such as job profile, duration, and GPU model; it can therefore serve as a stable basis for admission scoring. (3) Given that the capacity constraint on the placement side is not tight and job attributes cannot be changed: We narrow the adjustable variable to the admission order and propose an admission-ordering rule based on predicted readiness probability. Experiments show that the rule stably reduces ineffective occupancy across clusters and loads, and the allocation rate stays essentially unchanged (98.52% vs 98.53%). In summary, readiness status should become an explicit consideration for heterogeneous GPU cluster schedulers, just like placement success rate.

6.2

Limitations and Future Work

The applicability boundaries of our conclusions are as follows. The readiness label is an observable indicator of readiness status. The release notes define the field as “whether the pod has ever been ready”, and we use it accordingly as an observable indicator of whether a job can enter the ready state. We also exclude a known alternative explanation: ASI’s “Standby” mechanism allows users to voluntarily yield compute, a state recorded by the state_public field of the pod-hour table, which is not in the execution-summary table we use; hence “not ready” as defined here does not include voluntary yielding. Extrapolation requires checking the supply and demand structures. Our conclusions come from observations of a single production cluster. Before extrapolating to other heterogeneous clusters, one should check whether the supply structure (GPUs-per-machine distribution, rack homogeneity) and the demand structure (request-size distribution) are similar. CPU is not examined separately at the mechanism level. We treat CPU cores per GPU as supply-side structural background (its range is shown in Figure 2) and do not include CPU requests in the model. The replay evaluation assumes that readiness outcomes do not change with admission order. In reality, image-cache state (warm vs. cold), concurrent startup contention on the same machine, and network congestion can all cause the same job to obtain different readiness outcomes under different admission orders; end-to-end effects require online deployment tests. Future work can proceed in several directions: first, reproduce our conclusions on other heterogeneous clusters with similar supply and demand structures to test extrapolation; second, include CPU requests in the model and examine the effect of joint CPU–GPU matching on readiness; third, explore finer-grained admission-ordering policies, for example, jointly optimizing predicted readiness, job waiting time, and priority in a multi-objective formulation.

20

References [1] Suyi Li, Lingyun Yang, Haoxuan Yu, Sheng Yao, Tianyuan Wu, Xiaoxiao Jiang, Hanfeng Lu, Kangjin Wang, Chenhao Wang, Shenglin Xu, Lun Wang, Qingyang Duan, Shenghao Liang, Xiu Lin, Meng Zhang, Wenchao Wu, Yinghao Yu, Guodong Yang, Liping Zhang, Wei Wang. Heterogeneity at Hyperscale: Characterization and Scheduling of Large Production AI Clusters at Alibaba. In OSDI ’26 (20th USENIX Symposium on Operating Systems Design and Implementation), pp. 2187-2203, 2026. https://www.usenix.org/conference/osdi26/presentation/li-suyi. [2] Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, Yu Ding. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In NSDI ’22 (19th USENIX Symposium on Networked Systems Design and Implementation), pp. 945-960, 2022. https://www.usenix. org/conference/nsdi22/presentation/weng. [3] Cast AI. 2026 State of Kubernetes Optimization Report: average GPU utilization of 5 percent measured across 23,000 Kubernetes clusters. Cast AI industry report, 2026. https://cast.ai/reports/ state-of-kubernetes-optimization/. [4] Jiahua Huang, Weiwei Lin, Wentai Wu, others. On Efficiency, Fairness and Security in AI Accelerator Resource Sharing: A Survey. ACM Computing Surveys, 2025. doi:10.1145/3721427. https://doi.org/10.1145/3721427. [5] Jing Guo, Zihao Chang, Sa Wang, Haiyang Ding, Yihui Feng, Liang Mao, Yungang Bao. Who Limits the Resource Efficiency of My Datacenter: An Analysis of Alibaba Datacenter Traces. In IWQoS ’19 (IEEE/ACM International Symposium on Quality of Service), pp. 1-10, 2019. doi:10.1145/3326285.3329074. https://doi.org/10.1145/3326285. 3329074. [6] Kunkerkar, Sohan. Queue-Theoretic Admission Control for Multi-Tenant GPU Clusters. arXiv preprint arXiv:2607.28223, 2026. https://arxiv.org/abs/2607.28223. [7] Mamirov, Akhmadillo. Reducing Fragmentation and Starvation in GPU Clusters through Dynamic Multi-Objective Scheduling. arXiv preprint arXiv:2512.10980, 2025. https://arxiv.org/abs/2512.10980. [8] ThakiCloud AI Research. Ghost Occupancy: An LLM-Agent Protocol for Detecting and Reclaiming Idle GPU Deployments in Shared Kueue-Managed Kubernetes Clusters. ThakiCloud technical report, 2026. https://thakicloud.com/ tech-blog/en/research/ghost-occupancy-gpu-reclaim/. [9] Qizhen Weng, Lingyun Yang, Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, Liping Zhang. Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent. In USENIX Annual Technical Conference (ATC), 2023. https://www.usenix.org/conference/atc23/presentation/weng. [10] Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, Chuanxiong Guo. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2019. https://www.usenix.org/conference/nsdi19/presentation/gu. [11] Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, Matei Zaharia. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020. https://www.usenix.org/conference/osdi20/presentation/narayanan-deepak. [12] Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, Eric P. Xing. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2021. https://www.usenix.org/conference/ osdi21/presentation/qiao. [13] Suhas Jayaram Subramanya, Daiyaan Arfeen, Shouxu Lin, Aurick Qiao, Zhihao Jia, Gregory R. Ganger. Sia: Heterogeneity-aware, Goodput-optimized ML-cluster Scheduling. In ACM Symposium on Operating Systems Principles (SOSP), 2023. https://www.pdl.cs.cmu.edu/PDL-FTP/BigLearning/sia_sosp23-final.pdf. [14] ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, Michael Swift. Synergy: Resource Sensitive DNN Scheduling in Multi-Tenant Clusters. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2022. https://www.usenix.org/conference/nsdi22/presentation/lao. [15] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, John Wilkes. Large-scale Cluster Management at Google with Borg. In ACM European Conference on Computer Systems (EuroSys), 2015. https:// research.google/pubs/pub43438/. [16] Kubernetes Authors. Kubernetes Scheduler Configuration: Node Resources Fit and Scoring Plugins. 2024. https:// kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/. [17] Runway Engineering. No Idle GPUs: Managing Research Compute at Runway. Runway engineering blog, 2026. https: //runway.com/news/engineering/no-idle-gpus-managing-research-compute-at-runway. [18] Y. Hu, Y. Wang, C. Zhang, Y. Hong, P. Fan, J. Song, X. Zhou, and Z. Bao. A multi-cluster computing resource scheduling algorithm based on DDPG reinforcement learning. Journal on Communications, 46(11): 147–161, 2025. [19] Hongyi Liu, Yinping Ma, Xiaosong Huang, Lingzhe Zhang, Tong Jia, Ying Li. ORA: Job Runtime Prediction for HighPerformance Computing Platforms Using the Online Retrieval-Augmented Language Model. In ICS ’25 (39th ACM International Conference on Supercomputing), pp. 884-894, 2025. doi:10.1145/3721145.3725757. https://doi.org/10. 1145/3721145.3725757. [20] Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zongheng Yang, Mohammad Alizadeh. Learning Scheduling Algorithms for Data Processing Clusters. In ACM SIGCOMM, 2019. doi:10.1145/3341302.3342080. https: //dl.acm.org/doi/10.1145/3341302.3342080.

21

[21] Sheng Wang, Shiping Chen, Yumei Shi. GPARS: Graph predictive algorithm for efficient resource scheduling in heterogeneous GPU clusters. Future Generation Computer Systems, vol. 152, pp. 127-137, 2024. doi:10.1016/j.future.2023.10.022. https://doi.org/10.1016/j.future.2023.10.022. [22] Yuan, Yitao, others. PRISM: A Workload-Predictive Job Scheduler for Clusters with In-Network Aggregation Capability. In Proceedings of the ACM CoNEXT Workshop on In-Network Computing, 2025. doi:10.1145/3769699.3771587. [23] Yilei Lu, Dongbiao He, Jinlei Jiang, et al. Bridging the GPU Utilization Gap: Predictive Multi-Dimensional Resource Scheduling for AI Workloads. In EuroSys ’26 (21st European Conference on Computer Systems), 2026. doi:10.1145/3767295.3803579. [24] Duan, Jiaang, Xu, Shenglin, Qian, Shiyou, Yang, Dingyu, Wang, Kangjin, Liao, Chenzhi, Yu, Yinghao, Hua, Qin, Hu, Hanwen, Wang, Qi, Wu, Wenchao, Bao, Dongqing, Lu, Tianyu, Cao, Jian, Xue, Guangtao, Yang, Guodong, Zhang, Liping, Chen, Gang. GFS: A Preemption-aware Scheduling Framework for GPU Clusters with Predictive Spot Instance Management. arXiv preprint arXiv:2509.11134, 2025. https://arxiv.org/abs/2509.11134. [25] Haowei Tu, Siming Zhao, Samuel He. LLM-Augmented Salable GPU Supply Forecasting for Disaggregated Recommendation Serving. Journal of Advanced Computing Systems, 2024. http://scipublication.com/index.php/JACS/ article/view/394. [26] Himanshu Singh Baghel. ADAPT: A Self-Calibrating Proactive Autoscaler for Container Orchestration. arXiv preprint arXiv:2605.15788, 2026. https://arxiv.org/abs/2605.15788. [27] Anosan Jayahanthan, Isuru Gunarathne, Charindu Thisara, Sunimal Rathnayake, Malith Jayasinghe. A Self-Adaptive Framework for Predictive CPU Resource Allocation During Container Startup in Kubernetes. In UCC ’25 (18th IEEE/ACM International Conference on Utility and Cloud Computing), 2025. doi:10.1145/3773274.3774262. https: //doi.org/10.1145/3773274.3774262. [28] Alibaba Group. Alibaba Cluster Trace Program (clusterdata): cluster data collected from production clusters for cluster management research. GitHub repository, 2026. https://github.com/alibaba/clusterdata. [29] Leo Grinsztajn, Edouard Oyallon, Gael Varoquaux. Why do tree-based models still outperform deep learning on tabular data?. In NeurIPS 2022 Datasets and Benchmarks Track (also arXiv:2207.08815), 2022. https://arxiv.org/abs/ 2207.08815. [30] Ravid Shwartz-Ziv, Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, vol. 81, pp. 84-90, 2022. doi:10.1016/j.inffus.2021.11.011. https://doi.org/10.1016/j.inffus.2021.11.011. [31] Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, Su-In Lee. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, vol. 2, pp. 56-67, 2020. doi:10.1038/s42256-019-0138-9. https://doi.org/10.1038/ s42256-019-0138-9.

22

A

Description and Statistical Distributions of the Nine Observable Attributes

This appendix completes the value ranges and statistical distributions of the nine observable attributes in Table 3 of Section 3.2, for reproduction and verification. Table A1 gives descriptions and distribution highlights of the nine attributes, and Figure A1 gives the corresponding statistical distributions. Table A1. Description and descriptive statistics of the nine observable attributes. Feature

Meaning

Values

Distribution highlights

Relation to readiness

gpu_spec_public

Accelerator model (request side)

Categorical, 13 classes

Readiness 30.0 (XPU-C) to 58.4 (XPU-E), a range of 28.4 pp

priority_class

Priority class

job_type_public

Job type

Categorical, 3 classes Categorical, 6 classes

Most frequent XPU-E 28.5%; top three together 57.9%; smallest class A30 only 0.04% LP 82.8%, HP 16.1%, Other 1.1%

model_type_public

Model type

is_genai_request

Generative-request flag Requested GPUs

gpu_request

duration_hours

Declared duration (hours)

schedule_delay_sec Scheduling delay (seconds) server_req_count

Concurrent requests on the same machine

Categorical, 6 classes Binary 0/1

offline_inference 79.7%, online_inference 12.3%, training 7.4% (remaining 3 classes together 0.6%) genai 79.4%, cv 8.9%, rec 7.0% (remaining 3 classes together 4.7%) 77.8% of records take 1

Continuous, including fractional values Continuous, heavy-tailed

Median 1, P90 2; 11.4% below 1, 0.72% at 16 or above

Continuous, more than half zero Continuous

Median 0, 55.3% are 0, P90 224

Median 0.42, mean 5.20, P90 4.65, maximum 1,824

Median 145, mean 192, P90 380

Readiness 44.1 (LP) to 79.9 (Other), a range of 35.8 pp Readiness 42.4 (offline_inference) to 91.6 (training), a range of 49.2 pp Readiness 43.0 (genai) to 86.8 (rec), a range of 43.8 pp Readiness 64.9 at 0 and 42.3 at 1, a difference of 22.6 pp Readiness 58.7 at the low end and 56.6 at the high end, a difference of 2.1 pp Readiness 56.1 in the short bin and 64.3 in the long bin, a difference of 8.1 pp Readiness 44.7 at the low end and 53.8 at the high end, a difference of 9.1 pp Readiness 37.6 at the low end and 54.1 at the high end, a range of 16.5 pp

Note: statistics are computed on the modeling sample (2,006,496 records from 32,635 machines) and differ from the full-sample conventions of Section 3.2 by less than one percent; “low” and “high” refer to values at or below the 10th percentile and at or above the 90th percentile, respectively; for categorical attributes only categories with at least 1% of records are listed; all nine attributes have a missing rate of 0.

Table A1 and Figure A1 together give three facts relevant to the main line of this paper. First, the demand side is highly concentrated: the distributions of job type, model type, and the generative-request flag are nearly identical: offline inference accounts for about 80% of records by job type, and generative requests account for a similar share, indicating that the trace is dominated by offline generative inference. Second, request sizes are dominated by a single GPU, but fractional requests exist (11.4% below one), so requested GPUs are not strictly integer. Third, and most relevant here, the difference in readiness rates across the nine attributes is not reflected in request size (a range of only 2.1 percentage points) but is concentrated in job type, model type, and GPU model (ranges of 28.4 to 49.2 percentage points). This is consistent with the feature-group contribution analysis of Section 4.3: what drives whether a job becomes ready is job profile and GPU model, not how many GPUs it requests.

23

Accelerator type (13)

Job type (6) 28.53

15.25 14.13 11.05 10.13 7.15 5.08 2.68 2.10 2.09 1.51 0.27 0.04 0

10

79.67

offline_inference

7.37

training

other

0.18

embedding

1.24

dev

0.07

dev

0.06

75

100

16.14 1.06

0

80 60 40

22.2

20 0

Other

1

Records (log)

104

103 101

102

103

100

60 40 20

11.44

6.88

0 <1

=1

=2

1.03 1.38 0.72 =4

=8

≥ 16

Concurrent requests per server

55.3% = 0 s

104

105 104 103

103 Declared duration (h, log)

75

78.55

80

0

5

50

GPU request

Records (log)

10

100

25

Share of all records (%)

Schedule delay

105

10−1

0

Is generative request Share of all records (%)

Share of all records (%)

50

77.8

Declared duration

Records (log)

25

Share of all records (%)

50

10−2

7.02 3.39

0

75

HP

rec unknown

Priority class (3)

LP

8.92

0.37

30

82.80

25

cv

unknown

Share of all records (%)

100

79.37

genai

12.34

online_inference

20

Model type (6)

Share of all records (%)

XPU-E H20 XPU-A L20 A10 XPU-C XPU-B A100 XPU-D H20-141GB H800 A800 A30

100

102

104

Schedule delay (s, log)

100

101

102

103

Concurrent requests per server (log)

Figure A1: Statistical distributions of the nine observable attributes. (a)–(f) show the share of records by value; in (g)–(i), light color denotes all records and dark color the ready records, the horizontal axis is logarithmic, and the dashed line marks the median.

24

Related documents

• Ineffective Occupancy in Heterogeneous GPU Clusters: Prediction, Contribution Analysis, and Readiness-Aware Scheduling#1390012
• Arachne: Learning to Plan Parallel Training on Dynamic Heterogeneous Clusters#1119317
• Arachne: Learning to Plan Parallel Training on Dynamic Heterogeneous Clusters#1108717
• Beaver: Elastic GPU Sharing between ML and Latency-Critical vRAN Workloads#1186836
• Beaver: Elastic GPU Sharing between ML and Latency-Critical vRAN Workloads#1181594