MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs

This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013

1

MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs

arXiv:2609.02109v1 [cs.DC] 2 Sep 2026

Youssef Ennouri

and Soonhoi Ha , Fellow, IEEE

Abstract—Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across N ∈ {2, 3, 4, 5, 6} show high predictive accuracy (R2 ≈ 0.96) with an empirical sample budget that grows approximately linearly in N , in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to a N = 5 problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about 5× faster than exhaustive surrogate search. Index Terms—GPU Scheduling, Heterogeneous AI Inference, Performance Modeling, Runtime Systems, Surrogate Modeling

I. I NTRODUCTION odern inference deployments increasingly co-locate heterogeneous AI workloads such as large language models (LLMs) and vision models on shared GPUs to improve utilization. Concurrent execution, however, induces non-linear interference over shared resources, including streaming-multiprocessor (SM) occupancy, video memory (VRAM) capacity, and memory bandwidth. This interference depends jointly on model variants, runtime parameters, and workload intensity, making analytical modeling impractical. Efficient scheduling therefore requires a fast surrogate that predicts concurrent performance without live benchmarking. While established serving systems like Nexus [1], Clipper [2], and Clockwork [3] improve utilization through batching, placement, and allocation, their scope is limited to homogeneous or isolated workloads. This limitation, along with interference-aware GPU-sharing systems such as Orion [4], ElasticRoom [5], Miriam [6], and automated runtime-aware scheduling [7] underscores the need for accurate performance prediction under co-execution.

M

Corresponding author: Soonhoi Ha. This work was supported by the National Research Foundation of Korea (NRF), funded by the Korean government (MSIT), under Grant RS-2026-25483479, with research facilities provided by the Institute of Computer Technology (ICT), Seoul National University. The authors are with the Department of Computer Science and Engineering, Seoul National University, Seoul 08826, Republic of Korea (e-mail: youssefennouri@snu.ac.kr; sha@snu.ac.kr).

Scalability, however, remains the obstacle: training a surrogate on joint N -model configurations needs a profiling budget that grows combinatorially in N , and existing surrogate-based approaches [8] do not address this for heterogeneous GPU inference. To address this limitation, we propose a MeanField surrogate formulation with approximately linear empirical sample-complexity growth in N and integrate it into a practical runtime scheduler. This paper makes three main contributions: • We introduce the MeanField surrogate, a scalable, contention-aware predictor based on local model configurations and aggregate GPU state. • Under a stratified sampling protocol, we empirically show that the profiling budget required to reach R2 ≥ 0.95 grows approximately linearly over N ∈ {2, . . . , 6}, with n∗ ≈ 20N . • We develop a runtime scheduler by integrating the MeanField surrogate into a genetic algorithm (GA)-based framework, which is subsequently validated at N = 5 under dynamic workload scenarios.

Model 1 config λ1

Model 2 config λ2

Shared GPU

Runtime Monitor

SMs • VRAM • Memory BW

Workload + GPU state

⋮ Observed State

Model N config λN

apply λt*

st

GA + MeanField Runtime scheduler

Fig. 1. The runtime monitor samples the observed state st , which combines workload and aggregate GPU information. The MeanField surrogate guides the GA scheduler in selecting the joint configuration λ⋆t under SLA and VRAM constraints.

II. RUNTIME S CHEDULING M ODEL The system architecture is illustrated in Fig. 1. Because exhaustive online benchmarking is infeasible, the scheduler selects configurations for the co-executing models using a lightweight runtime surrogate to predict their performance. To formalize the scheduling problem, we consider N heterogeneous AI models concurrently executing on a shared GPU. The formulation is agnostic to the specific application types and applies to any heterogeneous workloads exposing

© 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013

a discrete configuration space; in this work it is instantiated and evaluated on two workload families: LLMs based on the Qwen2.5 series and vision models based on the YOLO11 family. Each model exposes a discrete configuration space Λi = Vi × Pi , where Vi denotes model variants and Pi runtime parameters. For the LLM, Vi = {FP16, AWQ} (16-bit floating-point and Activation-aware Weight Quantization variants, respectively) and Pi = {request concurrency} with values in {1, 2, 4}; for YOLO11, Vi = {n, s, m} and Pi = {skip rate, imgsz} with values in {1, 2, 4} and {320, 480, 640} respectively. Here, request concurrency controls the number of parallel LLM requests and therefore affects KV-cache pressure and SM occupancy. The scheduler must select one configuration for each model, forming the joint configuration space: Λ=

N Y

Λi

(1)

i=1

At each scheduling step, the runtime system observes the state st = (ρt , mt , ut ) (2) where ρt denotes the workload level, mt the current VRAM usage, and ut the aggregate GPU utilization. The scheduler selects λ∗ = arg maxλ∈Λ J(λ, st ), where the utility J rewards throughput-weighted quality while penalizing unnecessary reconfigurations and SLA violations: X X X J= ωi fi Qi −α ηi crel max(0, τi +δ−fi )2 (3) i −β i

i

P

i

The first term i ωi fi Qi is a throughput-weighted quality score: a configuration is rewarded only when it is both fast and accurate. The normalized throughput fi ∈ [0, 1] uses per-variant min–max scaling, fi = (fˆi − fvmin )/(fvmax − fvmin ), where fˆi is the raw predicted throughput (tokens/s for the LLM, FPS for vision) and fvmin , fvmax are the extrema observed for variant v during profiling. Per-variant rather than global normalization is essential: a faster variant (AWQ) would otherwise dominate the scale and render every configuration of a slower one (FP16) effectively infeasible. The quality term Qi captures variant accuracy (QLLM = {1.000, 0.979} for FP16/AWQ from relative perplexity; QVIS = {0.73, 0.96, 1.00} for YOLO11n/s/m from mean Average Precision, mAP50-95), scaled so the best variant scores 1. The priority weights ωi are operator-defined and encode application-specific importance; in our experiments all models are given equal priority ωi = 1 unless a scenario explicitly elevates one. They may further be adapted online as ωi = ω̄i + γ max(0, τi − f¯i ), with γ = 2.0. (t) Reconfiguration overhead is modeled through ηi = 1[vi ̸= (t−1) vi ], which indicates whether model i changes variant and therefore incurs an amortized reload cost crel i (cLLM = 30 s, cVIS = 5 s). The coefficient α = 1/60 ≈ 0.017 is the inverse of a 60-second amortization horizon, so a reload is accepted only when its predicted gain over the next 60 s exceeds the reload cost itself.

2

SLA compliance is enforced through a hard per-model throughput floor τi . In the N = 5 evaluation, τi is set to the empirical P25 of each model’s normalized performance distribution. The quadratic penalty uses a safety margin δ = 0.05 above this floor, discouraging configurations that remain feasible but operate too close to the SLA boundary. We set β = 5.0. Hard constraints enforce both VRAM feasibility and minimum throughput requirements: X riVRAM ≤ R, fi ≥ τi (4) i

VRAM-infeasible configurations are removed when constructing Λ. Candidates violating any throughput floor receive J = −∞; infeasible offspring are repaired by replacement with a randomly sampled feasible individual. III. M EAN F IELD S URROGATE FOR S CALABLE S CHEDULING A. Offline Profiling Experiments are conducted on a single NVIDIA RTX 3090 (24 GB GDDR6X) running CUDA 12.6 and Ubuntu 20.04. LLM workloads are served through vLLM [9] with FP16 and AWQ variants. Vision workloads use the Ultralytics engine. Offline profiling systematically explores the configuration space to generate training data for surrogate learning. The baseline N = 2 configuration space contains 162 concurrent configurations (6 LLM × 27 Vision), evaluated under four workload regimes (low, medium, high, burst), yielding 648 profiling measurements. For larger systems (N > 2), additional LLM and vision models were instantiated and profiled under the same workload regimes. Each cell is measured 10 times (two warmup passes discarded) under fixed GPU clocks, thermal stabilization below 70 ◦ C, and barrier-synchronized launches. A profiling sample denotes one sampled joint configuration evaluated under one workload regime. The ten repeated measurements are averaged into a single training row and are not counted as distinct samples. For N > 2, joint configurations are sampled stratified by model type, variant, and workload level rather than enumerating the full Cartesian product. The interference factor for model i is defined as: ∆i = fiisolated − ficoncurrent

(5)

Profiling reveals substantial interference: AWQ LLM variants lose 8.4% throughput on average under co-execution, while vision models lose 17–24% FPS depending on model size. Analysis of variance (ANOVA) [10] indicates workload intensity as the dominant factor for vision degradation (p < 0.01), whereas the vision-model skip rate dominates LLM slowdown by reducing shared SM and memory-bandwidth contention. These results suggest that interference is primarily driven by aggregate resource pressure rather than by specific model pairings, motivating the MeanField approximation proposed in this work.

© 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013

Mean Accuracy (R 2 )

(a) N = 3

(b) N = 4

0.8

0.6

1.0

100

50

150

200

(c) N = 5

250 0

50

100

150

200

250

100

150

200

250

(d) N = 6

0.8

0.6 0

100

50

150

200

Profiling Samples (n)

250 0

50

Profiling Samples (n)

Fig. 2. Mean R2 vs. training sample count n for the MeanField surrogate at N ∈ {3, 4, 5, 6}. At every N , the surrogate reaches the R2 ≥ 0.95 threshold within a modest sample budget, with the required budget increasing approximately linearly as N grows.

Fig. 2 shows the MeanField learning curves under increasing training budgets. At every value of N the surrogate reaches the target accuracy with a small number of samples and then plateaus near R2 ≈ 0.95–0.97, with low inter-seed and inter-fold variability. 120

Required samples n

C. Scalability Challenge The main limitation of direct surrogate modeling is the growth of the joint configuration space. A fully joint predictor requires profiling complexity of O(k N ), where k denotes the number of configurations per model, and its input dimensionality grows with N . The Joint MLP baseline in Table I therefore serves only as an N = 2 reference, as both profiling requirements and input dimensionality become prohibitive beyond small deployments. The MeanField formulation avoids this growth: its profiling cost scales as O(N · k) rather than O(k N ), while prediction accuracy remains high (Section IV). To characterize this trade-off, the MeanField surrogate is trained from scratch and evaluated using stratified cross-validation across N ∈ {2, 3, 4, 5, 6}, measuring both its asymptotic accuracy and the number of samples required to reach convergence.

1.0

0

Mean Accuracy (R 2 )

The observed state affects scheduling through the surrogate prediction fˆi = gi (λi , st ): raw throughput fˆi is normalized into fi , which then enters both the throughput-quality reward and the SLA penalty in Eq. (3). During scheduling, (mt , ut , ρt ) are observed context variables obtained from runtime monitoring or simulation traces and are held fixed during GA search. The surrogate thus ranks candidate configurations conditioned on the current workload state rather than predicting future GPU-state evolution. Our measurements support this approximation: aggregate utilization, memory pressure, and workload level provide sufficient predictive information to retain high accuracy across the evaluated model mixtures. The surrogate can therefore be trained directly from concurrent N -model executions, and its profiling cost grows as O(N · k) in the number k of sampled workload states rather than combinatorially. Each predictor gi is a 64–32 MLP with BatchNorm, ReLU, Dropout(0.1), and Sigmoid output. Categorical features are ordinally encoded and targets are min–max normalized per variant. This architecture was selected by ablation at N = 3 and n = 80, providing the best accuracy–latency trade-off and sub-millisecond local inference (Table I).

IV. E XPERIMENTAL R ESULTS A. Sample-Complexity Scaling We evaluate surrogate sample efficiency under identical training protocols across N ∈ {2, 3, 4, 5, 6}. Performance is measured using the mean R2 across targets, and convergence is defined as the minimum training set size n∗ required for the mean R2 to reach 0.95. Results are averaged over 25 evaluations (5 random seeds × 5-fold stratified cross-validation), capturing both data-sampling and optimization variance.

¤

B. MeanField Surrogate The name follows the mean-field approximation of statistical physics, in which the many-body interactions acting on a particle are replaced by a single aggregate field rather than modeled pairwise. By analogy, each model is predicted to respond not to the identities of its co-runners but to an aggregate field, the observed GPU state, that summarizes their combined resource pressure. The surrogate therefore trains one lightweight predictor per model from the model’s local configuration and aggregate GPU state: fˆi = gi (λi , mt , ut , ρt ). (6)

3

n ¤ = 20N measured

100 80 60 40

2

3

4 System Size (N)

5

6

Fig. 3. Empirical convergence sample count n∗ for the MeanField surrogate. The observed budget grows approximately linearly, n∗ ≈ 20N , over N ∈ {2, 3, 4, 5, 6}.

Fig. 3 summarizes this behavior through the empirical convergence sample count n∗ . MeanField reaches the target © 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013

accuracy after approximately (40, 60, 80, 100, 120) samples for N ∈ {2, 3, 4, 5, 6}, consistent with a near-linear trend (n∗ ≈ 20N ) and in sharp contrast to the combinatorial O(k N ) cost of a joint surrogate. B. Prediction Accuracy Table I reports the mean R2 of the MeanField surrogate at full training budget. Accuracy remains high (R2 > 0.95) up to N = 6 and stays within a narrow band of the N = 2 Joint MLP reference, confirming that the scalable MeanField formulation sacrifices little asymptotic accuracy while removing the joint surrogate’s combinatorial profiling cost. TABLE I A SYMPTOTIC PREDICTION ACCURACY (M EAN R2 ± STD OVER 25 EVALUATIONS ) OF THE M EAN F IELD SURROGATE ACROSS N , WITH THE JOINT MLP SHOWN AS AN N = 2 REFERENCE . N

Joint MLP

MeanField

2 3 4 5 6

0.9826 ± 0.002 – – – –

0.9860 ± 0.003 0.9779 ± 0.006 0.9689 ± 0.006 0.9645 ± 0.014 0.9588 ± 0.013

4

The GA remains within 0.10% of exhaustive search with zero SLA violations. StaticFull is also near-optimal, indicating limited adaptation benefit in these scenarios. Under realistic calibration budgets, however, StaticBudgetK loses 10.51% utility at K = 500 and 5.22% at K = 5000. The main benefit is therefore profiling-efficient near-optimal search rather than adaptation alone. V. C ONCLUSION This paper proposed a MeanField surrogate for scalable runtime scheduling of concurrent heterogeneous AI workloads on a shared GPU. Across N ∈ {2, 3, 4, 5, 6}, it achieved high prediction accuracy (R2 ≈ 0.96) with approximately linear empirical sample-complexity growth. Integrated into a GA scheduler, it scaled to an N = 5 problem with 78,732 feasible configurations, remaining within 0.10% of exhaustive search with zero SLA violations across eight dynamic scenarios, with a median end-to-end decision latency of 26 ms. Future work will extend the LLM workload model to capture prefill/decode phases and KV-cache occupancy. R EFERENCES

C. Runtime Scheduling Validation The MeanField surrogate drives a GA scheduler at N = 5, over a VRAM-feasible space of 78,732 configurations spanning three LLMs and two YOLO-family models. The 7B LLM is AWQ-only due to the 24 GB memory limit, and per-model SLA floors are set to the empirical P25. The GA uses 100 candidates, crossover probability 0.7, mutation probability 0.1, tournament selection (k = 3), elitism, and 30 generations. MeanField evaluates all local configurations in a median of 8.5 ms, followed by 17 ms for GA search, yielding a 26 ms end-to-end decision latency. Exhaustive surrogate search requires 131 ms, making the GA about 5× faster. TABLE II S CHEDULING RESULTS AT N = 5 OVER EIGHT DYNAMIC SCENARIOS . Scheduler

Mean J

Avg. Gap (%)

Worst Gap (%)

SLA (%)

Exhaustive Search StaticFull GA (ours) StaticBudget (K = 5000) StaticBudget (K = 500) Random

2.744 2.743 2.742 2.600 2.443 1.428

0.00 0.05 0.10 5.22 10.51 49.13

0.00 0.41 0.38 7.36 17.16 58.71

0.00 0.00 0.00 0.00 0.00 3.45

Table II reports mean utility over eight dynamic scenarios and three seeds. Exhaustive denotes a per-step argmax over the same surrogate-predicted landscape, so the reported gap measures search quality only. We compare against StaticFull, StaticBudgetK, and Random. StaticFull is an adaptivity ablation with full-space access, whereas StaticBudgetK selects one fixed configuration from only K offline samples.

[1] H. Shen, L. Chen, Y. Jin, L. Zhao, B. Kong, M. Philipose, A. Krishnamurthy, and R. Sundaram, “Nexus: A GPU cluster engine for accelerating DNN-based video analysis,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19), 2019, pp. 322–337. [2] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system,” in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017, pp. 613–627. [3] A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace, “Serving DNNs like clockwork: Performance predictability from the bottom up,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’20), 2020, pp. 443–462. [4] F. Strati, X. Ma, and A. Klimovic, “Orion: Interference-aware, fine-grained GPU sharing for ML applications,” in Proceedings of the 19th European Conference on Computer Systems (EuroSys ’24), 2024, pp. 1075–1092. [5] L. Ma, H. Chen, E. Shao, L. Wang, Q. Chen, and G. Tan, “Elasticroom: Multi-tenant DNN inference engine via co-design with resource-constrained compilation and strong priority scheduling,” in Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’24), 2024, pp. 1–14. [6] Z. Zhao, N. Ling, N. Guan, and G. Xing, “Miriam: Exploiting elastic kernels for real-time multi-DNN inference on edge GPU,” arXiv preprint arXiv:2307.04339, 2023. [7] F. Yu, S. Bray, D. Wang, L. Shangguan, X. Tang, C. Liu, and X. Chen, “Automated runtime-aware scheduling for multi-tenant DNN inference on GPU,” in Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2021. [8] Y. Han, Z. Nan, S. Zhou, and Z. Niu, “DVFS-aware DNN inference on GPUs: Latency modeling and performance analysis,” arXiv preprint arXiv:2502.06295, 2025. [9] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23), 2023, pp. 611–626. [10] D. C. Montgomery, Design and Analysis of Experiments, 9th ed. Wiley, 2017.

© 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Related documents

• MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs#822511
• SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters#151760
• On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend#353032
• ChurnBench: A Drift-Aware Benchmark Demonstrating That Refresh Scheduling, Not Cache Age, Governs Staleness in Agentic AI#673571
• Characterizing the Scalability and Performance of Large-Scale AI Training Under Multi-Tenancy#926227