arXiv CS open access

Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

Andrew Sheinberg et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

GPU clusters have become essential computing resources for AI and scientific research in academia and industry. However, their high cost and growing demand often result in long queueing delays. While previous monitoring efforts have operated at second- or minute-level frequencies, they primarily inf

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

arXiv:2610.36329v1 [cs.AI] 26 Oct 2026

ANDREW SHEINBERG∗ , SAMYAK GUPTA∗ , and KAI LI, Princeton University, USA GPU clusters have become essential computing resources for AI and scientific research in academia and industry. However, their high cost and growing demand often result in long queueing delays. While previous monitoring efforts have operated at second- or minute-level frequencies, they primarily informed job-level scheduling for clusters and lacked the detail needed to optimize GPU-level utilization. This paper presents an in-depth study of a university research GPU cluster, analyzing resource usage at 100 ms intervals. We have developed a lightweight monitoring tool that leverages Nvidia’s DCGM library to capture fine-grained “compute utilization” metrics within GPUs. Deployed on a cluster with 89 nodes and 319 GPUs, our tool collected data over several months. This paper focuses on a 10-day period with 7,399 jobs submitted by 150 users from 12 departments. Our study reveals three key findings: (1) the GPU cluster exhibits significant underutilization, with over 70% of jobs using less than 20% of GPU memory and 50% averaging below 5% Streaming Multiprocessor (SM) utilization; (2) 80% of jobs have a drastic mismatch between their requested and actual durations, significantly limiting the scheduler’s effectiveness; and (3) the multi-GPU jobs we examined show highly periodic communication patterns. These findings suggest there is plenty of room in the middle: substantial utilization gains are available through better job scheduling and GPU multitasking, independent of hardware or application improvements. Our monitoring solution is available as open-source software on GitHub. CCS Concepts: • Computer systems organization → Parallel architectures; Distributed architectures; • Hardware → Communication hardware, interfaces and storage. Additional Key Words and Phrases: GPU monitoring, GPU utilization, cluster scheduling, multiplexing

Introduction

GPU clusters have become the engines of modern AI, powering services and research across diverse domains. The insatiable appetite of large language models and the burgeoning demands of AI applications in science and other fields are driving an exponential growth in computing requirements. While GPU vendors continue to deliver increasingly powerful hardware with impressive performance, the associated costs have skyrocketed. For instance, Nvidia’s Blackwell GPUs command a hefty price tag of $30,000–$40,000 [Haddad 2024]. However, GPU clusters are significantly underutilized [Hu et al. 2021; Jeon et al. 2019; Xiao et al. 2020], raising the critical question: How can we substantially improve GPU cluster utilization? In many batch-oriented GPU clusters, GPUs are allocated exclusively to individual jobs for the duration of their reservations [Xiao et al. 2020]. Under this allocation model, a GPU remains unavailable to other jobs even when its assigned workload uses only a small fraction of its compute or memory resources. Sharing a GPU across independent jobs—through temporal multiplexing, concurrent execution, or a combination of the two—offers an opportunity to recover this unused capacity. Modern GPUs provide mechanisms for such sharing, but their hardware and runtime support imposes tradeoffs among allocation flexibility, performance isolation, and memory management. For example, NVIDIA’s Multi-Process Service (MPS) [Nvidia 2020b] enables concurrent execution ∗ Both authors contributed equally to this research.

Authors’ Contact Information: Andrew Sheinberg, asheinb@princeton.edu; Samyak Gupta, samyakg@princeton.edu; Kai Li, li@cs.princeton.edu, Princeton University, Department of Computer Science, Princeton, New Jersey, USA.

Andrew Sheinberg, Samyak Gupta, and Kai Li

across processes, while Multi-Instance GPU (MIG) [Nvidia 2020a] provides stronger isolation through predefined hardware partitions. These mechanisms provide useful building blocks, but efficiently adapting resource allocations to changing job demands remains a systems challenge. GPU memory introduces an additional constraint. Although GPUs support virtual addressing and memory protection, efficiently sharing their limited high-bandwidth memory requires managing the combined memory demands of multiple jobs. Conventional device allocations are not automatically pageable to host memory [Nvidia 2024b], while Unified Memory supports oversubscription with potentially substantial page-fault and data-migration costs [Nvidia 2024a]. These challenges motivate a closer examination of when GPU resources are unused, how long those intervals last, and which workloads could share resources effectively. Previous monitoring studies have relied on coarse-grained metrics, typically with sampling intervals of seconds to minutes, providing limited insights into the fine-grained utilization patterns of GPU resources. To close this gap, we have developed a fine-grained monitoring tool capturing utilization data at 100 ms intervals, the finest granularity supported by Nvidia’s Data Center GPU Manager (DCGM) [Nvidia 2020c] and the same length as Linux’s default round-robin time slice. We have deployed this lightweight, scalable tool across a university research cluster of 89 nodes and 319 GPUs for several months. This paper analyzes data collected during a 10-day observation period,1 encompassing 7,399 jobs submitted by 150 users across 12 departments. The analysis reveals three key findings: • Significant underutilization: Over 70% of jobs used less than 20% of GPU memory, and 50% averaged below 5% SM utilization. This underutilization highlights a substantial gap between the capabilities of modern GPUs and the actual resource demands of many workloads, leaving much of the cluster’s computational power untapped. • Pervasive mismatch between requested and actual job durations: Approximately 80% of jobs show a significant mismatch between their requested and actual runtimes, with one-third hitting their requested time limit. This discrepancy results in poor scheduling decisions, including inflated queue times and inefficient resource allocation, fundamentally undermining the effectiveness of current schedulers. • Highly periodic communication patterns in multi-GPU jobs: The multi-GPU workloads we examined demonstrated recurring communication patterns, including predictable bursts of NVLink traffic and synchronization phases. These periodic behaviors present opportunities for system designers to develop predictive resource allocation strategies, minimizing interference and optimizing the scheduling of tasks. These findings show great opportunities to improve GPU cluster efficiency. Much work on computing efficiency has focused on the bottom of the stack (faster hardware) [Feynman 1960] or the top (better-optimized applications) [Leiserson et al. 2020]. Our results show there is also plenty of room in the middle: the systems layer that allocates, schedules, and multiplexes GPUs across jobs can recover much of the capacity that today’s clusters leave idle. Using insights from fine-grained monitoring data, system designers can build scheduling and multitasking solutions that improve GPU utilization while preserving performance isolation, making research GPU clusters more effective and cost-efficient. 2

Monitoring Framework and Environment

We have developed and deployed a fine-grained monitoring tool to analyze activities on a university GPU cluster. 1 This study was conducted in fall 2024; the measurements, hardware, and related work discussed reflect that period.

Plenty of Room in the Middle

2.1

GPU Cluster Overview

We conducted our study on a university research cluster comprising 89 nodes interconnected through a shared general-purpose networked file system [Schmuck and Haskin 2002]. The cluster is shared by researchers across multiple departments and includes the following hardware configurations: • High-Performance Nodes (20): Each equipped with AMD EPYC Rome CPUs (128 cores), 768 GB RAM, and two NVIDIA A100 PCIe 40 GB GPUs. • Standard Nodes (69): Each featuring Intel Ice Lake CPUs (48 cores), 100 GB RAM, and four NVIDIA A100 SXM4 80 GB GPUs. All nodes are equipped with a 100 Gb/s ConnectX-6 InfiniBand NIC and a 1 Gb/s Ethernet NIC for out-of-band traffic. 2.2

Design Goals and Challenges

To effectively analyze GPU utilization without disrupting cluster operations, we established the following key design goals for the monitoring system: • Minimal Overhead: The monitoring tool must have minimal impact on cluster performance, with low CPU, memory, and storage usage, as data collection was planned over several months across hundreds of GPUs. • Operational Simplicity: The tool needed to be straightforward to deploy and operate, requiring minimal configuration and maintenance to avoid burdening system administrators. • Fine-Grained Metrics: Capturing high-resolution metrics for Streaming Multiprocessors (SMs) utilization, memory utilization, SM occupancy, PCIe traffic, NVLink traffic, and Tensor Core usage, providing a comprehensive view of GPU performance. • Robust and scalable: The tool should be resilient to node failures and deployable to a large-scale GPU cluster. We have observed that the standard nvidia-smi metric, commonly used to measure GPU utilization, often misrepresents actual resource usage. It reports only the fraction of cycles during which a kernel is running or a copy engine is active, failing to account for fractional SM usage. For example, a job using just one SM out of 108 on an A100 GPU could still be reported as having 100% utilization. To address this limitation, we utilized the “SM Activity” metric from Nvidia’s DCGM API [Nvidia 2020c], which provides a more precise measure of GPU compute utilization. This fine-grained metric allowed us to observe intra-GPU spatial utilization and uncover significant discrepancies in reported utilization. Figure 1a illustrates the discrepancy between nvidia-smi reported utilization and actual GPU usage by showing an example of an infinite-loop kernel (20 seconds) launched with a variable number of blocks (on an A100 with 108 SMs). DCGM-reported utilization gives an accurate view of intra-GPU spatial utilization. After deploying the monitoring system we compared the real-world differences. Figure 1b highlights the ratio of average utilization reported by the two approaches. This systematic discrepancy has significant implications for cluster efficiency analysis and scheduling decisions, especially in scenarios involving multitasking. 2.3

Design, Implementation and Deployment

To meet the design requirements, we design a monitoring tool to run on each host node, recording its data to a database. The main loop of the tool queries DCGM along with /proc to retrieve metrics of each attached GPU at a given frequency, writes the returned results to a buffer, and then goes to sleep. When the buffer is full, the tool writes its content to its dedicated DB. At a much lower

Andrew Sheinberg, Samyak Gupta, and Kai Li

1.0

0.8

25 20

0.6

NVML/SMI DCGM

% Jobs

Utilization %

0.4

10 0.2

0.0

0 0

40 50 60 70 Blocks Launched

100 108

(a) Running an infinitely looping kernel on a variable number of SMs. The popularly used NVML/SMI metric may significantly over-estimate the actual fraction of GPU SMs used.

20x 40x 60x 80x Mean over-estimation by nvidia-smi

100x

(b) Distribution of per-job utilization overestimates reported by nvidia-smi compared to the DCGM metric.

Fig. 1. Comparisons of nvidia-smi and actual (reported by DCGM) GPU SM utilization.

frequency (e.g., every hour), the tool queries the GPU cluster job scheduler (currently SLURM) for job information and writes to a separate table in the DB. To achieve the low-overhead and efficiency goals, we developed the monitoring tool as a standalone executable, written in a few hundred lines of C. We set up a SQLite DB for each host node to avoid any synchronization among multiple hosts. This design decision makes the monitoring tool robust (immune to host failures), though it requires some work to aggregate the DB data offline for data analysis. For our data collection, the sample period is configured to 100 ms, aligning with modern OS preemptive scheduling time slices. The local buffer size is configured to 600 samples to amortize the overhead of writing to DB. With such configurations, the buffer consumes < 2 MB. On a host node with 4 GPUs, the sample loop takes about 800 microseconds. Every 60 seconds, the tool spends about 60 ms to write ≈ 24k rows totaling 1 MB to DB. Thus, the overhead is ≈ 0.9% on a single CPU core. The total DB storage is about 1.5 GB/day per node. The overhead on GPUs is negligible. Because of its minimal overhead and robustness, the university IT administrators allowed us to deploy the monitoring system on a production research GPU cluster. Our tool is open source and available on GitHub. 3

Job Characteristics

Job Distribution by Department During the 10-day period, 7,399 jobs were submitted by 150 users from 12 departments. Table 1 summarizes the distribution of jobs by department (inferred from users’ group affiliations) and the proportion of GPU time reserved by each department, providing insights into the cluster’s diverse usage patterns. Computationally intensive fields such as Chemistry and Chemical & Biological Engineering dominate GPU usage, while departments like Neuroscience and Sociology, though contributing fewer jobs, add to the cluster’s workload diversity.

Plenty of Room in the Middle

Department

# of Jobs

Prop. of Reserved GPU-Time

Astronomy Chem & Bio Eng. Chemistry Comp. Science Elect. Eng. Genomics Mech. Eng Neuroscience Op. Research Physics Psychology Sociology

136 2184 2837 306 227 119 241 385 181 569 28 90

5.6% 22.7% 38.2% 4.9% 4.6% 0.9% 11.6% 3.4% 0.7% 3.1% 1.0% 1.6%

Table 1. Breakdown of jobs in the dataset by department. Departments which reserved ≤ 0.5% of overall GPU-time are omitted.

Job Duration and Categorization Figure 2 presents the cumulative distribution function (CDF) of job elapsed time, which we categorized into four groups for easier analysis: • Short: <600 seconds (10 minutes) • Medium: 600–7200 seconds (10 minutes–2 hours) • Long: 7200–28800 seconds (2–8 hours) • Very Long: >28800 seconds (8 hours) Each category represents roughly 25% of the jobs. The CDF reveals discrete jumps, suggesting that jobs are often submitted with specific common time limits, possibly reflecting users’ reliance on predefined configurations. 1.0

Density

0.8 0.6

Duration Category Short Medium Long Very Long

0.4 0.2 0.0 101

102 103 104 Job Elapsed Time (seconds)

105

Fig. 2. CDF of job elapsed time (𝑁 = 7, 399). Colored segments correspond to our different duration categories.

Andrew Sheinberg, Samyak Gupta, and Kai Li

GPU Resource Requests Figure 3a illustrates the distribution of jobs based on the number of GPUs and nodes requested at submission. A significant majority (95%) of jobs request a single GPU, with only a small fraction using multiple GPUs. However, multi-GPU and multi-node jobs take a disproportionately large amount of the GPU time on the cluster. These findings indicate that most workloads have limited parallelization in our dataset, potentially exacerbating the under-utilization of I/O and network bandwidth. This trend could be attributed to the complexity of parallelizing workloads across multiple GPUs or the longer queue times associated with multi-GPU jobs. This hypothesis is supported by Figure 3b; jobs which request multi-GPU or multi-node configurations have significantly longer queue times on average, potentially encouraging frustrated users to prefer issuing single-GPU jobs instead.

Multi-Node

Proportion Type Job GPU Time

Multi-Node

Single-Node, Multi-GPU Single-Node, Multi-GPU

Single-GPU Single-GPU

1S 0.0

0.2

(a)

0.4 0.6 Proportion

0.8

1.0

6H1D 1W r ay ee

Queue Time

(b)

Fig. 3. Classifying by Requested Resources. (𝑎) Proportions of all jobs and total allocated GPU-Time and (𝑏) Corresponding time spent in queue. We find multi-node and multi-GPU jobs face longer queue times than single-GPU jobs. Moreover, peak queue times even for single-GPU jobs can exceed several days.

The predominance of single-GPU jobs appears to be a factor contributing to the underutilization of the GPU cluster, including its network and I/O resources. The high timeout rate highlights a limitation in the current job-scheduling paradigm, requiring users to carefully estimate and request appropriate resources for their jobs. 4

Our Findings

Significant Queuing. To better understand the cluster dynamics we categorize jobs into three groups: single-GPU, single-node with multiple GPUs, and multi-node. Figure 3b portrays the distributions of queue-times for each class; it is clear that requesting more resources results in longer queuing. The gap between requesting 1 GPU vs. multiple is significant. The median queue time for Single-GPU jobs was already nearly 2 hours. However, 2 hours does not seem long compared to the 2-day median queue time for multi-GPU requests. Given these dynamics, it makes sense that the vast majority of jobs only request a single-GPU. Under-utilization. If jobs are sitting in a queue for so long, we might expect high overall cluster utilization. Figure 4 displays the aggregate cluster utilization over this 10-day window. Despite the inflow of jobs, cluster-utilization remains low with an overall GPU compute utilization of 29% and GPU-memory utilization of 16%, with CPU resources also sitting largely unused. These aggregate statistics echo prior monitoring studies (see §5). We investigate factors contributing to the widespread under-utilization and examine opportunities for future system designs.

Plenty of Room in the Middle

GPU Memory

System RAM

0.16 0

0.21

CPU Util

0.29

SM Util

0.2 0.4 0.6 0.8 Overall Cluster Utilization

0.17 0

(a) GPU Metrics

0.2 0.4 0.6 0.8 Overall Cluster Utilization

(b) CPU Metrics

Fig. 4. Aggregate cluster utilization. Averaged over the entire cluster throughout the 10-day sample. Overall, we find both GPU and CPU resources to be under-utilized across the cluster.

4.1

Significant Mismatch between Requested vs. Actual Durations

We find there is a widespread mismatch between requested and actual time durations of the monitored jobs. Figure 5 illustrates the ratio of requested time to actual elapsed time for jobs, categorized by job duration, with about 80% of jobs either underestimated or overestimated by a large factor. Only about 20% of requested durations fall within a factor of two of the actual runtime, and a substantial percentage are off by more than an order of magnitude. Job Duration 1

< 10 Min 10 Min to 2 Hr 2 to 8 Hrs

Fraction of Jobs

0.8

> 8 Hours All

0.6

0.4

0.2

0 <1x (=Timeout)

1-2x

2-10x

10-100x

>100x

Requested Time vs. True Running Time

Fig. 5. A significant portion of jobs over-request time. This is particularly troublesome for the longer jobs which present difficulties for scheduling algorithms as they are receiving inaccurate inputs. This can result in poor global decisions and thus excessive queuing.

Table 2 shows the distribution of job completion states. While 60.7% of jobs complete successfully, a notable 34.2% time out, and the remaining 5.1% fail due to user cancellations, errors, or host memory constraints. The high timeout rate suggests that many users underestimate their job runtimes, wasting the GPU time of prematurely terminated jobs, though some timeouts are intentional (e.g., interactive sessions or checkpointed jobs that run until their limit). Over-requesting resources also has adverse effects. From the users’ perspective, overestimating job durations increases queue delays, as the scheduler must allocate larger time slots for these jobs. An important set of jobs are essentially idle during their actual execution time periods. Table 3 depicts the breakdown of jobs with fully-idle GPU-usage by duration. The vast majority, 1,043 jobs,

Andrew Sheinberg, Samyak Gupta, and Kai Li

Completion State Completed Timeout Failed Canceled

Percentage of Jobs 60.70 34.17 3.15 1.99

Table 2. Completion states of jobs (𝑁 = 7, 399) on the cluster. A significant portion (34.17%) of jobs reach their requested time limit. A smaller fraction are canceled by the user, or fail due to errors or host memory constraints.

lasted less than a minute; of these 1,041 report successful completion, likely implying reservations for very small jobs or configuration errors where a GPU was accidentally requested. Duration < 1 Minute [1 Minute, 10 Minutes) [10 Minutes, 2 Hours] [2 Hours, 8 Hours] > 8 Hours

Prop. of All Jobs 14.1% 0.8% 2.2% 0.4% 0.6%

Prop. of Reserved GPU-Time 0.02% 0.001% 0.44% 0.12% 2.15%

Table 3. Fully-Idle GPU Jobs

A non-negligible number of jobs lasting multiple hours with fully-idle GPUs present a ripe opportunity for system designers. Out of all of the reserved GPU-time, 2.82% was spent in fully idle jobs, with 2.15% allocated as part of long-running (> 8 hours), completely idle jobs. Whether it be user-error in accidental GPU requests, or optimistic reservations that ended up not being needed, it is clear that an intelligent cluster would detect and take advantage of idle resources, rather than locking them away. Implications for Scheduling The widespread and drastic mismatch between requested and actual durations leads to poor scheduling decisions, causing low job completion rates, reduced cluster utilization, and increased queuing delays. There are several significant implications: • Motivation for Avoiding User-Specified Time: Our data reveal a severe and widespread mismatch between user-specified and actual durations, with one-third of jobs hitting their requested time limit. This discrepancy results in substantial waste of GPU resources and valuable human effort. • Motivation for Automatic Runtime Predictions: Since the effectiveness of schedulers depends on the accuracy of their input data, using historical data or learning-based approaches could help schedulers make better decisions. • Motivation for Multitasking: Multitasking approaches can significantly enhance GPU utilization by running multiple tasks concurrently, effectively leveraging underutilized resources. These implications motivate systems designers to explore new methods to manage GPU clusters.

Plenty of Room in the Middle

4.2

Great Opportunities for Multitasking

Most cluster management systems require users to reserve entire GPUs for fixed durations, effectively “locking away” resources that often remain underutilized. While some jobs fully saturate their GPUs, our data show that this is typically only for brief periods, if at all. Many reservations result in significant underutilization, raising the question of whether allocating entire GPUs on an all-or-nothing basis is the most effective approach, especially given the rising cost and demand for modern GPUs. Low SM and Memory Utilizations To study GPU utilization, we analyze the average Streaming Multiprocessor (SM) utilization and peak memory usage for single-GPU jobs, categorized by job duration. The results, illustrated in Figure 6, show significant underutilization: • 50% of jobs sustain an average SM utilization of under 5%, and • 75% of jobs have a peak memory usage of less than 20% and an average SM usage of under 20%.

0.9

0.8

0.7

Density of Jobs

These observations suggest that most jobs are far from fully utilizing GPU capabilities, a trend likely to worsen as per-GPU performance grows faster than typical workload demands.

0.6 0.5 0.4

< 10 Mins 10 Min to 2 Hr 2 to 8 Hrs > 8 Hrs All

0.3 0.2 0.1 0

0.2

0.4

0.6

Average SM Utilization

(a)

0.8

0.6 0.5 0.4 0.3 0.2 0.1

0.2

0.4

0.6

0.8

Peak GPU Memory Utilization

(b)

Fig. 6. CDFs of Single-GPU Job Utilization Metrics, stratified by duration. (a) Average SM Utilization (b) Peak Memory Usage

Efficient memory sharing is often a critical constraint for multitasking, as one task running out of memory can disrupt other tasks. Figure 6b shows that only 5% of jobs ever reach 75% of GPU memory capacity, and among long-running jobs (over 8 hours), just 10% exceed 90%. These results suggest that while some jobs may have large working sets, most leave ample memory headroom, making multitasking feasible. Figure 7 provides additional insights into memory stability. Memory-volatility data (Figure 7a) reveals how frequently memory usage increases during a job, while Figure 7b shows the typical time gap between memory growth events. Many jobs exhibit relatively stable memory usage for extended periods, reducing the risk of memory churn in multitasking scenarios.

Andrew Sheinberg, Samyak Gupta, and Kai Li 1

0.9

0.8

0.7

Density of Jobs

0.6 0.5 0.4 0.3

0.2

0.1

10−6

10−5 10−4 10−3 10−2 Frequency of Memory “Allocs”

(a)

10−1

10−1 100 101 102 103 104 Median Time Between Memory “Allocs”

(b)

Fig. 7. Memory Stability: (a) Fraction of recorded samples where memory usage increased across all jobs. (b) Median time between samples where memory usage increased, across all jobs. Line colors correspond to the duration categories in Figure 6.

Since single-GPU jobs are not gang-scheduled, they are well-suited for multitasking. We have performed a separate analysis of their SM and memory utilization, as illustrated in Figure 6a. The figure highlights the following: • Only 10% of single-GPU jobs achieve an average SM utilization exceeding 50%. • Among long-running jobs (over 8 hours), 33% surpass this threshold, indicating slightly higher but still limited utilization. • Over 70% of single-GPU jobs utilize less than 20% of memory capacity. These findings align closely with the overall utilization trends, as most jobs in the workload are single-GPU jobs. Fine Temporal Granularity The temporal granularity of GPU utilization sampling plays a crucial role in identifying underutilized intervals. Figure 8 compares sampling rates of 10 seconds, 1 second, and 100 milliseconds for both SM usage (Figure 8a) and NVLink throughput (Figure 8b). Coarser sampling masks significant fluctuations and bursty behavior, potentially misleading schedulers into overestimating GPU load. In contrast, 100 ms sampling, as shown in the zoomed-in trace in Figure 9, captures the more accurate peaks and troughs in SM utilization. Similarly, NVLink traffic reveals highly bursty patterns at fine-grained timescales. These data demonstrate the importance of granular telemetry when considering optimal resource allocation. A scheduler can take advantage of fine-grained utilization data to dynamically enqueue additional tasks onto a GPU without waiting for the current job to complete.

Plenty of Room in the Middle

11 400 GPU SM Utilization

10sec Granularity

100

60 40 20 0 t+50

t+75

t+100

t+125

t+150

t+175

100

t+200

80 60 40 20 0

t+25

t+50

t+75

t+100

t+125

t+150

t+175

t+200

t+25

t+50

t+75

t+100

t+125

t+150

t+175

t+200

t+25

t+50

t+75

t+100 Job Time (s)

t+125

t+150

t+175

t+200

2500 1sec Granularity

t+25

GPU NVLink (MB/sec)

1sec Granularity

GPU SM Utilization (%)

NVLink TX NVLink RX

200

0 t

100

2000 1500 1000 500 0

t+25

t+50

t+75

t+100

t+125

t+150

t+175

t+200 25000 100ms Granularity

100 100ms Granularity

300

80 60 40 20 0

20000 15000 10000 5000 0

t+25

t+50

t+75

t+100 Job Time (s)

t+125

t+150

t+175

t+200

(a) Comparison of querying GPU SM utilization at different sample rates. We see significant differences in observed program behavior as a result of the sampling rate; details such as the high frequency fluctuations and deep troughs in SM utilization are lost when sampling at lower rates.

(b) Comparison of querying NVLink throughput at different sample rates. Note the differing y-axis. As granularity increases, we gain more insight into the burstiness of NVLink traffic.

Fig. 8. Comparisons of querying GPU SM utilization and NVLink throughput at different sample rates.

100ms Granularity

100

GPU SM Utilization

80 60 40 20 0 t+40

t+41

t+42 t+43 Job Time (s)

t+44

t+45

Fig. 9. Zoomed-in view of Figure 8a

Implications for Multitasking Our data expose the severe underutilization of GPU SMs and memory, revealing great opportunities for multitasking on GPUs. There are several implications: • Motivation for multitasking: Dynamic multitasking could unlock significant available resources, particularly since over 70% of jobs consume less than 20% of SMs and memory. • Opportunity to Pack Single-GPU Jobs: Single-GPU jobs are good candidates for multiplexing as they are not gang-scheduled and do not incur inter-GPU traffic. These jobs account for a substantial fraction of aggregate cluster usage. • Motivation for Using Fine-Grained Time Slices: Fine-grained preemptive scheduling time slices such as 100–200 ms, on the order of the round-robin time slices OS schedulers use for CPUs, give more opportunities to improve GPU utilization. Nvidia’s Multi-Instance GPU (MIG) [Nvidia 2020a], Multi-Process Service (MPS) [Nvidia 2020b], and Unified Virtual Memory (UVM) [Nvidia 2017] are advancing toward greater maturity. These

Andrew Sheinberg, Samyak Gupta, and Kai Li

mechanisms offer promising opportunities to explore multitasking and reclaim underutilized GPU resources. 4.3

Highly Periodic, and Predictable, Communication Patterns

While most of the jobs on this cluster are isolated single-GPU jobs, Figure 3a indicates that 26% of allocated GPU time was for parallel workloads, and we presume this quantity to be significantly higher for AI and HPC dedicated clusters. The nature of parallel workloads and their accompanying synchronization poses a significant challenge to multitasking. Unintended interference can have compounding straggler effects [Wu et al. 2024]. To determine the possibility of interleaving tasks alongside parallel workloads we investigated communication patterns of Multi-GPU workloads at a link utilization granularity. Given the large number of jobs, we present three representative examples. Single-Node Patterns Neuroscience Job. Figures 10a and 10b illustrate NVLink usage in a parallel neuroscience workload using 4 GPUs within a single node. This job demonstrates a periodic communication phase of approximately 17 minutes, with additional bursts occurring roughly once per minute. The left panel (Figure 10a) shows the consistent repetition of large-scale patterns throughout the job’s runtime, while the right panel (Figure 10b) zooms in on the finer-grained bursts. This structured periodicity indicates that higher-level software systems can estimate NVLink traffic given a small initial sample of metrics.

NVLink Throughput (GB/Sec)

100

500

1000

1500

2000 2500 3000 Time (Sec)

3500

4000

(a) ≈ 17-Minute Phase Periodicity

4500

5000

50 60 Time (Sec)

100

(b) Localized, Per-Minute, Bursts

Fig. 10. Single-Node Neuroscience Job

Psychology Job. Figures 11a and 11b depict a psychology workload, also using 4 GPUs within a single node. This job exhibits mixed burst patterns, with bursts occurring at intervals of approximately 10 seconds and 60 seconds, respectively. While these intervals differ from the neuroscience example, the repetitive nature of NVLink traffic remains consistent, reflecting the structured communication demands of the application.

Plenty of Room in the Middle

13 100

NVLink Throughput (GB/Sec)

100

60 50

0 20k

21k

22k Time (Sec)

(a) “Bursty” Traffic at Two Different Rates

23k

0 21800

21805

21810

21815 Time (sec)

21820

21825

21830

(b) Focusing in on denser rectangular regions in the center of 11a still reveals sparsity and periodicity — once per 10 seconds vs. once per 60 seconds (surrounding spikes in 11a)

Fig. 11. Single-Node Psychology Job

Comprehensive Picture Periodicity among all resource dimensions can be seen in Figure 12, captured from a 4-Node, 16-GPU EE job, likely doing AI training; this pattern repeats for 12 hours with almost no noise. We can infer communication behavior through a combination of traffic metrics. The network metrics come from the node’s singular InfiniBand NIC, while the interconnect metrics are for a specific GPU within the node. The PCIe TX/RX metric traffic includes bytes sent to/received from the network card during RDMA in addition to bytes sent/received from system memory. Note that interconnect graphs are in network units (Gb/s). We see that this job is clearly bottlenecked by network bandwidth (purple/orange — the TX/RX lines are nearly identical), which reaches full saturation of ≈ 100 Gb/s for 2-second windows. In this scenario, both NVLink and PCIe are constrained to the network limit. We can deduce the selected GPU in Figure 12 is continually transmitting over the network as the PCIe TX readings (in pink, second plot) align with the node’s network TX readings. It is not transmitting to other intra-node GPUs (0 NVLink TX traffic from this selected GPU). The PCIe RX throughput implies inbound network traffic is split among multiple GPUs in this system. Other GPUs initially receive traffic (flat blue PCIe RX line) and then this GPU receives a partial share. These readings agree with the intra-node NVLink RX traffic this GPU receives (top plot in green) as it first is reading at the network limit and then drops off when it is receiving some data directly from the NIC via RDMA. We highlight that GPU SM Utilization (in red) fluctuates in lock-step with the overall communication pattern. Fully-idle GPUs result from what we presume to be global synchronization. In this example for every 7-second period, 1 second is spent idle. Accounting for 16 GPUs and a 12-hour duration, this implies around 21 hours of available GPU-computation that can be packed into these troughs. The motif of small pockets of SM idleness, occurring at frequent and regular intervals, is the common case for many network-bound distributed GPU applications. Despite its highly parallel nature, this type of job is a valid candidate for multitasking. GPU memory usage is shown staying flat at 50% (it does not change for hours) in lavender (second from bottom), leaving ample space for other jobs’ data.

CPU

GPU Memory

GPU Compute

Network (Gb/s)

PCIe (Gb/s)

NVLink (Gb/s)

Andrew Sheinberg, Samyak Gupta, and Kai Li 100 75 50 25 0

t 100 75 50 25 0 100 t 75 50 25 0 100 t 75 50 25 0 100 t 75 50 25 0 100 t 75 50 25 0 t

NVLink RX NVLink TX

30 PCIe RX PCIe TX

30 Net RX Net TX

30 SM Util % Tensor Active %

30 GPU Mem Util % GPU Mem Bus %

30 CPU Util % Host Mem Util %

Time (Sec)

Fig. 12. Slice from 4-Node, 16-GPU workload: Periodicity across all resources & frequent pockets of SM idleness (4th row, red)

Implications for Distributed Jobs Our analysis underscores the importance of high GPU-interconnect and network bandwidth. SingleNode jobs can exhibit full NVLink saturation during blocking communication phases. Figure 8 highlights how insufficient interconnect performance can degrade throughput during communicationintensive phases. Distributed jobs can be bottlenecked by network bandwidth and fail to saturate high-BW NVLinks and ultimately cause SM idle periods. The analysis indicates: • Opportunity for Accurate Traffic Predictions: Repetitive communication patterns present an opportunity for accurate traffic prediction. Scheduling tasks in an “out-of-phase” manner could help minimize contention and maintain consistent throughput. • Opportunity to Address Synchronization-Induced Idle Periods: Transparent multitasking alongside long-running distributed jobs can significantly improve cluster utilization, especially in bandwidth-limited environments where communication delays often cause extended GPU idleness. The general challenge to take advantage of these opportunities is to minimize interference during the bursty communication periods.

Plenty of Room in the Middle

Prior Work Philly [Jeon et al. 2019] AntMan [Xiao et al. 2020] Helios [Hu et al. 2021] MIT SuperCloud [Samsi et al. 2021] Weng et al. [2022] Moneo [Jiang et al. 2022] Ours

Spatial Granularity

Temporal Granularity

Studied Workloads

Coarse Coarse Coarse Coarse4

100 seconds ≈ 1 second1 100 seconds 100 ms

Deep Learning Deep Learning Deep Learning Mixed (incl. DL)

Coarse Fine Fine

15 seconds Variable2 100 ms

Deep Learning Individual DL Training Jobs3 Mixed (12 disciplines)

1 Inferred from time-series traces in Xiao et al. [2020] and confirmed with authors. 2 Supports DCGM monitoring, but dynamically reduces sampling rate. 3 Does not evaluate cluster-wide utilization; studies 3 specific workloads. 4 Samples nvidia-smi utilization, which does not capture fractional SM usage (§2).

Table 4. Comparison to several prior works studying cluster utilization.

Related Work

GPU Monitoring APIs. Despite growing awareness of underutilization, few production systems have deployed sub-second sampling to characterize bursty or partial GPU usage. We summarize key differences between our work and prior monitoring work in Table 4. The DCGM API supports high-resolution telemetry (e.g., SM Active, SM Occupancy, NVLink, PCIe) at 100–200 ms intervals, but it is rarely used at scale. The MIT SuperCloud dataset [Samsi et al. 2021] also records GPU time series every 100 ms, but through nvidia-smi, whose utilization metric can overstate SM usage by orders of magnitude (§2). To our knowledge, our work is among the first to collect DCGM’s SM-level metrics at 100 ms across an entire multi-disciplinary research cluster. This approach reveals patterns—such as frequent GPU idleness, partial SM occupancy, and bursty interconnect usage—that remain hidden when sampling at intervals of one second or longer. GPU Cluster Monitoring. Monitoring GPU-based clusters has been a significant topic of research, with numerous tools and frameworks designed to track resource usage. One of the earliest and most broadly adopted systems is Ganglia [Massie et al. 2004], which provides cluster-wide resource utilization metrics via a hierarchical design. Although Ganglia can be extended to expose GPU statistics (e.g., through an NVIDIA GPU monitoring module), it typically leverages the NVIDIA Management Library (NVML) [Nvidia 2007], offering only coarse-grained GPU usage data. Consequently, Ganglia-based monitoring lacks sub-second resolution and may miss rapid fluctuations in GPU compute, memory, or interconnect utilization. We instead designed our own monitoring solution to have more precise control over monitoring and sampling parameters. We are not the first to champion fine-grained, cluster-wide GPU monitoring; Moneo [Jiang et al. 2022] is a tool developed by Microsoft which also adopts fine-grained monitoring of GPU metrics. While Jiang et al. propose capturing fine-grained metrics, they only study a limited selection of hand-picked deep learning training jobs. However, despite the availability of such tools, they remain unpopular for use on HPC research clusters; a recent study by Weakley et al. [2025] surveys GPU monitoring practices in HPC clusters, finding that the newer NVIDIA Data Center GPU Manager (DCGM) API [Nvidia 2020c]—which supports sub-second sampling—remains under-adopted. They attribute this to limited compatibility with older GPUs and the overhead of integrating DCGM with existing monitoring systems. We hope that our work serves to raise broader awareness.

Andrew Sheinberg, Samyak Gupta, and Kai Li

In practice, the predominant monitoring infrastructure consists of deploying a container-based solution composed of Nvidia’s dcgm_exporter (for GPU metrics) and the Prometheus monitoring framework [Rabenstein and Volz 2015] (for CPU-side metrics and a time-series DB). This setup is targeted for Kubernetes-based clusters and is non-trivial to set up and control. Our monitoring tool can be deployed on any bare-metal Nvidia Data-Center GPU server (without dependencies) and is trivial to start/stop from the command line (start/kill the daemon process). Prior Characterization Studies. Several large-scale studies have examined GPU cluster utilization to identify inefficiencies and guide improvements in resource management; we provide a high-level summary of the differences with our work in Table 4. Philly [Jeon et al. 2019] from Microsoft was among the first to report considerable underutilization in a large-scale multi-tenant GPU cluster dedicated mostly to deep learning. Philly relied on NVML-based sampling at 100-second intervals, limiting its ability to capture sub-second bursts or fractional GPU usage. While underutilization in Philly was largely attributed to network constraints and gang scheduling, our results demonstrate that, even on nodes running a single job, significant idle periods occur that only become apparent when sampling at finer granularity. Helios [Hu et al. 2021] took a closer look at multiple production clusters running various GPU workloads. Like Philly, Helios employed NVML for coarse-grained (100-second) sampling and did not measure fractional SM utilization. Although Helios confirmed the prevalence of low GPU utilization and long queue times, it left open questions about the root causes of underutilization within each job’s runtime. More recently, Weng et al. [2022] characterize utilization in a production cluster which supports time-multiplexed sharing of GPUs between jobs, demonstrating low contention for SM resources in their analysis. While a strong preliminary step towards improving utilization, our results show that significant fractions of GPU SM resources remain unutilized by jobs; as time-slicing alone cannot allow for the sharing of these resources, we feel fine-grained monitoring of utilization reveals greater opportunities for multiplexing. Our sub-second analysis highlights internal idle phases and scheduling opportunities that standard job-level metrics cannot readily detect. GPU Multitasking and Scheduling Approaches. Many researchers have explored the idea of workload-independent GPU sharing [Adriaens et al. 2012; Chen et al. 2017; Liang et al. 2015; Mahajan et al. 2020; Park et al. 2015; Sedighi et al. 2024; Tanasic et al. 2014; Wang et al. 2016]. Much of recent work regarding GPU-multitasking has concentrated on machine-learning focused workloads. Gandiva [Xiao et al. 2018] considers time-slicing shared GPU resources between multiple jobs. AntMan [Xiao et al. 2020] introduces a software layer to multiplex GPU compute and memory resources, improving overall cluster usage. Like prior work, AntMan’s monitoring relies on NVML at roughly 1 s intervals, thus it still may miss quickly shifting utilization patterns. Nevertheless, its reported improvements underline the potential of GPU multi-tenancy—a conclusion our results reinforce. By measuring SM usage at sub-second granularity, we demonstrate that jobs often leave enough unused SM capacity for other processes to co-locate effectively. More recently, Orion [Strati et al. 2024] proposes a method for fine-grained sharing of GPU workloads on a single node, showing that performant and interference-aware multiplexing of GPU resources is possible. However, Orion relies on GPU utilization profiles collected offline, limiting its applicability to general user applications. We believe there is room for systems based on dynamic, real-time profiling of workloads to allow for flexible, application-agnostic sharing of GPUs. Based on the Helios dataset, Hu et al. [2021] proposed a “Quasi-Shortest-Service-First (QSSF)” scheduler, which uses historical job records to predict runtime and reduce queuing for large multi-GPU jobs. However, QSSF assumes relative stability in job behavior and easy access to training data, assumptions that may not hold in research environments where development

Plenty of Room in the Middle

and debugging are common. Our findings suggest that more adaptive scheduling strategies, fed by fine-grained performance metrics, could mitigate these shortcomings. 6

Limitations

Sampling Rate. We used the highest sampling frequency, 10 Hz, offered by DCGM [Nvidia 2020c]; however, we would have preferred to collect data at even higher rates if the API had allowed it, given our low-overhead monitoring tool supports rates 10–100× faster. While our temporal resolution exceeds that of prior monitoring studies, interconnect and network measurements are still aggregated over 100 ms windows, which may smooth out short-lived communication spikes and obscure burstiness. Moreover, individual GPU kernels often execute on a timescale of microseconds to single-digit milliseconds. Profiling tools such as Nvidia Nsight Systems (nsys) can capture this data (as a background process), but are not applicable for cluster-scale monitoring due to job degradation. Duration of Study. Although we ran the monitoring tool for several months, we only retrieved precise mappings of Job ID to GPU ID(s) for the final 10 days. Rather than making potentially inaccurate inferences about which jobs ran on which devices, we limited our analysis to the clean, fully instrumented dataset. Job Types and Diversity. Because most jobs in the cluster requested only a single GPU, we collected fewer samples of multi-GPU workloads (343 in total) and of these only 7 multi-node workloads. Our insight into the frequencies and behaviors of domain-specific workloads (e.g. ML or HPC workloads) is limited; our analysis aggregates data from all departments. Cluster Hardware and Heterogeneity. The cluster included two GPU types (A100 PCIe 40 GB and A100 SXM4 80 GB). We combined them into a single dataset for simplicity, as we report utilization in percentages. However, these GPUs differ in their interconnect bandwidth (PCIe vs. NVLink). Because most jobs were single-node, the impact of these differences on our overall findings was small. Nonetheless, future work should investigate how heterogeneous hardware and network capabilities affect cluster utilization and communication patterns. 7

Conclusion

This paper presents a fine-grained analysis of a university research GPU cluster through a lightweight monitoring tool operating at 100 ms granularity. Our study reveals three key findings: (1) severe underutilization, with over 70% of jobs using less than 20% of GPU memory and 50% of jobs averaging below 5% SM utilization; (2) pervasive mismatch between requested and actual job durations, affecting 80% of jobs and fundamentally limiting scheduler effectiveness; and (3) highly periodic communication patterns in multi-GPU jobs, suggesting opportunities for predictive resource allocation and compatible multiplexing. These results, coupled with emerging technologies like MIG, MPS, and UVM, indicate significant potential for improving cluster efficiency through dynamic resource sharing and intelligent scheduling. By leveraging fine-grained monitoring data, system designers can develop multitasking solutions that maximize GPU utilization while maintaining performance isolation, potentially transforming how research GPU clusters are managed and utilized. Acknowledgments We thank Josko Plazonic, Jon Halverson, Curtis Hillegas, and the rest of the Princeton Research Computing team for helping deploy the monitoring tool and for insightful feedback on the cluster

Andrew Sheinberg, Samyak Gupta, and Kai Li

environment. We also thank Sanjeev Arora, Danqi Chen, Karthik Narasimhan, and other members of the Princeton Language & Intelligence group for cooperation in our monitoring study. References Jacob T. Adriaens, Katherine Compton, Nam Sung Kim, and Michael J. Schulte. 2012. The case for GPGPU spatial multitasking. In Proceedings of the 2012 IEEE 18th International Symposium on High-Performance Computer Architecture (HPCA ’12). IEEE Computer Society, USA, 1–12. doi:10.1109/HPCA.2012.6168946 Guoyang Chen, Yue Zhao, Xipeng Shen, and Huiyang Zhou. 2017. EffiSha: A Software Framework for Enabling Efficient Preemptive Scheduling of GPU. In Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’17). Association for Computing Machinery, New York, NY, USA, 3–16. doi:10.1145/3018743.3018748 Richard P. Feynman. 1960. There’s Plenty of Room at the Bottom. Engineering and Science 23, 5 (1960), 22–36. CJ Haddad. 2024. Nvidia CEO Jensen Huang says demand for next-generation Blackwell AI chip is ‘insane’. https: //www.cnbc.com/2024/10/03/nvidia-ceo-demand-for-blackwell-ai-chip-is-insane.html Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, and Tianwei Zhang. 2021. Characterization and prediction of deep learning workloads in large-scale GPU datacenters. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’21). Association for Computing Machinery, New York, NY, USA, 1–15. doi:10.1145/3458817.3476223 Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of large-scale multi-tenant GPU clusters for DNN training workloads. In Proceedings of the 2019 USENIX Conference on USENIX Annual Technical Conference (USENIX ATC ’19). USENIX Association, USA, 947–960. Yuting Jiang, Yifan Xiong, Lei Qu, Cheng Luo, Chen Tian, Peng Cheng, and Yongqiang Xiong. 2022. Moneo: Monitoring Fine-grained Metrics Nonintrusively in AI Infrastructure. ACM SIGOPS Operating Systems Review 56, 1 (June 2022), 18–25. doi:10.1145/3544497.3544501 Charles E. Leiserson, Neil C. Thompson, Joel S. Emer, Bradley C. Kuszmaul, Butler W. Lampson, Daniel Sanchez, and Tao B. Schardl. 2020. There’s Plenty of Room at the Top: What Will Drive Computer Performance after Moore’s Law? Science 368, 6495 (2020), eaam9744. doi:10.1126/science.aam9744 Yun Liang, Huynh Phung Huynh, Kyle Rupnow, Rick Siow Mong Goh, and Deming Chen. 2015. Efficient GPU SpatialTemporal Multitasking. IEEE Transactions on Parallel and Distributed Systems 26, 3 (2015), 748–760. doi:10.1109/TPDS. 2014.2313342 Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. 2020. Themis: Fair and Efficient GPU Cluster Scheduling. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, 289–304. https://www.usenix.org/conference/ nsdi20/presentation/mahajan Matthew L. Massie, Brent N. Chun, and David E. Culler. 2004. The Ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30, 7 (July 2004), 817–840. doi:10.1016/j.parco.2004.04.001 Nvidia. 2007. NVIDIA Management Library (NVML). https://developer.nvidia.com/management-library-nvml Nvidia. 2017. Unified Memory for CUDA Beginners. https://developer.nvidia.com/blog/unified-memory-cuda-beginners/ Nvidia. 2020a. Multi-Instance GPU User Guide. https://docs.nvidia.com/datacenter/tesla/pdf/NVIDIA_MIG_User_Guide.pdf Nvidia. 2020b. Multi-Process Service User Guide. https://docs.nvidia.com/deploy/mps/index.html Nvidia. 2020c. NVIDIA DCGM Documentation. https://docs.nvidia.com/datacenter/dcgm/latest/index.html Nvidia. 2024a. CUDA C++ Programming Guide (Version 12.4.1). https://docs.nvidia.com/cuda/archive/12.4.1/cuda-cprogramming-guide/index.html Nvidia. 2024b. CUDA Runtime API: Memory Management (Version 12.4.1). https://docs.nvidia.com/cuda/archive/12.4.1/cudaruntime-api/group__CUDART__MEMORY.html Jason Jong Kyu Park, Yongjun Park, and Scott Mahlke. 2015. Chimera: Collaborative Preemption for Multitasking on a Shared GPU. In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’15). Association for Computing Machinery, New York, NY, USA, 593–606. doi:10.1145/ 2775054.2694346 Björn Rabenstein and Julius Volz. 2015. Prometheus: A Next-Generation Monitoring System (Talk). In SREcon15 Europe. USENIX Association, Dublin. Siddharth Samsi, Matthew L. Weiss, David Bestor, Baolin Li, Michael Jones, Albert Reuther, Daniel Edelman, William Arcand, Chansup Byun, John Holodnack, Matthew Hubbell, Jeremy Kepner, Anna Klein, Joseph McDonald, Adam Michaleas, Peter Michaleas, Lauren Milechin, Julia Mullen, Charles Yee, Benjamin Price, Andrew Prout, Antonio Rosa, Allan Vanterpool, Lindsey McEvoy, Anson Cheng, Devesh Tiwari, and Vijay Gadepally. 2021. The MIT Supercloud Dataset. In 2021 IEEE High Performance Extreme Computing Conference (HPEC). 1–8. doi:10.1109/HPEC49654.2021.9622850

Plenty of Room in the Middle

Frank Schmuck and Roger Haskin. 2002. GPFS, a Shared-Disk File System for Large Computing Clusters. In Proceedings of the FAST 2002 Conference on File and Storage Technologies. Monterey, California. Hoda Sedighi, Daniel Gehberger, Amin Ebrahimzadeh, Fetahi Wuhib, and Roch H. Glitho. 2024. Efficient Dynamic Resource Management for Spatial Multitasking GPUs. IEEE Transactions on Cloud Computing (2024), 1–18. doi:10.1109/TCC.2024. 3511548 Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 1075–1092. doi:10.1145/3627703.3629578 Ivan Tanasic, Isaac Gelado, Javier Cabezas, Alex Ramirez, Nacho Navarro, and Mateo Valero. 2014. Enabling preemptive multiprogramming on GPUs. In Proceeding of the 41st Annual International Symposium on Computer Architecture (ISCA ’14). IEEE Press, 193–204. doi:10.1145/2678373.2665702 Zhenning Wang, Jun Yang, Rami Melhem, Bruce Childers, Youtao Zhang, and Minyi Guo. 2016. Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 358–369. doi:10.1109/HPCA.2016.7446078 Le Mai Weakley, Scott Michael, Laura Huber, Abhinav Thota, Ben Fulton, and Matthew Kusz. 2025. Monitoring and Characterizing GPU Usage. Concurrency and Computation: Practice and Experience 37, 3 (2025), e8341. doi:10.1002/cpe.8341 Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, 945–960. https://www.usenix.org/conference/nsdi22/presentation/weng Tianyuan Wu, Wei Wang, Yinghao Yu, Siran Yang, Wenchao Wu, Qinkai Duan, Guodong Yang, Jiamang Wang, Lin Qu, and Liping Zhang. 2024. FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training. arXiv:2410.12588 [cs.DC] https://arxiv.org/abs/2410.12588 Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018. Gandiva: Introspective Cluster Scheduling for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, 595–610. https://www.usenix.org/conference/osdi18/presentation/xiao Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. 2020. AntMan: dynamic scaling on GPU clusters for deep learning. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI ’20). USENIX Association, USA, 533–548.

Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

Analyze & explore related research in Conceptio

Related documents