arXiv CSopen access

PRISM: Evaluating POSIX Storage Systems for AI Research Workflows

PRISM: Evaluating POSIX Storage Systems for AI Research Workflows Adithya Kumar1 , Aditya Basu1 , Jacob Kahn1 , Parth Malani1 , Leo Huang1 , Kalyan Saladi1

arXiv:2607.21746v1 [cs.DC] 23 Jul 2026

1

FAIR at Meta

The rapid advancement of AI research is driven by massive investments in GPU clusters, yet the critical role of storage systems in enabling efficient research workflows is often overlooked. Unlike traditional HPC workloads, AI research prioritizes researcher productivity and ease of iteration: practitioners rely on POSIX-compliant file systems for seamless prototyping, debugging, and experimentation before scaling to specialized storage backends. The primary selection criterion is therefore not peak throughput alone, but rather performance within a POSIX-compatible, researcher-friendly interface. However, existing benchmarks evaluate storage systems exclusively on peak performance and fail to capture the bursty, heterogeneous I/O patterns characteristic of real-world AI research—where workflows are dynamic,continuously evolving spanning all stages of research. We introduce PRISM, an evaluation framework that reproduces representative AI research workloads—spanning data ingestion, checkpoint I/O, and developer workflows—to assess and qualify POSIX storage systems along both usability and performance dimensions on GPU clusters. Using PRISM we were able to compare Lustre and NFS based POSIX storage systems across multiple research workload dimensions and select the appropriate storage solution for different environments. As a specific case study in our environment we observed that a flash backed NFS solution outperformed the flash backed Lustre solution by up to 3× for the distributed checkpoint load use-case which helped us make an informed cluster design. Date: July 27, 2026 Correspondence: Kalyan Saladi at [email protected]

1

Introduction

AI Research Clusters & Storage: The contemporary race toward super-intelligence is fueled by multi-billion

dollar investments in AI compute clusters with an intense pursuit of acquiring and efficiently operating the latest generation of GPUs and accelerators. Leading AI labs and organizations are competing for compute resources, whether by building in-house infrastructure or leveraging GPU neo-clouds. While GPUs performing costly computations grab all the attention, it is imperative that providers ensure the data fed into these GPUs is efficiently stored and handled. In the race to feature the latest generation GPUs, GPU neo-clouds often are not held accountable to provide competitive storage systems. Consequently, the burden of responsibility to keep these GPUs highly utilized falls squarely on researchers and infrastructure managers. AI research productivity is thus a highly important endeavor which necessitates allowing maximum flexibility on infrastructure with minimal impedance. This calls for a seamless integration of compute and data through a myriad of tools and frameworks, which is crucial as researchers need to rapidly prototype, design, test, and refine their models. Ultimately, storage systems are a critical component, as they impact the speed and efficiency of data processing, training times, and the overall performance of AI models being conceived on these billion-dollar GPU clusters. As we will show later, sub-optimal usage of a storage system on a notable cloud vendor can result in a 3× performance slowdown (4.2) for a common data loading job, and a critical bug in a notable storage vendor’s system caused an 8× increase in checkpoint latency (4.3) that resulted in a dramatic reduction in overall efficiency of our clusters. Role of storage in AI Research workflows: AI research workflows, by their nature, are fundamentally different

from stable production training. They are highly dynamic, fragmented, interactive, multi-modal, and often experimental, i.e., they are rapidly reshaped, frequently repetitive in the short-term, and can be discarded

1

entirely in the long-term. This demands an extreme degree of adaptability from the underlying infrastructure to accommodate ever-changing project needs. The typical lifecycle for an AI researcher progresses from prototyping a hypothesis to pre-processing datasets, running pre-training on a small scale, scaling to large datasets, performing post-training/fine-tuning, and ultimately publishing the model. In this fluid environment, POSIX (Portable Operating System Interface) based storage systems, such as NFS and Lustre, are a massive boon for research productivity, particularly in the earlier, more prototypical stages of the workflow. While pre-training massive frontier models may necessitate bespoke storage solutions for peak performance, the simplicity and effectiveness of a battle-tested POSIX storage system provide a crucial compatibility layer. These systems ensure the interoperability required by a vast ecosystem of offthe-shelf and user-defined software applications. Their critical role is evident across key stages, including: (i) Creating/managing developer environments (e.g., Cloning git repositories, building packages and conda environments, tar/untaring configuration files), (ii) Data generation (e.g., Downloading data from huggingface using curl, synthetic data generation from within the cluster), (iii) Data curation and consumption (e.g., Data loading for pretraining, Validating datasets, Pre-processing and extracting salient information), and (iv) Managing model checkpoints. This intricate dependency means the storage system must not become a bottleneck, as the proliferation of new datasets, formats, and processing techniques relies on its efficiency to maintain high productivity and keep GPUs optimally utilized. The scrappy, unpredictable, and exploratory nature of research means that storage access patterns are often unforeseen, which in our experience can lead to significant time spent debugging what turn out to be dead-end research paths. Evaluating POSIX storage systems using PRISM Ensuring that the storage system can consistently meet the ever-

changing and unpredictable demands of AI research workflows is paramount, particularly in light of the high relevance of modern GPU clusters. Failure to do so risks turning the storage layer into a bottleneck, which posed a significant challenge for us in determining how to effectively evaluate these systems. While numerous storage benchmark tools exist, from general-purpose ones like fioAxboe (2023) and elbenchoBreuner (2023) that measure basic bandwidth and IOPS, to application-specific suites like dbbench and MLPerf storage, we found these insufficient to fully characterize the performance implications for various research usecases and workflows. They fail to capture the full scope and dynamicity of real-world research. Specifically, we observed that alongside the substantial pre-training workloads (e.g., data pre-processing and synthetic data generation), our research cluster environments frequently generated highly unpredictable and competing workflows that are not represented by traditional benchmarks.

This gap motivated us to develop a canonical set of realistic applications and release it as PRISM 1 that accurately model the AI research workflow use-cases we had observed in our clusters over an 18-month period. The result is a robust evaluation framework that (i) measures real-application performance over low-level primitives for important storage use-cases, and (ii) make the framework extendable to a multitude of Python-based applications that faithfully replicate the complex access patterns used by our researchers. (iii) Test and prove that it works on small and medium sized GPU clusters (up to 1K GPUs) This framework has helped us to: (i) validate the performance and suitability of new storage vendors on neoclouds with large-scale GPU deployments, (ii) objectively explore, compare, and evaluate competing storage offerings on the public cloud, and (iii) successfully evaluate and perform multiple vendor and kernel upgrades across our clusters.

2

Storage Systems for AI Research Workflows

We summarize and compare different flavors of storage offerings available on the cloud in Table 1 along the dimensions of compatibility, deployability, configurability, and scalability. Cloud-native object storage offerings such as Amazon’s S3 and Google’s GCS are extremely easy to deploy and offer excellent scalability with simple HTTP-based APIs and are widely used for storing and retrieving datasets and other artifacts. They typically have higher access latency making them unsuitable for interactive use-cases, provide limited consistency guarantees, and their APIs necessitate application changes or require adopting intermediary shims (e.g.: s3fs, specialized libraries) for general purpose consumption. Similarly, distributed systems like HDFS and Ceph offer scalable, fault-tolerant storage, are deeply integrated with big data frameworks trading off 1 PRISM: POSIX Research Infrastructure Storage Measurement System

2

Storage offering

POSIX filesystems (NFS, Lustre) Object store (S3, GCS) Distributed systems (HDFS,Ceph)

Compatibility

Deployability

Configurability

Scalability

High

Easy

Moderate

Medium

Medium

Easy

Low

High

Low

Hard

High

High

Table 1 Tradeoffs between different cloud storage offerings for AI Research workflows

general file semantics for performance but often are operationally complex to deploy and configure especially on the public cloud. In contrast, POSIX-compliant file systems such as NFS (e.g., AWS EFS, FSx ONTAP, FSx OpenZFS) and Lustre (e.g., FSx Lustre, Azure Lustre) present a traditional file and directory abstraction making them highly ubiquitous and compatible with tools, users, and more recently with agents alike. They allow users and applications to treat remote storage just like local directories with well defined consistency semantics, drastically simplifying integration with existing software, scripts, and workflows. While portions of AI research workflows may exemplify under specific types of storage systems, we observed that POSIX-based ones have proven to be a game-changer for research productivity by drastically reducing the friction for our researchers. The seamless compatibility with the extensive ecosystem of open-source tools (e.g., PyTorch, NumPy, conda etc.) that inherently assume a filesystem abstraction lowers the barrier for experimentation and simplifies various operational activities such as scripting and debugging failures making POSIX storage systems the preferred ‘default’ for interactive work and training runs. While POSIX-like systems have an advantage with a simple deployment model on the cloud (e.g., FSx), the challenge remains in selecting, configuring, and tuning the right POSIX option (e.g., highly scalable Lustre vs. simpler NFS) to meet the performance demands of AI research workloads without introducing excessive operational complexity.

2.1

fsspec

fsspec is a Python library that provides a unified interface for working with different filesystem backends, and it has become increasingly valuable in AI research workflows through its deep integration with PyTorch. When training deep learning models, researchers often need to access data stored across various locations—local disks, cloud storage like S3 or Google Cloud Storage, HDFS clusters, or even HTTP endpoints. Rather than writing custom code for each storage backend, fsspec allows PyTorch data loaders to seamlessly read from any supported filesystem using a consistent URL-based syntax. PyTorch’s data loading utilities can work directly with fsspec-compatible file paths, meaning a researcher can switch from local development to cloud-based training simply by changing a path prefix from ‘file://’ to ‘s3://’ or ‘gs://’. The integration becomes even more critical in distributed training scenarios and when checkpointing models, where PyTorch’s ‘torch.load’ and ‘torch.save’ functions support fsspec backends, enabling models to checkpoint directly to cloud storage without requiring intermediate local copies. Many popular PyTorch-based libraries, including Hugging Face’s datasets library and PyTorch Lightning, leverage fsspec under the hood, making it an essential component of modern AI research infrastructure. Despite fsspec’s elegant abstractions for programmatic data access, researchers often find it essential to work directly with POSIX-compliant filesystems like NFS in practice because AI research workflows extend far beyond what happens inside Python code. While fsspec excels at making datasets accessible to PyTorch during training, it doesn’t help with the dozens of filesystem operations that happen outside the training loop. Package managers like ‘conda’ and ‘uv’ need to install dependencies, create environments, and cache packages—all operations that expect standard POSIX semantics with proper file locking, permissions, and hard links. Similarly, researchers frequently write bash scripts that orchestrate experiments, preprocess data with command-line tools like ‘awk’, ‘sed’, or ‘jq’, and chain together utilities that have no knowledge of fsspec’s URL schemes. These scripts assume they can use standard Unix tools like ‘ls’, ‘find’, ‘grep’, and

3

Dimension High level goals Workflow Performance target Resource demand Scaling Data lifecycle Data security standards

Production AI

Research AI

Repeatability, Efficiency, Strict SLOs Well-defined, Repeatable Predictable access pattern High throughput Stable, Predictable volume Right-sized, High Utilization Governed, Strict Schemas, Immutable Datasets, Lineage Tracking Sensitive 1P data High levels of access control

Flexibility, Interactivity, Low Friction Rapid, ad-hoc experiments Access pattern mix (Sequential/Random) Low latency Highly Bursty, Irregular Easy to Grow and Reconfigure Rebalance with minimal Coordination Store intermediate artifacts in scratch Frequent updates Open source 3P data Low barrier to access

Table 2 High level differences between how production workloads and research workloads leverage the storage system

‘rsync’ to navigate directory structures, search through files, and synchronize results, none of which work with cloud storage paths without additional wrapper layers. The logging and debugging aspects of research make POSIX filesystems a natural fit. During development, researchers constantly inspect training logs with ‘tail -f’, browse experiment outputs with ‘less’, ‘diff’ configuration files, and use text editors that expect local file access. When something goes wrong, researchers can navigate directories, grep through logs, visualize results with notebook interfaces that expect local paths, or run ad-hoc data munging scripts. Tools like ‘tensorboard’, ‘wandb’ sync utilities, and various profilers often expect to write to local filesystems with proper atomic write operations and file watching capabilities. The reality is that while deep learning frameworks have modernized to support cloud-native storage through fsspec, the broader ecosystem of development tools, shell utilities, and exploratory workflows still assumes POSIX semantics. This makes NFS-mounted shared storage practical for day-to-day research activities—even when the training data itself lives in cloud storage and is accessed via fsspec, researchers typically mount shared NFS volumes for everything else: code repositories, virtual environments, experiment tracking, intermediate outputs, and the collaborative workspace where team members can easily browse each other’s results.

2.2

Storage Infrastructure for Production vs. Research AI Workflows

The storage infrastructure requirements for research AI training as opposed to production AI infrastructure diverge along several key dimensions in practice, even when they use the same underlying storage technology. At a high level, production training pipelines are engineered for repeatability, efficiency, and strict servicelevel objectives, whereas research workflows prioritize flexibility, interactivity, and low friction for exploratory work. We summarize this distinction along various dimensions in Table 2 and talk about salient points below. Production AI training is engineered for strict service-level objectives (SLOs), efficiency, and repeatability, typically operating within well-defined, automated pipelines on predictable schedules. This environment relies on stable, predictable data volumes with governed data lifecycles, strict schemas, and strong lineage tracking, favoring specialized, high-performance object stores that offer strong guarantees and high utilization. The singular focus on consistent performance and predictable latency allows for “right-sizing” of resources and development of bespoke APIs which are quite often not fully POSIX compliant but tradeoff other things in favor of performance. In stark contrast, research AI training prioritizes flexibility, low friction, and rapid iteration to support ad hoc experimentation and heterogeneous workloads. These workloads are characterized by highly bursty resource demand, involving frequent updates to intermediate artifacts, scratch spaces, and a mix of structured and unstructured data, often resulting in a large number of small, short-lived files. The necessity to accommodate an ever-evolving toolchain and facilitate easy collaboration among researchers compels a preference for generic, POSIX-based storage systems (e.g., NFS, Lustre) that act as a common denominator for open-source tools. Storage systems in this context must be easy to grow, reconfigure, and rebalance with minimal coordination 4

0.00 0

20

40

60

80

0.6

P50

0.4

75%

50%

25%

0%

Time (days)

0.2 0.0

P99 P95 P90

READ_IOPS

(c)

Cumulative Probability

50.00

fs1 fs2 fs3 fs4

0.8

WRITE_IOPS

100%

Fraction of IOPS (%)

100.00

MD_IOPS

P99 P95 P90

0 3 6 10 13 16 19 22 25 29 32 35 38 41 45 48 51 54 57 61 64 67 70 73 76 80 83 86 89

Cumulative Probability

POSIX storage capacity (%)

150.00

CDF: iosize

1.0

CDF: read_write_ratio

1.0

0.8 0.6 P50

0.4

home project

0.2 0.0

0

50000 100000 150000 200000 250000 300000

iosize

Breakdown of read, read_write_ratio write, metadata IOPS - (d) Average size per IO oper(a) Capacity growth over 90 (b) Read vs. Write through- NAS. Significant amount of ation comparing ‘home’ and Metadata IOPS. days - Lustre put usage - NAS ‘project’ filesystems - NAS Time (Days)

0

20

40

60

80

100

Figure 1 Characterizing POSIX storage under research environments

to prevent friction in the exploratory process. The fundamental divergence in goals, access patterns, and data characteristics between production and research training necessitates a significant rethinking of how storage performance analysis is conducted. Traditional benchmarks Axboe (2023); Breuner (2023); Tarasov et al. (2016), optimized for the predictable, sequential, and large-file access patterns of production workloads, fail to accurately characterize the performance and behavioral needs of research environments, which are dominated by bursty demand, random I/O, and many small file operations. Consequently, to ensure that storage infrastructure effectively supports the unique demands of scientific discovery and rapid iteration, a novel set of tools and benchmarks is required to precisely characterize the performance of storage systems under the specific, highly variable access patterns inherent to research AI workloads. This paper focuses specifically on storage infrastructure for research AI workflows, where POSIX-based systems play a central role. Our goal is to characterize how these research workloads exercise underlying storage systems and to illustrate how storage benchmarking should be performed for such environments, so that stability, reliability, and performance can be ensured in practice.

2.3

Characterizing POSIX Storage on our research environments

To highlight the dependency of POSIX-based storage in research clusters, we characterize the storage capacity footprint and usage patterns across clusters. This system constitutes the primary working storage used by researchers for day-to-day experimentation, code development, and training, representing a substantial fraction of the I/O path for production-like workloads running under a research context. Each of these clusters have at least 8K Nvidia Hopper or later class of GPUs which primarily use POSIX-based storage for user home directories (code, conda environments etc.) and shared project directories (configuration, checkpoints, datasets, logs etc.). We consider the following types of POSIX-based storage solutions on 3 clusters: (i) Lustre (L) on cloud: is a Lustre based storage solution that is fully owned and operated by a cloud vendor. Representative examples of this type would be AWS-FSx Lustre or Azure-Lustre solution, (ii) NAS (NAS) on cloud: is is an off-the-shelf network-attached-storage solution that we buy from a storage vendor and deploy on a cloud provider. Both the hardware and software is fully designed and deployed by the storage vendor. We consider it as a fully managed offering on the cloud. Few options would include NetAPP, VAST or PureStorage. (iii) NAS_SW (N_SW) on cloud: is a hybrid aggregated network-attached-storage solution on the cloud. Representative examples for this type of solution include Hammerspace or Weka. While the first Lustre based solution is fully POSIX-compliant, the later two (NAS and NAS_SW) only provide NFS (v3 or v4) based client APIs which offer limited POSIX-compliance. In contrast to traditional HPC settings which require and leverage Lustre’s strong POSIX semantics, we note that for AI research usecases, POSIX semantics offered by NFS is sufficient to cover an overwhelming majority of use-cases effectively. We now present salient characteristics which enunciates the key implications that we had to consider while building out a benchmark to qualify these storage systems.

5

2.3.1

Capacity footprint and usage:

To illustrate the unpredictable growth of capacity, see Figure 1a on how capacity for Lustre filesystems grew in Q4 of 2025 in one particular cluster. As you can see the total capacity of storage for one of our clusters grew by over 125% in a sample period of 90 days with the biggest jump happening in a specific span of 2 weeks. Key implication: POSIX storage highly relevant and often has unpredictable growth. Additionally, since we

have a need to specifically careful about multi-tenant performance characteristics. 2.3.2

Read/write usage patterns:

To understand the throughput usage a little bit further, we shift our attention to understanding the read vs. write bandwidth usage for one specific NAS server. This file-server has 4 filesystems on it and supports a peak read/write capacity of about 150GB/s. We look at the 90 day read vs. write bandwidth ratio in Figure 1b and see that as expected reads dominate the writes. But interestingly, there are short dips where reads and writes are almost equal. This is another interesting characteristic where, datasets are frequently ingested and processed continuously in addition to loading data and reading/writing checkpoints. We see that beyond the writes of checkpoints which happen more in the background, there are short but sizeable time periods where data ingestion consumes a significant chunk of the bandwidth. Implication: While read pattern being dominant is expected and consistent with prior knowledge, we specif-

ically observe that for shared filesystems running AI research usecases there are often short spikes of data ingestion that creates significant read vs. write contention on the shared filesystem. 2.3.3

Impact of Metadata IOPS:

In Figure 1c, we plot the breakdown of operations between read, write, and Metadata operations on the cluster. Surprisingly, Metadata IOPS took the lion’s share Metadata IOPS has a significant share of IOPS usage across the cluster. Implication: We have to pay special attention to capture and validate metadata performance. 2.3.4

Blocksize per IO:

As mentioned before, we provision two kinds of filesystems for our users: (i) ‘home’ filesystem: to store all the code/conda environments and (ii) ‘project’ filesystem: to store checkpoints, config files, datasets, and logs. We observed that these two filesystems have contrasting IO size distributions. Figure 1d shows the average size (in kilobytes) per IO operation between a single ‘home’ filesystem and a ‘project’ filesystem on one particular cluster. By virtue of loading code/libraries, there is a much lower IO size on ‘home’ filesystem vs. the large IO operations on the ‘project’ filesystems. Implication: Different filesystem types have different access characteristics which means we have to suitably

study them with different benchmarks stressing different scenarios of small IOs as well as large IOs.

This centrality of POSIX-based storage to our AI research workflows necessitates us to ensure that our NFS/Lustre based systems remain stable, reliable, and performant. This is also quite challenging because these are quite diverse offerings across different storage vendors, and cloud providers. Our goal is to come up with a simplifying set of workloads that would help us easily qualify and validate these diverse storage systems while faithfully capturing some of these characteristics and workflow behaviors. While cloud environments provide excellent ease to bring up managed POSIX-like file systems (using Lustre or NFS), the challenge shifts from just provisioning to qualifying and validating these services against actual AI workloads and ensuring they meet our performance, scalability and failure-mode expectations before releasing it to our users.

6

Figure 2 POSIX Storage plays a central role for AI Research Workflows. The above figure illustrates how various

stages of the workflow depend on a shared storage system.

3

PRISM: A Benchmark Framework for AI research clusters

PRISM is a benchmark and validation framework targeting shared and distributed POSIX-based filesystems used in AI research infrastructure. We designed PRISM to address two primary challenges: (i) rapidly validate storage deployments when bringing up new ML clusters, and (ii) systematically compare performance characteristics across the proliferation of cloud providers and storage backends. Unlike production ML clusters with well-understood workloads, research settings exhibit highly dynamic, interactive, and non-repetitive access patterns, necessitating benchmarks that characterize both typical behavior and storage system limitations.

3.1

Benchmark Taxonomy

We organize the benchmark applications in PRISM along five research workflows that increasingly depend on POSIX storage, as illustrated in Figure 2. 3.1.1

Research Development Environments

Collaborative and reproducible research relies on consistent, portable development environments and reliable integration with version-controlled codebases. The git_clone benchmark measures repository cloning 7

performance, capturing the metadata-intensive operations characteristic of version control systems including directory traversal, small file creation, and permission handling. The run_tar and run_untar benchmarks measure archive creation and extraction, representative of environment packaging workflows where researchers share conda environments or container layers. These workloads are latency-sensitive due to the numerous small I/O operations involved in unpacking thousands of files during environment setup. 3.1.2

Data Preparation and Management

The file operation benchmarks (create_files, list_files, move_files, delete_files) measure metadata-intensive operations representative of data processing pipelines, where researchers frequently reorganize, filter, and transform datasets. The folder_bench benchmark creates directory hierarchies of configurable depth and breadth, measuring metadata performance for nested structures typical of organized research data repositories. 3.1.3

Data Loading

ML dataloading exhibits distinct access patterns that differ from traditional sequential file access. Read order is typically non-sequential due to stochastic optimization requiring random sampling, yet given the same random seed, access remains deterministic. The same data subset is repeatedly read across training epochs but in different orders, and multiple training jobs often access shared datasets simultaneously, creating contention. The md5_check benchmark reads files and computes checksums, supporting configurable access patterns including sequential access, randomized access with configurable seeds, and strided access for sparse reading patterns. This benchmark also supports page cache bypass via posix_fadvise(DONTNEED) to measure true storage performance without operating system caching effects. 3.1.4

Model Checkpointing

Checkpointing saves the state of an ML training process to enable recovery from failures and represents the primary artifact of training jobs. Checkpointing strategies differ significantly in their I/O patterns. In centralized checkpointing used with Data-Parallel training (DDP), a single rank writes the entire model state to one file while all ranks read from the same file during restore. In distributed checkpointing used with Fully-Sharded Data Parallel training (FSDP), each rank writes its parameter shard independently, and restore requires coordinated reads of all shards. PRISM provides comprehensive checkpointing benchmarks through ddp_save, ddp_load, fsdp_save, and fsdp_load. These benchmarks support both a configurable synthetic model for controlled experiments and the full range of HuggingFace models for validation against real-world architectures. The synthetic model provides reproducible checkpoint sizes through configurable layer count, input dimensions, and hidden dimensions, while HuggingFace integration enables testing with production models such as GPT-2, LLaMA, or BERT variants. In distributed mode, the framework ensures that only rank zero downloads HuggingFace models while other ranks wait at a synchronization barrier, then all ranks load from the shared cache directory. The rl_load benchmark addresses the asymmetric checkpoint access pattern common in reinforcement learning, where training runs on many ranks with sharded checkpoints while rollout and inference workers require the consolidated full model. This benchmark exercises the state dictionary consolidation path that aggregates distributed shards into a single model representation. 3.1.5

Dataset Generation

PRISM provides two complementary approaches for generating synthetic datasets that stress different aspects of storage system performance. Simple Dataset Creation. The create_dataset benchmark generates synthetic datasets with configurable file sizes ranging from 8KB to 1GB, using deterministic content generation based on SHA256 hashing for

8

Benchmark

ddp_save ddp_load fsdp_save fsdp_load rl_load md5_check create_dataset create_synthetic_workload create_files list_files move_files delete_files folder_bench git_clone run_tar run_untar

Description

Category

Access Pattern

Key Configuration

Train iteration followed by rank 0 checkpoint save All ranks load same checkpoint, run training iteration All ranks write shards in parallel via DCP All ranks load individual shards via DCP FSDP load with state dict consolidation Read files and compute MD5 checksums Generate synthetic files with deterministic content Generate files with configurable distributions Create N files of specified size Create files then stat each file Create then move N files between directories Create then delete N files Create nested directory hierarchies Clone repository N times Create TAR archive from dataset Extract TAR archive to directory

Checkpointing Checkpointing Checkpointing Checkpointing Checkpointing Dataloading Data generation Data generation Metadata Metadata Metadata Metadata Metadata Mixed Mixed Mixed

Single rank writes All ranks read same file All ranks write different files All ranks read different files All ranks read and consolidate Sequential, random, or strided Single rank writes Single rank writes Per-rank or shared directory Per-rank directories Per-rank directories Per-rank directories Per-rank directories Per-rank directories Per-rank archives Per-rank extraction

Synthetic or HuggingFace model Verifies via parameter checksum Creates sharded checkpoint directory Supports full state dict option For RL inference workers Chunk size, page cache control File sizes from 8KB to 1GB Arrival, size, data patterns File size, page cache bypass Measures readdir and stat Measures rename latency Measures unlink latency Depth and breadth configurable Repository URL configurable Compression options Compression options

Table 3 Complete benchmark taxonomy in PRISM, organized by category with access patterns and configuration

options.

reproducibility. Given the same seed, identical file contents are generated across runs, enabling verification and comparison studies. Multiple copies of each size can be created to produce datasets of specified total size. Workload-Aware Generation. The create_synthetic_workload benchmark provides a modular framework for generating files with realistic workload characteristics through three pluggable distribution components: • Arrival Distributions control when files are created. Options include deterministic (fixed interval), poisson (random arrivals at specified rate), and burst (grouped arrivals with quiet periods). When – simulate-arrivals is enabled, the generator actually sleeps between file creations to simulate temporal patterns. • Size Distributions control how large files are. Options include fixed (constant size), exponential (meanbased), uniform (range-based), and lognormal (heavy-tailed). Minimum and maximum size bounds can be applied to any distribution. • Data Generators control file content. Options include random (os.urandom), deterministic (SHA256based, reproducible), seeded_random (reproducible pseudo-random), zero (all zeros for deduplication testing), and pattern (repeating byte patterns for compression testing). This modular design enables researchers to construct workloads matching observed production patterns. For example, ML training datasets often follow lognormal size distributions with deterministic content, while streaming ingestion exhibits Poisson arrivals with uniform random data.

3.2

Framework Architecture

PRISM employs a three-layer architecture consisting of a command-line interface, a core measurement layer, and pluggable benchmark modules. The command-line interface implements a two-stage argument parser where common arguments such as dataset path and output directory are parsed first, followed by modulespecific arguments. This design enables each benchmark module to define its own parameters without modifying the core framework. The core measurement layer provides timing infrastructure through two Python decorators. The @benchable decorator registers functions in a global registry, enabling discovery and dispatch by name. When a function is decorated with @benchable, the framework automatically extracts its parameter signature and matches command-line arguments to function parameters during invocation. @BenchParams . b e n c h a b l e d e f c r e a t e _ f i l e s ( num_copies : i n t , dataset_path : str , f i l e _ s i z e : i n t = 4 0 9 6 ) -> b o o l : f o r i i n r a n g e ( num_copies ) : _ c r e a t e _ a n d _ w r i t e _ f i l e ( dataset_path , i , f i l e _ s i z e ) r e t u r n True

9

Listing 1 The benchable decorator registers functions for dispatch by name.

The @measurable decorator wraps functions with high-resolution timing using nanosecond-precision counters, accumulating measurements across invocations for statistical analysis. These decorators can be composed: outer functions decorated with @benchable define the benchmark entry point, while inner functions decorated with @measurable capture fine-grained sub-operation timings. @BenchParams . m e a s u r a b l e d e f _ c r e a t e _ a n d _ w r i t e _ f i l e ( path : s t r , i d x : i n t , s i z e : i n t ) : f i l e n a m e = o s . path . j o i n ( path , f " f i l e _ { i d x } . dat " ) with open ( f i l e n a m e , "wb" ) a s f : f . w r i t e ( o s . urandom ( s i z e ) ) Listing 2 The measurable decorator captures timing with nanosecond precision.

After benchmark execution, the framework computes statistics including mean, minimum, maximum, and percentiles (p50, p90, p99) for each measured operation. Since research workloads are interactive and bursty, we prioritize latency percentiles over peak throughput, as tail latencies often determine user-perceived performance. Results are serialized to JSON manifests containing all input parameters, host metadata, timing statistics, and pass/fail status. Each benchmark module implements a plugin interface consisting of a list of available functions, a modulespecific argument parser, and an execution entry point. This architecture enables straightforward extension where new benchmarks are added by creating a module that implements this interface and registering it in the module list.

3.3

Distributed Execution

Distributed benchmarks require coordinated execution across multiple processes, typically one per GPU. PRISM integrates with cluster schedulers such as SLURM to achieve gang scheduling semantics where all ranks execute synchronously. Process group initialization supports two methods. The environment variable method uses scheduler-provided master address and port information, suitable for most cluster deployments. The file-based method uses a shared filesystem path for coordination, useful when environment variables are unavailable or when testing the storage system’s synchronization primitives. For distributed checkpointing benchmarks, PRISM follows a barrier-based coordination pattern. All ranks synchronize before training to ensure readiness, synchronize after training to ensure completion, and synchronize after checkpoint I/O to ensure persistence before proceeding. This pattern isolates storage performance from process scheduling variations. The framework automatically handles GPU selection based on local rank, backend selection between NCCL for GPU clusters and Gloo for CPU-only environments, and derivation of global rank and world size from scheduler environment variables. Each rank produces its own JSON result file, enabling post-hoc aggregation that distinguishes per-rank performance variations from aggregate system behavior.

3.4

Extending PRISM

Adding new benchmarks to PRISM requires implementing the module plugin interface and decorating functions appropriately. The following example demonstrates adding a HuggingFace model loading benchmark that coordinates model downloads across distributed ranks. @BenchParams . b e n c h a b l e d e f hf_model_load ( model_name : s t r , cache_dir : str , batch_per_rank : i n t ) -> b o o l : # Rank 0 downloads model , o t h e r s w a i t

10

if

d i s t . i s _ i n i t i a l i z e d ( ) and d i s t . get_rank ( ) != 0 : dist . barrier ()

model = AutoModelForCausalLM . f r o m _ p r e t r a i n e d ( model_name , c a c h e _ d i r=c a c h e _ d i r ) # S i g n a l download c o m p l e t e i f d i s t . i s _ i n i t i a l i z e d ( ) and d i s t . get_rank ( ) == 0 : dist . barrier () # A l l r a n k s now l o a d from c a c h e r e t u r n s e t u p _ t r a i n i n g ( model , batch_per_rank ) Listing 3 Example extension: HuggingFace model loading with distributed coordination.

The framework automatically handles argument parsing from command-line or configuration files, timing collection for any functions decorated with @measurable, statistics computation across multiple iterations, warmup run exclusion from statistics, and result serialization with full provenance information. This design philosophy of minimal hand-rolled code with maximal leverage of existing tools enables researchers to focus on defining the I/O patterns of interest while the framework handles measurement infrastructure. PRISM also integrates external benchmarks such as fio for measuring peak I/O performance, providing a complete picture from synthetic microbenchmarks through ML-specific workloads. The combination of configurable synthetic models, HuggingFace model support, and extensible benchmark modules enables both controlled experiments for storage system characterization and validation against production training workloads.

4

Using PRISM

We built PRISM as a comprehensive framework for validating and qualifying POSIX-based storage systems consumed in a typical AI research cluster. Its primary purpose is to provide infrastructure teams with a systematic methodology for understanding the performance and scalability characteristics of a networked filesystem across different job sizes, benchmark categories, and cluster configurations. In this section, we demonstrate the breadth of PRISM’s applicability through four distinct use cases that we have encountered in operating storage infrastructure for large-scale AI research. We begin by characterizing the performance and scalability of a single storage system across the full suite of PRISM benchmarks (Section 4.1). We then leverage the framework to compare and contrast two fundamentally different storage architectures, a Lustre-based parallel filesystem and a NAS-based appliance. We highlight how each excel under different workload profiles (Section 4.2). Next, we examine the temporal dimension, demonstrating how we used PRISM to detect a critical performance regression introduced by a storage system upgrade (Section 4.3). Finally, we investigate how the choice of storage access interface, specifically, the fsspec Python library versus native POSIX access, affect observable performance (Section 4.4).

4.1

Performance and Scalability Characterization

The primary motivation for building PRISM was to validate and qualify the performance of a vendor’s storage solution before utilizing them on AI research clusters. For this use case, we focus on performing an in-depth, at-scale analysis of a storage system’s behavior under the key benchmark categories described in Section 3. These numbers ultimately help to establish baseline performance metrics and qualify the system’s suitability for research workloads. We executed PRISM on a cluster of 8K H100e GPUs provisioned with a NAS_SYS storage appliance. The storage system provides a total capacity of 8 PB with a peak sequential read/write bandwidth of around 150 GB/s with over 1M IOPS. We ran the full suite of checkpointing benchmarks (ddp_save, ddp_load, fsdp_save, fsdp_load) at scales ranging from 8 to 256 GPUs, dataloading benchmarks (md5_check) at scales up to 256 GPUs, and metadata benchmarks (create_files, list_files, move_files, delete_files) at 11

scales up to 256 GPUs. We apply this methodology to not only choose the right choice for a particular cluster, but also use it to qualify the readiness of the storage system as part of new cluster bring-ups. 4.1.1

Checkpoint Performance

We begin by evaluating the checkpoint performance of the storage system, as checkpoint save and restore latency directly impacts overall cluster level GPU utilization. Figure 3 presents the results for both DDP and FSDP checkpointing strategies across increasing task sizes.

(a) DDP Load

(b) DDP Save

(c) FSDP Load

(d) FSDP Save

Figure 3 Checkpoint load and save performance characterization on NAS_SYSusing PRISM.

Key Utility of PRISM: This characterization serves as a large-scale rapid smoke test. It verifies fundamental system operations (e.g., successful mounting, basic I/O correctness) across the entire cluster, confirms the filesystem’s ability to maintain performance and stability as the workload scales, and establishes quantitative baselines against which future measurements can be compared. From Figure 3, we observe that ddp_save latency remains stable as the job size scales from 8 to 256 GPUs, with a mean latency of 7700-8000 ms at the largest scale. On the contrary, we observe that the latency for ddp_load is 4000 ms up to task sizes of 128 GPUs but the P99 latency quickly saturates to up to 10000 ms

12

Figure 4 Dataloading latency characterization using the md5_check benchmark across increasing job scales under

different settings.

for 256 tasks indicating that the storage system started to show some performance variance as the scale of the jobs increased and there was some concurrent load imposed on the system. The fsdp_save (and fsdp_load) benchmark, which exercises parallel writes from all ranks, has a gradual reduction in latency as task sizes increase thanks to the benefits of parallelism available where the P99 latency (straggler rank) has reduced from 1600 ms to 500 ms (2500 ms to 600 ms respectively). These results confirm that the storage system can sustain the I/O demands of distributed checkpointing at the target cluster scale especially for highly parallel style of checkpointing. 4.1.2

Dataloading Performance

We next evaluate the dataloading performance using the md5_check benchmark, which reads files and computes checksums, exercising sequential and random read patterns representative of training data access. We compare the vanilla variant which utilizes the page-cache, sequentially access all files in the same order with a 8KB chunk size (shown as nas) with (i) no-caching option which fully bypasses the page-cache (nas_nocache), (ii) accessing the dataset in a random order (nas_nocache_random), and (iii) accessing the dataset in a random order with a larger chunk size of 32KB (nas_nocache_random_32K). Figure 4 shows the dataloading behavior across various task sizes under different settings. Overall we see that dataloading trends are similar for different task sizes indicating that the NAS_SYSis able to easily sustain the load imposed for this particular dataloading experiment. We can also observe that uncached latency (nas_nocache) is 4-13 % higher than the cached latency (nas) indicating that the OS page cache is helpful for dataloading. Randomly accessing the dataset imposes a penalty of 9-13 % overhead over the vanilla cached latency and improving the chunk size imposes an additional overhead of 6-15 %. 4.1.3

File Metadata Performance

Metadata operations affect research workflows involving environment setup, data preparation, and file management. Figure 5 presents the latency of the four metadata benchmarks across increasing client counts.

13

(a) create_files

(b) move_files

(c) delete_files

(d) list_files

Figure 5 File metadata performance characterization for NAS_SYS. Latency is reported as the mean across all ranks.

As shown in Figure 5, the create_files and move_files latency increases by 4.8 × and 12 × respectively as the client count grows from 8 to 256. The list_files operation exhibits the lowest sensitivity to scale, with latency remaining the same as we scale the number of clients. These results establish the metadata performance envelope and identify potential bottlenecks for metadata-heavy workflows.

4.2

Comparing Storage Architectures: Lustre vs. NAS

A second application of PRISM is to provide an objective, head-to-head comparison between competing storage solutions. In our research clusters, we deploy two fundamentally different POSIX-storage architectures: Lustre_SYS a Lustre-based parallel filesystem, and NAS_SYS a NAS-based storage appliance. These systems differ substantially in their inherent design, POSIX compliance, and consequently their performance characteristics. Lustre is optimized for high-throughput parallel I/O with a distributed metadata architecture, while NAS appliances unless explicitly designed usually offer weaker POSIX semantics and simpler deployment but may exhibit different scaling characteristics. Understanding their relative strengths and weaknesses across the diverse workloads of AI research is essential for making informed architectural decisions. We executed the full PRISM suite on both Lustre_SYS and NAS_SYS deployed on a cluster with compa-

14

rable hardware and software configurations. We scaled the number of tasks from 8 to 256 GPUs. Figure 6 presents the checkpointing results, and Figure 8 presents the metadata results. 4.2.1

Checkpoint operations

(a) DDP Load

(b) DDP Save

(c) FSDP Load

(d) FSDP Save

Figure 6 Checkpoint load/save performance characterization on Lustre_SYSand NAS_SYSusing PRISM

Overall, we observe that the NAS_SYSoutperforms Lustre_SYScomfortably for checkpoint loads while Lustre_SYSseems to be competitive for checkpoint writes (saves) owning to its parallel striping architecture to distribute write load across multiple object storage targets. For ddp_save, Lustre_SYSoutperforms NAS_SYSby 9 to 35 %. For fsdp_save, Lustre_SYScontinues to provide lower latency (up to 26 %) at lower task count but loses out to the NAS_SYS(up to 2 × worse) for larger task sizes. Comparing the load performance, NAS_SYSoutperforms Lustre_SYSacross most task-counts by up to 20 % and 3 × for ddp_load and fsdp_load respectively. 4.2.2

Dataloading Operations

We next compare the dataloading time as measured by the md5_check benchmark with files being loaded randomly. We load a synthetic dataset with 800K small files (32KiB size) laid out in a flat single directory. 15

Lustre_SYS’s are notoriously bad at handling multiple files residing in a single directory because of the prohibitively limited metadata performance. In order to mitigate that bottleneck, the general recommendation is to shard the files across multiple directories to ensure the metadata overhead is dispersed.

Figure 7 Dataloading performance comparing NAS_SYSand Lustre_SYSwith and without sharding of data for a

synthetic dataset of 800K small files (32KiB). For Lustre_SYS, sharding of dataset helps extract more performance since metadata performance becomes the bottleneck. Analysis: As you can see in Figure 7, the performance of the Lustre_SYSis more than 3× worse than the

NAS_SYS, especially at large scales (task size = 256). Thankfully, this dataloading performance limitation gets eased if we shard the dataset across multiple folders. The performance of Lustre_SYSwith sharding reduces by almost 3× when compared the the Lustre_SYS’s performance when the files are not sharded. 4.2.3

Metadata Operations

Analysis: Similar to dataloading benchmarks, Figure 8 demonstrates that NAS_SYS significantly outper-

forms Lustre_SYS on metadata-intensive operations. The list_files benchmark shows that NAS_SYS is orders of magnitude (80×) faster than Lustre_SYS at 256 clients. Lustre’s well-known metadata scalability limitations Shan and Shalf (2007) become increasingly apparent at higher client counts.

Key insight. These results demonstrate that no single storage architecture dominates across all workload categories. Lustre-based systems excel at large-file, throughput-intensive operations characteristic of checkpointing, while NAS-based systems provide substantially better metadata performance, which is critical for the interactive, small-file-heavy workflows of AI research. At scale, the performance divergence becomes more pronounced, underscoring the importance of workload-aware storage provisioning.

4.3

Detecting Temporal Performance Regressions

Storage systems in production clusters undergo frequent upgrades stemming from firmware patches, kernel updates, configuration changes, and hardware replacements. Any of these changes may inadvertently introduce functional and performance regressions that, if undetected, degrade the productivity of hundreds of researchers. PRISM serves as a critical tool for us to run continuous integration by providing standardized benchmarks that can be executed before and after system changes to reliably detect and quantify any unintended performance impact. We present a case study in which PRISM captured a severe performance regression following a storage system upgrade on one of our clusters. After the vendor deployed a routine software update, our automated PRISM runs revealed a dramatic increase in checkpoint latency. Specifically, the fsdp_load benchmark regressed by 16

(a) create_files

(b) move_files

(c) delete_files

(d) move_files

Figure 8 Metadata performance comparison between Lustre_SYS and NAS_SYS. (a) create_files. (b) list_files. (c) delete_files. (d) move_files.

approximately 4× for small jobs (2 nodes) and up to 8× for large-scale jobs (128 nodes - not shown in the graph). Upon investigation, we traced the root cause to a bug in the vendor’s NFS implementation. The openat system calls were incurring a pause of approximately 5 seconds when concurrent clients attempted to access the same file. This lock contention caused I/O stalls that cascaded into substantial increases checkpoint latency. Armed with the reproducible evidence from PRISM, we reported the issue to the vendor, who confirmed the bug in their NFS implementation and provided a patch. We then used PRISM to validate the fix before rolling it out to production. Figure 9 shows the checkpoint latency before and after the fix was applied, confirming that performance was restored to pre-upgrade levels. As shown in Figure 9, the fsdp_load mean latency at 16 tasks decreased from 2109 ms (before fix) to 891 ms (after fix), representing 2.3× improvement. Without PRISM’s continuous monitoring, this regression could have persisted undetected for weeks, resulting in a significant loss of GPU-hours across the cluster as jobs would have waited longer to resume from checkpoints.

17

(a) Checkpoint load (FSDP Load)

(b) Checkpoint save (FSDP Save)

Figure 9 Performance regression detected by PRISM after a storage system upgrade on the NAS_SYS. The latency

before and after the fix are shown and error bars represent 95% confidence intervals.

Key Utility of PRISM This case study illustrates the value of PRISM as a continuous integration tool. By establishing performance baselines and running standardized benchmarks after every system change, infrastructure teams can detect regressions early, provide vendors with actionable evidence, and validate fixes before they reach production users.

4.4

Impact of the Storage Access Interface: fsspec vs. Native Access

As discussed in Section 2.1, fsspec has become a useful component of the modern AI research software stack, providing a unified Python interface for accessing diverse storage backends. While this abstraction simplifies application code and enables portability across storage systems, it introduces an additional software layer between the application and the filesystem. We used PRISM to quantify the performance overhead introduced by fsspec compared to native POSIX filesystem access. We focused our evaluation on the list_files benchmark, as directory listing is a frequent operation in data exploration, dataset validation, and experiment management workflows. This benchmark is particularly sensitive to per-operation overhead because it involves a large number of individual metadata calls. We executed the benchmark in two configurations: (i) using native POSIX os.listdir() and os.stat() calls, and (ii) using the equivalent fsspec filesystem interface (fs.ls() and fs.info()). Figure 10 reveals a substantial performance gap between native access and fsspec specifically for list_files. For a directory containing 5000 files, native access completes in 104 ms, while fsspec requires 1651 ms—a 15× slowdown. The overhead is attributable to fsspec’s additional layers of abstraction, including parsing the filesystem type, filesystem dispatch, and result normalization into Python dictionaries. Key insight. While fsspec provides valuable developer convenience and portability, its performance overhead is non-trivial for metadata-intensive operations. Researchers and framework developers should be aware of this trade-off, particularly when building data pipelines that involve extensive directory traversal or file enumeration. For performance-critical paths, native POSIX access remains substantially faster. This finding also motivates future work on optimizing fsspec’s metadata handling, for example through batched stat calls or result caching. In production clusters, it becomes hard to root-cause where the reported performance slowdown originates, often the storage backend itself being suspected. In these cases, we found PRISM with interface optionality gives us an A/B test to evaluate the latency/throughput as seen by the workloads, helping focus energy on the right layer

18

(a) create_files

(b) move_files

(c) list_files

(d) delete_files

(e) move_files

(f) md5_check

Figure 10 Performance comparison of native POSIX access vs. fsspec for benchmarks which can leverage the fsspec API. The delete_files and list_files benchmarks had notable performance differences. Table 4 Comparison of PRISM with Prior Storage Benchmarks. We categorize benchmarks by their execution model, workload awareness, and target use case. PRISM is unique in its use of real PyTorch operations to benchmark a comprehensive set of AI research workflows on standard POSIX filesystems. Dimension

Execution Model ML Workload Aware Checkpointing Data Loading Env Setup / Metadata POSIX Interface Distributed Execution HuggingFace Integration Target Use Case

5

fio Axboe (2023)

IOR Shan and Shalf (2007)

Filebench Tarasov et al. (2016)

MLPerf Storage MLCommons (2025)

elbencho Breuner (2023)

PRISM

Synthetic × × × × ✓ × × Raw Perf.

Synthetic × × × Partial ✓ ✓ × HPC I/O

Synthetic × × × Partial ✓ × × Workloads

Simulated Partial × ✓ × ✓ ✓ × Prod. ML

Synthetic Partial × × × ✓ ✓ × Hybrid

AI Training Frameworks

✓ ✓ ✓ ✓ ✓ ✓ ✓

AI Research

Related Work

Storage benchmarking has evolved over several decades, with tools designed for diverse objectives ranging from raw device characterization to application-specific performance evaluation. We organize related work into two primary categories: general-purpose POSIX storage benchmarks that excel at measuring peak performance but lack awareness of ML workload characteristics, and ML-specific data management systems that optimize for training workflows but sacrifice POSIX generality. As summarized in Table 4, PRISM bridges these approaches by providing ML-aware benchmarks for general-purpose POSIX filesystems.

5.1

General-Purpose Storage Benchmarks

General-purpose benchmarks form the foundation of storage performance analysis. They can be subdivided into synthetic micro-benchmarks, HPC-oriented parallel benchmarks, and workload-driven simulators. Synthetic Micro-benchmarks like fio Axboe (2023), IOzone Norcott and Capps (2016), and Bonnie++ Coker (2023) are the de facto standards for storage benchmarking. They provide extensive configurability for block-level I/O patterns, excelling at characterizing raw storage performance by stressing specific subsystems.

19

However, they generate independent I/O streams that do not capture the correlated, bursty access patterns of distributed training, where storage operations are synchronized with gradient computation and collective communication. While these tools provide invaluable baseline measurements, the gap between their synthetic I/O and actual training workloads can lead to misleading capacity planning decisions. HPC Parallel Benchmarks such as IOR Shan and Shalf (2007) and mdtest Lawrence Livermore National Labo-

ratory (2023) target large-scale parallel filesystems, measuring aggregate bandwidth and metadata operation rates across distributed processes. IOR coordinates file-per-process and shared-file access patterns representative of scientific simulation checkpointing, while mdtest stress-tests directory operations for job staging. The IO500 Kunkel et al. (2017) benchmark combines these two to provide a holistic ranking of HPC storage systems. These tools assume relatively uniform access patterns across processes, which diverges from the heterogeneous I/O behavior of ML training where checkpoint writes are synchronous barriers and data loading is stochastic. Workload-Driven Simulators like Filebench Tarasov et al. (2016) introduce workload modeling through a

domain-specific language to describe file operations and access probabilities. Users can compose synthetic workloads mimicking database transactions or web servers. However, creating accurate ML workload models requires deep understanding of framework internals and data formats—knowledge that quickly becomes outdated. PRISM addresses this by executing actual PyTorch operations rather than modeling them.

Modern Hybrid Benchmarks like elbencho Breuner (2023) modernize storage benchmarking with support for

distributed execution, GPU memory as a source/sink, and unified interfaces across files, objects, and block devices. However, elbencho shares the fundamental limitation of synthetic I/O: it measures what storage can deliver rather than what ML training will demand. The distinction matters because training workloads exhibit complex temporal patterns that synthetic steady-state benchmarks cannot capture.

5.2

ML-Specific Data Systems and Benchmarks

A parallel line of work optimizes data management specifically for ML training, often departing from POSIX semantics entirely. ML-Specific Benchmarks like MLPerf Storage MLCommons (2025) and DLIO Devarajan et al. (2021) represent

the most direct effort to benchmark storage for ML workloads. They measure data loading throughput by simulating training loops that consume samples at rates derived from accelerator computation speeds. However, they focus exclusively on data loading and do not address checkpointing, environment setup, or the diverse metadata operations characteristic of research workflows. Additionally, by simulating rather than executing training, they may miss framework-specific behaviors. Data Orchestration Layers such as Alluxio Li et al. (2018) provide caching and tiering to accelerate data access. These systems excel when training jobs can be adapted to their APIs, but research environments often require direct POSIX access for interactive development and debugging. The overhead of data ingestion can negate caching benefits for rapidly evolving research workloads. Optimized Data Formats like LMDB Symas Corporation (2023) and WebDataset Breuel (2023) optimize se-

quential access by packing samples into large files. These formats eliminate per-sample filesystem overhead but require upfront data conversion, complicating dataset updates and interactive exploration.

Object Storage Systems, often benchmarked with tools like COSBench Zheng et al. (2013) and warp MinIO,

Inc. (2023), provide high-throughput sample serving but represent a different architectural choice. Replacing POSIX with purpose-built abstractions requires significant infrastructure investment and does not address the broader storage needs of research clusters, including code repositories and experiment artifacts.

5.3

Positioning PRISM

PRISM occupies a distinct position in this landscape. Rather than generating synthetic I/O or requiring specialized data formats, PRISM executes actual PyTorch operations—training iterations, DDP and FSDP checkpointing, distributed state dictionary operations—against real models, including the full HuggingFace model zoo. This approach ensures that benchmarks reflect current framework behavior, including optimizations and communication patterns that would be difficult to model accurately. 20

The key insight is that for AI research infrastructure, storage validation requires testing the actual operations researchers will perform. A filesystem might deliver excellent fio numbers yet exhibit poor checkpoint performance due to metadata overhead or lock contention. By providing benchmarks spanning research development environments, data loading, and checkpointing, PRISM enables comprehensive validation that synthetic benchmarks cannot provide. We view PRISM as contributing a pragmatic methodology. Our technical contribution lies in the systematic identification of research-relevant storage operations and a framework for extensible benchmark development. We believe the value proposition is strongest for organizations deploying storage for AI research clusters, where the gap between synthetic benchmarks and the interactive, exploratory workloads of research creates real engineering challenges.

6

Conclusion

The performance of storage systems is a critical, yet often overlooked, factor in the productivity of AI research clusters. The dynamic, metadata-intensive workloads common in research differ significantly from predictable, throughput-oriented production training patterns. This divergence means traditional benchmarks fail to adequately model real-world usage, leading to suboptimal infrastructure choices. To address this, we developed PRISM, a benchmark suite that provides a high-fidelity assessment of storage performance by executing real PyTorch operations that mirror the complete AI research lifecycle. We have demonstrated PRISM’s significant impact as a qualification and validation suite for the specialized storage needs of AI research. Its utility is proven in establishing performance baselines for new cluster deployments, enabling objective architectural comparisons that inform workload-aware provisioning, detecting performance regressions as a continuous integration tool, and quantifying the effects of cross-filesystem interference on shared backends. By capturing these complex, real-world behaviors, PRISM provides actionable insights that synthetic benchmarks cannot. In conclusion, PRISM bridges the gap between generalized storage testing and the specific demands of AI research. It offers a robust, data-driven methodology for infrastructure teams and vendors to select, validate, and optimize storage solutions. By ensuring storage is purpose-fit for its demanding role, PRISM empowers organizations to maximize GPU cluster efficiency, accelerate innovation, and realize the full potential of their research investments.

21

References Jens Axboe. fio - flexible I/O tester. https://github.com/axboe/fio, 2023. Accessed: 2025-01-28. Thomas Breuel. WebDataset: A library for large-scale data loading. https://github.com/webdataset/webdataset, 2023. Accessed: 2025-01-28. Sven Breuner. elbencho: A distributed storage benchmark for files, objects and blocks with support for GPUs. https://github.com/breuner/elbencho, 2023. Accessed: 2025-01-28. Russell Coker. Bonnie++ benchmark suite. https://www.coker.com.au/bonnie++/, 2023. Accessed: 2025-01-28. Hariharan Devarajan, Huihuo Zheng, Anthony Kougkas, Xian-He Sun, and Venkatram Vishwanath. DLIO: A datacentric benchmark for scientific deep learning applications. In Proceedings of the IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 81–91. IEEE, 2021. Julian M. Kunkel, Andreas Dilger, and Sven Breuner. Establishing the IO-500 benchmark. In Proceedings of the Workshop on Performance and Scalability of Storage Systems (PDSW-DISCS). Virtual Institute for I/O, 2017. Lawrence Livermore National Laboratory. mdtest: Metadata performance benchmark. https://github.com/hpc/ior, 2023. Part of the IOR repository. Haoyuan Li, Ali Ghodsi, Matei Zaharia, Scott Shenker, and Ion Stoica. Alluxio: A virtual distributed file system. In Technical Report, UC Berkeley, 2018. MinIO, Inc. warp: S3 benchmarking tool. https://github.com/minio/warp, 2023. Accessed: 2025-01-28. MLCommons. MLPerf storage benchmark suite. https://github.com/mlcommons/storage, 2025. Version 2.0. William Norcott and Don Capps. IOzone filesystem benchmark. https://www.iozone.org/, 2016. Accessed: 2025-0128. Hongzhang Shan and John Shalf. Using IOR to analyze the I/O performance for HPC platforms. In Proceedings of the Cray User Group. Lawrence Berkeley National Laboratory, 2007. Symas Corporation. LMDB: Lightning memory-mapped database. https://www.symas.com/lmdb, 2023. Accessed: 2025-01-28. Vasily Tarasov, Erez Zadok, and Spencer Shepler. Filebench: A flexible framework for file system benchmarking. USENIX ;login:, 41(1):6–12, 2016. Qingsong Zheng, Hai Chen, Yongjun Wang, Jianhua Zhang, and Jie Duan. COSBench: Cloud object storage benchmark. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE), pages 199–210. ACM, 2013.

22

Appendix A

Acknowledgments

We would like to thank Shubho Sengupta, Kim Hazelwood, Vivek Pai, Kevin Lee for their pioneering work on architecting and building high performance research clusters with a focus on usability for FAIR. Building successive versions of the clusters with varying POSIX storage solutions, shaped the vision for PRISM, from its origins as a simple file-creation validation tool to a benchmark suite tailored for AI research storage workloads. We are also grateful to the Cloud Foundation team(Chris Henry, Chandan Avdhut) at Meta for their generous support in standing up the filesystems and provisioning the compute clusters that made this work possible.

23