S TOR R EP: Storage Research Experiment Patterns on Chameleon Cloud and Trovi Ray A. O. Sinurat† *, Yuyang Huang†∗ , Nanqinqin Li⋆ , Mark Powers§† , Michael Sherman§† , Kate Keahey§† , Haryadi S. Gunawi†
arXiv:2606.16252v1 [cs.DC] 15 Jun 2026
†
University of Chicago
⋆
Princeton Unversity
Abstract
Argonne National Laboratory
rect results (i.e. lead to the wrong/opposite conclusion). This significantly hampers the communication and mutual understanding within our community. In the recent decade, the situation has drawn many researchers’ attention. Moreover, there have been many efforts in improving the reproducibility for papers’ artifact from both the author-side (e.g. developing an minimal-effort platform to package the artifact [4, 25]) and reader-side (e.g. serving open testbeds that provide numerous types of hardware to help the research match the environment configuration[2, 34]). Moreover, an increasing number of conferences are recommending and assigning ”Artifact Evaluation” or similar badge, which further highlights the importance of reproducibility in our community. Despite increasing attention and interest in the computer science community, reproducibility in the storage research field has been progressing at a relatively slow pace. We conducted an case study on the research papers’ artifacts related to Solid State Drives (SSDs) that were published over the last decade. We focused on 195 artifacts that claimed to run experiments on SSD simulators, such as SSDsim [40], SimpleSSD [42, 37], and NANDFlash [41] since these simulators can be run on almost any device equipped with CPU and memory, and, therefore, the artifacts that use them should be easier to reproduce compared to other types of artifacts that require special hardware and software configurations. To evaluate the reproducibility of these artifacts, we categorized them based on 6 criteria: Public, Compilation, Execution, Automation, Result, and Package. Surprisingly, we found that out of the 195 artifacts, only one (0.5% of the total artifacts) satisfied all 6 criteria and was deemed reproducible. We discuss our investigation and its implications further in Section 2. We understand the importance of conducting experiments that are not only accurate and insightful but also reproducible. As researchers, it is our responsibility to set up the ideal environment for experimentation, create experiment scripts, develop analysis and evaluation tools, and provide expected outcomes from these experiments (i.e. artifacts). However, we all know that this can be a rigorous and challenging process. Fortunately, Chameleon [2] platform addresses these issues
Storage experiments are vital to advancing storage research, but creating extensible and reproducible storage artifacts can be a challenging task. Our research has shown that only 1% of SSD simulator-based experiences are packaged and 0.5% of them can be easily reproduced. The lack of such artifacts without proper reproducibility can significantly impede the advancement of storage research. The biggest challenges in these types of experiments are ensuring that we have the correct environment to conduct them and creating extensible experiments that can be built upon in future research. To address this issue, we introduce S TOR R EP, a thorough study that provides six extensible and reproducible storage experiment artifacts that serve as the foundation for further storage research, utilizing the Chameleon infrastructure. Our study offers experiment patterns and guidelines that can help researchers create transparent and dependable storage experiments. We have successfully integrated our methods in several experiments in multiple community and educational events over several years and produced publicly accessible artifacts that can be extended and fully reproduced without any restrictions.
1
§
Introduction
In the computer science community, the evaluation of research projects typically involves the use of artifacts, which consist of a combination of environment configuration (e.g. hardware and software setup and configuration), experiment scripts, and analysis of the experiment output (e.g. generation of figures). Reproducing the artifacts described in the research papers plays a crucial role in helping other researchers understanding and evaluating the research projects. However, such process can be difficult since the researchers may not be able to match the exact environment configuration with the artifacts’ designed configuration and such discrepancy in the environment configurations can lead to results inconsistent with the original papers’ claimed results, or completely incor* The authors contribute equal amount of work and are sorted alphabetically based on their first names.
1
by integrating reliable and efficient tools that help researchers accomplish these arduous tasks with ease. One of the most challenging aspects of reproducing artifacts is providing the desired environment. Chameleon helps researchers overcome this hurdle by offering multiple types of storage and various types of nodes such as bare-metal machines and virtual machines, among other configurations. Additionally, Chameleon embraces the OpenStack [23] ecosystem, which provides a better user experience and ease of managing environments. This means that researchers can now focus on their experiments without worrying about environment configurations. One such feature of Chameleon is the integration of the Jupyter Notebook Interface [24], a widely recognized platform for running data science tasks. This integration allows researchers to run Python codes or shell scripts that can aid in the packaging of their experiments seamlessly. Moreover, Chameleon also offers the ability for users to share their experiment artifacts publicly on the Chameleon Trovi [4] platform, providing other users with the opportunity to reproduce their artifacts within the Chameleon infrastructure. This level of collaboration and transparency creates a vibrant and engaging community of researchers working together to achieve common goals. Our research endeavors have led us to leverage the Chameleon platform for conducting and packaging our artifacts. By studying the four basic patterns provided by Chameleon in their blog [3], we were able to take advantage of the platform’s infrastructure capabilities, including the straightforward setup of experiment containers and the efficient management of spawned server instances. Ultimately, we present S TOR R EP — six storage-related artifacts that can be quickly and easily shared, reproduced and extended within the platform. S TOR R EP includes (1) Simple Flexible I/O (FIO) [13] Benchmark: an artifact to run FIO benchmark on RAM disk (mounted with tmpfs) and a single SSD device, (2) Simple File System Benchmark: an artifact to run multiple FileBench [9] workloads on ext4 file system, (3) FIO Benchmark: an artifact extended the Simple FIO Benchmark to run FIO Benchmark on a RAID-0 SSDs array, (4) File Systems Benchmark: an artifact to simulate busy file server workload on multiple file systems in Linux, (5) Remote Direct Memory Access (RDMA): an artifact to test bandwidth and latency on local InfiniBand node, (6) F LASH N ET: neural network approach for fostering storage research in ML-based per-IO latency prediction. Over the past two years, we have practiced using S TOR R EP in various settings. These include a seminar class focusing on reproducibility with over 20 students, Chameleon Summer REPRO 2020 with seven graduate researchers, and FAST’23 Birds-of-a-Feather (BoF) with 15 expected registrants. The FAST’23 BoF specifically focused on the extensible and reproducible experiment pattern called ”breakable
pattern” methodology described in this work. Through these opportunities, we have learned several lessons to further improve artifact reproducibility and S TOR R EP’s usability for users. First, artifacts should be selfcontained, as we cannot assume all users have prior knowledge or familiarity with associated papers. Second, authors should manage users’ expectations throughout the reproduction process, including expected time consumption, output, figures, and other relevant information. Third, workflows should be idempotent to reduce user error and confusion. Finally, artifacts should only present immediate and relevant information to avoid overwhelming the user. Overall, we show that it is plausible to create reproducible and extensible experiment artifacts. We conclude with many interesting discussions to explore in the future. All experiments for S TOR R EP are made public.
2
The Bleak State of Reproducibility in Storage Research
Over the past decade, the storage market has experienced tremendous growth due to the explosion of big data and datadriven applications, such as machine learning (ML) and artificial intelligence (AI). Along with the market, research in storage systems continues to thrive in academia. Despite the success in research, the storage system research community has historically neglected the issue of reproducibility, leading us to question the extent of its current impact. Since many studies on storage systems [55, 36, 59, 39, 35, 53], particularly on solid-state disks (SSDs), have relied heavily on SSD simulators such as SSDsim [40], SimpleSSD [42, 37], and NANDFlash [41]. This situation arises because the majority of SSDs and their controllers/firmware are heterogeneous and proprietary to manufacturers. More importantly, the use of SSD simulators allows researchers to model and simulate all SSD characteristics and behavior along with the complete storage stack on standard CPUs (e.g. x86 or arm64) and DRAM, eliminating the need for special hardware. Hence, in this paper, we will use the research papers using SSD simulators as the case study. Leveraging simulators to conducting research should have made reproducing SSD research’s artifact (i.e. the combination of the environment configurations, experiment scripts, and the analysis of the results) much more manageable than other types of research requiring special hardware (e.g. ML/AI research with Graphics Cards or distributed and network systems research with Infiniband Network Interface Cards). However, the reality is far from what is expected. Throughout the case study, we investigate 193 SSD research papers published from 2010 to 2020 that used an SSD simulator for execution and evaluation. We determine the papers’ artifacts’ reproducibility formally based on six criteria: (1) Public — whether the artifact is publicly available and 2
Simulator Used
Total
Public
Compilation
Execution
SSDSim[40] MQsim[58] SimpleSSD[42] NANDFlash[41] gem5[31] FEMU[51] FlashSim[48] Total
31 2 35 28 3 4 90 193
1 1 4 1 1 2 3 13
1 1 3 0 0 0 0 5
0 1 1 0 0 0 0 2
Auto- Result mation 0 0 1 0 0 0 0 1
0 0 1 0 0 0 0 1
any restrictions. In the event of a corrupted OS, the instance can be rebuilt in a single click, without losing any machine reservations. The web interface is intuitive and easy to use, providing users with full control over their instances, reservations, networks, OS images, and object storage which can be accessed at [2].
Package 1 0 1 0 0 0 0 2
For straightforward cases without the need for OS or other low-level modifications, Chameleon also grants simple virtual machines by utilizing Kernel-based Virtual Machine (KVM) which will spawn the instance faster due to their lightweight nature. Up until now, Chameleon provides seven configurations of KVM nodes which differ in their number of virtual CPUs, memory size and storage size. KVM nodes are similar to bare metal nodes except that there is no performance isolation. Thus, these types of nodes will be beneficial for users who do not need precise hardware measurements and profiling.
Table 1: Artifacts Investigation. We investigate 193 artifacts published from 2010 to 2020 and found only 1 of them meets all the 6 criteria.
downloadable, (2) Compilation — whether the artifact can be compiled, (3) Execution — whether the compiled artifact can be run successfully, (4) Automation — whether the artifact is shipped with a script to automate the reproducing process, (5) Result — whether the results are consistent with the original researchers’ result, and finally (6) Package — whether the artifact is packaged into an easily accessed container (e.g. virtual machine or docker container). Surprisingly, as shown in Table 1, only 13 artifacts (6.7% of the total artifacts) are publicly available, 5 artifacts (2.6% of the total artifacts) can be compiled, 2 artifacts (1% of the total artifacts) are runnable, 1 artifact (0.5% of the total artifacts) produces similar results compared to the original claimed results, and 2 artifacts are packaged (1% of the total artifacts). This shocking revelation further motivate the development of this research project. These shocking results further urge and motivate us this paper.
3
Chameleon also encourages its users to reproduce their research on the Chameleon platform by introducing the Jupyter Notebook Interface [24] which is commonly used in data science and artificial intelligence fields. With Jupyter Notebook, users have the ability to attach their experiment scripts natively in Python programming language or shell scripts and execute these scripts in the Jupyter Notebook interface. Chameleon also integrates API access of Chameleon internal features, such as reservation and spawning instances using shell scripts and Python library [25], by wrapping the OpenStack APIs to the Jupyter Interface. Users will be able to set the experiment flows without any limitations. However, the most prominent feature of the Jupyter ecosystem lies in its replayability. Jupyter Notebook will save the output and variables obtained from the execution of the experiment code blocks. Hence, any important parts of the experiment can be re-run without re-executing the whole experiment and its preliminary steps, such as setting up the experiment environment and installing required softwares, subsequently minimizing the time to reproduce and modify the experiment. Indeed, we observed that the time needed to set up the experiment is profoundly longer than the runtime of the experiment itself.
Chameleon Cloud and Trovi
Chameleon [44] is an experimental testbed with the primary goal of supporting Computer Science research and education. The project, which has received full funding from the NSF FutureCloud [7], has a budget of $10 million and includes an impressive array of resources. These include 586 nodes (i.e.bare metal servers), each with unique features such as GPU and FPGA nodes, as well as newly added storage nodes. The nodes are hosted in two locations, the University of Chicago and Texas Advanced Computing Center (TACC), and are connected using a high-speed 100Gbps network. One of the key objectives of Chameleon is to provide its users with flexibility by leveraging the OpenStack ecosystem [23] which is widely used in industry to manage huge numbers of nodes in clusters. This system is employed throughout Chameleon, enabling the management of OS images, instances, and user artifacts through object storage. By default, Chameleon offers a bare-metal experimental environment, where researchers have complete control over the machines, which allow low-level modifications such as operating system (OS) hacking and changing root privileges without
The reproducibility feature in Chameleon is taken further by the newly added Trovi platform [4] in the Chameleon ecosystem. Trovi facilitates sharing and uploading of experiment artifacts containing Jupyter Notebook files and scripts. This feature of Trovi platform allows other researchers to validate and reproduce the experiments carried out by other researchers. Trovi allows users to: (1) run the experiments on Chameleon nodes or (2) download the artifacts and run the experiments on other computational platforms. With almost 60 public experiments hosted on Trovi that are publicly accessible at [4], the platform is an invaluable resource for researchers looking to collaborate and build upon each other’s work. 3
3.1
Chameleon Storage Infrastructure
their storage anywhere in any instance. Additionally, to facilitate shared storage between bare-metal and virtual machines, Chameleon provides pre-allocated shared file systems utilizing OpenStack Manila [22], which allows users to detach file systems while maintaining data persistence. To further enhance the user experience, Chameleon also provides utility scripts to manage, mount, upload, and download data in the storage mechanisms mentioned above, helping researchers accelerate their development and empower reproducible research.
Chameleon is an innovative project that addresses the critical needs of storage research community. The introduction of new types of storage nodes is a testament to Chameleon’s commitment to providing a better environment for storage research. To ensure that the storage nodes meet the needs of the researchers, the Chameleon team has ensured that the nodes are equipped with the latest technology such as solid-state drives (SSDs) and remote direct memory access (RDMA) over InfiniBand. In addition, Chameleon offers SSDs from various vendors to cater to a broader range of storage research needs. The nodes in Chameleon typically come with a single solid-state drive or hard disk drive with ample space of more than 300GB. However, this is not sufficient for storage system researchers. The operating system’s operations and other operations on users’ data can cause additional performance impacts on the storage experiments if they are run altogether in a single storage device. To address this, storage nodes in Chameleon have been designed with one or more empty SSDs that support a redundant array of independent disks (RAID) by default. This gives researchers the flexibility and performance isolation they need to carry out their storage experiments with ease. The storage nodes have a capacity of more than 1TB, which is sufficient for storing experiment data and results. The Chameleon project is making significant strides in supporting the research community with its cutting-edge technology. Currently, the project has 30 bare-metal storage nodes that are capable of running storage experiments hosted at the University of Chicago and TACC that can be tweaked without any restrictions. To this end, the Chameleon project has announced plans to add more storage nodes in the near future. The additional nodes will be specifically designed to support more researchers and enable them to carry out and reproduce experiments within the Chameleon platform. The Chameleon project is committed to advancing research and reproducibility by supporting a growing number of researchers, as evidenced by their efforts to add more nodes and cutting-edge technologies. Hopefully, in the future, researchers will have more resources at their disposal to carry out their experiments, leading to faster and more reliable results. Chameleon provides not only storage instances but also several mechanisms [5] that enable users to store and share their data effectively. Firstly, users can take a snapshot of their primary instance disk and create a bootable image containing all their data and configurations, which can be easily restored the next time they reserve servers on the Chameleon testbed. However, to mitigate longer boot times due to larger images, Chameleon provides binary objects storage, similar to enterprise-level storage options like Google Cloud Storage [14] and Amazon Web Services S3 [1], with enhanced FUSEbased file system integration. This enables users to mount
3.2
Storage Research on Chameleon
The Chameleon project is proven to be flexible for storage research by providing a wide range of storage nodes and disk vendors to researchers. These features have made it easier for researchers to evaluate their experiments, saving them from the hassle of acquiring multiple disks from different vendors, which can be an uphill task due to budgetary constraints or hardware scarcity in the market. As we all know, producing strong publications requires researchers to evaluate their works in multiple environments and vendors to demonstrate the efficacy of their research. With the availability of a range of storage nodes and disk vendors, storage researchers can now perform their experiments in different environments, leading to reliable research outcomes. One of the works that benefits from the Chameleon platform is LinnOS [38], published at OSDI 2020. They perform performance anomaly detection by leveraging latency prediction which becomes arduous over time because the device is getting faster. It is also well known that anomalies even in milliseconds granularity can cause monetary and productivity losses. In result, to adapt to this condition, they embrace the machine learning approach and build a model that can predict the speed, whether it is “fast” or “slow”, of every I/O that will be sent to an SSD device even at the millisecond granularity. [38] needs multiple enterprise-level SSD vendors to evaluate the efficacy of their model which fortunately is provided by Chameleon. The research team cited in [38] leveraged the Chameleon platform by replaying various storage traces on different devices to gather diverse performance profiles that were used to train their model. The support for heterogeneous SSDs provided by Chameleon played a significant role in enabling this type of research. In fact, LinnOS experiments can be accessed and reproduced at the Chameleon Trovi platform [16]. There are also some other experiments that could have utilized the Chameleon platform. One of them is the IODA [52] project published at SOSP 2021. IODA is an important research that aims to develop deterministic flash array design to provide predictable latencies that become a solution to severe service level objective (SLO) violations on a large scale. To fulfill the goal, the IODA team needs multiple SSDs for profiling tail-latency and evaluating software policy 4
4
designs. On top of that, they also require experiments to run on top of bare flash arrays to show performance limitations of Linux software RAID running on an array of SSDs. For that, they bought a high-performant and expensive machine with at least more than 4 PCIe slots to attach 4 SSDs and other peripherals that need further additional flash arrays set up using Linux software RAID. Nowadays, Chameleon supports flash array research with 2 or more nodes that can accommodate 4 SSDs or more. These other projects, Harmonia [47], Gecko [56], FlashOnRails [57], Purity [32], SWAN [45], RAIL [54], can also be adopted and reproduced in Chameleon storage infrastructure. Chameleon also supports persistent memory (PM) servers equipped with 3TB Optane PM and 4-socket of 112 cores Intel Xeon processors. This enables experiments using PM such as PACTree [46] published in SOSP 21 in the Chameleon storage infrastructure. PACTree is an improved persistent key-value store on top of Intel Optane PM. PACTree team performed a thorough study of Optane PM performance characteristics to figure out performance and scalability of the real PM hardware. The main discoveries are twofold: (1) PMoptimized storage stacks may face a bottleneck at the PM bandwidth level and (2) the scalability of these stacks could be limited by the slow write latency of PM. In order to address these obstacles, they developed PACTree, a key-value store that utilizes Intel Optane PM. PACTree minimizes the usage of PM bandwidth and prevents slow PM latency from becoming a scalability limitation. This type of experiment requires a multi-core machine with a high number of cores along with Optane PM integration to evaluate the effectiveness and efficiency of PACTree. Not only PACTree [46], other projects such as TIPS [50] and TimeStone [49] can be run and reproduced on Chameleon infrastructure. Recent works such as CrossFS [60], FusionFS [61], and KLOCs [43] focused on figuring out benefits, limitations and implications of near-storage computing and memory heterogeneity. CrossFS discovered the advantages of dividing file systems across userspace, the storage, and the OS to achieve better I/O performance and more sophisticated concurrency. FusionFS adopted and extended CrossFS design to implement CISC-style (Complex Instruction Set Computer) I/O operations that can naturally offload data for near-storage processing, reducing I/O operations such as data movement system calls and PCIe communication. On the other hand, KLOCs examined the advantages of memory heterogeneity using Optane memory. These works fortunately got the access to Optane PM [15] servers, however, emulating nearstorage computing devices is still a rigorous work. The availability of Optane PM servers and near-storage computing devices (such as SmartSSD [28]) that natively supported in Chameleon would ease these types of research to: (1) conduct more detailed studies using enterprise-level devices and (2) explore new research areas in utilizing near-storage computing and storage heterogeneity.
Reproducible Storage Experiments on Chameleon
S TOR R EP comprises a series of storage-related artifacts that can be easily reproduced and extended within the Chameleon infrastructure, which is publicly available through the Chameleon Trovi platform. S TOR R EP utilizes the Jupyter Notebook Interface and follows an opinionated and adaptable experiment pattern that we refer to as the ”breakable pattern” that is illustrated in Figure 1. Rather than relying on a single, unwieldy experiment script, we break the process into several manageable milestones. Setup: In this step, we allocate appropriate resources to serve as experiment containers in Chameleon, including the number of nodes, node types, network switches, router, and floating IPs. We then install or compile any necessary software, such as drivers or utilities, from various package managers and other source version control systems. Typically, the provisioning of experiment instances will take some time to complete, as we must spawn bare-metal instances, boot the operating system, and perform OS initialization tasks such as creating server users based on SSH keys and attaching disks to the server. Experimentation: Here, we can run the experiment code on the Chameleon nodes by executing scripts through an SSH connection based on the floating IPs we obtained in the setup stage. Users have the freedom to run the experiment code multiple times and adjust experiment configurations as desired, without needing to repeat the setup process. All experiment output generated by the experiment code is saved in files for later analysis. Analysis: This stage involves parsing and extracting crucial data from the output files generated during the experimentation stage. After identifying the key information, we encourage users to plot their results and save them in files, so that other users can view the anticipated outcomes when reproducing the experiments. Packaging: To enforce reproducibility, we recommend that researchers keep all experiment-related files, including experiment scripts, output files, plots, and any other generated materials, as part of their artifacts. These files can be useful to other researchers who wish to reproduce plots and view the analysis without running any experiments. Sharing: Finally, we motivate researchers to share their artifacts online. Chameleon provides the Trovi platform to facilitate researchers in uploading their artifacts and sharing them publicly, allowing other users to reproduce their artifacts inside Chameleon infrastructure. Native integration of Jupyter Notebook Interface and Chameleon platforms provides better user experience as other users only need to run whole artifacts without worrying about hardware reservation and software requirements. Even if other users do not want to reproduce the published artifacts in the Chameleon platform, Chameleon 5
Chameleon Platform Provisioners
Machines / VMs A
OS Images
Node Types
SSH Keys
Jupyter Notebook Interface
B
Trovi C
...
...
Z
...
Launch on Chameleon
Exp 1 (node A, ...)
ARTIFACTS
Step 1 Setup instances
Spawning instances Attaching Floating IPs Saving instances metadata
Analysis
...
Parsing output
Step N
Analysis
...
Reserve instances
Download artifacts
... Plotting
Object storage Block storage
Save plot and others
...
Installing experiment requirements
Storage Disk snapshot
Exp N (node X, ...) Step 1
upload and share
Step N
backup
Figure 1: ”Breakable” experiment pattern. Our opinionated, extensible and reproducible workflows by utilizing Jupyter Interface in Chameleon platform. Users can add their own personalized experiment on top of existing experiments by modifying existing notebook files or adding more notebook files.
other that can be seen in 1. In a single notebook file, such as provisioning instances, we split the big milestone into several small steps such as reserving instances, spawning instances and attaching the IP addresses to the spawned instances. However, when running multiple different sets of experiments, it is better to separate them into their own notebook file to provide better context and readability to other users.
offers flexibility by allowing users to download the artifacts in a single compressed file. This compressed file will contain all the notebook files, experiment scripts, plots and generated files, if any. Chameleon also offers API and scripts that wrap OpenStack ecosystem utilities to perform OS snapshots and storing objects. This will speed up the experimental setup when reproducing the artifacts. Modification: Other users are encouraged and invited to create new artifacts by easily copy-pasting the setup, experiment and analysis code from their peers’ artifacts. This extensible pattern will be helpful, especially in the context of education as it allows lecturers to give the base example that can be modified and extended freely, enhancing the students’ knowledge and creativity.
In this work, we also focus on storage research reproducibility. As mentioned earlier in 3.1, Chameleon provides storage nodes to be able to run storage-related experiments by enabling performance isolation to provide better performance and noise-free outputs while running the experiment. Well known and battle-tested benchmarks in storage fields such as FIO [13] for storage device benchmarking and FileBench [9] for file system benchmarking are used in several artifacts in this paper. Our main artifacts in this work are required to be run inside storage nodes. However, we commonly encounter situations where the storage nodes are fully reserved due to their high demand. The alternative experiment method is as follows: instead of running on an empty disk that is offered inside storage nodes, we run the artifacts on top of existing storage in any available nodes. This existing storage is not empty and already includes the OS, leading to reproducible experiment with the trade-off of having no performance isolation.
The most prominent benefit of adopting and integrating this pattern inside the Jupyter Interface is that users can extend and control the artifacts in any granularity. Users can easily break their milestones into several cells in a single Jupyter notebook file (coarse granularity) or in several Jupyter notebook files (fine granularity). None of them are better than the other. We use the former if we want to keep the variables and output that can be reused later in other cells inside the same notebook. The latter is used if users prefer more isolation such as running whole different experiments because each notebook file natively does not share variables and output to each other. Both granularities complement each 6
4.1
Artifact 1: Simple FIO Benchmark
sec / op 0.2
In this arifact, we run the FIO Benchmark [13] on a RAM disk (mounted using tmpfs) and a SSD device. With the FIO Benchmark, the artifact is aiming to benchmark and compare the read throughput of the RAM disk and SSD. This artifact can be reproduced on virtually any machines on Chameleon Cloud as it does not require any special hardware.
100000
0.1
50000
0.0 copyfiles makedirs remodirs
netsfs
createfiles
Bandwidth (MB/s)
netsfs
createfiles
# writes 40000
750 500
20000
250
5000
0 copyfiles makedirs remodirs
4000
150000
3000
100000
netsfs
createfiles
0 copyfiles makedirs remodirs
netsfs
createfiles
# ops
50000
2000
0
1000 0
0 copyfiles makedirs remodirs
# reads
1000
6000
# ops / sec
150000
copyfiles
makedirs
remodirs
netsfs
createfiles
Figure 3: Simple file system benchmark.
ssd_rd
Generated plot from analysis notebook file showing performance of ext4 file system in different types of workloads.
ramdisk_rd
Figure 2: Simple FIO Benchmark.
Generated plot from analysis notebook file comparing the read throughput among single SSD and RAM disk.
execute the file system workloads. After running the experiment code, FileBench will produce the output to standard output stream containing important variables such as number of: (1) seconds taken per operation, (2) operations per second, (3) operations, (4) reads, and (5) writes. The output of this experiment is shown in Figure 3 that is generated directly inside the analysis notebook file.
Following the breakable experiment patter, the artifact is separated into three notebooks: (1) environment setup (e.g. provision servers, install dependencies, etc.), (2) run experiment based on a FIO benchmark configuration file which can be easily modified to align with users’ interests (i.e. specifying the block size for each read operation, the maximum read operations in queue, etc.), (3) analysis results by fetching and parsing the bandwidth results a ynd plot them into a bar chart shown in Figure 2. The artifact can be accessed at [27].
4.3
Artifact 3: FIO Benchmark
12000
Artifact 2: Simple File System Benchmark
10000 Bandwidth (MB/s)
4.2
In this simple experiment, we run multiple FileBench [9] workloads for benchmarking default file system in Linux, ext4. This experiment can be reproduced in any nodes in Chameleon or even in personal computer or laptop. This experiment code simulates file system operations such as copying files, creating files, creating and removing directories, and performing heavy network file system operations. This artifact is accesible at [26]. The artifact also breaks into several parts such as (1) setup experiment container, (2) run experiment code, and (3) analyze experiment outputs and generate plots. The overall execution time of this experiment should be done in less than 30 minutes including provisioning of bare-metal instances to
8000 6000 4000 2000 0
raid_0_rd
Figure 4: FIO Benchmark.
ssd_rd
ramdisk_rd
Generated plot from analysis notebook file comparing the read throughput among RAID-0 SSD array, single SSD, and RAM disk.
7
This artifact further extends the Simple FIO Benchmark to benchmark the read throughput on a RAID-0 SSD array in addition to the RAM disk and SSD. As this artifact requires multiple SSDs, the artifact can only run on the storage node in the Chameleon Cloud. An example analysis figure are shown in Figure 4. The artifact can be accessed at [12].
4.4
code, the generated plot should be similar to Figure 5. The variables that we used are originated from FileBench software as defined in Section 4.2. This artifact is also published in Chameleon Trovi platform and can be accessed at [10].
4.5
Artifact 4: File Systems Benchmark sec / op 200000
0.10
150000 100000
0.05 0.00
50000 ext4
xfs
0
btrfs
ext4
# reads 40000
15000
30000
10000
20000
5000
10000 ext4
xfs
xfs
btrfs
# writes
20000
0
This simple artifact is intended to demonstrate capability of running RDMA benchmark on a local Chameleon node. RDMA is a mechanism to establish access to memory of two or more computers. InfiniBand is a popular implementation of RDMA that provides fast communication with very high throughput and very low latency and widely used in high performance computing (HPC) clusters. Fortunately, Chameleon provides support for utilizing InfiniBand in their platform by providing users with Mellanox [21] network device. Thus, specifically in this artifact, we reserve a node which has a Mellanox network device attached to it.
# ops / sec
0.15
0
btrfs
Latency test, 1000x iter with warm p
t_min[ sec]
ext4
xfs
1000
1000
500
500
500
0
btrfs
0
2
0.5
6
8
btrfs
0
0
2
4
6
0
8
Message Size 1e6
1000
1000
500
500
500
0
2
4
6
0
8
Message Size 1e6
0
2
4
6
Message Si%e
t_stdev[ sec]
0
t_typical[ sec]
2
4
6
8
Message Size 1e6
99% percentile[ sec]
99.9% percentile[ sec]
1000
0 xfs
4
Message Size 1e6
t_avg[ sec]
1.0
ext4
t_max[ sec]
1000
# ops
1e7
0.0
Artifact 5: Remote Direct Memory Access (RDMA)
0
8 1e6
0
2
4
6
8
Message Size 1e6
0.6 0.4
Figure 5: File systems benchmark. Generated plot showing
0.2
performance of ext4, btrfs, and XFS file system in different types of workloads.
0.0
This is an enhanced and extended version of simple file system benchmark. Instead of running only on ext4 file system, we will try to run the file system workloads on other well-known Linux file systems such as Btrfs and XFS. This artifacts requires a storage node in Chameleon to utilize its additional empty SSD. The steps in this artifact are quite similar to Section 4.2 with additional single step to prepare partitions to contain three file systems mentioned above. To create partition, we will format the empty disk and disable RAID partition, if any, because we do not need RAID features in this artifact. Then, we will create the three file systems inside that empty disk with the same configuration and enable minimum alignment as specified by the disk topology information. After completing the execution of the experiments
0
2
4
Message Size
6
8
1e6
Figure 6: RDMA latency benchmark. Produced plot from this artifact after running latency benchmark on a local InfiniBand node.
The Mellanox driver itself comes with several numbers of benchmarking tools. In this artifact, users can measure the bandwidth of local InfiniBand network performance using a program called ib send bw. We also provide a script to run another benchmark to calculate latency using ib send lat. After completing this experiment code, users should see generated plots, one of them looking similar to Figure 6. This artifact is publicly accessible at [17] and can hopefully be used as a stepping stone for other researchers to utilize and 8
5
familiarize themselves with InfiniBand communication standards. One possible expansion of this artifact is the ability to run benchmarks in multiple InfiniBand nodes simultaneously to mimic production HPC clusters.
4.6
Over the past two years, we have applied the methodology described above in a variety of scenarios beyond the S TOR R EP. These include classrooms with more than 20 students, the Chameleon Summer Repro 2020 event with 7 researchers, and the FAST’23 Birds-of-a-Feather event [8] (with an expected registration of over 15 attendees) from various academic levels (i.e. undergraduate, master’s, and doctoral students). Alongside our use of the S TOR R EP, we provide the following lessons for leveraging S TOR R EP experiment pattern to package users’ artifacts. Self-contain: The artifact should be self-contained because we cannot assume that all users are familiar with the technical details of the original paper. Therefore, it is crucial to include all the necessary background, methodology, and preconditions (e.g. access to resources, data sets, or recommended prior knowledge) in the artifact itself, rather than relying solely on the code. Managed Expectation: Users will need to have a clear understanding of what to expect when reproducing this artifact. Providing the answers to the following questions int the artifact can help the users manage their expectations:
Artifact 6: F LASH N ET, an ML for Storage Testbed
The present study delves into the ongoing research called F LASH N ET, which employs a neural network (NN) model to perform per-I/O admission control. The primary objective of this research is to provide a benchmark for stimulating further investigation in the area of ML-based per-I/O latency prediction in storage research. The study is an extended version of ION ET [33]. To train the model, the authors utilized Tencent I/O traces [29], which were replayed on their NVMe devices. Furthermore these traces is already preprocessed prior to their use for training the neural network model. To carry out this research, the authors employed TensorFlow [30], a neural network framework developed by Google. Though it is recommended to use GPU nodes for faster computation, the neural network model used in this study is relatively small and can be trained on CPU nodes. Therefore, users have the freedom to select their preferred node by modifying the experiment script. However, to ensure reproducibility, we opted to use CPU nodes since they are capable of executing the experiment adequately. The experiment is straightforward similar to usual neural network training pipeline and available at [11]. First, we will label the data to denote whether the specific I/O is ”slow” or ”fast” according to their trace. Then, we will extract feature from preprocessed data. Next, we will train the model using TensorFlow framework. Last, we will evaluate the model performance with the test data. After all steps are done, users should see the generated plot similar to Figure 7.
- How long is the artifact setup phase (i.e. hardware or software configuration) expected to take? - Does the artifact setup typically execute smoothly or are there some expected troubles? If so, are there ways to avoid them? - How long is the execution of the artifact itself going to take? If the time of the artifact run is variable on the inputs, consider at least informing the user at execution time of the expected time to completion for the workflow step. - Are there any common error messages that users might run into? - Can you provide a visual representation of the anticipated outcome?
FNR on Va ious Tencent I/O Dataset
FNR
0.15
Building idempotent workflow: It is common to re-run code cells in a notebook, and making sure that the cells are idempotent, if possible, can help reduce user errors and confusion for both the authors and users of the artifact. To this end, we provide a list of recommended procedures to maintain an idempotent workflow:
0.10 0.05 0.00
37
.5 ps
o d_i ran
cu
0
r_5
e t.p
_ ost k.m
r_5
c
pe ut.
Lessons Learned
st_ mo
.
0k
thp
e_ siz
t.p
cu
2 t.2
2 .r
k _50
er
.49
_40
60 w_
.r
k _50
er
t.p
cu
0
.21 _35
65 w_
t.p cu
er
1 .r
k _50
9 .37
_25 75 w_
- Avoid re-creating device reservations or re-initializing instances on the servers. - Avoid reinstalling packages unnecessarily. Most package managers should handle this automatically by default. - Avoid leaving any state (e.g. global environment variables, file modifications, etc.) that may affect a cell’s subsequent execution. For example, if you run a script on the server in cell A, and then modify it and re-run it
F LASH N ET performance. False negative rate (FNR) is a metric that is used in F LASH N ET experiment for evaluating performance of the neural network model. Figure 7:
9
in cell B, any user who runs cell A again will be executing the script with its modified contents, which can be confusing.
can be difficult to realize on the hardware side due to the heterogeneous nature of hardware used by users seeking to reproduce experiments. Fortunately, we can enforce hermeticity and idempotence on the software side, at the operating system level. In this regard, large-scale projects such as containerized builds using Docker [6], virtual machines, and pure hermetic operating systems like NixOS [20] can help solve this issue in the future. NixOS project consists of Nix [18], a purely functional package manager, and Nix functional programming language [19]. The Nix package manager, inspired by functional programming paradigm, produces deterministic and hermetic builds by taking side effects into an account. This package manager relies heavily on a Nix functional programming language offers low-level OS and users configurations that can be transported anywhere, giving users a precise replica of the experiment software and OS used by the authors. Users can also easily review the source code, simplifying the process from their perspective. In summary, S TOR R EP is a significant milestone in the advancement of reproducibility research, and we anticipate that it will stimulate further research in this area. We believe that the extensibility and flexibility of our ”breakable pattern” and the enforceable hermeticity and idempotence of the software side offer exciting prospects for researchers seeking to ensure the reliability, reproducibility, and transparency of their works. We are excited to continue contributing to the advancement of reproducible research in the future.
Handling errors gracefully: Researchers usually expect some errors while building and running the experiments. To handle them, they need to utilize programming language features such as exception and error number to mitigate errors from libraries or softwares that they used. Furthermore, this error handling will also provide more contexts for users to help them independently debug their experiment execution. Avoid noise: As the saying goes, “Brevity is the soul of wit.” If artifacts’ authors include too much information, users may not see the big picture of the artifact. Therefore, we should avoid including information that is available elsewhere, in which case providing a link is best. Similarly, lengthy explanations of things that are not immediately relevant to the purpose of the artifact should also be avoided. Structuring the information in a way that highlights high-level concepts while allowing users to dig deeper for details is a bonus. Last, we also encourage researchers to sanitize their script execution output because sometimes the output is too verbose making it harder to pinpoint the problems in their artifacts.
6
Conclusions and Discussions
We are pleased to introduce S TOR R EP, a simple and practical approach to reproducibility research that addresses the pressing need for reliable and transparent storage-related experiment artifacts. As far as we are aware, S TOR R EP is the first reproducibility research that accommodates the Chameleon platform, empowering researchers to construct reproducible storage-related experiment artifacts. Our research showcases the extensible nature of reproducible research without the need for cumbersome configuration of experiment containers, leveraging the Chameleon infrastructure. Additionally, we introduce a ”breakable pattern” that is possible thanks to the composability and flexibility of the Jupyter Notebook Interface. We are confident that this work will inspire further research in reproducibility across many fields of computer science. With regard to extensibility and flexibility, the ”breakable pattern” we present in S TOR R EP is generalizable and adaptable to various types of user experiments. While our work is focused on Jupyter Notebook Interface and Chameleon platform, we acknowledge the potential for vendor lock-in. Thus, we hope to see more examples and guidelines that utilize this pattern in any platform or interface. On the subject of reproducibility, S TOR R EP provides valuable advice and mechanisms for users to enhance the reproducibility of their research. However, creating pure idempotent and hermetic research artifacts is challenging and requires specific skills and platform support. These characteristics need to be enforced at the lowest possible level, but this
References [1] Amazon S3. https://aws.amazon.com/s3/. [2] Chameleon Cloud. https://chameleoncloud.org. [3] Chameleon Experiment Patterns: Making Complex Experiments Easy. https://www.chameleoncloud.org/blog/2022/09/26/ experiment-patterns-making-complex-experiments-easy/. [4] Chameleon Trovi Platform. https://chameleoncloud.org/experiment/share/. [5] Data Storage, Management, and Sharing on Chameleon. https://www.chameleoncloud.org/blog/2022/10/26/ data-storage-management-and-sharing-on-chameleon/. [6] Docker: Accelerated, Containerized Application Development. https://www.docker.com/. [7] Enabling a new future for cloud computing. https: //nsf.gov/news/news_summ.jsp?cntn_id=132377. [8] FAST ’23 Birds-of-a-Feather. https: //www.usenix.org/conference/fast23/activities. [9] File system and storage benchmark. https://github.com/filebench/filebench. [10] Filesystems Benchmark Artifact. https://chameleoncloud.org/experiment/share/ 552436c8-b628-4a77-911d-f21426645e4b.
10
[11] FlashNet Benchmark Artifact. https://chameleoncloud.org/experiment/share/ add27070-9ea5-4198-8ea5-a8ed412fdb7c. [12] Flexible I/O Benchmark Artifact. https://chameleoncloud.org/experiment/share/ 5fef1e0e-b585-4af9-b3ea-a4e53455f3d2. [13] Flexible I/O Tester. https://github.com/axboe/fio. [14] Google cloud storage. https://cloud.google.com/storage. [15] Intel® Optane™ Persistent Memory. https://www.intel.com/content/www/us/en/ products/docs/memory-storage/ optane-persistent-memory/overview.html. [16] LinnOS Experiment Artifact. https://chameleoncloud.org/blog/2020/11/01/ chameleon-and-reproducibility-linnos-case-study/. [17] Local RDMA Benchmark Artifact. https://chameleoncloud.org/experiment/share/ 631a39c8-befd-4911-880f-6f359e3982cb. [18] Nix package manager. https://github.com/NixOS/nix. [19] Nix programming language. https://nixos.org/manual/ nix/stable/language/index.html. [20] NixOS. https://nixos.org/. [21] Nvidia Mellanox. https://www.nvidia.com/en-us/ networking/management-software/. [22] OpenStack Manila. https://github.com/openstack/manila. [23] OpenStack: Open Source Cloud Computing Infrastructure. https://www.openstack.org/. [24] Project Jupyter. https://jupyter.org/. [25] Python CHI. https://github.com/ChameleonCloud/python-chi. [26] Simple Filesystem Benchmark Artifact. https://chameleoncloud.org/experiment/share/ b9a7f86e-b2ef-45e5-8be5-73d41047ceca. [27] Simple Flexible I/O Benchmark Artifact. https://chameleoncloud.org/experiment/share/ 3c1f55e3-6915-4795-aba8-2b7791c7ac3e. [28] SmartSSD. https: //www.xilinx.com/applications/data-center/ computational-storage/smartssd.html. [29] SNIA Block I/O Traces. http://iotta.snia.org/traces/block-io. [30] TensorFlow. https://www.tensorflow.org/. [31] Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, Tushar Krishna, Somayeh Sardashti, et al. The gem5 simulator. ACM SIGARCH computer architecture news, 39(2):1–7, 2011. [32] John Colgrove, John D. Davis, John Hayes, Ethan L. Miller, Cary Sandvig, Russell Sears, Ari Tamches, Neil Vachharajani, and Feng Wang. Purity: Building fast, highly-available
11
enterprise flash storage from commodity components. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD), 2015. [33] Daniar H. Kurniawan and Levent Toksoz and Mingzhe Hao and Henry Hoffmann and Tim Emami and Sandeep Madireddy and Anirudh Badam and Robert B. Ross and Haryadi S. Gunawi. Towards an Open Machine Learning Training Ground for I/O Performance Prediction. University of Chicago Technical Report, 2021. [34] Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, et al. The design and operation of cloudlab. In USENIX Annual Technical Conference, pages 1–14, 2019. [35] Nima Elyasi, Mohammad Arjomand, Anand Sivasubramaniam, Mahmut T Kandemir, Chita R Das, and Myoungsoo Jung. Exploiting intra-request slack to improve ssd performance. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, pages 375–388, 2017. [36] Congming Gao, Min Ye, Qiao Li, Chun Jason Xue, Youtao Zhang, Liang Shi, and Jun Yang. Constructing large, durable and fast ssd system via reprogramming 3d tlc flash memory. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 493–505, 2019. [37] Donghyun Gouk, Miryeong Kwon, Jie Zhang, Sungjoon Koh, Wonil Choi, Nam Sung Kim, Mahmut Kandemir, and Myoungsoo Jung. Amber: Enabling precise full-system simulation with detailed modeling of all ssd resources. In 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 469–481. IEEE, 2018. [38] Mingzhe Hao, Levent Toksoz, Nanqinqin Li, Edward Edberg Halim, Henry Hoffmann, and Haryadi S. Gunawi. LinnOS: Predictability on Unpredictable Flash Storage with a Light Neural Network. In Proceedings of the 14th Symposium on Operating Systems Design and Implementation (OSDI), 2020. [39] Jun He, Sudarsun Kannan, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. The unwritten contract of solid state drives. In Proceedings of the twelfth European conference on computer systems, pages 127–144, 2017. [40] Yang Hu, Hong Jiang, Dan Feng, Lei Tian, Hao Luo, and Shuping Zhang. Performance impact and interplay of ssd parallelism through advanced commands, allocation strategy and data granularity. In Proceedings of the international conference on Supercomputing, pages 96–107, 2011. [41] Myoungsoo Jung, Ellis Herbert Wilson, David Donofrio, John Shalf, and Mahmut Taylan Kandemir. Nandflashsim: Intrinsic latency variation aware nand flash memory system modeling and simulation at microarchitecture level. In 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST), pages 1–12. IEEE, 2012. [42] Myoungsoo Jung, Jie Zhang, Ahmed Abulila, Miryeong Kwon, Narges Shahidi, John Shalf, Nam Sung Kim, and Mahmut Kandemir. Simplessd: Modeling solid state drives for holistic system simulation. IEEE Computer Architecture Letters, 17(1):37–41, 2017.
[54] Heiner Litz, Javier Gonzalez, Ana Klimovic, and Christos Kozyrakis. Rail: Predictable, low tail latency for nvme flash. In ACM Trans. Storage Vol 18, No 1, 2022.
[43] Sudarsun Kannan, Yujie Ren, and Abhishek Bhattacharjee. Klocs: Kernel-level object contexts for heterogeneous memory systems. In Proceedings of the 26th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2021. [44] Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. Lessons learned from the chameleon testbed. In Proceedings of the 2020 USENIX Annual Technical Conference (ATC), 2020. [45] Jaeho Kim, Kwanghyun Lim, Youngdon Jung, Sungjin Lee, Changwoo Min, and Sam H. Noh. Alleviating garbage collection interference through spatial separation in all flash arrays. In Proceedings of the 2019 USENIX Annual Technical Conference (ATC), 2019. [46] Wook-Hee Kim, R. Madhava Krishnan, Xinwei Fu, Sanidhya Kashyap, and Changwoo Min. Pactree: A high performance persistent range index using pac guidelines. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP), 2021. [47] Youngjae Kim, Sarp Oral, Galen M. Shipman, Junghee Lee, David A. Dillow, and Feiyi Wang. Harmonia: A globally coordinated garbage collector for arrays of solid-state drives. In Proceedings of the 27th IEEE Symposium on Massive Storage Systems and Technologies (MSST), 2011. [48] Youngjae Kim, Brendan Tauras, Aayush Gupta, and Bhuvan Urgaonkar. Flashsim: A simulator for nand flash-based solid-state drives. In 2009 First International Conference on Advances in System Simulation, pages 125–131. IEEE, 2009. [49] R. Madhava Krishnan, Jaeho Kim, Ajit Mathew, Xinwei Fu, Anthony Demeri, Changwoo Min, and Sudarsun Kannan. Durable transactional memory can scale with timestone. ASPLOS20, 2020. [50] R. Madhava Krishnan, Wook-Hee Kim, Xinwei Fu, Sumit Kumar Monga, Hee Won Lee, Minsung Jang, Ajit Mathew, and Changwoo Min. TIPS: Making volatile index structures persistent with DRAM-NVMM tiering. In Proceedings of the 2021 USENIX Annual Technical Conference (ATC), 2021. [51] Huaicheng Li, Mingzhe Hao, Michael Hao Tong, Swaminathan Sundararaman, Matias Bjørling, and Haryadi S Gunawi. The CASE of FEMU: Cheap, accurate, scalable and extensible flash emulator. In Proceedings of the 16th USENIX Symposium on File and Storage Technologies (FAST), pages 83–90, 2018. [52] Huaicheng Li, Martin L. Putra, Ronald Shi, Xing Lin, Gregory R. Ganger, and Haryadi S. Gunawi. Loda: A host/device co-design for strong predictability contract on modern flash storage. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP), 2021. [53] Qiao Li, Liang Shi, Chun Jason Xue, Kaijie Wu, Cheng Ji, Qingfeng Zhuge, and Edwin H-M Sha. Access characteristic guided read and write cost regulation for performance improvement on flash memory. In 14th USENIX Conference on File and Storage Technologies (FAST 16), pages 125–132, 2016.
[55] Chun-Yi Liu, Jagadish B Kotra, Myoungsoo Jung, Mahmut T Kandemir, and Chita R Das. Soml read: Rethinking the read operation granularity of 3d nand ssds. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 955–969, 2019. [56] Ji Yong Shin, Mahesh Balakrishnan, Tudor Marian, and Hakim Weatherspoon. Gecko: Contention-Oblivious disk arrays for cloud storage. In Proceedings of the 11th USENIX Symposium on File and Storage Technologies (FAST), 2013. [57] Dimitris Skourtis, Dimitris Achlioptas, Noah Watkins, Carlos Maltzahn, and Scott Brandt. Flash on rails: Consistent flash performance through redundancy. In Proceedings of the 2014 USENIX Annual Technical Conference (ATC), 2014. [58] Arash Tavakkol, Juan Gómez-Luna, Mohammad Sadrosadati, Saugata Ghose, and Onur Mutlu. Mqsim: A framework for enabling realistic studies of modern multi-queue {SSD} devices. In 16th {USENIX} Conference on File and Storage Technologies ({FAST} 18), pages 49–66, 2018. [59] Shiqin Yan, Huaicheng Li, Mingzhe Hao, Michael Hao Tong, Swaminathan Sundararaman, Andrew A Chien, and Haryadi S. Gunawi. Tiny-tail flash: Near-perfect elimination of garbage collection tail latencies in nand ssds. ACM Transactions on Storage (TOS), 13(3):1–26, 2017. [60] Yujie Ren and Changwoo Min and Sudarsun Kannan. CrossFS: A Cross-layered Direct-Access File System. In Proceedings of the 14th Symposium on Operating Systems Design and Implementation (OSDI), 2020. [61] Jian Zhang, Yujie Ren, and Sudarsun Kannan. FusionFS: Fusing I/O operations using CISCOps in firmware file systems. In Proceedings of the 20th USENIX Symposium on File and Storage Technologies (FAST), 2022.
12