Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems Haochen Pan*† , Ryan Chard† , Song Young Oh* , Maxime Gonthier*† , Valérie Hayot-Sasson‡ , Geoffrey Lentner§ , Joe Bottigliero* , Rachana Ananthakrishnan*† , Kyle Chard*† , Ian Foster†* *
University of Chicago, Chicago, IL, USA Argonne National Laboratory, Lemont, IL, USA ‡ École de technologie supérieure, Montréal, QC, Canada § Purdue University, West Lafayette, IN, USA
arXiv:2604.10295v1 [cs.DC] 11 Apr 2026
†
Abstract—Modern HPC file systems can contain billions of files and hundreds of petabytes of data, making even simple questions increasingly intractable to answer. Traditional file system utilities such as find and du fail to scale to these sizes. While external indexing tools like GUFI and Brindexer improve query performance, they remain batch-oriented and unsuitable for heterogeneous, rapidly evolving environments. We present Icicle, a scalable framework for continuous file system metadata indexing and monitoring. Icicle maintains a unified, up-to-date, and queryable view of file system state while supporting both periodic snapshot-based ingestion for bulk metadata updates and event-based ingestion for real-time synchronization from production systems such as Lustre and IBM Storage Scale. Built on Apache Kafka and Apache Flink, Icicle provides highthroughput, fault-tolerant, and horizontally scalable ingestion of metadata events into two complementary search indexes, enabling both individual file discovery and aggregate summary statistics by user, group, and directory. This architecture enables efficient support for both coarse-grained administrative queries and interactive analytics over billions of objects. Our experimental evaluation on production-scale HPC datasets demonstrates orderof-magnitude throughput improvements over existing monitoring and indexing approaches, with tunable options for balancing consistency, latency, and metadata freshness. Index Terms—HPC storage, HPC file system, metadata indexing, real-time monitoring, stream processing
I. I NTRODUCTION High performance computing (HPC) file systems can store tens of billions of files and hundreds of petabytes of data, driven by large-scale scientific experiments, extreme-scale simulations, and AI-driven workflows. For example, Oak Ridge National Laboratory (ORNL) hosts a 700 PB Lustre file system, Argonne National Laboratory has two 100 PB Lustre file systems, and the National Energy Research Scientific Computing Center (NERSC) maintains a 650 PB High Performance Storage System (HPSS) tape archive [1]–[4]. Massive datasets produced routinely by scientific applications must be reliably stored and managed across tiered and heterogeneous storage systems, each with independent namespaces and quotas, and distinct lifecycle policies, interfaces, and representations. While this hierarchical design improves performance and optimizes data placement, it also fragments
metadata across multiple systems, making user and administrative tasks increasingly complex and time-consuming. Unfortunately, traditional utility tools such as find and du fail to scale to modern HPC file systems. Already in 2010, ORNL reported that a single find or du traversal on their 10 PB Spider file system could take more than 48 hours to complete [5]. As file systems continue to grow beyond petabyte scales, administrators have turned to external metadata indexing systems to improve observability and conduct data lifecycle management. Modern solutions such as Grand Unified File Indexer (GUFI) [6] and Brindexer [7] construct external, database-backed indexes of file system metadata, enabling faster query performance than in-place scans. However, these systems remain fundamentally batchoriented: they rely on periodic scans to update their databases and require parallel scripts to perform queries and analysis. As a result, obtaining even basic statistics such as per-user summaries or top-k usage reports remains slow, resourceintensive, and operationally cumbersome. Modern HPC file systems such as Lustre and IBM Storage Scale (GPFS) [8] provide real-time metadata event feeds, yet few indexing systems exploit these capabilities to enable both real-time event monitoring and cost-aware, periodic indexing. At the same time, metadata consumers have widely varying freshness requirements: anomaly detection may need nearreal-time updates, while usage trend analysis may tolerate daily snapshots. Existing approaches, while effective for file counting or hierarchical aggregation, are limited in their ability to support semantic or percentile-based queries. Examples include identifying which users created the most small files last week or calculating the 99th percentile of directory sizes per group. Executing such queries across a large collection of databases remains inefficient with current monitoring systems. Table I lists representative queries that motivate these needs, spanning individual file discovery, anomaly detection, and aggregate usage analysis. To address these emerging needs, we present Icicle, a scalable and extensible framework for continuous file system metadata indexing and monitoring across HPC file systems. Icicle aims to maintain an external, queryable, and unified
TABLE I: Representative queries for file system metadata management. Role General General General General General General General General Admin Admin Admin Admin Admin
Example Query Where is a specific file or directory? Which files or directories have world-writable permissions? Which files have not been accessed in over 12 months? Which large files have low access frequency? Which files are replicated across directories? Which directories contain more than 100,000 files? What is the total storage consumption per project? Which projects are approaching their quota limits? Which files are owned by deleted users? Which files exceed their retention period? Which users have the most small files? What is the per-user file count and storage usage? What is the 99th percentile directory size per project?
view of disparate file systems to support semantically rich queries. Icicle operates in two complementary modes: snapshot mode, which ingests periodic metadata exports or GUFIstyle database dumps; and update mode, which continuously consumes and reduces real-time metadata event streams such as Lustre changelogs and GPFS mmwatch [9] events. By integrating both modes using a modern stream-processing architecture built on Apache Kafka [10] and Apache Flink [11], Icicle provides scalable, fault-tolerant, and idempotent processing of metadata events. Its open architecture can support any POSIX-compliant environment that exposes metadata event streams or snapshots, and it can directly feed downstream policy engines, dashboards, and workflow systems for realtime automation. This paper makes three contributions: 1) The design of a unified and scalable metadata indexing framework. We introduce Icicle, a scalable metadata indexing and monitoring framework that unifies heterogeneous sources of file system state, including snapshots, GUFI-style databases, and real-time changelogs from Lustre and GPFS. Icicle is open source and available at https://github.com/globus-labs/icicle/. 2) High-throughput and idempotent stream processing. Icicle comprises three core components: (i) a highthroughput ingestion layer that collects file system metadata from existing tools and services; (ii) a distributed aggregation and processing layer built on Kafka and Flink that performs stream-based event reduction and state computation, to transform events into search-engine ingestion requests; and (iii) a web-based interface that enforces data visibility for administrators and users to perform queries and custom analytics. 3) Evaluation of scalability and throughput. We conduct a comprehensive evaluation of Icicle’s snapshot pipeline and event monitor on Lustre and GPFS file systems. In snapshot mode, we show that Icicle scales with both dataset size and available CPU resources, enabling efficient processing of multi-petabyte metadata exports. In update mode, Icicle’s Lustre monitor achieves 4– 100× higher throughput than the state-of-the-art FSMonitor [12], and both the Lustre and GPFS monitors scale
Query Expression name LIKE "*pattern*" mode = 777 atime < now() - 1y size > 100GB AND atime < now() - 6m GROUP BY checksum HAVING count > 1 file_count > 100000 SUM(size) GROUP BY project usage / quota > 0.9 uid NOT IN active_users mtime < retention_date COUNT(file_size < 1MB) DESC SUM(size), COUNT(*) GROUP BY uid PERCENTILE(size, 0.99) GROUP BY project
Granularity Individual Individual Individual Individual Individual Aggregate Aggregate Aggregate Individual Individual Aggregate Aggregate Aggregate
linearly with increasing changelog volumes. Icicle’s open and extensible design integrates seamlessly with existing HPC tools, establishing a foundation for federated, continuously updated, and queryable views of the file systems. Ultimately, Icicle aims to make data more discoverable, auditable, and manageable regardless of physical location, and advances the next generation of cyberinfrastructure. II. R EQUIREMENTS Table I presents representative queries performed by users and administrators across storage environments. These queries span multiple dimensions of file system management, from finding files of interest and identifying anomalies or security concerns to supporting data lifecycle decisions and resource allocation monitoring. Such queries define the functional requirements for Icicle’s user interfaces. From these, we derive the following technical requirements for a monitoring system. Based on our analysis of operational needs and the limitations of existing tools, we identify the following key requirements for a scalable metadata monitoring framework: 1) Snapshot ingestion. The system must efficiently ingest metadata exports, including GUFI-style databases and file system snapshots, to provide a consistent baseline view of file system state. Snapshot ingestion enables bulk updates and supports environments where real-time monitoring is unavailable or impractical. 2) Real-time monitoring. The system must continuously process metadata event streams from modern HPC file systems, such as Lustre changelogs and GPFS mmwatch events. Real-time processing enables second-level freshness for time-sensitive operations such as anomaly detection and quota enforcement. 3) Configurability. The system must support flexible tradeoffs between consistency, latency, and resource consumption. Anomaly detection may require near-instantaneous updates, while usage trend analysis may tolerate daily or weekly snapshots. The system should accommodate this spectrum of requirements without requiring separate infrastructure for each use case. 4) Heterogeneous source support. The system must support diverse storage backends without requiring signif-
HPC File Systems (e.g., Lustre, GPFS) Metadata Snapshots (e.g., TSV, SQLite Database)
Ingestion Layer
Preprocessing Primary Pipeline
Event Ingestion
Aggregation and Metadata Processing Processing Layer Update Notification
Counting Pipeline Aggregate Pipeline Snapshot Pipeline
File System Events (e.g., changelog, mmwatch)
Aggregate Index
Primary Index
Event Monitor Web Interface Pipeline data path
Graphical Query Builder
Interactive Visualization Interface
Monitor data path Web UI requests and response
TABLE II: Primary index schema for file system objects. Field
Description
path
Fully resolved absolute path; primary key uniquely identifying each object. Object type: regular file (f) or symbolic link (l). POSIX permission and mode bits (e.g., -rw-r--r--); enables permission-based filtering. Numeric user ID of the owner; enables per-user queries. Numeric group ID; supports group-level access and allocation analysis. Object size in bytes; used for storage consumption analysis. Last access time; identifies cold data and archive candidates. Last metadata change time; supports audit and compliance queries. Last content modification time; enables age-based lifecycle analysis. Containing fileset (GPFS only); supports fileset-scoped queries.
type mode uid gid size atime ctime mtime fileset
HPC User/Administrator
Fig. 1: Icicle architecture overview. Icicle comprises a snapshot pipeline, an event monitor, and a web interface. The snapshot pipeline ingests metadata snapshots to populate a per-object primary index and an aggregate index with per-user, group, and directory summaries. The event monitor statefully processes file system events to maintain the primary index. The web interface provides unified access to both indexes.
icant customization. Storing metadata using a common and open index abstraction enables unified querying regardless of the underlying data source. 5) Multi-granular semantic query support. The system must enable analysis at multiple granularities, ranging from individual files (e.g., locating specific files) to aggregated views reflecting users, groups, or projects. Beyond simple file counting and hierarchical aggregation, the system must support rich semantic queries, including percentile computations, temporal filtering, and crossattribute correlations. III. I CICLE Icicle is a scalable file system metadata indexing and monitoring framework. We first describe Icicle’s unified metadata model abstraction. We then present its ingestion and distributed aggregation layers, which maintain indexes through both snapshot-based bulk ingestion and event-based real-time synchronization. Finally, we present its web interface, which exposes access to metadata indexes through a unified frontend. Fig. 1 shows an overview of the system architecture. A. Metadata Indexes Icicle organizes file system metadata into two complementary indexes that together provide a comprehensive, queryable view of storage state. This dual-index architecture reflects a fundamental trade-off in metadata management: fine-grained, per-object records enable precise queries but incur high storage and query costs, while pre-computed summaries sacrifice granularity for dramatically improved query performance. Maintaining both indexes allows Icicle to efficiently support the full spectrum of administrative queries outlined in Table I.
The primary index serves as the authoritative record of individual file system objects. Each entry corresponds to a single file or link and captures the core POSIX metadata attributes (see Table II). These records support file-level queries that require object-specific filtering, such as identifying all files owned by a particular user within a directory, locating worldwritable files for security auditing, or identifying large files that have not been accessed within a specified period. The aggregate index provides pre-computed summary statistics for users, groups, and directories. Each aggregate record captures statistical summary information for all objects within its scope (see Table III). This design supports analytical queries, such as identifying the top storage consumers by project, computing per-user file count distributions, or determining the 99th percentile directory size, without requiring expensive scans over millions or billions of primary records. Icicle’s metadata model targets POSIX-compliant file systems. Extending support to non-POSIX environments, such as cloud object stores with ACL or IAM-based access control, would require schema changes (e.g., adding IAM principals, bucket identifiers, and storage-class fields) and adapters for cloud-native event streams such as S3 Event Notifications, Google Cloud Pub/Sub, or Azure Event Grid. Icicle’s modular architecture accommodates such extensions: new metadata fields require changes only to the preprocessing script and primary pipeline mapper, and new event sources require an additional ingestion-layer adapter. B. Data Ingestion, Processing, and Aggregation To produce and maintain these indexes, we employ two processing approaches. The snapshot pipeline runs periodically to convert static file system metadata snapshots (e.g., obtained with GUFI indexer or IBM Storage Scale’s mmapplypolicy) into the two indexes. The event monitor subscribes to file system event mechanisms provided by the underlying storage infrastructure to receive continuous updates about file creations, modifications, and deletions. These events are statefully processed into record ingestion and deletion requests that are applied to the primary index.
TABLE III: Aggregate index schema for precomputed summary statistics. Field
Description
principal
Aggregation key identifying the scope of the summary: numeric user ID, numeric group ID, or directory path. Total number of files within the aggregation scope; enables rapid object count queries without scanning the primary index. Distributional statistics over file sizes; supports capacity planning, anomaly detection, and usage reporting. Distributional statistics over access times, enabling identification of active versus dormant data. Distributional statistics over metadata change times, supporting audit and compliance queries. Distributional statistics over modification times, enabling age-based lifecycle analysis.
file_count size_{*} atime_{*} ctime_{*} mtime_{*}
Note: {*} denotes the set {min, p25, p50(median), p75, max, mean}; for size, the set additionally includes total.
Icicle’s architecture is designed for extensibility and scale. Both the snapshot pipeline and the event monitor can be applied to any storage system that supports snapshots or exposes an event stream. To support large deployments, the Icicle monitor scales horizontally by aligning monitor instances with the file system’s metadata partitioning. It deploys one monitor per Lustre MDT or GPFS mmwatch topic so that processing parallelism matches the underlying storage architecture. 1) Snapshot pipeline: The snapshot pipeline provides a scalable workflow for converting HPC metadata snapshots into searchable index structures. Existing metadata indexing tools such as Brindexer, GUFI, and IBM’s mmapplypolicy export file system metadata as per-directory SQLite databases or plain-text listings. This pipeline produces structured primary and aggregate records tailored for ingestion into search indexes such as Globus Search [13]. The snapshot pipeline also embeds version identifiers to support idempotent periodic pipeline execution (e.g., daily or weekly) and to automatically invalidate prior records, allowing users to detect object creation, updates, and deletion over time. 2) Event monitor: The real-time event monitor is designed to maintain index freshness between snapshots by processing file system events as they are emitted. Unlike snapshots, events describe file and directory operations, not state, and thus require stateful reconstruction to derive the final metadata changes. For example, file and directory create, update, and delete events may be canceled or coalesced, and directory rename events propagate recursively to all descendants. Icicle employs a stateful, rule-based reduction model that maintains in-memory directory hierarchy state and applies predefined coalescing and cancellation rules to reduce the event stream to a minimal, semantically equivalent set of metadata updates. C. Icicle Web Interface Icicle provides a web-based interface that unifies access to both primary and aggregate indexes, supporting administrative workflows such as ad hoc querying, capacity planning, and data lifecycle management, as well as user workflows including permission checks and large file or directory discovery.
The interface includes a graphical query builder and optional raw query mode supporting regex-match filters on any field; interactive visualizations of storage usage by user, group, and directory; and tools for generating file lists and scheduled reports for policy enforcement and remediation. In addition, the interface also generates summaries by populating structured templates with fields from the aggregate index, enabling rapid, high-level insight into file system behavior at scale. For automated or advanced workflows, both Elasticsearch and Globus Search APIs are directly accessible, enabling programmatic querying, filtering, and aggregation over both indexes. Examples of this web interface are shown in Fig. 2. IV. I MPLEMENTATION Here we describe how Icicle’s snapshot pipeline and event monitor are implemented. A. Pipeline Architecture Icicle’s snapshot pipeline processes file system metadata snapshots and delivers structured records to an external metadata index. Currently, the index is implemented using Globus Search [13], which is a scalable, secure, and managed indexing and query service for HPC-scale metadata; however, the specific metadata sink is replaceable, allowing use of alternative backends such as Elasticsearch [14] and OpenSearch [15]. At the ingestion layer, raw metadata snapshots (GUFI SQLite databases or GPFS TSV listings) are preprocessed into compact, uniform CSV files stored in Amazon S3. Then, at the distributed aggregation and processing layer, the snapshot pipeline operates exclusively on a reduced metadata representation, decoupling file system-specific formats from ingest logic and reducing data volume and I/O overhead. The snapshot pipeline is implemented using Apache Flink, deployed on Amazon Managed Flink. Managed Flink automatically handles operator retries, state persistence, and dynamic load balancing across Kinesis Processing Units (KPUs), each providing 1 vCPU, 4 GB RAM, and 50 GB ephemeral storage, ensuring high sustained throughput. The PyFlink code and its dependencies are packaged as a ZIP archive and stored in Amazon S3, from which workers retrieve them at start time. The snapshot pipeline comprises three logically distinct workflows, primary, counting, and aggregate, each expressed as a map–reduce computation. We use Amazon Managed Streaming for Apache Kafka (MSK), via Octopus [16] with Globus Auth integration, for ingest result logging in the primary and aggregate pipelines and for collecting the counting pipeline’s results. 1) From Preprocessed CSVs to Primary Records: The primary pipeline reads preprocessed CSV files from an S3 bucket and converts each row into a JSON record suitable for Globus Search ingestion. Each record contains three fields: subject: the file or link path from the file system root; visible_to: a list of users or groups who will be able to view the indexed record; and content: an arbitrary list of key-value pairs describing file attributes such as size, owner
(a) Top 10K users by storage, rendered from the (b) Graphical query builder issuing queries against (c) User summary populated from agaggregate index (Table III). the primary index (Table II). gregate index.
Fig. 2: Icicle web interface overview.
ID, group ID, permissions, and access, change, and modification timestamps normalized to ISO8601 with timezone offsets. Permission and mode bits are provided both in human-readable form (e.g., -rw-r--r--) and as integers (e.g., 100644) in content.raw. During the reduce stage, records are accumulated into batches of approximately 10 MB, the maximum payload accepted by the Globus Search ingest API. When either the batch reaches this limit or a 5-second timeout elapses, the accumulated records are submitted to the Globus Search ingest API. The asynchronous request IDs returned by the ingest API are published to a dedicated MSK topic for audit logging. Managed Flink executes the primary pipeline in streaming mode, allowing map and reduce operators to run concurrently and to be dynamically redistributed across KPUs, maintaining throughput and resilience. 2) Preparation for the Aggregate Pipeline: The counting pipeline computes object counts for each user, group, and directory prefix (truncated to a configurable maximum depth, directory_max) and prepares these counts as auxiliary input for the aggregate pipeline. Directory counts at this stage are non-recursive; recursive totals are computed in a subsequent post-processing step. For every input CSV row, the map worker assigns an integer shard ID in the range [0, 63] by applying zlib.crc32 to the row’s UTF-8 encoding. This hash-based sharding spreads work across workers to scale out processing and reduce skew. It then emits three intermediate tuples of the form (principal_id, shard_id, 1) to the reduce workers, where the principal IDs correspond to the owner ID (prefixed with ”u”), the group ID (prefixed with ”g”), and the directory prefix. During the reduce stage, all records sharing the same principal ID and shard ID are aggregated into a single (principal_id, shard_id, count) record. These outputs are written to a dedicated MSK topic. Once the counting pipeline completes, a script consumes the topic, reconstructs the full directory hierarchy to compute recursive directory counts (because the emitted messages contain only non-
recursive counts for directory shards), merges the user and group shard counts, and produces a compact CSV file that is staged alongside the code for use by the secondary pipeline. The counting pipeline operates in batch mode to emit one message per (principal_id, shard_id) shard. 3) From Preprocessed CSVs to Aggregate Records: The aggregate pipeline computes statistical summaries for each user, group, and directory principal. These summaries include quantiles as well as minimum, maximum, and average values for file sizes and timestamps. It reads preprocessed CSVs from S3 and also consumes the counting file produced by the counting pipeline. In the map stage, each CSV row is expanded into multiple records, one for each principal associated with that row’s path: user, group, and all directory prefixes between directory_min and directory_max. Each emitted tuple has the form (principal_id, shard_id, size, atime, ctime, mtime). Map workers maintain mergeable quantile sketches for the four numeric attributes (size, atime, ctime, and mtime), along with running minimums, maximums, and totals. These sketches ensure bounded memory usage and enable scalable quantile estimation, whereas exact quantile computation is expensive in terms of time and resource use. Once all shards for a principal are received, sketches are serialized and forwarded to the reduce workers. During the reduce stage, workers merge all sketch shards for a given principal and emit a single aggregate record. The subject field encodes the principal (e.g., ”user:123”, ”group:456”, or ”dir:/a/b/c”), and the visible_to field mirrors that of primary records. The content field contains quantile estimates (p10, p25, p50, p75, p90, p99), minimum and maximum values, total file size, and the file count. Flink executes the aggregate pipeline in streaming mode, as with the primary pipeline. However, the reduced volume of aggregate records allows them to be submitted to Globus Search immediately upon creation. a) Configurability, Scalability, and Extensibility: Resource allocation is flexible: Amazon Managed Flink allows each pipeline to scale from 1 to 256 KPUs, and the workflow
can be further decomposed along user, group, or directory dimensions (e.g., aggregate only on users or a subset of directories) to accommodate even larger workloads. Pipeline behavior is highly configurable: the operator can specify the pipeline sinks, define which users and groups are authorized to view indexed records, and tune batch size thresholds and submission timeouts. The pipeline also allows configuration of aggregation behavior, including the maximum number of shards per aggregation principal and the directory depth for aggregate records. The system is intentionally modular. Adding new metadata fields requires changes only to the preprocessing script and the primary pipeline’s mapper, while introducing additional aggregate statistics (such as 99.9th percentile) requires modifying a single line in the secondary pipeline’s reduce logic. This modularity allows the pipeline to evolve alongside file system features and analysis requirements. B. Monitor Architecture The monitor consists of three layers: event ingestion, metadata processing, and update notification. The ingestion layer unifies events into an internal format. The metadata-processing layer applies stateful reduction rules. The update notification layer emits path-resolved update and deletion events via MSK or directly to Globus Search. The design supports extensible reduction rules, pluggable storage models, and tunable retention windows for directory hierarchy tracking. 1) Event Ingestion Layer: The ingestion layer abstracts the heterogeneous event formats of Lustre and GPFS into uniformly structured key-value pairs containing the fields required for stateful reconstruction of metadata updates. For Lustre, the monitor invokes Lustre utilities (lfs changelog and lfs changelog_clear) via subprocess calls. It parses the raw textual output into structured dictionaries containing event ID, event type, timestamps, and file identifiers (FIDs). Different Lustre event types require different parsing logic; for example, creation and deletion events include a parent_fid, whereas rename events contain source_fid and source_parent_fid. A key performance optimization is to avoid lfs fid2path at this stage. Since this call incurs ∼10 ms of latency per invocation, path resolution is deferred to the metadata-processing layer, where it is invoked only when necessary (i.e., the changelog is not eliminated after applying stateful reduction rules). For GPFS, a mmwatch watcher can be configured to send fileset changelogs to a Kafka topic. The monitor can spawn one or more confluent_kafka [17] consumers operating in parallel, each assigned to different topic partitions. Events are serialized with the orjson [18] library and transported to the metadata-processing layer via a high-throughput, multipleproducer-single-consumer in-memory queue. The monitor supports optional ingestion-layer filtering to discard high-volume, low-information events (e.g., file open events), reducing metadata processing overhead. 2) Metadata Processing Layer: The metadata processing layer applies reduction rules to statefully transform event
streams into a minimal, semantically equivalent set of metadata updates. We include three types of reduction rules: (1) update coalescing: multiple events for the same FID are reduced to a single event, as a subsequent stat call can capture the final state of the object; (2) event cancellation: transient operation sequences, such as a CREAT followed by a UNLNK for the same file, or MKDIR followed by a UMDIR for the same directory, within the same batch are eliminated; (3) rename override: the above reductions are bypassed when a directory rename event occurs, as moving a directory affects both its original and destination parents as well as all descendant objects. Events that pass these reductions are stored in FID-keyed slots until a batch is ready. Batching is triggered either by reaching a size limit (e.g., 1000 events) or a time threshold (e.g., 5 seconds of inactivity), balancing throughput and update latency. The batch is then forwarded to the state manager (in the metadata processing layer), which maintains an in-memory representation of the file system hierarchy. Using parent–child relationships, the state manager resolves operations such as path construction for newly created objects without invoking lfs fid2path on Lustre, and recursively updates descendant paths affected by directory rename operations. Processing logic is dispatched to file system-specific handlers for creation, deletion, update, and rename events for both Lustre and GPFS. The state manager emits two lists: (1) to_update: with FID, path, and stat for each file and link object, where stat is gathered via stat calls on Lustre but directly carried from GPFS mmwatch changelogs. (2) to_delete: containing the FID and resolved path of objects whose primary records must be removed or invalidated in the index. 3) Update Notification Layer: The final stage converts the two lists from the state manager into Globus Search ingestion and deletion requests. Depending on deployment configuration, this layer can issue requests directly to Globus Search or publish them to a dedicated MSK topic for further inspection and integration with downstream consumers, such as user notification services. This design supports both synchronous ingestion for low-latency metadata updates and asynchronous processing for monitoring, auditing, or additional analytics. 4) Scalability and Persistence: To determine which objects are affected by directory rename events, each Lustre monitor maintains an in-memory directory hierarchy for the MDT from which events originate. Similarly, in GPFS, the monitor must consume all events from a given fileset topic to ensure correct path reconstruction. The monitor can be dedicated to a single MDT in Lustre or to a single Kafka topic for an individual GPFS fileset, allowing monitor deployment to scale linearly with the number of metadata servers and filesets. When resource sharing is desirable, a single monitor instance may watch multiple MDTs or multiple Kafka topics corresponding to different filesets. For both Lustre and GPFS, an optional LRU-based eviction policy limits the retention of inactive directory entries, reducing memory footprint while preserving correctness for active paths. Together, these mechanisms enable the monitor to scale with file system size and workload
intensity while keeping memory usage bounded. V. E VALUATION We evaluate the performance of the pipeline and the monitor to assess: (1) scalability of the pipeline; (2) effectiveness of the approximation algorithm; (3) scalability of the monitor on Lustre as the number of MDTs increases; and (4) scalability of the monitor on GPFS as the number of filesets increases. A. Pipeline Evaluation 1) Datasets: We evaluate the snapshot pipeline using three real-world HPC file system metadata snapshots: FS-small, FS-medium and FS-large. FS-small and FS-large are GUFI snapshots from project file systems at a large research computing center. FS-small is a partial index of a GPFS file system, covering the subtree that hosts organizational and staff home directories; it spans 67.63 TB of data and consists of 1.77 million GUFI entries tables with 8.46 million rows in total. FS-large is from an HPSS tape archive representing 53.59 PB of data across 40.88 million entries tables and 1.04 billion rows. Each entries row contains 22 metadata fields. FS-medium is from a national supercomputing facility and captures a full production GPFS. It comprises 145.59 million TSV records with 18 metadata fields, describing all files, links, and directories in a 1.55 PB file system. Important characteristics of these datasets are shown in Table IV. 2) Preprocessing: The preprocessing stage converts the input SQLite databases and TSV listings into CSV files that contain only the attributes required for the primary and aggregate indexes. File paths are escaped to safely handle special characters and ensure robust parsing. CSV files are generated with a target size of approximately one million rows; however, for FS-small and FS-large, individual files may exceed this threshold since splitting decisions are applied only after fully processing a GUFI database. The resulting CSVs are uploaded to a dedicated S3 bucket for each file system and consumed in parallel by the Flink pipeline. Preprocessing produced 9 CSV files (1.34 GB) for FS-small, 129 files (21.68 GB) for FS-medium, and 994 files (194.63 GB) for FS-large. Metadata preprocessing substantially reduces data volume by over 90% for FS-small and FS-large, and by over 40% for FS-medium. For GUFI-based snapshots, this reduction is achieved by retaining only ingestion-relevant attributes from the entries table and consolidating per-directory SQLite databases into million-row, headerless CSV files. For FSmedium, the total row count is reduced from 145.59 M to 128.50 M by filtering out directory entries and retaining only files and links. 3) Pipeline Runtime: In Table V, we report the runtime of each pipeline workflow across three file system datasets and two KPU configurations. Across all file systems, the aggregate pipeline consistently takes longer than the primary pipeline because it requires cross-KPU shuffles to aggregate records by users, groups, and directories, whereas the primary pipeline performs local, in-place aggregation within each KPU, emitting fixed-size (10 MB) bundles without shuffle.
With 128 KPUs, the total execution time of all three pipelines is approximately 8 minutes for FS-small, 29 minutes for FS-medium, and 217 minutes for FS-large. Increasing the number of KPUs yields a limited speedup for FS-small and FS-medium, as their inputs are preprocessed into only 9 and 129 CSV files, respectively. Because input files are assigned at the file granularity, additional KPUs cannot further parallelize ingestion, leaving the S3 read stage the dominant bottleneck. In contrast, FS-large contains 994 CSV files, and scaling from 128 to 256 KPUs reduces runtime by approximately 45%. These results show that preprocessing is a key scalability enabler: chunking metadata snapshots into smaller files is essential to fully utilize Flink parallelism and avoid underutilized KPUs. To further examine the impact of input granularity, we re-chunk the FS-small dataset into 100K-row CSV files (85 files total). This change improves performance across all pipelines, reducing overall execution time by 46% and aggregate pipeline runtime by 37%. The aggregate pipeline benefits most from finer partitioning because it runs in streaming mode, where map, shuffle, and reduce execute concurrently; increasing input files from 9 to 85 reduces its runtime from 263 s to 97 s. The counting pipeline runs in batch mode with barrier synchronization between stages, limiting its speedup. This difference explains why the counting pipeline runtime exceeds the aggregate pipeline runtime for FS-small*. The resulting index statistics are reported in Table VI. The primary index size scales with the number of records. FSmedium contains many more directories because we index two levels into user home directories (vs. one level for the others). In all cases, the aggregate index is under 1 GB, making it easy for the web interface to search and query. 4) Approximation Algorithms: We evaluate the aggregate pipeline’s quantile approximation accuracy by comparing the sketch estimates (q̂) against exact ground-truth quantiles (qexact ) across six target percentiles (p10–p99). The four sketching algorithms we chose are DDSketch [19], KLLSketch [20], ReqSketch [21], and t-Digest [22], drawn from the Datadog and Apache DataSketches libraries [23], [24]. We use their default relative-error guarantees. For each user and group, and for each distributional attribute of the aggregate index (size, atime, ctime, and mtime), we evaluate sketch accuracy using two error metrics over an aggregation of size N (the number of files): relative value error |q̂ − qexact |/|qexact | and normalized rank error |r̂ − r∗ |/N . Here, r̂ is the position of the estimated quantile q̂ in the exact sorted list, and r∗ is the expected rank of the target quantile (e.g., 0.5N for the median). The aggregate pipeline using these sketches exhibits comparable runtime across configurations, with DDSketch completing slightly faster (see the aggregate pipeline runtime column of Table VII). We observe a fundamental trade-off between rank and value accuracy across sketches. DDSketch provides the most stable value estimates, maintaining a mean relative error below 0.01 across all file systems, but at the cost of higher normalized rank error (worst-case mean = 0.31). In contrast, KLLSketch, ReqSketch, and t-Digest achieve superior rank accuracy
TABLE IV: File System Dataset Statistics. File system
Raw Metadata
Name
Size
FS-small FS-medium FS-large
67.63 TB 1.55 PB 53.59 PB
Source GUFI entries table mmapplypolicy LIST GUFI entries table
Preprocessed Metadata
Size
# Rows
# Cols
# CSVs
Size
# Rows
# Cols
90.75 GB 38.83 GB 2.15 TB
8.46M 145.59M 1.04B
22 18 22
9 129 994
1.34 GB 21.68 GB 194.63 GB
8.46M 128.50M 1.04B
9 10 9
TABLE V: Per-stage runtimes (seconds) and normalized total runtime per file system (normalized to 128 KPU = 1). File system
KPU
Primary Pipeline
Counting Pipeline
Aggregate Pipeline
Normalized Total
FS-small FS-small FS-small∗
128 256 128
79.82 74.55 62.69
166.88 107.64 116.48
263.11 242.95 97.56
1.00 0.83 0.54
FS-medium FS-medium
128 256
132.63 118.87
354.01 337.54
1,262.63 888.68
1.00 0.77
FS-large FS-large
128 256
926.83 483.72
3,554.24 1,832.08
8,552.87 4,820.03
1.00 0.55
∗ Uses a finer-grained CSV partitioning with a target of 100K rows per file.
(worst-case mean < 0.11) but exhibit large value errors, particularly around the median. This behavior stems from their internal bias toward accurately representing distribution tails, which leaves the central quantiles under-resolved and leads to large relative deviations in high-cardinality datasets such as FS-large. Given that our aggregate pipeline prioritizes value accuracy, we adopt DDSketch as the default sketch algorithm. Although exact aggregation is faster on FS-small (71 s vs. 97–98 s; Table VII), it requires reduce workers to hold complete value distributions in memory. For FS-medium and FS-large, this exceeds per-KPU memory capacity, as principals such as the root directory must aggregate across all files. DDSketch avoids this limitation with fixed-size summaries and a mean relative value error below 0.01, making sketch-based aggregation preferable for all production deployments. B. Monitor Evaluation We deploy Lustre and GPFS file systems on AWS to evaluate the Icicle monitor under backlogged metadata workloads. 1) Lustre setup: All Lustre experiments were conducted on an EC2–based Lustre cluster. The cluster comprised one Management Server (MGS) and two Object Storage Servers (OSSs) hosting four Object Storage Targets (OSTs), providing 1 TB of data storage. The number of OSSs and the available data capacity are not performance bottlenecks in our evaluation, since Icicle interacts exclusively with MDTs and does not issue data-path I/O to OSTs. Across different experiments, we configured 1, 2, or 4 Metadata Targets (MDTs) on 1, 2, or 4 Metadata Servers (MDSs). Each MDS was backed by a 32 GB gp3 EBS volume using the default baseline performance of 3000 IOPS and 125 MiB/s throughput. All servers were deployed on c5a.large instances in us-east-1a, while client nodes ran in us-east-1d, with an average server–client RTT of 0.757 ms. All servers run RHEL 8,
while clients run Ubuntu 24.04. Throughout this section, “FSMonitor” denotes an Icicle baseline that uses FSMonitor Algorithm 1 [12] for FID resolution, while leaving other components of the monitor unchanged; both baselines emit metadata update/delete requests to MSK (unlike the original FSMonitor, which outputs path-resolved changelogs rather than metadata changes). 2) Lustre baseline: We first evaluate Icicle monitor on a single-MDT cluster using the two workloads used to evaluate FSMonitor [12]: evaluate output script (eval_out) and evaluate performance script (eval_perf). eval_out repeatedly exercises a sequence of metadata operations within a directory: each iteration creates a uniquely named file, appends to it, renames it, creates a directory, moves the renamed file into the directory, and then recursively deletes the directory. In contrast, eval_perf stresses metadata throughput further by repeatedly performing a create–modify–delete cycle on uniquely named files, producing changelog events dominated by file create, opens, closes, and unlinks. Table VIII compares Icicle monitor and FSMonitor changelog processing throughput on a single-MDT Lustre cluster under the two workloads. Across all configurations, FSMonitor achieves substantially lower throughput (391–565 changelogs/s), reflecting the high cost of synchronous fid2path resolution (∼10 ms per call) performed for every changelog. In contrast, Icicle throughput is comparable to raw changelog ingestion, processing 32–33K changelogs/s, a 57–83× improvement over FSMonitor. This improvement stems from avoiding per-event fid2path: Icicle resolves an experiment’s directory FID once and then constructs descendant paths using parent-child directory state. Enabling changelog reduction further improves performance, particularly for the eval_perf workload, where Icicle with reduction reaches up to 41K changelogs/s, exceeding both Icicle baseline and Icicle receiving and emitting changelogs without stateful processing. These gains demonstrate that eliminating highly frequent but low-value 10OPENs (we keep 11CLOSEs) and 01CREAT/06UNLNK event pairs in the batch processor before they reach the state manager effectively reduces downstream processing overhead, yielding a consistent 1.1–1.2× throughput improvement over the base monitor configuration. Table VIII shows that increasing the client instance size from c5a.large to c5a.xlarge results in only modest throughput gains (∼2%), confirming that the single-MDT configuration, rather than client-side CPU or memory resources, is the dominant bottleneck in this setting.
TABLE VI: Primary and aggregate index statistics. Primary Index
File system FS-small FS-medium FS-large
Aggregate Index
# Records
Size
# Users
# Groups
8.46M 128.50M 1.04B
5.78 GB 90.85 GB 774.83 GB
37 240 2,091
12 178 325
Directory Depth One level into user home directories Two levels into user home directories One level into user home directories
# Directories
Size
1,133 65,190 16,724
3.48 MB 193.17 MB 56.95 MB
TABLE VII: Summary of sketch error across file systems for users and groups with at least 100 files. Minq and Maxq report errors over the six quantiles p10–p99, averaged across runs and aggregation keys. The aggregate pipeline runtime column reports the time to compute user and group aggregation (excluding directories), not live query retrieval; once built, the aggregate index supports sub-two-second query response times. Lower is better (↓). Algorithm
File system
Aggregate Pipeline Runtime (s)
Mean Normalized Rank Error (↓)
Mean Relative Value Error (↓)
Minq
Maxq
Minq
Maxq
FS-small
Exact DDSketch KLLSketch ReqSketch t-Digest
71 97 98 98 98
– 0.1023 0.0686 0.0705 0.0708
– 0.2620 0.1097 0.1095 0.1124
– 0.0048 0.0027 0.0017 0.0096
– 0.0057 0.0666 0.0317 0.0216
FS-medium
Exact DDSketch KLLSketch ReqSketch t-Digest
415 347 350 356 349
– 0.1493 0.0281 0.0276 0.0277
– 0.3110 0.0683 0.0693 0.0733
– 0.0066 0.0027 0.0040 0.0116
– 0.0080 0.1172 0.0372 0.3747
FS-large
Exact DDSketch KLLSketch ReqSketch t-Digest
4,993 2,478 2,526 2,607 2,554
– 0.1837 0.0189 0.0182 0.0182
– 0.2628 0.0406 0.0414 0.0456
– 0.0051 0.0056 0.0058 0.0135
– 0.0058 0.1629 2.2493 1.9477
Client
Workload
c5a.large c5a.large c5a.xlarge c5a.xlarge
eval_out eval_perf eval_out eval_perf
Chg FSMonitor 34,786 35,089 35,434 35,843
554 391 565 406
Icicle Icicle+Red. 32,162 32,553 32,678 33,179
35,680 40,471 36,194 41,120
3) Lustre scaling: We evaluate the monitor’s scaling performance using a Filebench [25] workload that generates realistic, metadata-intensive access patterns. The workload pre-populates a directory tree with 50K files whose sizes follow a Gamma distribution (mean size ∼16 KB, γ = 1.5), organized with an average directory width of 20 and a mean directory depth of 3.6. During execution, 32 concurrent threads repeatedly perform open–read–close operations on randomly selected files for 180 s, producing a sustained stream of finegrained metadata events.
30K
Throughput (changelogs/s)
TABLE VIII: Average throughput (changelogs/s) for one client and one MDT with workloads eval_out (evaluate output) and eval_perf (evaluate performance). Chg: Icicle receives and emits changelogs without stateful processing; FSMonitor: Icicle with FSMonitor-style FID resolution; Icicle: Icicle baseline; Icicle+Red.: Icicle with changelog reduction. c5a.large (2 vCPUs, 4 GiB RAM) and c5a.xlarge (4 vCPUs, 8 GiB RAM), each with AMD EPYC processor clocked up to 3.3 GHz. Each successive size (xlarge, 2xlarge, 4xlarge, . . . ) doubles the number of vCPUs and memory.
25K 20K 15K
FSMonitor (c5a.large) FSMonitor (c5a.xlarge) Icicle (c5a.large) Icicle (c5a.xlarge) Icicle+Red. (c5a.large) Icicle+Red. (c5a.xlarge)
10K 5K 0 1 MDT 1 client
2 MDTs 2 clients
4 MDTs 4 clients
Fig. 3: Filebench throughput scaling on Lustre as the MDT count increases with c5a.large or c5a.xlarge clients, with one client per MDT.
Fig. 3 shows throughput scaling using c5a.large and c5a.xlarge clients as the number of MDTs increases. Icicle scales nearly linearly, from ∼5.4K changelogs/s on one MDT to over 23K changelogs/s on four MDTs. Changelog reduction yields only modest gains (0–2%), as the Filebench workload lacks create–delete patterns, limiting reduction to filtering 10OPEN events before they reach the state manager. Placing monitors on c5a.xlarge yields only modest gains (2–7%), suggesting throughput is limited mainly by MDT parallelism
Throughput (changelogs/s)
400K Icicle+Red. (1p) Icicle+Red. (2p)
300K 200K 100K
Icicle (1p) Icicle (2p)
0K 1
2
3
4
5
Number of Clients
Fig. 4: Filebench throughput scaling on GPFS as a function of the number of clients, with one c5a.large client per fileset. 1p denotes a monitor consuming from one Kafka partition, while 2p uses two partitions merged into a single state manager. The limited gain from 2p indicates that throughput is constrained by fileset parallelism and event processing.
Throughput (changelogs/s)
150K 125K 100K 75K 50K
c5a.xlarge c5a.2xlarge c5a.4xlarge
25K 0K 1
2
4
c5a.8xlarge c5a.16xlarge 8
16
Kafka Topic Partitions
Fig. 5: Throughput versus Kafka topic partitions for a single GPFS client (one fileset). Solid lines denote Icicle, and dashed lines denote Icicle with event reduction. Throughput saturates beyond two partitions due to aggregation at the state manager.
and monitor event processing, not client resources. On the Filebench workload, FSMonitor achieves higher throughput in the single-MDT configuration (∼1.36K changelogs/s) than on the eval_out and eval_perf workloads (0.3–0.5K changelogs/s). This is because Filebench does not delete files after the initialization phase, allowing FSMonitor to reuse cached fid2path resolutions for repeated open, read, and close operations, thereby reducing lookup overhead. In contrast, although Icicle performs only a single fid2path resolution at the experiment directory root and derives all subsequent paths recursively from parent directory state, it exhibits lower throughput in this setting than in the previous two workloads in the baseline evaluation, because it performs additional per-event bookkeeping to maintain directory states, which may invoke per-file stat. Nevertheless, Icicle still achieves 3.68–4.05× higher throughput than FSMonitor.
4) GPFS Scaling: We evaluate Icicle’s scalability on GPFS by configuring a single-node cluster running on c5a.xlarge to stream changelog events directly to a local Kafka broker. We apply the same Filebench workload used in the Lustre experiments to each fileset. We vary the number of (1) watched filesets (1–5); (2) client nodes (one c5a.large client per fileset); and (3) Kafka topic partitions (1 or 2). To explore upper bounds, we additionally scale client instances up to c5a.16xlarge, increase Kafka topic partitions up to 16, and scale the GPFS+Kafka server to c5a.16xlarge. Servers and clients are set up in us-east-1d, and the roundtrip latency is ∼0.487 ms. Fig. 4 shows that Icicle scales linearly with the number of clients. This setup mirrors the Lustre experiments, where we add one client per MDT; here, we add one client per fileset. Without event reduction, throughput reaches 55.7K changelogs/s for a single fileset and increases to 279.5K changelogs/s for five filesets. With event reduction enabled, throughput improves from 62.9K to 315.7K changelogs/s, maintaining linear scaling even when both GPFS and Kafka run on a single c5a.xlarge server. We also evaluate configurations where each client runs two or more Kafka consumers (i.e., consuming a multi-partition topic), as shown in Fig. 4 for c5a.large clients and twopartition topics and Fig. 5 for c5a.xlarge clients and multi-partition topics. For c5a.large clients, performance is comparable to the single-partition configuration and is occasionally slightly lower, indicating limited benefit from additional partitions at this scale. In contrast, scaling up client resources has a clear impact: a c5a.xlarge client achieves 92.6K changelogs/s with a two-partition topic, compared to 55.0K changelogs/s for a c5a.large client, indicating that the client CPU is a primary constraint. Further scaling beyond c5a.2xlarge clients or more than four partitions yields diminishing returns, as performance becomes limited by the state manager’s ability to process GPFS changelogs. Notably, even the single-partition GPFS configuration significantly outperforms Lustre. This advantage stems from GPFS inotify-based changelogs, which include file metadata (e.g., stat information) directly in the event stream. As a result, Icicle avoids per-file stat calls in the state manager, substantially reducing processing overhead. To determine whether these findings hold beyond singlenode deployments, we also evaluated two-node and fournode GPFS configurations. Performance remained bounded by Kafka consumption throughput rather than event generation. Unlike Lustre, where adding MDTs introduces independent metadata partitions that the monitor can consume in parallel, GPFS scaling is constrained by the rate at which the monitor processes events from Kafka topics. Increasing the server instance size similarly provides minimal throughput improvement, confirming that the state manager’s processing capacity, not deployment topology or server-side resources, is the primary throughput bottleneck for GPFS.
VI. R ELATED W ORK
HPC File System Metadata Indexing. A large body of work has focused on improving the performance and scalability of metadata indexing and search in HPC file systems. Early efforts primarily employ tree-based structures such as k-d trees and R-trees to accelerate metadata queries over hierarchical namespaces [26]–[29]. TableFS [30] adopts an LSMtree-based design to optimize metadata insertion throughput. Subsequent distributed file system metadata planes [31]–[33] improve the scalability of metadata operations such as file creation and deletion. External metadata indexing systems decouple query performance from file system operations while reducing memory pressure [34], [35]. Brindexer [7] parallelizes namespace scans into SQLite databases, enabling efficient queries via RDBMSstyle partitioning and multi-threaded traversal; however, it still relies on static snapshots. GUFI [6] constructs a hierarchy of per-directory SQLite databases that preserve POSIX permissions and enable interactive metadata queries. Despite these improvements, GUFI’s architecture remains scan-based, requiring periodic rebuilds of read-only snapshots to maintain consistency with the underlying file system. The Robinhood Policy Engine [36] is another widely deployed external metadata indexing system. It mirrors system metadata in a local database (via scans or changelogs) and enforces policies for purge, hierarchical storage management (HSM), and OST-balancing. While its scan mode supports other POSIX-compliant systems, Robinhood mainly supports Lustre by polling changelogs from Lustre metadata servers and storing metadata (inferred from changelogs and stat calls) in a single SQL database, which limits scalability and flexibility. Recent work decomposes policy enforcement into event-driven agents. For example, QuickSilver [37] and PoliMOR [38] organize lifecycle management into scan, policy, and action agents that communicate over message queues with minimal shared state. Similarly, commercial engines, such as Cray ClusterStor Data Services [39] and GPFS Information Lifecycle Management [40], provide deep product integration, but they remain vendor-specific [38]. Changelog Monitoring and Event Detection. Unlike local file systems that expose OS-level event primitives such as inotify [41], kqueue [42], and FSEvents [43], HPC file systems provide dedicated telemetry interfaces. FSMonitor [12] unifies event sources across local (Linux, macOS, Windows) and Lustre file systems by defining a common event schema and aggregating notifications from them. While FSMonitor can ingest tens of thousands of events per second, its architecture remains centralized: collecting events and storing them in a single MySQL database without additional processing or exposing an external, queryable state index. Brindexer’s reindexer [7] listens to changelogs for modified directories but applies index updates in periodic, batch-oriented runs rather than maintaining a continuous real-time stream.
VII. S UMMARY Icicle is a comprehensive HPC file system metadata indexing framework that supports both static snapshot ingestion and real-time event monitoring. Our evaluation shows that the snapshot pipeline accommodates heterogeneous metadata sources and scales with resources to process larger snapshots, the quantile sketch maintains consistently low error while significantly improving performance, and the real-time monitor scales linearly with file system metadata parallelism. The web interface unifies access to indexed metadata with a graphical query builder and interactive visualizations of storage usage. Icicle supports efficient metadata ingestion, fulfilling the requirements for modern HPC file system management. ACKNOWLEDGMENT We thank the teams of the Diaspora Project and Globus for their valuable comments and feedback. This work was supported in part by the Diaspora Project, funded by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under Contract DE-AC0206CH11357, and by the Globus Search Project, funded by the National Science Foundation under Award 2411188. R EFERENCES [1] P. Braam, “The Lustre Storage Architecture,” Mar. 2019, arXiv:1903.01955 [cs]. [Online]. Available: http://arxiv.org/abs/1903. 01955 [2] M. Lakin, “OLCF announces storage specifications for Frontier exascale system,” 2021, retrieved Mar 25, 2026 from https://www.olcf.ornl.gov/2021/05/20/ olcf-announces-storage-specifications-for-frontier-exascale-system/. [3] N. Heinonen, “ALCF deploys powerful new file storage systems,” 2021, retrieved Mar 25, 2026 from https://www.alcf.anl.gov/news/ alcf-deploys-powerful-new-file-storage-systems. [4] National Energy Research Scientific Computing Center (NERSC), “Storage,” 2025, retrieved Mar 25, 2026 from https://www.nersc.gov/ what-we-do/computing-for-science/data-resources/storage. [5] R. Miller, J. Hill, D. A. Dillow, R. Gunasekaran, G. Shipman, and D. Maxwell, “Monitoring Tools for Large Scale Systems,” in Cray User Group Conference (CUG 2010), Edinburgh, Scotland, May 2010. [Online]. Available: https://cug.org/5-publications/proceedings attendee lists/CUG10CD/pages/1-program/final program/CUG10 Proceedings/pages/authors/06-10Tuesday/8C-Shipman-paper.pdf [6] D. Manno, J. Lee, P. Challa, Q. Zheng, D. Bonnie, G. Grider, and B. Settlemyer, “GUFI: Fast, Secure File System Metadata Search for Both Privileged and Unprivileged Users,” in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Nov. 2022, pp. 1–14. [Online]. Available: https://ieeexplore.ieee.org/document/10046106/ [7] A. K. Paul, B. Wang, N. Rutman, C. Spitz, and A. R. Butt, “Efficient Metadata Indexing for HPC Storage Systems,” in 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), May 2020, pp. 162–171. [Online]. Available: https://ieeexplore.ieee.org/document/9139660/ [8] F. B. Schmuck and R. L. Haskin, “GPFS: A shareddisk file system for large computing clusters,” in Proceedings of the Conference on File and Storage Technologies, ser. FAST ’02. USA: USENIX Association, 2002, pp. 231–244. [Online]. Available: https://www.usenix.org/legacy/publications/library/ proceedings/fast02/full papers/schmuck/schmuck.pdf [9] IBM, “mmwatch command — ibm storage scale 5.2.3 documentation,” 2025, retrieved Mar 25, 2026 from https://www.ibm.com/docs/ en/storage-scale/5.2.3?topic=reference-mmwatch-command. [10] J. Kreps, N. Narkhede, J. Rao et al., “Kafka: A distributed messaging system for log processing,” in Proceedings of the NetDB, vol. 11, no. 2011, Athens, Greece, 2011, pp. 1–7. [Online]. Available: https://notes.stephenholiday.com/Kafka.pdf
[11] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, “Apache flink: Stream and batch processing in a single engine,” The Bulletin of the Technical Committee on Data Engineering, vol. 38, no. 4, 2015. [Online]. Available: https://asterios.katsifodimos.com/assets/publications/flink-deb.pdf [12] A. K. Paul, R. Chard, K. Chard, S. Tuecke, A. R. Butt, and I. Foster, “FSMonitor: Scalable File System Monitoring for Arbitrary Storage Systems,” in 2019 IEEE International Conference on Cluster Computing (CLUSTER). Albuquerque, NM, USA: IEEE, Sep. 2019, pp. 1–11. [Online]. Available: https://ieeexplore.ieee.org/document/8891045/ [13] R. Ananthakrishnan, B. Blaiszik, K. Chard, R. Chard, B. McCollam, J. Pruyne, S. Rosen, S. Tuecke, and I. Foster, “Globus platform services for data publication,” in Proceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity, ser. PEARC ’18. New York, NY, USA: Association for Computing Machinery, 2018. [Online]. Available: https://doi.org/10.1145/3219104.3219127 [14] Elasticsearch, “Elasticsearch,” 2010, retrieved Mar 25, 2026 from https: //www.elastic.co/elasticsearch. [15] OpenSearch, “Opensearch,” 2021, retrieved Mar 25, 2026 from https: //opensearch.org/. [16] H. Pan, R. Chard, S. Zhou, A. Kamatar, R. Vescovi, V. Hayot-Sasson, A. Bauer, M. Gonthier, K. Chard, and I. Foster, “Octopus: Experiences with a hybrid event-driven architecture for distributed scientific computing,” in SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2024, pp. 496–507. [Online]. Available: https://arxiv.org/abs/2407.11432 [17] Confluent Inc., “confluent-kafka-python,” 2016, retrieved Mar 25, 2026 from https://github.com/confluentinc/confluent-kafka-python. [18] orjson Contributors, “orjson,” 2018, retrieved Mar 25, 2026 from https: //github.com/ijl/orjson. [19] C. Masson, J. E. Rim, and H. K. Lee, “Ddsketch: a fast and fully-mergeable quantile sketch with relative-error guarantees,” Proc. VLDB Endow., vol. 12, no. 12, pp. 2195–2205, Aug. 2019. [Online]. Available: https://doi.org/10.14778/3352063.3352135 [20] Z. Karnin, K. Lang, and E. Liberty, “Optimal quantile approximation in streams,” in 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), 2016, pp. 71–78. [Online]. Available: https://arxiv.org/abs/1603.05346 [21] G. Cormode, Z. Karnin, E. Liberty, J. Thaler, and P. Veselý, “Relative error streaming quantiles,” J. ACM, vol. 70, no. 5, Oct. 2023. [Online]. Available: https://doi.org/10.1145/3617891 [22] T. Dunning, “The t-digest: Efficient estimates of distributions,” Software Impacts, vol. 7, p. 100049, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2665963820300403 [23] Datadog, “sketches-py,” 2020, retrieved Mar 25, 2026 from https: //github.com/DataDog/sketches-py. [24] Apache Software Foundation, “datasketches-python,” 2024, retrieved Mar 25, 2026 from https://github.com/apache/datasketches-python. [25] V. Tarasov, “Filebench: A flexible framework for file system benchmarking,” ;login: The USENIX Magazine, vol. 41, no. 1, p. 6, 2016. [Online]. Available: https://www.usenix.org/publications/login/ spring2016/tarasov [26] A. W. Leung, M. Shao, T. Bisson, S. Pasupathy, and E. L. Miller, “Spyglass: fast, scalable metadata search for large-scale storage systems,” in Proceedings of the 7th conference on File and storage technologies, ser. FAST ’09. USA: USENIX Association, Feb. 2009, pp. 153– 166. [Online]. Available: https://www.usenix.org/conference/fast-09/ spyglass-fast-scalable-metadata-search-large-scale-storage-systems [27] Y. Hua, H. Jiang, Y. Zhu, D. Feng, and L. Tian, “SmartStore: a new metadata organization paradigm with semantic-awareness for next-generation file systems,” in Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, ser. SC ’09. New York, NY, USA: Association for Computing Machinery, Nov. 2009, pp. 1–12. [Online]. Available: https://dl.acm.org/doi/10. 1145/1654059.1654070 [28] A. Parker-Wood, C. Strong, E. L. Miller, and D. D. E. Long, “Security Aware Partitioning for efficient file system search,” in 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). Incline Village, NV, USA: IEEE, May 2010, pp. 1–14. [Online]. Available: http://ieeexplore.ieee.org/document/5496990/ [29] S. Patil and G. Gibson, “Scale and concurrency of GIGA+: file system directories with millions of files,” in Proceedings of the 9th USENIX conference on File and storage technologies, ser. FAST’11. USA: USENIX Association, Feb. 2011, pp. 177–190.
[30] K. Ren and G. Gibson, “TABLEFS: Enhancing metadata efficiency in the local file system,” in 2013 USENIX Annual Technical Conference (USENIX ATC 13). San Jose, CA: USENIX Association, Jun. 2013, pp. 145–156. [Online]. Available: https://www.usenix.org/conference/ atc13/technical-sessions/presentation/ren [31] K. Ren, Q. Zheng, S. Patil, and G. Gibson, “Indexfs: Scaling file system metadata performance with stateless caching and bulk insertion,” in SC’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2014, pp. 237–248. [Online]. Available: https://doi.org/10.1109/ SC.2014.25 [32] Q. Zheng, K. Ren, G. Gibson, B. W. Settlemyer, and G. Grider, “Deltafs: Exascale file systems scale better without dedicated servers,” in Proceedings of the 10th Parallel Data Storage Workshop, 2015, pp. 1–6. [Online]. Available: https://doi.org/10.1145/2834976.2834977 [33] W. Lv, Y. Lu, Y. Zhang, P. Duan, and J. Shu, “InfiniFS: An efficient metadata service for large-scale distributed filesystems,” in 20th USENIX Conference on File and Storage Technologies (FAST 22). Santa Clara, CA: USENIX Association, Feb. 2022, pp. 313–328. [Online]. Available: https://www.usenix.org/conference/fast22/presentation/lv [34] J. Cipar, G. Ganger, K. Keeton, C. B. Morrey, C. A. Soules, and A. Veitch, “LazyBase: trading freshness for performance in a scalable database,” in Proceedings of the 7th ACM european conference on Computer Systems, ser. EuroSys ’12. New York, NY, USA: Association for Computing Machinery, Apr. 2012, pp. 169–182. [Online]. Available: https://dl.acm.org/doi/10.1145/2168836.2168854 [35] SNIA, “Borgfs: File system metadata index search,” 2014, retrieved Mar 25, 2026 from https://www.snia.org/educational-library/ borgfs-file-system-metadata-index-search-2014. [36] T. Leibovici, “Taking back control of HPC file systems with Robinhood Policy Engine,” May 2015, arXiv:1505.01448 [cs]. [Online]. Available: http://arxiv.org/abs/1505.01448 [37] C. Brumgard, A. George, R. Mohr, K. Maheshwari, J. Simmons, and S. Oral, “QuickSilver: A Distributed Policy Driven Data Management System,” in Workshop: Women in HPC: Diversifying the HPC Community and Engaging Male Allies. Dallas, TX: Association for Computing Machinery, 2022. [Online]. Available: https://sc22.supercomputing.org/ proceedings/workshops/workshop pages/ws whpc103.html [38] A. George, C. Brumgard, R. Mohr, K. Maheshwari, J. Simmons, S. Oral, and J. Hanley, “Polimor: A policy engine made-to-order for automated and scalable data management in lustre,” in Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, ser. SC-W ’23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 1202–1208. [Online]. Available: https://doi.org/10.1145/3624062.3624190 [39] Hewlett Packard Enterprise, “Cray clusterstor data services user guide,” 2021, retrieved Mar 25, 2026 from https://support.hpe.com/hpesc/public/ docDisplay?docId=a00114855en us&docLocale=en US. [40] IBM, “Ibm spectrum scale information lifecycle management policies: Practical guide,” 2021, retrieved Mar 25, 2026 from https://www.ibm.com/support/pages/ ibm-spectrum-scale-information-lifecycle-management-policies-practical-guide. [41] R. Love, “Kernel korner: intro to inotify,” Linux J., vol. 2005, no. 139, p. 8, Nov. 2005. [Online]. Available: https://www.linuxjournal. com/article/8478 [42] J. Lemon, “Kqueue - A Generic and Scalable Event Notification Facility,” in Proceedings of the FREENIX Track: 2001 USENIX Annual Technical Conference. USA: USENIX Association, Jun. 2001, pp. 141–153. [Online]. Available: https://people.freebsd.org/∼jlemon/ papers/kqueue.pdf [43] Apple, “File system events,” 2012, retrieved Mar 25, 2026 from https: //developer.apple.com/documentation/coreservices/file system events.