arXiv CS open access

USDB: Scaling Storage for Next-Generation Recommender Systems at ByteDance

Xiaoran Zhu et al. · arxiv_cs

arXiv CS · Papers

Recommendation models consume increasingly long, attribute-rich user histories, making small continuous updates and repeated large reads a central storage asymmetry. We present USDB, ByteDance's unified sequence storage practice for streaming and batch workloads. CascadeLSM time-based compaction sta

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

USDB: Scaling Storage for Next-Generation Recommender Systems at ByteDance Xiaoran Zhu∗ , Jieming Cai∗ , Weiping Yu† , Xi Chen, Shu Cai, Jiahao Li, Yifan Xiao, Yan Deng, Yisha Zhou, Yu Lin, Shiru Ren, Siqiang Luo‡† , Peng Xu, Junjun Chai ByteDance Inc.; ‡ Nanyang Technological University

arXiv:2611.02618v1 [cs.DB] 2 Nov 2026

ABSTRACT Recommendation models consume increasingly long, attributerich user histories, making small continuous updates and repeated large reads a central storage asymmetry. We present USDB, ByteDance’s unified sequence storage practice for streaming and batch workloads. CascadeLSM time-based compaction stages materialize application-specific histories; differentiated replicas share persistent files; partial dumps reconstruct inference inputs from captured recent state and versioned history; and a hybrid columnar format accelerates projected reads. These designs reuse maintenance results across ingestion, inference, and training. Production migration records show 85.8% lower mean write latency, while normalized deployment accounting shows 42.1% lower infrastructure cost. Controlled comparisons with an Optimized RocksDB sequence baseline show faster long-history reads across projections and client loads. Aggregation pushdown sustains 8.5× the throughput of ordinary reads under a constrained receive budget. Production reconstruction and direct-file training results demonstrate how shared sequence maintenance supports fresh serving and repeated historical access.

INTRODUCTION

Recommendation systems [3, 55, 56], widely used in online content platforms and e-commerce, personalize billions of user experiences daily. At ByteDance, recommendation powers a diverse portfolio of products including videos in TikTok, Douyin, live streaming, and e-commerce platforms, serving billions of users worldwide. Over the past decade, the field has evolved from traditional collaborative filtering to Deep Learning Recommendation Models (DLRMs) [11, 14, 49, 76], and most recently, to Large Recommendation Models (LRMs) [8, 74, 75] that fundamentally reshape how recommendations are generated. Unlike DLRMs, which rely on heterogeneous feature engineering and do not scale efficiently with additional compute on more feature interactions or deeper layers [65, 74], LRMs adopt sequential transduction architectures that directly model user behavior sequences, exhibiting compelling scaling laws [8, 27, 32, 74]: model quality improves predictably with longer user behavior sequences, denser computation in attention layers, and increased model depth. These scaling trends increase the history retained and retrieved by recommendation infrastructure. At ByteDance, as shown in Figure 1, the sequence length consumed by recommendation models has increased 2,000-fold from 2023 to 2025 (50 to 100,000), with * equal contribution. † corresponding authors, siqiang.luo@ntu.edu.sg and weiping.yu@bytedance.com.

Legacy: separate stores 1M

Larger models

USDB: unified storage Sequence scaling

History retention 800K

History scale (actions) 0.5M 50 0 20 Storage per deployment (PB) 10 <1 0 2023 2024 Online (DRAM)

100K 20 2 2025 Writes 10×

Inference Materialization

Offline (SSD)

Training

Shared storage

2026 Reads 20× Inference Training

Figure 1: Recommendation-storage evolution through September 2026. Representative history and deployment-scale estimates accompany the storage organizations; traffic factors summarize approximate operational growth. representative retained-history configurations reaching 800,000 actions in September 2026, while the number of feature columns has grown from dozens to hundreds. Besides this algorithmic shift, the explosive growth in user base and engagement frequency has also amplified data volumes—Figure 1 summarizes the corresponding growth in deployment storage and the transition from separate stores to shared sequence storage. The plotted stages are representative internal estimates across applications; deployment capacity combines historical estimates with later scale anchors. Supporting these large-scale recommendation workloads requires robust data infrastructure capable of simultaneously serving both streaming (real-time) and batch (historical) data access patterns as shown in Figure 2. For streaming inference, the system must provide low-latency point lookups to retrieve recent user interactions. For batch training, it must deliver high-throughput historical lookups and scans over massive datasets. Feature stores [19, 43, 50] have emerged as an important component in ML infrastructure, providing feature management, discovery, and serving capabilities at the application layer. However, the underlying storage engine that powers these systems remains a critical yet under-explored challenge, particularly when handling petabytescale user behavior sequences [29, 36] with the dual access patterns described above. This dual requirement naturally suggests a Lambda architecture [34, 67] that processes real-time and

User events

App

click / like

USDB sequence state

Kafka

Recent

append

History

...

low latency

sample-time capture lookups

User + time range

Sample builder label

Read / model

recent

LRM Inference

tail

weights

History lookup

Top-K + columns

Top-K

reusable history; differentiated replicas publish and reuse its persistent files; recorded visibility connects inference inputs to later training reads; and a common indexed columnar format supports both consumers. This coordination turns the four challenges into the four corresponding mechanisms below. System Design. USDB (§ 4.1) integrates four components: • We propose CascadeLSM, a unified time-tiered LSM-tree that organizes streaming and batch data within a single structure. Through time-based data organization and differentiated compaction strategies, CascadeLSM eliminates the read-beforewrite pattern that plagued previous systems, reducing write latency by 85% while maintaining read latency (§ 4.2). • We design a differentiated multi-replica architecture over disaggregated storage-compute architecture, comprising read-write, read-heavy, and read-only replicas each tailored to its workload. Lightweight synchronization ensures replica consistency without redundant compaction, achieving a 42% reduction in total infrastructure cost. • We develop a sequence number-based versioning mechanism to maintain consistency between training and inference workloads. Combined with our partial dump mechanism, this reconstructs earlier inference inputs while replacing repeated full-sequence dumps with a captured recent fragment and reusable history (§ 4.4). • We design a hybrid columnar encoding format with adaptive encoding selection that serves both access patterns: columnar scans for training throughput and row-level indexing for inference latency. Our approach achieves 4.6–5.5× faster decompression than ZSTD in the codec evaluation, with encoded sizes between LZ4 and ZSTD (§ 4.5). Production Experience. Deployed since 2024 across Douyin, TikTok, and their live streaming platforms, USDB reduces mean write latency by 85.8%, mean CPU utilization by 72.6%, and normalized infrastructure cost by 42.1%. A 374-run study against Optimized RocksDB complements the production evaluation: repeated 10K/100K reads achieve 1.79×/1.75× throughput, and 100K client-read jobs complete 1.62–2.52× faster (§ 5).

Ranked items

Training throughput

LSN

Updates / capture

Results

Figure 2: Recommendation access. Inference reads recent Top-𝐾 histories; training repeats sample-time lookups. Both filter by user/time and select columns. Letters mark actions. batch data through separate pipelines, ultimately merging them for downstream consumption by training and inference systems. On the storage engine side, key-value storage using LSM-trees naturally aligns with our target due to its write-optimized structure [16, 18]. Combining these two techniques yields a promising storage architecture for modern recommendation systems in the large model era. Challenges. Scaling this architecture to billion-user platforms with petabyte-scale data introduces four fundamental challenges. First, existing approaches aggregate user sequences in memory using a read-before-write pattern; as streaming volumes exceed memory capacity and spill to SSDs, this transforms each small write into an SSD-bound bottleneck with severe amplification (§ 3.1). Second, LSM-tree compaction is CPU-intensive while batch storage demands large disk capacity, and coupling both on the same nodes drives up infrastructure costs and complicates capacity planning (§ 3.2). Third, maintaining consistency between training and inference becomes increasingly difficult at scale: out-of-order arrivals and delayed feedback signals cause the sequences observed during inference to diverge from those used in training, and LRMs are highly sensitive to such skew (§ 3.3). Finally, the conventional row-oriented SST format forces training workloads to deserialize entire records when scanning specific feature columns, wasting I/O bandwidth and yielding suboptimal compression ratios (§ 3.4). Our System. To address these challenges, we developed USDB, a universal user behavior sequence data backend for ByteDance’s entire recommendation system. This paper shares our industrial experience in building and operating a production-level storage system that serves one of the world’s largest recommendation platforms, involving billions of users, petabyte-scale retained histories, and large distributed training and serving fleets. Our industrial experience connects storage architecture, consistency mechanisms, and data formats within one sequence lifecycle. The contributions of this paper are as follows: Architecture Contribution. We organize streaming ingestion, historical materialization, inference, and training around a shared sequence lifecycle. Application-specific aggregation produces

2 BACKGROUND We introduce the sequence workloads, processing architecture, and storage primitives underlying USDB.

2.1

Long User Histories

LRMs use sequential transduction to model ordered user interactions, capture interests, and generate recommendations. Each interaction records a user, timestamp, and multidimensional features such as item identifiers, categories, and engagement signals. At ByteDance, a user’s history can contain tens of thousands of records, with petabytes in aggregate. Online inference retrieves a user’s sequence through a low-latency lookup keyed by uid to construct model input in real time. Training retrieves histories for many labeled samples through indexed lookups or bulk historical scans. These consumers impose different physical requirements: point lookups favor per-user locality and fast random access, whereas scans favor columnar efficiency and high I/O throughput 2

Table 1: Consumers of a shared logical user history. Consumer

Access

Dominant requirement

Inference

User, time range, Top-𝐾, columns Historical lookups for many samples Increments, materialization, backfill

Fresh observations; request latency Aggregate reads; original inference state Reusable history; bounded pending work

Training Maintenance

retains the requested number, and projects the selected columns. The application defines repeated-interaction, attribute-merge, and tie rules. Event time selects relevant behavior; ingestion version identifies what had arrived. Visibility selection precedes ordering and Top-𝐾, so a later-arriving event with an old timestamp cannot displace an event that inference observed. A partial dump carries the original selection parameters and a recent fragment; training retrieves a version-compatible older prefix from retained history.

3 MOTIVATION AND CHALLENGES

(Table 1). USDB combines these access patterns with applicationspecific aggregation and reconstruction of the history visible to an earlier inference request.

2.2

ByteDance’s early recommendation systems partitioned user sequences across distributed memory stores for low-latency inference. As the user base grew to billions and histories deepened, petabyte-scale sequence data exceeded memory capacity. Persistent storage must now support millisecond-level inference lookups and high-throughput historical training reads over a continuously growing dataset. Unifying these demands cost-effectively exposes four challenges.

Lambda Architecture

Lambda architecture [67] separates a speed layer that processes streams for low-latency serving from a batch layer that periodically recomputes views over complete history; a serving layer combines their results. Recommendation deployments use recent interactions for real-time inference and full histories to generate training datasets. Traditional implementations maintain separate stores, duplicating data and maintenance while complicating consistency and operation [34]. USDB unifies their storage while preserving both consumers’ latency and throughput needs.

2.3

3.1

In our early sequence storage implementations with moderate data volumes, we maintained streaming data entirely in memory and batch data on local SSDs, operating two independent databases. To optimize read performance for streaming queries, we aggregated user behavior sequences in memory using a key-key-value (KKV) format organized as date→uid→actions. This design enables a single point lookup to retrieve all interactions for a given user on a specific date. However, this aggregation strategy requires reading the entire existing sequence before each write operation to perform the merge, resulting in significant write amplification. While acceptable when all data resides in memory, this approach becomes untenable as write volumes grow. As user activity grew, streaming data spilled to SSD, turning this amplification into an I/O bottleneck. Appending one interaction to a sequence with thousands of actions requires reading prior data from SSD, merging in memory, and writing the complete sequence back. This amplifies small updates by orders of magnitude. Separate streaming and batch stores also duplicate data and maintenance. Periodic transfers create consistency windows and complicate recovery. We need a unified structure that preserves efficient streaming reads while eliminating read-before-write beyond memory capacity.

Log-Structured Merge-Tree

Log-Structured Merge-Tree (LSM-tree) [51] is a write-optimized data structure widely adopted in modern key-value stores. Writes are first appended to a Write-Ahead Log (WAL) for durability and buffered in memory; when the buffer fills, it is flushed as an immutable Sorted String Table (SST) file organized into multiple levels, with compaction periodically merging overlapping files to reclaim space and improve read performance. Since SST files are immutable, both compaction reads and writes exhibit sequential access patterns that align well with distributed file systems optimized for append-only operations. This design naturally suits our requirements: its write-optimized structure handles high-velocity user interaction streams, while compaction supports time-based data organization and aggregation.

2.4

Streaming Ingestion

Storage Formats and Encoding

Row-oriented storage formats store each key-value pair contiguously, optimizing for point lookups but performing poorly when scanning specific columns across many rows, a common pattern in training sample generation. Columnar formats instead store each column contiguously, enabling efficient scans and higher compression ratios, but incur overhead for point lookups and small updates. Since our workload requires both scanning specific feature columns across millions of user sequences for training and retrieving complete user records for inference, we adopt a hybrid design that adapts storage formats and encodings to access patterns.

3.2

Resource Imbalance

LSM-tree compaction consumes CPU to merge and recompress SST files, while batch storage needs disk capacity for extended histories. Traditional architectures couple both requirements on the same nodes, tying compute and storage provisioning together. This coupling creates several critical problems. First, different workload phases stress different resources. Streaming writes trigger compaction that consumes CPU, while batch reads demand sustained storage and network throughput, with CPU needs determined by decoding and sample construction. Provisioning for peak

Sequence selection and visibility. A history request specifies a user, event-time interval, maximum sequence length (Top-𝐾), and selected columns. Reconstruction also uses the ingestion-version boundary recorded by inference. The query first selects records visible at that boundary, then orders them by descending event time, 3

INPUT DATA recent updates

Behavior updates

§4.2

Inference client

§4.3

Maintain sequences

u7 | 10:03 | c | click

Serve sequences

Recent buffer

Fresh query

user | time | item | action

history a, b

Flush / compact

Kafka

ML APPLICATIONS

USDB storage service

recent c

Stable query: history only

Time-organized history write

read

One-user sequence u7:

capture

Replay §4.4 recent fragment

selected history query API

SHARED PERSISTENCE / HDFS SST

Bulk-built history file ingestion

Store and locate sequence columns §4.5 u7 u7

time 10:01 10:02

item a b

action … …

Training client

Key / time indexes

Direct file reads

Column blocks

Shared files + file-set metadata History / read paths

Updates / sample flow

Figure 3: USDB architecture: streaming updates and bulk-built history share maintenance, persistent files, and sequence access. Fresh queries combine recent state with history; training can query or read shared files and reconstruct earlier inference inputs from partial dumps. demand on both dimensions simultaneously leads to resource underutilization during normal operations. Second, capacity expansion is difficult: since each node’s data resides locally, scaling requires copying all shard data to new nodes, creating substantial migration overhead and risking service disruption. This is particularly problematic given the rapid growth in data volumes, where clusters frequently require expansion. The rigid CPU-to-storage ratio of container deployments exacerbates resource waste. For sequence feature workloads that are storage-heavy but compute-light, significant CPU cycles remain idle. Conversely, compaction-intensive periods may starve other operations of compute resources. Additionally, independent replicas using RocksDB [21, 22] must each execute redundant compaction operations, multiplying computational waste across the cluster.

3.3

LRM’s appetite for significantly longer and wider sequences has made the cost of full sample dumps prohibitive, across many overlapping samples. To address this, we designed a partial dump mechanism that captures only a subset of the inference state, specifically the most recent actions and relevant timestamps, then reconstructs complete training samples by merging with historical data from the offline cluster. Reconstruction must preserve inference visibility: timestamp ranges alone cannot exclude late arrivals within the middle of a sequence. Network-induced sideinfo loss and failover retransmission further complicate the merge. A versioning mechanism must identify which records inference observed and filter incompatible arrivals during training.

3.4

Storage Efficiency

Standard row-oriented SST formats optimize point lookups and small range scans. Complete key-value records are stored sequentially and indexed by key, serving inference efficiently but mismatching projected training reads. Training sample generation exhibits a columnar access pattern: it scans specific feature columns (e.g., item IDs, timestamps, interaction types) across millions of user sequences while ignoring other columns. Row-oriented storage forces the system to read entire records from disk and then discard unused columns, wasting I/O bandwidth and cache space. With user sequences containing dozens of feature columns and training workloads accessing only a subset, this inefficiency multiplies across petabyte-scale datasets. Additionally, row-oriented formats achieve suboptimal compression ratios. Compression algorithms exploit data locality and repetition, properties that manifest more strongly within individual columns than across heterogeneous column types in row-major layout. For instance, item ID columns exhibit strong categorical locality that enables aggressive compression through

Training-Inference Consistency

Recommendation models exhibit high sensitivity to traininginference data discrepancies, a phenomenon known as trainingserving skew. In production, we frequently observe scenarios where models show significant AUC improvements during offline training but exhibit metric degradation during online A/B testing. A primary contributing factor is inconsistency between the user behavior sequences observed during inference and those used during training. Interactions arrive out of order because of network delays, processing lag, or delayed feedback. Long-form video engagement signals may arrive minutes or hours later. Training must also exclude information unavailable at prediction time to prevent feature leakage, requiring both temporal and visibility alignment. The conventional approach to maintain consistency is to dump complete inference samples as snapshots for offline training. However, the combination of exponential growth in user activity and 4

(a) Time-tiered physical stages L0 | Current-window delta SST: u1 detail

(b) Closed-window user/item operator SST: u2 detail

Input detail | same window, user u1, item g1 Event time LSN Watch

Action bits

Capacity merge / flush

09:01

01 (click)

L1 | Current-window base u1 original rows

09:03

10 (like)

Group by uid/gid

u2 original rows

Output: one aggregated record (u1, g1) max watch = 5; flags = 01 OR 10 = 11

Window closes → aggregate

L2 | Previous-window base u1 aggregates

u2 aggregates

Sorted-run maintenance preserves these values.

Capacity + overlap

L3+ | Older windows older u1 groups

Optional detail-retention tree SST: original v5

older u2 groups

independent policy

SST: original v6

Fresh reads combine the base with uncovered recent rows.

Figure 4: CascadeLSM stages and aggregation. Current-window deltas and bases retain detail; closing a window aggregates by user/item. The example combines watch-time maxima and action-flag union. Older compactions preserve aggregates, while a separately retained detail tree keeps original rows. techniques like dictionary encoding, but mixing them with unrelated columns in row format dilutes these opportunities. The challenge is to develop a storage format that maintains row-oriented efficiency for inference point lookups while enabling columnar scans and improved compression for training workloads, all within a unified storage system that avoids maintaining duplicate copies in different formats.

Partial dumps bridge online serving and later sample generation. Inference records the recent fragment and visibility metadata; training combines it with retained historical data. The metadata identifies the original selection and compatible history, connecting reconstruction to the same lifecycle that maintains and publishes the sequence. Each component addresses the corresponding challenge in Section 3.

4.2

SOLUTION: USDB

USDB addresses the four challenges through coordinated storage architecture, sequence maintenance, input reconstruction, and physical layout. Four coordinated mechanisms address the challenges: CascadeLSM removes full-history rewriting from ingestion; differentiated replicas separate persistent capacity and maintenance from read serving; partial dumps with MVCC connect inference state to training; and hybrid columnar encoding accelerates selective historical access.

4.1

CascadeLSM: Adaptive Compaction

The core storage component of USDB is CascadeLSM, which unifies streaming and batch data management through a common LSM maintenance organization with time-based tiers. Rather than maintaining separate storage systems for real-time serving and batch training as in traditional Lambda architectures, CascadeLSM leverages adaptive compaction strategies to serve both workloads from reusable sequence state. This design reduces the separate representations and pipeline synchronization while enabling workload-specific optimizations at each tier through pluggable compaction logic.

Architecture

Figure 3 shows USDB’s four integrated components. At the storageengine layer, CascadeLSM organizes recent details and historical aggregates into time-based stages. Ingestion appends individual interactions; pluggable compaction constructs a user-local representation that inference and training can reuse. The hybrid columnar format combines per-user indexing with selective column access inside that representation. At the system layer, read-write replicas consume interaction logs, materialize sequences, and publish immutable SST files to shared HDFS storage. Read-heavy replicas combine a recent local staging region with those files for fresh reads. Read-only replicas reuse persistent views for bulk historical consumption. A batch builder can produce compatible history for the same access interface. The organization shares the cost of aggregation, encoding, and persistence across consumers while allowing compute to be provisioned by role.

4.2.1 Time-Tiered Organization. We organize the LSM-tree into multiple logical levels based on data recency, as illustrated in Figure 4. Level 0 serves as the delta layer containing the most recent incremental data within the current processing-time window. Level 1 holds the accumulated base data for the current window. Level 2 stores the previous window’s aggregated data, and deeper levels contain progressively older aggregated data. Each fixed processingtime interval [𝑤, 𝑤 + Δ) groups arrivals for materialization. The granularity is configurable: our production deployment typically uses daily windows to align with user activity and downstream training schedules; hourly or weekly windows are also supported. The upper levels store detailed, fine-grained interaction records to support real-time inference queries. Each record represents a single user action with its complete attributes. The lower levels store aggregated data for two reasons. First, the sheer volume of detailed historical data makes full retention impractical at 5

Local-file replicas Kafka

USDB: differentiated replicas Kafka Read/write (RW)

RW replica

Memtable

Flush + compaction

Write Read

Local SSTs + manifest

Streaming updates

Publish file set + source progress

Read-heavy (RH)

Read-only (RO)

File set + progress Staging region local buffer

Reads own state Write

Shared region adopted files

Fresh reads

Adopted file view

Stable reads

Shared HDFS Immutable SST files and manifest metadata File reads / writes

Metadata / adoption RH may flush/compact while shared progress lags.

Figure 5: Replica organization before and after disaggregation: local files versus differentiated roles over shared storage. scale. Second, downstream training workloads often require preaggregated features such as deduplicated or merged user behaviors within session windows. Pre-computing these aggregations reduces both data transfer costs and computation burden on training pipelines. For scenarios that require access to fine-grained historical records beyond the current window, CascadeLSM maintains a parallel backup LSM-tree that preserves detailed data without aggregation, with application-specific retention.

overlapping SST files are merged to reduce space amplification and maintain read performance. Since batch data is already aggregated, these compactions do not invoke external aggregation logic. Importantly, the aggregated data preserves the read efficiency characteristics of the original KKV (key-key-value) format: a single point lookup by uid retrieves all aggregated interactions for that user within a session window, enabling efficient sequence retrieval for training sample generation without requiring multiple random reads across scattered records. This cascading design makes compaction both a sorted-run maintenance operation and a reusable sequence transformation. Readers combine the materialized base with recent increments beyond its coverage. Publishing a new base advances that coverage together with the readable file set; the covered staging prefix can then be retired. The processing window chooses when aggregation runs, while event time continues to select relevant behavior. A late event or backfilled observation enters a new processing version even when its event timestamp belongs to an older query interval. Let 𝑚 increments accumulate between consolidations, 𝐶 (𝑚) be consolidation cost, and 𝑅(𝑚) the pending work per read. At update/read rates 𝜆𝑤 /𝜆𝑟 , maintenance-dependent work is

4.2.2 Compaction Strategies. Different tiers employ different compaction strategies optimized for their respective data characteristics and access patterns. For streaming data in the upper levels, we trigger compaction based on capacity thresholds rather than time boundaries. When the memtable reaches a configured size (e.g., 10 GB), it flushes to Level 0 as an SST file. Direct appends through LSM merge operations remove full-sequence read-beforewrite from foreground ingestion. Within a single processing window, data may compact multiple times between upper levels as capacity thresholds are exceeded. The key insight of CascadeLSM lies in how we handle the transition from streaming tiers to batch tiers. When a processing window closes, we trigger an aggregation compaction that transforms data from the upper levels to the lower levels. This compaction invokes an external CompactionService that implements a custom PartitionKeyMergeOp. This operator performs user-centric aggregation by treating uid as the partition key during multi-way merge: data is first partitioned by user, then within each partition, records sharing the same uid and gid (item identifier) are aggregated within the processing window. The aggregation logic is application-specific, including deduplication, merging interaction attributes (e.g., taking maximum watch time, bitwise-OR of action flags), and computing aggregate statistics. This design shifts aggregation from the write path, where our previous read-beforewrite system suffered severe amplification, to the background compaction path, transforming what was an online bottleneck into an offline batch operation. For batch data in the lower levels, we apply standard capacitybased compaction. When a level reaches its capacity threshold,

W (𝑚) = 𝜆𝑤 𝐶 (𝑚)/𝑚 + 𝜆𝑟 𝑅(𝑚).

(1)

Capacity triggers bound pending state and file count; window triggers select the application transformation boundary. This separates maintenance amortization from history representation. Production deployment experience: In production, compaction serves purposes beyond merging sorted runs. It provides a natural interception point for data transformation such as aggregation and reorganization without additional data movement. By externalizing this logic to a pluggable CompactionService, USDB decouples application-specific transformations from the storage engine core, allowing a single engine to serve heterogeneous upstream and downstream requirements effectively.

(a) Ordered, [2,9] Inference

History

Dump Replay

7 2

8 3

9 4

(b) Time only, [2,8]

6 2

7 3

8 6

6 2

7 3

8 4

h,i > g

Figure 6: Partial-dump replay. Circles show event times; letters below show LSNs. Gray: skipped history; orange: captured tail; red: late arrivals. Time-only replay adds 4 and 5; MVCC excludes ℎ, 𝑖 > 𝑔. Ranges are inclusive; Top-𝐾 is nonbinding.

4.3

Differentiated Replicas

using the shared region’s LSN as a filter to exclude redundant entries from the staging region. This design ensures that read-heavy replicas observe a consistent data state while minimizing redundant compaction work. The staging region supplies recent observations, while the shared region supplies the persisted prefix. For one ingestion partition, a shared frontier 𝑝 and staging frontier 𝑞 give the read view as shared records through 𝑝 plus staging records in (𝑝, 𝑞]. Advancing 𝑝 with the readable file set permits covered staging data to be reclaimed.

To address the resource imbalance challenge, we build a differentiated multi-replica architecture atop HDFS-based disaggregated storage (Figure 5). Rather than treating all replicas uniformly, we tailor each replica type to its specific workload characteristics, enabling workload-specific resource provisioning and lightweight inter-replica synchronization that avoids redundant compaction. 4.3.1 Differentiated Node Types. After disaggregation, we deploy three types of nodes tailored to different workload characteristics. Read-write replicas handle data ingestion and compaction, consuming from Kafka and persisting SST files to HDFS. These nodes require substantial CPU for compaction but minimal local storage since data resides on HDFS. Read-heavy replicas serve inference and streaming-training workloads that require fresh observations. These nodes perform minimal compaction, primarily maintaining a local memtable buffer while reading the majority of data from shared HDFS storage. Read-only replicas serve offline training workloads that can tolerate slightly stale data. These nodes perform no compaction and simply read from HDFS snapshots. This differentiation allows us to provision each node type with appropriate resources. Read-write nodes receive high CPU allocation with minimal SSD. Read-heavy nodes receive moderate CPU with fast network connectivity. Read-only nodes receive minimal CPU with high memory for caching.

4.3.3 Simplified Scaling. Disaggregation dramatically simplifies cluster scaling. In the previous architecture, scaling required copying complete shard data to new nodes. With disaggregation, we construct a separate database for each Kafka partition within a shard. Scaling then proceeds as follows: first, new read-only nodes load the target partition databases from HDFS; second, the old read-write nodes unload those partitions; third, new read-write nodes load the partitions and begin consuming from Kafka; finally, old read-only nodes unload the partitions. Historical SST files remain on HDFS and can be opened by the new owners; the handoff reuses the existing bulk data. Opening the target database, replaying its uncovered log suffix, and warming the reader cache determine readiness after handoff. Bulk history files stay reusable in shared storage. Production deployment experience: While disaggregated storage is increasingly common in modern KV systems [22, 58], the key challenge lies in how replicas are organized and kept consistent atop shared storage. By differentiating nodes into replicas with workload-specific provisioning, and coordinating them through lightweight synchronization, USDB avoids redundant compaction across replicas while maintaining consistency.

4.3.2 Replica Consistency. Read-heavy replicas reuse the writer’s persistent state and keep a recent staging region for online workloads. We achieve this through a two-region architecture within each read-heavy replica: a staging region and a shared region. The staging region subscribes to Kafka and buffers incoming data in a local memtable. This preserves recent observations for fresh reads while shared persistence catches up. When the memtable exceeds a configured size, the replica initiates a flush but first queries the shared region for its maximum Log Sequence Number (LSN). If the shared region’s LSN exceeds the staging region’s maximum LSN, the memtable can be safely discarded since the data is already persisted. Otherwise, only the delta beyond the shared region’s LSN needs to be flushed to HDFS. Read-heavy replicas may therefore perform fallback persistence and maintenance when the shared writer lags. The shared region opens the HDFS-resident RocksDB instance in read-only mode using the OpenAsFollower mechanism. It periodically applies metadata changes from the read-write replica, tracking the maximum persisted LSN. During read operations, a MergeRead with LSN filtering combines data from both regions,

4.4

Partial Dump with MVCC

To achieve high training-inference consistency while controlling storage costs, we design a two-part solution that balances these competing requirements. As discussed in Section 3.3, conventional full sample dump approach becomes prohibitively expensive at our scale. We address this through partial dump for cost efficiency combined with Multi-Version Concurrency Control (MVCC) based on Log Sequence Numbers (LSNs) to select the observations available to inference.

the LSN range. During training, we filter the training sequence to include only actions whose LSNs fall within the dumped LSN range. Actions 4 and 5 have LSNs (h,i) that exceed the maximum dumped LSN (g), so they are correctly excluded. The resulting training sequence (2,3,6,7,8) matches the inference sequence exactly. The saved version boundary likewise excludes observations introduced by a later retransmission. The source partition and version domain travel with the reconstruction metadata so that the cutoff is applied to the corresponding history.

Lookup: user u2, time [2, 8] Read fields: Time, Item, Watch One persisted SST file Data: columns encoded within each row group Row group A | u1 | stored time [1, 9] Time Item Watch Other Row group B | u2 | stored time [1, 7] Time Item Watch Other

Retaining the right version. For an aggregate whose values evolve, reconstruction also selects a compatible retained base or original detail. For example, watch-time observations 2 at 𝑣 1 and 5 at 𝑣 2 produce a latest maximum of 5; replay bounded by 𝑣 1 uses the earlier observation 2. Retaining detailed inputs or version-compatible materializations supplies that earlier value. Event time chooses behavior, ingestion version chooses available observations, and the published file view identifies the readable representation. Imported feature versions identify their corresponding historical base. The pipeline records version ranges and action counts and exposes reconstruction consistency to downstream consumers. Applications choose a partial window that bridges recent pipeline delays and their recovery interval. A larger captured window stores more of the inference state directly; a smaller one increases historical reuse. The same interface supports this choice without changing the training sample’s logical input.

Row group C | u2 | stored time [10, 12] Time Item Watch Other

Index Key / time locations u2, [1, 7] → B Skip A (key), C (time)

Column positions B: Time, Item, Watch offset + length per field

Schema: fields, types, stored encodings Footer: format version; index offset skipped group Index lookup

unused column Selected read

Figure 7: Indexed columnar layout: data blocks, column metadata, and per-user access share one file format.

Production deployment experience: While recommendation, advertising, and search ranking systems tolerate weak consistency in the traditional database sense, high training-inference consistency remains critical for model accuracy. Full sample dumps ensure consistency but incur prohibitive storage overhead at scale. USDB pairs lightweight partial dumps with LSN-based versioning for high consistency and lower snapshot cost.

4.4.1 Partial Dumps. Our partial dump mechanism captures only the most recent portion of user behavior sequences during inference, rather than dumping complete sequences. Complete training samples are then reconstructed by merging partial dumps with historical data stored in the training cluster. However, partial dump alone reintroduces consistency challenges that full dumps inherently avoided. A naive implementation uses the user’s event timestamp (when the action occurred) to align partial dump data with historical data during the merge. This works correctly when out-oforder events appear only at sequence boundaries, as shown in Figure 6(a). The problem arises when out-of-order events fall within the middle of a sequence. Consider Figure 6(b) where actions 4 and 5 arrive out of order due to network lag. During inference, the online sequence observes [2,3,6,7,8] while the training cluster has received [0,1,2,3,4,5,6,7,8]. The partial dump captures [6,7,8] with inclusive timestamp range [2,8]. During training, merging based on timestamps produces [2,3,4,5,6,7,8], which includes actions 4 and 5 that were not visible during inference. This training-inference inconsistency degrades model quality.

4.5

Hybrid Columnar Encoding

Typical LSM implementations use row-oriented SST formats optimized for point lookups, but this organization performs poorly for training workloads that scan specific feature columns across millions of user sequences. To address this, we developed a hybrid columnar format that balances inference and training access patterns, as illustrated in Figure 7. 4.5.1 Self-Describing Format. Our file format comprises three regions: a data region, an index region, and a footer. Inspired by PAX [2] and Parquet, the data region partitions data into row groups and stores values column-by-column within each group, but further introduces a fine-grained layout hierarchy for flexible IO and predicate pushdown. Concretely, the data region is organized as a RootLayout containing one ZoneLayout per column, each divided into ChunkLayouts (analogous to column chunks within a row group). Each ChunkLayout groups multiple FlatLayout(data) blocks, the basic unit of IO and computation. Each ZoneLayout also maintains a dedicated FlatLayout(zone) storing per-column zone maps that enable predicate pushdown

4.4.2 Versioned Replay. We resolve this through MVCC by assigning each action an LSN representing its system arrival time rather than its event time. As shown in Figure 6(c), each action carries both its event timestamp and its LSN. The online sequence (2,3,6,7,8) has LSNs (c,d,e,f,g). The training sequence (0,1,2,3,4,5,6,7,8) has LSNs (a,b,c,d,h,i,e,f,g), where actions 4 and 5 received later LSNs (h,i) despite having earlier event timestamps. During partial dump, we record both the timestamp range and 8

Table 2: Challenge → solution → evidence.

and fine-grained IO skipping. The index region includes Zone Maps & Range Indexes for range queries, BloomFilter for point lookup acceleration, and TableScheme for self-describing metadata. The footer stores FileVersion and IndexOffset for format versioning and index location. Embedded schema and file-version metadata identify stored fields and support additive column evolution. Applications supply feature identity and aggregation semantics. Column-oriented decoding feeds vectorized data processing while preserving the same indexed files for serving. Aggregation pushdown executes an aggregate over the selected user history near the stored data and returns a compact result instead of transferring feature rows. Our implementation supports approximate count while retaining the query’s user, time-range, and Top-𝐾 selection. Section 5.2 evaluates this aggregate under increasing query concurrency and constrained receive bandwidth.

Challenge

Replica overhead Shared files

Provisioning/cost

Repeated dumps

Tail + versioned prefix Reconstruction

Large reads

Columnar + pushdown Read/aggregate QPS

5.1.1 Length and Load. USDB’s advantage grows as reads retrieve more history (Figure 8). At eight clients, throughput is 1307.0 ± 15.7 versus 728.0 ± 23.6 QPS for 10K-action requests (1.79×), and 257.7±2.6 versus 146.8±0.3 for 100K (1.75×), using three runs per system and length (𝑛 = 3). At 20K, throughput is 903.7 ± 6.1 versus 473.3 ± 3.1 QPS (1.91×). The 100K mean latency is 31.13 versus 54.47 ms; both systems return roughly 90K rows per nonempty result. The same repetitions give similar 1K throughput, 2554.3±21.5 versus 2578.0 ± 79.5 QPS. This progression connects the benefit to repeated long-history retrieval. The crossed scan extends the result across five client loads and six requested lengths. At 30 clients, USDB reaches 4255/714.5 QPS at 10K/100K, compared with 2805/456.5: 1.52×/1.57×. At one client, the corresponding ratios are 1.91×/2.06×. Short 1K requests scale differently: at 30 clients the baseline reaches 11622.5 QPS and USDB 9102.5, with both reaching the request budget. An independent 100K concurrency sweep reaches 654.5/441.5 QPS at 30 clients and p99 100.67/102.89 ms.

Production deployment experience: File format flexibility is essential for serving diverse workloads within a unified storage system: smaller row groups favor point lookups for online inference, while larger row groups improve compression and scan throughput for offline training. USDB’s self-describing format enables the storage engine to adapt layout and encoding decisions to workload characteristics without requiring separate storage paths.

PRODUCTION EVALUATION

We evaluate four design questions: maintenance efficiency, replica provisioning, repeated sample payload, and selective history reads (Table 2). We first define the query patterns and compare the deployed read paths under controlled load, then evaluate resource accounting, reconstruction, and encoding. Production scale and fullday operation close the empirical study.

5.1

Evidence Write/CPU traces

Both use comparable production-workload samples, 15 common fixed-width columns, and identical query parameters. Method. Queries retrieve a user’s history within an eventtime range, in descending time order, with a Top-𝐾 limit and selected columns. Both clients use the same configuration and UID-sampling rule. Ordinary runs use 30 seconds of warm-up followed by up to 60 seconds of measurement; extended sweeps also cap each phase at 400K requests. Fixed-work jobs time retrieval under equal request budgets, including startup and shutdown, before feature decoding and model computation. The study contains 374 measurement runs across history length, projection, load, client resources, and access locality. Throughput is the median QPS over active five-second reporting intervals; latency uses cumulative request statistics. Errors and empty responses are included in the outcome analysis. Repeated conditions report mean ± sample standard deviation over three independent runs (𝑛 = 3), with individual points shown.

4.5.2 Adaptive Encoding Selection. Our format employs a decision tree-based encoding selector that chooses encodings based on column types and data characteristics. Inspired by recent columnar compression works [35, 64], the selector cascades lightweight encodings, including dictionary encoding, frame-of-reference, bitpacking, and run-length encoding, adaptively based on column statistics such as cardinality, value distribution, and data type. As shown in Section 5.5, this approach achieves significantly faster decompression than general-purpose compressors like ZSTD and LZ4 while maintaining encoded sizes between the tested LZ4 and ZSTD configurations. This trade-off is critical for training workloads where decompression throughput directly impacts data loading performance. The hybrid columnar encoding serves both access patterns: columnar scans with lightweight encoding for training throughput, and row-level indexing for inference latency.

Solution

Growing histories Time-based stages

5.1.2 Projection across Lengths. Selecting fewer columns increases useful long-read capacity in both systems. At 100K and eight clients, one versus 15 columns increases USDB throughput from 250.5 to 598 QPS and the baseline from 139.5 to 210.5; p99 falls from 51.11 to 26.08 ms and from 91.20 to 63.51 ms. At 30 clients, two/eight/15 columns yield USDB 1715/1102/714.5 QPS versus 780/570/456.5, giving 2.20×/1.93×/1.57× (Figure 8e). The six-width scan separates output width from requested length (Figure 9). Long reads benefit from reducing the materialized payload; short prefixes contain a larger share of fixed lookup and request work. This explains why one-column selection is not a simple inverse-width speedup. Three independent runs at 10K/100K and two/eight columns further check projected-query

Read Performance

Systems and workload. We compare USDB with Optimized RocksDB, a deployed sequence service with production indexing and column selection. Both systems run on matched hardware with the same deployment topology and data-center environment. 9

(a,b) QPS (shared log scale) 20

100

10K

(a) USDB QPS

1.5

2.3

309

213

162

113

206

123

1.50

1.73

1.91

2.02

2.22

2.06

1297

882

661

468

235

141

1104

575

367

254

128

1.17

1.53

1.80

1.84

1.78

2447

1597

1236

896

437

184

2500

961

664

434

216

142

0.98

1.66

1.86

2.06

2.03

1.29

5033

3410

2480

1777

783

462

6184

2274

1589

961

479

287

0.81

1.50

1.56

1.85

1.63

1.61

9102

5948

4255

2873

1295

714

30 11622 4176

2805

1702

792

456

0.78

1.42

1.52

1.69

1.64

1.57

10K

20K

50K

100K

10K

20K

50K

100K

10K

20K

50K

100K

Requested Top-K USDB

Optimized RocksDB

Requested Top-K

2 columns

8 columns

(f) Three runs, p=8 (log QPS)

1500

Queries/s

6000 4000 2000

15 columns

(e) Projected reads, K=100K Queries/s (log)

(d) Projected reads, K=10K

Clients

Requested Top-K

Queries/s

0.7

(b) Optimized RocksDB QPS

Clients

1000 500 0

Clients

3000 1000 300 100

10K 20K

Clients

100K

Requested Top-K

Figure 8: History length, projection, and client load. (a–b) QPS for 15 columns, with a shared logarithmic color scale. (c) USDB/baseline throughput ratio; 1 denotes parity. Color keys appear above the heatmaps; line keys apply to the lower row. (d–e) Two/eight/15-column reads. (f) Three runs per length at eight clients: mean ± sample SD, with individual points and logarithmic QPS. USDB

1600

500

800

250

Optimized RocksDB

(b) QPS, 100K

(d) p99, 100K 80

p99 (ms)

Queries/s

(a) QPS, 10K

20 40

10 0

Selected columns

Figure 9: Projection at eight clients across six column selections. (a–b) Throughput; (c–d) p99 latency. Each metric is shown separately for 10K- and 100K-action reads. variation. Together, the crossed load/width curves and repetitions show where selective materialization contributes within each complete read path.

indicating sufficient client parallelism. Buffer changes produce less than 2% throughput range for each system/width. USDB’s wide-query client CPU cost is 3.95/5.69 ms per completed attempt at 4/100 workers, showing why extra workers need not improve useful capacity. Five repeated default settings per system and projection quantify variation across the client sweeps. Mean 100K, 30client throughput is 1709.4/789.5 QPS for two columns and 725.8/459.6 for 15. The client study records six/45 request errors for USDB/baseline; USDB also records 316 empty results. The companion outcome analysis reports the per-load error and empty-response rates.

5.1.3 Client Scaling. Figure 10 increases concurrency from 30 to 128 clients for 10K/100K reads and two/15 columns. At 100K and 96 clients, USDB reaches 2457.5 QPS with two columns and 1001 with 15, compared with 1169 and 658.5. Beyond 96, throughput largely plateaus while p99 rises. At 128, two-column p99 is 322.62 versus 253.48 ms, and 15-column p99 is 381.88 versus 430.70 ms. The curves distinguish throughput capacity from the latency operating point of a consumer. We also vary client workers from 4 to 200 and read buffers from 4 to 256 KiB. For 15-column 100K reads, the baseline reaches 368/455.5/456.5 QPS with 4/100/200 workers; USDB reaches 719.5/731.5/721. The sweep reaches a stable throughput region,

5.1.4 Read Jobs and Mixed Loads. The fixed-work jobs test completion of a known amount of input reading. At eight 10

USDB

(a) 10K QPS

Optimized RocksDB

(b) 100K QPS

p99 (ms)

Queries/s

10000 1000

48 64

Clients

128

(d) Worker QPS

400

2000

5000

15 cols

15000

2 cols

48 64

128

1500 1000

200

500 0

48 64

Clients

128

Clients

200

Client workers

Figure 10: Client scaling with two/15 columns. (a–c) Throughput and p99 across client loads. (d) Client-worker sensitivity at 100K and 30 clients. Table 3: Lookback windows, paired USDB/Optimized RocksDB values. Rows/hit is rounded down. ∗ Emptywindow baseline latency uses the final reporting window.

clients, the longer jobs issue exactly 150K, 50K, or 30K requests for requested lengths 1K, 10K, or 100K. Each request succeeds with a nonempty result. At 100K, two/four/eight/15column completion times are 55.82/65.85/80.78/120.81 seconds for USDB and 130.74/166.01/170.64/195.69 for the baseline: 2.34×/2.52×/2.11×/1.62× speedups. At 10K, the corresponding speedups are 1.82×/2.13×/1.70×/1.61×. Figure 11a shows the longer 100K-history jobs. At the shorter 10K-request budget, the corresponding jobs take 21.01/30.87/40.89 seconds versus 50.87/66.01/71.17. Longer jobs amortize fixed startup and reporting exit costs, preserving the long-read advantage. At 1K and eight/15 columns, longer-job times are 51.61/60.83 versus 50.82/56.03 seconds, consistent with the different balance of fixed and payload work for short reads. Mixed consumers co-start short (1K) and long (100K) processes with initial client allocations 30:0, 27:3, 15:15, 3:27, and 0:30. Each stream has its own worker pool and bounded budget. Both streams execute concurrently; short streams can finish earlier when they reach the request budget. At 15:15 with 15 columns, USDB’s long stream reaches 423.5 QPS and p99 54.08 ms, versus 270 and 76.78 ms. Its short stream reaches 3876 QPS and p99 10.94 ms, versus 4436 and 8.51 ms. The combined view shows how long-read gains coexist with the latency needs of a fresh, short-history consumer.

Window 1h 6h 1d 3d 7d

5.2

Empty (%) Rows/hit Mean ms

p99 ms

2.33/0.41∗

100.00/100.00 –/– 3.77/0.57∗ ∗ 100.00/100.00 –/– 2.36/0.56 3.81/2.70∗ 34.22/29.95 90/90 2.44/1.74 3.85/8.73 0.88/0.86 840/1007 2.92/3.31 4.90/11.79 0.21/0.35 2359/2819 3.48/5.00 6.54/15.01

Aggregation Pushdown

We evaluate aggregation pushdown using approximate count over selected histories. USDB returns the aggregate as a scalar; the two comparison paths count timestamp rows retrieved through ordinary USDB and Optimized RocksDB reads. All paths use matched histories, the same time window, and the same Top-𝐾 cap. Sampled correctness checks agree exactly in 77 of 80 cases, with the remaining estimates differing by at most two rows. Matched histories under load. We fix a length-matched user cohort and a 30-day window, returning about 33K actions per query with a 10M Top-𝐾 cap. All client CPUs remain available. The foreground sweep uses eight seconds of warm-up and 20 seconds of measurement, repeated three times; QPS is the per-run median of active two-second reports. Figure 13a uses fixed 128 workers at 64/96/128 concurrent queries and fixed 256 workers at 128/160/192 queries, with both sweeps including the 128-query setting. At 192 queries, native count reaches 6.23K QPS versus 5.37K for USDB read-and-count and 5.16K for Optimized RocksDB, improving throughput by 1.16×/1.21×. Bandwidth-constrained counting. We fix 128 concurrent queries and 128 workers, then vary the aggregate client receive budget from 1 to 20 Gbit/s across the same three paths (Figure 13b). All client CPUs remain available, and socket counters independently verify the receive rate. Each point has 15 seconds of warm-up and 45 seconds of measurement, repeated three times. Successful QPS is the median active report rate after excluding failed and empty responses. At 1 Gbit/s, pushdown sustains 6.22K QPS versus about 0.73K for either scan path, yielding 8.52×/8.54× higher throughput. At 5 Gbit/s, the gains are 1.63×/1.63×. The scalar response requires only about 10 Mbit/s at 1 Gbit/s budget;

5.1.5 Sustained Reads and Locality. The 30-minute runs at eight clients, 100K, and 15 columns complete with zero errors or empty results. Median QPS is 263/150; cumulative mean latency is 30.69/53.09 ms and p99 51.24/74.81 ms. Figure 12 places the sustained report series beside the population and hot-set scans. Increasing the working set from larger uniform samples to the full sampled workload yields USDB 262–265.5 QPS and baseline 151–155.5. Keeping the same 10K subset while increasing the 1K hot-set share from 80% to 99% raises baseline throughput from 212.5 to 240; USDB changes from 261 to 258. The scan also includes smaller working sets. These distributions vary reuse while keeping the requested output fixed. Time-window selection changes the amount of eligible history (Table 3). One-hour and six-hour windows are empty at the fixed endpoint. One-day nonempty results average about 90 rows, and seven-day results 2359/2819 despite the 100K maximum. The baseline has lower mean latency for short windows; USDB has lower means at three/seven days and lower one/seven-day p99. Reporting coverage and actual rows explains these points without treating a requested maximum as the returned work. 11

USDB

1200 600 0

0 3

Selected columns

(b-d) 15 cols

(d) Long tail Long p99 (ms)

200

(b-d) 2 cols

Short p99 (ms)

400

Optimized RocksDB

(b) Long throughput Long queries/s

Job reads/s

(a) 100K read jobs

8 4 0

0 3

Initial long clients

80 40 0

0 3

Initial long clients

Figure 11: Read jobs and mixed loads. (a) 30K-request jobs on 100K-action histories, eight clients. (b–d) Long-read QPS and short/long p99 for co-started 1K/100K reads sharing 30 clients; each stream runs for up to 60 seconds or 400K requests. Solid/dashed lines indicate two/15 columns. USDB

Optimized RocksDB

200 150

75 60 45

Minutes

(d) Hot sets Queries/s

250

(b) Sustained tail p99 (ms)

Queries/s

(a) Sustained reads

240 200 160

Minutes

25K

Full

260 240 220 5K/80 10K/80 10K/90 10K/99

Working-set size

Population / hot (%)

Figure 12: Read stability and locality, eight clients, 100K, 15 columns. (a–b) Throughput and cumulative p99 from a 30-minute run; the initial report and incomplete final interval are omitted from the traces. (c) Uniform working-set throughput. (d) Throughput versus population/hot-request share; the hot set contains 1K users. Time-window results appear in Table 3.

6 5 4 64

128

160

192

6 4

Metric CPU cores Memory (GiB) Core:memory ratio Throughput (K/s/core) Compute cost Storage cost Total cost

2 0

Concurrent queries USDB pushdown

Table 4: Deployment provisioning and normalized cost. Shared-storage compute is the unit; the compared machine classes reflect their workload roles.

(b) Receive budget Count rate (kQPS)

Count rate (kQPS)

(a) Foreground load

USDB read+count

Receive budget (Gbit/s) Optimized RocksDB

Figure 13: Aggregation throughput with all client CPUs available; count queries. (a) Foreground QPS; filled/open markers: 128/256 workers. (b) QPS versus receive budget, 128 clients/workers. Means ± sample SD (𝑛 = 3).

Shared storage

6284 62840 1:10 0.32 2.73 1.93 4.66

4562 9124 1:2 0.44 1.00 1.70 2.70

2.70 (42.1%). CPU provisioning falls from 6284 to 4562 cores and memory from 62840 to 9124 GiB. The larger reduction in compute and memory reflects removing repeated maintenance from reader provisioning and selecting hardware by role. Persistent copies, writer compute, reader caches, and retention remain explicit cost components. The organization also lets an additional reader reuse existing historical files rather than first create a complete private shard image. This connects the publication protocol in Section 4.3 to an operational benefit beyond request throughput.

ordinary reads exhaust the smaller budgets and approach unrestricted throughput as the budget increases. Pushdown preserves throughput when result transfer limits ordinary reads.

5.3

Local disk

Replica Provisioning and Cost

Differentiated replicas change the provisioning unit from a complete private storage-and-maintenance stack to a workload role. Table 4 reports deployment accounting for local disk on memoryoriented machines and shared storage on balanced machines. At a normalized 1M reads/s and 50% CPU utilization, compute cost falls from 2.73 to 1.00, storage from 1.93 to 1.70, and total from 4.66 to

5.4

Reconstruction Consistency

The application study compares complete-sequence agreement with and without arrival-version filtering over two hours (Figure 14). All actions in a reconstructed sequence must agree with 12

100 50 0

60 90 Elapsed time (min)

120

ZSTD

Prior layout

3 2 1 512

3 2 1

512

Rows per group

Figure 15: Row-group size versus lookup latency and file size, normalized to the prior layout (dotted reference). K denotes thousands of rows per group.

Figure 14: Versioned reconstruction. Complete-sequence agreement over two hours, with and without MVCC. the inference input for a match. Across the 61 reported observations, matching is 97–100% with MVCC and 16–25% without it; 27 MVCC observations reach 100%. The later historical view contains observations that were unavailable to inference, and the saved arrival-version boundary selects the relevant state. Partial dumps capture recent state directly, while the versioned history supplies its reusable prefix. One production policy captures the most recent day, covering the dominant late-arrival interval and the typical outagedetection/recovery window. The deployment reports a captured sample fraction of 10–20%, motivating reuse of the older history during reconstruction. The following accounting includes the captured fragment and its metadata explicitly. For 𝑁 samples, let 𝐹 denote average encoded bytes of a full sequence, 𝑃 the captured sequence bytes, 𝑀 reconstruction metadata, and 𝑂 unchanged sample features. Sample storage becomes 𝐵 full = 𝑁 (𝑂 + 𝐹 ),

LZ4

Relative file size

Without MVCC

Relative lookup latency

Complete match (%)

With MVCC

Table 5: Codec measurements. Ratio is encoded/original size; throughput is measured at the codec boundary. Group Method Ratio Encode MB/s Decode GB/s 1K 8K 256K

5.5

USDB LZ4 ZSTD USDB LZ4 ZSTD USDB LZ4 ZSTD

12.8% 13.8% 9.9% 12.8% 13.1% 9.0% 11.0% 12.6% 8.6%

154.5 1950.1 637.6 407.1 2072.7 1083.8 420.4 2403.4 1297.9

9.05 5.36 1.66 13.80 5.38 2.57 14.88 6.88 3.21

Compression Performance

The physical-layout study uses a sample of production histories, keeping key columns separate and grouping value columns. Its reference layout takes 9994.45 𝜇s and 1.9G. Figure 15 compares nine configurations: LZ4/ZSTD with 512, 1,024, 2,048, or 4,096 rows per group, plus the reference layout. At 4K rows, LZ4 takes 9957.65 𝜇s and 2.1G, approaching reference latency with 10.5% more bytes; ZSTD uses 1.9G at 14045.65 𝜇s. At 512 rows, LZ4 takes 26269.50 𝜇s and 6.6G. Finer groups increase repeated metadata and fragmented reading in this study. The complete size/latency curve explains why grouping must be selected with the user-history access pattern, complementing the controlled projection scan. The codec benchmark (Table 5) shows the materialize-once/readrepeatedly choice. USDB decodes 4.6–5.5× faster than ZSTD, using 27.9–42.2% more encoded bytes and lower encode throughput. At 256K groups, it decodes at 14.88 GB/s versus 6.88 for LZ4 and 3.21 for ZSTD. Background construction amortizes encoding over repeated queries; the projection and read-job results evaluate the corresponding application-facing path.

𝐵 partial = 𝑁 (𝑂 + 𝑃 + 𝑀).

The sequence-payload reduction is 1−(𝑃+𝑀)/𝐹 , and entire-sample reduction is (𝐹 −𝑃−𝑀)/(𝑂+𝐹 ). This accounting exposes the benefit directly: overlapping historical prefixes are retained once in shared history rather than copied into every sample. Overall deployment accounting additionally includes the retained versions and detailed history used for reconstruction. Using mean captured/original lengths 195.7/2950, the storage model yields a 6.6% captured-length ratio. A separate 24-hour pipeline study of 2K sequences compares nine dump-window policies. One policy captures at most 660 actions under 30-second averaging, with item–timestamp agreement of 0.999–1.000; both metrics report the worst window of the day. These results guide the captured window toward recent pipeline delays while reusing the larger historical prefix. Length quantifies capture volume; item– timestamp agreement measures pipeline alignment; completesequence agreement measures reconstruction correctness. The controlled read jobs quantify the retrieval side of this design. Their 100K completion improvement persists across narrow and wide projections, showing how lower lookup time can support a sample pipeline that materializes history later. The storage model and read experiments thus address complementary parts of the same lifecycle: reduce repeated bytes at sample creation and supply reusable history efficiently at consumption.

5.6

Direct-File Training Access

The production training study compares service-mediated access and direct shared-file reads for the same model, with 2,000 datareader workers and 16 training nodes in both configurations and caching enabled. Trainer-reported throughput increases from 175K to 184K instances/s (5.1%), while weighted training-job CPU utilization remains about 32%. Cache hit rate increases from 92% to 96%; mean uncached sequence-fetch latency is 3.3/3.4 ms. Directfile access reuses the materialized historical representation in the training pipeline. These training results complement the controlled 13

5.8

fixed-work reads in Section 5.1, which measure client reading separately from model computation.

5.7

Deployment Lessons

Share the materialization investment. Aggregation and encoding both spend work during background construction to reduce subsequent reading. Their operating points should reflect repeated consumption and the read/write mix, alongside retained size. The production traces, codec study, and long-read jobs show these choices at their respective boundaries. Provision for useful work. Length and width change payload, UID reuse changes caching benefit, and concurrency changes throughput and tail latency. The client study finds little 100K throughput benefit beyond 96 clients; additional workers can increase CPU without increasing reads. Matching these needs to a replica role makes shared capacity useful to each consumer. Make history part of the sample contract. Recording selection and visibility together lets training reuse history while preserving the recent inference observation. Event time, ingestion order, and publication identify different parts of that state. Together, they give backfill and feature evolution a stable read interface. Future work extends aggregation over selected histories and overlaps historical prefetching with model computation. End-toend model jobs can quantify these interactions within each training pipeline.

Production at Scale

A September 2026 deployment record lists approximately 18 PB of retained history per region, including two copies, across three regions for one application family. This operational footprint complements the representative demand stages in Figure 1. The migration study below uses full-day records from an earlier deployment. The production migration replaced read-before-write sequence updates in separate local-disk streaming/batch stores. Its legacy deployment had 256 nodes with 32 cores and 100-Gbps interfaces, retaining about 20 TB of streaming and 600 TB of batch data including replication. This earlier migration baseline is a deployed Lambda-style sequence service with separate streaming/batch databases, user indexing, and aggregation. Optimized RocksDB in the controlled study is the subsequent optimized sequence service and supports projection; the new comparison uses both systems’ deployed read paths. Foreground efficiency. Across the complete 24-hour records for each deployment, mean write latency falls from 123.94 to 17.56 𝜇s (85.8%), and mean CPU utilization from 29.04% to 7.95% (72.6%). Mean read latency remains near 410 𝜇s, changing from 409.71 to 406.15; CPU peaks change from 64.3% to 14.7%. Figure 16 shows the complete daily time series. Foreground append decouples each update from full-history rewriting; background materialization preserves user-local read access. A write contributes one action, whereas an inference read typically retrieves about 1K actions in this workload. Read requests account for roughly 10–20% of write volume, making the write path a primary source of resource pressure. Maintenance and cache choices. The single-shard experiment uses a 32-core, 256-GB machine and random reads averaging 1.02K actions. At 12K reads/s, memory merge-on-read uses 48.8% CPU versus legacy’s 69%, with write latency 20 versus 86 𝜇s and read p99 1.437 versus 1.474 ms. The configuration study makes pending state and caching visible: the disk path with selective caching reaches 89% CPU and 15.617-ms p99 at 11.5K reads/s. Dedicated materialization and read-cache provisioning determine the operating point together. Engine write amplification is 55 for legacy and 66 for the USDB configurations; foreground append latency captures the benefit of moving application-level sequence rewriting into background maintenance. Long histories. The production long-read study also shows the shift in the cost balance (Figure 17). At 5K actions, mean latency is 7.28/2.71 ms for USDB/legacy. At 100K, it is 45.5/77.5 ms and p99 is 68.8/98.5 ms. The controlled comparison in Section 5.1 extends this evidence with a stronger deployed baseline, common columns, repetitions, and crossed workload parameters. Both studies make retained history length central to interpreting read performance.

6 RELATED WORK Recommendation and sequence infrastructure. Recommendation research spans collaborative filtering, feature engineering, and neural behavior modeling [3, 29, 36, 55, 56]. Wide&Deep, YouTube, and DIN established deep-model architectures [11, 14, 76]; training systems co-design software and hardware for their access patterns [49]. Generative and sequenceoriented models demand longer histories and more computation [8, 27, 41, 65, 74, 75], consistent with scaling laws [32]. Feature stores organize definitions, pipelines, and online/offline access [19, 43, 50]. TokaDB unifies training-serving data management [23]; Magnus integrates columnar ML data, versioning, and updates [59]; Krypton combines real-time serving and analytical SQL [10]; ABase supports multi-tenant workloads [31]. USDB couples incremental maintenance, shared histories, and inference-state reconstruction. Reconstruction and streaming state. Guo et al. [28] use versioned late materialization to separate reusable interaction history from training samples, with stream/batch visibility and projected prefetching. USDB’s partial dumps retain a recent inference fragment and reconstruct the older prefix from version-compatible history. Fresh reads reuse the same maintenance and replicas. Lambda-style architectures and unified stream processors address complementary execution concerns [7, 34, 67, 72]; Flink 2.0 and Ursa extend state separation and lakehouse integration [46, 47]. USDB coordinates user-history aggregation and inference visibility in shared state. LSM maintenance and access. LSM-trees and RocksDB supply the write-optimized foundation [21, 51]. Research explores compaction structures and workload-dependent tuning [16, 18, 30, 42, 48, 57, 70]; filters, learned indexes, and joins improve access within these structures [15, 17, 33, 45, 71]. WiscKey and PebblesDB reorganize values or sorted runs to reduce storage

Request outcomes in production. A separate 24-hour production window shows request success of 99.894–99.986% and client-error volume of 116–661 requests/s (Figure 18). The trace contains 282 paired five-minute observations; six missing intervals appear as gaps. Together with the latency and resource curves, these outcomes characterize the deployment’s operating point. 14

(a-c) Legacy (before)

(b) Production reads

160 80 0

450 350 250

Hour of day

50 25 0

Hour of day

(d) Write (μs) Legacy

550

CPU (%)

Latency (μs)

(a) Production writes

(a-c) USDB (after)

(e) Mean read (ms)

Hour of day

(f) Read p99 (ms)

0.42

0.48

0.51

1.30

1.42

(g) CPU (%)

1.47

Disk / full

15.6

0.40

0.50

0.55

1.16

1.52

1.59

Disk / selective*

13.8

0.93

1.05

3.58

1.84

2.17

15.62

Memory merge

0.35

0.41

0.48

1.02

1.24

1.44

48.8

12*

Read load (K queries/s)

Figure 16: Production operation and replica policies. (a–c) Full-day before/after records in five-minute averages. (d–g) Policy/load measurements; darker cells rank higher cost within each metric. Disk rows use full/selective caching; memory uses merge-on-read. *Selective caching uses 11.5K reads/s in the last column; other rows use 12K. Legacy

Latency (ms)

(b) p99

100

tables [26]. CascadeLSM places application-defined sequence aggregation at historical materialization boundaries. The resulting base is reused by serving and training. Distributed and disaggregated storage. Dynamo, Cassandra, Bigtable, and HBase establish distributed key-value and wide-table designs [9, 20, 37, 63]. Spanner, Aurora, and PolarDB explore distributed consistency and cloud resource separation [6, 13, 61, 68]. RocksDB disaggregation, CaaS-LSM, dLSM, LegoOS, and O3-LSM separate storage, compute, or maintenance duties [22, 40, 58, 66, 69]; GFS, HDFS, and Tectonic provide sharedfile foundations [4, 24, 52]. USDB separates replica duties and reconciles shared history with fresh increments, complementing established consistency and availability contracts [1, 5, 25, 53]. Columnar layouts and encoding. PAX, C-Store, Vertica, and Parquet improve locality and selective access [2, 38, 60, 62]. BtrBlocks, Vortex, and Bullion develop adaptive encodings and formats for modern analytics or ML [35, 39, 64]. F3 embeds portable decoders for extensible columnar representations [73]; OpenZL composes specialized codecs through a graph model and universal decoder [12]. USDB combines user indexing and columnar row groups for shared serving and projected training reads.

USDB

(a) Mean

100

Actions (K)

100

Actions (K)

Figure 17: Migration-era long reads. Mean and p99 latency across seven sequence lengths.

(a) Request success (%) 100.00 99.94 99.88

(b) Client errors (requests/s) 600

7 CONCLUSION

300

USDB shares materialized history across ingestion, inference, and training. Time-based compaction constructs reusable bases; differentiated replicas share their files; partial dumps reconstruct versioned inputs; and indexed columnar access accelerates selected histories. Production records show 85.8% lower mean write latency and 72.6% lower mean CPU with stable mean read latency; provisioning reduces normalized cost by 42.1%. Across 374 controlled runs, repeated 10K/100K reads improve throughput 1.79×/1.75×, and longer 100K client-read jobs complete 1.62–2.52× faster. Sharing the sequence lifecycle supports both fresh serving and growing historical access.

8 12 16 Elapsed time (h)

Figure 18: Production request outcomes over 24 hours. Gaps mark missing observations. work [44, 54], while AutoComp automates maintenance for lake

REFERENCES

[22] Siying Dong, Shiva Shankar P, Satadru Pan, Anand Ananthabhotla, Dhanabal Ekambaram, Abhinav Sharma, Shobhit Dayal, Nishant Vinaybhai Parikh, Yanqin Jin, Albert Kim, et al. 2023. Disaggregating rocksdb: A production experience. Proceedings of the ACM on Management of Data 1, 2 (2023), 1–24. [23] Peng Fang, Kelei Guo, Cheng Chen, Wei Zhang, Mingming Chen, Huaye Xu, Borong Meng, Zongjia Chen, Mingshuai Wang, Luping Wang, Yuan Zhang, Shiru Ren, Fang Wang, and Dan Feng. 2026. TokaDB: A Unified Storage Engine for Training-Serving Data Management in Large Recommendation Models. In Companion of the International Conference on Management of Data. ACM, New York, NY, USA, 293–306. https://doi.org/10.1145/3788853.3803078 [24] Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. The Google file system. In Proceedings of the nineteenth ACM symposium on Operating systems principles. 29–43. [25] Seth Gilbert and Nancy Lynch. 2002. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. Acm Sigact News 33, 2 (2002), 51–59. [26] Anja Gruenheid, Jesús Camacho-Rodríguez, Carlo Curino, Raghu Ramakrishnan, Stanislav Pak, Sumedh Sakdeo, Lenisha Gandhi, Sandeep K. Singhal, Pooja Nilangekar, and Daniel J. Abadi. 2025. AutoComp: Automated Data Compaction for Log-Structured Tables in Data Lakes. In Companion of the 2025 International Conference on Management of Data. 404–417. https://doi.org/10.1145/3722212. 3724430 [27] Lin Guan, Jia-Qi Yang, Zhishan Zhao, Beichuan Zhang, Bo Sun, Xuanyuan Luo, Jinan Ni, Xiaowen Li, Yuhang Qi, Zhifang Fan, et al. 2025. Make It Long, Keep It Fast: End-to-End 10k-Sequence Modeling at Billion Scale on Douyin. arXiv preprint arXiv:2511.06077 (2025). [28] Liang Guo, Ge Song, Litao Deng, Jianhui Sun, Chufeng Hu, Lu Zhang, Zhen Ma, Shouwei Chen, Weiran Liu, Sarang Masti Sreeshylan, Xiaoxuan Meng, and Yanzun Huang. 2026. Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale. https://doi.org/10.48550/arXiv. 2604.24806 arXiv:2604.24806 [cs.IR] [29] Zhicheng He, Weiwen Liu, Wei Guo, Jiarui Qin, Yingxue Zhang, Yaochen Hu, and Ruiming Tang. 2023. A survey on user behavior modeling in recommender systems. arXiv preprint arXiv:2302.11087 (2023). [30] Andy Huynh, Harshal A Chaudhari, Evimaria Terzi, and Manos Athanassoulis. 2022. Endure: a robust tuning paradigm for LSM trees under workload uncertainty. Proceedings of the VLDB Endowment 15, 8 (2022), 1605–1618. [31] Rong Kang, Yanbin Chen, Ye Liu, Fuxin Jiang, Qingshuo Li, Miao Ma, Jian Liu, Guangliang Zhao, Tieying Zhang, Jianjun Chen, and Lei Zhang. 2025. ABase: the Multi-Tenant NoSQL Serverless Database for Diverse and Dynamic Workhttps://arxiv.org/abs/2505.07692 loads in Large-scale Cloud Environments. arXiv:2505.07692. [32] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). [33] Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis. 2018. The case for learned index structures. In Proceedings of the 2018 international conference on management of data. 489–504. [34] Jay Kreps. 2014. Questioning the lambda architecture. Online article, July 205 (2014), 18–34. [35] Maximilian Kuschewski, David Sauerwein, Adnan Alhomssi, and Viktor Leis. 2023. Btrblocks: Efficient columnar compression for data lakes. Proceedings of the ACM on Management of Data 1, 2 (2023), 1–26. [36] Weijiang Lai, Beihong Jin, Yapeng Zhang, Yiyuan Zheng, Rui Zhao, Jian Dong, Jun Lei, and Xingxing Wang. 2025. Modeling Long-term User Behaviors with Diffusion-driven Multi-interest Network for CTR Prediction. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 289–298. [37] Avinash Lakshman and Prashant Malik. 2010. Cassandra: a decentralized structured storage system. ACM SIGOPS operating systems review 44, 2 (2010), 35–40. [38] Andrew Lamb, Matt Fuller, Ramakrishna Varadarajan, Nga Tran, Ben Vandier, Lyric Doshi, and Chuck Bear. 2012. The vertica analytic database: C-store 7 years later. arXiv preprint arXiv:1208.4173 (2012). [39] Gang Liao, Ye Liu, Jianjun Chen, and Daniel J. Abadi. 2025. Bullion: A Column Store for Machine Learning. In 15th Annual Conference on Innovative Data Systems Research (CIDR). Amsterdam, The Netherlands. https://vldb.org/cidrdb/ papers/2025/p26-liao.pdf [40] Qi Lin, Gangqi Huang, Te Guo, Chang Guo, Viraj Thakkar, Zhichen Zhu, Jianguo Wang, and Zhichao Cao. 2026. O3-LSM: Maximizing Disaggregated LSM Write Performance via Three-Layer Offloading. Proceedings of the ACM on Management of Data 4, 3, Article 216 (June 2026), 27 pages. https://doi.org/10.1145/ 3802093 [41] Tianhe Lin, Ziwei Xiong, Baoyuan Ou, Yingjie Qin, Lai Xu, Xiaocheng Zhong, Yao Hu, Zhiyong Wang, Tao Zhou, Yubin Xu, and Di Wu. 2026. LASER: An Efficient Target-Aware Segmented Attention Framework for Endto-End Long Sequence Modeling. https://doi.org/10.48550/arXiv.2602.11562 arXiv:2602.11562v1 [cs.IR] arXiv preprint, version 1, 12 February 2026.

[1] Daniel Abadi. 2012. Consistency tradeoffs in modern distributed database system design: CAP is only part of the story. Computer 45, 2 (2012), 37–42. [2] Anastassia Ailamaki, David J DeWitt, Mark D Hill, and Marios Skounakis. 2001. Weaving Relations for Cache Performance.. In VLDB, Vol. 1. 169–180. [3] Jesús Bobadilla, Fernando Ortega, Antonio Hernando, and Abraham Gutiérrez. 2013. Recommender systems survey. Knowledge-based systems 46 (2013), 109– 132. [4] Dhruba Borthakur et al. 2008. HDFS architecture guide. Hadoop apache project 53, 1-13 (2008), 2. [5] Eric A Brewer. 2000. Towards robust distributed systems. In PODC, Vol. 7. Portland, OR, 343–477. [6] Wei Cao, Yang Liu, Zhushi Cheng, Ning Zheng, Wei Li, Wenjie Wu, Linqiang Ouyang, Peng Wang, Yijing Wang, Ray Kuan, et al. 2020. { POLARDB } meets computational storage: Efficiently support analytical workloads in { CloudNative } relational database. In 18th USENIX conference on file and storage technologies (FAST 20). 29–41. [7] Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering 38, 4 (2015). [8] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al. 2025. Longer: Scaling up long sequence modeling in industrial recommenders. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 247–256. [9] Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C Hsieh, Deborah A Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E Gruber. 2008. Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS) 26, 2 (2008), 1–26. [10] Jianjun Chen, Rui Shi, Heng Chen, Li Zhang, Ruidong Li, Wei Ding, Liya Fan, Hao Wang, Mu Xiong, Yuxiang Chen, et al. 2023. Krypton: real-time serving and analytical SQL engine at ByteDance. Proceedings of the VLDB Endowment 16, 12 (2023), 3528–3542. [11] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7–10. [12] Yann Collet, Nick Terrell, W. Felix Handte, Danielle Rozenblit, Victor Zhang, Kevin Zhang, Yaelle Goldschlag, Jennifer Lee, Elliot Gorokhovsky, Yonatan Komornik, Daniel Riegel, Stan Angelov, and Nadav Rotem. 2025. OpenZL: A Graphhttps://doi.org/10.48550/arXiv.2510.03203 Based Model for Compression. arXiv:2510.03203v2 arXiv preprint, version 2. [13] James C Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, Jeffrey John Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, et al. 2013. Spanner: Google’s globally distributed database. ACM Transactions on Computer Systems (TOCS) 31, 3 (2013), 1–22. [14] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198. [15] Yifan Dai, Yien Xu, Aishwarya Ganesan, Ramnatthan Alagappan, Brian Kroth, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2020. From { WiscKey } to bourbon: A learned index for { Log-Structured } merge trees. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 155– 171. [16] Niv Dayan, Manos Athanassoulis, and Stratos Idreos. 2017. Monkey: Optimal navigable key-value store. In Proceedings of the 2017 ACM International Conference on Management of Data. 79–94. [17] Niv Dayan, Manos Athanassoulis, and Stratos Idreos. 2018. Optimal bloom filters and adaptive merging for LSM-trees. ACM Transactions on Database Systems (TODS) 43, 4 (2018), 1–48. [18] Niv Dayan and Stratos Idreos. 2018. Dostoevsky: Better space-time trade-offs for LSM-tree based key-value stores via adaptive removal of superfluous merging. In Proceedings of the 2018 International Conference on Management of Data. 505– 520. [19] Javier de la Rúa Martínez, Fabio Buso, Antonios Kouzoupis, Alexandru A Ormenisan, Salman Niazi, Davit Bzhalava, Kenneth Mak, Victor Jouffrey, Mikael Ronström, Raymond Cunningham, et al. 2024. The hopsworks feature store for machine learning. In Companion of the 2024 International Conference on Management of Data. 135–147. [20] Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon’s highly available keyvalue store. ACM SIGOPS operating systems review 41, 6 (2007), 205–220. [21] Siying Dong, Andrew Kryczka, Yanqin Jin, and Michael Stumm. 2021. Rocksdb: Evolution of development priorities in a key-value store serving large-scale applications. ACM Transactions on Storage (TOS) 17, 4 (2021), 1–32.

[42] Junfeng Liu, Fan Wang, Dingheng Mo, and Siqiang Luo. 2024. Structural designs meet optimality: Exploring optimized lsm-tree structures in a colossal configuration space. Proceedings of the ACM on Management of Data 2, 3 (2024), 1–26. [43] Rui Liu, Kwanghyun Park, Fotis Psallidas, Xiaoyong Zhu, Jinghui Mo, Rathijit Sen, Matteo Interlandi, Konstantinos Karanasos, Yuanyuan Tian, and Jesús Camacho-Rodríguez. 2023. Optimizing Data Pipelines for Machine Learning in Feature Stores. Proceedings of the VLDB Endowment 16, 13 (2023), 4230–4239. [44] Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Hariharan Gopalakrishnan, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2017. Wisckey: Separating keys from values in ssd-conscious storage. ACM Transactions On Storage (TOS) 13, 1 (2017), 1–28. [45] Siqiang Luo, Subarna Chatterjee, Rafael Ketsetsidis, Niv Dayan, Wilson Qin, and Stratos Idreos. 2020. Rosetta: A robust space-time optimized range filter for keyvalue stores. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 2071–2086. [46] Yuan Mei, Zhaoqian Lan, Lei Huang, Yanfei Lei, Han Yin, Rui Xia, Kaitian Hu, Paris Carbone, Vasiliki Kalavri, and Feng Wang. 2025. Disaggregated State Management in Apache Flink 2.0. Proceedings of the VLDB Endowment 18, 12 (2025), 4846–4859. https://doi.org/10.14778/3750601.3750609 [47] Matteo Merli, Sijie Guo, Penghui Li, Hang Chen, and Neng Lu. 2025. Ursa: A Lakehouse-Native Data Streaming Engine for Kafka. Proceedings of the VLDB Endowment 18, 12 (2025), 5184–5196. [48] Dingheng Mo, Fanchao Chen, Siqiang Luo, and Caihua Shan. 2023. Learning to optimize lsm-trees: Towards a reinforcement learning based key-value store for dynamic workloads. Proceedings of the ACM on Management of Data 1, 3 (2023), 1–25. [49] Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, et al. 2022. Software-hardware co-design for fast and scalable training of deep learning recommendation models. In Proceedings of the 49th Annual International Symposium on Computer Architecture. 993–1011. [50] Laurel Orr, Atindriyo Sanyal, Xiao Ling, Karan Goel, and Megan Leszczynski. 2021. Managing ML pipelines: feature stores and the coming wave of embedding ecosystems. Proceedings of the VLDB Endowment 14, 12 (2021), 3178–3181. [51] Patrick O’Neil, Edward Cheng, Dieter Gawlick, and Elizabeth O’Neil. 1996. The log-structured merge-tree (LSM-tree). Acta informatica 33, 4 (1996), 351–385. [52] Satadru Pan, Theano Stavrinos, Yunqiao Zhang, Atul Sikaria, Pavel Zakharov, Abhinav Sharma, Mike Shuey, Richard Wareing, Monika Gangapuram, Guanglei Cao, et al. 2021. Facebook’s tectonic filesystem: Efficiency from exascale. In 19th USENIX Conference on File and Storage Technologies (FAST 21). 217–231. [53] Dan Pritchett. 2008. BASE: An Acid Alternative: In partitioned databases, trading some consistency for availability can lead to dramatic improvements in scalability. Queue 6, 3 (2008), 48–55. [54] Pandian Raju, Rohan Kadekodi, Vijay Chidambaram, and Ittai Abraham. 2017. Pebblesdb: Building key-value stores using fragmented log-structured merge trees. In Proceedings of the 26th Symposium on Operating Systems Principles. 497– 514. [55] Paul Resnick and Hal R Varian. 1997. Recommender systems. Commun. ACM 40, 3 (1997), 56–58. [56] Francesco Ricci, Lior Rokach, and Bracha Shapira. 2021. Recommender systems: Techniques, applications, and challenges. Recommender systems handbook (2021), 1–35. [57] Subhadeep Sarkar, Dimitris Staratzis, Zichen Zhu, and Manos Athanassoulis. 2022. Constructing and analyzing the LSM compaction design space (updated version). arXiv preprint arXiv:2202.04522 (2022). [58] Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. 2018. { LegoOS } : A disseminated, distributed { OS } for hardware resource disaggregation. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 69–87. [59] Jun Song, Jingyi Ding, Irshad Kandy, Yanghao Lin, Zhongjia Wei, Zilong Zhou, Zhiwei Peng, Jixi Shan, Hongyue Mao, Xiuqi Huang, Xun Song, Cheng Chen, Yanjia Li, Tianhao Yang, Wei Jia, Xiaohong Dong, Kang Lei, Rui Shi, Pengwei Zhao, and Wei Chen. 2025. Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads. Proceedings of the VLDB Endowment 18, 12 (2025), 4964–4977. https://doi.org/10.14778/3750601.3750620 [60] Mike Stonebraker, Daniel J Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O’Neil, et al. 2018. C-store: a column-oriented DBMS. In Making Databases Work: the Pragmatic Wisdom of Michael Stonebraker. 491–518. [61] Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, and Xiaofeng Bao. 2017. Amazon aurora: Design considerations for high throughput cloud-native relational databases. In Proceedings of the 2017 ACM International Conference on Management of Data. 1041–1052. [62] Deepak Vohra. 2016. Apache parquet. In Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools. Springer, 325–335. [63] Mehul Nalin Vora. 2011. Hadoop-HBase for large-scale data. In Proceedings of 2011 International Conference on Computer Science and Network Technology,

Vol. 1. IEEE, 601–605. [64] Vortex Contributors. 2024. Vortex: An Extensible, State-of-the-Art Columnar File Format. https://github.com/vortex-data/vortex. Accessed: 2025. [65] Chunqi Wang, Bingchao Wu, Zheng Chen, Lei Shen, Bing Wang, and Xiaoyi Zeng. 2025. Scaling transformers for discriminative recommendation via generative pretraining. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2893–2903. [66] Ruihong Wang, Jianguo Wang, Prishita Kadam, M Tamer Özsu, and Walid G Aref. 2023. dlsm: An lsm-based index for memory disaggregation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). IEEE, 2835–2849. [67] James Warren and Nathan Marz. 2015. Big Data: Principles and best practices of scalable realtime data systems. Simon and Schuster. [68] Xinjun Yang, Yingqiang Zhang, Hao Chen, Feifei Li, Bo Wang, Jing Fang, Chuan Sun, and Yuhui Wang. 2024. PolarDB-MP: A Multi-Primary Cloud-Native Database via Disaggregated Shared Memory. In Companion of the 2024 International Conference on Management of Data. 295–308. https://doi.org/10.1145/3626246. 3653377 [69] Qiaolin Yu, Chang Guo, Jay Zhuang, Viraj Thakkar, Jianguo Wang, and Zhichao Cao. 2024. CaaS-LSM: compaction-as-a-service for LSM-based key-value stores in storage disaggregated infrastructure. Proceedings of the ACM on Management of Data 2, 3 (2024), 1–28. [70] Weiping Yu, Siqiang Luo, Zihao Yu, and Gao Cong. 2024. CAMAL: Optimizing LSM-trees via Active Learning. Proceedings of the ACM on Management of Data 2, 4 (2024), 1–26. [71] Weiping Yu, Fan Wang, Xuwei Zhang, and Siqiang Luo. 2024. Are Joins over LSM-Trees Ready? Take RocksDB as an Example. Proceedings of the VLDB Endowment 18, 4 (2024), 1077–1090. [72] Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. 2013. Discretized streams: Fault-tolerant streaming computation at scale. In Proceedings of the twenty-fourth ACM symposium on operating systems principles. 423–438. [73] Xinyu Zeng, Ruijun Meng, Martin Prammer, Wes McKinney, Jignesh M. Patel, Andrew Pavlo, and Huanchen Zhang. 2025. F3: The Open-Source Data File Format for the Future. Proceedings of the ACM on Management of Data 3, 4, Article 245 (Sept. 2025), 27 pages. https://doi.org/10.1145/3749163 [74] Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations. arXiv preprint arXiv:2402.17152 (2024). [75] Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, et al. 2024. Recommender systems in the era of large language models (llms). IEEE Transactions on Knowledge and Data Engineering 36, 11 (2024), 6889–6907. [76] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1059–1068.

USDB: Scaling Storage for Next-Generation Recommender Systems at ByteDance

Analyze & explore related research in Conceptio

Related documents