arXiv CSopen access

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training Ting Sun† , Junjie Zhang† , Xiao Yan‡ , Songxin Zhang† , Zhuoyang Song† Jingyi Xi† , Zunyao Mao† , Bingyi Jing§ , Jiaxing Zhang†∗ , Zejian Xie†∗ † Lionrock AI Lab, China Merchants Group, Hong Kong, China

arXiv:2605.09994v1 [cs.DC] 11 May 2026

‡ Wuhan University, Wuhan, China; § The Chinese University of Hong Kong, Shenzhen, China

{sunting2,junjayzhang,zhangsongxin,songzhuoyang}@cmhk.com {xijingyi,maozunyao,zhangjiaxing,xiezejian}@cmhk.com [email protected]; [email protected] ∗ Corresponding authors

Abstract Modern Large Foundation Model (LFM) training has transformed the data pipeline from a static ingestion layer into a dynamic component that must co-evolve with the training process. Existing systems are ill-equipped: colocated dataloaders offer no failure isolation, while message queue-based disaggregated dataloaders operate on a record/offset abstraction that cannot express the batch-level semantics required by distributed training. We present Lakestream, a brokerless, object-store-native training data plane with three key properties. First, it introduces the Transactional Global Batch (TGB), which builds on lakehouse-style ACID storage semantics and extends them with training-specific consistency, including atomic all-rank batch visibility, a globally ordered step sequence, checkpoint-aligned lifecycle management, and end-to-end exactly-once recovery. Second, it realizes recovery and retention directly in the storage layer, by inlining producer state in the manifest and tying reclamation to distributed checkpoint state. Third, its Decentralized Adaptive Commit (DAC) algorithm sustains stable ingestion throughput as the manifest grows, without any inter-producer communication. Evaluations on large-scale multimodal pre-training and SFT workloads using 64 GPUs show that Lakestream outperforms colocated dataloader throughput while providing full failure isolation, outperforms Apache Kafka in ingestion throughput, and achieves lower consumer read latency than Kafka.

Introduction

Data loading has long been a well-studied component of model training. In conventional deep learning workloads, the data pipeline is straightforward: a static, pre-processed dataset is shuffled offline, partitioned into fixed-size batches, and streamed to the training process at a predictable rate. Systems such as tf.data [26] and MosaicML Streaming [25] are designed around this model, optimizing for throughput and shuffle quality over a fixed corpus. To prevent preprocessing from bottlenecking GPU utilization, disaggregated variants offload preprocessing to independent worker pools: tf.data service [5] and Cachew [12] run preprocessing on separate CPU nodes and deliver batches to training ranks via RPC, while CoorDL [23] coordinates cache sharing across colocated training jobs. These systems improve resource efficiency, but all share a common assumption: the dataset is static, batch membership is determined

before training begins, and batches are ephemeral delivery artifacts that are discarded once consumed. Large Foundation Model (LFM) training breaks these assumptions. The data pipeline is no longer a passive feeder of a fixed corpus but a dynamic, runtime-coupled component that must coevolve with the training process. It is dynamic in three aspects that existing systems are not designed to handle. • Data volume. In multimodal pre-training and supervised finetuning with video or interleaved image-text data, raw inputs undergo heavy runtime preprocessing before reaching the optimizer. Video decoding, frame extraction, resolution rescaling, and token packing all introduce data expansion whose volume depends on the input content and current model configuration. Intermediate data can exceed the raw dataset size by one to three orders of magnitude. This unpredictable expansion makes static resource allocation infeasible and demands that preprocessing be decoupled from training, with elastic storage to absorb runtime-materialized data. • Batch membership. In workloads that use online token packing or model-dependent sample selection (such as long-context continued pre-training or reinforcement learning from human feedback), the preprocessing pipeline itself runs at training time, and the exact composition of each batch is determined by its output. Batch boundaries are known only after preprocessing completes; no static partitioning scheme can capture them in advance. The data plane must therefore treat batch membership as a runtime artifact and provide a mechanism to expose complete batches atomically to all training ranks. • Checkpoint alignment. Modern LFM pipelines interleave pretraining, fine-tuning, and reinforcement learning stages, each of which may require rolling back to an earlier checkpoint or replaying a specific batch sequence. Rather than a form of dynamism per se, this reflects a requirement for persistence and versioning: the data plane must maintain a durable, checkpointaligned batch history so that any live checkpoint can deterministically recover the exact data it consumed. Ephemeral delivery systems cannot provide this guarantee. These requirements call for a training-aware data plane that understands the semantic structure of training steps, not one that merely transports individual records. Existing solutions do not meet this bar. Colocated dataloaders provide natural semantic alignment but no failure isolation: a preprocessing crash stalls the entire job.

Sun et al.

They are also subject to structural resource contention, where preprocessing threads share CPU cycles and memory bandwidth with the training process on the same node, imposing a throughput ceiling that in-rank engineering cannot overcome under heavy preprocessing workloads. Disaggregated dataloaders with centralized services (tf.data service, Cachew) decouple preprocessing from training and support elasticity, but deliver batches ephemerally via RPC with no persistent history and no batch-level atomicity guarantees. Neither category can express the training-aware semantics that LFM workloads require: a batch spanning multiple dynamically produced objects must be either fully visible or fully invisible to all data-parallel ranks at once, and its membership must be recoverable across failures and rollbacks. Object storage is a natural substrate for a disaggregated training data plane. Its append-only write model, elastic scalability, and decentralized access require no broker provisioning. Inspired by lakehouse architectures [3, 19, 35], we propose to govern data visibility through versioned metadata manifests rather than record offsets, separating the act of writing data from the act of exposing it to consumers. A new training job requires only a fresh namespace prefix, with no partition assignment and no cold-start overhead. Object storage alone, however, provides only single-object atomicity: it offers no batch-level transactional semantics and no trainingprogress-aware lifecycle management. More fundamentally, a central coordinator is structurally misaligned with LFM training: preprocessing pools are elastic, training jobs require flexible scaling of producers and consumers independently, and any server-side component introduces a new failure domain and provisioning overhead orthogonal to the job. Lakestream’s design principle is therefore that the object store should be the only coordination medium, with no brokers, queues, or services to manage. Doing so requires solving problems at three levels. • Concurrent commit throughput. Decentralized writes rely on Optimistic Concurrency Control (OCC) over a versioned manifest. Under high producer concurrency and a monotonically growing manifest, the system must balance commit conflicts, commit overhead, and ingestion throughput with no interproducer communication. • Batch-level semantics. The system must guarantee atomic visibility of each training batch across all data-parallel ranks, a globally consistent batch ordering, and exactly-once semantics under producer failures and consumer restarts. • Checkpoint-coupled lifecycle. Checkpoint-based rollbacks and cross-run reuse require a versioned batch history, while training-progress-aligned reclamation is needed to bound storage costs without a persistent coordination service. We present Lakestream, a brokerless, object-store-native training data plane. Its core abstraction is the Transactional Global Batch (TGB), which treats each training batch as a first-class persistent entity with atomicity, durability, and globally consistent ordering guarantees. Lakestream materializes TGBs as versioned, manifest-referenced structures on object storage, decoupling data production from training consumption while preserving the semantic boundaries required by distributed optimization. It pairs the TGB abstraction with checkpoint-aligned lifecycle management, endto-end exactly-once semantics, and the Decentralized Adaptive

Commit (DAC) algorithm, which regulates each producer’s commit cadence based on online estimates of commit contention and manifest growth, sustaining stable ingestion throughput without any inter-producer communication. The technical contributions of this paper are as follows. • Brokerless, object-store-native architecture. We establish the object store as the sole coordination medium, with no serverside processes in the critical path. A new training run requires only a fresh namespace prefix; all mechanisms (batch visibility, fault recovery, and lifecycle management) are realized through object store primitives alone (§3). • Transactional Global Batch (TGB) abstraction. We introduce the TGB as a first-class persistent entity with atomicity, durability, and globally consistent ordering, unifying data visibility with training step semantics. All data-parallel ranks observe a globally consistent and atomically visible batch at every optimization step; exactly-once delivery and checkpoint-aligned recovery are achieved entirely through object store primitives (§4, §5.3). • Checkpoint-aligned lifecycle and DAC. We design a lifecycle management mechanism that ties data retention to distributed checkpoint progress, supporting safe rollback, crossrun reuse, and bounded storage overhead. The Decentralized Adaptive Commit (DAC) algorithm sustains stable ingestion throughput as the manifest grows, without any inter-producer communication (§5). We evaluate Lakestream on large-scale multimodal pre-training and SFT workloads using 64 GPUs. Lakestream outperforms the expert-optimized colocated pipeline by 2.68–7.73× in end-to-end throughput while providing full failure isolation, outperforms Kafka in producer ingestion throughput as the number of producers scales, and achieves lower consumer read latency than Kafka at all measured scales. DAC maintains stable throughput across long training runs where fixed-interval strategies degrade due to manifest growth.

2 Background 2.1 Requirements for LFM Training SPMD execution and the Global Batch. Modern LFM training follows the Single Program Multiple Data (SPMD) execution model, where devices are organized into a multi-dimensional device mesh [30, 40, 42]. At each training step 𝑠, the distributed optimizer consumes a Global Batch B𝑠 = {𝑥 1, 𝑥 2, . . . , 𝑥 𝑁 }, a finite set of 𝑁 training samples used to compute the gradient update: 𝜃 𝑠+1 ← 𝜃 𝑠 − 𝜂 ∇L (B𝑠 ; 𝜃 𝑠 ).

(1)

This batch is partitioned across the 𝐷 × 𝐶 data-relevant positions in the device mesh. Data Parallelism (DP) requires each replica to consume an independent subset of samples. Context Parallelism (CP) splits each sample’s token sequence across ranks within a replica, so CP ranks share the same samples but consume different token chunks. Tensor Parallelism (TP) and Pipeline Parallelism (PP) partition model parameters rather than input data, so ranks within the same TP or PP group receive identical input. The data plane therefore needs only to distinguish 𝐷 × 𝐶 positions in the mesh,

where 𝐷 is the DP world size and 𝐶 is the CP degree; TP and PP are transparent to data delivery. This structure imposes two hard requirements on the data plane. First, intra-batch consistency: all ranks must derive their projections from the same B𝑠 . The batch must become atomically visible across the cluster before any rank begins its optimization step. Partial visibility, where some ranks see a complete batch while others see only a prefix, leads to inconsistent gradient aggregation and parameter corruption. Second, inter-batch ordering: the sequence (B1, B2, . . .) must be strictly monotonic and globally agreed upon. If different ranks consume batches in different orders, their local model states diverge immediately. Checkpoint recovery tightens these requirements further. Periodic distributed checkpoints persist the model weights alongside the data-plane state. Upon recovery, training must resume from the exact batch where it left off, consuming the same sequence as in the original run. The batch sequence is therefore not merely a delivery artifact but a durable, replayable history that must be preserved independently of the training process. Runtime-dependent preprocessing. The samples that constitute a Global Batch do not arrive ready to consume. In modern LFM training, raw input passes through a multi-stage preprocessing pipeline before it can be packed into a batch and delivered to the optimizer. This pipeline is runtime-dependent: its compute cost, output volume, and latency are determined by input content and the current training configuration, not by static metadata. Multimodal training provides a representative example. A videotext sample requires video decoding, frame extraction at a resolution set by the model’s current input specification, and spatial or temporal subsampling. The output volume depends on video duration, codec, and target resolution, none of which can be bounded statically. In a LeRobot [9] cloth-folding episode with three 480 × 640 camera streams, the processed episode reaches 22.2 GiB while the raw episode ranges from 364.7 MiB to 2.5 MiB as the video CRF changes, implying 62×–9,068× expansion (Figure 1a). In our GR00T end-to-end experiments, replaying the dataset reader, sample transforms, Eagle batch transform, and Batch.to_bytes() serialization on three adjust_bottle episodes yields 288.3×–5,263.1× readyto-train expansion as resize resolution changes from 224 to 640 and observation history grows from 1 to 4 (Figure 1c); jumps are discrete because visual tokenization cost follows tile-count plateaus. Persample latency is also highly heterogeneous: short and long clips can differ in processing time by orders of magnitude, introducing stragglers that cannot be predicted or balanced statically. A second example is online experience rollout in reinforcement learning from human or verifiable feedback (RLHF/RLVR) [11, 28, 32]. In these workloads, the model itself generates training data by rolling out its current policy; the volume and composition of each rollout batch depend on the model state at that step and cannot be known in advance. The same pattern holds for image-text training: an OpenCLIP [14]-style WebDataset [1] sample of 74.0 KiB expands to 192.6 KiB–3.0 MiB as training resolution changes from 128 to 512, a 2.6×–41.5× expansion (Figure 1b). Across all three cases, preprocessing volume is large, configurationdependent, and unknown until execution. The data plane must accommodate bursty, dynamically sized production decoupled

Processed / raw size

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

(a) LeRobot Episode (b) OpenCLIP + WDS Sample

10,000x

H=1

H=2

H=4

100x

1.0x

12 24 36 48 Video CRF (H.264)

128 224 320 512 224 320 640 Preprocess resolution Resize resolution

Figure 1: Training-time preprocessing inflates data volume by large, configuration-dependent factors. The expansion ratio ranges from 62×–9,068× for LeRobot episodes (varying H.264 CRF), 2.6×–41.5× for OpenCLIP samples (varying resolution), and 288×–5,263× in the GR00T path used in our experiments (varying resize resolution and observation history). Dispatcher

CPU threads

shared memory

Rank 0

Rank 1

assign Worker 1 RPC Rank 0

(a) Colocated Pipeline

assign

Producer 1

Producer 2

write

Persistent Store

Worker 2 RPC Rank 1

read

Rank 0

Rank 1

(b) Centralized Service (c) Persistent Staging

Figure 2: Three architectural patterns for training dataflow. Colocated dataloaders couple preprocessing with training on the same node; centralized services decouple them but deliver batches ephemerally via RPC; Lakestream stages TGBs persistently on object storage, decoupling production from consumption while preserving batch-level semantics. Producer 1

Producer 2

Producer 1

Producer 2

Global Batch a

Global Batch b

Global Batch a

Global Batch b

shard a_1 shard a_2

shard b_1 shard b_2

shard a_1 shard a_2

shard b_1 shard b_2

commit at t

commit at t+1

commit at t+3

commit at t+2 shard a_2

shard b_1 shard a_1

Out of Order Commit

MQ Topic 1

shard b_1 shard b_2 shard a_1 shard a_2

MQ Topic

Redundant Read

shard b_2 shard a_1 shard a_2

shard a_1 shard a_2

MQ Topic 2 Rank 0

Rank 1

(a) Inconsistent Batch Order

Rank 0

Rank 1

(b) Read Amplification

Figure 3: Two structural limitations of message queues for LFM training. (a) Inconsistent Batch Order: concurrent producers commit shards out of order, causing DP ranks to consume shards from different batches at the same step and corrupting gradient aggregation. (b) Read Amplification: delivering a full batch as one message forces every rank to download all 𝐷 shards, consuming only 1/𝐷. from the synchronous demands of training ranks; static resource allocation and pre-partitioning are infeasible.

Sun et al.

2.2

Pitfalls of Existing Solutions

Three architectural patterns have emerged for organizing the training dataflow. All three fall short of the requirements identified in §2.1 for LFM workloads. Colocated pipeline. In a colocated pipeline, preprocessing runs within the same process group as the training ranks, typically on CPU threads or colocated nodes (Figure 2a). Representative systems include PyTorch DataLoader [29] and tf.data [26] in their default configurations, as well as CoorDL [23], which extends this model with coordinated cache sharing across colocated training jobs. This arrangement provides natural semantic alignment: batch boundaries, ordering, and recovery are managed within a single process group. The fundamental limitation is the absence of failure isolation and elastic scaling. A preprocessing failure stalls or terminates the entire training job, wasting the full GPU allocation for the duration of recovery. Since preprocessing and training share the same resource pool, the two cannot be scaled independently. Furthermore, even with careful threading and pipelining, preprocessing threads compete with the training process for CPU cycles and memory bandwidth on the same node, imposing a throughput ceiling that cannot be lifted without architectural decoupling. Disaggregated dataloaders with centralized services. To scale preprocessing independently, disaggregated systems move it to a separate worker pool coordinated by a centralized dispatcher (Figure 2b). Representative systems include tf.data service [5], Cachew [12], FastFlow [36], and MegaScale-Data [40]. This decoupling improves GPU utilization and allows preprocessing resources to scale independently. However, these systems deliver batches ephemerally: once a batch is transferred to the training rank via RPC, it is discarded. There is no persistent batch history, no atomic visibility guarantee across ranks, and no mechanism for replaying a specific batch after a failure or rollback. The centralized dispatcher is also a single point of failure: its crash stalls the entire pipeline, and all coordination traffic passing through it creates a scaling bottleneck. Disaggregated dataloaders with persistent staging. A third approach interposes a persistent storage layer between producers and consumers (Figure 2c). Producers write preprocessed data to a shared store; consumers read from it independently. This design supports failure isolation, elastic scaling, and data persistence. However, realizing correct training semantics on top of such a layer requires solving problems that existing persistent storage abstractions do not address: batch-level atomicity across concurrent producers, globally consistent ordering without a central coordinator, and checkpoint-aligned lifecycle management. These are the challenges that Lakestream is designed to solve, as we detail in §3. General-purpose message queues such as Kafka are a natural candidate for the persistent store, but their record abstraction is fundamentally misaligned with training semantics. As Figure 3 illustrates, MQ systems offer no batch-level primitive: a Global Batch is not an addressable entity but an implicit collection of independent records. This mismatch takes two forms. If each rank-specific shard is delivered as a separate message, out-of-order commits from concurrent producers cause different DP ranks to assemble shards from different batches at the same step, silently corrupting gradient

aggregation (a). If the entire Global Batch is packed into one message, every rank must download the full payload and discard all but its own shard, incurring 𝐷-fold read amplification (b). Imposing batch-level atomicity via a consumer-side barrier or external coordinator would satisfy the ordering requirement but reintroduces a centralized component with its own failure domain, which is precisely what a brokerless design avoids.

2.3

Opportunities With Object Storage

Object storage systems such as Amazon S3, Google Cloud Storage, and Azure Blob Storage exhibit three properties that make them a natural candidate for the persistent staging layer described above. First, objects are written atomically and immutably, which aligns with append-only data materialization. Second, access is fully decentralized: any authorized producer or consumer can read or write without routing through a broker, with no partitions to assign and no capacity to pre-allocate. Third, a new training job requires only a fresh namespace prefix, with no cold-start provisioning; storage capacity scales elastically with usage without operator intervention. Beyond these substrate properties, direct writes support elastic production: producers join and exit without registration, a crash affects only its own in-flight objects, and aggregate throughput scales with the producer pool. Eliminating a central coordinator removes both a failure domain and a scaling ceiling, and is structurally consistent with the SPMD execution model.

Lakestream Overview

Object storage provides durable, decentralized, immutable object writes, but offers no mechanism for expressing relationships between objects, no way to define what constitutes a complete batch, and no notion of training progress. Our key insight is to adapt the versioned-manifest architecture of lakehouse systems [3, 19, 35]: dataset state is a monotonically growing sequence of immutable manifest files, and successful publication of a new version is serialized by a conditional write on the next version name. Lakehouse manifests encode partition layouts for analytical query planning and are written infrequently. Applying this pattern to a training data plane requires the manifest to encode batch boundaries rather than partition layouts, the commit protocol to sustain high frequency under concurrent production, and the manifest to carry training-progress state (producer offsets and consumer watermarks) so that fault recovery and lifecycle management are driven by checkpoint progress. Figure 4 shows the overall architecture. Lakestream is implemented entirely as a client-side library with no server-side processes. It exposes two lightweight clients: a producer client, embedded in preprocessing workers, and a consumer client, embedded in each training rank. The object store is the sole shared substrate; no broker, dispatcher, or coordination service sits in the critical path. The central abstraction is the Transactional Global Batch (TGB), which gives B𝑠 a durable, first-class representation on object storage. A TGB is a logically atomic batch unit materialized as one or more immutable object-store objects. Its defining property is manifest-gated visibility: writing the objects alone does not make the TGB visible; the TGB becomes visible only when

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training List of Transactional Global Batch

Watermark Line Wglobal = min(Wi)

Buffer

Producers

Consumers Rank 0 Rank 1

Original Data

upload TGB

Rank 2 TGB7 TGB6 TGB5

TGB4 TGB3

TGB2

TGB1

Rank 3

TGB8 TGB4 TGB3

atomic commit manifest

manifest v3

List of Manifest

Figure 4: Lakestream architecture. Producers write TGBs directly to object storage and expose them atomically via a versioned manifest; consumers issue per-rank range reads against the objects of committed TGBs. The global watermark 𝑊global = min(𝑊𝑖 ) bounds which TGBs are eligible for reclamation. No broker or coordinator sits in the critical path. a committed manifest version records it and its position in the global step sequence. A single manifest commit may publish one or more newly written TGBs at once, so multiple TGBs can share the same first-visible manifest version. This provides atomicity through manifest-gated visibility, durability through object storage persistence, and globally consistent ordering through a linearized manifest version sequence. Lakestream makes no assumptions about producer count or stability. Producers may join or leave at any time, fail and restart, or produce data at varying rates, and producer and consumer lifecycles need not be synchronized.

3.1

Architecture and Data Flow

The data flow in Figure 4 proceeds in four stages. Stage 1: TGB materialization. The producer client serializes preprocessing output for one or more newly constructed TGBs into immutable object-store objects. This step requires no coordination: multiple producer clients write in parallel without communicating with each other or with consumer clients. Stage 2: Manifest commit. Once a producer client has accumulated one or more newly constructed TGBs, it commits them to the manifest sequence using a conditional put: the producer writes a new manifest object named with the next version number, succeeding only if no object with that name already exists. If a concurrent producer has already committed the same version, the write fails and the producer rebases onto the current manifest before retrying. The Decentralized Adaptive Commit (DAC) algorithm governs when each producer initiates a commit, balancing data freshness against conflict rate. A successful commit atomically makes the corresponding TGBs visible to all consumer clients. Stage 3: Batch consumption. The consumer client polls the manifest for new versions and fetches the rank-specific byte range from the objects of newly visible TGBs. It computes its projection locally from a lightweight footer index cached per TGB, requiring no inter-rank communication, and prefetches the referenced objects asynchronously to hide object store read latency.

Stage 4: Lifecycle management. After each successful distributed checkpoint, the training framework records the consumer’s current position as a watermark. Lakestream derives a global safety boundary 𝑊global = min𝑖 (𝑊𝑖 ); TGBs and manifest versions below this boundary are eligible for reclamation, tying data retention to checkpoint progress. Sections 4 and 5 describe the TGB data plane and the brokerless control plane in detail.

Transactional Global Batch

The Transactional Global Batch (TGB) abstraction gives each B𝑠 a durable, atomically visible representation on object storage. This section describes how TGBs are physically laid out to support the N-dimensional parallelism of modern LFM training, how the manifest represents their logical structure and ordering, how the combination enforces atomic visibility, and how the consumer client addresses rank-specific data without inter-rank coordination.

4.1

TGB Layout

Each TGB is materialized as one or more immutable objects written to the object store by a producer client, containing the trainingready data for one Global Batch B𝑠 . These objects are write-once: once persisted, their content never changes, allowing producer clients to write independently and consumer clients to cache aggressively without coherence overhead. The internal layout follows the data-sharing structure of the device mesh. As established in §2.1, only DP and CP require distinct data across ranks; TP and PP are transparent to data delivery. Accordingly, the data within a TGB is organized into 𝐷 ×𝐶 contiguous data slices, where slice (𝑑, 𝑐) contains the token chunk for CP rank 𝑐 of DP replica 𝑑. Each TGB includes a lightweight footer index recording the byte offset and length of every (𝑑, 𝑐) slice within its constituent objects. A consumer client reads this footer once per TGB, caches it locally, and thereafter fetches its data via targeted range reads. As a concrete example, consider a mesh with 𝐷 = 2 DP replicas and 𝐶 = 2 CP ranks, giving 4 slices per TGB. Replica 0 reads slices (0, 0) and (0, 1) for its two CP ranks; replica 1 reads (1, 0) and (1, 1). Any TP or PP ranks within the same DP replica and CP group derive identical (𝑑, 𝑐) coordinates and read the same slice. A 16-GPU job using 𝐷 = 2, 𝐶 = 2, TP = 2, PP = 2 therefore resolves exactly 4 distinct (𝑑, 𝑐) slices per TGB, one per (𝑑, 𝑐) pair, regardless of the TP and PP degree. This layout eliminates the read amplification of centralized disaggregated systems, where each rank receives the full batch and discards the portions belonging to other ranks, incurring 𝐷 ×𝐶-fold amplification. In Lakestream, each rank reads only its own data slice regardless of parallelism degree. Topology reconfiguration. Multi-stage LFM pipelines may use different parallelism degrees across stages, and practitioners frequently adjust TP, PP, or DP when resuming from a checkpoint. Rather than rewriting materialized TGBs, the consumer client remaps its position in the new device mesh locally. Changes to TP or PP degree leave data distribution unchanged; the consumer simply recomputes its (𝑑, 𝑐) coordinates. Changes to DP or CP degree alter how many data slices are needed per step: when DP world

Sun et al.

size doubles, the consumer reads two consecutive TGBs as one logical step; when it halves, the consumer reads alternating data slices from a single TGB across two steps. CP reconfiguration follows the same logic along the token-chunk dimension. Remapping requires no server-side intervention, no data rewrite, and no coordination with other ranks.

4.2

Manifest Structure

The manifest is the logical control structure of Lakestream. It is a versioned sequence of immutable metadata objects stored on the object store, where each version captures the complete state of the dataset at a point in time. A manifest version 𝑀𝑣 contains three components. The TGB list is an ordered sequence of TGB descriptors, each recording which step index the TGB corresponds to together with the object-store locations and layout metadata needed to read it. The TGB list defines the authoritative step sequence: the 𝑠-th step B𝑠 is the set of samples identified by the 𝑠-th entry in the list, regardless of when or by whom the corresponding TGB was written. The version sequence consists of immutable manifest objects named by their version number (e.g., 00000011.manifest). A producer commits by writing the next object under a conditional put that succeeds only if the name is unclaimed; conflict causes immediate failure with no further effect. This single write atomically advances the version and makes new TGBs visible, with no separate pointer to update. Readers follow progress by probing for higher-numbered manifest objects. The per-producer state map records the stream offset up to which each producer has successfully committed. It serves two purposes: allowing a producer that lost a commit race to determine, after rebasing, which of its TGBs are already incorporated; and providing the durable state for exactly-once recovery, as described in §5.3.

4.3

Atomic Visibility

A TGB is opaque to consumer clients until it is referenced in a committed manifest version. Consumer clients read only the TGB list from the current manifest version and fetch only the objects referenced by those TGB descriptors; they never scan the object store directly. Because a manifest version is a single atomically written object, every version a consumer observes is internally consistent: it references a complete, closed set of TGBs for each step it exposes. If a producer fails mid-commit, the version pointer is not updated and any partially written TGB data remains invisible. The next successful commit will include or exclude that data in a consistent way, as governed by the rebase protocol in §5.1.

4.4

Consumer Cursor and Deterministic Projection

Each consumer client maintains a cursor ⟨𝑉 , 𝑆⟩, where 𝑉 identifies the manifest version being read and 𝑆 the step index within that version’s TGB list. Given this cursor, the consumer independently computes its data slice: it derives its (𝑑, 𝑐) coordinates from environment variables (e.g., RANK, WORLD_SIZE, and the configured DP and CP degrees), looks up the byte offset and length for data slice

(𝑑, 𝑐) in the cached footer index, and issues targeted range reads against the corresponding object-store objects. Ranks within the same TP or PP group derive identical (𝑑, 𝑐) coordinates and read the same data slice without inter-rank communication; DP replicas derive disjoint coordinates and read disjoint data from the same TGB, keeping gradient aggregation mathematically consistent. After consuming step 𝑆, the consumer increments 𝑆; when 𝑆 reaches the end of the current TGB list, it polls for a new manifest version and updates 𝑉 . This polling is the only point at which the consumer touches the manifest; all data reads go directly to the object store from addresses resolved through the footer index. The cursor is the recovery interface between Lakestream and the training framework: after each successful distributed checkpoint, the framework persists the current cursor alongside the model weights. The persistence protocol and its recovery semantics are described in §5.3.

Brokerless Control Plane

The brokerless control plane of Lakestream manages three responsibilities: linearizing the global batch sequence across concurrent producers, regulating commit cadence to sustain throughput as the manifest grows, and ensuring fault tolerance and correct lifecycle management under producer failover and consumer restart. All three are realized without a central coordinator, using only object storage primitives and state embedded in the manifest.

5.1

Commit and Rebase Protocol

Producer clients commit TGBs to the manifest using a conditional put protocol that serializes concurrent updates without a broker. The protocol proceeds in three steps. First, the producer starts from its current local manifest view 𝑀𝑣 . Second, it constructs a candidate manifest 𝑀𝑣+1 by appending its local TGB references to the TGB list of 𝑀𝑣 and updating its own entry in the per-producer state map. Third, it attempts to write 𝑀𝑣+1 as (v+1).manifest via conditional put, succeeding only if the name is unclaimed. On conflict, the producer fetches the committed winner manifest 𝑀𝑣 ′ and rebases onto it: it appends its local TGB references to the winner’s list and updates only its own entry in the per-producer state map, copying all other entries verbatim from 𝑀𝑣 ′ , then retries. A producer need not observe the latest manifest before each attempt: starting from a stale base can only cause additional failed conditional writes, while rebasing on the winner preserves correctness. Because the conditional put is atomic, no two producers can commit the same version number, and the append-only union merge ensures no committed TGB is ever lost during a rebase. Version numbers are strictly monotonically increasing and never reused, so there is no ABA hazard: a producer that reads version 𝑣 and later finds version 𝑣 ′ in a conflict can safely take 𝑀𝑣 ′ as the rebase base, knowing its own TGBs are either already present (check via the state map) or must be appended. The result is a linearized step history consistent across all data-parallel ranks, with no inter-producer communication.

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

5.2

Algorithm 1 Decentralized Adaptive Commit (DAC)

Decentralized Adaptive Commit (DAC)

Sustaining high producer ingestion throughput requires careful regulation of commit cadence. The dominant cost per commit cycle is manifest I/O, and this cost grows monotonically as the manifest accumulates entries. A fixed commit interval becomes too aggressive over time, wasting an increasing fraction of producer time on failed commits. The central observation driving DAC is that manifest I/O time, the fragile window 𝜏𝑣 , is directly measurable at runtime and subsumes payload size, producer count, and network conditions into a single online-observable quantity. Adapting the commit interval to 𝜏ˆ𝑣 achieves stable throughput without static tuning. Reactive heuristics such as AIMD [16] adjust the commit interval based on observed outcomes but provide no quantitative guarantee on wasted time. DAC maintains an explicit overhead budget 𝛿 and conflict budget 𝜀, deriving a closed-form lower bound on 𝑇 from each constraint (Equations 7–8) and taking their maximum. Because 𝜏𝑣 directly captures manifest I/O cost, DAC adapts to manifest growth automatically with no separate tracking mechanism. Each producer client controls a post-attempt waiting gap 𝑇 : after any commit attempt, whether it succeeds or fails, the producer waits 𝑇 before initiating the next attempt. A commit proceeds by reading the current manifest version, constructing a new manifest object named with the next version number, and submitting it via conditional put. The fragile window 𝜏𝑣 is the interval from reading the current version to completing the write attempt: if another producer successfully commits the same version number within this window, the current commit fails. Under this control law, successive attempt starts are separated by approximately 𝑇 +𝜏𝑣 . Probabilistic model. We model the attempt starts of the other 𝑁 − 1 producers as independent Poisson processes with rate 1/(𝑇 + 𝜏𝑣 ). Under this renewal approximation, the probability that at least one competing attempt enters the fragile window of length 𝜏𝑣 is: 𝑝 conflict (𝑇 ) = 1 − 𝑒 − (𝑁 −1)𝜏𝑣 /(𝑇 +𝜏𝑣 ) (2) Attempt-duty budget. Each attempt cycle spends 𝜏𝑣 seconds on manifest I/O and lasts approximately 𝑇 +𝜏𝑣 seconds, so the resulting commit-duty factor is: 𝜏𝑣 𝑑 (𝑇 ) = (3) 𝑇 + 𝜏𝑣 DAC enforces two explicit budgets: a conflict budget 𝜀 on 𝑝 conflict (𝑇 ) and a duty budget 𝛿 on 𝑑 (𝑇 ). These define the feasible set F = {𝑇 ≥ 0 : 𝑝 conflict (𝑇 ) ≤ 𝜀, 𝑑 (𝑇 ) ≤ 𝛿 } (4) and DAC chooses the smallest feasible gap, 𝑇 ∗ = inf F

(5)

which maximizes freshness while respecting both budgets. Both 𝑝 conflict (𝑇 ) and 𝑑 (𝑇 ) decrease monotonically with 𝑇 , so 𝑇 ∗ is obtained by taking the maximum of the two resulting lower bounds. Online algorithm. In practice, 𝜏𝑣 is estimated online via an exponential moving average, and 𝑁 is read dynamically from the per-producer state map after each rebase, allowing DAC to track changes in the producer pool without inter-producer communication: 𝜏ˆ𝑣 ← (1 − 𝛼) 𝜏ˆ𝑣 + 𝛼 𝜏𝑣obs (6)

1: Parameters: 𝛿 (overhead budget), 𝜀 (conflict budget), 𝛼 (EMA

coefficient), 𝜌 (jitter magnitude)

2: Initialize: 𝜏ˆ𝑣 ← 0, gap ← 0, 𝑁 ← 1, 𝑡 last ← Now() 3: loop

WriteTGB() ⊲ Materialize data; no coordination needed if Now() − 𝑡 last ≥ gap then 6: 𝑡 0 ← Now() 7: success, 𝑀 ← TryCommit() 8: 𝜏𝑣obs ← Now() − 𝑡 0 9: 𝜏ˆ𝑣 ← (1 − 𝛼) 𝜏ˆ𝑣 + 𝛼 𝜏𝑣obs ⊲ Update EMA regardless of outcome 10: if success then 11: ClearBuffer() 12: else 13: 𝑀 ← Rebase() ⊲ Merge onto current manifest 14: end if 15: 𝑁 ← |ProducerStateMap(𝑀)| ⊲ Dynamic producer count from manifest 16: 𝑇conf ← max(0, (𝑁 − 1) 𝜏ˆ𝑣 / − ln(1 − 𝜀) − 𝜏ˆ𝑣 ) 17: 𝑇cost ← (1 − 𝛿) 𝜏ˆ𝑣 / 𝛿 18: gapnew ← max(𝑇conf, 𝑇cost ) · (1 + 𝜌 · Uniform(0, 1)) 19: gap ← gapnew 20: 𝑡 last ← Now() 21: end if 22: end loop 4: 5:

Rather than solving Equation 5 numerically, DAC computes the required gap directly from the two constraints. Solving 𝑝 conflict (𝑇 ) ≤ 𝜀 yields: (𝑁 − 1) 𝜏ˆ𝑣 𝑇 ≥ 𝑇conf = max 0, − 𝜏ˆ𝑣 (7) − ln(1 − 𝜀) and solving 𝑑 (𝑇 ) ≤ 𝛿 yields: 1−𝛿 𝜏ˆ𝑣 (8) 𝑇 ≥ 𝑇cost = 𝛿 The minimal feasible gap is therefore 𝑇 ∗ = max(𝑇conf, 𝑇cost )

(9)

and the target gap is a jittered version of this optimum used to desynchronize producers: gap = 𝑇 ∗ · (1 + 𝜌 · 𝑈 )

(10)

where 𝑈 ∼ Uniform(0, 1) and 𝜌 is the jitter magnitude. This gap is recomputed after every commit attempt and takes effect immediately for the next attempt. When bursts of correlated commits occur, the EMA update raises 𝜏ˆ𝑣 and widens the gap on the next cycle, self-correcting within a few attempt periods. On a successful commit, the producer clears its local data buffer. On a failed commit, the producer rebases onto the current manifest, merging its local TGB references and updating 𝑁 , before retrying. Algorithm 1 summarizes the full procedure, including the finalization phase in which the producer drains remaining uncommitted data before exiting.

Sun et al.

5.3

Fault Tolerance and Lifecycle Management

Lakestream provides end-to-end exactly-once semantics across the full pipeline. The producer and consumer sides use different mechanisms, each suited to the structure of the state they must preserve. A single shared primitive, the watermark, ties consumer fault tolerance to lifecycle management, serving both purposes without additional infrastructure. Producer fault tolerance. Producer resumption state (the stream offset up to which TGBs have been successfully committed) lives in process-local memory and is lost on failure. Without a recovery mechanism, a replacement process must restart from a conservative earlier position, re-producing and re-submitting data and yielding at-least-once semantics. Lakestream avoids this by inlining producer state directly into the manifest. Each producer is assigned a stable producer_id that persists across restarts; its resumption state is a single scalar, making inline storage practical without requiring a fixed producer count. On every commit, a producer updates only its own entry in the per-producer state map; all other entries are copied verbatim. Because the state map is committed atomically with the TGB list via conditional write, it is always consistent with the visible TGBs. A replacement process catches up to committed manifest state, looks up its producer_id, and resumes from the highest recorded offset with no coordination with other producers or consumers. Exactly-once delivery follows from two properties of the commit protocol: the state map entry and the TGB list advance atomically in the same write, so the recorded offset always matches the last visible TGB; and the conditional write prevents two processes sharing a producer_id from advancing the state map concurrently. End-to-end exactly-once argument. The two mechanisms are decoupled and jointly sufficient: neither a producer restart nor a consumer rollback disturbs the other side’s state. The only required invariant is that all data up to the rolled-back step remains available in the manifest, which is guaranteed by the watermark retention policy. Consumer fault tolerance and lifecycle management. After the training framework successfully writes a distributed checkpoint, each consumer records its current manifest version 𝑉 as a watermark persisted alongside the model weights. On rollback, the consumer restores its cursor from the checkpoint and resumes from the corresponding global batch with no data skipped and no batch consumed twice. The same watermarks drive lifecycle management. Existing streaming systems govern retention by time or capacity thresholds, with no awareness of checkpoint state; acknowledgment-based systems such as Pulsar treat delivery as a sufficient deletion signal, but a training rank that has consumed a batch may not yet have checkpointed the resulting model state, so a rollback can require replaying already-acknowledged data. Lakestream instead ties retention directly to checkpoint progress via: 𝑊global =

min 𝑊𝑖

𝑖 ∈ {1,...,𝑁 }

(11)

Any manifest version 𝑣 < 𝑊global and its associated TGB objects are unreachable from any live checkpoint and eligible for reclamation. A background process periodically computes 𝑊global from

the latest checkpoint watermarks and issues deletion requests accordingly. This process is outside the critical path: a failure delays reclamation but does not affect training correctness or throughput. Because object store deletions are idempotent and TGB objects are immutable, it can be restarted at any time without coordination. This design provides two guarantees. First, rollback safety: TGB objects are retained until 𝑊global advances past them, so any live checkpoint always finds the data it needs. Second, principled reclamation: as training progresses, 𝑊global advances and the retained window stays proportional to the checkpointing interval rather than the total training duration.

Implementation

Lakestream is implemented as a Python SDK with a Rust core. The Rust layer handles latency-critical execution: multi-threaded object prefetching, manifest polling, and the optimistic commit protocol. It is exposed to Python via PyO3-based foreign function interfaces, allowing users to implement custom preprocessing logic without recompilation. This split keeps systems-level throughput and memory safety alongside the rapid iteration cycles typical of LFM development. Lakestream uses If-None-Match semantics (available on S3, GCS, Azure Blob Storage, and BOS [21]) to implement the manifest commit protocol described in §5.1. Ordinary object reads are used only to fetch committed manifests during rebase and catch-up; observing a stale manifest can increase retries but cannot violate correctness. No other storage-side coordination is required. For the physical data layer, Lakestream materializes each TGB as one or more immutable objects on the object store. Our current prototype builds on an internal fork of Lance [19]. We extended Lance’s manifest format to support optional per-producer metadata fields, enabling the per-producer state map (§5.3) to be written atomically alongside the TGB list in a single manifest object; this is the primary divergence from upstream Lance. Because Lakestream is a pure client-side library, it integrates with any training framework that can invoke Python: a PyTorch training loop and a TensorFlow pipeline can share the same Lakestream namespace without modification, and switching frameworks requires only replacing the consumer call site.

Evaluation

We evaluate Lakestream on four questions aligned with its core claims: end-to-end training throughput, producer scalability and DAC stability, consumer efficiency under TGB-aware layout, and checkpoint-driven storage reclamation. We also measure the cost of manifest-inlined producer state for exactly-once semantics.

7.1

Experimental Setup

Infrastructure. All experiments run on a shared kjob cluster backed by Baidu Object Storage (BOS) [21]. Each node has 64 CPU cores. Training uses 8 GPUs and 8 trainer ranks per node; producer and consumer microbenchmarks use CPU-only nodes. The largest end-to-end run uses 8 trainer nodes (64 ranks) on NVIDIA H200 GPUs. Unless otherwise noted, the logical consumer world size in an end-to-end run equals the number of trainer nodes times 8.

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

Systems compared. Lakestream is our full object-store-native system. Kafka is a broker-based baseline using a dedicated Baidu Cloud Kafka cluster (16-core, 64 GB-memory instances, model kafka.ga2.c16m64, three replicas, 10 TB cloud disk per node) provisioned exclusively per experiment run; no two runs share a cluster. Kafka always uses strict TGB semantics: one message carries exactly one complete TGB, and producer count matches Lakestream. This is the only deployment mode for Kafka that satisfies our intra-batch consistency and inter-batch ordering requirements (§2.1) without reintroducing a centralized coordinator; we evaluate it to characterize the structural cost of broker-mediated delivery under LFM training semantics. For large-payload video workloads, the one-message-per-TGB constraint triggers two failure modes: TGB payloads exceeding per-message byte limits, and produce-request timeouts under peak broker load. We tuned all available parameters (message.max.bytes, request.timeout.ms, delivery.timeout.ms) to minimize failures, but could not eliminate them entirely for the Qwen3-VL configurations. Local is the expert-tuned colocated pipeline used in our production training stack. Each trainer rank launches 12 local worker threads for sample-level preprocessing, feeds transformed samples through a bounded sample queue into a dedicated collator thread for sequence packing and batch construction, and applies periodic garbage collection and cache cleanup to stabilize memory usage over long runs. This represents the engineering limit of the colocated architecture: further increasing worker count intensifies CPU and memory bandwidth contention with the GPU training process on the same node, and the preprocessing and training lifecycles remain coupled within the same process group. Commit policy baselines. Producer microbenchmarks compare five commit-policy baselines: Naive (commit every TGB), FIXED10 and FIXED100 (commit every 10 or 100 TGBs), INCR (start at 10 and increase by one on each conflict), and AIMD (Additive Increase Multiplicative Decrease, the classic TCP-style congestion control policy [16]: increase the interval by a fixed addend on success, halve it on conflict). Consumer microbenchmarks additionally compare dense-read, which reads the full TGB object span and filters locally. Workloads. We use four workload families: end-to-end GR00T [27] training, end-to-end HoloAssist [37] video SFT, end-to-end BEHAVIOR-1K [20] VLA training, and controlled data-plane microbenchmarks. All GR00T runs use a trainer-side DataLoader with num_workers=1 and prefetch_factor=4. HoloAssist runs train Qwen3-VL-30B-A3B [7] on the HoloAssist reasoning split, with online video decode, frame sampling at 2 FPS, and 8–16 frames per sample. BEHAVIOR-1K runs train the Qwen3-VL-30BA3B-AE-0.5B VLA model on multi-camera robot demonstrations stored in LeRobot [9] format, with online video decode, image augmentation, state/action windowing, and multimodal sequence packing. Payload sizes are 100 KB, 1000 KB, and 10000 KB. Producer experiments sweep 8–128 producers at logical world size 32, with a 300 s warmup and a 5-hour measurement window. The DAC commit-policy ablation fixes the producer count at 32 to isolate the effect of manifest growth over the same duration. Consumer experiments sweep logical world sizes 8 / 32 / 128 for 1,800 s per point. Consumer baselines always read from pre-materialized committed datasets so that all strategies observe identical input. The exactly-once microbenchmark sweeps payload sizes 100 KB,

1000 KB, and 10000 KB with TGB sizes 8 / 32 / 128. It alternates producer-state metadata with a dummy-metadata control on paired append inputs for one hour. Each operation commits one TGB, intentionally stressing per-commit metadata overhead; normal DAC-driven runs amortize this cost by committing larger batches. Metrics. End-to-end experiments report throughput (steps/s) and per-step latency over time. Producer experiments report aggregate ingestion throughput, commit success rate, and commit latency. Consumer experiments report effective per-rank throughput, P50/P95 read latency, and read amplification. Lifecycle experiments report object-store bytes over time. Methodology. Producer benchmarks exclude warmup; consumer benchmarks aggregate per-rank structured traces. DAC uses a conflict budget of 𝜀 = 0.05 in producer microbenchmarks and 𝜀 = 0.20 in end-to-end runs. End-to-end runs start producers and trainers together rather than from a pre-filled backlog, so reported step timing begins at first-batch arrival and excludes only the initial producer warm-up. We otherwise use all common recorded steps for each accepted run, without tail trimming. Omitted Kafka points indicate no usable strict-TGB run at that configuration.

7.2

End-to-End Training Performance

Unless otherwise noted, Lakestream uses dedicated CPU producer nodes and no train-side producers: 32 nodes for the Qwen3-VL workloads and 16 for GR00T. Local uses the optimized in-rank threaded pipeline described above. Kafka uses a separate producer job on CPU nodes of the same type, with the same producer count as Lakestream. The bottom row of Figure 5 shows that Lakestream sustains 3.09 steps/s versus 1.15 steps/s for Local and 0.11 steps/s for Kafka: a 2.68× gain over Local and 27.3× over Kafka. The all-rank steplatency P50/P95 values are 172/367 ms for Lakestream, 457/4,113 ms for Local, and 8,811/9,534 ms for Kafka. The gain over Kafka follows from removing the broker from the write path. The gain over Local reflects structural resource contention that in-rank engineering cannot eliminate: preprocessing threads share CPU cores and memory bandwidth with the GPU training process on the same node. Under GR00T’s heavy preprocessing expansion (§2.1), the batch queue periodically empties and the training rank stalls, producing the observed P95 of 4,113 ms, nearly 10× the P50 of 457 ms. Lakestream eliminates this contention by running preprocessing on dedicated nodes, yielding a P95 of 367 ms. HoloAssist video SFT. Lakestream sustains 0.222 steps/s versus 0.029 steps/s for Local on the HoloAssist [37] reasoning split (7.73× gain). The all-rank P50/P95 latencies are 2.60/2.79 s for Lakestream and 33.7/42.7 s for Local. The gain reflects the same structural contention as GR00T: video decode, frame sampling, and multimodal packing compete with training on the same nodes in the local baseline, whereas Lakestream offloads these to dedicated producer nodes. BEHAVIOR-1K VLA. Lakestream sustains 1.17 steps/s versus 0.227 steps/s for Local on BEHAVIOR-1K [20] demonstrations in LeRobot [9] format (5.17× gain). The all-rank P50/P95 latencies are 0.374/0.577 s for Lakestream and 3.83/7.96 s for Local. Normal steps are sub-second, with occasional fetch stalls when the trainer catches up to producer output; even with these stalls, disaggregating

0.08

Kafka

0.24

0.22

0.16

0.23 Lakestream 0.08 failed

0.16 3 0.08 103

GR00T Policy

Lakestream

failed 0.4

Qwen3-VL VLA

0.8

1.15 0.03

0.11 Kafka

Local

400

600

800

1000

0.16

104

0.08 103

Local 0.00

failed

Kafka

Lakestream

0.03

failed Sun et al.

Local

Kafka

3.09

Kafka

1.15 104 0.03

failed

0.8 104 0.4 103

0.0 0.00 Lakestream Local 600 0.11 Local Kafka 0 200 400 0.03Lakestream failed 0 Per-Step Latency (ms, log) Lakestream Local Kafka 0 2000 4000 6000 0 500 1000 1500 2000 2500 3000 Local Kafka 0 200 400 104600 800 1000 Training step 3.09 1.17 1.24 3 10 104 0.8 104 3

10 0.4

0.23 103 0.23 0.0 failed 0 200 400 600 Lakestream Local

failed

800 1000 Kafka

0.23

failed

GR00T Policy

3 10 0.4

103 1 1.15

1.15 103

104

failed Kafka1000 800 8000

10000

2 103 1

103 0.23

104

103

1.15 0.11 0 500 1000 Local 1500 2000 2500 Lakestream Kafka3000

0Lakestream 1000 1500 2500 3000 0.0 Local 2000 Kafka 0 2000 4000 (step/s) 6000 across 8000 three 10000 Figure 5: 500 Per-step latency (ms,Lakestream log) and end-to-end throughput Kafka3000 failed on the Qwen3-VL Local Kafka 0 500 1000 workloads. 1500 2000 2500 4 10 104 Training step 1.17 3.09GR00T it is shown over the longest measured prefix (9,694 steps), after workloads (no usable strict-TGB run at 8 nodes); for 4 1.2 10 3 104 which sustained broker backpressure caused per-step wait times to exceed an acceptable threshold. Lakestream outperforms 3.09 3 0.8 Local by 2.68–7.73× across all three workloads. 2

GR00T Policy

Qwen3-VL VLA 0.11 Kafka

Qwen3-VL VLA

0.23

Lakestream

Local 1

Throughput (steps/s) 0Lakestream 200 400Local 600 800 Kafka1000 0.00 Lakestream 0.22 0.24 4 10 3.09 1.17 3 1.2 0.16

Kafka

0.4

Qwen3-VL VLA

1.17

0.0

2 0.08 103 1 0.00

0.8 104

0 0.22200

0.24

0.0 Lakestream Throughput (steps/s) Per-Step Latency (ms, log) Local Kafka Kafka1000 0 500 1000 1500 2000 2500 3000 Local Kafka 0Lakestream 200 400 Local 600 800 1.17 Per-Step Latency Qwen3-VL SFT(ms, log) Qwen3-VL VLA GR00T Policy 1.2 0.22 0.244 10 Throughput (steps/s) Per-Step Latency (ms, log) 104

0.4

GR00T Policy Qwen3-VL ThroughputSFT

failed

1.17

Lakestream

0.8 104

Kafka

GR00T Policy

Qwen3-VL VLA Latency

1.2

0.03

1.2

Qwen3-VL SFT GR00T Policy

0.00

ps/s)

failed

Qwen3-VL VLA

0.16

Local

Per-Step Latency (ms, log)

0.22

Qwen3-VL SFT

stream

0.24

Lakestream

Qwen3-VL SFT

0.00

Throughput (steps/s)

103

preprocessing cuts total training Overhead0.11 of manifest-inlined producer state. Figure 8 isolates 1 time by 80.7% over the optimized 0.0 0 0 the Lakestream Local producer Kafka 500 1000 1500 the 2000 2500 3000 local pipeline. The 32-node allocation reflects heavier per-commit producer against a 0.11 Lakestream Local Kafka cost of manifest-inlined 0 2000 4000 6000state 8000 10000 0 2000 4000 60000 8000 10000 per-sample cost of multi-camera video decodeLocal and multimodal dummy-metadata control on paired append inputs. Training stepEvery TGB is Lakestream Kafka 0 2000 4000 6000 8000 10000 104 Training step Training intentionally step packing in the Qwen3-VL workloads compared with GR00T’s 16committed immediately, stressing per-commit meta3.09 3 node allocation. data overhead rather than the amortized cost in normal DAC-driven runs. The mean commit latency delta ranges from 9.4% to 74.9% 2 7.3 Producer Scalability and DAC103 across payload and TGB sizes; the upper end occurs at TGB=8 and 1.15 1 100 KB, where payload is minimal and metadata overhead is not Producer throughput scaling. Figure 6 sweeps producer count 0.11 amortized. The bottom panel of Figure 8 shows this relative cost and0 payload size, comparing Lakestream against Kafka and the Lakestream Local Kafka 0 2000 4000 6000 8000 10000 declining over the run: commit delta falls from 40.6% to 32.4% and commit-policy baselines. At 128 producers, Lakestream reaches Training step fragile-window delta from 50.5% to 29.5%, consistent with a fixed 1.42 GB/s, 7.03 GB/s, and 16.0 GB/s at 100 KB, 1000 KB, and 10000 KB. per-commit cost whose fraction shrinks as manifest commit work Against the strongest non-Lakestream baseline at the same producer grows. In normal operation, DAC further amortizes this cost by count, this is 6.00×, 3.31×, and 1.20× faster, respectively. Kafka committing larger batches. succeeds through 64 producers at 100 KB and 128 producers at 1000 KB, but no 10000 KB strict-TGB point succeeds. Lakestream 7.4 Consumer Efficiency and Read continues to scale, showing that removing the broker from the Amplification write path is the dominant win. The narrower margin at 10000 KB reflects a shift in the dominant bottleneck: at large payloads, raw Each rank issues a single range read targeting only its own data slice, object-store write bandwidth saturates for all strategies, leaving so Lakestream’s read amplification is near 1× at all world sizes. The less room for commit-protocol efficiency to differentiate them; at small residual overhead comes from reading the footer index and 100 KB, per-request overhead and conflict rate dominate, where object headers, a fixed absolute cost whose fractional contribution broker removal and DAC provide the largest relative gain. shrinks with payload size. At 100 KB and world size 128 it accounts DAC under manifest growth. Figure 7 shows that DAC is the for the measured 1.67×; dense-read and Kafka scale with world size only policy that sustains both success rate and throughput as the because both deliver the full TGB before filtering. At 128 consumers manifest grows over five hours. With 32 producers on one CPU and 100 KB, Lakestream reaches 1.61 MB/s per rank at 1.67× amplinode, DAC averages 431.9 MB/s at 96.3% commit success. INCR fication, versus 1.11 MB/s at 130.8× for dense-read and 0.061 MB/s reaches 110.5 MB/s at 64.2%, FIXED100 107.5 MB/s at 64.3%, AIMD at 128.0× for Kafka. At 10000 KB and 128 consumers, Lakestream 79.0 MB/s at 95.5%, FIXED10 50.0 MB/s at 13.1%, and Naive 7.1 MB/s cuts P50/P95 latency from 121.6 ms / 31.4 s to 22.8 ms / 3.21 s relat 9.0%. As manifest growth raises commit cost, policies that fail ative to dense-read, while improving per-rank throughput from to widen the commit interval waste an increasing fraction of time 1.46 MB/s to 16.2 MB/s (11.1×). Kafka has no successful strict-TGB on retries. The measured conflict rate of DAC stays close to the run at that configuration. The P95 result is operationally significant: target 𝜀 = 0.05 in the producer microbenchmarks, consistent with dense-read’s 31.4 s tail exceeds typical per-step training budgets, the Poisson model in practice. so a straggler rank that falls behind on a full-TGB read blocks the entire data-parallel group. Lakestream’s targeted range reads keep

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

Lakestream

Naive

FIXED10

FIXED100

INCR

AIMD

Kafka

Throughput (MB/s) 100 KB

1000 KB

10000 KB 15000

1200

6000

800

4000

10000

400

2000

5000

816 32

128

# Producers

816 32

# Producers

128

816 32

128

# Producers

FIXED10 FIXED100

AIMD Naive

Throughput (MB/s)

750 500 250 0

Commit Success (%) 90 60 30 50

100

150

200

Time (min)

250

300

Figure 7: DAC ablation with 32 producers over 5 hours. DAC is the only policy that sustains both high throughput and commit success as the manifest grows; all fixed-interval and heuristic policies degrade. the P95 below 4 s across all measured configurations, eliminating fetch stalls as a source of step-time variance.

7.5

Checkpoint-Driven Lifecycle Management

Figure 9 compares two otherwise identical 1,010-step runs with checkpoints every 10 steps and max_lag=80: one with physical deletion enabled after logical trim, and one without. The max_lag parameter caps how many unacknowledged TGBs producers may accumulate ahead of 𝑊global , bounding peak storage even if checkpointing temporarily stalls. Rank 0 samples total object-store bytes every 60 s and at each checkpoint boundary; the resulting curve shows plateaus between samples and drops when an advancing 𝑊global triggers reclamation at a checkpoint boundary. Without physical deletion, capacity grows monotonically to 34.85 GiB. With physical deletion, it is capped at 9.76 GiB, a 72.0%

Overhead Delta (%)

Lakestream INCR

Commit Delta (%)

Figure 6: Producer ingestion throughput versus producer count across three payload sizes. Lakestream is the only system that scales linearly with producer count; all baselines plateau or fail at high concurrency. Omitted Kafka points indicate no successful strict-TGB run at that configuration.

TGB=8

TGB=32 75

75 50

50 25 0

TGB=128

56 37

100KB

1000KB

10000KB

Payload

Commit Delta

Fragile Window Delta

30 20 10 0 0

Elapsed Time (min)

Figure 8: Overhead of manifest-inlined producer state for exactly-once semantics, measured against a dummymetadata control. The per-commit cost (9.4%–74.9% at run start) declines over time because the fixed metadata size becomes negligible relative to growing manifest I/O. reduction, confirming that checkpoint-written watermarks are a correct control signal: data remains available to live checkpoints and is reclaimed once no live checkpoint references it.

8 Related Work 8.1 Message Queues and Streaming Systems Distributed message queues such as Apache Kafka [18] and Apache Pulsar [34], along with cloud-native variants including WarpStream [38], AutoMQ [6], and Ursa [22], provide durable record delivery but retain the record/offset abstraction. A training batch is not an addressable entity in these systems; expressing batch-level atomicity requires an external coordination layer, and

Capacity (GiB)

Sun et al.

With GC

Without GC

15 0

Time (min)

Figure 9: Checkpoint-driven storage reclamation over a 1,010step run. With physical deletion enabled, peak capacity is capped at 9.76 GiB (72.0% reduction vs. 34.85 GiB without deletion), confirming that checkpoint-written watermarks are a correct and tight reclamation signal.

100KB

4.5

Lakestream baseline

Kafka

1000KB

7.5

10000KB 18

3.0

5.0

1.5

2.5

0.0

128

102

128

102

8.3 8

128

104

100 10

10−2

Read Amp. (x)

P95 Lat. (ms)

Eff. Thrpt. (MB/s)

Lakestream

128

8 10

128

100

101

101 100

103 8

128

100

Logical World Size

Figure 10: Consumer throughput, P95 latency, and read amplification across payload and world sizes. Lakestream maintains near-1× read amplification at all scales because each rank issues a single range read targeting only its own data slice; dense-read and Kafka amplification grow linearly with world size. Omitted Kafka points indicate no successful strictTGB run. retention is governed by time or capacity with no awareness of checkpoint state. Stream processing frameworks such as Flink [10], Spark Structured Streaming [4], and Dataflow [2] achieve fault tolerance through operator state snapshots, but restoring operator state does not make consumed input sets independently re-observable. Lakestream externalizes the batch sequence as a first-class persistent structure, making it replayable independently of any processing logic.

8.2

and Pecan [13] optimize sample-level cache efficiency. These systems treat the batch as a transient delivery artifact with no persistent boundary, no atomic cross-rank visibility guarantee, and no mechanism for deterministic replay. Dataset formats such as Petastorm [15] and MosaicML Streaming [25] target static, prematerialized datasets and cannot linearize out-of-order arrivals from concurrent dynamic producers into a durable step sequence. MegaScale-Data [40] externalizes data construction but does not provide a persistent, checkpoint-aligned batch history. Mixtera [8] provides declarative mixture control over static datasets but is a read-only layer with no transactional delivery or fault tolerance semantics. General-purpose frameworks such as Ray [24], Spark [39], and Dask [31] offer execution lineage, but lineage guarantees recomputability rather than membership stability: recomputing a non-deterministic batch produces a different result, which is insufficient for checkpoint recovery. Lakestream preserves the batch history as a stable, immutable record rather than deriving it on demand.

ML Training Data Pipelines

Disaggregated systems such as tf.data service [5], Cachew [12], FastFlow [36], FusionFlow [17], and Cedar [41] decouple preprocessing from training to prevent GPU stalls, while CoorDL [23]

Lakehouse Architectures

Lakehouse systems such as Delta Lake [3], Apache Iceberg [35], and Lance [19] use versioned metadata manifests over object storage to coordinate concurrent writers via optimistic concurrency control. Magnus [33] extends Iceberg for EB-scale ML data management with Git-like branching and multimodal storage optimizations. Lakestream is, to our knowledge, the first system to repurpose this pattern as a streaming training data plane. Lakestream adopts the versioned-manifest transaction pattern of lakehouse systems, but goes beyond conventional lakehouse semantics by making the Global Batch the fundamental unit and enforcing training-specific consistency, including atomic all-rank visibility, globally ordered batch progression, and checkpoint-aligned recovery and reclamation. The DAC algorithm addresses a problem absent in offline settings: sustaining high commit throughput as the manifest grows under continuous concurrent production.

Conclusion

Modern LFM training demands more from the data pipeline than record delivery: the Global Batch is the unit of distributed optimization, and its boundaries, ordering, and lifecycle must align with checkpoint state. Lakestream addresses this by adapting versionedmanifest lakehouse design into a training-aware data plane. It introduces the Transactional Global Batch, DAC for sustained ingestion under manifest growth, manifest-inlined producer state for exactly-once recovery, and checkpoint-aligned reclamation tied to live checkpoints. Implemented entirely as a client-side library with no server-side processes, Lakestream shows that object storage, combined with versioned metadata and decentralized coordination, is a sufficient substrate for a training-aware data plane.

References [1] Alex Aizman, Gavin Maltby, and Thomas Breuel. 2020. High Performance I/O For Large Scale Deep Learning. https://arxiv.org/abs/2001.01858. doi:10.48550/ arXiv.2001.01858 arXiv:2001.01858. [2] Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J Dagum, Sam Knight, Frances Perry, Reiner Schmidt, and Sam Whittle. 2015. The dataflow model: a practical approach to balancing correctness, latency, and

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment (PVLDB) 8, 12 (2015), 1792–1803. [3] Michael Armbrust, Tathagata Das, Liwen Sun, Burak Yavuz, Shixiong Zhu, Mukul Murthy, Joseph Torres, Herman van Hovell, Adrian Ionescu, Bogdan Ghit, Madhukara Bhat, Reynold Xin, Ali Ghodsi, Ion Stoica, and Matei Zaharia. 2020. Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores. Proceedings of the VLDB Endowment (PVLDB) 13, 12 (2020), 3411–3424. [4] Michael Armbrust, Tathagata Das, Joseph Torres, Burak Yavuz, Shixiong Liao, Yin Huai, Hossein Hosseini, Matei Zaharia, and Reynold Xin. 2018. Structured streaming: A declarative api for real-time applications in apache spark. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD). Association for Computing Machinery, New York, NY, USA, 601–613. [5] Andrew Audibert, Yang Chen, Dan Graur, Ana Klimovic, Jiri Simsa, and Chandramohan A. Thekkath. 2023. tf.data service: A Case for Disaggregating ML Input Data Processing. In Proceedings of the 2023 ACM Symposium on Cloud Computing (SoCC). Association for Computing Machinery, New York, NY, USA, 358–375. [6] AutoMQ Team. 2024. AutoMQ: Cloud-Native Streaming with Offloaded Storage. https://www.automq.com. [7] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. 2025. Qwen3-VL Technical Report. https://arxiv.org/abs/2511.21631 [8] Maximilian Böther, Xiaozhe Yao, Tolga Kerimoglu, Dan Graur, Viktor Gsteiger, and Ana Klimovic. 2026. Mixtera: A Data Plane for Foundation Model Training. Proc. ACM Manag. Data 4, 1 (April 2026). doi:10.1145/3786668 [9] Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, Steven Palma, Pepijn Kooijmans, Michel Aractingi, Mustafa Shukor, Dana Aubakirova, Martino Russi, Francesco Capuano, Caroline Pascal, Jade Choghari, Jess Moss, and Thomas Wolf. 2024. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in PyTorch. https://github.com/huggingface/lerobot. [10] Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Kostas. 2015. Apache Flink™: Stream processing at scale. ACM SIGMOD Record 44, 4 (2015), 28–39. [11] DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. [12] Dan Graur, Damien Aymon, Dan Kluser, Tanguy Albrici, Chandramohan A. Thekkath, and Ana Klimovic. 2022. Cachew: Machine Learning Input Data Processing as a Service. In Proceedings of the 2022 USENIX Annual Technical Conference (USENIX ATC). USENIX Association, Berkeley, CA, USA, 689–706. [13] Dan Graur, Oto Mraz, Muyu Li, Sepehr Pourghannad, Chandramohan A. Thekkath, and Ana Klimovic. 2024. Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement. In Proceedings of the 2024 USENIX Annual Technical Conference (USENIX ATC). USENIX Association, Berkeley, CA, USA, 649–665. [14] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. OpenCLIP. doi:10.5281/zenodo.5143773 [15] Uber Technologies Inc. 2018. Petastorm: Open source library to enable training deep learning models from Apache Parquet datasets. https://github.com/uber/ petastorm. [16] Van Jacobson. 1988. Congestion Avoidance and Control. In Symposium Proceedings on Communications Architectures and Protocols (SIGCOMM ’88). Association for Computing Machinery, New York, NY, USA, 314–329. doi:10.1145/52324.52356 [17] Taeyoon Kim, Youngbin Jeong, Myeongjae Jang, and Jong-Geun Lee. 2023. FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation. Proceedings of the VLDB Endowment (PVLDB) 17, 3 (2023), 488–502. [18] Jay Kreps, Neha Narkhede, and Jun Rao. 2011. Kafka: A Distributed Messaging System for Log Processing. In Proceedings of the 4th International Workshop on Networking Meets Databases (NetDB). Association for Computing Machinery, New York, NY, USA, 1–7. [19] Lance Format. 2025. Lance. https://github.com/lance-format/lance/. [20] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. 2023. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. Proceedings of Machine Learning Research 205 (2023), 80–93.

https://proceedings.mlr.press/v205/li23a.html [21] Jiahao Li, Biao Cao, Jielong Jian, Cheng Li, Sen Han, Yiduo Wang, Yufei Wu, Kang Chen, Zhihui Yin, Qiushi Chen, Jiwei Xiong, Jie Zhao, Fengyuan Liu, Yan Xing, Liguo Duan, Miao Yu, Ran Zheng, Feng Wu, and Xianjun Meng. 2025. Mantle: Efficient Hierarchical Metadata Management for Cloud Object Storage Services. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 928–943. doi:10.1145/3731569.3764824 [22] Matteo Merli, Sijie Guo, Penghui Li, Hang Chen, and Neng Lu. 2025. Ursa: A Lakehouse-Native Data Streaming Engine for Kafka. Proceedings of the VLDB Endowment (PVLDB) 18, 12 (2025), 5184–5196. [23] Jayashree Mohan, Amar Phanishayee, Janardhan Kulkarni, and Vijay Chidambaram. 2021. CoorDL: Co-ordinated Data Loading for Deep Learning. In Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC). USENIX Association, Berkeley, CA, USA, 305–319. [24] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Berkeley, CA, USA, 561–577. [25] MosaicML. 2023. StreamingDataset: A high-performance dataset for deep learning. https://github.com/mosaicml/streaming. [26] Derek G. Murray, Jiří Šimša, Ana Klimovic, and Ihor Indyk. 2021. tf.data: a machine learning data processing framework. Proc. VLDB Endow. 14, 12 (July 2021), 2945–2958. doi:10.14778/3476311.3476374 [27] NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. 2025. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. https://arxiv.org/abs/2503.14734 [28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Curran Associates, Inc., Red Hook, NY, USA, 27730–27744. [29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 32. Curran Associates, Inc., Red Hook, NY, USA, 8024–8035. [30] PyTorch Team. 2025. torch.distributed.checkpoint Documentation. https:// pytorch.org/docs/stable/distributed.checkpoint.html. [31] Matthew Rocklin. 2015. Dask: Parallel computation with blocked algorithms and task scheduling. In Proceedings of the 14th Python in Science Conference, Vol. 130. SciPy, Austin, TX, USA, 136. [32] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. [33] Jun Song, Jingyi Ding, Irshad Kandy, Yanghao Lin, Zhongjia Wei, Zilong Zhou, Zhiwei Peng, Jixi Shan, Hongyue Mao, Xiuqi Huang, Xun Song, Cheng Chen, Yanjia Li, Tianhao Yang, Wei Jia, Xiaohong Dong, Kang Lei, Rui Shi, Pengwei Zhao, and Wei Chen. 2025. Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads. Proc. VLDB Endow. 18, 12 (Aug. 2025), 4964–4977. doi:10.14778/3750601.3750620 [34] The Apache Software Foundation. 2015. Apache Pulsar. https://pulsar.apache. org. [35] The Apache Software Foundation. 2025. Apache Iceberg. https://iceberg.apache. org/. [36] Taegeon Um, Goeun Byun, Hwarim Choi, Mincheol Han, and Hyuck Park. 2023. FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline. Proceedings of the VLDB Endowment (PVLDB) 16, 11 (2023), 1086–1099. [37] Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. 2023. HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World. https://openaccess.thecvf. com/content/ICCV2023/html/Wang_HoloAssist_an_Egocentric_Human_ Interaction_Dataset_for_Interactive_AI_Assistants_ICCV_2023_paper.html. In Proceedings of the IEEE/CVF International Conference on Computer Vision

Sun et al.

(ICCV), pages 20270–20281. [38] WarpStream Labs. 2025. WarpStream: A Cloud-Native, Zero-Disk Apache Kafka Alternative. https://www.warpstream.com. [39] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). USENIX Association, Berkeley, CA, USA, 15–28. [40] Juntao Zhao, Qi Lu, Wei Jia, Borui Wan, Lei Zuo, Junda Feng, Jianyu Jiang, Yangrui Chen, Shuaishuai Cao, Jialing He, Kaihua Jiang, Yuanzhe Hu, Shibiao Nong, Yanghua Peng, Haibin Lin, and Chuan Wu. 2026. MegaScale-Data: Scaling

Dataloader for Multisource Large Foundation Model Training. https://arxiv.org/ abs/2504.09844 [41] Mark Zhao, Emanuel Adamiak, and Christos Kozyrakis. 2024. Cedar: Optimized and Unified Machine Learning Input Data Pipelines. Proceedings of the VLDB Endowment (PVLDB) 18, 2 (2024), 488–502. [42] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proc. VLDB Endow. 16, 12 (2023), 3848–3860. doi:10.14778/ 3611540.3611569

Record · ID 175205 · SHA-256 654aa8c428d68fcc
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.