arXiv CS open access

KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning

Bangjun Xiao et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning Bangjun Xiao†∗ Chiheng Lou†∗ Xiaodong Ji‡ Xiangwei Deng‡ Yikai Zhao‡ Gang Huang† Yun Ma† Hailin Zhang‡⋄

arXiv:2610.32389v1 [cs.DC] 23 Oct 2026

† School of Computer Science, Peking University

ABSTRACT

the policy, which in turn generates subsequent trajectories. As RL applications advance [5, 11, 22, 43] and training scales to larger batch sizes, larger clusters, and more complex workloads, effective and efficient execution depends not only on rollout and training individually, but also on trajectory transfer from rollout to training. This handoff determines which trajectories enter training, when they become available, how their data are stored, and how those data are transferred to training workers. Along this path, we focus on two aspects to improve learning quality and execution efficiency, respectively: enabling early admission under batch-wide training requirements, and aligning trajectory data access with the spatial and temporal needs of distributed training. RL training can impose batch-wide requirements: mixed RL enforces target data-source ratios [11, 43], while partialrollout completion quotas require sufficient progress from earlier rollout steps to mitigate long-tail starvation. Finegrained streaming [13, 53] makes completed trajectories incrementally available, but admitting too many from faster sources can leave insufficient capacity to satisfy slower sources’ quotas. Enforcing these requirements through whole-train-batch admission creates a train-batch admission barrier: training waits until all admission conditions hold, even when some trajectories are already ready (Figure 1). In off-policy disaggregated execution, this delays training and parameter feedback to rollout, resulting in higher architectural staleness—the additional parameter-version gap introduced by pipelining relative to colocated execution, even under identical staleness-control settings. This additional staleness degrades learning quality. The challenge is therefore to overcome the train-batch admission barrier while preserving batch-wide requirements (§2.2). Large trajectory payloads can also prolong training steps with inefficient trajectory transfer. Long trajectories carry not only token IDs but also per-token auxiliary data such as routing decisions [26] and top-𝑝 sampling masks [38], alongside multimodal inputs such as images and videos [8, 21]. Unlike pre-training on static datasets, RL consumes trajectories generated online by rollout, in both colocated and disaggregated execution, placing their preparation and transfer on the dependency path to policy updates. Spatially, storage

Trajectory transfer from rollout to training affects both learning quality and system efficiency in large language model reinforcement learning (RL). Enforcing batch-wide training requirements can delay the admission of completed trajectories, resulting in higher architectural staleness—the additional parameter-version gap from disaggregated pipelining relative to colocated execution. Beyond admission, misalignment between data access and training consumption causes redundant movement and unnecessary waiting when transferring large trajectory payloads. We present KnitTrans, a system that coordinates the rollout-to-training path through three designs. First, constraint-preserving mini-batch admission allows flexibility in early mini-batch composition, while checking that enough capacity remains to satisfy the remaining quotas. Second, on-demand trajectory data access matches storage units to consumer assignments and overlaps micro-batch data preparation with computation. Third, multimodal encoding deduplication supports reuse across a mini-batch while preserving gradient contributions from all backbone micro-batches. Experiments show up to 93% lower architectural staleness than with train-batch admission, narrowing the learning-quality gap between disaggregated execution and colocated execution. On multimodal workloads, KnitTrans reduces transfer latency by two orders of magnitude and achieves a speedup of up to 5.8× in transfer plus training over the baseline.

‡ LLM-Core Xiaomi

INTRODUCTION

Reinforcement learning (RL) has become a pivotal component in the post-training of large language models (LLMs), used to align model behavior with human preferences [31] and to elicit advanced capabilities such as reasoning [12, 30], tool use [7, 19], and multimodal reasoning [16, 21]. Despite the numerous RL algorithm variants [33, 34, 47], the workflow remains invariant: rollout generates trajectories; after rewards are computed, training consumes them to update ∗ Work done during internship at Xiaomi Corporation. ⋄ Corresponding author.

Bangjun Xiao et al. Naive Admission Mini-batch size: 2

mb1,2,3,4 mb1 mb2 mb3,4

Linear (1, 2, 3, 4)

datasource A

A target: 4

datasource B

B target: 4 interpolate

Rollout

incrementally in micro-batch consumption order and overlap with training computation, as illustrated in Figure 1. Multimodal Encoding Deduplication. GRPO-style RL [34] produces multiple trajectories sharing the same multimodal inputs. Sharing storage does not itself remove repeated encoding of these inputs during training. KnitTrans encodes each unique multimodal input once per mini-batch, reuses its encoding across the consuming trajectories, and accumulates their gradients for the encoder update. We evaluate KnitTrans on a 256-GPU cluster using models with 235B–671B total parameters, covering agentic and multimodal RL workloads under both disaggregated and colocated architectures. In disaggregated training, KnitTrans reduces architectural staleness by up to 93% relative to train-batch admission, narrowing the learning-quality gap with colocated training. In a separate evaluation of constraint preservation, KnitTrans satisfies all data-source quotas and reduces architectural staleness by 39.2% compared with linear-target admission. On multimodal workloads, it reduces exposed transfer latency by two orders of magnitude and speeds up transfer plus training by 5.8× over TransferQueue. We list the contributions of this paper as follows. • We analyze how train-batch admission barriers under batch-wide requirements delay consumption and increase architectural staleness, and how mismatches between data access and training consumption cause redundant movement and unnecessary waiting. • We develop constraint-preserving mini-batch admission using progressive readiness targets and feasibility guards, and on-demand trajectory data access through atomic-unit storage, topology-aware reorganization, and consumption-driven delivery. Mini-batch-wide multimodal encoding reuse further reduces repeated computation and accelerates training. • We build KnitTrans, an RL system that realizes these designs across disaggregated and colocated architectures. Large-scale evaluation demonstrates substantial gains in both learning quality and training efficiency through improvements to the rollout-to-training path.

KnitTrans (0, 0, 2, 4)

mb1

mb2 mb3 mb4

R0 Rollout

Storage (mm)

reorg

Forward

Storage (backbone) Train

reorg reorg

reorg

Backward enc

enc

Figure 1: Timeline of trajectory admission, data delivery, and training computation under KnitTrans. organization determines which data can be accessed independently, while the training topology determines which data each worker needs. Tensor-parallel (TP) ranks share inputs, context-parallel (CP) ranks consume sequence slices, and multimodal encoders can have assignments distinct from the backbone’s. Existing sample-level storage layouts [13, 27] organize data around sample boundaries rather than these consumer-specific requirements, causing redundant reads or transfer of unneeded data. Temporally, training consumes an admitted mini-batch through a sequence of micro-batches. Existing RL systems [35, 39] serialize payload retrieval and input packing with computation on the critical path, so these preparation costs prolong training time even without redundant data movement. The challenge is therefore to match both consumer-specific data requirements and micro-batch consumption order, avoiding redundant movement and unnecessary waiting (§3.2). We present KnitTrans, which coordinates trajectory admission, data access, and multimodal encoding along the rollout-to-training path through three designs. Constraint-Preserving Mini-Batch Admission. KnitTrans designs progressive readiness targets that accommodate uneven rollout progress, rather than require early admissions to mirror the final train-batch composition. In Figure 1, linear source-readiness targets delay the first mini-batch until the slower source makes sufficient progress, whereas relaxed early targets permit earlier admission. To keep this flexibility from making the final train batch infeasible, feasibility guards check that each admission leaves enough batch capacity to satisfy the remaining batch-composition quotas.

BACKGROUND

We briefly introduce the background of this work from the perspectives of RL trajectories and system architectures.

2.1

On-Demand Trajectory Data Access. KnitTrans aligns storage and data access with training consumption in space and time. It stores trajectory data as independently addressable atomic units and reorganizes the required units according to the training topology, reducing redundant data movement. After admission, data preparation and delivery proceed

Trajectory in RL Systems

A trajectory records one rollout from an input prompt, including generated tokens and any intermediate interactions. We use sample to refer to the prompt together with its associated trajectories. For example, GRPO-style algorithms roll out the same prompt 𝑛 times [34], producing 𝑛 trajectories. Beyond the generated token IDs and interaction history, 2

KnitTrans : Coordinated Trajectory Transfer for LLM Reinforcement Learning

Train

Stale. Roll.

Sync.

Partial

1.5

0.70

1.0

quality gap

0.5 0.0

architectural staleness 1 2

training step

Collocated staleness Disaggregated staleness (a)

0.75

avg@n

Rollout

avg. staleness

Disaggregated vs. Collocated

0.65 0.60 0.55

Collocated avg@n Disaggregated avg@n

(b)

Figure 2: Architectural staleness under partial rollout. (a) Colocated and disaggregated execution with batch-level streaming, where 𝑖 denotes the 𝑖-th version of parameters, assuming the durations of the two stages are balanced. (b) Average staleness and avg@𝑛 of VeRL (colocated) and slime (disaggregated). trajectory payloads also include rewards and per-token auxiliary data such as routing decisions [26] and top-𝑝 sampling masks [38], which typically occupy tens to thousands of times as many bytes as the token IDs. Trajectories may also carry multimodal inputs such as images and videos, which can account for an even larger share of the payload in some tasks [8, 21]. When the prompt contains multimodal inputs, these trajectories share the same inputs. We use trajectory transfer to denote the end-to-end process of making completed rollout trajectories available to training, including admission, storage, and data preparation and delivery to training workers. Trajectory completion and training admission are distinct: completion means that a trajectory has finished rollout, whereas admission involves not only determining whether it is accepted under the RL algorithm, but also whether it is selected to form the next training batch. This distinction is particularly relevant in streaming rollout, where trajectories progress individually rather than as a fixed rollout batch [10, 53]. The transfer path involves a batching hierarchy of trajectories already present in RL training [33, 35, 47]. A train batch is the batch whose training completion triggers parameter synchronization in disaggregated systems or the stage switch in colocated systems. A mini-batch is the unit whose completion triggers a parameter update on the training side. A micro-batch is the unit of one forward–backward pass.

2.2

to long-tail stalls, whether execution is synchronous or onestep asynchronous [15, 28]. Partial rollout, an off-policy approach to reducing such waits, pauses unfinished trajectories and resumes generation with updated parameters in later training steps. This technique applies not only to disaggregated systems but also to colocated systems [20]. The staleness measures the gap in parameter versions between data generation and training. It is defined per token: a token generated with parameter version 𝑣 and trained at version 𝑣 ′ has staleness 𝑣 ′ − 𝑣; the staleness of a trajectory usually refers to that of its oldest token. Stale trajectories are off-policy for the current model, and excessive staleness can degrade training quality. Existing asynchronous RL systems incorporate staleness-control mechanisms to limit this version gap [10, 23]. However, disaggregated pipelining [25, 28] introduces architectural staleness beyond that caused by cross-step generation: rollout continues during 𝑇𝑖 , but the resulting data is consumed by 𝑇𝑖+1 , after the parameters have advanced by one version (Figure 2(a), where shading highlights rollout intervals whose generated tokens are stale due to pipelining). Figure 2(b) confirms this in practice: under the same configuration and staleness-control algorithm, the disaggregated system has an average staleness about one step higher than that of the colocated system. Its avg@𝑛 — the mean reward over the 𝑛 rollouts of a prompt — falls behind accordingly. These observations motivate reducing architectural staleness in the rollout-to-training path.

Architectural Staleness

RL systems can be classified by where rollout and training are placed: disaggregated systems dedicate a separate resource pool to each stage, allowing the stages to be pipelined [14, 39, 53], while colocated systems time-share one pool of devices between rollout and training and switch stages at their boundaries [35, 46]. Regardless of placement, waiting for all scheduled trajectories to finish can expose training

MOTIVATION

We examine two challenges along the rollout-to-training path: enabling early admission while preserving batch-wide training requirements (§3.1), and aligning trajectory data access with the spatial and temporal needs of distributed training (§3.2). 3

Bangjun Xiao et al.

Trajectory Skewness

Rollout duration (s)

10000

Linear Admission Timeline MB 1

8000

R0 R0 R0

MB 2

6000

R1 R1 R1

Rollout

MB 3

4000

MB 4

2000 0

80%

0 1000 step begin

B (a)

R0 T0

2000

3000

4000

5000

Stale. Roll.

Train T2

Sync.

6000

Rollout time

(b)

(c)

Figure 3: Rollout skew, delayed admission and larger architectural staleness. (a) Per-source rollout durations: bars show means; arrows show ± one standard deviation. (b) Mini-batch admission times under linear cumulative source targets. (c) Delayed (top) versus earlier (bottom) admission at the same mini-batch granularity.

Admission under Train-Batch Constraints

100%

share of step time

3.1

RL training can impose batch-wide requirements that cannot be checked on each trajectory independently. Mixed RL uses target mixing ratios across data sources [11, 43], while partialrollout completion quotas require a minimum completion ratio for each rollout step to mitigate long-tail starvation. These requirements govern either the composition of a train batch or the rollout progress required for admission. Enforcing these requirements by assembling a complete train batch before releasing any trajectories creates a train-batch admission barrier: training waits until all admission conditions hold, even when some completed trajectories have already passed their individual acceptance checks. Existing trajectory streaming systems [13, 53] allow training to consume smaller groups of trajectories as rollout progresses, but do not address admission under these batch-wide requirements. When rollout completion times are skewed across data sources (Figure 3(a)), admitting trajectories as soon as they complete can violate the final batch composition. For example, consider a train batch of 100 trajectories requiring 50 from each of two sources, A and B, divided into four mini-batches of 25. If A completes faster, admitting 75 trajectories from A in the first three mini-batches leaves only 25 slots for B. Even filling the last mini-batch entirely with B yields a 75:25 composition rather than the required 50:50. Once these trajectories enter training, later admissions cannot repair the earlier allocation. Retaining the train-batch admission barrier to enforce these requirements, however, blocks early consumption and preserves the batch-level pipeline illustrated in Figure 2(a). The associated architectural staleness degrades training quality (§2.2). A natural strawman is to scale each source’s cumulative quota linearly with mini-batch admission progress. Under such skew, however, requiring every source to reach a proportional target at each admission boundary can delay even the first mini-batch. With four mini-batches per

Omni RL Composition

80% 60%

36%

54%

Grounding

Heavier Grounding

rollout transfer train

40% 20% 0%

Figure 4: RL step-time breakdown on two Omni RL workloads using VeRL with sample-level transfer. train batch, Figure 3(b) shows that nearly 80% of the interval from rollout start to final admission elapses before the first mini-batch is admitted. Figure 3(c) compares schedules at the same mini-batch granularity: delayed admission (top) postpones training completion and parameter synchronization, increasing the shaded rollout fraction and thus architectural staleness relative to earlier admission (bottom). Thus, finer admission granularity alone does not ensure early streaming. The challenge is to enable early mini-batch admission under uneven rollout progress by overcoming the train-batch admission barrier while preserving the final train-batch requirements. KnitTrans addresses this challenge through Constraint-Preserving Mini-Batch Admission (§5.1).

3.2

Trajectory Data Access

Even after trajectories are admitted, preparing and delivering their large payloads can delay training. In pre-training, inputs for the next training step can be prepared from an existing dataset while the current step executes [51]. RL instead consumes trajectories generated online by rollout, making it difficult to hide data preparation entirely in the background. Data preparation and delivery thus remain on the critical path between rollout and training. On two colocated Omni RL workloads, rollout-to-training transfer accounts for 36% and 54% of a training step, respectively, exceeding the training computation time in both cases (Figure 4). 4

KnitTrans : Coordinated Trajectory Transfer for LLM Reinforcement Learning (2) Rollout Constraint-Preserving Workers metadata Mini-Batch (1) Admission (§5.1)

Storage Granularity and Spatial Demand. Storage organization determines which data can be accessed independently, while the training topology determines which data each worker needs. Existing sample-level storage layouts [13, 27] organize data around sample boundaries rather than these consumer-specific requirements. Large-scale training combines data parallelism (DP), context parallelism (CP), and tensor parallelism (TP). TP requires all ranks in a TP group to process bit-wise identical micro-batches, whereas CP allows each rank to process only a slice of a long sequence. A samplelevel layout therefore either repeatedly transfers the same data to multiple ranks or transfers whole samples when a rank needs only a subset of their data. Alternatively, avoiding these redundant transfers requires fetching once and broadcasting (TP) or scattering (CP) locally, which removes the duplication but inserts an extra synchronization hop whose latency lands on the critical path, slowing down the training process. Omni RL further exposes this mismatch because multimodal encoders can have assignments distinct from the backbone’s. Their encoding cost is determined by the multimodal inputs and requires separate DP balancing [52]; their TP and CP degrees can also differ from the backbone’s [9, 36]. Some systems even place encoders in a separate cluster [49]. When multimodal inputs are fetched according to the backbone’s sample assignment, they must then be redistributed to the ranks assigned to encode them, introducing additional data transfer. Data access must therefore account for the encoder and backbone topologies separately, rather than using the backbone’s assignment for all trajectory data.

(3) DP balance & CP splitting

(4)Assigned Units

Training Workers Multimodal Encoding Deduplication(§5.3)

On-Demand Trajectory Data Access(§5.2) trajectory data ①

Atomic Unit Storage

③

Topology-Aware Reorganization

②

④

⑥

Local ⑤ shm Consumption-Driven Data Delivery

Figure 5: Overview of KnitTrans.

SYSTEM OVERVIEW

Figure 5 shows the end-to-end data flow of KnitTrans, from completed rollout trajectories to their admission and consumption by training workers. When a trajectory completes, rollout workers report its metadata to the control plane (1) and write its payload to dis1 Atomic-unit storage ○ 2 records the paytributed storage ○. load as independently addressable units organized through a hierarchical index, allowing subsequent access according to training assignments. Using the reported metadata, the control plane admits a mini-batch when its progressive readiness conditions and feasibility guards hold (2) (§5.1). It then determines DP balancing and CP splitting according to each consumer’s topology (3), and dispatches the resulting assignments (4) to TP-grouplocal reorganization processes. Following these assignments, the processes retrieve the required atomic units, using sliced 3 and pack them into reads for assigned sequence ranges ○, 4 publishing them incrementally in local training inputs ○, 5 shared memory in each consumer’s micro-batch order ○. Training workers prefetch the prepared inputs into pinned 6 overlapCPU memory and transfer them to the GPU ○, ping preparation and staging of subsequent micro-batches with ongoing computation (§5.2). For multimodal workloads, training workers reuse each unique input’s encoding across the admitted mini-batch and aggregate consumer gradients for encoder backward, while the encoder and backbone retain independent micro-batch partitions (§5.3).

Preparation Granularity and Temporal Demand. An admitted mini-batch does not require all of its data at once: training consumes it through a sequence of smaller microbatches. Unlike in pre-training, where inputs can be prepared a step ahead, retrieving large payloads from distributed storage and packing them into training inputs remain timeconsuming in RL, even without redundant data movement. Existing RL systems such as VeRL [35] and slime [39] support micro-batch execution, but their batch-oriented training paths leave payload retrieval and input packing serialized with training computation on the critical path, further prolonging training time. The underlying issue is the lack of on-demand data preparation aligned with the temporal needs of individual micro-batches.

DESIGN

The key is to separate logical training boundaries from physical execution granularity. Train-batch requirements need not force train-batch admission (§5.1); admitting a mini-batch need not require materializing all of its data at once (§5.2); and micro-batch execution need not repeat the encoding of shared inputs (§5.3).

5.1 The challenge of trajectory data access is therefore to align trajectory data access with both where data are consumed and when they are needed, avoiding redundant movement and unnecessary waiting. KnitTrans addresses this challenge through On-Demand Trajectory Data Access.

Constraint-Preserving Mini-Batch Admission

KnitTrans forms mini-batches for admission from completed rollout trajectories by combining progressive rollout-side readiness conditions with feasibility guards on candidate batch composition. We first express the original train-batch 5

Bangjun Xiao et al. admission gate as a conjunction of predicates: Ó BatchReady = 𝑖 𝑃𝑖 ,

Atomic Units

(1) Topology-aware Distributed Storage

where each predicate 𝑃𝑖 captures one requirement on the train batch, such as reaching the train-batch size, meeting a partial-rollout completion quota, or maintaining a target data-source ratio [43]. For example, ROLL assembles a training batch only after every domain meets its quota. For quota-based requirements, we first define rollout-side readiness conditions: 𝑃𝑖 : 𝑠𝑖 ≥ 𝜏𝑖 ,

local shm

Training Workers

reorg0

TP0 TP1

CP0

reorg1

TP0 TP1

CP1

traj. data

Figure 6: End-to-end workflow of On-Demand Trajectory Data Access.

(2) the cumulative count of class 𝑖 in the train batch if the candidate mini-batch is admitted, in contrast to the rollout-side count 𝑠𝑖 . Let 𝑏 0,𝑡 denote the total admitted count including this candidate. We partition batch-composition quotas into groups 𝐺 1, . . . , 𝐺𝑚 whose target classes compete for the same batch slots. For example, data-source quotas form one group, while partial-rollout completion quotas remain rollout-side readiness conditions rather than batch-capacity guards. For the candidate’s prospective status 𝐵𝑡 = {𝑏𝑖,𝑡 }, we check one guard for each group: + Í 𝑄𝑘 (𝐵𝑡 ) : ≤ 𝜏0 − 𝑏 0,𝑡 , (4) 𝑖 ∈𝐺𝑘 𝜏𝑖 − 𝑏𝑖,𝑡

where 𝑠𝑖 is the rollout-side count for quota 𝑖, and 𝜏𝑖 is its final target. For example, if a data source should contribute at least a fraction 𝑟𝑖 of a train batch of size 𝜏0 , the corresponding readiness condition is 𝑠𝑖 ≥ 𝜏𝑖 , where 𝜏𝑖 = 𝑟𝑖 𝜏0 ; analogous quotas apply to the other sources. Let 𝑠𝑖,0 denote the count at the beginning of the current train batch. For the 𝑡-th minibatch, we set a progressive target between 𝑠𝑖,0 and 𝜏𝑖 using predicate-specific interpolation: 𝜏𝑖,𝑡 = 𝑠𝑖,0 + 𝛼𝑖,𝑡 (𝜏𝑖 − 𝑠𝑖,0 ),

Hierarchical Index

𝑡 = 1, . . . , 𝑛,

(3)

where 0 ≤ 𝛼𝑖,1 ≤ · · · ≤ 𝛼𝑖,𝑛 = 1. The coefficients specify a configured schedule for each predicate. For equal-sized minibatches, the train-batch-size target grows linearly. Source targets instead use relaxed early coefficients and reach the original quotas at the final admission. This accommodates uneven rollout progress without requiring every readiness target to grow in proportion to the train-batch size. Before admitting the 𝑡-th mini-batch, the rollout-side counts must satisfy 𝑃𝑖,𝑡 : 𝑠𝑖 ≥ 𝜏𝑖,𝑡 , allowing earlier mini-batches to proceed before all final targets are reached. Figure 1 illustrates one such schedule, with cumulative source-readiness targets (0, 0, 2, 4) instead of linear targets (1, 2, 3, 4). Both schedules reach the same final source quotas, but the former permits earlier admission under the illustrated rollout skew. However, progressive readiness alone does not ensure final train-batch feasibility. Consider the two-source example from §3.1: a train batch requires 50 trajectories from each source, admitted through four mini-batches of 25. Suppose each source’s readiness target is 10 at the third admission and reaches 50 at the final admission. The current readiness conditions can hold even when the first three admitted minibatches contain 65 trajectories from A and 10 from B. Only 25 slots then remain for the 40 additional trajectories required from B, making the final quotas unattainable. We therefore append feasibility guards to the per-minibatch predicates. Once all progressive rollout-side readiness predicates 𝑃𝑖,𝑡 hold, an application-specific selection policy proposes a candidate mini-batch. We then apply the guards to check whether the candidate can be admitted without making the final train-batch quotas unattainable. Let 𝑏𝑖,𝑡 denote

where (𝑥) + = max(𝑥, 0). The right-hand side is the remaining capacity of the train batch, while the left-hand side is the outstanding quota deficit for the group. The guards prevent an irrevocable mini-batch decision from making the final train batch infeasible. The guard rejects any candidate that would produce the composition above. The admission predicate is therefore Ó Ó BatchReady𝑡 = 𝑖 𝑃𝑖,𝑡 ∧ 𝑘 𝑄𝑘 (𝐵𝑡 ). (5) The candidate for mini-batch 𝑡 is admitted only when BatchReady𝑡 holds. Individual mini-batches may therefore deviate from the final train-batch quotas, while the full train batch meets all original requirements.

5.2

On-Demand Trajectory Data Access

KnitTrans aligns trajectory data access with the spatial and temporal needs of training (Figure 6). Atomic-unit storage and topology-aware reorganization determine which data are prepared for each consumer. Together, they form the Topology-Aware Distributed Storage shown in Figure 6. Consumption-driven delivery coordinates data preparation with training progress within an admitted mini-batch. Atomic-Unit Storage. When the rollout stage writes into distributed storage, KnitTrans stores data as independently addressable atomic units that match the DP-balancing granularity specified by each training topology. In GRPO, for example, a complete trajectory is one atomic unit: once admitted for training, it is batched and processed independently 6

KnitTrans : Coordinated Trajectory Transfer for LLM Reinforcement Learning of the other trajectories generated from the same prompt, so the sample as a whole is not atomic. The storage representation does not fix CP slices at write time: CP splitting depends on the training batch and is determined after admission. For multimodal data, an atomic unit is an image or a group of consecutive video frames, as defined by the model architecture. A complete sample is then recorded with a hierarchical index: a sample maps to the storage keys of its trajectories, and each trajectory in turn maps to the storage keys of the other atomic units it references, mainly multimodal ones. The hierarchical index allows each atomic unit to be addressed independently, as illustrated in Figure 6. This storage representation is defined at the logical storage layer and is independent of the physical backend, allowing the same atomic-unit abstraction to operate over Mooncake, the Ray object store, or other backends. Topology-Aware Reorganization. When a mini-batch is admitted, the RL control plane determines data assignments according to each consumer’s topology, performing DP balancing [17, 41, 52] and CP splitting [6] based on metadata such as trajectory lengths and the computation costs of other atomic units, helping to reduce memory pressure. Each assignment is dispatched to a reorganization process on the local node of the corresponding TP group. Each process serves one CP partition and is shared by the TP ranks processing that partition. This organization addresses two complementary requirements: • A separate process for each CP partition can retrieve only the sequence slices assigned to it within the admitted mini-batch. To support these reads uniformly in the logical storage layer, KnitTrans adds a sequence-length field to atomic storage units where applicable and extends the read interface with an optional sequence-offset parameter. The logical storage layer coordinates with the physical backend to perform the corresponding slice reads, supporting both memory-based backends [32, 40] and disk-based backends. • Sharing one process across a TP group avoids repeated retrieval of the same trajectory data and centralizes input packing for the group’s forward computation. Since TP ranks are typically colocated on a physical node for efficient communication, node-local shared memory can distribute the prepared inputs without an extra fetchthen-broadcast step among the ranks.

for the group’s training workers. In Omni RL, multimodal encoders have distinct topologies, so each encoder type has its own storage configuration and reorganization processes, applying the same retrieval, packing, and publication flow separately from the backbone. Consumption-Driven Data Delivery. The assignment is determined over a mini-batch, but its payload need not be materialized all at once. The reorganization process executes reads and packing incrementally in each consumer’s microbatch consumption order, notifying the training workers as prepared inputs become available in shared memory. Each training worker uses a background thread to prefetch these inputs into pinned CPU memory. The training loop then asynchronously copies each prefetched micro-batch to the GPU for computation, instead of waiting for the entire minibatch to become ready. Figure 1 shows how the multimodal and backbone reorganization processes incrementally supply their respective computation stages. This coordination allows both reorganization and pinned-memory staging for later inputs to overlap with GPU computation on earlier inputs. This overlap reduces data-preparation delays on the training critical path without changing mini-batch admission or optimizer-update boundaries.

5.3

Multimodal Encoding Deduplication

GRPO-style RL produces multiple trajectories that share the same multimodal inputs. Using the storage keys recorded in the hierarchical index (§5.2), KnitTrans identifies shared multimodal inputs within each mini-batch. However, sharing stored input data does not itself eliminate repeated encoding during training. The training-side challenge is a mismatch in execution scope: encoding reuse spans an entire mini-batch, while the backbone consumes the shared encodings through separate micro-batch forward and backward passes. Without additional handling, the first backward pass through a shared encoding releases the saved encoder activations needed by later micro-batches. Keeping the encoded tensor available is therefore insufficient to reuse its encoder computation graph across all consumers. KnitTrans enables mini-batch-wide training with a single encoding of each unique input, preserving the gradient contributions of all consuming trajectories. First, KnitTrans stores each unique multimodal input as a single atomic unit in distributed storage, with all references to that input sharing the same storage key. The control plane also treats each unique input as a single unit for encoder-side DP balancing, assigning it to only one group of ranks for encoding. KnitTrans then separates the lifetime of the encoder graph from the backbone’s micro-batch execution. It encodes each unique multimodal input once per mini-batch and retains the corresponding encoder graph. Since multimodal inputs and

As shown in Figure 6, the process follows the hierarchical indices to locate the required atomic units. It then uses the training framework’s packing function to prepare the group’s inputs, including concatenating trajectories, applying padding where needed, and constructing attention masks or sequence-boundary metadata for packed attention [17, 29]. The resulting inputs are published in local shared memory 7

Bangjun Xiao et al. def train_minibatch(batch, plan): optimizer.zero_grad() 3 encoded = [] 4 # Encode unique inputs; retain graphs 5 for micro in encoder_micro_batches(batch, plan): 6 outputs = encoder(micro.inputs) 7 encoded.append((micro.ids, outputs)) 8 # Topology-aware routing and caching 9 cache, refs, route = route_detached( 10 encoded, batch, plan) 11 cache_grad = zeros_like(cache) 12 # Reuse encodings; accumulate gradients 13 for micro in backbone_micro_batches(batch): 14 ids = refs[micro.id] 15 features = cache[ids].detach().requires_grad_() 16 result = backbone(micro, features) 17 normalized_loss(result).backward() 18 cache_grad.index_add_(0, ids, features.grad) 19 # Return gradients; deferred encoder backward 20 owned_grad = tp_owned_slices(cache_grad, route) 21 encoder_grad = reverse_route_sum(owned_grad, route) 22 for ids, outputs in encoded: 23 backward(outputs, encoder_grad[ids]) 24 finalize_model_grads() 25 optimizer.step()

throughout the mini-batch, deduplication reduces the number of cached encodings, while releasing micro-batch-local copies as gradients accumulate before encoder backward alleviates peak memory pressure. The encoder partitions the mini-batch’s unique multimodal inputs into its own micro-batches, independently of the backbone’s micro-batch partitioning. Consumptiondriven delivery (§5.2) overlaps preparation of later encoder micro-batches with computation on earlier ones. The encoder graphs remain available until the gradients from all backbone micro-batches have been accumulated and returned for encoder backpropagation. Thus, both modules retain their own execution granularity while encoding reuse spans the entire mini-batch. Listing 1 summarizes the forward and backward flow for one mini-batch.

1 2

IMPLEMENTATION

KnitTrans is built on VeRL, with SGLang as the inference engine and Megatron as the training engine. We extend VeRL with partial rollout [20] and slime-style disaggregated execution. Its topology-aware distributed storage builds on TransferQueue [40], an open-source library for data storage in post-training workflows that disaggregates the control plane and data plane in RL trajectory transfer.

Listing 1: Mini-batch-wide multimodal encoding reuse.

TQClient Extensions. We override the write interfaces of TransferQueue’s TQClient to support both whole-trajectory writes and updates to individual fields. Each field’s topology identifier selects the corresponding splitting function, which decomposes its data into atomic units. The client stores these units and returns the index entries for the updated fields. On the training side, initializing TQClient with a TopologyConfig connects workers to a shared node-local Ray actor. After accepting admission and assignment results, the actor retrieves the required atomic units from TQClient. The actor then packs the retrieved data and materializes the prepared inputs in Ray’s Plasma object store. Local workers share the materialized arrays, while Ray’s inter-node object transfer also supports TP groups spanning multiple nodes. For NumPy-compatible CPU tensors, we expose NumPy views and reconstruct tensor views at the receivers, allowing local workers to access the shared arrays without additional copies. Multimodal data use a separate TQClient configured for the encoder topology, following the same storage and materialization path. By keeping this refactoring entirely within TransferQueue, we make the design of KnitTrans easy to adapt to other RL frameworks.

backbone data are independently DP-balanced, the encoding and consuming ranks need not coincide. Using both sides’ topologies and assignments, KnitTrans maps each unique encoding’s source to all consuming destinations, including trajectory repeats, TP replicas, and copies induced by CP boundaries. It distributes detached encoding values along this mapping and caches them at the consuming ranks for reuse within the mini-batch. Each backbone micro-batch gathers its required encodings from the local cache as detached inputs that require gradients. The backbone’s backward passes stop at this boundary without traversing the encoder graph. The resulting gradients are accumulated locally by encoding identity, combining contributions from repeated references within and across micro-batches. After all backbone micro-batches finish, KnitTrans returns and aggregates the accumulated gradients according to the encoding distribution mapping. Each TP rank contributes only its assigned gradient slice, avoiding duplicate accumulation across TP replicas. The encoding ranks then backpropagate the aggregated gradients through the retained encoder graphs before the optimizer update. Figure 1 places the encoder’s forward pass before the backbone micro-batch passes and its backward pass after them, reflecting this deferred encoder backward. Caching encodings and accumulating their gradients locally avoids repeating encoder–backbone transfers for each micro-batch. Regarding the additional GPU memory required to cache encodings

Sliced Reads. At the logical layer, TQClient’s read interface accepts optional field_ranges specifying per-field, persample sequence ranges, which the storage manager converts to byte offsets and slice shapes when required by the backend. At the physical layer, TransferQueue’s in-memory 8

architectural staleness (steps)

KnitTrans : Coordinated Trajectory Transfer for LLM Reinforcement Learning storage units slice tensors before transmission, Mooncake uses get_into_ranges to fetch the requested bytes, and the disk backend reads the corresponding ranges from sharedfilesystem shards with preadv. Micro-Batch Delivery. As each micro-batch is materialized, the reorganization actor publishes its Ray object handle in a thread-safe dictionary indexed by micro-batch. Training workers call await tq_client.get_next_micro() to retrieve the next handle, obtaining the materialized layout without waiting for the entire mini-batch. After the firstlayer kernel of the current micro-batch is launched, CUDA events coordinate staging of the next micro-batch into pinned host memory.

1.00

1.01

0.96

0.94

0.75 0.50 0.25 0.00

0.34 0.11

Qwen3-VL-235B train-batch

0.25

0.31

0.19

MiMo-v2.5-309B mini-batch(4)

0.07

DeepSeek-v3.2-671B mini-batch(8)

Figure 7: Architectural staleness of the disaggregated system on the CodeAgent task. over the global communication group according to the multimodal scheduling plan. • Miles (Miles) [27]: uses sample-level storage, like TransferQueue, with a Mooncake KV backend and RDMA transport for efficient data access. • MegaScale-Data (MegaData∗ ) [51]: a topology-aware data preprocessing system designed for pre-training and adapted here to RL trajectory transfer. Distributed storage optimization is not its focus, so this baseline still reads data from TQ’s sample-level storage. Since the original code is not open-sourced, our implementation is based on the paper’s description.

EVALUATION

We evaluate KnitTrans’s end-to-end performance (§7.2) and RL quality (§7.3), and use ablation studies (§7.4) to isolate the contributions of individual designs.

7.1

Architectural Staleness

1.25

Experimental Setup

Tasks and Datasets. We evaluate KnitTrans on two RL tasks. The first is a CodeAgent task, for which we train on a mixture of well-known software engineering datasets: SWE-smith [44], Multi-SWE-bench [48], Terminal-Lego [45], and Scale-SWE [50]. The second is Video Grounding, representing multimodal tasks. We use two in-house datasets, Grounding A and Grounding B, that differ in their fraction of video data to cover different multimodal payloads. The maximum context length is 256K for CodeAgent and 64K for Video Grounding.

Testbed and Other Settings. All experiments run on a GPU cluster of 32 nodes, each equipped with 8 NVIDIA Hopper- architecture GPUs and interconnected via highbandwidth NVLink and RDMA networks. We use GRPO [34] with 𝑛 = 16 rollouts per prompt as the RL algorithm, together with R3 [26], which replays the routing decisions of the rollout engine during training to reduce the training-inference discrepancy. We filter out samples whose staleness exceeds an upper bound to limit the parameter-version gap between generation and training. We also enforce a partial-rollout completion quota—a minimum completion ratio for every rollout step—to mitigate long-tail starvation. For rollouttraining placement, every module in the colocated setting occupies all the GPUs. In the disaggregated setting, we allocate separate GPU pools to balance their durations as closely as possible, using a 3:1 rollout-to-training GPU split in the CodeAgent experiments.

Models. We train three large models for large-scale RL experiments: Qwen3-VL-235B [1], MiMo-V2.5-309B [24], and DeepSeek-v3.2-671B [4]. The first two understand multimodal inputs and are trained on both tasks, whereas DeepSeek-v3.2 is trained only on the CodeAgent task. All models use FP8 quantization for both training and inference. Baselines. Our baselines are VeRL [35] for colocated execution and slime [39] for disaggregated execution. We align their staleness control algorithms to isolate the architectural contribution to staleness. We also use StreamRL [53] and AsyncFlow [13] as streaming baselines without batchwide admission constraints. For distributed storage and data movement, we compare three configurations within the same training framework:

7.2

End-to-End Performance

Architectural Staleness. We examine whether mini-batch admission narrows the architectural staleness gap between the two architectures. We use the CodeAgent task, which exhibits pronounced long-tailed rollout durations, with the staleness bound set to 4 in all runs. For each model, we run the disaggregated system with train-batch admission, i.e., batch-level streaming, and with KnitTrans’s mini-batch admission at 4 and 8 mini-batches per train batch. Architectural

• TransferQueue (TQ) [13, 40]: an open-source Python library with sample-level storage and no topology-aware reorganization. Each rank reads the backbone data for its assigned trajectories. Only backbone ranks with both TP rank 0 and CP rank 0 read multimodal data, which are then redistributed through an all-to-all collective 9

Bangjun Xiao et al.

Omni RL

seconds per step

1561 1500 505 476

2.2 s

1000 500 0

transfer

1083

1472

1175 1165 1173 1127 495 479

train

559 496

2.3 s

1005 2.1 s 583 523

338 332 337 291 443 454 458 380 180 193 1922.4 s 129 TQ Miles Mega* Knit TQ Miles Mega* Knit TQ Miles Mega* Knit TQ Miles Mega* Knit Data Trans Data Trans Data Trans Data Trans

Qwen3-VL Grounding A

MiMo-v2.5 Grounding A

Qwen3-VL Grounding B

MiMo-v2.5 Grounding B

Figure 8: Exposed transfer and training computation latency per step on Omni RL.

avg@n

staleness is measured as the average difference between the per-step staleness of each run and that of the colocated run under the same configuration, taken over the steps after the staleness has stabilized (step > staleness bound). Figure 7 shows that mini-batch admission substantially reduces architectural staleness on all three models. On Qwen3VL-235B, architectural staleness falls from 0.94 steps under train-batch admission to 0.34 at 4 mini-batches and 0.11 at 8; on MiMo-V2.5-309B, it falls from 0.96 to 0.25 and 0.19; and on DeepSeek-v3.2-671B, from 1.01 to 0.31 and 0.07, respectively. Deviations from the idealized results (0.25 at 4 mini-batches and 0.125 at 8 mini-batches) are consistent with per-step variation in rollout and training durations, which prevents perfectly balanced overlap. These results demonstrate the effectiveness of finer-grained admission. Four mini-batches per train batch already capture most of the benefit; increasing the number to eight yields diminishing returns without eliminating architectural staleness.

0.775 0.750 0.725 0.700 0.675 0.650 0.625

RL convergence

VeRL Slime KnitTrans

training step Figure 9: RL convergence: avg@𝑛 per training step on the CodeAgent task with MiMo-V2.5.

about 380–520 s per step, accounting for roughly 76–90% of TQ’s exposed transfer latency across these workloads. The transfer-plus-training speedup is 1.5–5.8×. Miles remains close to TQ in exposed transfer latency, mainly because of the large number of small tensors in trajectory data: transferring fragments individually incurs repeated memory-registration and Store-operation overheads [27]. Meanwhile, changing the transport backend alone does not remove the aforementioned main overhead. The adapted MegaData∗ baseline suffers from extensive redundant reads caused by sample-level storage, leading to severe congestion in distributed storage and stragglers among training workers that independently fetch trajectory data. This implies that the storage design—specifically, atomic-unit storage—remains essential even with topology-aware preprocessing. These results highlight the importance of matching storage units to the training topology, rather than optimizing transport or preprocessing in isolation. The evaluation results also show that KnitTrans reduces training time through Multimodal Encoding Deduplication. The combined benefit

Transfer Latency. We then measure how much KnitTrans shortens the rollout-to-training path. To compare this path across both architectures despite their different rollouttraining overlap, we measure transfer latency exposed on the training critical path and training computation time per step, comparing KnitTrans with the three baselines introduced in §7.1. Figure 8 reports all four configurations on both Omni RL models and datasets, averaged over the steps after warmup. Across all four model–dataset pairs, KnitTrans reduces exposed transfer latency by 217.6–264.3×, a reduction of two orders of magnitude, relative to TQ, leaving training computation as the dominant cost. The redistribution of multimodal inputs through collective communication takes 10

KnitTrans : Coordinated Trajectory Transfer for LLM Reinforcement Learning A

0.5

StreamRL/AsyncFlow

Final target (0.24)

Unmet target

w/o guards

Architectural staleness

Admitted / train batch

0.4

0.6

0.3

0.47

0.2 0.1

0.4

0.0 0.5

Linear

0.28

KnitTrans 0.2

0.4 0.3 0.2

0.0

0.1 0.0 1

Step

Linear

KnitTrans

(a) Per-source batch composition.

(b) Architectural staleness.

Figure 10: Constraint preservation and architectural staleness under different admission policies. is larger on Grounding B, where transfer and encoding account for more of the rollout-to-training path. Conversely, the gain is more limited on Qwen3-VL Grounding A, where backbone computation already dominates due to its fullattention architecture. The ablations in §7.4 separate the contributions of these mechanisms.

7.3

per-source targets, which are initially relaxed to allow earlier admission and tighten across mini-batches. KnitTrans further adds feasibility guards to these targets. Figure 10(a) shows batch composition during admission. Without considering batch-wide constraints, StreamRL/AsyncFlow violates a final quota in every step. w/o guards also incurs violations: admitting mini-batches once the current targets are met allows faster sources to occupy too many slots early. Linear and KnitTrans satisfy all final quotas, but differ in streaming benefits. During trace replay, we measure architectural staleness under these two mini-batch admission policies, as shown in Figure 10(b). KnitTrans lowers the average architectural staleness from 0.470 to 0.286, a 39.2% reduction relative to Linear. Relaxed targets with feasibility guards preserve final quotas while enabling earlier admission than linear targets.

RL Quality

We evaluate KnitTrans’s impact on RL quality from two perspectives: RL Convergence, which assesses how reduced architectural staleness affects learning, and Constraint Preservation, which tests whether mini-batch admission preserves algorithm-defined train-batch constraints while retaining the benefits of streaming. RL Convergence. §7.2 showed that mini-batch admission reduces architectural staleness; we further examine its effect on RL convergence. We conduct experiments on MiMo-V2.5 using three configurations: the colocated system (VeRL), the disaggregated system with train-batch admission (slime), and KnitTrans with 4 mini-batches per train batch. All other settings remain unchanged. We report avg@𝑛 once staleness has stabilized. Figure 9 reports the curves. KnitTrans closely follows the colocated run, with a mean gap of at most 0.007. Train-batch admission instead trails the colocated arm by 0.082 avg@𝑛 on average. Reducing architectural staleness from 0.96 to 0.25 steps thus narrows the learning-quality gap between the disaggregated and colocated systems. Constraint Preservation. We test the trade-off in §5.1 using a trace-driven admission replay. All policies use the same source-local rollout durations and acceptance decisions. We use the four data sources A–D in Figure 3(a), each with a target ratio of 24%. StreamRL/AsyncFlow admits accepted samples on completion without enforcing source quotas. Linear uses linear interpolation to set progressive per-source targets. w/o guards instead enforces the current cumulative

7.4

Ablation Study

We examine topology-aware storage and the two trainingside optimizations on the Omni RL task in the colocated setting, using both multimodal models on Grounding A and Grounding B. Each pair replays the same traced training batches. We report the mean over the steps after the warmup. Topology-Aware Distributed Storage. Figure 11(a) compares TQ with TADS, which organizes trajectory data, especially multimodal inputs, into independent atomic units and fetches them according to the training topology, so each rank reads exactly the multimodal units it consumes. Eager fetching and duplicated encoding remain unchanged. Across the four model–dataset pairs, exposed transfer falls from 495–583 s to 21.6–26.1 s per step, reducing total transfer-plustraining time by 28.8–72.0%. These gains come from reducing redundant fetches and redistributions of multimodal inputs, without relying on micro-batch fetch overlap. 11

1000 800 600 400 200

train transfer

505 25.9 s

1175 1170

559

495

21.6 s

26.1 s

338 335

443 458

583 21.6 s

180 192 TQ TADS TQ TADS TQ TADS TQ TADS Qwen3-VL MiMo-v2.5 Qwen3-VL MiMo-v2.5 Grounding A Grounding A Grounding B Grounding B

Multimodal Encoding Deduplication

Micro-Batch Fetch Overlap 30 25 20 15 10 5 0

train (compressed; labels in s)

26.1

25.9

transfer

21.6

2.5

2.6

1170 1161

335 334

458 455

192 193

seconds per step

Topology-Aware Distributed Storage

transfer time per step (s)

Bangjun Xiao et al. 57.0 s

500 400 300 200 100

eager iter eager iter eager iter eager iter Qwen3-VL MiMo-v2.5 Qwen3-VL MiMo-v2.5 Grounding A Grounding A Grounding B Grounding B

(a)

(b)

6.7 s 49.9 s

1104 1121

81 4.8 s

backbone encode

6.7 s

374 376

70 6.2 s 123 122 dup dedup dup dedup dup dedup dup dedup Qwen3-VL MiMo-v2.5 Qwen3-VL MiMo-v2.5 Grounding A Grounding A Grounding B Grounding B 285 284

(c)

Figure 11: Layer-wise latency ablations: (a) sample-level TQ versus topology-aware distributed storage (TADS); (b) eager versus iter fetching on TADS; (c) duplicated (dup) versus deduplicated (dedup) multimodal encoding with iterator fetching. The eager configuration is TADS from (a), and dup is iter from (b). Micro-Batch Fetch Overlap. Figure 11(b) holds topologyaware storage and the encoding policy fixed, comparing eager materialization of the mini-batch with iter fetching and background prefetch. Exposed transfer falls from 21.6– 26.1 s to 2.5–2.6 s per step across the four configurations, an additional 8.4–10.4× reduction over TADS alone. The remaining exposed latency includes metadata-based scheduling, cluster-wide dispatch of the resulting plans, and reorganization of the first micro-batch. Fetching upcoming micro-batches during computation removes most of the remaining data wait from the critical path.

and Miles [27] provide sample-level storage for rolloutto-training transfer. Beyond RL, MegaScale-Data [51] coordinates distributed preprocessing and topology-aware data placement, while BatchWeave [37] addresses consistent batch publication and recovery over object storage. KnitTrans extends sample-level exchange with independently retrievable units matched to consumer-specific assignments. Unlike pre-training loaders operating on existing datasets, it coordinates preparation with online trajectory admission and subsequent micro-batch consumption. Multimodal Training Optimization. Prior multimodal training systems address heterogeneous encoder and backbone workloads through data balancing [52], execution scheduling [9], and disaggregated training [49]. Repeated RL trajectories introduce another source of redundant work: the same multimodal input is consumed across multiple backbone forward and backward passes. KnitTrans addresses how to share its encoder computation while preserving every consuming trajectory’s gradient contribution.

Multimodal Encoding Deduplication. Figure 11(c) compares dup and dedup with iterator fetching enabled in both. The latter encodes each unique multimodal input once per mini-batch and reuses its encoding across trajectories of the same sample. With other configurations remaining the same, encoder forward-plus-backward time falls from 49.9–81.5 s to 4.8–6.7 s per step, corresponding to a 4.3–33.2% reduction in total training time. Gains are larger on Grounding B and on MiMo-V2.5, where the encoder accounts for a greater share of training time.

CONCLUSION

We presented KnitTrans, a coordinated trajectory-transfer system for LLM RL. Its constraint-preserving mini-batch admission enables early training without waiting for full train-batch readiness, reducing architectural staleness in off-policy disaggregated execution. On-demand trajectory data access matches storage units and incremental delivery to each consumer’s assignments and micro-batch schedule. Multimodal encoding deduplication supports mini-batchwide reuse while preserving gradient contributions across backbone backward passes. Experiments demonstrate preserved source quotas, a narrower learning-quality gap with colocated training, and reduced data-transfer and encoding costs. These results show that effective trajectory transfer requires coordinating admission and consumption, rather than merely moving completed trajectories faster.

RELATED WORK

RL Streaming and Training Admission. Prior systems refine streaming granularity to overlap rollout and training in on-policy settings [3, 18]. StreamRL [53] and AsyncFlow [13] support fine-grained streaming in asynchronous rollout– training pipelines. Batch-wide requirements introduce an additional restriction on admission: mixed RL enforces target data-source ratios through per-domain quotas [11, 43], and waiting for all quotas to be met delays otherwise ready trajectories. KnitTrans addresses how to admit mini-batches before all train-batch conditions hold without making the final quotas unattainable. Trajectory Storage and Data Access. Earlier RL data systems support trajectory exchange through distributed experience storage [2, 42]. For LLM RL, TransferQueue [13, 40] 12

KnitTrans : Coordinated Trajectory Transfer for LLM Reinforcement Learning

REFERENCES

//doi.org/10.1145/3627703.3629565. [18] Zhida Jiang, Zhaolong Xing, Jiawei Lu, et al. 2026. Rollout-Training CoDesign for Efficient LLM-Based Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2602.09578 (2026). https://arxiv.org/abs/2602.095 78. [19] Bowen Jin, Hansi Zeng, Zhenrui Yue, et al. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv preprint arXiv:2503.09516 (2025). https://arxiv.org/ab s/2503.09516. [20] Kimi Team. 2025. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599 (2025). https://arxiv.org/abs/25 01.12599. [21] Kimi Team. 2026. Kimi K2.5: Visual Agentic Intelligence. arXiv preprint arXiv:2602.02276 (2026). https://arxiv.org/abs/2602.02276. [22] Kimi Team. 2026. Kimi K3: Open Frontier Intelligence. arXiv preprint arXiv:2607.24653 (2026). https://arxiv.org/abs/2607.24653. [23] Haoyang Li, Sheng Lin, Fangcheng Fu, et al. 2026. StaleFlow: StalenessAware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training. arXiv preprint arXiv:2601.12784 (2026). https://arxiv.org/abs/2601.12784. [24] LLM-Core Xiaomi. 2026. MiMo-V2-Flash Technical Report. arXiv preprint arXiv:2601.02780 (2026). https://arxiv.org/abs/2601.02780. [25] Michael Luo, Sijun Tan, Roy Huang, et al. 2025. DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level. Notion Blog. (2025). https: //pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-1 4B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51. [26] Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. 2026. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers. In International Conference on Machine Learning (ICML). https://arxiv.org/abs/2510.11370. [27] Mooncake Community. 2026. Mooncake for Miles: From Fragmented Rollout Data to Efficient Bulk I/O. (2026). https://www.lmsys.org/bl og/2026-08-20-miles-mooncake-rollout-data-transfer. [28] Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. 2025. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2410.18252. [29] NVIDIA. [n. d.]. NeMo RL: Sequence Packing and Dynamic Batching. Official documentation. ([n. d.]). https://docs.nvidia.com/nemo/rl/late st/design-docs/sequence-packing-and-dynamic-batching.html. [30] OpenAI. 2024. OpenAI o1 System Card. arXiv preprint arXiv:2412.16720 (2024). https://arxiv.org/abs/2412.16720. [31] Long Ouyang, Jeffrey Wu, Xu Jiang, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS). [32] Ruoyu Qin, Zheyu Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation—A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, 155–170. https: //www.usenix.org/conference/fast25/presentation/qin. [33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347 (2017). https://arxiv.org/abs/1707.06347. [34] Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300 (2024). https://arxiv.org/abs/ 2402.03300.

[1] Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. 2025. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631 (2025). https://arxiv.org/abs/ 2511.21631. [2] Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, et al. 2021. Reverb: A Framework for Experience Replay. arXiv preprint arXiv:2102.04736 (2021). https://arxiv.org/abs/2102.04736. [3] Rongjian Chen, Jianmin Hu, Kejiang Ye, and Minxian Xu. 2026. RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated OnPolicy LLM Reinforcement Learning. arXiv preprint arXiv:2606.26997 (2026). https://arxiv.org/abs/2606.26997. [4] DeepSeek-AI. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv preprint arXiv:2512.02556 (2025). https: //arxiv.org/abs/2512.02556. [5] DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient MillionToken Context Intelligence. arXiv preprint arXiv:2606.19348 (2026). https://arxiv.org/abs/2606.19348. [6] Jiarui Fang and Shangchun Zhao. 2024. USP: A Unified Sequence Parallelism Approach for Long Context Generative AI. arXiv preprint arXiv:2405.07719 (2024). https://arxiv.org/abs/2405.07719. [7] Jiazhan Feng, Shijue Huang, Xingwei Qu, et al. 2025. ReTool: Reinforcement Learning for Strategic Tool Use in LLMs. arXiv preprint arXiv:2504.11536 (2025). https://arxiv.org/abs/2504.11536. [8] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. 2025. Video-R1: Reinforcing Video Reasoning in MLLMs. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 38. 99114–99137. https://doi.org/10.52202/085713- 3310 https: //arxiv.org/abs/2503.21776. [9] Weiqi Feng, Yangrui Chen, Shaoyu Wang, et al. 2024. Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation. arXiv preprint arXiv:2408.03505 (2024). https://arxiv.org/abs/2408.035 05. [10] Wei Fu, Jiaxuan Gao, Xujie Shen, et al. 2025. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. arXiv preprint arXiv:2505.24298 (2025). https://arxiv.org/abs/2505 .24298. [11] GLM-5 Team. 2026. GLM-5: from Vibe Coding to Agentic Engineering. arXiv preprint arXiv:2602.15763 (2026). https://arxiv.org/abs/2602.157 63. [12] Daya Guo et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025). https://arxiv.org/abs/2501.12948. [13] Zhenyu Han, Ansheng You, Haibo Wang, et al. 2025. AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM PostTraining. arXiv preprint arXiv:2507.01663 (2025). https://arxiv.org/abs/ 2507.01663. [14] Jian Hu, Xibin Wu, Wei Shen, et al. 2024. OpenRLHF: An Easy-touse, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143 (2024). https://arxiv.org/abs/2405.11143. [15] Shengyi Huang, Jiayi Weng, Rujikorn Charakorn, Min Lin, Zhongwen Xu, and Santiago Ontañón. 2023. Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform. arXiv preprint arXiv:2310.00036 (2023). https://arxiv.org/abs/2310.00036. [16] Wenxuan Huang, Bohan Jia, Zijie Zhai, et al. 2025. Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models. arXiv preprint arXiv:2503.06749 (2025). https://arxiv.org/abs/2503.067 49. [17] Chenyu Jiang, Zhen Jia, Shuai Zheng, Yida Wang, and Chuan Wu. 2024. DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines. In European Conference on Computer Systems (EuroSys). https: 13

Bangjun Xiao et al. [35] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. In European Conference on Computer Systems (EuroSys). https://arxiv.org/abs/2409.19256. [36] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al. 2019. MegatronLM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053 (2019). https://arxiv.org/ abs/1909.08053. [37] Ting Sun, Junjie Zhang, Xiao Yan, et al. 2026. BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training. arXiv preprint arXiv:2605.09994 (2026). https://arxiv.org/abs/ 2605.09994. [38] The Microsoft AI Team. 2026. MAI-Thinking-1: Building a Hill-Climbing Machine. Technical Report. Microsoft AI. https://microsoft.ai/pdf/m ai-thinking-1.pdf. [39] THUDM. 2025. slime: An LLM Post-Training Framework for RL Scaling. (2025). https://github.com/THUDM/slime. [40] TransferQueue Team. 2026. TransferQueue: An Asynchronous Streaming Data Management Module for Efficient Post-Training. (2026). https://github.com/Ascend/TransferQueue. [41] VeRL Contributors. [n. d.]. VeRL: RayPPOTrainer Token Balancing. Official source documentation, version 0.5.x. ([n. d.]). https://verl.rea dthedocs.io/en/v0.5.x/_modules/verl/trainer/ppo/ray_trainer.html. [42] Hanjing Wang, Man-Kit Sit, Congjie He, et al. 2023. GEAR: A GPUCentric Experience Replay System for Large Reinforcement Learning Models. arXiv preprint arXiv:2310.05205 (2023). https://arxiv.org/abs/ 2310.05205. [43] Weixun Wang, Shaopan Xiong, Gengru Chen, et al. 2025. Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library. arXiv preprint arXiv:2506.06122 (2025). https://arxiv.org/abs/2506.06122. [44] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025. SWE-smith: Scaling Data for Software Engineering Agents. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. https://arxiv.org/abs/2504 .21798. [45] Sidi Yang, Chaofan Tao, Jierun Chen, et al. 2026. What Makes Interaction Trajectories Effective for Training Terminal Agents? arXiv preprint arXiv:2606.03461 (2026). https://arxiv.org/abs/2606.03461. [46] Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, et al. 2023. DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. arXiv preprint arXiv:2308.01320 (2023). https://arxiv.org/abs/2308.01320. [47] Qiying Yu, Zheng Zhang, Ruofei Zhu, et al. 2025. DAPO: An OpenSource LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476 (2025). https://arxiv.org/abs/2503.14476. [48] Daoguang Zan, Zhirong Huang, Wei Liu, et al. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. https://arxiv.org/abs/2504.02605. [49] Zili Zhang, Yinmin Zhong, Yimin Jiang, et al. 2024. DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models. arXiv preprint arXiv:2408.04275 (2024). https://arxiv.org/abs/2408.04275. [50] Jiale Zhao, Guoxin Chen, Fanzhe Meng, et al. 2026. Immersion in the GitHub Universe: Scaling Coding Agents to Mastery. arXiv preprint arXiv:2602.09892 (2026). https://arxiv.org/abs/2602.09892. [51] Juntao Zhao, Qi Lu, Wei Jia, et al. 2026. MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training. In European Conference on Computer Systems (EuroSys). https: //doi.org/10.1145/3767295.3803568.

[52] Yijie Zheng, Bangjun Xiao, Lei Shi, et al. 2025. OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training. arXiv preprint arXiv:2503.23830 (2025). https://arxiv.org/abs/2503.23830. [53] Yinmin Zhong, Zili Zhang, Bingyang Wu, et al. 2025. StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation. arXiv preprint arXiv:2504.15930 (2025). https://arxiv.org/abs/2504.15930.

KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning

Analyze & explore related research in Conceptio

Related documents