arXiv:2606.24369v1 [cs.AI] 23 Jun 2026
Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation Sijie Wang
Zhengyu Qing
Zhiqiang Tan
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen [email protected]
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen [email protected]
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen [email protected]
Yiming Yin
Yeqing Zhang
Yaoyuan Wang
Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou) [email protected]
Huawei Technologies Ltd. [email protected]
Huawei Technologies Ltd. [email protected]
Qiang Wang
Xiaowen Chu
Shaohuai Shi∗
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen [email protected]
Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou) [email protected]
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen [email protected]
Abstract
QwenImage-20B generative models. Experimental results show that DigenRL achieves 1.56-2.10× throughput improvements over state-of-the-art diffusion RL systems, veRL-Omni and GenRL.
Reinforcement learning (RL) has become a dominant posttraining paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs). In parallel, diffusion-oriented RL algorithms (e.g., DanceGRPO and FlowGRPO), have rapidly expanded the scope of RL from language reasoning to diffusion-based visual and flow-based generation. However, efficient RL systems for diffusion generative LLMs remain underexplored. Existing implementations (e.g., veRL-Omni) still rely on colocated execution, which simplifies synchronization but couples rollout and training resources, limits heterogeneous deployment and constrains independent scaling. To this end, we introduce DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that supports flexible resource allocation, accommodates heterogeneous GPUs, and facilitates efficient task scheduling. To maximally reduce the execution bubbles in the disaggregated architecture, we propose 1) a generation-axis pipeline (GAP) and time-step parallelism (TSP) in the diffusion architecture to enable finer-grained pipelining between rollout and training, 2) an elastic trainer-assisted generation (TAG) approach to enable the trainer GPU resources to dynamically assist in executing rollout generations, and 3) a tightly one-step constrained asynchronous strategy to further utilize the tail bubble in the pipeline. Extensive experiments are conducted on three hardware testbeds with 16-32 GPUs using HunyuanVideo-13B, Wan2.1-14B, FLUX.1-12B,
CCS Concepts: • Computing methodologies → Reinforcement learning; • Computer systems organization → Distributed architectures; • Software and its engineering → Scheduling. Keywords: diffusion reinforcement learning, disaggregated post-training, pipeline scheduling, resource scheduling
1
Introduction
Reinforcement learning (RL) has re-emerged as one of the central paradigms for post-training large language models (LLMs) [10, 20]. In preference alignment, complex reasoning, and agentic training, RL typically boosts final model quality using proximal policy optimization (PPO) [23], group relative policy optimization (GRPO) [6], or its variants (e.g., DanceGRPO [38], FlowGRPO [18]). The basic workflow of RL for LLMs proceeds iteratively through: 1) sampling with the current policy model (generator or rollout), 2) evaluating the sampled outputs (reward model or critic), and 3) updating the policy based on these samples (actor update). The latter two steps are often grouped into a single trainer component, which computes rewards and losses to adjust the policy, after which the updated policy is synchronized with the generator. As a result, the generator and trainer share strong data dependencies, demand significant computation, and in practice often become the primary performance bottleneck. This
∗ Corresponding author.
1
Table 1. Comparison of RL frameworks for diffusion models. Feature
trainer. GAP and TSP together support finer-grained generation and training task pipelining and reduce the bubble time. Second, to further reduce the bubbles when the trainer waits for the generator, we propose elastic trainerassisted generation (TAG) to allow the trainer to execute part of the micro-batch generation workloads when it would otherwise be idle, as shown in Figure 1(d). Third, to better exploit the final micro-batch bubble, which in theory cannot be removed in synchronous Diffusion-RL, we propose a trajectory-consistent stale synchronization (TCSS) scheme. TCSS allows the generator to progress by at most one policy version while ensuring that each denoising trajectory is produced entirely under a single, fixed policy snapshot. The feature comparison of our DigenRL with existing RL systems is shown in Table 1. We conduct extensive experiments compared with state-of-the-art Diffusion-RL systems (veRLOmni [29] and GenRL [19]) using various visual generative models (HunyuanVideo-13B [15], Wan2.1-14B [30], FLUX.112B [4], QwenImage-20B [34]) on two homogeneous 32-GPU testbeds and one heterogeneous 16-GPU testbed. Experimental results (§5) show that our DigenRL achieves 1.56×–2.1× speedups over veRL-Omni and GenRL.
veRL GenRL veRL-Omni DigenRL
Diffusion-RL × ✓ Multi-model Support × Partial Algorithm Extensibility ✓ ✓ Disaggregated Training Partial × Asynchronous RL ✓ ×
✓ Partial Limited × ×
✓ ✓ ✓ ✓ ✓
has motivated the development of high-performance RL systems such as DeepSpeed-Chat [39], veRL [26], AReaL [8], StreamRL [41], OpenRLHF [12], LlamaRL [33], ROLL [31], etc. These systems illustrate that the training throughput of RL post-training is influenced not only by the choice of learning algorithms (e.g., PPO [23], GRPO [24], etc.), but also strongly by the way resources are organized and tasks are scheduled within the system. Recent RL systems (e.g., AReaL [8], StreamRL [41], and Laminar [25], OrchestrRL [27], WEAVE [32]) tend to exploit a disaggregated architecture due to its flexibility for resource allocation, asynchronous rollouts, and long-tail generation repacking, which are particularly effective for autoregressive LLM post-training (AR-RL). However, for the RL post-training of diffusion-based visual generative models (Diffusion-RL) [18, 38], current optimized systems (e.g., Long-RL [5], GenRL [19] and veRL-Omni [29]) still adopt a colocated architecture. This may be because 1) rollout workers have relatively uniform workloads, and 2) rollout and training share the same backend (e.g., PyTorch-FSDP [40]), avoiding efficiency issues. Yet, we argue that Diffusion-RL still needs a disaggregated architecture to enable more finegrained parallelism and task scheduling, thereby making better use of hardware resources and eventually increasing overall system throughput. However, disaggregated Diffusion-RL naturally incurs substantial pipeline bubbles, as shown in Figure 1(b)(c), because the trainer and generator must alternatively wait for one another, leaving compute devices underutilized and prolonging the overall training process. As illustrated in Figure 1(b), the disaggregated architecture introduces additional bubbles relative to the collocated architecture in Figure 1(a). Therefore, to enable disaggregated Diffusion-RL to fully exploit GPU resources during training, it is necessary to design finegrained pipeline scheduling strategies that minimize these bubbles as much as possible. In this paper, we introduce DigenRL, a disaggregated Diffusion-RL system for improving post-training throughput. To achieve this goal, we propose the following novel techniques. First, we design a diffusion-based parallelism (§3.2) according to the diffusion inference and training characteristics with 1) generation-axis pipeline (GAP) execution in the generator and 2) time-step parallelism (TSP) in the
2
Preliminaries and Motivations
2.1 Diffusion-RL Workflow Existing Diffusion-RL algorithms such as DanceGRPO [38] and Flow-GRPO [18] are typically executed in a colocated fashion: the same worker group, or tightly coupled device pool, is responsible for both sample generation and trainerside policy updates. Under this workflow, the generator uses the current policy 𝜋𝑘 to produce a denoising trajectory [7, 11, 17] old 𝑇 𝜏𝑖 = {(𝑥𝑖,𝑡 , 𝑥𝑖,𝑡 −1, ℓ𝑖,𝑡 )}𝑡 =1,
𝑥𝑖,𝑡 −1 ∼ 𝜋𝑘 (· | 𝑥𝑖,𝑡 , 𝑐𝑖 ),
where 𝑥𝑖,𝑡 is the latent at the denoising step 𝑡, 𝑐𝑖 is the condition or prompt, and old ℓ𝑖,𝑡 = log 𝜋𝑘 (𝑥𝑖,𝑡 −1 | 𝑥𝑖,𝑡 , 𝑐𝑖 )
is the step-level log-probability recorded during rollout. After the final sample 𝑥𝑖,0 is decoded by decoder [13, 28], a reward model [14, 37] evaluates its final reward 𝑟𝑖 = 𝑅(𝑥𝑖,0, 𝑐𝑖 ). Then, the trainer consumes these trajectories as one training micro-batch. In recent Diffusion-RL algorithms like DanceGPRO and Flow-GRPO [18, 38], final rewards are first converted into advantages before policy optimization. For example, for a group of samples generated from the same prompt, one may use 𝑟𝑖 − 𝜇𝑔 𝐴ˆ𝑖 = , (1) 𝜎𝑔 + 𝜖 where 𝜇𝑔 and 𝜎𝑔 denote the mean and standard deviation of rewards within the group. 2
Figure 2. Comparison between per-timestep loss computation and our proposed time-step parallelism (TSP). Figure 1. Comparison of four execution modes. We omit the reward stage in the figures as it occupies only a small fraction of the total time. (a) Colocated execution: all compute resources are assigned to the one stage at a time, with workers sharing model states and data. (b) Naive disaggregated execution: the resources are partitioned to two groups (Generator and Trainer), which suffers from strong inter-stage dependencies. (c) Disaggregated execution with micro-batch pipelining: a batch of data is split to multiple micro-batches to enable the generator and the trainer to be executed in a pipeline manner. (d) Our proposed Trainer-Assisted Generation (TAG): pipeline bubbles can be reduced by allowing idle trainer resources to participate in generation.
be written as 1 ∑︁ ∑︁ L𝑖,𝑡 , 𝑖 |S𝑖 | 𝑖 𝑡 ∈ S 𝑖 L𝑖,𝑡 = min 𝜌𝑖,𝑡 (𝜃 )𝐴ˆ𝑖 , clip 𝜌𝑖,𝑡 (𝜃 ), 1 − 𝜖, 1 + 𝜖 𝐴ˆ𝑖 .
Lpg = − Í
Under a fixed policy snapshot and a fixed collected trajectory batch, this objective is additive over the selected timesteps. If we define the per-step loss term as ℓ𝑖,𝑡 (𝜃 ), then 1 ∑︁ ∑︁ 1 ∑︁ ∑︁ Lpg = ℓ𝑖,𝑡 (𝜃 ), ∇𝜃 Lpg = ∇𝜃 ℓ𝑖,𝑡 (𝜃 ), 𝑍 𝑖 𝑍 𝑖 𝑡 ∈ S𝑖
𝑡 ∈ S𝑖
Í
where 𝑍 = 𝑖 |S𝑖 |. Therefore, timestep-parallel training does not assume that denoising decisions are independent. The workflow is shown in Figure 2(a); each time step can only begin after the previous time step has completed.
The trainer then recomputes the step-level log-probability under the current policy and forms an importance ratio old 𝜌𝑖,𝑡 (𝜃 ) = exp log 𝜋𝜃 (𝑥𝑖,𝑡 −1 | 𝑥𝑖,𝑡 , 𝑐𝑖 ) − ℓ𝑖,𝑡 . (2)
2.2
Policy optimization is typically performed only on a selected subset of denoising steps to reduce training cost [18, 38]. Let S𝑖 ⊆ {1, . . . ,𝑇 } denote the selected training timesteps for sample 𝑖. A PPO/GRPO-style clipped objective can then
Disaggregated Diffusion-RL
Compared with the colocated architecture, in which the generator and trainer jointly use the entire pool of GPU resources (Figure 1(a)), the disaggregated architecture splits the GPUs into two non-overlapping sets: a generator group 3
Figure 3. Scalability comparison across different stages.
Figure 4. The system overview of DigenRL.
using the trainer’s bubble time to support the generation process. Second, because synchronous Diffusion-RL inherently incurs a bubble at the final micro-batch, this last idle period is theoretically unavoidable. This observation motivates us to decouple generation and training into an asynchronous execution scheme, so that the remaining bubble time can be further utilized while still maintaining convergence guarantees.
(GenGroup) and a trainer group (TrainGroup). The generator and trainer then take turns executing their workloads (Figure 1(b)). During each RL training iteration, the generator produces a micro-batch of training data, which the trainer consumes to update the model weights; these updated weights are subsequently synchronized back to the generator. The disaggregated architecture in Diffusion-RL is useful in scenarios when the workloads of the generator and trainer vary and the GPU resources are heterogeneous [25, 36, 41]. In Diffusion-RL, the generator still advances sequentially along the denoising chain, while the optimization of the trainer can parallelize computations across multiple selected timesteps, which easily causes workload imbalance between the generator and trainer. As shown in Figure 3, when training FLUX.1-12B on a 32-GPU cluster, the trainer maintains good scalability as the number of GPUs increases because it is compute-bound. In contrast, the generator needs to process timestep generation sequentially, so it often fails to fully utilize all available compute resources as the resource scale grows. 2.3
3
DigenRL: System Design
3.1
System overview
The overall system design of DigenRL is shown in Figure 4. For any Diffusion-RL post-training workload executed on a GPU cluster, DigenRL allocates distinct GPUs for the generator and the trainer. Given user-specified mini-batch and micro-batch sizes, DigenRL leverages the generation-axis pipeline (GAP) module to pipeline generator and trainer operations along the generation dimension, thereby achieving finer-grained task pipelining. In the training phase, DigenRL adopts timestep parallelism (TSP) rather than conventional data parallelism to boost GPU utilization. During runtime, the trainer-assisted generation (TAG) module adaptively decides when the trainer should assist the generator with rollouts, aiming to shorten the generation time under a fixed mini-batch budget. Finally, when the one-step off-policy option is enabled, our TCSS leverages the last microbatch bubble to produce samples with at most a one-step stale policy, further enhancing system throughput.
Motivations
As shown in Figure 1(c), naively pipelining generation tasks and training tasks in Diffusion-RL with multiple micro-batches still causes many bubbles. First, the nature of data dependencies between the generator and trainer easily causes pipeline bubbles when their execution times are not comparable. Although one can assign different numbers of GPUs to the generator and trainer based on their respective workloads to balance their execution times, this is difficult in practice because RL algorithms such as DanceGRPO and Flow-GRPO involve hyperparameters (e.g., the time-step fraction for training) that influence the wall-clock time of both generation and training. On the other hand, when Diffusion-RL is executed on a heterogeneous GPU cluster, its runtime performance can fluctuate because of variations in hardware capabilities, including compute power, memory capacity/bandwidth, and interconnect bandwidth/latency. Thus, we propose (1) leveraging the computational properties of diffusion models during both inference and training to shorten bubble time, and (2) adaptively
3.2
Diffusion-based Parallelism
3.2.1 Generation-Axis Pipeline (GAP). Existing AR RL systems form micro-batches mainly along the prompt/sequence dimension and balance them by token length. This is effective for AR workloads, where RL training batches often contain hundreds or thousands of samples. In Diffusion RL, however, a train batch often contains only dozens, or even tens, of samples. Naively splitting along the batch dimension therefore produces too few pipeline units under small-batch Diffusion RL settings, as shown in Figure 5(a). DigenRL therefore introduces GAP, a micro-batch formation strategy that partitions work along the generation axis, creating finer pipeline units 4
the order of accumulation does not alter the final result. For example, in Figure 5, panel (a) accumulates the gradient of Prompt 1, Sample 0 at the second step, whereas panel (b) accumulates the gradient of Prompt 0, Sample 1 at the second step. Although the accumulation order changes, the final gradient remains unchanged. 3.2.2 Timestep Parallelism (TSP). Building on GAP, we further introduce Timestep Pipeline (TSP). As discussed in Section 2.1, the latent, next latent, and other per-timestep context are already known during training. Thus, once the rollout trajectory is fixed, the trainer can form a timestep batch, evaluate multiple selected timesteps in parallel, and aggregate them into the same update as shown in Figure 2(b). This is theoretically equivalent to sequentially accumulating the same per-step losses under the same policy snapshot. This consideration becomes even more critical in sharded training setups like FSDP [40] and ZeRO [22], where parameters are split across devices and must be all-gathered before each layer is computed. If the chosen timesteps are handled strictly one after another, each all-gather effectively benefits only a single timestep. In contrast, grouping several timesteps into a batch allows a single gathered parameter view to be reused for more computation, which amortizes the communication cost and enhances both parallelism and overall hardware utilization. One might wonder how TSP differs from simply enlarging the training micro-batch size, since both approaches increase the computation per forward/backward pass to better saturate the hardware. There are two key distinctions. First, for a fixed micro-batch size, TSP offers a higher potential speedup because it exposes parallelism not only across the batch dimension but also across the timestep dimension. Second, pipeline scheduling inherently balances the number of micro-batches against their size. Using smaller micro-batches increases the number of pipeline units and allows more finegrained overlap, but typically leads to poorer hardware utilization and lower compute efficiency. In practice, one must select a micro-batch size that trades off overlap efficiency against compute efficiency. TSP mitigates this trade-off: even when the generation-axis micro-batch size is small, timestep batching maintains high compute efficiency, while the increased number of micro-batches still supports fine-grained pipeline overlap. In combination, GAP and TSP largely eliminate the conflict between overlap efficiency and compute efficiency.
Figure 5. Comparison of different micro-batch partitioning strategies. (a) Partitioning along the batch dimension. (b) DigenRL partitions work along the generation dimension, producing more fine-grained micro-batches and enabling better overlap. while preserving the semantics of group reward normalization and policy update. As illustrated in Figure 5, panel (a) shows the mainstream partitioning method, which splits work along the batch dimension. Under this scheme, different GPUs sample different prompts, and reward and advantage can only be computed after all samples for one prompt have completed generation. In the example shown in the figure, there are two GPUs, the training batch size is four, and each prompt is sampled twice. Under this setup, the system can be partitioned into at most two micro-batches. GAP instead partitions work along the sampling dimension, as illustrated in Figure 5(b). Different GPUs sample the same prompt, and once all GPUs complete sampling, reward and advantage can be computed and used for training. Under the same configuration, this strategy yields four microbatches. More micro-batches enable finer-grained pipeline scheduling and more overlap. Our method also does not introduce numerical error. Before one training batch finishes all generation and training work, model weights are not updated; only gradient accumulation is performed. Since gradients are vectors, changing
3.3
Elastic Trainer-Assisted Generation (TAG)
In practical training, generation is usually slower than training. Each sample typically requires multiple rollouts, whereas the trainer often updates on only a subset of the generated samples and only a subset of their timesteps. Moreover, our timestep-parallel training further shortens trainer-side execution time. This creates a new inefficiency: as illustrated 5
Figure 6. Impact of different numbers of TAG micro-batches on end-to-end time. Figure 7. TAG execution workflow. in Figure 6(a), when one micro-batch finishes training, the next micro-batch may still be under generation, leaving the trainer idle because no new micro-batch is yet available for update. In disaggregated AR RL systems, a common solution is to repartition resources: more resources are assigned to the slower generation stage so that generator and trainer runtimes become closer, thereby increasing overlap. However, this strategy is less suitable for Diffusion RL. As discussed earlier, Diffusion RL typically uses much smaller training batches than AR RL, so resource repartitioning can easily create workload imbalance across GPUs. In practice, resource allocation in Diffusion RL is therefore often constrained by training batch size and cannot be adjusted as freely. AR RL also suffers from imperfect load balance, but because its batch size is much larger, the imbalance is usually acceptable and does not translate into severe resource waste. For this reason, we favor a workload-balanced resource allocation strategy for Diffusion RL, while allowing the trainer to take over part of the generation workload when it would otherwise be idle. We call this mechanism Elastic TrainerAssisted Generation (TAG), as illustrated in Figure 6. TAG execution workflow. A practical RL system must decide how to construct resource pools, and which components should be disaggregated or colocated. Taking VeRLOmni as an example, the system mainly consists of a generator and a reward model backed by vLLM-Omni, and a trainer backed by FSDP. After separating the generator from the trainer, we colocate the reward model with the generator in the same resource pool, because the trainer has high memory demand during training and is better suited to occupying an exclusive resource pool.
However, colocating scoring with generation under pipelined execution introduces another issue: dynamically arriving tasks may compete for the same compute resource. For example, after the trainer finishes a TAG task, the system may request scoring for that generated result. If the generator is still executing another generation task, the scoring task and generation task would contend for the same resource pool, sharply reducing efficiency. We therefore designed a buffering and synchronization mechanism to coordinate task execution. As shown in Figure 7, the driver process maintains two queues. The Scoring Queue (SQ) buffers generated images, videos, and other intermediate states waiting for reward evaluation, while the Update Queue (UQ) buffers reward-annotated samples waiting for trainer-side update. At the beginning of training, the driver first performs centralized data preprocessing and assigns micro-batches to the generator and trainer. Once a micro-batch finishes generation, it is pushed into SQ. After each task, the generator checks whether SQ is empty: if it is empty, the generator continues generation; otherwise, it takes images or videos from SQ for scoring. The scored samples are then pushed into UQ and consumed by the trainer. This gives scoring higher priority than generation, allowing the trainer to receive rewards earlier and proceed with update more quickly. As for Trainer, it may perform a context switch depending on the backend configuration before each TAG execution. For example, if generation and training use different backends, the trainer loads the generation model before TAG and offloads the training model to save memory, then reverses this operation after TAG completes. After each task, the trainer checks UQ: if UQ is non-empty, it runs a trainer-side update; otherwise, the driver may assign a TAG task according to the 6
TAG scheduling policy. Since each update micro-batch only accumulates gradients without changing model weights and we call the optimizer step only after all micro-batches have been trained, the overall training process preserves on-policy semantics. This workflow avoids severe efficiency degradation caused by resource contention, while also preventing the trainer from waiting idly for new training samples. A remaining question is when TAG should be applied. We will introduce our search strategy in Section 4. 3.4
Trajectory-Consistent Stale Synchronization (TCSS)
As shown in Figure 1(d), the generator idle time during the final trainer-side micro-batch update is unavoidable under strict synchronization. The generator cannot start producing the next train batch until the latest trainer weights have been synchronized. This barrier remains even after micro-batch pipelining and TAG, because it is caused by the global-step consistency requirement rather than by coarse scheduling granularity. To address the remaining resource waste caused by strict synchronization, we build on TAG and introduce TrajectoryConsistent Stale Sync (TCSS), a trajectory-consistent one-step stale synchronization strategy for disaggregated Diffusion RL. Instead of enforcing a strict global-step barrier after the last trainer micro-batch update, the generator is allowed to start a bounded number of next-step micro-batches using the previous policy snapshot. Once the updated policy is synchronized, subsequent micro-batches use the new policy. Importantly, TCSS differs from asynchronous AR schemes such as AReaL’s partial rollout [8]. In AR generation, a response can be naturally split along token prefixes, so different parts of a response may be produced under different policy versions. Directly applying such version mixing to diffusion is riskier because each sample is a tightly coupled denoising trajectory; mixing policy versions inside the same trajectory can amplify step-level log-probability mismatch and blur credit assignment across old and new denoising decisions. TCSS therefore mixes policy versions only across micro-batches, never within a single denoising trajectory. This design removes the unavoidable synchronization bubble while preserving trajectory-level policy consistency. Figure 8(a) illustrates that, under synchronous TAG, an unavoidable bubble appears at the global-step boundary because the generator must wait until the trainer finishes synchronizing the latest weights before continuing generation. Figure 8(b) shows our TCSS design. We define staleness as the maximum fraction of stale samples allowed within one global step, and allow the generator to produce a bounded number of micro-batches using the stale policy under this staleness constraint. In this way, the otherwise idle bubble can be used to start generation for the next global step, reducing the next step’s end-to-end latency.
Figure 8. Comparison between TAG and TCSS.
Asynchronism can affect training rewards because some micro-batches are generated by an old policy, so the staleness ratio should not be set too large. At the same time, GRPO already constrains excessive deviation between old and new policies in the loss through clip 𝜌𝑖,𝑡 (𝜃 ), 1 − 𝜖, 1 + 𝜖 𝐴ˆ𝑖 . Therefore, an appropriate staleness ratio can still preserve stable reward improvement. 3.4.1 TCSS Algorithm. Here, 𝐶 gen denotes the maximum number of concurrent generation futures. 𝑠 is the current global training step, and 𝑆 is the total number of training steps. 𝑠 sync records the latest step whose trainer weights have been synchronized to the generator workers. 𝑠 gen and 𝑠 upd denote the latest dispatched generation step and the latest trainer-update step, respectively, and 𝑊 bounds the maximum allowed policy staleness between generation and update. The core logic appears in lines 5–9. Instead of using a global step as the outer scheduling unit, TCSS runs a continuous while-loop that allows micro-batches from different global steps to enter execution as long as the staleness bound 𝑊 is satisfied. When the condition holds, the scheduler selects an idle trainer or generator worker and launches a generation task, thereby maximizing resource utilization. Lines 23–31 handle completion and synchronization. When a trainer-side micro-batch finishes, TCSS marks it as completed. Once all micro-batches of a global step are completed, the scheduler temporarily stops issuing trainer work for that step and performs weight synchronization when the generator is ready.
4
Recursive Bubble-Triggered TAG Search
A straightforward option is to insert several TAG microbatches before the trainer-side update starts. This can exploit the trainer-side bubble at the beginning of each training step 7
Algorithm 1 Trajectory-Consistent Stale Sync 1: Initialize trainer workers T , generator workers G, and
async queues 2: 𝑠 ← 0, 𝑠 sync ← −1 3: R, U, Qupd ← ∅ 4: while 𝑠 < 𝑆 do 5: if |R| < 𝐶 gen ∧ 𝑠 gen − 𝑠 upd < 𝑊 then 6: 𝑚 ← NextMicroBatch(D, 𝑠) 7: 𝑤 ← SelectWorker(𝑚, G, T ) 8: 𝑟 ← GenAsync(𝑚, 𝑤) 9: R ← R ∪ {𝑟 }
end if if 𝑟 ← FutureReturned(R) is not None then 𝜏 ← Get(𝑟 ) 𝐴 ← ComputeAdvantage(𝜏) B ← SelectBestOfN(𝜏, 𝐴) Qupd ← Qupd ∪ B 16: R ← R \ {𝑟 } 17: end if 18: while Qupd ≠ ∅ ∧ Free(T ) do 19: 𝑏 ← Pop(Qupd ) 20: 𝑢 ← UpdateAsync(𝑏, T ) 21: U ← U ∪ {𝑢} 22: end while 23: if 𝑢 ← FutureReturned(U) is not None then 24: MarkCompleted(𝑢) 25: U ← U \ {𝑢} 26: end if 27: while StepComplete(𝑠 sync + 1) do 28: 𝑠 sync ← 𝑠 sync + 1 29: SyncWeights(T , G) 30: 𝑠 ← 𝑠 sync + 1 31: end while 32: end while 10: 11: 12: 13: 14: 15:
Figure 9. Recursive bubble-triggered TAG search. At each emerging bubble, the scheduler either inserts one TAG microbatch or skips the bubble.
Algorithm 2 Recursive Bubble-Triggered TAG Search 1: Input: initial schedule state 𝑆 0 2: Output: minimum latency 𝑇 ∗ and TAG strategy Ω ∗ 3: (𝑇 ∗, Ω ∗ ) ← Search(𝑆 0, ∅) 4: return (𝑇 ∗, Ω ∗ ) 5: procedure Search(𝑆, Ω) 6: 𝑆 ← Advance(𝑆) 7: if Finished(𝑆) then 8: return (FinishTime(𝑆), Ω) 9: end if 10: 𝑏 ← EmergingBubble(𝑆) 11: (𝑇 ∗, Ω ∗ ) ← (+∞, ∅) 12: 𝑆 skip ← SkipBubble(𝑆, 𝑏) 13: (𝑇skip, Ωskip ) ← Search(𝑆 skip, Ω ◦ Skip(𝑏)) 14: if 𝑇skip < 𝑇 ∗ then 15: (𝑇 ∗, Ω∗ ) ← (𝑇skip, Ωskip ) 16: end if 17: 𝑚 ← NextTAG(𝑆, 𝑏)
and, by delaying the update, allows the update queue to accumulate enough ready samples, avoiding the case where the trainer waits for the generator at the very beginning. We call this approach Static TAG. However, inserting too many TAG micro-batches can make the trainer become the long tail: the trainer performs excessive TAG work, starts update too late, and eventually makes the generator wait for the trainer. Inserting too few TAG micro-batches, on the other hand, underutilizes trainer-side bubbles. Therefore, we prefer to adjust TAG decisions dynamically during execution. We propose a recursive global search strategy, called Bubble-Triggered TAG Search (BTS), to decide whether each emerging bubble should be filled by TAG. Because inserting a TAG micro-batch can fill the current trainer-side bubble, but it may also delay the training of subsequent micro-batches as shown in Figure 6; in other words, blindly inserting TAG at every bubble can be locally beneficial but globally suboptimal.
18: if 𝑚 ≠ ∅ then 19: 𝑆 tag ← InsertTAG(𝑆, 𝑏, 𝑚) 20: (𝑇tag, Ωtag ) ← Search(𝑆 tag, Ω ◦ Insert(𝑏, 𝑚)) 21: if 𝑇tag < 𝑇 ∗ then 22: (𝑇 ∗, Ω∗ ) ← (𝑇tag, Ωtag ) 23: end if 24: end if 25: return (𝑇 ∗, Ω ∗ ) 26: end procedure
Figure 9 illustrates the decision tree. Instead of fixing the number of TAG micro-batches in advance, the scheduler advances execution until a new emerging bubble appears, and 8
then decides whether to fill this bubble with one TAG microbatch or skip it. A skipped bubble is marked inactive and will not be reconsidered. The recursion terminates in two cases. First, even if a bubble appears, there may be no remaining micro-batch that can be assigned to TAG, as illustrated by node 2 in Figure 9. Second, with branch-and-bound pruning, a branch can be stopped early once its latency lower bound already exceeds the current best latency, as illustrated by node 5. In practical training, we apply TAG to the first bubble by default, i.e., the position of micro-batch 1 in Figure 9. This provides online profiling measurements for the generation time of the next micro-batch on the generator, as well as the generation and training times on the trainer. Training then proceeds normally. Whenever a new bubble appears, the driver process invokes Algorithm 2 to solve for the globally optimal strategy and decide whether the current bubble should be filled by TAG. Although Diffusion RL does not exhibit the same long-tail effect as AR workloads, and microbatch runtimes are usually same, small runtime variations can still be introduced by hardware instability, reward computation, and cold-start distortion during online profiling. These errors may accumulate as execution progresses, so a one-time global search is insufficient. Instead, whenever a bubble is encountered, we update the runtime estimates using historical online profiling measurements, use the averaged values as the new search input, and recompute the strategy to correct drift caused by measurement error. If the execution exposes 𝐾 emerging bubbles, Algorithm 2 has worst-case complexity 𝑂 (2𝐾 ), or 𝑂 (𝐵2𝐾 ) when each state simulation costs 𝑂 (𝐵) for 𝐵 micro-batches. However, the homogeneous micro-batch model allows us to substantially reduce the effective search space. Since all micro-batches have the same generation, TAG, and update costs, their identities are irrelevant to future scheduling decisions. We therefore compress a schedule state into aggregate counters, including the number of pending micro-batches, ready-toupdate micro-batches, completed micro-batches, and the current generator/trainer timestamps. Different decision paths that reach the same compressed state share the same optimal continuation. We memorize the optimal result for each compressed state and reuse it when the state is encountered again. For example, nodes 2 and 3 in Figure 9 follow different decision paths, but they reach equivalent compressed states and therefore have the same remaining optimal latency. Figure 10 reports the runtime of a single search. The y-axis varies the number of micro-batches, where a larger value implies a larger search space. The x-axis varies the ratio between the update time and the generation time of one micro-batch, where a smaller ratio indicates more frequent pipeline bubbles. Even in the most challenging setting (𝐵 = 128, 𝑡/𝑔𝐺 = 0.1), the search overhead remains negligible compared with generation and training.
Figure 10. Recursive TAG search time.
5
Evaluation
5.1
Experimental Settings
Testbed. Experiments are mainly carried out on: 1) Testbed A, a 32-GPU cluster comprising four interconnected 8-GPU nodes over a 200 Gb/s network, with each 48 GB GPU providing 336 third-generation Tensor Cores and up to 38.7 FP32 TFLOPS / 309.7 Tensor TFLOPS peak performance. 2) Testbed B, a 32-GPU cluster comprising four interconnected 8-GPU nodes over a 400 Gb/s network, with each 80 GB GPU providing 528 fourth-generation Tensor Cores and up to 67 FP32 TFLOPS / 1,979 BF16/FP16 Tensor TFLOPS peak performance. 3) Testbed C, a 16-GPU cluster comprising two interconnected 8-GPU nodes over a 200 Gb/s network, with each 96 GB GPU providing 752 fifth-generation Tensor Cores and up to 126 FP32 TFLOPS / 503.8 BF16/FP16 Tensor Core TFLOPS peak performance. Baselines. Our end-to-end evaluation consists of four parts. First, to validate the effectiveness of DigenRL, we integrate the Diffusion RL algorithm FlowGRPO and support the QwenImage 20B model. We compare its end-to-end time against VeRL-Omni, a state-of-the-art open-source Diffusion RL framework. For fairness, DigenRL uses the same inference backend, vLLM-Omni, and training backend, FSDP, as VeRL-Omni. Second, to verify that disaggregated execution outperforms colocated execution, we further integrate HunyuanVideo with DanceGRPO and Wan2.1 14B with FlowGRPO into DigenRL, and compare their end-to-end time against their native colocated execution. Third, to evaluate DigenRL under heterogeneous resources, we compare it with the colocated execution of HunyuanVideo and show that it can still maintain high throughput. Finally, we conduct ablation experiments to break down the gains from each optimization. We also verify that, when bounding the number of stale samples and limiting their impact on training, TCSS maintains a reward curve comparable to on-policy training. 9
Figure 11. QwenImage end-to-end time comparison with VeRL-Omni, colocated execution and DigenRL.
Figure 12. HunyuanVideo end-to-end time comparison with colocated execution and DigenRL. 5.2 End-to-End Time Comparison with Open-Source Frameworks
while controlling concurrency. Another reason is that image training does not impose severe memory pressure during the training stage, allowing us to use a high degree of TSP parallelism. As a result, DigenRL achieves higher efficiency in both the generation and training stages, which explains its substantial gains over VeRL-Omni and native colocation.
For open-source framework comparisons, GenRL is mainly characterized by its support for diverse models and algorithms, but provides relatively limited system-level optimization. Meanwhile, our measurements show that GenRL performs similarly to native colocation on Wan2.1, so the colocation comparison in §5.3 can serve as a performance reference between DigenRL and GenRL. Therefore, we mainly compare end-to-end time with VeRL-Omni. Meanwhile, to ensure a fair on-policy comparison, we disable TCSS. Our experiments are conducted on Testbed B with a training batch size of 32 and an image resolution of 512 × 512. We use intra-node FSDP full sharding and inter-node data parallelism (DP). Since FlowGRPO for QwenImage uses Qwen3VL-8B-Instruct [2] as the reward model, we colocate the generator and reward model to balance GPU memory pressure. Generation and scoring are then coordinated through the two-queue scheduling mechanism described in §3.3. Across all cases, DigenRL substantially outperforms both native colocation and VeRL-Omni. Compared with VeRLOmni, DigenRL achieves speedups ranging from 1.56× to 2.21×. A key reason is that vLLM-Omni schedules multiple requests through asynchronous queuing: as the batch size grows, scheduling queues and long-tail requests increasingly limit efficiency, and larger batches can even reduce overall throughput. This effect appears in both VeRL-Omni and native colocation. DigenRL mitigates the issue by splitting large batches into smaller micro-batches and pipelining them
5.3
End-to-End Time Comparison with Native Colocation
Figure 12 and Figure 13 compare DigenRL with native colocated Diffusion RL execution. The HunyuanVideo experiments are conducted on Testbed A, and the Wan2.1 14B experiments are conducted on Testbed B. We split the nodes evenly between the generator and trainer. The model uses FSDP as the backend, and the training batch size is 32. We use common sampling-step counts of 20, 30, and 40, and timestep fractions of 0.4 and 0.6, which denotes the proportion of denoising timesteps used for training. The resolution is 352 × 352 × 25. For HunyuanVideo, DigenRL achieves end-to-end speedups ranging from 1.04× to 1.37×. When TCSS is enabled, the speedups increase to 1.13×–1.49×. The gains are larger on 16 GPUs because scaling to 32 GPUs introduces additional driver-side scheduling pressure and GPU–CPU communication overhead, which partially offset the benefits of our optimizations. A similar trend appears on FLUX.1 12B, where the speedup improves from 1.06× to 1.21×. Because FLUX is an image generation model and has a shorter computation 10
Figure 13. Wan2.1 14B end-to-end time comparison with colocated execution and DigenRL.
Figure 14. FLUX.1 12B 32 GPUs end-to-end time comparison with colocated execution and DigenRL.
Figure 16. TCSS reward curve. 5.4
Figure 15. HunyuanVideo end-to-end time comparison on heterogeneous resources.
time, the scheduling overhead incurred on 32 GPUs becomes more prominent in the end-to-end runtime. For Wan2.1 14B, DigenRL achieves end-to-end speedups ranging from 1.13× to 1.34×. When TCSS is enabled, the speedups increase to 1.21×–1.45×. More importantly, it achieves similar gains on both 16 and 32 GPUs, mainly because Testbed B provides stronger CPU scheduling capability and higher CPU-GPU communication bandwidth than Testbed A, which mitigates the scheduling and communication pressure caused by scaling up. Another supporting observation is that DigenRL gains increase with the number of sampling steps, because computation occupies a larger fraction of the end-toend time and therefore partially amortizes communication and scheduling overhead. Overall, the performance gains come from two sources: TSP reduces trainer-side update time, while disaggregation with pipelining provides better scalability and overlap than native colocation. 11
End-to-End Time Comparison with Heterogeneous Resources
We conduct this set of experiments using eight GPUs from Testbed A and eight GPUs from Testbed C, connected through 200 Gb/s HDR InfiniBand. We choose HunyuanVideo as the evaluation model because RL for video diffusion models typically imposes higher compute and memory requirements, making the impact of heterogeneous resources more pronounced. As shown in Figure 15, DigenRL improves speedup over colocated execution from 1.46× to 1.85×. The overall speedup increases with the number of sampling steps and the timestep fraction, as larger values of these parameters amplify performance differences across heterogeneous hardware. Colocated execution is less effective under heterogeneous resources due to two key drawbacks. First, the more capable side often remains idle while waiting for the weaker side to complete the current stage, resulting in suboptimal resource utilization. Second, the training workload is constrained by the memory capacity of the weaker side, limiting support for higher-resolution videos and images. In contrast, DigenRL is well suited to this heterogeneous training setting. By assigning stronger compute resources to the trainer, it provides a higher memory ceiling for training, while allowing idle trainer resources to assist generation and thereby improving overall resource utilization. 5.5
Training Effectiveness of TCSS
Figure 16 shows the reward curves of on-policy training and TCSS when training FLUX.1-dev 12B. By strictly controlling staleness and limiting the impact of stale policies during
Figure 17. End-to-end speedup of different optimization methods on HunyuanVideo.
Figure 18. End-to-end speedup of different optimization methods on Wan2.1 14B. training, TCSS maintains a steadily increasing reward even when the staleness threshold is 0.5. 5.6
algorithms such as PPO [23], GRPO [1], DPO [21], and ReMax [16], and expose distributed training backends for large models. Disaggregated RL systems (AReaL [8], StreamRL [41], Laminar [25], RollArt [9], Weave [35], OrchestrRL [27], etc.) for post-training text-only LLMs or multimodal LLMs have been proposed to improve the training throughput in different aspects. AReaL [8] eliminates the synchronization bottleneck of traditional RL pipelines through fully asynchronous training. StreamRL [41] further improves disaggregated RL execution by addressing pipeline and workloadskew bubbles. Laminar [25] enhances resource utilization through asynchronous overlap between rollout and training. RollArt [9] focuses on scalable multi-task agentic RL training, while Weave [35] improves cluster efficiency via coscheduling of disaggregated RL workloads. OrchestrRL [27] jointly optimizes compute and network resource orchestration for large-scale disaggregated RL deployments. RLBoost [36] exploits otherwise idle training resources through partial response seeding. These systems mainly target autoregressive decoding, where requests can often be partitioned or resumed along token prefixes. DigenRL instead targets Diffusion-RL, where a sample is generated through a tightly coupled denoising trajectory and training must preserve step-level log-probabilities, rewards, and policy-version consistency. RL for diffusion and flow models. RL has likewise been explored in the context of diffusion and flow-based
Ablation Study
Figure 17 shows the speedup brought by the optimizations introduced on top of naive disaggregation. GAP improves the speedup from 1.52× to 1.84×. The 0.6 cases benefit more than the 0.4 cases because the training stage overlapped by GAP accounts for a larger fraction of the end-to-end time. Furthermore, TSP reduces the training time of each micro-batch, increasing the speedup from 1.62× to 2.14×. At this point, micro-batch training is no longer the bottleneck; generation becomes the dominant cost. TAG then alleviates this generation bottleneck by utilizing idle trainer resources, further improving the speedup from 2.28× to 3.05×, while TCSS increases it from 2.49× to 3.34×. The additional gain from TCSS is relatively small because TAG with already achieves high resource utilization; TCSS mainly extracts the remaining bubbles at synchronization boundaries. For Wan2.1-14B, as shown in Figure 18, GAP improves the speedup from 1.58× to 1.70×. TSP further increases it from 1.71× to 2.10×, TAG improves it from 2.27× to 2.79×, and TCSS increases it from 2.44× to 2.85×.
6
Related Work
RL post-training systems. Recent RL post-training systems for autoregressive foundation models have made largescale policy optimization increasingly practical. Generalpurpose systems such as veRL [26], OpenRLHF [12], ROLL [31], and LlamaRL [33] provide modular execution substrates for 12
generative models. DDPO [3] casts denoising as a sequential decision-making problem, while DanceGRPO [38], FlowGRPO [18], MixGRPO [1], and Long-RL [5] adapt grouprelative or flow-matching RL objectives for image and video synthesis. More recent toolkits, including GenRL [19] and veRL-Omni [29], offer software infrastructures for RL in visual generation. However, these approaches concentrate mainly on objectives, reward shaping, and model coverage, and continue to rely on a colocated architecture. DigenRL is orthogonal to these efforts: instead of proposing a new RL objective, it offers a high-performance execution substrate for existing Diffusion-RL methods based on a disaggregated architecture.
7
[8] Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. 2025. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. arXiv preprint arXiv:2505.24298. https://arxiv.org/abs/2505.24298. [9] Wei Gao, Yuheng Zhao, Tianyuan Wu, Shaopan Xiong, Weixun Wang, Dakai An, Lunxi Cao, Dilxat Muhtar, Zichen Liu, Haizhou Zhao, Ju Huang, Siran Yang, Yongbin Li, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, and Wei Wang. 2026. RollArt: Disaggregated Multi-Task Agentic RL Training at Scale. In Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’26). USENIX Association, Seattle, WA, USA. [10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. arXiv preprint arXiv:2006.11239. https://arxiv. org/abs/2006.11239. [12] Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. 2024. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143. https://arxiv.org/abs/2405.11143. [13] Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114. https://arxiv.org/abs/1312.6114. [14] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. arXiv preprint arXiv:2305.01569. https://arxiv.org/abs/2305.01569. [15] Weijie Kong, Qi Tian, Zijian Zhang, et al. 2024. HunyuanVideo: A Systematic Framework For Large Video Generative Models. arXiv preprint arXiv:2412.03603. https://arxiv.org/abs/2412.03603. [16] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. 2024. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. In International Conference on Machine Learning. PMLR, 29128–29163. [17] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2023. Flow Matching for Generative Modeling. arXiv preprint arXiv:2210.02747. https://arxiv.org/abs/2210.02747. [18] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. 2025. FlowGRPO: Training Flow Matching Models via Online RL. arXiv preprint arXiv:2505.05470. https://arxiv.org/abs/2505.05470. [19] ModelTC. 2025. GenRL: Reinforcement Learning Framework for Visual Generation. https://github.com/ModelTC/GenRL. [20] Long Ouyang, Jeff Wu, Xu Jiang, et al. 2022. Training Language Models to Follow Instructions with Human Feedback. arXiv preprint arXiv:2203.02155. https://arxiv.org/abs/2203.02155. [21] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290. https://arxiv.org/abs/2305.18290. [22] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). 1–16. https://arxiv.org/abs/1910.02054. [23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347. https://arxiv.org/abs/1707.06347. [24] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
Conclusion
This paper introduces DigenRL, a disaggregated RL system for post-training diffusion-based visual generative LLMs. We demonstrate that current colocated Diffusion-RL designs unnecessarily bind together sample generation and trainerside updates, and that a straightforward decoupling of these phases leads to pipeline stalls and synchronization-induced idle time. In contrast, DigenRL removes such bubbles via generation-axis pipelining to support more fine-grained task pipelining, leverages timestep-parallel training to raise training GPU utilization, adopts elastic trainer-assisted generation to opportunistically use trainer idle periods to speed up generation, and applies trajectory-consistent stale synchronization to exploit the final microbatch bubble for additional throughput gains. Across a variety of models, GRPO algorithms, hyperparameters, and hardware setups, our evaluation shows that DigenRL achieves higher end-to-end training throughput than state-of-the-art Diffusion-RL systems such as veRL-Omni and GenRL.
References [1] Anonymous. 2025. MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE. arXiv preprint arXiv:2507.21802. https://arxiv. org/abs/2507.21802. [2] Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. 2025. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631. https://arxiv.org/abs/2511. 21631. [3] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. 2024. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, Vol. 2024. 4965– 4987. [4] Black Forest Labs. 2024. FLUX.1. https://blackforestlabs.ai/ announcing-black-forest-labs/. [5] Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, Sifei Liu, Hongxu Yin, Yao Lu, and Song Han. 2025. Scaling RL to Long Videos. arXiv preprint arXiv:2507.07966. https://openreview.net/ forum?id=TxedB8hI5O. [6] DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12599 [cs.CL] [7] Prafulla Dhariwal and Alexander Quinn Nichol. 2021. Diffusion Models Beat GANs on Image Synthesis. arXiv preprint arXiv:2105.05233. https://arxiv.org/abs/2105.05233. 13
arXiv preprint arXiv:2402.03300 (2024). [25] Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, Yuqi Wu, Xiang Li, Chi Zhang, Yanghua Peng, et al. 2026. Laminar: A scalable asynchronous RL post-training framework. In Proceedings of the 21st European Conference on Computer Systems. 400–422. [26] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint arXiv:2409.19256. https://arxiv.org/abs/2409.19256. [27] Xin Tan, Yicheng Feng, Yu Zhou, Yimin Jiang, Yibo Zhu, and Hong Xu. 2026. OrchestrRL: Dynamic Compute and Network Orchestration for Disaggregated RL. In the 40th ACM Special Interest Group on Data Communication (SIGCOMM). [28] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. arXiv preprint arXiv:1711.00937. https://arxiv.org/abs/1711.00937. [29] verl-omni Contributors. 2026. verl-omni: A Multimodal Reinforcement Learning Extension for verl. https://github.com/verl-project/verlomni. [30] Team Wan, Ang Wang, Baole Ai, et al. 2025. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv preprint arXiv:2503.20314. https://arxiv.org/abs/2503.20314. [31] Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. 2025. Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library. arXiv preprint arXiv:2506.06122. https://arxiv.org/abs/2506.06122. [32] Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. 2026. WEAVE: Efficient Co-Scheduling for Disaggregated RL PostTraining. arXiv preprint arXiv:2602.11143. https://arxiv.org/abs/2602. 11143. [33] Bo Wu, Sid Wang, Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, Xiaocheng Tang, Yundi Qian, Beibei Zhu, and Rui Hou. 2025. LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training. arXiv preprint arXiv:2505.24034. https://arxiv.org/abs/2505.24034. [34] Chenfei Wu, Jiahao Li, Jingren Zhou, et al. 2025. Qwen-Image Technical Report. arXiv preprint arXiv:2508.02324. https://arxiv.org/abs/ 2508.02324. [35] Tianyuan Wu, Lunxi Cao, Yining Wei, Wei Gao, Yuheng Zhao, Dakai An, Shaopan Xiong, Zhiqiang Lv, Ju Huang, Siran Yang, Yinghao Yu, Jiamang Wang, Lin Qu, and Wei Wang. 2026. Weave: Efficient CoScheduling for Disaggregated RL Post-Training. In Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’26). USENIX Association, Seattle, WA, USA. [36] Yongji Wu, Xueshen Liu, Haizhong Zheng, Juncheng Gu, Beidi Chen, Z. Morley Mao, Arvind Krishnamurthy, and Ion Stoica. 2026. RLBoost: Harvesting Preemptible Cloud Resources for Cost-Efficient Reinforcement Learning on LLMs. arXiv preprint arXiv:2510.19225. https://arxiv.org/abs/2510.19225. [37] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. arXiv preprint arXiv:2304.05977. https://arxiv.org/abs/2304.05977. [38] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. 2025. DanceGRPO: Unleashing GRPO on Visual Generation. arXiv preprint arXiv:2505.07818. https://arxiv.org/abs/2505.07818. [39] Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. 2023. Deepspeed-chat: Easy, fast
and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320 (2023). [40] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proceedings of the VLDB Endowment 16, 12 (2023), 3848–3860. [41] Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, and Daxin Jiang. 2025. StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation. arXiv preprint arXiv:2504.15930. https: //arxiv.org/abs/2504.15930.
14