UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models Jiaqi Wang 1 2 * Haoge Deng 2 * Ting Pan 2 * Yang Liu 2 Chengyuan Wang 2 Fan Zhang 2 Yonggang Qi 1 Xinlong Wang 2
arXiv:2604.18518v1 [cs.CV] 20 Apr 2026
Abstract Uniform Discrete Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose UDM-GRPO, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurate and stable optimization signals; and (ii) reconstructing trajectories via the diffusion forward process better aligns probability paths with the pretraining distribution. Additionally, we introduce two strategies, Reduced-Step and CFG-Free, to further improve training efficiency. UDM-GRPO significantly improves base model performance across multiple T2I tasks. Notably, GenEval accuracy improves from 69% to 96% and PickScore increases from 20.46 to 23.81, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR benchmark, accuracy rises from 8% to 57%, further validating the generalization ability of our method. Code is available at https://github.com/Yovecent/UDM-GRPO.
Figure 1. Reward–step training curve. The baseline suffers from optimization collapse after 500 steps, characterized by violent reward oscillation and exploding KL divergence. In contrast, our UDM-GRPO achieves stable convergence with sustained reward improvement and bounded KL loss.
generation. By using parallel token updates and progressive refinement, it outperforms traditional mask-based methods (Xie et al., 2024). Despite these advances, pretrained models often struggle with tasks requiring precise alignment with human preferences (Lee et al., 2023) or complex compositional generation (Yan et al., 2025). Meanwhile, Reinforcement Learning (RL) (Sutton et al., 1998), particularly Group Relative Policy Optimization (GRPO) (Shao et al., 2024), has proven highly effective in enhancing the reasoning capabilities of Large Language Models (LLMs) (Guo et al., 2025). Motivated by this success, recent works have extended GRPO to visual generation (Xue et al., 2025; Wang et al., 2025b). Notably, approaches like Flow-GRPO (Liu et al., 2025) have demonstrated substantial gains by formulating the denoising process as a policy optimization problem. However, the integration of RL into Uniform Discrete Diffusion remains largely unexplored. This work takes the first step toward bridging this gap. Drawing inspiration from Flow-GRPO, we first construct a baseline adaptation for Uniform Discrete Diffusion. To circumvent the non-differentiability inherent in the discretized sampling process, we define the policy action as the intermediate predicted sample at each timestep, theoretically aligning the optimization objective with the Flow-GRPO framework. However, this direct adaptation proves fundamentally unstable. As shown in Figure 1 (red curve), the model achieves transient gains during the first 500 steps. It then collapses catastrophically, exhibiting violent reward oscillations and an exploding KL divergence. We attribute this instability to two critical misalignments: (i) Inaccurate intermediate actions. Early-step predictions are high-entropy and inaccurate, so treating them as actions forces the model
1. Introduction Recent advances in visual generative models have achieved remarkable generation quality (Lipman et al., 2022; Ho et al., 2020; Chang et al., 2022). In parallel, Uniform Discrete Diffusion (Gat et al., 2024; Wang et al., 2025a; Deng et al., 2025) has emerged as a promising paradigm for discrete *
Equal contribution . This work was done at Beijing 1 Academy of Artificial Intelligence. Beijing University of Posts and Telecommunications 2 Beijing Academy of Artificial Intelligence. Correspondence to: Xinlong Wang <[email protected]>, Yonggang Qi <[email protected]>. Preprint. April 21, 2026.
1
to learn from noisy and incorrect signals; (ii) Biased distribution of backward trajectory. Optimizing on the reverse process induces a distribution shift from the forward process during pretraining. This discrepancy biases the learned probability path, effectively leading to out-of-distribution (OOD) training (Liu et al., 2021).
domains introduces challenges due to the categorical nature of discrete variables. Early works (Austin et al., 2021; Hoogeboom et al., 2021) formalized diffusion over categorical spaces via multinomial transitions and discrete denoising objectives. Building on these foundations, one line of work adopts masked image modeling (MIM) for discrete image generation through iterative masked token prediction (Chang et al., 2022; 2023; Xie et al., 2024; Bai et al., 2024; Hong et al., 2022; Yu et al., 2023), showing strong performance with efficient parallel decoding. More recently, uniform discrete diffusion (Gat et al., 2024) has emerged as a simplified formulation by explicitly parameterizing a timedependent categorical corruption process. Fudoki (Wang et al., 2025a) and Next-Omni (Luo et al., 2025a) integrate this framework into unified models for image generation, while URSA (Deng et al., 2025) demonstrates competitive or superior performance to continuous diffusion on both image and video benchmarks. In this work, we adopt URSA as our baseline, as it provides a strong and representative implementation of uniform discrete diffusion for image generation.
To address these challenges, we introduce UDM-GRPO, the first framework integrating Uniform Discrete Diffusion with GRPO for text-to-image generation. Our approach relies on two core strategies to ensure stability: (1) We redefine the policy action at all timesteps as the final clean sample rather than the intermediate noisy predicted sample. This forces the model to optimize toward an accurate, rewardconsistent target. (2) Instead of using the reverse process, we reconstruct training trajectories using the forward diffusion process. This ensures the optimization strictly adheres to the probability path defined during pretraining, eliminating the distribution shift caused by sampling errors. We further introduce two strategies to enhance the efficiency of training. To mitigate the slow convergence caused by gradient dispersion across multi-step optimization, we propose a Reduced-Step training strategy that concentrates optimization on critical high-noise timesteps. Additionally, a CFGFree scheme is adopted to avoid simultaneous optimization of conditional and unconditional objectives, substantially reducing computational overhead.
Reinforcement Learning in Text-to-Image Generation Reinforcement learning has become a key research direction for aligning text-to-image models with human preferences through feedback signals. Existing approaches can be broadly divided into two paradigms: (1) Direct Preference Optimization (Rafailov et al., 2023) casts alignment as a preference classification task over ranked output pairs, allowing direct policy updates without explicit reward modeling (Wallace et al., 2024; Deng et al., 2024a; Yang et al., 2024). (2) Policy-based RL methods. Early efforts in this line primarily adopt Proximal Policy Optimization (PPO) (Schulman et al., 2017). DDPO (Black et al., 2023) formulates diffusion denoising as a multi-step Markov Decision Process, enabling RL beyond likelihood maximization. Following its success in large language models, GRPO (Shao et al., 2024) has been extended to visual generation, including autoregressive models (Wang et al., 2025b), mask-based diffusion (Luo et al., 2025b), continuous diffusion models and flow-matching (Xue et al., 2025; Liu et al., 2025; Li et al., 2025; He et al., 2025). However, stable and effective RL for Uniform Diffusion remains underexplored. Building on this progress, we introduce GRPO to Uniform Diffusion. We observe that a direct adaptation of the Flow-GRPO formulation leads to severe training instability. To address these challenges, we propose UDMGRPO, the first framework that enables stable and efficient reinforcement learning for Uniform Discrete Diffusion.
The improvement of our methods is evident in Figure 1. In contrast to the baseline, UDM-GRPO demonstrates a stable and sustained increase in reward without collapse, while maintaining a low and bounded KL divergence, validating the robustness of our framework. Our contributions are summarized as follows: (1) We propose the first method to integrate GRPO into Uniform Discrete Diffusion for T2I tasks. UDM-GRPO addresses the instability of direct integration by unifying the action across timesteps as the final clean sample and reconstructing the training trajectory via the forward process. (2) We propose two strategies to improve training efficiency: Reduced-Step optimization and CFG-Free training. (3) Extensive experiments across multiple benchmarks validate the effectiveness of our approach. In particular, UDM-GRPO enables the base model URSA to achieve state-of-the-art performance on GenEval (Ghosh et al., 2024) and PickScore (Kirstain et al., 2023) for both discrete and continuous generation.
2. Related work Discrete Diffusion Model Diffusion models have achieved remarkable success in continuous domains, demonstrating strong sample quality and scalability for visual synthesis (Labs, 2024; Seedream et al., 2025; Brooks et al., 2024; Wan et al., 2025). Extending diffusion to discrete
3. Initial Exploration Uniform Discrete Diffusion has emerged as a robust paradigm for discrete generation. However, standard train2
ing relies on supervised cross-entropy minimization, which limits its ability to optimize complex, non-differentiable objectives or handle intricate generation tasks. Therefore, we adopt GRPO (Shao et al., 2024) to solve these limitations. Motivated by the success of Flow-GRPO (Liu et al., 2025), we explore a direct integration of the Flow-GRPO framework with UDM, as described in this section. Specifically, we first review the fundamentals of Uniform Discrete Diffusion and the Flow-GRPO formulation in Sections 3.1 and 3.2, respectively, and then introduce our preliminary approach to combine the two in Section 3.3.
Figure 2. Illustration of the three trajectories. Xbackward denoises x0 via the reverse process to obtain x̂1 . In contrast, Xforward and Xpretrain share the same forward diffusion process but differ in their clean sources: x̂1 for Xforward and x1 from the pretraining dataset for Xpretrain , resulting in x̂t and xt , respectively.
3.1. Uniform Discrete Diffusion Discrete Flow Matching (DFM)/Diffusion (Gat et al., 2024; Shaul et al., 2024) is a class of generative models that transport a source distribution p0 (x) to a target data distribution p1 (x) on discrete state spaces S = [K]D , where [K] = {1, . . . , K} denotes the vocabulary and D is the sequence length. In contrast to masking-based diffusion, which typically performs non-refinable local generation, uniform discrete diffusion starts from categorical noise and iteratively refines all tokens, enabling higher-fidelity synthesis. Specifically, Uniform Discrete Diffusion Model (UDM) specifies p0 (x) ∼ U nif ([K])D as the uniform distribution over the vocabulary and generates samples from p1 (x) by jointly updating all tokens across timesteps, which has gradually attracted attention due to its particularly high generation quality.
solver with a two-stage conditional sampling scheme for efficient generation (Shaul et al., 2024). Specifically, given xt , we first sample an intermediate prediction xt1 ∼ pθ (· | xt ) from the model trained under Eq. 3, and then update the state through a parameter-free rule-based mapping: xt+∆t ∼ δxt (·) + ∆t ut (·, xt | xt1 ),
where ut (·, xt | xt1 ) denotes the conditional probability velocity. Trajectory Definition. Given a caption–image pair (c, x1 ) from the pretraining dataset, we define three trajectories over t ∈ [0, 1], using the reverse-process timesteps: (i) the backward trajectory Xbackward = {xt }1t=0 , where xt ∼ pθ (xt+∆t | xt , c) is generated by following the reverse process, and x1 denotes the model’s estimate x̂1 ; (ii) the pretraining trajectory Xpretrain = {xt }1t=0 , where xt ∼ pt (x | x1 ) is generated by the forward diffusion process; (iii) the forward-process-based trajectory Xforward = {x̂t }1t=0 , where x̂t ∼ pt (x | x̂1 ) is obtained by perturbing x̂1 via the same forward diffusion process. Figure 2 provides a detailed illustration of the three trajectories.
Probability paths. To connect p0 (x) and p1 (x), DFM defines continuous intermediate distributions {pt (x)}t∈[0,1] , X pt (x) ≜ pt (x | x1 ) p1 (x1 ), (1) x1 ∈S
where pt (x | x1 ) is the conditional forward distribution. Probability velocities. To traverse the probability path {pt (x)}, we model the generation process as a continuoustime Markov chain (CTMC) driven by a time-dependent probability velocity ut , which guides the state from xt toward the terminal state x1 . For a small step size ∆t, the state update rule is xt+∆t ∼ δxt (·) + ∆t ut (· | xt ).
(4)
3.2. DDPO and Flow-GRPO Most diffusion–RL methods are based on Denoising Diffusion Policy Optimization (DDPO) (Black et al., 2023), which formulates the reverse denoising process Xbackward as a multistep MDP. Formally, the induced MDP is (S, A, ρ0 , P, R). At timestep t, the state is defined as st ≜ (c, t, xt ) where c denotes the prompt and xt is the latent variable. The action corresponds to the denoised sample predicted by the model, at ≜ xt−1 , and the policy is given by π(at | st ) ≜ pθ (xt−1 | xt , c). The transition is deterministic and specified by P (st+1 | st , at ) ≜ (δc , δt−1 , δxt−1 ), where δy denotes the Dirac delta distribution centered at y. The initial state distribution is ρ0 (s0 ) ≜ (p(c), δT , N (0, I)) and the reward is terminal-only: R(st , at ) ≜ r(x0 , c).
(2)
Training. The model is trained to predict the original data x1 ∼ p1 (x) from the noised data xt ∼ pt (x | x1 ) by minimizing the cross-entropy objective: LCE (θ) = Et∼U [0,1], x1 , xt − log pθ (x1 | xt ) . (3) Inference. Although sampling can theoretically be performed according to Eq. 2, in practice UDM adopts an Euler
Following this formulation, Flow-GRPO (Liu et al., 2025) 3
{1, . . . , K}B×D is a sequence of discrete tokens, where D denotes the number of tokens and satisfies D = H × W in the continuous formulation. The model outputs per-token logits pθ (· | xt ) ∈ RB×D×K . Given the intermediate predicted samples xt1 ∈ {1, . . . , K}B×D , the policy probability can be computed as
converts the ODE-based denoising dynamics into an SDE to integrate Flow Matching with GRPO (Shao et al., 2024). Specifically, given a prompt c, the model generates G trajeci i i i i tories {τ i }G i=1 , where τ = (x0 , x∆t , . . . , x1 ) and |τ | = T . The group-normalized advantage for the i-th trajectory is then computed as: R(xi1 , c) − mean {R(xi1 , c)}G i=1 (5) Âi = std {R(xi1 , c)}G i=1
πθ (at | st ) =
D Y
Softmax pθ (· | xt ):,ℓ,: xt1,ℓ .
(8)
ℓ=1
Accordingly, Flow-GRPO optimizes the policy model by maximizing the following objective:
Here, t pθ (· | xt ):,ℓ,: denotes the logits at position ℓ, and x1,ℓ indicates indexing by the sampled token at ℓ.
J(θ) = Ec∼C,{τ i }G i=1 ∼πθold (·|c) " # G 1 1 X 1 X (t,i) J − β DKL πθ πref G i=1 T t=0 policy
By redefining the action as xt1 and choosing to optimize along the Xbackward trajectory, we implement a preliminary integration of GRPO with UDM, which empirically improves the performance of the base model (Table 3).
(6)
where
4. Method
(t,i) Jpolicy = min rti (θ)Âi , clip(rti (θ), 1 − ϵ, 1 + ϵ)Âi
In this section, we first analyze the limitations that arise from a naive integration of Uniform Diffusion with Flow-GRPO in Section 4.1. We then propose UDM-GRPO in Section 4.2 to address these limitations. Finally, we present training acceleration strategies in Sections 4.3 and 4.4, including Reduced-Step training and CFG-Free training, respectively.
pθ (xit+∆t | xit , c) rti (θ) = pold (xit+∆t | xit , c) 3.3. Pilot Integration of GRPO and Uniform Discrete Diffusion
4.1. Instability Challenges of Uniform Diffusion under GRPO
In our early study, we explored adapting Flow-GRPO to Uniform Discrete Diffusion, which focuses on the reverse sampling process Xbackward . Since the optimization objective remains unchanged, the main challenge is how to calculate the transition probability pθ (xt+∆t | xt , c) under Uniform Discrete Diffusion.
As described in Section 3, we integrate Uniform Diffusion with GRPO by explicitly redefining the transition probability. However, as illustrated by the red curve in Figure 1, the reward initially increases during the first 500 training steps but soon exhibits severe fluctuations, while the KL divergence grows sharply, leading to unstable training dynamics and degraded performance. Further analysis reveals that this instability arises from the following two factors:
Recalling the Euler solver described in Section 3.1, we derive the following two properties. First, the transition from xt to xt+∆t requires sampling the intermediate prediction xt1 ; however, this non-differentiable step blocks gradient propagation from xt to xt+∆t . Second, conditioned on xt1 , the distribution of xt+∆t is uniquely determined by a fixed, parameter-free mapping. Consequently, the generation of xt+∆t is entirely governed by xt1 , implying that learning pθ (xt+∆t | xt , c) is effectively equivalent to learning pθ (xt1 | xt , c).
Problem I: Inaccurate Intermediate Actions. As shown in Figure 4, we visualize the entropy of the model output pθ (· | xt ) for Xbackward and the corresponding predictions xt1 at different timesteps (first row). Early stages exhibit high entropy, reflecting inherent uncertainty and yielding incoherent and noisy predictions. As the diffusion process proceeds, the entropy gradually decreases, and the model eventually predicts a clean sample x̂1 . However, in our current RL-based formulation, each intermediate prediction xt1 is treated as the action. Although a positive advantage A > 0 is induced by the accurate final prediction x̂1 , maximizing pθ (xt1 | xt ) compels the model to imitate unreliable intermediate predictions xt1 at early timesteps. As a result, the model learns misleading information, which destabilizes the training process and can ultimately lead to collapse.
Based on the above considerations, we redefined the action and policy as at ≜ xt1 ,
πθ (at | st ) ≜ pθ (xt1 | xt , c).
(7)
This formulation preserves differentiability, retains Euler sampling efficiency, and enables policy optimization. In discrete diffusion models, the network outputs logits directly, which enables more convenient and accurate probability computation. Specifically, the latent state xt ∈ 4
Figure 3. Overview of UDM-GRPO. Given a prompt, we first sample G clean images x̂1 using the reverse process of UDM. To solve the instability caused by directly using this Xbackward as trajectory and xt1 as action, we construct the training trajectory Xforward by perturbing x̂1 with forward process at different timesteps. Then we use Xforward as trajectory and x̂1 as action to calculate the transition probability pθ (x̂1 | x̂t ). Finally, we compute the advantage from rewards of x̂1 and optimize the policy model using the GRPO loss. Moreover, to improve training efficiency, we adopt the CFG-Free Strategy during sampling and the Reduced-Step Strategy to select early timesteps for policy optimization (blue dashed box).
4.2. UDM-GRPO In this section, we propose UDM-GRPO, a new framework that resolves the aforementioned limitations. Key Insight I: To achieve more stable and precise optimization, it is desirable to select the accurate and rewardaligned denoised sample as the action. Standard diffusion pretraining inherently treats the clean image as the target for all t. Consistent with this principle, and given that our reward is defined solely on the final clean sample x̂1 , we redefine the action at all timesteps in Eq. 7 to be x̂1 : at ≜ x̂1 ,
π(at | st ) ≜ pθ (x̂1 | xt , c),
(9)
This modification not only provides a more rewardconsistent and precise optimization direction for RL, but also leads to further performance improvements (Table 3).
Figure 4. (i) The entropy of pθ (· | xt ) along the Xbackward trajectory, and the FID between Xbackward and Xpretrain as well as between Xforward and Xpretrain at different denoising timesteps (top). (ii) Visual comparison of the predicted xt1 images: Xbackward (first row), Xpretrain (second row), and Xforward (third row).
Key Insight II: To mitigate distribution shift and preserve consistency with pretraining, the training trajectory should closely adhere to the forward diffusion process.
Problem II: Biased Distribution of Backward Trajectory. During pretraining, the model is trained to predict x1 from the true forward process Xpretrain , whereas during RL finetuning, optimization is performed on the model’s own generated trajectory Xbackward . To investigate the discrepancy between these two distributions, we visualize the intermediate predictions xt1 in Figure 4. We observe a pronounced contrast: predictions from Xpretrain (second row) gradually become clearer starting from step 1, whereas those from Xbackward (first row) remain significantly noisy during the early denoising steps. This performance gap indicates that xt ∈ Xbackward has drifted away from the training manifold Xpretrain . Consequently, RL training exposes the model to out-of-distribution (OOD) (Liu et al., 2021) states, forcing it to learn from a biased probability trajectory.
Based on this, we adopt Xforward instead of Xbackward as the training trajectory. We validate this choice by quantifying the discrepancy between the intermediate predictions of each trajectory and the pretraining distribution Xpretrain using Fréchet Inception Distance (FID) (Heusel et al., 2017). Detailed experimental settings are provided in Appendix B. As illustrated in Figure 4, we can clearly see that Xforward yields consistently lower FID across all timesteps. In contrast, Xbackward suffers from significant deviation caused by error accumulation in the early predictions, particularly in steps 1 and 2. The qualitative visualization further exhibits the same trend, with Xforward (third row) producing predictions that are visibly more consistent with Xpretrain than those 5
from Xbackward . These results indicate that Xforward is better aligned with the pretraining distribution Xpretrain . Through the above two modifications, we reformulate the original T -step MDP as follows: st ≜ (x̂t , t, c),
at ≜ x̂1 ,
R(s, a) ≜ r(x̂1 , c),
π(at | st ) ≜ pθ (x̂1 | x̂t , c), ρ0 (s) ≜ U nif ([K])D . (10)
Thus, the optimization objective remains the same as in Eq. 6, with the policy ratio now reformulated as: rti (θ) =
pθ (x̂i1 | x̂it , c) pθold (x̂i1 | x̂it , c)
(11) Figure 5. Qualitative Comparison. We evaluate our model against SD3.5-L, Flux.1 Dev and URSA using prompts from GenEval and PickScore, respectively.
As shown in Figure 1, our approach demonstrates significant advantages in convergence speed, performance, and stability, and Table 3 provides a more quantitative evaluation of these improvements. Compared to prior methods, it optimizes the forward process, naturally avoiding sampler constraints, high memory overhead, and inconsistencies with pretraining. Moreover, under this formulation, the policy loss for both diffusion models and LLMs can be viewed as an advantage-weighted version of their respective pre-trained losses, which further supports the validity and naturalness of our approach. A detailed overview of the framework is provided in Figure 3.
initially degrades generation quality, the effect is transient: as training progresses, the model recovers and ultimately surpasses conventional CFG-based methods (Table 3).
5. Experiments 5.1. Experimental Settings Datasets and Reward Models. Our experiments cover three tasks: Compositional Image Generation, Visual Text Rendering, and Human Preference Alignment. Compositional Image Generation evaluates the model’s ability to understand and generate images with complex compositional constraints such as object count, color, and spatial relations. Visual Text Rendering focuses on accurately and consistently rendering text in realistic scenarios such as posters, advertisements, and books. Human Preference Alignment measures the alignment between generated images and human subjective preferences. For all tasks, we use the same datasets and corresponding reward models as in Flow-GRPO (Liu et al., 2025) for training and evaluation.
4.3. Reduced-Step Accelerated Optimization Uniform optimization over all diffusion timesteps disperses gradients across the denoising trajectory, leading to inefficient convergence (He et al., 2025). As illustrated in Figure 4, the diffusion denoising process exhibits temporally decreasing stochasticity: early timesteps maintain high entropy, enabling extensive exploration of the state space, while later timesteps become more deterministic. The visualization of prediction xt1 for Xbackward also shows that early denoising steps incur higher prediction errors, highlighting the need to focus optimization efforts on these stages. As a result, concentrating the optimization on steps with high noise leads to more significant gains. Motivated by this observation, we adopt a Reduced-Step training strategy to improve efficiency: for each sample x1 , we randomly select three consecutive timesteps from the first half of the diffusion timestep for training. As shown in Section 5.3, this approach significantly accelerates convergence.
Training and Evaluation. We fine-tune a 1.7B-parameter text-to-image generation model pre-trained by URSA (Deng et al., 2025). Group sampling is adopted during training, with each batch comprising 16 groups of 8 image samples. We use AdamW optimizer (Loshchilov & Hutter, 2019) with β1 = 0.9, β2 = 0.95, a weight decay of 0.01, and a constant learning rate of 1e-6. We default to 10 inference steps for group sampling and 25 inference steps for evaluation. All experiments are conducted on 32 A100 (40GB) GPUs.
4.4. CFG-Free Training 5.2. Main Results
Most prior Diffusion–RL methods rely on classifier-free guidance (CFG) training, which jointly optimizes conditional and unconditional models and substantially increases training complexity. To simplify optimization, we eliminate CFG during training. Although this CFG-Free approach
To evaluate the effectiveness of our method, we integrate pretrained URSA (Deng et al., 2025) text-to-image models with UDM-GRPO. As shown in Table 1, UDM-GRPO boosts the overall GenEval (Ghosh et al., 2024) score from 0.69 to 0.96, 6
Table 1. Comparison result on GenEval. Methods combined with GRPO are color-coded in gray. The best and second-best scores are marked in bold and underlined, respectively. Results for models other than ours are from (Liu et al., 2025) or their original papers. Model
#Params Overall Single Obj. Two Obj. Counting Colors Position Attr. Binding
▼ Continuous models SD2.1 (Rombach et al., 2022) SDXL (Podell et al., 2023) SANA-1.5 4.8B (Xie et al., 2025) NOVA (Deng et al., 2024b) FLUX.1 Dev (Labs, 2024) SD3.5-L (Esser et al., 2024) SD3.5-M (Esser et al., 2024) SD3.5-M (w/ Flow-GRPO) (Liu et al., 2025)
0.9B 2.6B 4.8B 1.4B 12B 8B 2.5B 2.5B
0.50 0.55 0.81 0.71 0.66 0.71 0.63 0.95
0.98 0.98 0.99 0.99 0.98 0.98 0.98 1.00
0.51 0.74 0.93 0.91 0.81 0.89 0.78 0.99
0.44 0.39 0.86 0.62 0.74 0.73 0.50 0.95
0.85 0.85 0.84 0.85 0.79 0.83 0.81 0.92
0.07 0.15 0.59 0.33 0.22 0.34 0.24 0.99
0.17 0.23 0.65 0.56 0.45 0.47 0.52 0.86
Emu3-Gen (Wang et al., 2024) SimpleAR (Wang et al., 2025b) MaskGen-XL (Kim et al., 2025) Show-o (Xie et al., 2024) Show-o (w/ Mask-GRPO) (Luo et al., 2025b) FUDOKI (Wang et al., 2025a) Emu3.5 (DiDA) (Cui et al., 2025) URSA (Deng et al., 2025)
8.5B 1.5B 1.1B 1.3B 1.3B 1.5B 34B 1.7B
0.54 0.63 0.57 0.53 0.73 0.77 0.86 0.69
0.98 0.61 0.95 0.99 0.96 0.99
0.71 0.90 0.55 0.52 0.90 0.85 0.91
0.34 0.81 0.49 0.69 0.56 0.60
0.81 0.13 0.82 0.85 0.88 0.87
0.17 0.28 0.31 0.11 0.35 0.68 0.28
0.21 0.45 0.57 0.28 0.59 0.67 0.49
URSA (w/ UDM-GRPO)
1.7B
0.96
1.00
1.00
0.95
0.97
0.97
0.85
▼ Discrete models
visual fidelity and text-image alignment for UDM models.
Table 2. Comparison results on GenEval, PickScore, and OCR. Our method, UDM-GRPO, is highlighted in gray. The best and secondbest results are indicated by bold and underlined, respectively. Model
GenEval
PickScore
OCR
SDXL (Podell et al., 2023) SD3.5-L (Esser et al., 2024) FLUX.1-Dev (Labs, 2024) URSA (Deng et al., 2025) URSA (w/o CFG) (Deng et al., 2025)
0.55 0.71 0.66 0.69 0.36
22.42 22.91 22.84 21.79 20.46
0.14 0.68 0.59 0.08 0.04
UDM-GRPO
0.96
23.81
0.57
We further evaluate our approach on two downstream tasks: text rendering and human preference alignment. As shown in Table 2, UDM-GRPO achieves the state-of-the-art performance on PickScore. For text rendering, although the pre-trained model exhibits poor ocr performance, UDMGRPO still achieves a substantial improvement. 5.3. Reduced-Step Accelerated Optimization
Table 3. Ablation study of different methods for integrating GRPO into our base model on GenEval, PickScore and OCR. The best and second-best results are indicated by bold and underlined, respectively. Model URSA URSA URSA URSA URSA (w/o CFG)
In this section, we investigate the strategies for few-step optimization to demonstrate the efficiency of our approach. We evaluate three timestep selection strategies: (1) Early highnoise timesteps: optimizing the three consecutive timesteps from the first half of the diffusion timestep; (2) Random consecutive timesteps: randomly sampling the three consecutive timesteps; and (3) All timesteps: optimizing over the entire diffusion timesteps.
Action Trajectory GenEval PickScore OCR xt1 x̂1 x̂1 x̂1
backward backward forward forward
0.69 0.84 0.89 0.94 0.96
21.79 21.99 23.10 23.51 23.81
0.08 0.23 0.23 0.34 0.57
We conduct head-to-head comparisons of the three strategies described above on GenEval, PickScore, and OCR tasks. As shown in subplots (d1, d2, d3) of Figure 6, optimizing the early high-noise timesteps yields a clear advantage on GenEval and PickScore, while the performance differences remain minor for OCR. These results demonstrate both the efficiency and effectiveness of few-step training approach.
surpassing both prior RL methods and pre-trained baselines across a range of model variants and sizes, establishes a new state-of-the-art under both continuous and discrete settings. Figure 5 presents qualitative comparisons of T2I generation among SD3.5, FLUX, URSA, and our method. After RL post-training, the URSA model exhibits substantial improvements in spatial arrangement, attribute binding, and object counting on the compositional prompts of GenEval. On the trival prompts of PickScore (Kirstain et al., 2023), UDM-GRPO produces images with finer details and fewer artifacts, preserving style consistency. These improvements highlight the effectiveness of our approach in enhancing
5.4. Ablation Study Action Choice: Final Clean Sample x̂1 vs. Intermediate Predicted Sample xt1 . We study how the action parameterization affects backward optimization. We compare two choices: (i) using the final denoised output at t = 1 as the 7
Figure 7. Qualitative Comparison. We compare different methods for integrating GRPO into our base model. From left to right, the results correspond to (a): backward + xt1 , (b): backward + x̂1 , (c): forward + x̂1 , and (d): forward + x̂1 + CFG-free.
CFG-free. vs. CFG. We compare forward optimization with and without classifier-free guidance (CFG). As shown in Figure 6 and Table 3, although the CFG-Free (blue) setting performs poorly early in training due to lower sample quality, it surpasses the CFG-based (orange) training as optimization progresses. Overall, CFG-Free converges faster and achieves a lower KL divergence, indicating improved training efficiency and stability. Notably, on the OCR task, CFG-Free outperforms the CFG-based setting, suggesting that removing CFG broadens the policy distribution and enables more effective exploration during RL training.
Figure 6. Experimental results. Performance metrics and KL loss on GenEval (a1, a2), PickScore (b1, b2), and OCR (c1, c2). The effects of different timestep optimization strategies across tasks are shown in (d1, d2, d3).
action, a = x̂1 , and (ii) using the model’s step-t estimate of the final output as the action, a = xt1 for t ∈ (0, 1). Starting from the original Flow-GRPO-based formulation, we replace the intermediate-prediction action (red) with the final-sample action (green) and perform head-to-head comparisons on GenEval, PickScore, and OCR in Figure 6 and Table 3. Using a = x̂1 consistently improves training performance on GenEval and PickScore, and leads to a lower KL divergence to the reference policy, suggesting more stable optimization. The gain is less pronounced on OCR, indicating that the underlying challenge for text rendering remains.
Qualitative Result. We also provide visualizations corresponding to the ablation experiments described above, as shown in Figure 7. From left to right, the results correspond to: backward + xt1 , backward + x1 , forward + x̂1 , and forward + x̂1 + CFG-Free. It can be observed that our final method demonstrates clear advantages over the previous approaches.
Trajectory Choice: Forward vs. Backward. We compare forward and backward optimization, which differ only in the construction of the state xt . Specifically, forward optimization resamples xt from the forward diffusion process, whereas backward optimization uses intermediate denoising states from the reverse process. The action is defined as the clean sample in both settings. As shown in Figure 6 and Table 3, although both methods exhibit similar performance in the early training stage, backward optimization (green) later suffers from slow convergence and eventually collapses when the GenEval score reaches approximately 0.89. In contrast, forward optimization (orange) improves smoothly and stably, reaching a score of 0.95. Moreover, forward optimization consistently yields lower KL divergence, indicating superior training stability. Similar performance trends are also observed on the other two tasks.
6. Conclusion In this paper, we propose UDM-GRPO, the first method that integrates the Uniform Discrete Diffusion Model with GRPO for text-to-image generation. By treating the final clean sample as the action and reconstructing the trajectory through the forward diffusion process, our method effectively addresses the instability caused by naive adaptation. Furthermore, we enhance training efficiency via ReductionStep and CFG-free strategy. UDM-GRPO significantly improves the performance of the base model across multiple T2I tasks. In future work, we will investigate extending our framework to text-to-video generation and to more challenging multi-reward optimization task. 8
Impact Statement
Deng, F., Wang, Q., Wei, W., Hou, T., and Grundmann, M. Prdp: Proximal reward difference prediction for largescale reward finetuning of diffusion models. In CVPR, 2024a.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Deng, H., Pan, T., Diao, H., Luo, Z., Cui, Y., Lu, H., Shan, S., Qi, Y., and Wang, X. Autoregressive video generation without vector quantization. In arXiv preprint arXiv:2412.14169, 2024b.
Acknowledgement This work was supported by the Hainan Provincial Joint Project of Li’an International Education Innovation Pilot Zone (Grant No.624LALH008), BUPT Kunpeng&Ascend Center of Cultivation, NSFC (Grant No.61601042), and the Super Computing Platform of BUPT.
Deng, H., Pan, T., Zhang, F., Liu, Y., Luo, Z., Cui, Y., Wang, W., Shen, C., Shan, S., Zhang, Z., et al. Uniform discrete diffusion with metric path for video generation. In arXiv preprint arXiv:2510.24717, 2025. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
We would like to acknowledge Shilin Lu and Yuanzhi Zhu for the insightful discussions. We are grateful to Jiazhen Yan, Junwei Liu, Yuanyuan Li, Shaqi Luo, and Shuchen Weng for their significant support to this work. We also thank Yuanzhi Zhu, Zhipeng Chen, Jing Zuo, and Hongcan Xiao for their valuable feedback on the draft.
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching. In NeurIPS, 2024.
References
Ghosh, D., Hajishirzi, H., and Schmidt, L. Geneval: An object-focused framework for evaluating text-to-image alignment. In NeurIPS, 2024.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. In NeurIPS, 2021.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. In arXiv preprint arXiv:2501.12948, 2025.
Bai, J., Ye, T., Chow, W., Song, E., Chen, Q.-G., Li, X., Dong, Z., Zhu, L., and Yan, S. Meissonic: Revitalizing masked generative transformers for efficient highresolution text-to-image synthesis. In ICLR, 2024.
He, X., Fu, S., Zhao, Y., Li, W., Yang, J., Yin, D., Rao, F., and Zhang, B. Tempflow-grpo: When timing matters for grpo in flow models. In arXiv preprint arXiv:2508.04324, 2025.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. In arXiv preprint arXiv:2305.13301, 2023. Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models as world simulators. In https://openai.com/research/video-generation-modelsas-world-simulators, 2024.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In CVPR, 2022.
Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-tovideo generation via transformers. In arXiv preprint arXiv:2205.15868, 2022.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image generation via masked generative transformers. In arXiv preprint arXiv:2301.00704, 2023.
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. In NeurIPS, 2021.
Cui, Y., Chen, H., Deng, H., Huang, X., Li, X., Liu, J., Liu, Y., Luo, Z., Wang, J., Wang, W., et al. Emu3. 5: Native multimodal models are world learners. In arXiv preprint arXiv:2510.26583, 2025.
Kim, D., He, J., Yu, Q., Yang, C., Shen, X., Kwak, S., and Chen, L.-C. Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. In arXiv preprint arXiv:2501.07730, 2025. 9
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurIPS, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06347, 2017. Seedream, T., Chen, Y., Gao, Y., Gong, L., Guo, M., Guo, Q., Guo, Z., Hou, X., Huang, W., Huang, Y., et al. Seedream 4.0: Toward next-generation multimodal image generation. In arXiv preprint arXiv:2509.20427, 2025.
Labs, B. F. Flux. In https://github.com/black-forest-labs/flux, 2024. Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text-to-image models using human feedback. In arXiv preprint arXiv:2302.12192, 2023.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. In arXiv preprint arXiv:2402.03300, 2024.
Li, J., Cui, Y., Huang, T., Ma, Y., Fan, C., Yang, M., and Zhong, Z. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde. In arXiv preprint arXiv:2507.21802, 2025.
Shaul, N., Gat, I., Havasi, M., Severo, D., Sriram, A., Holderrieth, P., Karrer, B., Lipman, Y., and Chen, R. T. Flow matching with general discrete paths: A kinetic-optimal perspective. In arXiv preprint arXiv:2412.03487, 2024.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In arXiv preprint arXiv:2210.02747, 2022.
Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction. In MIT press Cambridge, 1998.
Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards out-of-distribution generalization: A survey. In arXiv preprint arXiv:2108.13624, 2021.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct preference optimization. In CVPR, 2024.
Liu, J., Liu, G., Liang, J., Li, Y., Liu, J., Wang, X., Wan, P., Zhang, D., and Ouyang, W. Flow-grpo: Training flow matching models via online rl. In NeurIPS, 2025.
Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., et al. Wan: Open and advanced large-scale video generative models. In arXiv preprint arXiv:2503.20314, 2025.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In ICLR, 2019.
Wang, J., Lai, Y., Li, A., Zhang, S., Sun, J., Kang, N., Wu, C., Li, Z., and Luo, P. Fudoki: Discrete flow-based unified understanding and generation via kinetic-optimal velocities. In arXiv preprint arXiv:2505.20147, 2025a.
Luo, R., Xia, X., Wang, L., Chen, L., Shan, R., Luo, J., Yang, M., and Chua, T.-S. Next-omni: Towards anyto-any omnimodal foundation models with discrete flow matching. In arXiv preprint arXiv:2510.13721, 2025a.
Wang, J., Tian, Z., Wang, X., Zhang, X., Huang, W., Wu, Z., and Jiang, Y.-G. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. In arXiv preprint arXiv:2504.11455, 2025b.
Luo, Y., Hu, X., Fan, K., Sun, H., Chen, Z., Xia, B., Zhang, T., Chang, Y., and Wang, X. Reinforcement learning meets masked generative models: Mask-grpo for text-toimage generation. In arXiv preprint arXiv:2510.13418, 2025b.
Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need. In arXiv preprint arXiv:2409.18869, 2024.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In arXiv preprint arXiv:2307.01952, 2023.
Xie, E., Chen, J., Zhao, Y., Yu, J., Zhu, L., Wu, C., Lin, Y., Zhang, Z., Li, M., Chen, J., et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. In arXiv preprint arXiv:2501.18427, 2025.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.
Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. In arXiv preprint arXiv:2408.12528, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 10
Xue, Z., Wu, J., Gao, Y., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., et al. Dancegrpo: Unleashing grpo on visual generation. In arXiv preprint arXiv:2505.07818, 2025. Yan, Z., Ye, J., Li, W., Huang, Z., Yuan, S., He, X., Lin, K., He, J., He, C., and Yuan, L. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation. In arXiv preprint arXiv:2504.02782, 2025. Yang, S., Chen, T., and Zhou, M. A dense reward view on aligning text-to-image diffusion with preference. In arXiv preprint arXiv:2402.08265, 2024. Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In CVPR, 2023.
11
Appendix In this appendix, implementation details, experiments, and qualitative results are organized as follows: • Training Details (A) • Distribution Analysis (B) • Extended Experimental Results (C)
A. Training Details A.1. Pseudo Code for UDM-GRPO We present the detailed pseudo code of the proposed UDM-GRPO in Algorithm 1. Algorithm 1 UDM-GRPO 1: Input: KL weight β, clip parameter ϵ, reference policy πref , candidate timestep groups Tgroup 2: Initialize: θ ← θold 3: for each iteration n = 1, 2, . . . do 4: for each prompt c ∼ C do 5: Sample G trajectories {τ i }G // CFG-Free i=1 ∼ πθold (· | c) G 6: Extract clean samples {xˆ1 i }G and advantages { Â } i i=1 i=1 7: Sample candidate timestep groups {(t1i , t2i , t3i )}G // Reduction-Step i=1 ∼ Tgroup 8: end for 9: Initialize total loss L ← 0 10: for i = 1, . . . , G do 11: for j = 1, 2, 3 do 12: Sample noisy state x̂ij ∼ ptj (x | x̂i1 ) // Forward-Process 13:
rtij (θ) ← i
ti pθ (x̂i1 |x̂ij ,c) t
i
// Accurate Action Strategy
i
pθold (xˆi1 |x̂ij ,c) t
(tj ,i)
i
14:
i Jpolicy ← min rtij (θ)Âi , clip(rtij (θ), 1 − ϵ, 1 + ϵ)Âi
15:
(tji ,i) L ← L − Jpolicy + β DKL
i
i
pθ (· | x̂itj , c) ∥ pref (· | x̂itj , c) i
i
16: end for 17: end for 18: θ ← θ − λ∇θ L 19: θold ← θ 20: end for
// Policy Optimization
B. Distribution Analysis In Section 4.2, we validate that Xforward is closer to the forward trajectory than Xbackward by comparing the FID between each trajectory and Xpretrain , as well as through visual comparisons. In this section, we provide a detailed description of the experimental setup and computation procedure. Specifically, we first sample 2,048 pairs of captions and corresponding images (c, x1 ) from the URSA pretraining dataset. Following the trajectory definitions in Section 3.1, we generate three types of trajectories for each pair: the forward trajectory Xforward , the backward trajectory Xbackward , and the pretraining trajectory Xpretrain . For each trajectory, we sample xt1 from the conditional distribution pθ (xt1 | xt ) given xt . The resulting index predictions are subsequently converted to image space using the model’s standard decoding procedure. To quantitatively measure how closely each trajectory aligns with the pretraining trajectory Xpretrain , we compute the Fréchet Inception Distance (FID) between the distributions of predicted images at the same timestep. Specifically, for a given timestep t, we treat the set of predicted images from Xforward and Xbackward as two empirical distributions and compute their FID with respect to Xpretrain . All FID computations are performed using the official PyTorch implementation, ensuring consistency with standard evaluation practices. 12
C. Extended Experimental Results In this section, we conduct additional experiments to systematically demonstrate the effectiveness of our method from multiple perspectives. C.1. Generalized Validation To further validate the generality of our method, we conduct additional experiments on FUDOKI(Wang et al., 2025a), a UDM-based multimodal large language model that unifies visual understanding and image generation. We adopt the same experimental setup as used for URSA and conduct training across three benchmarks—GenEval, PickScore, and OCR—using 15 inference steps for group sampling and 32 inference steps for evaluation. As shown in the Table 4, UDM-GRPO consistently and significantly improves the performance of the original model. These findings demonstrate that our method is not only effective for standalone T2I models, but also generalizes well to MLLMs, further confirming its broad applicability. Table 4. Performance of FUDOKI with UDM-GRPO on GenEval, PickScore and OCR tasks. Model
#Params PickScore OCR GE(Overall) GE(Single Obj.) GE(Two Obj.) GE(Counting ) GE(Colors) GE(Position) GE(Attr. Binding)
FUDOKI
1.5B
21.32
0.04
0.76
0.96
0.86
0.51
0.90
0.67
0.64
FUDOKI (w/ UDM-GRPO)
1.5B
23.40
0.26
0.86
0.99
0.90
0.89
0.99
0.87
0.72
C.2. Different Model Performance Comparison We provide additional visualizations comparing the baseline, our method, SD3.5-L, and Flux.1 Dev to further highlight the superior performance of our approach as shown in Figure 8. In particular, our model demonstrates enhanced generative capacity, including the ability to accurately model complex scenes with a larger number of objects and to produce outputs that better align with human perceptual quality metrics. These results further illustrate the potential of UDM-GRPO to augment the capabilities of the base model.
Figure 8. Qualitative Comparison. The prompts are taken from GenEval, PickScore respectively, where we compare the SD3.5-L and Flux.1 Dev with our model.
13
C.3. Different Method Qualitative Comparision In this section, we present the results of models trained with different methods, as discussed in Section 5, thereby enabling a comprehensive comparison of their performance. As shown in Figures 9, the methods from left to right correspond to backward + xt1 , backward + x1 , forward + x̂1 , and forward + x̂1 + CFG-free. We observe that the initial integration with GRPO already improves the model’s capabilities, while the addition of Accurate Action and the forward strategy further improves performance. Moreover, the results indicate that CFG does not compromise generative quality, demonstrating the model’s strong generative capacity.
a photo of a blue umbrella
a photo of a green skis and a brown airplane
a photo of an orange truck and a pink sink
a photo of four frisbees
a photo of a toothbrush and a carrot
a photo of an elephant below a horse
Figure 9. Visualization for different method.
14
C.4. Training Process To better understand the training dynamics of our UDM-GRPO framework, we visualize the evolution of generated samples corresponding to fixed evaluation prompts at regular intervals during training. As shown in Figure 10, it is evident that the quality of generated samples improves progressively over time, with their accuracy steadily increasing.
Training Process
An ant in the shape of a letter a, insanely detailed, photorealistic, 8k, ultra high resolution, volumetric lighting, taken with canon eos,
wizard head, 2d, vector illustration, logo, 2d flat, centered, fitness company, white background, paul rand
a photo of a bus above a boat
Painting of a brane sphere interior, metallic shimmer, noctilucent, intricate, photorealistic, colorful, psychic style
Lunch in Bavaria - oil painting
Figure 10. We visualize the generated samples across successive training iterations during the optimization.
15