: Generate Images by Optimizing What You Reward Onkar Susladkar1 Dong-Hwan Jang1 Tushar Prakash2 Adheesh Juvekar1 Vedant Shah1 Ayush Barik1 Nabeel Bashir1 Muntasir Wahed1 Ritish Shrirao2 Ismini Lourentzou1 1
University of Illinois Urbana-Champaign
2
Sony Research, India
arXiv:2604.08536v1 [cs.CV] 9 Apr 2026
{onkarks2, lourent2}@illinois.edu
Change forest to a wizard house
Change pitcher to wooden
ard w e R
Flow
Make a Ghibli style portrait
Steam rising from a ceramic cup of coffee on a wooden table, shallow depth of field, warm morning light, ultra-detailed foam, realistic photography style
Make “CLOSE” to “OPEN”
Change cake to square cake
Stormy ocean waves crashing against jagged cliffs, dramatic sky with rays of sunlight breaking through clouds
Photorealistic dragon perched on a mountain peak at sunset, detailed scales and wings, warm rim lighting, cinematic composition
Figure 1. RewardFlow enables accurate, localized, inversion-free image editing and generation using multi-reward Langevin guidance.
Abstract
throughout sampling. Across several image editing and compositional generation benchmarks, RewardFlow delivers state-of-the-art edit fidelity and compositional alignment. PLAN Lab https://plan-lab.github.io/rewardflow
We introduce RewardFlow, an inversion-free framework that steers pretrained diffusion and flow-matching models at inference time through multi-reward Langevin dynamics. RewardFlow unifies complementary differentiable rewards for semantic alignment, perceptual fidelity, localized grounding, object consistency, and human preference, and further introduces a differentiable VQA-based reward that provides fine-grained semantic supervision through language-vision reasoning. To coordinate these heterogeneous objectives, we design a prompt-aware adaptive policy that extracts semantic primitives from the instruction, infers edit intent, and dynamically modulates reward weights and step sizes
1. Introduction Text-guided image generation and editing have become the most active frontiers in generative modeling, driven by recent advances in diffusion and flow-matching models [2, 8, 16, 24, 38]. The ability to generate or modify an image based solely on natural-language instructions has enabled diverse applications in visual design, content cre1
ation, and interactive editing. While significant progress has been made in fine-tuning–based approaches [5, 11, 25, 37], these methods require expensive optimization and exhibit limited generalization beyond the training distribution. Consequently, training-free and inversion-free methods have emerged as a practical alternative. By operating directly on pretrained models at inference time without modifying model weights, such methods offer broad applicability and efficient deployment [1, 12, 15, 19, 23, 49]. However, on one hand, inversion often distorts layout or identity and introduces brittle forward–reverse sampling loops. On the other hand, inversion-free methods circumvent reconstruction but lose access to a faithful latent representation of the original image. As a result, they frequently suffer from content drift, semantic leakage, weak object localization, and insufficient fine-grained controllability. More recently, reward-guided frameworks [9, 27, 45] attempt to address controllability, but they employ coarse rewards with weak semantic grounding, lack adaptive policies, and provide no mechanism for harmonizing heterogeneous objectives over the sampling trajectory. These shortcomings prevent existing methods from achieving precise, consistent, and semantically faithful edits in a truly zero-shot setting. To address these challenges, we propose RewardFlow, a zero-shot, training-free, and inversion-free framework for text-guided image editing and generation based on multireward Langevin dynamics. RewardFlow fuses complementary signals, including global semantics, perceptual alignment, spatial grounding, aesthetic quality, and semantic faithfulness, into a differentiable objective that guides a pretrained flow-matching model at inference time. Specifically, we introduce two new reward formulations that significantly improve localized accuracy and compositional alignment: (i) a SAM2 [35] text-guided object reward that produces differentiable localization signals, enforcing mask-consistent edits and penalizing leakage outside target regions, and (ii) a differentiable VQA reward that enforces fine-grained semantic correctness through language–vision reasoning. To balance coarse-to-fine scheduling of these heterogeneous reward signals, we introduce a novel prompt-aware adaptive policy that extracts semantic primitives from the instruction, and dynamically adjusts reward weights throughout the denoising process, enabling stable convergence and improving sampling efficiency. Furthermore, to prevent drift during inference-time optimization, we incorporate a cleanlatent KL tether that anchors the sampling trajectory to the original latent representation. As shown in Figure 1, RewardFlow supports high-quality text-to-image generation and precise edits across diverse instruction types, including localized style, attributes, and text modifications. In summary, our contributions are: • We introduce RewardFlow, a training-free multi-rewardguided Langevin framework that integrates complemen-
tary differentiable signals to enable controllable, inversionfree editing and generation. Across multiple benchmarks, RewardFlow achieves state-of-the-art zero-shot performance in editing fidelity and compositional generation. • We design a novel prompt-aware adaptive policy that parses semantic primitives from the text instruction, infers intent, and dynamically modulates reward weights, providing coarse-to-fine efficient optimization. • We propose a novel differentiable VQA-based reward that provides fine-grained semantic supervision, ensuring accurate attribute changes and improved compositional alignment, alongside a SAM-guided reward that supports localized edits, penalizing leakage outside target regions. • Moreover, we provide a principled theoretical justification, showing that our update corresponds to a valid discretization of a Langevin SDE targeting a prompt-tilted density, establishing a sound foundation for stable reward-guided convergence under our framework.
2. Related Work Training-Based Methods. Early image generation and editing approaches rely on fine-tuning large diffusion [3] or GAN [13] models, often achieving high fidelity but at significant computational cost. DreamBooth [37] fine-tunes text-to-image models on a few subject images to bind unique identities, while Imagic [22] enables identity-preserving edits from a single image. Other approaches modify latent representations rather than model weights, e.g., StyleCLIP [31] manipulates StyleGAN [21] latents using CLIP [34] guidance, and Textual Inversion [10] introduces new token embeddings without retraining. Although training-based methods exhibit strong alignment, they require model updates and do not generalize well to unseen edits, making them impractical for interactive applications. Inference-Time Controllable Generation and Editing. Diffusion inversion seeks a noise latent whose denoising trajectory exactly reconstructs the source image. Recent methods such as Null-text inversion [28], Direct Inversion [19], and LEDITS++ [4] achieve faithful reconstructions but require costly forward–reverse passes and rely heavily on accurate inversion. A parallel line of work performs inferencetime editing without training or inversion. SDEdit [26] and PostEdit [39] perturb and denoise the input image with stochastic or posterior sampling, while FlowEdit [23], FreeFine [49], and Edicho [1] steer pre-trained models through ODE paths, attention control, or correspondence cues. Despite their speed and convenience, these methods lack a faithful latent reconstruction of the input image, which often leads to content drift, weak identity and layout preservation, hallucinated details, and limited fine-grained controllability. Reward-Guided Optimization. Recent work explores reward-based alignment of generative models. ReNO [9] optimizes latent trajectories based on multi-objective feed2
back, while ORIGEN [27] applies reward-guided Langevin sampling for zero-shot grounding. Recent formulations [6] cast this as trajectory optimal control. While effective for global alignment, existing methods lack localized reward models and adaptive control, often leading to drift, overediting, or weak spatial consistency. Our work introduces a unified Langevin dynamics framework that integrates a suite of coarse-to-fine rewards and a prompt-aware policy to adaptively and precisely steer generation.
(described in Section 3.2) predicts a time-dependent weight for each reward wi (tk ) and an adaptive step size ηk , which controls the magnitude of the update. We standardize each reward using its running mean µi and standard deviation σi , (k) (k) as R̄i = (Ri − µi )/(σi + ϵ) to ensure consistent scalP (k) (k) ing. We then define the fused reward Rtot = i wi (tk ) R̄i , whose gradient is mapped to the latent space exactly as above, yielding the total reward drift gRtot ,k . This mechanism naturally modulates the generation trajectory so that different objectives activate at appropriate times, for example, global semantic alignment early in the trajectory and fine spatial refinements later. The reverse-time update t = t̄ → 0 is z (k+1) = z (k) + ηk fk + gRtot ,k + gKL,k + ξk , (2) tk+1 = tk − ηk , ξk ∼ N 0, 2 γk ηk I ,
3. RewardFlow Method RewardFlow performs training-free, reward-guided generation by treating each denoising step as an instance of testtime optimization over a set of differentiable rewards. The key idea is to evaluate a set of rewards on the intermediate decoded image, combine them through a prompt-aware adaptive policy that dynamically sets time-varying reward weights and step sizes (Section 3.2), and map the resulting fused gradient back into the latent space through the decoder–denoiser chain rule. This produces a reward-guided drift term that augments the native flow-matching dynamics. Our method integrates a diverse set of heterogeneous rewards (Section 3.3), covering semantic, perceptual, regional, object-level, and QA-based, into a unified guidance signal that steers the sampling trajectory of a pretrained flowmatching model. An identity-preserving KL tether further anchors the generation to the source image to preserve identity and layout (Section 3.4).
where fk =vθ (z (k) , tk , p) is the backbone drift, gRtot ,k is the fused multi-head reward, and gKL,k is a clean-space KL tether (described in Section 3.4) that preserves identity. As we show in the supplementary material, this update corresponds to a valid discretization of a Langevin SDE that targets a prompt-tilted density, ensuring consistent convergence toward an image that satisfies the semantic, spatial, and structural constraints of the prompt.
3.2. Prompt-Aware Adaptive Policy A central contribution of our method is the adaptive policy that acts as a closed-loop controller for the Langevin sampler, and determines how strongly each reward should influence the trajectory and how aggressively the sampler should move at each step. Intuitively, different prompts require different forms of controllable generation, such as adding an object, removing a region, or changing a style attribute, each with distinct reward priorities and step sizes, and these requirements change over time as the image becomes closer to the target. To capture this, rather than relying on fixed inference schedules, the policy 1 extracts semantic primitives from p once before sampling, and then each step dynamically adjusts 2 reward weights wi and object direction sobj , and 3 the reward-aware step size (ηk ).
3.1. Multi-Reward Langevin-Based Generation Given a text prompt p and optional input image x, we obtain the initial clean latent z0 = Enc(x) by encoding the image (for image-conditioned cases) or sampling from noise (for unconditional text-to-image generation). We then initialize the forward trajectory z (0) = αt̄ z0 + σt̄ ε, with ε ∼ N (0, I) at a variance-preserving noise level t̄ and begin reverse-time sampling. At each step k with time tk , a flow-matching (rectified-flow) denoiser produces a clean latent z̃ (k) = Denθ (z (k) , tk , p) and its decoded image I (k) = Dec(z̃ (k) ). The decoded image I (k) is treated as an optimization variable evaluated by a set of differentiable rewards {Ri (I (k) , p)}. Each reward produces an image(k) space gradient gI,i = ∇I (k) Ri (I (k) , p), which is mapped into the latent-space drifts using the decoder JDec (·) and denoiser JDen (·) Jacobians, i.e., (k)
gRi ,k = λR JDen (z (k) , tk , p)⊤ JDec (z̃ (k) )⊤ gI,i .
1 Semantic Primitives (SP). As a one-time pre-process, before sampling, we parse the prompt p once using an LLM to extract Semantic Primitives SP(p) as a set of atomic, actionable concepts parsed from p. Each primitive corresponds to a self-contained generative objective, e.g., p =“Remove the cap from the person and add sunglasses” → SP(p) = {“Remove Cap”, “add sunglasses”}). SPs enables computing per-primitive perceptual, region, and object-level rewards, preventing interference between unrelated objectives and improving controllability. 2 Dynamic Reward Weighting. Different prompts require different reward strengths, and the optimal weighting changes over time as the generation progresses. At
(1)
Because different rewards should dominate at different moments in the generation process, we fuse the individual reward signals into a total reward that is adaptively weighted based on (i) the text prompt, (ii) the current denoising time, and (iii) the evolving generation state. A lightweight policy 3
[ Orange ---> Apple ] Juice
3 Reward-Aware Step Size (ηk ). Finally, the policy adapts (k) the step size ηk based on the current total reward Rtot . A high reward indicates we are close to the target, so we take smaller, more careful steps (refinement). A low reward indicates we are far, prompting larger steps (exploration). This is controlled by a logistic map (k) ηk = ηmin + (ηmax − ηmin ) · σ − γη (Rtot − r0 ) , (3)
[Red ---> Black] High heels
where σ(u) is the logistic function and r0 target threshold. Ghibli style portraits
Add Glasses
3.3. Differentiable Rewards
Figure 2. Gradient localization of our differentiable rewards. We visualize the image-space gradient ∇I Rtot (·) for various edit prompts. Our proposed rewards prevent semantic leakage by concentrating the gradient precisely on target semantic regions, demonstrating the fine-grained spatial control enabled by RewardFlow.
Relying on a single, global reward offers semantic alignment but lacks spatial precision; its gradients tend to diffuse across the entire image, leading to semantic leakage where unrelated regions are unintentionally modified. To counter this, we construct a hierarchical reward toolkit that provides fine-grained spatial, perceptual, and object-level control. We employ two levels of differentiable rewards: SP-level rewards, marked with ⊛ which are computed separately for each semantic primitive, and global prompt-level rewards, marked with ⟳, which operate over the entire input. Figure 2 illustrates how these rewards focus gradients on the intended edit region, preventing spillover. ⊛ Global and Perceptual Alignment (Rglb and Rper ). To ensure overall semantic correctness, we compute two complementary alignment rewards for each semantic SP: the global alignment reward that measures cosine similarity between image and text embeddings from SigLIP [40] encoders and the perceptual alignment reward that applies the same cosine formulation using Perception encoders. ⊛ Region-level Grounding (Rrg ). Global alignment alone cannot ensure that the modified or generated content occurs in the correct image region. To provide spatial specificity, we compute region–text relevance scores between region proposals and the SP phrases using RegionCLIP-style embeddings [48]. These scores are then softly pooled using a temperature-controlled attention mechanism, yielding a differentiable reward that encourages content changes to appear in spatial areas that are most relevant to the prompt. ⊛ Object Consistency (Roc ). Region grounding ensures localized generation in the right spatial region but does not guarantee that the correct object appears or disappears, or is modified as intended. To capture this object-level behavior, we employ text-conditioned SAM2 [35] to obtain soft masks Mj with confidence scores aj and mixture weights ωj = softmax(aj /τSP ) with τSP temperature. For each SP, we compute an object alignment score Fobj (Mj , gSP ), which measures the cosine similarity between the masked image and the SP, minus a small “leakage” penalty Pfor similarity outside the mask. The final reward Roc = j ωj Fobj (Mj , gSP ) evaluates whether the intended object semantics are correctly realized and spatially confined to the appropriate region.
each step, the policy therefore computes the reward weights wi using three sources of information. First, for a given SP and the full prompt, we classify the generation intent into three coarse categories (add, remove, or style) which broadly capture how content should evolve. These intent probabilities are fused into a single base profile prior τ = πadd τ (add) + πremove τ (remove) + πstyle τ (style) , where each reward template τ (·) encodes the characteristic importance pattern of the differentiable rewards for that type of intent. For example, if the given prompt is removal-heavy, this profile downweights region/object rewards. To avoid drift as (k−1) (k) sampling proceeds, we compute δi = max(0, Ri −Ri ). Moreover, we incorporate a lightweight schedule term hi (tk ) that nudges the policy to emphasize different reward families at different noise levels (e.g., localization early, semantics late). This prevents premature overfitting and stabilizes the trajectory. Because each prompt yields multiple SPs, weighting occurs in two stages. First, for each SP-level reward family, we compute τi , δi , and hi (tk ) for every SPspecific reward and apply a softmax across SPs to obtain one representative reward for that family. Second, we combine these four representative SP-level rewards with the global prompt-level rewards using wi (tk ) = softmaxi β (τi+κfb δi+ κsch hi (tk )) , where κfb and κsch , control the influence of the feedback and schedule terms, respectively, τi is the base profile prior, and β is the softmax temperature. Object Direction. Object-level changes are inherently directional: some prompts require introducing an object while others require removing or suppressing an object. A single object reward cannot distinguish between these two cases on its own, unless we explicitly encode direction. To handle this, the policy predicts a direction multiplier based on the intent classifier sobj = πadd −πremove , and the final reward ′ used per SP is Robj = sobj · Robj . A positive (add-intent) makes the sampler increase object presence (sobj ≈ +1), while a negative value (remove-intent) makes it decrease object presence (sobj ≈ −1). 4
Table 1. PIE-B ENCH image editing results. RewardFlow consistently improves edit fidelity and spatial localization across all metrics while maintaining competitive runtime. Best results are in bold and strong baselines are underlined. Method EF [18] ProxG [14] P2P [15] DI [19] AREdit [41] InfEdit [46] TurboEdit [44] InstantEdit [12] FlowEdit [23] FlowChef [32] KV-Edit [50] Flux+RewardFlow Qwen Image+RewardFlow InfEdit [46] TurboEdit [44] InstantEdit [12] Flux+RewardFlow Qwen Image+RewardFlow
Distance ↓ (×103 )
PSNR ↑
LPIPS ↓ (×103 )
MSE ↓ (×104 )
SSIM ↑ (×102 )
Whole ↑
Edited ↑
NFE ↓
Step ↓
8.39 8.39 9.58 11.60 13.12 13.87 15.11 12.57 11.56 9.67 8.47 7.78 7.64 16.19 18.57 17.14 13.44 10.33
27.49 28.45 27.72 27.25 28.78 28.63 26.04 29.63 28.33 29.03 29.04 31.21 32.09 26.75 24.59 27.96 29.57 29.92
44.38 38.27 44.98 49.25 42.67 39.80 69.54 35.27 43.57 43.11 43.44 40.55 38.47 50.79 77.53 44.39 40.03 37.92
29.79 25.63 30.02 32.87 37.88 33.19 55.12 24.57 37.48 36.67 35.47 26.47 23.57 42.33 58.48 34.94 29.66 26.95
85.61 85.87 85.01 84.86 87.19 86.28 84.27 87.40 86.23 87.44 86.66 89.67 90.21 84.71 82.64 86.44 88.13 90.12
25.87 25.04 24.94 25.83 26.57 25.84 26.09 26.06 26.43 27.05 28.21 29.44 29.78 25.68 25.70 26.28 27.31 28.55
22.14 21.64 21.57 22.39 24.51 22.44 23.35 22.73 23.03 23.09 23.92 26.62 27.57 22.27 22.30 22.82 24.69 27.52
70 100 100 100 16 12 24 24 33 28 27 43 54 4 4 8 6 6
50 50 50 50 30 12 12 12 30 30 40 20 25 4 4 4 4 4
⟳ Human Preference Alignment (Rhps ). This reward is defined as the normalized scalar output of a differentiable predictor HHPS (I (k) , p) (e.g., HPS v2 [43]) measuring image prompt consistency: Rhps I (k) , p = norm HHPS (I (k) , p) .
This regularizer moderates aggressive reward-driven updates, preventing drift, and anchoring the generation around the source identity and spatial layout.
⟳ VQA Reward (Rvqa ). For fine-grained semantic correctness, we construct a QA pair (q, a⋆ ) from the prompt and evaluate it with a frozen language model. Given token logits ℓt with pt = softmax(ℓt ), the reward is the negated lengthnormalized cross-entropy plus margin objective:
4. Experiments
Rvqa = −T1
T h P t=1
i ℓt [u] . log pt [a⋆t ]+λm max 0, m−ℓt [a⋆t ]+max ⋆ u̸=at
We evaluate RewardFlow using state-of-the-art diffusion backbones, Flux [2], Qwen [42], and PixArt-α [7] on two established benchmarks: PIE-B ENCH for image editing [20] and T2I-C OMP B ENCH for compositional generation [17]. Additional details are provided in the Appendix.
(4)
4.1. Image Editing Results
To the best of our knowledge, this is the first work to integrate a differentiable VQA-based reward into inference-time controllable image generation and editing.
Quantitative Results. We report quantitative editing performance in Table 1, comparing RewardFlow to a broad suite of open-source, training-free editing methods on PIE-Bench. All diffusion-based baselines use the same Flux backbone for a controlled comparison, with TurboEdit (SDXL) and AREdit (autoregressive) as exceptions. Under this shared backbone, Flux+RewardFlow achieves consistent state-ofthe-art performance. Relative to the strongest prior Fluxbased baseline, Flux+RewardFlow reduces Distance by 7.3% (7.78 vs. 8.39) while keeping LPIPS within 6.0% of the best Flux-based method (40.55 vs. 38.27), indicating better preservation of background and identity at comparable perceptual similarity. RewardFlow also improves PSNR by 5.3% (31.21 vs. 29.63) and SSIM by 2.6% (89.67 vs. 87.44), yielding sharper and more structurally consistent reconstructions. For edit alignment, RewardFlow increases Whole accuracy by 4.4% (28.21→29.44) and Edited accuracy by 8.6% (24.51→26.62), outperforming all Flux-based editors. These fidelity gains are obtained with competitive efficiency
3.4. Identity-Preserving KL Tether Strong reward guidance can cause the sampler to “chase” high reward values at the expense of the input’s identity, producing drift, layout distortion, or reward hacking. To prevent this, we introduce an identity-preserving KL tether that softly pulls the predicted clean latent z̃ (k) back toward the original latent representation z0 . Conceptually, this term corresponds to the gradient of a Kullback–Leibler divergence between the current clean-prediction distribution q(z̃|z (k) ) and a reference Gaussian prior centered at the input latent z0 . Minimizing this KL encourages the clean prediction to remain close to the input’s content and structure. Taking the derivative of this KL with respect to latent z (k) yields ⊤ gKL,k = −λKL JFMDenoise z̃ (k) − z0 (5) 5
Input Img
InfEdit
FlowEdit
FlowChef
InstantEdit
KV-Edit
Change the perspective of this sculpture from the front to the right side of viewer
Replace the surfboards with flowers
Change the animal from tiger to brown cat
Change the fruits to pizza
Make the parrots kiss each other
Change the forest to a wizard house Figure 3. Image editing qualitative comparison across diverse instruction types. RewardFlow produces edits that are both semantically accurate and spatially localized, while better preserving background structure, lighting, and identity compared to prior methods.
(43 NFEs and 20 sampling steps), corresponding to roughly 60–80% fewer sampling steps than gradient-based editors that typically require 50–100 steps. In the few-step setting (4 sampling steps), RewardFlow with Flux and Qwen Image further improves over prior fast editors (InstantEdit and TurboEdit), reducing Distance by up to 44.4% and LPIPS by up to 25.8%, while increasing Whole and Edited accuracies by up to 11.1% and 23.4%, respectively.
of the sculpture, RewardFlow changes the perspective without introducing major distortions compared to baselines. For object replacement tasks, such as replacing the surfboards with flowers or changing the fruits to pizza, competing methods often under-edit, over-edit, or generate implausible replacements, whereas RewardFlow performs the intended substitution cleanly while maintaining the surrounding layout and appearance. Similarly, in the animal replacement example, baselines such as InfEdit and InstantEdit fail to replace the tiger or cannot adhere to the specified attributes, while RewardFlow generates the brown cat with coherent
Qualitative Results. Figure 3 shows that RewardFlow produces edits that are both more instruction-faithful and spatially precise than baselines. In the viewpoint transformation 6
Table 2. T2I compositional generation on T2I-C OMP B ENCH. Accuracy across fine-grained attribute binding (color, shape, texture), object relationships (spatial and non-spatial), and complex compositions. RewardFlow consistently improves all base models (PixArt-α, Flux, and Qwen Image). Best results are in bold. Model
Color
Shape
Texture
Spatial
Non-Spatial
Complex
SD v1.4 [36] SD v2.1 [36] SDXL [33] PixArt-α [7] DALL-E 2 [29] DALL-E 3 [30]
0.38 0.51 0.64 0.69 0.57 0.81
0.36 0.42 0.54 0.56 0.55 0.68
0.42 0.49 0.56 0.70 0.64 0.81
0.12 0.13 0.20 0.21 0.13 –
0.31 0.31 0.31 0.32 0.30 –
0.31 0.34 0.41 0.41 0.37 –
(1) PixArt-α DMD [47] (1) + ReNO [9] (1) + RewardFlow
0.38 0.64 0.74
0.34 0.57 0.66
0.47 0.72 0.75
0.19 0.25 0.30
0.30 0.31 0.39
0.36 0.46 0.52
(2) Flux [2] (2) + ReNO [9] (2) + RewardFlow
0.75 0.81 0.88
0.61 0.64 0.69
0.69 0.72 0.78
0.26 0.29 0.33
0.33 0.35 0.42
0.47 0.49 0.57
(3) Qwen Image [42] (3) + ReNO [9] (3) + RewardFlow
0.83 0.84 0.91
0.72 0.75 0.83
0.80 0.84 0.90
0.35 0.36 0.39
0.39 0.43 0.51
0.61 0.63 0.78
Flux
Flux + ReNO
Flux+
A dramatic close-up portrait of a young woman standing in soft rain at night, neon city lights reflecting in the water on her skin
A legendary dark fantasy knight standing on a battlefield of ashes, black armor with glowing red cracks, tattered cape moving in the wind, giant ruined castle in the background
Table 3. Ablation on key RewardFlow components. Setting / Variant ✗ Dynamic Reward Weighting. ✗ Semantic Primitives (SPs) ✗ Reward-Aware Step Size ✗ KL-Tether RewardFlow (Full)
Distance ↓ (×103 )
PSNR ↑
LPIPS ↓ (×103 )
MSE ↓ (×104 )
SSIM ↑ (×102 )
Whole ↑
Edited ↑
8.47 9.03 9.15 9.56 7.64
30.77 31.19 31.38 29.98 32.09
38.78 39.12 39.56 40.26 38.47
34.67 35.01 35.21 35.28 33.57
89.37 88.47 89.12 87.23 90.21
29.02 27.45 28.32 27.98 29.78
27.01 26.51 26.92 26.13 27.57
A transparent glass bowl containing three striped marbles: one blue-and-white, one red-andyellow, and one green-and-black
shape, color, and pose. In relational or compositional edits, such as making the parrots kiss or changing the forest into a wizard house, RewardFlow again yields outputs that better reflect the target semantics without the leakage, structural drift, or background corruption observed in other methods.
Figure 4. Text-to-image qualitative results. Across all prompts, RewardFlow produces images that exhibit higher alignment with the textual descriptions while also generating outputs with more visually appealing composition and aesthetics.
4.2. Image Generation Results Quantitative Results. Table 2 reports text-to-image generation performance on T2I-C OMP B ENCH across six categories spanning fine-grained attribute binding, object relationships, and complex multi-constraint prompts. Across all three base models (PixArt-α, Flux, and Qwen Image), RewardFlow consistently improves composition accuracy and outperforms the training-free reward-based baseline ReNO across every category. For Flux, RewardFlow improves overall performance by approximately 12.5%, while for Qwen Image, the improvement reaches 12.8%, with particularly strong boosts in non-spatial and complex compositional categories. These results underscore RewardFlow’s effectiveness in enhancing attribute binding and relational coherence across diverse generation backbones. Qualitative Results. Figure 4 compares examples generated from ReNO [9] and RewardFlow with Qwen Image backbone. Across diverse prompts, RewardFlow produces images with stronger semantic alignment and improved aesthetic quality. Compared to the base model and ReNO, our method consistently enhances color vibrancy, local detail, and prompt adherence. In the portrait example, RewardFlow better captures the neon reflections and wet-skin appearance, while in the fantasy knight example, RewardFlow generates a more coherent dark atmosphere, sharper armor details,
A notebook pen and open Laptop on wooden Table beside it Coffee cup and Mobile
Llama standing in the grass field
Figure 5. Gradient localization across reward combinations. Including all rewards concentrates gradients to accurate object contours and eliminates leakage.
and the ruined castle that other methods fail to realize. In the striped marbles example, RewardFlow more faithfully satisfies the specified object count and patterns.
5. Ablation Studies Reward Components. We analyze both quantitative and qualitative effects of each reward component and visualize their gradient localization behavior. As shown in Figure 5 and Table 4, with only the global alignment reward (Rglb , first column), gradients are broadly distributed, yielding weak spatial focus (Distance 11.23, SSIM 84.09). Including the perceptual reward (Rper ) sharpens local structure (LPIPS 45.22 → 43.12), slightly improving focus around relevant regions. The human preference reward (Rhps ) improves realism and global coherence (PSNR 27.57 → 28.82), reducing 7
Src. Image
✘
✘
✘
Src. Image
✘
✘Dynamic Steps
✘ SP
✘ DW
✘ KL-Penalty
Convert to Anime Portrait
Two [white -> wooden] lanterns
Change goat colour to yellow
Disney Pixar style Replace the bench’s material with marble
Outer [black -> blue] blazer Add a coffee cup on the table in the foreground
A bar stool
Figure 7. Ablations illustrating the effect of removing key components. RewardFlow (all components) achieves the best visual consistency and instruction alignment.
Transfer the image into a dramatic charcoal-drawing style
Table 4. Ablation on reward components. Each column indicates whether the corresponding reward is enabled (✓) or disabled (✗).
A dog with a blue house in the background
Figure 6. Effect of removing reward components. (✗ RC, SAM, LLM, HPS, and PE) denote excluding (Rrg , Roc , Rvqa , Rhps , and Rper ), respectively. RewardFlow (all rewards) achieves semantically precise edits, modifying only instruction-relevant content while maintaining background and context integrity.
Rglb Rper Rhps Roc Rrg Rvqa ✓ ✓ ✓ ✓ ✓ ✓
perceptual noise and producing smoother tones in tasks such as “marble bench” and portrait stylization (Figure 6). The object consistency reward (Roc ) improves gradient concentration around intended objects (Figure 5), reducing leakage and improving spatial precision (Distance 9.77 → 8.39). This effect is clearly seen in localized edits such as “yellow goat” and “blue house dog” (Figure 6). Adding the regionlevel grounding reward (Rrg ) further localizes gradients to prompt-relevant regions (PSNR 29.44 → 30.12, Whole 21.47 → 26.47), ensuring edits such as the “coffee cup” placement occur in the correct area without disturbing context (Figure 6). Finally, the VQA reward (Rvqa ) provides the strongest fine-grained supervision (PSNR 32.09, SSIM 90.21), producing sharply focused gradient activations that align with object contours and yield semantically precise results across all tasks. Overall, global rewards (Rglb , Rper , Rhps ) ensure semantic and perceptual coherence, while localized rewards (Roc , Rrg , Rvqa ) progressively concentrate gradients to target regions, achieving precise instruction-faithful edits.
✗ ✓ ✓ ✓ ✓ ✓
✗ ✗ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✓
Distance ↓ LPIPS ↓ MSE ↓ SSIM ↑ PSNR ↑ Whole ↑ Edited ↑ (×103 ) (×103 ) (×104 ) (×102 ) 11.23 10.21 9.77 8.39 8.01 7.64
26.33 27.57 28.82 29.44 30.12 32.09
45.22 43.12 41.67 40.58 40.02 38.47
39.34 37.45 37.12 36.55 35.28 33.57
84.09 85.17 86.39 87.71 88.92 90.21
19.33 19.88 20.12 21.47 26.47 29.78
21.22 22.85 23.33 24.75 25.91 27.57
Without semantic primitives (SPs), Distance rises to 9.03 and Whole drops by 2.33, leading to interference between objectives and inconsistent stylization, as seen in the “disney pixar style” example. Fixing the step size worsens Distance (9.15) and PSNR (31.38), producing unstable updates such as uneven recoloring in the “outer [black→blue] blazer” edit. Excluding the KL tether causes the most severe degradation (PSNR -2.11, SSIM -1.89) and structural drift, exemplified by distortions in the bar stool geometry. Overall, SPs enable disentangled control, dynamic weighting maintains balanced optimization, adaptive steps ensure stable convergence, and the KL tether preserves structural fidelity, collectively supporting coherent and precise controllable generation.
6. Conclusion We introduce RewardFlow, a training-free framework that steers pretrained text-guided image editing and generation models using multi-reward Langevin dynamics. By combining global, localized, and VQA-based rewards with a prompt-aware adaptive policy and a KL tether, RewardFlow achieves fine-grained, spatially precise control while preserving identity and layout. Extensive experiments demonstrate consistent improvements in edit fidelity, compositional alignment, and generation quality over strong training-free baselines. We believe treating controllable generation as reward-guided sampling offers a general test-time alignment strategy, with promising extensions to video editing.
RewardFlow Method Components. In Table 3 and Figure 7, we ablate each component of our adaptive policy to assess its contribution. The full model achieves the best results (Distance 7.64, PSNR 32.09, SSIM 90.21), confirming the effectiveness of jointly adaptive control. Removing dynamic reward weighting reduces PSNR by 1.32 and SSIM by 0.84, as fixed weights fail to adapt to evolving reward satisfaction—evident in the “two [white→wooden] lanterns” edit, where color shifts occur but texture consistency degrades. 8
Acknowledgments
[10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2022. 2 [11] Chenjian Gao, Lihe Ding, Xin Cai, Zhanpeng Huang, Zibin Wang, and Tianfan Xue. Controllable first-frame-guided video editing via mask-aware lora fine-tuning. In International Conference on Learning Representations (ICLR), 2025. 2 [12] Yiming Gong, Zhen Zhu, and Minjia Zhang. Instantedit: Text-guided few-step image editing with piecewise rectified flow. In International Conference on Computer Vision (ICCV), 2025. 2, 5 [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020. 2 [14] Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-free real image editing with proximal guidance. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024. 5 [15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-Prompt Image Editing with Cross Attention Control. In ACM SIGGRAPH Asia, 2022. 2, 5 [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020. 1 [17] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for openworld compositional text-to-image generation. Advances in Neural Information Processing Systems (NeurIPS), 2023. 5 [18] Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5 [19] Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct Inversion: Boosting Diffusion-Based Editing with 3 Lines of Code. arXiv preprint arXiv:2310.01506, 2023. 2, 5 [20] Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. In International Conference on Learning Representations (ICLR), 2024. 5 [21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2 [22] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2 [23] Vladimir Kulikov, Matan Kleiner, Inbar HubermanSpiegelglas, and Tomer Michaeli. FlowEdit: Inversion-Free
This research was partially supported by Google, the Google TPU Research Cloud (TRC) program, the U.S. Defense Advanced Research Projects Agency (DARPA) under award HR001125C0303, and the U.S. Army under contract W5170125CA160. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of Google, DARPA, the U.S. Army, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References [1] Qingyan Bai, Hao Ouyang, Yinghao Xu, Qiuyu Wang, Ceyuan Yang, Ka Leong Cheng, Yujun Shen, and Qifeng Chen. Edicho: Consistent image editing in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. 2 [2] Black Forest Labs. FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space. arXiv preprint arXiv:2506.15742, 2025. 1, 5, 7 [3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2 [4] Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos. Ledits++: Limitless image editing using text-to-image models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2 [5] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2 [6] Jinho Chang, Jaemin Kim, and Jong Chul Ye. Training-free reward-guided image editing via trajectory optimal control. arXiv preprint arXiv:2509.25845, 2025. 3 [7] Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations (ICLR), 2024. 5, 7 [8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024. 1 [9] Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimization. Advances in Neural Information Processing Systems (NeurIPS), 2024. 2, 7
9
Text-Based Editing Using Pre-Trained Flow Models. In International Conference on Computer Vision (ICCV), 2025. 2, 5 [24] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2022. 1 [25] Jingyi Lu, Xinghui Li, and Kai Han. Regiondrag: Fast regionbased image editing with diffusion models. In European Conference on Computer Vision (ECCV), 2024. 2 [26] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021. 2 [27] Yunhong Min, Daehyeon Choi, Kyeongmin Yeo, Jihyun Lee, and Minhyuk Sung. Origen: Zero-shot 3d orientation grounding in text-to-image generation. arXiv preprint arXiv:2503.22194, 2025. 2, 3 [28] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2 [29] OpenAI. Introducing dall-e 2. OpenAI Blog (2022). https: //openai.com/index/dall-e-2/, 2022. 7 [30] OpenAI. Introducing dall-e 3. OpenAI Blog (2023). https: //openai.com/dall-e-3, 2023. 7 [31] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In International Conference on Computer Vision (ICCV), 2021. 2 [32] Maitreya Patel, Song Wen, Dimitris N. Metaxas, and Yezhou Yang. FlowChef: Steering of Rectified Flow Models for Controlled Generations. In International Conference on Computer Vision (ICCV), 2025. 5 [33] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations (ICLR), 2023. 7 [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. 2 [35] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In The Thirteenth International Conference on Learning Representations, 2024. 2, 4 [36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 7 [37] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine
tuning text-to-image diffusion models for subject-driven generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2 [38] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020. 1 [39] Feng Tian, Yixuan Li, Yichao Yan, Shanyan Guan, Yanhao Ge, and Xiaokang Yang. Postedit: Posterior sampling for efficient zero-shot image editing. In The Thirteenth International Conference on Learning Representations, 2024. 2 [40] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 4 [41] Yufei Wang, Lanqing Guo, Zhihao Li, Jiaxing Huang, Pichao Wang, Bihan Wen, and Jian Wang. Training-free text-guided image editing with visual autoregressive model. In International Conference on Computer Vision (ICCV), 2025. 5 [42] Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324, 2025. 5, 7 [43] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv preprint arXiv:2306.09341, 2023. 5 [44] Zongze Wu, Nicholas Kolkin, Jonathan Brandt, Richard Zhang, and Eli Shechtman. TurboEdit: Instant Text-Based Image Editing Using Few-Step Diffusion Models. In European Conference on Computer Vision (ECCV), 2024. 5 [45] Xin Xie and Dong Gong. Dymo: Training-free diffusion model alignment with dynamic multi-objective scheduling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2 [46] Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-Free Image Editing with Natural Language. arXiv preprint arXiv:2312.04965, 2023. 5 [47] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 7 [48] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. RegionCLIP: RegionBased Language-Image Pretraining. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4 [49] Hanshen Zhu, Zhen Zhu, Kaile Zhang, Yiming Gong, Yuliang Liu, and Xiang Bai. Training-free geometric image editing on diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. 2
10
[50] Tianrui Zhu, Shiyi Zhang, Jiawei Shao, and Yansong Tang. KV-Edit: Training-Free Image Editing for Precise Background Preservation. In International Conference on Computer Vision (ICCV), 2025. 5
11
: Generate Images by Optimizing What You Reward Supplementary Material
Figure 8. High-resolution images generated by RewardFlow.
1
7. SDE Formulation
mation vθ (z, t, p) ≈ ∇z log qt (z | p) we obtain ∇z log ρt (z | p, x) ≈ vθ (z, t, p) + gRtot (z, t, p)
In this section, we detail the stochastic differential equation (SDE) that grounds the Langevin-style reverse update in Eq. (2), specify the diffusion-strength schedule γk , and provide a derivation showing how Eq. (2) emerges from sampling a prompt-tilted latent density. Prompt-tilted target density. Let qt (z | p) denote the unconditional latent distribution at time t for prompt p. Given total reward Rtot (z, t, p) obtained by combining the differentiable rewards and KL potential 2
K(z, t; z0 ) := 12 z̃(z, t, p) − z0 2 ,
+ gKL (z, t; z0 ).
Evaluating this at (z (k) , tk ) in Eq. (10) yields z (k+1) = z (k) + ηk fk + gRtot,k + gKL,k + ξk , (12) ξk ∼ N 0, 2γk ηk I , where fk := vθ (z (k) , tk , p) is the backbone drift. This is exactly the stochastic update stated in Eq. (2) of the main paper, now seen as an Euler–Maruyama discretization of the Langevin SDE in Eq. (10) targeting the prompt-tilted density in Eq. (7), with the reward terms defining the controllability potential and the KL tether stabilizing identity and layout.
(6)
we define the prompt-tilted target density ρt (z | p, x) ∝ qt (z | p) exp λR Rtot (z, t, p)−λKL K(z, t; z0 ) .
(7)
where z0 = Enc(x) denotes the clean latent of the (optional) source image x. Taking the gradient of the log-density in Eq. (7) yields
7.2. Noise variance schedule γk The Gaussian perturbation in Eq. (2) is parameterized by the time-dependent variance γk . We instantiate γk with a monotonically decreasing schedule tk ρ γk = γmin + γmax − γmin , t̄ (13) where γmin , γmax > 0, ρ > 0,
∇z log ρt (z | p, x) = ∇z log qt (z | p)
+ λR ∇z Rtot (z, t, p)
− λKL ∇z K(z, t; z0 ). Each reward Ri (I, p) is defined in image space. For (k) step k, let gI,i := ∇I Ri (I (k) , p) denote the image-space gradient. Using the decoder and denoiser Jacobians, the reward drift in Eq. (1) can be written as (k)
gRi ,k = λR JDen (z (k) , tk , p)⊤ JDec (z̃ (k) )⊤ gI,i .
(11)
so that early steps at high noise levels (tk ≈ t̄) use larger diffusion (exploration), while late steps near tk ≈ 0 use smaller diffusion, focusing on refinement. In the special case γmin = γmax , Eq. (13) reduces to a constant-noise Langevin sampler.
(8)
Summing over rewards then yields the fused reward drift gRtot,k = λR ∇z(k) Rtot (z (k) , tk , p). In addition, differentiating K(z (k) , tk ; z0 ) with respect to z (k) yields Eq. (5), so gKL,k = −λKL ∇z(k) K(z (k) , tk ; z0 ).
8. Datasets and Evaluation To ensure a fair comparison, we adopt the same evaluation protocols and metrics as defined in the original papers of each respective dataset. T2I-C OMP B ENCH . T2I-C OMP B ENCH is a large-scale benchmark designed to evaluate compositional text-to-image generation in open-world settings, and consists of approximately 6,000 prompts categorized into three key tasks: attribute binding, object relationships, and complex compositions. Each prompt describes scenes with multiple objects and attributes, requiring precise alignment between textual semantics and visual structure. For evaluation, the benchmark employs a set of compositional metrics. Attribute binding is assessed using BLIP-based VQA that queries each object’s attribute independently (e.g., “What color is the bench?”). Spatial relations are evaluated using UniDet, a pre-trained object detector, to check the relative positioning of objects via bounding box analysis. For complex scenes, a compositional consistency score is computed by aggregating CLIPScore, BLIP-VQA accuracy, and UniDet spatial relation correctness. This framework enables a detailed understanding of how well models handle fine-grained compositional constraints beyond conventional image-text similarity.
7.1. Langevin SDE and Discrete Update We introduce an algorithmic time variable s ∈ [0, S] and a monotone schedule t(s) from algorithmic time to diffusion time, with t(0) = t̄ and t(S) = 0. We consider the overdamped Langevin SDE whose stationary distribution at each t is the prompt-tilted density ρt : p dzs = ∇z log ρt(s) zs | p, x ds + 2 γ(s) dWs , (9) where Ws is standard Brownian motion and γ(s) > 0 controls the diffusion strength. Let s0 < s1 < · · · < sK be a discretization of [0, S] with step sizes ηk = sk+1 − sk , and write tk = t(sk ), γk = γ(sk ), and z (k) ≈ zsk . Applying Euler–Maruyama to Eq. (9) yields z (k+1) = z (k) + ηk ∇z log ρtk z (k) | p, x (10) p ξk ∼ N (0, I). + 2 γk ηk ξk , Substituting Eq. (7) and using flow-matching score approxi2
in Figure 9. Unless otherwise stated, we use the same hyperparameters across all backbones, datasets, and tasks. All experiments are run on a single node with 2× NVIDIA A100 GPUs (80 GB each). We implement RewardFlow in PyTorch with automatic mixed precision (AMP) for all backbones and reward networks, which reduces memory footprint and latency without affecting visual quality. Unless otherwise noted, we use a batch size of 1 per GPU for editing experiments and 2 for text-to-image generation. Backbones and Resolution. We instantiate RewardFlow on three pretrained flow-matching / diffusion backbones: PixArt-α, Flux, and a Qwen-based latent diffusion model. All images are generated and edited at 1024 × 1024 resolution. We use the official checkpoints and sampling schedules for each backbone and do not fine-tune any model weights; RewardFlow operates purely at inference time. For image editing, given a prompt p and source image x, we encode the image into a clean latent z0 = Enc(x), initialize a noisy latent z (0) at a fixed noise level t̄ as in the backbone, and run K = 35 reverse steps following the update in Eq. (2). For unconditional text-to-image generation, z0 is sampled from the backbone’s prior and the KL tether is disabled (λKL = 0). Prompt Parsing and Semantic Primitives. Before sampling, we parse each prompt p once using GPT-5 to extract: • A set of Semantic Primitives SP(p) = {pm }M m=1 , where each pm is a short, atomic instruction (e.g., “remove cap”, “add sunglasses”, etc.). Jvqa • A small set of VQA pairs {(qj , a⋆j )}j=1 that probe finegrained aspects of the intended edit (e.g., “What is on the person’s head?” → “Nothing”). As shown in the prompt template in Figure 10, we instruct the model to ensure that each SP is self-contained and that the VQA questions are answerable from the final image without ambiguity. This one-time parsing step is performed offline and cached for all subsequent sampling runs with the same prompt. For multi-instruction prompts, SPs prevent interference between unrelated objectives and enable perprimitive reward computation. Rewards and Feature Extractors. At every denoising step, each reward is evaluated on I (k) and the corresponding SPs, producing both a scalar score and an image-space gradient. We briefly summarize implementation choices for each. Global and perceptual rewards (Rglb , Rper ). For the global semantic reward Rglb we use a SigLIP-style vision– sig language model ϕsig img , ϕtext and compute cosine similarity between the image and each SP: (k) Rglb (I (k) , p) = cos ϕsig ), ϕsig text (p) . img (I
Figure 9. Overview of the RewardFlow framework.
G EN E VAL . G EN E VAL is a structured evaluation suite targeting fine-grained text-to-image alignment at the object level. It introduces prompts designed to probe a model’s ability to generate images with correct object presence, cooccurrence, counting, spatial arrangement, and color attribution. Each generated image is evaluated using automated pipelines based on pre-trained vision models. Object detectors verify the existence and number of instances for specified entities, while spatial metrics assess whether objects appear in the correct geometric configuration (e.g., left/right or above/below). Color attributes are checked by segmenting object regions and comparing predicted colors with prompt specifications. Each task yields binary correctness judgments, and the results are reported as per-category accuracies along with an overall compositional accuracy score. GenEval has been shown to correlate strongly with human judgments and helps isolate specific failure modes such as incorrect object counts or attribute swaps. PIE-B ENCH . PIE-B ENCH is a comprehensive benchmark for evaluating text-guided image editing systems. It comprises 700 real-world and artistic images, each paired with a source prompt, a target prompt, a natural language editing instruction, and a binary editing mask. The edits are drawn from ten categories, including object addition, removal, replacement, attribute changes (e.g., color, pose), material substitution, background edits, and global style transformations. The benchmark evaluates two core criteria: (1) Edit Fidelity, which measures how well the edited image aligns with the target prompt, typically using CLIPScore or similar semantic similarity metrics; and (2) Content Preservation, which assesses how much of the non-edited image content remains unchanged, computed via PSNR or SSIM on unmasked (nonedit) regions. PIE-Bench allows for quantitative and targeted assessment of how effectively models perform localized or global edits while preserving image realism and structure.
For the perceptual reward Rper we employ a Perception Enper coder ϕper img , ϕtext and cosine similarity. Prompt-level scores Rglb (I (k) , p) and Rper (I (k) , p) are obtained by aggregating over SPs (uniform averaging modulated by the policy). Region grounding reward (Rrg ). Region-level grounding
9. Implementation Details In this section, we provide additional implementation details for RewardFlow. An overview of the method is illustrated 3
Vision-language Editing Assistant. You are a vision-language assistant. You receive an image and a short edit instruction. 1) Extract short edit prompts: output a compact list of 5–12 atomic, actionable tags/phrases that guide the image edit. Include: • Visible subject descriptors (pose, angle, clothing items) actually present. • The edit action(s) and key visual attributes (style, color, size, placement). • Constraints to preserve identity, lighting, composition, realism, and continuity. • Any practical rendering notes (alignment, shadows, reflections, edges). 2) Create exactly one Q&A pair focused on the final edited image’s appearance. • Ask **one** question that would most affect the final look (e.g., style, colorway, size/scale, placement, material/finish, mood/lighting continuity). • Give **one** concise answer based on the image/instruction; if not determinable, answer "Unspecified from image." ## Rules - **Output JSON only** in the exact schema below—no extra text. - Keep each short prompt <= 6 words; imperative, neutral wording. - Do not invent details not visible or implied by the instruction. - Avoid sensitive inferences (e.g., ethnicity, health, etc.). - American English. ## Input EDIT_INSTRUCTION: {edit_instruction} ## Output schema (JSON only) { "short_prompts": ["<tag1>", "<tag2>", "..."], "qna": { "question": "<visual-outcome question>", "answer": "<concise answer or ’Unspecified from image’>" } }
Figure 10. Prompt template used for semantic primitives and Rvqa . reg uses RegionCLIP image-region ψimg (I, rm ) and text embedreg dings ψtext (p). Given region proposals {rm } we compute sm (p) and soft attention weights αm (p), and define X Rrg (I (k) , p) = αm (p) sm (p).
in the background. The object reward for SP p is X Roc (I (k) , p) = ωj Fobj (Mj , gSP (p)), j
and is further modulated by the add/remove intent scalar sobj ∈ [−1, 1] predicted by the adaptive policy. Human Preference Reward (Rhps ). For Rhps , we use HPSv2, a pretrained human preference scorer that takes (I (k) , p) as input and outputs a scalar score. We normalize this score with a fixed running mean and variance so that it is numerically comparable to the other rewards and can be combined without further scaling: Rhps (I (k) , p) = norm HHPS (I (k) , p) .
m
This reward encourages gradients to concentrate on spatial regions that are both semantically and visually aligned with each SP, matching the behavior illustrated in Figure 9. Object consistency reward (Roc ). For object-level localization, we use text-guided SAM2 [35] (Florence-SAM21 ) to obtain soft masks {Mj } and confidences {aj } for each semantic primitive. For each SP p, we query SAM2 with the text description and optional point prompts derived from its coarse localization (e.g., from the region-level gradients), yielding soft foreground masks Mj and their confidences aj . Mixture weights ωj are formed via a softmax over aj . We compute an object alignment score Fobj (Mj , gSP (p)) that rewards correct semantics in the mask and penalizes leakage
In practice, Rhps is evaluated on the full prompt and primarily stabilizes overall aesthetic quality and prompt adherence. VQA reward (Rvqa ). For Rvqa we use the Qwen-2.5-VL 3B model, accessed via the HuggingFace Transformers interface. For each Q&A pair (q, a⋆ ) produced by ChatGPT, we feed (I (k) , p) into Qwen-2.5-VL and obtain the token-level logits ⋆ ⋆ {ℓt }Tt=1 for the answer sequence a⋆−(a⋆t )Tt=1 . We then form the VQA reward from these logits. In practice, we cap T ⋆ to a reasonable answer length (e.g., T ⋆ ≤ 70 tokens).
1 https : / / huggingface . co / spaces / SkalskiP / florence - sam / blob / main / checkpoints / sam2 _ hiera _ large.pt
4
Table 5. T2I generation on G EN E VAL. Overall ↑ Single ↑ Two ↑ Counting ↑ Colors ↑ Position ↑ Color Attribution ↑
Model
Figure 11. Reward progression over time.
SD v2.1 SDXL IF-XL PixArt-α DALL-E 2 DALL-E 3 SD3 (8B)
0.50 0.55 0.61 0.48 0.52 0.67 0.68
0.98 0.98 0.97 0.98 0.94 0.96 0.98
0.51 0.74 0.74 0.50 0.66 0.87 0.84
0.44 0.39 0.66 0.44 0.49 0.47 0.66
0.85 0.85 0.81 0.80 0.77 0.83 0.74
0.07 0.15 0.13 0.08 0.10 0.43 0.40
0.17 0.23 0.35 0.07 0.19 0.45 0.43
(1) PixArt-α DMD (1) + ReNO (1) + RewardFlow
0.45 0.59 0.65
0.95 0.98 0.99
0.38 0.72 0.77
0.46 0.58 0.65
0.76 0.85 0.89
0.05 0.15 0.21
0.09 0.27 0.33
(2) Flux (2) + ReNO (2) + RewardFlow
0.64 0.72 0.81
0.98 0.99 0.99
0.80 0.90 0.97
0.64 0.79 0.90
0.78 0.87 0.95
0.18 0.21 0.39
0.43 0.56 0.72
(5) Qwen (5) + ReNO (5) + RewardFlow
0.83 0.85 0.91
0.99 0.99 0.99
0.98 0.98 0.99
0.92 0.94 0.97
0.92 0.95 0.98
0.27 0.35 0.47
0.71 0.75 0.84
Table 6. VLM Comparison on PIE-B ENCH.
Cosine similarity backbone. All rewards except Rhps and Rvqa are implemented as cosine similarities between the embeddings of semantic primitives and the current image at step k. Gradients are obtained via automatic differentiation through the corresponding vision–language encoders. KL Tether for Image Editing. For all image editing experiments, we enable the clean-latent KL tether gKL,k from Eq. (5). The tether is computed in the clean latent space z̃ (k) and back-propagated through Denθ using its Jacobian JDen . We keep the KL strength λKL = 1.5 fixed across steps and applied only when a source image is provided. For pure text-to-image generation, we set λKL = 0 so that the sampler targets the prompt-tilted distribution without anchoring to a particular source latent. Editing vs. Generation Configurations. For text-toimage experiments, we use all rewards except the objectconsistency reward Roc , which is less relevant in the absence of a reference layout. For image editing, we enable the full set of rewards {Rglb , Rper , Roc , Rrg , Rhps , Rvqa }, dynamic reward weighting, reward-aware step sizes, and the KL tether. As shown in Figure 11, all reward components in RewardFlow exhibit consistent and stable improvement over the course of sampling. Starting from an initial value of −1, the global semantics reward Rglb , perceptual reward Rper , region grounding reward Rrg , object consistency reward Roc , human-preference reward Rhps , and VQA reward Rvqa all trend upward with natural fluctuations, eventually converging to high positive values. The smooth yet spiky trajectories indicate that the system is actively exploring while steadily refining the sample quality under each objective, rather than overfitting to any single reward. Taken together, these qualitative dynamics demonstrate that RewardFlow effectively coordinates and optimizes all reward signals, confirming that the full reward pipeline operates as intended.
VLMs Qwen 2.5VL 3B† LLaMa-4-8B Qwen 3 Next-34B
Distance ↓ LPIPS ↓ MSE ↓ SSIM ↑ PSNR ↑ Whole ↑ Edited ↑ (×103 ) (×103 ) (×104 ) (×102 ) 7.64 6.57 6.53
32.09 33.43 32.34
38.47 37.19 38.05
33.57 31.31 32.76
90.21 91.33 91.49
29.78 30.44 31.01
27.57 28.82 29.03
the ReNO baseline. Starting from weaker backbones such as PixArt-α DMD and Flux, RewardFlow lifts the mean score from 0.45→0.65 and 0.64→0.81, respectively, and further improves over ReNO by +0.06 and +0.09 in overall performance. The gains are largest on the most compositional sub-tasks: for PixArt-α, Two objects and Counting increase from 0.38/0.46 to 0.77/0.65, and for Flux from 0.80/0.64 to 0.97/0.90. Even on the strong Qwen backbone, RewardFlow improves the overall performance from 0.83 to 0.91 and surpasses ReNO on all metrics, notably boosting Position from 0.27→0.47 and Color Attribution from 0.71→0.84. As a result, Qwen + RewardFlow achieves the best overall G EN E VAL performance, outperforming powerful off-the-shelf models such as SDXL, DALL-E 3, and SD3 (8B), whose mean scores remain in the 0.55–0.68 range. These quantitative gains stem from the way RewardFlow integrates diverse, task-aligned rewards into test-time optimization. Instead of relying primarily on a global alignment signal as in ReNO, RewardFlow evaluates a heterogeneous set of differentiable rewards covering semantic and perceptual alignment, regional and object-level consistency, and QA-style reasoning and fuses their gradients through a prompt-aware adaptive policy that adjusts reward weights and step sizes along the denoising trajectory. This richer, spatially and semantically grounded feedback allows the sampler to correct fine-grained failures such as incorrect counts, swapped colors, or mislocalized objects, while preserving the overall realism of the backbone generator. Consequently, RewardFlow is better able to satisfy complex multi-object, attribute, and localization constraints, which is reflected in its strong improvements on Two objects, Counting, Position, and Color Attribution compared to both unmodified backbones and prior reward-guided baselines. Ablation on VLMs. We further conduct an ablation study by replacing the visual-language model (VLM) used for Rvqa with different architectures. As shown in Table 6, the overall
10. Additional Results Text-to-Image Generation. We perform additional text-to-image generation evaluation on G EN E VAL. As shown in Table 5, RewardFlow consistently improves compositional faithfulness over both backbone models and 5
⇒
⇒
⇒
fill the patterns with colors as if you're on psychedelics
put a spider on the cake
turn the plushies into real animals
⇒
⇒
⇒
remake this scene in Monet's painting-style
fill the jar with random candy
ancient power seeps from the cauldron
⇒
⇒
⇒
the metal bike is covered in rust
draw the ornament like a child
cover his skin in diamonds
⇒
⇒
⇒
make a young girl wear the hoodie
⇒
turn her into a hologram
the open lid shows a reflection of the watch
⇒
make the house look 3D
turn it into pixel art
⇒
create a fictional character using this art style
Figure 12. Image Editing Qualitative Results with Flux + RewardFlow. For each input image on the left, RewardFlow is instructed to apply a targeted edit (text below), and the right image shows the generated result. Tasks span from global scene modifications and object-level edits to very fine-grained, localized edits.
performance remains relatively stable when scaling from 3B to 8B parameters, indicating that moderate model scaling yields limited benefit for this task. However, substituting with the larger and more recent Qwen3-Next-34B model leads to a noticeable ∼7% improvement across most evaluation metrics, suggesting that more capable VLMs enhance semantic reasoning in the reward estimation process, however, at the expense of increased computational overhead.
a natural landscape into Monet’s painting style, and turning a portrait into pixel art, all while keeping shapes and composition intact. Our proposed method also accurately handles object insertion and modification: a spider is added on top of the cake, plush toys are turned into realistic animals, a jar is filled with random candy, and “ancient power” is made to seep from the cauldron with coherent lighting. Local attribute edits are also precisely localized, e.g., metal parts of the bike are rusted without any corruptions, the ornament is redrawn in a child-like manner, the subject’s skin is covered with diamonds, and the pocket watch lid reflects the watch face without hallucinating unrelated content. Finally, RewardFlow successfully performs more abstract edits such as making a young girl wear the same hoodie, turning the woman into a hologram, making the cartoon house appear 3D, and synthesizing a new fictional character inspired
11. Additional Qualitative Results Image Editing Qualitative Results. Using Flux as the base model, as shown in Figure 12, RewardFlow follows a wide variety of fine-grained instructions while preserving background layout and image identity. RewardFlow can perform strong stylistic changes, such as recoloring the carved wooden ornament “as if on psychedelics,” translating 6
⇒
⇒
⇒
put a velvet curtain behind her
the floor is glass
put the cookies in a glass bowl
⇒
⇒
⇒
turn the helmet into gold
the town is experiencing a rainy day
turn it into an anime style image
⇒
⇒
⇒
the greek temple is amidst a zombie apocalypse
replace the dog with a cat
remove a slice from the cake; show the insides
⇒
⇒
⇒
turn the frosting blue
remove the white embroidering
turn the dog into a cartoon
⇒
⇒
⇒
place the gingerbread man on a table
cover the city in fog
change to a night sky
Figure 13. Image Editing Qualitative Results with Qwen Image + RewardFlow. For each input image on the left, RewardFlow is instructed to apply a targeted edit (text below), and the right image shows the generated result. Tasks span from global scene modifications and object-level edits to very fine-grained, localized edits.
by a textured input image. Across all examples, edits are restricted to instruction-relevant regions and avoid semantic leakage into the rest of the scene. With Qwen Image as the backbone, shown in Figure 13, RewardFlow exhibits similarly precise and diverse editing capabilities. Global scene edits include converting a sunny town into a rainy day, covering a city with fog, and changing a bright skyline to a night sky, while preserving camera pose and urban geometry. Attribute and material changes are handled cleanly, e.g., a velvet curtain is placed behind the bride, the showroom floor becomes glass, helmet material is changed to gold, and cake frosting is recolored blue without affecting decorations. RewardFlow also supports challenging object-level manipulations, such as putting cookies into a glass bowl, turning a dog into a cartoon, and placing the gingerbread man onto a table with consistent perspective. Fine, localized modifications, such as removing the embroidered
text on the hat, removing a slice from the cake and revealing the inside, and staging a “zombie apocalypse” around a Greek temple, are executed while maintaining sharp structure and coherent lighting. Results demonstrate RewardFlow generalizes across backbones and instruction types, delivering semantically faithful, spatially localized edits from global scene transforms down to pixel-level adjustments. Figure 14 presents a qualitative comparison between RewardFlow and recent image editing methods, including InfEdit, FlowEdit, FlowChef, InstantEdit, and KV-Edit, under the same input image and text instruction. The figure covers a range of challenging edit types, including material transformation, object-level semantic replacement, and color editing. In the first row, the instruction asks to make the frame of the bike rusty. Baseline methods exhibit different failure modes, e.g., some methods under-edit the image and leave large parts of the bicycle frame nearly unchanged (such as In7
Input Img
InfEdit
FlowEdit
FlowChef
InstantEdit
KV-Edit
Make the frame of the bike rusty
Change cat to labrador retriever
Change the cat from a real one to a silver sculpture
Change the color of the cat from orange to black Figure 14. Image Editing Qualitative Comparisons. Comparison across a range of challenging edits, such as diverse attribute-, style-, and object-level transformations. Each row shows the source image followed by results from strong baselines and RewardFlow.
fEdit, FlowChef, and KV-Edit), while others apply the rusty texture too aggressively or inconsistently, affecting broader regions and introducing unnatural appearance changes (such as FlowEdit and InstantEdit). In contrast, RewardFlow successfully transfers the rusty material appearance onto the bicycle frame while preserving the overall structure, viewpoint, wheel geometry, and background scene, resulting in a more coherent and realistic edit. In the cat-to-labrador transformation (second row), several baselines either fail to fully realize the target dog breed or generate inconsistent appearances, whereas RewardFlow produces a more convincing labrador retriever that remains in the same position on the chair, while keeping the surrounding environment intact. For the real-cat to silver-sculpture edit, baselines either fail to fully impose the metallic sculptural material or introduce artifacts in shape and surface reflectance, whereas RewardFlow renders metallic texture and reflective highlights, while preserving the original pose, object boundaries, and scene composition. Finally, in the fourth row, baselines sometimes over-darken the image, alter contrast unnaturally, or fail to perform a clean color transition, whereas RewardFlow produces a cleaner black cat while maintaining the
Figure 15. RewardFlow counting failure case.
cat’s silhouette, eye color, and overall scene context. Failure Modes. While robust, RewardFlow is bounded by its components. A primary failure mode arises from VQA limitations in fine-grained reasoning like counting. As shown in Figure. 15, if VQA model fails to accurately count small objects, the reward signal becomes uninformative. Text-to-Image Generation Qualitative Results. Figure 16 presents qualitative comparisons for text-to-image generation with the Flux backbone under three inference settings: 8
vanilla Flux, Flux guided by a global matching reward (Flux + GlobalReward), and the full reward-augmented model, RewardFlow (Flux + RewardFlow). Across a diverse set of prompts, including a chef portrait in a restaurant kitchen, a street-fashion scene in nighttime Tokyo, a multi-person family cooking scene, and a culturally specific festival portrait, the vanilla backbone generally captures the coarse scene semantics but frequently under-specifies fine-grained attributes, weakens environmental grounding, and exhibits limited compositional precision. Incorporating only the global reward improves overall prompt alignment and image aesthetics, yet the generations still miss localized details and precise relational cues, particularly in clothing structure, scene context, object placement, and human interaction. In contrast, RewardFlow consistently produces samples with stronger semantic fidelity, improved spatial and contextual grounding, and higher perceptual coherence. In the chef example, RewardFlow better realizes the warm kitchen environment, apron texture, flour details, and realistic skin appearance. In the Tokyo street scene, RewardFlow more faithfully captures the wet-pavement reflections, while in the family cooking example, RewardFlow yields more natural multi-person interaction, and better localized food and countertop details. In the festival portrait, RewardFlow more accurately renders traditional Indian attire through richer embroidery, more convincing jewelry, and a stronger festive lighting atmosphere.
Algorithm 1 RewardFlow: Prompt-aware multi-reward Langevin editing 1: Input: image x, prompt p, steps K 2: z0 ← Enc(x) 3: {pm }M m=1 ← E XTRACT SP S (p){semantic primitives} 4: (q, a⋆ ) ← M AKE QA(x, p) {fixed once} 5: Initialize running stats {µi , σi }i for all heads 6: Sample ε ∼ N (0, I), set z (0) ← αt̄ z0 + σt̄ ε, t0 ← t̄
7: for k = 0 to K − 1 do 8: z̃ (k) ← Den(z (k) , tk , p) 9: I (k) ← Dec(z̃ (k) ) 10: {SP-wise rewards: compute over p ∈ SP(p)} 11: Initialize score vectors: vglb , vper , vrg , voc ← [ ] 12: for each p ∈ SP(p) do 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25:
vglb .append(G LOB R EWARD(I (k) , p)) vper .append(P ERC R EWARD(I (k) , p)) vrg .append(R EGION R EWARD(I (k) , p)) voc .append(O BJ R EWARD(I (k) , p)) end for {Adaptive fusion of SP scores per head} for h ∈ {glb, per, rg, oc} do ℓh ← C OMPUTE W EIGHTS(vh , tk , SP(p)) ωh ← softmax(ℓh ) P (k) Rh ← j ωh [j] · vh [j] {weighted fusion} end for {Prompt-wise rewards: computed once per step} (k) Rhps ← HPSR EWARD(I (k) , p) (k)
Rvqa ← VQAR EWARD(I (k) , q, a⋆ ) {Normalize each head} for each head i do (k) (k) 29: Update µi , σi ; R̄i ← (Ri − µi )/(σi + ϵ) 30: end for 31: {Prompt-aware adaptive weights (Sec. 3.2)} (k) (k) 32: {wi }i ← C OMPUTE W EIGHTS {R̄i }, tk , p P (k) (k) (k) 33: Rtot ← i wi R̄i 34: {Fused reward gradient in latent space (Sec. 3.1)} P (k) (k) (k) 35: gI ← i wi ∇I (k) R̄i (k) 36: gRtot ,k ← λR JDen (z (k) , tk , p)⊤ JDec (z̃ (k) )⊤ gI 37: {Backbone drift and KL tether (Sec. 3.4)} 38: fk ← vθ (z (k) , tk , p) 39: gKL,k ← − λKL JDen (z (k) , tk , p)⊤ (z̃ (k) − z0 ) 40: {Reward-aware step size (Sec. 3.2)} (k) 41: ηk ← S TEP S IZE(Rtot ); ξk ∼ N (0, I) 42: {Langevin update} 43: z (k+1) ← z (k) + p ηk (fk + gRtot ,k + gKL,k ) + 2γ(tk )ηk ξk 44: tk+1 ← tk − ηk 45: end for 46: return Iˆ ← Dec Denθ (z (K) , 0, p) 26: 27: 28:
9
Flux
Flux + GlobalReward
Flux+
A confident middle-aged chef standing in a warm restaurant kitchen, arms crossed, flour on apron, soft window light, realistic skin texture
A stylish young man walking through Tokyo at night, layered streetwear, reflective wet pavement, neon signs glowing around him, candid fashion photography
A happy family cooking together in a bright modern kitchen, natural sunlight, fresh vegetables on the counter, candid laughter
A young woman in traditional Indian attire during a festival, intricate embroidery, jewelry, colorful lights in the background Figure 16. Text-to-image qualitative results with Flux as backbone. Qualitative comparison of Flux, Flux + Global Reward, and Flux + RewardFlow across diverse prompts.
10