Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF Eric Zhu Carnegie Mellon University
Abhinav Shrivastava University of Maryland, College Park
Soumik Mukhopadhyay University of Maryland, College Park [email protected]
Per-timestep Weighting
Historical Hardmining DDPO
Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of diffusion RLHF while preserving generalization to unseen prompts. Our key observation is that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6× improvement in sample efficiency compared to widely used diffusion RLHF baselines.
Ours DDPO
Trajectory Database Select top-k
arXiv:2607.07693v1 [cs.LG] 8 Jul 2026
Abstract
Ours
Figure 1. Our method has two parts. In the per-timestep weighting, we consider the relative importance of each transition step in the denoising trajectory. In the historical hardmining part, we look at previous trajectories with high advantage and repeat those trajectories in the training.
Figure 2. A graph of a baseline method (bottom curve) vs. the same baseline method using our method (top curve). As shown above, augmenting existing RLHF frameworks with our method achieves significant sample efficiency compared to the baseline.
are trained to reproduce the distribution of their training data, they do not inherently reflect human preferences. Recent work [2, 8] addresses this limitation through reinforcement learning from human feedback (RLHF) [6], which fine-tunes diffusion models using scalar feedback from human or reward models to explicitly optimize for preference alignment. One major challenge with RLHF in diffusion models is the credit assignment problem: the difficulty of determining which intermediate timesteps in the diffusion process actually contributed to the final reward. Because human feedback is only given on the final image, methods such as DDPO [2] simply assign the same loss to every timestep. This setup ignores the structure of the denoising process and
1. Introduction Diffusion models [41] have become the leading framework for high-quality image generation. However, because they 1
how different timesteps edit the image at different levels of granularity [25]. As a result, the model optimizes transitions uniformly without considering the non-uniform nature of the denoising trajectory, leading to inefficient training. Prior work attempts to address this issue by contrasting paired denoising trajectories generated from the same initial noise [14, 52]. In these approaches, trajectories remain identical until a designated branching timestep and diverge afterward. While contrasting two trajectories with shared initial noise can highlight the effect of the timestep at which they diverge, the approach does not contrast other timesteps in the sequence. Earlier steps remain identical and therefore uninformative, and later steps still inherit the same uniform advantage, leaving the broader credit assignment problem unresolved. Moreover, these methods require multiple reward evaluations per starting noise, reducing sample efficiency and increasing computational cost. In this paper, we argue that reward information in diffusion trajectories is inherently unevenly distributed across timesteps and trajectories. Some denoising steps and trajectories contribute significantly more learning signal than others. To exploit this observation, we propose two complementary strategies that emphasize informative training data. First, we introduce a per-timestep weighting scheme that selectively reweights denoising steps during optimization. We theoretically motivate this weighting by analyzing the relationship between Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO), showing that each timestep should ideally be weighted proportionally to the variance of the TD-error PPO advantage. Although this quantity is infeasible to estimate efficiently during training, we analyze its behavior and approximate the trend using a simple heuristic based on the squared magnitude of per-timestep latent change. Empirically, this weighting improves training sample efficiency. Second, we introduce a trajectory replay mechanism inspired by replay buffers commonly used in robotics-based reinforcement learning [1, 22, 34]. Instead of discarding previously sampled trajectories, we reuse informative trajectories during training. Drawing inspiration from prioritized replay buffers [34], we find that hard-mining trajectories with the largest advantages [40] provides the most sample-efficient learning. By emphasizing informative trajectories, the model can continue improving without repeatedly sampling new trajectories and reward evaluations. Overall, our method improves the efficiency of the baseline by 2–6× while remaining simple and compatible with existing diffusion RLHF pipelines. In summary, our main contributions are: • A computationally practical and mathematically motivated per-timestep weighting scheme that mitigates the credit assignment problem in diffusion RLHF. • A replay buffer mechanism that retrieves informative past
trajectories during training, reducing the need for repeated reward queries. • A simple, plug-and-play method that integrates seamlessly into existing diffusion RLHF pipelines without architectural changes. • Extensive experiments demonstrating consistent gains across diverse reward functions, underscoring the generality of our approach.
2. Related Work 2.1. Replay Buffers A replay buffer stores previous environment interactions for a policy to train on. It is standard in off-policy methods such as DDPG [20], Soft-Actor Critic [9], and Deep QNetworks [24], but not in on-policy methods like PPO [37] and TRPO [36], or in current RLHF methods, since these rely on sampling from the most recent weights to stay indistribution. Many replay-buffer variants exist. Hindsight experience replay [1] relabels failed trajectories with new goals to synthesize successful ones. Others prioritize transitions under certain criteria. The approach [34] samples by high temporal-difference (TD) error as a proxy for how undertrained a network is at a given point, and Energy-Based Hindsight [54] prioritizes high-kinetic-energy trajectories. Synthetic trajectories generated by diffusion models were proposed by [22], and retrieval was shown by [47] to enhance robot imitation learning. Finally, [19] hardmines previous PPO trajectories using cumulative TD error.
2.2. RLHF RLHF has become the central paradigm for aligning large language models (LLMs). Modern RLHF was introduced by [27], who scaled it to pretrained LLMs via a threestep pipeline of supervised fine-tuning, reward modeling, and reinforcement learning. Subsequent approaches such as Group Relative Policy Optimization [38] remove the reward model entirely in favor of group-normalized advantage, and [39] shows GRPO prevents catastrophic forgetting better than supervised fine-tuning. Preference-based approaches such as [31] have a human rank one sample over another rather than give a numerical reward. More recently, [4] segments the LLM token stream before computing segment-level reward, and [29] applies GRPO with prompt hardmining. RLHF has also been explored in robotics: [16, 45] train policies on preference-based evaluation of robot trajectories, [51] uses reward-guided evolutionary search over a pretrained trajectory diffusion model, and [32] finetunes pretrained imitation-learning diffusion policies such as diffusion policy [5]. 2
2.3. Diffusion RLHF RLHF is also effective for aligning diffusion models with human preference. Diffusion RLHF began with DPOK and DDPO [2, 8], which applied RL to diffusion and outperformed reward-weighted regression (RWR) [28]. Later approaches such as B2-DiffuRL [14] and Branch-GRPO [18] branch from shared input noise to sharpen contrast between trajectories, and another line of work demonstrates GRPO can be applied in flow matching and diffusion [18, 21, 50, 55]. In contrast to GRPO, preference-based methods [3, 44, 52] have a human pick between generated image pairs rather than assign a numerical reward and train on this feedback. Diffusion RLHF has been shown effective in video diffusion [30] and 2-step diffusion [17] settings. In contrast, some methods [7, 23, 48] assume a differentiable reward (e.g. a classifier or VLM) and backpropagate through it directly, while others localize spatial regions of reward importance [15, 48]. The work SIPO [53] reweights trajectory timesteps for more stable training under non-numerical pair-wise preference feedback, and a recent work TempFlow [10] explores timestep weighting in flow matching rather than DDPM-style models, but without hardmining. Moreover, they evaluate on a single reward function, whereas our evaluation across 5 reward functions shows their weighting scheme underperforms ours.
Figure 3. Change in reward of predicted x0 from step 0 to 4, 4 to 8, and so on. This can be used as a proxy for the effect of different timesteps t on the final reward. Interestingly, most of the details in the final image are determined by step 12.
Figure 4. Stochastic estimation of σt =
p Var(At ).
noise from the diffusion model, the state being the latent space of images, and the transition function being a sampler such as DDIM [42] and DDPM [12] that inputs the current latent and predicted noise.
3. Preliminaries 3.1. Markov Decision Process Formally, a MDP [43] can be represented as a tuple (S, A, P, R, γ), where S is the state space, A is the action space, P (s′ |s, a) is the probability transition function from current state s and action a to new state s′ , R(s, a) is the reward function, and γ is the discount factor. At each timestep t, the agent has access to state s and chooses an action a. The result is a new state s′ sampled from P(s′ |s, a) and a reward, denoted as R(s, a). The goal in reinforcement learning is to maximize the sum ofP decayed rewards, formally denoted as J(π) = n i Eτ ∼pπ i=0 γ R(si , ai ) .
3.3. PPO and Diffusion RLHF PPO [37] is a commonly used method in classical reinforcement learning. Formally, the PPO uses the loss function: πθ (a|s) LPPO = min At ππθθ (a|s) (a|s) , clip( πθ (a|s) , 1 − ϵ, 1 + ϵ)At old
old
where At is the advantage at timestep t, ϵ is the clipping value to prevent overly large weight shifts, and πθold (a|s) is the probability of taking action a with the old parameters in θold as opposed to the current parameters θ. In classical robotics RL, the advantage is given by the TD-error At = γV (st+1 )+Rt+1 −V (st ), where V is the value function. In the context of diffusion RLHF, prior methods do not use the TD-error but rather compute the uniform advantage A = r−rmean std(r) , where r is the reward at the end of the trajectory, rmean is the average reward in a batch of samples, and std(r) is the standard deviation of the rewards in a batch. The same advantage value A is used for all timesteps in the trajectory.
3.2. Diffusion Models For a given data sample x0 ∈ Rd , the forward diffusion process gradually adds noise over T steps to attain the noised trajectory (x0 , x1 , ..., xT ). Mathematically, the √ forward process can be defined as q(xt |xt−1 ) = N (xt ; 1 − βt xt−1 ; βt I), where βt is the monotonic noise schedule. The reverse diffusion process consists of a trained model pθ where pθ (xt−1 |xt ) learns to approximate the posterior distribution q(xt−1 |xt , x0 ), which is also a normal distribution with monotonic variance schedule β̃t . With reward feedback, the reverse diffusion process can be interpreted as a MDP with the action being the predicted
4. Methodology In existing diffusion RLHF pipelines, informative training signals are often obscured by uninformative or low-reward 3
expansion gives R(xt+1 ) ≈ R(xt ) + h⊤ t ∇R(xt ). Summing over timestepsP(t) yields a telescoping decomposition R(xT ) − R(x0 ) ≈ t ∆R(xt ) under the assumption of local linearity. This provides an approximate decomposition of the final reward in terms of ∆Rt and initial reward. If we disregard the initial reward R(x0 ) (which should average to the same quantity given random initial noise), ∆Rt can be viewed as a crude proxy for the contribution of timestep t to the final reward. In Fig. 3, we observe that ∆Rt varies significantly across timesteps, suggesting that different parts of the denoising trajectory contribute unequally to the final reward signal. We emphasize that ∆Rt should not be used directly for credit assignment, since it captures only the immediate reward change and does not account for the long-term effect of the state xt on future denoising steps.
Figure 5. Various timestep weighting schemes. For fairness, all timestep weightings are normalized to have a mean of 1.
data. Our approach addresses this issue at two levels: (1) timestep-level credit assignment and (2) trajectory-level data selection. We first analyze the credit assignment problem and derive a timestep-dependent weighting scheme, and then introduce a replay-based hard-mining strategy to prioritize informative trajectories. First, at the timestep level (Sec. 4.2), we try to address the credit assignment problem. While standard diffusion RLHF methods assign uniform credit to all denoising steps, we introduce a practical approximation to a theoretically motivated weighting scheme that assigns greater importance to diffusion steps that contribute most to the final reward (Fig. 1 (left)). Second, at the trajectory level (Sec. 4.4), we introduce a hard-mining mechanism that selectively reuses past trajectories with high absolute advantage, ensuring that the most informative experiences are emphasized during learning (Fig. 1 (right)).
4.1.2. Advantage Variance Proportional Weighting as a Natural Consequence. Motivated by this observation, we next connect diffusion RLHF to PPO to derive a principled timestep-level credit assignment scheme. A key difference between PPO and GRPO is that PPO does per-timestep advantages based on TD-error, thereby reducing the credit assignment issue that occurs with GRPO. We derive an equation to reconstruct the PPO timestep-level advantages from a singular final advantage that is computed using GRPO. We show that the PPO timestep advantages are proportional to the trajectory level advantage Afinal for some timestep-dependent constants w(t). By using these PPO advantage values over the uniform GRPO advantages, we can attain better convergence. The original PPO uses TD-error advantages as:
4.1. Non-Uniform Credit Assignment Traditional diffusion RLHF uses GRPO math to compute a mean single advantage A = R−R std(R) . Depending on the diffusion RLHF method, Rmean and Rstd either refer to the group or batch mean and standard deviation. This advantage constant A is applied evenly in its loss function. The same constant is used in each timestep: Lddpo =
n X
At = γV (st+1 ) + Rt+1 − V (st ) where At is the timestep-level advantage and V is the value function from reinforcement learning. For simplicity of notation, we assume a 20 step diffusion process. Let state sk = {xk , Tk } where Tk is the diffusion timestep and xk is the noisy image. Consider a diffusion trajectory (s0 , s1 , ..., s20 ) where state s0 corresponds to pure noise, s20 corresponds to the final image, and sk corresponds to intermediate latents. Consider the series of actions a0 , ..., a19 . In a RLHF scenario, there is only a single reward from s20 , so denote the final reward as Rfinal and all intermediate rewards as 0. Finally, we assume that γ = 1, as is typical in RLHF. We have that for t = 0, ..., 19 our equation for At simplifies to:
−A · log (Pθ (xt−1 |xt , c))
t=1
This loss function is problematic because it does not account for how much each step contributes to the final reward. To better understand the credit assignment issue, we first analyze how reward evolves along a diffusion trajectory. 4.1.1. Case Study. Let’s look at the reward corresponding to the predicted clean image x̂0 (xt ) at timestep t be defined as Rt = R(x̂0 (xt )) for some reward function R. For brevity, we denote this as R(xt ) since x̂0 varies smoothly with xt . Consider the quantity ∆Rt = R(xt+1 ) − R(xt ). For a small diffusion update xt+1 = xt + ht , a first order Taylor series
At = γV (st+1 ) + Rt+1 − V (st ) = V (st+1 ) − V (st ) We have that V (s20 ) = Rfinal . This leads us with A0 + A1 + ... + A19 4
Generate Trajectories
Calculate Reward
Update
DDPO Calculate Reward
Generate Trajectories Store
Trajectory Database
Sample top-k Reward Trajectories
Historical Hardmining
Update
Per-timestep Weighting Ours
Figure 6. An overview of our method. Compared to DDPO, we add a timestep-dependent weight that accounts for the asymmetric nature of the denoising process and add a replay buffer for storing previous trajectories and hardmining the important ones.
= [V (s1 ) − V (s0 )]+[V (s2 ) − V (s1 )]+...+[V (s20 ) − V (s19 )] Note that w(t) is larger when σk is larger, which happens in timesteps that change the final reward the most. Thus, = V (s20 ) − V (s0 ) = Rfinal − V (s0 ) if we align a schedule w(t) correctly with σt , then we can Since s0 is pure noise sampled from N (0, 1), there is construct accurate estimates of the PPO timestep-level ada very high diversity of denoising trajectory that can origvantages At . In our formulation, σt does not depend on st inate from the same start noise due to the high resamor at and solely depends on the timestep t. This allows us pling noise of existing samplers. Thus, we assume that to use the same w(t) universally for different trajectories. V (s0 ) ≈ Es0 ∼N (0,1) (Rfinal ) = Rmean . Thus, 4.1.3. Empirical Measurement of Var(At ) A0 +...+A19 = Rfinal −V (s0 ) ≈ Rfinal −Rmean = Afinal ·std(R) Intuitively, the advantage variance Var(At ) = σt2 should vary across diffusion timesteps, since the forward proUnder the assumption that we only know the final advancess adds noise with timestep-dependent variance βt . In tage Afinal and nothing else, we can estimate each At . We Fig. 4 we estimate σt2 by denoising a trajectory partially till have that At = V (st+1 ) − V (st ) is zero-meaned over all timestep t, and branching off into multiple denoising trajectrajectories because tories and computing variance of their final rewards. This is undesirable during training due to the need of large numV (st ) = Est+1 |st [γV (st+1 ) + Rt ] = Est+1 |st [V (st+1 )] ber of branches needed to accurately compute the variance. =⇒ Est+1 |st [At ] = Est+1 |st [V (st+1 ) − V (st )] = 0 However, this clearly indicates the need for a non-uniform weighting scheme and provides an initial indication of the Let σt be the standard deviation of At over the distribuexpected weighting trend. We see here a monotonically detion of trajectories generated by π. We further approximate creasing trend (except for a slight increase in the final steps, 2 each At as coming from a gaussian N (0, σt ). which might be due to image generation stochasticity). distribution for We want P19 to find the conditional P Ak | i=0 Ai = Afinal ·std(R) . This is the conditional 4.2. Strategy 1: Per-Timestep Weighting distribution of a multivariate normal under linear constraint. Previously, we motivated the case for using a non-uniform This gives us the distribution: weighting scheme using a case study. We then demonstrated that the PPO timestep-level advantage can be derived 19 X from the original GRPO advantage under certain assumpAk Ai = Afinal · std(R) ∼ N µ, σ 2 , tions, and that it depends directly on the variance of the i=0 advantage. While this derivation provides theoretical guid2 2 σk std(R) σk 2 2 P where µ = P A , σ = σ 1 − ance, estimating these quantities exactly during training is final n n k 2 2 i=0 σi i=0 σi impractical. We therefore seek a simple, practical weightRefer to appendix for more details on how this distribuing scheme that approximates this behavior. tion was derived. Thus, the expected value/mean of Ak is a We experiment with multiple weighting schemes as constant times Afinal where the constant is: shown in Fig. 5. An obvious choice for such a monotonically decreasing weighting qscheme is using the reverse difσt2 · std(R) w(t) = P n 2 fusion standard deviation β̃t used for adding randomness i=0 σi 5
trajectories after the current epoch is done. While this online sampling allows the method to train on trajectories that are generated by the most up-to-date weights, we argue that this is still highly sample inefficient and storing previous trajectories is more effective at increasing sample efficiency. TD error is not available in RLHF settings because rewards are provided only at the end. This prevents us from implementing a prioritized replay buffer. Instead, we implement a buffer that prioritizes high absolute advantage values.
4.4. Strategy 2: Historical Hardmining Building on this replay buffer formulation, we propose a hard-mining strategy that prioritizes trajectories with high absolute advantage values. Our intuition is that a trajectory with high absolute advantage, whether strongly positive or strongly negative, contains one or more timesteps that substantially changed the reward, making it highly informative training data. We perform two different methods of trajectory replay: (1) Trajectory-level hardmining, where we find the highest k trajectories based on |A| in the previous few epochs, and (2) Random sampling of previous trajectories. We find that keeping a replay buffer of only the last few epochs of trajectories was the most advantageous because sampling from an epoch too long ago resulted in out-ofdistribution trajectories. Overall, our method combines these two strategies: timestep-level reweighting and trajectory-level hard mining. Both strategies are motivated by the idea of reweighting the training data so that more informative samples are emphasized during training. See Algorithm 1 for details.
Figure 7. Qualitative comparisons between our method vs. default RLHF method vs. no training. In parentheses is how many reward queries were used during finetuning. All images within triplet are rendered using the same input seed and noise.
during sampling. (A recent work [10] tried to use a similar strategy of weighting using the standard deviation of noise in Flow-GRPO SDE.) We additionally consider the change in diffusion model’s mean absolute latent change |zt −zt−1 | and mean squared latent change |zt −zt−1 |2 as other choices for weighting schemes, which share the same monotonicity. Here, zt is the diffusion state in the latent space of Stable Diffusion VAE. As opposed to the previous static weighing schemes, this last scheme is dynamic, as it is dependent on the latents which in turn are dependent on how the current model denoises. Finally, we also try gaussian weighting schemes with early, middle, and late focus w.r.t. to the sampling steps. These are not monotonic but can signal towards which sampling step should be focused on more. We replace the GRPO advantage Afinal with our new advantage w(t)Afinal . Our new loss is:
Algorithm 1 Overall Two Strategy Algorithm Initialize buffer D = {}, pretrained diffusion model π for epoch = 1 to num epochs do for i = 1 to n do Sample trajectory τi = (xT , xT −1 , . . . , x0 ) end for for i = 1 to n do Compute weighted loss Lreweighted using Eq. (2) with trajectory τi θ = θ − ∇θ Lreweighted end for if D is not empty then Retrieve top k samples from D Compute Lreweighted using all top k samples θ = θ − ∇θ Lreweighted end if D ← D ∪ {τ1 , . . . , τn } D ← Remove Old Entries(D) # remove trajectories older than a few epochs end for
\mathcal {L}_{\text {reweighted}} = \sum _{t=1}^n -A_\text {final} \cdot w(t)\cdot \log \left (P_\theta (x_{t-1} | x_t, c)\right ) where w(t) is a scalar that only depends on the timestep of the transition.
4.3. Replay Buffers So far, we focused on improving credit assignment at the timestep level. We now address the second source of inefficiency: how trajectories are reused during training. In classical RL for robotics, methods [9, 20, 24] often make use of a replay buffer that stores previous trajectory from previous epochs. During training, episodes are drawn from the replay buffer and used to train the model alongside data from the current epoch. This allows the policy to sample from a larger dataset than just the trajectories from the current epoch. An extension of replay buffer is the prioritized replay buffer [34] which replays transitions with a high TD training error and has been shown to speed up training. In contrast to replay buffers, GRPO disregards diffusion
5. Experiments Since our method is highly applicable to existing diffusion RLHF frameworks, we choose to incorporate our method 6
DDPO
5.2. Sample Efficiency
DDPO
Ours
DDPO
Ours
DDPO
Ours
DDPO
Ours
In Figure 9, we see that our modifications are able to achieve higher reward than all existing methods within the same budget of reward queries. For fair comparison, each baseline and its augmented counterpart share identical hyperparameters. For qualitative examples of training images, see Figures 7 and 8. We also show that our method is capable of generalizing to novel prompts with higher sample efficiency than existing methods. For the list of unseen prompts, we prompted ChatGPT [26] for animals not present in the prompt dataset. As seen in Table 1, we find that our method still allows for more generalization than previous methods on the same budget of reward queries.
5.3. Ablations Since we do both per-timestep weighting and historical hardmining, we show that both of these methods independently make diffusion RLHF more sample efficient. We consider the 4 possibilities: (1) Neither hardmining nor timestep weighting (baseline), (2) Hardmining but not timestep weighting, (3) Timestep weighting but not hardmining, and (4) Both timestep weighting and hardmining. Our results show that each of our techniques contributes significantly to the overall efficiency of the finetuning method. See Figure 10 where, for all the rewards, using hardmining and weighting together is more effective than using only one.
Ours
Aesthetic Score (Artistic and Colorful)
JPEG Incompressibility (Add Details)
JPEG Compressibility (Reduce Details)
ImageReward (Learned Human Preference)
HPS V2 (Learned Human Preference)
back on generated images.
Training
Figure 8. Image samples from training with the same seed. For each reward function, there are two green boxes, one for our method and one for theirs. These boxes correspond to DDPO and our approach reaching the same reward level. We obtain the same amount of reward in approximately half the steps in all the reward variants. Note that the later images in our method’s row correspond to over-optimization on the specific reward function, a phenomena common to all RLHF platforms.
5.4. Timestep Weighting
with existing baselines to show the significant increase in sample efficiency. For all experiments, we use the same list of animal training prompts as [2]. We use LoRA [13] weights with rank 4 matrices that were inserted into RunwayML Stable Diffusion v1.5 [33]. For our experiments, we test our method applied to DDPO [2], DPOK [8], and B2-DiffuRL [14]. See full list of hyperparameters in the appendix.
Empirically, we find that timestep weighting consistently provides the most sample-efficient results, sometimes up to 6× efficiency. Despite our reward functions focusing on different aspects of an image, our timestep weighting boosts sample efficiency for all cases, as seen in Figures 10 and 9. Choice of Timestep Weighting. Using our Gaussian weighting schemes, we find that weighting earlier timesteps more heavily works better than weighting middle timesteps, which in turn works better than weighting later timesteps. This suggests that a monotonically decreasing weighting scheme is preferable. For reference, uniform (unweighted) weighting falls somewhere between the mid and early focus approaches in sample efficiency. In our experiments, the most consistently wellperforming scheme across all reward functions is the mean squared latent change, |zt − zt−1 |2 (see Fig. 11). This even outperforms the DDIM sampling standard deviation proposed in the recent work TempFlow [10]. We believe this scheme works well because it assigns higher weight to the timesteps where the model makes the largest changes to the latent representation. By prioritizing these high-change
5.1. Reward Functions We test our method on 5 reward functions. We select 3 rewards from the DDPO [2] paper. We also experiment with HPS v2 [46] and Image Reward [49]. The first reward function we test on is JPEG compressibility, which prefers compressed images and favors smoother textures that are easier to compress. Conversely, our second reward function is JPEG incompressibility, which favors highly detailed images with diverse textures. Third, we test on the reward function given by the Aesthetic Score classifier from [35] that is trained on human aesthetic preferences. Finally, both HPS v2 and Image Reward simulate human preference by training on aesthetic and prompt-adherence human feed7
Table 1. Generalization To Novel Prompts at 4k Reward Queries.
Untrained DDPO DDPO + Ours
Aesthetic (↑)
Jpeg Comp (↑)
Jpeg Incomp (↑)
Image Reward (↑)
HPS v2 (↑)
5.44 5.48 5.66
-118.75 -112.79 -44.25
118.758 124.48 233.28
0.48 0.53 0.71
0.284 0.2888 0.3004
Table 2. Prompt Adherence via CLIP Score (↑) at 4k Reward Queries
DDPO DDPO + Ours
Aesthetic
Jpeg Comp
Jpeg Incomp
Image Reward
HPS v2
0.289 0.311
0.305 0.298
0.299 0.310
0.308 0.312
0.304 0.300
5.5. Historical Hardmining We compare to historical sampling, where samples are randomly selected, to hardmining. As seen in Figure 12, we see that for all reward functions, hardmining is more effective than simply rerunning random samples from the previous epochs. This validates our hypothesis that large advantage trajectories are more informative than other trajectories.
Figure 9. Performance of our augmentation of existing diffusion RLHF methods. As seen in the graph, our method is able to train significantly faster than simply the method alone. All results are averaged over 3 runs and are smoothed with a running average of 500 reward queries.
5.6. CLIP Score In addition to testing sample efficiency, we also test CLIP score [11] on the finetuned models to measure promptimage adherence. CLIP score measures the embedding similarity between an image and its prompt, serving as a proxy for how faithfully the image reflects the text. We generated 1k images using training prompts and computed the average CLIP score, as seen in Table 2. We find that there is no visible degradation of prompt adherence compared to the baseline DDPO.
Figure 10. Ablation study on DDPO comparing the 4 cases: default DDPO, hardmine only DDPO, weighted timestep only DDPO, and weighted timestep and hardmined DDPO. As seen in the figure, both hardmining previous trajectories and reweighing the timesteps leads to significant increases in the performance of DDPO. Results are averaged over 3 seeds. Error margin is the std of the 3 runs. We show a running average of 500 rewards.
5.7. Generalization Score Following experiments [2], we also test generalization score. For our experiments, we first finetune a model based on a reward function for a given number of queries. Afterwards, we find a new list of prompts that were not in the list of training prompts and test how well our model keeps the reward high in those scenarios. As seen in Table 1, we find that our method also generalizes well to other prompts.
Figure 11. Performance of various weighting schemes. We observe that using the latent change squared gives the best results across all the rewards. In some rewards, Early focus is slightly better, but it is much inferior in HPS v2 and Image Reward.
6. Conclusion In this paper, we provide a simple but effective method to enhance the sample efficiency of diffusion RLHF. Our method is easily adaptable to existing RLHF platforms and has the potential to speed up training up to 6 times while retaining prompt-adherence and the ability to generalize to novel prompts. Our method departs from the common RLHF assumption that each timestep affects the output reward to the same extent. We ground our method on the intuition that both certain timesteps and certain trajectories are more informative than others during training. Future directions could analyze how different schedulers and αschedules affect the training sample efficiency or extend the method preference-based models such as D3PO [52].
Figure 12. Mean reward with different retrieval methods. All results were done with timestep weighting. For all methods, we ran for 3 seeds for consistency. We show with running average of 500 rewards. All results are averaged over 3 runs.
timesteps during reward-alignment training — rather than using fixed coefficients — the model is guided more effectively toward better rewards. 8
References
adaptation of large language models (2021), https: //arxiv.org/abs/2106.09685 7 [14] Hu, Z., Zhang, F., Chen, L., Kuang, K., Li, J., Gao, K., Xiao, J., Wang, X., Zhu, W.: Towards better alignment: Training diffusion models with reinforcement learning against sparse rewards (2025), https: //arxiv.org/abs/2503.11240 2, 3, 7 [15] Huang, Q., Dai, W., Liu, J., He, W., Jiang, H., Song, M., Song, J.: Patchdpo: Patch-level dpo for finetuning-free personalized image generation (2025), https://arxiv.org/abs/2412.03177 3 [16] Huang, S., Levy, M., Gupta, A., Ekpo, D., Zheng, R., Shrivastava, A.: Trend: Tri-teaching for robust preference-based reinforcement learning with demonstrations (2025), https : / / arxiv . org / abs / 2505.06079 2 [17] Jia, Z., Nan, Y., Zhao, H., Liu, G.: Reward fine-tuning two-step diffusion models via learning differentiable latent-space surrogate reward (2025), https : / / arxiv.org/abs/2411.15247 3 [18] Li, Y., Wang, Y., Zhu, Y., Zhao, Z., Lu, M., She, Q., Zhang, S.: Branchgrpo: Stable and efficient grpo with structured branching in diffusion models (2025), https://arxiv.org/abs/2509.06040 3 [19] Liang, X., Ma, Y., Feng, Y., Liu, Z.: Ptr-ppo: Proximal policy optimization with prioritized trajectory replay (2021), https://arxiv.org/abs/2112. 03798 2 [20] Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015) 2, 6 [21] Liu, J., Liu, G., Liang, J., Li, Y., Liu, J., Wang, X., Wan, P., Zhang, D., Ouyang, W.: Flow-grpo: Training flow matching models via online rl (2025), https: //arxiv.org/abs/2505.05470 3 [22] Lu, C., Ball, P.J., Teh, Y.W., Parker-Holder, J.: Synthetic experience replay (2023), https://arxiv. org/abs/2303.06614 2 [23] Luo, G., Granskog, J., Holynski, A., Darrell, T.: Dual-process image generation (2025), https:// arxiv.org/abs/2506.01955 3 [24] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning (2013), https://arxiv.org/abs/1312.5602 2, 6 [25] Mukhopadhyay, S., Gwilliam, M., Yamaguchi, Y., Agarwal, V., Padmanabhan, N., Swaminathan, A., Zhou, T., Ohya, J., Shrivastava, A.: Do text-free diffusion models learn discriminative visual representations? (2024), https://arxiv.org/abs/ 2311.17921 2
[1] Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., Zaremba, W.: Hindsight experience replay (2018), https://arxiv.org/abs/1707.01495 2 [2] Black, K., Janner, M., Du, Y., Kostrikov, I., Levine, S.: Training diffusion models with reinforcement learning (2024), https://arxiv.org/abs/2305. 13301 1, 3, 7, 8, 12 [3] Cai, M., Li, S., Li, W., Huang, X., Chen, H., Hu, J., Wang, Y.: Dspo: Direct semantic preference optimization for real-world image super-resolution (2025), https://arxiv.org/abs/2504.15176 3 [4] Chai, Y., Sun, H., Fang, H., Wang, S., Sun, Y., Wu, H.: Ma-rlhf: Reinforcement learning from human feedback with macro actions (2025), https://arxiv. org/abs/2410.02743 2 [5] Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., Song, S.: Diffusion policy: Visuomotor policy learning via action diffusion. In: Proceedings of Robotics: Science and Systems (RSS) (2023) 2 [6] Christiano, P., Leike, J., Brown, T.B., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences (2023), https://arxiv. org/abs/1706.03741 1 [7] Clark, K., Vicol, P., Swersky, K., Fleet, D.J.: Directly fine-tuning diffusion models on differentiable rewards (2024), https : / / arxiv . org / abs / 2309.17400 3 [8] Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., Lee, K.: Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models (2023), https://arxiv.org/abs/2305.16381 1, 3, 7 [9] Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018), https://arxiv.org/abs/1801.01290 2, 6 [10] He, X., Fu, S., Zhao, Y., Li, W., Yang, J., Yin, D., Rao, F., Zhang, B.: Tempflow-grpo: When timing matters for grpo in flow models. arXiv preprint arXiv:2508.04324 (2025) 3, 6, 7 [11] Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning (2022), https://arxiv.org/ abs/2104.08718 8 [12] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models (2020), https://arxiv.org/ abs/2006.11239 3 [13] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank 9
[26] OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H.W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S.P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S.S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N.S., Khan, T., Kilpatrick, L., Kim, J.W., Kim, C., Kim, Y., Kirchner, J.H., Kiros, J., Knight, M., Kokotajlo, D., Łukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C.M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S.M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H.P., Michael, Pokorny, Pokrass, M., Pong, V.H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y.,
Staudacher, N., Such, F.P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M.B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J.F.C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J.J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., Zoph, B.: Gpt-4 technical report (2024), https://arxiv.org/abs/2303.08774 7 [27] Ouyang, L., Wu, J., Jiang, X., Almeida, D., et al.: Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022) 2 [28] Peters, J., Schaal, S.: Reinforcement learning by reward-weighted regression for operational space control. In: Proceedings of the 24th International Conference on Machine Learning. p. 745–750. ICML ’07, Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1273496.1273590, https : //doi.org/10.1145/1273496.1273590 3 [29] Pikus, B., Tiwari, P.R., Ye, B.: Hard examples are all you need: Maximizing grpo post-training under annotation budgets (2025), https://arxiv.org/ abs/2508.14094 2 [30] Prabhudesai, M., Mendonca, R., Qin, Z., Fragkiadaki, K., Pathak, D.: Video diffusion alignment via reward gradients (2024), https://arxiv.org/abs/ 2407.08737 3 [31] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model (2024), https://arxiv.org/abs/ 2305.18290 2 [32] Ren, A.Z., Lidard, J., Ankile, L.L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., Simchowitz, M.: Diffusion policy policy optimization (2024), https://arxiv.org/abs/2409. 00588 2 [33] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) 7 [34] Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay (2016), https://arxiv. org/abs/1511.05952 2, 6 [35] Schuhmann, C., Beaumont, R.: Laion-aesthetics. LAION. AI (2022) 7 10
[36] Schulman, J., Levine, S., Moritz, P., Jordan, M.I., Abbeel, P.: Trust region policy optimization (2017), https://arxiv.org/abs/1502.05477 2 [37] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017), https://arxiv.org/abs/ 1707.06347 2, 3 [38] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y., Guo, D.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models (2024), https: //arxiv.org/abs/2402.03300 2 [39] Shenfeld, I., Pari, J., Agrawal, P.: Rl’s razor: Why online reinforcement learning forgets less (2025), https://arxiv.org/abs/2509.04259 2 [40] Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining (2016), https://arxiv.org/abs/ 1604.03540 2 [41] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics (2015), https:// arxiv.org/abs/1503.03585 1 [42] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models (2022), https://arxiv.org/ abs/2010.02502 3 [43] Sutton, R., Barto, A.: Reinforcement learning: An introduction. IEEE Transactions on Neural Networks 9(5), 1054–1054 (1998). https://doi.org/10.1109/TNN.1998.712192 3 [44] Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., Naik, N.: Diffusion model alignment using direct preference optimization (2023), https://arxiv. org/abs/2311.12908 3 [45] Wang, Y., Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., Erickson, Z.: Rl-vlm-f: Reinforcement learning from vision language foundation model feedback (2024), https://arxiv.org/abs/2402. 03681 2 [46] Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., Li, H.: Human preference score v2: A solid benchmark for evaluating human preferences of textto-image synthesis. arXiv preprint arXiv:2306.09341 (2023) 7 [47] Xie, A., Chand, R., Sadigh, D., Hejna, J.: Data retrieval with importance weights for few-shot imitation learning (2025), https://arxiv.org/abs/ 2509.01657 2 [48] Xing, X., Saha, A., He, J., Hao, S., Vicol, P., Ryu, M., Li, G., Singla, S., Young, S., Li, Y., Yang, F., Ramachandran, D.: Focus-n-fix: Region-aware fine-
tuning for text-to-image generation (2025), https: //arxiv.org/abs/2501.06481 3 [49] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36, 15903–15935 (2023) 7 [50] Xue, Z., Wu, J., Gao, Y., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., Luo, P.: Dancegrpo: Unleashing grpo on visual generation (2025), https://arxiv.org/abs/2505.07818 3 [51] Yang, B., Su, H., Gkanatsios, N., Ke, T.W., Jain, A., Schneider, J., Fragkiadaki, K.: Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following (2024), https://arxiv.org/abs/2402.06559 2 [52] Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Li, Q., Shen, W., Zhu, X., Li, X.: Using human feedback to fine-tune diffusion models without any reward model (2024), https://arxiv.org/abs/ 2311.13231 2, 3, 8, 12 [53] Yang, X., Tan, Z., Wang, J., Zhou, Z., Li, H.: Sdpo: Importance-sampled direct preference optimization for stable diffusion training (2025), https: //arxiv.org/abs/2505.21893 3 [54] Zhao, R., Tresp, V.: Energy-based hindsight experience prioritization (2020), https://arxiv.org/ abs/1810.01363 2 [55] Zhou, Y., Ling, P., Bu, J., Wang, Y., Zang, Y., Wang, J., Niu, L., Zhai, G.: Fine-grained grpo for precise preference alignment in flow models (2025), https: //arxiv.org/abs/2510.01982 3
11
A. Appendix A.1. Hyperparameters For Baselines We chose to use the same hyperparameters for baselines as those chosen for all baselines in the D3PO [52] paper. The only difference is that we chose a smaller number samples per epoch as we found that sampling more often led to better performance. We also changed to AdamW over Adam, as consistent with most diffusion RLHF methods. Between baseline and baseline + our method, we kept all hyperparameters the same. All results were obtained with 4 RTX A5000 GPUs or 4 RTX A4000 GPUs. See a comprehensive list of hyperparameters in Tables 3 and 4.
A.2. Prompt List
Figure 13. Comparison of Hardmining to increasing the LR and to randomly replaying samples from the current epoch. All results are averaged over 3 seeds.
We used the same training prompt list as in [2]. The list of animals is listed below: cat dog horse monkey rabbit zebra spider bird sheep deer cow goat lion tiger bear raccoon fox wolf lizard beetle ant butterfly fish shark whale dolphin squirrel mouse rat snake turtle frog chicken duck goose bee pig turkey fly llama camel bat gorilla hedgehog kangaroo For generalization experiments, we used the following list of animals generated by ChatGPT: elephant giraffe hippopotamus rhinoceros leopard cheetah hyena bison moose elk reindeer antelope armadillo sloth otter beaver badger lynx bobcat cougar jaguar capybara porcupine platypus echidna koala wallaby wombat manatee walrus seal narwhal orca penguin albatross flamingo peacock owl eagle hawk parrot crocodile alligator chameleon salamander
A.3. Comparisons with Hardmining When using hardmining, even though we use lesser number of reward calls, we use more backpropagation through the network. Hence, an obvious question might be whether using hardmining helps just because there are more samples whose gradients are being backpropagrated or is hardmining helping because of the selection of more important trajectories. In this section, we compare hardmining with two more methods. First, we compare to simply increasing the learning rate proportional to how many more gradient steps we take. Second, we compare to randomly replaying entries from the current epoch. These two settings correspond to using faster learning rate and more gradient calls respectively, both of which are alternatives to hardmining. Note that this is different from Figure. 12 where we compared to random retrieval from previous epochs as opposed to only the current epoch. As seen in Figure 13, our hardmining
based on the absolute advantages does better than both increasing the learning rate and repeating samples from the current epoch, showing that hardmining helps not just because of increased samples.
σ 2 · std(R) 2 Afinal i=1 σi
k A.4. Proof That Ak has mean P n
We want to find the probability distribution of Ak where we assume that each Ai has mean 0 and standard deviation σi for i ∈ {0, 1, 2, ..., n}. Furthermore, we have the condition that for a given constant C,
A0 + A1 + A2 + ... + An = C
For simplicity of notation, assume that k = 0. The proof can easily generalize to other values of k. Let B = A1 + A2 + ... + An . The mean of B is the sum of the means of A1 + A2 , ...An which is 0. The variance of B is the sum of the variances of A1 + A2 , ...An . 2 This means that σB = σ12 + σ22 + ... + σn2 . Finally, we have that from the summation of Gaussian distributions is still a Gaussian distribution. Thus,
2 B = A1 + A2 + A3 + ... + An ∼ N 0, σB
We can rewrite our question as finding the distribution A0 | A0 + B = C. We have that the for a given value a such that A0 = a,