arXiv CS open access

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

arXiv:2608.02332v1 [cs.LG] 3 Aug 2026

Botao Dong, Longyang Huang, Ning Pang, and Hongtian Chen

Abstract—In offline reinforcement learning (RL), the distribution shift between behavioral data and the learned policy can lead to erroneous Q-value estimation, thereby misguiding the direction of policy optimization. To address this issue, we develop a behavioral advantage corrected policy evaluation (BACPE) approach, which utilizes the Q-function of the behavior policy to correct the learned policy’s Q-function, thus mitigating pessimistic conservatism and overestimation bias. Furthermore, the convergence of BAC-PE is analyzed theoretically, and an upper bound on the difference between the learned Q-function and the true Q-function is derived. To alleviate distribution shift, this work employs diffusion models to represent both the behavior policy and the learned policy, performing distribution matching for accurate policy regularization. Additionally, Qvalue guidance is incorporated into the training process to achieve effective policy improvement. By combining BAC-PE with diffusion policy modeling, we propose the diffusion policy with behavioral advantage correction (DPBAC) algorithm. Compared to existing offline methods, DPBAC demonstrates stronger policy representation capabilities and effectively mitigates the bias in Q-value estimation. Experimental results on multiple domains of D4RL tasks show that DPBAC achieves superior performance, with notable advantages over state-of-the-art (SOTA) algorithms. Index Terms—Distribution shift, policy optimization, diffusion models, policy regularization.

I. I NTRODUCTION S a pivotal subfield of machine learning, RL enables agents to learn optimal policies through iterative interactions with the environment and feedback-driven adaptation [1]. This methodology has achieved substantial advancements in various domains such as robotics [2]–[4], autonomous driving [5]–[7], and industrial processes [8], underscoring its versatility and potential in solving complex decision-making tasks [9]. However, online RL training requires extensive agent-environment interactions to collect samples [10], leading to safety risks [11] and high data acquisition costs [12]. These limitations are particularly pronounced in scenarios requiring real-time decision-making and stringent safety guarantees. Compared with online RL, offline RL relies on pre-collected datasets for training [13], which eliminates the necessity of real-time interaction with the environment [14]. This not only enhances agent safety but also ensures the efficient utilization of available data. However, directly applying online RL methods in offline settings often results in training failures due to inherent challenges in offline RL [15], such as undesirable distribution shift [16] and the overestimation of out-

Botao Dong is with the School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore 637459 (e-mail: botao.dong@ntu.edu.sg).

of-distribution (OOD) actions [17]. To address these issues, existing studies primarily focus on two main strategies: value regularization, which aims to mitigate overestimation biases in Q-values [18], and policy regularization, which enhances the consistency between the learned policy and the behavior policy [19]. Value regularization approaches aim to correct valueestimation errors caused by Bellman backups over actions outside the support of the offline dataset. Since the critic is directly trained only on state-action pairs from the given dataset, OOD actions queried by the actor or target policy may be assigned unrealistically large value estimates due to the lack of empirical transition support. Such positive extrapolation errors can be recursively amplified through temporal-difference backups, driving the learned policy toward unsupported regions. To break this error-amplification loop, value regularization modifies the critic objective by lowering the values of unsupported actions, suppressing uncertain estimates, or restricting backups to reliable in-dataset actions [20], [21]. Representative methods include conservative Q-learning (CQL) [22], which penalizes OOD actions to learn a lowerbound critic; implicit Q-learning (IQL) [23], which avoids explicit OOD action sampling by estimating expectile values and extracting high-advantage in-dataset actions; falSe COrrelation REduction (SCORE) [24], which suppresses unreliable estimates using bootstrapped ensembles; and supported value regularization (SVR) [25], which preserves Bellman updates on supported actions while penalizing OOD regions. Although these methods differ in identifying unsupported or uncertain actions, they all inject pessimism into policy evaluation to prevent OOD value exploitation. However, excessive pessimism may also limit policy improvement, since conservative penalties cannot always distinguish unreliable OOD actions from potentially high-return but under-sampled actions in partially covered datasets. Moreover, broad penalties may spill over through function approximation to nearby or potentially valuable in-distribution regions, and the induced negative bias can further propagate through Bellman backups, especially in sparse-reward or long-horizon tasks [26]. Consequently, value regularization can reduce overestimation, but it may also mis-rank promising actions and provide weak signals for policy improvement. This limitation motivates the proposed behavioral advantage corrected policy evaluation mechanism. Instead of imposing a uniformly conservative penalty, the proposed mechanism uses the Q-value of the behavior policy as an adaptive reference to correct both overestimation and excessive pessimism. Unlike value regularization, which focuses on constraining

the learned Q-values, policy regularization addresses distribution shift from the actor side by constraining the learned policy to remain close to the behavior policy induced by the dataset [27]. Its objective can be interpreted as maximizing the learned value while imposing a constraint or penalty on the divergence between the learned policy and the behavior policy [28]. Existing methods mainly differ in how the behavior support is represented and how strictly the learned policy is allowed to deviate from it. At the most conservative end, behavior cloning (BC) [29] performs pure supervised imitation, which achieves strong behavioral consistency but ignores rewarddriven improvement [30]. Action-support constraint methods such as batch-constrained Q-learning (BCQ) [31] relax this restriction by learning a generative model of dataset actions and applying a small perturbation, so that the actor can improve values while remaining close to sampled behavior actions. Divergence-based or trust-region-based methods further formulate this idea as an explicit regularized optimization problem, where a penalty term controls the distance between the learned policy and the behavior policy [32], [33]. Datasetgeometry-aware approaches such as DOGE [34] refine the notion of support by using geometric information of the dataset, allowing policy improvement in regions that are close to or generalizable from the data while remaining conservative in uncertain areas. More recently, diffusion-based behavior regularization uses expressive diffusion behavior models to estimate heterogeneous behavior distributions and regularize policy optimization more accurately [35]. Thus, policy regularization methods form a spectrum from strict imitation to valueaware constrained improvement, differing mainly in whether they constrain pointwise actions, distributional divergence, trust regions, dataset geometry, or generative behavior scores. Their common limitation is that an overly tight constraint inherits the suboptimality of the behavior policy, whereas an overly loose constraint reintroduces OOD actions and extrapolation error [36], [37]. Moreover, practical regularization relies on an accurate representation of the behavior policy and an appropriate discrepancy measure. When the behavior distribution is multimodal, long-tailed, or generated by multiple skills, simple deterministic or unimodal Gaussian multi-layer perceptron policies may collapse different action modes and yield inaccurate divergence estimates. In this case, feasible behavior modes can be mistakenly treated as OOD, and the learned policy may either be over-regularized or drift toward unsupported actions, resulting in degraded offline RL performance. Inspired by the aforementioned analysis, we identify three critical challenges faced by existing offline RL methods: (1) improving the Q-value estimation mechanism to avoid both optimistic overestimation and excessive pessimism, (2) implementing appropriate policy regularization to control distribution shift without simply inheriting the suboptimality of the behavior policy, and (3) constructing an expressive policy representation capable of accurately modeling complex behavior distributions while supporting value-guided policy improvement. To address these challenges, this work develops a behavioral advantage corrected policy evaluation (BAC-PE) mechanism, which leverages the Q-function of the behavior

policy as an adaptive reference for correcting the learned policy’s Q-function. Moreover, diffusion models are employed to represent both the behavior policy and the learned policy, enabling high-fidelity behavior-policy modeling and Q-guided diffusion policy improvement. The key contributions of this work are outlined as follows:

1) A behavioral advantage corrected policy evaluation mechanism, termed BAC-PE, is proposed for offline RL. Rather than broadly penalizing unsupported actions, BAC-PE uses the behavior-policy Q-value as an adaptive reference to correct the learned critic, thereby mitigating both OOD overestimation and excessive pessimism. 2) The convergence of BAC-PE is rigorously established via the corrected Bellman operator. An explicit upper bound on the discrepancy between the corrected and true Q-functions is further derived, showing its dependence on transition estimation bias and the distributional discrepancy between the learned and behavior policies. 3) BAC-PE is integrated with diffusion-based behaviorpolicy modeling and Q-guided policy improvement to construct DPBAC. This design combines adaptive critic correction with an expressive diffusion policy class, enabling accurate policy regularization and reward-driven improvement beyond behavior imitation. 4) Comprehensive experiments are conducted on multiple domains of the D4RL benchmark to compare DPBAC with SOTA offline RL algorithms. Evaluations of task performance, policy expressiveness, sensitivity, sparsereward learning, and Q-value estimation bias demonstrate the effectiveness and superiority of DPBAC.

Compared with representative offline RL methods, the distinctions of DPBAC are elaborated as follows. Unlike value-regularization methods such as CQL, IQL, SCORE, and SVR, which suppress OOD value exploitation through conservative, in-dataset, uncertainty-aware, or support-aware value estimation [22]–[25], DPBAC introduces BAC-PE to correct the learned critic using the behavior policy Q-value as an adaptive reference, thereby mitigating both overestimation and excessive pessimism. Compared with policy-regularization methods such as BC, BCQ, DOGE, and behavior-regularized approaches, which constrain policy learning through imitation losses, generative action constraints, dataset geometry, or divergence/trust-region penalties [29], [31]–[34], DPBAC combines diffusion-based behavior modeling with valueguided improvement to maintain dataset consistency while seeking higher-return actions. Unlike diffusion-based offline RL methods such as DQL [44] and SRPO [35], which mainly employ diffusion models as expressive policy classes or behavior score regularizers, DPBAC further integrates diffusion policy modeling with behavioral advantage corrected critic learning and provides theoretical guarantees for policy evaluation. Thus, DPBAC jointly addresses critic bias correction, expressive behavior modeling, and Q-guided policy improvement within a unified offline RL framework.

II. P RELIMINARIES A. Online Reinforcement Learning As the mathematical foundation of RL, the Markov decision process is formally expressed by a tuple (S, A, P, r, γ, ϱ0 ), where S denotes the state space, A represents the action space, P corresponds to the state transition probability, r defines the reward function satisfying |r(s, a)| < r, γ specifies the discount factor, and ϱ0 characterizes the initial state distribution. To derive the optimal policy π ∗ (·|s), RL approaches typically involve learning a Q-function Qπ (s, a), which quantifies the expected cumulative reward of a given policy π(·|s) a given state-action pair (s, a), i.e., Qπ (s, a) ≜ Pfor ∞ Eρπ [ i=t γ i−t r(si , ai )|st = s, at = a], where ρπ represents the trajectory distribution arising from π. Notice that Qπ (s, a) is uniquely determined as the solution to the classic Bellman equation Qπ (s, a) = (B π Qπ )(s, a), where the Bellman operator B π is formulated as (B π Q)(s, a) = r(s, a) + γPπ Q(s, a),

(1)

with Pπ Q(s, a) ≜ Es′ ∼P(·|s,a),a′ ∼π(·|s′ ) [Q(s′ , a′ )]. B. Offline Reinforcement Learning Unlike online RL, offline RL operates solely on a static dataset D collected by a behavior policy µ(·|s), with no interaction with environment throughout the learning process. As a result, the state transition probability P(·|s, a) and the Bellman operator B π are not directly accessible in offline RL. b To this end, an estimated state transition probability P(·|s, a) can be approximated based on the given dataset, resulting in an estimated Bellman operator Bbπ defined as b π Q(s, a), Bbπ Q (s, a) = r(s, a) + γ P (2) ′ ′ b π Q(s, a) ≜ E ′ b where P s ∼P(·|s,a),a′ ∼π(·|s′ ) Q(s , a ) . Let b π (s, a) denote the unique fixed point of Bbπ , i.e., Q b π (s, a) = Q π bπ b B Q (s, a). C. Diffusion Model Diffusion models utilize a forward process of progressive noise injection combined with a reverse denoising mechanism to generate data samples [38]. By capturing the probabilistic structure of the given data distribution, diffusion models can generate samples that closely match the statistical characteristics of the training data. The core of training diffusion models lies in optimizing a neural network to accurately predict the noise introduced at each step of the forward diffusion process. This ability enables the network to progressively remove the corresponding noise during the reverse denoising steps, ultimately generating samples that closely resemble the original data [39]. III. BAC-PE A NALYSIS AND D IFFUSION P OLICY D ESIGN Due to the distribution shift, offline RL may involve the selection of OOD actions, which lack sample support, leading to potential overestimation of the Q-function. This can cause the learned policy to diverge from the behavior policy, resulting in

degraded performance or even policy failure. To address this issue, offline RL attempts to maximize performance within the data-supported distribution of the behavior policy µ(·|s), with the optimization objective being: b π (s, a) − 1 ΥKL π(·|s)∥µ(·|s) , max Es∼D,a∼π(·|s) Q π λ (3) where λ represents the weighting factor and ΥKL (·∥·) denotes the KL divergence. As demonstrated in [35], the optimization problem (3) has an analytical solution given by: π ∗ (a|s) =

1 b π (s, a) , µ(a|s) exp λQ W (s)

(4)

where W (s) denotes the partition function. As inferred from (4), achieving the optimal policy π ∗ (·|s) requires accurate modeling of the behavior policy µ(·|s) and reliable estimation of the Q-function. To this end, we develop a policy evaluation approach aimed at providing accurate and trustworthy Q-function estimation, as detailed in Section III-A, while facilitating high-fidelity representation of the behavior policy µ(·|s) and empowering the learned policy π(·|s) to model complex and multimodal behavior distributions, as elaborated in Section III-B. A. Behavioral Advantage Corrected Policy Evaluation To concurrently address the issues of pessimistic bias and overestimation in Q-function, a policy evaluation method incorporating behavioral advantage correction is designed. Prior to introducing the update law for the Q-function, the formal definition of the behavioral advantage is provided as follows. Definition 1: The behavioral advantage AµQ , defined as the difference between the Q-function of the behavior policy Qµ and a given Q-function Q(s, a), is expressed as: AµQ (s, a) = Qµ (s, a) − Q(s, a), ∀(s, a).

(5)

Remark 1: Based on the above definition, a positive behavioral advantage indicates that the Q-function of the behavior policy, Qµ (s, a), suggests a higher expected return for executing the behavior policy at the current state-action pair (s, a). Conversely, a negative behavioral advantage implies a potential overestimation in the given Q-function, Q(s, a), for the corresponding state-action pair. Such overestimation may propagate through the successive policy iteration process, potentially compounding errors and leading to a significant deterioration in overall policy performance. To address the challenges of pessimistic bias and erroneous overestimation in the learned Q-function, the classic Bellman operator B π in (1) is refined by incorporating the behavioral advantage AµQ in (5). The formulation of the corrected Bellman operator T π is presented as follows: T π Q (s, a) = B π Q (s, a) + ηAµQ (s, a), (6) where η denotes the correction coefficient. In offline settings, where the state transition probability P(·|s, a) is unavailable,

the estimated form of the corrected Bellman operator, denoted by Tb π , is formulated as: bµ (s, a), Tb π Q (s, a) = Bbπ Q (s, a) + η A (7) Q where the estimated behavioral advantage is described as bµ (s, a) = Q b µ (s, a) − Q(s, a), and Q b µ (s, a) denotes the A Q estimated Q-function of the behavior policy. Motivated by the corrected Bellman operator Tb π in (7), a novel iterative rule for policy evaluation is developed: e k+1 (s, a) = Tb π Q e k (s, a), k = 0, 1, 2 · · · , Q (8) e k+1 (s, a) denotes the value at each iteration step, where Q e 0 (s, a). starting from the initial iteration value Q The vectorized form of the Q-function is denoted by the bold symbol Q, whose dimensions correspond to the product of the state space and action space dimensions. Each component of Q represents the Q-value associated with a specific (s, a) pair. Accordingly, the vectorized representation of (8) e k+1 = Tb π Q e k . Then, the fixed point of Tb π , is given by Q along with the convergence analysis of the iterative rule (8), is subsequently established in Theorem 1. Theorem 1: Given a policy π and an initial iterative value e 0 , the sequence of iterative values will converge to Q eπ, Q e π = Tb π Q e π , prowhich satisfies the fixed-point equation Q vided that the iterative rule (8) is employed. Proof: See Appendix A. Following this, when the estimation bias in the state tranb sition probability P(·|s, a) is eliminated, Tb π degenerates into π T . The contraction mapping property of T π , along with its π fixed point Q , is presented in the subsequent corollary. Corollary 1: The corrected Bellman operator T π is a contraction mapping, with its fixed-point equation formulated π π as Q = T π Q . Proof: The proof of Corollary 1 can be readily derived from that of Theorem 1 by substituting Tb π with T π . Theorem 1 presents the convergence and fixed-point properties of the iterative rule (8). However, the relationship between e π and Qπ , which are induced by the the fixed points, Q π b operators T and B π respectively, is further investigated in Theorem 2. e π of Theorem 2: The relationship between the fixed point Q π π b the operator T and the fixed point Q of the operator Bπ is characterized as follows: π e π = Qπ + γ(I − γPπ )−1 P b π − Pπ Q e Q π −1 b µ eπ . + η(I − γP ) Q −Q (9) Proof: See Appendix B. Assuming that the state transition probability of the environb ment can be accurately estimated, i.e., P(·|s, a) = P(·|s, a), π π b the equivalence between T and T holds, thereby yielding π the following corollary concerning the fixed point Q of T π . π Corollary 2: The relationship between Q and Qπ is π π characterized by Q = Qπ + η(I − γPπ )−1 Qµ − Q . b π = Pπ Proof: The proof can be established by setting P π π e and replacing Q with Q in (9). π To formally quantify the discrepancy between Q and Qπ , Theorem 3 presents a rigorous analysis of the associated

bound. Before establishing this result, Lemma 1 is introduced to provide the necessary preliminary results for the subsequent derivation. Lemma 1: [40] Given a policy π, the difference between its Q-function, Qπ , and that of the behavior policy, Qµ , is bounded as follows: 2γr max ΥT V π(·|s)∥µ(·|s) , (10) Qπ − Qµ ∞ ≤ 2 (1 − γ) s where ΥT V (·∥·) refers to the total variation divergence. Building upon Lemma 1, Theorem 3 formally characterizes π the difference between Q and Qπ as follows. Theorem 3: For an arbitrary policy π, the difference between π Q and Qπ is governed by the following bound: maxs ΥT V π(·|s)∥µ(·|s) π π ≤ 2ηγr · Q −Q . (11) (1 − γ e)(1 − γ)2 ∞ Proof: See Appendix C. Furthermore, taking into account the estimation bias beb e π and tween P(·|s, a) and P(·|s, a), the difference between Q π Q is analyzed in Theorem 4. In preparation for this analysis, Lemma 2 provides the requisite preliminary results as follows. b Lemma 2: Considering the estimation bias in P(·|s, a), the difference between Qπ and the estimated Q-function of b µ , is characterized by: behavior policy, Q 2γr bµ Qπ − Q ≤ max ΥT V π(·|s)∥µ(·|s) 2 (1 − γ) s ∞ h i 2γr b b µ −1 + I − γP max ΥT V P(·|s, a) P(·|s, a) . 1−γ ∞ s,a (12) Proof: See Appendix D. Drawing from Lemma 2, Theorem 4 establishes the result e π and Qπ . concerning the difference between Q Theorem 4: Under an arbitrary policy π, the difference e π and Qπ is bounded as follows: between Q maxs ΥT V π(·|s)∥µ(·|s) e π − Qπ Q ≤ 2ηγr · (1 − γ e)(1 − γ)2 ∞ max Υ P b P s,a T V µ −1 b , (13) + 2γr 1 + η I − γ P (1 − γ)(1 − γ e) ∞ b P represents the abbreviated notation for where ΥT V P b ΥT V P(·|s, a) P(·|s, a) . Proof: See Appendix E. Remark 2: As analyzed in Theorem 4, the difference e π and Qπ is primarily governed by the sum of between Q two terms. One term represents the discrepancy between the actual state transition probability P(·|s, a) and its estimation b b P(·|s, a), quantified as ΥT V P(·|s, a) P(·|s, a) , which is inherently influenced by the statistical properties and distribution of the given dataset. The other term captures the discrepancy between the learned policy π(·|s) and the behavior policy µ(·|s), expressed as ΥT V π(·|s)∥µ(·|s) . To reduce this policy distributional discrepancy, it is essential for the learned policy π(·|s) to be capable of modeling complex, multimodal behavior distributions, thereby enabling better alignment with the behavior policy µ(·|s). To this end, the following diffusion-based policy mechanism is developed and elaborated in Section III-B.

B. Diffusion-Based Policy Scheme

Diffusion Policy with Behavioral Advantage Correction (DPBAC)

To capture the complex and multimodal behavior distribution inherent in the given static dataset D = {(s, a, r, s′ )}, a diffusion model is adopted. By learning the intricate distributional properties, the diffusion model facilitates the generation of actions that align with the dataset’s statistical characteristics. The modeling process of the diffusion-based policy, which comprises the subsequent two stages, is detailed below. 1) Forward Action Diffusion: For every state-action pair (s, a) ∈ D, the diffusion model progressively injects noise into the initial action a0 = a ∼ µ(·|s), transforming it stepby-step into Gaussian noise aId ∼ N (0, I) over Id diffusion steps. At each step, noise is added to satisfy the transition: √ (14) ℏ ai ai−1 = N ai ; αi ai−1 , (1 − αi )I , where αi represents the noise scheduling parameter. From (14), the marginal distribution QId associated with the joint distribution ℏ a1:Id |a0 = i=1 ℏ ai |ai−1 can be formulated analytically as: √ (15) ℏ ai a0 = N ai ; ᾱi a0 , (1 − ᾱi )I , Qi where ᾱi = m=1 αm . By utilizing the reparameterization trick on (15), the following expression is obtained: √ √ ai = ξ 1 − ᾱi + a0 ᾱi , i ∈ {1, 2, · · · , Id }, (16) where ξ ∼ N (0, I) represents a noise sample. 2) Reverse Denoising for Action Generation: The reverse denoising process facilitates the recovery of actions by progressively reducing noise and reconstructing the target distribution from an initial Gaussian sample aId ∼ N (0, I). To this end, the policy distribution is defined as: π(a|s) ≜ ℏ a0:Id s = N aId ; 0, I

Id Y

ℏ ai−1 ai , s ,

(17)

i=1

where ℏ a a , s = N ai−1 ; χ(ai , s, i), φ2 (ai , s, i) characterizes the distribution of a single denoising step. Here, χ(ai , s, i) and φ2 (ai , s, i) denote the corresponding denoising mean and covariance, respectively. i−1

IV. A LGORITHM I MPLEMENTATION Building upon the BAC-PE developed in Section III-A and the diffusion policy scheme introduced in Section III-B, this section formally presents the DPBAC algorithm, with its structural diagram shown in Fig. 1. The detailed algorithm workflow is illustrated in Algorithm 1, and the optimization objectives for all employed networks are elaborated as follows. A. Diffusion-Based Behavior Policy Generation In the DPBAC algorithm, a diffusion model parameterized by ϕ is employed to model the behavior policy µ(·|s), capturing its complex and multimodal data distribution characteristics. Following [38], ℏϕ ai−1 ai , s can be parameterized as

Pre-training: diffusion policy Diffusion policy

µ (φ )

Diffusion-based target policy training Behavior critic

Qθµ

Behavior advantage

Qθ j

µφµ

π (φ ) Policy enhancement

Q-function networks Output: behavior policy

( s, a, r , s ') Pre-collected dataset 

Test and deployment

g (φ )

πφ ( ⋅ s ') Output: diffusion-based target policy parameter φ

Fig. 1. Schematic representation of the developed DPBAC algorithm.

a noise prediction module, with the covariance φ2 ai , s, i = (1 − αi )I and the mean formulated as: 1 1 − αi χϕ (ai , s, i) = √ ai − √ ξϕ (ai , s, i) . (18) αi 1 − ᾱi The practical implementation involves predicting the Gaussian noise ξ in (16) using a neural network ξϕ , which is trained with the following loss function: h i 2 Lµ (ϕ) = Eξ∼N (0,I),(s,a)∼D ξ − ξϕ (ai , s, i) , (19) √ √ where ai = ξ 1 − ᾱi +a0 ᾱi and i ∼ U{1, Id }. As outlined in Algorithm 1, once the behavior policy has been trained using the diffusion policy loss in (19), its parameters are set as ϕµ = ϕ, yielding the parameterized behavior policy µϕµ (·|s′ ). When generating an action a for the current state s, the process begins by sampling Gaussian noise aId ∼ N (0, I), followed by a stepwise reverse denoising procedure based on the following formulation: ai−1 |ai = χϕ ai , s, i + φ ai , s, i ξ, i ∈ {1, · · · , Id }, (20) ultimately yielding the generated action a = a0 . To expedite the action sampling process in (20), the DPM-Solver [41] is utilized to accelerate the reverse denoising process. B. Behavior Critic Training A behavior critic network parameterized by θµ is employed to approximate the Q-function Qµ (s, a) of the behavior policy. The training objective is to minimize the difference between its estimated value and the TD target zµ , with the loss function formulated as follows: h 2 i LµQ (θµ ) = E(s,a,r,s′ )∼D zµ − Qθµ (s, a) , (21) where zµ = r + γQθµ (s′ , ã′ ), and ã′ ∼ µϕµ (·|s′ ). C. Behavioral Advantage Corrected Critic Learning To implement the BAC-PE rule (8), the following loss function is derived from (7) for training the Q-function networks corresponding to the learned policy, each parameterized by θj : h 2 i LπQ (θj ) = E(s,a,r,s′ )∼D zπ − Qθj (s, a) , (22)

Algorithm 1 DPBAC Initialize: Q-function networks parameters θ1 , θ2 , diffusion policy network parameter ϕ, behavior critic network parameter θµ , pre-collected dataset D, target networks parameters for Q-function θ̄1 ← θ1 , θ̄2 ← θ2 , target network parameters for diffusion policy ϕ̄ ← ϕ, target policy update periodicity δ, target network smoothing factor σ, behavior policy training steps nmax , policy training steps Nmax . 1: # Pre-training: diffusion-based behavior policy training 2: for behavior policy training step n = 1, 2, · · · , nmax do 3: Randomly extract a mini-batch from the pre-collected dataset D. 4: Optimize the diffusion policy parameter ϕ via minimizing (19). 5: end for 6: Record the current diffusion policy parameter ϕ as ϕµ , and set ϕ̄ = ϕ. 7: # Diffusion-based learned policy training 8: for policy training step n = 1, 2, · · · , Nmax do 9: Randomly extract a mini-batch from the pre-collected dataset D. 10: Optimize the behavior critic parameter θµ via minimizing (21). 11: Optimize Q-function networks parameters θj via minimizing (22). 12: if δ|n then 13: Optimize the diffusion policy parameter ϕ via minimizing (23). 14: Update the corresponding target network parameters with θ̄j ← (1 − σ)θ̄j + σθj , ϕ̄ ← (1 − σ)ϕ̄ + σϕ. 15: end if 16: end for Output: The diffusion-based learned policy network parameters ϕ.

where the TD target zπ incorporates the reward signal, the discounted estimate of future returns, and the behavioral advantage correction term: zπ = r + γ minj=1,2 {Qθ̄j (s′ , â′ )} + η(Qθµ (s, a) − Qθ̄j (s, a)), with â′ ∼ πϕ̄ (·|s′ ).

Lµ (ϕ), and the performance improvement loss, Lg (ϕ), defined as follows: Lπ (ϕ) = Lµ (ϕ) + ς¯Lg (ϕ),

(23)

scalwhere ς¯ = ς/E(s,a)∼D Qθ1 (s, a) serves as a normalized ing coefficient, and Lg (ϕ) = −Es∼D,a∼πϕ (·|s) Qθ1 (s, a) . Remark 3: In order to improve performance on the benchmark task and fully leverage the knowledge embedded in the pretrained behavior policy, the optimization objective (23) is designed to enable further optimization of the diffusion policy network parameter ϕ. As presented in Section IV-A, the pretrained model provides a strong initialization, facilitating faster convergence and lowering both training time and computational requirements for the target task. V. E XPERIMENTS In this section, we present a comprehensive evaluation of the proposed DPBAC algorithm on the D4RL benchmark [42]. To demonstrate its effectiveness across a diverse range of tasks, we compare the performance of DPBAC with the SOTA offline RL algorithms. In addition, we conduct sensitivity analysis to investigate the individual contributions of each component within the DPBAC framework. Beyond standard benchmark evaluations, we further examine the ability of DPBAC in solving sparse-reward tasks. Finally, we analyze the expressiveness and intrinsic regularization properties of the diffusion-based policy. A. Experimental Configurations

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 2. Representative task environments from the D4RL benchmark used for evaluation. (a) Gym-MuJoCo: halfcheetah. (b) Gym-MuJoCo: hopper. (c) Gym-MuJoCo: walker2d. (d) AntMaze: umaze. (e) AntMaze: medium. (f) AntMaze: large. (g) Adroit: pen. (h) FrankaKitchen: kitchen.

D. Diffusion Actor Improvement To achieve the optimal policy described in (3), the diffusionbased policy network presented in Section IV-A is further enhanced using Q-function guidance. This guidance optimizes the pre-trained diffusion-based policy network parameters by directing the learned policy to prioritize actions with higher Q-values, thereby enabling the generation of a learned policy capable of surpassing the performance of the behavior policy. To this end, the loss function for the learned policy is formulated as a weighted combination of the diffusion policy loss,

1) Dataset for Benchmarking and Evaluation: We evaluate the proposed DPBAC algorithm on four domains from the D4RL benchmark: Gym-MuJoCo, AntMaze, FrankaKitchen, and Adroit. These environments are depicted in Fig. 2, with detailed descriptions provided in Table I. Gym-MuJoCo consists of continuous control tasks with smooth and dense reward functions, making it a widely adopted benchmark for performance evaluation. AntMaze features sparse rewards and requires the agent to navigate an ant robot through a maze by stitching together suboptimal trajectories. FrankaKitchen involves four sequential sub-tasks, testing the algorithm’s ability for long-horizon value optimization and generalization. Adroit, collected from human demonstrations, exhibits narrow state-action distributions and demands strong regularization to prevent policy drift caused by extrapolation errors. 2) Baseline Offline RL Methods for Comparison: To ensure a comprehensive and fair evaluation, we compare the proposed method against a broad set of representative baseline algorithms. Specifically, for value regularization approaches, we include CQL1 , SVR2 , IQL3 , and SCORE4 . For policy regularization methods, we consider BC5 , BCQ6 , and DOGE7 . 1 CQL [22]: https://github.com/aviralkumar2907/CQL. 2 SVR [25]: https://github.com/MAOYIXIU/SVR. 3 IQL [23]: https://github.com/ikostrikov/implicit q learning. 4 SCORE [24]: https://github.com/familyld/SCORE. 5 BC [43]: https://github.com/CherryPieSexy/imitation learning. 6 BCQ [31]: https://github.com/sfujim/BCQ. 7 DOGE [34]: https://github.com/Facebear-ljx/DOGE.

4 6 Training steps

60 SCORE IQL DQL

×105

(f) walker2d-m-r-v2

0 0

BC BCQ CQL DOGE

SCORE IQL DQL

4 6 Training steps

SVR SRPO DPBAC

×105

(k) kitchen-partial-v0

SCORE IQL DQL

SVR SRPO DPBAC

40 20 4 6 Training steps

BC BCQ CQL DOGE

20 0 0

×105

SCORE IQL DQL

4 6 Training steps

×105

60 50 40 30 20 10 0 0

BC BCQ CQL DOGE

(p) antmaze-m-d-v2

SVR SRPO DPBAC

×105

0 0

SCORE IQL DQL

4 6 Training steps

BC BCQ CQL DOGE

SVR SRPO DPBAC

4 6 Training steps

BC BCQ CQL DOGE

(q) antmaze-l-p-v2

×105

SCORE IQL DQL

4 6 Training steps

SCORE IQL DQL

4 6 Training steps

SVR SRPO DPBAC

Average normalized score

×105

SVR SRPO DPBAC

(r) antmaze-l-d-v2

×105

BC BCQ CQL DOGE

SCORE IQL DQL

SVR SRPO DPBAC

20 0 2

BC BCQ CQL DOGE

0 0

SCORE IQL DQL

SVR SRPO DPBAC

4 6 Training steps

×105

SCORE IQL DQL

4 6 Training steps

×105

120 100 80 60 40 20 0 0

SVR SRPO DPBAC

(s) pen-human-v1

BC BCQ CQL DOGE

SCORE IQL DQL

4 6 Training steps

SVR SRPO DPBAC

×105

(o) antmaze-m-p-v2

pen-human-v1

BC BCQ CQL DOGE

antmaze-medium-play-v2

120 100 80 60 40 20 0 0

(j) kitchen-complete-v0

20 0

4 6 Training steps

SVR SRPO DPBAC

(n) antmaze-u-d-v2

0 0

SCORE IQL DQL

×105

0 0

(m) antmaze-u-v2

SVR SRPO DPBAC

BC BCQ CQL DOGE

kitchen-complete-v0

antmaze-large-diverse-v2

SCORE IQL DQL

(e) hopper-m-r-v2

antmaze-umaze-diverse-v2

40 20

×105

(i) walker2d-m-e-v2

60 BC BCQ CQL DOGE

antmaze-umaze-v2

×105

(h) hopper-m-e-v2

SVR SRPO DPBAC

4 6 Training steps

SVR SRPO DPBAC

walker2d-medium-expert-v2

SCORE IQL DQL

4 6 Training steps

antmaze-large-play-v2

(l) kitchen-mixed-v0 Average normalized score

BC BCQ CQL DOGE

100

0 0

antmaze-medium-diverse-v2

120

SVR SRPO DPBAC

100

BC BCQ CQL DOGE

SCORE IQL DQL

100

kitchen-mixed-v0

60 50 40 30 20 10 0 0

0 0

BC BCQ CQL DOGE

(d) halfcheetah-m-r-v2

(g) halfcheetah-m-e-v2 Average normalized score

Average normalized score

BC BCQ CQL DOGE

SCORE IQL DQL

4 6 Training steps

kitchen-partial-v0

60 50 40 30 20 10 0 0

×105

Average normalized score

4 6 Training steps

SVR SRPO DPBAC

hopper-medium-expert-v2

4 6 Training steps

SVR SRPO DPBAC

100

SCORE IQL DQL

0 0

halfcheetah-medium-expert-v2

Average normalized score

walker2d-medium-replay-v2

BC BCQ CQL DOGE

×105

(b) hopper-m-v2

BC BCQ CQL DOGE

Average normalized score

SVR SRPO DPBAC

Average normalized score

0 0

SCORE IQL DQL

Average normalized score

×105

BC BCQ CQL DOGE

Average normalized score

4 6 Training steps

SVR SRPO DPBAC

Average normalized score

SCORE IQL DQL

(a) halfcheetah-m-v2 Average normalized score

Average normalized score

0 0

BC BCQ CQL DOGE

Average normalized score

hopper-medium-replay-v2

100

halfcheetah-medium-replay-v2

Average normalized score

walker2d-medium-v2

×105

pen-cloned-v1

120

Average normalized score

hopper-medium-v2

Average normalized score

halfcheetah-medium-v2

BC BCQ CQL DOGE

100 80

SCORE IQL DQL

SVR SRPO DPBAC

60 40 20 0 0

4 6 Training steps

×105

(t) pen-cloned-v1

Fig. 3. Average normalized scores of DPBAC and SOTA baselines on Gym-MuJoCo, AntMaze, FrankaKitchen, and Adroit tasks. The central line and shaded region for each color represent the mean score curve and the standard deviation, respectively, of the corresponding algorithm across eight random seeds. m=medium, r=replay, e=expert, u=umaze, d=diverse, p=play, l=large.

TABLE I OVERVIEW OF TESTING TASK ENVIRONMENT CONFIGURATIONS Task environments

State/Action dimensions

Gym-MuJoCo: halfcheetah Gym-MuJoCo: hopper Gym-MuJoCo: walker2d AntMaze: umaze AntMaze: medium AntMaze: large Adroit: pen FrankaKitchen: kitchen

17 / 6 11 / 3 17 / 6 27 / 8 27 / 8 27 / 8 45 / 24 59 / 9

Action range − 1.0, 1.0 − 1.0, 1.0 − 1.0, 1.0 − 1.0, 1.0 − 1.0, 1.0 − 1.0, 1.0 − 1.0, 1.0 − 1.0, 1.0

In addition, we include diffusion-based policy approaches, namely SRPO8 and DQL9 , to highlight comparisons with recent diffusion-driven offline RL algorithms. All baseline results are obtained using the official implementations provided by the original authors to ensure reproducibility and 8 SRPO [35]: https://github.com/thu-ml/SRPO. 9 DQL [44]: Diffusion-Policies-for-Offline-RL.

https://github.com/Zhendong-Wang/

consistency. 3) Algorithm Implementation Details: During training, we evaluate the performance of DPBAC every 5000 steps. To mitigate the impact of randomness, all methods are executed eight times with different random seeds, and the average return is reported as the final result. All experiments are conducted on NVIDIA RTX 4090 GPUs with 24 GB of VRAM. The hyperparameter settings for DPBAC are summarized in Table II. Based on the results of the sensitivity analysis presented in Section V-C, we set the diffusion steps Id to 6 for GymMuJoCo, 8 for AntMaze, and 10 for both FrankaKitchen and Adroit. The regularization weight ς is configured as follows: 5 for medium and medium-replay datasets, 1 for medium-expert datasets, 6.5 for AntMaze, 0.01 for FrankaKitchen, and 0.1 for the Adroit domain. B. Score Comparison with SOTA RL Algorithms We evaluate the performance of DPBAC across four task domains and compare it against a set of baseline algorithms. To ensure a fair comparison, we normalize the scores using the protocol provided in [42], where a score of 100 corresponds to expert-level performance. The performance comparisons

Value

Target network smoothing parameter σ Q-function network learning rate Policy network learning rate Behavior policy training steps nmax Policy training steps Nmax Diffusion noise schedule Parameters optimizer Activation functions in all networks Correction coefficient η Mini-batch size Hidden layer neuron count Target policy update periodicity δ Hidden layer count across all networks

5 × 10−3 3 × 10−4 3 × 10−4 2 × 105 1 × 106 cosine Adam Mish 0.35 256 256 5 2

are presented in Fig. 3 and Table III. Specifically, Fig. 3 illustrates the learning curves of all evaluated algorithms, where solid lines represent the average normalized scores over eight random seeds, and shaded areas indicate the standard deviation. Table III reports the mean and standard deviation of the normalized scores over the last ten evaluation steps. Bolded values denote the highest scores for each task, while underlined values indicate the second-best results. Fig. 4 presents a comprehensive comparison of the performance of all baseline methods over the entire suite of tasks. A detailed analysis of the results for each task domain is provided below. -d-v

antm

aze-m

70 68 66 64

0.75 0.95

0.15

0.35

0.55

0.75

0.95

0.35

0.55

0.75

0.95

0.55

0.75

0.95

(a) medium dataset

80 60 40 20 0 5

0.15

v0 d-

0.95

74 72 70 68 66 64

0.75

0.55

ixe

D-QL

0.55

0.15

en kit ch

IQL

0.35

-m

-v 1 an

-v

ned

SVR

0.15

Average normalized score

2 -v l-p eaz tm

an antmaze-l-d-v2

antmaze-u-v2

walker2d-m-r-v2

-e

pen

-clo

r2d alke CQL

0.35

kit

m SCORE

halfcheetah-m-v2

ah 2

-v2 -m-e

80.10

et -e-v

-p-v 2

-v

he lfc ha hop per-m

antm

-m er

er -m

aze-m

-rv2

-v2

Average normalized score

-r-v2 walker 2d-m

tah-m

e che half

79.30

Average normalized score

Hyperparameter

2) Results in AntMaze Domain: The AntMaze domain presents a significant challenge due to its sparse reward signals and the absence of optimal trajectories. Consequently, accurate Q-value estimation is crucial for achieving satisfactory performance. In addition, algorithms must be capable of composing near-optimal trajectories by stitching together suboptimal segments to accomplish goal-directed navigation. As shown in Table III, traditional approaches such as BC and DOGE yield near-zero scores in medium and large maze variants. In contrast, DPBAC obtains the best average AntMaze score and achieves the best or second-best performance on most variants, even in the most complex large maze scenarios. This advantage stems from its reliable Q-value estimation mechanism, which remains effective under sparse-reward conditions. By providing value-guided feedback to the diffusion policy, DPBAC leverages the generative capacity of the diffusion model to generate effective trajectories that approximate optimal behavior.

Average normalized score

TABLE II H YPERPARAMETERS OF DPBAC IN OUR EXPERIMENTS

SRPO

-v1

-par

en itch

k DOGE

(b) medium-replay dataset

tial-

BCQ

DPBAC

1) Results in Gym-MuJoCo Domain: As shown in Fig. 3 and Table III, DPBAC achieves the best performance on the majority of locomotion control tasks. Most baseline methods perform well on the medium-expert datasets, which contain a subset of high-quality trajectories. DPBAC not only matches but further improves upon these results. However, in the medium and medium-replay datasets, where high-quality data is scarce, baseline methods often experience significant performance degradation. In contrast, DPBAC consistently delivers superior results, owing to the high expressiveness of its diffusion-based policy and the effectiveness of the behavioral advantage correction mechanism. These findings demonstrate the robustness and efficacy of the proposed approach.

0 5

0.15

0.35

0.55

Average normalized score

Fig. 4. Radar chart visualization of the averaged normalized scores of various comparative algorithms across all evaluation tasks.

Average normalized score

100

102.70

100

95 90 85 80 75 70

0.75 0.95

0.15

0.35

(c) medium-expert dataset Fig. 5. Sensitivity test on the correction coefficient η and regularization weight ς in Gym-MuJoCo tasks. Results are reported across eight random seeds for η ∈ {0.15, 0.35, 0.55, 0.75, 0.95} and ς ∈ {1, 2, 3, 4, 5} on medium, medium-replay, and medium-expert datasets.

3) Results in FrankaKitchen and Adroit Domains: The Adroit domain is characterized by narrow data distribution and high-dimensional state-action space, making it particularly

TABLE III T HE AVERAGE NORMALIZED SCORES AND CORRESPONDING STANDARD DEVIATIONS OF DPBAC AND SOTA BASELINES ON G YM -M U J O C O , A NT M AZE , F RANKA K ITCHEN , AND A DROIT DURING THE FINAL 10 EVALUATIONS ACROSS 8 RANDOM SEEDS Gym-MuJoCo

SCORE

CQL

SVR

IQL

DQL

SRPO

DOGE

BCQ

DPBAC

halfcheetah-m-v2 42.5 ± 0.4 55.3 ± 0.7 48.0 ± 0.4 55.7 ± 0.3 47.4 ± 0.3 51.8 ± 0.4 55.4 ± 0.7 49.9 ± 0.5 47.9 ± 0.3 56.4 ± 0.6 hopper-m-v2 52.1 ± 4.1 90.9 ± 0.7 73.2 ± 6.4 88.2 ± 10.0 54.6 ± 5.7 79.9 ± 13.0 90.8 ± 2.1 62.7 ± 6.1 58.6 ± 4.3 92.7 ± 6.7 65.3 ± 9.8 87.4 ± 3.6 82.6 ± 1.6 77.6 ± 24.8 76.1 ± 8.5 87.7 ± 0.6 83.2 ± 5.5 84.9 ± 3.2 77.0 ± 4.1 88.8 ± 2.0 walker2d-m-v2 halfcheetah-m-r-v2 35.0 ± 2.9 47.7 ± 0.8 48.1 ± 19.4 49.9 ± 0.7 44.0 ± 0.9 47.8 ± 0.3 49.8 ± 1.4 43.8 ± 1.0 39.7 ± 2.1 50.3 ± 0.8 hopper-m-r-v2 25.6 ± 8.1 95.6 ± 7.0 96.5 ± 2.3 103.3 ± 0.9 57.1 ± 16.7 100.5 ± 0.1 87.9 ± 20.3 31.9 ± 3.4 29.8 ± 6.1 100.6 ± 5.6 walker2d-m-r-v2 17.7 ± 7.6 86.5 ± 3.5 80.7 ± 3.5 88.3 ± 2.2 70.9 ± 13.5 89.3 ± 7.8 88.1 ± 4.3 22.6 ± 7.4 51.5 ± 9.5 89.5 ± 8.4 halfcheetah-m-e-v2 57.1 ± 6.1 72.1 ± 11.8 38.1 ± 8.2 94.5 ± 1.7 88.8 ± 5.2 91.5 ± 0.7 78.1 ± 9.4 85.2 ± 7.0 92.9 ± 2.7 95.0 ± 8.8 hopper-m-e-v2 51.5 ± 4.0 84.5 ± 23.4 72.3 ± 35.5 99.6 ± 1.1 39.2 ± 7.8 101.8 ± 9.2 69.7 ± 27.4 97.6 ± 12.7 83.6 ± 14.1 100.9 ± 15.6 walker2d-m-e-v2 100.5 ± 7.7 111.2 ± 0.7 108.9 ± 3.2 109.4 ± 0.1 108.8 ± 2.5 109.7 ± 0.1 109.5 ± 4.2 100.2 ± 11.2 110.9 ± 0.4 112.1 ± 0.5 Average AntMaze antmaze-u-v2 antmaze-u-d-v2 antmaze-m-p-v2 antmaze-m-d-v2 antmaze-l-p-v2 antmaze-l-d-v2 Average FrankaKitchen

49.7 ± 5.6 81.2 ± 5.8 72.0 ± 8.9 85.2 ± 4.6 65.2 ± 6.8 84.4 ± 3.6 79.2 ± 8.4 BC

SCORE

CQL

SVR

IQL

DQL

SRPO

64.3 ± 5.8

65.8 ± 4.8

87.4 ± 5.4

DOGE

BCQ

DPBAC

49.6 ± 18.2 86.3 ± 13.6 63.2 ± 18.9 0.0 ± 0.0 86.4 ± 14.5 82.3 ± 4.1 53.6 ± 16.6 93.8 ± 8.5 84.8 ± 8.7 95.2 ± 10.6 42.8 ± 22.0 30.8 ± 32.5 67.0 ± 10.5 45.6 ± 13.1 57.1 ± 19.9 62.8 ± 18.9 53.7 ± 41.8 49.5 ± 25.5 46.2 ± 15.2 65.5 ± 23.6 0.0 ± 0.0 60.4 ± 18.3 2.0 ± 3.5 0.0 ± 0.0 15.3 ± 10.1 55.8 ± 3.0 48.8 ± 20.9 0.9 ± 1.6 0.0 ± 0.0 67.4 ± 12.8 0.0 ± 0.0 57.2 ± 22.6 1.0 ± 1.7 0.6 ± 1.1 3.6 ± 5.8 69.5 ± 13.0 68.1 ± 26.0 2.4 ± 3.0 0.1 ± 0.2 61.1 ± 16.1 0.0 ± 0.0 20.4 ± 18.1 0.0 ± 0.0 0.1 ± 0.2 0.3 ± 0.5 36.9 ± 8.0 18.2 ± 45.8 0.0 ± 0.0 0.0 ± 0.0 41.3 ± 14.8 0.0 ± 0.0 27.0 ± 15.2 0.0 ± 0.0 0.1 ± 0.2 0.5 ± 1.0 35.5 ± 9.3 45.8 ± 41.9 0.0 ± 0.0 0.0 ± 0.0 46.8 ± 12.0 15.4 ± 6.7 47.0 ± 20.1 22.2 ± 5.8 BC

SCORE

CQL

7.7 ± 2.4

27.2 ± 8.6 57.1 ± 9.4 48.0 ± 32.2 24.4 ± 6.4

SVR

IQL

DQL

SRPO

21.9 ± 4.0

62.9 ± 15.0

DOGE

BCQ

DPBAC

kitchen-complete-v0 11.2 ± 9.2 kitchen-partial-v0 35.8 ± 6.0 41.2 ± 8.5 kitchen-mixed-v0

0.8 ± 1.1 2.2 ± 2.9 1.1 ± 1.8

7.6 ± 6.9 33.2 ± 10.2 37.8 ± 20.7 55.0 ± 32.0 0.0 ± 0.0 0.0 ± 0.0 0.5 ± 0.8 39.5 ± 17.3 46.8 ± 14.7 6.2 ± 10.8 0.0 ± 0.0 0.4 ± 0.6 33.5 ± 9.6 60.0 ± 7.9 0.0 ± 0.0

0.0 ± 0.0 4.0 ± 5.1 1.0 ± 1.6

2.8 ± 2.9 3.6 ± 2.7 5.4 ± 4.2

55.5 ± 12.9 60.9 ± 6.6 62.7 ± 3.3

Average

29.4 ± 7.9

1.4 ± 1.9

2.5 ± 2.3

1.7 ± 2.2

3.9 ± 3.3

59.7 ± 7.6

Adroit

SCORE

CQL

DOGE

BCQ

DPBAC

pen-human-v1 pen-cloned-v1 Average 1 2

11.4 ± 3.9 36.9 ± 15.9 53.9 ± 18.2 2.1 ± 3.6 SVR

IQL

DQL

SRPO

57.1 ± 20.3 −1.0 ± 4.1 33.0 ± 18.9 67.9 ± 30.1 62.6 ± 28.3 67.2 ± 12.8 −3.0 ± 0.2 13.5 ± 20.2 69.0 ± 17.5 70.1 ± 12.9 9.3 ± 12.9 2.7 ± 7.5 15.9 ± 12.1 53.2 ± 22.1 33.8 ± 24.3 49.1 ± 27.2 3.5 ± 5.3 16.6 ± 11.7 23.0 ± 12.6 54.3 ± 23.1 33.2 ± 16.6 0.9 ± 5.8 24.5 ± 15.5 60.6 ± 26.1 48.2 ± 26.3 58.2 ± 20.0 0.3 ± 2.8

15.1 ± 16.0 46.0 ± 15.1 62.2 ± 18.0

In the Gym-MuJoCo suite of tasks, m denotes medium, r stands for replay, and e refers to expert. In the AntMaze suite of tasks, u represents umaze, d denotes diverse, m stands for medium, p refers to play, and l indicates large.

susceptible to extrapolation errors. Consequently, the policy regularization ability becomes critical for ensuring reliable performance. The strong results achieved by DPBAC on Adroit domain can be attributed to the expressive ability of its diffusion policy, as enforced by the loss term Lµ (ϕ) in (23), which imposes strong regularization and effectively mitigates performance degradation caused by erroneous policy shift. The FrankaKitchen domain poses a long-horizon, multi-stage manipulation challenge that requires algorithms to model multigoal behaviors and address the issue of long-horizon credit assignment. Most existing offline RL algorithms struggle to achieve satisfactory performance in this domain. However, DPBAC demonstrates consistently strong results, underscoring both the applicability and the effectiveness of diffusion-based policies in addressing such complex and compositional tasks. C. Sensitivity Studies and Parameter Analysis To assess the contribution of each component within DPBAC to the overall performance, we conduct a series of sensitivity tests. Specifically, we investigate the impact of three key

hyperparameters: the correction coefficient η, which controls the degree of Q-value correction; the regularization weight ς, which regulates the strength of policy regularization; and the number of diffusion steps Id , which governs the expressiveness and precision of the generative process. Detailed test results for each hyperparameter are presented below. 1) Evaluating the Impact of η and ς via Sensitivity Tests: We conduct the following sensitivity tests by varying η ∈ {0.15, 0.35, 0.55, 0.75, 0.95} and ς ∈ {1, 2, 3, 4, 5}, with the corresponding results depicted in Fig. 5. Overall, across different datasets, the performance of DPBAC exhibits a nonmonotonic trend with respect to η, initially increasing and then decreasing. This phenomenon can be attributed to the trade-off in Q-value correction: a small η provides insufficient correction of the Q-values, while a large η may lead to overcorrection, thereby degrading performance. Although the optimal value of η varies across datasets, η = 0.35 consistently yields strong performance. In addition, we observe that the algorithm’s performance increases with larger values of ς on the medium and medium-replay datasets, but decreases

halfcheetah-medium-v2 hopper-medium-v2

Average normalized score

kitchen-partial-v0 kitchen-mixed-v0

(b) AntMaze

Average normalized score

(a) Gym-MuJoCo

antmaze-medium-play-v2 antmaze-medium-diverse-v2

pen-human-v1 pen-cloned-v1

(d) Adroit

Fig. 6. Sensitivity test on the number of diffusion steps Id , with results averaged over eight random seeds for Id ∈ {4, 6, 8, 10}.

with larger ς on the medium-expert datasets. This is due to the lower quality of trajectories in the former datasets, where weaker policy regularization (i.e., larger ς) allows the policy to deviate more effectively from suboptimal behavior. In contrast, the medium-expert datasets contain higher-quality expert demonstrations, and stronger policy regularization helps preserve desirable behavior, thus enhancing performance. 2) Evaluating the Impact of Id via Sensitivity Tests: To investigate the effect of diffusion steps Id ∈ {4, 6, 8, 10} on algorithm performance across different task domains, we conduct sensitivity tests, with the corresponding results shown in Fig. 6. In the figure, triangles and pentagrams indicate individual experimental outcomes. As illustrated by the box plots, increasing the diffusion steps Id generally leads to improved performance. This trend can be attributed to the enhanced generative capacity of the diffusion policy with longer denoising steps, which facilitates the modeling of more complex action distributions. However, larger values of Id also incur higher computational costs. In high-dimensional domains such as FrankaKitchen and Adroit, selecting a larger Id leads to a more pronounced performance improvement due to the increased complexity of the state-action space. D. Policy Shift Investigation To evaluate the expressiveness and regularization ability of the diffusion policy in DPBAC, we randomly sample 150 stateaction pairs from the dataset, where the actions are regarded as those taken by the behavior policy. For each sampled state, we generate a corresponding action using the trained diffusion policy, referred to as the learned policy action. We then visualize both sets of state-action pairs using t-SNE, as shown in Fig. 7. Circles represent state-action pairs from the behavior policy, while squares represent those generated by the learned policy. To assess the quality of these actions, we

compute the Q-values of all visualized pairs using the trained Q-function network, which are used for color encoding. The visualization reveals that the learned policy closely covers the distribution of the behavior policy, demonstrating the strong expressive capacity and regularization ability of the diffusion model. Moreover, the learned policy’s state-action pairs exhibit generally higher Q-values compared to those of the behavior policy, indicating that DPBAC effectively guides the diffusion policy toward higher-value regions of the action space, thereby achieving superior performance over the behavior policy. E. Behavioral Advantage Study To investigate the impact of the behavioral advantage bµ on policy training, we vary the correction coefficient A e Q η ∈ {0.15, 0.25, 0.35, 0.55, 0.75, 0.95}. Fig. 8 illustrates the evolution of the behavioral advantage throughout training under different η values. It can be observed that stronger correction strengths (i.e., larger values of η) lead to smaller behavioral advantages, indicating that the learned Q-function becomes more closely aligned with the Q-function of the behavior policy. This observation is consistent with our theoretical analysis. A smaller behavioral advantage suggests reduced policy deviation from the behavior policy, which helps stabilize training in offline settings where distributional shift is detrimental. However, an excessively strong correction with a large η may impede the policy’s ability to explore higher-reward regions beyond those induced by the behavior policy, thereby potentially constraining overall performance improvement. F. Sparse Reward Tasks Evaluation To evaluate the ability of DPBAC in handling sparse-reward tasks, we conduct experiments on the maze navigation tasks in the Maze2D domain, which includes four levels of maze complexity: open, umaze, medium, and large. We consider both dense and sparse reward settings. In the dense setting, rewards are distributed stepwise and are inversely proportional to the distance to the goal, whereas in the sparse setting, rewards are provided only upon reaching the goal. We compare DPBAC against the diffusion-based method DQL [44], the value regularization algorithm CQL [22], and the conditional sequence modeling approach QDT [45]. The experimental results are presented in Fig. 9. It can be observed that while CQL and DQL perform well under the dense reward setting, their performance significantly deteriorates under sparse rewards due to inaccurate value estimation. Similarly, QDT, which relies on CQL for return-to-go token relabeling, also struggles to adapt to sparse reward tasks. In contrast, DPBAC demonstrates consistently strong performance across both dense and sparse reward settings. This highlights the effectiveness of the proposed Q-value correction mechanism and the powerful regularization ability of the diffusion policy, which together help mitigate the negative impact of Q-value estimation bias. G. Q-value Estimation Bias Analysis To evaluate the accuracy of Q-value estimation in DPBAC algorithm, Fig. 10 reports the estimation bias of the learned Q-

100

150 100

150

100

400

Q Value

10 5

300

200

5 10

100

Behavior Policy Learned Policy 10 20

20 15 10 5 0 5 10 15 20

(i) walker2d-medium-expert-v2

Behavior Policy Learned Policy

0.6

0.4 0.3

1000 900

800

700

600

500

400

(j) antmaze-umaze-diverse-v2

Q Value

150 100 50 0

0.2

0.0

300

100 50

Behavior Policy Learned Policy

0.8 0.7

0.6 0.5 0.4 0.3 0.2

0.1

(k) antmaze-medium-play-v2

350

(h) hopper-medium-expert-v2

0.4

150

200

250

0.8

Behavior Policy Learned Policy 20 10 0

Behavior Policy Learned Policy

20 15 10 5 0 5 10 15

200

0.6

0.1

250

300

(d) halfcheetah-medium-replay-v2

0.2

0.0

Behavior Policy Learned Policy

350

(g) halfcheetah-medium-expert-v2

0.5

(f) walker2d-medium-replay-v2 500

200

Behavior Policy Learned Policy

250

(e) hopper-medium-replay-v2 25

100

Q Value

300

150

350

Q Value

200

Q Value

250

150

(b) hopper-medium-v2 300

Behavior Policy Learned Policy

250

(a) halfcheetah-medium-v2 20

300

Q Value

200

100

10 20

400

Behavior Policy Learned Policy

Q Value

Behavior Policy Learned Policy 10 20

250

350

Q Value

200

Q Value

300

400

Behavior Policy Learned Policy

300

Q Value

400

Behavior Policy Learned Policy

Q Value

15 10 5 0 5 10 15 20

0.0

(l) antmaze-large-diverse-v2

0 10 20 30 40 50 60 700

halfcheetah-medium-v2

= 0.15 = 0.25

= 0.35 = 0.55

4 6 Training steps

halfcheetah-medium-replay-v2

= 0.75 = 0.95

Behavioral advantage

Fig. 7. Analysis of policy expressiveness and regularization ability: t-SNE visualization of state-action pair distributions from dataset D and those generated by the learned policy.

= 0.35 = 0.55

= 0.75 = 0.95

maze2d-open-dense-v0 maze2d-medium-dense-v1

894.7

D-QL

769.4

= 0.35 = 0.55

= 0.75 = 0.95

QDT

4 6 Training steps

×105

100 150 200 250 0

4 6 Training steps

×105

hopper-medium-replay-v2

Behavioral advantage

= 0.15 = 0.25

maze2d-umaze-v1 maze2d-large-v1

DPBAC

878.9

D-QL

579.4

488.8

QDT

291.7

(b) halfcheetah-medium-replay-v2

halfcheetah-medium-expert-v2

maze2d-open-v0 maze2d-medium-v1

CQL

542.1 0

(a) halfcheetah-medium-v2

maze2d-umaze-dense-v1 maze2d-large-dense-v1

DPBAC

×105

= 0.15 = 0.25

= 0.35 = 0.55

= 0.75 = 0.95

200

400

600

CQL

800

Average normalized score

(a) Dense reward setting

402.8 0

200

400

600

800

Average normalized score

(b) Sparse reward setting

Fig. 9. Test results of algorithm performance on sparse-reward tasks. Evaluations are conducted on Maze2D tasks under both dense and sparse reward settings, with results averaged over eight random seeds.

20 30 0

4 6 Training steps

×105

(d) hopper-medium-replay-v2

Fig. 8. Behavioral advantage curves of DPBAC over the full training process with varying η values.

values under different correction strengths η on the walker2dmedium-v2 and walker2d-medium-expert-v2 datasets. For a given initial state-action pair (s0 , a0 ), the Q-value bias is computed as the absolute difference between the learned Qvalue Qθ1 (s0 , a0 ) and the corresponding ground-truth Q-

value. Following [13], the ground-truth Q-value is estimated using the Monte Carlo method and serves as a reference for assessing the estimation bias of the learned Q-values. As shown in Fig. 10, when η = 0, no correction is applied to the learned Q-values, resulting in a large absolute estimation error. As η increases, this error gradually decreases, indicating that the proposed Q-value correction mechanism effectively alleviates overestimation bias. However, when η becomes excessively large, the bias increases again, suggesting that overly strong correction may lead to underestimation of the Q-values. These results highlight the importance of selecting an appropriate correction strength η to balance the mitigation

×102

1.0

1.4

Q-value bias

0.9 0.8 0.7 0.6

Taking the absolute value on both sides of (24), one has e ⋄ (s, a) − Tb π Q e † (s, a) Tb π Q h i e ⋄ (s′ , a′ ) − Q e † (s′ , a′ ) ≤ γ Es′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q

1.2

1.0

0.8 η=0

η = 0.15

η = 0.35

η = 0.55

(a) walker2d-medium-v2

η = 0.75

η=0

η = 0.15

η = 0.35

η = 0.55

η = 0.75

(b) walker2d-medium-expert-v2

Fig. 10. Statistics of Q-value bias for DPBAC under different correction coefficients η on walker2d-medium-v2 and walker2d-medium-expert-v2 datasets.

e ⋄ (s, a) − Q e † (s, a) +η Q e ⋄ (s′ , a′ ) − Q e † (s′ , a′ ) ≤ γEs′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q e ⋄ (s, a) − Q e † (s, a) +η Q e ⋄ (s′ , a′ ) − Q e † (s′ , a′ ) Q ≤ γ max ′ ′ s ,a

of overestimation bias and the risk of underestimation, thereby enabling more accurate Q-value estimation and more effective policy learning.

e ⋄ (s, a) − Q e † (s, a) + η max Q s,a

e ⋄ (s, a) − Q e † (s, a) , ∀(s, a). ≤γ e max Q

(25)

s,a

where γ e = γ + η. Equation (25) can be reformulated in its vector representation as: VI. C ONCLUSION

To address the pessimistic bias and Q-function overestimation issues in existing offline RL algorithms, this study proposed a BAC-PE method. We established the convergence of BAC-PE and derived both the relationship between its converged Q-function and the true Q-function, and an upper bound on their difference. This upper bound is positively correlated with the distributional difference between the learned policy and the behavior policy. To minimize this distributional discrepancy, we utilized a diffusion model for policy modeling, enhancing the expressive ability of the learned policy to better capture multimodal behavior policies, leading to the development of DPBAC. Extensive experiments, including comparisons with SOTA baselines, sensitivity studies, sparsereward evaluations, and Q-value bias analysis across multiple D4RL task domains, demonstrate the superior performance of DPBAC.

e ⋄ − Tb π Q e† e⋄ − Q e† , Tb π Q ≤γ e Q ∞ ∞

(26)

which indicates Tb π constitutes a contraction mapping, ensure π satisfying Q e π = Tb π Q eπ. ing the existence of a fixed point Q According to (26), one has e k − Tb π Q eπ e k+1 − Q eπ Q = Tb π Q ∞ ∞ e k−1 − Tb π Q eπ ek − Q eπ bπQ T ≤γ e Q = γ e ∞ ∞ e k−1 − Q eπ e0 − Q eπ ≤γ e2 Q ≤ ··· ≤ γ ek+1 Q ∞

∞

, (27)

which demonstrates that the iterative sequence initiated from e 0 converges to the fixed point Q eπ. Q A PPENDIX B P ROOF OF T HEOREM 2 Proof: Based on Theorem 1 and equation (7), the following result holds: e π = Tb π Q eπ Q e π + ηA bµ = Bbπ Q

eπ Q

b πQ eπ + η Q bµ − Q eπ . = r + γP A PPENDIX A P ROOF OF T HEOREM 1

e ⋄ and Q e † represent two distinct Q-functions. Proof: Let Q Based on the iterative rule (8), it follows that:

(28)

From equation (1), it follows that Qπ = B π Qπ = r+γPπ Qπ . Consequently, the vectorized form of reward function r can be expressed as r = (I − γPπ )Qπ , which further leads to the derivation of (28) as: e π = (I − γPπ )Qπ + γ P b πQ eπ + η Q bµ − Q eπ Q e π − γPπ Q eπ = (I − γPπ )Qπ + γPπ Q b πQ eπ + η Q bµ − Q eπ . + γP (29)

Reorganizing the terms in (29) yields: e ⋄ (s, a) − Tb π Q e † (s, a) Tb π Q π e π = (I − γPπ )Qπ + γ P b π − Pπ Q e (I − γPπ )Q e ⋄ (s, a) − Bbπ Q e † (s, a) − η Q e ⋄ (s, a) − Q e † (s, a) = Bbπ Q h i bµ − Q eπ . +η Q (30) e ⋄ (s′ , a′ ) − Q e † (s′ , a′ ) = γEs′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q By left-multiplying both sides of (30) by (I − γPπ )−1 , e ⋄ (s, a) − Q e † (s, a) . −η Q (24) equation (9) is obtained.

A PPENDIX C P ROOF OF T HEOREM 3

A PPENDIX D P ROOF OF L EMMA 2

Proof: According to the analysis about the fixed-point of T π presented in Corollary 1, one has π π Q (s, a) = T π Q (s, a) π µ = B π Q (s, a) + ηAQ π (s, a) h π i = r(s, a) + γEs′ ∼P(·|s,a) Q (s′ , a′ ) + ηAµQπ (s, a). (31)

Proof: By the triangle inequality, it follows that: bµ Qπ − Q

′

bµ ≤ Qπ − Q µ + Q µ − Q ≤ Qπ − Q µ

Substituting (32) into (31) results in π

Q (s, a) = Qπ (s, a) − γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) [Qπ (s′ , a′ )] i h π + γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) Q (s′ , a′ ) + ηAµQπ (s, a), (33)

Q (s, a) − Qπ (s, a) = η Qµ (s, a) − Q (s, a) h π i + γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) Q (s′ , a′ ) − Qπ (s′ , a′ )

(38)

b µ − Qµ = γ P b µQ b µ − γPµ Qµ Q b µQ bµ − γP b µ Qµ + γ P b µ Qµ − γPµ Qµ = γP bµ Q b µ − Qµ + γ P b µ − P µ Qµ = γP b µ −1 P b µ − P µ Qµ . = γ I − γP (39) Applying the maximum norm to both sides of (39) yields:

∞

=γ

bµ I − γP

−1

∞

b µ − P µ Qµ P

∞

(40)

+ ηQ (s, a) − ηQ (s, a) π = η Qπ (s, a) − Q (s, a) + η Qµ (s, a) − Qπ (s, a) h π i + γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) Q (s′ , a′ ) − Qπ (s′ , a′ ) . (34)

b µ − Pµ Qµ in (40) is analyzed as follows: Next, the term P b µ − Pµ Qµ (s, a) = P b µ Qµ (s, a) − Pµ Qµ (s, a) P µ ′ ′ = Es′ ∼bP(·|s,a) Q (s , a ) − Es′ ∼P(·|s,a) Qµ (s′ , a′ ) ′

Computing the absolute value on both sides of (34) yields: π

a′ ∼µ(·|s′ )

′

Z a∼µ(·|s ) b ′ |s, a) − P(s′ |s, a) Ea′ ∼µ(·|s′ ) Qµ (s′ , a′ ) ds′ . = P(s

Q (s, a) − Qπ (s, a)

(41)

≤ η Q (s, a) − Q (s, a) + η Q (s, a) − Q (s, a) h π i + γ Es′ ∼P(·|s,a),a′ ∼π(·|s′ ) Q (s′ , a′ ) − Qπ (s′ , a′ )

Considering the absolute value of both sides of (41), one has:

≤ η Q (s, a) − Qπ (s, a) + η Qµ (s, a) − Qπ (s, a) π

+ γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) Q (s′ , a′ ) − Qπ (s′ , a′ ) h π i ≤ η max Q (s, a) − Qπ (s, a) + Qµ (s, a) − Qπ (s, a) s,a

Q (s′ , a′ ) − Qπ (s′ , a′ ) + γ max ′ ′ s ,a

≤γ e max Q (s, a) − Qπ (s, a) µ

+ η max Q (s, a) − Q (s, a) , s,a

(35)

with its vectorized form represented as: π

Q − Qπ ∞ ≤ γ e Q − Q π ∞ + η Qµ − Q π ∞ η ≤ Qµ − Q π ∞ . (36) 1−γ e Following Lemma 1, (11) is straightforwardly obtained.

s,a

(37)

which implies that

b µ − Qµ Q

Qµ = B µ Qµ = r + γPµ Qµ , b µ = Bbµ Q bµ = r + γP b µQ bµ, Q

which implies that

∞

′

r(s, a) = Q (s, a) − γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) [Q (s , a )]. (32)

∞

bµ + Qµ − Q

Based on the definitions of B π in (1) and Bbπ in (2), one has

a′ ∼π(·|s′ )

Based on the fixed-point equation of Bellman operator B π in (1), it follows that Qπ (s, a) = (B π Qπ )(s, a) = r(s, a) + γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) [Qπ (s′ , a′ )], which leads to

∞

Due to the property maxs,a |Qµ (s, a)| < r/(1 − γ), it can be deduced from (42) that h i 2r b b µ − P µ Qµ P ≤ max ΥT V P(·|s, a) P(·|s, a) . 1 − γ s,a ∞ (43) By synthesizing (10), (40), and (43), equation (12) can be derived from (37).

With the property that maxs,a |Qπ (s, a)| < r/(1 − γ), (48) can be reformulated as:

A PPENDIX E P ROOF OF T HEOREM 4 Proof: From the fixed-point equation of Tb π presented in Theorem 1, one can derive: e π (s, a) = Tb π Q e π (s, a) Q bµ (s, a) e π (s, a) + η A = Bbπ Q eπ Q h i e π (s′ , a′ ) = r(s, a) + γEs′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q bµ (s, a). + ηA eπ Q

(44)

e π − Qπ Q

b µ − Qπ +η Q ∞ h i 2γr b + max ΥT V P(·|s, a) P(·|s, a) 1 − γ s,a h i 2γr b ≤ max ΥT V P(·|s, a) P(·|s, a) (1 − γ)(1 − γ e) s,a η b µ − Qπ . (49) Q + 1−γ e ∞ ∞

e π − Qπ ≤γ e Q

∞

By combining (12) with (49), (13) can be obtained.

Incorporating (32) into (44) leads to: e π (s, a) = Qπ (s, a) − γEs′ ∼P(·|s,a),a′ ∼π(·|s′ ) Qπ (s′ , a′ ) Q π ′ ′ e (s , a ) + η A bµ (s, a). (45) + γE ′ b Q ′ ′

R EFERENCES

[1] R. S. Sutton and A. G. Barto, Reinforcement Learning. Cambridge, MA, USA: MIT Press, 1998. [2] Y. Feng, Z. Wu, J. Wang, S. Li, Y. Huang, J. Yu, and M. Tan, “Decentralized reinforcement learning for multiple robotic fish in cooperative It can be observed that pursuit task,” IEEE Trans. Cogn. Develop. Syst., vol. 17, no. 4, pp. h i h i 1022–1034, Aug. 2025. e π (s′ , a′ ) − Es′ ∼P(·|s,a) Qπ (s′ , a′ ) Es′ ∼bP(·|s,a) Q [3] C. Sun, X. Wu, Y. Su, X. Shi, and C. Sun, “Multithreaded asynchronous ′ ′ ′ ′ a ∼π(·|s ) a ∼π(·|s ) deep reinforcement learning with multisensor fusion for robot collision i h i h avoidance,” IEEE Trans. Neural Netw. Learn. Syst., vol. 36, no. 9, pp. π ′ ′ π ′ ′ e = Es′ ∼bP(·|s,a) Q (s , a ) − Es′ ∼bP(·|s,a) Q (s , a ) 16128–16142, Sep. 2025. ′ ′ ′ ′ a ∼π(·|s ) a ∼π(·|s ) [4] C. Pan, Z. Peng, Y. Li, B. Han, and D. Wang, “Flocking of underh i h i actuated unmanned surface vehicles via deep reinforcement learning and + Es′ ∼bP(·|s,a) Qπ (s′ , a′ ) − Es′ ∼P(·|s,a) Qπ (s′ , a′ ) model predictive path integral control,” IEEE Trans. Instrum. Meas., vol. ′ ′ a′ ∼π(·|s′ ) 73, 2024, Art. no. 2505011. Z a ∼π(·|s ) [5] Y. Liu, Q. Zhang, Y. Gao, and D. Zhao, “Deep-reinforcement-learning π ′ ′ ′ ′ ′ b |s, a) − P(s |s, a) Ea′ ∼π(·|s′ ) Q (s , a ) ds = P(s based driving policy at intersections utilizing lane graph networks,” IEEE Trans. Cogn. Develop. Syst., vol. 16, no. 5, pp. 1759–1774, Oct. 2024. h i [6] L.-Y. Hao, G. Dong, T. Li, and Z. Peng, “Path-following control with e π (s′ , a′ ) − Qπ (s′ , a′ ) . (46) + Es′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q obstacle avoidance of autonomous surface vehicles subject to actuator faults,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 4, pp. 956–964, Apr. Building on (46), (45) can be derived as follows: 2024. h i [7] X. Huang, Y. Cheng, Q. Yu, and X. Wang, “Deep reinforcement learning π ′ ′ for autonomous driving based on safety experience replay,” IEEE Trans. e π (s, a) − Qπ (s, a) = γE ′ b eπ ′ ′ Q s ∼P(·|s,a) Q (s , a ) − Q (s , a ) Cogn. Develop. Syst., vol. 16, no. 6, pp. 2070–2084, Dec. 2024. ′ ′ a ∼π(·|s ) [8] O. Dogru, J. Xie, O. Prakash, R. Chiplunkar, J. Soesanto, H. Chen, Z K. Velswamy, F. Ibrahim, and B. Huang, “Reinforcement learning in b ′ |s, a) − P(s′ |s, a) Ea′ ∼π(·|s′ ) Qπ (s′ , a′ ) ds′ +γ P(s process industries: review and perspective,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 2, pp. 283–300, Feb. 2024. e π (s, a) + Q b µ (s, a) − Qπ (s, a) . [9] X. Hai, Q. Feng, W. Chen, C. Wen, and A. W. Khong, “Capability+ η Qπ (s, a) − Q (47) oriented decision-making in multi-uav deployment and task allocation: A hierarchical game-based framework,” IEEE Trans. Syst., Man, Cybern., Considering the absolute value of both sides of (47), one has: Syst., vol. 55, no. 7, pp. 4562–4574, Jul. 2025. [10] W. Huang, Y. Cui, H. Li, and X. Wu, “Practical probabilistic modele π (s, a) − Qπ (s, a) ≤ η Q e π (s, a) − Qπ (s, a) based reinforcement learning by integrating dropout uncertainty and Q trajectory sampling,” IEEE Trans. Neural Netw. Learn. Syst., vol. 36, Z π ′ ′ ′ no. 7, pp. 12812–12826, Jul. 2025. ′ ′ b |s, a) − P(s |s, a) Ea′ ∼π(·|s′ ) Q (s , a ) ds +γ P(s [11] S. Wei, X. Wang, X. Feng, and H. Yu, “MAST: Multiagent safe transformer for reinforcement learning,” IEEE Trans. Cogn. Develop. h i Syst., vol. 17, no. 4, pp. 976–986, Aug. 2025. e π (s′ , a′ ) − Qπ (s′ , a′ ) + γ Es′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q [12] Y. Yang, Z. Ding, R. Wang, H. Modares, and D. C. Wunsch, “Datadriven human-robot interaction without velocity measurement using offb µ (s, a) − Qπ (s, a) +η Q policy reinforcement learning,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 47–63, Jan. 2022. e π (s, a) − Qπ (s, a) + η Q b µ (s, a) − Qπ (s, a) [13] L. Huang, B. Dong, W. Xie, and W. Zhang, “Offline reinforcement ≤η Q learning with behavior value regularization,” IEEE Trans. Cybern., vol. Z 54, no. 6, pp. 3692–3704, Jun. 2024. b ′ |s, a) − P(s′ |s, a) Ea′ ∼π(·|s′ ) Qπ (s′ , a′ ) ds′ +γ P(s [14] K. Jiang, W. Liu, Y. Wang, L. Dong, and C. Sun, “Discovering latent h i variables for the tasks with confounders in multi-agent reinforcement e π (s′ , a′ ) − Qπ (s′ , a′ ) learning,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 7, pp. 1591–1604, + γEs′ ∼bP(·|s,a),a′ ∼π(·|s′ ) Q Jul. 2024. h i [15] R. F. Prudencio, M. R. O. A. Maximo, and E. L. Colombini, “A b ≤ 2γΥT V P(·|s, a) P(·|s, a) max Qπ (s, a) survey on offline reinforcement learning: Taxonomy, review, and open s,a problems,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 8, pp. π π e + (γ + η) max Q (s, a) − Q (s, a) 10237–10257, Aug. 2024. s,a [16] J. Wang, J. Zhang, H. Jiang, J. Zhang, L. Wang, and C. Zhang, “Offline meta reinforcement learning with in-distribution online adaptation,” in b µ (s, a) − Qπ (s, a) . + η max Q (48) Proc. Int. Conf. Mach. Learn., vol. 202, 2023, pp. 36626–36669. s,a s ∼P(·|s,a),a ∼π(·|s )

eπ Q

[17] J. Liu, Z. Zhang, Z. Wei, Z. Zhuang, Y. Kang, S. Gai, and D. Wang, “Beyond ood state actions: Supported cross-domain offline reinforcement learning,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 12, 2024, pp. 13945–13953. [18] S. Hu, Z. Fan, C. Huang, L. Shen, Y. Zhang, Y. Wang, and D. Tao, “Q-value regularized transformer for offline reinforcement learning,” in Proc. Int. Conf. Mach. Learn., 2024, pp. 19165–19181. [19] Z. Yuan, Z. Zhang, X. Li, Y. Cui, M. Li, and X. Ban, “Controlling partially observed industrial system based on offline reinforcement learning–a case study of paste thickener,” IEEE Trans. Ind. Informat., vol. 21, no. 1, pp. 49–59, Jan. 2025. [20] N. Pang, L. Huang, B. Dong, H. Chen, X. Wang, and W. Zhang, “Sarsaaugmented off-policy reinforcement learning,” IEEE Trans. Cogn. Develop. Syst., vol. 18, no. 2, pp. 504–517, Apr. 2026. [21] Y. Zhang, J. Liu, C. Li, Y. Niu, Y. Yang, Y. Liu, and W. Ouyang, “A perspective of q-value estimation on offline-to-online reinforcement learning,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 15, 2024, pp. 16908–16916. [22] A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative Q-learning for offline reinforcement learning,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 1179–1191. [23] I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit Q-learning,” in Proc. Int. Conf. Learn. Represent., 2021, pp. 1–13. [24] Z. Deng, Z. Fu, L. Wang, Z. Yang, C. Bai, T. Zhou, Z. Wang, and J. Jiang, “False correlation reduction for offline reinforcement learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 2, pp. 1199–1211, Feb. 2024. [25] Y. Mao, H. Zhang, C. Chen, Y. Xu, and X. Ji, “Supported value regularization for offline reinforcement learning,” in Proc. Adv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 40587–40609. [26] J. Shao, Y. Qu, C. Chen, H. Zhang, and X. Ji, “Counterfactual conservative q learning for offline multi-agent reinforcement learning,” in Proc. Adv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 77290–77312. [27] C. Ma, D. Yang, T. Wu, Z. Liu, H. Yang, X. Chen, X. Lan, and N. Zheng, “Improving offline reinforcement learning with in-sample advantage regularization for robot manipulation,” IEEE Trans. Neural Netw. Learn. Syst., vol. 36, no. 6, pp. 11215–11227, Jun. 2025. [28] R. Zhang, Z. Luo, J. Sjölund, T. Schön, and P. Mattsson, “Entropyregularized diffusion policy with q-ensembles for offline reinforcement learning,” in Proc. Adv. Neural Inf. Process. Syst., vol. 37, 2024, pp. 98871–98897. [29] S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforcement learning,” in Proc. Adv. Neural Inf. Process. Syst., vol. 34, 2021, pp. 20132–20145. [30] G. Peng, J. Yang, X. Li, and M. O. Khyam, “Deep reinforcement learning with a stage incentive mechanism of dense reward for robotic trajectory planning,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 53, no. 6, pp. 3566–3573, Jun. 2023. [31] S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in Proc. Int. Conf. Mach. Learn., vol. 97, 2019, pp. 2052–2062. [32] Y. Ran, Y.-C. Li, F. Zhang, Z. Zhang, and Y. Yu, “Policy regularization with dataset constraint for offline reinforcement learning,” in Proc. Int. Conf. Mach. Learn., vol. 202, 2023, pp. 28701–28717. [33] Z. Zhang and X. Tan, “An implicit trust region approach to behavior regularized offline reinforcement learning,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 15, 2024, pp. 16944–16952. [34] J. Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y. Q. Zhang, “When data geometry meets deep function: Generalizing offline reinforcement learning,” in Proc. Int. Conf. Learn. Represent., 2023, pp. 1–35. [35] H. Chen, C. Lu, Z. Wang, H. Su, and J. Zhu, “Score regularized policy optimization through diffusion behavior,” in Proc. Int. Conf. Learn. Represent., 2024, pp. 1–20. [36] S. Cao, X. Wang, and Y. Cheng, “Dual behavior regularized offline deterministic actor-critic,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 54, no. 8, pp. 4841–4852, Aug. 2024. [37] Z. Zhou, G. Liu, and M. Zhou, “A robust mean-field actor-critic reinforcement learning against adversarial perturbations on agent states,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 10, pp. 14370– 14381, Oct. 2024. [38] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 6840–6851. [39] J. Zhang, Y. Cheng, S. Cao, and X. Wang, “Offline reinforcement learning with reverse diffusion guide policy,” IEEE Trans. Ind. Informat., vol. 20, no. 10, pp. 11785–11793, Oct. 2024.

[40] L. Huang, B. Dong, and W. Zhang, “Efficient offline reinforcement learning with relaxed conservatism,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5260–5272, Aug. 2024. [41] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps,” in Proc. Adv. Neural Inf. Process. Syst., vol. 35, 2022, pp. 5775–5787. [42] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, “D4RL: Datasets for deep data-driven reinforcement learning,” 2020, arXiv:2004.07219. [43] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in Proc. Joint Conf. Artif. Intell., 2018, pp. 4950–4957. [44] Z. Wang, J. J. Hunt, and M. Zhou, “Diffusion policies as an expressive policy class for offline reinforcement learning,” in Proc. Int. Conf. Learn. Represent., 2023, pp. 1–17. [45] T. Yamagata, A. Khalil, and R. Santos-Rodriguez, “Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl,” in Proc. Int. Conf. Mach. Learn., vol. 202, 2023, pp. 38989–39007.

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

Analyze & explore related research in Conceptio

Related documents