Offline Deep Q* Estimation with Diffusion Models

Offline Deep Q∗ Estimation with Diffusion Models Xiaohong Chen ∗

Yuling Jiao †

Lican Kang ‡

Jerry Zhijian Yang §

arXiv:2608.14401v1 [stat.ML] 14 Aug 2026

Chen Zhong ¶

Abstract In offline reinforcement learning (RL), estimating the optimal action-value function Q∗ can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and transition kernel are unknown, so the optimal Bellman operator is not directly observable from data. To address this issue, we propose a novel framework that decouples operator estimation from value function learning. In this approach, we first formulate conditional diffusion models to estimate the reward law and transition kernel, which induces a data-driven approximation of the optimal Bellman operator. We then plug these estimators into the Bellman equation and obtain a deep estimator of Q∗ by minimizing the empirical Bellman residual over a neural network function class. This formulation separates high-dimensional conditional distribution estimation from the fixed point estimation problem and yields a coherent end-to-end statistical framework. In particular, conditional diffusion models provide a flexible and expressive approach to modeling unknown environment dynamics under limited data, addressing a central ∗ †

Department of Economics, Yale University, New Haven, 06511, USA. Email: xiaohong.chen@yale.edu School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University,

Wuhan, 430072, China. Email: yulingjiaomath@whu.edu.cn ‡ Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. Email: kanglican@whu.edu.cn § School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. Email: zjyang.math@whu.edu.cn ¶ School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China. math@whu.edu.cn

1

Email:

zc-

challenge in offline RL. Theoretically, we first establish sharp nonasymptotic convergence rates for learning the optimal Bellman operator through an end-to-end analysis of conditional diffusion estimation in total variation distance. We then establish the 2β
e n− dx +da +2β for the excess Bellman residual risk. Finally, oracle value-stage rate O under a concentrability condition, we translate this residual bound into an L2 converβ
e n− dx +da +2β for the resulting deep estimator of Q∗ , where dx and da gence rate of O denote the dimensions of the state and action spaces, respectively, and β denotes the Hölder smoothness index of Q∗ . Importantly, our theoretical analysis does not rely on completeness assumptions commonly used in deep RL theory. Extensive numerical experiments demonstrate the effectiveness of the proposed method and its strong empirical performance. KEY WORDS: Deep RL, Bellman residual minimization, Deep Q∗ estimation, Conditional diffusion model.

1

Introduction

In deep reinforcement learning (DRL), value functions or policies are represented by deep neural networks (DNNs), which makes RL applicable to high-dimensional control and planning problems (Mousavi et al., 2016; Li, 2023). The empirical success of DRL in Atari games (Mnih et al., 2015) and Go (Silver et al., 2016) demonstrates that deep function approximation can support effective decision-making in complex environments. At the same time, many practical problems do not permit repeated online interaction with the environment, because data collection may be costly, unsafe, or otherwise infeasible. In such settings, the learner must rely on previously collected data, often generated by an unknown behavior policy (Levine et al., 2020; Fujimoto et al., 2019; Prudencio et al., 2023). This offline regime fundamentally changes the nature of the problem: the primary challenge is no longer exploration, but the reliable estimation of value functions or policies from limited and potentially biased data. From the value function modeling perspective, a central object in offline RL is to estimate the optimal action-value function Q∗ , defined as the fixed point of the optimal Bellman equa-

2

tion. Classical fitted-value methods, including fitted Q-iteration (FQI) (Ernst et al., 2005) and its deep extensions, such as deep Q-networks (DQN) (Mnih et al., 2015) and deep fitted Q-iteration (DFQI) (Fan et al., 2020), approximate Q∗ through repeated Bellman regression with DNNs. These methods directly connect sampled transitions with value estimation, but recursive bootstrapping can propagate approximation and optimization errors, and maxbased targets in deep Q-learning may induce overestimation bias (Van Hasselt et al., 2016; Fan et al., 2020). Deep approximate policy iteration (DAPI) (Jiao et al., 2025a) follows a related route based on the Bellman equation, where each policy evaluation step solves a Bellman equation over a DNN class. These iterative methods provide a natural approach to value learning, but repeated fitted regression or policy evaluation steps can be computationally demanding when many trajectories or repeated Bellman solves are required. Another line focuses on direct value estimation. Methods such as Minimax Squared Bellman Optimality Error Minimization (MSBO) and Minimax Average Bellman Optimality Error Minimization (MABO) estimate Q∗ through adversarial Bellman-error formulations (Xie and Jiang, 2020). These minimax formulations provide a direct statistical criterion for value estimation and can be applied to batch data, but they usually require solving coupled optimization problems over multiple function classes, which can be computationally demanding. Theoretical analyses of these methods typically rely on concentrability coefficients and Bellman completeness assumptions, which respectively control distribution mismatch and the representability of Bellman updates within a prescribed function class (Chen and Jiang, 2019; Fan et al., 2020; Feng et al., 2023; Jiao et al., 2025a). While these assumptions have enabled important nonasymptotic guarantees, they may become restrictive when flexible DNN approximators are used in offline RL, particularly in high-dimensional settings where Bellman updates may no longer be accurately represented within a fixed function class. A complementary direction is model-based RL and its offline variants, including ModelBased Policy Optimization (MBPO) (Janner et al., 2019), Model-based Offline Policy Optimization (MOPO) (Yu et al., 2020), MOReL: Model-Based Offline RL (Kidambi et al., 2020), and Conservative Offline Model-Based Policy Optimization (COMBO) (Yu et al., 2021). Model-based methods make the environment mechanism explicit by estimating rewards or transitions, and therefore provide a natural alternative to directly fitting value 3

functions. However, existing model-based offline RL methods often use the learned model for rollout generation, uncertainty penalization, or conservative policy optimization. Their performance therefore depends heavily on the quality and stability of the learned environment model, especially in high-dimensional and continuous settings. Taken together, these approaches highlight a fundamental tradeoff: direct value-learning methods remain close to the Bellman equation but rely on recursive or coupled optimization, whereas model-based methods make the environment mechanism explicit but depend on the quality of reward and transition estimation. This tradeoff suggests a statistical perspective that separates environment modeling from value estimation. Since the Bellman equation is fully determined by the reward law and transition kernel, it is natural to first estimate these conditional objects and then use them to construct an estimator of Q∗ . Related ideas appear in inverse RL (IRL), where demonstrations are used to recover reward information (Ziebart et al., 2008; Wulfmeier et al., 2015; Arora and Doshi, 2021), and in conditional moment and instrumental variable models, where nuisance components are estimated and then incorporated into a target equation (Dikkala et al., 2020; Xu et al., 2020; Kim et al., 2025). From this viewpoint, offline RL can be formulated as a statistical problem in which the propagation of estimation error from environment modeling to value estimation becomes an explicit object of analysis. The main difficulty in this formulation lies in estimating high-dimensional conditional distributions for rewards and transitions. Diffusion models (Ho et al., 2020; Song et al., 2020) provide a flexible generative framework for learning complex conditional distributions, and recent theoretical work establishes strong statistical guarantees for diffusion-based estimators (Oko et al., 2023; Chen et al., 2023b; Jiao et al., 2024; Chang et al., 2026). Among these results, Tang et al. (2025) established minimax optimal rates in total variation distance for conditional distribution estimation and proved manifold adaptivity under the Wasserstein metric. Our analysis further quantifies how estimation errors in the conditional reward and transition laws propagate through the optimal Bellman operator to the estimator of Q∗ . In particular, diffusion models are well suited for complex and multimodal distributions, which makes them attractive for modeling conditional reward laws and transition kernels in continuous offline RL. Recently, diffusion models have also become an active direction in RL, with 4

applications to conditional decision making and trajectory generation (Ajay et al., 2023), offline policy learning and behavior regularization (Wang et al., 2022; Chen et al., 2023b; Mao et al., 2024), value estimation (Mazoure et al., 2023), and policy optimization in online or continuous-control settings (Ren et al., 2024; Ma et al., 2025; Frans et al., 2025). Motivated by these developments, we propose a framework for offline Q∗ estimation that integrates conditional diffusion modeling with value function learning. We use conditional diffusion models to estimate the reward law and transition kernel, which define a data-driven approximation of the optimal Bellman operator. These estimates are then incorporated into b is obtained via empirical Bellman residual minimization over a the Bellman equation, and Q neural network function class. This construction yields an end-to-end statistical framework in which the interaction between generative estimation and value learning can be rigorously analyzed. In theoretical analysis, we establish convergence rates for the proposed deep Q∗ estimator. In particular, we first derive convergence rates for the conditional diffusion estimators and then quantify how these errors propagate through the Bellman operator and affect value function estimation. This yields an excess Bellman residual risk bound of order 2β

e − dx +da +2β ), where n is the sample size used for value function learning, dx and da are the O(n dimensions of the state and action spaces, respectively, and β denotes the Hölder smoothness index of the target value function. Furthermore, under a concentrability condition, we obtain β

an L2 convergence rate O(n− dx +da +2β ) for value function estimation. Notably, our analysis avoids the Bellman completeness assumption commonly imposed in theoretical analyses of DRL (Chen and Jiang, 2019; Fan et al., 2020; Feng et al., 2023; Jiao et al., 2025a); see Section 4 for details. The key point is that our framework for Bellman operator learning and loss decomposition reduces the problem to approximating Q∗ of the optimal Bellman operator, rather than requiring the Bellman updates of an entire function class to be representable within the same class. Empirically, we conduct numerical experiments on OpenAI Gym tasks, including Pendulum and Swimmer (https://gymnasium.farama.org/) and on the Medical Information Mart for Intensive Care III (MIMIC-III) clinical dataset (https:// physionet.org/content/mimiciii/1.4/). We compare our method with MOPO, DFQI, and MABO in terms of both training rewards and evaluation rewards. The results support the benefit of separating environment-law estimation from value estimation in offline RL. 5

1.1

Contribution

Our main contributions are summarized as follows: • We propose a framework for offline RL that integrates conditional diffusion modeling with value function estimation. Conditional diffusion models are used to estimate the reward law and transition kernel, which induce a data-driven approximation of the optimal Bellman operator. These estimates are incorporated into a Q∗ estimation procedure via empirical Bellman residual minimization over a DNN class. Extensive numerical experiments demonstrate the effectiveness of the proposed method. • We establish rigorous theoretical guarantees for the proposed framework. Specifically, we derive sharp convergence rates for the conditional diffusion estimators in total variation distance, and obtain oracle value-stage rates for the resulting estimator of Q∗ , as summarized in Theorem 1.1. Moreover, our analysis avoids the Bellman completeness assumption commonly imposed in theoretical analyses of deep RL, while explicitly characterizing how the generative estimation error propagates through the Bellman operator and interacts with the value estimation error. b be the proposed estimator of Q∗ . Under some regular Theorem 1.1 (Informal). Let Q conditions, the Bellman residual satisfies h i b − LT ∗ (Q∗ ) ≲ E LT ∗ (Q)

εopt |{z}

+

Optimal Bellman operator learning error

ε |value {z }

.

Value function learning error

In particular, the operator learning error εopt consists of the reward law estimation error α α

1+dx +da +2α e n− e nx− 2dx +da +2α , and the Monte Carlo O , the transition kernel estimation error O r dx +da
e ne dx +da +2β induced by approximating the learned Bellman operator. Meanwhile, the error O m
2β e n− dx +da +2β . By choosing appropriate value function estimation error satisfies εvalue = O sample sizes nr , nx , and Monte Carlo sample size m, the Bellman residual achieves conver2β

e − dx +da +2β ). Furthermore, under a concentrability condition, we have gence rate O(n β

b − Q∗ ∥L2 (ρ) ≲ n− dx +da +2β . E∥Q

6

1.2

Related Work

We review the literature on DRL and diffusion-based methods in RL. Deep RL. DRL uses DNNs to represent value functions or policies, and has been widely studied for high dimensional control and planning problems (Mousavi et al., 2016; Li, 2023). A basic route to value function modeling is to approximate the optimal action-value function Q∗ through Bellman regression. FQI casts Bellman updates as supervised regression problems (Ernst et al., 2005), and DFQI extends this procedure to a class of DNNs (Fan et al., 2020; Kang et al., 2023). DQN combines the fitted-value idea with deep representation learning, experience replay, and target networks, achieving strong performance on Atari games (Mnih et al., 2015; Bellemare et al., 2013; Hester et al., 2018; Fan et al., 2020). DAPI follows a related iterative framework in which each iteration solves a Bellman equation over a DNN class to obtain an updated value function estimator (Jiao et al., 2025a). However, these methods rely on recursive fitted targets, so approximation and optimization errors may accumulate across iterations. In addition, max-based updates may cause overestimation bias, which Double DQN reduces by separating action selection from value evaluation (Van Hasselt et al., 2016). Despite these improvements, DQN-type methods are mainly suited to discrete action spaces. Existing non-asymptotic theory for deep value learning has established convergence guarantees under concentrability coefficients and Bellman completeness assumptions (Chen and Jiang, 2019; Fan et al., 2020; Feng et al., 2023; Jiao et al., 2025a). These assumptions control distribution mismatch and the propagation of recursive Bellman approximation errors. In contrast, our framework separates estimation of the reward and transition laws from value function learning. The limitation of DQN-type methods in continuous action spaces motivates policygradient and actor-critic (AC) methods. In AC methods, the actor updates the policy, while the critic estimates value functions through Bellman-type equations (Konda and Tsitsiklis, 1999; Grondman et al., 2012). Proximal Policy Optimization (PPO) stabilizes policygradient updates using a clipped surrogate objective (Schulman et al., 2017). Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) extend AC learning to continuous-control tasks (Haarnoja et al., 2018). Although these methods improve policy7

learning flexibility, their critics still face the challenge of controlling fitted value function errors. The error propagation of fitted Bellman targets motivates another line of work that seeks more direct criteria for estimating Q∗ . MSBO and MABO formulate this task through minimax Bellman optimality error objectives, replacing simple recursive regression with adversarial Bellman error minimization (Xie and Jiang, 2020). Minimax Bellman error ideas have also been extended to heterogeneous offline RL; Miao et al. (2025) estimate individual Q functions using minimax Bellman error criteria and combine them with pessimistic personalized policy learning. These formulations provide a clearer statistical criterion for value estimation. However, they typically require solving coupled optimization problems over multiple function classes, which can be computationally demanding and sensitive to optimization design (Uehara et al., 2020; Xie and Jiang, 2020). A complementary response is to model the environment mechanism before computing values or policies. MBPO learns an ensemble dynamics model and uses short rollouts generated by the model to augment real data for policy optimization, reducing the accumulation of model error over long horizons (Janner et al., 2019). In offline RL, MOPO introduces rewards penalized by uncertainty to discourage rollouts through unreliable model regions (Yu et al., 2020). MOReL constructs a pessimistic MDP by detecting uncertain state action pairs and assigning conservative values outside the reliable region of the model (Kidambi et al., 2020). COMBO combines data generated by the model with conservative value learning to use synthetic rollouts while controlling overestimation on poorly modeled samples (Yu et al., 2021). These methods show that estimating rewards and transitions is a useful alternative to direct value regression. Our work follows this environment modeling perspective, but focuses on estimating the Bellman operator from reward laws and transition kernels and analyzing the resulting excess Bellman loss, rather than on policy optimization with learned rollouts. Diffusion RL. Diffusion models have recently been used in RL as flexible generative tools across several settings. They have been applied to conditional decision making and trajectory generation (Ajay et al., 2023; Zhu et al., 2023), policy representation and behavior regularization in offline RL (Wang et al., 2022; Lu et al., 2023; Chen et al., 2022a, 2024a; Mao et al., 2024), value estimation (Mazoure et al., 2023), and policy improvement or online

8

optimization for continuous control (Ren et al., 2024; Fang et al., 2024; Wang et al., 2024; Ma et al., 2025; Frans et al., 2025). For offline decision making, Decision Diffuser treats decision making as conditional generation of state trajectories. It conditions the generated trajectories on returns, constraints, or skills and recovers actions using an inverse dynamics model (Ajay et al., 2023). Related diffusion methods use conditional generation or guidance to model complex behavior, select actions with high value, or improve offline policy learning (Zhu et al., 2023; Lu et al., 2023; Chen et al., 2022a, 2024a; Mao et al., 2024). Another group of methods uses diffusion models to represent policies or behavior distributions. Diffusion Q-learning represents the policy by a conditional diffusion model and combines behavior cloning with maximization of the Q function (Wang et al., 2022). Contrastive energy prediction learns a guidance function for diffusion sampling and applies it to offline RL (Lu et al., 2023). Selecting from behavior candidates uses a generative behavior model together with action evaluation to avoid actions outside the support of the offline data. Score regularized policy optimization uses the score function of a pretrained diffusion behavior model to regularize policy optimization (Chen et al., 2022a, 2024a). Diffusion DICE uses diffusion guidance within the support of the behavior distribution to move this distribution toward the optimal policy distribution (Mao et al., 2024). These methods use diffusion primarily for action or trajectory generation within policy learning. Another line combines diffusion policies with RL optimization and policy improvement. Diffusion Policy Policy Optimization fine tunes diffusion policies through policy gradient updates for continuous control and robot learning (Ren et al., 2024). Diffusion AC methods use diffusion policies in actor critic learning. They formulate constrained policy iteration as a diffusion noise regression problem for offline RL (Fang et al., 2024), or introduce an entropy regulator for online RL with maximum entropy (Wang et al., 2024). Soft Diffusion AC uses the state action value function as an energy function to train diffusion policies (Ma et al., 2025). Diffusion guidance can also define a controllable policy improvement operator without requiring an explicit value function (Frans et al., 2025). These methods focus on policy learning rather than on estimating the environment laws that define the Bellman operator. 9

Other adjacent studies address different statistical targets or decision problems. Diffuser uses diffusion as a trajectory planner rather than to estimate the environment laws for a single transition (Janner et al., 2022). Diffusion models have also been used to quantify uncertainty in trajectory planning, construct visual world models, and generate trajectories adapted to a target policy (Sun et al., 2023; Alonso et al., 2024; Fang and Lan, 2024). They have further been used to estimate action values from sampled future state sequences and to represent successor state measures through the Bellman flow equation (Mazoure et al., 2023; Schramm and Boularias, 2024). Beyond diffusion models, related studies estimate transition distributions with contrastive energy models and analyze offline RL based on learned models under partial coverage or in tabular MDPs (Chen et al., 2024c; Uehara and Sun, 2022; Li et al., 2024). The work most closely related to ours is the diffusion world model developed by Ding et al. (2024). Both this model and ours use conditional diffusion models to generate future states and rewards for value function learning. The diffusion world model jointly generates multistep future states and rewards and uses them for value estimation and offline policy learning. We instead estimate the one-step conditional reward law and transition kernel and plug these estimates into the optimal Bellman operator for Q∗ estimation.

1.3

Outlines

The rest of this paper is organized as follows. Section 2 introduces preliminaries. Section 3 presents the proposed method. Section 4 establishes the theoretical results. Section 5 reports numerical experiments. Section 6 concludes the paper. Proofs of all theoretical results are deferred to the Appendix.

2

Preliminaries

2.1

Markov Decision Process

A discounted Markov decision process (MDP) (Sutton et al., 1998; Agarwal et al., 2019) consists of a quintuple (X , A, P, R, γ), where X ⊂ Rdx denotes the state space, A ⊂ Rda 10

denotes the action space, P : X × A → M(X ) denotes the transition kernel, R(·|x, a) denotes the distribution of the immediate reward R(x, a) ∈ R, and γ ∈ [0, 1) denotes the discount factor. Here, M(X ) denotes the set of probability measures on (X , B(X )). For each state–action pair (x, a) ∈ X × A, the measure P(·|x, a) gives the distribution of the next state after taking action a at state x. In addition, for every measurable set D ∈ B(X ), the map (x, a) 7→ P(D|x, a) is measurable. The setting in this paper involves continuous state-action spaces, and, without loss of generality, assumes that X × A = [0, 1]dx × [0, 1]da . Let π(·|x) denote a stochastic policy, namely, a probability distribution over actions given the current state x. Given an initial distribution ν ∈ M(X ) with X1 ∼ ν, the offline dataset {(Xi , Ai , Ri , Xi′ )}ni=1 , with Xi′ = Xi+1 , satisfies X1 ∼ ν, Ai ∼ π(·|Xi ), Ri ∼ R(·|Xi , Ai ), Xi′ ∼ P(·|Xi , Ai ), i = 1, . . . , n. For theoretical analysis, the sample data are assumed to be independent and identically distributed (i.i.d.), and we let µπ ∈ M(X × A) denote the joint distribution of the stateaction pair (X, A) under policy π. For a given policy π, the action-value function is defined by Qπ (x, a) := E

"∞ X

# γ t Rt+1 | X1 = x, A1 = a, π .

t=0

In the general case, we assume that there exists a positive constant Rmax such that R(x, a) ∈ h i max [0, Rmax ] for each pair (x, a) ∈ X × A, then Qπ take values in 0, R1−γ . Suppose there exists ∗

a policy π ∗ that maximizes Qπ such that Q∗ := Qπ . The optimal action-value function Q∗ satisfies the optimal Bellman equation Q∗ = T ∗ Q∗ , where the optimal Bellman operator T ∗ is given by T ∗ Q(x, a) = E[R(x, a)] + γEX ′ ∼P (·|x,a) max Q(X ′ , a′ ). ′ a ∈A

In this work, we consider three independent batch datasets collected under a behavior policy r x πb : Sr := {(Xi , Ai , Ri , Xi′ )}ni=1 , Sx := {(Xi , Ai , Ri , Xi′ )}ni=1 , and D := {(Xi , Ai , Ri , Xi′ )}ni=1 ,

where the three collections are independent copies, and nr , nx , and n denote their respective sample sizes. Throughout the paper, we assume that all observations are independently and identically distributed under the behavior policy πb . Moreover, we use µπb (abbreviated as µ) to denote the joint distribution of the state–action pair (X, A) induced by πb . The datasets 11

Sr and Sx are used for optimal Bellman operator learning, while D is used for value function learning, as detailed in Section 3.

2.2

Diffusion Models

We briefly review diffusion models (Song et al., 2020; Yang et al., 2023; Chen et al., 2024b) and their role in conditional generative modeling. Mathematically, diffusion models are described by a pair of stochastic differential equations (SDEs): a forward diffusion process that gradually perturbs data into noise, and a reverse-time generative process that transforms noise back into samples from the target distribution. The key ingredient in this construction is the score function, which is typically learned through score matching, while sample generation is carried out by numerically discretizing the reverse SDE. Forward Process. The forward process starts from an initial data sample Ȳ0 ∼ ζ and gradually corrupts it by injecting Gaussian noise. Its dynamics are governed by the SDE dȲt = f (Ȳt , t) dt + g(t) dBt ,

Ȳ0 ∼ ζ,

(1)

where Ȳt ∈ Rd denotes the data state at time t ∈ [0, T ], Bt is a standard Brownian motion, and ζ is the target data distribution. The drift term f : [0, T ] × Rd → Rd characterizes the deterministic evolution, while the diffusion coefficient g : [0, T ] → R controls the stochastic noise injection. As time evolves, the forward process progressively smooths the original data distribution through Gaussian perturbations, eventually mapping ζ to a tractable prior distribution, typically a standard Gaussian. Backward Process. By the time-reversal theory of diffusion processes (Anderson, 1982; Haussmann and Pardoux, 1986), the forward SDE in (1) admits a reverse-time representation involving the score function ∇ log ζt , where ζt denotes the marginal distribution of Ȳt . The corresponding reverse-time SDE is given by h i et , dȲtR = f (ȲtR , t) − g 2 (t)∇ log ζt (ȲtR ) dt + g(t) dB et denotes a standard Brownian motion in reverse time. For numerical implementawhere B tion, it is convenient to rewrite the reverse dynamics as a forward-time process. Applying 12

the time transformation t 7→ T − t with a sufficiently large terminal time T , we obtain the SDE on [0, T ]: h i dYt = − f (Yt , T − t) + g 2 (T − t)∇ log ζT −t (Yt ) dt + g(T − t) dBt , Y0 ∼ ζT .

(2)

Here, Yt = ȲT −t . Starting from the prior distribution ζT at time t = 0, the process evolves toward YT , which approximates a sample from the target distribution ζ. Therefore, the main task is to learn the time-dependent score function ∇ log ζT −t (Yt ) from data. Conditional diffusion models generalize this framework by replacing the target distribution ζ with a conditional distribution ζ(·|X) depending on a covariate X. In this case, both the forward and reverse diffusion processes become conditional on X, and the corresponding score function changes to ∇ log ζt (·|X). Consequently, the drift term in the reverse SDE also depends on the conditioning variable X. Since the conditional score function is unknown in practice, it must be estimated from data. A standard approach is score matching (Hyvärinen and Dayan, 2005; Vincent, 2011), where the score function is parameterized by a DNN and learned by minimizing a denoising objective. Once the score estimator is obtained, samples from the target conditional distribution can be generated by numerically solving the reverse SDE (2). In practice, this is typically implemented through stochastic discretization schemes such as the Euler–Maruyama (EM) method, which yields the final sampling procedure for the conditional diffusion model.

2.3

Deep Neural Networks

We now introduce DNNs equipped with Rectified Linear Unit (ReLU) activation functions. A ReLU DNN is given by hθ (x) = AD ◦ σ ◦ AD−1 ◦ σ ◦ · · · ◦ σ ◦ A1 ◦ σ ◦ A0 (x) ∈ Rq ,

x ∈ Rd ,

where σ(x) = max{0, x} denotes the ReLU activation function applied element-wise, θ denotes the entire set of network parameters, and D denotes the depth of the network. For each i = 0, 1, . . . , D, the affine transformation Ai (·) is defined as Ai (x) = Mi x + bi ,

Mi ∈ Rki+1 ×ki , 13

bi ∈ Rki+1 ,

where (k0 , k1 , . . . , kD , kD+1 ) denotes the sequence of layer dimensions. Specifically, k0 = d corresponds to the input dimension, and kD+1 = q corresponds to the output dimension. We write hθ = (hθ,1 , . . . , hθ,q )⊤ , where hθ,j denotes the j-th component of hθ . Consequently, the network consists of D hidden layers and a total of D + 1 layers. Accordingly, the parameter set of the DNN hθ (·)can be explicitly written as
θ := (M0 , b0 ), (M1 , b1 ), . . . , (MD , bD ) . We denote the number of non-zero elements in θ as ∥θ∥0 =

D  X



∥ vec(Mi )∥0 + ∥bi ∥0 ,

i=0

and the largest absolute value among all parameters in θ as   ∥θ∥∞ = max max ∥ vec(Mi )∥∞ , max ∥bi ∥∞ , i∈{0,...,D}

i∈{0,...,D}

where vec(M) denotes the column-wise vectorization of the matrix M. For convenience, we define the width W, the size S, the weight bound B, and the uniform function bound CNN of the network as W = max{k1 , . . . , kD }, S = ∥θ∥0 , ∥θ∥∞ ≤ B, max ∥hθ,j ∥∞ ≤ CNN , j∈[q]

respectively, where [q] := {1, . . . , q} and ∥hθ,j ∥∞ := sup |hθ,j (x)|. x∈[0,1]d

Definition 2.1 (Hölder Class). For s > 0 with s = r + τ , where r ∈ N0 , τ ∈ (0, 1], and d ∈ N, define the Hölder class Hs (Rd , Σ) by   |∂ ξ g(x) − ∂ ξ g(y)| s d d ξ H (R , Σ) := g : R → R : max ∥∂ g∥∞ ≤ Σ, max sup ≤Σ . ∥ξ∥1 ≤r ∥ξ∥1 =r x̸=y ∥x − y∥τ For the domain [0, 1]d ⊆ Rd , write  Hs := g : [0, 1]d → R : g ∈ Hs (Rd , Σ) . Definition 2.2 (Total Variation Distance). Let ν1 and ν2 be two probability distributions over a sample space Ω ⊆ Rd . The total variation distance between ν1 and ν2 is defined by TV(ν1 , ν2 ) = sup |ν1 (D) − ν2 (D)| . D∈B(Ω)

14

For a function g : Ω → R, let ∥g∥∞,Ω := sup |g(x)|. Then, an equivalent characterization of x∈Ω

the TV distance is 1 TV(ν1 , ν2 ) = sup 2 ∥g∥∞,Ω ≤1

Z

Z g(x)ν1 (dx) − Ω

g(x)ν2 (dx) . Ω

Definition 2.3 (Covering Number). Let G be a set endowed with a semi-metric ρ̃. For any ε > 0, the ε-covering number N (G, ε, ρ̃) of G with respect to ρ̃ is defined as   N (G, ε, ρ̃) = inf k ∈ N ∃ g1 , g2 , . . . , gk ∈ G, such that ∀ g̃ ∈ G, min ρ̃(g̃, gj ) ≤ ε . 1≤j≤k

Definition 2.4 (Admissible distributions (Chen and Jiang, 2019)). A distribution ρ ∈ M(X × A) is called admissible if there exist h ≥ 1 and a possibly nonstationary randomized
policy π such that ρ(B) = P (Xh , Ah ) ∈ B|X1 ∼ ν, π for every B ∈ B(X × A), where Ah ∼ π(·|Xh ) and Xh+1 ∼ P(·|Xh , Ah ). Fix in advance a nonempty restricted class Πtar of possibly nonstationary randomized policies, rather than the class of all policies with Lebesguedensity action kernels. For π ∈ Πtar and h ≥ 1, let ρπh denote the law of (Xh , Ah ) under X1 ∼ ν and π, and set M := {ρπh : h ≥ 1, π ∈ Πtar }.

2.4

Notations

We introduce some additional notation used throughout the paper. For any u, v ∈ R, write u = O(v) or u ≲ v if u ≤ Cv for some constant C > 0, and write u = Ω(v) or u ≳ v if u ≥ C ′ v for some constant C ′ > 0. Moreover, write u ≍ v if both u ≲ v and u ≳ v e to suppress logarithmic factors. For any s, t ∈ R, let ⌈s⌉ and ⌊s⌋ hold. We also use O denote the ceiling and floor of s, and define s ∨ t := max{s, t} and s ∧ t := min{s, t}. Let N0 and N denote the sets of non-negative integers and positive integers, respectively. For P a vector z = (z1 , . . . , zd )⊤ ∈ Rd and q ∈ [1, ∞), define ∥z∥q := ( dj=1 |zj |q )1/q , and define P ∥z∥∞ := max1≤j≤d |zj | for q = ∞. In particular, ∥z∥2 := ( dj=1 zj2 )1/2 denotes the Euclidean norm. For a probability measure ν and a measurable function h : Rd → R, define the Lq (ν) norm by ∥h∥qLq (ν) := Ez∼ν |h(z)|q .

15

3

Method

In this section, we introduce the proposed framework for offline deep Q∗ estimation based on conditional diffusion models. Recall that the optimal action-value function Q∗ is characterized as the fixed point of the optimal Bellman equation. This naturally motivates estimating Q∗ through Bellman residual minimization. We then define the population Bellman residual loss LT ∗ (Q) := E(X,A)∼µ |T ∗ Q(X, A) − Q(X, A)|2 ,

(3)

where T ∗ is the optimal Bellman operator and µ is the data generation distribution of stateaction pair (X, A). By definition of the Bellman equation, Q∗ ∈ arg min LT ∗ (Q). Q

A fundamental difficulty is that the optimal Bellman operator T ∗ is unknown, since both the reward law and the transition kernel are unavailable in offline RL. This observation motivates learning these conditional distributions directly from offline data and then constructing a data-driven approximation of the optimal Bellman operator. To this end, we use conditional diffusion models to estimate the reward law and transition kernel, as described in Section 3.1. The resulting estimators induce an estimated optimal Bellman operator, which is subsequently incorporated into the value-learning procedure in Section 3.2. Accordingly, the proposed method requires three independent batch datasets generated under a behavior r x policy πb : Sr = {(Xi , Ai , Ri , Xi′ )}ni=1 , Sx = {(Xi , Ai , Ri , Xi′ )}ni=1 , D = {(Xi , Ai , Ri , Xi′ )}ni=1 .

The datasets Sr and Sx are used for reward law and transition kernel learning, respectively, while D is used for value function learning. Next, we present the detailed construction of the optimal Bellman operator learning and the corresponding value-learning procedure in the following two sections.

3.1

Optimal Bellman Operator Learning

To learn the optimal Bellman operator T ∗ , it is sufficient to estimate the underlying reward law and transition kernel that define the Bellman equation. Consequently, we use conditional diffusion models to learn these conditional distributions from offline datasets. Specifically, 16

the reward distribution R(·|x, a) is estimated from the batch dataset Sr , while the transition distribution P(·|x, a) is learned using Sx . The resulting approximated counterparts are b b denoted as R(·|x, a) and P(·|x, a), respectively. For notational convenience, let S := Sr ∪ Sx denote the combined dataset. We write w := (x, a) ∈ Rdw for a fixed state–action observation with dimension dw := dx + da . Once these two conditional models have been learned, we can b generate rewards and next states for any given state-action pair w by sampling from R(·|w) b and P(·|w). Since the constructions of the reward and transition models are parallel, we focus on the transition model below; the reward model can be developed analogously. To estimate the transition kernel, we let p(·|w) denote the distribution of the next state given current state-action pair w, that is, p(·|w) ≡ P(·|w), and introduce the forward diffusion process dx̄t = −2x̄t dt + 2dBt , x̄0 ∼ p(·|w), t ∈ [0, T ]. Then, the corresponding reverse process xt := x̄T −t is given by dxt = [2xt + 4∇ log pT −t (xt |w)]dt + 2dBt .

(4)

Here, pt (·|w) denotes the conditional density function of x̄t , and ∇ log pt (·|w) denotes the corresponding conditional score function to be estimated. Score Estimation. The reverse process described in (4) theoretically transforms noise into samples from the target conditional distribution p(x|w). However, direct simulation is intractable because the true conditional score function ∇x log pt (x|w) is unknown, rendering the drift term s∗ (t, x, w) := ∇x log pt (x|w) uncomputable. To address this, we employ score matching techniques (Hyvärinen and Dayan, 2005; Vincent, 2011) to learn an approximate score estimator b s(t, x, w). We define the following loss function Z T −T1 1 L(s) := Ex̄t ,W ∥s(t, x̄t , W ) − ∇ log pt (x̄t |W )∥2 dt, T − 2T1 T1 where T1 ∈ (0, T ) is introduced as a time truncation parameter to avoid instability near the endpoints of the diffusion process, and W := (X, A) denotes the random state–action pair. Using denoising score matching, the objective can be equivalently expressed, up to a

17

constant, as 1 L(s) = T − 2T1 =

1 T − 2T1

Z T −T1 Ex̄0 ,W Ex̄t |x̄0 ,W s(t, x̄t , W ) + T1 Z T −T1

x̄t − x̄0 µt σt2

Ex̄0 ,W EZ s(t, x̄0 µt + Zσt , W ) + T1

Z σt

2

dt

2

dt,

where µt = e−2t and σt2 = 1 − e−4t . For the empirical objective, write W i := (Xi , Ai ), x̄0,i = x Xi′ , of the i-th transition in Sx . Let {(tj , Zj )}m j=1 be i.i.d. auxiliary samples with tj ∼

U [T1 , T − T1 ] and Zj ∼ N (0, Idx ). The estimator b s is given by b := b s ∈ arg min L(s) s∈G

mx X nx 1 X Zj s(tj , x̄0,i µtj + Zj σtj , W i ) + mx nx j=1 i=1 σtj

2

,

(5)

where G denotes ReLU DNNs in Section 2.3. The resulting drift estimator is then given by bb(t, x, w) = 4b s(t, x, w) + 2x. EM Discretization. With the estimated drift term bb, we can define an SDE initializing from the prior distribution pT1 (x|w), i.e., b t , w)dt + 2dBt , db xt = bb(T − t, x b T1 ∼ pT −T1 (x|w), T1 ≤ t ≤ T − T1 . x

(6)

Subsequently, we can employ a discrete-approximation method for this dynamics (6). Let T1 = t0 < t1 < · · · < tK = T − T1 ,

K ∈ N+ ,

be the discretization points on [T1 , T − T1 ]. We consider the explicit EM scheme: e tk , w)dt + 2dBt , de xt = bb(T − tk , x e t0 = xT1 ∼ pT −T1 (x|w), x

t ∈ [tk , tk+1 ],

(7)

for k = 0, 1, · · · , K − 1. However, the conditional distribution pT −T1 (x|w) is unknown, rendering it impossible to utilize dynamics (7) for generating new samples. Thus, we substitute pT −T1 (x|w) with N (0, Idx ) since pT −T1 (x|w) converges to N (0, Idx ) as T tends to infinity. This adaptation presents a practically viable strategy. Therefore, the sampling dynamics

18

take the following form: dx̌t = bb(T − tk , x̌tk , w)dt + 2dBt , x̌t0 ∼ N (0, Idx ),

t ∈ [tk , tk+1 ].

(8)

Consequently, we can employ the dynamics in (8) to generate new samples that are approximately distributed according to the target conditional distribution. We work with the state space X = [0, 1]dx and reward space [0, Rmax ]. We therefore truncate the generated samples to their respective domains. For l < u, define CLIP(z, l, u) := min{u, max{l, z}}, where the operation is applied componentwise when z is a vector. We present the conditional diffusion model for transition kernel learning in Algorithm 1. Algorithm 1 Transition Kernel Learning via Conditional Diffusion Models x Require: N , Sx , {(tj , Zj )}m j=1 ∼ U[T1 , T − T1 ] and N (Z; 0, Idx ).

1: Obtain the score estimation b s by (5) 2: Initialize x̌t0 ∼ N (0, Idx ). 3: for k = 0 to K − 1 do 4:

Sample ϵk ∼ N (0, Idx ).

5:

√ Update x̌tk+1 ← x̌tk + bb(T − tk , x̌tk , w)(tk+1 − tk ) + 2 tk+1 − tk ϵk ;

6: end for 7: Output CLIP(x̌tK , 0, 1).

3.2

Value Function Learning

b b With the learned reward law and transition kernel R(·|x, a) and P(·|x, a), constructed in Section 3.1, we can draw m samples from these conditional distributions for any given state– b j }m b b′ m b action pair (x, a), denoted by {R j=1 ∼ R(·|x, a), {Xj }j=1 ∼ P(·|x, a). For each fixed (x, a), the empirical operator uses the same reward and next-state samples for all candidate Q-functions. Therefore, we can define the empirical optimal Bellman operator: m m 1 Xb γ X ∗ b b ′ , a′ ). T Q(x, a) := Rj + max Q(X j m j=1 m j=1 a′ ∈A

19

(9)

We can verify that the empirical optimal Bellman operator Tb ∗ is a γ-contraction (See Lemma C.1). Accordingly, we define the corresponding Bellman residual loss: 2

LTb ∗ (Q) := E(X,A)∼µ Tb ∗ Q(X, A) − Q(X, A) .

(10)

b as the empirical risk minimizer Given the batch dataset D, we define the final estimator Q associated with the empirical optimal Bellman operator Tb ∗ . Specifically, n 2 1 X b∗ b b T Q(Xi , Ai ) − Q(Xi , Ai ) . Q ∈ arg min LTb ∗ (Q) := Q∈Q n i=1

(11)

Here, Q denotes the class of ReLU DNNs used to estimate the optimal action-value function max Q∗ . By the definition of the MDP, the action-value function is bounded above by R1−γ .

Consequently, without loss of generality, we restrict Q to functions uniformly bounded by Rmax . 1−γ

Therefore, in the proposed framework, once the reward model and transition model are learned, the optimal Bellman operator can be estimated accordingly. The remaining step then reduces to estimating Q∗ through Bellman residual minimization under the learned Bellman operator. We summarize the overall structure of the proposed method in the following Algorithm 2. Algorithm 2 Offline Deep Q∗ Estimation with Diffusion Models Require: Batch datasets Sr , Sx , D, and sample size m. b of the optimal action-value function Q∗ . Ensure: Estimator Q 1: Learn the reward law and transition kernel via Algorithm 1 using the datasets Sr and

Sx , respectively. 2: Construct the estimator of the optimal Bellman operator, Tb ∗ , via (9).

b via (11). 3: Obtain the estimator of the optimal action-value function, Q,

Algorithm 2 provides a concise description of the proposed estimation procedure. The reward law and transition kernel are first estimated from offline data and then incorporated into an empirical approximation of the optimal Bellman operator. Specifically, for each state and action pair (x, a), the learned transition model generates m next-state samb ′ (x, a)}m . For each sampled next state X b ′ (x, a), the maximization step solves ples {X j j=1 j 20

b ′ (x, a), a′ and substitutes Q(X b ′ (x, a), amax ) into the empirical Bellman amax ∈ arg max Q X j j j j ′ a ∈A

backup. The resulting maximum values are averaged over the m next-state samples and combined with the Monte Carlo average of the generated rewards to form Tb ∗ Q(x, a). In this work, we consider a continuous action space. Hence, the maximization involved in the estimated Bellman operator is generally approximated numerically, for example, through action discretization or gradient-based optimization. Our theoretical analysis assumes access to the exact maximization operator. An approximate action optimization procedure introduces an additional optimization error, which is not considered in the present analysis. Finally, b is obtained by empirical Bellman residual minimization with respect to the the estimator Q learned operator Tb ∗ . Existing offline RL methods usually use data or learned models in different ways. Fitted value learning methods, such as FQI, DQN, DFQI, and DAPI, estimate Q∗ through recursive Bellman regression (Ernst et al., 2005; Mnih et al., 2015; Fan et al., 2020; Jiao et al., 2025a). Minimax Bellman error methods, such as MSBO and MABO, replace recursive regression with optimization over Bellman error criteria and auxiliary function classes (Xie and Jiang, 2020). Model based offline RL methods learn reward or transition models, but these models are typically used for rollout generation, uncertainty penalization, or conservative policy optimization (Janner et al., 2019; Yu et al., 2020; Kidambi et al., 2020; Yu et al., 2021). In contrast, the proposed method uses the learned reward and transition laws directly to construct the optimal Bellman operator. This leads to a simple final stage: after the conditional diffusion models are trained, Q∗ is estimated by a standard empirical risk minimization problem. The use of conditional diffusion models further allows flexible modeling of complex reward and transition laws, including continuous and potentially multimodal conditional distributions, without adding rollout trajectories generated by the model to the replay buffer.

4

Theoretical Analysis

In this section, we develop the theoretical analysis of the proposed method. Our main goal b The analysis uses a is to derive a value function estimation error bound for the estimator Q. 21

randomized comparison kernel supported on a δ-greedy comparison set of positive Lebesgue measure. Under Assumption 4.6, we establish that √ ∗ b − Q ∥L2 (ρ) ≤ Cδ ∥Q b − T ∗ Q∥ b L2 (µ) + γδ . ∥Q 1−γ 1−γ Here, µ is the data generating distribution of (X, A), ρ is the target distribution used to evaluate the value function error, and Cδ quantifies the mismatch between the data distribution and the target and successor comparison distributions. This result shows that bounding the value function estimation error reduces to controlling the Bellman residual, b − LT ∗ (Q∗ ). As described in Section 3, our method consists of two key compoi.e., LT ∗ (Q) nents: learning the reward law and transition kernel through conditional diffusion models, and then estimating the value function through Bellman residual minimization with the learned Bellman operator. Therefore, the final error depends jointly on the accuracy of the learned reward and transition models and the error from value function learning. To make this interaction explicit, we decompose the Bellman residual using the following lemma (see Appendix D for the proof). b satisfies Lemma 4.1 (Error decomposition). Conditional on S, T, Z, the estimator Q h i   ∗ ∗ 2 b − Tb ∗ Q∥ b 22 b − LT ∗ (Q∗ ) ≤ 2ED ∥T ∗ Q b∗ ∗ ED LT ∗ (Q) L (µ) + 8∥T Q − T Q ∥L2 (µ) {z } | {z } | Part II Part I
+ 2ED sup L b ∗ (Q) − 2Lb b ∗ (Q) + 8(1 + γ)2 inf ∥Q − Q∗ ∥2 . Q∈Q

T

T

|

Q∈Q

{z Part III

∞

}

In Lemma 4.1, the Parts I and II measure the discrepancy between the true and learned Bellman operators and therefore capture the effect of conditional diffusion estimation. The Part III is the sum of a statistical error and an approximation error arising from the valuelearning step under the learned Bellman operator. This decomposition forms the basis of our end-to-end error analysis. Consequently, the analysis reduces to establishing upper bounds for the optimal Bellman operator learning error and the value function learning error separately. These two components are analyzed in Sections 4.1–4.2, respectively. Combining b presented these two bounds yields the final convergence rate for the proposed estimator Q, in Section 4.3. 22

4.1

Error Bound for Optimal Bellman Operator Learning

In this section, we aim to bound the error of optimal Bellman operator learning, that is, bound Parts I and II. By direct calculation (See Appendix B.5) and recalling that W = (X, A), we obtain i h h
i 2 2 b b − Tb ∗ Q∥ b 22 ≲ R E E TV R(·|W ), R(·|W ) ES ∥T ∗ Q S W max L (µ) 2 dx +da Rmax log6 n(log m + log n) dx +da +2β n (1 − γ)2 m h 2
i Rmax 2 b ES EW TV P(·|W ), P(·|W ) , + (1 − γ)2

+

2 Rmax (1 − γ)2 m h 2
i Rmax 2 b + E E TV P(·|W ), P(·|W ) . S W (1 − γ)2

(12)

h i h
i 2 b ES ∥Tb ∗ Q∗ − T ∗ Q∗ ∥2L2 (µ) ≲ Rmax ES EW TV2 R(·|W ), R(·|W ) +

(13)

Therefore, it suffices to bound the errors of two conditional diffusion models. Recall the b t in (6), x e t in (7), and x̌t in (8); for any w ∈ [0, 1]dw , let pT −t (·|w), processes xt in (4), x pbT −t (·|w), peT −t (·|w), and p̌T −t (·|w) denote their corresponding distributions, respectively. By the data processing property of total variation, coordinatewise clipping of the generated outputs to their prescribed domains does not enlarge the bounds below. We then decompose the TV error into three parts: h h
i
i 2 2 ESx ,T,Z EW TV p̌T1 (·|W ), p(·|W ) ≲ ESx ,T,Z EW TV peT1 (·|W ), pT1 (·|W ) h
i + ESx ,T,Z EW TV2 p̌T1 (·|W ), peT1 (·|W ) h
i + EW TV2 pT1 (·|W ), p(·|W ) .

(14)

The three terms on the right-hand side of (14) correspond respectively to the score estimation error and EM discretizations, the error from replacing the initialization distribution, and the time-truncation error. We then proceed to establish individual bounds for these three terms in Subsections 4.1.1–4.1.3. To this end, we introduce several assumptions below. Assumption 4.1 (Bounded Conditional Density). The conditional density p(x|w) is supported on [0, 1]dx × [0, 1]dw and is uniformly bounded above and below by positive constants Bu and Bl , respectively. 23

Assumption 4.2 (Hölder Continuity). The conditional density p(x|w) belongs to the Hölder class Hα ([0, 1]dx +dw , Σ) for some α > 0 and Σ > 0, where α = p+q with p ∈ N0 and q ∈ (0, 1]. Assumption 4.3 (Boundary Smoothness). For some constants a0 ∈ (0, 12 ) and Σ0 > 0,
the restriction of p(x|w) to [0, 1]dx \[a0 , 1 − a0 ]dx × [0, 1]dw admits a Whitney extension in H3α+2 ([−1, 1]dx × [0, 1]dw , Σ0 ). Assumption 4.4 (Bounded Partial Derivatives). For every multi-index γ ∈ Nd0w satisfying γ p(x|w) exists and is uniformly bounded on ∥γ∥1 ≤ ⌊6α + 3⌋ + 1, the partial derivative ∂w

[0, 1]dx +dw by a constant Bγ . Remark 4.1. The regularity conditions on the target density in Assumptions 4.1–4.4 are imposed to facilitate the theoretical analysis of diffusion models and are consistent with commonly used assumptions in the literature. The compact support condition in Assumption 4.1 is standard in related works such as Lee et al. (2023); Li et al. (2023); Oko et al. (2023). Assumption 4.2 requires Hölder continuity, which is a classical assumption in nonparametric estimation and is closely related to Assumption 3.1 in Fu et al. (2024). In addition, Assumptions 4.3 and 4.4 impose boundary smoothness and bounded derivatives, following the technical framework of Oko et al. (2023), and thereby permit comparable theoretical analysis. 4.1.1

h  i Bound ESx ,T,Z,W TV2 peT1 (·|W ), pT1 (·|W )

We begin by establishing an error bound for score function estimation. To state the error decomposition, we introduce two auxiliary losses. For any score function s, define nx nx 1 X 1 X b x̄0,i , W i ), b LSx (s) := ℓ(s, x̄0,i , W i ), LSx ,T,Z (s) := ℓ(s, nx i=1 nx i=1 where 1 ℓ(s, x̄0 , W ) := T − 2T1 and

Z T −T1 T1

Z 2 EZ s t, x̄0 µt + Zσt , W + dt, σt

m

x
Zj 1 X b ℓ(s, x̄0 , W ) := s tj , x̄0 µtj + Zj σtj , W + mx j=1 σtj

2

.

With these definitions, the excess risk of the estimated score function b s satisfies     L(b s)−L(s∗ ) ≤ L(b s) − 2LSx (b s) + L(s∗ ) +2 LSx (b s) − LbSx ,T,Z (b s) +2 inf LbSx ,T,Z (s) − L(s∗ ) . | {z } s∈G | {z } | {z } Statistical error Approximation error Statistical error 24

The derivation of this decomposition is provided in Appendix A.7. We then bound the approximation error in Lemma 4.2 and two statistical errors in Lemma 4.3. Lemma 4.2 (Approximation Error). Suppose that Assumptions 4.1-4.4 hold. Let N ≫ 1, 2α T1 = N −Cµ and T = Cλ log N , where Cµ = α∧1 and Cλ = α are constants. Then there exists

a ReLU neural network s ∈ G(D, W, S, B) satisfying p s(t, x, w) = s(t, xclip (x, −C0 , C0 ), w) for ∥x∥∞ > C0 , where C0 = O( log N ), with network size
D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N ) . For any w ∈ [0, 1]dw and t ∈ [T1 , T − T1 ], we have Z N −2α log N . ∥s(t, x, w) − ∇ log pt (x|w)∥2 dx ≲ σt2 x∼pt (·|w) √

Moreover, we can take s satisfying ∥s(t, ·, ·)∥∞ ≲

log N . σt

We now restrict the following subclass of ReLU networks within G(D, W, S, B): √ n log N C := s ∈ G(D, W, S, B) ∥s(t, ·, ·)∥∞ ≲ , σt p
s(t, x, w) = s t, sclip (x, −C0 , C0 ) for ∥x∥∞ > C0 , where C0 = O( log N ), D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N )

o .

2α Throughout the paper, we set T1 = N −Cµ and T = Cλ log N , where Cµ = α∧1 and Cλ = α.

Lemma 4.3 (Statistical Error). Suppose that Assumptions 4.1–4.3 hold. Let N ≫ 1, then the score estimator b s defined in (5), with neural network architectures restricted to the class C, satisfies N dx +dw log14 N log4 N + log nx ∗ ESx ,T,Z L(b s) − 2LSx (b s) + L(s ) ≲ nx h

i

and
13 3 w 3 2 h i N Cµ +4Cλ + dx +d 2 N log N + log N log m + log 2 m 2 log x x ESx ,T,Z LSx (b s) − LbSx ,T,Z (b s) ≲ . √ mx 25

Combining the approximation error bound in Lemma 4.2 with the statistical error estimates in Lemma 4.3, we obtain the following upper bound for the score estimator. Theorem 4.1 (Error Bound for Score Estimation). Suppose that Assumptions 4.1-4.4 hold. 1

(dx +dw +12α)(α∧1)+4α

1

By choosing N = ⌊nxdx +dw +2α ⌋ + 1 ≲ nxdx +dw +2α , mx = nx (dx +dw +2α)(α∧1) , the score estimator b s defined in (5), with the neural network structure belonging to class C, satisfies   Z T −T1 2α − 1 2 Ext ,W ∥b s(T −t, xt , W )−∇ log pT −t (xt |W )∥ dt ≲ nx dx +dw +2α log18 nx . ESx ,T,Z T − 2T1 T1 Remark 4.2. Existing diffusion model theory places wide emphasis on the L2 -accuracy of score estimators. This is evident in the assumptions adopted by references (Chen et al., 2023a; Conforti et al., 2023; Lee et al., 2022, 2023; Benton et al., 2024; Li et al., 2023; Gao et al., 2023; Jiao et al., 2024). Rigorous theoretical investigations into score estimation itself, however, remain limited. Among the few studies that address this gap, (Wang et al., 2021, Theorem 5) establishes the consistency of score estimation. (Oko et al., 2023, Theorem 3.1) and (Chen et al., 2023b, Theorem 2) further quantify its convergence rates. These works, though, are confined entirely to a continuous-time setting. They ignore time discretization effects and assume integrals over [T1 , T −T1 ] are tractable. In contrast, our analysis explicitly accounts for time discretization and provides a convergence rate tailored to the discrete-time regime. A key strength of our result is that the bound achieves the minimax optimal rate in nonparametric statistics (Stone, 1980; Tsybakov, 2009) up to logarithmic terms. h  i We now derive an upper bound for ESx ,T,Z EW TV2 peT1 (·|W ), pT1 (·|W ) . By Theorem 4.1, for any ϵ > 0, there exists a constant ∆ϵ > 0 such that, if the maximum time step satisfies

ESx ,T,Z

max (tk+1 − tk ) ≤ ∆ϵ , then

0≤k≤K−1

"K−1 Z X tk+1 k=0

tk

# Extk ,W ∥b s(T − tk , xtk , W ) − ∇ log pT −tk (xtk |W )∥2 dt 

≤ (T − 2T1 ) · ESx ,T,Z

1 T − 2T1

Z T −T1

 Ext ,W ∥b s(T − t, xt , W ) − ∇ log pT −t (xt |W )∥ dt + ϵ 2

T1

2α − ≲ nx dx +dw +2α log19 nx + ϵ.

−

(15)

2α

Taking ϵ = nx dx +dw +2α and choosing ∆ϵ = ∆nx , we obtain the following lemma. 26

Lemma 4.4. Suppose that the conditions of Theorem 4.1 hold. By taking max (tk+1 − 0≤k≤K−1    10α(α∧1)+4α − (d +d +2α)(α∧1) tk ) = O min ∆nx , nx x w , we then have h  i 2α 2 ESx ,T,Z EW TV peT1 (·|W ), pT1 (·|W ) ≲ nx − dx +dw +2α log19 nx . Remark 4.3. The quantity ∆nx introduced above controls the error caused by approximating the score loss integrated over time in (15) using the values at the left endpoints. The architecture of C gives the following uniform Lipschitz bound with respect to the time and state 2α p − variables: LN := dx (1 + dx ) B D+1 W D . For ϵ = nx dx +dw +2α , we can choose ∆nx :=

N −2α . (log N )21 (L2N + N 8Cλ +2Cµ )

The corresponding number of steps for a uniform Euler discretization is   
 T − 2T1 K := = (T − 2T1 )(log N )21 N 2α L2N + N 8Cλ +2Cµ . ∆nx This choice bounds the approximation error in (15) by ϵ. 4.1.2

h  i Bound ESx ,T,Z EW TV2 p̌T1 (·|W ), peT1 (·|W )

In this section, we bound the error introduced by the initialization replacement in the sampling dynamics. Given that (7) and (8) differ only with respect to their initial distributions, the data processing inequality implies h  i h  i ESx ,T,Z EW TV2 p̌T1 (·|W ), peT1 (·|W ) ≲ EW TV2 pT −T1 (·|W ), N (0, I dx ) .

(16)

To bound this TV distance, we first analyze the KL divergence between the two multivariate distributions. By Lemma E.2 and Pinsker’s inequality, we derive the upper bound stated in the following lemma. 1

1

Lemma 4.5. Under Assumption 4.1, taking N = ⌊nx dx +dw +2α ⌋ + 1 ≲ nx dx +dw +2α , we have h

ESx ,T,Z EW TV

2



i 2α p̌T1 (·|W ), peT1 (·|W ) ≲ nx − dx +dw +2α .

27

4.1.3

h  i Bound EW TV2 pT1 (·|W ), p(·|W )

In this section, we bound the time-truncation error, which arises from the early stopping at time T1 . This term captures the discrepancy between the intermediate distribution pT1 (·|w) and the target distribution p(·|w). For small T1 , the forward diffusion process only mildly perturbs the target distribution. Assumption 4.1 ensures that the conditional density p(x|w) is supported on [0, 1]dx , while Assumption 4.2 gives its Hölder continuity in x on this domain. These properties allow us to control the perturbation induced by the short diffusion time, leading to the following bound. 1

1

Lemma 4.6. Under Assumption 4.1 and 4.2, taking N = ⌊nxdx +dw +2α ⌋ + 1 ≲ nx dx +dw +2α , we have h  i 2α − EW TV2 pT1 (·|W ), p(·|W ) ≲ nx dx +dw +2α . The bounds in Lemmas 4.4–4.6 together imply the following error bound for transition kernel estimation. Theorem 4.2. Suppose that the conditions of Lemmas 4.4–4.6 are satisfied. Then, h  i 2α − b ESx ,T,Z EW TV2 P(·|W ), P(·|W ) ≲ nx 2dx +da +2α log19 nx . Similarly, we can also derive an error bound for reward law estimation in the following corollary. Corollary 4.1. Under the conditions of Theorem 4.2 with the same Hölder index α, we have h  i 2α − b ESr ,T,Z EW TV2 R(·|W ), R(·|W ) ≲ nr 1+dx +da +2α log19 nr . Remark 4.4. Theorem 4.2 and Corollary 4.1 establish an end-to-end theoretical guarantee for diffusion-based learning of the transition kernel and reward law. Our result strengthens earlier studies that do not provide end-to-end guarantees, including Chen et al. (2023a); Conforti et al. (2023); Lee et al. (2022, 2023); Benton et al. (2024); Li et al. (2023); Gao et al. (2023), and improves upon end-to-end analyses based on Lipschitz-continuous score networks, such as Chen et al. (2023b); Jiao et al. (2025b), which typically yield suboptimal rates. It is also comparable to the guarantees established in Oko et al. (2023); Jiao et al. 28

(2024); Fu et al. (2024); Chang et al. (2026); in particular, unlike Fu et al. (2024), our analysis explicitly accounts for the numerical discretization error of the reverse process. By (12) and (13), Theorem 4.2, and Corollary 4.1, we obtain the following error bound for optimal Bellman operator learning. Theorem 4.3. Suppose that the conditions of Theorem 4.2 and Corollary 4.1 hold and that

dx +da dx +da the ReLU DNN class Q has S = O n dx +da +2β log5 n , D = O(log n), W = O n dx +da +2β log3 n ,
dx +da and B = O n dx +da +2β . Then, h i 2 2α 2α − 1+d +d − 2d +d Rmax 19 2 x a +2α x a +2α b − Tb ∗ Q∥ b 22 ≲ R n ED,S,T,Z ∥T ∗ Q n log n + log19 nx r x r max L (µ) (1 − γ)2 2 dx +da log6 n(log m + log n) Rmax d +d +2β x a n , + (1 − γ)2 m h i 2 2α 2α − − 2d +d Rmax ∗ ∗ ∗ ∗ 2 2 x a +2α b nr 1+dx +da +2α log19 nr + ED,S,T,Z ∥T Q − T Q ∥L2 (µ) ≲ Rmax n log19 nx x 2 (1 − γ) 2 Rmax + . (1 − γ)2 m

4.2

Error Bound for Value Function Learning

In this section, we derive an upper bound on the value function estimation error by controlling Part III in the preceding decomposition. Part III contains a statistical error term h i sup LTb ∗ (Q) − 2LbTb ∗ (Q) and an approximation error term inf ∥Q−Q∗ ∥2∞ . We bound these Q∈Q

Q∈Q

two terms separately below.   Bound statistical error sup LTb ∗ (Q) − 2LbTb ∗ (Q) .

We bound this term using the offset

Q∈Q

Rademacher complexity technique (Liang et al., 2015). Conditionally on the learned Bellman operator Tb ∗ , for each Q ∈ Q, define 2

ℓQ (X, A) := Tb ∗ Q(X, A) − Q(X, A) . The empirical offset Rademacher complexity of Q is defined as " # n n 1 X  ηX 2 off Rn (Q, η|D) := Eτ sup τi ℓQ (Xi , Ai ) − ℓ (Xi , Ai ) D , n i=1 Q Q∈Q n i=1

29

(17)

where η > 0 and τ := {τi }ni=1 are i.i.d. Rademacher random variables satisfying P(τi = 1) = P(τi = −1) = 12 . The corresponding offset Rademacher complexity is " # n n 1 X  X η off Roff τi ℓQ (Xi , Ai ) − ℓ2Q (Xi , Ai ) . n (Q, η) := ED Rn (Q, η|D) = ED,τ sup n Q∈Q n i=1 i=1 By the empirical covering number and Hoeffding’s inequality (see Lemma E.6), we can obtain Roff n (Q, η) ≤ where C :=

2 4Rmax . (1−γ)2

1 + log N (Q, δ, ∥ · ∥∞ ) 4Rmax (1 + γ) + (1 + 2ηC)δ, 2ηn 1−γ

(18)

In our analysis, we transform the statistical error into the offset

Rademacher complexity for estimation purposes, yielding  b ED sup LTb ∗ (Q) − 2LTb ∗ (Q) ≤ ED,τ sup 

Q∈Q

Q∈Q

! n n 3X 1 X 2 τi ℓQ (Xi , Ai ) − ℓ (Xi , Ai ) . n i=1 Cn i=1 Q

1 From these and choosing η = 3C in (18), we can obtain the statistical error bound, as shown

in the following lemma.
dx +da Lemma 4.7. Assume that the ReLU DNNs class Q is set with the size S = O n dx +da +2β log5 n ,
dx +da dx +da depth D = O(log n), W = O(n dx +da +2β log3 n), and weights bound B = O n dx +da +2β , then the statistical error satisfies  b ED sup LTb ∗ (Q) − 2LTb ∗ (Q) ≲ 

Q∈Q

2 Rmax − d +d2β +2β x a n log4 n. 2 (1 − γ)

(19)

Remark 4.5. To derive the statistical error bound, we assume that the batch data are i.i.d., following the standard setting in the error analysis of DRL documented in Jiang and Li (2016); Fan et al. (2020); Uehara et al. (2022). In our analysis, we use the offset Rademacher complexity technique, which yields a fast statistical rate of order

1 ,∥·∥∞ ) log N (Q, n . Moreover, this n

i.i.d. assumption can be extended to dependent data by taking into account the sequential structure of MDPs. In that case, a standard approach imposes suitable mixing conditions to establish the corresponding statistical error bound, as in Feng et al. (2023); Jiao et al. (2025a). Bound approximation error inf ∥Q − Q∗ ∥2∞ . This error can be controlled using ReLU Q∈Q

approximation theory for Hölder classes under the following Assumption 4.5. Then, the 30

desired bound for this approximation error follows from the ReLU approximation result in Jiao et al. (2025a). See Appendix C.2. The error decomposition in Lemma 4.1 and the form of this approximation term together explain why our analysis does not require Bellman completeness. In DFQI, DAPI, and related recursive value learning methods, the approximation error typically involves the Bellman updates of a function class. Typical examples include inf ∥T ∗ Q − Q∗ ∥∞ , inf ∥T π Q − Qπ ∥∞ , Q∈Q

Q∈Q

∗

and the inherent Bellman error sup inf ∥T f − g∥∞ , where Q denotes the DNN class and F f ∈F g∈F

denotes a candidate value function class (Munos and Szepesvári, 2008). Controlling these terms usually requires that the Bellman updates remain in the same function class or can be well approximated by it. This is the role of Bellman completeness. In contrast, our approximation error reduces to inf ∥Q − Q∗ ∥2∞ , so it is controlled directly by the approximation Q∈Q

capacity of the DNN class for Q∗ , as formalized in Assumption 4.5. Assumption 4.5. The optimal action-value function Q∗ belongs to the Hölder class Hβ for some constant β > 0. Consequently, the above statistical and approximation error analyses yield the following error bound for value function learning. Theorem 4.4. Suppose that Assumption 4.5 holds, and the ReLU DNNs class Q is con
dx +da dx +da figured with S = O n dx +da +2β log5 n , W = O(n dx +da +2β log3 n), D = O(log n), and B =
dx +da O n dx +da +2β . Then, h i b 2ED sup LTb ∗ (Q) − 2LTb ∗ (Q) + 8(1 + γ)2 inf ∥Q − Q∗ ∥2∞ ≲ Q∈Q

Q∈Q

2 Rmax − d +d2β +2β x a n log7 n. 2 (1 − γ)

Remark 4.6. Under the Hölder smoothness assumption Q∗ ∈ Hβ , Theorem 4.4 yields an
2β e n− dx +da +2β . The Hölder condition may be replaced by other regularity error bound of order O assumptions. For instance, suppose that Q∗ belongs to a Besov class with mixed smoothness 1
1
e n 2β+1 e n 2β+1 β. Let the ReLU DNN class Q have S = O , D = O(log n), W = O , and 2β
e n− 2β+1 (Montanelli and Du, B = nO(1) . The same analysis yields the convergence rate O 2019; Suzuki, 2018). Alternatively, suppose that Q∗ satisfies a spectral Barron condition.

+da +da e (dx + da )n 2dxdx+2d e n 2dxdx+2d a +2 , D = O(1), W = O a +2 , and B = O(1). The Let Q have S = O +da +2
e n− 2ddxx +2d a +2 resulting convergence rate is O (Klusowski and Barron, 2018; Ma et al., 2022). 31

4.3

Main Results

This section presents our main theoretical results, with emphasis on the convergence rate of b By the error decomposition of Lemma 4.1, Theorem 4.3, and Theorem 4.4, the estimator Q. we first establish the excess risk bound in the following theorem. Theorem 4.5. Suppose that the conditions of Theorem 4.3 and Theorem 4.4 hold. Then, h i 2 2α 2α − 2d +d Rmax 19 1+dx +da +2α x a +2α b − LT ∗ (Q∗ ) ≲ R2 n− ED,S,T,Z LT ∗ (Q) log n + n log19 nx x r max r (1 − γ)2 2 dx +da log6 n(log m + log n) Rmax d +d +2β x a + n (1 − γ)2 m 2 Rmax − d +d2β +2β + n x a log7 n. (1 − γ)2 Furthermore, if   β(2dx +da +2α)   β(1+dx +da +2α) 1+dx +da +2α α n α(dx +da +2β) , nx ≥ Ω n α(dx +da +2β) , m ≥ Ω(n), nr ≥ Ω (1 − γ) then h i ∗ b ∗ ∗ ED,S,T,Z LT (Q) − LT (Q ) ≲

2 Rmax − d +d2β +2β x a n log19 n. 2 (1 − γ)

b and it consists Remark 4.7. Theorem 4.5 provides convergence rates for the excess risk of Q, −

2α

2

−

2α

γRmax 2dx +da +2α 2 nr 1+dx +da +2α log19 nr , (1−γ) log19 nx , of four terms. In particular, the terms Rmax 2 nx 2

dx +da

6

Rmax log n(log m+log n) dx +da +2β and (1−γ) are all attributable to the conditional diffusion estima2n m 2

2β

Rmax − d +d +2β x a log7 n is tion error characterized in Theorem 4.3. The remaining term (1−γ) 2n

attributable to the approximation and statistical errors arising in the estimation of Q∗ as established in Lemma 4.7. Theorem 4.5 controls the Bellman residual under the sampling distribution µ. To transb we need to compare the sampling late this residual risk into a direct estimation error of Q, distribution µ with the target distribution ρ under which the value function error is evaluated. This motivates the following concentrability condition (Chen and Jiang, 2019; Xie and Jiang, 2020; Fan et al., 2020). Assumption 4.6 (Concentrability coefficient). For δ > 0 and Q ∈ Q, define   ∗ ∗ GQ,δ (x) := a ∈ A : max{Q(x, a), Q (x, a)} ≥ max max{Q(x, e a), Q (x, e a)} − δ . e a∈A

32

Assume that GQ,δ (x) has positive Lebesgue measure for every x ∈ X , and set R 1{a ∈ GQ,δ (x)} da πQ,δ (B|x) := RB . 1{a ∈ GQ,δ (x)} da A Let Mδ be the smallest class containing M and closed under ν 7→ P(ν) ⊗ πQ,δ for Q ∈ Q, where P(ν) ⊗ πQ,δ is the law of (X ′ , A′ ) under (X, A) ∼ ν, X ′ ∼ P(·|X, A), and A′ ∼ πQ,δ (·|X ′ ). For every δ > 0, there exists Cδ < ∞ such that every ν ∈ Mδ satisfies ν ≪ µ,

dν dµ

∞

≤ Cδ .

Assumption 4.6 can be viewed as the continuous version of the concentrability condition in Chen and Jiang (2019). Under this assumption, for every δ > 0 and ρ ∈ M, we have √

b L2 (µ) + γδ . Consequently, we can establish the following b − Q∗ ∥L2 (ρ) ≤ Cδ ∥Q b − T ∗ Q∥ ∥Q 1−γ 1−γ value function estimation error bound. Theorem 4.6. Suppose that Assumption 4.6 and the conditions of Theorem 4.5 hold, then h i R √C 19 γδ δ − d +dβ +2β b − Q∗ ∥L2 (ρ) ≲ max n x a log 4 n + . E ∥Q 2 (1 − γ) 1−γ β

e for a constant C e < ∞ gives Furthermore, taking δ = n− dx +da +2β and Cδ ≤ C p h i R e β b − Q∗ ∥L2 (ρ) ≲ max C n− dx +da +2β log 194 n. E ∥Q (1 − γ)2 b Theorem Remark 4.8. While Theorem 4.5 characterizes the excess risk convergence of Q, 4.6 directly establishes an error bound for the value function, building on the concentrability assumption and Theorem 4.5. The detailed proof is provided in Appendix D. After ignoring logarithmic orders and other constants, our method attains the oracle value-stage rate
β O n− dx +da +2β in Theorem 4.6, which outperforms the convergence rate of MABO (Xie and β

Jiang, 2020), given by O(n− 2dx +2da +4β ). Moreover, compared with existing DRL convergence results, our convergence rate is also superior. For instance, Feng et al. (2023) obtained a

β β convergence rate O n− 2dx +2da +4β+2 , while Jiao et al. (2025a) achieved O n− dx +da +4β . We highlight that our analysis avoids the Bellman completeness assumption commonly used in existing DRL theory (Chen and Jiang, 2019; Fan et al., 2020; Feng et al., 2023; Jiao et al., 2025a). In fitted value iteration, approximate policy iteration, and minimax Bellman

33

error methods, the approximation error is often tied to the Bellman updates of the working function class. This is commonly captured by the inherent Bellman error sup inf ∥f − g∈F f ∈F

∗

T g∥L2 (µ) , where F denotes a candidate value function class (Munos and Szepesvári, 2008). Controlling this error requires the Bellman image T ∗ F to be contained in the same function class or to be well approximated by it. This requirement is precisely the role of Bellman completeness. In contrast, our approximation error is controlled directly by approximating Q∗ , the fixed point of the optimal Bellman operator T ∗ . It suffices to assume that Q∗ is Hölder continuous. Bellman completeness of the whole function class is not required. The preceding result provides a refined error decomposition with respect to the individual sample sizes n, nr , and nx . Since offline data are typically collected under a fixed total budget, we next derive the corresponding convergence rate under a prescribed allocation of the total sample size. We further establish a lower bound under the same total offline budget, which characterizes the fundamental statistical limit of the proposed framework. Theorem 4.7. Suppose that the conditions of Theorem 4.6 hold. Let n e be the total offline sample size and set n = ωn n e, nr = ωr n e, and nx = ωx n e, where ωn , ωr , ωx ∈ (0, 1) and ωn + ωr + ωx = 1. If m ≥ Ω(n), then h i b − LT ∗ (Q∗ ) ≲ ED,S,T,Z LT ∗ (Q)

2 2β 2α Rmax n e− min{ 2dx +da +2α , dx +da +2β } log19 n e. 2 (1 − γ)

β α Taking δ = n e− min{ 2dx +da +2α , dx +da +2β } , then

p h i R e − min { 2dx +dαa +2α , dx +dβa +2β } log 192 n b − Q∗ ∥L2 (ρ) ≲ max C n ED,S,T,Z ∥Q e e. (1 − γ)2 Theorem 4.8 (Information-theoretic lower bound). Let M denote an MDP and Eα,β the class of MDPs satisfying Assumptions 4.1 and 4.2 for both the conditional transition and reward densities, and Assumption 4.5 for Q∗ . Suppose that µ = ρ = Unif(X × A) and let n e := n + nr + nx . Then, α b − Q∗ ∥L2 (ρ) ≳ n inf sup EM ∥Q e− max{ dx +da +2α , dx +da +2β } . M β

b M ∈Eα,β Q

Here, Q∗M denotes the optimal action-value function under model M . The infimum is taken b with respect to σ(Sr , Sx , D), and EM over all possibly randomized measurable estimators Q 34

is taken with respect to the joint distribution of the observations and any estimator randomization induced by M . Remark 4.9. Theorems 4.7–4.8 characterize the statistical difficulty of offline reinforcement learning from two different perspectives. Theorem 4.7 provides an achievable upper bound for the proposed conditional diffusion model-based Bellman learning procedure under a fixed offline sample budget. Since the proposed procedure first learns the conditional reward distribution and transition kernel and subsequently constructs the Bellman operator, its convergence rate reflects the additional statistical complexity of estimating the underlying environment dynamics. In particular, the transition distribution estimation term involves the joint space (x, a, x′ ) and therefore depends on the dimension 2dx + da . Furthermore, if we set √ h i β 19 e − α(dx +da ) Rmax C ∗ b β ≤ 2dx +da in Theorem 4.7, then ED,S,T,Z ∥Q − Q ∥L2 (ρ) ≲ (1−γ)2 n e dx +da +2β log 2 n e. This matches the rate in Theorem 4.6. In contrast, Theorem 4.8 provides an information-theoretic lower bound for direct estimation of the optimal action-value function over the considered MDP class. The corresponding minimax problem allows arbitrary estimators of Q∗ and does not impose the intermediate requirement of recovering the underlying conditional environment distributions. Consequently, this lower bound characterizes the intrinsic difficulty of estimating the value function itself, rather than the additional complexity induced by explicitly learning the conditional dynamics. The distinction between the two results highlights the difference between value function estimation and conditional environment modeling. The proposed framework targets the latter objective by estimating the conditional distributions that determine the Bellman operator. The resulting rate therefore reflects the statistical cost of learning the environment dynamics before performing Bellman optimization, which is absent in direct Q∗ estimation.

5

Numerical Experiments

In this section, we evaluate the empirical performance of the proposed method on both synthetic benchmark environments and real-world offline RL tasks. In Section 5.1, we first consider OpenAI Gymnasium environments, including Pendulum and Swimmer. In Section 5.2, we then examine the method on the real-world MIMIC-III clinical database. We 35

compare the proposed method with three representative offline RL baselines: MOPO (Yu et al., 2020), DFQI (Fan et al., 2020; Feng et al., 2023), and MABO (Xie and Jiang, 2020).

5.1

OpenAI Gymnasium Environments

We consider two continuous control benchmarks from Gymnasium, Pendulum and Swimmer (https://gymnasium.farama.org/). Pendulum is a classic low-dimensional control task in which the agent applies continuous torque to swing a pendulum upward and keep it balanced near the upright position. It mainly tests whether an algorithm can learn smooth and stable control from continuous actions. Swimmer is a MuJoCo locomotion task in which a multi-link body must coordinate joint torques to move forward in a fluid-like environment. Compared with Pendulum, Swimmer has higher-dimensional dynamics and requires coordinated control across multiple joints. These two environments therefore provide complementary tests of offline continuous-control performance. For both tasks, we construct the offline dataset by rolling out a random policy for 200 episodes. The baseline methods are trained on the full offline dataset. Our method uses 20,000 samples for diffusion model training and another 20,000 samples to estimate Q∗ through empirical Bellman residual minimization. We train all methods for 300 epochs and evaluate each learned policy over 50 episodes. All reported results are averaged over three seeds. In the training curves, the solid line denotes the mean training reward over the three seeds, and the shaded region represents the minimum-to-maximum range across these seeds.

36

Figure 1: Illustrations of the benchmark environments: Pendulum (Left) and Swimmer (Right).

Pendulum. In the Pendulum task, the agent generates continuous torque actions to swing the pendulum up and maintain it near the upright position. Figure 2 shows the training rewards and evaluation rewards. The training curves show that all methods improve from low initial rewards and reach a stable reward region during training. Although DFQI shows competitive training behavior, its evaluation performance is substantially lower than that of our method. MABO and MOPO also improve during training, but they obtain lower final evaluation rewards. As reported in Table 1, our method achieves the best evaluation reward of −177.06, improving over the strongest baseline result of −206.39 from MABO.

37

Figure 2: Performance comparison on the Gymnasium Pendulum task. Left: training reward over 300 epochs, where the solid line is the mean over three seeds and the shaded region is the min-max range. Right: box plot of evaluation rewards over 150 returns.

Swimmer. In the Swimmer task, the agent controls a multi-link body by applying torques at its joints, with the objective of producing coordinated forward movement. Figure 3 shows the corresponding results. The training curves indicate that the methods exhibit different reward patterns on this more challenging task. Our method maintains a relatively high reward level during training and achieves the best final evaluation performance. As shown in Table 1, our method obtains an evaluation reward of 71.90, while the best baseline result is 39.33 from MABO. These results suggest that the proposed framework can remain effective in a task that requires coordinated multi-joint control.

38

Figure 3: Performance comparison on the Gymnasium Swimmer task. Left: training reward over 300 epochs, where the solid line is the mean over three seeds and the shaded region is the min-max range. Right: box plot of evaluation rewards over 150 returns.

Table 1: Evaluation results on the Gymnasium environment tasks.

5.2

Task

DFQI

MABO

MOPO

Ours

Pendulum

-212.92

-206.39

-232.40

-177.06

Swimmer

30.93

39.33

34.98

71.90

Real-World Data

We examine the proposed method on a real-world sepsis treatment task constructed from the MIMIC-III database (https://physionet.org/content/mimiciii/1.4/). MIMIC-III is a publicly available critical care clinical database constructed from anonymized electronic health records of ICU patients at Beth Israel Deaconess Medical Center (Johnson et al., 2016). It contains longitudinal ICU records, including demographics, vital signs, laboratory tests, medication records, fluid input and output, diagnosis codes, nursing records, and disease severity scores. These temporal records allow patient disease trajectories to be reconstructed for sequential treatment decision making. We construct the study cohort following standard preprocessing protocols used in RL 39

studies of sepsis treatment (Raghu et al., 2017; Nanayakkara et al., 2022). Suspected infection events are identified from antibiotic administration and microbiology culture records, and organ dysfunction is assessed by the Sequential Organ Failure Assessment (SOFA) score. We retain adult ICU patients and include only the first eligible sepsis-related ICU stay for each patient. Patients are required to have a maximum SOFA score of at least 2 within 24 hours after sepsis onset, and ICU stays with severe missingness or invalid treatment records are excluded. The final processed cohort contains approximately 12,600 patients. Each ICU stay is divided into consecutive four-hour decision windows. Clinical measurements within a window are aggregated as the current state, and treatments administered in the same window define the action. Two adjacent windows form a transition sample (xt , at , rt , xt+1 ). The original state contains 41 clinical variables. We apply principal component analysis to the non-SOFA variables, retain the first 10 principal components, and concatenate the raw SOFA score, resulting in an 11-dimensional state representation. The action is defined by intravenous fluids and vasopressors, each discretized into three dosage levels, yielding 9 discrete treatment actions. The reward is the negative next-step SOFA score, rt = −SOFAt+1 , so a larger reward indicates lower subsequent organ dysfunction. This SOFA-based reward has clear clinical interpretability and has been used in related studies (Wang et al., 2026; Miao et al., 2025). After processing, the dataset contains approximately 215,000 transition samples. We select 5,000 patients as the offline training set, corresponding to approximately 85,000 transitions. All compared algorithms are trained on this fixed dataset, and simulatorgenerated transitions are not added to the replay buffer. Since the observed data only contain outcomes under clinician actions, we train a personalized simulator following the idea of PerSim (Agarwal et al., 2021). Given a current state and a candidate action, the simulator predicts the next state and reward. It is used only for evaluation and does not provide additional training samples. All algorithms are trained for 300 epochs. Since early intervention is important for sepsis progression, we set γ = 0.7 to emphasize immediate and short-term treatment effects. During training, we periodically evaluate each policy by simulated rollouts from fixed held-out initial states. In the training curves, the solid line denotes the mean reward over three seeds, and the shaded region represents the minimum40

to-maximum range. Final evaluation is performed on held-out patient initial states that are not used in offline training, where we perform 50 independent rollouts. Figure 4 and Table 2 summarize the experimental results. The proposed method achieves the highest final reward of −5.33, improving over the clinician policy with reward −5.59. Since the reward is defined as the negative future SOFA score, this improvement suggests lower predicted organ dysfunction in subsequent decision windows. Our method also outperforms MOPO, MABO, and DFQI. To further interpret the relative quality of different policies, we additionally evaluate a random policy that uniformly selects from the 9 treatment actions. The random policy obtains an average reward of −8.27, which is substantially lower than the rewards of the clinician policy and all learned policies. This gap indicates that the compared algorithms learn useful treatment structure from the offline data, while the proposed method achieves the best simulated return among them.

Figure 4: Performance comparison on the MIMIC-III sepsis treatment task. Left: training reward over 300 epochs, where the solid line is the mean over three seeds and the shaded region is the min-max range. Right: box plot of evaluation rewards over 150 returns.

Table 2: Evaluation results on the MIMIC-III sepsis treatment task. Method

Clinician

MABO

MOPO

DFQI

Ours

Reward

-5.59

-5.92

-5.88

-6.69

-5.33

41

6

Conclusion

We develop an offline RL framework for deep Q∗ estimation based on conditional diffusion models. The proposed method learns the reward law and transition kernel from offline data through conditional diffusion modeling, and then incorporates the resulting Bellman operator estimator into a deep Q∗ learning procedure via Bellman residual minimization. This design cleanly separates conditional generative modeling from value learning and enables a precise analysis of how optimal Bellman operator estimation errors propagate to value estimation. Notably, this separation also allows our analysis to avoid the Bellman completeness assumption commonly used in existing DRL theory. Ignoring logarithmic factors, the theoretical analysis yields convergence rates of order dx +da 2α 2α − − d +d +2β
O(nr 1+dx +da +2α ), O(nx 2dx +da +2α ) and O n x ma for reward law, transition kernel estimation and the Monte Carlo error induced by approximating the learned Bellman operator, 2β

respectively. It further gives an excess Bellman residual risk rate of order O(n− dx +da +2β ), and β

b attains the rate O(n− dx +da +2β ), which matches the the value function estimation error of Q oracle value-stage convergence rate of the nonparametric least-squares estimator. Numerical experiments support the theoretical findings and demonstrate the practical effectiveness of the proposed method. Several directions remain for future investigation. It would be of interest to extend the present framework to broader offline policy optimization problems and to more challenging settings, such as high dimensional locomotion, partially observable systems, and large-scale world-modeling problems.

Appendix In this appendix, Appendix A provides the technical analysis for score function estimation. Appendix B establishes the optimal Bellman operator learning bounds, Appendix C proves the value function learning results, Appendix D contains the proof of the main theorem, and Appendix E collects auxiliary lemmas and technical approximation results used throughout the proofs.

42

Table of Contents A Bounds on the Score Function

44

A.1 Bounds on pt (x|w) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

44

A.2 Bounds on the Derivatives of pt (x|w), ∇ log pt (x|w) and Clipping . . . . . .

46

A.3 Approximating of p(x|w) via Local Polynomials . . . . . . . . . . . . . . . .

53

A.4 Approximating of pt (x|w) and σt ∇pt (x|w) via Local Polynomial Integrals .

57

A.5 Approximating of Local Polynomial Integrals via ReLU Neural Networks . .

62

A.6 Error Bound for Approximating ∇ log pt (x|w) with ReLU Neural Networks .

76

A.7 Statistical Error of the Score Function . . . . . . . . . . . . . . . . . . . . .

89

A.8 Error Bound for the Score Function . . . . . . . . . . . . . . . . . . . . . . .

98

B Bounds on Optimal Bellman Operator Learning B.1 Proof of Lemma 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99 99

B.2 Proof of Lemma 4.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 B.3 Proof of Lemma 4.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 B.4 Proofs of Theorem 4.2 and Corollary 4.1 . . . . . . . . . . . . . . . . . . . . 108 B.5 Proof of Theorem 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 C Bounds on Value Function Learning

112

C.1 Proof of Lemma 4.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 C.2 Proof of Theorem 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 D Main Results

116

E Auxiliary Results

125

E.1 Construction of a Large ReLU Network . . . . . . . . . . . . . . . . . . . . . 128 E.2 Approximation of Basic Functions with ReLU Networks . . . . . . . . . . . . 129 E.3 Clipping and Switching Functions . . . . . . . . . . . . . . . . . . . . . . . . 129

43

A

Bounds on the Score Function

We first outline the main idea before presenting the formal lemmas. The conditional score can be written as ∇pt (x|w) . pt (x|w) Thus, it suffices to approximate both the diffused density pt (x|w) and its gradient. For the ∇ log pt (x|w) =

density term, the starting point is the decomposition   Z 1 ∥x − µt z∥2 exp − p(z|w) dz. pt (x|w) = | {z } σ dx (2π) d2x 2σt2 Rdx t |{z} Density | {z } Domain

Kernel

The approximation proceeds in four steps. • First (Domain localization). By Lemma A.2, the integration domain can be localized   √ √ Qdx xi −Cσt log ϵ−1 xi +Cσt log ϵ−1 dx from R to a neighborhood Ax = i=1 ai,x , ai,x = , , µt µt with the clipping error controlled by the Gaussian tail. • Second (Density approximation). On the localized region, we use a Taylor expansion to approximate the conditional density p(z|w) (See Appendix A.3). • Third (Kernel approximation). On the localized region, we use another Taylor expansion to approximate the Gaussian transition kernel (See Appendix A.4). • Finally (Neural network approximation). After the previous approximations, the localized integral reduces to a finite combination of polynomial-type functions, which can be approximated by ReLU DNNs (See Appendix A.5). Following the work of Oko et al. (2023); Jiao et al. (2024), we establish several highprobability bounds for the conditional density pt (x|w) and the corresponding score function in the following section.

A.1

Bounds on pt (x|w)

Lemma A.1. Under Assumption 4.1, let Dt (x) := max {(−xi )+ , (xi − µt )+ }. Then, for 1≤i≤dx

dx

any x ∈ R

dw

and w ∈ [0, 1] ,       (∥x∥∞ − µt )2+ dx Dt (x)2 Dt (x)2 exp − ≲ pt (x|w) ≲ exp − ≤ exp − . σt2 2σt2 2σt2 44

Proof. For each i, define   Z 1 (xi − µt zi )2 1 √ exp − qi (xi ) := dzi , ∆i := (−xi )+ ∨ (xi − µt )+ , 2σt2 0 σt 2π By Assumption 4.1, Bl

dx Y

qi (xi ) ≤ pt (x|w) ≤ Bu

i=1

dx Y

qi (xi ).

i=1

For each i, choose

Ii =

h i  σt  , 0, 1 ∧   µt    h  i    xi , xi + 1 1 ∧ σt ,  µt

µt

2

µt

xi < 0, 0 ≤ xi ≤ µ2t ,

h   i   xi xi 1 σt  − 1 ∧ , , µ2t < xi ≤ µt ,  µt 2 µt µt     h   i    1 − 1 ∧ σt , 1 , xi > µt . µt Then Ii ⊂ [0, 1], |Ii | ≳ 1 ∧ µσtt , and for any zi ∈ Ii , we conclude |xi − µt zi | ≤ ∆i + σt . Indeed, if xi < 0, then   σt ≤ ∆i + σt . |xi − µt zi | ≤ −xi + µt 1 ∧ µt If 0 ≤ xi ≤ µt , then ∆i = 0 and, by the definition of Ii ,   µt σt xi ≤ 1∧ ≤ σt = ∆i + σt . |xi − µt zi | = µt zi − µt 2 µt If xi > µt , then  |xi − µt zi | ≤ xi − µt + µt

σt 1∧ µt

 ≤ ∆i + σt .

Thus,   (xi − µt zi )2 qi (xi ) ≥ exp − dzi 2σt2 Ii σt 2π   |Ii | (∆i + σt )2 ≥ √ exp − 2σt2 σt 2π   ∆2 ≳ exp − 2i , σt Z

1 √

45

where the last inequality follows from

|Ii | σt

σ

≳

1∧ µt

t

σt

= min

n

1 1 , σt µt

o

2

∆2

+σt ) ≳ 1, (∆i2σ ≤ σ2i + 1. 2 t

t

Therefore, dx Y

dx Y

    dx Dt (x)2 ∆2i pt (x|w) ≥ Bl . qi (xi ) ≳ exp − 2 ≥ exp − 2 σ σ t t i=1 i=1 For the upper bound, since µt zi ∈ [0, µt ] and ∆i = (−xi )+ ∨(xi −µt )+ , we have |xi −µt zi | ≥ ∆i . Thus   1 ∆2i qi (xi ) ≤ √ exp − 2 . 2σt σt 2π t zi Moreover, by u = xi −µ , σt

1 qi (xi ) = µt

Z xi

 2   1 u 1 ∆2i √ exp − du ≲ exp − 2 , xi −µt 2 µ 2σt 2π t σ σt

t

t zi | i where the last inequality follows from |u| = |xi −µ ≥∆ . Therefore, σt σt

 qi (xi ) ≲

1 1 ∧ σ t µt



    ∆2i ∆2i exp − 2 ≲ exp − 2 , 2σt 2σt

where the last inequality follows from µ2t + σt2 = 1. Consequently, dx Y

dx Y

∆2 pt (x|w) ≤ Bu qi (xi ) ≲ exp − i2 2σt i=1 i=1 



  Dt (x)2 ≤ exp − . 2σt2

Since Dt (x) = max {(−xi )+ , (xi − µt )+ } ≥ max (|xi | − µt )+ = (∥x∥∞ − µt )+ , we further 1≤i≤dx

1≤i≤dx

have   (∥x∥∞ − µt )2+ pt (x|w) ≲ exp − . 2σt2 This completes the proof.

A.2

Bounds on the Derivatives of pt (x|w), ∇ log pt (x|w) and Clipping

In this section, we give the upper bounds on the derivatives of pt (x|w) and ∇ log pt (x|w). We first state the following lemma.

46

Lemma A.2 (Integral Clipping). Under Assumption 4.1, for any x ∈ Rdx , w ∈ [0, 1]dw , β ∈ Ndx and 0 < ϵ < e−1 , there exists a constant C > 0 such that β   Z Y dx  xi − µt zi i ∥x − µt z∥2 1 − dz dx p(z|w) exp σt 2σt2 σtdx (2π) 2 Rdx i=1 β   Z Y dx  xi − µt zi i 1 ∥x − µt z∥2 − dz ≲ ϵ, − dx p(z|w) exp σt 2σt2 σtdx (2π) 2 Ax i=1   √ √ Qd x xi −Cσt log ϵ−1 xi +Cσt log ϵ−1 where Ax = i=1 ai,x with ai,x = , . µt µt Proof. It follows that   Z dx β Y xi − µt zi i 1 ∥x − µt z∥2 dz − dx p(z|w) exp σt 2σt2 σtdx (2π) 2 Rdx \Ax i=1   Z dx β Y ∥x − µt z∥2 Bu xi − µt zi i 1{z∈[0,1]dx } exp − ≲ d dz dx σt 2σt2 σt x (2π) 2 Rdx \Ax i=1   dx Z dx β X Y xi − µt zi i ∥x − µt z∥2 Bu 1{zi ∈[0,1]} exp − ≲ d dz dx σt 2σt2 σt x (2π) 2 j=1 R×···×R×(R\aj,x )×R×···×R i=1 "   ! Z dx dx β Y X (xi − µt zi )2 1 xi − µt zi i √ 1{zi ∈[0,1]} exp − dzi ≲ Bu 2 σ 2σ σ 2π t R t t j=1 i=1,i̸=j   # Z β 1 (xj − µt zj )2 xj − µt zj j · √ 1{zj ∈[0,1]} exp − dzj . σt 2σt2 σt 2π R\aj,x t zi For i ̸= j, let yi = xi −µ . Then, it holds that σt   Z β 1 xi − µt zi i (xi − µt zi )2 √ 1{zi ∈[0,1]} exp − dzi σt 2σt2 σt 2π R  2 Z 1 y 1 βi ≤√ |yi | exp − i dyi ≲ . 2 µt 2πµt R

Notice that xi − µt zi σt

βi

  βi (xi − µt zi )2 βi
exp − ≤ (βi ) 2 exp − ≲ 1. 2 2σt 2

Then 1 √ σt 2π

xi − µt zi σt R

Z

βi

  Z (xi − µt zi )2 1 1 1{zi ∈[0,1]} exp − dzi ≲ 1{zi ∈[0,1]} dzi ≲ . 2 2σt σt R σt

Thus, by (20), we can derive   Z β 1 xi − µt zi i (xi − µt zi )2 √ 1{zi ∈[0,1]} exp − dzi ≲ O(1), σt 2σt2 σt 2π R 47

(20)

since σt2 + µ2t = 1. For i = j, we have   β xj − µt zj j (xj − µt zj )2 dzj 1{zj ∈[0,1]} exp − σt 2σt2 σt 2π R\aj,x  2 Z yj 1 βj |yj | exp − ≲ dyj √ µt |yj |≥C log ϵ−1 2  2 Z yj 1 βj dyj yj exp − ≲ √ µt yj ≥C log ϵ−1 2
βj 1 C2 ≲ ϵ 2 log ϵ−1 2 . µt Z

1 √

Choosing C ≥ 2, we obtain

σt

1 √

xj − µt zj σt 2π R\aj,x Z

βj

  (xj − µt zj )2 ϵ 1{zj ∈[0,1]} exp − dzj ≲ . 2 2σt µt

By setting C ≥ max{2, ∥β∥1 } and ϵ < e−1 , we obtain that when xj − µt zj σt xj −µt zj σt

decreases as

βj

xj −µt zj σt

≥ βj ,

  (xj − µt zj )2 exp − 2σt2

increases. Therefore, we obtain that

  β xj − µt zj j (xj − µt zj )2 1{zj ∈[0,1]} exp − dzj σt 2σt2 σt 2π R\aj,x βj Z C2 ϵ ϵ 2 (log ϵ−1 ) 2 1{zj ∈[0,1]} dzj ≲ . ≲ σt σt R\aj,x Z

1 √

Then by (21), we have 1 √ σt 2π

Z

βj

β dx  Y xi − µt zi i

  ∥x − µt z∥2 dz − dx p(x|w) exp 2σt2 σtdx (2π) 2

xj − µt zj σt R\aj,x

  (xj − µt zj )2 1{zj ∈[0,1]} exp − dzj ≲ ϵ, 2σt2

which implies that Z

Rdx \Ax i=1

≲

dx X

σt

1

(O(1))dx −1 · ϵ ≲ ϵ.

j=1

This completes the proof. 48

(21)

Lemma A.3 (Boundedness of Derivatives). Under Assumptions 4.1 and 4.4, for any k ∈ N0 , 0 ≤ l ≤ ⌊6α + 3⌋ + 1, 1 ≤ i1 , . . . , ik ≤ dx , and 1 ≤ j1 , . . . , jl ≤ dw , the following bound holds: ∂xi1 · · · ∂xik ∂wj1 · · · ∂wjl pt (x|w) ≲

1 . σtk

(22)

Moreover, we have   1 Dt (x) ∥∇ log pt (x|w)∥ ≲ · ∨1 , σt σt   1 Dt (x)2 ∥∂xi ∇ log pt (x|w)∥ ≲ 2 · ∨1 , σt σt2 and ∂t µt + ∂t σt ∥∂t ∇ log pt (x|w)∥ ≲ · σt2



Dt (x)2 ∨1 σt2

(23) (24)  32 .

(25)

Proof. For notational convenience, we define   ∥x − µt z∥2 N (z|w) := d − , dx exp 2σt2 σt x (2π) 2   γ p(z|w) ∂w ∥x − µt z∥2 γ N (z|w) := d − . dx exp 2σt2 σt x (2π) 2 p(z|w)

Turning first to the proof of (22), we let   Z Z p(z|w) ∥x − µt z∥2 f1 (x|w) = pt (x|w) = − dz = N (z|w)dz. dx exp 2σt2 σtdx (2π) 2 (β,γ)

For multi-indices β ∈ Ndx and γ ∈ Ndw , let f1

β

γ

(x|w) := ∂xβ11 · · · ∂xddxx ∂wγ11 · · · ∂wddww f1 (x|w),

we define Bβ := {u ∈ Ndx |ui ≤ βi , 1 ≤ i ≤ dx }. Then, it holds that 2

∂xβ11 ∂xβ22 · · · ∂xβddxx e−∥x∥ /2 =

X

2

Bu ∂xu11 ∂xu22 · · · ∂xuddxx e−∥x∥ /2 ,

u∈Bβ (β,γ)

where Bu are specific constants. Consequently, f1 (β,γ) f1 (x|w) =

1 Pdx

σt

i=1 βi

·

Z X

Bu

(x|w) can be expressed as

u dx  Y xi − µt zi i i=1

u∈Bβ

σt

N γ (z|w)dz.

For multi-indices β ∈ Nd0x and γ ∈ Nd0w satisfying ∥β∥1 = k and ∥γ∥1 = l ≤ ⌊6α + 3⌋ + 1, Assumption 4.4 gives γ ∂w p(z|w) ≤ Bγ ,

Z X u∈Bβ

Bu

u dx  Y xi − µt zi i i=1

σt 49

N γ (z|w)dz ≲

1 1 ∧ k ≲ 1. k σt µt

Hence (22) follows from the identity σt2 + µ2t = 1. We now proceed to establish (23) and (24). Without loss of generality, we analyze the first coordinate of ∇ log pt (x|w), the bounds for the remaining coordinates follow identically. Let f2 (x|w) := [σt ∇pt (x|w)]1 . Then, we have 1 f2 (x|w) · , σt f1 (x|w)   1 ∂xi f2 (x|w) f2 (x|w)(∂xi f1 (x|w)) [∂xi ∇ log pt (x|w)]1 = · − . σt f1 (x|w) f12 (x|w)

[∇ log pt (x|w)]1 =

Explicitly expanding the terms yields R  x1 −µt z1  N (z|w)dz σt f2 (x|w) R , =− f1 (x|w) N (z|w)dz R  xi −µt zi  N (z|w)dz σt ∂xi f1 (x|w) 1 R , =− f1 (x|w) σt N (z|w)dz  R x1 −µt z1 xi −µt zi − 1 + N (z|w)dz {i=1} σt σt ∂xi f2 (x|w) 1 R = . f1 (x|w) σt N (z|w)dz The integral terms in the three equations above can be expressed in the following unified form:

R Qdx  xi −µt zi βi i=1

F(x|w) = with

Pdx

i=1 βi ≤ 2.

where Ax =

N (z|w)dz ,

N (z|w)dz

According to Lemma A.2, for any 0 < ϵ < e−1 , there exists a constant

C > 0 such that β Z Y dx  xi − µt zi i Rdx i=1

σt

R

σt

β dx  Y xi − µt zi i

Z N (z|w)dz −

Ax i=1

σt

N (z|w)dz ≲ ϵ,

 xi −Cσt √log ϵ−1 xi +Cσt √log ϵ−1  , . i=1 ai,x with ai,x = µt µt

Qdx

Consequently, conditioned on pt (x|w) ≥ ϵ, we have R Qdx xi −µt zi βi N (z|w)dz i=1 σt Ax R |F(x|w)| ≤ + O(1) N (z|w)dz Ax "d # βi x Y xi − µt zi ≤ max + O(1) z∈Ax σt i=1 2

≤ C log ϵ ≲ log ϵ

−1

−1

Pdx β i=1 i 2

Pdx β i=1 i 2

50

,

+ O(1)

(26)

which implies that p f2 (x|w) ∂xi f1 (x|w) ≲ log ϵ−1 , ≲ f1 (x|w) f1 (x|w)

p log ϵ−1 ∂xi f2 (x|w) log ϵ−1 , ≲ . σt f1 (x|w) σt

Then, we obtain p log ϵ−1 log ϵ−1 ∥∇ log pt (x|w)∥ ≲ , ∥∂xi ∇ log pt (x|w)∥ ≲ . σt σt2   2 For given a constant 0 < Bl,1 < 1, we replace ϵ with Bl,1 exp − dx Dσt2(x) , then t

  dx Dt (x)2 pt (x|w) ≥ Bl,1 exp − . σt2 Then we have 1 · ∥∇ log pt (x|w)∥ ≲ σt



 Dt (x) ∨1 . σt

and for 1 ≤ i ≤ dx , 1 ∥∂xi ∇ log pt (x|w)∥ ≲ 2 · σt



 Dt (x)2 ∨1 . σt2

Finally, we prove (25). By a simple calculation, we have   1 f2 (x|w) · ∂t [∇ log pt (x|w)]1 = ∂t σt f1 (x|w)   1 ∂t f1 (x|w) f2 (x|w) 1 ∂t f2 (x|w) 1 f2 (x|w) − · · + · = ∂t σt f1 (x|w) σt f1 (x|w) f1 (x|w) σt f1 (x|w) ∂t σt =− [∇ log pt (x|w)]1 σt i
Rh ∂t σt ∥x − µt z∥2 σt−3 − dx σt−1 + (∂t µt )z T (x − µt z)σt−2 N (z|w)dz f (x|w) 2 R − · f1 (x|w) σt N (z|w)dz   " # (dx + 1)(∂t σt )σt−1 − ∥x − µt z∥2 (∂t σt )σt−3 R (∂t µt )z1 + (x1 − µt z1 )  N (z|w)dz ⊤ −2 − (∂t µt )z (µt z − x)σt R + . σt2 N (z|w)dz According to (26) and Lemma A.1, we obtain ∂t µt + ∂t σt · ∥∂t ∇ log pt (x|w)∥ ≲ σt2 The proof is complete.

51



Dt (x)2 ∨1 σt2

 32 .

Lemma A.4 (Error Bounds for Clipping). Under Assumptions 4.1 and 4.4, let 0 < ϵ < e−1 and 0 < ϵp < 1. For all 0 < t < ∞, there exists a constant C > 0 such that Z ϵ pt (x|w)∥∇ log pt (x|w)∥2 dx ≲ , √ σt Dt (x)≥Cσt log ϵ−1 Z pt (x|w)dx ≲ σt ϵ. √ Dt (x)≥Cσt

(27)

log ϵ−1

p Moreover, for Dt (x) ≤ Cσt log ϵ−1 , it follows that Z dx +2 ϵp pt (x|w)1{pt (x|w)≤ϵp } ∥∇ log pt (x|w)∥2 dx ≲ 2 (log ϵ−1 ) 2 , √ σt Dt (x)≤Cσt log ϵ−1 Z dx pt (x|w)1{pt (x|w)≤ϵp } dx ≲ ϵp (log ϵ−1 ) 2 . √ log ϵ−1

Dt (x)≤Cσt

Proof. We begin by establishing (27). For the region Dt (x) ≥ Cσt

p log ϵ−1 , applying Lemma

A.1 and Lemma A.3 yields   Dt (x)2 Dt (x)2 1 . pt (x|w)∥∇ log pt (x|w)∥ ≲ 2 exp − σt 2σt2 σt2 2

Since {x : Dt (x) ≤ r} = [−r, µt + r]dx , its volume is (µt + 2r)dx , and hence the shell volume is bounded by (µt + 2r)dx −1 dr. Therefore, Z pt (x|w)∥∇ log pt (x|w)∥2 dx √ log ϵ−1

Dt (x)≥Cσt Z ∞

1 ≲ 2 σt

r2 exp − √ 2σt2 Cσt log ϵ−1

1 ≲ σt

Z ∞



r2 (µt + 2r)dx −1 dr σt2

z2

√ C



e− 2 z 2 (1 + σt z)dx −1 dz log ϵ−1

Z ∞

z2 1 e− 2 z dx +1 dz √ σt C log ϵ−1 dx +1 1 C2 ≲ ϵ 2 · (log ϵ−1 ) 2 σt ϵ ≲ , σt

≲

where we used z = σrt and chose C sufficiently large. Similarly, Z ∞

Z √ Dt (x)≥Cσt

log ϵ−1

pt (x|w)dx ≲ σt C

52

2

√

log ϵ−1

e−z /2 z dx −1 dz ≲ σt ϵ.

On Dt (x) ≤ Cσt

p

log ϵ−1 , Lemma A.3 gives Dt (x)2 log ϵ−1 ∥∇ log pt (x|w)∥ ≲ ≲ . σt4 σt2 2

Therefore, Z √ Dt (x)≤Cσt

Similarly, Z √

pt (x|w)1{pt (x|w)≤ϵp } ∥∇ log pt (x|w)∥2 dx

dx p dx +2 ϵp log ϵ−1  ϵp −1 µ + 2Cσ log ϵ ≲ 2 (log ϵ−1 ) 2 . t t 2 σt σt

≲

Dt (x)≤Cσt

log ϵ−1

log ϵ−1

dx  p dx ≲ ϵp (log ϵ−1 ) 2 . pt (x|w)1{pt (x|w)≤ϵp } dx ≲ ϵp µt + 2Cσt log ϵ−1

This completes the proof.

A.3

Approximating of p(x|w) via Local Polynomials

Lemma A.5 (Approximating p(x|w) via local polynomials). Suppose that Assumption 4.2 holds. Let N ≫ 1, there exists g(x|w) such that p(x|w) − g(x|w) ≲ N −α , x ∈ [0, 1]dx , w ∈ [0, 1]dw . Proof. We write f (x, w) := p(x|w). By Assumption 4.2, we know that ∥f ∥Hα ([0,1]dx ×[0,1]dw ) ≤ 2p Σ. For any m = (m1 , m2 , · · · , mdx )⊤ ∈ [N ]dx := {1, · · · , N }dx , x = (x1 , x2 , · · · , xdx )⊤ ∈ [0, 1]dx , n = (n1 , n2 , · · · , ndw )⊤ ∈ [N ]dw := {1, · · · , N }dw , w = (w1 , w2 , · · · , wdw )⊤ ∈ [0, 1]dw , for n ∈ [N ], define the local weight function ηn,N : [0, 1] → [0, 1] by     1, 0 ≤ w ≤ N1 ,    η1,N (w) := 2 − N w, 1 < w ≤ 2 , N N      2 0, < w ≤ 1, N     N w − n + 1, n−1 < w ≤ Nn ,  N   ηn,N (w) := n + 1 − N w, n < w ≤ n+1 , 2 ≤ n ≤ N − 1, N N      0, otherwise, 53

and ηN,N (w) := For n ∈ [N ]dw , define ηn,N (w) := ψm,n (x, w) := 1

  0,

0 ≤ w ≤ NN−1 ,

 N w − N + 1,

N −1 < w ≤ 1. N

Qd w

i=1 ηni ,N (wi ). We define

x∈( m−1 ,m N N ]

ηn,N (w) =

dx Y

1x ∈ mi −1 , mi ηn,N (w).

i=1

i

( N

N

]

  For 1 ≤ i ≤ dx , the interval corresponding to mi = 1 is understood as 0, N1 . The functions {ηn,N }N n=1 form a partition of unity on [0, 1]. Consequently, the functions {ψm,n }m∈[N ]dx ,n∈[N ]dw form a partition of unity on [0, 1]dx × [0, 1]dw , that is, X

ψm,n (x, w) ≡ 1, x ∈ [0, 1]dx , w ∈ [0, 1]dw .

m∈[N ]dx ,n∈[N ]dw

Denote 1 u v m n  Cm,n,u,v = ∂ ∂ f , , u!v! x w N N  m u n v gm,n,u,v (x, w) = ψm,n (x, w) x − w− , N N  X m u  n v Gm,n (x, w) = Cm,n,u,v x − w− . N N ∥u∥1 +∥v∥1 ≤p

Then gm,n,u,v (x, w) is supported on     m−1 m ni 1 dx dw , , wi − ≤ , 1 ≤ i ≤ dw , (x, w) ∈ [0, 1] × [0, 1] : x ∈ N N N N where the first x-cell is understood coordinatewise to contain the left endpoint 0. For x ∈ [0, 1]dx and w ∈ [0, 1]dw , we define X

g(x, w) :=

ψm,n (x, w)Gm,n (x, w).

m∈[N ]dx ,n∈[N ]dw

Then, f (x, w) − g(x, w) =

X

ψm,n (x, w)f (x, w) −

m,n

≤

X

X

ψm,n (x, w)Gm,n (x, w)

m,n

ψm,n (x, w) f (x, w) − Gm,n (x, w) .

m,n

54

Using the multivariate Taylor formula, there exists θ ∈ [0, 1] such that  X m u  n v f (x, w) = Cm,n,u,v x − w− N N ∥u∥1 +∥v∥1 <p  X 1 m u  n v u v ′ ′ + ∂ ∂ f (x , w ) x − w− , u!v! x w N N p(u,v)

where
p(u, v) := ∥u∥1 + ∥v∥1 = p , n m x′ = (1 − θ) + θx, w′ = (1 − θ) + θw. N N Subtracting Gm,n from this expansion gives ψm,n (x, w) f (x, w) − Gm,n (x, w) m n  X 1 v v ≤ ψm,n (x, w) ∂xu ∂w f (x′ , w′ ) − ∂xu ∂w f , u!v! N N p(u,v) m u

n v N N q X 1 m u n v θ(m, n) p ≤ 2 Σ ψm,n (x, w) x− w− θ(x, w) − . u!v! N N N ∞ × x−

w−

p(u,v)

Whenever ψm,n (x, w) ̸= 0, we have xi − mNi ≤ N1 for 1 ≤ i ≤ dx and wi − nNi ≤ N1 for 1 ≤ i ≤ dw . Therefore, ψm,n (x, w) f (x, w) − Gm,n (x, w) X 1 ≤ 2p Σ ψm,n (x, w) ∥u∥ u!v! N 1 +∥v∥1 +q p(u,v)

= 2p Σ ψm,n (x, w)

(dx + dw )p ≲ ψm,n (x, w)N −α . p! N p+q

It follows that f (x, w) − g(x, w) ≲

X

ψm,n (x, w)N −α = N −α .

m,n

Finally, define g(x|w) = g(x, w), x ∈ [0, 1]dx , w ∈ [0, 1]dw . Then p(x|w) − g(x|w) = f (x, w) − g(x, w) ≲ N −α . The proof is complete. 55

By additionally incorporating Assumption 4.3, we can derive the following results. Lemma A.6. Suppose Assumptions 4.1-4.3 hold. Let N ≫ 1, there exists g(x|w) that satisfies p(x|w) − g(x|w) ≲ N −α , x ∈ [0, 1]dx , w ∈ [0, 1]dw , and |p(x|w) − g(x|w)| ≲ N −(3α+2) , x ∈ [0, 1]dx \[a0 , 1 − a0 ]dx , w ∈ [0, 1]dw . Moreover, g(x|w) has the following form g(x|w) =

X

(0) Cm,n,u,v gm,n,u,v (x, w)1{x∈[a0 ,1−a0 ]dx }

X

m,n ∥u∥1 +∥v∥1 <α

+

X

(1) Cm,n,u,v gm,n,u,v (x, w)1{x∈[0,1]dx \[a0 ,1−a0 ]dx } ,

X

m,n ∥u∥1 +∥v∥1 <3α+2 (0)

(1)

where Cm,n,u,v and Cm,n,u,v satisfy (0) |Cm,n,u,v |≲

1 1 (1) , |Cm,n,u,v . |≲ u!v! u!v!

Proof. Since p(x|w) ∈ Hα ([0, 1]dx × [0, 1]dw , Σ), by Lemma A.5, we can construct g (0) (x, w) =

X

(0) Cm,n,u,v gm,n,u,v x, w 1{x∈[0,1]dx }

X

m,n ∥u∥1 +∥v∥1 <α

such that p(x|w) − g (0) (x, w) ≲ N −α , x ∈ [0, 1]dx , w ∈ [0, 1]dw , where (0) |Cm,n,u,v |≤

1 2α Σ ≲ . u!v! u!v!

By Assumption 4.3, there exists a function g ′ ∈ H3α+2 ([−1, 1]dx × [0, 1]dw , Σ0 ) such that
g ′ (x, w) = p(x|w) on [0, 1]dx \[a0 , 1 − a0 ]dx × [0, 1]dw . Hence, |g ′ (x, w) − p(x|w)| = 0 ≲ N −(3α+2)
on [0, 1]dx \[a0 , 1 − a0 ]dx × [0, 1]dw .

By using Lemma A.5 and replacing p(x|w) with

g ′ (x, w), we obtain g (1) (x, w) =

X

(1) Cm,n,u,v gm,n,u,v x, w 1{x∈[0,1]dx }

X

m,n ∥u∥1 +∥v∥1 <3α+2

56

such that g ′ (x, w) − g (1) (x, w) ≲ N −(3α+2) , x ∈ [0, 1]dx , w ∈ [0, 1]dw , where (1) |Cm,n,u,v |≤

23α+2 Σ0 1 ≲ . u!v! u!v!

Let g(x|w) = g (0) (x, w)1{x∈[a0 ,1−a0 ]dx } + g (1) (x, w)1{x∈[0,1]dx \[a0 ,1−a0 ]dx } , that is, g(x|w) =

X

(0) Cm,n,u,v gm,n,u,v (x, w)1{x∈[a0 ,1−a0 ]dx }

X

m,n ∥u∥1 +∥v∥1 <α

+

X

X

(1) Cm,n,u,v gm,n,u,v (x, w)1{x∈[0,1]dx \[a0 ,1−a0 ]dx } .

m,n ∥u∥1 +∥v∥1 <3α+2

Then, g(x|w) satisfies that p(x|w) − g(x|w) ≲ p(x|w) − g (0) (x, w) 1{x∈[a0 ,1−a0 ]dx }   + p(x|w) − g ′ (x, w) + g ′ (x, w) − g (1) (x, w) 1{x∈[0,1]dx \[a0 ,1−a0 ]dx } ≲N −α 1{x∈[a0 ,1−a0 ]dx } + N −(3α+2) 1{x∈[0,1]dx \[a0 ,1−a0 ]dx } , which implies that |p(x|w) − g(x|w)| ≲ N −α + N −(3α+2) ≲ N −α , x ∈ [0, 1]dx , w ∈ [0, 1]dw , and |p(x|w) − g(x|w)| ≲ N −(3α+2) , x ∈ [0, 1]dx \[a0 , 1 − a0 ]dx , w ∈ [0, 1]dw . The proof is complete.

A.4

Approximating of pt (x|w) and σt ∇pt (x|w) via Local Polynomial Integrals

In this section, we approximate pt (x|w) and σt ∇pt (x|w) by means of local polynomial integrals. Under Assumptions 4.1–4.3, the expression for g(x|w) given in Lemma A.6 can

57

be reformulated as follows g(x|w) =

X

(0) Cm,n,u,v gm,n,u,v (x, w)1{x∈[a0 ,1−a0 ]dx }

X

m,n ∥u∥1 +∥v∥1 <α

+

X

(1) Cm,n,u,v gm,n,u,v (x, w)1{x∈[0,1]dx }

X

m,n ∥u∥1 +∥v∥1 <3α+2

−

X

(1) Cm,n,u,v gm,n,u,v (x, w)1{x∈[a0 ,1−a0 ]dx } .

X

m,n ∥u∥1 +∥v∥1 <3α+2

It can be formulated as the combination of the following functional forms X

X

Cm,n,u,v gm,n,u,v (x, w)1{x∈[Cβ ,Cγ ]dx } ,

(28)

m,n α(u,v)<Cα

where Cα ∈ {α, 3α + 2}, Cβ ∈ {0, a0 }, Cγ ∈ {1 − a0 , 1}, (0) (1) α(u, v) = ∥u∥1 + ∥v∥1 , Cm,n,u,v ∈ {Cm,n,u,v , Cm,n,u,v }.

By replacing p with g, we denote Z g1 (t, x, w) := g(z|w)1{z∈[0,1]dx } · Rdx

∥x − µt z∥2 exp − dx 2σt2 σtdx (2π) 2 1



 dz.

(29)

By virtue of Lemma A.5 or Lemma A.6, the discrepancy between pt (x|w) and g1 (t, x, w) admits the following bound

1{z∈[0,1]dx }

  ∥x − µt z∥2 p(z|w) − g(z|w) · d − pt (x|w) − g1 (t, x, w) ≤ dz dx exp 2σt2 σt x (2π) 2 Rdx   Z 1{z∈[0,1]dx } ∥x − µt z∥2 −α − ≲N · dz ≲ N −α . dx exp 2 d x 2σt Rdx σt (2π) 2 Z

Therefore, it remains to approximate g1 (t, x, w). By Lemma A.2, for any ϵ > 0, there exists a constant C > 0 such that, for all x ∈ Rdx ,

1{z∈[0,1]dx }

  ∥x − µt z∥2 − dz ≲ ϵ, pt (x|w) − p(z|w) · d dx exp 2σt2 σt x (2π) 2 Ax √ √ i h Qdx xi −Cσt log ϵ−1 xi +Cσt log ϵ−1 where Ax = i=1 ai,x with ai,x = , . We denote µt µt Z

1{z∈[0,1]dx }

  ∥x − µt z∥2 − dz. g2 (t, x, w) := g(z|w) · d dx exp 2σt2 σt x (2π) 2 Ax Z

58

Since p(x|w) is bounded, g(z|w) inherits this boundedness. Substituting p(z|w) with g(z|w) in Lemma A.2, we obtain a bound for the gap between g1 (t, x, w) and g2 (t, x, w) g1 (t, x, w) − g2 (t, x, w) ≲ ϵ. Note that g2 (t, x, w) involves an integral of the exponential function, which poses non-trivial analytical challenges. To mitigate this difficulty, we leverage polynomial approximations for t zi | the exponential function. For any 1 ≤ i ≤ dx and z ∈ Ax , the inequality |xi −µ ≤ σt p C log ϵ−1 holds. Exploiting this bound, we invoke the Taylor expansion to derive the

following result   X  j k−1 1 (xi − µt zi )2 C 2k logk ϵ−1 (xi − µt zi )2 ≤ , zi ∈ [Cl (xi ), Cu (xi )], exp − − − 2 k 2σt2 j! 2σ k!2 t j=0 where

( Cl (xi ) = max

and

) p xi − Cσt log ϵ−1 , −1 , µt

( Cu (xi ) = min

) p xi + Cσt log ϵ−1 ,1 . µt

By choosing k ≥ 32 C 2 u log ϵ−1 and applying the inequality k! ≥ ( k3 )k when k ≥ 3, we obtain (xi − µt zi )2 exp − 2σt2 



 j k−1 X 3 2 1 (xi − µt zi )2 ≤ ϵ 2 C u log u . − − 2 j! 2σt j=0

Accordingly, we may set    log dx 2 1+ u = max e, 3C 2 log ϵ−1 such that 3

2

ϵ 2 C u log u ≤

59

ϵ , dx

where k = O (log ϵ−1 ). By multiplying over the dx dimensions indexed by i, we have   Y  j dx X k−1 ∥x − µt z∥2 (xi − µt zi )2 1 exp − − − 2σt2 j! 2σt2 i=1 j=0   Y  j dx dx X k−1 Y (xi − µt zi )2 1 (xi − µt zi )2 exp − = − − 2σt2 j! 2σt2 i=1 i=1 j=0  X  j  k−1 dx X 1 (xi − µt zi )2 (xi − µt zi )2 − − ≤ exp − 2σt2 j! 2σt2 j=0 i=1 ( ) dx k−1  (x − µ z )2  X  (x − µ z )2 j Y 1 i t i i t i · max exp − − , 2 2 2σ j! 2σ t t j=0 j=0,j̸=i ≤ dx ·

ϵ
dx +1 ϵ · 1+ dx dx

≲ ϵ. Therefore, we only need to approximate Z g3 (t, x, w) :=

g(z|w)1{z∈[0,1]dx } ·

Ax

where Mki (z) :=

dx Y Mk (z) i

σ (2π)1/2 i=1 t

dz,

 j k−1 X 1 (xi − µt zi )2 − . j! 2σt2 j=0

Observe that the difference between g2 (t, x, w) and g3 (t, x, w) can be bounded as follows Z 1 g2 (t, x, w) − g3 (t, x, w) ≲ ϵ · d g(z|w)1{z∈[0,1]dx } dz dx σt x (2π) 2 Ax Z
1 p(z|w) + N −α 1{z∈[0,1]dx } dz ≲ϵ· d dx σt x (2π) 2 Ax  d x p 2Cσt log ϵ−1 ≲ ϵ · (Bu + N −α ) µdt x σtdx dx

log 2 ϵ−1 ≲ϵ· . µdt x

60

Additionally, we derive a second bound g2 (t, x, w) − g3 (t, x, w) ≲ ϵ · ≲ϵ· ≲ϵ·

Z

1 dx

σtdx (2π) 2 Ax Z 1 dx 2

σtdx (2π) Bu + N −α

g(z|w)1{z∈[0,1]dx } dz
p(z|w) + N −α 1{z∈[0,1]dx } dz

Ax

dx

σtdx (2π) 2 dx

log 2 ϵ−1 . ≲ϵ· σtdx Combining the two estimates and using the identity σt2 + µ2t = 1, we conclude that dx

g2 (t, x, w) − g3 (t, x, w) ≲ ϵ log 2 ϵ−1 . This shows that it suffices to approximate g3 (t, x, w) to sufficiently high accuracy using a ReLU neural network. Note that g(z|w) can be expressed as the combination of (28). Thus, g3 (t, x, w) can be expressed as the combination of the following functional forms Z

g(z|w)1{z∈[0,1]dx } ·

g3 (t, x, w) = Ax

=

X

X

m,n α(u,v)<Cα

=

X

X

Z Cm,n,u,v

dx Y Mk (z) i

σ (2π)1/2 i=1 t

dz

gm,n,u,v (z, w)1{z∈[Cβ ,Cγ ]dx }

Ax

dx Y Mk (z) i

σ (2π)1/2 i=1 t

dz

(30)

Cm,n,u,v Ψm,n,u,v (t, x, w),

m,n α(u,v)<Cα

In this representation, the z-cell indicator is incorporated into the integration domains Di,m , whereas the factor ηn,N (w) remains outside the integral. Here dx X k−1 ni  v i Y · ψ(t, xi , mi , ui , j), Ψm,n,u,v (t, x, w) := ηn,N (w) wi − N i=1 j=0 i=1  j Z  1 m u 1 (r − µt z)2 ψ(t, r, m, u, j) := 1{z∈[Cβ ,Cγ ]} z − − dz σt (2π)1/2 Di,m N j! 2σt2 m − 1 mi Di,m := ai,x ∩ , . N N dw  Y

Similarly, we can define h1 (t, x, w) to approximate σt ∇pt (x|w), where   Z µt z − x ∥x − µt z∥2 h1 (t, x, w) := g(z|w)1{z∈[0,1]dx } · d +1 − dz. dx exp 2σt2 σt x (2π) 2 Rdx 61

The difference between h1 (t, x, w) and σt ∇pt (x|w) can be bounded as ∥h1 (t, x, w) − σt ∇pt (x|w)∥ ≲ N −α . We can also define h2 (t, x, w) and h3 (t, x, w) as follows   Z ∥x − µt z∥2 µt z − x − h2 (t, x, w) : = g(z|w)1{z∈[0,1]dx } · d +1 dz, dx exp 2σt2 σt x (2π) 2 Ax Z dx µt z − x Y h3 (t, x, w) : = g(z|w)1{z∈[0,1]dx } · d +1 Mki (z)dz. dx σt x (2π) 2 i=1 Ax Then, we have ∥h1 (t, x, w) − h2 (t, x, w)∥ ≲ ϵ, and ∥h2 (t, x, w) − h3 (t, x, w)∥ ≲ ϵ · log

dx +1 2

ϵ−1 .

The ℓ-th element of h3 (t, x, w) can be expressed as the combination of the following functional forms. h3,ℓ (t, x, w) =

X

X

m,n α(u,v)<Cα

×

Y  dw  ni vi wi − Cm,n,u,v ηn,N (w) N i=1

Y k−1 dx X

 X  k−1 ∇ ψ(t, xi , mi , ui , j) ψ (t, xℓ , mℓ , uℓ , j) .

i̸=ℓ j=0

j=0

(31)

where  j m u 1 (x − µt z)2 ψ(t, x, m, u, j) := 1{z∈[Cβ ,Cγ ]} z − − dz, 1 N j! 2σt2 σt (2π) 2 Di,m  j Z  1 m  u µt z − x 1 (x − µt z)2 ∇ ψ (t, x, m, u, j) := 1{z∈[Cβ ,Cγ ]} z − − dz. 1 N σt j! 2σt2 σt (2π) 2 Dℓ,m 1

A.5

Z



Approximating of Local Polynomial Integrals via ReLU Neural Networks

In this section, we employ ReLU neural networks to approximate the local polynomial integrals we have derived. Specifically, we focus on the approximation of the quantity in (30) under the condition that ∥x∥∞ ≤ C0 , where C0 > 0 is a fixed constant. Before constructing a ReLU approximation of ψ(t, x, m, u, j), we first approximate the coefficients µt = e−2t and σt2 = 1 − e−4t appearing in its representation. 62

Lemma A.7 (Approximating the OU Coefficients). Let T ≥ 1, a ∈ {2, 4}, and 0 < η < 1/2. There exists a ReLU network Ea,T,η : R → [0, 1] such that sup Ea,T,η (t) − e−at ≤ η. t∈[0,T ]

Its parameters satisfy

D = O (T + log η −1 )2 , W = O (T + log η −1 )2 ,

S = O (T + log η −1 )3 , B = exp O(T + log η −1 ) . l n  om PM (−aT )q t
q Proof. Let M := max 2eaT, log2 η2 , and PM (t) := . By Taylor’s q=0 q! T
M +1 theorem and (M + 1)! ≥ Me+1 , we have  M +1 eaT (aT )M +1 η −at ≤ sup |PM (t) − e | ≤ ≤ 2−(M +1) ≤ . (M + 1)! M +1 2 t∈[0,T ] −aT

Set δ := ηe 2 . Following the convention above, put Π0,δ (t) := 1 and Π1,δ (t) := Tt . For 2 ≤ q ≤ M , Lemma E.11, with C = 1 and the shared input Tt , gives  q  t t t  Πq,δ (t) := sprod,q , sup Πq,δ (t) − ≤ δ. ,..., T t∈[0,T ] |T {z T} q times

By Lemma E.10, PeM (t) :=

M X (−aT )q q=0

q!

Πq,δ (t),

sup |PeM (t) − PM (t)| ≤ δ t∈[0,T ]

M X (aT )q q=0

q!

η ≤ . 2

Define Ea,T,η (t) := sclip (PeM (t), 0, 1). Since e−at ∈ [0, 1] and clipping is nonexpansive, sup |Ea,T,η (t) − e−at | ≤ η. t∈[0,T ]

For 2 ≤ q ≤ M , Lemma E.11 gives
Dq = O log q [log q + log δ −1 ] ,

Wq = 48q,

Sq = O q[log q + log δ −1 ] ,

Bq = 1.

Because M = O(T + log η −1 ) and log δ −1 = O(T + log η −1 ), parallelization yields


D = O (T + log η −1 )2 , W = O (T + log η −1 )2 , S = O (T + log η −1 )3 . q

Finally, max (aTq!) ≤ eaT , so the affine output layer and final clipping layer give B = 0≤q≤M

exp (O (T + log η −1 )). 63

Using Lemma A.7, we now construct a ReLU neural network to approximate ψ(t, x, m, u, j) over the domain ∥x∥∞ ≤ C0 . Lemma A.8. Given N ≫ 1, C0 > 0, for any ϵ0 > 0, 1 ≤ m ≤ N , 0 ≤ u < Cα , and 0 ≤ j ≤ k − 1, there exists a ReLU neural network ψR ∈ G(D, W, S, B) with
2 2 4 −1 D = O log2 ϵ−1 , 0 + log C0 + log N + log ϵ
3 3 6 −1 , W = O log3 ϵ−1 0 + log C0 + log N + log ϵ
4 4 8 −1 S = O log4 ϵ−1 , 0 + log C0 + log N + log ϵ 2 2 4 −1 B = exp O log2 ϵ−1 0 + log C0 + log N + log ϵ

,

such that |ψR (t, x, m, u, j) − ψ(t, x, m, u, j)| ≲ ϵ0 , x ∈ [−C0 , C0 ], t ∈ [T1 , T − T1 ]. tz Proof. By the definition of ψ(t, x, m, u, j), we take the transformation y = x−µ , then σt

 j m u 1 (x − µt z)2 1{z∈[Cβ ,Cγ ]} z − dz − N j! 2σt2 Di,m u  2 j Z  1 x − σt y m y dy = − − j!(2π)1/2 µt D µt N 2 u−l Z   2 j u   X 1 m σt y l y u x = − − − dy 1/2 j!(2π) µt l=0 l µt N µt 2 D u−l u   l Z X u x m 1 l+j σt − (−1) y l+2j dy = j!(2π)1/2 µt l=0 l µt N 2j µlt D   u t u−l l X ) σt DUl+2j+1 (x) − DLl+2j+1 (x) 1 (−1)l+j u (x − mµ N = · , j!(2π)1/2 l=0 2j l l + 2j + 1 µu+1 t

1 ψ(t, x, m, u, j) = σt (2π)1/2

Z



where D = [DL (t, x), DU (t, x)] with DL (t, x) = ψclip

!
t p p , x − µ C , x − µ C ψclip x − mµ t γ t β N , −C log ϵ−1 , C log ϵ−1 , σt

and  DU (t, x) = ψclip 

  t ψclip x − (m−1)µ , x − µ C , x − µ C t γ t β N σt

64

 p p , −C log ϵ−1 , C log ϵ−1  .

Thus, we only need to approximate the function of the form  mµt u−l DUl+2j+1 (x) − DLl+2j+1 (x) ψm,l,j (t, x) = σtl x − · . N µu+1 t For a fixed accuracy η > 0, Lemmas A.7 and E.14 provide the ReLU networks
µ bt := sclip (E2,T,η (t), µT , 1) , σbt2 := sclip 1 − E4,T,η (t), σT21 , 1 , such that 

sup t∈[T1 ,T −T1 ]

|b µt − µt | ∨ σbt2 − σt2

  For r ∈ {m − 1, m}, define cr := min Cγ , max Nr , Cβ



≤ η.

. Since µt > 0 and µ bt ≥ µT > 0, we

have ψclip



 rµt x− , x − µ t C γ , x − µ t C β = x − µ t cr , N

and   rb µt ψclip x − ,x − µ bt Cγ , x − µ bt Cβ = x − µ bt cr . N Thus, the analytic definitions of DL and DU remain unchanged. In the network constructions below, their inner clipping terms are represented exactly by x − µ bt cm and x − µ bt cm−1 , respectively, while σ 2 is replaced by σb2 . We first consider the approximation of Dl+2j+1 (t, x). t

t

L

We may choose a ReLU neural network (L)

ψm,j,l (t, x) = ψprod,D1 ( ·, · · · , · ) ◦ ψclip (·, −C | {z }

p

log ϵ−1 , C

p log ϵ−1 )

l+2j+1 times



 2 b ◦ψprod,D2 x − µ bt cm , ψrec ψroot (σt ) . For DUl+2j+1 (t, x), we use the following ReLU neural network for approximation (U )

ψm,j,l (t, x) = ψprod,D1 ( ·, · · · , · ) ◦ ψclip (·, −C | {z }

p

log ϵ−1 , C

p log ϵ−1 )

l+2j+1 times



 ◦ψprod,D2 x − µ bt cm−1 , ψrec ψroot (σbt2 ) . Thus, the difference DUl+2j+1 (t, x) − DLl+2j+1 (t, x) can be approximated by (D)

(U )

(L)

ψm,j,l (t, x) = ψm,j,l (t, x) − ψm,j,l (t, x).

65

u−l  t t Next, we turn to the term x − mµ . Given that x − mµ ≤ C0 + 1, by Lemma E.11, N N there exists a ReLU neural network  mb µt  (N ) ψm,j,l (t, x) = ψprod,N ( ·, · · · , · ) ◦ x − | {z } N u−l times

t that approximates x − mµ N

u−l

. For σtl , we select

  (σt ) ψm,j,l (t) = ψprod,σt (·, · · · , ·) ◦ ψroot σbt2 . | {z } l times

1 , the chosen is For µu+1 t

(µ )

t ψm,j,l (t) = ψprod,µt ( ·, · · · , · ) ◦ ψrec (b µt ) . | {z }

u+1 times

Finally, we can construct a ReLU neural network (F ) ψm,j,l (t, x) = ψprod,F



(σt ) ψm,j,l (t),

(N ) ψm,j,l (t, x),

(D) ψm,j,l (t, x),



(µt ) ψm,j,l (t)

to approximate ψm,l,j (t, x). (F )

Next, we derive the error bound between ψm,j,l (t, x) and ψm,l,j (t, x). For convenience, we introduce notation for approximation errors: Let ϵ(U ) , ϵ(L) , ϵ(D) , ϵ(µt ) , ϵ(σt ) , ϵ(N ) , ϵ(F ) corre(U )

(L)

(D)

(µ )

(σ )

(N )

(F )

t t spond to the errors of ψm,j,l , ψm,j,l , ψm,j,l , ψm,j,l , ψm,j,l , ψm,j,l , ψm,j,l , respectively. Separately,

let ϵprod,D1 , ϵprod,D2 , ϵprod,D , ϵprod,N , ϵprod,σt , ϵprod,µt , ϵprod,F denote the approximation errors of ψprod,D1 , ψprod,D2 , ψprod,D , ψprod,N , ψprod,σt , ψprod,µt , ψprod,F , respectively. p p u−l t u−l −1 , |D (t, x)| ≤ C Since x − mµ ≤ (C + 1) , |D (t, x)| ≤ C log ϵ log ϵ−1 , σtl ≤ 0 L U N 1 1 1 and µu+1 ≤ µu+1 , we set t

T

 C1 = max (C0 + 1)

u−l

, 1, 2(C

p

−1 l+2j+1

log ϵ )

,

1 µu+1 T

 .

By Lemma E.11, we have (F )

|ψm,j,l (t, x) − ψm,l,j (t, x)| ≤ ϵprod,F + 4C13 · max{ϵ(D) , ϵ(N ) , ϵ(σt ) , ϵ(µt ) }. By Lemmas E.12 and E.13, we can bound ϵ(U ) and ϵ(L) . Since Cβ , Cγ ∈ [0, 1] and the clipping function is 1-Lipschitz, replacing µt by µ bt changes each inner clipped factor by at most η. 66

 Let C2 = max C0 + 1, √

1

 . Then −4T

1−e

1

" p ϵ(L) ≤ ϵprod,D1 + (l + 2j + 1)(C log ϵ−1 )l+2j ϵprod,D2 + 2C2

η + ϵrec +

ϵroot + √ϵηroot

!#

ϵ2rec

and " !# ϵroot + √ϵηroot p . ϵ(U ) ≤ ϵprod,D1 + (l + 2j + 1)(C log ϵ−1 )l+2j ϵprod,D2 + 2C2 η + ϵrec + ϵ2rec Since ϵ(D) ≤ ϵ(U ) + ϵ(L) , we obtain " !# ϵroot + √ϵηroot p ϵ(D) ≤ 2ϵprod,D1 + 2(l + 2j + 1)(C log ϵ−1 )l+2j ϵprod,D2 + 2C2 η + ϵrec + . ϵ2rec We can also bound ϵ(N ) , ϵ(σt ) and ϵ(µt ) as (N )

ϵ

  η ≤ ϵprod,N + (u − l)(C0 + 1) η, ϵ ≤ ϵprod,σt + l ϵroot + √ , ϵroot   η u+1 (µt ) ϵrec + 2 . ϵ ≤ ϵprod,µt + u µT ϵrec (u−l−1)+

(σt )

ϵ0 ∗ To ensure that ϵ(F ) ≤ ϵ0 , we take ϵprod,F = ϵ20 and max{ϵ(D) , ϵ(N ) , ϵ(µt ) , ϵ(σt ) } ≤ 8C 3 =: ϵ . In 1

detail, we take ϵ∗ ϵ∗ p , ϵprod,D2 = , 4 8(l + 2j + 1)(C log ϵ−1 )l+2j ϵ∗ ϵ∗ ϵ∗ ϵprod,N = , ϵprod,µt = , ϵprod,σt = . 2 2 2

ϵprod,D1 =

Moreover, we take (

ϵ∗ µuT ϵ∗ p ϵrec = min µT , σT1 , , 4(u + 1) 64C2 (l + 2j + 1)(C log ϵ−1 )l+2j   ϵ∗ 2 3 ϵroot = min σT1 , ϵrec , . 4(l ∨ 1) Finally, we choose  η := min

1 3/2 3 ϵ∗ , ϵroot , ϵrec , 4 2 [1 + (u − l)(C0 + 1)(u−l−1)+ ] 67

 .

) ,

These choices give η

ϵroot + √ϵroot η η ϵroot + √ ≤ 2ϵroot , ϵrec + 2 ≤ 2ϵrec , η + ϵrec + ≤ 4ϵrec . ϵroot ϵrec ϵ2rec Therefore, max{ϵ(D) , ϵ(N ) , ϵ(σt ) , ϵ(µt ) } ≤ ϵ∗ . Note that l ≤ u ≤ Cα = O(1) and j ≤ k − 1 = O(log ϵ−1 ). Moreover, the choices above imply
2 −1 . log η −1 = O log ϵ−1 0 + log(C0 + 2) + log N + log ϵ Since T = Cλ log N , Lemma A.7 shows that the additional networks µ bt and σbt2 are absorbed into the parameter bounds below. Subsequently, we can obtain the network structures of (·)

(U )

(L)

ψm,j,l . We now detail the network complexity for each component. For ψm,j,l , ψm,j,l , and (D)

ψm,j,l , the parameters satisfy
2 2 4 −1 D = O log2 ϵ−1 , 0 + log N + log C0 + log ϵ
3 3 6 −1 W = O log3 ϵ−1 , 0 + log N + log C0 + log ϵ
4 4 8 −1 S = O log4 ϵ−1 , 0 + log C0 + log N + log ϵ 2 2 4 −1 B = exp O log2 ϵ−1 0 + log N + log C0 + log ϵ

.

.

(N )

Regarding ψm,j,l , the additional composition with µ bt gives
2 2 4 −1 D = O log2 ϵ−1 + log C + log N + log ϵ , 0 0
3 3 6 −1 W = O log3 ϵ−1 , 0 + log C0 + log N + log ϵ
4 4 8 −1 , S = O log4 ϵ−1 0 + log C0 + log N + log ϵ 2 2 4 −1 B = exp O log2 ϵ−1 0 + log C0 + log N + log ϵ (σ )

t Similarly, for ψm,j,l , we obtain

2 2 4 −1 D = O log2 ϵ−1 , 0 + log C0 + log N + log ϵ
3 3 6 −1 W = O log3 ϵ−1 , 0 + log C0 + log N + log ϵ
4 4 8 −1 S = O log4 ϵ−1 , 0 + log C0 + log N + log ϵ

2 −1 B = exp O log ϵ−1 . 0 + log C0 + log N + log ϵ

68

(µ )

t For the component ψm,j,l , the complexity is

2 2 4 −1 , D = O log2 ϵ−1 0 + log N + log C0 + log ϵ
3 3 6 −1 W = O log3 ϵ−1 , 0 + log N + log C0 + log ϵ
4 4 8 −1 , S = O log4 ϵ−1 0 + log C0 + log N + log ϵ 2 2 4 −1 B = exp O log2 ϵ−1 0 + log N + log C0 + log ϵ

. (F )

Therefore, by Lemma E.8 and Lemma E.10, we finally derive the network structure of ψm,j,l as follows
2 2 4 −1 D = O log2 ϵ−1 + log C + log N + log ϵ , 0 0
3 3 6 −1 W = O log3 ϵ−1 , 0 + log C0 + log N + log ϵ
4 4 8 −1 S = O log4 ϵ−1 , 0 + log C0 + log N + log ϵ 2 2 4 −1 B = exp O log2 ϵ−1 0 + log C0 + log N + log ϵ

.

Now, we consider the following ReLU neural network   u X (−1)l+j u 1 1 (F ) ψm,j,l (t, x). ψR (t, x, m, u, j) = 1/2 j j!(2π) 2 l l + 2j + 1 l=0 Then, we have 1 ψR (t, x, m, u, j) − ψ(t, x, m, u, j) ≤ j!(2π)1/2 ≤

u   X u l=0

1 j l 2 (l + 2j + 1)

! · ϵ0

ϵ0 ≲ ϵ0 . j!2j−u (2π)1/2

By Lemma E.10, the parameters of the network ψR satisfy
2 2 4 −1 D = O log2 ϵ−1 , 0 + log C0 + log N + log ϵ
3 3 6 −1 W = O log3 ϵ−1 , 0 + log C0 + log N + log ϵ
4 4 8 −1 S = O log4 ϵ−1 , 0 + log C0 + log N + log ϵ 2 2 4 −1 B = exp O log2 ϵ−1 0 + log C0 + log N + log ϵ

.

The proof is complete. With Lemma A.8, we can construct a ReLU neural network to approximate Ψm,n,u,v (t, x, w).

69

Lemma A.9. Given N ≫ 1, C0 > 0, for any ϵ0 > 0, there exists a ReLU neural network ΨR ∈ G(D, W, S, B) with
2 2 4 −1 , D = O log2 ϵ−1 + log C + log N + log ϵ 0 0
3 3 6 −1 W = O log ϵ−1 (log3 ϵ−1 0 + log C0 + log N + log ϵ ) ,
4 4 8 −1 ), S = O log ϵ−1 (log4 ϵ−1 0 + log C0 + log N + log ϵ

2 2 4 −1 B = exp O log2 ϵ−1 , 0 + log C0 + log N + log ϵ such that |ΨR (t, x, w) − Ψm,n,u,v (t, x, w)| ≲ ϵ0 , which is given by Ψm,n,u,v (t, x, w) = ηn,N (w)

dw  Y

wi −

i=1

dx X k−1 ni vi Y · ψ(t, xi , mi , ui , j), N i=1 j=0

for x ∈ [−C0 , C0 ]dx , w ∈ [0, 1]dw , t ∈ [T1 , T − T1 ]. Proof. By Lemma E.10, we first construct a ReLU neural network ψsum ∈ G(D, W, S, B), which satisfies ψsum (t, x, m, u) =

k−1 X

ψR (t, x, m, u, j).

j=0

Here, the network parameters are
2 2 4 −1 C + log N + log ϵ , D = O log2 ϵ−1 + log 0 R
3 3 6 −1 W = O log ϵ−1 (log3 ϵ−1 R + log C0 + log N + log ϵ ) ,
4 4 8 −1 ), S = O log ϵ−1 (log4 ϵ−1 R + log C0 + log N + log ϵ

2 2 4 −1 B = exp O log2 ϵ−1 , R + log C0 + log N + log ϵ where ϵR is the approximation error of ψR . Then, for any m ∈ [N ]dx and u ∈ Ndx , ∥u∥1 ≤ α(u, v) < Cα , we can construct the following ReLU network ψm,u (t, x) = ψprod (ψsum (t, x1 , m1 , u1 ), · · · , ψsum (t, xdx , mdx , udx ))

70

to approximate the term

dx k−1 Q P

ψR (t, xi , mi , ui , j). The approximation error can be written

i=1 j=0

as ϵψm,u ≤ ϵprod + dx C3dx −1 · kϵR , where C3 =

max

k−1 X

|xi |≤C0 ,1≤i≤dx

|ψ(t, xi , mi , ui , j)|.

j=0

The term |ψ(t, xi , mi , ui , j)| (1 ≤ i ≤ dx ) can be bounded as follows   ui X (−1)l+j ui (xi − mNi µt )ui −l σtl DUl+2j+1 (xi ) − DLl+2j+1 (xi ) 1 · ψ(t, xi , mi , ui , j) = j!(2π)1/2 l=0 2j l l + 2j + 1 µut i +1 ui   X p 1 ui ui −l ≤ j (C + 1) (C log ϵ−1 )l+2j+1 0 2 j!(2π)1/2 µuTi +1 l=0 l p ui   (C log ϵ−1 )ui +2(k−1)+1 X ui (C0 + 3)ui −l ≤ ui +1 j 1/2 l 2 j!(2π) µT l=0 p u +2k−1 (C log ϵ−1 ) i ≤ · (C0 + 4)ui . ui +1 1/2 j!(2π) µT It implies that k−1 X log ϵ−1 )ui +2k−1 1 ui · (C + 4) ψ(t, xi , mi , ui , j) ≤ 0 u +1 i j! (2π)1/2 µT j=0 j=0 p e(C log ϵ−1 )ui +2k−1 ≤ · (C0 + 4)ui . (2π)1/2 µuTi +1

k−1 X

(C

p

Taking ϵprod =

ϵ1 ϵ1 , ϵR = , 2 2dx C3dx −1 k

we have ϵψm,u ≤ ϵ1 . Note that k = O(log ϵ−1 ), therefore, the parameters of ψprod satisfy
2 −1 D = O log ϵ−1 , W = O(1), 1 + log C0 + log ϵ

2 −1 S = O log ϵ−1 , B = exp O(log C0 + log2 ϵ−1 ) . 1 + log C0 + log ϵ Thus we obtain the network parameters of ψm,u , which satisfy
2 2 4 −1 D = O log2 ϵ−1 + log C + log N + log ϵ , 0 1
3 3 6 −1 W = O log ϵ−1 (log3 ϵ−1 1 + log C0 + log N + log ϵ ) ,
4 4 8 −1 S = O log ϵ−1 (log4 ϵ−1 ), 1 + log C0 + log N + log ϵ

2 2 4 −1 B = exp O log2 ϵ−1 . 1 + log C0 + log N + log ϵ 71

We now approximate the complete w-dependent factor ηn,N (w)

dw  Y

wi −

i=1

ni vi . N

(32)

For w ∈ [0, 1], the local weights introduced above have the exact ReLU representations η1,N (w) = 1 − ReLU(N w − 1) + ReLU(N w − 2), ηn,N (w) = ReLU(N w − n + 1) − 2 ReLU(N w − n) + ReLU(N w − n − 1), 2 ≤ n ≤ N − 1, and ηN,N (w) = ReLU(N w − N + 1). Hence, each ηni ,N (wi ) is represented exactly by a fixed-depth, fixed-width ReLU subnetwork whose weights are bounded by O(N ). For each 1 ≤ i ≤ dw , by Lemma E.11, there exists a ReLU neural network  ni  ψ(wi , ni , vi ) = ψprod,w (·, . . . , ·) ◦ wi − | {z } N vi times

n vi

approximating wi − Ni

, with approximation error ϵψni ,vi ,wi . Parallelizing the exact sub-

networks for ηni ,N (wi ), the networks ψ(wi , ni , vi ), and one product network, we construct
ψn,v (w) = ψprod,W ηn1 ,N (w1 ), . . . , ηndw ,N (wdw ), ψ(w1 , n1 , v1 ), . . . , ψ(wdw , ndw , vdw ) . This network approximates the factor in (32), and its error satisfies ϵψn,v ≤ ϵprod,w + dw Cw2dw −1 max ϵψni ,vi ,wi , 1≤i≤dw

where Cw := max{1, 2Cα }. Taking ϵprod,w =

ϵ1 , 2

max ϵψni ,vi ,wi =

1≤i≤dw

ϵ1 2dw Cw2dw −1

gives ϵψn,v ≤ ϵ1 . The parameters of ψn,v (w) satisfy

−1 D = O log ϵ−1 1 + log N , W = O(1), S = O log ϵ1 + log N , B = exp (O(log N )) . The exact local-weight subnetworks add only constant depth and width, and their O(N ) weights are absorbed by the displayed bound.

Finally, we can construct a ReLU neural

network for the term 

 ΨR (t, x, w) = Ψprod ψn,v (w), ψm,u (t, x) . 72

The approximation error of ΨR (t, x, w) satisfies ϵΨ ≤ ϵprod,Ψ + 2C max{ϵψn,v , ϵψm,u } ≤ ϵprod,Ψ + 2Cϵ1 , where C = max Taking ϵprod,Ψ =

ϵ0 2

(d w Y

2vi ,

i=1 ϵ0 and ϵ1 = 4C

dx Y e(C i=1

) p log ϵ−1 )ui +2k−1 · (C0 + 4)ui . ui +1 1/2 (2π) µT

gives ϵΨ ≤ ϵ0 . Because the factors ηni ,N are represented

exactly, they introduce no additional approximation term. The resulting network parameters are
2 2 4 −1 D = O log2 ϵ−1 , 0 + log C0 + log N + log ϵ
3 3 6 −1 W = O log ϵ−1 (log3 ϵ−1 0 + log C0 + log N + log ϵ ) ,
4 4 8 −1 S = O log ϵ−1 (log4 ϵ−1 + log C + log N + log ϵ ), 0 0

2 2 4 −1 B = exp O log2 ϵ−1 . 0 + log C0 + log N + log ϵ This completes the proof. With Lemma A.9, we can construct a ReLU neural network to approximate (30). Lemma A.10. Given N ≫ 1 and C0 > 0, for any ϵ0 > 0, there exists a ReLU neural network gR ∈ G(D, W, S, B) with
2 2 4 −1 D = O log2 ϵ−1 , Ψ + log C0 + log N + log ϵ

3 3 6 −1 , W = O N dx +dw log ϵ−1 log3 ϵ−1 Ψ + log C0 + log N + log ϵ

4 4 8 −1 S = O N dx +dw log ϵ−1 log4 ϵ−1 , Ψ + log C0 + log N + log ϵ

2 2 4 −1 B = exp O log2 ϵ−1 , Ψ + log C0 + log N + log ϵ such that |gR (t, x, w) − g3 (t, x, w)| ≲ ϵ0 for x ∈ [−C0 , C0 ]dx , w ∈ [0, 1]dw , and t ∈ [T1 , T − T1 ]. Proof. By Lemmas A.8 and A.9, each network ΨR is constructed from the local networks containing µ bt and σb2 . Therefore, its approximation error ϵΨ already includes the corresponding t

schedule approximation errors. 73

We can construct a ReLU neural network gR (t, x, w) =

X

X

Cm,n,u,v ΨR (t, x, w)

m,n α(u,v)<Cα

with network parameters
2 2 4 −1 D = O log2 ϵ−1 , Ψ + log C0 + log N + log ϵ

3 3 6 −1 , + log C + log N + log ϵ W = O N dx +dw log ϵ−1 log3 ϵ−1 0 Ψ

4 4 8 −1 S = O N dx +dw log ϵ−1 log4 ϵ−1 , Ψ + log C0 + log N + log ϵ

2 2 4 −1 B = exp O log2 ϵ−1 . Ψ + log C0 + log N + log ϵ Taking ϵΨ =

ϵ0 , (dx + dw )Cα N dx +dw

the approximation error between gR (t, x, w) and (30) satisfies |gR (t, x, w) − g3 (t, x, w)| = gR (t, x, w) −

X

X

Cm,n,u,v Ψm,n,u,v (t, x, w)

m,n α(u,v)<Cα

≤

X

X

|Cm,n,u,v | ϵΨ

m,n α(u,v)<Cα

≲

X

X

m,n α(u,v)<Cα

ϵΨ u!v!

≲ (dx + dw )Cα N dx +dw

ϵ0 (dx + dw )Cα N dx +dw

≲ ϵ0 . Each ΨR approximates the complete local summand containing ηn,N (w), and ϵΨ already accounts for the OU schedule approximation. Thus, the summation introduces no additional approximation term. This completes the proof. In the same way, we can choose a ReLU neural network hR (t, x, w) ∈ Rdx to approximate (31). We omit the proof and give the following lemma.

74

Lemma A.11. Given N ≫ 1, C0 > 0, and 1 ≤ ℓ ≤ dx . For any ϵ0 > 0, there exists a ReLU neural network hR,ℓ ∈ G(Dℓ , Wℓ , Sℓ , Bℓ ) with
2 2 4 −1 , Dℓ = O log2 ϵ−1 + log C + log N + log ϵ 0 Ψ
3 3 6 −1 Wℓ = O N dx +dw log ϵ−1 (log3 ϵ−1 Ψ + log C0 + log N + log ϵ ) ,
4 4 8 −1 ), Sℓ = O N dx +dw log ϵ−1 (log4 ϵ−1 Ψ + log C0 + log N + log ϵ

2 2 4 −1 , Bℓ = exp O log2 ϵ−1 Ψ + log C0 + log N + log ϵ such that |hR,ℓ (t, x, w) − h3,ℓ (t, x, w)| ≲ ϵ0 for x ∈ [−C0 , C0 ]dx , w ∈ [0, 1]dw , t ∈ [T1 , T − T1 ]. Proof. Repeat the construction of Lemma A.9 for each local summand in (31), replacing the last factor ψ by ψ ∇ . Under the same change of variables used in Lemma A.8, the additional factor µtσz−x increases the exponent in the antiderivative from l + 2j + 1 to l + 2j + 2. Hence, t ψ ∇ admits the same product-network construction as ψ. bt and σbt2 as in Lemma A.8, The corresponding network ψR∇ uses the same schedule networks µ while the target ψ ∇ retains the exact coefficients µt and σt2 . Using the same schedule-error allocation gives ψR∇ (t, x, m, u, j) − ψ ∇ (t, x, m, u, j) ≲ ϵΨ , and the increase of the exponent by one does not change the stated network-complexity bounds. The factor ηn,N (w) is represented by the same exact subnetworks. Taking ϵΨ =

ϵ0 , (dx + dw )Cα N dx +dw

the same coefficient summation as in Lemma A.10 gives |hR,ℓ (t, x, w) − h3,ℓ (t, x, w)| ≲ ϵ0 . The exact representation of ηn,N (w) introduces no additional approximation term, and the schedule approximation is already included in ϵΨ . Therefore, the network parameter bounds remain unchanged.

75

Similarly, by applying Lemma A.11 and using the parallelization property of ReLU neural networks, there exists a neural network hR (t, x, w) = [hR,1 (t, x, w), · · · , hR,dx (t, x, w)]⊤ ∈ Rdx that approximates (31). The ℓ2 -norm error between hR (t, x, w) and (31) is bounded by O(ϵ0 ).

A.6

Error Bound for Approximating ∇ log pt (x|w) with ReLU Neural Networks

In this section, we construct two ReLU neural networks to approximate the true score function ∇ log pt (x|w) on different time intervals. The approximation error analysis is inspired by the diffusion approximation arguments of Oko et al. (2023); Fu et al. (2024). Lemma A.12. Let N ≫ 1, there exists a ReLU network s(1) ∈ G(D, W, S, B) with D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N ) that satisfies Z N −2α log N pt (x|w)∥s(1) (t, x, w) − ∇ log pt (x|w)∥2 dx ≲ , t ∈ [T1 , 3N −1 ]. 2 σ Rdx t Moreover, we can take s(1) satisfying ∥s(1) (t, x, w)∥∞ ≲

√

log N . σt

Proof. The approximation error admits a decomposition into three terms Z pt (x|w)∥s(1) (t, x, w) − ∇ log pt (x|w)∥2 dx d x RZ = pt (x|w)∥s(1) (t, x, w) − ∇ log pt (x|w)∥2 dx √ Dt (x)>Cσt

log ϵ−1 1

{z

|

}

I

Z +

√ Dt (x)≤Cσt

log ϵ−1 1

{z

| Z +

II

√ Dt (x)≤Cσt

|

pt (x|w)1{pt (x|w)<ϵp } ∥s(1) (t, x, w) − ∇ log pt (x|w)∥2 dx

log ϵ−1 1

}

pt (x|w)1{pt (x|w)≥ϵp } ∥s(1) (t, x, w) − ∇ log pt (x|w)∥2 dx . {z III

76

}

Bound Term I. By Lemma A.4, there exists a constant C > 0 such that for any ϵ1 > 0, Z 2 (1) Term I ≤ 2 √ −1 pt (x|w) s (t, x, w) dx Dt (x)>Cσt log ϵ1 Z +2 pt (x|w) ∥∇ log pt (x|w)∥2 dx √ log ϵ−1 1

Dt (x)>Cσt

≲

ϵ1 + σt ϵ1 ∥s(1) (t, x, w)∥2∞ . σt √ C

log ϵ−1

1 We then invoke Lemma A.3, which yields that ∥∇ log pt (x|w)∥ ≤ whenever Dt (x) ≤ σt p Cσt log ϵ−1 we choose the function s(1) such that its supremum 1 . Motivated by this bound, √ −1 log ϵ1 . Taking ϵ1 = N −(2α+1) , we have norm satisfies s(1) (t, x, w) ∞ ≲ σt

Term I ≲

ϵ1 log ϵ−1 N −(2α+1) 1 ≲ log N. σt σt

(33)

Bound Term II. Choosing ϵp = N −(2α+1) and leveraging Lemma A.4 as before, the following bound holds: Z Term II ≤ 2

√ Dt (x)≤Cσt

log ϵ−1 1

pt (x|w)1{pt (x|w)<ϵp } ∥s(1) (t, x, w)∥2 dx

Z +2

√ Dt (x)≤Cσt

log ϵ−1 1

pt (x|w)1{pt (x|w)<ϵp } ∥∇ log pt (x|w)∥2 dx

ϵp −1 dx2+2 −1 d2x (1) 2 (log ϵ + ϵ (log ϵ ) p 1 1 ) ∥s (t, x, w)∥∞ 2 σt dx +2 ϵp 2 ≲ 2 (log ϵ−1 1 ) σt dx +2 N −(2α+1) ≲ log 2 N. 2 σt

(34)

≲

√

Bound Term III. Since ∥∇ log pt (x|w)∥ ≲

log N σt

on the region of Term III, there exists √

N a constant C1 > 0 such that ∥∇ log pt (x|w)∥∞ ≤ ∥∇ log pt (x|w)∥ ≤ C1 σlog . By (29), we t

define   h (t, x, w) := max min ′

p h1 (t, x, w) , C log N 1 g1 (t, x, w) ∨ N −(2α+1)

77

 , −C1

p

 log N

.

We decompose Term III into two subterms Z 2 h′ (t, x, w) (1) Term III ≲ pt (x|w)1{pt (x|w)≥ϵp } s (t, x, w) − dx √ σt Dt (x)≤Cσt log ϵ−1 1 | {z } III.A

Z +

√ Dt (x)≤Cσt

log ϵ−1 1

pt (x|w)1{pt (x|w)≥ϵp }

|

h′ (t, x, w) − ∇ log pt (x|w) σt

{z

III.B

2

dx . }

To bound the approximation error in (III.A), we construct the ReLU neural network s(1) . p √ We take C0 = 1 + C (2α + 1) log N = O( log N ). In every network constructed below, we replace the state input x by sclip (x, −C0 , C0 ) and retain the same notation. On the region of Term III, ∥x∥∞ ≤ C0 , so this composition leaves the following estimates unchanged and ensures the required input-clipping identity on the whole space. By Lemmas A.10 and A.11, for any ϵ0 > 0, we can construct two ReLU neural networks s3 (t, x, w) and s4 (t, x, w) such that |s3 (t, x, w) ∨ N −(2α+1) − g1 (t, x, w) ∨ N −(2α+1) | ≤ |s3 (t, x, w) − g1 (t, x, w)| ≤ |s3 (t, x, w) − g3 (t, x, w)| + |g1 (t, x, w) − g3 (t, x, w)| dx

≲ ϵ0 + ϵ log 2 ϵ−1 , and ∥s4 (t, x, w) − h1 (t, x, w)∥ ≤ ∥s4 (t, x, w) − h3 (t, x, w)∥ + ∥h1 (t, x, w) − h3 (t, x, w)∥ ≲ ϵ0 + ϵ log

dx +1 2

ϵ−1 .

Note that g1 (t, x, w) ∨ N −(2α+1) can be rewritten using the ReLU function as g1 (t, x, w) ∨ N −(2α+1) = ReLU(g1 (t, x, w) − N −(2α+1) ) + N −(2α+1) , which is approximated by s5 (t, x, w) := ReLU(s3 (t, x, w) − N −(2α+1) ) + N −(2α+1) . For 1 ≤ i ≤ dx , we define s6 (t, x, w) to approximate [h′ (t, x, w)]i as   p p [s6 (t, x, w)]i := sclip sprod,1 ([s4 (t, x, w)]i , srec,1 (s5 (t, x, w))) , −C1 log N , C1 log N . 78

For a fixed ηsch > 0, Lemmas A.7 and E.14 provide the ReLU network
σbt2 := sclip 1 − E4,T,ηsch (t), σT21 , 1 , Set qt := srec,2



sup t∈[T1 ,3N −1 ]

σbt2 − σt2 ≤ ηsch .

  ϵroot + √ηϵsch root 2 b sroot σt , δσ := ϵrec,2 + . ϵ2rec,2

Then sup

qt −

t∈[T1 ,3N −1 ]

1 ≤ δσ . σt

For b ≥ 0, define clipR (z, b) := −b + ReLU(z + b) − ReLU(z − b). This map is an exact ReLU network and satisfies clipR (z, b) = max{min{z, b}, −b} even when b is the output of another network. Subsequently, s(1) (t, x, w) is defined to approximate [h′ (t,x,w)]i σt

as   p [s(1) (t, x, w)]i := clipR sprod,2 ([s6 (t, x, w)]i , qt ) , C1 log N (qt + δσ ) .

√ Since qt + δσ ≥ σ1t and |[h′ ]i | ≤ C1 log N , the clipping interval contains the target [h′ ]i /σt . Therefore, ϵs(1) ,i ≤ ϵprod,2 + 2C2 max {ϵs6 ,i , δσ } , where

(

)  Cµ  1 C2 := 1 + max sup , sup |[h′ (t, x, w)]i | = O N 2 , t∈[T1 ,3N −1 ] σt t,x,w

and ϵs6 ,i :=

|[s6 (t, x, w)]i − [h′ (t, x, w)]i | .

sup t∈[T1 ,3N −1 ], x∈[−C0 ,C0 ]dx , w∈[0,1]dw

The error ϵs6 ,i is bounded as follows ( ϵs6 ,i ≲ ϵprod,1 + 2C3 max ϵ0 + ϵ log  with C3 = max

ϵ−1 , ϵrec,1 +

dx 2

−1

ϵ0 + ϵ log ϵ ϵ2rec,1

) ,

 sup |[h1 (t, x, w)]i |, sup |g1 (t, x, w) ∨ N −(2α+1) | = O(1). By taking

t,x,w

ϵ=N

dx +1 2

  3C − 3α+ 2µ +2

t,x,w

, ϵprod,1 =

N −α N −α N −α N −α , ϵprod,2 = , ϵrec,1 = , ϵrec,2 = , 8C2 2 32C2 C3 8C2 79

and

( ϵ0 = min

N −α , 16C2 C3



N −α 32C2 C3

3 )

1 , ϵroot = 2



N −α 8C2

3

3 2 , ηsch = ϵroot ,

we obtain N −α N −α , ϵs6 ,i ≲ , ϵs(1) ,i ≲ N −α . δσ ≤ 4C2 C2 Cµ

−1 Since σT−1 = O(N 2 ), the above choice gives log ηsch = O(log N ). Hence, Lemma A.7 and 1

the additional exact clipping layer do not change the stated network-complexity orders. Moreover, p p [s (t, x, w)]i ≤ C1 log N (qt + δσ ) ≤ C1 log N (1)



1 + 2δσ σt



√ log N . ≲ σt

Subsequently, we can obtain the network parameters of [s(1) (t, x, w)]i , 1 ≤ i ≤ dx :
Di = O(log4 N ), Wi = O(N dx +dw log7 N ), Si = O(N dx +dw log9 N ), Bi = exp O(log4 N ) . Combining [s(1) (t, x, w)]i , 1 ≤ i ≤ dx , and by Lemma E.10, we construct the ReLU neural network ⊤  s(1) (t, x, w) := [s(1) (t, x, w)]1 , [s(1) (t, x, w)]2 , · · · , [s(1) (t, x, w)]dx , with network parameters
D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N ) , and it satisfies ∥s(1) (t, x, w)∥∞ ≲

√

log N . σt

Thus, the approximation error (III.A) is bounded

by Term III.A ≲ dx N −2α ≲ N −2α .

(35)

Next, we bound the approximation error (III.B), we always have pt (x|w) ≥ ϵp = N −(2α+1) , √ p N and in this region, ∥∇ log pt (x|w)∥ ≲ log and ∥x∥∞ ≤ µt + Cσt (2α + 1) log N ≤ σt p √ 1 + C (2α + 1) log N ≲ C0 where we let C0 = O( log N ). Now, we split {x : Dt (x) ≤ p Cσt (2α + 1) log N } into two parts: D1 := {x : Dt (x) = 0} = [0, µt ]dx , D2 := {x : 0 < Dt (x) ≤ Cσt

80

p (2α + 1) log N }.

Region D1 . For x ∈ D1 , we have x ∈ [0, µt ]dx and Lemma A.1 gives pt (x|w) ≳ 1. For 1 ≤ i ≤ dx , we obtain [h′ (t, x, w)]i [h′ (t, x, w)]i − σt [∇ log pt (x|w)]i − [∇ log pt (x|w)]i = σt σt [h1 (t, x, w)]i 1 σt [∇pt (x|w)]i ≲ − −(2α+1) σt g1 (t, x, w) ∨ N pt (x|w) 1 [h1 (t, x, w)]i pt (x|w) − σt [∇pt (x|w)]i (g1 (t, x, w) ∨ N −(2α+1) ) ≲ · σt (g1 (t, x, w) ∨ N −(2α+1) )pt (x|w) [h1 (t, x, w)]i · pt (x|w) − g1 (t, x, w) ∨ N −(2α+1)

[h1 (t, x, w)]i − σt [∇pt (x|w)]i

. σt · pt (x|w) √ Using pt (x|w) ≳ 1, g1 (t, x, w) ∨ N −(2α+1) ≳ 1, and |[h1 (t, x, w)]i | ≲ log N , we obtain √ [h′ (t, x, w)]i log N − [∇ log pt (x|w)]i ≲ pt (x|w) − g1 (t, x, w) ∨ N −(2α+1) σt σt 1 + · [h1 (t, x, w)]i − σt [∇pt (x|w)]i σ (36) √t log N  ≲ pt (x|w) − g1 (t, x, w) σt  + ∥h1 (t, x, w) − σt ∇pt (x|w)∥ . ≲

+

σt · (g1 (t, x, w) ∨ N −(2α+1) )

Here, we use pt (x|w) ≥ N −(2α+1) and |pt (x|w) ∨ N −(2α+1) − g1 (t, x, w) ∨ N −(2α+1) | ≤ |pt (x|w) − g1 (t, x, w)|. Substituting (36) into the integral gives Z

h′ (t, x, w) − ∇ log pt (x|w) σt

2

pt (x|w)1{pt (x|w)≥N −(2α+1) } dx Z
log N ≲ |pt (x|w) − g1 (t, x, w)|2 + ∥h1 (t, x, w) − σt ∇pt (x|w)∥2 dx 2 σt D1 2   Z Z 1 ∥x − µt z∥2 log N − dz ≲ dx [p(z|w) − g(z|w)] exp dx σt2 2σt2 D1 Rdx σt (2π) 2 2!   Z x − µt z ∥x − µt z∥2 + − dz dx dx [p(z|w) − g(z|w)] exp dx +1 2σt2 (2π) 2 Rdx σt 2   Z Z t z∥ 1 + ∥x−µ log N ∥x − µt z∥2 σt2 2 ≲ − dzdx dx |p(z|w) − g(z|w)| exp dx σt2 2σt2 D1 Rdx σt (2π) 2 Z log N ≲ 2 dx N −2α dx σt µt D1 N −2α log N ≲ . σt2 D1

81

Region D2 . For 1 ≤ i ≤ dx , we have [h′ (t, x, w)]i [h1 (t, x, w)]i − [∇ log pt (x|w)]i ≤ − [∇ log pt (x|w)]i σt σt · (g1 (t, x, w) ∨ N −(2α+1) ) |[h1 (t, x, w)]i − σt [∇pt (x|w)]i | ≲ σt · (g1 (t, x, w) ∨ N −(2α+1) ) 1 1 + |[∇pt (x|w)]i | · − . −(2α+1) g1 (t, x, w) ∨ N pt (x|w) For the first term on the right-hand side, we use g1 (t, x, w) ∨ N −(2α+1) ≥ N −(2α+1) to get |[h1 (t, x, w)]i − σt [∇pt (x|w)]i | |[h1 (t, x, w)]i − σt [∇pt (x|w)]i | ≲ . −(2α+1) σt · (g1 (t, x, w) ∨ N ) σt N −(2α+1) t (x|w)]i | = |[∇ log pt (x|w)]i | ≲ For the second term, using |[∇p pt (x|w)

√

log N and g1 (t, x, w)∨N −(2α+1) ≥ σt

N −(2α+1) yields 1 1 − |[∇pt (x|w)]i | · −(2α+1) g1 (t, x, w) ∨ N pt (x|w) √ −(2α+1) log N |pt (x|w) − g1 (t, x, w) ∨ N | ≲ · σt g1 (t, x, w) ∨ N −(2α+1) √ N 2α+1 log N ≲ |pt (x|w) − g1 (t, x, w)|. σt Combining these two bounds gives

√ N 2α+1 log N  h′ (t, x, w) − ∇ log pt (x|w) ≲ |pt (x|w) − g1 (t, x, w)| σt σt  + ∥h1 (t, x, w) − σt ∇pt (x|w)∥ .

Substituting (37) into the integral gives Z 2 h′ (t, x, w) pt (x|w)1{pt (x|w)≥N −(2α+1) } − ∇ log pt (x|w) dx σt D2 Z   N 4α+2 log N 2 2 ≲ |pt (x|w) − g1 (t, x, w)| + ∥h1 (t, x, w) − σt ∇pt (x|w)∥ dx σt2 D2 ! 2 Z Z 1 ∥x − µt z∥2 N 4α+2 log N − dz ≲ dx [p(z|w) − g(z|w)] exp dx σt2 2σt2 D2 Rdx σt (2π) 2 2!   Z x − µt z ∥x − µt z∥2 + − dz dx dx [p(z|w) − g(z|w)] exp dx +1 2σt2 (2π) 2 Rdx σt   Z Z N 4α+2 log N 1 ∥x − µt z∥2 2 ≲ − dz dx |p(z|w) − g(z|w)| exp dx σt2 2σt2 D2 Rdx σt (2π) 2   ! Z 2 ∥x − µt z∥2 ∥x − µ z∥ t 2 + − dz dx. dx |p(z|w) − g(z|w)| exp dx +2 2σt2 (2π) 2 Rdx σt 82

(37)

By setting ϵ = N −(6α+2) in Lemma A.2 and replacing p(z|w) with |p(z|w) − g(z|w)|2 , we obtain Z Z

|p(z|w) − g(z|w)|2 dx

σtdx (2π) 2

D2

Rdx

Z

Z

≲ D2

and Z Z

∥x − µt z∥2 exp − 2σt2



|p(z|w) − g(z|w)|2

  ∥x − µt z∥2 dzdx + N −(6α+2) , exp − 2σt2



√ ∥x−µt z∥∞ ≤O(1)σt log N

dx

σtdx (2π) 2

∥x − µt z∥2 |p(z|w) − g(z|w)|2

dzdx

 ∥x − µt z∥2 dzdx exp − dx 2σt2 σtdx +2 (2π) 2 D2 Rdx   Z Z log N |p(z|w) − g(z|w)|2 ∥x − µt z∥2 exp − ≲ dzdx + N −(6α+2) . dx 2 √ dx 2σ 2 σt (2π) D2 ∥x−µt z∥∞ ≤O(1)σt log N t √ On the clipped region, we have ∥x − µt z∥∞ ≤ Cσt log N , and hence ∥x − µt z∥2∞ ≲ 

σt2 log N ≲ log N . Now take x ∈ D2 . Since Dt (x) > 0, there exists an index j such that either xj < 0 or xj > µt . Since z ∈ [0, 1]dx , if xj < 0, then µt zj ≤ |xj − µt zj | ≤ Cσt

p log N ,

and if xj > µt , then µt (1 − zj ) ≤ |xj − µt zj | ≤ Cσt

p log N .

√ For sufficiently large N , µt = e−2t ≍ 1 for t ∈ [T1 , 3N −1 ]. Hence zj ≤ Cσt log N or √ √ 1 − zj ≤ Cσt log N . For sufficiently large N , Cσt log N ≤ a0 . Hence, for this index j, either zj ≤ a0 or zj ≥ 1 − a0 , which implies z ∈ [0, 1]dx \[a0 , 1 − a0 ]dx . Thus, by Lemma A.6, it holds that Z Z

|p(z|w) − g(z|w)|2

 ∥x − µt z∥2 exp − dzdx dx √ 2σt2 σtdx (2π) 2 D2 ∥x−µt z∥∞ ≤O(1)σt log N   Z Z |p(z|w) − g(z|w)|2 ∥x − µt z∥2 ≲ exp − dzdx dx 2σt2 σtdx (2π) 2 D2 {z∈[0,1]dx \[a0 ,1−a0 ]dx } Z |p(z|w) − g(z|w)|2 dz ≲ {z∈[0,1]dx \[a0 ,1−a0 ]dx }

≲ N −(6α+4) .

83



Thus, we have Z

h′ (t, x, w) − ∇ log pt (x|w) σt

pt (x|w)1{pt (x|w)≥N −(2α+1) }

D2

2

dx

N 4α+2 log N N 4α+2 log2 N −(6α+4) · N + · N −(6α+2) σt2 σt2 N −2α log N ≲ . σt2

(38)

≲

Combining (35) and (38), we finally establish that Term III ≲ N

−2α log N

σt2

. Together with the

bounds for Term I and Term II (cf. (33), (34)), this completes the proof of the lemma. Next, we employ ReLU neural networks to approximate ∇ log pt (x|w) for all t ∈ [2N −1 , T ]. Set t′0 = N −1 . By virtue of the Markov property, the probability density function admits the following integral representation: pt (x|w) =

∥x − µt−t′0 z∥2 pt′0 (z|w) exp − 2 2σt−t ′ Rdx

Z

1 dx

dx (2π) 2 σt−t ′

0

! dz.

0

For t ∈ [2N −1 , T ] and t′0 = N −1 , we have 1≥

2 σt−t ′

0

σt2

′

−1

e4t0 − 1 e4·N − 1 = 1 − 4t ≥ 1 − 4·2N −1 . e −1 e −1

Since the last quantity is bounded away from zero for sufficiently large N . Hence σt−t′0 ≍ σt .

µt−t′0 ≍ µt ,

We first approximate gt′0 and present the following lemma. Lemma A.13. Let N ≫ 1, there exists a constant C > 0 and a function gt′0 such that Z

 21 pt′0 (x|w) − gt′0 (x|w) dx ≲ N −(3α+1) , 2

Rdx

where gt′0 has the following form gt′0 (x|w) =t′0

− k2

X

X

Cm,n,u,v

m,n ∥u∥1 +∥v∥1 <k

 · gm,n,u,v

 x 1 √ + , w 1{∥x∥∞ ≤1+C √log N } , 2(1 + C log N ) 2

with k = ⌊6α + 3⌋ + 1. 84

(39)

Proof. Let k ≥ 1, 0 ≤ l ≤ k. According to Lemma A.3, for any x ∈ Rdx , w ∈ [0, 1]dw , we have k 1 ′ −2 . ≲ t 0 σtk′

∥∂xi1 xi2 ···xik−l ∂wj1 ···wjl pt′0 (x|w)∥ ≲

0

k

Thus pt′0 t′0 2 is k-Hölder continuous on Rdx with some constant Ck . By replacing pt′0 with p2t′ 0 in Lemma A.4, we can claim that there exists a constant C such that ! 12 Z  12 Z p2t′0 (x|w)dx ≤ p2t′0 (x|w)dx ≲ N −(3α+1) . √ √ ∥x∥∞ >1+C log N

∥x∥∞ >µt′ +Cσt′ 0

log N

0

Following a similar approach to Lemma A.5, we let   p k f (x, w) = t′0 2 pt′0 (1 + C log N )(2x − 1)|w , x ∈ [0, 1]dx , w ∈ [0, 1]dw . √ k Here, ∥f ∥H([0,1]dx +dw ) ≲ 2k (1 + C log N )k = O(log 2 N ). Thus, there exists a function gt,0 satisfying k

k

gt,0 (x|w) − t′0 2 pt′0 (x|w) ≲ N −k log 2 N, ∥x∥∞ ≤ 1 + C

p log N ,

where gt,0 (x|w) =

X

X

′ Cm,n,u,v

m,n ∥u∥1 +∥v∥1 <k

 · gm,n,u,v

 x 1 √ + , w 1{∥x∥∞ ≤1+C √log N } , 2(1 + C log N ) 2 k

k

′ and Cm,n,u,v = O(log 2 N ). We then set gt′0 (x|w) = t′0 − 2 gt,0 (x|w), yielding k

k

pt′0 (x|w) − gt′0 (x|w) ≲ N − 2 log 2 N. Choosing k = ⌊6α + 3⌋ + 1 ≥ 3α + 2, we find Z  12 2 pt′0 (x|w) − gt′0 (x|w) dx ≲ N −(3α+1) √ ∥x∥∞ ≤1+C log N

for sufficient large N . Finally, gt′0 satisfies Z  12 2 pt′0 (x|w) − gt′0 (x|w) dx Rdx

Z ≤

√ ∥x∥∞ ≤1+C log N

2

pt′0 (x|w) − gt′0 (x|w) dx +

≲ N −(3α+1) . We have therefore established the desired result. 85

Z √ ∥x∥∞ >1+C log N

 21

p2t′0 (x|w)dx

Based on Lemma A.13, we can construct a ReLU neural network to approximate ∇ log pt (x|w) on [2N −1 , T ]. Lemma A.14. Let N ≫ 1, there exists a ReLU neural network s(2) ∈ G(D, W, S, B) with D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N )

that satisfies Z N −2α log N pt (x|w)∥s(2) (t, x, w) − ∇ log pt (x|w)∥2 dx ≲ , t ∈ [2N −1 , T ]. 2 σ d x R t Moreover, we can take s(2) satisfying ∥s(2) (t, x, w)∥∞ ≲

√

log N . σt

Proof. The proof is similar to the proof of Lemma A.12. We also decompose the approximation error into three terms: Z pt (x|w)∥s(2) (t, x, w) − ∇ log pt (x|w)∥2 dx d x RZ = pt (x|w)∥s(2) (t, x, w) − ∇ log pt (x|w)∥2 dx √ Dt (x)>Cσt

log ϵ−1 1 ,

{z

|

}

I

Z +

√ Dt (x)≤Cσt

log ϵ−1 1

| Z +

{z

}

II

√ Dt (x)≤Cσt

pt (x|w)1{pt (x|w)<ϵp } ∥s(2) (t, x, w) − ∇ log pt (x|w)∥2 dx

log ϵ−1 1

pt (x|w)1{pt (x|w)≥ϵp } ∥s(2) (t, x, w) − ∇ log pt (x|w)∥2 dx .

|

{z

}

III

Bound Term I. To begin with, we analyze Term I using Lemma A.4, there exists a constant C > 0 such that for any ϵ1 > 0, Z 2 (2) Term I ≤ 2 √ −1 pt (x|w) s (t, x, w) dx Dt (x)>Cσt log ϵ1 , Z +2 pt (x|w) ∥∇ log pt (x|w)∥2 dx √ Dt (x)>Cσt

≲

log ϵ−1 1 ,

ϵ1 + σt ϵ1 ∥s(2) (t, x, w)∥2∞ . σt √ C

log ϵ−1

1 We then invoke Lemma A.3, which yields that ∥∇ log pt (x|w)∥ ≤ whenever Dt (x) ≤ σt p (2) Cσt log ϵ−1 such that its supremum 1 . Motivated by this bound, we choose the function s

86

√ (2)

norm satisfies s (t, x, w) ∞ ≲

log ϵ−1 1 . Taking ϵ1 = N −(2α+1) , we have σt

Term I ≲

ϵ1 log ϵ−1 N −(2α+1) 1 ≲ log N. σt σt

Bound Term II. Choosing ϵp = N −(2α+1) , Lemma A.4 gives Z (2) 2 Term II ≤ 2 √ −1 pt (x|w)1{pt (x|w)<ϵp } ∥s (t, x, w)∥ dx Dt (x)≤Cσt log ϵ1 Z pt (x|w)1{pt (x|w)<ϵp } ∥∇ log pt (x|w)∥2 dx +2 √ Dt (x)≤Cσt

log ϵ−1 1

dx dx +2 ϵp (2) 2 2 2 (log ϵ−1 + ϵp (log ϵ−1 1 ) ∥s (t, x, w)∥∞ 1 ) 2 σt dx +2 ϵp 2 ≲ 2 (log ϵ−1 1 ) σt dx +2 N −(2α+1) 2 N. ≲ log σt2 √ Bound Term III. We first take C1 ≥ C 2α + 1 (Replace C in Lemma A.13 with C1 ) √ √ m m and C0 = 1 + C1 log N = O( log N ). Then, we replace z − N with 2(1+C1z√log N ) + 12 − N

≲

in ψ(t, x, m, u, j), and replace α(u, v) < Cα and Cβ , Cγ with α(u, v) ≤ k = O(1) and √ 1 + C1 log N . We keep the notations ! Z 2 ′ z∥ ∥x − µ 1 t−t 0 dz, g1 (t, x, w) := gt′0 (z|w)1{∥z∥∞ ≤1+C1 √log N } · d − dx exp 2 x 2σt−t ′ σt−t Rdx ′ (2π) 2 0 0 ! Z 2 ′ z − x ′ z∥ µ ∥x − µ t−t t−t 0 gt′0 (z|w)1{∥z∥∞ ≤1+C1 √log N } · d +10 h1 (t, x, w) := − dz, dx exp 2 x 2σ d σt−t′ (2π) 2 R x t−t′0 0     p p h1 (t, x, w) ′ h (t, x, w) := max min , O( log N ) , −O( log N ) . g1 (t, x, w) ∨ N −(2α+1) Applying the argument of Lemma A.12, we can construct a ReLU neural network s(2) with network parameters D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N ) that satisfy s(2) (t, x, w) − and

p h′ (t, x, w) ≲ N −α , ∥x∥∞ ≤ 1 + C1 log N , σt−t′0 ∞

√ √ log N log N ∥s (t, x, w)∥∞ ≲ ≲ (By (39)). σt−t′0 σt (2)

87

Therefore, we have Z √ Dt (x)≤Cσt

log ϵ−1 1

pt (x|w)1{pt (x|w)≥N −(2α+1) }

h′ (t, x, w) s (t, x, w) − σt−t′0 (2)

2

dx ≲ dx N −2α ≲ N −2α .

By (37) in Lemma A.12, √ N 2α+1 log N  h′ (t, x, w) − ∇ log pt (x|w) ≲ |pt (x|w) − g1 (t, x, w)| σt−t′0 σt  + ∥h1 (t, x, w) − σt−t′0 ∇pt (x|w)∥ . Then, we have Z √ Dt (x)≤Cσt

pt (x|w)1{pt (x|w)≥N −(2α+1) }

log ϵ−1 1

N 4α+2 log N ≲ 2 σt−t ′ 0

Z Rdx

h′ (t, x, w) − ∇ log pt (x|w) σt−t′0

2

dx

|pt (x|w) − g1 (t, x, w)|2 + ∥h1 (t, x, w) − σt−t′0 ∇pt (x|w)∥2 dx

! ∥x − µt−t′0 z∥2 |pt′0 (z|w) − gt′0 (z|w)|2 1+ dx 2 dx σt−t dx dx ′ σt−t R R ′ (2π) 2 0 0 0   ∥x − µt−t′0 z∥2 × exp − dzdx 2 2σt−t ′ 0 Z N 4α+2 log N 2 pt′0 (z|w) − gt′0 (z|w) dz ≲ 2 σt−t′ Rdx N 4α+2 log N ≲ 2 σt−t ′

Z

Z

0

N 4α+2 log N · N −(6α+2) ≲ σt2 N −2α log N ≲ . σt2 Therefore, we finally obtain N −2α log N , Term III ≲ σt2 which implies that Z

pt (x|w)∥s(2) (t, x, w) − ∇ log pt (x|w)∥2 dx ≲

Rdx

N −2α log N . σt2

The proof is complete. Combining Lemma A.12 and Lemma A.14, we immediately obtain Lemma 4.2.

88

Proof of Lemma 4.2. By Lemma A.12 and Lemma A.14, there exist two ReLU neural networks s(1) (t, x, w) and s(2) (t, x, w) approximating the score function ∇ log pt (x|w) over the time intervals [T1 , 3N −1 ] and [2N −1 , T ], respectively. Combined with Lemma E.11 and E.15, we are able to assemble a piecewise ReLU neural network via switching weighting:   s(t, x, w) : = sprod sswitch,1 (t, 2N −1 , 3N −1 ), s(1) (t, x, w)   −1 −1 (2) + sprod sswitch,2 (t, 2N , 3N ), s (t, x, w) . Lemma E.11 with ϵ = N −α and ϵ0 = 0, gives s − sswitch,1 s(1) − sswitch,2 s(2) ∞ ≤ 2N −α . The product networks add only O(log N ) depth and size per coordinate, which are dominated by the original network complexities. Hence, D = O(log4 N ),

W = O(N dx +dw log7 N ),

S = O(N dx +dw log9 N ),

B = exp O(log4 N ) .

We next establish the approximation error. Since sswitch,1 + sswitch,2 = 1, Z pt (x|w) ∥s − ∇ log pt (x|w)∥2 dx d R x Z 2 2 ≲ sswitch,1 pt (x|w) s(1) − ∇ log pt (x|w) dx dx RZ 2 + s2switch,2 pt (x|w) s(2) − ∇ log pt (x|w) dx + N −2α Rdx

≲ s2switch,1 + s2switch,2

N −2α log N N −2α log N −2α + N ≲ . σt2 σt2

Here, we use 0 ≤ sswitch,1 , sswitch,2 ≤ 1, sswitch,1 + sswitch,2 = 1, and σt2 ≤ 1. This completes the proof.

A.7

Statistical Error of the Score Function

Recall that in Section 4.1.1, the auxiliary losses are given by nx nx 1 X 1 X b x̄0,i , W i ), b LSx (s) = ℓ(s, x̄0,i , W i ), LSx ,T,Z (s) = ℓ(s, nx i=1 nx i=1

where 1 ℓ(s, x̄0 , W ) = T − 2T1

Z T −T1 EZ T1

89

Z s t, x̄0 µt + Zσt , W + σt

2

dt,

and

m

x X
Zj b x̄0 , W ) = 1 ℓ(s, s tj , x̄0 µtj + Zj σtj , W + mx j=1 σtj

2

.

Then, for any   L(b s) − L(s∗ ) = L(b s) − 2LSx (b s) + L(s∗ ) + 2 LSx (b s) − L(s∗ )     = L(b s) − 2LSx (b s) + L(s∗ ) + 2 LSx (b s) − LbSx ,T,Z (b s) + 2 LbSx ,T,Z (b s) − L(s∗ )     ∗ ∗ b b ≤ L(b s) − 2LSx (b s) + L(s ) + 2 LSx (b s) − LSx ,T,Z (b s) + 2 inf LSx ,T,Z (s) − L(s ) . s∈G

Taking expectations, followed by taking the infimum over s ∈ G on both sides of the above inequality, it holds that   Z T −T1 1 2 ESx ,T,Z Ex̄t ,W ∥b s(t, x̄t , W ) − ∇x̄t log pt (xt |W )∥ dt T − 2T1 T1 h i = ESx ,T,Z L(b s) − L(s∗ ) h i h i h i ∗ ∗ b ≤ ESx ,T,Z L(b s) − 2LSx (b s) + L(s ) + 2ESx ,T,Z LSx (b s) − LSx ,T,Z (b s) + 2 inf L(s) − L(s ) . s∈G

In the above inequality, the terms h

i h i b ESx ,T,Z L(b s) − 2LSx (b s) + L(s ) , 2ESx ,T,Z LSx (b s) − LSx ,T,Z (b s) ∗

and h

i inf L(s) − L(s ) ∗

s∈G

denote the statistical error and approximation error, respectively. We begin by providing an upper bound for ℓ(s, x̄0 , W ). Upper bound for ℓ(s, x̄0 , W ) First, we have EZ s(t, x̄0 µt + Zσt , W ) +

Z σt

2

≲

90

log N E∥Z∥2 log N + ≲ . 2 2 σt σt 1 − e−4t

Then, substituting the above estimate yields Z T −T1
Z 2 1 ℓ(s, x̄0 , W ) = EZ s t, x̄0 µt + Zσt , W + dt T − 2T1 T1 σt Z T −T1 log N 1 ≲ dt T − 2T1 T1 1 − e−4t log(e4(T −T1 ) − 1) − log(e4T1 − 1) ≲ log N · T − 2T1 Cλ log N ≲ log N · Cλ log N − 2N −Cµ ≤ log N for all sufficiently large N . Lipschitz continuity for ℓ(s, x̄0 , W ). Now we derive the Lipschitz continuity for ℓ(s, x̄0 , W ). Note that we restrict ReLU neural networks into class G. For any s1 , s2 ∈ G, by the construction structure of s1 , s2 , we have ℓ(s1 , x̄0 , W ) − ℓ(s2 , x̄0 , W )   Z T −T1 1 2Z ≤ EZ ∥s1 − s2 ∥ · s1 + s2 + dt T − 2T1 T1 σt !1 Z T −T1 2 2 1
2Z 1 dt ≤ EZ ∥s1 − s2 ∥2 2 EZ s1 + s2 + T − 2T1 T1 σt  1 Z T −T1
1 log N 1 dx 2 2 2 ≲ EZ ∥s1 − s2 ∥ + 2 dt T − 2T1 T1 σt2 σt  12   21  Z T −T1 Z T −T1 1 log N 1 ≲ EZ ∥s1 − s2 ∥2 dt dt T − 2T1 T1 T − 2T1 T1 σt2   12 Z T −T1 1 1 ≲ log 2 N · EZ ∥s1 − s2 ∥2 dt T − 2T1 T1 1

≲ log 2 N · ∥s1 − s2 ∥L∞ ([T1 ,T −T1 ]×Rdx ×[0,1]dw ) 1

≲ log 2 N · ∥s1 − s2 ∥L∞ ([N −Cµ ,Cλ log N ]×[−O(1)√log N ,O(1)√log N ]dx ×[0,1]dw ) .

91

Covering number evaluation. The covering number of the neural network class is evaluated as follows: log N G, δ, ∥ · ∥L∞ ([T1 ,T −T1 ]×Rdx ×[0,1]dw )

≲ log N G, δ, ∥ · ∥L∞ ([N −Cµ ,Cλ log N ]×[−O(1)√log N ,O(1)√log N ]dx ×[0,1]dw ) √   DWB(Cλ log N ∨ O(1) log N ∨ 1) ≲ SD log δ   1 13 4 dx +dw ≲N log N log N + log , δ

where we use
D = O(log4 N ), W = O(N dx +dw log7 N ), S = O(N dx +dw log9 N ), B = exp O(log4 N ) . The details of this derivation can be found in Chen et al. (2022b, Lemma 5.3). x Proof of Lemma 4.3. Let ℓ̃(s, x̄0 , W ) = ℓ(s, x̄0 , W ) − ℓ(s∗ , x̄0 , W ) and S′x = {W ′i , x̄′0,i }ni=1

be an independent copy of Sx , then for any s1 , s2 ∈ G, there exists a constant C1 > 0 such that ℓ̃(s1 , x̄0 , W ) − ℓ̃(s2 , x̄0 , W ) = ℓ(s1 , x̄0 , W ) − ℓ(s2 , x̄0 , W ) 1

≤ C1 log 2 N ∥s1 − s2 ∥L∞ ([T1 ,T −T1 ]×Rdx ×[0,1]dw ) . h i We first estimate ESx ,T,Z L(b s) − 2LSx (b s) + L(s∗ ) . It follows that h i ESx ,T,Z L(b s) − 2LSx (b s) + L(s∗ ) # " nx   1 X = ESx ,T,Z ES′x ℓ(b s, x̄′0,i , W ′i ) − ℓ(s∗ , x̄′0,i , W ′i ) nx i=1 ! nx   2 X − ℓ(b s, x̄0,i , W i ) − ℓ(s∗ , x̄0,i , W i ) nx i=1 " # nx 1 X = ESx ,T,Z G(b s, x̄0,i , W i ) , nx i=1 where, for any s ∈ G, define h i G(s, x̄0,i , W i ) := ES′x ℓ̃(s, x̄′0,i , W ′i ) − 2ℓ̃(s, x̄0,i , W i ).

92

Let Gδ be the δ-covering of G with minimum cardinality Nδ := N (G, δ, ∥ · ∥L∞ ([T1 ,T −T1 ]×Rdx ×[0,1]dw ) ), then for any s ∈ G, there exists a sδ ∈ Gδ such that 1

|ℓ̃(s, x̄0 , W ) − ℓ̃(sδ , x̄0 , W )| ≤ C1 log 2 N ∥s − sδ ∥L∞ ([T1 ,T −T1 ]×Rdx ×[0,1]dw ) 1

≤ C1 δ log 2 N. Therefore, 1

G(b s, x̄0,i , W i ) ≤ G(sδ , x̄0,i , W i ) + 3C1 δ log 2 N. Since |ℓ̃(sδ , x̄0,i , W i )| ≲ log N , there exists a constant C2 > 0 such that |ℓ̃(sδ , x̄0,i , W i )| ≤ C2 log N . We have that |ℓ̃(sδ , x̄0,i , W i )−Eℓ̃(sδ , x̄0,i , W i )| ≤ 2C2 log N . Let σ 2 = Var[ℓ̃(sδ , x̄0,i , W i )], then 2  σ 2 ≤ ESx ℓ̃(sδ , x̄0,i , W i )   Z T −T1 1 2 ∗ 2 ≤ C1 log N · ESx EZ ∥sδ − s ∥ dt T − 2T1 T1   = C12 log N · L(sδ ) − L(s∗ )   = C12 log N ESx ℓ(sδ , x̄0,i , W i ) − ℓ(s∗ , x̄0,i , W i ) = C12 log N ESx [ℓ̃(sδ , x̄0,i , W i )]. We obtain ESx [ℓ̃(sδ , x̄0,i , W i )] ≥

σ2 . C12 log N

For each fixed sδ ∈ Gδ , define " F (Sx , x̄0,i , W ) := ESx

#

nx 1 X

n

x 1 X ℓ̃(sδ , x̄0,i , W ) − ℓ̃(sδ , x̄0,i , W i ). nx i=1 nx i=1

93

By Bernstein’s inequality (See Lemma E.6), we have " # nx 1 X PSx ,T,Z G(sδ , x̄0,i , W i ) > t nx i=1 h

i

!

= PSx ,T,Z ES′x ℓ̃(sδ , x̄′0,i , W ′ ) − 2ℓ̃(sδ , x̄0,i , W ) > t " #! nx X t 1 = PSx ,T,Z F (S′x , x̄′0,i , W ′i ) > + ES′x ℓ̃(sδ , x̄′0,i , W ′i ) 2 2n i=1 " #! nx X 1 t ℓ̃(sδ , x̄0,i , W i ) = PSx ,T,Z F (Sx , x̄′0,i , W i ) > + ESx 2 2n i=1   t σ2 ≤ PSx ,T,Z F (Sx , x̄0,i , W i ) > + 2 2C12 log N ! nv 2 ≤ exp − 2σ 2 + 4vC23log N ! nt
≤ exp − , 8 C12 + C32 log N 2

where v = 2t + 2C 2σlog N , and we use v ≥ 2t and σ 2 ≤ 2vC12 log N . Hence, for any t > 1

1 2

3C1 δ log N , we have " PSx ,T,Z

nx 1 X

nx i=1 "

# G(b s, x̄0,i , W i ) > t

≤ PSx ,T,Z sup

nx 1 X

s∈G nx

" ≤ PSx ,T,Z max

1

" sδ ∈Gδ

≤ Nδ exp −

G(s, x̄0,i , W i ) > t

i=1 nx X

sδ ∈Gδ nx

≤ Nδ max PSx ,T,Z

#

# 1 2

G(sδ , x̄0,i , W i ) > t − 3C1 δ log N

i=1 nx 1 X

nx i=1

# 1 2

G(sδ , x̄0,i , W i ) > t − 3C1 δ log N 1 2

nx (t − 3C1 δ log N )
8 C12 + C32 log N

94

! .

1

By setting ϵ = 3C1 δ log 2 N + ϵ0 , δ = n1x and ϵ0 = "

C

8(C12 + 32 ) log N log Nδ , then we obtain nx

#

nx 1 X

G(b s, x̄0,i , W i ) nx i=1 ! 1 Z ∞ 2 nx (t − 3C1 δ log N )
dt ≤ ϵ + Nδ exp − 8 C12 + C32 log N ϵ
1 3C1 log 2 N + 8 C12 + C32 log N log Nδ ≤ nx
C2 2 8 C1 + 3 log N + nx   1 log N 14 4 dx +dw ≲N log N log N + log δ+ δ nx
14 4 dx +dw log N log N + log nx N ≲ . nx

ESx ,T,Z

(40)

Next, we bound nx  i h1 X i b b ESx ,T,Z LSx (b s) − LSx ,T,Z (b s) = ESx ,T,Z ℓ(b s, x̄0,i , W i ) − ℓ(b s, x̄0,i , W i ) . nx i=1

h

We decompose n1x

Pnx  i=1

 b s, x̄0,i , W i ) into the following three terms ℓ(b s, x̄0,i , W i ) − ℓ(b

nx  nx    X 1 X b s, x̄0,i , W i ) = 1 ℓ(b s, x̄0,i , W i ) − ℓ(b ℓ(b s, x̄0,i , W i ) − ℓtrunc (b s, x̄0,i , W i ) nx i=1 nx | i=1 {z } I

+

nx  1 X

nx | i=1

 ℓtrunc (b s, x̄0,i , W i ) − ℓbtrunc (b s, x̄0,i , W i ) {z

}

II

nx   1 X trunc b b + ℓ (b s, x̄0,i , W i ) − ℓ(b s, x̄0,i , W i ) , nx i=1 {z } | III

where ℓtrunc (b s, x̄0 , W ) = EZ and

1 T − 2T1

Z T −T1 T1

Z b s(t, x̄0 µt + Zσt , W ) + σt

m

x Zj 1 X trunc b b ℓ (b s, x̄0 , W ) = s(tj , x̄0 µtj + Zj σtj , W ) + mx j=1 σtj

95

2

2

dt · 1{∥Z∥∞ ≤M }

1{∥Zj ∥∞ ≤M } .

! ,

We analyze the three terms individually. To begin with, we have nx   1 X Term I = ℓ(b s, x̄0,i , W i ) − ℓtrunc (b s, x̄0,i , W i ) nx i=1 ! Z T −T1 nx 2 1 X 1 Z b = EZ s(t, x̄0,i µt + Zσt , W i ) + dt · 1{∥Z∥∞ >M } nx i=1 T − 2T1 T1 σt Z T −T1 
 1 1 2 dt ≲ log N P(∥Z∥∞ > M ) + E ∥Z∥ 1{∥Z∥∞ >M } · T − 2T1 T1 σt2 1

1

≲ log N P(∥Z∥∞ > M ) + [E(∥Z∥4 )] 2 P(∥Z∥∞ > M ) 2 1

≲ log N P(∥Z∥∞ > M ) 2   M2 ≲ log N exp − . 4

Therefore, there exists a constant C3 > 0 such that !   nx   M2 1 X trunc ℓ(b s, x̄0,i , W i ) − ℓ (b s, x̄0,i , W i ) ≤ C3 log N exp − ESx ,T,Z . nx i=1 4 Let h(s, t, x̄0,i , Z) := ∥s(t, x̄0,i µt + Zσt , W i ) + σZt ∥2 1{∥Z∥∞ ≤M } , then there exists a constant C4 > 0 such that 0 ≤ h(s, t, x̄0,i , Z) ≲ 4t

M 2 + log N ≤ C4 N Cµ +4Cλ (M 2 + log N ), 2 σt 4T

1 −1 where we use σt2 = 1 − e−4t = e e4t−1 ≥ e e4T . For any δ1 > 0, s ∈ G, there exists a sδ1 ∈ Gδ1

and a constant C5 > 0 such that 2Z h(s, t, x̄0,i , Z) − h(sδ1 , t, x̄0,i , Z) ≲ δ1 s + sδ1 + 1{∥Z∥∞ ≤M } σt √ M + log N ≲ · δ1 σt p Cµ ≤ C5 N 2 +2Cλ (M + log N )δ1 . Then, for fixed x̄0,i and W i , we have Z T −T1 mx 1 1 X EZ h(b s, t, x̄0,i , Z)dt − h(b s, tj , x̄0,i , Zj ) T − 2T1 T1 mx j=1 ! Z T −T1 mx X 1 1 ≤ sup EZ h(s, t, x̄0,i , Z)dt − h(s, tj , x̄0,i , Zj ) T − 2T1 T1 mx j=1 s∈G ! Z T −T1 mx 1 1 X ≤ max EZ h(sδ1 , t, x̄0,i , Z)dt − h(sδ1 , tj , x̄0,i , Zj ) sδ ∈Gδ1 T − 2T1 T1 mx j=1 p Cµ + 2C5 N 2 +2Cλ (M + log N )δ1 . 96

Let Cµ

f (M, N ) := C4 N Cµ +4Cλ (M 2 + log N ), g(M, N ) := 2C5 N 2 +2Cλ (M +

p log N )δ1 .

For t > g(M, N ), using Hoeffding’s inequality (See Lemma E.6) implies ! Z T −T1 mx 1 1 X PT,Z EZ h(b s, t, x̄0,i , Z)dt − h(b s, tj , x̄0,i , Zj ) > t T − 2T1 T1 mx j=1 ! ! Z T −T1 mx 1 1 X EZ h(sδ1 , t, x̄0,i , Z)dt − h(sδ1 , tj , x̄0,i , Zj ) > t − g(M, N ) ≤ PT,Z max sδ1 ∈Gδ1 T − 2T1 T1 mx j=1 ! Z T −T1 mx 1 1 X h(sδ1 , tj , x̄0,i , Zj ) > t − g(M, N ) ≤ Nδ1 max PT,Z EZ h(sδ1 , t, x̄0,i , Z)dt − sδ1 ∈Gδ1 T − 2T1 T1 mx j=1   2   2mx t − g(M, N )  ≤ Nδ1 exp − . f 2 (M, N ) Thus, by taking the expectation with respect to T and Z, we deduce that for any ϵ > 0, Term II satisfies the following bound   ET,Z ℓtrunc (b s, x̄0,i , W i ) − ℓbtrunc (b s, x̄0,i , W i ) Z +∞   = PT,Z ℓtrunc (b s, x̄0,i , W i ) − ℓbtrunc (b s, x̄0,i , W i ) > t dt 0

Z +∞

 2mx t2 dt ≤ g(M, N ) + ϵ + Nδ1 exp − 2 f (M, N ) ϵ   √ π 2mx ϵ2 f (M, N ) √ ≤ g(M, N ) + ϵ + Nδ1 exp − 2 . 2 f (M, N ) 2mx 

Thus, we have ! nx   1 X ESx ,T,Z ℓtrunc (b s, x̄0,i , W i ) − ℓbtrunc (b s, x̄0,i , W i ) nx i=1 nx h  i 1 X ESx ET,Z ℓtrunc (b s, x̄0,i , W i ) − ℓbtrunc (b s, x̄0,i , W i ) = nx i=1   √ π 2mx ϵ2 f (M, N ) √ Nδ1 exp − 2 . ≤ g(M, N ) + ϵ + 2 f (M, N ) 2mx The last term can be expressed as Term III = −

nx X mx 1 X Zj b s(tj , x̄0,i µtj + Zj σtj , W i ) + mx nx i=1 j=1 σtj

97

2

1{∥Zj ∥∞ >M } ≤ 0,

which implies   b s, x̄0,i , W i ) ≤ 0. ESx ,T,Z ℓbtrunc (b s, x̄0,i , W i ) − ℓ(b q √ log Nδ1 Combining the above inequalities, by setting ϵ = f (M, N ) , M = 2 log mx , and 2mx δ1 = m1x , we have # nx   1 X b s, x̄0,i , W i ) ℓ(b s, x̄0,i , W i ) − ℓ(b ESx ,T,Z nx i=1   p Cµ M2 Cµ +4Cλ log N exp − ≤ C3 N + 2C5 N 2 +2Cλ (M + log N )δ1 4   √ f (M, N ) π 2mx ϵ2 √ + Nδ1 exp − 2 +ϵ 2 f (M, N ) 2mx Cµ √ √ N Cµ +4Cλ log N + N 2 +2Cλ ( log mx + log N ) ≲ mx p log N1/mx + 1 Cµ +4Cλ √ +N (log mx + log N ) · 2mx √ 13 dx +dw N 2 log 2 N (log2 N + log mx ) Cµ +4Cλ ≲N (log mx + log N ) · √ mx "

≲

N Cµ +4Cλ +

dx +dw 2

13

(41)

3

log 2 N (log3 N + log2 N log mx + log 2 mx ) . √ mx

The proof is complete.

A.8

Error Bound for the Score Function

Using Lemma 4.2 and Lemma 4.3, we prove Theorem 4.1. 1

1

Proof of Theorem 4.1. With the choice N = ⌊nxdx +dw +2α ⌋ + 1 ≲ nxdx +dw +2α from Lemma 4.2, one can bound the approximation error as h i inf L(s) − L(s∗ ) ≲

s∈G

1 T − 2T1

Z T −T1 T1

N −2α log N dt σt2

≲ N −2α log N 2α

−

≲ nx dx +dw +2α log nx . (dx +dw +12α)(α∧1)+4α (dx +dw +2α)(α∧1)

Substituting the choices of N , mx = nx

98

2α , Cµ = α∧1 and Cλ = α into (40)–(41),

we derive the final bound on the statistical error h i h i ESx ,T,Z L(b s) − 2LSx (b s) + L(s∗ ) + 2ESx ,T,Z LSx (b s) − LbSx ,T,Z (b s) 2α

−

−

2α

19

≲ nx dx +dw +2α log18 nx + nx dx +dw +2α log 2 nx . Thus, we finally have   Z T −T1 1 2 ESx ,T,Z Ext ,W ∥b s(T − t, xt , W ) − ∇ log pT −t (xt |W )∥ dt T − 2T1 T1   Z T −T1 1 2 = ESx ,T,Z Ex̄t ,W ∥b s(t, x̄t , W ) − ∇ log pt (x̄t |W )∥ dt T − 2T1 T1 −

2α

−

2α

−

2α

19

−

2α

≲ nx dx +dw +2α log18 nx + nx dx +dw +2α log 2 nx + nx dx +dw +2α log nx ≲ nx dx +dw +2α log18 nx . The proof is complete.

B

Bounds on Optimal Bellman Operator Learning

B.1

Proof of Lemma 4.4

With Lemma E.3, we can prove Lemma 4.4. ew denote the path laws of the true reverse process and Proof of Lemma 4.4. Let Pw and P the continuous time EM interpolation, respectively. The estimates below imply EPw

K−1 X Z tk+1 k=0

tk

For m ≥ 1, define ( τm := inf

∥bb(T − tk , Xtk , w) − b(T − t, Xt , w)∥2 dt < ∞.

t ∈ [T1 , T − T1 ] :

K−1 X Z tk+1 k=0

1{ u ≤ t}

tk

) × ∥bb(T − tk , Xtk , w) − b(T − u, Xu , w)∥2 du ≥ m

∧ (T − T1 ).

Since the common diffusion coefficient is 2Idx , the drift difference stopped at τm satisfies the exponential moment condition in Lemma E.3. Applying Lemma E.3 up to τm and then 99

letting m → ∞, the lower semicontinuity of relative entropy and monotone convergence give K−1 X Z tk+1 1 e KL(Pw ∥Pw ) ≤ EPw ∥bb(T − tk , Xtk , w) − b(T − t, Xt , w)∥2 dt. 8 k=0 tk

Consequently, the data processing inequality and Pinsker’s inequality yield K−1 X Z tk+1 1 TV (e pT1 , pT1 ) ≤ EPw ∥bb(T − tk , Xtk , w) − b(T − t, Xt , w)∥2 dt. 16 k=0 tk 2

For t ∈ [T1 , T − T1 ], the Gaussian convolution score identity and Jensen’s inequality give ∇ log pT −t (xt |w) = −

1 σT −t

E [Z|xt , w] , EP ∥∇ log pT −t (xt |w)∥2 ≤

dx . σT2 −t

N The network class also gives ∥b s(T − tk , ·, w)∥2∞ ≲ σlog . Since 2 T −tk

  b b(T − tk , xtk , w) − b(T − t, xt , w) = 2(xtk − xt ) + 4 b s(T − tk , xtk , w) − ∇ log pT −t (xt |w) , we have 2

EP b b(T − tk , xtk , w) − b(T − t, xt , w)

≲ EP ∥xtk − xt ∥2 +

log N dx + 2 . 2 σT −tk σT −t

Here T − t, T − tk ≥ T1 , so σT −t , σT −tk ≥ σT1 > 0. The OU representation and the bounded support of the initial distribution give

sup t∈[T1 ,T −T1 ]

EP

"K−1 Z X tk+1 k=0

EP ∥xt ∥2 < ∞. It follows that

#   2 (2Idx )−1 b b(T − tk , xtk , w) − b(T − t, xt , w) dt < ∞,

tk

The preceding bounds imply sup

sup

0≤k≤K−1 t∈[tk ,tk+1 ]

  2 EP (2Idx )−1 b b(T − tk , xtk , w) − b(T − t, xt , w) < ∞.

Moreover, the true reverse process and the continuous time EM interpolation are both initialized from pT −T1 (·|w) and have the same diffusion coefficient 2Idx . Therefore, by Pinsker’s inequality, the TV distance between the true terminal distribution pT1 and the discretized terminal distribution peT1 satisfies 2

TV (e pT1 , pT1 ) ≲ KL(pT1 ∥e pT1 ) ≲ EP

"K−1 Z X tk+1 k=0

tk

100

2

b b(T − tk , xtk , w) − b(T − t, xt , w)

# dt .

Hence,   ES,T,Z,W TV2 (e pT1 , pT1 ) # "K−1 Z X tk+1 2 ≲ ES,T,Z,W EP b s(T − tk , xtk , W ) − ∇ log pT −t (xt |W ) dt + ES,T,Z,W

k=0 tk "K−1 Z X tk+1

# 2

EP xtk − xt dt .

tk

k=0

First, we have EP ∥xtk − xt ∥2 = EP ∥x̄T −tk − x̄T −t ∥2 −2(t−tk )

= EP

e

Z T −tk

− 1 x̄T −t + 2

2

e−2(T −tk −u) dB u

T −t

Z T −tk

= 1−e

−2(t−tk ) 2

EP ∥x̄T −t ∥ + 4dx

= 1−e

−2(t−tk ) 2

EP ∥x̄T −t ∥ + dx 1 − e−4(t−tk )

2

e−4(T −tk −u) du

T −t 2

≲ t − tk . The same representation and the Burkholder–Davis–Gundy inequality give, for q ∈ {2, 4, 6}, EP xtk − xt

q

q

≲ (t − tk ) 2 .

(42)

max (tk+1 − tk ), # K−1 "K−1 Z X X tk+1 2 (tk+1 − tk )2 ≲ T ∆ ≲ ∆ log N. EP xtk − xt dt ≲ ES,T,Z,W

Thus, with ∆ =

0≤k≤K−1

k=0

tk

k=0

Moreover, ES,T,Z EW

"K−1 Z X tk+1

# 2

EP b s(T − tk , xtk , W ) − ∇ log pT −t (xt |W ) dt

tk

k=0 "K−1 Z X tk+1

≲ ES,T,Z EW

k=0

tk

# 2

EP b s(T − tk , xtk , W ) − ∇ log pT −tk (xtk |W ) dt

|

{z

}

I

+ ES,T,Z EW

"K−1 Z X tk+1 k=0

{z

}

II

"K−1 Z X tk+1 k=0

|

2

EP ∇ log pT −tk (xtk |W ) − ∇ log pT −tk (xt |W ) dt

tk

| + ES,T,Z EW

#

# 2

EP ∇ log pT −tk (xt |W ) − ∇ log pT −t (xt |W ) dt .

tk

{z III

101

}

(43)

Bound I. By (15), for any ϵ > 0, there exists ∆ϵ > 0 such that, if

max (tk+1 − tk ) ≤ ∆ϵ ,

0≤k≤K−1

then 2α

−

Term I ≲ nx dx +dw +2α log19 nx + ϵ.

(44)

Let E denote the score error integrated over time in (15), and let V∆ denote the integrated mean square variation of the same score residual between tk and t. All expectations are taken over the training randomness, W , and the same OU path. By the Lipschitz bound in the preceding remark and (42), s(T − tk , xtk , W ) − b s(T − t, xt , W )∥2 ≲ L2N (t − tk ). EP ∥b Combining this estimate with the bounds for the true score in (46) and (47) gives V∆ ≲ log N




L2N + N 8Cλ +2Cµ ∆ + N 12Cλ +3Cµ ∆2 + N 16Cλ +4Cµ ∆3 + N 24Cλ +6Cµ ∆8 .

For ∆ ≤ ∆nx , the terms involving ∆2 , ∆3 , and ∆8 are bounded by the first term. Hence, V∆ ≲ N −2α (log N )−20 . Moreover, Theorem 4.1 gives E ≲ N −2α log19 N. Expanding the squared norm and applying the Cauchy–Schwarz inequality yield Term I − E ≤ 2

p 1 EV∆ + V∆ ≲ N −2α (log N )− 2 + N −2α (log N )−20 .

For all sufficiently large N , the last expression is bounded by −

2α

N −2α ≤ nx dx +dw +2α = ϵ. Therefore, the explicit value of ∆nx given in the preceding remark provides the deterministic threshold required in (15). Bound II. Under P, xt ∼ pT −t (·|w). By the Gaussian convolution representation, d

xt = µT −t x̄0 + σT −t Z,

x̄0 ∼ p(·|w),

102

Z ∼ N (0, Idx ),

where x̄0 and Z are independent and x̄0 ∈ [0, 1]dx . Hence, for each 1 ≤ i ≤ dx , (−xt,i )+ ≤ σT −t (−Zi )+ ,

(xt,i − µT −t )+ ≤ σT −t (Zi )+ .

It follows that DT −t (xt ) ≤ σT −t ∥Z∥∞ , and therefore, for every fixed positive integer q, EP [DT −t (xt )q ] ≤ σTq −t E∥Z∥q∞ ≲ σTq −t .

(45)

For θ ∈ [0, 1], let xθ = (1 − θ)xt + θxtk . Since DT −tk is 1-Lipschitz, DT −tk (xθ ) ≤ DT −tk (xtk ) + ∥xθ − xtk ∥ ≤ DT −tk (xtk ) + ∥xtk − xt ∥. By the fundamental theorem of calculus, Jensen’s inequality, and Lemma A.3, ∥∇ log pT −tk (xtk |w) − ∇ log pT −tk (xt |w)∥2 Z 1 2 = ∂x ∇ log pT −tk (xθ |w)(xtk − xt ) dθ 0 Z 1 ≤ ∥∂x ∇ log pT −tk (xθ |w)∥2 dθ ∥xtk − xt ∥2 0 2 Z 1 DT −tk (xθ )2 1 ≲ 4 ∨ 1 dθ ∥xtk − xt ∥2 2 σT −tk 0 σT −tk   1 DT −tk (xtk )4 ∥xtk − xt ∥4 ≲ 4 1+ + ∥xtk − xt ∥2 . σT −tk σT4 −tk σT4 −tk By (42), (45) and the Cauchy–Schwarz inequality, h i
1
1 EP DT −tk (xtk )4 ∥xtk − xt ∥2 ≤ EP DT −tk (xtk )8 2 EP ∥xtk − xt ∥4 2 ≲ σT4 −tk (t − tk ). Taking expectations, we obtain EP ∥∇ log pT −tk (xtk |w) − ∇ log pT −tk (xt |w)∥2 1

1

(EP DT −tk (xtk )8 ) 2 (EP ∥xtk − xt ∥4 ) 2 EP ∥xtk − xt ∥6 ≲ 4 (t − tk ) + + σT −tk σT4 −tk σT4 −tk   σT4 −tk (t − tk ) (t − tk )3 1 ≲ 4 (t − tk ) + + σT −tk σT4 −tk σT4 −tk 1

≲

t − tk (t − tk )3 + . σT4 −tk σT8 −tk

103

!

Consequently, Term II ≲

K−1 X Z tk+1  tk

=

k=0 K−1 X

t − tk (t − tk )3 + σT4 −tk σT8 −tk

 dt

K−1 1 (tk+1 − tk )2 1 X (tk+1 − tk )4 + 2 k=0 σT4 −tk 4 k=0 σT8 −tk K−1

K−1

∆ X tk+1 − tk ∆3 X tk+1 − tk ≤ + . 2 k=0 σT4 −tk 4 k=0 σT8 −tk Then Term II ≲ T ∆σT−4 + T ∆3 σT−8 ≲ ∆N 8Cλ +2Cµ log N + ∆3 N 16Cλ +4Cµ log N. 1 1

(46)

Bound III. For every u ∈ [T − t, T − tk ], σu ≥ σT −t . Moreover, the bounded support of the initial distribution and |∂u µu | ≲ 1 imply Du (xt ) ≲ DT −t (xt ) + |µu − µT −t | ≲ DT −t (xt ) + (t − tk ). Hence, by the preceding moment bound, "   3 # EP DT −t (xt )6 + (t − tk )6 (t − tk )6 Du (xt )2 ∨ 1 ≲ 1 + ≲ 1 + . EP σu2 σT6 −t σT6 −t Since |∂u µu | ≲ 1 and |∂u σu | ≲ σu−1 , by Lemma A.3, we obtain EP ∂u ∇ log pu (xt |w)

2

≲

1 σT6 −t

+

(t − tk )6 . σT12−t

By the Cauchy–Schwarz inequality, 2

EP ∇ log pT −tk (xt |w) − ∇ log pT −t (xt |w) Z T −tk 2 = EP ∂u ∇ log pu (xt |w) du T −t Z T −tk 2 ≤ (t − tk ) EP ∂u ∇ log pu (xt |w) du T −t

(t − tk ) (t − tk )8 + . σT6 −t σT12−t 2

≲ Thus, we conclude that

Term III ≲

K−1 X Z tk+1  k=0

tk

(t − tk )2 (t − tk )8 + σT6 −t σT12−t

 dt

≲ ∆2 N 12Cλ +3Cµ log N + ∆8 N 24Cλ +6Cµ log N. 104

(47)

Combining (43), (44), (46), and (47), we obtain h  i 2α − ES,T,Z EW TV2 peT1 (·|W ), pT1 (·|W ) ≲ nx dx +dw +2α log19 nx + ϵ + ∆ log N + ∆N 8Cλ +2Cµ log N + ∆3 N 16Cλ +4Cµ log N + ∆2 N 12Cλ +3Cµ log N + ∆8 N 24Cλ +6Cµ log N. −

2α

Set ϵ = nx dx +dw +2α , ∆nx = ∆ϵ and  n o 10α(α∧1)+4α 
− ∆ = O min ∆nx , N −(2α+8Cλ +2Cµ ) = O min ∆nx , nx (dx +dw +2α)(α∧1) . Consequently, 2α 
 − ES,T,Z EW TV2 peT1 (·|W ), pT1 (·|W ) ≲ nx dx +dw +2α log19 nx .

B.2

Proof of Lemma 4.5

Proof. By the convexity of the KL divergence and Lemma E.2, we have Z    KL pT −T1 (·|w) N (0, I dx ) = KL pT −T1 (·|x̄0 , w)p(x̄0 |w)dx̄0 N (0, I dx ) Rdx Z   ≲ KL pT −T1 (·|x̄0 , w) N (0, I dx ) p(x̄0 |w)dx̄0 Rdx

=

 1  −4(T −T1 ) e E∥x̄0 ∥2 − dx log(1 − e−4(T −T1 ) ) − dx e−4(T −T1 ) 2

≲ e−4T . By the Pinsker’s inequality, we have   1   TV2 pT −T1 (·|w), N (0, I dx ) ≤ KL pT −T1 (·|w) N (0, I dx ) . 2 By (16), we obtain h  i 2α − ES,T,Z,W TV2 p̌T1 (·|W ), peT1 (·|W ) ≲ e−4T ≲ nx dx +dw +2α . The proof is complete.

105

B.3

Proof of Lemma 4.6

Proof. Since p(·|w) is supported on D = [0, 1]dx , then Z Z Z |pT1 (x|w) − p(x|w)|dx = |pT1 (x|w) − p(x|w)|dx + pT1 (x|w)dx . Rdx D Dc | {z } | {z } A

B

Bound term A. For x ∈ D, since µdTx1

  ∥x − µT1 z∥2 exp − dz = 1, dx d /2 2σT21 Rdx σT1 (2π) x

Z

we have Z p(x|w) − pT1 (x|w) =

µdTx1 p(x|w) − p(z|w) σTdx1 (2π)dx /2

D

|

∥x − µT1 z∥2 exp − 2σT21 {z 



I

dz }

+ (µdTx1 − 1)pT1 (x|w) {z } | II

µdTx1

∥x − µT1 z∥2 + p(x|w) exp − dx d /2 2σT21 Dc σT1 (2π) x {z | Z



III

For the Term I, let z =

 dz . }

x+σT1 u , by Assumption 4.2, we have µT1

µdTx1 p(x|w) − p(z|w)

  ∥x − µT1 z∥2 exp − dz dx 2σT21 σTdx1 (2π)dx /2 D D   Z Z dx µT1 |p(x|w) − p(z|w)| ∥x − µT1 z∥2 exp − dzdx ≤ 2σT21 σTdx1 (2π)dx /2 D D   Z Z dx µT1 ∥x − z∥α∧1 ∥x − µT1 z∥2 ≲ exp − dzdx 2σT21 σTdx1 (2π)dx /2 D D Z Z α∧1 x + σT1 u 2 = x− (2π)−dx /2 e−∥u∥ /2 dudx µT D−x µT 1 1 D σT 1 Z Z α∧1 x + σT1 u 2 ≤ x− (2π)−dx /2 e−∥u∥ /2 dudx µT1 D Rdx # α∧1  α∧1 Z Z " σT1 1 − µ 2 T1 ≲ ∥u∥α∧1 + ∥x∥α∧1 (2π)−dx /2 e−∥u∥ /2 dudx µT1 µT1 D Rdx  α∧1  α∧1 σT1 1 − µT1 ≲ + . µT1 µT 1

Z

Z

106

For the Term II, since pT1 (·|w) is a probability density, we have Z Z dx dx (µT1 − 1)pT1 (x|w) dx = |µT1 − 1| pT1 (x|w)dx D D Z dx ≤ |µT1 − 1| pT1 (x|w)dx Rdx

= |µdTx1 − 1| ≲ 1 − µ T1 . The last inequality follows from µT1 ∈ (0, 1] and 1 − µdTx1 = (1 − µT1 )(1 + µT1 + · · · + µdTx1 −1 ) ≤ dx (1 − µT1 ). Moreover, by Assumption 4.1, the boundary Term III satisfies µdTx1

  ∥x − µT1 z∥2 exp − dzdx p(x|w) dx d /2 2σT21 D Dc σT1 (2π) x   Z Z µdTx1 ∥x − µT1 z∥2 exp − ≤ Bu dzdx dx d /2 2σT21 D Dc σT1 (2π) x  Z  x σT1 = Bu P + ξ∈ / D dx, ξ ∼ N (0, Idx ) µT1 µT1 D    dx Z   X xi µT 1 − x i ≤ Bu P ξi < − + P ξi > dx σ σ T T D 1 1 i=1    dx Z 1   X s − µT1 s ≤ Bu +Φ ds Φ − σT1 σT1 i=1 0 "Z #   Z µT  Z 1 ∞  1 µT 1 − s s Φ − ds + Φ − ds + 1ds ≤ Bu dx σT1 σT1 0 0 µT1

Z

Z

≲ σT1 + (1 − µT1 ).

107

Bound Term B. By Assumption 4.1, we have   Z Z Z 1 ∥x − µT1 z∥2 pT1 (x|w)dx = p(z|w) dxdz exp − dx d /2 2σT21 Dc D Dc σT1 (2π) x Z ≤ Bu P(µT1 z + σT1 ξ ∈ / D) dz, ξ ∼ N (0, Idx ) D d x Z X

     µT1 z i 1 − µT1 z i ≤ Bu P ξi < − + P ξi > dz σT1 σT1 i=1 D    dx Z 1   X µT1 s µT1 s − 1 Φ − ≤ Bu +Φ ds σT1 σT1 i=1 0 "Z    # Z 1 ∞  µT 1 s 1 u ≤ Bu dx Φ − ds + Φ − du σT1 µT1 1−µT1 σT1 0 σT ≲ 1. µT1 Combining the preceding inequalities, for every w, we have    σ α∧1  1 − µ α∧1 σ T1 T1 T + + 1 + (1 − µT1 ). TV pT1 (·|w), p(·|w) ≲ µT1 µT1 µT1 Therefore, "Z h  i EW TV2 pT1 (·|W ), p(·|W ) ≤ EW

2 # |pT1 (x|W ) − p(x|W )| dx

Rdx

" ≲

σT1 µT1

α∧1

 +

1 − µT1 µT1

α∧1

#2 σT1 + (1 − µT1 ) + µT1

≲ T1α∧1 ≲ N −Cµ (α∧1) −

2α

≲ nx dx +dw +2α . Hence, the proof is complete.

B.4

Proofs of Theorem 4.2 and Corollary 4.1

Proof of Theorem 4.2. By applying Lemmas 4.4–4.6 to the three terms, respectively, and replacing p̌T1 (·|X, A) with P(·|X, A), we obtain the desired bound. Similarly, we obtain the following error bound for reward law estimation. 108

Proof of Corollary 4.1. The proof follows the same argument as that of Theorem 4.2, with the next-state variable replaced by the reward variable.

B.5

Proof of Theorem 4.3

Proof. By (9), we obtain  2 ∗ ∗ b T Q(X, A) − T Q(X, A) m

2 1 Xb ≲ E[R(X, A)] − Rj (X, A) m j=1

+γ

2

 E

X ′ ∼P(·|X,A)

m 2 1 X ′ ′ b max Q (X , a ) − max Q(Xj , a ) a′ ∈A m j=1 a′ ∈A ′

′

m

2 1 Xb Rj (X, A) ≲ ER∼R [R(X, A)] − ER∼Rb [R(X, A)] + ER∼Rb [R(X, A)] − m j=1 2

+ EX ′ ∼P(·|X,A) max Q(X ′ , a′ ) − EX ′ ∼P(·|X,A) max Q(X ′ , a′ ) b ′ ′ a ∈A

(48)

2

a ∈A

max Q(X ′ , a′ ) − + EX ′ ∼P(·|X,A) b ′ a ∈A

m 1 X

2

b ′ , a′ ) . max Q(X j ′

m j=1 a ∈A

Since the reward is bounded by Rmax , we have   2 m   2 X 1 b1 (X, A) ≤ Rmax . bj (X, A)  = 1 Var R ESr  ER∼Rb [R(X, A)] − R m j=1 m 4m and

(49)

2

ER∼R(·|X,A) [R] − ER∼R(·|X,A) [R] b Z Z = R R(dR|X, A) − [0,Rmax ]

Z =

2

b R R(dR|X, A)

[0,Rmax ]

(50)

h i2 b R R(dR|X, A) − R(dR|X, A)

[0,Rmax ]

  2 b ≲ Rmax TV2 R(·|X, A), R(·|X, A) . b since Q b ∈ Q, we have For Q = Q,  b ′ , a′ ) − ESx  EX ′ ∼P(·|X,A) max Q(X b ′ a ∈A

m 1 X

2

b X b ′ , a′ )  max Q( j ′

m j=1 a ∈A

 b ′ , a′ ) − ≤ ESx  sup EX ′ ∼P(·|X,A) max Q(X b ′ b Q∈Q



a ∈A

109

m X

2



1 b X b ′ , a′ )  . max Q( j m j=1 a′ ∈A

b ∈ Q, there exists a Q bε ∈ Qε such that Let Qε be an ε-cover of Q under ∥ · ∥∞ . For every Q b−Q bε ∥∞ ≤ ε. Since ∥Q b ′ , a′ ) − max Q bε (x′ , a′ ) ≤ max |Q(x b ′ , a′ ) − Q bε (x′ , a′ )| ≤ ε, max Q(x ′ ′ ′ a ∈A

a ∈A

a ∈A

we obtain m 1 X ′ ′ b b X b ′ , a′ ) sup EX ′ ∼P(·|X,A) max Q(X , a ) − max Q( b j ′ ∈A ′ ∈A a a m b Q∈Q j=1

2

2

m 1 X ′ ′ b b ε (X b ′ , a′ ) + ε2 . ≲ max EX ′ ∼P(·|X,A) max Qε (X , a ) − max Q b j ′ ∈A ′ ∈A a a b m Qε ∈Qε j=1

The Hoeffding’s lemma implies that the centered Monte Carlo fluctuation associated with 2

bε ∈ Qε is sub-Gaussian with variance factor v = Rmax 2 . Therefore, each Q 4m(1−γ)   2 m X bε (X ′ , a′ ) − 1 bε (X bj′ , a′ ) > t max Q max Q PSx  max EX ′ ∼P(·|X,A) b ′ ∈A ′ a a ∈A b m j=1 Qε ∈Qε   t ≤ 2N (Q, ε, ∥ · ∥∞ ) exp − . 2v Integrating the preceding tail bound yields  bε (X ′ , a′ ) − ESx  max EX ′ ∼P(·|X,A) max Q b ′ b ε ∈Qε Q

a ∈A

m X

2



1 b ε (X bj′ , a′ )  max Q ′ a ∈A m j=1

  2 Rmax ≲ log 2N (Q, ε, ∥ · ∥∞ ) . m(1 − γ)2 Combining the preceding inequalities and setting ε = m1 , we obtain " # m X 2 1 b ′ , a′ ) − b X bj′ , a′ ) ESx EX ′ ∼P(·|X,A) max Q(X max Q( b a′ ∈A m j=1 a′ ∈A  2
 1 Rmax 1 ≲ 2+ log N Q, m , ∥ · ∥∞ . m m(1 − γ)2 With the size, depth, width, and weight bound of the ReLU DNN class Q as  dx +da   dx +da   dx +da  S = O n dx +da +2β log5 n , D = O(log n), W = O n dx +da +2β log3 n , B = O n dx +da +2β . By Lemma E.5, we have 
 dx +da 1 log N Q, m , ∥ · ∥∞ ≲ SD log (mBWD) ≲ n dx +da +2β log6 n(log m + log n). 110

Substituting this bound into the preceding estimate yields   2 m X b ′ , a′ ) − 1 b X bj′ , a′ )  ESx  EX ′ ∼P(·|X,A) max Q(X max Q( b ′ a ∈A m j=1 a′ ∈A

(51)

6

2 Rmax

dx +da 1 log n(log m + log n) . + n dx +da +2β 2 2 m (1 − γ) m

≲

b ′ , a′ ) ≤ Rmax , we have Since 0 ≤ max Q(X 1−γ ′ a ∈A

2

E

X ′ ∼P(·|X,A)

Z =

a ∈A

a ∈A

b ′ , a′ ) P(dX ′ |X, A) − max Q(X ′

2

Z

[0,1]dx a ∈A

Z =

b ′ , a′ ) − E ′ b b ′ , a′ ) max Q(X Q(X X ∼P(·|X,A) max ′ ′ ′ b ′ , a′ ) P(dX b max Q(X |X, A) ′

[0,1]dx a ∈A

(52)

i2 ′ b ′ , a′ ) P(dX ′ |X, A) − P(dX b max Q(X |X, A) ′ h

[0,1]dx a ∈A   2 Rmax 2 b ≲ TV P(·|X, A), P(·|X, A) , (1 − γ)2

max For Q = Q∗ , since 0 ≤ max Q∗ (X ′ , a′ ) ≤ R1−γ , we have ′

a ∈A



2

m 1 X



b ′ , a′ )  max Q∗ (X j m j=1 a′ ∈A    1 ∗ ′ ′ = ESx VarX ′ ∼P(·|X,A) max Q (X , a ) b a′ ∈A m 2 Rmax . ≤ 4(1 − γ)2 m

ESx  EX ′ ∼P(·|X,A) max Q∗ (X ′ , a′ ) − b ′ a ∈A

(53)

and 2

EX ′ ∼P(·|X,A) max Q∗ (X ′ , a′ ) − EX ′ ∼P(·|X,A) max Q∗ (X ′ , a′ ) b ′ ′ a ∈A

Z =

max Q∗ (X ′ , a′ ) P(dX ′ |X, A) − ′

[0,1]dx a ∈A

Z =

a ∈A

Z

2 ′ b max Q∗ (X ′ , a′ ) P(dX |X, A) ′

[0,1]dx a ∈A

h i2 ∗ ′ ′ ′ ′ b max Q (X , a ) P(dX |X, A) − P(dX |X, A) ′

[0,1]dx a ∈A   2 Rmax 2 b ≲ TV P(·|X, A), P(·|X, A) , (1 − γ)2

111

(54)

Therefore, by (48)–(52), we have (12) and (13). h  i b − Tb ∗ Q∥ b 2 2 ≲ R2 ES E(X,A) TV2 R(·|X, A), R(·|X, b ES ∥T ∗ Q A) max L (µ) 2 dx +da log6 n(log m + log n) Rmax dx +da +2β n (1 − γ)2 m h 2
i Rmax 2 b E E TV P(·|X, A), P(·|X, A) . + S (X,A) (1 − γ)2

+

and by (48)–(50), (53)–(54) h
i 2 b ES ∥Tb ∗ Q∗ − T ∗ Q∗ ∥2L2 (µ) ≲ Rmax ES E(X,A) TV2 R(·|X, A), R(·|X, A) + +

h 2 Rmax 2 b E E S (X,A) TV P(·|X, A), P(·|X, A) (1 − γ)2

2 Rmax (1 − γ)2 m
i

With (12), (13), Theorem 4.2 and Corollary 4.1, we complete the proof.

C

Bounds on Value Function Learning

C.1

Proof of Lemma 4.7

Proof. By Lemma C.1, for any (x, a) ∈ [0, 1]dw , we have |Tb ∗ Q1 (x, a) − Tb ∗ Q2 (x, a)| ≤ γ ∥Q1 − Q2 ∥∞ . By (17), we establish 2

Tb ∗ Q1 (x, a) − Q1 (x, a) − Tb ∗ Q2 (x, a) − Q2 (x, a)   ≤ Tb ∗ Q1 (x, a) − Q1 (x, a) − Tb ∗ Q2 (x, a) + Q2 (x, a)   · Tb ∗ Q1 (x, a) − Q1 (x, a) + Tb ∗ Q2 (x, a) − Q2 (x, a)

2

|ℓQ1 (x, a) − ℓQ2 (x, a)| =

4Rmax 4Rmax b ∗ T Q1 (x, a) − Tb ∗ Q2 (x, a) + Q1 (x, a) − Q2 (x, a) 1−γ 1−γ 4Rmax (1 + γ) ≤ ∥Q1 − Q2 ∥∞ . 1−γ ≤

112

To bound the statistical error, we employ the technique of offset Rademacher complexity. Starting with the expectation over the distribution D, we obtain that   ED sup LTb ∗ (Q) − 2LbTb ∗ (Q) Q∈Q

= ED sup

! n 2X E(X,A) [ℓQ (X, A)] − ℓQ (Xi , Ai ) n i=1

= ED sup

! n n 3 1 3 X 1 X E(X,A) [ℓQ (X, A)] − E(X,A) [ℓQ (X, A)] − ℓQ (Xi , Ai ) − ℓQ (Xi , Ai ) 2 2 2n i=1 2n i=1

≤ ED sup

! n n X X 3 1 3 1 E(X,A) [ℓQ (X, A)] − E(X,A) [ℓ2Q (X, A)] − ℓQ (Xi , Ai ) − ℓ2 (Xi , Ai ) 2 2C 2n i=1 2Cn i=1 Q

Q∈Q

Q∈Q

Q∈Q

2

4Rmax max and ℓQ (X, A) ≤ (1−γ) where we use the fact that |Q(X, A)| ≤ R1−γ 2 =: C. Now, we introduce

a ghost sample D∗ = {(Xi∗ , A∗i )}ni=1 independent of D. Further, let τ = {τi }ni=1 denote a set of Rademacher variables. Substituting the expectation with the empirical mean derived from the ghost sample D∗ , we obtain ED sup

Q∈Q

! n X 3 1 1 E(X,A) [ℓQ (X, A)] − E(X,A) [ℓ2Q (X, A)] − ℓQ (Xi , Ai ) − ℓ2Q (Xi , Ai ) 2 2C 2n i=1 2Cn i=1 n 3 X

n n 3 1X 1X 2 1 ∗ ∗ ∗ ∗ ED [ ED [ ℓQ (Xi , Ai )] − ℓQ (Xi∗ , A∗i )] 2 n i=1 2C n i=1 ! n n 3 X 1 X 2 − ℓQ (Xi , Ai ) − ℓ (Xi , Ai ) 2n i=1 2Cn i=1 Q

= ED sup

Q∈Q

n n 3 X 1 X 2 ℓQ (Xi∗ , A∗i ) − ℓQ (Xi∗ , A∗i ) 2n i=1 2Cn i=1 Q∈Q ! n n 3 X 1 X 2 − ℓQ (Xi , Ai ) − ℓ (Xi , Ai ) 2n i=1 2Cn i=1 Q

≤ ED,D∗ sup

≤ ED,D∗ ,τ sup

Q∈Q

! n n X 3 X 1 τi ℓQ (Xi∗ , A∗i ) − ℓ2Q (Xi∗ , A∗i ) 2n i=1 2Cn i=1

! n n 3 X 1 X 2 + ED,D∗ ,τ sup − τi ℓQ (Xi , Ai ) − ℓ (Xi , Ai ) 2n i=1 2Cn i=1 Q Q∈Q ! n n 3X 1 X 2 ≤ ED,τ sup τi ℓQ (Xi , Ai ) − ℓ (Xi , Ai ) n i=1 Cn i=1 Q Q∈Q = 3Roff n (Q,

1 ). 3C 113

Let N (Q, δ, ∥ · ∥∞ ) be the covering number of Q with cover Qδ , there exists a Qδ ∈ Qδ such that ∥Q − Qδ ∥∞ ≤ δ, that is, |ℓQ (Xi , Ai ) − ℓQδ (Xi , Ai )| ≤

4Rmax (1 + γ) 4Rmax (1 + γ) ∥Q − Qδ ∥∞ ≤ δ. 1−γ 1−γ

Thus, n n n 1X 1X 1X τi ℓQ (Xi , Ai ) ≤ τi ℓQδ (Xi , Ai ) + |τi ||ℓQ (Xi , Ai ) − ℓQδ (Xi , Ai )| n i=1 n i=1 n i=1 n n 1X 4Rmax (1 + γ) 1 X ≤ τi ℓQδ (Xi , Ai ) + Σ |τi | ∥Q − Qδ ∥∞ n i=1 1−γ n i=1 n

≤

1X 4Rmax (1 + γ) τi ℓQδ (Xi , Ai ) + δ. n i=1 1−γ

Using 0 ≤ ℓQ (X, A) ≤ C and 0 ≤ ℓQδ (X, A) ≤ C, we also have n n n  1X 2 1X 2 1 X 2 ℓQδ (Xi , Ai ) − ℓ2Q (Xi , Ai ) − ℓQ (Xi , Ai ) = − ℓQδ (Xi , Ai ) + n i=1 n i=1 n i=1 n n 1X 2 1X ≤− ℓ (Xi , Ai ) + 2C |ℓQδ (Xi , Ai ) − ℓQ (Xi , Ai )| n i=1 Qδ n i=1

≤−

n n 1X 2 8Rmax (1 + γ)C 1 X ℓQδ (Xi , Ai ) + ∥Q − Qδ ∥∞ n i=1 1−γ n i=1 n

1X 2 8Rmax (1 + γ)C δ. ≤− ℓQδ (Xi , Ai ) + n i=1 1−γ Since τ = {τi }ni=1 is a sequence of i.i.d. Rademacher variables independent of D, then conditionally on D we have Roff n (Q, η|D) ≤ Eτ +

 max

Qδ ∈Qδ

n 1 X

n  ηX 2 τi ℓQδ (Xi , Ai ) − ℓ (Xi , Ai ) n i=1 n i=1 Qδ

4Rmax (1 + γ) (1 + 2ηC)δ, 1−γ

for any η > 0. Recall that {τi ℓQδ (Xi , Ai )}ni=1 are independent random variables conditioning on D, Eτ [τi ℓQδ (Xi , Ai )] = 0, and −ℓQδ (Xi , Ai ) ≤ τi ℓQδ (Xi , Ai ) ≤ ℓQδ (Xi , Ai ), 114

i = 1, . . . , n.

By Hoeffding’s inequality (See Lemma E.6), it yields that for any Qδ ∈ Qδ and ξ > 0, ( n ) n 1X ηX 2 Pτ τi ℓQδ (Xi , Ai ) > ξ + ℓ (Xi , Ai ) n i=1 n i=1 Qδ   2  n P 2  nξ + η i=1 ℓQδ (Xi , Ai )   ≤ exp  n  − P 2 ℓ2Qδ (Xi , Ai ) i=1

≤ exp(−2ηnξ). N For simplicity, we denote N (Q, δ, ∥ · ∥∞ ) as N . Therefore, setting A = log , we have 2ηn n 1X

Eτ max

n ( i=1

Qδ ∈Qδ

Z ∞ ≤

τi ℓQδ (Xi , Ai ) −

max

n 1X

n ηX

n i=1

! ℓ2Qδ (Xi , Ai )

τi ℓQδ (Xi , Ai ) −

n ηX

!) ℓ2Qδ (Xi , Ai ) > ξ

dξ n i=1 n i=1 ( n ) Z ∞ n X X 1 η N max Pτ τi ℓQδ (Xi , Ai ) > ξ + ℓ2Qδ (Xi , Ai ) dξ ≤ Q ∈Q n i=1 n i=1 δ δ 0 Z ∞ ≤A+ N exp(−2ηnξ)dξ Pτ

0

Qδ ∈Qδ

A

N log N log N + exp(−2ηn ) ≤ 2ηn 2ηn 2ηn 1 + log N = . 2ηn 1 By setting δ = n1 , applying Lemma E.5, and choosing η = 3C , we obtain the following bound   ! n n 9C 1 + log N X X 3 1 20Rmax (1 + γ) Eτ |D sup τi ℓQ (Xi , Ai ) − ℓ2Q (Xi , Ai ) ≤ + δ n i=1 Cn i=1 2n 1−γ Q∈Q   2 18Rmax 1 + log N 20Rmax (1 + γ) = + δ (1 − γ)2 n 1−γ  R2 SD log(nBWD)  ≤ O max . (1 − γ)2 n

dx +da Considering the size S = O n dx +da +2β log5 n , network depth D = O(log n), width W =
dx +da dx +da O(n dx +da +2β log3 n), and weight bound B = O n dx +da +2β , we derive the statistical error bound (19).

115

C.2

Proof of Theorem 4.4

We first show that the empirical optimal Bellman operator Tb ∗ is a γ-contraction and then give the proof of Theorem 4.4. Lemma C.1. The operator Tb ∗ defined in (9) is a γ-contraction with respect to the supremum norm. max Proof. Let Q1 , Q2 be arbitrary measurable functions satisfying 0 ≤ Q1 , Q2 ≤ R1−γ . For any

(x, a) ∈ [0, 1]dx × [0, 1]da , Tb ∗ uses the same samples at (x, a) for Q1 and Q2 , so the reward terms cancel and  m 

γ X ∗ ∗ ′ ′ ′ ′ b b b b T Q1 (x, a) − T Q2 (x, a) = max Q1 Xj (x, a), a − max Q2 Xj (x, a), a a′ ∈A m j=1 a′ ∈A m

γ X bj′ (x, a), a′ − max Q2 X bj′ (x, a), a′ max Q1 X ≤ ′ ′ a ∈A m j=1 a ∈A m

γ X b ′ (x, a), a′ − Q2 X b ′ (x, a), a′ max Q1 X ≤ j j ′ m j=1 a ∈A ≤ γ∥Q1 − Q2 ∥∞ . Taking the supremum over (x, a) ∈ [0, 1]dx × [0, 1]da gives Tb ∗ Q1 − Tb ∗ Q2

∞

≤ γ∥Q1 − Q2 ∥∞ .

Since γ < 1, Tb ∗ is a γ-contraction.
dx +da Proof of Theorem 4.4. By Lemma E.4, with S = O n dx +da +2β log5 n , D = O(log n), W =
dx +da dx +da O(n dx +da +2β log3 n), and B = O n dx +da +2β , we have 2β

inf ∥Q − Q∗ ∥2∞ ≲ n− dx +da +2β .

Q∈Q

(55)

Therefore, incorporating the statistical error bound (19) together with the approximation error bound (55) yields the theorem.

D

Main Results

In this section, we present the proof of error decomposition and our main results. 116

Proof of Lemma 4.1. Since Q∗ = T ∗ Q∗ , we have LT ∗ (Q∗ ) = ∥T ∗ Q∗ − Q∗ ∥2L2 (µ) = 0. Therefore, b − LT ∗ (Q∗ ) = ∥T ∗ Q b − Tb ∗ Q b + Tb ∗ Q b − Q∥ b 22 LT ∗ (Q) L (µ) b − Tb ∗ Q∥ b 2 2 + 2L b ∗ (Q) b ≤ 2∥T ∗ Q L (µ) T   b − Tb ∗ Q∥ b 2 2 + 2 L b ∗ (Q) b − 2Lb b ∗ (Q) b + 4Lb b ∗ (Q) b = 2∥T ∗ Q L (µ) T T T   b − Tb ∗ Q∥ b 2 2 + 2 sup L b ∗ (Q) − 2Lb b ∗ (Q) ≤ 2∥T ∗ Q L (µ) T T Q∈Q

+ 4 inf LbTb ∗ (Q), Q∈Q

b = inf Q∈Q Lb b ∗ (Q). Taking expectation over D where the last inequality follows from LbTb ∗ (Q) T gives h

i h i ∗ ∗b ∗b 2 b b ED LT ∗ (Q) − LT ∗ (Q ) ≤ 2ED ∥T Q − T Q∥L2 (µ)   + 2ED sup L b ∗ (Q) − 2Lb b ∗ (Q) + 4ED inf Lb b ∗ (Q). T

Q∈Q

T

Q∈Q

T

Conditional on the learned operator Tb ∗ , the value sample D remains independent and identically distributed according to µ. Therefore, ED inf LbTb ∗ (Q) ≤ inf ED LbTb ∗ (Q) = inf LTb ∗ (Q). Q∈Q

Q∈Q

Q∈Q

For every Q ∈ Q, the identity T ∗ Q∗ = Q∗ gives Tb ∗ Q − Q = Tb ∗ Q − Tb ∗ Q∗ + Tb ∗ Q∗ − T ∗ Q∗ + Q∗ − Q. Hence, 2

LTb ∗ (Q) = Tb ∗ Q − Tb ∗ Q∗ + Tb ∗ Q∗ − T ∗ Q∗ + Q∗ − Q L2 (µ)  2 ≤ 2∥Tb ∗ Q∗ − T ∗ Q∗ ∥2L2 (µ) + 2 ∥Tb ∗ Q − Tb ∗ Q∗ ∥L2 (µ) + ∥Q − Q∗ ∥L2 (µ) ≤ 2∥Tb ∗ Q∗ − T ∗ Q∗ ∥2L2 (µ) + 2(1 + γ)2 ∥Q − Q∗ ∥2∞ . The last inequality uses ∥Tb ∗ Q − Tb ∗ Q∗ ∥L2 (µ) ≤ ∥Tb ∗ Q − Tb ∗ Q∗ ∥∞ ≤ γ∥Q − Q∗ ∥∞

117

and ∥Q − Q∗ ∥L2 (µ) ≤ ∥Q − Q∗ ∥∞ , since µ is a probability measure. Taking the infimum over Q ∈ Q gives 4 inf LTb ∗ (Q) ≤ 8∥Tb ∗ Q∗ − T ∗ Q∗ ∥2L2 (µ) + 8(1 + γ)2 inf ∥Q − Q∗ ∥2∞ . Q∈Q

Q∈Q

Combining the preceding bounds, we obtain i h i h ∗ ∗ 2 b∗ ∗ b − LT ∗ (Q∗ ) ≤ 2ED ∥T ∗ Q b − Tb ∗ Q∥ b 22 ED LT ∗ (Q) L (µ) + 8∥T Q − T Q ∥L2 (µ)   b + 2ED sup LTb ∗ (Q) − 2LTb ∗ (Q) + 8(1 + γ)2 inf ∥Q − Q∗ ∥2∞ . Q∈Q

Q∈Q

This completes the proof. Proof of Theorem 4.5. By the error decomposition of Lemma 4.1, together with Theorems 4.3–4.4, the desired theorem follows directly. Using the similar argument of Chen and Jiang (2019), we can prove Theorem 4.6 below. Proof of Theorem 4.6. Since A = [0, 1]da is compact, every Q ∈ Q is continuous, and Q⋆ ∈ Hβ , the positive-measure condition in Assumption 4.6 ensures that πQ,δ is a well defined measurable stochastic kernel. For every a ∈ GQ,δ (x), we first show that max Q(x, e a) − max Q∗ (x, e a) ≤ |Q(x, a) − Q∗ (x, a)| + δ. e a∈A

e a∈A

Suppose first that max Q(x, e a) ≥ max Q∗ (x, e a). e a∈A

e a∈A

For a ∈ GQ,δ (x), the definition of GQ,δ (x) gives max{Q(x, a), Q∗ (x, a)} ≥ max Q(x, e a) − δ. e a∈A

If Q(x, a) ≥ Q∗ (x, a), then max Q(x, e a) − max Q∗ (x, e a) ≤ max Q(x, e a) − Q∗ (x, a) e a∈A

e a∈A

e a∈A

= max Q(x, e a) − Q(x, a) + Q(x, a) − Q∗ (x, a) e a∈A

≤ δ + |Q(x, a) − Q∗ (x, a)|.

118

(56)

If Q∗ (x, a) > Q(x, a), then max Q(x, e a) − max Q∗ (x, e a) ≤ max Q(x, e a) − Q∗ (x, a) ≤ δ. e a∈A

e a∈A

e a∈A

The opposite ordering follows by interchanging Q and Q∗ . This proves (56). For every ν ∈ Mδ , Jensen’s inequality and (56) give ∗

∥T Q − T

∗

Q∗ ∥2L2 (ν) = γ 2

Z 

Z X ×A

 2 ′ max Q(x , e a) − max Q (x , e a) P(dx |x, a) ν(dx, da) ′

e a∈A

X

′

e a∈A

2

Z

Z

∗

max Q(x , e a) − max Q (x , e a) P(dx′ |x, a)ν(dx, da) e a∈A e a∈A ZX ×A ZX Z 2 ≤ γ2 (|Q(x′ , a′ ) − Q∗ (x′ , a′ )| + δ)

≤γ

2

X ×A

X

′

∗

′

A

× πQ,δ (da′ |x′ )P(dx′ |x, a)ν(dx, da) = γ 2 ∥ |Q − Q∗ | + δ ∥2L2 (P(ν)⊗πQ,δ ) . Taking square roots and applying Minkowski’s inequality give ∥T ∗ Q − T ∗ Q∗ ∥L2 (ν) ≤ γ∥Q − Q∗ ∥L2 (P(ν)⊗πQ,δ ) + γδ.

(57)

The density ratio condition in Assumption 4.6 gives Z dν ∗ 2 ∥Q − T Q∥L2 (ν) = |Q(x, a) − T ∗ Q(x, a)|2 (x, a)µ(dx, da) dµ X ×A Z ≤ Cδ |Q(x, a) − T ∗ Q(x, a)|2 µ(dx, da) X ×A

= Cδ ∥Q − T ∗ Q∥2L2 (µ) . Hence, ∥Q − T ∗ Q∥L2 (ν) ≤

p Cδ ∥Q − T ∗ Q∥L2 (µ) .

Using Q∗ = T ∗ Q∗ together with (57) and (58), we obtain ∥Q − Q∗ ∥L2 (ν) ≤ ∥Q − T ∗ Q∥L2 (ν) + ∥T ∗ Q − T ∗ Q∗ ∥L2 (ν) p ≤ Cδ ∥Q − T ∗ Q∥L2 (µ) + γ∥Q − Q∗ ∥L2 (P(ν)⊗πQ,δ ) + γδ. By the definition of Mδ , P(ν) ⊗ πQ,δ ∈ Mδ . Taking the supremum over ν ∈ Mδ gives sup ∥Q − Q∗ ∥L2 (ν) ≤

p Cδ ∥Q − T ∗ Q∥L2 (µ) + γ sup ∥Q − Q∗ ∥L2 (ν) + γδ.

ν∈Mδ

ν∈Mδ

119

(58)

The supremum is finite because Q and Q∗ are bounded. Rearranging gives √ γδ Cδ ∗ sup ∥Q − Q ∥L2 (ν) ≤ ∥Q − T ∗ Q∥L2 (µ) + . 1−γ 1−γ ν∈Mδ Since M ⊆ Mδ , for every ρ ∈ M, √ Cδ γδ ∥Q − Q ∥L2 (ρ) ≤ ∥Q − T ∗ Q∥L2 (µ) + . 1−γ 1−γ ∗

Applying Jensen’s inequality, and using Q∗ = T ∗ Q∗ yield √ Cδ ∗ b − T ∗ Q∥ b L2 (µ) + γδ b E∥Q E∥Q − Q ∥L2 (ρ) ≤ 1−γ 1−γ √ 1   Cδ γδ 2 b − T ∗ Q∥ b 22 ≤ E∥Q + L (µ) 1−γ 1−γ √ 1  h i Cδ b − LT ∗ (Q∗ ) 2 + γδ E LT ∗ (Q) = 1−γ 1−γ √ 19 γδ Rmax Cδ − d +dβ +2β 2 n + x a ≲ n log , 2 (1 − γ) 1−γ β

where the last inequality follows from Theorem 4.5. Taking δ = δn := n− dx +da +2β and using e we obtain Cδn ≤ C, p e − β β 19 γ R C max b − Q∗ ∥L2 (ρ) ≲ E∥Q n dx +da +2β log 2 n + n− dx +da +2β 2 (1 − γ) 1−γ p e − β 19 Rmax C n dx +da +2β log 2 n. ≲ 2 (1 − γ) This completes the proof. Proof of Theorem 4.7. Since ωn , ωr , and ωx are fixed and positive, n, nr , and nx are all of order n e. Moreover, dx ≥ 1 implies 2 2 2 2α 2α 2α 19 − − 2d +d Rmax Rmax Rmax 19 x a +2α nr 1+dx +da +2α log 2 nr + n log n ≲ n e− 2dx +da +2α log19 n e. x x 2 2 1−γ (1 − γ) (1 − γ)

The choice of m gives dx +da

2β n e dx +da +2β ≲n e− dx +da +2β . m

Substituting these bounds into Theorem 4.5 and Theorem 4.6 proves Theorem 4.7.

120

Proof of Theorem 4.8. Set s = max{α, β} and n e = n + nr + nx . We construct a finite family of MDPs contained in Eα,β . Let every MDP in this family have the common uniform transition density pX (x′ |x, a) = 1, x′ ∈ X . Choose a nonzero nonnegative infinitely differentiable function ψ : Rdx +da → [0, ∞) whose
dx +da support lies strictly inside − 12 , 12 . For sufficiently small h > 0, choose Jh ≍ h−(dx +da ) centers z1 , . . . , zJh ∈ [h, 1 − h]dx +da such that ∥zj − zk ∥∞ ≥ 2h whenever j ̸= k. Then each   z−z translated and rescaled function z 7→ ψ h j vanishes outside [0, 1]dx +da , and no two of these functions are nonzero at the same point. For v ∈ {0, 1}Jh and z = (x, a), define s

fv (z) = c0 h

Jh X j=1

 vj ψ

z − zj h



Z , bv =

max fv (x, a) dx,

X a∈A

where c0 > 0 is a fixed constant chosen below. Since max fv (x, a) ≤ ∥fv ∥∞ , then a∈A

0 ≤ bv ≤ ∥fv ∥∞ ≤ c0 hs ∥ψ∥∞ . For each t ∈ {α, β}, write t = k + τ , where k ∈ N0 and τ ∈ (0, 1]. By the chain rule, the disjointness of the bump regions, and their mutual separation, max ∥∂ ξ fv ∥∞ ≤ c0 max hs−∥ξ∥1 ∥∂ ξ ψ∥∞ ≲ c0 hs−t ≲ c0 ,

∥ξ∥1 ≤k

∥ξ∥1 ≤k

|∂ ξ fv (z) − ∂ ξ fv (z ′ )| ≲ c0 hs−k−τ = c0 hs−t ≲ c0 . ∥ξ∥1 =k z̸=z ′ ∥z − z ′ ∥τ max sup

The last bound follows from s = α ∨ β and 0 < h ≤ 1. Hence, by choosing c0 sufficiently small, the functions fv belong uniformly to the prescribed α-Hölder and β-Hölder balls. Next, choose an infinitely differentiable function φ : R → R with compact support such that, for r ∈ [0, Rmax ],
12 r − Rmax 2 . φ(r) = 3 Rmax Direct integration gives Z Rmax Z Rmax Z Rmax 12 φ(r) dr = 0, rφ(r) dr = 1, φ(r)2 dr = 3 , Rmax 0 0 0 Define Qv (x, a) =

Rmax + fv (x, a) 2(1 − γ) 121

sup r∈[0,Rmax ]

|φ(r)| =

6 2 Rmax

.

and pR,v (r|x, a) =

1 Rmax

+ fv (x, a) − γbv φ(r), r ∈ [0, Rmax ].

Let Mv denote the resulting MDP, where R and X ′ are conditionally independent given (X, A). The identities for φ imply Z Rmax Z Rmax Rmax rpR,v (r|x, a) dr = pR,v (r|x, a) dr = 1, + fv (x, a) − γbv . 2 0 0 Moreover,
6(1 + γ)c0 hs ∥ψ∥∞ fv (x, a) − γbv φ(r) ≤ . 2 Rmax Hence, for sufficiently small fixed c0 , 3 1 ≤ pR,v (r|x, a) ≤ . 2Rmax 2Rmax Therefore, pR,v is a valid conditional density supported on [0, Rmax ] and satisfies fixed positive density bounds. The product rule and the preceding Hölder bounds show that pR,v belongs uniformly to a fixed α-Hölder ball. Reducing c0 if necessary also ensures that Qv belongs to the prescribed β-Hölder ball and satisfies 0 ≤ Qv (x, a) ≤

Rmax . 1−γ

Thus, Mv ∈ Eα,β . The conditional mean reward under Mv is E[R|X = x, A = a] =

Rmax + fv (x, a) − γbv . 2

Since the transition density is uniform,   ′ ′ E max Qv (X , a )|X = x, A = a = ′ a ∈A

Rmax + bv . 2(1 − γ)

Consequently, Rmax (TM∗ v Qv )(x, a) = + fv (x, a) − γbv + γ



Rmax + bv 2(1 − γ)



2 Rmax = + fv (x, a) = Qv (x, a). 2(1 − γ)

The optimal Bellman operator is a γ-contraction under the supremum norm. Its fixed point is unique, and hence Q∗Mv = Qv . 122

By Lemma E.7, there exists a set Vh ⊂ {0, 1}Jh containing the zero vector such that Jh Jh , dH (v, v ′ ) ≥ , 8 4

log |Vh | ≥

v ̸= v ′ .

Rmax Since Q∗Mv (z) = 2(1−γ) + fv (z), the common constant term cancels when Q∗Mv and Q∗Mv′ are

subtracted. Moreover, ρ = Unif(X × A) and X × A has unit volume. The disjointness of the bump regions eliminates all cross terms. Using the change of variables u = Jh P (vj − vj′ )2 = dH (v, v ′ ), we obtain, for distinct v, v ′ ∈ Vh ,

z−zj h

and

j=1

∥Q∗Mv − Q∗Mv′ ∥2L2 (ρ) =

Z

|fv (z) − fv′ (z)|2 dz

X ×A

= c20 h2s

Z Jh X ′ 2 (vj − vj )

ψ

2



X ×A

j=1

z − zj h

 dz

= c20 h2s+dx +da ∥ψ∥2L2 (Rdx +da ) dH (v, v ′ ) ≳ h2s+dx +da Jh ≍ h2s , where the last line uses dH (v, v ′ ) ≳ Jh and Jh ≍ h−(dx +da ) . Consequently, there exists a constant c1 > 0, independent of h and v, such that ∥Q∗Mv − Q∗Mv′ ∥L2 (ρ) ≥ 2c1 hs ,

v ̸= v ′ .

Let Pv denote the distribution of one observation (X, A, R, X ′ ) under Mv . Since f0 = b0 = 0, pR,0 (r|x, a) =

1 Rmax

.

By Lemma E.7, Z KL(Pv ∥P0 ) =

KL (pR,v (·|x, a)∥pR,0 (·|x, a)) µ(dx, da) Z Z Rmax ≤ Rmax |fv (x, a) − γbv |2 φ(r)2 dr dx da ZX ×A 0 12 = 2 |fv (x, a) − γbv |2 dx da. Rmax X ×A X ×A

Disjointness of the bump supports gives Z

fv (x, a)2 dx da = c20 h2s+dx +da ∥ψ∥22

X ×A

Jh X j=1

123

vj ≲ h2s ,

and b2v ≲ h2s . Hence KL(Pv ∥P0 ) ≲ h2s . Let Pv denote the joint distribution of Sr , Sx , D under Mv . Since the three batches are mutually independent and contain a total of n e independent observations, Lemma E.7 gives KL(Pv ∥P0 ) = nr KL(Pv ∥P0 ) + nx KL(Pv ∥P0 ) + nKL(Pv ∥P0 ) = n e KL(Pv ∥P0 ) ≲ n eh2s . 1

Choose h = ch n e− dx +da +2s , where ch > 0 is sufficiently small. Since Jh ≍ h−(dx +da ) , log |Vh | ≳ h−(dx +da ) . Therefore, n eh2s ≲n ehdx +da +2s = cdhx +da +2s . log |Vh | Choosing ch sufficiently small gives 1 X 1 1 X KL(Pv ∥P0 ) ≤ log |Vh | |Vh | v∈V |Vh | v∈V 16 h

h

1 1 = |Vh | log |Vh | |Vh | 16 1 = log |Vh |. 16 Let V be uniformly distributed on Vh . Conditionally on V = v, let Y = (Sr , Sx , D) have distribution Pv . The marginal distribution of Y is therefore P :=

1 X Pv . |Vh | v∈V h

The mutual information between V and Y is I(V ; Y ) :=

1 X KL(Pv ∥P). |Vh | v∈V h

Then, we obtain 1 X 1 X KL(Pv ∥P0 ) = KL(Pv ∥P) + KL(P∥P0 ) |Vh | v∈V |Vh | v∈V h

h

= I(V ; Y ) + KL(P∥P0 ) ≥ I(V ; Y ). Consequently, I(V ; Y ) ≤

1 log |Vh |. 16

124

Since Jh ≍ h−(dx +da ) and log |Vh | ≥ J8h , there exist constants cV > 0 and h0 > 0 such that log |Vh | ≥ cV h−(dx +da ) ,

0 < h ≤ h0 .

(59)

1 log |Vh |, gives Applying Lemma E.7 with θv = Q∗Mv and δ = c1 hs , together with I(V ; Y ) ≤ 16

  b − Q∗ ∥L2 (ρ) ≥ c1 hs ≥ 15 − log 2 . inf sup PMv ∥Q Mv b v∈Vh 16 log |Vh | Q
Fix c2 ∈ 0, 15 and define 16 & ( N0 := max

ch h0

dx +da +2s

, cdhx +da +2s



log 2 cV ( 15 − c2 ) 16

)'  dxd+d+d a +2s x

a

.

1

By (59) and h = ch n e− dx +da +2s , for every n e ≥ N0 ,   b − Q∗M ∥L2 (ρ) ≥ c1 hs ≥ c2 . inf sup PMv ∥Q v b v∈Vh Q

Finally, E[Z] ≥ tP(Z ≥ t) and {Mv : v ∈ Vh } ⊂ Eα,β yield b − Q∗M ∥L2 (ρ) ≥ c1 c2 hs inf sup EM ∥Q b M ∈Eα,β Q

max{α,β}

≍n e− dx +da +2 max{α,β} =n e− max{ dx +da +2α , dx +da +2β } . α

β

This completes the proof.

E

Auxiliary Results

Lemma E.1 (Theorem 2.5 of Boucheron et al. (2013)). Let Z1 , . . . , ZN be real-valued random variables such that for every λ ∈ (0, b) and i = 1, . . . , N , the logarithm of the moment generating function of Zi satisfies ψZi (λ) ≤ ψ(λ) where ψ is a convex and continuously differentiable function on [0, b) with 0 < b ≤ ∞ such that ψ(0) = ψ ′ (0) = 0. Then E max Zi ≤ ψ ∗−1 (log N ). i=1,...,N

In particular, if the Zi are sub-Gaussian with variance factor v, that is, ψ(λ) = λ2 v/2 for every λ ∈ (0, ∞), then E max Zi ≤

p

i=1,...,N

125

2v log N .

Lemma E.2 (Lemma 3 of Jiao et al. (2025b)). Let p(y) = N (µ1 , Σ1 ), q(y) = N (µ2 , Σ2 ), then the KL divergence between p and q satisfies KL(p∥q) =

 1 −1 −1 (µ1 − µ2 )⊤ Σ−1 (µ − µ ) − log Σ Σ + Tr Σ Σ − d . 1 1 y 1 2 2 2 2 2

In particular, when q(y) = N (0, I dy ), the result simplifies to KL(p∥q) =

 1 ∥µ1 ∥2 − log |Σ1 | + Tr(Σ1 ) − dy . 2

Lemma E.3 (Theorem 6 of Chen et al. (2022c)). Let P and P′ be two probability measures on C([0, T ]; Rd ). Suppose that under P and P′ , respectively, the canonical process z = (z t )t∈[0,T ] satisfies dz t = st dt + σ(t) dB t ,

dz t = s′t dt + σ(t) dB ′t ,

z 0 ∼ p0 ,

where st and s′t are progressively measurable processes and σ(t) > 0 is deterministic. Suppose that   Z T  1 ∥st − s′t ∥2 EP exp dt < ∞. 2 0 σ 2 (t) Then  Z T  Z dP′ (st − s′t )⊤ 1 T ∥st − s′t ∥2 = exp − dB t − dt . dP σ(t) 2 0 σ 2 (t) 0 Consequently, if pT and p′T are the time-T marginal distributions under P and P′ , respectively, then 1 KL(pT ∥p′T ) ≤ KL(P∥P′ ) = EP

Z T

2

0

∥st − s′t ∥2 dt. σ 2 (t)

Lemma E.4 (Lemma 6.1 of Jiao et al. (2025a)). Assume that f ∈ Hς with ς = s + r, s ∈ N0 and r ∈ (0, 1]. For any ε ∈ (0, 1), there exists a ReLU DNN function ψ with depth

D ≤ O (log(1/ε)), size S ≤ O ε−d/ς log(1/ε) , and weight bound B ≤ O ε−d/ς such that ∥f − ψ∥∞ ≤ ε. Lemma E.5 (Lemma 20 of Feng et al. (2024)). Let F be the ReLU DNNs with width W, depth D, and size S. Assume that the parameters of F are bounded by a constant B > 0, then for each δ > 0, log N (F, δ, ∥ · ∥∞ ) ≤ O(SD log(BWD/δ)). 126

Lemma E.6 (Theorem 2.8 and Equation 2.10 of Boucheron et al. (2013)). Let X1 , . . . , Xm be independent real-valued random variables, X :=

Pm

i=1 Xi

m

, and σ 2 :=

Xi ≤ bi almost surely, then, for every ϵ > 0,  
2m2 ϵ2 Pr X − E[X] ≥ ϵ ≤ 2 exp − Pm . 2 i=1 (bi − ai )

Pm

i=1 Var(Xi )

m

. If ai ≤

(Hoeffding’s inequality)

If |Xi − E[Xi ]| ≤ M almost surely, then, for every ϵ > 0,  
mϵ2 Pr X − E[X] ≥ ϵ ≤ 2 exp − 2 2 . 2σ + 3 M ϵ

(Bernstein’s inequality)

Lemma E.7 (Section 2 and Section 9 of Duchi (2026)). The following results hold. (i) For every sufficiently large integer J, there exists a set V ⊂ {0, 1}J containing the zero vector such that log |V| ≥

J , 8

dH (v, v ′ ) ≥

J , 4

v ̸= v ′ .

(ii) Let P and Q be probability measures satisfying P ≪ Q. Then KL(P ∥Q) ≤ Dχ2 (P ∥Q). If PX,Y and QX,Y have the same marginal distribution for X, then   KL(PX,Y ∥QX,Y ) = EX KL(PY |X ∥QY |X ) . Moreover, if Y1 , . . . , YN are independent under both P and Q, with respective marginal distributions Pi and Qi , then KL(P∥Q) =

N X

KL(Pi ∥Qi ).

i=1

(iii) Let {θv : v ∈ V} satisfy d(θv , θv′ ) ≥ 2δ for every v ̸= v ′ . Let V be uniformly distributed on V, and conditionally on V = v, let Y have distribution Pv . Then   b θv ) ≥ δ ≥ 1 − I(V ; Y ) + log 2 . inf sup Pv d(θ, log |V| θb v∈V The same conclusion holds for randomized estimators.

127

E.1

Construction of a Large ReLU Network

In this and the subsequent sections, we summarize existing results and fundamental tools for function approximation using neural networks. See Nakada and Imaizumi (2020); Oko et al. (2023); Fu et al. (2024) for more details. Lemma E.8 (Neural Network Concatenation). For a series of ReLU networks s1 : Rd1 → Rd2 , s2 : Rd2 → Rd3 , · · · , sk : Rdk → Rdk+1 with si ∈ S(Di , Wi , Si , Bi ) (i = 1, 2, · · · , k), there exists a neural network s ∈ G(D, W, S, B) satisfying s(x) = sk ◦ sk−1 ◦ · · · ◦ s1 (x) for all x ∈ Rd1 , with D=

k X

Di , W ≤ 2

k X

i=1

Wi , S ≤ 2

i=1

k X

Si , and B ≤ max Bi .

i=1

1≤i≤k

Lemma E.9 (Identity Function). Given d ∈ N+ and D ≥ 2, there exists a neural network sId,D ∈ G(D, W, S, B) that realizes d-dimensional identity function sId,D (x) = x, x ∈ Rd . Here W = 2d, S = 2dD, B = 1. Lemma E.10 (Neural Network Parallelization). For a series of ReLU networks si : Rdi → ′

Rdi with si ∈ G(Di , Wi , Si , Bi ) (i = 1, 2, · · · , k), there exists a neural network ϕ ∈ G(D, W, S, B) ′

′

′

⊤ d1 +d2 +···+dk ⊤ → Rd1 +d2 +···+dk for all satisfying s(x) = [s⊤ 1 (x1 ), s2 (x2 ), · · · , sk (xk )] :→ R ⊤ ⊤ ⊤ d1 +d2 +···+dk x = (x⊤ (xi can be shared), with 1 , x2 , · · · , xk ) ∈ R

D = D, W ≤ 2

k X

Wi , S ≤ 2

i=1

D = max Di , W ≤ 2 1≤i≤k

k X

k X

Si , and B ≤ max Bi (when D = Di holds for all i), 1≤i≤k

i=1

Wi , S ≤ 2

i=1

k X

(Si + Dd′i ), and B ≤ max { max Bi , 1} (otherwise). 1≤i≤k

i=1

Moreover, for x1 = x2 = · · · = xk = x ∈ Rd and d′1 = d′2 = · · · = d′k = d′ , there exists a P neural network ssum ∈ G(D, W, S, B) that realizes ssum (x) = ki=1 si (x) with D = max Di + 1, W ≤ 4 1≤i≤k

k X i=1

Wi , S ≤ 4

k X

(Si + Dd′i ) + 2W, and B ≤ max { max Bi , 1}.

i=1

128

1≤i≤k

E.2

Approximation of Basic Functions with ReLU Networks

Lemma E.11 (Approximating the Products). Let d ≥ 2, C ≥ 1. For any ϵ > 0, there exists a neural network sprod ∈ G(D, W, S, B) with D = O(log d(log d + log ϵ−1 + d log C)), W = 48d, S = O(d(log d + log ϵ−1 + d log C), B = C d such that sprod (x′1 , x′2 , · · · , x′d ) −

d Y

xi ≤ ϵ + dC d−1 ϵ0 ,

i=1

for all x ∈ [−C, C]d and x′ ∈ Rd with ∥x − x′ ∥∞ ≤ ϵ0 . Moreover, |sprod (x′ )| ≤ C d for all x′ ∈ Rd , and sprod (x′1 , x′2 , · · · , x′d ) = 0 if at least one of x′i is 0. Q Remark E.1. We note that some of xi , xj (i ̸= j) can be shared. For Ii=1 xiui with P ui ∈ N+ (i = 1, 2, · · · , I) and Ii=1 ui = d, there exists a neural network satisfying the same bounds as above. Lemma E.12 (Approximating the Reciprocal Function). For any 0 < ϵ < 1, there exists a neural network srec ∈ G(D, W, S, B) with D = O(log2 ϵ−1 ), W = O(log3 ϵ−1 ), S = O(log4 ϵ−1 ) and B = O(ϵ−2 ) such that srec (x′ ) −

1 |x′ − x| ≤ϵ+ , x ϵ2

for all x ∈ [ϵ, ϵ−1 ] and x′ ∈ R. Lemma E.13 (Approximating the Square Root Function). For any 0 < ϵ < 1, there exists a neural network sroot ∈ G(D, W, S, B) with D = O(log2 ϵ−1 ), W = O(log3 ϵ−1 ), S = O(log4 ϵ−1 ) and B = O(ϵ−1 ) such that |sroot (x′ ) −

√ |x′ − x| x| ≤ ϵ + √ , ϵ

for all x ∈ [ϵ, ϵ−1 ] and x′ ∈ R.

E.3

Clipping and Switching Functions

Lemma E.14 (Clipping Function). For any a, b ∈ Rd with ai ≤ bi (i = 1, 2, · · · , d), there exists a clipping function sclip (x, a, b) ∈ G(D, W, S, B) with   D = 2, W = 2d, S = 7d, B = max max{|ai |, |bi |} ∨ 1, 1≤i≤d

129

such that sclip (x, a, b)i = min{bi , max{xi , ai }} (i = 1, 2, · · · , d). When ai = cmin and bi = cmax for all i, we sometimes denote sclip (x, a, b) as sclip (x, cmin , cmax ) using scalar values cmin and cmax . Lemma E.15 (Switching Function). Let t1 < t2 < s1 < s2 , and f (t, x) be a scalar-valued function (for a vector-valued function, we just apply this coordinate-wise). Assume that |ϕ1 (t, x) − f (t, x)| ≤ ϵ on [t1 , s1 ] and |ϕ2 (t, x) − f (t, x)| ≤ ϵ on [t2 , s2 ]. Then, there exist two neural networks sswitch,1 (t, t2 , s1 ) and sswitch,2 (t, t2 , s1 ) ∈ G(D, W, S, B) with D = 3, W = 2, S = 8, and B = max{s1 , (s1 − t2 )−1 } such that |sswitch,1 (t, t2 , s1 )ϕ1 (t, x) + sswitch,2 (t, t2 , s1 )ϕ2 (t, x) − f (t, x)| ≤ ϵ holds for any t ∈ [t1 , s2 ], where sswitch,1 (t, t2 , s1 ) =

1 ReLU (s1 − sclip (t, t2 , s1 )) , s1 − t 2

sswitch,2 (t, t2 , s1 ) =

1 ReLU (sclip (t, t2 , s1 ) − t2 ) . s1 − t 2

References Agarwal A, Jiang N, Kakade SM, Sun W (2019) Reinforcement learning: Theory and algorithms. CS Dept, UW Seattle, Seattle, WA, USA, Tech Rep 32:96 Agarwal A, Alomar A, Alumootil V, Shah D, Shen D, Xu Z, Yang C (2021) Persim: Dataefficient offline reinforcement learning with heterogeneous agents via personalized simulators. Advances in Neural Information Processing Systems 34:18564–18576 Ajay A, Du Y, Gupta A, Tenenbaum JB, Jaakkola TS, Agrawal P (2023) Is conditional generative modeling all you need for decision making? In: The Eleventh International Conference on Learning Representations

130

Alonso E, Jelley A, Micheli V, Kanervisto A, Storkey A, Pearce T, Fleuret F (2024) Diffusion for world modeling: Visual details matter in Atari. In: Advances in Neural Information Processing Systems, vol 37, pp 58757–58791 Anderson BD (1982) Reverse-time diffusion equation models. Stochastic Processes and their Applications 12(3):313–326 Arora S, Doshi P (2021) A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence 297:103500 Bellemare MG, Naddaf Y, Veness J, Bowling M (2013) The arcade learning environment: An evaluation platform for general agents. Journal of artificial intelligence research 47:253–279 Benton J, De Bortoli V, Doucet A, Deligiannidis G (2024) Nearly d-linear convergence bounds for diffusion models via stochastic localization. In: International Conference on Learning Representations, vol 2024, pp 36916–36936 Boucheron S, Lugosi G, Massart P (2013) Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, DOI 10.1093/acprof:oso/9780199535255. 001.0001 Chang J, Jiao Y, Kang L, Shi J (2026) Deep bootstrap. arXiv preprint arXiv:260210587 Chen H, Lu C, Ying C, Su H, Zhu J (2022a) Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:220914548 Chen H, Lee H, Lu J (2023a) Improved analysis of score-based generative modeling: Userfriendly bounds under minimal smoothness assumptions. In: International Conference on Machine Learning, PMLR, pp 4735–4763 Chen H, Lu C, Wang Z, Su H, Zhu J (2024a) Score regularized policy optimization through diffusion behavior. In: The Twelfth International Conference on Learning Representations Chen J, Jiang N (2019) Information-theoretic considerations in batch reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 1042–1051

131

Chen M, Jiang H, Liao W, Zhao T (2022b) Nonparametric regression on low-dimensional manifolds using deep relu networks: function approximation and statistical recovery. Information and Inference: A Journal of the IMA 11(4):1203–1253 Chen M, Huang K, Zhao T, Wang M (2023b) Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. International Conference on Machine Learning pp 4672–4712 Chen M, Mei S, Fan J, Wang M (2024b) An overview of diffusion models: Applications, guided generation, statistical rates and optimization. arXiv preprint arXiv:240407771 Chen R, Jia C, Huang Z, Liu TS, Liu XH, Yu Y (2024c) Offline transition modeling via contrastive energy learning. In: Proceedings of the 41st International Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, vol 235, pp 6992–7014 Chen S, Chewi S, Li J, Li Y, Salim A, Zhang A (2022c) Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In: The Eleventh International Conference on Learning Representations Conforti G, Durmus A, Silveri MG (2023) Score diffusion models without early stopping: finite fisher information is all you need. arXiv preprint arXiv:230812240 Dikkala N, Lewis G, Mackey L, Syrgkanis V (2020) Minimax estimation of conditional moment models. In: Advances in Neural Information Processing Systems, vol 33, pp 12248– 12262 Ding Z, Zhang A, Tian Y, Zheng Q (2024) Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:240203570 Duchi JC (2026) Statistics and information theory, lecture Notes for Statistics 311/Electrical Engineering 377, Stanford University, version July 23, 2026 Ernst D, Geurts P, Wehenkel L (2005) Tree-based batch mode reinforcement learning. Journal of Machine Learning Research 6(18):503–556

132

Fan J, Wang Z, Xie Y, Yang Z (2020) A theoretical analysis of deep Q-learning. In: Proceedings of the 2nd Conference on Learning for Dynamics and Control, Proceedings of Machine Learning Research, vol 120, pp 486–489 Fang L, Liu R, Zhang J, Wang W, Jing BY (2024) Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint arXiv:240520555 Fang Z, Lan T (2024) Learning from random demonstrations: Offline reinforcement learning with importance-sampled diffusion models. arXiv preprint arXiv:240519878 Feng X, Jiao Y, Kang L, Zhang B, Zhou F (2023) Over-parameterized deep nonparametric regression for dependent data with its applications to reinforcement learning. Journal of Machine Learning Research 24(383):1–40 Feng X, He X, Jiao Y, Kang L, Wang C (2024) Deep nonparametric quantile regression under covariate shift. Journal of Machine Learning Research 25(385):1–50 Frans K, Park S, Abbeel P, Levine S (2025) Diffusion guidance is a controllable policy improvement operator. arXiv preprint arXiv:250523458 Fu H, Yang Z, Wang M, Chen M (2024) Unveil conditional diffusion models with classifierfree guidance: A sharp statistical theory. arXiv preprint arXiv:240311968 Fujimoto S, Meger D, Precup D (2019) Off-policy deep reinforcement learning without exploration. In: Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 97, pp 2052–2062 Gao X, Nguyen HM, Zhu L (2023) Wasserstein convergence guarantees for a general class of score-based generative models. arXiv preprint arXiv:231111003 Grondman I, Busoniu L, Lopes GA, Babuska R (2012) A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, part C (applications and reviews) 42(6):1291–1307

133

Haarnoja T, Zhou A, Hartikainen K, Tucker G, Ha S, Tan J, Kumar V, Zhu H, Gupta A, Abbeel P, et al. (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:181205905 Haussmann UG, Pardoux E (1986) Time reversal of diffusions. The Annals of Probability pp 1188–1205 Hester T, Vecerik M, Pietquin O, Lanctot M, Schaul T, Piot B, Horgan D, Quan J, Sendonaris A, Osband I, et al. (2018) Deep q-learning from demonstrations. In: Proceedings of the AAAI conference on artificial intelligence, vol 32 Ho J, Jain A, Abbeel P (2020) Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems, vol 33, pp 6840–6851 Hyvärinen A, Dayan P (2005) Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research 6(4) Janner M, Fu J, Zhang M, Levine S (2019) When to trust your model: Model-based policy optimization. In: Advances in Neural Information Processing Systems, vol 32, pp 12519– 12530 Janner M, Du Y, Tenenbaum JB, Levine S (2022) Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:220509991 Jiang N, Li L (2016) Doubly robust off-policy value evaluation for reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 48, pp 652–661 Jiao Y, Kang L, Lin H, Liu J, Zuo H (2024) Latent Schrödinger bridge diffusion model for generative learning. arXiv preprint arXiv:240413309 Jiao Y, Kang L, Liu J, Lu X, Yang JZ (2025a) Deep approximate policy iteration. The Annals of Statistics 53(2):802–821 Jiao Y, Kang L, Liu J, Peng H, Zuo H (2025b) Model free prediction with uncertainty assessment. IEEE Transactions on Information Theory 71(9):7229–7253 134

Johnson AE, Pollard TJ, Shen L, Lehman LwH, Feng M, Ghassemi M, Moody B, Szolovits P, Anthony Celi L, Mark RG (2016) Mimic-iii, a freely accessible critical care database. Scientific data 3(1):1–9 Kang L, Luo Y, Yuan H, Zhu C (2023) Error analysis of fitted $q$-iteration with reLUactivated deep neural networks Kidambi R, Rajeswaran A, Netrapalli P, Joachims T (2020) MOReL: Model-based offline reinforcement learning. In: Advances in Neural Information Processing Systems, vol 33, pp 21810–21823 Kim J, Meunier D, Gretton A, Suzuki T, Li Z (2025) Optimality and adaptivity of deep neural features for instrumental variable regression. arXiv preprint arXiv:250104898 Klusowski JM, Barron AR (2018) Approximation by combinations of ReLU and squared ReLU ridge functions with ℓ1 and ℓ0 controls. IEEE Transactions on Information Theory 64(12):7649–7656 Konda V, Tsitsiklis J (1999) Actor-critic algorithms. In: Solla S, Leen T, Müller K (eds) Advances in Neural Information Processing Systems, MIT Press, vol 12 Lee H, Lu J, Tan Y (2022) Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems 35:22870–22882 Lee H, Lu J, Tan Y (2023) Convergence of score-based generative modeling for general data distributions. In: International Conference on Algorithmic Learning Theory, PMLR, pp 946–985 Levine S, Kumar A, Tucker G, Fu J (2020) Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:200501643 Li G, Wei Y, Chen Y, Chi Y (2023) Towards faster non-asymptotic convergence for diffusionbased generative models. arXiv preprint arXiv:230609251

135

Li G, Shi L, Chen Y, Chi Y, Wei Y (2024) Settling the sample complexity of modelbased offline reinforcement learning. The Annals of Statistics 52(1):233–260, DOI 10.1214/23-AOS2342 Li SE (2023) Deep reinforcement learning. In: Reinforcement learning for sequential decision and optimal control, Springer, pp 365–402 Liang T, Rakhlin A, Sridharan K (2015) Learning with square loss: Localization through offset rademacher complexity. In: Conference on Learning Theory, PMLR, pp 1260–1285 Lu C, Chen H, Chen J, Su H, Li C, Zhu J (2023) Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 22825–22855 Ma H, Chen T, Wang K, Li N, Dai B (2025) Efficient online reinforcement learning for diffusion policy. arXiv preprint arXiv:250200361 Ma L, Siegel JW, Xu J (2022) Uniform approximation rates and metric entropy of shallow neural networks. Research in the Mathematical Sciences 9(3):46 Mao L, Xu H, Zhan X, Zhang W, Zhang A (2024) Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. Advances in Neural Information Processing Systems 37:98806–98834 Mazoure B, Talbott W, Bautista MA, Hjelm D, Toshev A, Susskind J (2023) Value function estimation using conditional diffusion models for control. arXiv preprint arXiv:230607290 Miao R, Shahbaba B, Qu A (2025) Reinforcement learning for individual optimal policy from heterogeneous data. Annals of statistics 53(4):1513 Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

136

Montanelli H, Du Q (2019) New error bounds for deep relu networks using sparse grids. SIAM Journal on Mathematics of Data Science 1(1):78–92 Mousavi SS, Schukat M, Howley E (2016) Deep reinforcement learning: an overview. In: Proceedings of SAI intelligent systems conference, Springer, pp 426–440 Munos R, Szepesvári C (2008) Finite-time bounds for fitted value iteration. Journal of Machine Learning Research 9:815–857 Nakada R, Imaizumi M (2020) Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. The Journal of Machine Learning Research 21(1):7018– 7055 Nanayakkara T, Clermont G, Langmead CJ, Swigon D (2022) Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment. PLOS Digital Health 1(2):e0000012 Oko K, Akiyama S, Suzuki T (2023) Diffusion models are minimax optimal distribution estimators. In: Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 202, pp 26517–26582 Prudencio RF, Maximo MR, Colombini EL (2023) A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE transactions on neural networks and learning systems 35(8):10237–10257 Raghu A, Komorowski M, Ahmed I, Celi L, Szolovits P, Ghassemi M (2017) Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:171109602 Ren AZ, Lidard J, Ankile LL, Simeonov A, Agrawal P, Majumdar A, Burchfiel B, Dai H, Simchowitz M (2024) Diffusion policy policy optimization. arXiv preprint arXiv:240900588 Schramm L, Boularias A (2024) Bellman diffusion models. arXiv preprint arXiv:240712163 Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:170706347

137

Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, et al. (2016) Mastering the game of go with deep neural networks and tree search. nature 529(7587):484–489 Song Y, Sohl-Dickstein J, Kingma DP, Kumar A, Ermon S, Poole B (2020) Score-based generative modeling through stochastic differential equations. In: International Conference on Learning Representations Stone CJ (1980) Optimal rates of convergence for nonparametric estimators. The annals of Statistics pp 1348–1360 Sun J, Jiang Y, Qiu J, Nobel P, Kochenderfer MJ, Schwager M (2023) Conformal prediction for uncertainty-aware planning with diffusion dynamics model. Advances in Neural Information Processing Systems 36:80324–80337 Sutton RS, Barto AG, et al. (1998) Reinforcement learning: An introduction, vol 1. MIT press Cambridge Suzuki T (2018) Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint arXiv:181008033 Tang R, Lin L, Yang Y (2025) Conditional diffusion models are minimax-optimal and manifold-adaptive for conditional distribution estimation. In: International Conference on Learning Representations Tsybakov AB (2009) Introduction to Nonparametric Estimation. Springer series in statistics, Springer Uehara M, Sun W (2022) Pessimistic model-based offline reinforcement learning under partial coverage. In: International Conference on Learning Representations Uehara M, Huang J, Jiang N (2020) Minimax weight and Q-function learning for off-policy evaluation. In: Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 119, pp 9659–9668

138

Uehara M, Shi C, Kallus N (2022) A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:221206355 Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, vol 30 Vincent P (2011) A connection between score matching and denoising autoencoders. Neural computation 23(7):1661–1674 Wang G, Jiao Y, Xu Q, Wang Y, Yang C (2021) Deep generative learning via schrödinger bridge. In: International conference on machine learning, PMLR, pp 10794–10804 Wang J, Qi Z, Shi C (2026) Blessing from human-ai interaction: Super reinforcement learning in confounded environments. arXiv preprint arXiv:220915448 Wang Y, Wang L, Jiang Y, Zou W, Liu T, Song X, Wang W, Xiao L, Wu J, Duan J, et al. (2024) Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems 37:54183–54204 Wang Z, Hunt JJ, Zhou M (2022) Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:220806193 Wulfmeier M, Ondruska P, Posner I (2015) Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:150704888 Xie T, Jiang N (2020) Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In: Conference on Uncertainty in Artificial Intelligence, PMLR, pp 550–559 Xu L, Chen Y, Srinivasan S, de Freitas N, Doucet A, Gretton A (2020) Learning deep features in instrumental variable regression. arXiv preprint arXiv:201007154 Yang L, Zhang Z, Song Y, Hong S, Xu R, Zhao Y, Zhang W, Cui B, Yang MH (2023) Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys 56(4):1–39

139

Yu T, Thomas G, Yu L, Ermon S, Zou J, Levine S, Finn C, Ma T (2020) MOPO: Modelbased offline policy optimization. In: Advances in Neural Information Processing Systems, vol 33, pp 14129–14142 Yu T, Kumar A, Rafailov R, Rajeswaran A, Levine S, Finn C (2021) COMBO: Conservative offline model-based policy optimization. In: Advances in Neural Information Processing Systems, vol 34, pp 28954–28967 Zhu Z, Zhao H, He H, Zhong Y, Zhang S, Guo H, Chen T, Zhang W (2023) Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:231101223 Ziebart BD, Maas AL, Bagnell JA, Dey AK, et al. (2008) Maximum entropy inverse reinforcement learning. In: Aaai, Chicago, IL, USA, vol 8, pp 1433–1438

140

Related documents

• Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design#1014606
• ChronoSSM: Training for Temporally Aware Representations in Autoregressive State Space Models#444599
• Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference#1014915
• dQwen3.5: Hybrid-Attention Diffusion Language Models#1034547
• dQwen3.5: Hybrid-Attention Diffusion Language Models#978416