Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control

Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control Zhaoyu Zhu1 1

Shuang Li3

University of Illinois Urbana-Champaign 2 The University of Texas at Austin 3 The Chinese University of Hong Kong, Shenzhen

zhaoyuz4@illinois.edu

arXiv:2608.07433v1 [math.OC] 7 Aug 2026

Rui Gao2

rui.gao@mccombs.utexas.edu

lishuang@cuhk.edu.cn

Abstract Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space. We study entropy-regularized discounted linear-quadratic (LQ) control. A Bellman verification argument shows that the unrestricted problem has a linear-Gaussian optimal policy, and the discounted-occupancy-weighted statewise Wasserstein gradient is tangent to this policy class. WPG therefore reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance. We prove that this ODE is globally well posed and converges exponentially from every admissible initialization. For each fixed LQ problem, the exponent has a positive limit as the entropy temperature tends to zero and contains no perturbative factor of the form exp(−c/τ ), while retaining the usual dependence on the conditioning of the control problem.

1

Introduction

The geometry used to update a policy is central to continuous-control reinforcement learning. Classical policy gradient optimizes a finite-dimensional actor in Euclidean parameter space. Natural policy gradient and mirror descent instead use information geometry, usually through the KL divergence. Wasserstein policy gradient (WPG) takes a different approach: for each state, it transports the conditional action distribution in the action space. With entropy regularization, the resulting Wasserstein policy gradient flow is a Fokker-Planck equation whose drift is generated by the action gradient of the current soft action-value function and whose diffusion is generated by entropy. This formulation connects policy optimization with Wasserstein gradient flows, particle methods, and optimal-transport trust-region and proximal methods (Zhang et al., 2018; Arbel et al., 2020; Moskovitz et al., 2021; Terpin et al., 2022; Song et al., 2023; Pfau et al., 2025; Zhu et al., 2026b). General WPG convergence theory is largely based on Langevin-dynamics analysis of the policy distribution and its moving Gibbs target (Zhu et al., 2026a; Šiška and Zhang, 2026). These results rely on boundedness assumptions on rewards that exclude unbounded quadratic costs and, importantly, do not establish well-posedness of the underlying continuous-time flow. The convergence rate is expressed in terms of a uniform log-Sobolev coefficient for the moving Gibbs law and deteriorates exponentially in the rate of O( τ1 ) exp(O(− τ1 )) as the entropy temperature τ tends to zero. In this paper, we show that these general results can be sharpened in the linear-quadratic (LQ) setting. We proceed in two steps. First, we solve the unrestricted entropy-regularized control problem over admissible stationary randomized Markov policies. A Bellman verification argument, 1

based on the soft Bellman equation and the discounted Riccati equation, identifies a linear-Gaussian policy π⋆ (· | x) = N (−K⋆ x, Σ⋆ ) is optimal over the full admissible policy class. Therefore, without loss of optimality, we formulate the learning problem over πK,Σ (· | x) = N (−Kx, Σ). Note that this is an optimality statement rather than a distributional assumption on the state process: the initial state and disturbances need not be Gaussian. Second, we make the policy-space geometry explicit. Tangent perturbations are represented by action-space continuity equations, and their squared norm is defined as the corresponding statewise 2-Wasserstein kinetic energy averaged with respect to the normalized discounted stateoccupancy measure. Combining this metric with the policy-gradient theorem yields the Wasserstein policy-gradient flow (Pfau et al., 2025; Zhu et al., 2026a,b; Šiška and Zhang, 2026) ∂t πt (u | x) = ∇u · πt (u | x)∇u Qπt (x, u) + τ log πt (u | x)

.

For every admissible linear-Gaussian policy, the soft action-value function is quadratic in the action. Its action gradient is therefore affine, and the WPG vector field is tangent to the linear-Gaussian policy class. Under the parameterization (K, Σ), the flow closes exactly as K̇ = −2EK ,

Σ̇ = −2MK Σ − 2ΣMK + 2τ I.

(1)

Thus, the gain-covariance ODE is an exact representation of the policy-space gradient flow on the linear-Gaussian class, rather than a projection or approximation. Every solution of this closed parameter system generates a linear-Gaussian solution of WPGF. This exact closure allows the convergence analysis to proceed through the global well-posedness of the parameter ODE, without requiring a separate uniqueness theory for all solutions of the unrestricted nonlinear Fokker-Planck equation. The closed parameter equations also make the contrast with Fisher-Rao flow transparent. Writing EK for the feedback Bellman residual, WPG satisfies K̇ = −2EK , while Fisher-Rao flow satisfies K̇ = −2ΣEK . Fisher-Rao therefore scales learning in each action direction by the current exploration variance in that direction. If a fixed total variance is spread over many redundant action coordinates, the variance in task-relevant directions decreases and Fisher-Rao slows down, whereas the WPG gain equation is unchanged. Figure 1 illustrates this difference. Contributions.

Our contributions are threefold.

• Linear-Gaussian optimality and exact policy-space reduction. We first consider the unrestricted class of admissible stationary randomized Markov policies and prove, through an exact relativeentropy identity, that an optimal policy is linear-Gaussian. Restricting the optimization problem to admissible linear-Gaussian policies therefore entails no loss of optimality. We then define the discounted-occupancy-weighted statewise 2-Wasserstein metric on conditional action distributions and derive WPGF as the negative gradient flow of the objective under this metric. At every admissible linear-Gaussian policy, the soft action-value function is quadratic in the action, so the WPG vector field is tangent to the linear-Gaussian class and admits the exact parameter representation (1). Using the Lyapunov representation of policy evaluation and compact objective sublevel sets, we further prove that this parameter system has a unique global admissible solution from every finite-cost linear-Gaussian initialization.

2

Figure 1: Robustness to redundant action dimensions. The plot gives a closed-form local comparison near the optimal policy. An eight-dimensional task-relevant action subspace is embedded in ambient dimensions m ∈ {8, 32, 128, 512, 2048, 8192}. The optimal covariance eigenvalues in this subspace are 10−(i−1)/7 , i = 1, . . . , 8, and each redundant coordinate has optimal variance 10−6 . Both flows start from the same isotropic covariance with total variance equal to ten times the optimal total variance, and the horizontal axis uses the rescaled time s = τ t. As m grows, the fixed variance budget is spread over more redundant coordinates. The WPG gain equation is independent of the current covariance, so all WPG curves coincide. The Fisher-Rao gain equation is scaled by the covariance, so learning in the relevant directions becomes slower.

• Global exponential convergence with explicit temperature dependence. We prove that the objective gap satisfies 

C(Kt , Σt ) − C(K⋆ , Σ⋆ ) ≤ exp −

4λmin (Γ0 )λmin (R) t ∥S⋆ (τ )∥



C(K0 , Σ0 ) − C(K⋆ , Σ⋆ ) .

Here Γ0 is the initial-state second-moment matrix, R is the action-cost matrix, and S⋆ (τ ) is the discounted state-correlation matrix under the optimal policy. For each fixed LQ problem, (0)

(1)

S⋆ (τ ) = S⋆ + τ S⋆ , and hence the convergence exponent approaches a positive limit as τ ↓ 0. The LQ-specific analysis therefore avoids the additional perturbative factor exp(−c/τ ) that appears in general WPG bounds. The exponent retains the standard dependence on the discounted state correlation, initial-state excitation, action-cost curvature, and problem scaling. The same objective estimate also controls the squared feedback-gain error and the entropy-weighted relative covariance error. • An exact LQ analysis without perturbative or density-ratio losses. The proof exploits three identities specific to the LQ setting. The Gibbs law induced by the current soft action-value function is exactly Gaussian. Its sharp log-Sobolev coefficient is αK =

2λmin (MK ) . τ

Combining this bound with the τ 2 -weighted Fisher-information dissipation and the identity between the Bellman residual and τ KL cancels the explicit temperature factor without invoking 3

a Holley-Stroock perturbation argument. Consequently, the rate does not require a bounded value-function estimate, a uniform oscillation bound for the soft action-value function, or a ratio bound between policy densities. Organization. Section 2 reviews related work. Section 3 formulates the unrestricted entropyregularized LQ problem and proves optimality of a linear-Gaussian policy. Section 4 defines the policy-space metric, derives WPGF, and proves exact closure of its direction on the linear-Gaussian policy class. Section 5 establishes global well-posedness and global exponential convergence of the parameter flow, with explicit discussion of temperature and problem conditioning. The appendices compare policy geometries and give the complete proofs.

2

Related Work

Policy optimization for LQ control. The modern convergence theory of policy optimization for infinite-horizon LQ control begins with Fazel et al. (2018), who show that the deterministic feedback-gain objective is nonconvex but satisfies a gradient-dominance property. This structure yields global guarantees for policy gradient, natural policy gradient, and Gauss-Newton methods. Subsequent work develops derivative-free guarantees (Malik et al., 2019), finite-horizon theory for noisy LQ control (Hambly et al., 2021), and model-free convergence and sample-complexity bounds (Mohammadi et al., 2022). These results primarily optimize a deterministic feedback gain and do not treat the full conditional action distribution as the policy variable. For entropy-regularized LQ control, Guo et al. (2026) study the same discounted infinite-horizon model as our considered setting, but develop different algorithms and a different convergence analysis. Their regularized policy-gradient (RPG) and iterative policy-optimization (IPO) methods are formulated directly in the linear-Gaussian policy parameters. After aligning sign and step-size conventions, the RPG gain direction coincides with the WPG gain direction, but the covariance direction is different; the IPO covariance update is an exact one-step policy improvement, while the WPG covariance update is a Wasserstein gradient step. Giegrich et al. (2024) consider a further distinct setting: finite-horizon continuous-time exploratory LQ control with time-dependent linear-Gaussian policies in a noncoervive setting. Their method uses Fisher geometry for the policy mean and Bures-Wasserstein geometry for the covariance. The use of Bures-Wasserstein geometry is related to the covariance component of the Gaussian 2-Wasserstein metric, but their algorithm combines two separate geometries on the Gaussian parameters; it is not derived from a single statewise Wasserstein metric on the full conditional action law. Wasserstein policy optimization. Wasserstein policy methods update policies by transport in action space rather than by a Euclidean parameter change or a KL-based mirror step. Early work formulates policy optimization as a Wasserstein gradient flow (Zhang et al., 2018). Related developments include kernelized and computationally efficient Wasserstein natural gradients (Arbel et al., 2020; Moskovitz et al., 2021), optimal-transport trust-region methods (Terpin et al., 2022), metric-aware trust-region analysis (Song et al., 2023), Wasserstein proximal updates (Zhu et al., 2026b), and Wasserstein Policy Optimization (Pfau et al., 2025). These methods differ in how the policy-space direction is represented or approximated. Wasserstein Policy Optimization, for example, approximates a distributional policy flow in an actor-critic method and projects the direction onto a finite-dimensional actor. In the LQ model studied here, the linear-Gaussian class contains 4

an unrestricted optimum and the WPG direction is tangent to this class, so the gain-covariance representation is exact and requires no projection. General WPG convergence. The closest convergence result is Zhu et al. (2026a). They develop a general Bellman-based theory for WPG over continuous action distributions under boundedness and regularity assumptions and prove a discrete-time contraction up to a discretization error. Their continuous-time analysis is formulated for a sufficiently regular density flow, and the moving Gibbs family is controlled by the Holley-Stroock perturbation principle. The present analysis is complementary and specific to LQ control. Quadratic value functions yield an exact finite-dimensional parameter flow and permit a direct global well-posedness proof. The Gibbs law is Gaussian with sharp coefficient without a perturbation factor. The LQ identities also allow unbounded quadratic costs and give convergence statements directly for the feedback gain and the entropy-weighted covariance error. Thus the broader general theory and the sharper LQ analysis address different levels of generality.

3

Entropy-Regularized Linear-Quadratic Control

3.1

Problem setup

We consider the discounted stochastic linear system xt+1 = Axt + But + wt ,

t ≥ 0,

where xt ∈ Rn , ut ∈ Rm , A ∈ Rn×n , and B ∈ Rn×m . The noise sequence (wt )t≥0 is i.i.d., independent of the initial state and of the past, has mean zero, and has covariance W ⪰ 0. The initial state x0 ∼ D has full-rank second moment Γ0 := E[x0 x⊤ 0 ] ≻ 0. The stage cost is

c(x, u) = x⊤ Qx + u⊤ Ru,

Q ⪰ 0,

R ≻ 0.

We use the cost-minimization convention. For a discount factor γ ∈ (0, 1), entropy temperature τ > 0, and Markov policy π(· | x) with density, define V (x) := E π

π

"∞ X

#

γ c(xt , ut ) + τ log π(ut | xt ) t

x0 = x ,

C(π) := Ex0 ∼D [V π (x0 )].

t=0

The +τ log π term is a negative-entropy cost and therefore favors randomized action laws under minimization. For comparisons beyond the Gaussian class, let Πadm denote stationary randomized Markov policies with densities such that V π (x) is finite and the following discounted transversality condition holds for every initial state: γ T Eπx [1 + ∥xT ∥2 ] −→ 0,

x ∈ Rn .

This condition is automatic for the admissible linear-Gaussian policies considered below. We write µ := λmin (Γ0 ) > 0,

λR := λmin (R) > 0,

for the two scalar problem constants that enter the convergence rate. Throughout, ∥·∥ denotes the Euclidean norm for vectors and the spectral norm for matrices. We first introduce the linear-Gaussian family and its stability condition. We then prove, by a Bellman verification argument, that this family contains an optimal policy for the unrestricted control problem. 5

3.2

Linear-Gaussian policies and admissibility

We parameterize stochastic feedback policies by the linear-Gaussian family πK,Σ (· | x) = N (−Kx, Σ),

K ∈ Rm×n ,

Σ ∈ Sm ++ ,

(2)

where K parameterizes the state-dependent mean, Σ is the state-independent action covariance, πK,Σ and Sm ++ denotes the cone of symmetric positive definite m × m matrices. We write VK,Σ := V and C(K, Σ) := C(πK,Σ ). Equivalently, under πK,Σ , ut = −Kxt + ξt ,

ξt ∼ N (0, Σ),

with (ξt )t≥0 independent across time and independent of the system noise. The term linear-Gaussian policy refers only to the conditional action law in (2): the conditional law is Gaussian, its mean is linear in the state, and its covariance is state independent. Neither the initial state nor the disturbance is assumed Gaussian. Consequently, the state process need not be Gaussian. For a gain K, define FK := A − BK, LK := Q + K ⊤ RK. The closed loop becomes xt+1 = FK xt + ηt ,

ηt := Bξt + wt ,

Cov(ηt ) = ΩΣ := BΣB ⊤ + W.

E[ηt ] = 0,

Definition 3.1 (Admissibility). A pair (K, Σ) is called admissible if √ ρ( γFK ) < 1.

Σ ≻ 0,

We denote the admissible set by Aadm , and we write Gadm := {πK,Σ : (K, Σ) ∈ Aadm } for the corresponding class of admissible linear-Gaussian policies. For an admissible pair, define the discounted state-correlation matrix SK,Σ := E

K,Σ

"∞ X

#

γ xt x⊤ t t

.

(3)

t=0

Then SK,Σ is finite and SK,Σ ⪰ Γ0 ≻ 0, because the t = 0 term equals Γ0 . Throughout the paper, S denotes discounted state-correlation matrices. Assumption 3.1 (Standing assumptions). Throughout the paper, we assume: √ √ √ (i) ( γA, γB) is stabilizable: there exists K such that ρ( γ(A − BK)) < 1. √ √ (ii) (Q1/2 , γA) is detectable: there exists L such that ρ( γA − LQ1/2 ) < 1. (iii) The initial linear-Gaussian policy (K0 , Σ0 ) has finite cost: C(K0 , Σ0 ) < ∞. These are the discounted analogues of the standard stabilizability and detectability assumptions for LQ control (Anderson and Moore, 2007). Detectability rules out unstable closed loops with finite cost caused by modes unobserved by Q; see Lemma D.2 in Appendix D.

6

3.3

Value representation and Bellman notation

The next lemma gives the value and soft action-value functions in closed form. Its proof is a standard Lyapunov calculation and appears in Appendix B. Lemma 3.1 (Quadratic value representation). For every admissible pair (K, Σ), define PK :=

∞ X

t ⊤ t . γ t (FK ) LK FK

t=0

Then PK is the unique positive semidefinite solution of the discounted Lyapunov equation ⊤ PK = LK + γFK PK FK .

(4)

Moreover, with MK := R + γB ⊤ PK B,

NK := γB ⊤ PK A,

EK := MK K − NK ,

we have MK ⪰ R ≻ 0, and the entropy-regularized value function is V πK,Σ (x) = x⊤ PK x + qK,Σ , where

1 qK,Σ = 1−γ



τ Tr(ΣMK ) − m + log((2π)m det Σ) + γ Tr(W PK ) . 2 

The corresponding soft state-action value is QπK,Σ (x, u) = u⊤ MK u + 2u⊤ NK x + ψK,Σ (x), where 



ψK,Σ (x) = x⊤ Q + γA⊤ PK A x + γ Tr(W PK ) + γqK,Σ . Consequently, C(K, Σ) = Tr(Γ0 PK ) + qK,Σ .

(5)

The matrix EK is the feedback component of the Bellman residual: EK = 0 precisely when the policy mean equals the current quadratic Bellman minimizer.

3.4

Optimality of linear-Gaussian policies

Proposition 3.1 (Optimality over the unrestricted policy class). Under Assumption 3.1(i)-(ii), there is a linear-Gaussian policy π⋆ (· | x) = N (−K⋆ x, Σ⋆ ) such that, for every π ∈ Πadm and every x ∈ Rn , V π (x) − V π⋆ (x) = τ Eπx

"∞ X t=0

≥ 0.

7

#

γ t KL(π(· | xt ) ∥π⋆ (· | xt ))

(6)

Hence, up to equality of the conditional densities almost everywhere, π⋆ is the unique policy in Πadm that is optimal for every initial state. In particular, min C(π) =

π∈Πadm

min

(K,Σ)∈Aadm

C(K, Σ) = C(K⋆ , Σ⋆ ).

(7)

Thus the policy optimization problem can be restricted to admissible linear-Gaussian policies without loss of optimality. The optimal parameters satisfy EK⋆ = 0,

−1 K⋆ = MK NK⋆ , ⋆

Σ⋆ =

τ −1 M . 2 K⋆

Moreover, P⋆ := PK⋆ is the unique stabilizing solution of the discounted algebraic Riccati equation 

P⋆ = Q + γA⊤ P⋆ A − γ 2 A⊤ P⋆ B R + γB ⊤ P⋆ B

−1

B ⊤ P⋆ A.

The proof is given in Appendix B. Proposition 3.1 establishes the order of the analysis. Having first solved the unrestricted control problem, we may now solve the finite-dimensional learning problem min

(K,Σ)∈Aadm

C(K, Σ).

By (7), this restriction is without loss of optimality for the original control problem. The next section derives the WPG direction on Gadm and shows that it has an exact representation in the gain and covariance. Thus optimality of the policy class and closure of the update are established as separate results. Remark 3.1 (Limit as the entropy temperature tends to zero). The Riccati equation, and hence P⋆ , K⋆ , and MK⋆ , is independent of τ . The temperature parameter affects only the optimal covariance: Σ⋆ =

τ −1 M . 2 K⋆

Consequently, Σ⋆ → 0 linearly as τ ↓ 0, and, for every fixed x, π⋆ (· | x) = N (−K⋆ x, Σ⋆ ) −→ δ−K⋆ x in W2 . Thus the entropy-regularized optimal policy converges to the deterministic discounted LQ controller as τ ↓ 0.

4

Closure of the WPG Direction on the Linear-Gaussian Policy Class

By Proposition 3.1, it suffices to optimize over Gadm . This section defines the policy-space metric, derives WPGF as its gradient-descent flow, and obtains its exact gain-covariance representation on this policy class. Because the state space is unbounded and the costs have quadratic growth, the soft Bellman operator is naturally defined on (

V2 :=

)

|v(x)| v : Rn → R : ∥v∥V2 := sup <∞ . 2 x∈Rn 1 + ∥x∥ 8

For a Markov policy π, define its closed-loop transition operator by (Pπ v)(x) := Eπ [v(x1 ) | x0 = x], and write PK,Σ := PπK,Σ . For v ∈ V2 , define

Qv (x, u) := c(x, u) + γ E[v(Ax + Bu + w)].

For a policy π, write Qπ := QV π and define its soft policy-evaluation operator by (Tπ v)(x) :=

Z

Qv (x, u) + τ log π(u | x) π(du | x).

Rm

Whenever the action minimization is finite, the soft Bellman operator is (T∗ v)(x) := inf

Z

Qv (x, u) + τ log ϱ(u | x) ϱ(du | x).

ϱ(·|x) Rm

Definition 4.1 (Relative entropy, Fisher information, and LSI). Let p be a probability law on Rm with positive smooth density, and let ν ≪ p. Define dν dν, KL(ν∥p) := log dp Rm Z

I(ν∥p) :=

Z Rm

∇u log

dν 2 dν. dp

We use the convention I(ν∥p) = +∞ if log(dν/dp) does not have a weak gradient with respect to u. We say that p satisfies an α-log-Sobolev inequality if I(ν∥p) ≥ 2α KL(ν∥p)

4.1

for all probability laws ν ≪ p.

Policy-space Wasserstein metric and gradient flow

We now state the policy-space geometry used to define WPG. It is the statewise 2-Wasserstein geometry used in the WPG literature, with the statewise metric weighted by the current discounted state-occupancy measure (Zhang et al., 2018; Pfau et al., 2025; Zhu et al., 2026a). For a policy π, let dπD (A) := (1 − γ)

∞ X

γ t PπD (xt ∈ A),

A ∈ B(Rn ),

t=0

be its normalized discounted state-occupancy measure. Definition 4.2 (Discounted-occupancy-weighted statewise Wasserstein metric). Fix a probability measure ν on the state space. For two policies whose conditional action laws belong to P2 (Rm ) and for which the integral below is finite, define 2 e ) := W2,ν (π, π

1 1−γ

Z Rn

e (· | x)) ν(dx). W22 (π(· | x), π

Policies that agree for ν-almost every state are identified. At a base policy π with positive smooth conditional densities, WPG uses ν = dπD . To describe the corresponding tangent metric, begin with a test potential φ ∈ Cc∞ (Rn × Rm ), set v = ∇u φ, and define the mass-preserving tangent perturbation ξv (u | x) := −∇u · π(u | x)v(x, u) .

9

The tangent velocity space is the closure of these action-gradient fields in the norm below. Equivalently, each tangent perturbation is represented by its minimum-L2 (π(· | x))-norm action-gradient velocity. For tangent perturbations ξ and ζ, with gradient representatives vξ and vζ , define 1 dπ (dx) ⟨vξ (x, u), vζ (x, u)⟩π(du | x), 1 − γ Rn D Rm ∥ξ∥2π,W := ⟨ξ, ξ⟩π,W . Z

⟨ξ, ζ⟩π,W :=

Z

For fixed ν, W2,ν is the direct integral of the statewise 2-Wasserstein distances. WPG uses its tangent metric with the state weight evaluated at the base policy. The factor (1 − γ)−1 converts the normalized occupancy measure into the usual discounted sum. If dπD does not have full support, the tangent norm is understood after identifying perturbations that agree dπD -almost everywhere in the state variable. We next compute the first variation at an admissible linear-Gaussian policy. Fix (K, Σ) ∈ Aadm , write π = πK,Σ , take φ ∈ Cc∞ (Rn × Rm ), and set v = ∇u φ,

Tε,x (u) = u + εv(x, u),

π ε (· | x) = (Tε,x )# π(· | x).

Lemma C.1 proves directly that this transport path is well defined for all sufficiently small |ε|, that C(π ε ) is differentiable at zero, and that 1 DC(π)[ξv ] = 1−γ

Z

Z

Qπ (x, u) + τ log π(u | x) ξv (u | x) du. dπD (dx) Rn Rm

Subtracting the conditional mean of the expression in parentheses gives the usual centered firstR variation representative and does not change the value because ξv (u | x) du = 0. Integration by parts gives 1 DC(π)[ξv ] = 1−γ

Z

Z

dπD (dx) Rm Rn

∇u Qπ (x, u) + τ log π(u | x) , v(x, u) π(du | x).

(8)

The same lemma verifies the domination, differentiation of the discounted series, Fubini interchange, and boundary terms used in these formulas. Importantly, the perturbing policies π ε need not be Gaussian; thus the calculation identifies the policy-space gradient at a linear-Gaussian policy before the closure result below is invoked. Proposition 4.1 (WPGF as a policy-space gradient flow). At every admissible linear-Gaussian policy π = πK,Σ , the gradient of C under the metric in Definition 4.2 is gradW C(π)(u | x) = −∇u · π(u | x)∇u Qπ (x, u) + τ log π(u | x) . 



(9)

Consequently, the gradient-descent flow ∂t πt = − gradW C(πt ) is ∂t πt (u | x) = ∇u · πt (u | x)∇u Qπt (x, u) + τ log πt (u | x) . 



(WPGF)

Proof. Fix π = πK,Σ and set gπ (x, u) := ∇u Qπ (x, u) + τ log π(u | x) .

The function inside the action gradient is quadratic in (x, u), so gπ is affine. Admissibility gives a finite discounted second state moment, and the conditional action law has finite second moments. Hence Z Z 1 dπD (dx) ∥gπ (x, u)∥2 π(du | x) < ∞. 1−γ 10

Moreover, gπ belongs to the completed tangent velocity space in Definition 4.2. It is the action gradient of a quadratic function, and smooth cutoffs in the state and action variables approximate it in the metric norm; the details are given at the end of the proof of Lemma C.1. For every test velocity v = ∇u φ, Equation (8) gives DC(π)[ξv ] = ⟨−∇u · (πgπ ), ξv ⟩π,W . The test velocities are dense by definition, so the identity extends by continuity to the tangent space. This proves (9); changing the sign gives (WPGF). The same discounted occupancy measure and normalization appear in the first variation and in the metric, so no additional state-dependent factor enters the WPG velocity. The metric determines the gradient for dπD (dx)π(du | x)-almost every (x, u). As in the statewise WPG convention, we use the displayed Bellman field as the representative at every state where it is defined. This gives the full-state policy update used in the closure calculation below. For each state x, define the Gibbs density associated with the current soft action-value function by
exp −Qπt (x, u)/τ pt (u | x) := Z (10)
. exp −Qπt (x, a)/τ da Rm

Since the normalizing constant in (10) is independent of u, ∇u Qπt (x, u) + τ log πt (u | x) = τ ∇u log

πt (u | x) . pt (u | x)

Consequently, WPGF has the equivalent Fokker-Planck representation πt (u | x) . ∂t πt (u | x) = τ ∇u · πt (u | x)∇u log pt (u | x) 



(11)

The Gibbs law pt (· | x) is not fixed: it is determined by Qπt , which depends on the current policy through the Bellman equation. Thus, at each time, (11) is the relative-entropy gradient flow toward the current Gibbs law, while that Gibbs law changes with the policy.

4.2

Closure on the linear-Gaussian policy class

We now evaluate the statewise WPG vector field on Gadm . Lemma 3.1 gives QπK,Σ (x, u) = u⊤ MK u + 2u⊤ NK x + ψK,Σ (x),

(12)

where ψK,Σ is independent of u. Hence ∇u QπK,Σ (x, u) = 2MK u + 2NK x, which is affine in u. The following proposition shows that the WPG vector field is tangent to the linear-Gaussian policy class and identifies its exact parameter representation. Proposition 4.2 (Exact closure on linear-Gaussian policies). For every (K, Σ) ∈ Aadm , the righthand side of the statewise WPG equation evaluated at πK,Σ is tangent to Gadm . More precisely, let

11

DπK,Σ [K̇, Σ̇] denote the directional derivative of the Gaussian density with respect to its parameters. The unique parameter velocity (K̇, Σ̇) satisfying DπK,Σ [K̇, Σ̇](u | x) = ∇u ·[πK,Σ (u | x)∇u (QπK,Σ (x, u) + τ log πK,Σ (u | x))]

(13)

for every x and u is K̇ = −2EK ,

Σ̇ = −2MK Σ − 2ΣMK + 2τ I.

(14)

Consequently, on every interval on which (Kt , Σt ) is an admissible solution of K̇t = 2(Nt − Mt Kt ) = −2Et ,

(15)

Σ̇t = −2Mt Σt − 2Σt Mt + 2τ I, the curve πt = πKt ,Σt is a linear-Gaussian solution of the statewise WPG equation (WPGF).

Remark 4.1 (Meaning of the closure result). Proposition 4.2 identifies the statewise WPG vector field exactly on the policy class that is sufficient for optimality. The parameter ODE is therefore the WPG dynamics on Gadm , rather than a projection or approximation of the policy-space direction. The proof is given in Appendix C. Proposition 4.2 defines the closed linear-Gaussian WPG parameter flow analyzed below. Here all Bellman quantities are evaluated at the current gain Kt . Namely, with Pt := PKt ,

Mt := R + γB ⊤ Pt B,

Nt := γB ⊤ Pt A,

Et := Mt Kt − Nt ,

the matrix Pt is the unique positive semidefinite solution of Pt = Q + Kt⊤ RKt + γ(A − BKt )⊤ Pt (A − BKt ). Consequently, the Gibbs law in (10) is also Gaussian: pt (· | x) = N (−Mt−1 Nt x, Ξt ),

Ξt :=

τ −1 M . 2 t

(16)

The gain equation follows the feedback residual Et = Mt Kt − Nt . The covariance equation combines contraction generated by the quadratic action-cost matrix Mt with the additive entropy term 2τ I. Remark 4.2 (Relation to other policy geometries). For comparison, Table 1 in Appendix A summarizes the joint gain and covariance updates induced by WPGF and other policy geometries.

5

Global Convergence Analysis

We now analyze the linear-Gaussian parameter flow. A Bellman resolvent identity converts the relative Fisher-information dissipation at each state into discounted value decrease. The Gaussian Bellman residual and the LQ performance-difference identity then compare this decrease with the global objective gap. All estimates are expressed through the matrices PK , MK , EK , and SK,Σ ; no bounded value-function or soft-Q estimate is used.

12

5.1

Value descent through the Bellman resolvent

Because the parameter ODE is the exact closure of the statewise Wasserstein policy gradient on Gadm , its descent is most naturally expressed at the value-function level. Let Vt := V πt , Define

Pt := Pπt = PKt ,Σt .

gt (x) := τ 2 I πt (· | x)∥pt (· | x) ,

where pt is the current Gibbs policy in (16). Then gt (x) ≥ 0, and the following identity converts local action-space dissipation into value decrease. Lemma 5.1 (Value resolvent identity). Along the linear-Gaussian parameter flow in Proposition 4.2, (I − γPt ) Consequently,

d Vt = −gt . dt

(17)

∞ X d Vt (x) = − γ i (Pti gt )(x) ≤ −gt (x) ≤ 0, dt i=0

∀x ∈ Rn .

(18)

The proof is given in Appendix C. This identity propagates the action-space dissipation at each state through the closed-loop dynamics and converts it into monotone value improvement.

5.2

Bellman identities

For an admissible (K, Σ), define τ −1 −1 NK x, ΞK ). , pK (· | x) := N (−MK ΞK := MK 2 The next two identities are the LQ-specific inputs to the convergence proof; both are proved in Appendix B. Lemma 5.2 (Bellman residual and Gaussian LSI). Fix an admissible (K, Σ). The Bellman residual RK,Σ (x) := VK,Σ (x) − (T∗ VK,Σ )(x) satisfies RK,Σ (x) = τ KL(πK,Σ (· | x) ∥ pK (· | x))  τ −1 −1 ⊤ Tr(Ξ−1 = x⊤ EK MK EK x + K Σ) − m − log det(ΞK Σ) . 2 n Moreover, for every x ∈ R , define 2λmin (MK ) , τ Then αK ≥ α, and the Gibbs law pK (· | x) satisfies αK := λmin (Ξ−1 K )=

α :=

2λR . τ

I(ν∥pK (· | x)) ≥ 2αK KL(ν ∥ pK (· | x)) ≥ 2αKL(ν ∥ pK (· | x))

(19)

(20)

for every probability law ν ≪ pK (· | x). The coefficient αK is sharp for this Gaussian law. Equivalently, in the convention KL(ν∥pK ) ≤ (CLSI (pK )/2)I(ν∥pK ), the sharp constant is τ CLSI (pK ) = . 2λmin (MK ) 13

−1 The sharp coefficient follows directly from the covariance ΞK = (τ /2)MK . The convergence proof uses the uniform lower bound α = 2λR /τ , obtained from MK ⪰ R; this lower bound need not be sharp for a fixed K. In the general WPG analysis of Zhu et al. (2026a), the moving Gibbs law is controlled as a bounded perturbation of a Gaussian, leading to a coefficient of the form   β 2Umax αgen = exp − . τ τ

In the present LQ model, no perturbation step is needed. Since WPG dissipates τ 2 I(π∥p) and the Bellman residual is R = τ KL(π∥p), the uniform bound gives τ 2 I(π∥p) ≥ 4λR R. Thus this part of the contraction estimate contains no factor of the form exp(−c/τ ). Lemma 5.3 (Performance-difference identity). Let (K, Σ) and (K ′ , Σ′ ) be admissible, and define τ 1 ∆K := K ′ − K, fK (Σ) := log det Σ − Tr(MK Σ). 2(1 − γ) 1−γ Then    C(K ′ , Σ′ ) − C(K, Σ) = Tr SK ′ ,Σ′ ∆K ⊤ MK ∆K + 2∆K ⊤ EK (21) + fK (Σ) − fK (Σ′ ). Consequently, C is differentiable on the admissible set, with   τ 1 MK − Σ−1 . (22) ∇K C(K, Σ) = 2EK SK,Σ , ∇Σ C(K, Σ) = 1−γ 2 Together, these identities connect the stationary conditions of the reduced flow to the global objective gap.

5.3

Global well-posedness

To prove global existence, we must show that the trajectory cannot lose discounted stability or approach a singular covariance. The next proposition identifies admissibility with finite cost and shows that every finite-cost sublevel set is a compact subset of the admissible set. Proposition 5.1 (Finite cost, stability, and compact sublevels). Under Assumption 3.1, for every K ∈ Rm×n and Σ ∈ Sm ++ , √ C(K, Σ) < ∞ ⇐⇒ ρ( γ FK ) < 1. Moreover, for every c ∈ R, the sublevel set Sc := {(K, Σ) ∈ Aadm : C(K, Σ) ≤ c} is compact. In particular, it has positive distance from the boundary of Aadm . The proof is given in Appendix D. Detectability is essential: without it, a discounted-unstable mode that is invisible to the state cost may still have finite objective value. Lemma 5.4 (Global well-posedness of the closed parameter flow). Under Assumption 3.1, the ODE (15) with initial condition (K0 , Σ0 ) has a unique solution {(Kt , Σt )}t≥0 for all t ≥ 0. The trajectory remains admissible and stays in the sublevel set SC(K0 ,Σ0 ) := {(K, Σ) ∈ Aadm : C(K, Σ) ≤ C(K0 , Σ0 )} . Thus the closed WPG parameter flow is well defined for all t ≥ 0. The proof is given in Appendix D. It is a finite-dimensional ODE argument based on descent of the cost and compactness of finite-cost sublevel sets. 14

5.4

Global convergence and temperature dependence

The next theorem gives the global objective rate. Two corollaries make its parameter and temperature dependence explicit. Theorem 5.1 (Global exponential convergence). Under Assumption 3.1, let {(Kt , Σt )}t≥0 be the WPG parameter trajectory from an admissible initialization (K0 , Σ0 ), and let (K⋆ , Σ⋆ ) be the optimizer in Proposition 3.1. Define S⋆ := SK⋆ ,Σ⋆ ,

λτ :=

4µλR > 0. ∥S⋆ ∥

Then, along the WPG parameter trajectory, −

d C(Kt , Σt ) ≥ λτ C(Kt , Σt ) − C(K⋆ , Σ⋆ ) . dt

(23)

Consequently, for every t ≥ 0, C(Kt , Σt ) − C(K⋆ , Σ⋆ ) ≤ exp(−λτ t) C(K0 , Σ0 ) − C(K⋆ , Σ⋆ ) .

(24)

Equivalently, if 0 < ε < C(K0 , Σ0 ) − C(K⋆ , Σ⋆ ), then the objective gap is at most ε once C(K0 , Σ0 ) − C(K⋆ , Σ⋆ ) ∥S⋆ ∥ log . t≥ 4µλR ε 



(25)

Corollary 5.1 (Gain and entropy-weighted covariance convergence). Define Φ(X) := Tr(X) − m − log det X,

X ∈ Sm ++ .

Along the trajectory in Theorem 5.1, µλR ∥Kt − K⋆ ∥2F +

  τ −1/2 −1/2 Φ Σ⋆ Σt Σ⋆ 2(1 − γ)

(26)

≤ exp(−λτ t) C(K0 , Σ0 ) − C(K⋆ , Σ⋆ ) .

In particular, the squared gain error and the entropy-weighted relative covariance error decay with the same exponential exponent as the objective gap. Proof. Apply the performance-difference identity in Lemma 5.3 with the optimal pair (K⋆ , Σ⋆ ) as the reference policy. Since EK⋆ = 0, 



C(K, Σ) − C(K⋆ , Σ⋆ ) = Tr SK,Σ (K − K⋆ )⊤ MK⋆ (K − K⋆ )   τ −1/2 −1/2 . + Φ Σ⋆ ΣΣ⋆ 2(1 − γ)

Using SK,Σ ⪰ Γ0 ⪰ µI and MK⋆ ⪰ R ⪰ λR I, and then applying Theorem 5.1, proves the claim. Corollary 5.2 (Temperature dependence of the exponential rate). Let F⋆ := A − BK⋆ ,

M⋆ := MK⋆ .

The matrices K⋆ , P⋆ , M⋆ , and F⋆ do not depend on τ . Moreover, (0)

(1)

S⋆ (τ ) = S⋆ + τ S⋆ , 15

(27)

(0)

where S⋆

(1)

and S⋆

are the unique positive semidefinite solutions of (0)

(0)

S⋆ = Γ0 + γF⋆ S⋆ F⋆⊤ + (1)

(1)

S⋆ = γF⋆ S⋆ F⋆⊤ + Consequently, λτ ≥

γ W, 1−γ

γ BM⋆−1 B ⊤ . 2(1 − γ)

4µλR (0) S⋆

and lim λτ = τ ↓0

,

(1)

+ τ S⋆ 4µλR (0)

S⋆

> 0.

Thus, for each fixed LQ problem, the exponent has the displayed positive limit. The time constant (0) ∥S⋆ (τ )∥ /(4µλR ) in (25) is at most affine in τ and converges to S⋆ /(4µλR ) as τ ↓ 0. Proof. Proposition 3.1 shows that the Riccati equation, and hence K⋆ , P⋆ , M⋆ , and F⋆ , is independent of τ , while τ Σ⋆ (τ ) = M⋆−1 . 2 For any admissible (K, Σ), the discounted state-correlation matrix satisfies ⊤ SK,Σ = Γ0 + γFK SK,Σ FK +

γ (BΣB ⊤ + W ). 1−γ

Substituting the optimal pair and the formula for Σ⋆ (τ ) gives (27) and the two displayed Lyapunov equations. The norm bound follows from the triangle inequality, and the limit follows from continuity of the spectral norm. Remark 5.1 (Interpretation of the rate bound). Corollary 5.2 isolates the temperature dependence for a fixed LQ problem and a fixed choice of coordinates. Its magnitude continues to reflect the usual conditioning of the control problem: discounting and the optimal closed-loop stability margin enter through ∥S⋆ ∥, initial-state excitation enters through µ, and action-cost curvature enters through λR . State and action rescalings change these quantities as well. The result therefore removes the additional exponential low-temperature factor while preserving the familiar instance dependence. Remark 5.2 (Interpretation of the covariance estimate). Set −1/2

Xt := Σ⋆

−1/2

Σt Σ⋆

.

The covariance quantity in (26) is τ Φ(Xt ), up to the fixed factor 2(1 − γ), and is exactly the covariance contribution to the entropy-regularized objective. Since Φ(X) is locally equivalent to ∥X − I∥2F near I, this yields exponential convergence of the relative covariance for every fixed τ > 0. On a fixed spectral neighborhood of I, an unweighted bound for ∥Xt − I∥2F has a prefactor proportional to 1/τ . Proof sketch of Theorem 5.1. The proof has two quantitative steps. First, the Bellman resolvent identity and the Gaussian log-Sobolev inequality convert Wasserstein dissipation into an accumulated soft Bellman residual. Second, the LQ performance-difference identity compares this residual with the global objective gap. 16

Let

Rt (x) := Vt (x) − (T∗ Vt )(x),

St := SKt ,Σt ,

and define

Rt := Ex0 ∼D

h

i

(I − γPπt )−1 Rt (x0 ) .

Lemma 5.1 expresses objective dissipation as the discounted accumulation of relative Fisher information under the current policy. Lemma 5.2 gives the sharp coefficient αKt = 2λmin (Mt )/τ , its uniform lower bound 2λR /τ , and the identity Rt = τ KL(πt ∥pt ). Hence the explicit factors of τ cancel and d − C(Kt , Σt ) ≥ 4λR Rt . dt The Gaussian KL formula yields 



Rt = Tr St Et⊤ Mt−1 Et + Bt ,

Bt := fKt (Ξt ) − fKt (Σt ) ≥ 0.

The LQ performance-difference identity gives 



C(Kt , Σt ) − C(K⋆ , Σ⋆ ) ≤ Tr S⋆ Et⊤ Mt−1 Et + Bt . Using

St ⪰ Γ0 ⪰ µI,

S⋆ ⪯ ∥S⋆ ∥ I,

and Bt ≥ 0, we obtain

µ C(Kt , Σt ) − C(K⋆ , Σ⋆ ) . ∥S⋆ ∥ Combining these estimates proves (23), and Gronwall’s inequality gives (24). Appendix E contains the complete proof.

Rt ≥

6

Conclusion

We studied entropy-regularized discounted LQ control in two stages. A Bellman verification argument first shows that a linear-Gaussian policy is optimal over the unrestricted admissible stationary policy class. The policy optimization problem can therefore be restricted to the gain and covariance without loss of optimality. We then stated the discounted-occupancy-weighted statewise 2-Wasserstein metric explicitly and derived WPGF as the negative gradient flow of the control objective. At every admissible linear-Gaussian policy, the WPG direction is tangent to the same class and is represented exactly by the feedback-gain and covariance velocities. The resulting finite-dimensional ODE generates a linear-Gaussian solution of WPGF without projection and is globally well posed from every admissible initialization. This is the well-posedness result needed for the parameter-flow analysis; it does not rely on a uniqueness statement for every solution of the unrestricted nonlinear policy equation. The parameter equations also clarify the comparison with Fisher–Rao flow: the two gain update formulas coincide when the action covariance is the identity, whereas their covariance update laws are fundamentally different. The objective gap and squared gain error converge exponentially. The sharp Gaussian log-Sobolev coefficient of the current Gibbs law is 2λmin (MK )/τ , and its uniform lower bound 2λR /τ cancels the explicit temperature factor in the Fisher-information dissipation. Consequently, for each fixed LQ problem and action metric, the proved exponent has a positive limit as τ ↓ 0 and contains no perturbative exp(−c/τ ) factor. Its magnitude continues to reflect the usual LQ conditioning through discounting, closed-loop stability, initial-state excitation, action-cost scaling, and coordinate choice. Covariance convergence is controlled in the entropy-weighted relative error; an unweighted relative covariance estimate can have a prefactor proportional to 1/τ . 17

A

Comparison with Other Policy Geometries

Table 1 writes several LQ control policy-gradient directions in a common notation. Gain-only methods optimize K while fixing or omitting the exploration covariance. In the joint linearGaussian setting, regularized policy gradient (RPG) and the Fisher-Rao natural gradient for the full linear-Gaussian policy both precondition the covariance gradient on the left and right, but their gain components differ: only the Fisher-Rao gain direction is premultiplied by Σ. The closed WPG parameter equations instead follow directly from the statewise Wasserstein direction of the conditional action distributions. Table 1: Descent directions for the infinite-horizon discounted (entropy-regularized) LQ control problem. Method

Geometry / preconditioner

Gain direction

Covariance direction

Theory / distinction

Deterministic-policy methods for standard LQ control: only K is optimized PG (Fazel et al., Euclidean in K 2018)

−2EK S

0

Gain-only LQ control baseline; covariance is not learned.

Gain NPG (Fazel State-correlation et al., 2018) preconditioning in K

−2EK

0

Standard LQ control NPG.

Entropy-regularized LQ control over joint linear-Gaussian policies: both K and Σ are optimized 1 Euclidean GF Product Euclidean −2EK S − GΣ Formal Euclidean gradient 1 − γ geometry on (K, Σ) flow for the joint parameter (K, Σ); it does not preserve the geometry of positive-definite covariance matrices. −

β ΣGΣ Σ 1−γ

RPG (Guo et al., Separate gain and 2026) covariance preconditioners

−2EK

Full Fisher-Rao NPG

Fisher-Rao geometry of the complete conditional linear-Gaussian family

−2ΣEK

−2ΣGΣ Σ

Natural gradient for the full conditional linear-Gaussian policy; its gain component differs from RPG and WPGF.

WPGF (ours)

Conditional action-space W2 ; on the linear-Gaussian family, the induced metric is Euclidean for the mean and the W2 metric for the covariance

−2EK

−2(GΣ Σ + ΣGΣ )

Global well-posedness and an explicit global exponential objective rate.

Discrete global linear rate under its stated conditions; not the full linear-Gaussian Fisher-Rao NPG.

Notation and conventions. S := SK,Σ and GΣ := MK − (τ /2)Σ−1 . All rows use the cost-minimization sign convention. PG and gain NPG freeze Σ. For RPG, the displayed vector field is the infinitesimal direction of the Guo-Li-Xu iteration with η1 = h, η2 = βh, and h ↓ 0; thus β > 0 records the covariance-to-gain step-size ratio, while its stated guarantee is for the discrete iterates. The Fisher-Rao row uses the discounted Fisher metric.

These directions are not interchangeable. Gain-only PG and NPG do not define a joint flow on (K, Σ); RPG is a discrete method with separately chosen gain and covariance step-size scalings; and Euclidean and full Fisher-Rao flows use different joint policy geometries. The main theorem therefore analyzes the exact gain-covariance dynamics induced by WPG, including global well-posedness and the explicit global convergence.

18

Construction of Figure 1. Let the task-relevant action subspace have dimension r = 8. In an eigenbasis of M⋆ , set the optimal covariance eigenvalues to ξi⋆ = 10−(i−1)/(r−1) ,

i = 1, . . . , r,

and set every redundant-coordinate eigenvalue to 10−6 . Since ξi⋆ = τ /(2mi ), these values determine the corresponding eigenvalues mi of M⋆ . With the rescaled time s = τ t, the local WPG gain error in mode i is ⋆ eW i (s) = exp(−s/ξi ). For the Fisher-Rao flow, write qi = σi /ξi⋆ . To first order in the gain error, the modal covariance and gain equations are dqi deFR i = qi (1 − qi ), = −qi eFR i , ds ds so, from qi (0) = qi,0 and eFR i (0) = 1, eFR i (s) =

1 . 1 + qi,0 (es − 1)

Figure 1 plots the worst error over the eight relevant modes. For each ambient dimension m, the common isotropic initialization is Σ0 = cm I, where mcm = 10 Tr(Σ⋆ ); the trace includes the m − r redundant-coordinate eigenvalues. Hence qi,0 = cm /ξi⋆ decreases as redundant coordinates are added, which slows Fisher-Rao but leaves WPG unchanged.

B

Linear-Quadratic Identities for Linear-Gaussian Policies

This appendix proves the LQ identities used in the convergence analysis.

B.1

Quadratic value representation

We begin with the quadratic representation of the value and soft action-value functions for a fixed linear-Gaussian policy. Proof of Lemma 3.1. Fix an admissible pair (K, Σ). Under πK,Σ , ut = −Kxt + ξt ,

ξt ∼ N (0, Σ),

and hence xt+1 = FK xt + ηt ,

FK := A − BK,

Cov(ηt ) = ΩΣ := BΣB ⊤ + W.

√ Since ρ( γFK ) < 1, the series PK =

∞ X

t ⊤ t γ t (FK ) LK FK ,

LK := Q + K ⊤ RK,

t=0

converges. Shifting the summation index gives ⊤ PK = LK + γFK PK FK .

19

Conversely, iterating this Lyapunov equation shows that any solution must coincide with the preceding series. Thus PK is its unique positive-semidefinite solution. We next determine the constant term of the value function. Define

τ rΣ := Tr(RΣ) − m + log (2π)m det Σ . 2 The expected one-step regularized cost under πK,Σ is E[c(x, u) + τ log πK,Σ (u | x) | x] = x⊤ LK x + rΣ . Consider the quadratic function Ve (x) := x⊤ PK x + q. Using E[ηt ] = 0 and Cov(ηt ) = ΩΣ , we have ⊤ E[Ve (xt+1 ) | xt = x] = x⊤ FK PK FK x + Tr(PK ΩΣ ) + q.

Therefore

h

i

E c(x, u) + τ log πK,Σ (u | x) + γ Ve (xt+1 ) xt = x 



⊤ = x⊤ LK + γFK PK FK x + rΣ + γ Tr(PK ΩΣ ) + γq.

By the Lyapunov equation, the quadratic coefficient equals PK . Thus Ve satisfies the policy Bellman equation provided (1 − γ)q = rΣ + γ Tr(PK ΩΣ ). Since



Tr(RΣ) + γ Tr(PK BΣB ⊤ ) = Tr Σ(R + γB ⊤ PK B)



= Tr(ΣMK ), where

MK := R + γB ⊤ PK B,

the required constant is q = qK,Σ =

1 1−γ



Tr(ΣMK ) −

τ m + log (2π)m det Σ + γ Tr(W PK ) . 2 

Iterating the policy Bellman identity over a finite horizon and using h

i

γ T E 1 + ∥xT ∥2 x0 = x −→ 0 under discounted stability shows that V πK,Σ (x) = x⊤ PK x + qK,Σ . We now compute the corresponding soft state-action value. By definition, QπK,Σ (x, u) = c(x, u) + γE[V πK,Σ (Ax + Bu + w)] . Substituting the quadratic value representation gives QπK,Σ (x, u) = x⊤ Qx + u⊤ Ru + γ(Ax + Bu)⊤ PK (Ax + Bu) + γ Tr(W PK ) + γqK,Σ . 20

Hence, with

NK := γB ⊤ PK A,

we obtain

QπK,Σ (x, u) = u⊤ MK u + 2u⊤ NK x + ψK,Σ (x),

where





ψK,Σ (x) = x⊤ Q + γA⊤ PK A x + γ Tr(W PK ) + γqK,Σ .

Finally, taking expectation over x0 ∼ D yields C(K, Σ) = Tr(Γ0 PK ) + qK,Σ .

B.2

Proof of the optimality of a linear-Gaussian policy

Proof of Proposition 3.1. Apply the standard discrete-time Riccati theorem to the scaled system √ e := √γB. Ae := γA, B Under Assumption 3.1(i)-(ii), the discounted algebraic Riccati equation 

P = Q + γA⊤ P A − γ 2 A⊤ P B R + γB ⊤ P B has a unique stabilizing solution P⋆ ⪰ 0. Define M⋆ := R + γB ⊤ P⋆ B, and

−1

B⊤P A

N⋆ := γB ⊤ P⋆ A,

τ (M⋆ )−1 . 2 Since M⋆ ⪰ R ≻ 0, we have Σ⋆ ≻ 0. Moreover, the stabilizing property of P⋆ gives √ ρ( γ(A − BK⋆ )) < 1. K⋆ := (M⋆ )−1 N⋆ ,

Set

Σ⋆ :=

1 τ τm q⋆ := γ Tr(P⋆ W ) + log det M⋆ − log(πτ ) , 1−γ 2 2 and define the quadratic candidate 



V (x) := x⊤ P⋆ x + q⋆ . Fix x ∈ Rn . Define the density det(M⋆ )1/2 1 gx (u) := exp − (u + K⋆ x)⊤ M⋆ (u + K⋆ x) . m/2 τ (πτ ) 

This is the density of



π⋆ (· | x) = N (−K⋆ x, Σ⋆ ).

Let ρ(· | x) be any probability density for which the following expression is finite. Since M⋆ K⋆ = N⋆ , completing the square gives u⊤ M⋆ u + 2u⊤ N⋆ x = (u + K⋆ x)⊤ M⋆ (u + K⋆ x) − x⊤ (N⋆ )⊤ (M⋆ )−1 N⋆ x. 21

By the definition of gx , u⊤ M⋆ u + 2u⊤ N⋆ x + τ log ρ(u | x) τ τm ρ(u | x) = −x⊤ (N⋆ )⊤ (M⋆ )−1 N⋆ x + log det M⋆ − log(πτ ) + τ log . 2 2 gx (u) Integrating with respect to ρ(· | x) yields Z



Rm



u⊤ M⋆ u + 2u⊤ N⋆ x + τ log ρ(u | x) ρ(du | x)

τm τ log det M⋆ − log(πτ ) 2 2 + τ KL (ρ(· | x) ∥ π⋆ (· | x)) .

= −x⊤ (N⋆ )⊤ (M⋆ )−1 N⋆ x +

On the other hand, c(x, u) + γE[V (Ax + Bu + w)] 



= x⊤ Q + γA⊤ P⋆ A x + γ Tr(P⋆ W ) + γq⋆ + u⊤ M⋆ u + 2u⊤ N⋆ x. Combining the preceding two identities, we obtain Z



Rm



c(x, u) + γE[V (Ax + Bu + w)] + τ log ρ(u | x) ρ(du | x) 



= x⊤ Q + γA⊤ P⋆ A − (N⋆ )⊤ (M⋆ )−1 N⋆ x τm τ log(πτ ) + γ Tr(P⋆ W ) + γq⋆ + log det M⋆ − 2 2 + τ KL (ρ(· | x) ∥ π⋆ (· | x)) . The Riccati equation implies Q + γA⊤ P⋆ A − (N⋆ )⊤ (M⋆ )−1 N⋆ = P⋆ , while the definition of q⋆ implies γ Tr(P⋆ W ) + γq⋆ + Therefore

Z Rm



τ τm log det M⋆ − log(πτ ) = q⋆ . 2 2 

c(x, u) + γE[V (Ax + Bu + w)] + τ log ρ(u | x) ρ(du | x)

= V (x) + τ KL (ρ(· | x) ∥ π⋆ (· | x)) . Since relative entropy is nonnegative and vanishes only when its two arguments agree almost everywhere, taking the infimum over all action densities gives T∗ V = Tπ⋆ V = V . Moreover, π⋆ (· | x) is the unique minimizer of the soft Bellman expression for every x. Let π ∈ Πadm . Applying (∗) with ρ(· | x) = π(· | x) along a trajectory generated by π gives h

Eπx c(xt , ut ) + τ log π(ut | xt ) + γV (xt+1 ) xt = V (xt ) + τ KL (π(· | xt ) ∥ π⋆ (· | xt )) . 22

i

Multiplying by γ t , taking expectations, and summing from t = 0 to T − 1, the value terms telescope and yield " # Eπx

TX −1

γ t (c(xt , ut ) + τ log π(ut | xt ))

t=0

= V (x) − γ T Eπx [V (xT )] + τ Eπx

"T −1 X

#

γ t KL (π(· | xt ) ∥ π⋆ (· | xt )) .

t=0

By the definition of Πadm and the quadratic growth of V , γ T Eπx [|V (xT )|] −→ 0. Letting T → ∞, we obtain V π (x) = V (x) + τ Eπx

"∞ X

#

γ KL (π(· | xt ) ∥ π⋆ (· | xt )) t

t=0

≥ V (x). For π = π⋆ , every relative-entropy term vanishes. Furthermore, discounted stability implies the required transversality, and hence V π⋆ (x) = V (x). Combining this equality with (∗∗) gives (6). If another policy is optimal for every initial state, then the time-zero relative-entropy term vanishes for every x, so its conditional density agrees almost everywhere with π⋆ (· | x). This proves uniqueness. Because π⋆ is admissible and linear-Gaussian, equality (7) follows. It remains to express the result in the policy-evaluation notation. Using K⋆ = (M⋆ )−1 N⋆ , the Riccati equation can be rewritten as P⋆ = Q + (K⋆ )⊤ RK⋆ + γ(A − BK⋆ )⊤ P⋆ (A − BK⋆ ). Since K⋆ is stabilizing, this discounted Lyapunov equation has a unique positive-semidefinite solution. Therefore PK⋆ = P⋆ . Consequently, and hence as well as

MK⋆ = M⋆ , −1 K⋆ = MK NK⋆ , ⋆

NK⋆ = N⋆ ,

EK⋆ = MK⋆ K⋆ − NK⋆ = 0, Σ⋆ =

τ −1 M . 2 K⋆

23

B.3

Soft Bellman residual and Gaussian log-Sobolev inequality

Lemma B.1 (Gaussian KL formula). Let ν1 = N (m1 , Σ1 ),

ν2 = N (m2 , Σ2 ).

Then KL(ν1 ∥ν2 ) =

i 1h −1 −1 (m1 − m2 )⊤ Σ−1 2 (m1 − m2 ) + Tr(Σ2 Σ1 ) − m − log det(Σ2 Σ1 ) . 2

Proof. Using the Gaussian density formula, log

1 det Σ2 1 dν1 1 ⊤ −1 (u) = log + (u − m2 )⊤ Σ−1 2 (u − m2 ) − (u − m1 ) Σ1 (u − m1 ). dν2 2 det Σ1 2 2

Taking expectation under ν1 , and using h

i

Eν1 (u − m1 )⊤ Σ−1 1 (u − m1 ) = m and

h

i

−1 ⊤ −1 Eν1 (u − m2 )⊤ Σ−1 2 (u − m2 ) = Tr(Σ2 Σ1 ) + (m1 − m2 ) Σ2 (m1 − m2 ),

gives the result. Proof of Lemma 5.2. Gibbs policy and Bellman residual. By Lemma 3.1, we have VK,Σ (x) = x⊤ PK x + qK,Σ . For the fixed policy (K, Σ), the state-action function QK,Σ (x, u) := c(x, u) + γE[VK,Σ (Ax + Bu + w)] is quadratic in u. More precisely, expanding c(x, u) = x⊤ Qx + u⊤ Ru and the term γE[(Ax + Bu + w)⊤ PK (Ax + Bu + w)], and collecting the u-quadratic and u-linear coefficients, gives QK,Σ (x, u) = u⊤ MK u + 2u⊤ NK x + ψK,Σ (x), where ψK,Σ is independent of u, MK = R + γB ⊤ PK B, and NK = γB ⊤ PK A. It follows that  1 1 exp − QK,Σ (x, u) ∝ exp − u⊤ MK u + 2u⊤ NK x . τ τ 







Completing the square yields 



−1 pK (· | x) = N −MK NK x, ΞK ,

ΞK :=

τ −1 M . 2 K

For any density ρ, the Gibbs variational identity gives Z

(QK,Σ (x, u) + τ log ρ(u)) ρ(du) = (T∗ VK,Σ )(x) + τ KL(ρ ∥ pK (· | x)) .

Indeed, if ZK,Σ (x) :=

R

exp(−QK,Σ (x, u)/τ ) du, then (T∗ VK,Σ )(x) = −τ log ZK,Σ (x), 24

and the preceding identity follows from τ KL(ρ ∥ pK (· | x)) =

Z

(QK,Σ (x, u) + τ log ρ(u)) ρ(du) + τ log ZK,Σ (x).

Taking ρ = πK,Σ (· | x) and using the Bellman identity VK,Σ = TπK,Σ VK,Σ , we obtain

RK,Σ (x) := VK,Σ (x) − (T∗ VK,Σ )(x) = τ KL(πK,Σ (· | x) ∥ pK (· | x)) .

Applying the Gaussian KL formula in Lemma B.1 to πK,Σ (· | x) = N (−Kx, Σ)

−1 pK (· | x) = N (−MK NK x, ΞK ),

and

we get −1 −1 RK,Σ (x) = x⊤ (K − MK NK )⊤ MK (K − MK NK )x +

i τh −1 Tr(Ξ−1 K Σ) − m − log det(ΞK Σ) . 2

Since EK = MK K − NK , −1 −1 −1 ⊤ (K − MK NK )⊤ MK (K − MK NK ) = EK MK EK .

This gives the explicit formula (19). Gaussian log-Sobolev inequality. The Gibbs conditional action law −1 pK (· | x) = N (−MK NK x, ΞK )

has covariance

τ −1 M . 2 K A nondegenerate Gaussian measure GΞ with covariance Ξ satisfies ΞK =

I(ν∥GΞ ) ≥ 2λmin (Ξ−1 )KL(ν∥GΞ ), and the coefficient λmin (Ξ−1 ) is sharp; see, for example, Bakry et al. (2014). Hence, for the current Gibbs law, 2λmin (MK ) αK := λmin (Ξ−1 . K )= τ Equivalently, if the LSI is written as KL(ν∥pK ) ≤

CLSI (pK ) I(ν∥pK ), 2

CLSI (pK ) =

1 τ = . αK 2λmin (MK )

then its sharp constant is Since

MK = R + γB ⊤ PK B ⪰ R ⪰ λR I,

we also have the uniform lower bound αK ≥ α := Therefore

2λR . τ

I(ν∥pK (· | x)) ≥ 2αK KL(ν∥pK (· | x)) ≥ 2αKL(ν∥pK (· | x)).

Translations do not change either Gaussian coefficient, so the uniform estimate is independent of x and of the admissible gain K. 25

B.4

Performance-difference identity and gradients

Proof of Lemma 5.3. Let

∆K := K ′ − K.

Let (xt , ut )t≥0 be the trajectory generated by the policy (K ′ , Σ′ ). Since VK,Σ has quadratic growth and (K ′ , Σ′ ) is admissible, Lemma C.2 implies ′

′

γ T EK ,Σ [VK,Σ (xT )] → 0

as T → ∞.

Using C(K, Σ) = Ex0 ∼D [VK,Σ (x0 )], the usual telescoping argument gives ′

′

C(K ′ , Σ′ ) − C(K, Σ) = EK ,Σ

∞ X



γ t c(xt , ut ) + τ log πK ′ ,Σ′ (ut | xt )

t=0



+ γVK,Σ (xt+1 ) − VK,Σ (xt ) . We next compute the conditional one-step contribution. Conditioning on xt = x, let u ∼ N (−K ′ x, Σ′ ),

x+ := Ax + Bu + w.

Lemma 3.1 gives VK,Σ (x) = x⊤ PK x + qK,Σ ,

MK = R + γB ⊤ PK B,

NK = γB ⊤ PK A.

A direct computation then gives E c(x, u) + τ log πK ′ ,Σ′ (u | x) + γVK,Σ (x+ ) − VK,Σ (x) xt = x 







= x⊤ K ′⊤ MK K ′ − 2K ′⊤ NK − K ⊤ MK K + 2K ⊤ NK x τ τ + Tr(MK Σ′ ) − log det Σ′ − Tr(MK Σ) + log det Σ. 2 2 As a quadratic form in x, the matrix coefficient agrees with ∆K ⊤ MK ∆K + 2∆K ⊤ EK ,

EK = MK K − NK .

Indeed, the two matrix representatives can differ only by a skew-symmetric term, which vanishes inside x⊤ (·)x. The constant part is (1 − γ) fK (Σ) − fK (Σ′ ) .

Substituting the one-step expansion into the telescoping identity and summing the quadratic term along the trajectory of (K ′ , Σ′ ), we obtain ′

EK ,Σ

′

∞ X





⊤ ⊤ γ t x⊤ t ∆K MK ∆K + 2∆K EK xt

t=0





= Tr SK ′ ,Σ′ ∆K ⊤ MK ∆K + 2∆K ⊤ EK



.

The constant term contributes ∞ X

γ t (1 − γ) fK (Σ) − fK (Σ′ ) = fK (Σ) − fK (Σ′ ).

t=0

26

Therefore 



C(K ′ , Σ′ ) − C(K, Σ) = Tr SK ′ ,Σ′ ∆K ⊤ MK ∆K + 2∆K ⊤ EK



+ fK (Σ) − fK (Σ′ ).

This proves (21). It remains to identify the gradients. We differentiate (21) at (K ′ , Σ′ ) = (K, Σ). First fix Σ′ = Σ and set

K ′ = K + εH.

Then ∆K = εH. Since the bracketed term in (21) vanishes at ε = 0, the first variation of SK+εH,Σ does not contribute. Thus 



C(K + εH, Σ) − C(K, Σ) = 2ε Tr SK,Σ H ⊤ EK + o(ε). With respect to the Frobenius inner product, this gives ∇K C(K, Σ) = 2EK SK,Σ . Next fix K ′ = K and set

Σ′ = Σ + εU,

U = U ⊤.

The quadratic term in (21) vanishes identically, so C(K, Σ + εU ) − C(K, Σ) = fK (Σ) − fK (Σ + εU ). Using fK (Σ) =

τ 1 log det Σ − Tr(MK Σ), 2(1 − γ) 1−γ

we obtain C(K, Σ + εU ) − C(K, Σ) = Hence ∇Σ C(K, Σ) =

ε Tr 1−γ



1 1−γ

τ MK − Σ−1 U 2 



+ o(ε).

τ MK − Σ−1 . 2





This proves (22).

C

Closure on Linear-Gaussian Policies and Value Dissipation

This appendix verifies the policy first variation along compactly supported action transports at a linear-Gaussian policy and proves the closure and dissipation identities.

27

C.1

Policy-space metric and Wasserstein gradient

Lemma C.1 (First variation at a linear-Gaussian policy). Fix (K, Σ) ∈ Aadm and write π = πK,Σ . Let φ ∈ Cc∞ (Rn × Rm ), set v = ∇u φ, and define Tε,x (u) := u + εv(x, u),

π ε (· | x) := (Tε,x )# π(· | x).

There exists ε0 > 0 such that, for every |ε| < ε0 , the map Tε,x is a smooth bijection of Rm for every x, the policy π ε has a positive smooth density and belongs to Πadm , and the map ε 7→ C(π ε ) is differentiable at zero. Its derivative is
1 d = C(π ε ) dπD (dx) Qπ (x, u) + τ log π(u | x) ξv (u | x) du dε 1 − γ Rn Rm ε=0 Z Z
1 = dπD (dx) ∇u Qπ (x, u) + τ log π(u | x) , v(x, u) π(du | x), 1 − γ Rn Rm Z

Z

where ξv (u | x) :=

(30)

d ε π (u | x) = −∇u · π(u | x)v(x, u) . dε ε=0

In particular, the differentiation under the discounted infinite sum, the Fubini interchange, and the integration by parts in (30) are valid under the standing assumptions. Proof. The transport path. Because v and ∇u v are bounded and vanish outside a compact subset of Rn × Rm , choose ε0 > 0 so that |ε|∥∇u v∥∞ < For every fixed x,

1 2

for |ε| < ε0 .

∥Tε,x (u) − Tε,x (a)∥ ≥ 1 − |ε|∥∇u v∥∞ ∥u − a∥.

Thus Tε,x is injective and has an invertible Jacobian. It agrees with the identity outside a compact set in the action variable, and is therefore proper. Its image is open by the inverse function theorem and closed by properness. Since Rm is connected, the image is all of Rm , and the inverse is smooth. The change-of-variables formula gives π ε (Tε,x (u) | x) det Im + ε∇u v(x, u) = π(u | x).

(31)

Hence π ε has a positive smooth density. Differentiating at zero yields ξv = −∇u · (πv).

(32)

Writing sv := ξv /π, sv (x, u) = −∇u · v(x, u) − ⟨v(x, u), ∇u log π(u | x)⟩ . Since ∇u log π(u | x) = −Σ−1 (u + Kx) and v has compact support in (x, u), the score sv is bounded R and compactly supported. Moreover, Rm ξv (u | x) du = 0 for every x. Uniform discounted moment bound. Set F = A − BK. Admissibility permits a number b ∈ max{1, ρ(F )}, γ −1/2 28

and a constant cF < ∞ such that ∥F j ∥ ≤ cF bj for every j ≥ 0. A draw from π ε (· | x) can be represented as u = −Kx + ξ + εv(x, −Kx + ξ), ξ ∼ N (0, Σ). The state recursion is therefore xt+1 = F xt + Bξt + wt + εBv(xt , −Kxt + ξt ). The last term is uniformly bounded, while Bξt + wt has a finite second moment. Iterating the recursion and applying Minkowski’s inequality shows that, for every deterministic initial state x, ε

Eπx ∥xt ∥2

1/2

≤ Cbt (1 + ∥x∥),

uniformly for sufficiently small ε. The action representation gives the same type of bound for ut . Hence, for some C < ∞ and q ∈ (0, 1), γ t Eπx

ε

h

i

1 + ∥xt ∥2 + ∥ut ∥2 ≤ Cq t (1 + ∥x∥2 ),

t ≥ 0,

(33)

uniformly for small ε. One may take any q ∈ (γb2 , 1) after increasing C. This proves the discounted transversality condition and shows that π ε ∈ Πadm . After integration over x0 ∼ D, the same estimate holds without the factor 1 + ∥x∥2 on the right. The change-of-variables identity also gives, uniformly for small ε, |c(x, u) + τ log π ε (u | x)| ≤ C 1 + ∥x∥2 + ∥u∥2 .

(34)

Indeed, the Gaussian log density is quadratic, the Jacobian determinant is bounded above and away from zero, and the transport differs from the identity only on a fixed compact set. Differentiation of the discounted series. On a trajectory generated under the reference policy π, define t Y π ε (u | x) ε rε (xj , uj ). , Lt := rε (x, u) := π(u | x) j=0 The initial-state law and transition kernel are the same under the two policies, so C(π ε ) =

∞ X

γ t Eπ c(xt , ut ) + τ log π ε (ut | xt ) Lεt . 



(35)

t=0

By (31) and compact support of v, there is c < ∞ such that, uniformly in (x, u), e−c|ε| ≤ rε (x, u) ≤ ec|ε| ,

|∂ε log rε (x, u)| ≤ c,

for sufficiently small ε. The same argument bounds |∂ε log π ε | on the set where it is nonzero. By (34), the derivative of the t-th integrand in (35) is bounded in absolute value by C(t + 1)ec|ε|(t+1) 1 + ∥xt ∥2 + ∥ut ∥2 .

Choose ε0 smaller if needed so that qecε0 < 1, where q is from (33). The discounted expectations of these bounds form a summable series. Dominated convergence therefore justifies differentiation under the expectation and the infinite sum. At ε = 0, ∂ε rε (x, u)|0 = ∂ε log π ε (u | x)|0 = sv (x, u). 29

Consequently, 



∞ t X
X d γ t Eπ  c(xt , ut ) + τ log π(ut | xt ) sv (xj , uj ) C(π ε ) = dε 0 t=0 j=0 ∞ X

+τ

γ t Eπ [sv (xt , ut )].

(36)

t=0

The second sum vanishes because Eπ [sv (xt , ut ) | xt ] =

Z Rm

ξv (u | xt ) du = 0.

Fubini interchange. The score sv is bounded. By (33), ∞ X t X

γ t Eπ [|sv (xj , uj )| |c(xt , ut ) + τ log π(ut | xt )|] ≤ C

t=0 j=0

∞ X

(t + 1)q t < ∞.

t=0

Fubini’s theorem therefore permits exchange of the two sums in (36). Writing t = j + r and conditioning on (xj , uj ), the conditional expected cost from time j onward is Qπ (xj , uj ) + τ log π(uj | xj ). Hence ∞ X 
 d ε γ j Eπ sv (xj , uj ) Qπ (xj , uj ) + τ log π(uj | xj ) C(π ) = dε 0 j=0

1 = 1−γ

Z

Z

Qπ (x, u) + τ log π(u | x) ξv (u | x) du. dπD (dx) m n R R

This is the first equality in (30). Integration by parts and completion of the tangent space. For a linear-Gaussian policy, Qπ (x, u) + τ log π(u | x) is a smooth quadratic function of (x, u). Since v has compact support in the action variable, (32) gives, with no boundary term, Z Rm

Qπ (x, u) + τ log π(u | x) ξv (u | x) du

=

Z Rm

∇u Qπ (x, u) + τ log π(u | x) , v(x, u) π(du | x).

The right-hand side is absolutely integrable because the action gradient is affine and the discounted state and action second moments are finite. This proves the second equality in (30). Finally, let
gπ (x, u) := ∇u Qπ (x, u) + τ log π(u | x) . It is the action gradient of a quadratic function hπ . Let χR (x) and ηL (u) be smooth cutoffs that equal one on balls of radii R and L, and define φR,L (x, u) := χR (x)ηL (u)hπ (x, u). Choose the action cutoff so that ∥∇u ηL ∥∞ ≤ C/L. For fixed R, on the annulus where ∇u ηL ̸= 0, the quadratic growth of hπ gives |hπ (x, u)∇u ηL (u)| ≤ CR (1 + ∥u∥). 30

The remaining cutoff error is bounded by a constant times 1 + ∥x∥ + ∥u∥. First letting L → ∞ for fixed R, and then letting R → ∞, the finite second moments and dominated convergence give ∇u φR,L −→ gπ in the metric norm. Thus the affine WPG velocity lies in the completed tangent space, and compactly supported test velocities are sufficient to identify the metric gradient. Remark C.1 (Linear-Gaussian parameter directions). Let s 7→ (Ks , Σs ) be a C 1 linear-Gaussian parameter curve through (K, Σ), with derivative (K̇, Σ̇). If LΣ̇ is the unique symmetric solution of LΣ̇ Σ + ΣLΣ̇ = Σ̇, then the action-gradient velocity vK̇,Σ̇ (x, u) = −K̇x + LΣ̇ (u + Kx) satisfies

d πKs ,Σs (u | x) = −∇u · πK,Σ (u | x)vK̇,Σ̇ (x, u) . ds s=0

The cutoff argument above places this affine velocity in the completed tangent space, so the policy-space first-variation formula applies directly to every linear-Gaussian parameter direction.

C.2

Proof of closure on the linear-Gaussian class

Proof of Proposition 4.2. Fix (K, Σ) ∈ Aadm , and abbreviate π := πK,Σ ,

M := MK ,

For fixed x, set

N := NK ,

E := EK = M K − N.

z := u + Kx.

Then π(· | x) = N (−Kx, Σ) and ∇u log π(u | x) = −Σ−1 z. By (12),

∇u Qπ (x, u) = 2M u + 2N x.

Since u = z − Kx, the WPG transport field becomes ∇u Qπ (x, u) + τ log π(u | x) = 2M (z − Kx) + 2N x − τ Σ−1 z

= 2M − τ Σ−1 z − 2Ex.

Let

H := 2M − τ Σ−1 .

Using ∇u π = −πΣ−1 z, the right-hand side of WPGF is ∇u · π(u | x) Hz − 2Ex 



h

i

= π(u | x) Tr(H) − z ⊤ HΣ−1 z + 2z ⊤ Σ−1 Ex . 31

(37)

We next compute a general tangent vector to the linear-Gaussian family. Differentiating the Gaussian density with respect to (K, Σ) in a direction (K̇, Σ̇) gives 

DπK,Σ [K̇, Σ̇](u | x) = π(u | x) − z ⊤ Σ−1 K̇x −

1 Tr(Σ−1 Σ̇) 2

1 + z ⊤ Σ−1 Σ̇Σ−1 z . 2 

Choose

K̇ = −2E,

The linear term in (38) is then

(38)

Σ̇ = −2M Σ − 2ΣM + 2τ I.

−z ⊤ Σ−1 K̇x = 2z ⊤ Σ−1 Ex.

For the constant term, cyclicity of the trace gives 1 − Tr(Σ−1 Σ̇) = Tr(2M − τ Σ−1 ) = Tr(H). 2 Finally, since M and Σ are symmetric,
1 ⊤ −1 −1 z Σ Σ̇Σ z = −z ⊤ 2M − τ Σ−1 Σ−1 z 2 = −z ⊤ HΣ−1 z.

Substitution into (38) yields exactly (37). This proves the tangent identity (13) and the parameter velocity (14). The velocity is unique because the derivative of the conditional mean is −K̇x and the derivative of the conditional covariance is Σ̇; these first two moments uniquely determine the parameter velocity for every x. Therefore, if (Kt , Σt ) satisfies (15), the chain rule and (13) show that πt = πKt ,Σt satisfies WPGF on every interval on which the parameter path is admissible. This proves exact closure on the linear-Gaussian policy class.

C.3

The closed-loop Markov resolvent on V2

Lemma C.2 (Value resolvent on V2 ). Let (K, Σ) be admissible. Then PK,Σ : V2 → V2 , and there exist cK,Σ < ∞ and q̄K,Σ ∈ (0, 1) such that j j ∥γ j PK,Σ v∥V2 ≤ cK,Σ q̄K,Σ ∥v∥V2 ,

Consequently, (I − γPK,Σ )−1 =

∞ X

j ≥ 0.

j γ j PK,Σ

j=0

as a bounded operator on V2 , and the inverse is positivity preserving. √ Proof. Write F = FK , Ω = ΩΣ , and P = PK,Σ . Since ρ( γF ) < 1, there exist cF < ∞ and qF ∈ (0, 1) such that γ j ∥F j ∥2 ≤ cF qFj , j ≥ 0. For the closed-loop process, xj = F x + j

j−1 X

F j−1−ℓ ηℓ ,

ℓ=0

32

E[ηℓ ηℓ⊤ ] = Ω,

and hence E[∥xj ∥2 | x0 = x] = ∥F j x∥2 +

j−1 X





Tr F r Ω(F r )⊤ .

r=0

Choose

q̄K,Σ ∈ max{γ, qF }, 1 .

Using Tr(F r Ω(F r )⊤ ) ≤ Tr(Ω)∥F r ∥2 , we obtain j γ j E[∥xj ∥2 | x0 = x] ≤ cK,Σ q̄K,Σ (1 + ∥x∥2 ).

Therefore, for v ∈ V2 ,

γ j |P j v(x)| ≤ ∥v∥V2 γ j 1 + E[∥xj ∥2 | x0 = x] ,

which gives

j ∥γ j P j v∥V2 ≤ cK,Σ q̄K,Σ ∥v∥V2 .

Thus the Neumann series converges absolutely in operator norm. Its partial sums satisfy (I − γP)

T X

γ j P j = I − γ T +1 P T +1 ,

j=0

and the remainder converges to zero in operator norm. Hence the limit is (I − γP)−1 . Since every P j is positivity preserving, so is the inverse.

C.4

Dissipation identity for the value function

Proof of Lemma 5.1. Let Vt := VKt ,Σt and Pt := Pπt = PKt ,Σt , and define pt (· | x) := N (−Mt−1 Nt x, Ξt ),

Ξt :=

τ −1 M . 2 t

Equivalently, pt (· | x) is the Gibbs density associated with the function u 7→ Qπt (x, u): 1 pt (u | x) = Zt (x)−1 exp − Qπt (x, u) . τ 



By Lemma 3.1, Vt (x) = x⊤ PKt x + qKt ,Σt , so all quantities below belong to V2 . For fixed t and x, introduce the action functional Jt (ν; x) :=

Z 



Qπt (x, u) + τ log ν(u) ν(u) du.

Then the soft Bellman identity reads Vt (x) = Jt (πt (· | x); x). Differentiating this identity in t gives two contributions: V̇t (x) = ∂t Jt (πt (· | x); x) + Dν Jt (πt (· | x); x)[∂t πt (· | x)]. The first contribution comes only from the variation of Qπt . Since ∂t Qπt (x, u) = γE[V̇t (Ax + Bu + w) | x, u], 33

we have

∂t Jt (πt (· | x); x) =

Z

∂t Qπt (x, u) πt (du | x) = γPt V̇t (x).

It remains to compute the action derivative. Since pt (u | x) = Zt (x)−1 exp(−Qπt (x, u)/τ ), Qπt (x, u) + τ log ν(u) = τ log

ν(u) − τ log Zt (x). pt (u | x)

The last term is constant in u, and therefore disappears when paired with a signed perturbation of R total mass zero. Hence, for any admissible perturbation δν with δν(u) du = 0, Dν Jt (πt (· | x); x)[δν] = τ

Z

log

πt (u | x) δν(u) du. pt (u | x)

Moreover, by (11), the WPGF can be written in relative-entropy gradient-flow form as πt ∂t πt = τ ∇u · πt ∇u log . pt 



Taking δν = ∂t πt (· | x) and integrating by parts gives Dν Jt (πt (· | x); x)[∂t πt (· | x)] = −τ 2 Equivalently,

Z

∇u log

πt (u | x) 2 πt (du | x). pt (u | x)

Dν Jt (πt (· | x); x)[∂t πt (· | x)] = −τ 2 I(πt (· | x)∥pt (· | x)).

Define

gt (x) := τ 2 I(πt (· | x)∥pt (· | x)).

Combining the two contributions yields V̇t (x) = γPt V̇t (x) − gt (x), or equivalently

(I − γPt )V̇t = −gt .

By Lemma C.2, the inverse (I − γPt )−1 is well defined on V2 and admits the positive Neumannseries representation (I − γPt )−1 =

∞ X

γ k Ptk .

k=0

Therefore

V̇t = −(I − γPt )−1 gt .

Since gt ≥ 0 and the resolvent is positivity preserving, we obtain V̇t ≤ −gt ≤ 0. This proves (17) and (18).

34

C.5

Objective dissipation along the linear-Gaussian parameter flow

Lemma C.3 (Objective dissipation). Along every admissible solution of the parameter ODE (15), define τ . Gt := Mt − Σ−1 2 t Then Σ̇t = −2(Gt Σt + Σt Gt ), and

    4 d C(Kt , Σt ) = −4 Tr SKt ,Σt Et⊤ Et − Tr Σt G2t ≤ 0. dt 1−γ

Proof. The covariance equation in (15) can be rewritten as Σ̇t = −2Mt Σt − 2Σt Mt + 2τ I = −2(Gt Σt + Σt Gt ). Using the gradient identities (22) and K̇t = −2Et , we obtain 1 d C(Kt , Σt ) = ⟨2Et SKt ,Σt , −2Et ⟩F + ⟨Gt , −2(Gt Σt + Σt Gt )⟩F dt 1−γ     4 = −4 Tr SKt ,Σt Et⊤ Et − Tr Σt G2t . 1−γ Both terms on the right-hand side are nonpositive because SKt ,Σt ≻ 0, Σt ≻ 0, and Gt is symmetric.

D

Stability, Compactness, and Global Well-Posedness

This appendix proves the stability, compactness, and continuation results used for global wellposedness.

D.1

Closed-loop detectability

Lemma D.1 (Detectability passes to the closed loop). Under Assumption 3.1, for every feedback matrix K, the scaled pair 1/2 √ (LK , γFK ) is detectable. Proof. Suppose, to the contrary, that the pair 1/2 √ (LK , γFK )

is not detectable. Then there exist v ̸= 0 and λ ∈ C such that FK v = λv, Since

√ | γλ| ≥ 1,

LK = Q + K ⊤ RK ⪰ 0 1/2

and R ≻ 0, the identity LK v = 0 implies Q1/2 v = 0,

Kv = 0. 35

1/2

LK v = 0.

Consequently,

Av = FK v + BKv = λv.

Thus v is an unobservable eigenvector of the pair √ (Q1/2 , γA) √ 1/2 √ with | γλ| ≥ 1, contradicting Assumption 3.1. Therefore (LK , γFK ) is detectable.

D.2

Finite cost and discounted stability

Lemma D.2 (Finite cost is equivalent to discounted stability). Under Assumption 3.1, for every K and every Σ ≻ 0, √ C(K, Σ) < ∞ if and only if ρ( γFK ) < 1. Proof. First assume that

√ ρ( γFK ) < 1.

Then the Lyapunov series defining PK converges. Moreover, the discounted state correlation SK,Σ =

∞ X

γ t E[xt x⊤ t ]

t=0

is finite. Hence C(K, Σ) =

∞ X







γ t Tr LK E[xt x⊤ t ] + Tr(RΣ) −

t=0

τ m + log (2π)m det Σ < ∞. 2 

Conversely, suppose that C(K, Σ) < ∞. Define Xt := E[xt x⊤ t ]. Since

t xt = FK x0 + zero-mean noise independent of x0 ,

we have

t t ⊤ Xt ⪰ FK Γ0 (FK ) .

For N ≥ 0, define the finite-horizon observability Gramian PN :=

N X

t ⊤ t γ t (FK ) LK FK .

t=0

Let

cΣ := Tr(RΣ) −

Then C(K, Σ) =

τ m + log (2π)m det Σ . 2

∞ X

γ t Tr(LK Xt ) +

t=0

rΣ . 1−γ

Therefore, for every N , Tr(Γ0 PN ) ≤

N X

γ t Tr(LK Xt ) ≤ C(K, Σ) −

t=0

36

rΣ < ∞. 1−γ

Since Γ0 ⪰ µI, it follows that µ Tr(PN ) ≤ Tr(Γ0 PN ) ≤ C(K, Σ) −

cΣ . 1−γ

The sequence (PN )N ≥0 is monotone increasing in the positive semidefinite order and has uniformly bounded trace. Hence it converges to some P ⪰ 0. Passing to the limit in ⊤ PN = LK + γFK PN −1 FK

gives

⊤ P = LK + γFK P FK .

We now show that the closed loop must be discounted-stable. Suppose instead that √ ρ( γFK ) ≥ 1. Then there exist v ̸= 0 and λ ∈ C such that √ | γλ| ≥ 1.

FK v = λv, Testing the Lyapunov identity against v gives

(1 − γ|λ|2 )v ∗ P v = v ∗ LK v. The left-hand side is nonpositive, while the right-hand side is nonnegative. Therefore v ∗ LK v = 0, and hence

1/2

LK v = 0.

This contradicts Lemma D.1. Thus

√ ρ( γFK ) < 1.

Remark D.1 (The detectability hypothesis is necessary). The detectability condition cannot be removed. For instance, if Q = 0, then √ (0, γA) √ is detectable only when γA is already stable. Consider the scalar example A = 2,

B = 1,

Q = 0,

1 γ= , 2

R = 1,

W = 0,

Γ0 = 1,

τ = 1.

The unrestricted finite-cost minimizer is 1 , (K, Σ) = 0, 2 



which is not discounted-stable. By contrast, the stationarity equations restricted to the admissible class select a different admissible point, which is strictly suboptimal for the unrestricted problem. Thus, without Assumption 3.1(ii), finite cost and discounted stability need not coincide. 37

D.3

Compactness of finite-cost sublevel sets

Proof of Proposition 5.1. The finite-cost characterization is Lemma D.2. It remains to prove compactness of the sublevel sets. Boundedness of sublevels. From the Lyapunov equation (4), ⊤ PK = Q + K ⊤ RK + γFK PK FK ⪰ K ⊤ RK.

Hence, using (5), Γ0 ⪰ µI, and R ⪰ λR I,   τ τm 1 ⊤ λR Tr(Σ) − log det Σ − (1 + log(2π)) C(K, Σ) ≥ Tr(Γ0 K RK) + 1−γ 2 2(1 − γ)   τ τm 1 λR Tr(Σ) − log det Σ − (1 + log(2π)). ≥ µλR ∥K∥2F + 1−γ 2 2(1 − γ) The right-hand side tends to +∞ if ∥K∥F → ∞, if λmax (Σ) → ∞, or if λmin (Σ) ↓ 0. Therefore every sublevel set Sc := {(K, Σ) ∈ Aadm : C(K, Σ) ≤ c} is bounded, and its Σ-components remain in a compact subset of Sm ++ . Closedness of sublevels. Let (Kj , Σj ) ∈ Sc be a convergent sequence with Σj → Σ ≻ 0.

Kj → K,

By the boundedness just proved, the matrices Σj remain in a compact subset of Sm ++ . Moreover, Kj is bounded. Hence the constants qKj ,Σj in the value functions are uniformly bounded below: there exists a constant C0 such that qKj ,Σj ≥ C0 for all j. Since C(Kj , Σj ) ≤ c, we get Tr(Γ0 PKj ) = C(Kj , Σj ) − qKj ,Σj ≤ c − C0 . Using Γ0 ⪰ µI, this implies

µ Tr(PKj ) ≤ Tr(Γ0 PKj ) ≤ c − C0 .

Thus (PKj ) is bounded in the cone of positive semidefinite matrices. Passing to a subsequence if necessary, we may assume PKj → P ⪰ 0. Taking limits in the Lyapunov equation ⊤ PKj = Q + Kj⊤ RKj + γFK P F j Kj Kj

gives

⊤ P = Q + K ⊤ RK + γFK P FK .

By Lemma D.1 and the eigenvector argument used in Lemma D.2, this identity with P ⪰ 0 implies √ ρ( γFK ) < 1. Hence (K, Σ) is admissible. On the admissible set, K 7→ PK is continuous, and therefore C is continuous. Consequently, C(K, Σ) = lim C(Kj , Σj ) ≤ c. j→∞

Thus Sc is closed. It is also bounded, its covariance matrices are uniformly positive definite, and every limit point satisfies the discounted-stability condition. Therefore Sc is a compact subset of Aadm . In particular, C attains its minimum on each nonempty finite sublevel set. By Assumption 3.1(iii) and Lemma D.2, such a sublevel set is nonempty, and hence a minimizer exists. 38

D.4

Global well-posedness of the closed parameter flow

Proof of Lemma 5.4. By Assumption 3.1(iii) and Lemma D.2, the finite-cost initialization (K0 , Σ0 ) is admissible. On the admissible set, the map K 7→ PK is smooth. Indeed, the Lyapunov equation can be vectorized as  −1 ⊤ ⊤ vec(PK ) = I − γFK ⊗ FK vec(LK ), √ and the inverse exists whenever ρ( γFK ) < 1. Hence the vector field in (15) is smooth on the admissible set, and the standard ODE theorem gives a unique local solution. Along this local solution, Lemma C.3 gives d C(Kt , Σt ) ≤ 0. dt Therefore the trajectory remains in the finite sublevel set SC(K0 ,Σ0 ) = {(K, Σ) : C(K, Σ) ≤ C(K0 , Σ0 )}. By Proposition 5.1, this sublevel set is a compact subset of the admissible set. Since the trajectory remains in this compact set, it stays bounded, the covariance remains positive definite, and discounted stability is preserved. The standard continuation theorem for ODEs therefore extends the solution uniquely to all t ≥ 0. Since the trajectory remains in the same finite sublevel set, every point on the trajectory is admissible and has finite cost.

E

Proof of the Main Theorem

We use the Bellman resolvent identity and the LQ performance-difference identity to compare objective dissipation with the optimality gap. Proof of Theorem 5.1. By Lemma 5.4, the trajectory is globally defined, remains admissible, and has finite cost for all t ≥ 0. For brevity, write Ct := C(Kt , Σt ), and Let

Mt := MKt ,

C⋆ := C(K⋆ , Σ⋆ ), Nt := NKt ,

St := SKt ,Σt , Et := EKt ,



pt (· | x) := N −Mt−1 Nt x, Ξt

Ξt :=

S⋆ := SK⋆ ,Σ⋆ , τ −1 M . 2 t



be the Gibbs policy associated with the current value function Vt := VKt ,Σt . Step 1: Gaussian LSI and objective decrease. Let Pt := Pπt denote the closed-loop transition operator induced by πt = πKt ,Σt , and define gt (x) := τ 2 I πt (· | x)∥pt (· | x) .

By the Bellman resolvent identity (17), (I − γPt )V̇t = −gt .

39

Therefore,

h i
d Ct = Ex0 ∼D (I − γPt )−1 gt (x0 ) . dt The Gaussian log-Sobolev inequality (20) gives

−

I(ν∥pt (· | x)) ≥ 2αt KL ν∥pt (· | x) ≥

4λR KL ν∥pt (· | x) , τ

αt =

2λmin (Mt ) . τ

Taking ν = πt (· | x) and using the Bellman residual identity (19), Rt (x) := Vt (x) − (T∗ Vt )(x) = τ KL πt (· | x)∥pt (· | x) ,

we obtain

gt (x) ≥ 4λR Rt (x).

The temperature cancels in this inequality. This step introduces no factor of the form exp(−c/τ ). By Lemma C.2, the resolvent (I − γPt )−1 =

∞ X

γ j Ptj

j=0

is positivity preserving. It follows that − where

d Ct ≥ 4λR Rt , dt

Rt := Ex0 ∼D

h

(39) i

(I − γPt )−1 Rt (x0 ) .

Step 2: LQ performance comparison. The Gaussian KL formula (19) for the Bellman residual is Rt (x) = x⊤ Et⊤ Mt−1 Et x + bt , where

i τh −1 Tr(Ξ−1 t Σt ) − m − log det(Ξt Σt ) . 2 Consequently, by the definition (3) of the discounted state-correlation matrix,

bt :=





Rt = Tr St Et⊤ Mt−1 Et + Recall that fKt (Σ) =

bt . 1−γ

τ 1 log det Σ − Tr(Mt Σ). 2(1 − γ) 1−γ

Since Ξt = τ2 Mt−1 , direct substitution gives fKt (Ξt ) − fKt (Σt ) = Define We thus have

bt . 1−γ

Bt := fKt (Ξt ) − fKt (Σt ). 



Rt = Tr St Et⊤ Mt−1 Et + Bt . 40

(40)

We next compare this expression with the global objective gap. Applying the performancedifference identity (21) with (K ′ , Σ′ ) = (K⋆ , Σ⋆ ),

(K, Σ) = (Kt , Σt ), and setting yields

∆K⋆ := K⋆ − Kt , 



Ct − C⋆ = − Tr S⋆ ∆K⋆⊤ Mt ∆K⋆ + 2∆K⋆⊤ Et



+ fKt (Σ⋆ ) − fKt (Σt ).

The gain term satisfies 



Tr S⋆ ∆K⋆⊤ Mt ∆K⋆ + 2∆K⋆⊤ Et 1/2

= Mt

≥ − Tr



1/2

∆K⋆ S⋆

−1/2

+ Mt

S⋆ Et⊤ Mt−1 Et



 1/2 2

Et S⋆

F



− Tr S⋆ Et⊤ Mt−1 Et



.

Moreover, fKt is strictly concave on Sm ++ , and ∇Σ fKt (Σ) = Its unique critical point is therefore Σ=

1 1−γ



τ −1 Σ − Mt . 2 

τ −1 M = Ξt , 2 t

which is its unique global maximizer. Hence fKt (Σ⋆ ) ≤ fKt (Ξt ), It follows that

Ct − C⋆ ≤ At + Bt ,

Let Since

Bt ≥ 0. 



At := Tr S⋆ Et⊤ Mt−1 Et .

(41)

Ht := Et⊤ Mt−1 Et ⪰ 0. St ⪰ Γ0 ⪰ µI

we have St ⪰ Therefore, Tr(St Ht ) ≥

and

S⋆ ⪯ ∥S⋆ ∥ I,

µ S⋆ . ∥S⋆ ∥

µ µ Tr(S⋆ Ht ) = At . ∥S⋆ ∥ ∥S⋆ ∥

Also, ∥S⋆ ∥ ≥ µ, and hence, using Bt ≥ 0, Bt ≥

µ Bt . ∥S⋆ ∥

Combining these inequalities with (40) and (41) gives Rt ≥

µ µ (At + Bt ) ≥ (Ct − C⋆ ). ∥S⋆ ∥ ∥S⋆ ∥ 41

(42)

Step 3: Exponential rate. Combining (39) and (42), we obtain −

4µλR d Ct ≥ (Ct − C⋆ ). dt ∥S⋆ ∥

Defining λτ :=

4µλR , ∥S⋆ ∥

we have

d (Ct − C⋆ ) ≤ −λτ (Ct − C⋆ ). dt This proves (23). Since Ct ≥ C⋆ by Proposition 3.1, Grönwall’s inequality yields 0 ≤ Ct − C⋆ ≤ e−λτ t (C0 − C⋆ ). This proves (24).

42

References Brian D. O. Anderson and John B. Moore. Optimal Control: Linear Quadratic Methods. Dover Publications, 2007. Michael Arbel, Arthur Gretton, Wuchen Li, and Guido Montufar. Kernelized wasserstein natural gradient. In International Conference on Learning Representations, 2020. Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffusion Operators, volume 348 of Grundlehren der mathematischen Wissenschaften. Springer, 2014. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 1467–1476. PMLR, 2018. Michael Giegrich, Christoph Reisinger, and Yufei Zhang. Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems. SIAM Journal on Control and Optimization, 62(2):1060–1092, 2024. Xin Guo, Xinyu Li, and Renyuan Xu. Fast policy learning for linear-quadratic control with entropy regularization. SIAM Journal on Control and Optimization, 64(1):124–151, 2026. Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods for the noisy linear quadratic regulator over a finite horizon. SIAM Journal on Control and Optimization, 59(5):3359–3391, 2021. Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter L. Bartlett, and Martin J. Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 89 of Proceedings of Machine Learning Research, pages 2916–2925. PMLR, 2019. Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R. Jovanović. Convergence and sample complexity of gradient methods for the model-free linear–quadratic regulator problem. IEEE Transactions on Automatic Control, 67(5):2435–2450, 2022. Ted Moskovitz, Michael Arbel, Ferenc Huszar, and Arthur Gretton. Efficient wasserstein natural gradients for reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. David Pfau, Ian Davies, Diana L. Borsa, João Guilherme Madeira Araújo, Brendan Daniel Tracey, and Hado Van Hasselt. Wasserstein policy optimization. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49128–49149. PMLR, 2025. David Šiška and Yufei Zhang. A note on convergence of wasserstein policy optimization. arXiv preprint arXiv:2605.22622, 2026. Jun Song, Niao He, Lijun Ding, and Chaoyue Zhao. Provably convergent policy optimization via metric-aware trust region methods. Transactions on Machine Learning Research, 2023. 43

Antonio Terpin, Nicolas Lanzetti, Batuhan Yardim, Florian Dörfler, and Giorgia Ramponi. Trust region policy optimization with optimal transport discrepancies: Duality and algorithm for continuous actions. In Advances in Neural Information Processing Systems, volume 35, pages 19786–19797. Curran Associates, Inc., 2022. Ruiyi Zhang, Changyou Chen, Chunyuan Li, and Lawrence Carin. Policy optimization as wasserstein gradient flows. In International Conference on Machine Learning, pages 5737–5746. PMLR, 2018. Zhaoyu Zhu, Rui Gao, and Shuang Li. Global convergence of wasserstein policy gradient for entropy-regularized reinforcement learning. arXiv preprint arXiv:2605.26078, 2026a. Zhaoyu Zhu, Shuhan Zhang, Rui Gao, and Shuang Li. Wasserstein proximal policy gradient. arXiv preprint arXiv:2603.02576, 2026b.

44

Related documents

• Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning#224500
• Hessian Schatten-Norm Regularization for Linear Inverse Problems#235364
• Wasserstein Policy Learning for Distributional Outcomes#287149
• Wasserstein Stability and Free Boundaries in Measure-Parameterized Bilevel Obstacle Problems#985807
• CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents#343462