Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions Tomasz R. Bielecki∗

Thibaut Mastrolia†

Haoze Yan‡

arXiv:2608.19151v1 [cs.LG] 19 Aug 2026

August 20, 2026

Abstract We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.

1

Introduction

Cybersecurity has become a major operational concern for large and highly interconnected organizations, where a single security incident can rapidly disrupt essential services and expose sensitive information. A recent illustration is the May 2026 cybersecurity incident affecting Canvas, a cloud-based learning management system used by thousands of schools and universities. The breach led for example UC Berkeley to temporarily restrict access to the platform and potentially exposed information including names, email addresses, student identification numbers, and messages exchanged through Canvas, highlighting the vulnerability of organizations to attacks propagated through third-party digital infrastructure. Such events also illustrate an important feature of cyber risk: attacks rarely occur as isolated and independent events, but may instead arrive in clusters, with previous incidents increasing the likelihood of subsequent malicious activity. Hawkes processes provide a natural probabilistic framework for representing this selfexciting behavior [4]. At the same time, defenders must dynamically decide how to allocate limited security resources as the threat environment evolves. This motivates the combination of Hawkes-process models of cyber-event arrivals with reinforcement learning, which provides a flexible framework for learning adaptive defense strategies in an uncertain and dynamically changing environment. The objective of this study is twofold: study the controlled system of a Hawkes-driven system for general self-excitation processes and develop reinforcement techniques in continuous time to minimize costs related to cascade incidents in an unknown environment. ∗

Hinsdale, Illinois, USA, tuslecki@gmail.com UC Berkeley, Department of Industrial Engineering and Operations Research, USA, mastrolia@berkeley.edu ‡ UC Berkeley, Department of Industrial Engineering and Operations Research, USA, haoze.yan@berkeley.edu †

1

1.1

Hawkes processes: theory, applications, and control

In his pioneering work [28], Alan G. Hawkes introduced the theory of self-exciting point processes, in which the occurrence of an event raises the likelihood of subsequent events; the intensity then decays over time before returning to a baseline level. Since this seminal work, the theory and applications of it have known a growing interest. We refer to for example some recent studies focusing on multivariate processes [11, 10], parameters estimation for linear Hawkes process with general kernel [15], limit and convergence theorems and approximation results [35, 40, 17, 46], expansion formulas and Malliavin calculus [32, 31] or extensions to rough diffusions [36, 22, 12]. These processes have found a remarkably wide range of applications, including earthquake modeling [52, 42], finance and microstructure [23, 34], market order flow modeling [38, 37, 26, 25], see also the review papers [3, 29], credit risk [24], epidemiology [48], soccer timing of threat events and ball touches of players [6], more recently, cyber risk [4, 20, 9, 14, 33] or cascade accidents in power plant investment [2]. This last application leads naturally to the question of controlling Hawkes processes, in which a hacker and a defender each seek to optimally manage their losses over the course of an attack episode. Numerical schemes for simulating Hawkes processes via the so-called thinning method were first investigated in [43, 44], and later rigorously formalized through a two-dimensional Poisson measure by Brémaud and Massoulié [13]. Numerical methods for generalized Hawkes have also been recently developed in [54]. The emergence of cyber risk has renewed the mathematical community’s interest in control of Hawkes processes, inducing new challenges related to the control of the intensity process. In its simplest form, a Hawkes process N is a counting process whose self-exciting intensity λ is given by Z t− Φ(t − s) dNs , λt = µ∞ (t) + 0

where µ∞ is the baseline intensity and Φ is a kernel that measures the magnitude of the self-excited jumps. The aim of this paper is to study the control of such dynamics in a multidimensional framework, namely dXt = b(t, Xt , at ) dt + σ(t, Xt , at ) dWt +

m X

γi (t, Xt− , at ) dNti ,

i=1 ⊤ ∈ Rm , and is where the intensity of N = (N 1 , . . . , N m ) is denoted by λt = (λ1t , . . . , λm + t ) assumed to be defined componentwise by m Z t− X i λt = µi (t, Xt− , at ) + Φij (t − s) dNsj , j=1

0

where Φ is a mutual excitation matrix between each component of N . This kernel lies at the heart of the complexity of Hawkes processes, and developing a self-contained control theory for them remains an open challenge, with only partial preliminary results available so far. When m = 1 and Φ takes the exponential form Φ(s) = q e−βs , known as the exponential kernel, the intensity λ admits an explicit expression and, in particular, becomes a Markov process. [8] exploited this structure to reduce the stochastic control of a Hawkes-driven diffusion with an exponential kernel to a standard Markovian optimization problem with state variables (N, λ) later extended in [14, 45] to the control of cyber systems, again restricting attention to a single exponential kernel. To the best of our knowledge, the main recent breakthrough is due to [41], who approximate the value of a stochastic control problem driven by a general-kernel Hawkes process by that of a regularized version driven by a mixture of exponentials. This construction, known as the Markovianization of the generalized Hawkes model, has also been studied for Volterra processes in [1]. 2

1.2

Hawkes CT-DDPG: the big picture

The first ingredient of our Hawkes CT-DDPG method is to reconstruct the approximate, Markovianized control problem using only the observed realizations of the controlled diffusion together with its event (jump) times, and to store in a replay buffer the new state, composed of the updated diffusion value and its jump components. More precisely, the control process a at time t governs how the existing Hawkes memory is read at that time, while past jumps continue to contribute through a classical Hawkes kernel. The Markovian approximation of a kernel Φ controlled by a takes the form Φ(τ, a) ≈ ΦK (τ, a) =

K X

Qk (a) e−βk τ .

k=1

Note that mixtures of exponentials with possibly negative weights are dense in Lp , see [39]. We use this fundamental result for the Markovianization of the kernel Φ above through the matrices Qk (a) which may have non-positive entries. This may produce negative raw memory values. To avoid that, we define the componentwise positive part to the approximated memory term: ! K X K,k K λK Qk (at ) Zt− , t = µ(t, Xt− , at ) + k=1

+

K where Z K,k = (Z K,k,j )m j=1 , driven by the jumps of N , componentwise by K,k,j Zt− :=

Z

e−βk(t−s) dNsK,j ,

j = 1, . . . , m.

(0,t)

A key technical point is that the memory variables entering the intensity must be predictable. For this reason, throughout the paper the approximating intensity is evaluated at the left-limit K,k memory Zt− . The process ZtK,k itself is càdlàg and jumps at event times; this convention keeps the intensity predictable while preserving the Markov property of the lifted process. This is the purpose of our first layer, summarized in Algorithm 1 below. The second ingredient builds on the recent works on continuous time reinforcement learning, investigated in [18] and used in stochastic control in [53] with a relaxed-control formulation and more recently in [16] which uses a Continuous-Time Deep Deterministic Policy Gradient (CT-DDPG) method to update both the control (actor) and the value function (critic) with neural networks. The central idea is to define the training losses for the network weights by extracting as much information as possible from the martingale property of stochastic integrals (the so-called generalized moment method), together with an advantage rate designed to learn the Hamiltonian of the corresponding HJB equation, which is the only quantity containing the unknown parameters. This second layer is summarized in Algorithm 2 below. Finally, the convergence of our algorithm rests on two main contributions: the Markovianization approximation and the error incurred when training the actor and critic networks via the Hawkes CT-DDPG method. The former can be analyzed theoretically, whereas the latter remains poorly understood within current reinforcement learning theory and can only be assessed numerically, by benchmarking our algorithm against an oracle and against analytical solutions, where these exist, as a sanity check. The paper is organized as follows. Section 2 introduces the controlled Hawkes jump–diffusion model together with the main standing assumptions. Section 3 develops the exponential Markov approximation based on mixtures of exponentials, establishes the corresponding convergence 3

theorem, and presents the online Markov-state update algorithm. Section 4 formulates the Hawkes CT-DDPG method, including the Bellman equation, the advantage rate, the deterministic policy gradient identity, the martingale characterization, and the implementation algorithm. Finally, Section 5 presents numerical experiments under three kernels: a single-exponential kernel, an Erlang kernel, and a power-law kernel. For the first two cases, the exact finite-dimensional Markov representations allow us to construct known-parameter DGM/HJB oracle benchmarks. For the power-law case, which has no exact finite-dimensional Markov representation, we instead use a DGM/HJB benchmark obtained after approximating the kernel by a finite mixture of exponentials. We compare Hawkes CT-DDPG with these model-based benchmarks, a validationselected static policy, and the discrete-time reinforcement-learning methods SAC and DDPG, and we show, in particular, that Hawkes CT-DDPG remains the closest approximator of the oracle in all scenarios and consistently outperforms the other discrete RL methods, thereby extending the findings of [16] to Hawkes-driven diffusions.

2

Mathematical framework and controlled Hawkes jump-diffusions

In the full paper we set a probability space (Ω, F, P) endowed with a dW -dimensional Browninan motion W . Let T > 0 be a finite horizon. Let Π := (Πi )i≤m be a family of m independent Poisson random measures on (0, T ] × R+ each with compensator ds dθ. We set
Ft := σ X0 , Ws , Πi ((0, s] × B) : 0 ≤ s ≤ t, i = 1, . . . , m, B ∈ B(R+ ), Leb(B) < ∞ , and let F be the usual augmentation of F0 = (Ft0 )0≤t≤T . We call F the environmental filtration. Let A ⊂ Rda be the action space with da > 0 supposed to be compact. An admissible control, also called a policy in the optimization problem, is an A-valued F-predictable process a = (at )0≤t≤T . The random variable at is the local action applied at time t. We set A := {a : [0, T ] × Ω −→ A : a is F-predictable} . Remark 2.1. The class A is fixed independently of the kernel approximation index K defined in the following section. In particular, when the original and approximating systems are compared later, the same value at (ω), ; ω ∈ Ω is inserted into both systems. Let b, σ, γ be the drift, volatility and jump severity coefficients defined for dx > 0 by b : [0, T ] × Rdx × A → Rdx ,

σ : [0, T ] × Rdx × A → Rdx ×dW ,

γi : [0, T ] × Rdx × A → Rdx ,

i = 1, . . . , m.

We now define the controlled Hawkes-driven SDE system. Let (Ω, F, F, P) support an Rdx -valued initial condition X0 , a dW -dimensional Brownian motion W , and mutually independent Poisson random measures Π1 , . . . , Πm , independent of (X0 , W ), each with intensity dt dθ. Let a ∈ A. A controlled Hawkes jump–diffusion under a is a strong solution (X a , N a , λa ) of the coupled

4

Brownian-Poisson measure system  Z t Z t  a a a  σ(s, Xs− , as ) dWs b(s, X , a ) ds + X = X +  0 s− s t   0 0    Z ∞ m Z  X   a  Πi (ds, dθ), X0 ∈ Rdx , + γi (s, Xs− , as )1{θ≤λa,i   s }   i=1 (0,t] 0     Z Z ∞

Πi (ds, dθ), i = 1, . . . , m, 1{θ≤λa,i Nta,i =  s }   (0,t] 0       λa,i = µ (t, X a , a )  i  t− t t   Z ∞  m  XZ    + Φij (t − s, at )1{θ≤λa,j Πj (ds, dθ),  s }  (0,t) 0

(1)

i = 1, . . . , m.

j=1

Remark 2.2. The control inside the kernel Φ is given by the current action at , not the historical action (as )s≤t . The coefficient Φij (τ, a) represents the lag-τ excitation of component i caused by a past jump in component j, as read under the current action at . Here X a and N a are càdlàg, and λa is required to be nonnegative and F-predictable. Equivalently, Z a a λt = µ(t, Xt− , at ) + Φ(t − s, at ) dNsa , (2) (0,t)

and the state equation satisfies a a dXta = b(t, Xt− , at ) dt + σ(t, Xt− , at ) dWt +

m X

a γi (t, Xt− , at ) dNta,i .

i=1

For a generic current action a ∈ A, define the current-action memory readout by Z a Ht (a) := Φ(t − s, at ) dNsa .

(3)

(0,t) a , a ) + H a (a ). Thus the action used to read the accumulated Hawkes Therefore λat = µ(t, Xt− t t t memory at time t is the current action at , rather than the historical actions applied when the past events occurred.

Remark 2.3. The strict upper limit (0, t) in (2) ensures that λat depends only on accepted a , this avoids ill-posedness events strictly before t. Together with the predictability of at and Xt− of the equation and any algebraic loop at time t. e i (ds, dθ) := Πi (ds, dθ) − ds dθ. Note that We define the compensated Poisson measure by Π R T a,i whenever 0 λs ds < ∞ almost surely, Z t Z Z ∞ e i (ds, dθ) Mta,i := Nta,i − λa,i ds = 1{θ≤λa,i Π (4) s s } 0

is an F-local martingale. Consequently, λa,i is its intensity.

(0,t)

0

Rt

a,i a,i 0 λs ds is the F-predictable compensator of N , and

Remark 2.4 (Environmental and observation filtrations). For a fixed admissible control a, one may define the observation filtration Fa,obs as the usual augmentation of Fta,obs := σ(Xsa , Nsa : 0 ≤ s ≤ t) . Then Fa,obs ⊆ F. This policy-dependent filtration is useful for describing what is supplied to the learning algorithm, but it is not used to define the common admissible class A in the approximation theorem. 5

Throughout, ∥ · ∥ denotes the Euclidean norm on Rdx and Rda , and ∥ · ∥F denotes the Frobenius norm on Rdx ×dW . For vectors in Rm , we use the standard ℓ1 -norm ∥ · ∥1 , while for M ∈ Rm×m , m

X ∥M v∥1 ∥M ∥1→1 := sup = max |Mij |. 1≤j≤m v̸=0 ∥v∥1 i=1

This norm is natural for the Hawkes memory because a jump in component j contributes the column M ej , and ∥M ∆Ns ∥1 ≤ ∥M ∥1→1 ∥∆Ns ∥1 . We now set the standing assumption, enforced along this study. Assumption 2.1 (Standing assumptions). The following conditions hold. (S1) The maps b, σ, and γi , i = 1, . . . , m, are Borel measurable and continuous in the time and action variable. There exists L > 0 such that, uniformly in (t, u) ∈ [0, T ] × A, ∥b(t, x, u) − b(t, x′ , u)∥ + ∥σ(t, x, u) − σ(t, x′ , u)∥ m X + ∥γi (t, x, u) − γi (t, x′ , u)∥ ≤ L∥x − x′ ∥, i=1

and ∥b(t, x, u)∥ + ∥σ(t, x, u)∥ +

m X

∥γi (t, x, u)∥ ≤ L(1 + ∥x∥).

i=1

(S2) The baseline map µ is Borel measurable, nonnegative, and continuous in the time and action variable. Uniformly in (t, u) ∈ [0, T ] × A, ∥µ(t, x, u) − µ(t, x′ , u)∥1 ≤ L∥x − x′ ∥,

∥µ(t, x, u)∥1 ≤ L(1 + ∥x∥).

(S3) The kernel Φ is Borel measurable and entrywise nonnegative. For every T > 0, ϕT (·) := sup ∥Φ(·, u)∥1→1 ∈ L1 ([0, T ]). u∈A

Moreover, for every a ∈ A, the process t 7→ Hta (at ) defined by (3) admits an F-predictable version. (S4) For every a ∈ A, the system (1) admits a pathwise unique strong solution (X a , N a , λa ) on [0, T ]. For every p ≥ 1, there exists Cp,T < ∞, independent of a, such that " Z p # sup E a∈A

sup ∥Xta ∥p + ∥NTa ∥p1 +

0≤t≤T

T

∥λat ∥1 dt

≤ Cp,T .

0

Remark 2.5 (Scope of the well-posedness assumption). This assumption isolates strong existence, pathwise uniqueness, nonexplosion, and uniform moment control from the Markovapproximation argument. All these properties can be verified under appropriate Lyapunov and stability conditions for state-dependent Hawkes intensities, see for example [41, Proposition 1.4]. The focus of this paper is the finite-dimensional kernel approximation and its use in continuous-time actor–critic learning, rather than a complete existence theory for the nonlinear closed-loop Hawkes system. In particular, these properties are satisfied in the numerical example we investigate below, see Section 5. When no confusion can arise, we suppress the superscript a and write (X, N, λ) for the system controlled by a fixed a ∈ A. 6

3

Finite-Dimensional Markov Approximation

In order to develop an RL method in continuous time extending [16] to Hawkes process, we first need to reduce the study to a Markovian optimization with its associated HJB equation. The key idea to to approach any integrable kernel potentially memory dependent with a family of exponential kernels, known as a mixture of exponentials, by using a density argument and convergence of value functions associated to this Markovianized version. This process has been previously developed in [1] for Volterra processes and [41] for Hawkes processes. In our model however, we face a technical challenge since the Markovianized kernel is itself controlled and must approximated the general one unformly with respect to the control. We first set the following assumption to approach any kernel Φ ∈ Lp for some p ≥ 1. Then we turn to the Markovianization of the state process and finally to the convergence of the value and objective of the Markovianization controlled process and Hawkes diffusion to the general one.

3.1

Signed exponential approximation

We enforce the following assumption in this study. Assumption 3.1. Fix a decay scale β > 0. For each K ≥ 1, there exists a kernel ΦK and a measurable continuous matrix coefficient functions Qk : A → Rm×m , defined by ΦK (τ, a) :=

PK

k=1 Qk (a)e

k = 1, . . . , K,

−βkτ . such that

Z T ε̄K (τ ) dτ −→ 0,

δK (T ) :=

K→∞

0

where ε̄K (τ ) := sup ∥ΦK (τ, a) − Φ(τ, a)∥1→1 . a∈A

Lemma 3.1. Suppose Assumption 2.1(S3) and Assumption 3.1 hold. (i) Let ϕ̄K (τ ) := sup ∥ΦK (τ, a)∥1→1 . a∈A

Then

Z T ϕ̄K (τ ) dτ < ∞.

sup K≥1 0

(ii) For every K ≥ 1, the scalar Volterra equation Z t ϕ̄K (t − s)RK (s) ds RK (t) = ϕ̄K (t) + 0

admits a unique solution RK ∈ L1 (0, T ). This solution is nonnegative and satisfies Z T sup RK (t) dt < ∞. K≥1 0

Proof. Let φT (τ ) := sup ∥Φ(τ, a)∥1→1 . a∈A

For almost every τ ∈ [0, T ], the reverse triangle inequality and the elementary inequality sup f (a) − sup g(a) ≤ sup |f (a) − g(a)| a∈A

a∈A

a∈A

7

(5)

give ϕ̄K (τ ) − φT (τ ) ≤ sup |∥ΦK (τ, a)∥1→1 − ∥Φ(τ, a)∥1→1 | a∈A

≤ sup ∥ΦK (τ, a) − Φ(τ, a)∥1→1 a∈A

= ε̄K (τ ). Consequently, ϕ̄K − φT

L1 (0,T )

≤ δK (T ) −→ 0.

(6)

In particular, Z T

Z T ϕ̄K (τ ) dτ ≤

φT (τ ) dτ + δK (T ).

0

0

Since φT ∈ L1 (0, T ) by Assumption 2.1(S3), and the convergent sequence (δK (T ))K≥1 is bounded, we obtain Z T

sup

ϕ̄K (τ ) dτ < ∞.

K≥1 0

This proves part (i). We next prove part (ii). For α > 0, define the weighted L1 -norm Z T ∥f ∥1,α := e−αt |f (t)| dt. 0

Since e−αT ∥f ∥L1 (0,T ) ≤ ∥f ∥1,α ≤ ∥f ∥L1 (0,T ) , this norm is equivalent to the usual L1 (0, T )-norm. For f, g ∈ L1 (0, T ), Fubini’s theorem and the change of variable u = t − s yield Z TZ t ∥f ∗ g∥1,α ≤ e−α(t−s) |f (t − s)|e−αs |g(s)| ds dt 0 0 Z T −s  Z T −αs −αu = e |g(s)| e |f (u)| du ds 0

0

≤ ∥f ∥1,α ∥g∥1,α .

(7)

Fix K ≥ 1. By part (i), ϕ̄K ∈ L1 (0, T ). Moreover, dominated convergence gives Z T ϕ̄K 1,α = e−αt ϕ̄K (t) dt −→ 0 as α → ∞. 0

Therefore, one may choose αK > 0 such that qK := ϕ̄K 1,α < 1. K

Consider the affine map TK f := ϕ̄K + ϕ̄K ∗ f on the Banach space L1 (0, T ) equipped with ∥·∥1,αK . By (7), ∥TK f − TK g∥1,αK ≤ qK ∥f − g∥1,αK . Thus TK is a contraction. The Banach fixed-point theorem therefore gives a unique RK ∈ L1 (0, T ) satisfying (5) almost everywhere on (0, T ). The solution is nonnegative. Indeed, starting from (0) RK = 0 and defining (n+1) (n) RK := TK RK , 8

(n)

all the iterates are nonnegative, because ϕ̄K ≥ 0, and RK → RK in the weighted L1 -norm. P P∞ ∗j (n) 1 Equivalently, RK = nj=1 ϕ̄∗j j=1 ϕ̄K with convergence in the weighted L K , so that RK = norm ∥·∥1,αK . It remains to prove the uniform bound in K. Since φT ∈ L1 (0, T ), dominated convergence gives Z T

∥φT ∥1,α =

e−αt φT (t) dt −→ 0

as α → ∞.

0

Choose α > 0 and ρ ∈ (0, 1) such that ∥φT ∥1,α < ρ < 1. By (6), ϕ̄K − φT

1,α

≤ ϕ̄K − φT

L1 (0,T )

≤ δK (T ) −→ 0.

Hence, after enlarging K0 if necessary, ϕ̄K 1,α ≤ ρ,

K ≥ K0 .

Taking the weighted L1 -norm in (5) and using (7), we obtain, for K ≥ K0 , ∥RK ∥1,α ≤ ϕ̄K 1,α + ϕ̄K 1,α ∥RK ∥1,α . Therefore, ∥RK ∥1,α ≤

ϕ̄K 1,α 1 − ϕ̄K 1,α

≤

ρ . 1−ρ

Using the equivalence of the weighted and unweighted norms gives Z T 0

RK (t) dt ≤ eαT ∥RK ∥1,α ≤ eαT

ρ , 1−ρ

K ≥ K0 .

RT For the finitely many indices K < K0 , 0 RK (t) dt < ∞ by the existence argument above. Combining these finitely many quantities with the preceding bound yields Z T sup

RK (t) dt < ∞.

K≥1 0

Remark 3.2 (Example: compact action set, separability and continuity of Qk (·)). Assume that m = 1 and A is a compact set; the kernel Φ(τ, a) is linearly separable Φ(τ, a) = Φ̂(τ )Q(a); the application a 7−→ Q(a) is continuous. Then we choose ΦK (τ, a) = Q(a)Φ̂K (τ ), with Φ̂K (τ ) := PK −βkτ for some coefficient qk such that Φ̂K −→ Φ̂. Therefore, Assumption 3.1 is k=1 qk e satisfied and so Lemma 3.1. This is in particular satisfied in the numerical section by choosing cef f a Φ(τ, a) = QHill (a)Φ̂(τ ), with A = [0, 1] and QHill (a) = 1 − ahalf +a , cef f > 0, ahalf ∈ A.

3.2

Markovianization procedure

We now turn to introduce the Markovian version of (Hawkes-SDE). We set a level of truncation K > 0 and define the truncated Markovianized version of (1) by

9

 Z t Z t  K,a K,a K,a  Xt = X0 + b(s, Xs− , as ) ds + σ(s, Xs− , as ) dWs    0 0    Z ∞ m Z  X   K,a  + γi (s, Xs− , as )1{θ≤λK,a,i } Πi (ds, dθ), X0 ∈ Rdx ,   s   i=1 (0,t] 0     Z Z ∞  1{θ≤λK,a,i } Πi (ds, dθ), i = 1, . . . , m, NtK,a,i = s   (0,t] 0       λK,a,i = µ (t, X K,a , a )  i t  t t−       Z ∞ m Z  X    i = 1, . . . , m. + (ΦK )ij (t − s, at )1{θ≤λK,a,j } Πj (ds, dθ) ,   s  (0,t) 0 j=1

+

(8) For each k = 1, . . . , K, define the Rm -valued exponentially-weighted Hawkes memory factor K Z K,k = (Z K,k,j )m j=1 , driven by the jumps of N , componentwise by Z Z Z ∞ K,k,j −βk(t−s) K,j e dNs = e−βk(t−s) 1{θ≤λK,j } Πj ( ds, dθ), Zt− := (0,t)

(0,t)

s

0

j = 1, . . . , m.

Equivalently, Z K,k is the Rm -valued càdlàg solution of the linear jump-ODE dZtK,k = −βk ZtK,k dt + dNtK ,

Z0K,k = 0,

(9)

where N K = (N K,1 , . . . , N K,m )⊤ ∈ Rm ; that is, the j-th component decays at rate βk and jumps by 1 at each accepted event of N K,j . Therefore, we define the approximate memory of the Hawkes process by HtK (at ) :=

K X

K,k Qk (at ) Zt− =

k=1

Z

K X

Qk (at )e

−βk(t−s)

dNsK =

(0,t) k=1

Z

ΦK (t − s, at ) dNsK ,

(0,t)

P −βkτ of Assumption 3.1.We which recovers the Markovianized kernel ΦK (τ, a) = K k=1 Qk (a)e define the càdlàg exponential memory variables Z t K,k Zt := e−βk(t−s) dNsK ∈ Rm , k = 1, . . . , K. 0

Then, we define the signed approximate memory HtK (at ) :=

K X

K,k Qk (at )Zt− .

k=1

The approximating Markov Hawkes-SDE is thus given by  P K,i K  dXtK,a = b(t, XtK,a , a ) dt + σ(t, XtK,a , at) dWt + m t  i=1 γi (t, Xt− , at ) dNt ,  PK K,k K , (Hawkes−SDE)K λK t = µ(t, Xt− , at ) + k=1 Qk (at )Zt− +    K,k K,k dZt = −βk Zt dt + dNtK , for a choice of Qk associated with the kernel Φ as defined under Assumption 3.1 and where for any vector u ∈ Rm , the vector u+ denotes the componentwise positive part: u+ = (max{u1 , 0}, . . . , max{um , 0})⊤ . 10

Remark 3.3 (Signed approximation and positivity). Note that the positive part is set to preserve the usual density argument for finite linear combinations of {e−βkt }k≥1 in Lp (R+ ). Since signed approximants may create negative raw memory values, the approximating intensity applies a componentwise positive part only to the approximated memory term. Thus signed approximation and valid intensities are both retained. Definition 3.4 (State Space). The finite-dimensional Markov state is given by StK := (t, XtK,a , ZtK,1 , . . . , ZtK,K ). We denote this space by SK where SK := [0, T ] × Rdx × (Rm )K . The process StK is càdlàg and Markov. Intensities and controls are evaluated using predictable information. Thus, when a Markov control is used in continuous time, the precise convention is YtK = (XtK,a , ZtK,1 , . . . , ZtK,K ).

K at = π(t, Yt− ),

K outside Inside Lebesgue time integrals we often write π(t, YtK ) for readability, since YtK = Yt− jump times and jump times have zero Lebesgue measure.

Remark 3.5 (Model-free state construction). The matrices Qk (a), the baseline µ, and the intensity λ are used only in the theoretical Markov approximation. They are not needed to construct the state StK . The memory update induced by solving (9) only requires observed event times, component labels, and the chosen decay scale β. Hence the representation is compatible with model-free CT-DDPG. Similarly to the previous standing assumption (S4) we assume that the system (Hawkes − SDE)K admits a solution with enough integrability so that the following assumption is enforced along the study. Assumption 3.2 (Uniform well-posedness and localization moments for the approximations). For every K ≥ 1 and every admissible control a ∈ A, the approximating Poisson-embedded system admits a pathwise unique strong solution (X K,a , N K,a , λK,a ) on [0, T ]. Moreover, there exist p⋆ > 1 and Cp⋆ ,T < ∞, independent of K and a, such that for any t ∈ [0, T ] " Z p # sup sup Et,x,zK ;a

K≥1 a∈A

sup t≤u≤T

XuK,a

p⋆

+ NTK,a

p⋆ 1

T

+ 0

⋆

λK,a t

1

dt

≤ Cp⋆ ,T ,

where Et,x,zK ;a [·] := E[·|Xt = x, (ZtK,1 , . . . , ZtK,K ) = zK ] following the policy a. We provide a sufficient condition for Assumption 3.2 in the proposition below, also satisfied in our numerical simulations (see Appendix A). Proposition 3.6. Fix p⋆ > 1. In addition to Assumption 2.1(S1)–(S3) and 3.1, suppose that the following conditions hold. (i) The baseline intensity is uniformly bounded: sup (t,x,u)∈[0,T ]×Rdx ×A

for some µ < ∞. 11

∥µ(t, x, u)∥1 ≤ µ

(ii) The jump amplitudes are uniformly bounded: m X

sup

∥γi (t, x, u)∥ ≤ Γ

(t,x,u)∈[0,T ]×Rdx ×A i=1

for some Γ < ∞. (iii) The nonnegative kernel envelopes satisfy the uniform subcriticality condition Z T ϕK (s) ds < 1, ϕK (s) := sup ∥ΦK (s, u)∥1→1 . ρT := sup u∈A

K≥1 0

Then Assumption 3.2 holds with exponent p⋆ . Proof. Fix K ≥ 1 and a ∈ A, and set NtK,a,Σ := NtK,a

1

=

=

1

NtK,a,i ,

i=1

and ℓK,a := λK,a t t

m X

m X

λK,a,i . t

i=1

Since the componentwise positive-part map satisfies ∥z+ ∥1 ≤ ∥z∥1 , we have the estimate ℓK,a ≤ t

K,a µ(t, Xt− , at )

Z ≤µ+

Z 1

ΦK (t − s, at ) dNsK,a

+ (0,t)

1

ϕK (t − s) dNsK,a,Σ .

(10)

(0,t)

Let N ϕK :

K

be the scalar linear Hawkes process with constant baseline µ and nonnegative kernel K λt = µ +

Z

K

ϕK (t − s) dN s . (0,t)

With the common Poisson embedding and the monotonicity of the right-hand side of (10), we can deduce K NtK,a,Σ ≤ N t , 0 ≤ t ≤ T, a.s. K

Extend ϕK by zero outside [0, T ]. In the cluster representation of N , each immigrant generates a Galton–Watson cluster whose number of direct offspring has Poisson mean Z T ρK := ϕK (s) ds ≤ ρT < 1. 0 K

Let MT denote the number of immigrants of N t occurring in [0, T ], so we have MT ∼ Pois(µT ). Let SK,1 , SK,2 , ... be the independent total cluster sizes associated with these immigrants. Every K point of N observed by time T belongs to one of those clusters, so we have K

NT ≤

MT X j=1

12

SK,j .

By Holder’s inequality, we have  p  MT n X X p E  SK,j  MT = n ≤ np−1 E[SK,j ] j=1

j=1 p = np E[SK ].

Therefore,  h E

K

NT

p i

≤ E 

MT X

p  SK,j  

j=1 p ≤ E[MTp ]E[SK ].

By Hawkes and Oakes [30, p. 496], the cluster SK is almost surely finite because ρK < 1. In addition, its total progeny has the Borel distribution; see [21, 51]. Since ρK ≤ ρT < 1, a Poisson(ρK ) offspring variable can be coupled as a thinning of a Poisson(ρT ) offspring variable. Applying this recursively to the Galton–Watson trees gives SK ≤ SρT a.s. under a suitable coupling. Then, because of the exponential tail of the Borel(ρT ) distribution, we have ∗

p⋆ sup E[SK ] ≤ E[SρpT ] < ∞.

K≥1

Since the number of immigrants before T , MT , is bounded by a Poisson random variable with mean µT , it follows that h
p i (11) sup sup E NTK,a,Σ ⋆ < ∞. K≥1 a∈A

In particular, the approximating counting processes are nonexplosive. Integrating (10) over time and applying Fubini’s theorem gives  Z T Z T Z K,a ℓt dt ≤ µT + ϕK (t − s) dt dNsK,a,Σ 0

(0,T ) s K,a,Σ . ≤ µT + ρT NT

Consequently, (11) implies "Z T sup sup E K≥1 a∈A

0

λK,a t

1

 p⋆ # dt < ∞.

It remains to estimate the physical state. The uniform boundedness of the jump amplitudes gives m Z X K,a sup γi (s, Xs− , as ) dNsK,a,i ≤ ΓNtK,a,Σ . 0≤u≤t

i=1

(0,u]

Using the linear-growth assumptions on b and σ, the Burkholder–Davis–Gundy inequality, Young’s inequality, we obtain    ! Z t    p p ⋆ ⋆ K,a,Σ p E sup XuK,a ≤ Cp⋆ ,T 1 + E ∥X0 ∥ ⋆ + E (NT )p⋆ + E sup XrK,a ds . 0≤u≤t

0

Hence, Gronwall’s inequality yields " sup sup E

sup

K≥1 a∈A

0≤t≤T

13

XtK,a

p⋆

# < ∞.

0≤r≤s

Strong existence and pathwise uniqueness up to explosion follow from the standard interlacing construction for jump-diffusions; see [47, Chapter V.10, Theorem 57] and [5, Proposition 2.1]. Between consecutive accepted events, the physical state solves a globally Lipschitz diffusion SDE, while at each accepted event the counting-process and state updates are uniquely determined by the common Poisson embedding and the pre-jump state. Furthermore, the pathwise domination K

NtK,a,Σ ≤ N t

K

and the almost-sure finiteness of the subcritical Hawkes count N T imply that only finitely many accepted events occur on [0, T ]. Hence explosion is impossible, and so we have a unique nonexplosive strong solution. Together with the preceding uniform moment estimates, this proves Assumption 3.2. Remark 3.7. (ii) and (iii) are satisfied, for example if µ, γ are uniformly bounded in x and A is compact.

3.3

Objective functions and convergence of value functions

We assume that the agent is minimizing a running cost c : [0, T ] × Rdx × A −→ R along the time duration [0, T ] and a terminal cost g : Rdx −→ R, satisfying the following assumption. Assumption 3.3 (Cost regularity). c is Borel measurable and continuous in time and action variable. In addition, there exist constants C > 0 and q ≥ 0 such that, uniformly in (t, a), |c(t, x, a)| + |g(x)| ≤ C(1 + ∥x∥q+1 ), and

|c(t, x, a) − c(t, x′ , a)| + |g(x) − g(x′ )|
≤ C 1 + ∥x∥q + ∥x′ ∥q ∥x − x′ ∥.

For the objective and value convergence conclusions, the exponent p⋆ in Assumption 3.2 is assumed to satisfy p⋆ > q + 1. For an admissible control a, we define the objective function for the problem by Z T  J(a) := E c(t, Xt , at ) dt + g(XT ) . 0

We aim at solving the following problem V0 := inf J(a). a∈A

We define its Markovian approximation by V0K := inf JK (a), a∈A

Z T JK (a) := E 0



c(t, XtK,a , at ) dt + g(XTK,a )

.

Let ei ∈ Rm be the i-th coordinate vector. For a smooth test function f = f (t, x, zK ), continuously differentiable with respect to the time component, twice continuously differentiable with respect to the diffusion variable x and one continuously differentiable with respect to each memory variable z i , we define the extended generator i 1 h LaK f (t, x, zK ) = ∂t f (t, x, zK ) + b(t, x, a) · ∇x f (t, x, zK ) + Tr σσ ⊤ (t, x, a)∇2xx f (t, x, zK ) 2 K X − βk z k · ∇z k f (t, x, zK ) k=1

+

m X

h i K 1 K K λK (t, x, z , a) f (t, x + γ (t, x, a), z + e , . . . , z + e ) − f (t, x, z ) , i i i i

i=1

14

h P  i K k K where λK . i (t, x, z , a) = µ(t, x, a) + k=1 Qk (a)z + i

Lemma 3.8 (Stopped physical-state stability). Suppose that Assumption 2.1(S1) holds. Fix K ≥ 1 and a ∈ A. Let (X a , N a , λa )

(X K,a , N K,a , λK,a )

and

be constructed with the same Brownian motion, the same Poisson random measures, the same initial condition, and the same admissible control process a. For R ≥ 1, define n o τRK,a := inf t ∈ [0, T ] : NtK,a + ∥Nta ∥1 > R or XtK,a > R or ∥Xta ∥ > R ∧ T. 1

Set K,a DX,R (t) := E



a sup X K,aK,a − Xu∧τ K,a u∧τR R 0≤u≤t

and K,a Dλ,R (t) := E

Z t∧τ K,a R

 ,

λK,a − λas 1 ds. s

0

Then, for every R ≥ 1, there exists CT,R < ∞, independent of K and a ∈ A, such that K,a DX,R (t) ≤ CT,R

Z t 0

K,a K,a DX,R (s) ds + CT,R Dλ,R (t),

0 ≤ t ≤ T.

(12)

Consequently, K,a K,a DX,R (t) ≤ CT,R Dλ,R (t),

0 ≤ t ≤ T.

Proof. Fix K ≥ 1 and a ∈ A. For readability, suppress the superscript a, and write τR := τRK,a ,

∆Xt := XtK − Xt .

Define SR (t) := sup ∥∆Xu∧τR ∥ . 0≤u≤t

Thus, DX,R (t) = E[SR (t)]. For each i = 1, . . . , m, set IsK,i (θ) := 1{θ≤λK,i } ,

Isi (θ) := 1{θ≤λis } .

s

Define the counting process for the common jump from two processes by Z Z ∞ K,i Ct := 1{θ≤λK,i ∧λi } Πi ( ds, dθ), (0,t]

s

0

s

and define the counting process for the discrepancy between two processes by Z Z ∞ K,i IsK,i (θ) − Isi (θ) Πi ( ds, dθ). Ξt := (0,t]

0

Indeed, Z (0,t]

Z R+

IsK,i (θ) − Isi (θ) Πi ( ds, dθ) =

Z (0,t]

15

|dNtK.i − dNti |,

(13)

because for any fixed atom (s, θ), if NsK,i and NsK accept or reject it simultaneously, then IsK,i (θ) − Isi (θ) = |dNtK.i − dNti | = 0; if they disagree on this point, then IsK,i (θ) − Isi (θ) = |dNtK.i − dNti | = 1. Furthermore, Z ∞

i IsK,i (θ) − Isi (θ) dθ = λK,i s − λs ,

0

so we have E[ΞK,i t ]=E Set CtK :=

m X

Z

i λK,i s − λs ds

(14)

(0,t]

CtK,i ,

ΞK t :=

i=1

m X

ΞK,i t ,

i=1

decomposing the total counts of two processes by common jumps and the discrepancies, we have NtK 1 + ∥Nt ∥1 = 2CtK + ΞK (15) t . Let K QK + ∥Nt ∥1 . t := Nt 1

On a set of probability one, each component counting process has jumps of size at most one. Consequently, ∆QK 0 ≤ t ≤ T. t ≤ 2m, By the definition of τR , QK τR − ≤ R. It follows that, for every t ∈ [0, T ], QK t∧τR ≤ R + 2m. Using (15) and the nonnegativity of ΞK , we obtain K Ct∧τ ≤ R

R + 2m . 2

(16)

Write K γsK,i := γi (s, Xs− , as ),

γsi := γi (s, Xs− , as ).

The jump integrand admits the decomposition
γsK,i IsK,i (θ) − γsi Isi (θ) = γsK,i − γsi 1{θ≤λK,i ∧λi } s

s

+ γsK,i 1{λi <θ≤λK,i } s

s

− γsi 1{λK,i <θ≤λi } . s

s

Let L be the Lipschitz and growth constant in Assumption 2.1(S1). By the definition of τR , for K ∨ ∥X ∥ ≤ R. Therefore, every s ≤ τR , Xs− s− γsK,i − γsi ≤ LSR (s−), and γsK,i ∨ γsi ≤ ΓR ,

ΓR := L(1 + R).

Consequently, γsK,i IsK,i (θ) − γsi Isi (θ) ≤ Lγ SR (s−)1{θ≤λK,i ∧λi } + ΓR IsK,i (θ) − Isi (θ) . s

16

s

Subtracting the two stopped state equations, and using the Lipschitz continuity of b, gives Z t Z K SR (s) ds + MR (t) + ΓR Ξt∧τR + L SR (s−) dCsK , (17) SR (t) ≤ L 0

(0,t∧τR ]

where

Z u∧τR MR (t) := sup 0≤u≤t

  K σ(s, Xs− , as ) − σ(s, Xs− , as ) dWs .

0

Set

Z t HR (t) := L 0

SR (s) ds + MR (t) + ΓR ΞK t∧τR .

The process HR is nonnegative and nondecreasing. Iterating (17) over the jump times of C K gives Y
SR (t) ≤ HR (t) 1 + L∆CsK . 0<s≤t∧τR

Since 1 + x ≤ ex for x ≥ 0, Y

K 1 + L∆CsK ≤ exp LCt∧τ . R

0<s≤t∧τR

Using (16), we conclude that  Z t  K SR (t) ≤ κR L SR (s) ds + MR (t) + ΓR Ξt∧τR ,

(18)

0

where

 κR := exp

L(R + 2m) 2

 .

By the Burkholder–Davis–Gundy inequality and the Lipschitz continuity of σ, "Z 1/2 # t∧τR

E[MR (t)] ≤ CBDG LE

∥∆Xs− ∥2 ds

.

0

Moreover, Z t∧τR

Z t

2

∥∆Xs− ∥ ds ≤ SR (t) 0

SR (s) ds. 0

Hence, by the Cauchy–Schwarz inequality, " E[MR (t)] ≤ CBDG LE

1/2 #

Z t

SR (t)

SR (s) ds 0

 1/2 Z t DX,R (s) ds . ≤ CBDG L DX,R (t) 0

Rt Let A = κR CBDG L( 0 DX,R (s) ds)1/2 , B = (DX,R (t))1/2 , then we have AB ≤ 12 A2 + 12 B 2 , which implies Z t 1 κR E[MR (t)] ≤ DX,R (t) + CT,R DX,R (s) ds, (19) 2 0 where CT,R is constant depending on T, R. Note that, by (14) ΞK t −

Z t 0

λK s − λs 1 ds 17

is a local martingale. Therefore, from (14) and Fubini’s theorem, m   X E ΞK = E t∧τR

=

i=1 m X

(0,T ]

1{s≤t∧τR } 1{θ≤λK,i } − 1{θ≤λis } Πi ( ds, dθ) s

0

Z TZ ∞ 1{s≤t∧τR } 1{θ≤λK,i } − 1{θ≤λis } dθ ds

E 0

i=1

=E

Z ∞

Z

s

0

Z t∧τR X m 0

i λK,i s − λs ds

i=1

Z t∧τR

λK s − λs 1 ds

=E 0

= Dλ,R (t).

(20)

K Notice that ΞK t∧τR ≤ Qt∧τR ≤ R + 2m, so the stopped discrepancy count is integrable. Taking expectations in (18), using (19) and (20), gives

Z t DX,R (t) ≤ CT,R

1 DX,R (s) ds + DX,R (t) + CT,R Dλ,R (t). 2 0

Absorbing the term 12 DX,R (t) into the left-hand side yields Z t DX,R (t) ≤ CT,R

DX,R (s) ds + CT,R Dλ,R (t). 0

This proves (12). Finally, Dλ,R is nonnegative and nondecreasing. The inhomogeneous Gronwall inequality therefore gives DX,R (t) ≤ CT,R Dλ,R (t),

0 ≤ t ≤ T,

after enlarging CT,R if necessary. This proves (13). We now give a fundamental lemma related to Volterra resolvent. Lemma 3.9 (Volterra-resolvent comparison). Let k ∈ L1 ([0, T ]; R+ ), and let R be the solution of the Volterra equation Z t R(t) = k(t) + k(t − s)R(s) ds, t ∈ [0, T ]. 0

Suppose f, F are nonnegative integrable functions on [0, T ] and Z t f (t) ≤ F (t) +

k(t − s)f (s) ds,

0 ≤ t ≤ T.

0

Then

Z t f (t) ≤ F (t) +

R(t − s)F (s) ds. 0

Consequently, Z t

 f (u) du ≤

0

Z T

Z t R(u) du F (s) ds,

0

0

1+

18

0 ≤ t ≤ T.

Proof. Define Z t R(t − s)F (s) ds.

G(t) := F (t) + 0

Using the resolvent identity R = k + k ∗ R, we have G = F + R ∗ F = F + k ∗ F + k ∗ R ∗ F = F + k ∗ G. Since f ≤ F + k ∗ f and G = F + k ∗ G, the Volterra comparison principle (see [7]) gives f ≤ G. Hence Z t

f (t) ≤ F (t) +

R(t − s)F (s) ds. 0

Integrating over [0, t] gives Z t

Z t

Z tZ u

f (u) du ≤

F (u) du + R(u − s)F (s) ds du 0 0   Z t Z t = F (s) 1 + R(u − s) du ds 0 s  Z t Z T ≤ 1+ R(r) dr F (s) ds.

0

0

0

0

Theorem 3.10 (Convergence of the Markov approximation). Suppose Assumptions 2.1, 3.1, and 3.2 hold. For every K ≥ 1 and a ∈ A, let (X a , N a , λa )

(X K,a , N K,a , λK,a )

and

be the respective solutions of (1) and (8), constructed with the same Brownian motion, the same Poisson random measures, and the same admissible control process a = (at )0≤t≤T . Then ( " # ) Z lim sup E

K→∞ a∈A

T

XtK,a − Xta

sup 0≤t≤T

+E 0

λK,a − λat t

1

dt

= 0.

(21)

Moreover, lim sup E

"m X

K→∞ a∈A

# sup

i=1 0≤t≤T

NtK,a,i − Nta,i

= 0.

If, in addition, Assumption 3.3 holds with its growth exponent q satisfying p⋆ > q + 1, where p⋆ is the exponent in Assumption 3.2, then sup |JK (a) − J(a)| −→ 0. a∈A

Consequently, for V0K := inf JK (a),

V0 := inf J(a), a∈A

a∈A

we have V0K − V0 −→ 0.

19

(22)

Proof. Fix K ≥ 1 and a ∈ A. Throughout the process-convergence part of the proof, suppress the superscript a. Let τR := τRK,a be the stopping time introduced in Lemma 3.8. Thus, n o τR = inf t ∈ [0, T ] : NtK 1 + ∥Nt ∥1 > R or XtK > R or ∥Xt ∥ > R ∧ T. Define  DX,R (t) := E

sup 0≤u≤t

Z t∧τR



K Xu∧τ − Xu∧τR R

,

Dλ,R (t) := E 0

λK s − λs 1 ds.

Also set   dλ,R (t) := E 1{t≤τR } λK t − λt 1 . Since the singleton {τR } has zero Lebesgue measure, we have Z t Dλ,R (t) =

dλ,R (s) ds. 0

For each component i = 1, . . . , m, recall the discrepancy counting process is defined as ΞK,i := t

Z ∞

Z (0,t]

For every i,

i

1{θ≤λK,i } − 1{θ≤λis } Π ( ds, dθ) s

0

and

ΞK t :=

m X i=1

Z ∞ 0

i 1{θ≤λK,i } − 1{θ≤λis } dθ = λK,i s − λs . s

Moreover, (s, θ) 7−→ 1{s≤t∧τR } 1{θ≤λK,i } − 1{θ≤λis } s

is nonnegative and predictable. Step 1: global control of the kernel-approximation error.

Define

Z [ΦK (t − s, at ) − Φ(t − s, at )] dNs

AK (t) := E (0,t)

. 1

By the definition of ε̄K ,

Z AK (t) ≤ E (0,t)

ε̄K (t − s) d ∥Ns ∥1 .

Consequently, Fubini’s theorem and the compensator of N give Z T

Z TZ AK (t) dt ≤ E

0

0

ε̄K (t − s) d ∥Ns ∥1 dt

(0,t)

Z

Z T

=E (0,T )

s

Z T Z T =E 0

s

 ε̄K (t − s) dt d ∥Ns ∥1

 ε̄K (t − s) dt ∥λs ∥1 ds

Z T ≤ δK (T )E 0

∥λs ∥1 ds

≤ CT δK (T ). The last estimate is uniform over a ∈ A by Assumption 2.1(S4). 20

ΞK,i t .

Step 2: intensity stability before τR . Define the predictable raw memories Z Z K K b Φ(t − s, at ) dNs . ΦK (t − s, at ) dNs , Ht := Ht := (0,t)

(0,t)

The two intensities satisfy K bK λK t = µ(t, Xt− , at ) + (Ht )+ ,

and, since Φ is entrywise nonnegative, λt = µ(t, Xt− , at ) + Ht = µ(t, Xt− , at ) + (Ht )+ . The componentwise positive-part map is 1-Lipschitz in the ℓ1 -norm. Hence, by Assumption 2.1(S2), K bK λK . (23) t − λt 1 ≤ L Xt− − Xt− + Ht − Ht 1

Decompose b K − Ht = H t

Z

ΦK (t − s, at ) d(NsK − Ns ) +

(0,t)

Z [ΦK (t − s, at ) − Φ(t − s, at )] dNs .

(24)

(0,t)

For the first term, define "

#

Z

BK,R (t) := E 1{t≤τR }

ΦK (t − s, at ) d(NsK − Ns )

(0,t)

. 1

Pathwise, the total variation of d(N K − N ) is bounded by the discrepancy count. Therefore, " # m Z X K,i BK,R (t) ≤ E 1{t≤τR } ϕ̄K (t − s) dΞs i=1

≤E

m Z X i=1

Z t = 0

(0,t)

(0,t)

1{s≤τR } ϕ̄K (t − s) dΞK,i s

  ϕ̄K (t − s)E 1{s≤τR } λK s − λs 1 ds

Z t ϕ̄K (t − s)dλ,R (s) ds.

=

(25)

0

In the second line, we used 1{t≤τR } ≤ 1{s≤τR } for any s < t, and the third line follows from the compensation formula. Multiplying (23) by 1{t≤τR } , taking expectations, and using (24), (25), and the global quantity AK (t), gives Z t dλ,R (t) ≤ LDX,R (t) + AK (t) + ϕ̄K (t − s)dλ,R (s) ds. 0

Apply Lemma 3.9 with f = dλ,R , F = LDX,R + AK , k = ϕ̄K . Using the uniform resolvent bound gives Z t  Z t Dλ,R (t) ≤ CT DX,R (s) ds + AK (s) ds 0 0 Z t ≤ CT DX,R (s) ds + CT δK (T ), 0 ≤ t ≤ T. (26) 0

Here CT is independent of K, a, and R. By Lemma 3.8, DX,R (t) ≤ CT,R Dλ,R (t). 21

Substituting this estimate into (26) gives Z t Dλ,R (t) ≤ CT,R

Dλ,R (s) ds + CT δK (T ). 0

Gronwall’s inequality implies Dλ,R (T ) ≤ CT,R δK (T ). Applying Lemma 3.8 once more gives DX,R (T ) ≤ CT,R δK (T ). Therefore, for every fixed R ≥ 1, h i K,a K,a lim sup DX,R (T ) + Dλ,R (T ) = 0.

(27)

K→∞ a∈A

Step 3: removal of the state/total-count localization. Let p := p⋆ . By Markov’s inequality, " # " # p 1 1 K,a K,a a p + p E sup ∥Xt ∥ P(τR < T ) ≤ p E sup Xt R R 0≤t≤T 0≤t≤T h p i 1 + pE NTK,a + ∥NTa ∥1 . (28) R 1 Since (x + y)p ≤ 2p−1 (xp + y p ),

x, y ≥ 0,

Assumptions 2.1(S4) and 3.2 imply sup sup P(τRK,a < T ) ≤

K≥1 a∈A

For the physical states, " E

sup 0≤t≤T

(29)

"

#

XtK,a − Xta

Cp,T . Rp

!

K,a (T ) + E ≤ DX,R

sup 0≤t≤T

By Hölder’s inequality, "

XtK,a + sup ∥Xta ∥ 1{τ K,a <T } .

sup

E

0≤t≤T

≤

sup 0≤t≤T

+ sup 0≤t≤T

∥Xta ∥

# 1{τ K,a <T } R

1− p1

XtK,a + sup ∥Xta ∥ 0≤t≤T

P(τRK,a < T )

sup sup E

sup

K≥1 a∈A

0≤t≤T

.

Lp

Using the uniform moment bounds and (29), we obtain ! " XtK,a

(30)

R

0≤t≤T

! XtK,a

#

#

Cp,T . Rp−1

(31)

  dt 1{τ K,a <T } .

(32)

+ sup ∥Xta ∥ 1{τ K,a <T } ≤ R

0≤t≤T

Similarly, Z T E 0

λK,a − λat t

1

K,a dt ≤ Dλ,R (T ) + E

Z T 0

22

λK,a t

1

+ ∥λat ∥1

R

Hölder’s inequality, the moment assumptions, and (29) give   Z T
Cp,T K,a a + ∥λt ∥1 dt 1{τ K,a <T } ≤ p−1 . sup sup E λt R R 1 K≥1 a∈A 0 Combining (27), (30), (31), (32), and (33), we obtain # ( " Z lim sup sup E

sup

K→∞ a∈A

0≤t≤T

XtK,a − Xta

)

T

λK,a − λat t

+E 0

1

≤

dt

(33)

Cp,T . Rp−1

Letting R → ∞ proves (21). Step 4: convergence of the counting processes. For each component i, the common Poisson embedding gives the pathwise total-variation bound sup NtK,a,i − Nta,i ≤ ΞK,a,i . T

0≤t≤T

By (14), E

"m X

# sup

i=1 0≤t≤T

NtK,a,i − Nta,i

≤E

"m X

# ΞK,a,i T

Z T =E 0

i=1

λK,a − λat t

1

dt.

Taking the supremum over a ∈ A and using (21) proves (22). Step 5: objective and value convergence. ∆aK := sup

0≤t≤T

XtK,a − Xta

For every a ∈ A, define (

a UK := max

,

sup 0≤t≤T

XtK,a

)

, sup ∥Xta ∥ . 0≤t≤T

By the weighted Lipschitz condition on c and g, Z T    q K,a K,a a q a 1 + Xt + ∥Xt ∥ Xt − Xt dt |JK (a) − J(a)| ≤ CE 0 i h  q + CE 1 + XTK,a + ∥XTa ∥q XTK,a − XTa a q ≤ CT E [(1 + (UK ) ) ∆aK ] .

(34)

If q = 0, then (21) directly gives sup |JK (a) − J(a)| ≤ CT sup E[∆aK ] −→ 0.

a∈A

a∈A

⋆ Suppose now that q > 0, and set s := pq⋆ , and r := p⋆p−q . Then r, s > 1 and 1r + 1s = 1. Moreover, since p⋆ > q + 1 then p⋆ − q > 1 and so r < p⋆ . First note that Assumptions 2.1(S4) and 3.2 imply a p⋆ sup sup E [(UK ) + (∆aK )p⋆ ] < ∞. (35)

K≥1 a∈A

Indeed, a p⋆ (UK ) ≤ sup

0≤t≤T

XtK,a

p⋆

+ sup ∥Xta ∥p⋆ , 0≤t≤T

and (∆aK )p⋆ ≤ 2p⋆ −1

sup 0≤t≤T

XtK,a 23

p⋆

! + sup ∥Xta ∥p⋆ 0≤t≤T

.

Define ϑ :=

p⋆ − q − 1 . p⋆ − 1

0 < ϑ < 1,

1−ϑ=

Since q > 0 and p⋆ > q + 1, we have q . p⋆ − 1

Furthermore, ϑ+

p⋆ − q − 1 1−ϑ q = + p⋆ p⋆ − 1 p⋆ (p⋆ − 1) p⋆ − q 1 = = . p⋆ r

Define the conjugate Hölder exponents κ1 := Then we have

1 , rϑ

κ2 :=

p⋆ . r(1 − ϑ)

1 r(1 − ϑ) 1 + = rϑ + = 1, κ1 κ2 p⋆

and therefore κ1 , κ2 > 1. Applying Hölder’s inequality yields h i E [(∆aK )r ] = E (∆aK )rϑ (∆aK )r(1−ϑ) h i1/κ1 h i1/κ2 ≤ E (∆aK )rϑκ1 E (∆aK )r(1−ϑ)κ2 = E[∆aK ]rϑ E [(∆aK )p⋆ ]r(1−ϑ)/p⋆ . Taking the r-th root gives the inequality E [(∆aK )r ]1/r ≤ E[∆aK ]ϑ E [(∆aK )p⋆ ](1−ϑ)/p⋆ .

(36)

Equivalently, ∥∆aK ∥Lr ≤ ∥∆aK ∥ϑL1 ∥∆aK ∥1−ϑ Lp⋆ . Taking the supremum over a ∈ A in (36), we obtain sup ∥∆aK ∥Lr ≤ a∈A

ϑ



sup E[∆aK ] a∈A

 ×

(1−ϑ)/p⋆

sup E [(∆aK )p⋆ ] a∈A

.

The first factor converges to zero by (21), while the second factor is uniformly bounded by (35). Hence sup ∥∆aK ∥Lr −→ 0. (37) a∈A

We next estimate the other factor in (34). Since qs = p⋆ , a q s a q s ∥1 + (UK ) ∥Ls = E [(1 + (UK ) ) ] a qs ≤ 2s−1 (1 + E [(UK ) ]) a p⋆ = 2s−1 (1 + E [(UK ) ]) .

Therefore, by (35), a q sup sup ∥1 + (UK ) ∥Ls < ∞.

K≥1 a∈A

24

(38)

Applying Hölder’s inequality with the conjugate exponents r and s in (34), and using (37) and (38), gives a q sup |JK (a) − J(a)| ≤ CT sup ∥1 + (UK ) ∥Ls sup ∥∆aK ∥Lr

a∈A

a∈A

a∈A

−→ 0. Thus, supa∈A |JK (a) − J(a)| −→ 0. Finally, define εK := supa∈A |JK (a) − J(a)| . For every a ∈ A, J(a) − εK ≤ JK (a) ≤ J(a) + εK . Taking the infimum over a ∈ A yields V0 − εK ≤ V0K ≤ V0 + εK . Consequently, V0K − V0 ≤ εK = sup |JK (a) − J(a)| −→ 0. a∈A

3.4

Online Markov-state update

We now describe how the finite-dimensional Markov state can be constructed online from observable data, in preparation for the reinforcement-learning method developed in the next section. The decay scale β > 0 and the number of filters K are user-chosen design parameters, while m is the number of observed event types. At each decision time, the learner only needs the current physical state and the times and component labels of the events observed since the preceding decision time. These observations determine the exponential filters through the recursion X Ztkn+1 = e−βk∆n Ztkn + e−βk(tn+1 −τℓ ) ejℓ , k = 1, . . . , K. (τℓ ,jℓ )∈En

Thus, the lifted state can be updated without knowing the kernel coefficients Qk (·), the baseline µ, the intensity λ, the original kernel Φ, or the state-dynamics coefficients b, σ, γ. The effect of the applied action on the unknown dynamics is learned from the observed transitions and rewards rather than computed from an explicit model. Algorithm 1 summarizes the resulting update.

25

Algorithm 1 Online Markov-state update for current-action readout Require: Decay scale β > 0; number of filters K; Hawkes dimension m; grid 0 = t0 < t1 < · · · < tN = T . Require: Observed state Xtn and event times/component labels (τℓ , jℓ ), where jℓ ∈ {1, . . . , m}. Ensure: Markov state StKn = (tn , Xtn , Zt1n , . . . , ZtKn ). 1: Initialize Z0k ← 0 ∈ Rm for k = 1, . . . , K. 2: for n = 0, 1, . . . , N − 1 do 3: Form StKn ← (tn , Xtn , Zt1n , . . . , ZtKn ). 4: Choose action an using the actor, for example an = πθ (StKn ) + εn . 5: Apply an on [tn , tn+1 ). 6: Observe reward sample Rn , next state Xtn+1 , and events En := {(τℓ , jℓ ) : tn < τℓ ≤ tn+1 }. Set ∆n ← tn+1 − tn . for k = 1, . . . , K do 9: Ztkn+1 ← e−βk∆n Ztkn . 10: for all (τℓ , jℓ ) ∈ En do 11: Ztkn+1 ← Ztkn+1 + e−βk(tn+1 −τℓ ) ejℓ . 12: end for 13: end for 14: Form StKn+1 ← (tn+1 , Xtn+1 , Zt1n+1 , . . . , ZtKn+1 ). 15: Store (StKn , an , Rn , StKn+1 , ∆n ) in the replay buffer. 16: end for 7:

8:

Remark 3.11 (Binned observations). If exact event times inside [tn , tn+1 ] are unavailable and only counts ∆Nn = Ntn+1 − Ntn are observed, one may use the endpoint approximation Ztkn+1 ≈ e−βk∆n Ztkn + ∆Nn . The event-exact update in Algorithm 1 is preferable whenever timestamps are available.

4

Controlled Hawkes and CT-DDPG

4.1

Hawkes-Markov Decision Process and value function

Throughout this section we fix an integer K. We consider a (time augmented) Markov Decision Process with state space given at any time t by dx m K YtK = (t, XtK,a , ZK t ) ∈ YK := [0, T ] × R × (R ) , K,1 where ZK , . . . , ZtK,K , the random variable XtK represents the observed value of X K t := Zt solving (8) at time t while Z K,i are derived from the observation of the event time, see Section 3.4 and Remark 3.11. For the sake of simplicity, we remove the superscript K in this section to alleviate the notations. We equip (Rm )K with the product norm

zK Z :=

K X

K

k=1

zk

1

.

For each i = 1, . . . , m, define m K eK i := (ei , . . . , ei ) ∈ (R ) ,

so that, at an event of type i, the memory vector changes from zK to zK + eK i . 26

Definition 4.1 (Deterministic policy and neural network parametrization). We denote by πζ : YK −→ A a deterministic Markov policy as a neural network parametrized by the weight vector ζ ∈ Z ⊂ Rp for some p ≥ 1. We denote by U the set of admissible policies π ζ that Assumption 3.2 is satisfied by choosing a = π ζ and there exists a locally bounded function ϖ1 : [0, ∞) → [0, ∞) ′ such that for all ζ ∈ Z, t ∈ [0, T ], and (t, x, zK ), (t′ , x′ , zK ) ∈ [0, T ] × Rdx × (Rm )K ,   ′ ′ |πζ (t, x, zK ) − πζ (t′ , x′ , zK )| ≤ ϖ1 (|ζ|) |t − t′ | + x − x′ + z − zK ZK

and |πζ (t, 0, 0)| ≤ ϖ1 (|ζ|). We denote by ρ ≥ 0 the reward discount rate. The fixed policy objective is given for any t ∈ [0, T ], zK = (z1 , . . . , zK ) ∈ (Rm )K by ζ JK (t, x, zK ) := Et,x,zK ;πζ

Z T

−ρ(s−t)

e



c(s, XsK , πζ (YsK )) ds + e−ρ(T −t) g(XTK )

.

t

The value function at time t starting at the point x, z is thus given by ζ VK (t, x, zK ) = inf JK (t, x, zK ). π ζ ∈U

We recall the infinitesimal operator defined above by using the policy π ζ , π ζ (t,x,z)

LK

f (t, x, z) = ∂t f (t, x, z) + b(t, x, π ζ (t, x, z)) · ∇x f (t, x, z) + −

K X

i 1 h ⊤ Tr σσ (t, x, π ζ (t, x, z))∇2xx f (t, x, z) 2

βk z k · ∇z k f (t, x, z)

k=1

+

m X

h i ζ ζ 1 K λK i (t, x, z, π (t, x, z)) f (t, x + γi (t, x, π (t, x, z)), z + ei , . . . , z + ei ) − f (t, x, z) .

i=1

Theorem 4.2. Fix K ≥ 1 and a deterministic Markov policy π ζ ∈ U. Suppose that   ζ JK ∈ C 1,2,1 [0, T ] × Rdx × (Rm )K where C 1,2,1 means once continuously differentiable in time, twice continuously differentiable in x, and once continuously differentiable in each memory variable zk . Assume moreover that there exist C > 0 and ℓ ∈ [1, p∗ − 2] such that, ζ ζ ζ ζ JK (u, x, zK ) + ∂t JK (u, x, zK ) + ∇x JK (u, x, zK ) + ∇zK JK (u, x, zK )   ℓ ζ + ∇xx JK (u, x, zK ) ≤ C 1 + ∥x∥ℓ + zK Z , K

(39)

ζ Then JK satisfies π ζ (t,x,zK ) ζ ζ JK (t, x, zK ) − ρJK (t, x, zK ) + c

LK



 t, x, π ζ (t, x, zK ) = 0,

(40)

for (t, x, zK ) ∈ [0, T ) × Rdx × (Rm )K , with terminal condition ζ JK (T, x, zK ) = g(x).

(41)

Conversely, any function v satisfying the same smoothness, growth, and integrability conditions ζ and solving (40)–(41) coincides with JK . 27

Remark 4.3. Note that (39) together with Assumptions 2.1(S1)-(S2) and 3.1 there exists a π ζ (t,x,zK ) ζ JK (t, x, zK )| ≤ C̃(1 + |x|p⋆ + ∥zK ∥p⋆ ).

constant C̃ > 0 such that |LK

Proof of Theorem 4.2. Fix (t, x, zK ) ∈ [0, T ] × Rdx × (Rm )K . For every s ∈ [t, T ], we have "Z #   s
ζ K −ρ(u−t) K ζ K −ρ(s−t) ζ K JK (t, x, z ) = Et,x,zK ;πζ e c Yu , π (Yu ) du + e JK Ys . (42) t

For n ≥ 1, define the local time n o τn := inf u ∈ [t, T ] : XuK + ZK ≥ n ∧ T. u Z K

From the definition of U, and by Assumption 3.2 we have   sup XuK + ZK <∞ u Z K

t≤u≤T

P − a.s.

Consequently, limn τn = T . By the jump-diffusion Itô formula on [t, s ∧ τn ] we get we obtain Z s∧τn  
ζ ζ K K K −ρ(u−t) K ζ K e−ρ(s∧τn −t) JK s ∧ τn , Xs∧τ , Z − J (t, x, z ) + e c Y , π (Y ) du s∧τn u u K n t " # Z s∧τn  

ζ aζu ζ K K K K K ζ K −ρ(u−t) du = e LK JK u, Xu− , Zu− − ρJK u, Xu− , Zu− + c Yu , π (Yu ) t

+ MK s∧τn , where Z s∧τn


⊤  ζ K K e−ρ(u−t) ∇x JK u, Xu− , ZK σ u, Xu− , π ζ (YuK ) dWu u− t " # m Z s∧τn     X
ζ ζ K K K fK,i , + e−ρ(u−t) JK u, Xu− + γi u, Xu− , π ζ (YuK ) , ZK − JK YuK dM u− +ei u

MK s∧τn :=

i=1

t

with fuK,i := NuK,i − N K,i − M t

Z u

  K K ζ K K λK s, X , Z , π (s, X , Z ) ds. i s− s− s s−

t

Then, taking the expectation under condition (39) together with (42) we get for any s ∈ [t, T ] "Z s∧τn E t

"

ζ ζ e−ρ(u−t) LaKu JK

 

ζ K K K K ζ K u, Xu− , ZK u− − ρJK u, Xu− , Zu− + c Yu , π (Yu )

#

# du = 0.

By the dominated convergence theorem using Remark 4.3 and Assumption 3.3 we deduce that   π ζ (t,x,zK ) ζ ζ LK JK (t, x, zK ) − ρJK (t, x, zK ) + c t, x, π ζ (t, x, zK ) = 0. Conversely, let a function v satisfying (40)–(41) and the same growth and integrability conditions as in (39). Applying Itô’s formula to
e−ρ(u−t) v u, XuK , ZK u

28

on [t, τn ], and taking expectations we get "Z τn

v(t, x, zK ) = Et,x,zK ;πζ



e−ρ(u−t) c u, XuK , aζu



t

#
du + e−ρ(τn −t) v τn , XτKn , ZK . τn

(43)

With τn ↑ T , using the polynomial-growth and the dominated convergence we get Z τn lim E

n→∞

e

−ρ(u−t)

c



u, XuK , π ζ (YuK )



Z T du = E

t

  e−ρ(u−t) c u, XuK , π ζ (YuK ) du,

t

and h h
i
i −ρ(T −t) K K lim E e−ρ(τn −t) v τn , XτKn , ZK = E e v T, X , Z τn T T n→∞ h i
= E e−ρ(T −t) g XTK . ζ Letting n → ∞ in (43) therefore yields v(t, x, zK ) = JK (t, x, zK ).

4.2

Advantage rate, performance and critic-advantage update

Definition 4.4 (Advantage rate). The advantage-rate function associated with policy π ζ for any action a is given by ζ ζ AζK (t, x, zK , a) := LaK JK (t, x, zK ) − ρJK (t, x, zK ) + c(t, x, a).

Theorem 4.5 (Martingale characterization of the value
and advantage rate). Fix K ≥ 1, ζ ∈ Z, and π ζ ∈ U. Let V ∈ C 1,2,1 [0, T ] × Rdx × (Rm )K satisfy the same polynomial-growth and integrability conditions as in Theorem 4.2. Let Q : [0, T ] × Rdx × (Rm )K × A −→ R be continuous with polynomial growth in x, zK of order p⋆ . Suppose further that   V(T, x, zK ) = g(x), Q t, x, zK , π ζ (t, x, zK ) = 0. (44) ζ ζ K For each (t, x, zK ) ∈ [0, T ) × Rdx × (Rm )K , let Ot,x,z K ⊂ A be a neighborhood of π (t, x, z ). ζ Assume that, for every a ∈ Ot,x,z K , there exists an admissible A-valued F-predictable process α = (αs )s∈[t,T ] such that lims↓t αs = a, P-a.s. Assume moreover that for s ∈ [t, T ], K Mst,x,z ;α := e−ρ(s−t) V

s, XsK,α , ZK,α s

Z s +

h

i e−ρ(u−t) c u, XuK,α , αu − Q u, XuK,α , ZK,α , α du, u u

t

(45) ζ is an F-martingale. Then for every (t, x, zK ) ∈ [0, T ] × Rdx × (Rm )K and a ∈ Ot,x,z K, ζ V(t, x, zK ) = JK (t, x, zK ),

Q(t, x, zK , a) = AζK (t, x, zK , a).

ζ Proof. For (t, x, zK ) ∈ [0, T ) × Rdx × (Rm )K and a ∈ Ot,x,z K . Let α be an exploratory control

satisfying the assumptions of the theorem. To simplify notation, write Xs := XsK,α , Zs := ZK,α s . By the jump–diffusion Itô formula, for every s ∈ [t, T ], e−ρ(s−t) V(s, Xs , Zs ) − V(t, x, zK ) Z s h i = e−ρ(u−t) LαKu V (u, Xu− , Zu− ) − ρV (u, Xu− , Zu− ) du + Ns , t

29

(46)

where N = (Ns )s∈[t,T ] is the local martingale Z s e−ρ(u−t) ∇x V (u, Xu− , Zu− )⊤ σ (u, Xu− , αu ) dWu Ns = t

+

m Z s X i=1

fK,α,i , e−ρ(u−t) V u, Xu− + γi (u, Xu− , αu ), Zu− + eK − V(u, Xu− , Zu− ) dM i u

t

with fK,α,i := N K,α,i − N K,α,i − M s s t

Z s

λK i (u, Xu− , Zu− , αu ) du.

t

Combining (46) with (45) yields K ;α K ;α Mt,x,z − Mt,x,z = s t

Z s

e−ρ(u−t) FV,Q (u, Xu− , Zu− , αu ) du + Ns ,

t

where FV,Q (t, x, zK , a) := LaK V(t, x, zK ) − ρV(t, x, zK ) + c(t, x, a) − Q(t, x, zK , a). K

By assumption, Mt,x,z ;α is a martingale, while N is a local martingale. Therefore, Z s  −ρ(u−t) e FV,Q (u, Xu− , Zu− , αu ) du t

s∈[t,T ]

is a local martingale, which has the continuous paths with finite-variation. Hence, we have Z s e−ρ(u−t) FV,Q (u, Xu− , Zu− , αu ) du = 0, s ∈ [t, T ], P-a.s. (47) t

Since the process is càdlàg, lim (Xu− , Zu− ) = x, zK , P-a.s. Together with lim αu = a, and the u↓t

u↓t

continuity of FV,Q , we get lim e−ρ(u−t) FV,Q (u, Xu− , Zu− , αu ) −→ FV,Q (t, x, zK , a), u↓t

P-a.s.

Setting s = t + h with h > 0, dividing (47) by h, and taking h ↓ 0, we obtain Z 1 t+h −ρ(u−t) 0 = lim e FV,Q (u, Xu− , Zu− , αu ) du h↓0 h t = FV,Q (t, x, zK , a). ζ Consequently, for every a ∈ Ot,x,z K,

Q(t, x, zK , a) = LaK V(t, x, zK ) − ρV(t, x, zK ) + c(t, x, a). Taking a = π ζ (t, x, zK ) in (48) and using (44), we obtain   π ζ (t,x,zK ) LK V(t, x, zK ) − ρV(t, x, zK ) + c t, x, π ζ (t, x, zK ) = 0, with terminal condition V(T, x, zK ) = g(x). Then Theorem 4.2 yields ζ V(t, x, zK ) = JK (t, x, zK ).

Substituting this identity into (48) and using Definition 4.4 gives Q(t, x, zK , a) = AζK (t, x, zK , a).

30

(48)

Connection with critic-advantage learning. Theorem 4.5 provides the identification principle used to train the value and advantage-rate critics. For a fixed policy π ζ , we approximate ζ JK by a value network parametrized by a vector of real numbers θ Vθ : [0, T ] × Rdx × (Rm )K −→ R, and introduce an action-dependent network parametrized by a vector of real numbers ψ Qψ : [0, T ] × Rdx × (Rm )K × A −→ R. The corresponding normalized advantage-rate approximator is   Qψ,ζ (t, x, zK , a) := Qψ (t, x, zK , a) − Qψ t, x, zK , π ζ (t, x, zK ) . It satisfies

(49)

  Qψ,ζ t, x, zK , π ζ (t, x, zK ) = 0,

ζ by construction, consistently with the normalization of AζK . For the exact pair (JK , AζK ), the discounted process in Theorem 4.5 is a martingale under locally exploratory controls with candidates V = Vθ and Q = Qψ,ζ . We therefore train (Vθ , Qψ,ζ ) by driving discrete empirical increments of this process toward zero. The resulting multi-step martingale residual and advantage-critic loss are defined in Subsection 4.4.

4.3

Policy loss and actor update.

From Theorem 4.2 we recall that AζK (t, x, zK , π ζ (t, x, zK )) = 0. ′

Lemma 4.6. Let π ζ , π ζ be two deterministic Markov policies parameterized by ζ, ζ ′ ∈ Z and let ζ the growth conditions in Theorem 4.2 be satisfied for JK . Then ′

ζ ζ JK (t, x, zK ) − JK (t, x, zK ) = Et,x,z;πζ ′

Z T t

 ′ ′ ′ e−ρ(u−t) AζK (Yuζ , π ζ (Yuζ )) du .

Proof. Using a similar localization in the proof of Theorem 4.2, we get h i ′ ζ Et,x,z;πζ ′ e−ρ(T −t) g(YTζ ) − JK (t, x, zK )   i ′ hZ T ′ π ζ (Yuζ ) ζ ζ −ρ(u−t) ζ′ ζ′ ′ = Et,x,zK ;πζ e LK JK (Yu ) − ρJK (Yu ) du . t

Therefore, by adding Et,x,z;πζ ′

hR

′ ′ ′ T −ρ(u−t) c(Yuζ , π ζ (Yuζ )) du t e

i

to both sides we get the equality.

Assumption 4.1. Fix K ≥ 1 and ζ ∈ Z. There exist an open neighborhood Oζ ⊂ Z of ζ, a constant Cζ > 0, and exponents qπ , qA ≥ 0 such that qπ + qA < p⋆ and the following conditions hold. 1. (ζ, y) 7−→ π ζ (y) is continuous on Oζ × YK ; for every y ∈ YK , the map ζ 7−→ π ζ (y) is continuously differentiable, and (ζ, y) 7−→ Dζ π ζ (y) is continuous on Oζ × YK . Moreover, π ζ (y) ∈ int(A) and
Dζ π ζ (y) ≤ Cζ 1 + ∥y∥qπ , (ξ, y) ∈ Oζ × YK .

31

2. Y ζε −→ Y ζ in (P ⊗ dt)-measure on Ω × [0, T ], for any ζε −→ ζ when ε −→ 0 3. For every y ∈ YK , the map a 7−→ AζK (y, a) admits a continuously differentiable extension  to an open neighborhood of π ζ (y) : ζ ∈ Oζ . Moreover, the map (y, a) 7−→ ∇a AζK (y, a) is continuous on the corresponding domain and

∇a AζK y, π ζ (y) ≤ Cζ 1 + ∥y∥qA , (ζ, y) ∈ Oζ × YK . Theorem 4.7. Under Assumption 4.1 Z T  ζ ζ ⊤ ζ ζ ζ K −ρt ζ ζ ∇ζ JK (x, z ) = E e Dζ π (Yt ) ∇a AK (Yt , π (Yt )) dt . 0

Proof. Fix ζ ∈ Z, ε > 0 and a direction η ∈ Rr where ε > 0 is sufficiently small so that ζε := ζ + εη is in Oζ . From Lemma 4.6, we compute ζε ζ h JK (x, zK ) − JK (x, zK ) = E0,x,zK ,;πζε ε

Z T e

ζ ζε ζε −ρt AK (Yt , πζε (Yt ))

0

ε

i dt .

(50)

Recall that AζK (y, πζ (y)) = 0 holds for every state y. Therefore AζK (Ytζε , πζ (Ytζε )) = 0, and hence

AζK (Ytζε , πζε (Ytζε )) Aζ (Y ζε , πζε (Ytζε )) − AζK (Ytζε , πζ (Ytζε )) = K t . ε ε

Since a 7−→ AζK (y, a) is continuously differentiable from Assumption 4.1, by the fundamental theorem of analysis with respect to the action variable, for any y, we have Z 1  π ζε (y) − π ζ (y)  AζK (y, πζε (y)) − AζK (y, πζ (y)) = ∇a AζK (y, πζ+εrη (y))⊤ dr, ε ε 0 where the derivative with respect to a is understood with respect to the neural network parameterizing the action variable. Hence, ζε ζ JK (x, zK ) − JK (x, zK ) ε" # Z T Z 1  ⊤ π ζε (Y ζε ) − π ζ (Y ζε ) t t e−ρt ∇a AζK Ytζε , π ζ+rεη (Ytζε ) × dr dt . = E0,x,zK ;πζε ε 0 0

Recall from Assumption 4.1 that π ζε (y) − π ζ (y) −→ Dζ π ζ (y)η, with π ζ+rεη (y) −→ π ζ (y) ε→0 ε→0 ε uniformly on compact sets. Moreover, there exists some constant Cζ′ > 0 such that  ⊤ π ζε (y) − π ζ (y) ∥∇a AζK y, π ζ+rεη (y) × ∥ ≤ Cζ′ (1 + ∥y∥qπ +qA ). ε Since qπ + qA < p⋆ , one can choose q̃ > 1 such that q̃(qπ + qA ) ≤ p⋆ . Therefore, hZ T  ⊤ π ζε (Y ζε ) − π ζ (Y ζε ) i t t sup E ∥∇a AζK Ytζε , π ζ+rεη (Ytζε ) × ∥q̃ dt < ∞. ε 0 ε>0,ζε ∈Oζ , r∈[0,1] 32

  ⊤  π ζε (Ytζε )−π ζ (Ytζε ) × Then, the family ∇a AζK Ytζε , π ζ+rεη (Ytζε ) ε

ε>0,ζε ∈Oζ , r∈[0,1]

is uniformly

integrable. By Vitali’s convergence theorem, taking the limit when ε −→ 0 in (50) we deduce that Z T    ζ ζ ⊤ ζ ζ ζ K −ρt ζ ζ ∇ζ JK (x, z ) = E0,x,zK ;πζ e Dζ π (Yt ) ∇a AK Yt , π (Yt ) dt . 0

Connection with actor learning. Theorem 4.7 provides the theoretical basis for the actor update. Once the advantage-rate critic has been trained locally around the current policy action, we use     ∇a Qψ t, x, zK , π ζ (t, x, zK ) ≈ ∇a AζK t, x, zK , π ζ (t, x, zK ) . From (49) we get ∇a Qψ,ζ (t, x, zK , a) = ∇a Qψ (t, x, zK , a). ζ Substituting these results yields a sample-based approximation of ∇ζ JK and the precise actor loss and its joint implementation with the critic update are given in Subsection 4.4.

4.4

Full Hawkes CT-DDPG Algorithm

The numerical implementation follows the CT-DDPG method of [16] extended to Hawkes processes with additional terms in the advantage rate function. We extend the algorithm to the
K K K lifted Hawkes state Yn := tn , Xtn , Ztn , where the memory filters ZK tn are updated from the observed event times according to Algorithm 1. We therefore record only the main critic and actor updates. Let Vθ : YK −→ R be the value network and let Qψ : YK × A −→ R be the raw action-dependent advantage-rate network. For a fixed actor π ζ , let Qψ,ζ (y, a) be the normalized advantage-rate critic defined in (49)   Qψ,ζ (y, a) := Qψ (y, a) − Qψ y, π ζ (y) ,
with Qψ,ζ y, π ζ (y) = 0, consistently with the normalization of AζK . For simplicity, we consider a time-discretization (tk )k≥0 with t0 = 0 and path h > 0, that is tn+1 − tn = h. Given a contiguous L-step trajectory segment sampled from the replay buffer, where L ≥ 1 denotes the multi-step TD horizon, define θ,ψ,ζ δn,L := e−ρLh Vbn+L − Vθ (YnK ) + h

L−1 X

h  
i K e−ρℓh c tn+ℓ , XtKn+ℓ , an+ℓ − Qψ,ζ Yn+ℓ , an+ℓ ,

ℓ=0

where

( g(XTK ), tn+L = T, Vbn+L := K Vθ− (Yn+L ), tn+L < T,

and Vθ− denotes a target value network. If the replay buffer stores the cost accumulated over an interval rather than a cost rate, the corresponding interval cost replaces h c(tn , XtKn , an ). For a batch B of replayed path segments, the critic loss is Lcritic (θ, ψ) :=

  2 1 X 1 X θ,ψ,ζ 2 δn,L + λT Vθ T, XTK,b , ZK,b − g(XTK,b ) , T |B| |BT | n∈B

b∈BT

where BT is a batch of terminal states. The critic parameters are updated by θ ← θ − ηθ ∇θ Lcritic ,

ψ ← ψ − ηψ ∇ψ Lcritic . 33

During this update, the actor and target-network parameters are held fixed. On the other hand, Theorem 4.7 motivates the actor loss   1 X −ρtb Lactor (ζ) := e Qψ YbK , π ζ (YbK ) , (51) |BA | b∈BA

where BA is a batch of nonterminal lifted states. Indeed, holding the sampled states and ψ fixed, ∇ζ Lactor (ζ) =

  1 X −ρtb e Dζ π ζ (YbK )⊤ ∇a Qψ YbK , π ζ (YbK ) . |BA | b∈BA

The raw network Qψ is used in (51), since fully differentiating the normalized quantity Qψ,ζ (y, π ζ (y)) would give zero. Because the present problem is formulated as cost minimization, the actor update is ζ ← ζ − ηζ ∇ζ Lactor (ζ). Replay sampling, exploration noise, target networks, and Polyak averaging are implemented as in [16]. The complete procedure is summarized in Algorithm 2. Algorithm 2 Hawkes CT-DDPG Require: Actor π ζ , value critic Vθ , raw advantage critic Qψ , target networks, replay buffer D, and filter parameters (β, K). 1: for each episode do 2: Initialize X0K and ZK 0 = 0. 3: for n = 0, . . . , N − 1 do 4: Form YtK = (tn , XtKn , ZK tn ). n 5:

Select an exploratory action
an = ProjA π ζ (YtK ) + ε . n n

Apply an , and observe the interval cost, XtKn+1 , and the events occurring on (tn , tn+1 ]. K 7: Update ZK tn+1 using Algorithm 1, form Yn+1 , and store the transition in D. 8: Sample replayed path segments and update (θ, ψ) by minimizing Lcritic . 9: Sample nonterminal lifted states and update ζ by minimizing Lactor . 10: Update the target networks by Polyak averaging. 11: end for 12: end for 6:

5

Numerical illustrations

We now illustrate our result by comparing the Hawkes CT-DDPG efficiency in three settings for the Hawkes kernels. For the sake of simplicity, we work in the one-dimensional setting, that is m = dx = dW = 1. We thus set a generative model to test the efficiency of our algorithm. The dynamics of the outcome X is given by
 
dXt = b0 − κXt − ba at dt + σ0 + σx Xt + σa at + dWt + clip γ0 + γx Xt− + γa at , γmin , γmax dNt , where N is a Hawkes process with intensity Z t− ϕ(t − s) dNs ,

λt = µ(t, Xt− , at ) + α Q(at ) 0

34

with
µ(t, X, a) = clip µ0 + µx X + µa a, µmin , µmax , and Q(a) = 1 −

ceff a . ahalf + a

Q represents a saturating Hill-type attenuation kernel. The control a reduces the excitation gain, with half-effect at a = ahalf , and maximal reduction parameter ceff . Regarding the stochastic control optimization parameters, we choose ct = cx Xt2 + ca a2t ,

g(XT ) = c̃XT2 .

The numerical setup with choices of values for each parameter is given in Appendix A. We recall that these dynamics and all the introduced parameters remain unknown for the Hawkes CT-DDPG method and are only set for data generation purposes and comparison with modeldependent methods (named the Oracle when Markovian solutions are available). The three scenarios tested are described below. • Single-exponential kernel. In this case, ϕexp (τ ) = e−ρexp τ , with ρexp > 0, and the system is Markovian so that our Markovianization procedure is exact. We compare Hawkes CT-DDPG with an Oracle reducing the stochastic control problem to an HJB equation with state variable X, λ similarly to [8, 14]. • Erlang Kernel. We consider ϕErlang (τ ) = ρer τ e−ρer τ , with ρer > 0. In particular, the system can still be Markovianized introducing two auxiliary processes L1 , L2 with dynamics dL1t = −ρer L1t dt + dNt ,

dL2t = ρer L1t − L2t dt.

Then, a λat = µ(t, Xt− , at ) + α Q(at ) L2t− .

Here α is the excitation-scale parameter in the general intensity specification above; it is not generated by the Erlang Markovianization. We refer to [19] for more details. • Power-law kernel. This fully non-Markovian kernel is given by ϕ(τ ) = hη h (τ + η)−(1+h) , where τ, η, h > 0. This kernel has infinite memory and is not Markovian. Known-parameter HJB benchmarks. To separate the error arising from model-free learning from the difficulty of the underlying control problem, we compare the learned policies with model-based benchmarks that are given full knowledge of the environment parameters, including the coefficients b, σ, γ, µ, the Hawkes kernel and its control dependence, and the cost functions. For the single-exponential kernel, the Hawkes memory admits an exact one-factor Markov representation. For the Erlang kernel, it admits an exact two-phase Markov representation with auxiliary states (L1 , L2 ). In these two experiments, we formulate the corresponding finitedimensional HJB equation and solve it numerically using the Deep Galerkin Method introduced in [50]. We refer to the resulting feedback policies as the known-parameter DGM/HJB oracles, up to the numerical approximation error of the HJB solver. The power-law kernel does not admit an exact finite-dimensional Markov representation. In this case, we first approximate the true kernel by a finite exponential mixture, mix

φ

(τ ) =

M X

wj e−βj τ ,

j=1

construct the corresponding finite-dimensional Markov state, and solve the HJB equation of this approximating model using DGM. Since this benchmark contains both kernel-approximation error and numerical HJB error, it is referred to below as the mixture DGM/HJB benchmark,

35

Figure 1: Single Kernel Environment: Costs Comparison and Sanity Check. Exact-Oracle (red, left), CT-DDPG (blue, center), and validation-selected static strategy (grey, right). rather than an exact oracle for the original power-law problem. All benchmark and learned policies are evaluated using the same Monte Carlo testing protocol. All these kernels are tested with Hawkes CT-DDPG combining Algorithm 1 (Markovianization) and Algorithm 2 (Hawkes CT-DDPG). We compare our results with an Oracle, the best static strategy and two other reinforcement learning methods in discrete-time: Soft Actor-Critic (SAC), see [27] and Deep Deterministic Policy Gradient (DDPG), see [49]. Finally, we also emphasize the effect of the Markovianization Algorithm 1. We compare the Hawkes CT-DDPG algorithm without taking into account the memory (current state, Algorithm 2 only) with the Hawkes CT-DDPG Markovianized (Algorithms 1 and 2, named filtered) for Erlang and power-law kernels. The exact numerical values for each experiment with confidence intervals are given in Appendix A. Single-exponential Figure 1: sanity check with Oracle comparison. The true exponential kernel is represented exactly by one member of the preselected filter. We note that CT-DDPG attains cost 0.1744, reducing cost by 46.3% relative to the validation-selected static control a ≡ 0.39, whose cost is 0.3247. The known-parameter DGM/HJB oracle, constructed from the exact one-factor Markov representation, remains better at 0.1658, as expected. This experiment therefore checks that the observed-filter implementation can exploit an exact finite-dimensional memory state; it is not a comparison against SAC or standard DDPG, and it does not contain a current-state ablation. Erlang kernel Figure 2: Oracle vs. Hawkes CT-DDPG vs discrete time reinforcement learning. The oracle exploits the two-phase Markovianization [19] reducing the problem to a standard stochastic control optimization with state (X, L1 , L2 ), while our Markovianization in Algorithm 2 is applied to a 12-dimensional exponential filter bank. The DGM/HJB oracle remains as expected the best overall policy, with cost 0.1037 and is set as benchmark for the efficiencies of Hawkes CT-DDPG. The filtered Hawkes CT-DDPG (with cost 0.1056) performs better than static, SAC and DDPG and remains the closest to the oracle. Relative to currentstate observations, filtering reduces mean cost by 13.4% for CT-DDPG; 2.55% for SAC, and 12.9% for DDPG. Filtered CT-DDPG is numerically the best learned policy, improving on filtered SAC and filtered DDPG by 0.89% and 4.31%, respectively. Power-law kernel Figure 3: Hawkes CT-DDPG vs. discrete time reinforcement learning. Because the power-law model has no exact finite-dimensional Markov representation, no exact HJB oracle is available. We instead report the mixture DGM/HJB benchmark obtained by first approximating the power-law kernel with a finite exponential mixture and then solving the corresponding finite-dimensional HJB equation by DGM. The learning methods observe a 20-dimensional exponential filter approximation. Filtering reduces mean cost by 4.67% for 36

Figure 2: Erlang Kernel Environment Costs Comparison. Top from left to right: oracle (red), best static, CT-DDPG no Markovianization (current) and with Markovianization (filtered), SAC current and filtered and DDPG current and filtered. Bottom: value and confidence intervals for CT-DDPG current and filtered, SAC current and filtered, DDPG current and filtered.

37

Figure 3: Power Kernel Environment Costs Comparison. Top from left to right: mixture oracle (red), best static, CT-DDPG no Markovianization (current) and with Markovianization (filtered), SAC current and filtered and DDPG current and filtered. Bottom: value and confidence intervals for CT-DDPG current and filtered, SAC current and filtered, DDPG current and filtered. CT-DDPG, 4.82% for SAC, and 1.47% for DDPG. Filtered CT-DDPG again has the lowest cost among learned policies. Its cost is 6.96% below filtered SAC and 8.85% below filtered DDPG, while the mixture DGM/HJB oracle remains lower at 1.5696. Cross-experiment conclusion. The single-exponential experiment serves as a sanity check, confirming that Hawkes CT-DDPG can effectively exploit a finite-dimensional Markov representation. The Erlang and power-law experiments provide more demanding tests: both models are non-Markovian in the physical state alone, although the Erlang kernel admits an exact finite-dimensional Markov lift, whereas the power-law kernel requires an exponential-mixture approximation. In both settings, augmenting the physical state with observable exponential filters improves the performance of CT-DDPG, SAC, and DDPG relative to their state-only counterparts. Moreover, when all model-free methods are supplied with the same preselected filter bank and filter-update rule, Hawkes CT-DDPG achieves the lowest mean cost in both experiments. These results provide consistent evidence that the proposed method is well suited to the control of Hawkes-driven systems with unknown dynamics and kernel parameters.

References [1] Eduardo Abi Jaber and Omar El Euch. Multifactor approximation of rough volatility models. SIAM journal on financial mathematics, 10(2):309–349, 2019. [2] Isabel Agostino and Thibaut Mastrolia. Approximation of singular-stopping control driven by hawkes processes via rescaled mdps. arXiv preprint arXiv:2602.05025, 2026.

38

[3] Emmanuel Bacry, Iacopo Mastromatteo, and Jean-François Muzy. Hawkes processes in finance. Market Microstructure and Liquidity, 1(01):1550005, 2015. [4] Adrian Baldwin, Iffat Gheyas, Christos Ioannidis, David Pym, and Julian Williams. Contagion in cyber security attacks. Journal of the Operational Research Society, 68(7):780–791, 2017. [5] David R. Baños, Salvador Ortiz-Latorre, and Oriol Zamora Font. Change of measure in a heston–hawkes stochastic volatility model. Advances in Applied Probability, 56(4):1370– 1399, 2024. [6] Ali Baouan, Sebastien Coustou, Mathieu Lacome, Sergio Pulido, and Mathieu Rosenbaum. Crediting football players for creating dangerous actions in an unbiased way: the generation of threat (got) indices. Journal of Quantitative Analysis in Sports, 2025. [7] Paul R. Beesack. Comparison theorems and integral inequalities for volterra integral equations. Proceedings of the American Mathematical Society, 20(1):61–66, 1969. [8] Alain Bensoussan and Benoı̂t Chevalier-Roignant. Stochastic control for diffusions with selfexciting jumps: An overview. Mathematical Control and Related Fields, 14(4):1452–1476, 2024. [9] Yannick Bessy-Roland, Alexandre Boumezoued, and Caroline Hillairet. Multivariate hawkes process for cyber insurance. Annals of Actuarial Science, 15(1):14–39, 2021. [10] Tomasz R. Bielecki, Jacek Jakubowski, and Mariusz Niewȩglowski. Construction and simulation of generalized multivariate hawkes processes. Methodology and Computing in Applied Probability, 24(4):2865–2896, May 2022. [11] Tomasz R. Bielecki, Jacek Jakubowski, and Mariusz Niewȩglowski. Multivariate Hawkes processes with simultaneous occurrence of excitation events coming from different sources. Stochastic Models, 39(3):537–565, November 2022. [12] Alessandro Bondi, Sergio Pulido, and Simone Scotti. The rough hawkes heston stochastic volatility model. Mathematical Finance, 34(4):1197–1241, 2024. [13] Pierre Brémaud and Laurent Massoulié. Stability of nonlinear hawkes processes. The Annals of Probability, pages 1563–1588, 1996. [14] Giorgia Callegaro, Claudio Fontana, Caroline Hillairet, and Beatrice Ongarato. A stochastic Gordon–Loeb model for optimal cybersecurity investment under clustered attacks. arXiv:2505.01221, 2025. [15] Álvaro Cartea, Samuel N Cohen, and Saad Labyad. Gradient-based estimation of linear hawkes processes with general kernels. arXiv preprint arXiv:2111.10637, 2021. [16] Ziheng Cheng, Xin Guo, and Yufei Zhang. Deterministic policy gradient for reinforcement learning with continuous time and state. arXiv:2509.23711, 2026. [17] Laure Coutin, Benjamin Massat, and Anthony Réveillac. Quantification of limit theorems for hawkes processes. arXiv preprint arXiv:2503.21273, 2025. [18] Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1):219–245, 2000. [19] Aline Duarte, Eva Löcherbach, and Guilherme Ost. Stability, convergence to equilibrium and simulation of non-linear hawkes processes with memory kernels given by the sum of erlang kernels. ESAIM: Probability and Statistics, 23:770–796, 2019. 39

[20] Elisabeth V Dubois, Omer F Keskin, and Unal Tatar. Cyber risk modeling methods and data sets: A systematic interdisciplinary literature review for actuaries. Society of Actuaries Research Institute, 2022. [21] Meyer Dwass. The total progeny in a branching process and a related random walk. Journal of Applied Probability, 6(3):682–686, 1969. [22] Omar El Euch and Mathieu Rosenbaum. The characteristic function of rough heston models. Mathematical Finance, 29(1):3–38, 2019. [23] Paul Embrechts, Thomas Liniger, and Lu Lin. Multivariate hawkes processes: an application to financial data. Journal of Applied Probability, 48(A):367–378, 2011. [24] Eymen Errais, Kay Giesecke, and Lisa R Goldberg. Affine point processes and portfolio credit risk. SIAM Journal on Financial Mathematics, 1(1):642–665, 2010. [25] Alberto Gennaro, Thibaut Mastrolia, and Francesca Primavera. Signature methods for optimal market making. arXiv preprint arXiv:2606.19772, 2026. [26] Julius Graf and Thibaut Mastrolia. Learning market making with closing auctions. arXiv preprint arXiv:2601.17247, 2026. [27] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018. [28] Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83–90, 1971. [29] Alan G Hawkes. Hawkes processes and their applications to finance: a review. Quantitative Finance, 18(2):193–198, 2018. [30] Alan G. Hawkes and David Oakes. A cluster process representation of a self-exciting process. J. Appl. Probability, 11:493–503, 1974. [31] Caroline Hillairet, Thomas Peyrat, and Anthony Réveillac. Multivariate self-exciting processes with dependencies. arXiv preprint arXiv:2503.15958, 2025. [32] Caroline Hillairet and Anthony Réveillac. Explicit correlations for the hawkes processes. Stochastics, pages 1–29, 2025. [33] Caroline Hillairet, Anthony Réveillac, and Mathieu Rosenbaum. An expansion formula for hawkes processes and application to cyber-insurance derivatives. Stochastic Processes and their Applications, 160:89–119, 2023. [34] Ulrich Horst and Wei Xu. The microstructure of stochastic volatility models with self-exciting jump dynamics. The Annals of Applied Probability, 32(6):4568–4610, 2022. [35] Thibault Jaisson and Mathieu Rosenbaum. Limit theorems for nearly unstable hawkes processes. The Annals of Applied Probability, 25(2):600–631, 2015. [36] Thibault Jaisson and Mathieu Rosenbaum. Rough fractional diffusions as scaling limits of nearly unstable heavy tailed hawkes processes. The Annals of Applied Probability, 26(5):2860–2882, 2016. [37] Paul Jusselin. Optimal market making with persistent order flow. SIAM Journal on Financial Mathematics, 12(3):1150–1200, 2021. 40

[38] Paul Jusselin and Mathieu Rosenbaum. No-arbitrage implies power-law market impact and rough volatility. Mathematical Finance, 30(4):1309–1336, 2020. [39] David W Kammler. Approximation with sums of exponentials in lp[0, ∞). Journal of Approximation Theory, 16(4):384–408, 1976. [40] Mahmoud Khabou, Nicolas Privault, and Anthony Réveillac. Normal approximation of compound hawkes functionals. Journal of Theoretical Probability, 37(1):549–581, 2024. [41] Mahmoud Khabou and Mehdi Talbi. Markov approximation for controlled Hawkes jumpdiffusions with general kernels. arXiv:2507.11294, 2025. [42] Junhyeon Kwon, Yingcai Zheng, and Mikyoung Jun. Flexible spatio-temporal hawkes process models for earthquake occurrences. Spatial Statistics, 54:100728, 2023. [43] Peter AW Lewis and GS Shedler. Simulation of nonhomogeneous poisson processes with log linear rate function. Biometrika, 63(3):501–505, 1976. [44] Yosihiko Ogata. On lewis’ simulation method for point processes. IEEE transactions on information theory, 27(1):23–31, 1981. [45] Beatrice Ongarato et al. Hawkes and affine processes in risk modeling: Theory and applications in finance and cybersecurity. 2026. [46] Tristan Pace and Gordan Zitkovic. Convergence of nonhomogeneous hawkes processes and feller random measures. arXiv preprint arXiv:2412.15999, 2024. [47] Philip E. Protter. Stochastic Integration and Differential Equations. Springer, Berlin, second edition, 2005. [48] Marian-Andrei Rizoiu, Swapnil Mishra, Quyu Kong, Mark Carman, and Lexing Xie. Sir-hawkes: Linking epidemic models and hawkes processes to model diffusions in finite populations. In Proceedings of the 2018 world wide web conference, pages 419–428, 2018. [49] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387–395. Pmlr, 2014. [50] Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, December 2018. [51] J. C. Tanner. A derivation of the borel distribution. Biometrika, 48(1/2):222–224, 1961. [52] Nicholas J Van Der Elst and Emily E Brodsky. Connecting near-field and far-field earthquake triggering to dynamic strain. Journal of Geophysical Research: Solid Earth, 115(B7), 2010. [53] Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1–34, 2020. [54] Xiao-Wei Zhang, Peter W Glynn, Kay Giesecke, and Jose Blanchet. Rare event simulation for a generalized hawkes process. In Proceedings of the 2009 Winter Simulation Conference (WSC), pages 1291–1298. IEEE, 2009.

41

A

Numerical Setup

The simulator reports discounted reward internally, whereas the tables and figures below report the equivalent positive discounted cost. Thus, lower values indicate better policies. All Monte Carlo comparisons use common test seeds within an experiment. Table 1: High-level experiment definitions. Item

Single exponential

True kernel

ϕ(u) = e−1.30u

True Markovization

CT-DDPG memory state DGM/HJB oracle

Generic RL baselines

Erlang kernel

Power-law kernel

ϕ(u) = ρue−ρu , ρ = 1.15 (the scale is carried by α) Exact, one exponential Exact, two Erlang memory is included in the auxiliary states Y 1 , Y 2 filter bank

ϕ(u) = bη b (u + η)−(1+b) , η = 0.12, b = 0.80 No finite exact Markovization; simulator uses event-history power-law memory Exponential filter bank Z k , Exponential filter bank Z k , Exponential filter bank Z k , K=8 K = 12 K = 20 Known-parameter exact Known-parameter exact Known-parameter single-exponential Markov Erlang Markov state exponential-mixture state approximation with positive memory link Not used in this SAC and DDPG with SAC and DDPG with sanity-check experiment current-state and filtered current-state and filtered observations observations

Table 2: Environment and objective parameters. Parameter

Single exponential

Erlang kernel

Power-law kernel

Horizon T Time step ∆t Discount rate βdisc Baseline µ(t, x, a)

5.0 0.02 0.02 clip(2.05 + 0.10x − 0.08a, 10−6 , 6) 1.25 ca = 1.25, a1/2 = 0.35 0.325k, k = 1, . . . , 8 0.02 − 0.45x − 0.72a [0.05 + 0.015x + 0.08a]+ clip(0.10 + 0.015x − 0.015a, 10−4 , 0.25) 0.80

5.0 0.02 0.02 clip(2.00 + 0.03x − 0.04a, 10−6 , 5) 1.05 ca = 1.30, a1/2 = 0.35 0.23k, k = 1, . . . , 12 0.02 − 0.45x − 0.70a [0.05 + 0.012x + 0.08a]+ clip(0.085 + 0.005x − 0.010a, 10−4 , 0.20) 0.3000

8.0 0.02 0.02 clip(0.60 + 0.002x − 0.002a, 10−6 , 5) 0.99 ca = 1.08, a1/2 = 0.10 1.00k, k = 1, . . . , 20 −1.00x − 0.02a [0.03 + 0.002x + 0.005a]+ clip(0.50 + 0.002x − 0.05a, 10−4 , 0.90) 1.00

0.18

0.2250

0.50

0.60

0.2250

0.50

Excitation scale α Control effect q(a) Filter decays Drift b(x, a) Volatility σ(x, a) Jump size γ(x, a) State running-cost coefficient cx Control running-cost coefficient ccost a Terminal-cost coefficient cT

For a transparent sufficient-condition check, let q̄ = supa∈[0,1] q(a) = 1. On [0, T ], the true

42

excitation masses are Z 5 Mexp = 1.25

e−1.30u du = 0.96009,

0

Z 5 MErlang = 1.05

1.15ue−1.15u du = 0.89343,

0

Z 8 Mpower = 0.99

0.80(0.12)0.80 (u + 0.12)−1.80 du = 0.95601.

0

All are strictly below one. The signed exponential approximations are checked through their positive-link envelope, whose corresponding masses are also below one as reported in the table. Together with compact controls, bounded baseline and jump coefficients, finite horizon, and the displayed Lipschitz coefficients, this gives sufficient conditions in Proposition 3.6, which verify the moment requirement in Assumption 3.2 (with the numerical diagnostic using p∗ = 4). Since the quadratic costs have growth index q = 1, this also gives p∗ > q + 1 = 2. Table 3: Mean discounted cost with 90% Monte Carlo intervals. Lower is better. The intervals measure rollout uncertainty conditional on one trained policy; bold values identify the best model-free learned policy in each comparison. Policy

Single exponential

Erlang kernel

Power-law kernel

DGM/HJB oracle CT-DDPG current CT-DDPG filtered SAC current SAC filtered DDPG current DDPG filtered Validation-selected static

0.1658 ± 0.0070 – 0.1744 ± 0.0073 – – – – 0.3247 ± 0.0214

0.1037 ± 0.0046 0.1219 ± 0.0067 0.1056 ± 0.0049 0.1093 ± 0.0050 0.1065 ± 0.0046 0.1266 ± 0.0051 0.1103 ± 0.0050 0.1314 ± 0.0060

1.5696 ± 0.0617 1.6647 ± 0.0634 1.5870 ± 0.0607 1.7920 ± 0.0665 1.7056 ± 0.0641 1.7669 ± 0.0644 1.7410 ± 0.0680 1.9242 ± 0.0734

In the single-exponential column, CT-DDPG filtered denotes the proposed CT-DDPG supplied with the eight-filter bank. Because the fourth decay equals the true decay, this bank contains an exact Markov representation. SAC and standard DDPG are not included in this sanity-check experiment. Implementation and Reporting Notes. All neural policies are trained using normalized observations, bounded controls, minibatch optimization, gradient clipping, and a replay buffer following an initial data-collection period. Slowly updated target value networks are used for stability, with target actor networks additionally employed in the Erlang and power-law experiments, and actor updates are made more conservatively than critic updates when necessary. The reported actor skip terms are direct affine contributions from the current state and selected summaries of the observed filter bank to the actor logit; they provide a stable policy initialization but introduce neither additional observations nor knowledge of the true kernel. SAC and DDPG are trained with replay-based off-policy updates, vectorized simulation, exploration noise where appropriate, and exactly the same current-state or filtered observations used in the corresponding comparison. The DGM oracle is trained by sampling the HJB domain, with additional reachable-state sampling for the power-law approximation. During training, policies are periodically evaluated without exploration on held-out trajectories, and the checkpoint with the lowest validation cost is restored; validation episodes are used only for model selection and are separate from the final Monte Carlo sample. Warm starts, observation normalization, and reward rescaling are used only as numerical stabilization devices, while all reported results are evaluated on independent common-random trajectories and expressed in the original, unscaled cost units. 43

Related documents

• Continuous control with deep reinforcement learning#7158
• Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning#974961
• Adaptive Laser Welding Control: A Reinforcement Learning Approach#448950
• Bayesian reinforcement learning with Dirichlet processes#291862
• Online Material Estimation for Conditioned Diffusion Policy in Shaping Deformable Linear Objects#1070120