Global Convergence of DGM and PINN Algorithms for Solving Nonlinear PDEs

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Global Convergence of DGM and PINN Algorithms for Solving Nonlinear PDEs

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

Global Convergence of DGM and PINN Algorithms for Solving Nonlinear PDEs Justin Sirignano∗, Konstantinos Spiliopoulos†, and Samuel Cohen‡

arXiv:2607.24726v1 [cs.LG] 27 Jul 2026

June 2026

Abstract The Deep Galerkin Method (DGM) and Physics Informed Neural Networks (PINNs) have become widely-used methods for solving partial differential equations (PDEs) in the rapidly growing field of scientific machine learning. In these methods, a neural network is trained to approximate the PDE solution by using (stochastic) gradient descent to minimize the PDE residual of the neural network. Due to the non-convexity of the PDE residual objective function, the trained neural network may, in principle, only converge to a local minimizer of the objective function (which would not be a solution of the PDE). Therefore, there is a longstanding question regarding the mathematical foundations of these algorithms, and it is highly valuable to establish that the trained neural network will converge to the PDE solution. For a class of semi-linear PDEs (nonlinear in the solution and its first derivative), we prove that neural networks trained with gradient descent to minimize the PDE residual objective function will converge to the PDE solution.

Introduction

The Deep Galerkin Method (DGM, [4]) and Physics Informed Neural Networks (PINNs, [3]) methods solve Partial Differential Equations (PDEs) using neural networks. Both methods share a common objective function, the PDE residual of the neural network, which is used to train the neural network to approximate the solution of the PDE. Specifically, the neural network parameters are trained with (stochastic) gradient descent to minimize the PDE residual. These methods have become widely-used in scientific machine learning across many diverse fields including fluid dynamics, physics, financial mathematics, biology, and optimal control. The PDE residual of a neural network is a non-convex objective function (of the trainable parameters) and therefore, in principle, the trained neural network ∗ Mathematical Institute, University of Oxford

† Department of Mathematics and Statistics, Boston University ‡ Mathematical Institute, University of Oxford

may converge to a local minimizer which is not the PDE solution. This would of course be a major limitation to using neural networks as a numerical method for PDE applications. It has been recently proven that DGM/PINNs trained with gradient descent will globally converge to the solution of linear elliptic PDEs [2]. The mathematical proof in [2] cannot however be applied to nonlinear PDEs. In this paper, for a class of nonlinear PDEs, we prove global convergence of DGM/PINNs trained with gradient descent to the PDE solution. The proof requires the development of a new mathematical approach different from [2] to address the non-trivial technical challenges introduced by the nonlinearity of the PDE. We consider a class of semi-linear PDEs Au u

x ∈ Ω,

x ∈ ∂Ω,

(1)

where Ω ⊂ Rn , Au is a nonlinear operator of the form Au = L + f (u, ux ), f (v, w) : R × Rn → R, and L is the uniformly linear elliptic operator Lu = −

n n X X ∂u ∂u ∂ [aij (x) ]+ ci (x) − d(x)u. ∂x ∂x ∂x i j i i=1 i,j=1

We will approximate the solution u(x) to the PDE (1) with a neural network QN (x; θ) = η(x)N −β

N X

ci σ(wi x + bi ),

(2)

i=1

where η(x) is a smooth function which vanishes on the boundary ∂Ω but has positive normal derivatives on the boundary, N −β is a normalization factor with 1 2 < β < 1, σ(·) is a nonlinear activation function, and the parameters which i i i will be trained are θ = (ci , wi , bi )N i=1 . H = σ(w x + b ) is the i-th hidden unit N of the neural network Q (x; θ) and there are N total hidden units. The related assumptions are stated in Assumption 4. The neural network QN (x; θ) is trained to minimize the objective function Z 1 N J (θ) = (A N QN (x; θ) − g(x))2 µ(dx) (3) 2 Ω Q using clipped gradient descent (see Section 3), where we denote for convenience AQN QN (x; θ) = [AQN QN ](x; θ). The objective function (3) is the PDE residual of the neural network QN (x; θ). Since the objective function J N (θ) is nonconvex in the neural network parameters θ, the gradient descent algorithm for training θ may potentially converge to a sub-optimal local minimizer of J N (θ). A local minimizer would of course not be a solution of the PDE (1). We will prove that, as the number of hidden units N → ∞ and the training time t → ∞, the neural network will converge to a global minimizer which is the solution of the PDE (1). 2

µ(dx) in the objective function (3) is a probability distribution, which can be sampled from in order to perform SGD-based training. In the original PINNs algorithm, µ is fixed before training begins, which is suitable for low-dimensional PDEs. The DGM algorithm randomly samples a minibatch from µ at each stochastic gradient descent step to calculate an unbiased stochastic estimate of ∇θ J N (θ) for high dimensional PDEs. The random sampling of spatial points from µ at each stochastic gradient descent step yields a mesh-free algorithm which addresses the curse-of-dimensionality and is one of the main distinctions between the PINNs and DGM algorithm. In this paper, we will focus on the core optimization question of whether a neural network trained with gradient descent to minimize (3) will converge to the PDE solution; further extensions to the stochastic gradient descent-variant are left for future research. Analyzing the optimization of (3) must address several challenges. First and foremost, the objective function is non-convex, and therefore in principle the trained neural network may converge to a local minimizer which is not the solution of the PDE. In typical supervised learning problems (e.g., least squares regression), as the number of hidden units N → ∞, the neural network will "convexify", leading to a limit problem which is convex in the neural network function and allowing one to prove global convergence [1]. However, for semilinear PDEs, the neural network does not "convexify", with the limit neural network satisfying a nonlinear PDE. That is, the limit neural network (in the limit N → ∞) satisfies a gradient flow of a non-convex objective function. Therefore, it would a priori seem that the neural network, even in the overparametrized limit of N → ∞, may converge to a local minimizer. The analysis of the limit neural network evolution has to address several fundamental challenges. The limit neural network satisfies a nonlinear, nonlocal PDE involving a nonlinear kernel function SQ(t) (x, y); see (19) in Section 3. SQ(t) (x, y) is a nonlinear function of the limit neural network Q(t) and its R derivatives Qx (t). Due to this dependence, the kernel operator SQ(t) f = SQ(t) (·, y)f (y)dy is a function of time t and therefore its eigenvalues and eigenfunctions also change over time. Consequently, it is not clear if we can study a projection of the PDE residual on a fixed eigenfunction basis (the proof approach in the paper [2]). [2] only considered linear PDEs and the limit neural network evolution satisfied a linear PDE. Importantly, for linear elliptic PDEs in [2], the limit kernel operator is a constant S(x, y) which does not depend upon the solution Q(t), allowing for the decomposition of the limit PDE dynamics via the fixed eigenfunctions of S. Projection onto time-dependent eigenfunctions in the nonlinear case is challenging. Furthermore, the positive eigenvalues of SQ(t) do not have a spectral gap (they accumulate at zero) and SQ(t) may also have zero eigenvalues. This makes it challenging to prove convergence as a priori it would seem the zero eigenvalues and eigenvalues accumulating to zero would both lead to a local minimizer (and not a global minimizer). Finally, any attempt at analysis requires establishing appropriate uniform-in-time bounds for Q(t, x) and the PDE residual AQ(t) Q(t, x) − g, which is challenging due to their satisfying a non-local nonlinear PDE. This paper develops a new mathematical approach which is able to prove 3

global convergence of the DGM/PINNs algorithm for semi-linear PDEs. We first study the evolution of the objective function during training. We establish that when training the parameters θ(t) with clipped gradient descent, the objective function J N (θ(t)) will converge to an appropriate limit objective function J(t) = J(Q(t)) as N → ∞ and the limit neural network’s training trajectory satisfies a nonlinear, non-local PDE (see Section 3). We then prove in Theorem 5.10 that the limit objective function must converge to zero (a global minimizer). The first step in our analysis is to prove that the limit training dynamics (which satisfy the nonlinear PDE) converge to a fixed point by establishing the monotonicity of the PDE residual evolution, uniform a priori bounds on the neural network in H 2 , and a uniform regularity bound for the RHS of dJ dt ; see (20). We analyze this fixed point, which we prove is a global minimizer of the objective function. The trained neural network at the fixed point furthermore has zero PDE residual. The final step is to prove that the neural network converges to this fixed point as training time t → ∞. Using the a priori bounds, we prove that, for every sequence of times tn → ∞, there is a convergent sub-sequence which must converge to a solution of the fixed point. We then prove that there is a unique limit point which has zero PDE residual and is the unique solution of the PDE. Consequently, the trained neural network converges to the solution of the PDE. The rest of the paper is organized as follows. Our assumptions and main results are presented in Section 2. Section 3 contains some preliminary calculations related to the clipped gradient descent algorithm which trains the neural network to minimize the PDE residual objective function. Section 4 establishes that fixed points of the neural network’s limit PDE are weak solutions in the appropriate space to the PDE (1). In Section 5 we prove that the PDE residual converges to zero as first N → ∞ and then t → ∞. That is, the limit objective function J(t) converges to zero as t → ∞ and that the neural network converges to a global minimizer which is the solution of the PDE (1). Section 6 proves the large N limit of the trained neural network. We conclude with Section 7 where we establish uniform-in-time error bounds on the objective function J N (θ(t)) for the PDE residual of the finite neural network, which is possible due to the monotonicity of J N (θ(t)). In particular, we are able to show that limN →∞ limt→∞ J N (θ(t)) = limt→∞ limN →∞ J N (θ(t)) = 0.

Assumptions and Main Results

In this section, we state our assumptions and present the main results of the paper.

2.1

Main Assumptions

We first impose proper assumptions on the nonlinearity f (v, w) and on the PDE (1). Assumption 1. We assume that f (v, w) and its first two derivatives are uniformly bounded (and measurable functions). L is a uniformly linear elliptic 4

operator with the dissipativity condition that its eigenvalues satisfy inf n λn > 0 (and therefore the coercivity condition holds for L). Furthermore, the coefficients of L are smooth. Ω ⊂ Rn is a compact domain with a smooth boundary. The function g is bounded on Ω. The smoothness assumption on the coefficients of the operator L can be relaxed, but we do not do so here for presentation purposes. Assumption 2. The PDE (1) has a unique weak solution in H01 . Pn ∂h . In Define the operator DQ h = Lh + fv (Q, Qx )h + i=1 fwi (Q, Qx ) ∂x i 1 the following assumption, we will assume that the operator DQ : H0 → L2 is surjective. Assumption 3. For any R ∈ L2 and Q ∈ H01 , the PDE DQ h

x ∈ Ω,

x ∈ ∂Ω,

(4)

has a solution h ∈ H01 . A sufficient condition for Assumption 3 to hold is Assumption 1 and the additional assumption that fv (v, w) ≥ 0. No additional assumptions on fw (v, w) are necessary. Under these assumptions, due to Theorems 8.9 and 8.12 in [7], 2 2 there exists a unique weak solution h ∈ H(0) to (4), where we have set H(0) = 2 1 H ∩ H0 . As discussed in the introduction section we will approximate the solution u(x) to the PDE (1) with a neural network QN (x; θ) = η(x)N −β

N X

ci σ(wi x + bi ),

i=1

where η(x) is a smooth function which vanishes on the boundary ∂Ω but has positive normal derivatives on the boundary and N −β is a normalization factor with 12 < β < 1. For the trainable parameters θ = (ci , wi , bi )N i=1 and η(x) we make the following assumption. Assumption 4. The neural network parameters (ci , wi , bi ) are i.i.d. randomly initialized with distribution µ0 (dc, dw, db). The random variables wi , bi have full support on Rn where wi has bounded moments, ci are mean-zero, bounded random variables and ci , wi , and bi are independent. The neural network activation function σ is bounded and has at least three bounded derivatives. σ is a discriminatory function (such as a sigmoid or tanh function). η ∈ Cb3 (with bounded derivatives) where η > 0 in the interior of Ω and η = 0 for x ∈ ∂Ω. In addition, ∇η · nx ̸= 0 for x ∈ ∂Ω where nx is the outward unit normal vector at x. Finally for the measure µ, which is used in the objective function (3) we have Assumption 5. 5

Assumption 5. Assume that µ(dx) is a uniform probability measure on Ω. We note that Assumption 5 is mainly posed for convenience and concreteness, and can be relaxed to any positive probability density function on Ω that is bounded away from zero in Ω.

2.2

Summary of Main Results

Let us now discuss the main results of the paper. Theorem 6.4 proves that the neural network QN (y; θ(t)) trained with clipped gradient descent (13) will converge as N → ∞ to the solution of the nonlocal, nonlinear PDE Z ∂Q(t, x) = −α UQ(t) (x, y) AQ(t) Q(t, y) − g(y) µ(dy), ∂t Ω with the initial condition is Q(0, x) = 0 and the kernel function UQ(t) (y, x) = ∇θ η(y)cσ(wy + b) · DQ(t) [∇θ η(x)cσ(wx + b)], µ0 . Before we continue, it is important to clarify the notation we use. When we write AQ(t) Q(t, y) and DQ(t) g(x, θ) for some function g we shall mean AQ(t) Q(t, y) = [AQ Q](t, y) = AQ(t,y) Q(t, y) DQ(t) g(x, θ) = DQ(t,x) g(x, θ) respectively. Analogously DQN g(x, θ) = DQN (x;θ) g(x, θ). Continuing now the description of our main results, the limit equation stated above for Q(t, x) governs the evolution of the (limit) neural network output as a function of training time t. In addition, we establish that ∂xi QN (t, x) converges almost surely to ∂xi Q(t, x) on t ∈ [0, T ] as N → ∞ for |i| ≤ 2: a.s.

sup |∂xi QN (t, x) − ∂xi Q(t, x)| → 0,

x∈Ω

where T is arbitrary. As a direct consequence, for arbitrarily chosen T < ∞, as N → ∞, a.s.

sup ∥QN (t) − Q(t)∥H 2 → 0. t∈[0,T ]

Even though the aforementioned convergence is in [0, T ] for arbitrary T < ∞, Theorem 5.11 proves that the limit neural network Q(t, x) converges to the solution u(x) of the PDE (1) in H 1 as t → ∞: lim ∥Q(t) − u∥H 1 = 0.

t→∞

(5)

Theorem 7.1 also establishes uniform-in-time error bounds for the objective function for the finite neural network with N hidden units. For any ϵ > 0, there exists a t ≥ 0 and an N0 < ∞ such that E[J N (θ(s))] ≤ ϵ, 6

(6)

for all s ≥ t and N ≥ N0 . Consequently, if we make the neural network sufficiently large, we can achieve an arbitrarily low PDE residual. We highlight that such uniform-in-time estimates are typically difficult to achieve in the analysis of mean field systems. That is, typically mean-field analysis can only establish a result such as lims→∞ limN →∞ E[J N (θ(s))] = 0, which is directly implied by our Theorems 6.4 and 5.11. Typically, without additional assumptions, mean-field analysis is not able to prove that the limits are interchangeable, i.e. it cannot prove that the relation holds limN →∞ lims→∞ E[J N (θ(s))] = 0. In Theorem 7.1, we are able to additionally prove (7)

lim lim E[JtN ] = 0,

N →∞ t→∞

where the proof leverages the fact that the pre-limit objective function J N (θ(t)) is monotonically decreasing in t.

Gradient Descent Algorithm for DGM and PINNs

In this section we present preliminary calculations deriving the formulas for the evolution of the objective function J N (θ(t)) and the neural network QN (x; θ(t)) as a function of the training time t. An important ingredient in our subsequent proofs will be an appropriate clipping of the gradient of the objective function used in the gradient descent algorithm. Clipping of the gradient is a standard procedure followed in practice in gradient descent algorithms in machine learning. Here, we shall see that it is an important also for the convergence analysis to hold as N → ∞. We emphasize though that while implementing gradient clipping is important for establishing the necessary a priori bounds for the analysis, its effect fades away as N → ∞: the clipping does not appear in the limit training dynamics (5). The gradient of the objective function is Z N N N ∇θ J (θ) = (AQN Q (x; θ) − g(x)) × ∇θ AQN Q (x; θ) µ(dx) ZΩ = (AQN QN (x; θ) − g(x)) × DQN ∇θ QN (x; θ)µ(dx), (8) Ω

where the differential operator DQ is DQ h = Lh + fv (Q, ∂x Q)h +

n X

fwi (Q, Qx )

i=1

∂h . ∂xi

For example, denoting QN (t) = QN (·; θ(t)), we get DQN (t) h = Lh + fv (QN (x; θ(t)), QN x (x; θ(t)))h n X ∂h + fwi (QN (x; θ(t)), QN . x (x; θ(t))) ∂x i i=1 7

(9)

The gradient with respect to a single parameter triplet θi = (ci , wi , bi ) is: Z ∇θi J N (θ) = N −β (AQN QN (x; θ) − g(x)) × DQN ∇θi ci σ(wi x + bi )µ(dx) . Ω

(10)

We will apply a clipping function to the term inside the brackets for the clipped gradient descent algorithm used to train the neural network. Clipped gradient descent is a standard method in deep learning for training large models; clipping increases stability during training. We introduce a clipping function ϕN which satisfies the following properties: Definition 3.1 (Smooth clipping function). A function class {ϕN }N ∈N+ forms a family of smooth clipping functions with parameter γ > 0 if for any N ∈ N+ • ϕN ∈ C2b (R) is increasing on R. • |ϕN | is bounded by 2N γ . • ϕN (x) = x for x ∈ [−N γ , N γ ]. • |(ϕN )′ | ≤ 1 on R. Assumption 6. Functions {ϕN }N ∈N+ are smooth clipping functions with parameters γ where γ > 0, and for β ∈ ( 21 , 1) as in the definition of the neural network (2), we assume γ + β < 1. The training algorithm will clip the gradient (10), and the clipped gradient will be used to train the neural network following the equation Z i i i N −β N N N ∇ c σ(w x + b )µ(dx) , Q (x; θ) − g(x)) × D GN (θ) = N ϕ (A θi Q Q i Ω

(11)

where the scalar clipping function ϕN is applied element-wise to the vector in the above equation. Note that (11) can equivalently be written in the more general vector form Z GN (θ) = N −β ϕN N β (AQN QN (x; θ) − g(x)) × DQN ∇θ QN (x; θ)µ(dx) . Ω

(12)

Notice that, due to Definition 3.1, as N → ∞, the effect of the clipping vanishes. The parameters are trained with (clipped) gradient descent: dθ = −αN GN (θ(t)), dt where the learning rate αN = αN 2β−1 and α > 0 is a positive constant. 8

(13)

Next step is to use the chain rule to derive an evolution equation for the neural network output, ∂QN (x; θ(t)) = −αN ∇θ QN (x; θ(t)) · GN (θ(t)) ∂t Z = −αN N −β ϕN N β

(AQN (t) QN (y; θ(t)) − g(y)) × DQN (t) [∇θ QN (y; θ(t))]µ(dy) · ∇θ QN (x; θ(t)).

(14)

Next define the quantity BtN (x) = αN N −β ϕN

(AQN (t) QN (y; θ(t)) − g(x)) × DQN (t) [∇θ QN (y; θ(t))]µ(dy) · ∇θ QN (x; θ(t)), N

(15)

and the kernel UtN (x, y) = N 2β−1 DQN (t) ∇θ QN (y; θ(t)) · ∇θ QN (x; θ(t)) N = ∇θ η(x)cσ(wx + b) · DQN (t) [∇θ cσ(wy + b)], µt , PN −1 where µN t =N i=1 δθi (t) (dc, dw, db) is the empirical measure of the trained parameters. Then we obtain Z ∂QN (x; θ(t)) = −αN UtN (x, y)(AQN (·;θ(t)) QN (y; θ(t)) − g(y))µ(dy) ∂t Ω N N + Lt (x) − Bt (x) , (16) R N where LN U N (x, y)(AQN (·;θ(t)) QN (y; θ(t)) − g(y))µ(dy). We will t (x) = α Ω t N later show that term Lt (x) − BtN (x) vanishes as N → ∞ (i.e., the effect of the clipping function disappears as the neural network becomes larger). As N → ∞, Theorem 6.4 proves that (UtN (x, y), QN (x; θ(t)) will converge to (UQ(t) (x, y), Q(t, x)) (almost surely in C 2 (Ω) for each t ≥ 0) where: UQ (x, y) =

∇θ (η(x)cσ(wx + b)) · DQ [∇θ (η(y)cσ(wy + b))], µ0 ,

where µ0 is the distribution of the randomly initialized parameters at t = 0 (the overparameterized Neural Tangent Kernel (NTK) limit, e.g, [1]. The limit neural network’s training evolution satisfies the PDE Z ∂Q(t, x) = −α UQ(t) (x, y) AQ(t) Q(t, y) − g(y) µ(dy), (17) ∂t Ω 9

with the initial condition is Q(0, x) = 0. The PDE (17) governs the evolution of the (limit) neural network output as a function of training time t (17) is a nonlocal nonlinear PDE due to the integral, the nonlinear dependence of UQ(t) on (Q, Qx ), and the nonlinear dependence of AQ(t) on (Q, Qx ). We will sometimes use the notation Ut (x, y) = UQ(t) (x, y) for notational convenience. Next, we use the notation established above to derive an equation for the evolution of the PDE residual: ∂Q ∂ [AQ(t) Q(t, x) − g](x) = DQ(t) (t, x) ∂t ∂t Z = −αDQ(t) UQ(t) (x, y) AQ(t) Q(t, y) − g(y) µ(dy) . Ω Z = −α St (x, y) AQ(t) Q(t, y) − g(y) µ(dy), (18) Ω

where the kernel SQ (x, y) is defined to be SQ (x, y) = DQ ∇θ (η(x)cσ(wx + b)) · DQ ∇θ (η(y)cσ(wy + b)) , µ0 . (19) We will sometimes use the notation St (x, y) = SQ(t) (x, y) for notational convenience. Our calculations in Section 6 will show that the pre-limit objective function J N (θ(t)) will converge to the following limit objective function J(t) = J(Q(t)) as N → ∞: Z 1 J(t) = (AQ(t) Q(t, x) − g(x))2 µ(dx). 2 Ω Differentiating with respect to time yields Z dJ = −α (AQ(t) Q(t, x) − g(x))SQ(t) (x, y)(AQ(t) Q(t, y) − g(y))µ(dx)µ(dy). dt Ω (20) In Section 4 we prove that fixed points of the right hand side of (20) are weak solutions to the PDE (1).

Fixed Points of the Neural Network’s Limit PDE

In this section we study the fixed points Q of (20). We will return later to the question of convergence as t → ∞. Define R(t, x) = AQ(t) Q(t, x) − g(x) and R(x) = AQ Q(x) − g(x). If dJ dt = 0, this implies that any fixed point Q(x) and R(x) = AQ Q(x) − g(x) must satisfy: Z 0 = R(x)SQ (x, y)R(y)µ(dx)µ(dy), (21)

which can be lower bounded using the following calculations: Z 0= R(x)SQ (x, y)R(y)µ(dx)µ(dy) 2 ZΩ = R(x)⟨DQ [η(x)σ(wx + b)]DQ [η(y)σ(wy + b)], µ0 ⟩R(y)µ(dx)µ(dy) 2 ZΩ + R(x)⟨DQ [η(x)cσ ′ (wx + b)x] · [DQ η(y)cσ ′ (wy + b)y], µ0 ⟩R(y)µ(dx)µ0 (dy) Ω2 Z R(x)DQ [η(x)σ(wx + b)]DQ [η(y)σ(wy + b)]R(y)µ(dx)µ(dy) = Ec,w,b∼µ0 2 ZΩ + Ec,w,b∼µ0 R(x)(DQ [η(x)cσ ′ (wx + b)x]· Ω2 ′ · DQ [η(y)cσ (wy + b)y])R(y)µ(dx)µ(dy) Z 2 = Ec,w,b∼µ0 R(x)DQ [η(x)σ(wx + b)]µ(dx) Z Ω 2 R(x)DQ [η(x)cσ ′ (wx + b)x]µ(dx) + Ec,w,b∼µ0

(22)

Ω

Z ≥ Ec,w,b∼µ0

2 R(x)DQ [η(x)σ(wx + b)]µ(dx) .

(23)

Ω

The latter calculation implies that Z 2 Ec,w,b∼µ0 R(x)DQ [η(x)σ(wx + b)]µ(dx) = 0.

(24)

Ω

In fact we have the following result Lemma 4.1. Suppose that R ∈ L2 , Q ∈ H01 , and that (24) holds. Then, we have that for all (w, b) ∈ Rd × R Z R(x)DQ [η(x)σ(wx + b)]µ(dx) = 0. Ω

R Proof. Equation (24) immediately implies that E(w, b) = R(x)DQ [η(x)σ(wx+ b)]µ(dx) is zero almost everywhere in (w, b) ∈ Rd ×R. Since E(w, b) is continuous (due to our assumptions and R ∈ L2 ), E(w, b) = 0 everywhere. Consequently, for any fixed point R ∈ L2 and Q ∈ H01 , Z R(x)DQ [η(x)σ(wx + b)]µ(dx) = 0, Ω

for all (w, b) ∈ R × R. 2 Now, this implies that if R ∈ L2 , then ⟨R, DQ f ⟩ = 0 for any f ∈ H(0) = 2 1 H ∩ H0 . d

Lemma 4.2. If R ∈ L2 and Q ∈ H01 satisfy the fixed point equation (21), then 2 ⟨R, DQ f ⟩ = 0 for any f ∈ H(0) = H 2 ∩ H01 . Proof. Recall that Z R(x)DQ [η(x)σ(wx + b)]µ(dx) = 0,

(25)

Ω

for all (w, b) ∈ Rd × R. The linear span of {η(x)σ(w · x + b)}w,b∈Rn+1 is dense 2 in H(0) ; see [5] for the proof. Due to this density result, there exists a function PN 2 F N (x) = η(x) i=1 ci σ(wi · x + bi ) which converges to any f ∈ H(0) : lim ∥FN − f ∥H 2 = 0.

N →∞

Then, due to equation (25), ⟨R, DQ f ⟩ = lim ⟨R, DQ F N ⟩ N →∞

= lim

N →∞

N Z X i=1

R(x)DQ [η(x)σ(wi · x + bi )]µ(dx)⟩

Ω

= 0, concluding the proof of the lemma. 2 Lemma 4.3. If R ∈ L2 and for any Q ∈ H01 , then there exists an h ∈ H(0) such that DQ h = R.

Proof. This is directly from Assumption 3 with a sufficient Pn condition being ∂h . fv (Q, ∂x Q) ≥ 0. Recall that DQ h = Lh + fv (Q, Qx )h + i=1 fwi (Q, Qx ) ∂x i Due to Assumption 1, fv and fwi are bounded. Consider the PDE DQ h = R, where h satisfies Dirichlet boundary conditions and the source term R ∈ L2 . Due 2 to Theorems 8.9 and 8.12 in [7], there exists a unique weak solution h ∈ H(0) . Theorem 4.4. Any (R, Q) ∈ L2 × H01 satisfying the fixed point characterized by equation (21) is a global minimizer with zero PDE residual: ∥R∥L2 = 0. 2 Proof. Due to Lemma 4.3, there exists an h ∈ H(0) such that DQ h = R (weakly). 1 Therefore, for every test function v ∈ H0 , Z X n n X ∂h ∂v ∂h aij + ci v(x) − d(x)h(x)v(x) ∂xj ∂xi ∂xi Ω i,j=1 i Z n X ∂h (x)v(x) dx = R(x)v(x)dx. + fv (Q, Qx )h(x)v(x) + fwi (Q, Qx ) ∂xi Ω i=1

Since h ∈ H 2 and v ∈ H01 , we can use integration by parts to find that Z Z DQ h(x) v(x)dx = R(x)v(x)dx. Ω

Ω

Recall that H01 is dense in L2 (Ω) (since Cc∞ is a subset of H01 ). Consider a sequence of functions vn ∈ H01 where ∥vn − R∥L2 → 0. Then, for each n, Z DQ h(x) − R(x) vn dx = 0, Ω

and by the Cauchy-Schwarz inequality Z Z | (DQ h(x) − R(x))vn dx − (DQ h(x) − R(x))R(x)dx| → 0, Ω

Ω

as n] → ∞. Consequently, we obtain Ω (DQ h(x) − R(x))R(x)dx = 0. We next apply Lemma 4.2 to get Z 2 ∥R∥L2 = DQ h(x)R(x)dx R

Ω

= 0. 2 Note that h ∈ H(0) and R ∈ L2 , which satisfy the assumptions of Lemma 4.2. These conclude the proof or the theorem.

Furthermore, under suitable technical conditions, if the PDE residual is zero, then, the neural network function (at the fixed point) is a solution to the PDE (1). Corollary 4.5. Under Assumption 1 and Assumption 2, if Q ∈ H01 and ∥R∥2L2 = ∥AQ Q − g∥L2 = 0, then Q is a unique weak solution in H01 to the PDE (1). Proof. For any test function ϕ ∈ H01 , we can apply the Cauchy-Schwarz inequality to show that ⟨AQ Q − g, ϕ⟩2 ≤ ∥AQ Q − g∥2L2 × ∥ϕ∥2L2 = 0. This implies that ⟨AQ Q − g, ϕ⟩ = 0. Since Q and ϕ are in H01 , the above equation can be integrated by parts to produce n Z X ∂ϕ ∂Q ∂Q aij (x) (x) (x) + ci (x) (x)ϕ(x) − d(x)Q(x) ∂x ∂x ∂x i j i i=1 Ω + f (Q(x), Qx (x))ϕ(x) − g(x)ϕ(x) dx = 0. (26)

The definition of a weak solution in H01 is a Q ∈ H01 which satisfies (26) for every ϕ ∈ H01 . We recall that we have assumed in Assumption 2 that there is a unique weak solution to (26). This concludes the proof of the Corollary.

Global Convergence of Neural Network’s Limit PDE as t → ∞

Now that we have characterized the fixed points of the neural network’s limit PDE, we will analyze its convergence as t → ∞. We have establish the monotonicity of the limit objective function J(t). Lemma 5.1. J(t) is monotone decreasing in t and for the residual R(t, x) = AQ(t) Q(t, x) − g(x) we have that there is a uniform in time constant C < ∞ such that supt≥0 ∥R(t, ·)∥L2 ≤ C < ∞. Proof. Using similar calculations as in equation (23), we can establish the lower bound Z R(t, x)SQ(t) (x, y)R(t, y)µ(dx)µ(dy) ≥ Ω×Ω

Z ≥ Ec,w,b∼µ0

R(t, x)DQ(t) η(x)σ(wx + b)µ(dx) Ω

≥ 0. Since dJ dt =

R Ω×Ω

R(t, x)SQ(t) (x, y)R(t, y)µ(dx)µ(dy), we obtain dJ ≤ 0. dt

(27)

Consequently, we get J(t) = ∥R(t, x)∥2L2 ≤ J(0) < ∞. The above lemma establishes that the PDE residual of the neural network Q(t, x) is uniformly bounded in L2 in time: sup∥LQ(t) + f (Q(t), Qx (t))∥2L2 ≤ C < ∞,

(28)

t≥0

where C is uniform in time. Since f (v, w) is uniformly bounded, we can use the triangle inequality to show that ∥LQ(t)∥L2 = ∥LQ(t) + f (Q(t), Qx (t)) − f (Q(t), Qx (t))∥L2 ≤ ∥LQ(t) + f (Q(t), Qx (t))∥L2 + ∥f (Q(t), Qx (t))∥L2 (29)

≤ C1 < ∞,

2 also uniformly in time. It is also true that Q(t, x) ∈ H(0) = H01 ∩ H 2 and is continuous in time.

2 Lemma 5.2. Any solution Q(t, x) to the equation (17) is in C([0, ∞); H(0) ).

Proof. Differentiating (17) yields Z ∂Q(t, x) = −α UQ(t) (x, y) AQ(t) Q(t, y) − g(y) µ(dy), ∂t ZΩ ∂∇x Q(t, x) = −α ∇x UQ(t) (x, y) AQ(t) Q(t, y) − g(y) µ(dy), ∂t ZΩ ∂∇xx Q(t, x) = −α ∇xx UQ(t) (x, y) AQ(t) Q(t, y) − g(y) µ(dy). (30) ∂t Ω Pn Pn Define V (t, x) = |Q(t, x)| + i=1 |Qxi (t, x)| + i,j=1 |Qxi xj (t, x)|. Integrating (30) with respect to time, taking an absolute value, and summing over i, j = 0, 1, . . . , n and using the boundedness assumptions on f produces V (t, x) ≤ C1

2 Z tZ X i=0

Ω

max(|UQ(s) (x, y)| + |∂xi UQ(s) (x, y)| + |∂xi xj UQ(s) (x, y)|) i,j

× V (t, y)dy + C2

2 Z tZ X i=0

Ω

max(|UQ(s) (x, y)| + |∂xi UQ(s) (x, y)| + |∂xi xj UQ(s) (x, y)|)(1 + g(y))dy. i,j

(31)

Here maxi,j (|UQ(s) (x, y)| + |∂xi UQ(s) (x, y)| + |∂xi xj UQ(s) (x, y)| is uniformly bounded (in both the spatial variables x, y ∈ Ω and time t ≥ 0) due to the assumptions on the moments of w0i , the derivatives of σ being bounded, and Ω being a compact domain. Therefore, the last term in (31) is bounded via the Cauchy-Schwarz inequality since g ∈ L2 . Taking a supremum first on the RHS and then on the LHS: Z tZ sup V (t, x) ≤ C1 sup V (t, y)dy + C2 t x∈Ω

Ω y∈Ω

Z t ≤ C3

sup V (t, y)dy + C2 t. 0 y∈Ω

By Gronwall’s inequality, ≤ Ct.

sup V (t, x)

(32)

x∈Ω

This of course implies that ∥Q(t, x)∥H 2 < ∞ for every t ≥ 0. In addition Q(t, x) is continuous in time. For i = 0, 1, 2, we have ∂xi Q(t + τ, x) − ∂xi Q(t, x) = −α

Z t+τ Z t

∂xi UQ(s) (x, y) AQ(s) Q(s, y) − g(y) µ(dy).

Ω

(33)

Therefore, using the previous bound, 2 X

sup |∂xi Q(t + τ, x) − ∂xi Q(t, x)|

≤

Cτ,

(34)

i=0 x∈Ω

which implies that ∥Q(t + τ ) − Q(t)∥H 2 ≤ C3 τ , where C3 depends upon the interval [0, T ] where T > t + τ . 2 Finally, we note that Q(t, x) is in H(0) for all t ≥ 0 since Q(0, x) = 0 and UQ(t) (x, y) = 0 for x ∈ ∂Ω, which leads to Q(t, x) = 0 for x ∈ ∂Ω. Lemma 5.3. There exists a unique solution Q(t, x) to the equation (17) in 2 C([0, T ]; H(0) ), where T > 0 is arbitrary. Proof. We first consider the RHS of equation (17): Z t+τ Z Gt,∆ (q)(τ, x) = Q(t, x) − α UQ(s) (x, y) Aq q − g(y) µ(dy)ds, t

(35)

Ω

where 0 ≤ τ ≤ ∆. Then, we have that Z t+τ Z |Gt,∆ (Q1 )(τ, x) − Gt,∆ (Q2 )(τ, x)| ≤ C

AQ1 Q1 − AQ2 Q2 µ(dy)ds t

Z t+τ Z (|Q1 − Q2 | +

+C t

Ω

n X ∂Q2 ∂Q1 − |) × AQ2 Q2 − g µ(dy)ds | ∂xi ∂xi i=1

Z t+τ ≤

C(1 + ∥AQ2 Q2 − g∥L2 ) × ∥Q1 − Q2 ∥H 2 ds. t

≤ ∆C(1 +

sup ∥AQ2 Q2 (s) − g∥L2 ) s∈[t,t+∆]

(36)

sup ∥Q1 (s) − Q2 (s)∥H 2 , s∈[t,t+∆]

where we have used the bounded moments of w0i and the uniform boundedness of fvv , fvw , and fww in the second line. Similarly, bounds can be established on the first and second derivatives |∂xi Gt,∆ (Q1 )(τ, x) − ∂xi Gτ,∆ (Q2 )(t, x)| and |∂xi xj Gt,∆ (Q1 )(τ, x) − ∂xi xj Gτ,∆ (Q2 )(t, x)|, yielding sup ∥Gt,∆ (Q1 )(s) − Gt,∆ (Q2 )(s)∥H 2 ≤ s∈[t,t+∆]

≤ ∆ × C1 (1 +

sup ∥AQ2 Q2 (s) − g∥L2 ) × s∈[t,t+∆]

sup ∥Q1 (s) − Q2 (s)∥H 2 . s∈[t,t+∆]

(37) Using similar calculations, sup ∥Gt,∆ (q)(s)∥H 2 ≤ ∥Q(t)∥H 2 + ∆ × C2 × s∈[t,t+∆]

sup ∥q(s)∥H 2 . s∈[t,t+∆]

(38)

Note that C1 , C2 do not depend upon time t. Let C3 = max(C1 , C2 ). Define the map M : C([0, ∆]; H 2 ) → C([0, ∆]; H 2 ) where M (q)(s, x) = G0,∆ (q)(s, x). Define the set B = {Q : ∥Q∥C([0,∆];H 2 ) ≤ 2C(T )}.We will consider the iteration Q̄k+1 = M (Q̄k ) with Q̄0 ∈ B. Consider a time interval [0, T ], where from our previous estimates any solution must satisfy ∥Q∥H 2 ≤ C(T ). Consider two functions Q1 , Q2 ∈ C([0, ∆]; H 2 ) where supt∈[0,∆] ∥Q1 (t)∥H 2 , supt∈[0,∆] ∥Q2 (t)∥H 2 ≤ 2C(T ). Then, M (q) ∈ B for 1 any q ∈ B if ∆ ≤ C3 ×4C(T ). Furthermore, from (36), if we select ∆ sufficiently small, we can establish a contraction ∥M (Q1 ) − M (Q2 )∥C([0,∆];H 2 ) ≤ q∥Q1 − Q2 ∥C([0,∆];H 2 ) for 0 < q < 1 and Q1 , Q2 ∈ B. By the Banach fixed point theorem, M (q) has a unique fixed point in B. Therefore, there is a unique solution on the time interval [0, ∆]. Once a unique solution is established on [0, ∆], we can – using the same method – prove a unique solution on the time intervals [∆, 2∆], [2∆, 3∆], . . .. Furthermore, in Lemma 5.2, we proved that any solution in H 2 much take 2 values in H(0) (i.e., the solution vanishes on the boundary ∂Ω), which completes the proof. We are now in position to prove a uniform H 2 bound on Q(t, x): Lemma 5.4. Q(t, x) is uniformly bounded in H 2 : ∥Q(t)∥H 2 ≤ C2 < ∞.

(39)

2 Proof. Due to Lemma 5.3, Q(t) ∈ H(0) . Similar to equation (29), we can show that Q(t) is a solution to a PDE LQ(t) = q where q ∈ L2 . Specifically, LQ(t) + f (Q(t), Qx (t)) = R(t). Re-arranging, LQ(t) = −f (Q(t), Qx (t)) + R(t). Due to f (v, w) being uniformly bounded and ∥R(t)∥L2 also being uniformly bounded (see Lemma 5.1), Q(t) satisfies the PDE LQ(t) = q(t) where q(t) is uniformlyin-time bounded. Therefore, from Theorem 8.12 (page 186) in [7],

∥Q(t)∥H 2 ≤ K(∥LQ(t)∥L2 + ∥Q(t)∥L2 ) ≤≤ K1 + K∥Q(t)∥L2 . It only remains to establish the L2 bound on Q(t). Due to our assumptions L satisfies the coercivity condition: K2 ∥Q(t)∥2L2 ≤ ⟨LQ(t), Q(t)⟩, where K2 > 0. Applying the Cauchy-Schwarz inequality on the RHS, K2 ∥Q(t)∥2L2 ≤ ∥LQ(t)∥L2 ∥Q(t)∥L2 . Dividing by ∥Q(t)∥L2 for the case where ∥Q(t)∥L2 > 0, we have the uniformin-time bound 1 C1 ∥Q(t)∥L2 ≤ ∥LQ(t)∥L2 ≤ . K2 K2 17

When ∥Q(t)∥L2 = 0, we also of course have that ∥Q(t)∥L2 ≤ C2 where C1 C2 = K > 0. 2 Lemma 5.5. For any sequence of times tn → ∞, there is a convergent subsequence Q(tnk ) ∈ H 1 . Proof. By the Rellich-Kondrachov theorem (see Theorem 7.29 in [9]), the bounded set A = {u ∈ H 2 : ∥u∥H 2 ≤ C} is precompact in H 1 . Since ∥Qtn ∥H 2 ≤ C, Qtn ∈ A where A is precompact in H 1 . Therefore, there is a further subsequence Qtnk which converges to a limit Q in H 1 (where the limit point may depend upon the sequence tnk ). Lemma 5.6. For any sequence of times tn → ∞, there is a weakly convergent subsequence R(tnk , x) in L2 . Proof. Since R(tn ) is uniformly bounded in L2 , there is a weakly convergent w subsequence R(tnk ) → R ∈ L2 (Theorem 3, Appendix D, page 639 in [6]). We now establish a uniform bound on ∂Q ∂t (and its first and second spatial derivatives) which will be essential for the subsequent analysis. ∂Q

∂Qxi xj ∂t (t, x)|

Proof. Recall that Q(t, x) satisfies the evolution equation Z ∂Q(t, x) = −α UQ(t) (x, y)R(t, y)µ(dy), ∂t Ω Ut (x, y) = ∇θ η(x)cσ(wx + b) · DQ(t) [∇θ η(y)cσ(wy + b)], µ0 , where the initial condition is Q(0, x) = 0. Then, |

Z 2 ∂Q(t, x) 2 | ≤ α2 |UQ(t) (x, y)||R(t, y)|µ(dy) , ∂t Z Ω Z ≤α |UQ(t) (x, y)|2 µ(dy) × |R(t, y)|2 µ(dy) Ω

Ω

(40)

≤ C < ∞,

where we have applied Assumption 1 and Assumption 4 as well as the bound from Lemma 5.1. Similarly, we can calculate uniform bounds for the first and second spatial derivatives of ∂Q ∂t . Details are omitted due to the similarity of the argument. Lemma 5.8. We have that lim ⟨R(t), SQ(t) R(t)⟩ = 0.

t→∞

Proof. Recall that dJ = −α⟨R(t), SQ(t) R(t)⟩. (41) dt We will apply Barbalat’s Lemma (See page 205, Section 5.5 of [8]) to prove that limt→∞ ⟨R(t), SQ(t) R(t)⟩ = 0, which requires showing that (a) limt→∞ J(t) exists and is finite and (b) dJ dt = −α⟨R(t), SQ(t) R(t)⟩ is uniformly continuous in time t. First, we show that dJ dt (t) is uniformly continuous. Due to Lemma 5.3, 2 Q(t, x) ∈ C([0, ∞); H(0) ). We start by considering the following decomposition ⟨R(t + δ), St+δ R(t + δ)⟩ − ⟨R(t), St R(t)⟩ = ⟨R(t + δ), (St+δ − St )R(t + δ)⟩ + ⟨R(t + δ), St R(t + δ)⟩ − ⟨R(t), St R(t)⟩ = ⟨R(t + δ), (St+δ − St )R(t + δ)⟩ + ⟨R(t + δ), St (R(t + δ) − Rt )⟩ + ⟨R(t + δ), St R(t)⟩ − ⟨R(t), St R(t)⟩ = ⟨R(t + δ), (St+δ − St )R(t + δ)⟩ + ⟨R(t + δ), St (R(t + δ) − R(t))⟩ (42)

+ ⟨R(t + δ) − R(t), St R(t)⟩,

where for notational convenience we have denoted St = SQ(t) . We next bound each of the terms in the last line above. Due to Assumption 1 and Assumption 4, we have the uniform bound |St (x, y)| ≤ C1 . Recall that ∥R(t)∥L2 ≤ C0 and that we have also proven that ∥Q(t, x)∥H 2 < C2 in Lemma 5.4. Using the Cauchy-Schwarz inequality, |⟨R(t + δ) − R(t), St R(t)⟩|2 ≤ ∥R(t + δ) − R(t)∥2L2 ∥St R(t)∥2L2 ≤ C3 ∥R(t + δ) − R(t)∥2L2 ∥R(t)∥2L2 ≤ C4 ∥Q(t + δ) − Q(t)∥2H 2 .

(43)

Using similar calculations |⟨R(t + δ), St (R(t + δ) − R(t))⟩|2 ≤ C5 ∥Q(t + δ) − Q(t)∥2H 2 .

(44)

Using the Cauchy-Schwarz inequality, ⟨R(t + δ), (St+δ − St )R(t + δ)⟩2 ≤ ∥R(t + δ)∥2L2 ∥(St+δ − St )R(t + δ)∥2L2 . We first apply the following re-arrangement of terms: St+δ (x, y) − St (x, y) = = DQ(t+δ) ∇θ η(x)cσ(wx + b) · DQ(t+δ) ∇θ η(y)cσ(wy + b) , µ0 − DQ(t) ∇θ η(x)cσ(wx + b) · DQ(t) ∇θ η(y)cσ(wy + b) , µ0 = (DQ(t+δ) − DQ(t) ) ∇θ η(x)cσ(wx + b) · DQ(t+δ) ∇θ η(y)cσ(wy + b) , µ0 + DQ(t) ∇θ η(x)cσ(wx + b) · (DQ(t+δ) − DQ(t) ) ∇θ η(y)cσ(wy + b) , µ0 .

Using a Taylor expansion, [DQ(t+δ) − DQ(t) ]h(x) = = fv (Q(t + δ, x), Qx (t + δ, x)) − fv (Q(t, x), Qx (t, x))h N X

∂h + fwi (Q(t + δ, x), Qx (t + δ, x)) − fwi (Q(t, x), Qx (t, x)) ∂x i i=1 ∗ = fvv , fvw (z (x)) · Q(t + δ, x) − Q(t, x), Qx (t + δ, x) − Qx (t, x) h +

n X i=1

∂h fwi v , fwi w (m∗ (x)) · Q(t + δ, x) − Q(t, x), Qx (t + δ, x) − Qx (t, x) , ∂xi

where z ∗ (x), m∗ (x) are points on the line between (Q(t, x), Qx (t, x)) and (Q(t + δ, x), Qx (t + δ, x)). Due to Assumption 1, the derivatives of f are uniformly bounded. Therefore, |DQ(t+δ) − DQ(t) ]h(x)| ≤ C |Q(t + δ, x) − Q(t, x)| ∂h (x)|. + |Qx (t + δ, x) − Qx (t, x)| × sup |h(x) + ∂xi i Consequently, due to Assumption 1 and Assumption 4, |St+δ (x, y) − St (x, y)| ≤ C(|Q(t + δ, x) − Q(t, x)| + |Q(t + δ, y) − Q(t, y)| + |Qx (t + δ, x) − Qx (t, x)| + |Qy (t + δ, y) − Qy (t, y)|). Therefore, we have that Z

∥(St+δ − St )R(t + δ)∥2L2 ≤ ∥

(St+δ (·, y) − St (·, y))R(t + δ, y)µ(dy)∥2L2

Ω

Z Z = Ω

Ω Z Z

≤ Ω

(St+δ (x, y) − St (x, y))R(t + δ, y)µ(dy) µ(dx) Z 2 2 (St+δ (x, y) − St (x, y)) µ(dy) R(t + δ, y) µ(dy) µ(dx)

Ω

≤ C(∥Q(t + δ) − Q(t)∥2H 1 , which proves the bound ⟨R(t + δ), (St+δ − St )R(t + δ)⟩2 ≤ C(∥Q(t + δ) − Q(t)∥2H 1 . Combining the above bounds yields |⟨R(t + δ), St+δ R(t + δ)⟩ − ⟨R(t), St R(t)⟩|2 ≤ C(∥Q(t + δ) − Q(t)∥2H 2 . Furthermore, due to Lemma 5.7, ∂Q ∂t (t, x) , |∂t,xi Q(t, x)| , and ∂t,xi xj Q(t, x) are uniformly bounded in both space x ∈ Ω and time t ≥ 0. Therefore, we have 20

that Z t+δ |Q(t + δ, x) − Q(t, x)| ≤

| t

dQ (s, x)|ds ≤ Cδ, dt

Z t+δ

Therefore, | dJ dt | is in fact uniformly continuous for all t ≥ 0. Since J(t) is monotone decreasing (see Lemma 5.1) and J(t) ≥ 0, the monotone convergence theorem indicates that limt→∞ J(t) exists (and is finite). Therefore, the conditions for Barbalat’s Lemma (See page 205, Section 5.5 of [8]) are satisfied, proving the statement of this lemma. Lemma 5.9. For any sequence tn → ∞, there is a subsequence tnkj such that w

R(tnkj ) → R and ∥Q(tnkj ) − Q∥H 1 → 0 (where R and Q may depend on the subsequence). Consequently, ⟨R(tnkj ), SQ(tnk )) R(tnkj )⟩ → ⟨R, SQ R⟩,

(45)

DQ ∇θ η(x)cσ(wx + b) · DQ ∇θ η(y)cσ(wy + b) , µ0 .

(46)

where SQ is SQ (x, y) =

Proof. By Lemma 5.6, there is a subsequence tnk such that Rtnk → R for some R ∈ L2 (where R may depend upon the sequence tnk ). By Lemma 5.5, there is a further sub-subsequence tnkj where ∥Q(tnkj ) − Q∥H 1 → 0 for some Q ∈ H 1 (where Q may depend upon the sequence tnkj ). For notational convenience, define ti = tnkj . We first establish the strong convergence of Sti = SQ(ti ) . SQ (x, y) − SQ(ti ) (x, y) = = DQ [∇θ η(x)cσ(wx + b)] · DQ [∇θ η(y)cσ(wy + b)], µ0 − DQ(ti ) [∇θ η(x)cσ(wx + b)] · DQ(ti ) [∇θ η(y)cσ(wy + b)], µ0 = DQ − DQ(ti ) [∇θ η(x)cσ(wx + b)] · DQ [∇θ η(y)cσ(wy + b)], µ0 + DQ(ti ) [∇θ η(x)cσ(wx + b)] · DQ − DQ(ti ) [∇θ η(y)cσ(wy + b)], µ0 .

j=1

j,m=1

× |Q(ti , x) − Q(x)| + |Q(ti , y) − Q(y)| +

n X

|Qxj (ti , x) − Qxj (x)| + |Qxj (ti , y) − Qxj (y)| .

(47)

j=1

Similarly, due to Assumption 1 and Assumption 4, we have the uniform bound |SQ (x, y)| ≤ C. Now, let us next consider the decomposition ⟨R(ti ), SQ(ti ) R(ti )⟩ = ⟨R(ti ), (SQ(ti ) − SQ + SQ )R(ti )⟩ = ⟨R(ti ), SQ R(ti )⟩ + ⟨R(ti ), (SQ(ti ) − SQ )R(ti )⟩ = ⟨R(ti ), SQ R⟩ + ⟨R(ti ), SQ (R(ti ) − R)⟩ + ⟨R(ti ), (SQ(ti ) − SQ )R(ti )⟩ R

Let us begin by proving the function G = SQ R(x) is in L2 , where G(x) = S (x, y)R(y)µ(dy). Using the Cauchy-Schwarz inequality, Ω Q Z Ω

2 SQ (x, y)R(y)µ(dy) µ(dx) Ω Ω Z Z Z ≤ SQ (x, y)2 µ(dy) × R(y)2 µ(dy) µ(dx)

G(x)2 µ(dx) =

Z Z

Ω

(48)

≤ C.

Therefore, since SQ (x, y) is uniformly bounded and R ∈ L2 , SR ∈ L2 . By the definition of weak convergence, ⟨R(ti ), SQ R⟩ → ⟨R, SQ R⟩. The next term we consider is ⟨R(ti ), (SQ(ti ) − SQ )R(ti )⟩. Define the function H(t, x) where Z H(ti , x) = (SQ(ti ) (x, y) − SQ (x, y))R(ti , y)µ(dy). (49) Ω

Then, using the inequality (47) and the Cauchy-Schwarz inequality, Z

Z Z

H(ti , x) dx = (SQ(ti ) (x, y) − SQ (x, y))R(ti )(y)dy Ω Ω Z Z ≤C |Q(ti , x) − Q(x)| + |Q(ti , y) − Q(y)|

µ(dx)

Ω

n X

Ω

2 |Qxj (ti , x) − Qxj (x)| + |Qxj (ti , y) − Qxj (y)| × |R(ti , y)|dy µ(dx)

j=1

Z Z ≤C +

|Q(ti , x) − Q(x)| + |Q(ti , y) − Q(y)| Ω Ω n X

|Qxj (ti , x) − Qxj (x)| + |Qxj (ti , y) − Qxj (y)|

Z Ω

j=1

Z Z ≤C +

|R(ti , y)|2 µ(dy)µ(dx)

dy ×

|Q(ti , x) − Q(x)|2 + |Q(ti , y) − Q(y)|2

Ω Ω n X

|Qxj (ti , x) − Qxj (x)|2 + |Qxj (ti , y) − Qxj (y)|2

Z dy × |R(ti , y)|2 µ(dy)µ(dx) Ω

j=1

≤ C∥Q(ti ) − Q∥H 1 × ∥R(ti )∥2L2 ≤ C∥Q(ti ) − Q∥H 1 , where we have used the uniform bound for R(t) from Lemma 5.1 in the last line. Consequently, due to the strong convergence of Q(ti ) in H 1 , ∥H(ti )∥L2 → 0. This allows us to conclude ⟨R(ti ), (SQ(ti ) − SQ )R(ti )⟩2

⟨R(ti ), H(ti )⟩2

≤

∥R(ti )∥2L2 ∥H(ti )∥2L2 → 0,

since ∥R(t)∥2L2 is uniformly bounded. The third term to be analyzed is ⟨R(ti ), SQ (R(ti ) − R)⟩. Define V (t, x) by Z V (t, x) = SQ (x, y)(R(t, y) − R(y))µ(dy). Ω

For each x ∈ Ω, S(x, ·) ∈ L2 . Therefore, for each x ∈ Ω, V (ti , x) → 0 since R(ti ) weakly converges to R in L2 . Then, Z Z 2 2 |⟨R(ti ), SQ (R(ti ) − R)⟩| ≤ R(ti , x) µ(dx) × V (ti , x)2 µ(dx). (50) Ω

Ω

Using Young’s inequality and the previous bounds, Z Z 1 1 2 2 V (t, x) ≤ S(x, y) µ(dy) + (R(t, y) − R(y))2 µ(dy) 2 Ω 2 Ω Z Z Z 1 ≤ S(x, y)2 µ(dy) + R(t, y)2 µ(dy) + R(y)2 µ(dy)µ(dy) 2 Ω Ω Ω ≤ C < ∞. Therefore, V (t, x) is bounded by an integrable function on Ω and the dominated convergence theorem yields Z V (ti , x)2 µ(dx) → 0. (51) Ω

Combining this with the bound in equation (50), |⟨R(ti ), S(R(ti ) − R)⟩|2 → 0. We can now prove that the PDE residual of the neural network converges to zero as training time t → ∞. Theorem 5.10. The PDE residual R(t, x) = AQ Q(t, x) − g(x) of the neural network Q(t, x) converges to zero in L2 as training time t → ∞. In particular, this also means that the objective function J(t) → 0 and the neural network converges to a global minimizer. (52)

lim J(t) = lim ∥R(t)∥2L2 = 0.

t→∞

Proof. For any sequence tn → ∞, there exists a subsequence tnkj such that ⟨R(tnkj ), SQ(tnk ) R(tnkj )⟩ → ⟨R, SQ R⟩. Due to Lemma 5.8, we have that j

limt→∞ ⟨R(t), SQ(t) R(t)⟩ = 0. Therefore, ⟨R(tnkj ), SQ(tnk ) R(tnkj ) → 0. The j

limit point ⟨R, SQ R⟩ (which may depend upon the specific subsequence tnkj ) must then also satisfy ⟨R, SQ R⟩ = 0. Due to Theorem 4.4, ∥R∥L2 = 0. Therefore, for any sequence tn , there exists a subsequence tnkj such that J(tnkj ) = ∥R(tnkj )∥2L2 → 0. Since J(tnkj ) is a scalar, this implies that J(tn ) → 0. Since the sequence tn is arbitrary, limt→∞ J(t) = 0. We can now prove that limit neural network weakly converges to a fixed point which is a solution of the PDE.

Theorem 5.11. The neural network function Q(t, x) converges to the solution u(x) of the PDE (1) in H 1 as t → ∞: lim ∥Q(t) − u∥H 1 = 0.

t→∞

(53)

Proof. Recall from Lemma 5.4 that ∥Q(t)∥H 2 is uniformly bounded. Then, for any sequence of times tn → ∞, there exists a subsequence Q(tnk ) ∈ H 1 which converges to a limit point Q (which may depend upon the subsequence). Due to Lemma 5.2 and Lemma 5.3, each Q(tnk ) furthermore takes values in H01 . H01 is a closed subset of H 1 . Therefore, Q(tnk ) (strongly) converges to a limit point in H01 . From Theorem 5.10, we know that ∥AQ(tnk ) Q(tnk ) − g∥L2 → 0. Then, for any ϕ ∈ H01 , Z ϕ(x) AQ(tnk ) Q(tnk , x) − g(x) dx → 0.

(54)

(55)

Ω

For notational convenience, define the sequence Q(tnk ) as Q(tℓ ), where ℓ = nk . Integrating by parts yields n Z X

∂ϕ ∂Q ∂Q (x) (tℓ , x) + ci (x) (tℓ , x)ϕ(x) − d(x)Q(tℓ , x) ∂x ∂x ∂x i j i i=1 Ω + f (Q(tℓ , x), Qx (tℓ , x))ϕ(x) − g(x)ϕ(x) dx → 0. aij (x)

Since Q(tℓ ) converges strongly in H01 , f and its derivatives are uniformly bounded, and ϕ, ai j, ci , d are smooth functions on a compact domain, the limit point satisfies n Z X

∂ϕ ∂Q ∂Q (x) (x) + ci (x) (x)ϕ(x) − d(x)Q(x) ∂xi ∂xj ∂xi i=1 Ω + f (Q(x), Qx (x))ϕ(x) − g(x)ϕ(x) dx = 0. aij (x)

(56)

Since H01 is a closed space, the limit point Q of the subsequence tnk is an element of H01 . Note that equation (56) holds on the subsequence tnk for all ϕ ∈ H01 . Due to Corollary 4.5, Q is then the unique weak solution of the original PDE (1). Since the sequence tn is arbitrary and the limit point Q = u ∈ H01 is unique, Q(t) converges to u in H01 as t → ∞.

Convergence of neural network to limit PDE solution as N → ∞

In this section, we prove convergence of the finite-width neural network QN (t, x) to the solution of the limit neural network PDE for Q(t, x). We first establish a priori bounds on the parameter evolution. |θi (t) − θi (0)| ≤ CN 2β−1 N γ−β ≤ CN β−1+γ ,

(57)

where we have chosen γ + β < 1. The constant C = CT < ∞ is a finite constant depending on T , with t ∈ [0, T ]. This furthermore allows us to appropriately bound the moments of the trained parameters on t ∈ [0, T ] E[|θi (t)|p ] ≤ E[|θi (0) + CN β−1+γ |p ].

(58)

Importantly, since the parameters ci0 are initialized as bounded random variables on a compact set, |cit | ≤ |ci0 + CN β−1+γ | ≤ K < ∞.

(59)

Lemma 6.1. Define the pre-limit PDE residual for the neural network as RtN (y) = AQN (t) QN (y; θ(t)) − g(y). Then, there is a deterministic constant C2 < ∞ such that the PDE residual is almost surely bounded on a finite time interval [0, T ]: P[ sup

sup

N ≥1 t,x∈[0,T ]×Ω

|RtN (x)| ≤ C2 ] = 1,

(60)

where the constant C2 depends upon T . Proof. The pre-limit PDE residual satisfies Z ∂RtN (y) N −β N β = −α N ϕ N (AQN (t) QN (x; θ(t)) − g(x)) ∂t N × DQN (t) [∇θ Q (x; θ(t))]µ(dx) · DQN [∇θ QN (y; θ(t))].

(61)

Therefore, we can bound the pre-limit PDE residual as Z Z t N β (AQN (t) QN (x; θ(t)) − g(x)) |RtN (y)| ≤ |R0N (y)| + αN β−1 Ω

× DQN (t) [∇θ QN (x; θ(t))]µ(dx) · |DQN [∇θ QN (y; θ(t))]| ≤ |R0N (y)| + αN 2β−1

Z tZ 0

|RtN (x)| × |DQN (t) [∇θ QN (x; θ(t))]|µ(dx)

Ω

× |DQN [∇θ QN (y; θ(t))]| Z tZ N X |RtN (x)| × ≤ |R0N (y)| + αN −1 |DQN [∇θi cit σ(wti y + bit )]|µ(dx). 0

Ω

i=1

(62)

Taking a supremum over the RHS and then the LHS yields sup |RtN (y)| ≤

sup |R0N (y)| + C

y∈Ω

Z T 0

sup |RtN (x)| ×

x∈Ω

1 X (1 + |w0i |)2 . N i=1

(63)

Due to the uniform strong law of large numbers and since g is uniformly bounded on Ω, there is a constant C < ∞ such that P[supN ≥1 supy∈Ω |R0N (y)|] = 1. By the strong law of large numbers and since the i.i.d. RVs w0i have bounded PN moments, there is a constant C1 such that P[supN ≥1 N1 i=1 (1+|w0i |)2 < C1 ] = 1. Applying Gronwall’s inequality to (63) yields that there is a constant C2 such that P[supN ≥1 supt,x∈[0,T ]×Ω |RtN (x)| ≤ C2 ] = 1. Using the bound from Lemma 6.1, we can now the second term in R study N N N N (y, x)R U equation (16). Recall the definition LN t (x)µ(dx). Bt t t (y) = α Ω can be expressed as the empirical average N

BtN (y) =

1 X N ϕ N i=1

RtN (x)DQN (t) [∇θi cit σ(wti x + bit ]µ(dx) (64)

× ∇θi cit σ(wti y + bit ). LN t can be re-arranged in a similar form: N

LN t (y) =

1 X N i=1

RtN (x)DQN (t) [∇θi cit σ(wti x + bit ]µ(dx)

× ∇θi cit σ(wti y + bit )

Lemma 6.2. The effect of the clipping function ϕN vanishes almost surely as N → ∞: P[ lim

sup

N →∞ t,y∈[0,T ]×Ω

|∂yi BtN (y) − ∂yi LN t (y)| = 0] = 1,

where |i| ≤ 2. 27

(65)

R Proof. Define AN,k = RtN (x)DQN (t) [∇θi cit σ(wti x + bit ]µ(dx). Then, for |i| ≤ 2, t we can use (57) to establish the bound N

|∂yi BtN (y) − ∂yi LN t (y)| ≤

2C X (1 + |w0k |)2 |AN,k |1|AN,k |≥N γ . t t N

(66)

k=1

Due to the bound (57), we can bound the |AN,k | in terms of a polynomial t of the initial parameters w0i : |AN,k |≤C× t

|RtN (x)| × (1 + |w0k |)2 .

sup

(67)

t,x∈[0,T ]×Ω

In Lemma 6.1, it was proven that there exists a deterministic constant C2 such that P[supN ≥1 supt,x∈[0,T ]×Ω |RtN (x)| ≤ C2 ] = 1. Consequently, N

P[ sup

sup

N ≥1 t,y∈[0,T ]×Ω

|∂yi BtN (y) − ∂yi LN t (y)| ≤

C X (1 + |w0k |)4 1C(1+|w0k |)2 ≥N γ ] N k=1 N

≤

C X (1 + |w0k |)4 1C(1+|w0k |)4 ≥N γ ]. N k=1

PN 1

Define h(w) = C(1 + |w|)4 and H N = N ϵ > 0, P[H

> ϵ] ≤

k i=1 |h(w0 )||h(w0k )|≥N γ .

E[|h(w0k )||h(w0k )|≥N γ ] ϵ k

≤

For any

|h(w0 )| E[|h(w0k )| × N mγ

]

C ≤ N −mγ , ϵ where we have used the assumption that w0k has bounded moments. We now select m such that mγ > 1. Then, ∞ X

P[H N > ϵ] ≤

N =1

C < ∞. ϵ

Therefore, by the First Borel-Cantelli Lemma, P(H N > ϵ i.o.) = 0. Consequently, for any ϵ > 0, P(lim supN →∞ H N > ϵ) = 0. This implies that P(limN →∞ H N = 0) = 1. We can therefore conclude that P[ lim

sup

N →∞ t,y∈[0,T ]×Ω

|∂yi BtN (y) − ∂yi LN t (y)| = 0] = 1.

x,y∈Ω2

x∈Ω

N X

|QN xm (x; θ(t)) − Qxm (t, x)|),

(68)

m=1

PN a.s. a.s. where K N,2 = CN1 i=1 (|w0i |+1)2 . supt∈[0,T ] K N,1 (t) → 0 and limN →∞ K N,2 = C, where C is some finite constant. Proof. By direct computation we have ∂yi [UtN (y, x) − UQ(t) (y, x)] N i [∇θ cσ(wx + b)] − DQ(t) [∇θ cσ(wx + b)] , µt = ∂y ∇θ η(y)cσ(wy + b) · DQN t i N N + ∂y ∇θ η(y)cσ(wy + b) · DQ(t) [∇θ η(x)cσ(wx + b)], µt − µ0 i N + ∂y ∇θ η(y)cσ(wy + b) · DQ(t) [∇θ η(x)cσ(wx + b)], µ0 − µ0 . (69) The second line of (69) converges to zero uniformly in t, x, y ∈ [0, T ] × Ω × Ω at a rate O(N β−1+γ ) due to (57). Using a Taylor expansion as well as the bounds for trained parameters (57) N and (59), the Pn first term of (69) can be bounded as a function of |Q (x; θ(t)) − Q(t, x)| + m=1 |QN (x; θ(t)) − Q (t, x)|: xm xm | ∂yi ∇θ η(y)cσ(wy + b) · DQN (t) [∇θ cσ(wx + b)] − DQ(t) [∇θ cσ(wx + b)] , µN | t ≤ C1 |QN (x; θ(t)) − Q(t, x)| +

N X

N ′ 1 X |QN (x; θ(t)) − Q (t, x)| × 1 + |1 + w0i |2 . xm xm N ′ j=1 i =1

Recall that w0i are i.i.d. random variables. By the strong law of large numbers, N

1 X i a.s. |w0 + C2 |2 → E|w0i + C2 | = C3 , N i=1 as N → ∞. The third line of (69) can be bounded by: N

1 X [q(ci0 , w0i , bi0 , t, x, y) − Eµ0 [q(ci0 , w0i , bi0 , t, x, y)]]|,(70) t,x,y∈[0,T ]×Ω2 N i=1 sup

where q(c, w, b, t, x, y) = ∂yi ∇θ η(y)cσ(wy + b) · DQ(t) [∇θ η(x)cσ(wx + b)]. For a δ > 0, construct a finite sub-cover Uk = {(t, x, y) : |(t, x, y) − (tk , xk , yk )| ≤ δ with k = 0, 1, . . . , K for the compact set [0, T ] × Ω2 . Define AN (t, x, y) = PN 1 i i i i i i i=1 [q(c0 , w0 , b0 , t, x, y) − Eµ0 [q(c0 , w0 , b0 , t, x, y)]]. Note that the quantity N {AN (tk , xk , yk )}k=0,1,...,K converges almost surely to zero. We begin our calculations by considering the bound |AN (t, x, y) − AN (tk , xk , yk )| ≤ N 1 X ≤ [|q(ci0 , w0i , bi0 , t, x, y) − q(ci0 , w0i , bi0 , tk , xk , yk )| N i=1 + Eµ0 [|q(ci0 , w0i , bi0 , t, x, y) − q(ci0 , w0i , bi0 , tk , xk , yk )|] .

(71)

Using a Taylor expansion, the fact that σ has at least three bounded derivatives, the bound on ci0 , and the uniform bounds on the time derivatives of Q and Qx from Lemma 5.7: |q(ci0 , w0i , bi0 , t, x, y) − q(ci0 , w0i , bi0 , tk , xk , yk )| ≤ C(1 + |w0i |)5 × (|y − yk | + |x − yk | (72)

+ (t − tk )). Combining (71) with (72) and setting f (w) = (1 + |w|)5 , N C1 δ X f (w0j ) + C2 δ, sup |A (t, x, y) − A (tk , xk , yk )| ≤ N t,x,y∈Uk j=1 N

where f (w0j ) is a fifth-order polynomial. By the strong law of large numbers, the RHS converges almost surely to δE[f (w0j )]. Consequently, a.s.

lim sup

sup

(73)

|AN (t, x, y)| ≤ Cδ.

N →∞ t,x,y∈[0,T ]×Ω2 a.s.

Since δ was arbitrary, limN →∞ supt,x,y∈[0,T ]×Ω2 |AN (t, x, y)| = 0. Consequently, we have calculated the following bound sup |∂yi [UtN (y, x) − Ut (y, x)]| ≤ K N,1 (t)

x,y∈Ω2

+ K N,2 |QN (x; θ(t)) − Q(t, x)| +

N X

|QN xm (x; θ(t)) − Qxm (t, x)| ,

m=1 a.s.

C1 PN

a.s.

2 N,1 i (t) → 0 and limN →∞ K N,2 = where K N,2 = N i=1 (|w0 |+1) . supt∈[0,T ] K C, where C is some finite constant.

Theorem 6.4. ∂xi QN (t, x) converges almost surely to ∂xi Q(t, x) on t ∈ [0, T ] as N → ∞ for |i| ≤ 2: a.s.

sup |∂xi QN (t, x) − ∂xi Q(t, x)| → 0,

x∈Ω

where T is arbitrary. As a direct consequence, as N → ∞, a.s.

∥QN (t) − Q(t)∥H 2 → 0. Proof. For sufficiently large N , the difference V N (t, x) = Q(t, x) − QN (t, x) satisfies Z dV N (t, x) = −α UQ(t) (x, y) AQ(t) Q(t, y) − g(y) dt Ω N N − Ut (y, x) AQN (t) QN (x; θ(t)) − g(x) dy + (LN t (x) − Bt (x)) Z = −α (UQ(t) (x, y) − UtN (x, y)) AQ(t) Q(t, y) − g(y) dy ZΩ −α UtN (x, y) AQ(t) Q(t, y) − g(y) − (AQN (t) QN (t, y) − g(y)) dy Ω N + (LN t (x) − Bt (x)).

Summing over |i| ≤ 2 and using the bound from equation (32): X Z tZ E N (t, x) ≤ E N (0, x) + C |∂xi (UQ(s) (x, y) − UsN (x, y))|dyds |i|≤2

Ω

Z tZ

+C |∂xi UsN (x, y)| × E N (s, y)dyds. 0 Ω Z t i N + |∂xi LN (x) − ∂ B (x)|dt . t x t 0

Taking the supremum over the RHS: E N (t, x) ≤ sup E N (0, x) x∈Ω

X Z tZ + C |i|≤2

Z tZ

sup |∂xi (UQ(s) (x, y) − UsN (x, y))|dyds

Ω x,y∈Ω2

sup |∂xi UsN (x, y)| × sup E N (s, y)dyds

Ω x,y∈Ω2

y∈Ω

i N |∂xi LN t (x) − ∂x Bt (x)|

sup

(74)

t,x∈[0,T ]×Ω

Using the bound (74), the above equation can be simplified to give: E N (t, x) ≤ sup E N (0, x) + C sup K N,1 (t) x∈Ω

t∈[0,T ]

Z t +C

( sup |∂xi UsN (x, y)| + K N,2 ) × sup E N (s, y)ds x,y∈Ω2

y∈Ω

sup

|i|≤2

t,x∈[0,T ]×Ω

i N |∂xi LN t (x) − ∂x Bt (x)|.

Since the above equation holds for every x ∈ Ω, we can take supremum of the LHS: sup E N (t, x) ≤ sup E N (0, x) + C sup K N,1 (t) x∈Ω

x∈Ω

t∈[0,T ]

Z t +C

sup 0

t,x,y∈[0,T ]×Ω2

|∂xi UsN (x, y)| + K N,2

× sup E N (s, y)ds y∈Ω

+ K N,4 , (75) P i N where K N,4 = C |i|≤2 supt,x∈[0,T ]×Ω |∂xi LN t (x) − ∂x Bt (x)|. Due to the bound (57) and the assumptions on the initial parameters, we can prove that K N,3 = supt,x,y∈[0,T ]×Ω2 |∂xi UtN (x, y)| is almost surely bounded due to the strong law of large numbers. That is, there exists a finite constant C3 such that P[ sup

sup

E N (0, x) =

N ≥1 t,x,y∈[0,T ]×Ω2

|∂xi UtN (x, y)| ≤ C3 ] = 1,

for |i| ≤ 2. Recall that |∂xi η(x)N −β

N X cj0 σ(w0j x + bj0 )|, j=1

|i|≤2

where θ0j = (cj0 , w0j , bj0 ) are i.i.d. RVs and cj0 is mean zero. By the MarcinkiewiczPN j j j i −β Zygmund strong law of large numbers, AN i (x) = ∂x η(x)N j=1 c0 σ(w0 x+b0 ) 32

converges almost surely to zero for each x ∈ Ω. For a δ(N ) > 0, construct a finite sub-cover UkN = {x : |x − xk | ≤ δ(N ) with k = 1, . . . , K(N ) for the compact set Ω. We select δ(N ) = N −q for q = 2 − β. Then, there exists a finite sub-cover with K(N ) = ⌊Cδ(N )−n ⌋ = ⌊CN qn ⌋ where n is the spatial dimension of Ω ⊂ Rn . That is, for each N ≥ 1, we construct a new finite sub-cover, where the total number of members of the sub-cover grows with N . Define fi,j (x) = ∂xi η(x)σ(w0j x+bj0 ). Note that supx∈Ω | ∂x∂m [fi,j (x)]| ≤ C(1+ N |w0j |)3 . Define YiN (x) = AN i (x) − Ai (xk ). Using a Taylor expansion, YiN (x) = N −β

N X cj0 fi,j (x) − fi,j (xk ) j=1

N X = N −β cj0 ∇x fi,j (x∗j ) · (x − xk ), j=1

where x∗j is the line segment connecting x and xk . ∇x fi,j (x∗j ) is polynomially bounded by w0j ; cj0 is bounded, and cj0 is furthermore mean zero. Using the triangle inequality, sup |AN i (x)| ≤

x∈Ω

max 1≤m≤K(N )

|AN i (xm )|

sup x,m:|x−xm |≤δ(N ),m=1,...,K(N )

|YiN (x)|.

(76)

We first address the second term in (76) using the identity q + β = 2 sup x,m:|x−xm |≤δ(N ),m=1,...,K(N )

|YiN (x)| ≤ CN −q−β

N X

(1 + |w0j |)3

j=1

= CN −1 × N −1

N X

(1 + |w0j |)3 .

j=1

PN By the strong law of large numbers, the N −1 j=1 (1 + |w0j |)3 converges a.s. to a finite constant. Therefore, due to the additional factor of N −1 , P lim sup |YiN (x)| = 0 = 1. (77) N →∞ x,m:|x−xm |≤δ(N ),m=1,...,K(N )

We now consider the first term in (76) using the union bound for probabilities. For an (arbitrary) ϵ > 0, define the event GN = {max1≤m≤K(N ) |AN i (xm )| ≥ ϵ}. Then, K(N )

P[GN ] ≤

P[|AN i (xm )| ≥ ϵ].

m=1

(78)

Define the i.i.d. mean-zero random variables Wj = ∂xi η(xm )cj0 σ(w0j xm + For a positive integer p > 2+(2−β)n 2(2β−1) , we can use Markov’s inequality, the Marcinkiewicz-Zygmund inequality, and Jensen’s inequality to bound the probability terms in the sum in (78): bj0 ).

≤ Cϵ−4p N 2p(1−2β) . Therefore, substituting this bound into (78), P[

max 1≤m≤K(N )

−4p 2p(1−2β) |AN N i (xm )| ≥ ϵ] ≤ CK(N )ϵ

≤ C(N qn + 1)N 2p(1−2β) ϵ−4p ≤ C(N (2−β)n+2p(1−2β) + N 2p(1−2β) )ϵ−4p ≤ CN −2 ϵ−4p ,

(79)

where the last line is due to our a priori choice of p > 2+(2−β)n 2(2β−1) . Due to the bound (79), ∞ X N =1

max 1≤m≤K(N )

|AN i (xm )| ≥ ϵ] ≤ C < ∞.

(80)

a.s.

Therefore, supx∈Ω E N (0, x) → 0.

(81)

Applying Gronwall’s inequality to (75) yields sup E N (t, x) ≤ sup E N (0, x) + K N,4 + C sup K N,1 (t) x∈Ω

x∈Ω

t∈[0,T ]

+ t sup E N (0, x) + K N,4 + C sup K N,1 (t) x∈Ω

t∈[0,T ]

N,2 N,3 N,2 N,3 × (K + K ) × exp (K + K )t . Since we have that a.s.

sup E N (0, x) → 0, x∈Ω a.s.

K N,4 → 0, a.s.

sup K N,1 (t) → 0 t∈[0,T ]

P[ sup K N,2 ≤ C] = 1, N ≥1

P[ sup K N,3 ≤ C] = 1, N ≥1

we obtain that a.s.

sup E N (t, x) → 0, x∈Ω

as N → ∞. This concludes the proof of the theorem.

Global convergence of finite neural network to PDE solution for finite N as t → ∞

The convergence of the finite neural network QN (t, x) to the solution Q(t, x) of the limit PDE in Theorem 6.4 is not uniform in time t ≥ 0. Due to this lack of uniform convergence, it is often difficult in typical mean-field analysis to establish uniform-in-time error bounds on the pre-limit process QN (t, x). However, in the following analysis, we are able to prove uniform-in-time error bounds on the objective function JtN = J N (θ(t)) for the PDE residual of the finite neural network by leveraging the monotonicity of JtN .

The PDE residual is monotonically decreasing since Z t N N N ∇θ J N (θ(τ )) · GN (θ(τ ))dτ Jt = Js − α s

= JsN − αN

Z t N

−2β

≤ JsN − αN

Z t s

N X

N β ∇θi J N (θ(τ )) · ϕN N β ∇θi J N (θ(τ )) dτ

i=1

N −2β

N X

ϕN N β ∇θi J N (θ(τ )) · ϕN N β ∇θi J N (θ(τ )) dτ

i=1

≤ JsN . Then, for any 0 ≤ s ≤ t, E[JtN ] ≤ E[JsN ]. Due to Assumption 4, E[J0N ] ≤ C < ∞. Since JtN ≥ 0, E[JtN ] converges to a limit as t → ∞ due to the monotone convergence theorem. Theorem 6.4 directly implies that, for each t ≥ 0, JtN converges almost surely to J(t) as N → ∞. Then, for each t ≥ 0, E[JtN ] converges to J(t) due to the dominated convergence theorem. Theorem 5.10 has proven that limt→∞ J(t) = 0. Therefore, for any ϵ > 0, there exists a t ≥ 0 such that J(t) ≤ 2ϵ and an N0 such that |E[JtN ]−J(t)| ≤ 2ϵ for any N ≥ N0 . Consequently, we have that E[JtN ] = |E[JtN ] − J(t) + J(t)| ≤ |E[JtN ] − J(t)| + |J(t)| (82)

≤ ϵ. Furthermore, due to monotonicity, E[JsN ] ≤ ϵ,

for all s ≥ t and N ≥ N0 . That is, we can prove a uniform error bound for the neural network with a finite number of hidden units. Theorem 7.1. For any ϵ > 0, there exists a t ≥ 0 and an N0 such that E[JsN ] ≤ ϵ, for all s ≥ t and N ≥ N0 . Consequently, we obtain that lim lim E[JsN ] = 0.

N →∞ s→∞

Acknowledgements KS and JS were partially supported by the project “DMS-EPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks" (NSF DMS-2311500). 36

References [1] A. Jacot, F. Gabriel, and C. Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks, In Advances in Neural Information Processing Systems, 8571-8580, 2018. [2] D. Jiang, J. Sirignano, and S. Cohen. Global Convergence of Deep Galerkin and PINN Methods for Solving Partial Differential Equations, SIAM Journal on Financial Mathematics, 17(2), 2026. [3] M. Raissi, P. Perdikaris, and G. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics, 378, 686–707, 2019. [4] J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations, Journal of Computational Physics, 375, 1339–1364, 2018. [5] S. Cohen, D. Jiang, and J. Sirignano. Neural Q-learning for solving PDEs. Journal of Machine Learning Research, 24(236), 1–49, 2023 [6] L. Evans. Partial Differential Equations (2nd ed.), American Mathematical Society (Graduate Studies in Mathematics, Vol. 19), 2010. [7] D. Gilbarg and N. Trudinger. Elliptic Partial Differential Equations of Second Order, Springer, 2001. [8] S. Sastry. Nonlinear Systems: Analysis, Stability, and Control, Springer, 1999. [9] M. Renardy and R. Rogers. An introduction to partial differential equations, Springer, 2004.

Record · ID 405638 · SHA-256 2b99747c176da771
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.