IMA Journal of Numerical Analysis (2026) 00, 1–38 https://doi.org/DOI HERE
arXiv:2605.27133v1 [cs.LG] 26 May 2026
Deep-layer limit and stability analysis of the basic forward-backward-splitting induced network (II): learning problems X UAN L IN China Academy of Aerospace System and Innovation, 100088, Beijing, China School of Mathematical Sciences, Nankai University, 94 Weijin Road, 300071, Tianjin, China AND
C HUNLIN W U * School of Mathematical Sciences, Nankai University, 94 Weijin Road, 300071, Tianjin, China * Corresponding author. [email protected] [Received on 3 May 2026; revised on Date Month 2026; accepted on Date Month 2026] Deep unfolding neural networks derived from iterative optimization schemes and numerical ordinary/partial differential equations (ODEs/PDEs) have attracted much attention in data science over the last decade. Therein, numerous important network architectures were constructed from the basic forward-backward-splitting (FBS) algorithm. In this paper, we continue our research on the most basic FBS-induced network, an architecture unrolled from the original FBS algorithm by incorporating direct parameter relaxations. Following the difference/differential inclusion formulations in our previous forward system analyses, we here consider some theoretical aspects of corresponding learning problems. Under some mild assumptions, we establish a general convergence property of the training problem of the basic FBS-induced network to the learning problem of the deep-layer limit system, implying a Γconvergence argument showing that any cluster point of the optimal learning parameters for the network is a solution to the learning problem of the deep-layer limit system. A qualitative analysis of perturbation stabilities of these learning problems is also presented. A simple numerical experiment is conducted to validate our main general convergence result. Keywords: Forward-backward splitting, Deep unfolding/unrolling, Dynamical inclusion, Γ-convergence, Stability.
1. Introduction In this paper, we continue our previous study in [1] on deep-layer limit analysis of the socalled basic FBS-network. Here, "basic" means that the neural network is unrolled from the original FBS algorithm with direct parameter relaxation. Using our previous dynamical inclusion modeling of forward systems in [1], we here consider the deep-layer limit of the learning problem and related stability aspects. Indeed, there has been growing interest in deep-layer limit modeling and analysis for deep neural networks (DNNs) in data science in recent years, which helps to interpret the behaviors and mechanisms in DNNs. In particular, the well-known ResNets can be characterized as an © The Author(s) 2026. Published by Oxford University Press on behalf of the Institute of Mathematics and its Applications. All rights reserved.
2
XUAN LIN, CHUNLIN WU
ODE [2, 3, 4] or an ordinary differential inclusion (ODI) [5] in the deep-layer limit. Recently, Transformer architecture without layer normalization and its variants have been interpreted as a (coupled) ODE systems [6, 7]. Moreover, densely connected network (DenseNet) with a general nonlocal framework has been mathematically modeled as a nonlinear integral equation in the deep-layer limit [8]. For networks induced from numerical ODEs [9, 10] or timedependent PDEs [11, 12], such as trainable reaction diffusion [13], PDE-Net and its variant [14, 15], stable/reversible networks [4, 16, 17, 18], LM-ResNet [19], neural ODE [20], networks with multigrid structures [21, 22], and more from a continuous viewpoint [23], their deep-layer limits are naturally the systems from which they originated. These studies help to indicate some properties such as stabilities of network architectures [4, 5, 16, 17, 18, 19, 24], or design some novel and interesting architectures [4, 5, 16, 19, 21, 22, 23]. However, for those networks [25, 26, 27, 28, 29, 30] unrolled from iterative optimization algorithms [31, 32, 33] which play important roles in imaging-related inverse problems, there is so far little study on deep-layer limit analysis [1, 34]. Therein, [34] studied a dynamical system modeling based on ODE for learned primal-dual method, while [1] proposed a dynamical inclusion modeling for the most basic FBS-network. As the FBS algorithm and its variants have induced several related significant networks like LISTA [25], ALISTA [26], ISTA-Net [35], FISTA-Net [36], variational networks [29], PFBS-IR and PFBS-AIR [37], we now focus on this algorithm and continue our previous study on the most basic induced network structure in [1]. The main contributions of this paper can be summarized as follows • We show the existence of solutions to the learning problems of the basic FBS-network and its deep-layer limit system, under certain conditions. • We establish a general convergence result from the learning problem of the basic FBSnetwork to that of its deep-layer limit system, indicating Γ-convergence and related consequences. • We present perturbation stabilities of the learning problems for the basic FBS-network and its deep-layer limit system, in terms of the initial state, the observed data, and the label. These results demonstrate a deep consistency between the learning problems of the basic FBSnetwork and its deep-layer limit system, as well as their stabilities. Our analysis method admits a simplified variant capable of analyzing the deep-layer limit of other networks unrolled from the FBS type of algorithms, such as LISTA and ALISTA. The remainder of the paper is organized as follows. In Section 2, we formulate the learning problems of the FBS-network and its deep-layer limit under dynamical inclusion modeling. We establish in Section 3 not only the existence of solutions to the learning problems, but also convergence properties from the network training problem to the learning problem of the deep-layer limit system. In Section 4, we discuss the stabilities of both learning problems. Our main convergence result is verified numerically in Section 5. Section 6 concludes our study.
3
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
2. Learning problems of FBS-induced networks and related deep-layer limit systems The FBS algorithm is fundamental and efficient for solving the following signal and image processing related linear inverse problem minn
x∈R
n1 2
o ∥Ax − b∥22 + λ R(x) ,
where b ∈ Rm is an observation, A ∈ Rm×n (usually m ≤ n) is a measurement or representation matrix, R is a smooth or nonsmooth regularization function, and λ > 0 is a trade-off parameter. This classical algorithm can be represented as the following iterative scheme xk+1 = (I + α k λ ∂ R)−1 (xk − α k A⊤ (Axk − b)),
k = 0, 1, 2, · · · ,
(2.1)
where α k > 0 is the iterative step size, and ∂ R is the subdifferential of R. From Eq.(2.1) and related variants, numerous effective DNNs [1, 25, 26, 29, 35, 36, 37] have been constructed in sparse coding and imaging applications. 2.1. The basic FBS-network case By directly unrolling the FBS algorithm and relaxing parameters, we presented the basic FBSnetwork in [1], where dynamical inclusion modeling and some theoretical analyses for forward systems are also provided. As in [1], the forward systems of the basic FBS-network with N layers and its related deep-layer limit system are as follows N,k+1 x − xN,k + hN α N,k (AN,k )⊤ (AN,k xN,k − b) + hN α N,k λ N,k (∂ R)(xN,k+1 ) ∋ ⃗0, k = 0, 1, · · · , N − 1, N,0 0 x =x , ( α (t)(A A(t))⊤ (A A(t)xx(t) − b) +α α (t)λ λ (t)(∂ R)(xx(t)) ∋ ⃗0, a.e. t ∈ [0, T ], ẋx(t) +α x (0) = x0 ,
(2.2)
(2.3)
where N ∈ N+ is a predefined number of layers of the network, hN ≡ T /N, x0 is the given initial m×n , α N,: ≡ {α N,k }N−1 ⊂ R , λ N,: ≡ {λ N,k }N−1 ⊂ R , A , value, and AN,: := {AN,k }N−1 + + k=0 ⊂ R k=0 k=0 α , λ are learnable parameters that generalize the original A, α, λ . In supervised learning, we determine learnable parameters through minimizing the loss functions over given datasets. Suppose we are given a training set {(b j , y j )}Jj=1 with J samples, where (b j , y j ) ∈ Rm × Rn is the j-th pair of the input data and the label. Let {x0j }Jj=1 ⊂ Rn be the initial system states. We can then formulate the optimal control learning problem of the
4
XUAN LIN, CHUNLIN WU
FBS-network as follows: 1 J (1) N,: J N,: N,: N,: min J (A , α , λ ) = L (xN,N ∑ N j ; y j ) + β1 HN (A ) N,: N,: N,: J (A ,α ,λ )∈DN j=1 (2) (3) N,: N,: + β H (α ) + β H (λ ) 2 3 N N N,k N,k ⊤ N,k N,k s.t. ⃗0 ∈ xN,k+1 (A ) (A x j − b j ) + hN α N,k λ N,k (∂ R)(xN,k+1 − xN,k ), (2.4) j j + hN α j k = 0, 1, . . . , N − 1, N,0 0 xj = xj, j = 1, 2, . . . , J, where DN ⊂ (Rm×n )N × RN × RN is an admissible set for the control variables AN,: , α N,: , λ N,: with α N,: , λ N,: being nonnegative, L : Rn × Rn → R is a continuous, lower bounded (1) loss function (e.g., L (x; y) := 12 ∥x − y∥22 ) for a single sample, and HN : (Rm×n )N → R+ , (2)
(3)
HN : RN → R+ , HN : RN → R+ are the regularization functions with weight parameters β1 ≥ 0, β2 ≥ 0, β3 ≥ 0, respectively. Similarly, we have an optimal control learning problem in the continuous-time setting as 1 J J (1) (2) (3) A,α α ,λ λ ) = ∑ L (xx j (T ); y j ) + β1 H (A A) + β2 H (α α ) + β3 H (λ λ) inf J (A J j=1 A,α α ,λ λ )∈D D (A α (t)(A A(t))⊤ (A A(t)xx j (t) − b j ) +α α (t)λ λ (t)(∂ R)(xx j (t)), a.e. t ∈ [0, T ], s.t. ⃗0 ∈ ẋx j (t) +α x j (0) = x0j , j = 1, 2, . . . , J, (2.5) where D ⊂ L∞ ([0, T ]; Rm×n × R × R) is an admissible set for the control variables A , α , λ with α , λ being nonnegative, and H (1) : L∞ ([0, T ]; Rm×n ) → R+ , H (2) : L∞ ([0, T ]) → R+ , H (3) : L∞ ([0, T ]) → R+ are the regularization functions with weight parameters β1 ≥ 0, β2 ≥ 0, β3 ≥ 0, respectively. 2.2. Some other cases like LISTA The above inclusion-based problems (2.4) and (2.5) can be adapted to some other cases induced from the FBS algorithm. As LISTA is the earliest one in this topic, we here take it as an example. Recall from [1] that the forward systems of the LISTA network with N layers
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
5
(N ∈ N+ ) and its related deep-layer limit system are as follows ( N,k+1 x − xN,k + hN (W1 xN,k −W2 b) + hN θ (∂ ∥ · ∥1 )(xN,k+1 ) ∋ ⃗0, k = 0, 1, . . . , N − 1, xN,0 := x0 . ( ẋx(t) + (W1x (t) −W2 b) + θ (∂ ∥ · ∥1 )(xx(t)) ∋ ⃗0,
a.e. t ∈ [0, T ],
x (0) = x0 .
(2.6)
(2.7)
where θ := αλ , W1 := αA⊤ A, and W2 := αA⊤ are all set to be learnable with consideration of the constraint θ > 0. Therefore, in supervised learning, the optimal control problem of the LISTA network (2.6) can be written as below: n J fJ (W1 ,W2 , θ ) := 1 ∑ L (xN,N ; y j ) + β1 H f(1) (W1 ) min J N j J e (W ,W ,θ )∈ D j=1 1 2 o f(2) (W2 ) + β3 H f(3) (θ ) + β2 H N,k N,k+1 s.t. ⃗0 ∈ xN,k+1 − xN,k ), j j + hN (W1 x j −W2 b) + hN θ (∂ ∥ · ∥1 )(x j k = 0, 1, . . . , N − 1, 0 xN,0 j = xj, j = 1, 2, . . . , J,
(2.8)
e ⊂ Rn×n × Rn×m × R is an admissible set with W1 ∈ Rn×n , W2 ∈ Rn×m , θ ∈ R, and where D (1) f f(2) : Rn×m → R+ , H f(3) : R → R+ are the regularization functions with H : Rn×n → R+ , H weight parameters β1 ≥ 0, β2 ≥ 0, β3 ≥ 0, respectively. Similarly, we write the optimal control problem of the related deep-layer limit system (2.7) as ( J f(1) (W1 ) fJ (W1 ,W2 , θ ) := 1 ∑ L (xx j (T ); y j ) + β1 H J inf J e (W1 ,W2 ,θ )∈D j=1 ) f(2) (W2 ) + β3 H f(3) (θ ) + β2 H s.t. ⃗0 ∈ ẋx j (t) + (W1x j (t) −W2 b) + θ (t)(∂ ∥ · ∥1 )(xx j (t)), a.e. t ∈ [0, T ], x j (0) = x0j , j = 1, 2, . . . , J.
(2.9)
6
XUAN LIN, CHUNLIN WU
2.3. The learning problems to be analyzed and further notations In the following sections, we will study the convergence and stability properties of the learning problems for the basic FBS-network and its deep-layer limit system. Since the number of samples J in practical applications is always finite, we can, without loss of generality, consider the single-sample case of (2.4) and (2.5), which we denote by (QN ) and (Q), respectively, as follows
n (1) min JN (AN,: , α N,: , λ N,: ) = L (xN,N ; y) + β1 HN (AN,: ) N,: N,: N,: (A ,α ,λ )∈DN o (2) (3) + β2 HN (α N,: ) + β3 HN (λ N,: )
(2.10) s.t. ⃗0 ∈ xN,k+1 − xN,k + hN α N,k (AN,k )⊤ (AN,k xN,k − b) + hN α N,k λ N,k (∂ R)(xN,k+1 ), k = 0, 1, . . . , N − 1, xN,0 = x0 , A,α α ,λ λ ) = L (xx(T ); y) + β1 H (1) (A A) + β2 H (2) (α α ) + β3 H (3) (λ λ) inf J (A (AA,αα ,λλ )∈D D (2.11) α (t)(A A(t))⊤ (A A(t)xx(t) − b) +α α (t)λ λ (t)(∂ R)(xx(t)), a.e. t ∈ [0, T ], s.t. ⃗0 ∈ ẋx(t) +α x (0) = x0 .
There is no essential difference between convergence and stability analysis for problems (2.10) (2.11) and that for multi-sample problems (2.4) (2.5). Moreover, similar to the forward analyses in [1], our analysis procedure can be simplified to discuss the learning problems in other cases like LISTA and its continuous-time analog. For our analyses on learning problems of the basic FBS-network and the related limit system in later sections, we give some assumptions on the regularizer R and the single-sample loss function L : (A1) R : Rn → R is convex on Rn ; (A2) ⃗0 ∈ (∂ R)(⃗0); (A3) there exists a constant M > 0, for each x ∈ Rn , either sup{∥z∥2 : z ∈ (∂ R)(x)} ≤ M∥x∥2 (case 1), or sup{∥z∥2 : z ∈ (∂ R)(x)} ≤ M (case 2); (A4) L : Rn × Rn → R is continuous and nonnegative. We mention that (A1)-(A3) have been proposed in [1] for forward system analyses with numerous useful examples. Examples for the loss function L satisfying (A4) include L (x; y) := 1p ∥x − y∥ pp with any given p ∈ (0, +∞). Note that when p = 2, the function L is exactly the widely-used empirical mean square error (MSE) in deep learning literature.
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
7
According to the forward stability results in [1], we consider the regularization functions of the problem (QN ) and (Q) as follows. For some p ∈ [1, +∞),
(1)
1 N−1
(2)
1 N−1
HN (AN,: ) := ψ
∥AN,k ∥2p , ∀AN,k ∈ Rm×n , ∀k = 0, 1, . . . , N − 1, ∀N ∈ N+ , N ∑ k=0
HN (α N,: ) := ψ (3)
HN (λ N,: ) := ψ
N,k p |α | , ∀α N,k ∈ R+ , ∀k = 0, 1, . . . , N − 1, ∀N ∈ N+ , ∑
N k=0
1 N−1
|λ N,k | p , ∀λ N,k ∈ R+ , ∀k = 0, 1, . . . , N − 1, ∀N ∈ N+ , N ∑ k=0
A) := ψ H (1) (A
1 Z
α ) := ψ H (2) (α λ ) := ψ H (3) (λ
T [0,T ] 1 Z T [0,T ] 1 Z T
A(t)∥2p dt , ∀A A ∈ L p ([0, T ]; Rm×n ), ∥A α (t)| p dt , ∀α α ∈ L p ([0, T ]), |α λ (t)| p dt , ∀λ λ ∈ L p ([0, T ]), |λ
[0,T ]
unless otherwise specified. We use the following assumption for ψ that (A5) ψ : R → R+ is locally Lipschitz continuous. Note that if ψ ≡ 0, the problems (2.10) (2.11) reduce to a special case without parameter regularization. We recall some notations from [1]. We denote by Bn (z; r) and Bn (z; r) in Rn the ndimensional open ball centered at z with radius r > 0 and its closure, respectively. For N ∈ N+ , the interval [0, T ] is partitioned into N subintervals as follows:
[0, T ] :=
N−2 [h
T T [h T i k , (k + 1) (N − 1) , T , N N N k=0
where each subinterval is denoted by ΩNk for k ∈ {0, 1, . . . , N − 1}. Let X represent a finite dimensional Banach space, and denote XN ≡ X × X × · · · × X. We define a piecewise constant | {z } N
extension operator IN : XN → L∞ ([0, T ]; X) such that for every V ∈ XN , N−1
(IN V )(t) := ∑ Vk χΩN (t), k=0
k
∀t ∈ [0, T ],
8
XUAN LIN, CHUNLIN WU
( 1, t ∈ C ,
being the characteristic function of the set C . Additionally, we 0, t ∈ / C. employ a projection operator PN : L1 ([0, T ]; X) → XN , such that for every V ∈ L1 ([0, T ]; X), with χC (t) :=
(PNV )k :=
1 |ΩNk |
Z ΩN k
V (s)ds,
∀k ∈ {0, 1, . . . , N − 1}.
We thus can connect the system (2.10) and the system (2.11) by introducing the functional A,α α ,λ λ ) := JN ◦ PN as follows. For any given N ∈ N+ , we let the function (JN ◦ PN )(A A,α α ,λ λ) ∈D. JN (PN A , PN α , PN λ ) for any (A We also introduce some further notations and conventions. For any function U ∈ L1 ([0, T ]; Rm×n ), we extend its definition by setting U (t) := 0 for t ∈ R\[0, T ], while still denoting the extended function as U . Hence, the shift operator τh is defined such that for any h ∈ R and any function U ∈ L1 (R; Rm×n ), (τhU )(t) := U (t + h),
∀t ∈ R.
We endow the space Rm×n × R × R with the norm ∥(U, η, ζ )∥Rm×n ×R×R := ∥U∥2 + |η| + |ζ |,
∀(U, η, ζ ) ∈ Rm×n × R × R.
We use this to further equip norms for spaces (Rm×n )N × RN × RN and L p ([0, T ]; Rm×n × R × R). In particular, the ℓ p (p ∈ [1, +∞]) norm of (U N,: , η N,: , ζ N,: ) ∈ (Rm×n )N × RN × RN is as follows ∥(U N,: , η N,: , ζ N,: )∥ℓ p ((Rm×n )N ×RN ×RN ) N−1 1 T p ∥(U N,k , η N,k , ζ N,k )∥Rp m×n ×R×R , ∑ N k=0 := max ∥(U N,k , η N,k , ζ N,k )∥Rm×n ×R×R , 0≤k≤N−1
p ∈ [1, +∞), p = +∞.
For p ∈ [1, +∞], the norm in L p ([0, T ]; Rm×n × R × R) naturally reads
U ,η η ,ζζ )∥L p ([0,T ];Rm×n ×R×R) := ∥(U
Z
[0,T ]
1 U (s),η η (s),ζζ (s))∥Rp m×n ×R×R ds p , ∥(U
ess sup
U (s),η η (s),ζζ (s))∥2 , s∈[0,T ] ∥(U
p ∈ [1, +∞), p = +∞.
U ,η η ,ζζ ) ∈ L p ([0, T ]; Rm×n × R × R). Note that the above factor NT is introduced for for any (U consistency between (Rm×n )N × RN × RN and L p ([0, T ]; Rm×n × R × R).
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
9
3. Convergence properties from the learning problem of the basic FBS-network to that of its deep-layer limit system We start from the existence of solutions to (2.10) and (2.11). 3.1. Existence of solutions to the learning problems of the basic FBS-network and its deep-layer limit system 3.1.1. Existence of solutions to the learning problem (2.10) The existence of solutions to the learning problem (2.10) is straightforward by noting the following lemma, which is included here for completeness. L EMMA 3.1 Suppose the assumptions (A1)(A2) hold and a vector y ∈ Rn be given. Denote 1 the minimizer of the optimization problem minx∈Rn {R(x) + 2ρ ∥x − y∥22 } with ρ > 0 as xρ = proxρR (y). If ∆ρ → 0, then xρ+∆ρ → xρ . 1 Proof. By denoting fρ (x) := R(x) + 2ρ ∥x − y∥22 , we first show
fρ (xρ ) = lim fρ (xρ+∆ρ ),
(3.1)
∆ρ→0
i.e., R(xρ ) +
1 ρ 1 ρ+∆ρ ∥x − y∥22 = lim R(xρ+∆ρ ) + ∥x − y∥22 . ∆ρ→0 2ρ 2ρ
By the definition, we have 1 ρ+∆ρ 1 ρ ∥x − y∥22 ≤R(xρ+∆ρ ) + ∥x − y∥22 , 2ρ 2ρ 1 1 R(xρ+∆ρ ) + ∥xρ+∆ρ − y∥22 ≤R(xρ ) + ∥xρ − y∥22 . 2(ρ + ∆ρ) 2(ρ + ∆ρ) R(xρ ) +
(3.2) (3.3)
It then follows that 1 ρ 1 ρ+∆ρ ∥x − y∥22 ≤ lim inf R(xρ+∆ρ ) + ∥x − y∥22 , ∆ρ→0 2ρ 2ρ (3.4) 1 1 ρ+∆ρ ρ+∆ρ 2 ρ ρ 2 lim sup R(x )+ ∥x − y∥2 ≤ lim sup R(x ) + ∥x − y∥2 . 2(ρ + ∆ρ) 2(ρ + ∆ρ) ∆ρ→0 ∆ρ→0 (3.5) R(xρ ) +
10
XUAN LIN, CHUNLIN WU
Since the nonexpansiveness of the proximal operator and proxρR (⃗0) =⃗0 indicate ∥xρ ∥2 ≤ ∥y∥2 for every ρ > 0, we obtain 1 ρ+∆ρ lim sup R(xρ+∆ρ ) + ∥x − y∥22 2ρ ∆ρ→0 1 ∥xρ+∆ρ − y∥22 = lim sup R(xρ+∆ρ ) + 2(ρ + ∆ρ) ∆ρ→0 1 ≤ lim sup R(xρ ) + ∥xρ − y∥22 (by Eq.(3.5)) 2(ρ + ∆ρ) ∆ρ→0 =R(xρ ) +
1 ρ ∥x − y∥22 2ρ
≤ lim inf R(x ∆ρ→0
ρ+∆ρ
1 ρ+∆ρ )+ ∥x − y∥22 2ρ
(by Eq.(3.4)),
which proves Eq.(3.1). We then show the lemma by using Eq.(3.1). Since fρ (x) is strongly convex with the constant ρ1 > 0, then for every g ∈ ∂ f (xρ ), 1 ρ+∆ρ ∥x − xρ ∥22 ≤ fρ (xρ+∆ρ ) − fρ (xρ ) − g⊤ · (xρ+∆ρ − xρ ). 2ρ As xρ is the minimizer of fρ , we can choose g = ⃗0 and complete the proof.
□
Now we show that the learning problem (QN ) can attain its minimum. T HEOREM 3.2 Suppose the assumptions (A1)(A2)(A4)(A5) hold. Let the observed data b ∈ Rm , the initial value x0 ∈ Rn , and the label y ∈ Rn be given. Consider the optimal control problem (QN ). If the admissible set DN is closed and bounded, then (QN ) has a solution. Proof. By Lemma 3.1, it is not difficult to see that for every k = 0, 1, . . . , N −1, the state xN,k+1 is continuous with respect to (w.r.t.) (AN,: , α N,: , λ N,: , xN,k ), and thus xN,N is continuous w.r.t. (AN,: , α N,: , λ N,: ). Therefore, by the assumption of L , we see that the objective functional JN is continuous w.r.t. (AN,: , α N,: , λ N,: ). If the admissible set DN is closed and bounded, the optimal control problem (QN ) can attain its minimum over the feasible set of learnable parameters. □ In the following discussion, we denote the solution set of the problem (QN ) as SN .
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
11
3.1.2. Existence of solutions to the learning problem (2.11) We now provide an existence theorem of solutions to the optimal control problem to the deeplayer limit system as the continuous-time analog of that of the basic FBS-network. We first give two lemmas, which indicate a Bochner version of Kolmogorov-Riész-Fréchet Theorem [38, Thm 4.26] and the sufficiency of the upper semicontinuity (u.s.c.) of a set-valued mapping to its outer semicontinuity (o.s.c.). L EMMA 3.3 (Matrix-valued version of Kolmogorov-Riész-Fréchet Theorem) Let F be a bounded set in L p (R; Rm×n ) with p ∈ [1, +∞). Assume that U ∥L p (R;Rm×n ) = 0 uniformly in U ∈ F , lim ∥τhU −U
|h|→0
U ∥L p (R;Rm×n ) < ε i.e., ∀ε > 0, ∃δ > 0 such that ∀h ∈ R with |h| < δ , the inequality ∥τhU −U holds for all U ∈ F . Then the closure of F |Ω in L p (R; Rm×n ) is compact for any measurable set Ω ⊂ R with finite measure. Here F |Ω denotes the restrictions to Ω of the functions in F . U i, j (t)| ≤ ∥U U (t)∥2 for a.e. t ∈ Ω, any i ∈ {1, 2, · · · , m}, and any j ∈ Proof. By noting |U {1, 2, · · · , n}, we can apply the classical Kolmogorov-Riész-Fréchet Theorem A.2 to an entry V i, j : V ∈ F } to obtain its compactness. By progressively extracting mn (a finite Fi, j ≡ {V n U (t)∥2p ≤ Cm,n,p ∑m U i, j (t)| p for a.e. t ∈ Ω with number) times of subsequences and ∥U i=1 ∑ j=1 |U some certain constant Cm,n,p > 0, we derive the compactness of F |Ω . □ Clearly, we can extend Lemma 3.3 to the case for F being a bounded set in L p (R; Πi∈Θ Rmi ×ni ) with a finite index set Θ, which can be seen as a Bochner version of Lemma 3.3. L EMMA 3.4 (U.s.c. implies o.s.c) Let the set-valued mapping F : Rn ⇒ Rm be closed-valued. If F is u.s.c., then F is also o.s.c.. Proof. Consider any given y ∈ lim supx→x F(x). Thus there exist xν → x and yν ∈ F(xν ) such that yν → y. Since F is u.s.c., then we have dist(yν , F(x)) → 0. Note that F is closed-valued, then if yν → y, we can derive y ∈ F(x). Hence, lim supx→x F(x) ⊆ F(x), which completes the proof. □ Now, we give the existence of the solution to the problem (Q). T HEOREM 3.5 Suppose the assumptions (A1)-(A5) hold and p ∈ [1, +∞). Let the observed A,α α ,λ λ ) with data b ∈ Rm , the initial value x0 ∈ Rn , and the label y ∈ Rn . Recall D as a set of (A α , λ both being nonnegative. Consider the optimal control problem (Q) (2.11) of the related deep-layer limit system. If the following conditions
12
XUAN LIN, CHUNLIN WU
(1) (Uniform boundedness in L∞ space) D is bounded in L∞ ([0, T ]; Rm×n × R × R); (2) (Closedness in L p space) D is closed in L p ([0, T ]; Rm×n × R × R); (3) (Equi-continuity in L p sense) D satisfies A,α α ,λ λ ) − (A A,α α ,λ λ )∥L p (R;Rm×n ×R×R) = 0, lim ∥τh (A
|h|→0
A,α α ,λ λ) ∈D, uniformly for (A
hold, then the minimization problem (Q) has at least a solution. Proof. We divide the proof into four steps. Step 1: By the Existence and Uniqueness Theorem [1, Thm.3.2], we see that for each A,α α ,λ λ ) ∈ D , the state inclusion (2.3) has a unique absolute continuous solution x , and the (A objecctive function J is well defined. For later bound estimations, we denote three bound A,α α ,λ λ) ∈D, constants MA > 0, Mα > 0, and Mλ > 0 such that for every (A A∥L∞ ([0,T ];Rm×n ) < MA , ∥A
α ∥L∞ ([0,T ]) < Mα , ∥α
λ ∥L∞ ([0,T ]) < Mλ , ∥λ
by the uniform boundedness given in assumption (1). A( j) ,α α ( j) ,λ λ ( j) )}+∞ Suppose {(A j=1 ⊂ D be a minimizing sequence of the problem (Q) in ( j) n (2.11), and x : [0, T ] → R be the unique associated state to the state inclusion (2.3) with A( j) ,α α ( j) ,λ λ ( j) ), j = 1, 2, · · · . Hence, x ( j) satisfies the tuple (A ( ( j) α ( j) (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b) +α α ( j) (t)λ λ ( j) (t)(∂ R)(xx( j) (t)) ∋ ⃗0, a.e. t ∈ [0, T ], ẋx (t) +α x ( j) (0) = x0 . (3.6) Due to the assumptions (1)-(3) and a Bochner version of Lemma 3.3, there exist a A( j) ,α α ( j) ,λ λ ( j) )}+∞ A∗ ,α α ∗ ,λ λ ∗) ∈ D subsequence of {(A j=1 ⊂ D (not relabeled) and a limit point (A ∗ ∗ (with α , λ both being nonnegative) such that Lp
A( j) ,α α ( j) ,λ λ ( j) ) → (A A∗ ,α α ∗ ,λ λ ∗ ), (A
j → +∞.
By Theorem A.1, this sequence has a subsequence (not relabeled) that converges a.e. on [0, T ]. 1 n Step 2: We will show the uniform boundedness of {xx( j) }+∞ j=1 in H ((0, T ); R ) and thus the ∗ existence of a weak cluster point x . By Bound Estimation Theorem [1, Thm.3.2], one has, for each j ∈ N+ , Z α ( j) (s)|∥A A( j) (s)∥22 ds ∥xx( j) (t)∥2 ≤∥x0 ∥2 · exp |α [0,t]
Z
+ [0,t]
Z
α ( j) (s)|∥A A( j) (s)∥2 ∥b∥2 · exp |α
[s,t]
α ( j) (r)|∥A A( j) (r)∥22 dr ds |α
13
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
≤∥x0 ∥2 · exp(Mα MA2 T ) + Mα MA T ∥b∥2 · exp(Mα MA2 T ) =: M0 ,
∀t ∈ [0, T ],
where M0 := M0 (MA , Mα , b, x0 , T ) ≥ 0 is clearly independent of t ∈ [0, T ] and j ∈ N+ . This leads to ∥xx( j) ∥L2 ([0,T ];Rn ) =
Z [0,T ]
∥xx( j) (t)∥22 dt
1
2
1
≤ T 2 M0 ,
∀ j ∈ N+ .
Moreover, for every j ∈ N+ and every t ∈ [0, T ], sup{∥z∥2 : z ∈ (∂ R)(xx( j) (t))} ≤ sup{∥z∥2 : z ∈ (∂ R)(Bn (⃗0; M0 + 1))} =: M1 , where the constant M1 := M1 (MA , Mα , b, x0 , T ) > 0 is independent of j ∈ N+ . Using these, we derive from the state inclusion (3.6) that, α ( j) (t)|∥A A( j) (t)∥2 (∥A A( j) (t)∥2 ∥xx( j) (t)∥2 + ∥b∥2 ) ∥ẋx( j) (t)∥2 ≤|α α ( j) (t)||λ λ ( j) (t)| sup{∥z∥2 : z ∈ (∂ R)(xx( j) (t))} + |α α ( j) (t)||λ λ ( j) (t)|M1 α ( j) (t)|∥A A( j) (t)∥22 M0 + |α α ( j) (t)|∥A A( j) (t)∥2 ∥b∥2 + |α ≤|α ≤Mα MA2 M0 + Mα MA ∥b∥2 + Mα Mλ M1 =: M2 ,
a.e. t ∈ [0, T ],
where the constant M2 := M2 (M1 , MA , Mα , Mλ , b, x0 ) > 0 is independent of j ∈ N+ . Hence, ∥ẋx( j) ∥L2 ([0,T ];Rn ) =
Z [0,T ]
∥ẋx( j) (t)∥22 dt
1
2
1
≤ T 2 M2 ,
∀ j ∈ N+ .
2 n x( j) }+∞ The boundedness of {xx( j) }+∞ j=1 and {ẋ j=1 in L ([0, T ]; R ) imply the boundedness 1 n x( j) }+∞ of {xx( j) }+∞ j=1 in H ((0, T ); R ), since {x j=1 are univariate functions. By Embedding Theorem [39, Thm 10.13] and noting the absolute continuity of x ( j) , we derive that x ( j) → x ∗ in C0 ([0, T ]; Rn ) uniformly up to a subsequence (and without relabeled). Besides, there exists a subsequence (not relabeled) such that for j → +∞, x ( j) ⇀ x ∗ in H 1 ((0, T ); Rn ), indicating by [39, E.g. 8.4(3), page 230] that
ẋx( j) ⇀ ẋx∗ in L2 ((0, T ); Rn ).
(3.7)
Step 3: Now we prove that x ∗ is the unique solution to the state inclusion (2.3) with A∗ ,α α ∗ ,λ λ ∗ ). Here we follow the proof framework in [40, Proof of Thm 2.1], parameters (A but we are under weaker assumptions on the differential inclusion and need to construct the subgradient function by ourselves carefully from the subdifferentiation term.
14
XUAN LIN, CHUNLIN WU
λ ( j) (t) possibly equals to zero, we Step 3-1: In this substep, for each j ∈ N+ , since α ( j) (t)λ ( j) n wish to construct a measurable mapping g : [0, T ] → R such that g ( j) (t) ∈ (∂ R)(xx( j) (t)) for each t ∈ [0, T ], and satisfies α ( j) (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b) +α α ( j) (t)λ λ ( j) (t)gg( j) (t) = ⃗0, ẋx( j) (t) +α
a.e. t ∈ [0, T ], (3.8)
2 n and moreover, {gg( j) }+∞ j=1 has a weak cluster point in L ([0, T ]; R ). According to the state inclusion (3.6), for each j ∈ N+ , there always exists a selection function ge( j) with ge( j) (t) ∈ (∂ R)(xx( j) (t)) for every t ∈ [0, T ] satisfying
α ( j) (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b) +α α ( j) (t)λ λ ( j) (t)e ẋx( j) (t) +α g ( j) (t) = ⃗0,
a.e. t ∈ [0, T ]. (3.9)
Note that ge( j) is not necessarily measurable on [0, T ]. For each j ∈ N+ , we define two measurable sets ( j)
E1 := {t ∈ [0, T ] : ẋx( j) (t) does not exist},
( j)
λ ( j) (t) ̸= 0}, E2 := {t ∈ [0, T ] : α ( j) (t)λ ( j)
( j)
( j)
where µ(E1 ) = 0. We try to construct g ( j) on two disjoint sets: E2 \E1 ( j) ( j) ( j) ( j) [0, T ]\(E2 \E1 ). On the one hand, it is easy to see that on E2 \E1 , ge( j) (t) := −
and
1 A( j) (t)xx( j) (t) − b)), α ( j) (t)(A A( j) (t))⊤ (A (ẋx( j) (t) +α λ ( j) (t) α ( j) (t)λ ( j)
( j)
is measurable. On the other hand, let us consider [0, T ]\(E2 \E1 ). Note that ∂ R is closedvalued and u.s.c. on Rn due to the assumptions (A1)-(A3) and [41, Prop 6.1.1]. Then by Lemma 3.4, we derive that ∂ R is o.s.c. on Rn . Since x ( j) ∈ C0 ([0, T ]; Rn ), then for each y ∈ lim supt→t (∂ R ◦ x ( j) )(t), there exist t ν → t (hence x ( j) (t ν ) → x ( j) (t)) and yν ∈ (∂ R)(xx( j) (t ν )) ≡ (∂ R ◦ x ( j) )(t ν ) such that yν → y. Thus, ∂ R ◦ x ( j) is also o.s.c. on [0, T ]. By [42, E.g. 14.9] (closed-valued + o.s.c. derive measurability) and Measurable Selection Theorem [42, Coro.14.6], there exists a measurable selection g ( j) : [0, T ] → Rn such that g ( j) (t) ∈ (∂ R ◦xx( j) )(t) for each t ∈ [0, T ]. Therefore, for each j ∈ N+ , we define the mapping g ( j) : [0, T ] → Rn as ge( j) (t), t ∈ E ( j) \E ( j) , 2 1 ( j) g (t) := g ( j) (t), t ∈ [0, T ]\(E ( j) \E ( j) ), 2 1 which is measurable on [0, T ] and satisfies the state equation (3.8).
15
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
( j) It is straightforward that ∈ (∂ R)(xx( j) (t)) ⊆ Bn (⃗0, M1 ) for a.e. t ∈ [0, T ] and every R g (t) ( j) j ∈ N+ , which indicates [0,T ] ∥gg (t)∥22 dt ≤ T M12 for every j ∈ N+ . Here, M1 is a constant introduced in Step 2. This indicates the existence of a subsequence of {gg( j) }+∞ j=1 (not relabeled) satisfying g ( j) ⇀ g ∗ in L2 ([0, T ]; Rn ) as j → +∞. Step 3-2: We will prove in this substep that in L2 ((0, T ); Rn ), as j → +∞,
α ( j) (A A( j) )⊤ (A A( j)x ( j) − b) +α α ( j)λ ( j)g ( j) ⇀ ẋx∗ +α α ∗ (A A∗ )⊤ (A A∗x ∗ − b) +α α ∗λ ∗g ∗ . (3.10) ẋx( j) +α Since ẋx( j) ⇀ ẋx∗ in L2 ((0, T ); Rn ) is known from Eq.(3.7), we next show A( j) )⊤ (A A( j)x ( j) − b) → α ∗ (A A∗ )⊤ (A A∗x ∗ − b), α ( j) (A
(3.11)
α ( j)λ ( j)g ( j) ⇀ α ∗λ ∗g ∗ ,
(3.12)
as j → +∞ in L2 ([0, T ]; Rn ). Using the inequality (∑ni=1 ai ) p ≤ n p−1 ∑ni=1 aip , Eq.(3.11) is deduced by α ( j) (A A( j) )⊤ (A A( j)x ( j) − b) −α α ∗ (A A∗ )⊤ (A A∗x ∗ − b)∥2L2 ([0,T ];Rn ) ∥α Z
α ( j) (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b) −α α ∗ (t)(A A∗ (t))⊤ (A A∗ (t)xx∗ (t) − b)∥22 dt ∥α
= [0,T ]
≤4
Z [0,T ]
α ( j) (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b) −α α ∗ (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b)∥22 dt ∥α
Z
+4 [0,T ]
Z
+4 [0,T ]
Z
+4 [0,T ]
≤4
Z [0,T ]
α ∗ (t)(A A( j) (t))⊤ (A A( j) (t)xx( j) (t) − b) −α α ∗ (t)(A A∗ (t))⊤ (A A( j) (t)xx( j) (t) − b)∥22 dt ∥α α ∗ (t)(A A∗ (t))⊤A ( j) (t)xx( j) (t) −α α ∗ (t)(A A∗ (t))⊤A ∗ (t)xx( j) (t)∥22 dt ∥α α ∗ (t)(A A∗ (t))⊤A ∗ (t)xx( j) (t) −α α ∗ (t)(A A∗ (t))⊤A ∗ (t)xx∗ (t)∥22 dt ∥α
α ( j) (t) −α α ∗ (t)∥2 · 1dt · 2Mα · MA2 (MA M0 + ∥b∥)2 ∥α
Z
+4 [0,T ]
Z
+4 [0,T ]
Z
+4 [0,T ]
A( j) (t) −A A∗ (t)∥2 · 1dt · 2MA · Mα2 (MA M0 + ∥b∥)2 ∥A A( j) (t) −A A∗ (t)∥2 · 1dt · 2MA · Mα2 MA2 M02 ∥A ∥xx( j) (t) −xx∗ (t)∥2 · 1dt · 2M0 · Mα2 MA4 1
α ( j) −α α ∗ ∥L p ([0,T ]) · T q · 2Mα · MA2 (MA M0 + ∥b∥)2 ≤4∥α
16
XUAN LIN, CHUNLIN WU 1
A( j) −A A∗ ∥L p ([0,T ];Rm×n ) · T q · 2MA · Mα2 (MA M0 + ∥b∥)2 + 4∥A 1
A( j) −A A∗ ∥L p ([0,T ];Rm×n ) · T q · 2MA · Mα2 MA2 M02 + 4∥A 1
+ 4∥xx( j) −xx∗ ∥L p ([0,T ];Rn ) · T q · 2M0 · Mα2 MA4 →0 + 0 + 0 + 0 = 0,
j → +∞.
In addition, Eq.(3.12) is due to that for every v ∈ L2 ([0, T ]; Rn ), α ( j)λ ( j)g ( j) −α α ∗λ ∗g ∗ ,vv⟩| = |⟨α
Z
α ( j) (t)λ λ ( j) (t)gg( j) (t) −α α ∗ (t)λ λ ∗ (t)gg∗ (t),vv(t)⟩dt ⟨α
[0,T ]
≤
Z
α ( j) (t)λ λ ( j) (t)gg( j) (t) −α α ∗ (t)λ λ ( j) (t)gg( j) (t),vv(t)⟩ dt ⟨α
[0,T ]
Z
λ ∗ (t)gg( j) (t),vv(t)⟩ dt α ∗ (t)λ λ ( j) (t)gg( j) (t) −α α ∗ (t)λ ⟨α
+ [0,T ]
Z
+
α ∗ (t)λ λ ∗ (t)gg( j) (t) −α α ∗ (t)λ λ ∗ (t)gg∗ (t),vv(t)⟩dt ⟨α
[0,T ] 1
1
1
1
α ( j) −α α ∗ ∥L2 p ([0,T ]) · T 2q · 2 2 Mα2 · Mλ M1 ∥vv∥L2 ([0,T ];Rn ) ≤∥α 1
1
1
1
λ ( j) −λ λ ∗ ∥L2 p ([0,T ]) · T 2q · 2 2 Mλ2 · Mα M1 ∥vv∥L2 ([0,T ];Rn ) + ∥λ Z
+
α ∗ (t)λ λ ∗ (t)vv(t)⟩dt ⟨gg( j) (t) −gg∗ (t),α
[0,T ]
→0 + 0 + 0 = 0,
j → +∞,
where we used Hölder inequality and g ( j) ⇀ g ∗ in L2 ([0, T ]; Rn ). One can then follow the last part of the proof framework of Theorem 2.1 in [40, pp.10-11] to check that x ∗ satisfies α ∗ (t)(A A∗ (t))⊤ (A A∗ (t)xx∗ (t) − b) +α α ∗ (t)λ λ ∗ (t)gg∗ (t) = ⃗0, ẋx∗ (t) +α
a.e. t ∈ [0, T ],
(3.13)
and g ∗ is a.e. a selection of ∂ R ◦xx∗ , that is, g ∗ (t) ∈ (∂ R)(xx∗ (t)) for a.e. t ∈ [0, T ]. A( j) ,α α ( j) ,λ λ ( j) ) = J (A A∗ ,α α ∗ ,λ λ ∗ ). Note that Step 4: Finally, we check that lim j→+∞ J (A A( j) ,α α ( j) ,λ λ ( j) ) − J (A A∗ ,α α ∗ ,λ λ ∗) J (A A( j) ) − H (1) (A A∗ ) ≤ L (xx( j) (T ); y) − L (xx∗ (T ); y) + β1 H (1) (A λ ( j) ) − H (3) (λ λ ∗) . α ( j) ) − H (2) (α α ∗ ) + β3 H (3) (λ + β2 H (2) (α
17
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
By the assumption (A4) and lim j→+∞ x ( j) (T ) = x ∗ (T ), one has lim j→+∞ |L (xx( j) (T ); y) − L (xx∗ (T ); y)| = 0. Besides, since A( j) ∥L p ([0,T ];Rm×n ) − ∥A A∗ ∥L p ([0,T ];Rm×n ) ≤ ∥A A( j) −A A∗ ∥L p ([0,T ];Rm×n ) → 0, ∥A
j → +∞,
one has, by the assumption (A5), that, as j → +∞, A( j) ) − H (1) (A A∗ ) = ψ H (1) (A
1
1 A( j) ∥Lp p ([0,T ];Rm×n ) − ψ A∗ ∥Lp p ([0,T ];Rm×n ) → 0, ∥A ∥A T T
α ( j) )−H (2) (α α ∗ )| = 0, lim j→+∞ |H (3) (λ λ ( j) )−H (3) (λ λ ∗ )| = 0. and similarly lim j→+∞ |H (2) (α Therefore, A( j) ,α α ( j) ,λ λ ( j) ) − J (A A∗ ,α α ∗ ,λ λ ∗) lim J (A
j→+∞
A( j) ,α α ( j) ,λ λ ( j) ) − J (A A∗ ,α α ∗ ,λ λ ∗) ≤ lim J (A j→+∞
A( j) ) − H (1) (A A∗ ) ≤ lim L (xx( j) (T ); y) − L (xx∗ (T ); y) + lim β1 H (1) (A j→+∞
j→+∞
α ( j) ) − H (2) (α α ∗ ) + lim β3 H (3) (λ λ ( j) ) − H (3) (λ λ ∗) + lim β2 H (2) (α j→+∞
j→+∞
=0 + 0 + 0 + 0 = 0, which completes the proof.
□
R EMARK 3.6 Of course, one can assume "compactness in L∞ topology" to show the same result, which seems simplifying the proof. However, this assumption is too restrictive. # Consider a sequence {I3N P3N α # }+∞ N=1 with α (t) := 1 [ T2 ,T ] (t) for every t ∈ [0, T ]. We see that it has no convergent subsequence in L∞ topology by [43, Chapter IV, Section 8, Thm.18, pp.297]. In the following discussion, we denote the solution set of (Q) as S, and by Theorem 3.5, S ̸= 0. / 3.2. Convergence properties of the learning problem of the basic FBS-network to that of its deep-layer limit system In this subsection, we establish our main general convergence result of the objective functional of the optimal control problem of the FBS-network to that of the deep-layer limit analog, which implies the Γ-convergence, and thus the convergence property of minimizers of learning problems.
18
XUAN LIN, CHUNLIN WU
T HEOREM 3.7 (Convergence property of objective functionals of learning problems) Suppose the assumptions (A1)-(A5) hold, and p ∈ [1, +∞). Let the observed data b ∈ Rm , the initial value x0 ∈ Rn , and the label y ∈ Rn . If F ⊂ L∞ ([0, T ]; Rm×n × R × R) is nonempty e, α e , λe ) ∈ F and any sequence {(A A(N) ,α α (N) ,λ λ (N) )}+∞ and bounded, then for any point (A N=1 ⊂ p L e, α e, α e , λe ) as N → +∞, J (A e , λe ) = lim (JN ◦ A(N) ,α α (N) ,λ λ (N) ) → (A F satisfying (A N→+∞
A(N) ,α α (N) ,λ λ (N) ) holds. PN )(A A(N) ,α α (N) ,λ λ (N) ) ∈ F , we denote {xN,k }Nk=0 as the network state of Proof. For a point (A the FBS-network (2.2) with parameters (PN A (N) , PN α (N) , PN λ (N) ). We then define the piecewise linear mapping x N as x N (t) := xN,k +
N N,k+1 (x − xN,k )(t − t N,k ), ∀t ∈ [t N,k ,t N,k+1 ], ∀k ∈ {0, 1, . . . , N − 1}. T
e, α e , λe ) ∈ F and any sequence {(A A(N) ,α α (N) ,λ λ (N) )}+∞ For any point (A N=1 ⊂ F satisfying p L (N) (N) (N) e e e , λ ) as N → +∞, we first derive A ,α α ,λ λ ) → (A , α (A e∥L p ([0,T ];Rm×n ) ≤ ∥IN PN A (N) − A e∥L p ([0,T ];Rm×n ) ∥IN PN A (N) ∥L p ([0,T ];Rm×n ) − ∥A e∥L p ([0,T ];Rm×n ) + ∥IN PN A e −A e∥L p ([0,T ];Rm×n ) ≤∥IN PN A (N) − IN PN A e∥L p ([0,T ];Rm×n ) + ∥IN PN A e −A e∥L p ([0,T ];Rm×n ) A(N) − A ≤∥A →0 + 0 = 0,
N → +∞,
where the last two steps use [1, Prop.3.7]. Similarly, we obtain limN→+∞ ∥IN PN α (N) − e ∥L p ([0,T ]) = 0, limN→+∞ ∥IN PN λ (N) − λe ∥L p ([0,T ]) = 0. Noting that α (1)
A(N) ) =ψ (HN ◦ PN )(A
1 N−1
1 N−1
∥(PN A (N) )k ∥2p = ψ hN ∥(PN A (N) )k ∥2p N ∑ T ∑ k=0
=ψ
1 T
k=0
∥IN PN A (N) ∥Lp p ([0,T ];Rm×n ) ,
we see that as N → +∞, (1)
A(N) ) = ψ (HN ◦ PN )(A
1
1 e∥ pp e), ∥IN PN A (N) ∥Lp p ([0,T ];Rm×n ) → ψ ∥A = H (1) (A m×n L ([0,T ];R ) T T (2)
according to the assumption (A5). Similarly, (HN λ (N) ) → H (3) (λe ) as N → +∞. PN )(λ
(3)
e ), (HN α (N) ) → H (2) (α ◦ PN )(α
◦
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
19
Lp e Lp Lp e ∞ (N) → e , λ (N) → A(N) }+∞ Besides, since {A A , α (N) → α λ, N=1 is bounded in L space and A N N,k N we obtain by [1, Cor.3.9] that x , which is constructed by {x }k=0 satisfying the basic FBS-network (2.2) with parameters (PN A (N) , PN α (N) , PN λ (N) ), converges uniformly to the e, α e , λe ). It follows that x of the differential inclusion (2.3) with parameters (A unique solution e xN (T ) → e x(T ) as N → +∞. Hence, by the assumption (A4), one has
L (xN,N ; y) = L (xxN (T ); y) → L (e x (T ); y),
N → +∞.
Therefore, we obtain A(N) ,α α (N) ,λ λ (N) ) lim (JN ◦ PN )(A i h (3) (2) (1) λ (N) ) α (N) ) + β3 (HN ◦ PN )(λ A(N) ) + β2 (HN ◦ PN )(α = lim L (xxN (T ); y) + β1 (HN ◦ PN )(A N→+∞
N→+∞
e) + β2 H (2) (α e ) + β3 H (3) (λe ) =L (e x (T ); y) + β1 H (1) (A e, α e , λe ), =J (A which completes the proof.
□
R EMARK 3.8 This theorem gives a general convergence of objective functional of the learning problems from the discrete- to continuous-time setting, over a relatively more general domain. However, we cannot borrow the usual techniques in optimization to derive the convergence of the minimizers directly from this result, because the form of objective functional keeps changing along N, and each of the functional is in general not strongly convex. The following corollary, although straightforward from Theorem 3.7, presents the Γconvergence from the objective functional of the FBS-network training problem to that of its continuous-time analog. C OROLLARY 3.9 (Γ-convergence of objective functionals of learning problems) Suppose the assumptions (A1)-(A5) hold, and p ∈ [1, +∞). Let the observed data b ∈ Rm , the initial value x0 ∈ Rn , and the label y ∈ Rn . If F ⊂ L∞ ([0, T ]; Rm×n × R × R) is nonempty and bounded, and it is closed in L p sense, then J (·, ·, ·) = Γ- lim JN ◦ PN (·, ·, ·) over L p ([0, T ]; Rm×n × N→+∞
R × R), where A,α α ,λ λ ) := J (A
( A,α α ,λ λ ), (A A,α α ,λ λ) ∈F, J (A
A,α α ,λ λ ) ∈ L p ([0, T ]; Rm×n × R × R)\F F, + ∞, (A ( A,α α ,λ λ ), (A A,α α ,λ λ) ∈F, JN ◦ PN (A A,α α ,λ λ ) := JN ◦ PN (A A,α α ,λ λ ) ∈ L p ([0, T ]; Rm×n × R × R)\F F. + ∞, (A
20
XUAN LIN, CHUNLIN WU
Proof. Let us check the "liminf" and "limsup" conditions; see Appendix A.3. We will use the closedness of F in L p ([0, T ]; Rm×n × R × R) and Theorem 3.7. e, α e , λe ) ∈ L p ([0, T ]; Rm×n × R × R) and a For the "liminf" condition, we arbitrarily take a (A Lp
p m×n × R × R) satisfying (A A(N) ,α α (N) ,λ λ (N) )}+∞ A(N) ,α α (N) ,λ λ (N) ) → sequence {(A N=1 ⊂ L ([0, T ]; R e, α e, α e , λe ) as N → +∞. If (A e , λe ) ∈ L p ([0, T ]; Rm×n × R × R)\F F , then the "liminf" (A e e e A α λ F condition holds trivially. If ( , , ) ∈ , it is enough to consider the case with A(N) ,α α (N) ,λ λ (N) )}+∞ {(A N=1 ⊆ F by the definition of JN ◦ PN , and the "liminf" condition also holds due to Theorem 3.7. e, α e , λe ) ∈ L p ([0, T ]; Rm×n × R × For the "limsup" condition, we arbitrarily take a (A p m×n e e e , λ ) ∈ L ([0, T ]; R F , then again the “limsup" condition holds R). If (A , α × R × R)\F e e e A(N) ,α α (N) ,λ λ (N) )}+∞ trivially. If (A , α , λ ) ∈ F , we can choose a sequence {(A N=1 ⊂ F satisfying p L (N) (N) (N) e e e A ,α α ,λ λ ) → (A , α , λ ) as N → +∞, and the “limsup" condition holds again by using (A
Theorem 3.7.
□
So far, the convergence of the objective function of the learning problem associated with the basic FBS-network to that of its corresponding deep limit system has been established. In what follows, we build connections between the minimizers of JN and JN ◦ PN , and combine them with the above Γ-convergence to derive the convergence properties of the learning problem minimizers of the basic FBS-network to those of its continuous-time counterpart. For convenience of description, we denote A,α α ,λ λ ) ∈ F }, PN F := {(PN A , PN α , PN λ ) : (A for F ⊆ L∞ ([0, T ]; Rm×n × R × R), and IN FN := {(IN AN,: , IN α N,: , IN λ N,: ) : (AN,: , α N,: , λ N,: ) ∈ FN }, A,α α ,λ λ ) : (PN A , PN α , PN λ ) ∈ FN }, [FN ] := {(A for FN ⊆ (Rm×n )N × RN × RN , N ∈ N+ . The following proposition and corollary build connections between the minimizers of JN and JN ◦ PN . P ROPOSITION 3.10 Suppose F ⊆ L∞ ([0, T ]; Rm×n × R × R). Let [PN F ] ⊆ F for any N ∈ N+ . Then the point ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ (Rm×n )N × RN × RN is a minimizer A∗,N ,α α ∗,N ,λ λ ∗,N ) ∈ L∞ ([0, T ]; Rm×n × R × of JN (·, ·, ·) in PN F if and only if the point (A ∗,N ∗,N ∗,N N,: ∗ N,: ∗ N,: A ,α α ,λ λ ) = ((A ) , (α ) , (λ )∗ ) is a minimizer of (JN ◦ R) satisfying PN (A PN )(·, ·, ·) in F . Proof. Our proof is motivated by [34].
21
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
"=⇒": We assume by contradiction that the point ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ (Rm×n )N × A∗,N ,α α ∗,N ,λ λ ∗,N ) ∈ is a minimizer of JN (·, ·, ·) in PN F , but the point (A ∞ m×n ∗,N ∗,N ∗,N N,: ∗ N,: ∗ N,: ∗ A ,α α ,λ λ ) = ((A ) , (α ) , (λ ) ) is not a L ([0, T ]; R × R × R) satisfying PN (A A#,N ,α α #,N ,λ λ #,N ) be a point in F satisfying minimizer of (JN ◦ PN )(·, ·, ·) over F . Let (A RN × RN
A#,N ,α α #,N ,λ λ #,N ) < (JN ◦ PN )(A A∗,N ,α α ∗,N ,λ λ ∗,N ). (JN ◦ PN )(A We see (PN A #,N , PN α #,N , PN λ #,N ) ∈ PN F , and λ #,N ) = JN (PN A #,N , PN α #,N , PN λ #,N ). A#,N ,α α #,N ,λ (JN ◦ PN )(A A∗,N ,α α ∗,N ,λ λ ∗,N ) = ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ PN F , According to PN (A ∗,N ∗,N ∗,N A ,α α ,λ λ ) ∈ [PN F ] ⊆ F , and thus (A
one has
λ ∗,N ) A#,N ,α α #,N ,λ λ #,N ) <(JN ◦ PN )(A A∗,N ,α α ∗,N ,λ (JN ◦ PN )(A =JN (PN A ∗,N , PN α ∗,N , PN λ ∗,N ) =JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ), which means JN (PN A #,N , PN α #,N , PN λ #,N ) < JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ), indicating a contradiction. A∗,N ,α α ∗,N ,λ λ ∗,N ) ∈ L∞ ([0, T ]; Rm×n × "⇐=": We assume by contradiction that the point (A A∗,N ,α α ∗,N ,λ λ ∗,N ) = ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) is a minimizer of (JN ◦ R × R) satisfying PN (A PN )(·, ·, ·) in F , but the point ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ (Rm×n )N × RN × RN is not a minimizer of JN (·, ·, ·) in PN F . Let ((AN,: )# , (α N,: )# , (λ N,: )# ) be a point in PN F satisfying JN ((AN,: )# , (α N,: )# , (λ N,: )# ) < JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ). We see (IN (AN,: )# , IN (α N,: )# , IN (λ N,: )# ) ∈ [PN F ] ⊆ F , and thus JN ((AN,: )# , (α N,: )# , (λ N,: )# ) =JN (PN IN (AN,: )# , PN IN (α N,: )# , PN IN (λ N,: )# ) =(JN ◦ PN )(IN (AN,: )# , IN (α N,: )# , IN (λ N,: )# ). A∗,N ,α α ∗,N ,λ λ ∗,N ) ∈ PN F According to ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) = PN (A ∗,N ∗,N ∗,N A ,α α ,λ λ ) ∈ F , one has (A JN ((AN,: )# , (α N,: )# , (λ N,: )# ) <JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) =JN (PN A ∗,N , PN α ∗,N , PN λ ∗,N ) A∗,N ,α α ∗,N ,λ λ ∗,N ), =(JN ◦ PN )(A
and
22
XUAN LIN, CHUNLIN WU
A∗,N ,α α ∗,N ,λ λ ∗,N ), which means (JN ◦PN )(IN (AN,: )# , IN (α N,: )# , IN (λ N,: )# ) < (JN ◦PN )(A leading to a contradiction. □ C OROLLARY 3.11 Suppose F ⊆ L∞ ([0, T ]; Rm×n × R × R). Let IN PN F ⊆ F for any N ∈ N+ . Then the point ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ (Rm×n )N × RN × RN is a minimizer of JN (·, ·, ·) in PN F if and only if the point (IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ) ∈ L∞ ([0, T ]; Rm×n × R × R) is a minimizer of (JN ◦ PN )(·, ·, ·) over F . Proof. "=⇒": We assume by contradiction that the point ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ (Rm×n )N × RN × RN is a minimizer of JN (·, ·, ·) in PN F , but the point (IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ) ∈ L∞ ([0, T ]; Rm×n × R × R) is not a minimizer of A#,N ,α α #,N ,λ λ #,N ) in F satisfying (JN ◦ PN )(·, ·, ·) in F . Let (A A#,N ,α α #,N ,λ λ #,N ) < (JN ◦ PN )(IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ). (JN ◦ PN )(A We see (PN A#,N , PN α #,N , PN λ #,N ) ∈ PN F , and A#,N ,α α #,N ,λ λ #,N ) = JN (PN A #,N , PN α #,N , PN λ #,N ). (JN ◦ PN )(A According to ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ PN F , N,: ∗ N,: ∗ (IN (A ) , IN (α ) , IN (λ N,: )∗ ) ∈ IN PN F ⊆ F , and thus
one
has
A#,N ,α α #,N ,λ λ #,N ) <(JN ◦ PN )(IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ) (JN ◦ PN )(A =JN (PN IN (AN,: )∗ , PN IN (α N,: )∗ , PN IN (λ N,: )∗ ) =JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ), which means JN (PN A #,N , PN α #,N , PN λ #,N ) < JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ), indicating a contradiction. "⇐=": We assume by contradiction that the point (IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ) ∈ ∞ L ([0, T ]; Rm×n × R × R) is a minimizer of (JN ◦ PN )(·, ·, ·) in F , but the point ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ (Rm×n )N × RN × RN is not a minimizer of JN (·, ·, ·) in PN F . Let ((AN,: )# , (α N,: )# , (λ N,: )# ) in PN F satisfying JN ((AN,: )# , (α N,: )# , (λ N,: )# ) < JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ). We see (IN (AN,: )# , IN (α N,: )# , IN (λ N,: )# ) ∈ IN PN F ⊆ F , and thus JN ((AN,: )# , (α N,: )# , (λ N,: )# ) =JN (PN IN (AN,: )# , PN IN (α N,: )# , PN IN (λ N,: )# ) =(JN ◦ PN )(IN (AN,: )# , IN (α N,: )# , IN (λ N,: )# ).
23
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
According to (IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ) ∈ F and ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) = (PN IN (AN,: )∗ , PN IN (α N,: )∗ , PN IN (λ N,: )∗ ) ∈ PN F , one has JN ((AN,: )# , (α N,: )# , (λ N,: )# ) <JN ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) =JN (PN IN (AN,: )∗ , PN IN (α N,: )∗ , PN IN (λ N,: )∗ ) =(JN ◦ PN )(IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ), which means (JN ◦ PN )(IN (AN,: )# , IN (α N,: )# , IN (λ N,: )# ) N,: ∗ N,: PN )(IN (A ) , IN (α )∗ , IN (λ N,: )∗ ), leading to a contradiction. □
<
(JN
◦
R EMARK 3.12 In fact, one cannot directly derive Corollary 3.11 from Proposition 3.10, but the proof framework of Corollary 3.11 is similar to that of Proposition 3.10. We are now at the position to prove the existence of cluster points of solutions of the problems {(QN )}+∞ N=1 , each of which is exactly a solution of the problem (Q) (2.11). T HEOREM 3.13 (Convergence property of minimizers of learning problems) Suppose that the assumptions (A1)-(A5) hold, p ∈ [1, +∞), D is bounded in L∞ ([0, T ]; Rm×n × R × R) and is closed, equi-continuous in L p sense. Assume IN PN D ⊆ D for any N ∈ N+ . Denote ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ PN D as a solution of the problem (QN ) (2.10) in PN D for any N ∈ N+ . Then there exist a subsequence {((ANi ,: )∗ , (α Ni ,: )∗ , (λ Ni ,: )∗ )}+∞ i=1 and a limit point A∗ ,α α ∗ ,λ λ ∗ ) ∈ D such that (A Lp
A∗ ,α α ∗ ,λ λ ∗ ), (INi (ANi ,: )∗ , INi (α Ni ,: )∗ , INi (λ Ni ,: )∗ ) → (A
as i → +∞.
A∗ ,α α ∗ ,λ λ ∗ ) being a solution to the problem (Q) (2.11) in D . with (A Proof. We define c(A A,α α ,λ λ ) := J
( A,α α ,λ λ ), (A A,α α ,λ λ) ∈D, J (A
A,α α ,λ λ ) ∈ L p ([0, T ]; Rm×n × R × R)\D D, + ∞, (A ( A,α α ,λ λ ), (A A,α α ,λ λ) ∈D, JN ◦ PN (A A,α α ,λ λ ) := J\ N ◦ PN (A A,α α ,λ λ ) ∈ L p ([0, T ]; Rm×n × R × R)\D D, + ∞, (A for our proof. Since ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ ) ∈ PN D is a minimizer of the problem (QN ) (2.10) over PN D , one has, by Corollary 3.11, that (IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ ) is a minimizer of (JN ◦ PN )(·, ·, ·) over D , i.e., a minimizer of J\ N ◦ PN (·, ·, ·) over L p ([0, T ]; Rm×n × R × R). Moreover, by IN PN D ⊆ D , a Bochner version of
24
XUAN LIN, CHUNLIN WU
Lemma 3.3 implies that {(IN (AN,: )∗ , IN (α N,: )∗ , IN (λ N,: )∗ )}+∞ N=1 has a subsequence A∗ ,α α ∗ ,λ λ ∗ ) ∈ D , i.e., {(INi (ANi ,: )∗ , INi (α Ni ,: )∗ , INi (λ Ni ,: )∗ )}+∞ converging to some point (A i=1 Lp
A∗ ,α α ∗ ,λ λ ∗ ), as N → +∞. (INi (ANi ,: )∗ , INi (α Ni ,: )∗ , INi (λ Ni ,: )∗ ) → (A p m×n × c(·, ·, ·) = Γ-limN→+∞ J\ Using the above facts and J N ◦ PN (·, ·, ·) over L ([0, T ]; R R × R) similarly in Corollary 3.9, we then apply the Fundamental Theorem of Γc(·, ·, ·), i.e., a solution to the A∗ ,α α ∗ ,λ λ ∗ ) is a minimizer of J Convergence A.4 to see that (A problem (Q) (2.11). □ 4. Stability properties of learning problems of the basic FBS-network and its deep-layer limit system In this section, we discuss the stability properties of the learning problems (2.10) and (2.11), i.e., the sensitivity of their optimums and optimal solutions w.r.t x0 , b, y. 4.1. The stability of the learning problem (2.10) of the basic FBS-network The following theorem describes the stability of the FBS-network in terms of the given initial value x0 , observed data b, and label y. Its proof is inspired by [44] for a bilevel optimization problem, and we put it in the appendix for completeness. T HEOREM 4.1 (Stability of the learning problem of the basic FBS-network w.r.t. (x0 , b, y)) Suppose the assumptions of the problem (QN ) in Theorem 3.2 hold and p ∈ [1, +∞]. Let n m n 0 n m n {((x0 )(r) , b(r) , y(r) )}+∞ r=1 ⊂ R × R × R converge to (x , b, y) ∈ R × R × R . For any given (r) e ): r ∈ N+ , we consider the following perturbed problem (Q N n f(r) (AN,: , α N,: , λ N,: ) := L (xN,N ; y(r) ) + β1 H (1) (AN,: ) min J N N (AN,: ,α N,: ,λ N,: )∈DN o (2) (3) + β2 HN (α N,: ) + β3 HN (λ N,: ) s.t. ⃗0 ∈ xN,k+1 − xN,k + hN α N,k (AN,k )⊤ (AN,k xN,k − b(r) ) + hN α N,k λ N,k (∂ R)(xN,k+1 ), k = 0, 1, . . . , N − 1, xN,0 = (x0 )(r) , (4.1) n o+∞ (r) N,: ∗,(r) N,: ∗,(r) N,: ∗,(r) and denote its solution set as Se . Let the sequence (A ) , (α ) , (λ ) N
(r) satisfy ((AN,: )∗,(r) , (α N,: )∗,(r) , (λ N,: )∗,(r) ) ∈ SeN , ∀r ∈ N+ . Then,
r=1
(1) {((AN,: )∗,(r) , (α N,: )∗,(r) , (λ N,: )∗,(r) )}+∞ r=1 has a convergent subsequence, and its all cluster points belong to SN , i.e., the solution set of the problem (QN );
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
25
(2) as r → +∞, inf
((AN,: )∗ ,(α N,: )∗ ,(λ N,: )∗ )∈SN
∥((AN,: )∗,(r) , (α N,: )∗,(r) , (λ N,: )∗,(r) ) − ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ )∥ℓ p ((Rm×n )N ×RN ×RN ) → 0.
(4.2)
e (r) ) converges to that of (QN ) as r → +∞, that is, (3) the optimal value of (Q N inf
(AN,: ,α N,: ,λ N,: )∈DN
f(r) (AN,: , α N,: , λ N,: ) → J N
inf
(AN,: ,α N,: ,λ N,: )∈DN
JN (AN,: , α N,: , λ N,: ),
r → +∞.
4.2. The stability of the learning problem (2.11) of the related deep-layer limit system The following theorem gives the stability of the deep-layer limit system in terms of the given initial value x0 , observed data b, and label y. Its proof is an infinite dimensional extension of that of Theorem 2.1 in [44]. T HEOREM 4.2 (Stability of the learning problem of the deep-layer limit system w.r.t. (x0 , b, y)) Suppose the assumptions of the problem (Q) in Theorem 3.5 hold and p ∈ [1, +∞). n m n 0 n m n Let {((x0 )(r) , b(r) , y(r) )}+∞ r=1 ⊂ R × R × R converge to (x , b, y) ∈ R × R × R . For any e (r) ): given r ∈ N+ , we consider the following perturbed problem (Q n f(r) (A A,α α ,λ λ ) := L ((xx(T ))(A A,α α ,λ λ ; (x0 )(r) , b(r) ); y(r) ) + β1 H (1) (A A) min J (AA,αα ,λλ )∈D D o α ) + β3 H (3) (λ λ) + β2 H (2) (α α (t)(A A(t))⊤ (A A(t)xx(t) − b(r) ) +α α (t)λ λ (t)(∂ R)(xx(t)), a.e. t ∈ [0, T ]; s.t. ⃗0 ∈ ẋx(t) +α x (0) = (x0 )(r) ; A∗,(r) ,α α ∗,(r) ,λ λ ∗,(r) )}+∞ and denote its solution set as Se(r) . Let the sequence {(A r=1 satisfy ∗,(r) ∗,(r) ∗,(r) (r) e A α λ (A ,α ,λ ) ∈ S , ∀r ∈ N+ . Then, p A∗,(r) ,α α ∗,(r) ,λ λ ∗,(r) )}+∞ (1) {(A r=1 has a convergent subsequence in L topology, and its all cluster +∞ ∗,(r) ∗,(r) ∗,(r) A α λ points of {(A ,α ,λ )}r=1 belong to S, i.e., the solution set of the problem (Q); ∗,(r) A α ∗,(r) ,λ λ ∗,(r) ) − (A A∗ ,α α ∗ ,λ λ ∗ )∥L p ([0,T ];Rm×n ×R×R) = 0; (2) limr→+∞ inf(AA∗ ,αα ∗ ,λλ ∗ )∈S ∥(A ,α (r) e ) converges to that of (Q) as r → +∞, that is, (3) the optimal value of (Q
inf
A,α α ,λ λ )∈D D (A
f(r) (A A,α α ,λ λ) → J
inf
A,α α ,λ λ )∈D D (A
A,α α ,λ λ ), J (A
Proof. Step 1: We first prove the following two preliminary results:
r → +∞.
26
XUAN LIN, CHUNLIN WU
f(r) (A A,α α ,λ λ ) − J (A A,α α ,λ λ )| = 0. (i) limr→+∞ sup(AA,αα ,λλ )∈D D |J +∞ (l) (l) (l) A ,α α ,λ λ )}l=1 ⊂ D A(l) ,α α (l) ,λ λ (l) ) − (ii) If the sequence {(A satisfies ∥(A (l) (l) e e, α e , λ )∥L p ([0,T ];Rm×n ×R×R) → 0 as l → +∞, then liml→+∞ J (A A ,α α ,λ λ (l) ) = (A e, α e , λe ). J (A A,α α ,λ λ ; x0 , b) as the value at T of the unique solution For clarity, we here denote (xx(T ))(A A,α α ,λ λ ) with the initial value x0 and the x of the differential inclusion (2.3) determined by (A observed data b. For (i), since D is bounded in L∞ ([0, T ]; Rm×n × R × R), [1, Thm.4.2] with p = +∞ provides that x(T ))(A A,α α ,λ λ ; (x0 )(r) , b(r) ) : (A A,α α ,λ λ) ∈D} ∪+∞ r=1 {(x
A,α α ,λ λ ; x0 , b) : (A A,α α ,λ λ) ∈D} {(xx(T ))(A
[
is a subset of a compact set (exactly a closed bounded ball) of Rn . Moreover, [1, Thm.4.2] and the boundedness of D indicate that, there exist two constants ce1 ≥ 0, ce2 ≥ 0 both independent A,α α ,λ λ) ∈D, of r, A , α , λ such that for any (A A,α α ,λ λ ; (x0 )(r) , b(r) ) − (xx(T ))(A A,α α ,λ λ ; x0 , b)∥2 ≤ ce1 ∥(x0 )(r) − x0 ∥2 + ce2 ∥b(r) − b∥2 . ∥(xx(T ))(A Hence, for any δ > 0, there exists r1,δ ∈ N+ such that for every integer r ≥ r1,δ , 1 A,α α ,λ λ ; (x0 )(r) , b(r) ) − (xx(T ))(A A,α α ,λ λ ; x0 , b)∥2 ≤ δ . ∥(xx(T ))(A 3 A,α α ,λ λ )∈D D (A sup
Similarly, limr→+∞ y(r) = y indicates that for any δ > 0, there exists r2,δ ∈ N+ such that for every integer r ≥ r2,δ , we have 1 ∥y(r) − y∥2 ≤ δ . 3 These two results indicate that for any δ > 0, there exists r3,δ := max{r1,δ , r2,δ } ∈ N+ such that for every integer r ≥ r3,δ , 2 A,α α ,λ λ ; (x0 )(r) , b(r) ) − (xx(T ))(A A,α α ,λ λ ; x0 , b)∥2 + ∥y(r) − y∥2 ≤ δ < δ . ∥(xx(T ))(A 3 A,α α ,λ λ )∈D D (A sup
By further noting the continuity assumption of L from (A4) and thus the uniform continuity of L over compact sets, we obtain that for any ε > 0, there exists r3,δ (ε) ≡ max{r1,δ (ε) , r2,δ (ε) } ∈ N+ such that for every integer r ≥ r3,δ (ε) , sup A,α α ,λ λ )∈D D (A
f(r) (A A,α α ,λ λ ) − J (A A,α α ,λ λ )| |J
27
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
=
sup A,α α ,λ λ )∈D D (A
=
sup A,α α ,λ λ )∈D D (A
A,α α ,λ λ ; (x0 )(r) , (b)(r) ); (y)(r) + β1 H (1) (A A) + β2 H (2) (α α ) + β3 H (3) (λ λ) L (xx(T ))(A A,α α ,λ λ ; x0 , b); y − β1 H (1) (A A) − β2 H (2) (α α ) − β3 H (3) (λ λ) −L (xx(T ))(A A,α α ,λ λ ; (x0 )(r) , (b)(r) ); (y)(r) − L (xx(T ))(A A,α α ,λ λ ; x0 , b); y L (xx(T ))(A
<ε, which proves (i). The proof of (ii) is similar to that of (i). By the assumption (A5) of ψ, we see that e)| ≤CD ∥A e∥L p ([0,T ];Rm×n ) → 0, A(l) ) − H (1) (A A(l) − A |H (1) (A
l → +∞,
e. Similarly, |H (2) (α e )| ≤ CD ∥α α (l) ) − H (2) (α α (l) − where CD > 0 is independent of A (l) and A e ∥L p ([0,T ]) and |H (3) (λ α λ (l) ) − H (3) (λe )| ≤ CD ∥λ λ (l) − λe ∥L p ([0,T ]) . Hence, for any ε > 0, there exists l1,ε ∈ N+ such that for every l ≥ l1,ε , 1 e)| + β2 |H (2) (α e )| + β3 |H (3) (λ A(l) ) − H (1) (A α (l) ) − H (2) (α λ (l) ) − H (3) (λe )| < ε. β1 |H (1) (A 2 Besides, by [1, Thm.4.2] and the boundedness of D , we know A(l) ,α α (l) ,λ λ (l) ; x0 , b) − (xx(T ))(A A,α α ,λ λ ; x0 , b)∥2 ∥(xx(T ))(A e∥L p ([0,T ];Rm×n ) + ∥α e ∥L p ([0,T ]) + ∥λ A(l) − A α (l) − α λ (l) − λe ∥L p ([0,T ]) ), ≤c(∥A e, α (l) , α e , λ (l) , λe . Hence, for any δ > 0, there where c > 0 is a constant independent of l, A (l) , A exists l2,δ ∈ N+ such that for every integer l ≥ l2,δ , e, α e , λe ; x0 , b)∥2 < δ . A(l) ,α α (l) ,λ λ (l) ; x0 , b) − (xx(T ))(A ∥(xx(T ))(A By using the continuity assumption of L in (A4) and the above result, we obtain that for any ε > 0, there exists l3,ε := max{l1,ε , l2,δ (ε) } ∈ N+ , such that for every l ≥ l3,ε , e, α e , λe )| A(l) ,α α (l) ,λ λ (l) ) − J (A |J (A A(l) ,α α (l) ,λ λ (l) ; x0 , b); y + β1 H (1) (A A(l) ) + β2 H (2) (α α (l) ) + β3 H (3) (λ λ (l) ) = L (xx(T ))(A e, α e) − β2 H (2) (α e , λe ; x0 , b); y − β1 H (1) (A e ) − β3 H (3) (λe ) −L (xx(T ))(A e, α e , λe ; x0 , b); y A(l) ,α α (l) ,λ λ (l) ; x0 , b); y − L (xx(T ))(A ≤ L (xx(T ))(A
28
XUAN LIN, CHUNLIN WU
e)| + β2 |H (2) (α e )| + β3 |H (3) (λ A(l) ) − H (1) (A α (l) ) − H (2) (α λ (l) ) − H (3) (λe )| + β1 |H (1) (A 1 1 < ε + ε = ε, 2 2 which proves (ii). Step 2: In this step, we will prove (1). The existence of the convergent subsequence of p A∗,(r) ,α α ∗,(r) ,λ λ ∗,(r) )}+∞ {(A r=1 in L topology is obvious by the uniform boundedness of D , the p equi-continuity of D in L sense, and a Bochner version of Lemma 3.3. A∗,∗ ,α α ∗,∗ ,λ λ ∗,∗ ) of We next show that, in L p topology, every cluster point (A Lp
A∗,(r) ,α α ∗,(r) ,λ λ ∗,(r) )}+∞ A∗,(ri ) ,α α ∗,(ri ) ,λ λ ∗,(ri ) ) → (A A∗,∗ ,α α ∗,∗ ,λ λ ∗,∗ ) {(A r=1 belongs to S. Assume that (A ∗,∗ ∗,∗ ∗,∗ A ,α α ,λ λ ) ∈ D . Using again the continuity of L in (A4), we as i → +∞. We see that (A obtain A∗,∗ ,α α ∗,∗ ,λ λ ∗,∗ ) J (A A∗,∗ ,α α ∗,∗ ,λ λ ∗,∗ ; x0 , b); y + β1 H (1) (A A∗,∗ ) + β2 H (2) (α α ∗,∗ ) + β3 H (3) (λ λ ∗,∗ ) =L (xx(T ))(A h A∗,(ri ) ,α α ∗,(ri ) ,λ λ ∗,(ri ) ; x0 , b); y + β1 H (1) (A A∗,(ri ) ) = lim L (xx(T ))(A i→+∞ i α ∗,(ri ) ) + β3 H (3) (λ λ ∗,(ri ) ) + β2 H (2) (α h A∗,(ri ) ,α α ∗,(ri ) ,λ λ ∗,(ri ) ; (x0 )(ri ) , (b)(ri ) ); (y)(ri ) ) + β1 H (1) (A A∗,(ri ) ) = lim L ((xx(T ))(A i→+∞ i α ∗,(ri ) ) + β3 H (3) (λ λ ∗,(ri ) ) + β2 H (2) (α h i A,α α ,λ λ ; (x0 )(ri ) , (b)(ri ) ); (y)(ri ) ) + β1 H (1) (A A) + β2 H (2) (α α ) + β3 H (3) (λ λ) , ≤ lim L ((xx(T ))(A i→+∞
A,α α ,λ λ) ∈D, ∀(A A,α α ,λ λ ; x0 , b); y) + β1 H (1) (A A) + β2 H (2) (α α ) + β3 H (3) (λ λ ), =L ((xx(T ))(A A,α α ,λ λ ), =J (A
A,α α ,λ λ) ∈D, ∀(A
A,α α ,λ λ) ∈D, ∀(A
where the second equality is due to Step 1 (ii), the third and the fifth equalities are both due α ∗,(ri ) ,λ λ ∗,(ri ) ) ∈ Se(ri ) . This indicates A∗,(ri ) ,α to Step 1 (i), and the fourth inequality is from (A ∗,∗ ∗,∗ ∗,∗ A ,α α ,λ λ ) ∈ S, and one has, similar to the derivation for Eq.(A.3), that (A f(ri ) (A A∗,(ri ) ,α α ∗,(ri ) ,λ λ ∗,(ri ) ) = lim J
i→+∞
inf
A,α α ,λ λ )∈D D (A
A,α α ,λ λ ). J (A
(4.3)
Step 3: We now prove (2) and (3) by contradiction. To prove (2), we assume that the sequence inf
A∗ ,α α ∗ ,λ λ ∗ )∈S (A
A∗,(r) ,α α ∗,(r) ,λ λ ∗,(r) ) − (A A∗ ,α α ∗ ,λ λ ∗ )∥L p ([0,T ];Rm×n ×R×R) ↛ 0. ∥(A
29
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
Then there exist an ε0 > 0 and a subsequence n o+∞ A∗,(ri ) ,α α ∗,(ri ) ,λ λ ∗,(ri ) ) − (A A∗ ,α α ∗ ,λ λ ∗ )∥L p ([0,T ];Rm×n ×R×R) inf ∥(A A∗ ,α α ∗ ,λ λ ∗ )∈S (A
i=1
such that for every i ∈ N+ , inf
A∗ ,α α ∗ ,λ λ ∗ )∈S (A
A∗,(ri ) ,α α ∗,(ri ) ,λ λ ∗,(ri ) ) − (A A∗ ,α α ∗ ,λ λ ∗ )∥L p ([0,T ];Rm×n ×R×R) ≥ ε0 . ∥(A ∗,(r )
∗,(r )
(4.4)
∗,(r )
A i j ,α α i j ,λ λ i j )}+∞ By Step 2, we can further extract a convergent subsequence {(A j=1 +∞ ∗,(r ) ∗,(r ) ∗,(r ) p ∗,+ ∗,+ ∗,+ A i ,α α i ,λ λ i )}i=1 in L topology with a limit point (A A ,α α ,λ λ ) ∈ S, and from {(A then ∗,(ri j )
inf
A∗ ,α α ∗ ,λ λ ∗ )∈S (A ∗,(ri j )
A ∥(A
∗,(ri j )
A ≤∥(A
α ,α
→0,
j → +∞,
λ ,λ
α ,α
∗,(ri j )
∗,(ri j )
λ ,λ
∗,(ri j )
A∗ ,α α ∗ ,λ λ ∗ )∥L p ([0,T ];Rm×n ×R×R) ) − (A
A∗,+ ,α α ∗,+ ,λ λ ∗,+ )∥L p ([0,T ];Rm×n ×R×R) ) − (A
which contradicts with Eq.(4.4). A similar argument and noting Eq.(4.3) give (3).
□
R EMARK 4.3 According to Theorem 4.1 and Theorem 4.2, if we set b(r) ≡ b, y(r) ≡ y, ∀r ∈ N+ , we can derive the perturbation stability of the minimums and the solutions to the discrete- and continuous-time control problems w.r.t. the initial value x0 . Similarly, we can obtain the perturbation stability w.r.t. b and y respectively under the settings fixing (x0 )(r) ≡ x0 , y(r) ≡ y or fixing (x0 )(r) ≡ x0 , b(r) ≡ b. 5. Numerical experiment In this section, we conduct a simple experiment to investigate the behavior of the training loss for the basic FBS-network, providing numerical validation for our main general convergence result Theorem 3.7. We examine how the training loss varies with an increasing number of network layers N in sparse signal reconstruction tasks. Note that our main result focuses on the behavior of the training problem rather than generalization, and we thus observe only the training loss here. All implementations are carried out in PyTorch and executed on 4 NVIDIA GeForce RTX 3060 GPUs. We construct a customized dataset {(b j , y j )}J+J j=1 with J = 16384 and J = 2048 being the numbers of training and validation samples, respectively. The dimensions of the observation vector b j ∈ Rm and the corresponding ground truth sparse signal vector y j ∈ Rn of the j-th sample are set to m = 256 and n = 1024, as in Eq.(2.2). The observations are generated as b j = Ay j + ε j for j = 1, 2, . . . , J + J using a fixed matrix A ∈ Rm×n , where {ε j }J+J j=1 are small observation errors. No additional preprocessing is applied.
30
XUAN LIN, CHUNLIN WU
We construct an N-layer FBS-network unfolded from the FBS structure as in Eq.(2.2) with hN = 1/N for N ∈ N+ . Each layer comprises a linear transformation followed by a nonlinear activation, for which we choose the soft-thresholding function. This corresponds to setting the function R as R(·) := ∥ · ∥1 in Eq.(2.2). The training loss, introduced in Eq.(2.4), consists of a data fitting term and several regularization terms. The data fitting term L is the mean squared error (MSE) between the network output and the ground truth: 1 N,N 2 L xN,N x − y j 2, j ; y j := 2 j where j = 1, 2, · · · , J. The regularization terms are defined as (1)
HN
1 N−1 N,k 2 AN,: = ∑ A 2, N k=0
(2)
HN
1 N−1 N,k 2 α N,: = ∑α , N k=0
(3)
HN
1 N−1 N,k 2 λ N,: = ∑λ , N k=0
N,: = {α N,k }N−1 , and λ N,: = {λ N,k }N−1 denote the collections of where AN,: = {AN,k }N−1 k=0 , α k=0 k=0 trainable parameters across the N layers. The corresponding regularization coefficients are set to β1 = β2 = β3 = 1 × 10−7 . We now describe the training configuration for the FBS-network models. All models are trained for 800 epochs using stochastic gradient descent (SGD) with a momentum of 0.9 and an initial learning rate r0 := 8.0 × 10−3 . Following the approach in [45], we assign different learning rates to different parameter groups. Specifically, the learning rates for AN,: and λ N,: are set to rA := r0 ∗ N and rλ := r0 ∗ N, respectively, while the learning rate for α N,: is set to rα := r0 ∗ N 3 . The batch size is set to 256. The trainable linear transformation matrices AN,: are all initialized as ((A⊤ )orth )⊤ , i.e., the transpose of the orthogonal decomposition of A⊤ . Similarly, the trainable scalar parameters α N,: and λ N,: are initialized as 10 and 0.05, respectively. To verify our main convergence result, we evaluate the performance of the FBS-network with increasing N ∈ {5, 10, 15, 20, 25}. Figure 1 presents the training loss curves versus epochs for these networks. We can observe two phenomena. First, the training loss consistently decreases as N increases, with larger N yielding a lower loss. Second, the marginal improvement gradually diminishes as the network becomes deeper, indicating that the network performance gradually tends to a limit. These empirical findings verify our theoretical analysis in Theorem 3.7.
6. Conclusion In the last decade, the FBS algorithm has been unrolled to build several effective neural networks in sparse coding and image restoration. In this paper, we studied the deep-layer limit and stability behaviors of the learning problems of the basic FBS-network with direct parameter relaxations as given in [1]. Under the dynamical inclusion forward modeling, we established some convergence properties of the network training problem to the related
31
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
Training loss curves N=5 N = 10 N = 15 N = 20 N = 25
0.006
Training loss
0.005 0.004 0.003 0.002 0.001 0
100
200
300
400
Epoch
500
600
700
800
F IG . 1. The training loss curves of the FBS-network with increasing network layer numbers N = 5, 10, 15, 20, 25.
deep-layer limit learning problem, and presented stability results of the above two learning problems. Our analysis procedure can be simplified to derive similar results for other unrolling networks from FBS-type algorithms, such as LISTA and ALISTA networks. These theoretical results indicate, in some sense, that such FBS-induced learning methods can work stably and consistently with different numbers of layers. Acknowledgments This work was supported by the National Natural Science Foundation of China (grants 12271273), the Key Program (21JCZDJC00220) of the Natural Science Foundation of Tianjin, China. R EFERENCES 1. 2. 3. 4.
X. Lin, C. Wu, Deep-layer limit and stability analysis of the basic forward-backward-splitting induced network (i): feed-forward systems, IMA Journal of Numerical Analysis (2025) draf068. W. E, A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics 1 (5) (2017) 1–11. M. Thorpe, Y. van Gennip, Deep limits of residual neural networks, Research in the Mathematical Sciences 10 (1) (2023) 6. B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, E. Holtham, Reversible architectures for arbitrarily deep residual neural networks, in: Proceedings of the 32th AAAI Conference on Artificial Intelligence, 2018, pp. 2811–2818.
32 5.
XUAN LIN, CHUNLIN WU
L. Zhang, H. Schaeffer, Forward stability of ResNet and its variants, Journal of Mathematical Imaging and Vision 62 (3) (2020) 328–351. 6. Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, T.-y. Liu, Understanding and improving Transformer from a multi-particle dynamic system point of view., in: ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020. 7. J. Liang, J. Huang, M. Sun, C. Wu, A deep layer limit analysis of transformer, Journal of Machine Learning, accepted (2026). 8. J. Huang, H. Su, X.-C. Tai, C. Wu, Mathematical modeling and convergence analysis of deep neural networks with dense layer connectivities in deep learning, arXiv preprint arXiv:2510.02049 (2025). 9. C. W. Gear, L. R. Petzold, ODE methods for the solution of differential/algebraic systems, SIAM Journal on Numerical Analysis 21 (4) (1984) 716–728. 10. J. C. Butcher, The numerical analysis of ordinary differential equations: Runge-Kutta and general linear methods, Wiley-Interscience, 1987. 11. S. Larsson, V. Thomée, Partial differential equations with numerical methods, Vol. 45, Springer, 2003. 12. J. C. De los Reyes, Numerical PDE-constrained optimization, Springer, 2015. 13. Y. Chen, T. Pock, Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration, IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (6) (2016) 1256–1272. 14. Z. Long, Y. Lu, X. Ma, B. Dong, PDE-Net: Learning PDEs from data, in: Proceedings of the 35th International Conference on Machine Learning, PMLR, 2018, pp. 3208–3216. 15. Z. Long, Y. Lu, B. Dong, PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network, Journal of Computational Physics 399 (2019) 108925. 16. E. Haber, L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems 34 (1) (2017) 014004. 17. E. Haber, K. Lensink, E. Treister, L. Ruthotto, Imexnet a forward stable deep neural network, in: International Conference on Machine Learning, PMLR, 2019, pp. 2525–2534. 18. L. Ruthotto, E. Haber, Deep neural networks motivated by partial differential equations, Journal of Mathematical Imaging and Vision 62 (3) (2020) 352–364. 19. Y. Lu, A. Zhong, Q. Li, B. Dong, Beyond finite layer neural networks: bridging deep architectures and numerical differential equations, in: Proceedings of the 35th International Conference on Machine Learning, PMLR, 2018, pp. 3276–3285. 20. R. T. Chen, Y. Rubanova, J. Bettencourt, D. K. Duvenaud, Neural ordinary differential equations, Advances in Neural Information Processing Systems 31 (2018). 21. J. He, J. Xu, MgNet: A unified framework of multigrid and convolutional neural network, Science China Mathematics 62 (2019) 1331–1354. 22. X.-C. Tai, H. Liu, R. Chan, PottsMGNet: A mathematical explanation of encoder-decoder based neural networks, SIAM Journal on Imaging Sciences 17 (1) (2024) 540–594. 23. W. E, C. Ma, L. Wu, Machine learning from a continuous viewpoint, I, Science China Mathematics 63 (11) (2020) 2233–2266. 24. C. Ma, J. Wang, E. Weinan, Model reduction with memory and the machine learning of dynamical systems, Communications in Computational Physics 25 (4) (2019) 947–962. 25. K. Gregor, Y. LeCun, Learning fast approximations of sparse coding, in: Proceedings of the 27th International Conference on Machine Learning, 2010, pp. 399–406.
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
33
26. J. Liu, X. Chen, Z. Wang, W. Yin, ALISTA: Analytic weights are as good as learned weights in LISTA, in: Proceedings of the 7th International Conference on Learning Representations, 2019. 27. Y. Yang, J. Sun, H. Li, Z. Xu, Deep ADMM-Net for compressive sensing MRI, Advances in Neural Information Processing Systems 29 (2016). 28. J. Adler, O. Öktem, Learned primal-dual reconstruction, IEEE Transactions on Medical Imaging 37 (6) (2018) 1322–1332. 29. E. Kobler, T. Klatzer, K. Hammernik, T. Pock, Variational networks: connecting variational methods and deep learning, in: Proceedings of the 39th German Conference on Pattern Recognition, Springer, 2017, pp. 281–293. 30. K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, F. Knoll, Learning a variational network for reconstruction of accelerated MRI data, Magnetic Resonance in Medicine 79 (6) (2018) 3055–3071. 31. A. Beck, First-order methods in optimization, SIAM, 2017. 32. R. Glowinski, S. J. Osher, W. Yin, Splitting methods in communication, imaging, science, and engineering, Springer, 2017. 33. R. Glowinski, P. Le Tallec, Augmented Lagrangian and operator-splitting methods in nonlinear mechanics, SIAM, 1989. 34. J. Huang, Y. Gao, C. Wu, On dynamical system modeling of learned primal-dual with a linear operator K : stability and convergence properties, Inverse Problems 40 (7) (2024) 075006. 35. J. Zhang, B. Ghanem, ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1828–1837. 36. J. Xiang, Y. Dong, Y. Yang, FISTA-Net: Learning a fast iterative shrinkage thresholding network for inverse problems in imaging, IEEE Transactions on Medical Imaging 40 (5) (2021) 1329–1339. 37. Q. Ding, G. Chen, X. Zhang, Q. Huang, H. Ji, H. Gao, Low-dose ct with deep learning regularization via proximal forward–backward splitting, Physics in Medicine & Biology 65 (12) (2020) 125009. 38. H. B. (auth.), Functional Analysis, Sobolev Spaces and Partial Differential Equations, 1st Edition, Universitext, Springer-Verlag New York, 2010. 39. H. W. Alt, Linear functional analysis: An application-oriented introduction, Springer, 2016. 40. C. M. Elliott, On the convergence of a one-step method for the numerical solution of an ordinary differential inclusion, IMA Journal of Numerical Analysis 5 (1) (1985) 3–21. 41. J. M. Borwein, J. D. Vanderwerff, Convex functions: constructions, characterizations and counterexamples, Vol. 172, Cambridge University Press, 2010. 42. R. T. Rockafellar, R. J.-B. Wets, Variational analysis, Vol. 317, Springer Science & Business Media, 2009. 43. N. Dunford, J. T. Schwartz, Linear operators. Part I: General theory, Vol. 10, John Wiley & Sons, 1988. 44. J. Frecon, S. Salzo, M. Pontil, Bilevel learning of the group lasso structure, Advances in Neural Information Processing Systems 31 (2018). 45. J. Ludziejewski, J. Mala’snicki, M. Pi’oro, M. Krutul, K. Ciebiera, M. Stefaniak, J. Krajewski, P. Sankowski, M. Cygan, K. Adamczewski, S. Jaszczur, Decoupled relative learning rate schedules, arXiv preprint arXiv:2507.03526 (2025). 46. L. Grafakos, Classical Fourier analysis, Vol. 2, Springer, 2008. 47. G. Dal Maso, An introduction to Γ-convergence, Springer Science & Business Media, 1993.
34
XUAN LIN, CHUNLIN WU
48. J. Matias, M. Morandotti, Homogenization problems in the calculus of variations: an overview, São Paulo Journal of Mathematical Sciences 9 (2) (2015) 162–180.
A. Appendix A.1. Backgrounds of some convergence T HEOREM A.1 (L p convergence implies a.e. pointwise convergence up to a subsequence) p p Let (X, A , µ) be a measure space, { fk }+∞ k=1 ⊆ L (X), and f ∈ L (X). Here, 1 ≤ p ≤ +∞. If ∥ fk − f ∥L p (X) → 0 as k → +∞, then there exists a subsequence { fkr }+∞ k=1 such that | f kr (x) − f (x)| → 0 as r → +∞ for a.e. x ∈ X. Proof. It is just a simple combination of [46, Prop.1.1.9] and [46, Thm.1.1.11].
□
T HEOREM A.2 (Kolmogorov-Riesz-Fréchet Theorem, a generalization of Arzela-Ascoli Theorem in L p spaces) [38, Thm.4.26] Let F be a bounded set in L p (RN ) with p ∈ [1, +∞). Assume that lim ∥τh f − f ∥L p (RN ) = 0 uniformly for f ∈ F ,
|h|→0
i.e., ∀ε > 0, ∃δ > 0 such that ∀h ∈ RN with |h| < δ , the inequality ∥τh f − f ∥L p (RN ) < ε holds for all f ∈ F . Then the closure of F|Ω in L p (RN ) is compact for any measurable set Ω ⊆ RN with finite measure. Here F|Ω denotes the restrictions to Ω of the functions in F . D EFINITION A.3 (Γ-convergence) [47, Prop.8.1] Let X be a topological space satisfying the first axiom of countability, {Fk }+∞ k=1 be a sequence of functionals such that for every k ∈ N+ , Fk : X → R := R ∪ {+∞}. Then {Fk }+∞ k=1 Γ-converges to the functional F : X → R if and only if the following conditions hold: (a) For every x ∈ X and every sequence {xk }+∞ k=1 ⊂ X converging to x as k → +∞, there is F(x) ≤ lim inf Fk (xk ). k→+∞
(b) For every x ∈ X, there exists a sequence {xk }+∞ k=1 converging to x as k → +∞ such that F(x) ≥ lim sup Fk (xk ). k→+∞
T HEOREM A.4 (The fundamental theorem of Γ-convergence) [47, Cor 7.20] [48, Thm 3.3] Let (X, d) be a metric space, {Fε }ε be a sequence of functionals defined on X, F := Γ- lim Fε ε→0+
be a functional defined on X. For ∀ε > 0, let (xε )∗ be a minimizer of Fε in X. (1) If x∗ is a cluster point of {(xε )∗ }ε , then x∗ is a minimizer of F, and F(x∗ ) = lim sup Fε ((xε )∗ ). ε→0+
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
35
(2) If {(xε )∗ }ε converges to a point x∗ in X, then x∗ is a minimizer of F, and F(x∗ ) = lim Fε ((xε )∗ ).
ε→0+
A.2. Proof of Theorem 4.1 Proof. For clarity, we denote {xN,k (AN,: , α N,: , λ N,: ; x0 , b)}Nk=0 as the state of the network (2.2) determined by the parameters (AN,: , α N,: , λ N,: ) with the initial value x0 and the observed data b. Step 1: We first prove that for any given N ∈ N+ , lim
r→+∞
sup (AN,: ,α N,: ,λ N,: )∈DN
f(r) (AN,: , α N,: , λ N,: ) − JN (AN,: , α N,: , λ N,: ) = 0. J N
(A.1)
Since DN is compact, Stability Theorem 4.1 in [1] gives that N,N N,: ∪+∞ (A , α N,: , λ N,: ; (x0 )(r) , b(r) ) : (AN,: , α N,: , λ N,: ) ∈ DN } r=1 {x
{xN,N (AN,: , α N,: , λ N,: ; x0 , b) : (AN,: , α N,: , λ N,: ) ∈ DN }
[
is a subset of a compact set in Rn . It follows from Stability Theorem 4.1 in [1] that, there exist two constants ce1 ≥ 0, ce2 ≥ 0 both independent of r, AN,: , α N,: , λ N,: , such that ∥xN,N (AN,: , α N,: , λ N,: ; (x0 )(r) , b(r) ) − xN,N (AN,: , α N,: , λ N,: ; x0 , b)∥2 ≤e c1 ∥(x0 )(r) − x0 ∥2 + ce2 ∥b(r) − b∥2 . Therefore, for any δ > 0, there exists r1,δ ∈ N+ such that for every integer r ≥ r1,δ , sup (AN,: ,α N,: ,λ N,: )∈D
1 ∥xN,N (AN,: , α N,: , λ N,: ; (x0 )(r) , b(r) ) − xN,N (AN,: , α N,: , λ N,: ; x0 , b)∥2 ≤ δ . 3 N
Besides, limr→+∞ y(r) = y implies that for any δ > 0, there exists r2,δ ∈ N+ such that for every integer r ≥ r2,δ , we have 1 ∥y(r) − y∥2 ≤ δ . 3 These results indicates that for any δ > 0, there exists r3,δ := max{r1,δ , r2,δ } such that for every integer r ≥ r3,δ , sup (AN,: ,α N,: ,λ N,: )∈DN
2 ≤ δ < δ. 3
∥xN,N (AN,: , α N,: , λ N,: ; (x0 )(r) , b(r) ) − xN,N (AN,: , α N,: , λ N,: ; x0 , b)∥2 + ∥y(r) − y∥2
36
XUAN LIN, CHUNLIN WU
By further noting the continuity assumption of L in (A4) and thus its uniform continuity on compact sets, we obtain that for any ε > 0, there exists an integer r3,δ (ε) such that for every integer r ≥ r3,δ (ε) , (r)
sup (AN,: ,α N,: ,λ N,: )∈DN
=
f (AN,: , α N,: , λ N,: ) − JN (AN,: , α N,: , λ N,: )| |J N L (xN,N (AN,: , α N,: , λ N,: ; (x0 )(r) , b(r) ); y(r) )
sup (AN,: ,α N,: ,λ N,: )∈D
N
(1)
(2)
(3)
+ β1 HN (AN,: ) + β2 HN (α N,: ) + β3 HN (λ N,: ) − L (xN,N (AN,: , α N,: , λ N,: ; x0 , b); y) (1)
(2)
(3)
− β1 HN (AN,: ) − β2 HN (α N,: ) − β3 HN (λ N,: ) =
L (xN,N (AN,: , α N,: , λ N,: ; (x0 )(r) , b(r) ); y(r) ) − L (xN,N (AN,: , α N,: , λ N,: ; x0 , b); y)
sup (AN,: ,α N,: ,λ N,: )∈DN
<ε, which indicates Eq.(A.1). Step 2: In this step, we will prove (1). The existence of a cluster point of {((AN,: )∗,r , (α N,: )∗,r , (λ N,: )∗,r )}+∞ r=1 is obvious due to the compactness of DN . We next show that its every cluster point belongs to SN . Suppose that {((AN,: )∗,ri , (α N,: )∗,ri , (λ N,: )∗,ri )}+∞ i=1 converges to ((AN,: )∗,∗ , (α N,: )∗,∗ , (λ N,: )∗,∗ ). By using again the continuity of L , we obtain JN ((AN,: )∗,∗ , (α N,: )∗,∗ , (λ N,: )∗,∗ ) (1)
=L (xN,N ((AN,: )∗,∗ , (α N,: )∗,∗ , (λ N,: )∗,∗ ; x0 , b); y) + β1 HN ((AN,: )∗,∗ ) (2)
(3)
+ β2 HN ((α N,: )∗,∗ ) + β3 HN ((λ N,: )∗,∗ ) h = lim L (xN,N ((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ; x0 , b); y) i→+∞
i (1) (2) (3) + β1 HN ((AN,: )∗,(ri ) ) + β2 HN ((α N,: )∗,(ri ) ) + β3 HN ((λ N,: )∗,(ri ) ) h = lim L (xN,N ((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ; (x0 )(ri ) , b(ri ) ); y(ri ) ) i→+∞
i (1) (2) (3) + β1 HN ((AN,: )∗,(ri ) ) + β2 HN ((α N,: )∗,(ri ) ) + β3 HN ((λ N,: )∗,(ri ) ) h (1) ≤ lim L (xN,N (AN,: , α N,: , λ N,: ; (x0 )(ri ) , b(ri ) ); y(ri ) ) + β1 HN (AN,: ) i→+∞
37
DEEP-LAYER LIMIT AND STABILITY OF FBS-NETWORK(II)
i (3) (2) + β2 HN (α N,: ) + β3 HN (λ N,: ) (1)
(2)
(3)
=L (xN,N (AN,: , α N,: , λ N,: ; x0 , b); y) + β1 HN (AN,: ) + β2 HN (α N,: ) + β3 HN (λ N,: ), ∀(AN,: , α N,: , λ N,: ) ∈ DN =JN (AN,: , α N,: , λ N,: ),
∀(AN,: , α N,: , λ N,: ) ∈ DN ,
(A.2)
where the third and the fifth equalities are due to the result in Step 1, and the fourth (r ) inequality is from ((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ) ∈ SeN i . Obviously, Eq.(A.2) indicates N,: ∗,∗ N,: ∗,∗ N,: ∗,∗ ((A ) , (α ) , (λ ) ) ∈ SN , and f(ri ) ((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ) JN ((AN,: )∗,∗ , (α N,: )∗,∗ , (λ N,: )∗,∗ ) = lim J N i→+∞
≤
inf
(AN,: ,α N,: ,λ N,: )∈DN
JN (AN,: , α N,: , λ N,: )
=JN ((AN,: )∗,∗ , (α N,: )∗,∗ , (λ N,: )∗,∗ ). That is, f(ri ) ((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ) = lim J N
i→+∞
inf
(AN,: ,α N,: ,λ N,: )∈DN
JN (AN,: , α N,: , λ N,: ). (A.3)
Step 3: Now we prove (2) and (3). For (2), we assume by contradiction that inf
((AN,: )∗ ,(α N,: )∗ ,(λ N,: )∗ )∈SN
((AN,: )∗,(r) , (α N,: )∗,(r) , (λ N,: )∗,(r) ) − ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ )
ℓ p ((Rm×n )N ×RN ×RN )
↛ 0.
Then there exist an ε0 > 0 and a subsequence n inf ((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ) ((AN,: )∗ ,(α N,: )∗ ,(λ N,: )∗ )∈SN
− ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ )
o+∞ ℓ p ((Rm×n )N ×RN ×RN ) i=1
such that for every i ∈ N+ , inf
((AN,: )∗ ,(α N,: )∗ ,(λ N,: )∗ )∈SN
((AN,: )∗,(ri ) , (α N,: )∗,(ri ) , (λ N,: )∗,(ri ) ) − ((AN,: )∗ , (α N,: )∗ , (λ N,: )∗ )
ℓ p ((Rm×n )N ×RN ×RN )
≥ ε0 .
(A.4)
38 By n
XUAN LIN, CHUNLIN WU
Step
2,
we
can
∗,(r ) ∗,(r ) ∗,(r ) (AN,: ) i j , (α N,: ) i j , (λ N,: ) i j
further o +∞ j=1
extract a convergent subsequence n o+∞ N,: ∗,(r ) N,: ∗,(r ) N,: i i from (A ) , (α ) , (λ )∗,(ri ) i=1
with the limit point belonging to SN to get a contradiction with Eq.(A.4). A similar argument and noting Eq.(A.3) give (3). □