Recurrent neural networks approximate continuous functions Valentin Abadie, Clemens Hutter and Helmut Bölcskei 2026-06-19
arXiv:2606.20325v1 [cs.LG] 18 Jun 2026
Abstract Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [−1, 1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.
1
Introduction
The work [14] introduced an approximation paradigm for recurrent neural networks (RNNs), which asks whether RNNs can approximate a function f : [−1, 1] → R in a sense that we will detail below. The starting point for this approximation paradigm is the classical universal approximation theorem for feedforward neural networks [6, 9, 13], which asserts that every continuous function on a compact domain can be approximated arbitrarily well by a shallow neural network with sigmoidal activation function. Subsequent quantitative results relating the smoothness of the target function and the prescribed approximation error to the size of the approximating network were later obtained in [1, 2]. In the past two decades the rectified linear unit (ReLU) has become the dominant activation function in theory and practice. Beginning with [26], quantitative approximation results for deep ReLU networks have been developed [23, 17, 19], culminating in [7], which shows that deep ReLU networks approximate a wide range of function classes in metric-entropy–optimal manner. The quantitative approximation results in the literature typically take the following form: for a given function f and a given approximation error ϵ > 0, there exists a neural network Ψ that approximates f to within error ϵ, formalized as ∀f : ∀ϵ : ∃ Ψ such that Ψ approximates f to within error ϵ.
(1)
Thus the network architecture and weights depend on the chosen value of ϵ. If a smaller error is later required, a new (typically larger) network must be instantiated. For instance, [7, Proposition III.5] shows that for every polynomial f and every ϵ > 0, there exists a deep ReLU network of ϵ-independent width and depth O(log (ϵ−1 )) that approximates f to within error ϵ. In [14], the authors introduced an approximation framework in which the approximating network is no longer chosen as a function of the target tolerance ϵ. The guiding idea is to exchange the usual quantifier order ∀ϵ, ∃ Ψ for a statement of the form ∀f : ∃ Ψ, such that ∀ϵ : Ψ approximates f to within error ϵ.
(2)
In practice, the accuracy is improved by iterating one and the same neural network, whose architecture and weights remain fixed throughout the process. The paper [14] realizes this idea with Recurrent Neural Networks (RNNs) [11]; in the case where f : [−1, 1] → R is a polynomial, the iterates yield errors that tend to zero, and moreover do so at an exponential rate in the number of compositions. In this paper, we generalize this result by showing that (2) holds for every continuous function f : [−1, 1] → R. Note that this result is not a mere consequence of the density of polynomials in the space of continuous functions, as in [14], the RNN that approximates a given polynomial function f has a hidden state size that grows linearly with the degree of f . A density-type result would thus yield an RNN with infinite hidden state size, which goes out of our formalism. In fact, our proof relies on a completely different approach. 1
Neural state:
A finite-dimensional real-valued vector
Turing machine
x1
Discrete control head
q1
q3
qn−1
q2
q4
qn
Updated by application of some piecewise-linear map fi : R d → R d , i ∈ {1, . . . , m}
x2 x3 x4
The Turing machine chooses the map
Interaction through transition table
0
1
1
1
0
1
0
0
1
Infinite tape storing bits
xd−1 xd
Figure 1: Schematic representation of a TMNU. The idea is to introduce a new model that has the same approximation properties as RNNs, but that one can manipulate more easily than RNNs. The intuition is to use the well-known fact that RNNs are Turing-complete, i.e., they can simulate Turing machines [21, Chapter 3.1] in some sense to be made precise later. Turing machines are easier to manipulate than RNNs, because they can be specified simply by writing an algorithm [21, Chapter 3.3], whereas RNNs are specified by writing matrices and vectors, which is more cumbersome. The key principle of the proof is to create an RNN whose hidden state is separated in two parts: one part that simulates the computation of a Turing machine, and one part that manipulates the input x ∈ R according to the instructions of the Turing machine. To make explicit this separation of the hidden state into two parts, we in fact introduce a new model of computation, which we call the TMNU (for "Turing Machine with Neural Units"), that is essentially made of a Turing machine and of a real computation unit that applies piecewise-linear transformations to an input x ∈ R according to the instructions of the Turing machine. TMNUs have the same flexibility of design as Turing machines, but they also incorporate the ability to manipulate real numbers through piecewise-linear transformations, which belongs to the realm of RNNs. However, we will show that TMNUs are essentially equivalent to RNNs, and that showing (2) can be reduced to the problem of approximating continuous functions with TMNUs, which is much simpler to solve, because TMNUs can be designed by writing algorithms, and notably we can design simple TMNUs to be used later as subroutines in the design of more complex TMNUs. This flexibility of design is crucial to the proof, and is not available when working directly with RNNs. The idea of the TMNU detour comes from two observations: (i) RNNs are known to be Turing-complete [20], and (ii) the theory of computable analysis [25] studies the approximation of functions by Turing machines, in a paradigm that is very similar to (2). The functions that can be approximated by Turing machines in the paradigm of computable analysis are called computable functions, and include most functions of interest in analysis, such as polynomials, trigonometric functions, exponential functions, etc. In fact, it is known that every computable function f : [−1, 1] → R is continuous [5, Proposition 6.1] and “Often, a function that is not computable [...] is so for purely topological reasons; that is, it is not continuous” [5, p452]. Therefore, computable functions make a good proxy for continuous functions. A first naive approach would be to use observations (i) and (ii) to conclude that RNNs can approximate any computable function according to (2). However, this is not the case because the simulation of Turing machines by RNNs use encodings of the input to the Turing machine in a way that is not compatible with the approximation paradigm (2). We explain this in greater detail in Section 4. The TMNU model is designed to circumvent this issue. We conclude this section by mentioning that the TMNU model has to be viewed as a proof tool that is specifically designed to prove the main result of the paper, but that can certainly be used in other proofs regarding the approximation properties of RNNs. We next formalize the RNN approximation paradigm, state the main results, and then develop the TMNU model used in the proofs.
2
The RNN Approximation Paradigm
We proceed to give a formal definition of RNNs, and to operationalize the approximation paradigm (2) in terms of RNNs.
2
Definition 2.1 (RNN [8, 11, 14]). We denote by ρ : R → R, ρ(x) := max(0, x) the ReLU function which acts component-wise, i.e., ρ(x1 , . . . , xm ) := (ρ(x1 ), . . . , ρ(xm )). An RNN is an ordered set R := (d, m, d′ ; Ah , bh , Ax , Ao , bo ) where d ∈ N is the input dimension, d′ ∈ N is the output dimension, m ∈ N is ′ ′ the hidden state size, Ah ∈ Rm×m , Ax ∈ Rm×d , Ao ∈ Rd ×m , and bh ∈ Rm , bo ∈ Rd . To R we associate N0 N the hidden state operator H : Rd → (Rm ) 0 mapping an input sequence (x[t])t∈N0 recursively to the sequence of hidden states (h[t])t∈N0 according to h[−1] := 0 ∈ Rm
(3)
(Hx)[t] = h[t] = ρ(Ah h[t − 1] + Ax x[t] + bh ). We finally view R as a map R : Rd
N 0
(4)
′ N 0 → Rd defined by
(Rx)[t] = Ao (Hx)[t] + bo ,
t ∈ N0 ,
(x[t])t∈N0 ∈ Rd
N0
.
We also let m(R) := m denote the hidden state size of R and M(R) denote the magnitude of the weights of R, i.e., M(R) := max{∥Ah ∥∞ , ∥bh ∥∞ , ∥Ax ∥∞ , ∥Ao ∥∞ , ∥bo ∥∞ }. Following [14], to conform with (2), the aim is, for a given function f , to find an RNN that achieves any arbitrarily small approximation error ϵ provided we let it run sufficiently long. To this end, if we desire an approximation of the function f at the point x ∈ R, we take the input sequence of the RNN as x̃[t] = x1{t=0} , t ∈ N0 . Here, 1{·} denotes the truth function which takes on the value 1 if the statement inside {·} is true and equals 0 otherwise. To formalize this, the following operator was introduced in [14]. N 0 Definition 2.2. The mapping D : Rd → Rd is defined according to (Dx)[t] = x1{t=0} ,
t ∈ N0 .
The corresponding output sequence of the RNN then produces increasingly accurate approximations of f (x) as time t evolves. The approximation paradigm (2) has been operationalized in terms of RNNs in [14] as ∀f : ∃ R : ∀ϵ : ∃t0 : sup sup |(RDx)[t] − f (x)| < ϵ. t≥t0
x
(5)
Every approximation theorem fitting this paradigm must have the size, topology, and weights of the approximating RNN R be independent of the approximation error ϵ, simply by virtue of the quantifier order in (5). Only the runtime required to achieve the desired approximation error ϵ will depend on ϵ. This approximation paradigm exhibits interesting practical properties as storing the fixed RNN on digital devices requires little memory and the approximation error can be controlled simply by adjusting the runtime of the RNN. Before proceeding to the main results, we want to make clear the necessity of allowing the RNN to compute for an arbitrary long time to reach arbitrary precision ε. Indeed, let f : x 7→ x2 , R be an RNN, ε > 0 and suppose that there exists t0 ∈ N0 such that sup
sup |(RDx)[t] − f (x)| < ε.
(6)
t≥t0 x∈[−1,1]
Then, in particular, we have supx∈[−1,1] |(RDx)[t0 ] − f (x)| < ε. Note that (RDx)[t0 ] is a piecewise-linear function of x, and, in fact, can be identified to a deep ReLU neural network of depth t0 +1 and width m(R). As shown in Appendix E.1, it has at most (m(R) + 1)t0 +1 − 1 breakpoints, while any piecewise-linear approximation of f within precision ε requires at least (2ε)−1/2 − 1 breakpoints, hence we obtain the lower bound 2(t 1+1) 0 1 m(R) ≥ − 1 −→ ∞. (7) ε→0 2ε Therefore, in order to comply with the approximation paradigm (5), we need to allow the RNN to compute for an arbitrary long time, which is the only way to possibly reach arbitrary precision ε while keeping the size and weights of the RNN fixed. The core of this paper is to show that, indeed, the approximation paradigm (5) can be achieved for a large class of functions f .
3
Main Results
We now state the main result of the paper, which shows that every continuous function f : [−1, 1] → R can be approximated by an RNN in the sense of (5), when the RNN is allowed to compute for an arbitrary long time. 3
Theorem 3.1. Let f : [−1, 1] → R be a continuous function. Then, there exists an RNN R such that lim
sup |RDx[t] − f (x)| = 0.
(8)
t→∞ x∈[−1,1]
Moreover, the hidden state dimension and the magnitude of the weights of R satisfy m(R) ≤ 5320, and M(R) ≤ 5 1 + ∥f ∥L∞ ([−1,1]) . Sketch of the proof. This theorem is a direct consequence of combining the link between RNNs and TMNUs given by Theorem 5.4 with the construction of a TMNU that approximates f given by Theorem 6.1. The explicit bounds on the hidden state dimension and the weight magnitude of R are obtained by plugging the quantities appearing in Lemma C.26 into the bounds given by Theorem 5.4. Surprisingly perhaps, and in sharp contrast to the main result of [14], the hidden state dimension of the RNN R does not depend on the function f . Still, we are able to recover exponential decay of the approximation error as a function of the runtime t for the class of polynomial functions. This is stated in the following result. Theorem 3.2. Let N ∈ N, and a0 , . . . , aN ∈ R. Then, there exists an RNN R satisfying m(R) ≤ 5320, such that sup
RDx[t] −
x∈[−1,1]
M(R) ≤ 5 (1 + ∥a∥1 )
and N X
ai xi ≤ 2 · 2−t/ta ,
(9)
t ≥ τa ,
i=0
where ta := 72(N + 1) and τa := ta (6∥a∥1 + N + 17). Sketch of the proof. The proof of this theorem follows exactly that of Theorem 3.1, and the exponential rate is specifically obtained by application of Theorem 6.2 that construct a TMNU that approximates the target polynomial with an exponential rate. Note that the exponential rate is optimal, in the sense that there are some polynomials for which this exponential rate cannot be beaten. Indeed, (7) can be reformulated as a lower bound on the approximation error of x 7→ x2 as a function of the runtime t as 2 1 1 ε := sup |RDx[t] − f (x)| ≥ (10) = 2−2t log2 (m(R)+1) , t ∈ N. t 2(m(R) + 1) 4 x∈[−1,1] Therefore, among the class of polynomial functions, the function x 7→ x2 has an approximation rate that decays exponentially with t, and cannot be approximated with a faster rate. In a minimax sense, this shows that the class of polynomial functions has an approximation rate that decays exponentially with t. More generally, we link the decay of the approximation error as a function of the runtime t to the rate of convergence of the Chebyshev series of f . Given a continuous function f : [−1, 1] → R, its Chebyshev coefficients are defined as ˆ ˆ 1 1 f (x)T0 (x) 2 1 f (x)Tn (x) (f ) ) √ √ c0 := dx, c(f := dx, n ∈ N, (11) n π −1 π −1 1 − x2 1 − x2 and the Chebyshev partial sums are defined as Sn(f ) (x) :=
n X (f ) ck Tk (x),
n ∈ N,
x ∈ [−1, 1],
(12)
k=0
where Tn is the n-th Chebyshev polynomial given by T0 (x) = 1,
T1 (x) = x,
Tn+1 (x) = 2xTn (x) − Tn−1 (x),
n ∈ N,
x ∈ [−1, 1].
(13)
In accordance with our approximation paradigm (5), we are interested in functions for which the Chebyshev partial sums converge uniformly to f on [−1, 1], i.e., whether the quantity S(f, n) := sup
(f ) sup |f (x) − Sm (x)|,
n ∈ N,
(14)
m≥n x∈[−1,1]
converges to 0, in which case we say that f has a Chebyshev series. In fact, Chebyshev series essentially reduce to Fourier series under the change of variable x = cos(θ), because Tn (cos θ) = cos(nθ), and therefore (f ) the Chebyshev partial sums Sn converge uniformly to f on [−1, 1] if and only if the Fourier partial 4
sums of the function θ 7→ f (cos(θ)) converge uniformly to θ 7→ f (cos(θ)) on [0, π]. It is known that the Fourier partial sums of a continuous function converge uniformly to the function if the function satisfy the property of being Dini-Lipschitz continuous [10, Theorem 3.10], which definition is given as follows. Definition 3.1 (Dini-Lipschitz continuity). A continuous function f : [−1, 1] → R is said to be DiniLipschitz continuous if lim ωf (ε) log(ε) = 0, ε→0
where ωf is the modulus of continuity of f defined by ωf (ε) :=
sup
|f (x) − f (y)|,
ε ∈ [0, 2].
x,y∈[−1,1],|x−y|≤ε
Therefore, we restrict our attention to the vast class of Dini-Lipschitz continuous functions, which includes for example all Lipschitz and Hölder-continuous functions. One may be interested in the approximation of functions that are not Dini-Lipschitz continuous, but this would require going beyond Chebyshev series, which we reserve for future work. The rate of approximation S(f, n) can be linked to smoothness properties of f . We consider the following normalized function classes: • For α ∈ (0, 1], the class Hα of α-Hölder functions f satisfying ∥f ∥L∞ ([−1,1]) ≤ 1 and |f (x) − f (y)| ≤ 1. |x − y|α x̸=y
sup
Note that for α = 1, the class H1 coincides with the class Lip of 1-Lipschitz functions. • For k ∈ N, the class C k Hα of k-times continuously differentiable functions f with derivatives up to order k satisfying ∥f (j) ∥L∞ ([−1,1]) ≤ 1, and f (k) ∈ Hα . • The classes S and S1 of smooth functions with derivatives of order n satisfying ∥f (n) ∥L∞ ([−1,1]) ≤ n! and ∥f (n) ∥L∞ ([−1,1]) ≤ 1, respectively. We summarize the corresponding Chebyshev approximation rates in Table 1. Such approximation rates are classical and can be found in standard references on numerical approximation [24] completed with references on approximation with Fourier series [22]. Name of the class Hα C k Hα S S1
Rate of approximation S(f, n) O(n−α log(n)) O(n−(k+α) log(n)) exp(−Ω(n)) exp(−Ω(n log n))
Table 1: Link between the smoothness of f and the rate of approximation of f by its Chebyshev series. Our third main result shows that there is a direct link between the decay of the approximation error as a function of the runtime t and the rate of approximation of f by its Chebyshev series. Specifically, the approximation error decays as the sum of two terms: the first term that accounts for RNN-specific computation errors, and a second term that accounts for the error of approximating f by its Chebyshev partial sums. The tradeoff between these two terms is controlled by the choice of a nondecreasing function η : [1, ∞) → N, which, depending on the rate of approximation of f by its Chebyshev series, can be chosen to get the fastest possible decay of the approximation error as a function of t. The choices of η for the different function classes are summarized in Table 3, together with the resulting approximation error decay as a function of t. Theorem 3.3. Let f : [−1, 1] → R be a Dini-Lipschitz continuous function. Let η : [1, ∞) → N be nondecreasing, with η(t) → ∞ and t/η(t) nondecreasing. Assume that there exists A > 0 such that η(t) log2 (η(t) + 1) ≤ At,
t ≥ 1.
Then, there exist a constant τ := τf,A > 0 and an RNN R satisfying m(R) ≤ 6700,
and
M(R) ≤ 5 3 + S(f, 0) + ∥f ∥L∞ ([−1,1])
such that for every t ≥ τ , t/τ
sup |RDx[t] − f (x)| ≤ 2 · 2− η(t/τ ) + S (f, η(t/τ )) . x∈[−1,1]
5
(15)
Sketch of the proof. Once again, this theorem follows from combining the link between RNNs and TMNUs given by Theorem 5.4 with the construction of a TMNU that computes Chebyshev partial sums of f specified in Theorem 6.3. The explicit bounds on the hidden state dimension and the weight magnitude of R are also obtained with Lemma C.26. Theorem 3.3 gives an upper bound on the convergence speed of the RNN computations to the target function f . The natural follow-up question is whether this convergence speed is optimal. To answer this question, we show that, for fixed hidden dimension and fixed weight magnitude, the approximation error of RNNs cannot decay faster than a certain rate that depends on some information-theoretic properties of the class of target functions. In particular, this shows that the approximation rates given in Table 3 are almost optimal, up to logarithmic factors. The intuition behind this result is that, for fixed hidden dimension and fixed weight magnitude, the set of functions that can be approximated by RNNs after a fixed number of time steps t must have limited metric entropy [15], a concept that we now recall the definition of. In the following, we denote by C([−1, 1]) the set of continuous functions from [−1, 1] to R, equipped with the uniform norm ∥ · ∥L∞ ([−1,1]) . Definition 3.2 (Covering number). Let X ⊂ C([−1, 1]) be compact and ε > 0. An ε-covering of X is a finite collection of functions g1 , . . . , gN ∈ C([−1, 1]) such that for every f ∈ X , there exists j ∈ {1, . . . , N } with ∥f − gj ∥L∞ ([−1,1]) ≤ ε. We denote by N (ε, X ) the size of the smallest ε-covering of X . The metric entropy of X is defined as ε 7→ log2 N (ε, X ). The function classes that we have introduced above have well-known metric entropy estimates [15], that we summarize in Table 2. Name of the class X Hα C k Hα S S1
Metric entropy log2 N (ε, X ) ≍ (ε−1/α ) ≍ (ε−1/(k+α) ) ≍(log(1/ε)2 ) ≍
log(1/ε)2 log log(1/ε)
Table 2: Link between the smoothness of f and the metric entropy of the corresponding normalized function class. We now turn these metric entropy estimates into lower bounds on the approximation error of RNNs with fixed hidden dimension and fixed weight magnitude. We denote by Rm,B the set of RNNs with hidden dimension at most m and weight magnitude at most B. Then, for compact X ⊂ C([−1, 1]) and t ∈ N0 , we investigate the worst-case approximation error at time t of functions in X by RNNs in Rm,B , defined as Et (X ; m, B) := sup inf sup |(RDx)[t] − f (x)| . f ∈X R∈Rm,B x∈[−1,1]
We find the following lower bound on Et (X ; m, B) in function of the metric entropy of X . Theorem 3.4. Let X ⊂ C([−1, 1]) be compact, let m ∈ N, and let B ≥ 1. Let h : (0, 1] → (0, ∞) be nonincreasing and assume that log(ε−1 ) = 0, ε→0 h(ε) lim
and
log N (ε, X ) ≥ h(ε),
ε ∈ (0, 1].
Then there exist constants K > 0 and t0 ∈ N, depending only on m, B and h, such that, for every t ≥ t0 , Et (X ; m, B) ≥
1 −1 h (K(t + 1)), 2
where h−1 (T ) := inf {ε ∈ (0, 1] : h(ε) ≤ T }, for T > 0. Sketch of the proof. The intuition behind the result is that the metric entropy of the set if functions that can be realized by an RNN in at most t time steps grows as O(t). Moreover, to every function of a set X with precision ε, that set of approximation functions must have a metric entropy of the same order as the metric entropy of X at precision ε. Therefore, we must have h(ε) = O(t), which gives the desired lower bound on ε as a function of t. The full proof is given in Section E.2. We can now apply this general lower bound to the function classes that we have introduced above, by plugging in the corresponding metric entropy estimates. For example, for the class Hα , we have h(ε) ≍ ε−1/α , and therefore h−1 (t) ≍ t−α . By applying Theorem 3.4, we get that Et (Hα ; m, B) = Ω(t−α ). We summarize the resulting lower bounds in Table 3.
6
Name of the class
Choice of η(t)
Hα
≍ t/ log t
C k Hα S S1
RNN approximation rate (Upper bound)
≍ t/ log t √ ≍ t p ≍ t/ log t
O
O
(log t)1+α tα
1+k+α
(log t) tk+α
√ exp(−Ω( t)) √ exp(−Ω( t log t))
RNN approximation rate (Lower bound) Ω Ω
1 tα
1 tk+α
√ exp(−O( t)) √ exp(−O( t log t))
Table 3: Link between the smoothness of f , the choice of η, and the approximation error decay as a function of the runtime t. The lower bounds are minimax lower bounds for fixed hidden dimension and fixed weight magnitude. Note that the upper bounds are almost tight with the lower bounds, up to logarithmic factors. For the sets S and S1 , the upper and lower bounds are tight at the exponential scale. The rest of the paper is devoted to proving these statements. The main technical step is to introduce TMNUs as an intermediate computational model: they are rigid enough to be simulated by RNNs with controlled dimension and weight magnitude, but flexible enough to implement the polynomial and Chebyshev computations appearing above. We first define this model in Section 4, show its simulation by RNNs in Section 5, and then build the machines that realize the approximation schemes and the corresponding convergence estimates in Section 6.
4
Turing machines, TMNUs, and the approximation paradigm for TMNUs
In this section, we introduce formally the TMNU model. Before doing so, we recall the definition of a Turing machine [21, Chapter 3.1] and related concepts, which will be used to define the TMNU model. We will also explain in greater details why the classical correspondence between RNNs and TMNUs is not enough to solve the approximation paradigm (5), and why we need to introduce the TMNU model as an intermediate computational model to solve the approximation paradigm (5).
4.1
Turing machines
A Turing machine is a mathematical model of computation that informally consists of three components: 1. a finite number n ∈ N of internal states, 2. a tape of cells, which cells can either be empty or having the symbol 0 or the symbol 1 written on them. For the sake of mathematical convenience, we assume that the tape is infinite in both directions, i.e. that the tape is a bi-infinite sequence of cells, which we can model as a function τ : Z → Γ := {−1, 0, 1}, where τ (i) is the symbol written on the i-th cell of the tape, and where τ (i) = −1 means that the i-th cell is empty, τ (i) = 1 means that the i-th cell has the symbol 1 written on it, and τ (i) = 0 means that the i-th cell has the symbol 0 written on it. For sake of clarity, we let □ := −1 denote the symbol that represents an empty cell. Depending on the context, we might write either τ (i) or τi to denote the symbol written on the i-th cell of the tape. 3. A control head that decides of an internal state to transition to, can read and write symbols on the cell numbered 0 of the tape, and that can decide to shift the tape to the left, to the right or leave it as it is. Specifically, if the head decides to write a symbol σ ∈ Γ, then the tape is updated by changing the symbol written on the cell numbered 0 to σ, i.e., τ (0) ← σ. If the head decides to shift the tape to the right, then the tape is updated by shifting all the symbols on the tape one cell to the right, i.e., τ (i) ← τ (i − 1) for every i ∈ Z. If the head decides to shift the tape to the left, then the tape is updated by shifting all the symbols on the tape one cell to the left, i.e., τ (i) ← τ (i + 1) for every i ∈ Z. In general, the Turing machine model can be made more general, i.e., the cells of the tape may contain more than three symbols, there might be multiple tapes, etc. However, we here consider only the simple version of the Turing machine model described above, which is sufficient for our purposes and mathematically lighter. This informal description of a Turing machine can be made formal as follows. Definition 4.1. (Turing machine) A Turing machine is a pair M := (n, δ), such that n ∈ N is the number of states, and δ : {1, . . . , n} × Γ → {1, . . . , n} × Γ × {−1, 0, 1} is the transition function, that given an internal state q ∈ {1, . . . , n} and a symbol σ ∈ Γ read by the head, specifies a triple (q ′ , σ ′ , m) := δ(q, σ), where q ′ is the internal state to transition to, σ ′ is the symbol to write on the tape, and m is the shift of the tape to be effected, with −1 denoting a left shift, 0 denoting no shift and 1 denoting a right shift. We conveniently define δ q , δ σ and δ m to be such that δ = (δ q , δ σ , δ m ), i.e., δ q (q, σ) = q ′ , δ σ (q, σ) = σ ′ and δ m (q, σ) = m for every q ∈ {1, . . . , n} and σ ∈ Γ. 7
A precise description of the evolution of a Turing machine can be given as follows. We first describe the actions that the control head of a Turing machine can perform on the tape. We define the reading operation R : ΓZ → Γ, the writing operations Wσ : ΓZ → ΓZ for σ ∈ Γ, and the shifting operation Sm : ΓZ → ΓZ for m ∈ {−1, 0, 1} by Rτ := τ (0), (16) ( σ, i = 0, Wσ τ (i) := (17) τ (i), i ∈ Z \ {0}, and
Sm τ (i) := τ (i − m),
i ∈ Z,
(18)
for every τ ∈ Γ . The evolution of a Turing machine is then defined as follows. Assuming that M has some internal state q ∈ {1, . . . , n} and some tape τ ∈ ΓZ , we update the internal state of M to q ′ and the tape of M to Sm Wσ τ , where (q ′ , σ, m) := δ(q, Rτ ) is the triple specified by the transition function δ given the current internal state q and the symbol R = τ (0) read by the head. This can be formulated more compactly by using the formalism of configurations [21, Chapter 3.1], that here again, we define in a slightly different way as in the classical literature, in order to make it more convenient for our purposes. The definition of configuration and the update of configuration are given as follows. Z
Definition 4.2. (Configuration of a Turing machine) Given a Turing machine M := (n, δ), we call configuration of M any pair c := (q; τ ) such that q ∈ {1, . . . , n} and τ ∈ ΓZ . We let CM := {1, . . . , n} × ΓZ be the set of configurations of M . We give a meaning of M as a map from CM to CM by letting M (q; τ ) := (δ q (q, Rτ ), Sδm (q,Rτ ) Wδσ (q,Rτ ) τ ),
(19)
for every configuration (q; τ ) ∈ CM . The formalism above can be used to define a computation of a Turing machine M := (n, δ) starting from some initial configuration c ∈ CM as the sequence of configurations (M t (c))t∈N0 obtained by iterating the map M : CM → CM starting from c. Finally, we say that M halts on configuration c if there exists t ∈ N0 such that (qt ; τt ) := M t (c) satisfies qt = n. We call n the halting state of M . Often, the initial configuration of a Turing machine is of a specific form. Specifically, we will often assume that the initial tape τ is empty on its left part, i.e., τ (i) = □ for every i < 0. In order to condense notations, we introduce the following convention. We denote by {0, 1}∗ the set of finite binary sequences, by {0, 1}N the set of infinite binary sequences, and define {0, 1}# := {0, 1}∗ ∪ {0, 1}N as the set of all binary sequences. For every u ∈ {0, 1}∗ , ℓ (u) denotes the length of u, and for every u ∈ {0, 1}N , we let ℓ (u) := ∞. We will write τ = |u to mean that τ (i) = ui+1 for every i ∈ {0, . . . , ℓ (u) − 1}, and that τ (i) = □ for every i < 0 or i > ℓ (u) − 1, for some u ∈ {0, 1}∗ ∪ {0, 1}N . For instance, if u = 101, then τ = |u means that τ (0) = 1, τ (1) = 0, τ (2) = 1, and τ (i) = □ for every i < 0 and every i > 2. We now explain why the classical correspondence between RNNs and Turing machines is not enough to solve the approximation paradigm (5).
4.2
The necessity of the TMNU detour
In Section 1, we have mentioned that it is not enough to use the simulation of Turing machines by RNNs to solve the approximation paradigm (5). Here, we show more formally why this is the case. We say Turing machine M computes a function f : {0, 1}∗ → {0, 1}∗ , if upon being started in configuration c0 := (1; |u), the computation reaches the halting configuration c := (n; |f (u)) after some finite number of computation steps, for every u ∈ {0, 1}∗ . It was notably proven in [20] that if a Turing machine M computes a function f : {0, 1}∗ → {0, 1}∗ in the aforementioned sense, then there is an RNN that computes f in the following way. One can give a reformulation of f in the form of a function fˆ : C → C, where C ⊆ [−1, 1] is a Cantor set, i.e., a set of real numbers that is totally disconnected. Specifically, by defining the map ηCantor : {0, 1}∗ → [−1, 1] by ηCantor (u) =
ℓ(u) X
(2ui + 1)4−i
for every u ∈ {0, 1}∗ ,
i=1
the set C is defined as C := ηCantor ({0, 1}∗ ), and the function fˆ : C → C is defined to be such that as fˆ(ηCantor (u)) := ηCantor (f (u)) for every u ∈ {0, 1}∗ . Then, the result by [20] states that for every Turing machine M that computes a function f : {0, 1}∗ → {0, 1}∗ , there exists an RNN R such that for every u ∈ {0, 1}∗ , there exists t0 ∈ N0 such that for all t ≥ t0 , RDηCantor (u)[t] = fˆ(ηCantor (u)) for every t ∈ N0 , thus establishing the Turing completeness of RNNs. 8
One could be tempted to use this bridge between Turing machines and RNNs to solve the approximation paradigm (5). Indeed, one could try to exploit the theory of computable analysis [25, 5], that studies the properties of functions f : [−1, 1] → R by means of an underlying Turing machine that computes it. Specifically, this theory stipulates a natural representation of real numbers as binary sequences, which is called the Cauchy representation [5, Definition 4.4], that we denote here ηCauchy : {0, 1}N → R. The choice of such a representation is far from arbitrary, and is motivated by a well-established theory to compare representations of real numbers in the context of computable analysis. Then, given a function f : [−1, 1] → R, we say that f is computed by a Turing machine M if for every x ∈ [−1, 1], if we initialize the tape of M with an infinite binary sequence u ∈ {0, 1}N such that ηCauchy (u) = x, then, in the long run, the machine generates a sequence v ∈ {0, 1}N that satisfies ηCauchy (v) = f (x) [5, Definition 4.9]. A function f : [−1, 1] → R that can be computed in such a way by some Turing machine is said to be computable. As mentioned in Section 1, computable functions form a very large class that includes basically every function of interest, and solving the approximation paradigm (5) for computable functions would be a significant leap towards solving the approximation paradigm for continuous functions. The temptation here is to think that this is possible to design an RNN that, given an input x ∈ [−1, 1], can somehow extract some prefix p of an infinite binary sequence u ∈ {0, 1}N satisfying ηCauchy (u) = x, encode it as ηCantor (p) as in the approach in [20] and manipulate this sequence in the same way as some Turing machine M does. The RNN in question would then extract increasingly long prefixes of u and manipulate them in the same way as M does to generate increasingly long prefixes of some sequence v ∈ {0, 1}N such that ηCauchy (v) = f (x), and therefore to approximate f . However, the major obstruction to this approach is precisely the fact that given a real number x ∈ [−1, 1], the RNN in question should be able to extract some prefix p of an infinite binary sequence u ∈ {0, 1}N such that ηCauchy (u) = x, and encode it as ηCantor (p). Therefore, the RNN should be able to realize, in a finite number of computation steps, a map g : [−1, 1] → C. However, such a function g cannot be realized by an RNN with ReLU activation function, because g is not continuous, and every function that can be realized by an RNN in a finite number of time steps is continuous. To see why g is not continuous, note that [−1, 1] connected set, while C is a totally disconnected set, and therefore there is no continuous function from [−1, 1] to C by the intermediate value theorem [16, Theorem 4.7]. Therefore, going in this direction would require to circumvent this obstruction, which does not appear to be evident. In other words, it is not clear how to use the theory of computable analysis to show that every function f : [−1, 1] → R that is computed by a Turing machine can be approximated by an RNN. This is precisely here that the need to investigate other models of computations arise, especially models that can manipulate directly real-valued variables. Such models are already present in the literature. To the best of our knowledge, there exist three main models of computation that can manipulate real-valued variables. Firstly, there is the Blum-Shub-Smale (BSS) machine [3], which is a natural extension of Turing machines to manipulate real-valued variables. However, the BSS machine model has a fundamental mismatch with RNNs, because at its core is the use of discontinuous functions: during its computation, a BSS machine has the ability to apply the function f : R → R defined by f (x) = 0 if x < 0, and f (x) = 1 if x ≥ 0, which is not continuous. Therefore, the BSS machine model cannot be used to study RNNs with ReLU activation function, which are fundamentally continuous. Secondly, there is the General Purpose Analog Computer (GPAC) [18], which is a model of computation that can manipulate real-valued variables through the use of ordinary differential equations. Such models have been proven to approximate continuous functions in a sense that is similar to paradigm (5) for computable functions [4]. This model is probably the closest to RNNs, because it manipulates real-valued variables through the use of continuous transformations. However, the computations of the GPAC model are based on a continuous-time dynamics, while the computations of RNNs are based on a discrete-time dynamics and use update functions that are not piecewise-linear but rather polynomial, which leaves no direct simulation of the GPAC model by RNNs. However, we strongly believe that there exist deep connections between the two models, and we leave the study of these interactions as future work. Finally, there is a class of models of computations called hybrid models [12], that are constituted of a discrete controller that has access to some continuous variables that can be manipulated. The discrete controller may decide to apply some function to the continuous variables, based on its internal state and on the value of the continuous variables. When the discrete controller does not depend on the continuous variables, we say that the hybrid model is in open-loop. The TMNU model that we introduce in this paper can be seen as a particular instance of open-loop hybrid models, where the discrete controller is a Turing machine, and where the continuous variables can only be manipulated through the use of piecewise-linear transformations. To the best of our knowledge, there is no literature investigating the approximation capabilities of such models in the sense of paradigm (5). Moreover, in order to keep the proof direct, we do not study the TMNU model in the context of hybrid models, but we believe that significant generalizations may be possible.
9
4.3
TMNUs and the approximation paradigm for TMNUs
Informally speaking, a TMNU is a Turing machine that has an extra component, which we call the "neural state", that is a vector of real numbers ω ∈ Rd of some fixed dimension d ∈ N, that we call neural dimension. The neural state is updated at each step by application of some function f : Rd → Rd that depends on the current internal state and the symbol read by the head. The functions that can be applied to the neural state are restricted to be of the form f (x) = Ax + b for some matrix A ∈ Rd×d and some vector b ∈ Rd , or f (x) = ρS (x) where S ⊆ {1, . . . , d} and ρS (x)i := ρ(xi ) if i ∈ S and ρS (x)i := xi if i ∈ /S for every x = (x1 , . . . , xd ) ∈ Rd , where ρ(x) := max(x, 0) is the ReLU activation function. For d ∈ N, we let Rd := {ρS : S ⊆ {1, . . . , d}}, and Ad,d the set of affine functions from Rd to Rd . We cast this informal description of TMNUs into the following definition. Definition 4.3. A TMNU is a quadruple M := (n, δ, d, κ), where (n, δ) is a Turing machine, d ∈ N is the neural dimension, and κ : n × Γ → Ad,d ∪ Rd is the command function which specifies a function f := κ(q, σ) to be applied to the neural state given an internal state q and a symbol σ read by the head. We define FM := {κ(q, σ) : q ∈ {1, . . . , n}, σ ∈ Γ} ⊆ Ad,d ∪ Rd , and νM := #(FM ). (20) Given an internal state q, a symbol σ read by the head, and a neural state ω, the TMNU M := (n, δ, d, κ) updates its internal state and tape according to the transition function δ given q and σ, and updates its neural state according to the function specified by the command function κ given q and σ. More precisely, if we denote by f := κ(q, σ) the function specified by the command function κ given the internal state q and the symbol σ read by the head, then the neural state is update to f (ω). Again, this can be formulated more compactly by using the formalism of configurations, as follows. Definition 4.4. Given a TMNU M := (n, δ, d, κ), we call configuration of M any triple c := (q; τ ; ω) such that q ∈ {1, . . . , n}, τ : Z → Γ and ω ∈ Rd . We let CM := {1, . . . , n} × ΓZ × Rd be the set of configurations of M. We give a meaning of M as a map from CM to CM by letting M(q; τ ; ω) := (M (q, τ ); κ(q, Rτ )(ω)),
(21)
for every configuration (q; τ ; ω) ∈ CM , where M := (n, δ) is the Turing machine component of M. As for Turing machines, the formalism above can be used to define a computation of a TMNU M := (n, δ, d, κ) starting from some initial configuration c ∈ CM as the sequence of configurations (Mt (c))t∈N0 obtained by iterating the map M : CM → CM starting from c. In order to ease notations, we define projections on the state, tape and neural state of M as follows. Given a configuration c := (q; τ ; ω) of M, we let πq (c) := q, πτ (c) := τ, πω (c) := ω. (22) As a very important remark, note that the evolution of the internal state πq Mt (c) and tape πτ Mt (c) of M is governed entirely by the Turing machine component of M, i.e., (πq Mt (c), πτ Mt (c)) = M t (πq c, πτ c)
for every t ∈ N0 .
This fact makes the TMNU model easy to manipulate, because one can first establish the evolution of the internal state and tape of M by reasoning in terms of Turing machines, and then establish the evolution of the neural state of M computing the successive neural states that are obtained by applying the piecewise-linear transformations specified by the command function κ, given the evolution of the internal state and tape. As mentioned above, beyond their ease of manipulation, the other interest of TMNUs is that they correspond, in a sense to be made clear in the later sections, to the computations of an RNN. Hence, to show the existence of an RNN that approximates a function f : [−1, 1] → R, it is enough to show the existence of a TMNU that approximates f , in a sense that is consistent with paradigm (5), that we now describe. We will say that a TMNU M := (n, δ, d, κ) approximates a function f : [−1, 1] → R if there exists some u ∈ {0, 1}# such that for every x ∈ [−1, 1], if we start the computation of M from the initial configuration c := (q; τ ; ω) where q = 1, τ = |u and ω is the vector of Rd whose first coordinate is x and whose other coordinates are 0, then the last coordinate of the neural state of M at time t converges to f (x) as t → ∞, in a uniform way with respect to x. To formalize this notion of approximation, we introduce a notation that describes the evolution of the last coordinate of the neural state of M at time t when we start the computation of M from the initial configuration c as described above. Definition 4.5. Given a TMNU M := (n, δ, d, κ), x ∈ R and u ∈ {0, 1}# , we let Mu x : N0 → R be the sequence defined by Mu x[t] := πd ◦ πω (Mt (1; |u; x, 0, . . . , 0)), (23) for every t ∈ N0 , where πd : Rd → R is the projection on the last coordinate of Rd , and where πω is the projection on the neural state defined in (22). 10
In the remainder of the paper, we show that given a function f : [−1, 1] → R, there exists a TMNU M and a sequence u ∈ {0, 1}# such that lim
sup |Mu x[t] − f (x)| = 0,
t→∞ x∈[−1,1]
(24)
which is consistent with the paradigm (5) for RNNs, and we will study the convergence rate of supx∈[−1,1] |Mu x[t]−f (x)| as t → ∞ for some specific functions f : [−1, 1] → R with particular smoothness properties. To sum up, the proof of the main result is based on the two main steps: (a) Showing that, under some mild conditions, the paradigm (24) for TMNUs is essentially equivalent to paradigm (5) for RNNs, in a sense to be made precise in Section 5. (b) Showing that TMNUs can approximate every continuous function f : [−1, 1] → R in the sense described above, and studying the convergence rate of the approximation for some specific functions f : [−1, 1] → R. This is done in Section 6. The combination of these two steps delivers the main results of the paper.
5
Simulation of TMNU by RNNs
In this section, we show that RNNs can simulate the computation of TMNUs, under some boundedness assumption of the TMNU. Specifically, we introduce a notion of magnitude of a configuration of a TMNU, and we show that if the configuration of a TMNU remains bounded during the computation, then there exists an RNN that simulates the computation of this TMNU. Specifically, given a TMNU M and a real number C > 0 we define the set of configurations of M with neural state bounded by C as the set given by
where
BM (C) := {c ∈ CM : ∥c∥ ≤ C},
(25)
∥c∥ := ∥πω (c)∥∞ ,
(26)
and we define the set of initial configurations whose trajectory under M has neural state bounded by C as the set given by ∞ BM (C) := {c ∈ CM : Mt (c) ∈ BM (C) for all t ∈ N0 }. (27) In this section, we will then show that given a TMNU M := (n, d, δ, κ) and C > 0, there exists an RNN R := RM,C , a natural number tM ∈ N0 and a mapping γM : CM → Rn+2+d , such that for every ∞ configuration c ∈ BM (C) and every t ∈ N0 , we have RDγM (c)[t] = γM (Mt//tM (c)),
(28)
where // denotes the integer division. The design of R and γM is largely inspired by the construction in [20] of an RNN that simulates a Turing machine, by means of simulating an intermediate machine called a stack machine. Our construction here does not rely on such an intermediate machine, and directly simulates the TMNU. The construction is effected in two steps. First, we show that there exists a continuous piecewise-linear function F := FM,C that implements M under some boundedness condition on its neural state, in the sense that γM ◦ M(c) = F ◦ γM (c),
c ∈ BM (C).
(29)
Then, we show how to use F to design an RNN R that simulates the computation of M as in (28). We first explain how γM is designed. Given a configuration c := (q; τ ; ω), we want to encode it as a vector xc ∈ Rn+2+d , precisely of the form xc := (1n,q , xτ , ω), where 1n,q is the one-hot encoding of the state q, specifically given by 1n,q := (δ1,q , δ2,q , . . . , δn,q ) ∈ {0, 1}n ,
(30)
such that δi,q = 1 if i = q and δi,q = 0 otherwise is the usual Kronecker symbol, and xτ ∈ [−1, 1]2 is a pair of real numbers that encodes the tape τ that we define as follows. xτ1 encodes the right part of the tape, and xτ2 encodes its left part, i.e., by denoting τ + := (τi−1 )i∈N ,
and
11
τ − := (τ−i )i∈N ,
(31)
M
BM (C)
CM γM
γM
Rm
Rm
FM,C
Figure 2: Principle of the simulation of M with a piecewise-linear function FM,C under some boundedness condition on its neural state. The simulation is expressed by the fact that the above diagram commutes. One challenge is ensure sure that M maps BM (C) to itself, so that one can iterate the simulation. we have that xτ1 encodes τ + and xτ2 encodes τ − . Specifically, we define the encoding of a sequence (ui )i∈N ∈ ΓN as the real number given by η̃ :
ΓN
→ 7 →
(ui )i∈N
4
1] P[−1, ∞ −i u i=1 i 4 .
(32)
The interest of such an encoding resides in the fact that one can read the first symbol u1 of a sequence (ui )i∈N by application of a continuous piecewise-linear function to η̃((ui )i∈N ), which is a key ingredient for the simulation of the reading operation of the tape by a neural network. We can then define the encoding of a tape τ as the pair of real numbers given by η̃(τ + ) and η̃(τ − ), i.e., we define the encoding of a tape τ as the vector given by η : ΓZ → [−1, 1]2 (33) τ 7→ (η̃(τ + ), η̃(τ − )). To sum up, we have defined the mapping γM as follows. Given a configuration c := (q; τ ; ω), we have γM :
Q × Γ Z × Rd (q; τ ; ω)
→ Rn+2+d 7 → (1n,q , η(τ ), ω).
(34)
We now explain the two steps of the construction of the RNN R that simulates the computation of M. We only sketch the main ideas of the construction here, but the exact details can be found in Appendix B. The first step is to show that there exists a piecewise-linear function F such that (29) holds. Following the spirit of [20], we simply notice that every component of M can be realized by a piecewise-linear function. Specifically, the reading, writing and shifting operations on the tape can be implemented by continuous piecewise-linear functions over the encoding of the tape. Moreover, the transition function δ being simply a lookup table, it can be implemented by a continuous piecewise-linear function by linear interpolation. The command function κ can be seen as choosing a function f among the finite set of functions FM depending on the state and the symbol read on the tape, which can also be formulated as lookup table and hence be implemented by a continuous piecewise-linear function by linear interpolation. Finally, applying the function f chosen by the command function κ is the more delicate part, and the reason for the boundedness assumption. The strategy is the following. First, upon noting that the functions fi ∈ FM , i ∈ {1, . . . , νM } are piecewise-linear, the function ω ∈ Rd 7→ (f1 (ω), f2 (ω), . . . , fνM (ω)) ∈ RνM d . Now, assuming that the command function has chosen a function fi among the functions of M, we want to design a piecewise-linear function that takes as input the vector (f1 (ω), f2 (ω), . . . , fνM (ω)) and returns fi (ω). This can be done by remarking that fi (ω) =
νM X
δi,j fj (ω),
(35)
j=1
and that, by extending slightly [20, Lemma 4.1], one has the following property. Lemma 5.1. Let M > 0, z ∈ [0, M ] and b ∈ {0, 1}. Then, we have ρ(z + M (b − 1)) = bz. Proof. If b = 0, then we have ρ(z + M (b − 1)) = ρ(z − M ) = 0. If b = 1, then we have ρ(z + M (b − 1)) = ρ(z) = z. This concludes the proof. Note that for every j ∈ {1, . . . , νM }, fj is continuous, and hence for every C > 0, we have sup
∥fj (ω)∥∞ < ∞.
ω∈[−C,C]d
12
(36)
Accordingly, we define
∥M∥C :=
max
sup
j∈{1,...,νM } ω∈[−C,C]d
∥fj (ω)∥∞ ,
(37)
and by application of Lemma 5.1 with M = ∥M∥C , we can reformulate (35) as fi (ω) =
νM X
ρ(fj (ω) + ∥M∥C (δi,j − 1)),
(38)
j=1
which is a piecewise-linear function. Combining the piecewise-linear functions defined above results in establishing the existence of a piecewise-linear function F such that (29) holds. In fact, in Appendix B.2, we get more quantitative about the shape of this piecewise-linear function F . In fact, the piecewise-linear function F that we construct is a deep ReLU neural network, defined as follows. Definition 5.1. (ReLU Neural network) We call ReLU neural network an ordered sequence Ψ := (N0 , N1 , . . . , NL ; A1 , b1 , A2 , b2 , . . . , AL , bL ),
(39)
where L ∈ N and N0 , N1 , . . . , NL ∈ N, Aℓ = (Aℓjk ) ∈ RNℓ ×Nℓ−1 and bℓ = (bℓj ) ∈ RNℓ for ℓ ∈ {1, . . . , L}. We consider the ReLU neural network Ψ as a function Ψ : RN0 → RNL defined by Ψ(x) := AL ρ AL−1 ρ . . . ρ A1 x + b1 + bL−1 + bL for all x ∈ RN0 , (40) where ρ is applied coordinate-wise. We let W(Ψ) :=
max
ℓ∈{0,...,L}
Nℓ ,
L(Ψ) := L,
M(Ψ) :=
max
ℓ∈{1,...,L}
∥Aℓ ∥∞ ∨ ∥bℓ ∥∞ .
We let Nd,k be the set of ReLU neural networks with input dimension N0 = d and output dimension NL = k. We generally say that a neural network Ψ is deep if L(Ψ) > 2. Our final simulation theorem stipulates that we can simulate the iterations of a M by the iterations of a deep ReLU neural network (of depth 4), and that the width and the magnitude of the weights and biases of this neural network can be upper bounded by a quantity that depends on n, d, νM and ∥M ∥C . Theorem 5.2. Let M := (n, d, δ, κ) be a TMNU and C > 0. Then, there exists a ReLU neural network Ψ ∈ Nn+2+d,n+2+d such that L(Ψ) = 4,
W(Ψ) ≤ max{3n, n + νM + 9} + 2dνM + 18,
and
M(Ψ) ≤ max{4, ∥M ∥C },
(41)
satisfying Ψ(γM (c)) = γM (M (c)),
c ∈ BM (C).
(42)
Proof. The proof is carried out by designing concrete ReLU neural networks that implement the continuous piecewise-linear functions defined above, and combining them into a larger neural network. For the full proof, see the proof of Theorem B.14. We now convert this result into the existence of an RNN R that simulates the computation of M as in (28). Specifically, we show that the iterations of a deep ReLU neural network can be simulated by an RNN. Theorem 5.3. Let n ∈ N and Ψ ∈ Nn,n be a ReLU neural network. Then, there exists an RNN R such that m(R) ≤ (2L(Ψ) + 2)(2n ∨ W(Ψ) + 1), and M(R) ≤ max{4, M(Ψ)}, (43) satisfying RDx[t] = Ψt//L(Ψ) (x),
t ∈ N,
x ∈ Rn .
Sketch of the proof. The proof relies on designing an RNN Rf that reflects the structure of Ψ. Essentially, at each time step, the RNN performs one of the affine transforms Aℓ , adds the corresponding bias bℓ , and applies the ReLU function. After L := L(Ψ) time steps, the RNN will have computed Ψ(x), and then it can repeat this process to compute Ψ(Ψ(x)), and so on. This construction comes with some technicalities. For example, the RNN cannot hold negative values in its hidden state, so we need to design the RNN to hold both the positive and negative parts of the intermediate computations, and then combine them appropriately to recover the correct output. Moreover, we need a mechanism to retain the value of Ψi (x) for L iterations, while the RNN is performing the computations for Ψi+1 (x), and replace this value with Ψi+1 (x) after L iterations. This can be achieved by using some additional coordinates in the hidden state of the RNN to keep track of the current iteration and to store the intermediate values. The details of this construction can be found in Appendix B.3. 13
We finally conclude that the approximation paradigm (5) for RNNs corresponds to the approximation paradigm (24) for TMNUs, given that a certain boundedness condition holds. Given a TMNU M and C > 0, we say that M has uniformly C-bounded trajectories at u ∈ {0, 1}N if ∞ {cx = (1; |u; x, 0, . . . , 0) : x ∈ [−1, 1]} ⊆ BM (C).
(44)
Theorem 5.4. Let M := (n, d, δ, κ) be a TMNU, C > 0 and u ∈ {0, 1}N such that M has uniformly C-bounded trajectories at u. Then, there exists an RNN R such that m(R) ≤ 10(max{3n, n + νM + 9} + 2dνM + 19) + 1,
and
M(R) ≤ max{4, ∥M∥C },
(45)
satisfying RDx[t] = Mu x[t//4],
t ∈ N0 ,
x ∈ [−1, 1].
(46)
Sketch of the proof. Let M = (n, d, δ, κ) be a TMNU, C > 0 and u ∈ {0, 1}N such that M is uniformly C-bounded at u. Upon invoking Theorems 5.2 and 5.3 together, we get that there exists an RNN R such that m(R) ≤ 10(max{3n, n + νM + 9} + 2dνM + 19), and M(R) ≤ max{4, ∥M∥C }, satisfying
RD(γM (c))[t] = γM (Mt//4 (c)),
∞ c ∈ BM (C).
t ∈ N0 ,
(47)
Now, note that since M is uniformly C-bounded at u, we have that for every x ∈ [−1, 1], cx := ∞ (1; |u; x, 0, . . . , 0) ∈ BM (C), so that for every x ∈ [−1, 1] and t ∈ N0 , we have RDγM (cx )[t] = γM (Mt//4 (cx )).
(48)
Moreover, for every x ∈ [−1, 1], we have γM (cx ) = (1n,1 , η(|u), x, 0, . . . , 0) = (0n+2 , x, 0d−1 )x + (1n,1 , η(|u), 0, . . . , 0) =: Ax + b, and moreover,
Mu x[t] = πd πω Mt (cx ) = πn+2+d γM (Mt (cx )) =: A′ γM (Mt (cx )).
We show in Lemmata B.15 and B.16 that RNNs are essentially stable through affine transformations of their inputs and outputs, and that we can design an RNN R′ such that m(R′ ) = m(R) + 1 satisfying
and
M(R′ ) = max{4, ∥M ∥C },
R′ Dx[t] = A′ RD(Ax + b)[t],
t ∈ N0 ,
x ∈ [−1, 1].
(49) (50)
Therefore, we have R′ Dx[t] = A′ RD(Ax + b)[t] = A′ γM (Mt//4 (cx )) = πd πω γM (Mt//4 (cx )) = Mu x[t//4].
(51)
For more details, see the proof of Theorem B.22. This concludes the proof. We hence have reduced approximation paradigm (5) for RNNs to approximation paradigm (24) for TMNUs. In the next section, we will solve the approximation paradigm (24) for continuous functions, and hence conclude that approximation paradigm (5) for RNNs can be solved.
6
TMNU constructions
In this section, we explain the construction of a TMNU that approximates every continuous function f : [−1, 1] → R in the sense of paradigm (24). The mathematical strategy is supplied by the Weierstrass approximation theorem: every continuous function on [−1, 1] is the uniform limit of a sequence of polynomials. By truncating the binary expansions of their coefficients, we obtain a sequence of finitely encodable dyadic polynomials (Pi )i∈N0 that still converges uniformly to f . Thus, at an abstract level, it suffices to design a machine that reads the successive finite encodings of each Pi from its tape, evaluates the polynomials successively at the input x, and outputs increasingly accurate approximations of f (x). The construction below may therefore be viewed as an implementation of the Weierstrass approximation theorem by a TMNU. The key question is how one fixed machine can implement this increasingly complex sequence of polynomial computations. Our answer is to treat TMNUs as programs assembled from reusable subroutines. Rather than defining the final machine C directly and analyzing one large transition system, we first construct machines performing elementary arithmetic operations. We then embed them as subroutines of 14
progressively more expressive machines. This subroutine mechanism is not merely a device for making the proof manageable; it is also the intuition behind the main theorem of the paper. It suggests that one fixed TMNU can implement the Weierstrass approximation procedure for every continuous function by composing a finite collection of elementary operations according to the information encoded on its tape. Since a uniformly bounded TMNU computation can be transferred to an RNN by Theorem 5.4, this gives a conceptual reason to expect that a fixed-size RNN can approximate arbitrary continuous functions. At the technical level, the subroutine mechanism transfers the complete trajectory of a smaller machine into every larger machine that calls it. Specifically, if a machine M contains another machine N as a subroutine, then on prescribed states and neural coordinates, M follows exactly the trajectory of N, while its remaining neural coordinates are carried along unchanged. The shadow of an ambient configuration extracts the configuration seen by N, while the lift inserts the resulting subroutine configuration back into M. Schematically, throughout a call to the subroutine, trajectory in M = lift trajectory in N . This identity means that a lemma proved for N immediately describes the corresponding portion of the trajectory of M. Consequently, the input-output behavior, running time, approximation error, and trajectory bound of a machine can all be reused at the next level of the construction. This makes it possible to control the two quantitative properties needed later: the convergence rate of C and the uniform boundedness of its complete trajectory. The latter is precisely the hypothesis required by Theorem 5.4 to transfer the final TMNU construction to an RNN.
6.1
Overview of the TMNU constructions for the main result
The construction is organized as the following hierarchy of subroutine calls: ± , Hom+ −→ Hom,
Hom, × −→ π,
π, S −→ Π,
Π −→ C.
The machine S reads a block 1k 0 and sends x to 2k x. The machines ± and Hom+ respectively apply a sign and multiply by a nonnegative dyadic number a ∈ [0, 1), and are combined as subroutines of Hom to implement x 7→ ax for any dyadic a ∈ (−1, 1). Finally, × reads a computational parameter 1n 0 and approximates (x, y) 7→ xy up to an error of order 2−2n . These elementary machines form the reusable arithmetic instructions of the construction. At the next level, π calls Hom and × to perform one monomial-accumulation update. If the neural state contains the input x, a current approximation y of the monomial xi , and a current partial sum z, while the tape contains a dyadic coefficient ai together with a multiplication parameter n, then π approximately sends (x, y, z) 7→ (x, yx, z + ai y). Starting from y = 1 and z = 0, the machine Π repeatedly calls the π subroutine to compute the successive monomial contributions ai xi and add them to the partial sum. For the boundedness reasons explained below, it actually evaluates a rescaled polynomial P ∗ and then calls S to recover the corresponding approximation of P . Thus, for a dyadic polynomial P and a multiplication parameter n, Π produces an approximation FP,n (x) of P (x). Its behavior and trajectory bound follow by composing the previously established properties of π and S. At the top of the hierarchy, C repeatedly calls Π. Its tape is an infinite encoding of dyadic polynomials Pi and multiplication parameters ni , and during its i-th cycle the Π subroutine computes FPi ,ni (x), which C stores as its current output. If Pi converges uniformly to a continuous function f and the parameters ni are chosen so that FPi ,ni is close to Pi , then these successive outputs converge to f . Once again, the
S
±
Hom+ Hom π Π
× Machine S ± Hom+ Hom × π Π C
Purpose multiply by a power of 2 flips the sign of the input apply a positive dyadic contraction y 7→ ay apply a dyadic contraction y 7→ ay approximate multiplication one polynomial micro-update evaluate one encoded polynomial iterate through a sequence of polynomials
C Figure 3: Subroutine hierarchy for evaluating dyadic polynomials in the monomial basis. 15
subroutine lemma turns the analysis of each complete cycle of C into the already established analysis of Π. The main additional difficulty is uniform boundedness. Convergence of the outputs alone would establish an approximation result for C, but would not yet allow us to obtain the desired RNN: the correspondence result of Theorem 5.4 applies only when the simulated TMNU computation is uniformly bounded. Although each individual computation of Π is bounded by a finite quantity depending on the ℓ1 -norm of its coefficients, deg(Pi ) X ∥Pi ∥1 := |aj |, j=0
the sequence (∥Pi ∥1 )i∈N0 need not be uniformly bounded. To avoid transferring this possible growth to the neural state, Π first evaluates a rescaled polynomial Pi∗ satisfying ∥Pi∗ ∥1 < 1, so that all monomials and partial sums remain bounded by one, and then calls S to recover the desired output. Consequently, the i-th cycle is bounded in terms of 1 ∨ |FPi ,ni (x)|, rather than ∥Pi ∥1 . Since the outputs remain close to f , the complete trajectory of C is uniformly bounded in terms of 1 ∨ ∥f ∥L∞ ([−1,1]) . This verifies the hypothesis needed to apply Theorem 5.4 and transfer the construction from C to an RNN. By composing the subroutine lemmas along the hierarchy above, we obtain the following main TMNU approximation statement, which is the basis for Theorem 3.1. Theorem 6.1. For every continuous function f : [−1, 1] → R, there exists u ∈ {0, 1}N such that the machine C is uniformly Cf -bounded at u and satisfies lim ∥Cu x[t] − f (x)∥L∞ ([−1,1]) = 0,
t→∞
where Cf := 1 + ∥f ∥L∞ ([−1,1]) . Proof. See Theorem C.14.
6.2
Convergence rates for polynomials
The same modular analysis also gives quantitative rates. Each subroutine lemma records its running time and approximation error; composing these estimates gives the cost and error of every cycle of C. Importantly, the output produced during cycle i is reached only after all preceding polynomials P0 , . . . , Pi have been read and evaluated. The relevant time cost is therefore the cumulative time Ni ≍
i X
cost(Pj ),
j=0
rather than the cost of evaluating Pi alone. To obtain a sharp rate in the machine time t, the sequence (Pi )i∈N0 must be chosen so that its approximation error decreases rapidly compared with this cumulative cost. Consider first a fixed polynomial P with real coefficients. A naive first choice would be to let cycle i evaluate the dyadic polynomial obtained by truncating the coefficients of P after i binary digits. This gives an error of order 2−i and, since the degree of P is fixed, a cost of order i for cycle i. However, the cumulative time then satisfies i X Ni ≍ j ≍ i2 , j=0 √
so an error of order 2−i becomes only an error of order 2− t when expressed in terms of machine time. To avoid this loss, cycle i instead uses a dyadic polynomial Pi obtained by truncating the coefficients i of P after a number of bits of order 2i . Its approximation error is then of order 2−2 , while its encoding i and evaluation cost is of order 2 . Moreover, the geometric growth makes the cost of evaluating the latest polynomial comparable with the cost of evaluating the entire sequence up to that point: cost(Pi ) ≍ 2i
and
Ni ≍
i X
2j ≍ 2i .
j=0 i
Thus, by the time t is of order 2i , the output error is of order 2−2 , which becomes an error of order 2−t . In this sense, the exponential truncation schedule is optimal up to constant factors for this sequential strategy: evaluating Pi alone already costs order 2i , and evaluating all earlier approximations adds only a comparable amount of work. This strategy results in the following exponential convergence rate for polynomials.
16
PN Theorem 6.2. Let N ∈ N, and let P (x) := i=0 ai xi for some a0 , . . . , aN ∈ R. Then, there exists u ∈ {0, 1}N such that C is uniformly CP -bounded at u, where CP := 1 + ∥P ∥L∞ ([−1,1]) , and satisfying |Cu x[t] − P (x)| ≤ 2 · 2
− tt
P
,
for all x ∈ [−1, 1], t ≥ τP ,
where tP := 18(N + 1), τP := tP (6∥a∥1 + N + 11) + 1. Proof. See Theorem C.16 in Appendix C.
6.3
Convergence rates for general continuous functions
For more general continuous functions, the strategy is to specifically choose the approximating polynomials Pi to be the Chebyshev truncations of f . However, the monomial construction above is not well-suited for this choice, because its would require first expanding the Chebyshev truncation in powers of x and then evaluating the resulting monomial sum. The coefficients of the monomial expansion grow exponentially with the degree, so the the downscaling-upscaling scheme implemented in the machine Π to keep the trajectory bounded would cause an additional time cost that leads to poor convergence rates. To avoid this issue, the construction no longer passes through the monomial machine Π. Instead, it uses the Chebyshev basis directly. The relevant hierarchy of subroutines is × −→ TStep,
Hom, TStep, S −→ TSum,
TSum −→ TC.
Here, upon receiving a parameter n ∈ N, TStep is used to compute approximations of the Chebyshev polynomials by calling the machine × to approximate the recurrence relation Tk+2 (x) = 2xTk+1 (x)−Tk (x), with a precision of order 2−2n . Upon being iterated, TStep hence compute approximations T̃k,n (x) of the Chebyshev polynomials Tk (x) with a uniform error of k 2 2−2n−1 , which constitutes a mild blows up with k. This mild blow-up is key in the obtained convergence rates, and is established by a careful stability analysis of the Chebyshev recurrence in Lemma C.20, which notably relies on the use of Chebyshev polynomials of the second kind. The machine TSum evaluates one finite Chebyshev sum with dyadic coefficients. Given dyadic coefficients a = (a0 , . . . , ad ) and a precision parameter n, it successively forms the Chebyshev values by repeated calls to TStep, uses Hom to multiply the current value by the next dyadic coefficient, and adds the result to an accumulator. As in the monomial construction, it first works with a rescaled coefficient vector a∗ satisfying 2∥a∗ ∥1 < 1, so that the accumulator stays bounded during the computation, and finally calls S to undo the rescaling. The output is a function denoted Ha,n , that satisfies Ha,n (x) −
d X
ak Tk (x) ≤ (1 + 2∥f ∥L∞ ([−1,1]) )d2 2−2n−1 .
k=0
Finally, given a function f : [−1, 1] → R, TC repeatedly calls TSum on successive degrees di , precision parameters ni , and dyadic coefficient vectors c̃i := (c̃i,0 , . . . , c̃i,di ) ∈ Rdi +1 that approximate the Chebyshev (f ) (f ) (f ) coefficient vectors ci := (c0 , . . . , cdi ) of f to precision pi . Therefore, during its i-th cycle, TC computes Hc̃i ,ni (x), which by the previous estimate gives an approximation of the Chebyshev truncation |TCu x[t] − f (x)| ≤ Hc̃i ,ni (x) −
di X
c̃i,k Tk (x) +
di X
(f )
(f )
c̃i,k Tk (x) − Sdi (x) + Sdi (x) − f (x)
k=0
k=0 2 −2ni −1 ≤ (1 + 2∥f ∥L∞ ([−1,1]) )di 2 + di 2−pi + S(f, di ),
while requiring a time of order Ti ≃ di (qi + ni + log(di )) to read the coefficients and perform the calls to TStep. We choose the parameters di , pi , and ni to balance the three terms in the error estimate above,
S
±
Hom+
×
Hom
TStep
TSum TC
Machine S ± Hom+ Hom × TStep TSum TC
Purpose multiply by a power of 2 flips the sign of the input apply a positive dyadic contraction y 7→ ay apply a dyadic contraction y 7→ ay approximate multiplication computes one step of the Chebyshev recursion computes a dyadic Chebychev partial sum computes a Chebyshev expansion
Figure 4: Subroutine hierarchy for evaluating dyadic Chebyshev sums. 17
and to make Ti of order 2i , so that the cumulative time up to cycle i is also of order 2i . Specifically, given a function η : [1, ∞) → N, we set di := η(2i ),
pi ≃ ni ≃ 2i /di + O(log2 (di )).
(52)
With these choices, we have Ti = O(2i ) and the error estimate becomes i
i
|TCu x[t] − f (x)| ≃ 2−2 /η(2 ) + S(f, η(2i )). Since the cumulative time up Ni to cycle i is of order O(2i ), this gives a convergence rate of order t/τ
2− η(t/τ ) + S(f, η(t/τ )), for some constant τ > 0, which is the intended tradeoff between the Chebyshev truncation error and the machine error, expressed in terms of machine time t. Theorem 6.3. Let f : [−1, 1] → R be continuous. Let η : [1, ∞) → N be nondecreasing, with η(t) → ∞ and t/η(t) nondecreasing. Assume that there exists A > 0 such that η(t) log2 (η(t) + 1) ≤ At,
t ≥ 1.
Define γf := log2 2∥f ∥L∞ ([−1,1]) + 1 and τ := τf,A := 4 10 + γf + A(80 + 6γf ) . Then there exists u ∈ {0, 1}N such that TC is uniformly Cf -bounded at u with Cf := 3 + S(f, 0) + ∥f ∥L∞ ([−1,1]) , and for every x ∈ [−1, 1] and t ≥ τ , t/τ
|TCu x[t] − f (x)| ≤ 2 · 2− η(t/τ ) + S (f, η(t/τ )) . Proof. See Theorem C.23 in Appendix C.
References [1] A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Transactions on Information Theory, vol. 39, no. 3, p. 930–945, May 1993. [2] ——, “Approximation and estimation bounds for artificial neural networks,” Machine Learning, vol. 14, no. 1, p. 115–133, Jan. 1994. [3] L. Blum, F. Cucker, M. Shub, and S. Smale, Complexity and real computation.
Springer, 1998.
[4] O. Bournez, M. L. Campagnolo, D. S. Graça, and E. Hainry, “Polynomial differential equations compute all real computable functions on computable compact intervals,” Journal of Complexity, vol. 23, no. 3, pp. 317–335, 2007. [5] V. Brattka, P. Hertling, and K. Weihrauch, “A tutorial on computable analysis,” New computational paradigms: changing conceptions of what is computable, pp. 425–491, 2008. [6] G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, no. 4, pp. 303–314, 1989. [7] D. Elbrächter, D. Perekrestenko, P. Grohs, and H. Bölcskei, “Deep neural network approximation theory,” IEEE Transactions on Information Theory, vol. 67, no. 5, pp. 2581–2623, 2021. [8] J. L. Elman, “Finding structure in time,” Cognitive Science, vol. 14, no. 2, pp. 179–211, 1990. [9] K.-I. Funahashi, “On the approximate realization of continuous mappings by neural networks,” Neural Networks, vol. 2, no. 3, pp. 183–192, 1989. [10] A. Gil, J. Segura, and N. M. Temme, Numerical methods for special functions. [11] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. www.deeplearningbook.org 18
SIAM, 2007.
MIT Press, 2016. [Online]. Available:
[12] T. A. Henzinger, “The theory of hybrid automata,” in Proceedings 11th Annual IEEE Symposium on Logic in Computer Science, 1996, pp. 278–292. [13] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, 1989. [14] C. Hutter, V. Abadie, and H. Bölcskei, “A quantifier-reversal approximation paradigm for recurrent neural networks,” arXiv preprint arXiv:2511.15326, 2025. [15] A. N. Kolmogorov, “ε-entropy and ε-capacity of sets in functional spaces,” Uspekhi Matematicheskikh Nauk, vol. 14, no. 2(86), pp. 3–86, 1959. [16] J. M. Lee, Introduction to Topological Manifolds, ser. Graduate Texts in Mathematics. 2011, vol. 202.
Springer,
[17] J. Schmidt-Hieber, “The Kolmogorov–Arnold representation theorem revisited,” Neural Networks, vol. 137, pp. 119–126, May 2021. [18] C. E. Shannon, “Mathematical theory of the differential analyzer,” Journal of Mathematics and Physics, vol. 20, no. 1-4, pp. 337–354, 1941. [19] J. W. Siegel, “Optimal approximation rates for deep ReLU neural networks on Sobolev and Besov spaces,” Journal of Machine Learning Research, vol. 24, no. 357, pp. 1–52, 2023. [20] H. T. Siegelmann and E. D. Sontag, “On the computational power of neural nets,” in Proceedings of the fifth annual workshop on Computational learning theory, 1992, pp. 440–449. [21] M. Sipser, Introduction to the Theory of Computation, 3rd ed.
Cengage Learning, 2013.
[22] A. I. Stepanets, Uniform approximations by trigonometrical polynomials.
VSP, 2001.
[23] M. Telgarsky, “Neural networks and rational functions,” in Proceedings of the 34th International Conference on Machine Learning. PMLR, Jul. 2017, pp. 3387–3393. [24] L. N. Trefethen, Approximation theory and approximation practice, extended edition. [25] K. Weihrauch, Computable analysis: an introduction.
SIAM, 2019.
Springer, 2000.
[26] D. Yarotsky, “Error bounds for approximations with deep ReLU networks,” Neural Networks, vol. 94, pp. 103–114, 2017.
A
Notation and definitions
(a) Given A ⊆ R, we let ±A := A ∪ {−x : x ∈ A}. (b) Let A ∈ Rn×m . We say that A is a left-selector matrix if A ∈ {0, 1}n×m and every row of A contains at most one entry equal to 1. We say that A is a right-selector matrix if A ∈ {0, 1}n×m and every column of A contains at most one entry equal to 1. (c) Let A ∈ Rm×n and b ∈ Rm . We define Ω(A) := {Ajk : j ∈ {1, . . . , m}, k ∈ {1, . . . , n}},
Ω(b) := {bj : j ∈ {1, . . . , m}}.
(53)
Given a ReLU neural network Ψ := (N0 , N1 , . . . , NL ; A1 , b1 , . . . , AL , bL ), we define Ω(Ψ) :=
L [
Ω(Aℓ ) ∪ Ω(bℓ ).
(54)
ℓ=1
Given an affine map W : Rn → Rm , note that W is uniquely defined by W (x) = Ax + b for all x ∈ Rn , for some A ∈ Rm×n and b ∈ Rm . We then define Ω(W ) := Ω(A) ∪ Ω(b).
(55)
Given a TMNU M := (n, d, δ, κ), we define Ω(M) :=
[ q∈Q,σ∈Γ
19
Ω(κ(q, σ)).
(56)
(d) We define the lexicographical ordering on Rm by x <L y ⇐⇒ ∃i ∈ {1, . . . , m} such that xi <L yi and xj = yj for all j ∈ {1, . . . , i − 1}.
(57)
We define the lexicographical ordering on Rn×m by A <L B ⇐⇒ ∃i ∈ {1, . . . , n} such that Ai <L Bi and Aj = Bj for all j ∈ {1, . . . , i − 1},
(58)
where Ai and Bi denote the i-th rows of A and B, respectively. We define the lexicographical ordering on An,m by f <L g ⇐⇒ A <L A′ or A = A′ and b <L b′ , (59) where f (x) = Ax + b and g(x) = A′ x + b′ for all x ∈ Rn . Given A ⊆ {1, . . . , n}, wew define χA := (χA (1), . . . , χA (n)) ∈ Rn , where χA (i) := 1 if i ∈ A and χA (i) := 0 otherwise. We define the lexicographical ordering on Rn by ρA <L ρB ⇐⇒ χA <L χB .
(60)
Finally, we define the lexicographical ordering on An,n ∪ Rn by f <L g ⇐⇒ f, g ∈ An,n and f <L g, or f, g ∈ Rn and f <L g, or f ∈ An,n and g ∈ Rn .
(61)
Note that the lexicographical ordering defined in this way is a total order.
B
Construction of an RNN that simulates a TMNU
This appendix is devoted to the detailed construction of an RNN that simulates a TMNU. The construction will be mainly divided into three parts: first, we show that the update of a TMNU can be implemented by a continuous piecewise-linear (CPWL) function (CPWL), then we will construct a ReLU neural network that implements this CPWL function, and finally we will show how to use this ReLU neural network to construct an RNN that simulates the TMNU.
B.1
Construction of a piecewise-linear function that implements the transition function of a TMNU
This section is devoted to showing that the transition function of a TMNU can be implemented by a continuous piecewise-linear function. We divide the analysis in 3 parts: first, we show how the actions on the tape can be implemented by CPWL functions, then we show how the transition and command functions can be implemented by CPWL functions, and finally we show how to combine these CPWL functions to obtain a CPWL function that implements the transition function of the TMNU. B.1.1
Reading, writing and shifting operations as continuous piecewise-linear functions over the encoding of the tape
In this part, we show that the reading, writing and shifting operations on the tape can be implemented by continuous piecewise-linear functions over the encoding of the tape. The first lemma isolates these three elementary tape operations at the level of the Cantor encoding. Lemma B.1. There exists continuous piecewise-linear functions r : R2 → R, wσ : R2 → R2 for σ ∈ {−1, 0, 1} and sm : R2 → R2 for m ∈ {−1, 0, 1} such that for every tape τ , we have r(η(τ )) = Rτ, Proof. First note that
wσ (η(τ )) = Wσ τ,
η̃(x) ∈ u1 + [−1/3, 1/3],
sm (η(τ )) = Sm τ.
(62)
u ∈ ΓN .
Therefore, depending on the value of u1 , the value of η(u) belongs to one of the three disjoint intervals I−1 := [−4/3, −2/3], I0 := [−1/3, 1/3] and I1 := [2/3, 4/3]. We can then design a piecewise-linear function that is equal to −1 on I−1 , 0 on I0 and 1 on I1 can be used to read the first element of u. We denote by f such a function, i.e., f : R → R satisfies f (η̃(u)) = u1 ,
u ∈ ΓN .
(63)
A graphical representation of f is given on Figure 5, which can be easily cast into the following piecewise-
20
linear function defined by
−1 3x + 1 f (x) := 0 3x − 1 1
if x ≤ −2/3, if − 2/3 < x < −1/3, if − 1/3 ≤ x ≤ 1/3, if 1/3 < x < 2/3, if x ≥ 2/3.
(64)
f (x) 1 x 0.5
−2 −1.5 −1 −0.5
1
1.5
2
−1 I−1
I0
I1
Figure 5: A piecewise-linear function that is equal to −1 on I−1 , 0 on I0 and 1 on I1 . Now, we can use this function f to realize the reading, writing and shifting operations. Namely, we define the reading operation r : R2 → R by r(x, y) := f (x),
(65)
x, y ∈ R,
the writing operations wσ : R2 → R2 for σ ∈ {−1, 0, 1} by wσ (x, y) := (x + σ − f (x), y),
(66)
x, y ∈ R,
and the shifting operations sm : R2 → R2 for m ∈ {−1, 0, 1} by if m = 0, (x, y) sm (x, y) := (4(x − f (x)), f (x) + y/4) if m = 1, (f (y) + x/4, 4(y − f (y))) if m = −1,
x, y ∈ R.
(67)
First note that we have defined r, wσ and sm as linear combinations of x, y and f (x), and hence these functions are continuous piecewise-linear. To see that these functions indeed implement the claimed operations, fix a tape τ ∈ ΓZ . Then, first have (33)
(65)
(63)
(31)
r(η(τ )) = r(η̃(τ + ), η̃(τ − )) = f (η̃(τ + )) = τ1+ = τ0 = Rτ. Second, fix σ ∈ {−1, 0, 1}, and define τσ+ ∈ ΓN by τσ+ (1) := σ and τσ+ (i) := τ + (i) for every i ≥ 2. Note that (Wσ τ )+ = τσ+ and (Wσ τ )− = τ − . Then, we have (33)
(66)
(63)
wσ (η(τ )) = wσ (η̃(τ + ), η̃(τ − )) = (η̃(τ + ) + σ − f (η̃(τ + )), η̃(τ − )) = (η̃(τ + ) + σ − τ1+ , η̃(τ − )). Note that (Wσ τ )+ = (τσ+ , τ − ). Moreover, we have (32)
η(τ + ) + σ − τ1+ = 4
∞ X
τi+ 4−i + σ − τ1+ = 4 · σ · 4−1 + 4
i=1
∞ X
τi+ 4−i = η̃(τσ+ ).
i=2
and hence
(33)
wσ (η(τ )) = (η̃(τσ+ ), η̃(τ − )) = (η̃((Wσ τ )+ ), η̃((Wσ τ )− )) = η(Wσ τ ). Finally, for m = 0, we have s0 (η(τ )) = η(τ ) = η(S0 τ ). For m = 1, we have (67)
(63)
s1 (η(τ )) = (4(η̃(τ + ) − f (η(τ + ))), f (η(τ + )) + η̃(τ − )/4) = (4(η̃(τ + ) − τ1+ ), τ1+ + η̃(τ − )/4).
21
Note that (32)
4(η(τ + ) − τ1+ ) = 4
∞ X
τi+ 4−i+1 = 4
i=2
τi−1 4−i+1 = 4
i=2
and, similarly, Hence, we have
∞ X
∞ X
τi−2 4−i = 4
i=1
∞ X
−i S(τ, 1)+ = η̃((S1 τ )+ ). i 4
i=1
τ1+ + η̃(τ − )/4 = η̃((S1 τ )− ).
(68)
s1 (η(τ )) = (η̃((S1 τ )+ ), η̃((S1 τ )− )) = η(S1 τ ).
We can analogously show that s−1 (η(τ )) = η(S−1 τ ). This concludes the proof. In order to condense the incoming notation, we combine δ m and δ σ into a single function δ τ : Q × Γ → {1, . . . , 9} defined by δ τ (q, σ) := 3 · δ m (q, σ) + δ σ (q, σ) + 5, q ∈ Q, σ ∈ Γ. (69) Note that (69) implicitly associates a number k ∈ {1, . . . , 9} to each pair σk , mk ∈ {−1, 0, 1}2 , given by k = 3 · mk + σk + 5.
(70)
Accordingly, for k ∈ {1, . . . , 9}, the map Uk : ΓZ → ΓZ defined by U3m+σ+5 τ := Sm Wσ τ,
τ ∈ ΓZ , m ∈ {−1, 0, 1}, σ ∈ {−1, 0, 1}.
(71)
We then define by uk : R2 → R2 the continuous piecewise-linear function given by uk (x, y) := smk (wσk (x, y)),
x, y ∈ R,
(72)
and we have by Lemma B.1 that uk (η(τ )) = η(Uk τ )
τ ∈ ΓZ , k ∈ {1, . . . , 9}.
(73)
We also let u : R2 → R18 be the continuous piecewise-linear function given by u(x, y) := (u1 (x, y), u2 (x, y), . . . , u9 (x, y)),
x, y ∈ R.
(74)
We now move to show that we can also implement the transition function δ and command function κ of M by continuous piecewise-linear functions. B.1.2
Transition function as a continuous piecewise-linear function
In this section, we show that the transition function δ and command function κ of M can be implemented by continuous piecewise-linear functions. Before proceeding, note that we can reformulate the definition of the command function κ as follows. We begin by fixing a convenient enumeration of the neural commands used by the machine. Definition B.1. Let M := (n, d, δ, κ) be a TMNU. We define f1 , . . . , fνM : Rd → Rd be an enumeration FM in increasing order according to the ordering <L over Ad,d ∪ Rd defined in (61), and let fM := (f1 , f2 , . . . , fνM ) : Rd → RνM d . Further, we define the function κ̃ : Q × Γ → {1, . . . , νM } by κ̃(q, σ) := i
⇐⇒ κ(q, σ) = fi ,
q ∈ Q, σ ∈ Γ.
Under the above reformulation of the command function, we can proceed to simulate the transition and command functions by continuous piecewise-linear functions. The next lemma shows that the discrete transition choices can be recovered by CPWL selectors. Lemma B.2. Let M := (n, d, δ, κ) be a TMNU, and γ ∈ {q, τ, ω}. Then, there exists continuous piecewise-linear functions ∆γ : Rn+2 → Rnγ , where nq := n, nτ := 9 and nω := νM , such that for every q ∈ {1, . . . , n} and τ ∈ ΓZ , we have ∆γ (1n,q , η(τ )) = 1nγ ,hγ (q,Rτ ) , where hq := δ q , hτ := δ τ as defined by (69) and hω := κ̃ as defined in Definition B.1.
22
(75)
Proof. We make the proof for γ = q only, as the remaining cases are exactly analogous. For every q ∈ {1, . . . , n} and σ ∈ Γ, we let xq,σ := (1n,q , σ) ∈ Rn+1 ,
yq,σ := 1n,δq (q,σ) ∈ Rn .
(76)
{(xq,σ , yq,σ ) : q ∈ {1, . . . , n}, σ ∈ Γ} ⊆ Rn+1 × Rn
(77)
Note that the set defined by
is finite (actually has at most 3n elements), so by linear interpolation, there exists a continuous piecewiselinear function fq : Rn+1 → Rn such that fq (xq,σ ) = yq,σ for every q ∈ {1, . . . , n} and σ ∈ Γ. Now, we can define ∆q : Rn+2 → Rn by ∆q (x, y) := fq (x, r(y)), where r : R2 → R is the continuous piecewise-linear function defined in Lemma B.1. Note that ∆q is continuous piecewise-linear as a composition of continuous piecewise-linear functions. Moreover, for every q ∈ {1, . . . , n} and τ ∈ ΓZ , we have (65)
∆q (1n,q , η(τ )) = fq (1n,q , r(η(τ ))) = fq (1n,q , Rτ ) = fq (xq,Rτ ) = yq,Rτ = 1n,δq (q,R(τ )) . This concludes the proof. We next combine the results of Lemmata B.1 and B.2 to design a continuous piecewise-linear function that implements M. B.1.3
Definition of a continuous piecewise-linear function that implements M
Before proceeding, we need the following Lemma, which shows that we can use continuous piecewise-linear functions to select some coordinates of a vector. This selection device will be used to choose the update prescribed by the current state and scanned symbol. Lemma B.3. Let n, d ∈ N. Then, there exists a continuous piecewise-linear function gn,d : Rn ×Rnd → Rd such that for every M > 0, ℓ ∈ {1, . . . , n} and z1 , . . . , zn ∈ [0, M ]d , we have gn,d (M (1n,ℓ − 1n ), (z1 , . . . , zn )) = zℓ .
(78)
Proof. Let n, d ∈ N. We first define gn,d : Rn × Rnd → Rd by gn,d (x, z1 , . . . , zn ) :=
n X
ρ(zi + (xi − 1)1d )
i=1
for every x ∈ Rn , z1 , . . . , zn ∈ Rd . Note that gn,d is continuous piecewise-linear as a sum of continuous piecewise-linear functions. Moreover, for every M > 0, ℓ ∈ {1, . . . , n} and z1 , . . . , zn ∈ [0, M ]d , we have gn,d (M (1n,ℓ − 1n ), z1 , . . . , zn )i =
n X
n (a) X
ρ(zi + M (1n,ℓ (i) − 1)1d ) =
i=1
i=1
(30) X zi 1n,ℓ (i)
n i=1
zi δℓ,i = zℓ
(79)
where (a) follows from Lemma 5.1. This concludes the proof. We can now exploit the results of Lemmata B.1, B.2 and B.3 to design a continuous piecewise-linear function that implements M. The theorem below assembles the three ingredients into a single CPWL transition map. Theorem B.4. Let M := (n, d, δ, κ) be a TMNU, and C > 0. Then, there exists a continuous piecewiselinear function F := FM,C : Rn+2+d → Rn+2+d such that F (γM (c)) = γM (M(c)),
c ∈ BM (C).
(80)
Proof. We will define three piecewise-linear functions Fq : Rn+2+d → Rn , Fτ : Rn+2+d → R2 and Fω : Rn+2+d → Rd such that for every configuration c := (q; τ ; ω) of M, we have Fq (γM (c)) = 1n,πq M(c) ,
Fτ (γM (c)) = η(πτ M(c)),
Fω,C (γM (c)) = πω M(c).
(81)
We then define F : Rn+2+d → Rn+2+d by F (x) := (Fq (x), Fτ (x), Fω,C (x)) for every x ∈ Rn+2+d . Note that F is continuous piecewise-linear as a combination of continuous piecewise-linear functions. Moreover, for every configuration c of M, we have F (γM (c)) = (Fq (γM (c)), Fτ (γM (c)), Fω,C (γM (c))) = (1n,πq M(c) , η(πτ M(c)), πω M(c)) = γM (πq M(c), πτ M(c), πω M(c)) = γM (M(c)). 23
We now move to the definition of the functions Fq , Fτ and Fω . (a) We let Fq (x, y, z) := ∆q (x, y) for every x ∈ Rn , y ∈ R2 and z ∈ Rd , where ∆q is as in Lemma B.2. Then, (75)
(a)
Fq (γM (c)) = ∆q (1n,q , η(τ )) = 1n,δq (q,Rτ ) = 1n,πq M(c) ,
(82)
where (a) is by Definition 4.2. (b) We here proceed to the definition of Fτ . We let Fτ (x, y, z) := g9,2 (∆τ (x, y), u(y)),
x ∈ R n , y ∈ R2 , z ∈ R d ,
(83)
where ∆τ is as in Lemma B.2, u is defined by (74) and g9,2 is as in Lemma B.3. Note that Fτ is continuous piecewise-linear as a combination of continuous piecewise-linear functions. Moreover, for every configuration c := (q; τ ; ω) of M, we have (a)
Fτ (γM (c)) = g9,2 (∆τ (1n,q , η(τ )), u(η(τ ))) = g9,2 (19,δτ (q,Rτ ) , u1 (η(τ )), . . . , u9 (η(τ ))) (b)
(73)
(69)
= uδτ (q,Rτ ) (η(τ )) = η(Uδτ (q,Rτ ) τ ) = η(U3δm (q,Rτ )+δσ (q,Rτ )+5 τ )
(71)
(c)
= η(Sδm (q,Rτ ) Wδσ (q,Rτ ) τ ) = η(πτ M(c)),
where (a) follows from Lemma (75), (b) is by Lemma B.3 and (c) is by Definition 4.2. (c) We here proceed to the definition of Fω,C . We let fM = (f1 , . . . , fνM ) and κ̃ be as in Definition B.1, and we define Fω,C (x, y, z) := gνM ,d (∥M∥C (∆κ (x, y) − 1νM ), ρ(fM (z))) − gνM ,d (∥M∥C (∆κ (x, y) − 1νM ), ρ(−fM (z))),
(84)
where gνM ,d is as in Lemma B.3, and ∆κ is as in Lemma B.2. Note that Fω,C is continuous piecewise-linear as a combination of continuous piecewise-linear functions. Now note that, for every configuration c ∈ BM (C), we have ∥ω∥ ≤ C
⇒
∥fM (ω)∥∞ ≤ ∥M∥C
ρ(fM (ω)), ρ(−fM (ω)) ∈ [0, ∥M∥C ]dνM .
⇒
(85)
Therefore, for c := (q, τ, ω) ∈ BM (C), we have (a)
gνM ,d (∥M∥C (∆ω (1n,q , η(τ )) − 1νM ), ρ(fM (ω))) = gνM ,d (∥M∥C (1νM ,κ̃(q,Rτ ) − 1νM ), ρ(fM (ω))) (b)
(c)
= ρ(fκ̃(q,Rτ ) (ω)) = ρ(κ(q, Rτ )(ω)),
where (a) follows from Lemma B.2, (b) is by Lemma B.3 and (85), and (c) is by Definition B.1, and similarly, we have gνM ,d (∥M∥C (∆ω (1n,q , η(τ )) − 1νM ), ρ(−fM (ω))) = ρ(−κ(q, Rτ )(ω)), so that (a)
Fω,C (γM (c)) = ρ(κ(q, Rτ )(ω)) − ρ(−κ(q, Rτ )(ω)) = κ(q, Rτ )(ω) = πω M(c), where (a) is by ρ(x) − ρ(−x) = x for every x ∈ R. This concludes the proof.
B.2
Construction of a neural network that simulates a TMNU
In Section B.1, we have established that given a TMNU M and C > 0, there exists a CPWL function FM,C such that FM,C (γM (c)) = γM (M(c)), c ∈ BM (C). Here, we construct explicitly a ReLU neural network that realizes this function FM,C . We first introduce useful notation for ReLU neural networks. Let Ψ := (N0 , N1 , . . . , NL ; A1 , b1 , A2 , b2 , . . . , AL , bL ) be a ReLU neural network as in (39). (a) For ℓ ∈ {1, . . . , L}, we let Ψℓ : RN0 → RNℓ be the function defined by Ψℓ (x) := Aℓ ρ Aℓ−1 ρ . . . ρ A1 x + b1 + bℓ−1 + bℓ for all x ∈ RN0 . 24
(86)
We call Ψℓ the ℓ-th ancestor subnetwork of Ψ. By convention, we let Ψ0 (x) = x for every x ∈ RN0 . (b) We let
di (Ψ) := N0 ,
do (Ψ) := NL .
(87)
(c) We say that Ψ is a left-selector if A1 is a left-selector matrix and b1 = 0, and we say that Ψ is a right-selector if AL is a right-selector matrix and bL = 0. This section is organized as follows. First, we establish some useful properties of ReLU neural networks. Then we reproduce the construction scheme from Section B.1 by designing explicit networks for the tape operations, transition and command functions, and neural-state update. B.2.1
Auxiliary results on ReLU neural networks
This part is devoted to the proof of some auxiliary results on ReLU neural networks that will be useful for the construction of a neural network that simulates a TMNU. We first need an identity network whose depth can be prescribed. Lemma B.5. Let n, d ∈ N. Then, there exists a left- and right-selector neural network In,d with W(In,d ) = n, L(In ) = d and Ω(In ) ⊆ {0, 1} such that In (x) = ρ(x) for all x ∈ Rn . Proof. For every x ∈ Rn , we have In,1 (x) = In x = x. Now, for d ∈ N, we define In,d := (n, . . . , n; In , 0, . . . , In , 0), where In is the n × n identity matrix and there are d layers. Note that indeed, L(In,d ) = d, W(In,d ) = n and Ω(In,d ) = {0, 1}. Now, let x ∈ Rn , and note that by definition of In , we have In,d (x) = ρ ◦ . . . ◦ ρ(x) = ρ(x). | {z } d−1 times
Note also that In is a left-selector matrix and a right-selector matrix, and 0 is the zero vector, so In,d is indeed a left- and right-selector neural network. This concludes the proof. The next lemma asserts that neural networks are closed under composition. Lemma B.6. Let Ψ1 , Ψ2 be two neural networks such that do (Ψ1 ) = di (Ψ2 ) and either Ψ1 is a rightselector or Ψ2 is a left-selector. Then, there exists a neural network Ψ such that L(Ψ) = L(Ψ1 )+L(Ψ2 )−1, W(Ψ) ≤ max{W(Ψ1 ), W(Ψ2 )}, Ω(Ψ) ⊆ Ω(Ψ1 ) ∪ Ω(Ψ2 ) and Ψ(x) = Ψ2 (Ψ1 (x)) for all x ∈ Rdi (Ψ1 ) . Proof. Suppose that (1)
(1)
(1)
(2)
(2)
(2)
L1 1 Ψ1 = (N0 , N1 , . . . , NL1 ; A1(1) , b1(1) , A2(1) , b2(1) , . . . , AL (1) , b(1) )
and
L2 2 Ψ2 = (N0 , N1 , . . . , NL2 ; A1(2) , b1(2) , A2(2) , b2(2) , . . . , AL (2) , b(2) ). (1)
(2)
Note that by assumption, we have NL1 = N0 . We define (1)
(1)
(1)
(2)
(2)
Ψ :=(N0 , N1 , . . . , NL1 −1 , N1 , . . . , NL2 ; L1 −1 L1 L1 L2 L2 1 1 1 2 2 1 −1 A1(1) , b1(1) , . . . , AL (1) , b(1) , A(2) A(1) , A(2) b(1) + b(2) , A(2) , b(2) , . . . , A(2) , b(2) ).
Note that indeed, L(Ψ) = L(Ψ1 ) + L(Ψ2 ) − 1, W(Ψ) ≤ max{W(Ψ1 ), W(Ψ2 )}. Moreover, if Ψ1 is L1 L1 1 1 1 a right-selector, then AL (1) is a right-selector matrix and b(1) = 0, so Ω(A(2) A(1) ) ⊆ Ω(A(2) ) and 1 1 1 1 1 Ω(A1(2) bL (1) + b(2) ) ⊆ Ω(b(2) ). If Ψ2 is a left-selector, then A(2) is a left-selector matrix and b(2) = 0, so L1 L1 L1 1 1 1 Ω(A1(2) AL (1) ) ⊆ Ω(A(1) ) and Ω(A(2) b(1) + b(2) ) ⊆ Ω(b(1) ). In either case, we have Ω(Ψ) ⊆ Ω(Ψ1 ) ∪ Ω(Ψ2 ).
Now, let x ∈ Rdi (Ψ1 ) , and note that by definition of Ψ, we have Ψ≤L1 (x) = A1(2) Ψ1 (x) + b1(2) , and therefore
Ψ(x) = Ψ2 (Ψ1 (x)).
This concludes the proof. We shall also use a parallelization device to run several networks on selected input coordinates.
25
Lemma B.7. Let m, d ∈ N, Ψ1 , Ψ2 , . . . , Ψm be m neural networks such that L(Ψ1 ) = . . . = L(Ψm ) and let ιi : {1, . . . , di (Ψi )} → {1,P . . . , d}, i ∈ {1, . . . , m}. SmThen, there exists a neural network Ψ such that m L(Ψ) = L(Ψ1 ), W(Ψ) = d ∨ i=1 W(Ψi ), Ω(Ψ) ⊆ i=1 Ω(Ψi ) ∪ {0} and Ψ(x) = (Ψ1 (πι1 x), Ψ2 (πι2 x), . . . , Ψm (πιm x)) for all x ∈ Rd . Proof. Suppose that (i)
(i)
(i)
L Ψi = (N0 , N1 , . . . , NL ; A1(i) , b1(i) , A2(i) , b2(i) , . . . , AL (i) , b(i) ),
for every i ∈ {1, . . . , m}. We define Ψ := (d, N1 , . . . , NL ; A1 , b1 , . . . , AL , bL ), ℓ ℓ A(1) 0 ... 0 b(1) 0 bℓ ℓ A . . . 0 (2) Pm (2) (i) ℓ where Nℓ = i=1 Nℓ for ℓ ∈ {1, . . . , L} and Aℓ = .. .. .. .. and b = .. for . . . . . 0 0 . . . Aℓ(m) bℓ(m) ℓ ∈ {2, . . . , L}, and 1 1 A(1) πι1 b(1) A1(2) πι2 b1(2) A1 = , b1 = . . .. . . . 1 1 A(m) πιm b(m) Pm Sm Note that indeed, L(Ψ) = L, W(Ψ) = d ∨ i=1 W(Ψi ) and Ω(Ψ) ⊆ i=1 Ω(Ψi ) ∪ {0}. Now, let x ∈ Rd , and note that by definition of Ψ, we have Ψ(x) = (Ψ1 (πι1 x), Ψ2 (πι2 x), . . . , Ψm (πιm x)). This concludes the proof. B.2.2
Neural networks for writing and shifting operations
This part is devoted to the construction of neural networks that realize the reading, writing and shifting operations of a TMNU. To this end, we construct a neural network that realizes the CPWL function that implements the reading operation. The following lemma gives an explicit ReLU realization of the one-symbol reader. Lemma B.8. Let f : R → R be the function defined by (64). Then, there exists a neural network Nf with W(Nf ) = 4, L(Nf ) = 2 and Ω(Nf ) = {−2, −1, 0, 1, 2, 3}, such that Nf (x) = f (x) for all x ∈ R. Proof. We define Nf : R → R by Nf := (1, 4, 1; A1f , b1f , A2f , b2f ), where 3 3 1 Af := 3 , 3
2 1 b1f := −1 , −2
A2f := 1
−1
1
−1 ,
b2f := −1.
Note that indeed, L(Nf ) = 2, W(Nf ) = 4 and Ω(Nf ) = {0, 1, 2, 3, −1, −2}. Now, let x ∈ R, and note that by definition of Nf , we have Nf (x) = ρ(3x + 2) − ρ(3x + 1) + ρ(3x − 1) − ρ(3x − 2) − 1 = f (x).
Once the reader is available, the combined write-and-shift candidates can be realized by a single shallow network. Lemma B.9. Let u : R2 → R18 be the function defined by (74). Then, there exists a neural network Ψ satisfying W(Ψ) = 18, L(Ψ) = 2, Ω(Ψ) ⊆ ±{0, 1/4, 1, 2, 3, 4} such that Ψ(x) = u(x) for all x ∈ R2 .
26
Proof. First, let 1 0 A1 := A1f 0
0 1 , 0 A1f
1 0 A2 := 0 0
0 0 b1 := b1f , b1f
0 1 0 0
0 0 A2f 0
0 0 , 0 A2f
0 0 b2 := b2f , b2f
where A1f , b1f , A2f , b2f are as in the proof of Lemma B.8. Note that for every x, y ∈ R+ , we have A2 ρ(A1 (x, y) + b1 ) + b2 = (x, y, f (x), f (y)),
(88)
where f is the function defined by (64). Now, consider, for σ ∈ {−1, 0, 1} and m ∈ {−1, 0, 1}, the functions fσ,−1 , fσ,0 , fσ,1 : R4 → R2 defined by 1 x1 + σ − x3 σ/4 0 − 14 1 fσ,−1 (x) := x4 + , 4(x2 − x4 ) = 4 x+ =: Aσ,−1 x + bσ,−1 , (89) 0 0 4 0 −4 4 fσ,0 (x) := (x1 + σ − x3 , x2 ) =
1 0
0 1
−1 0
4 0
0
and fσ,1 (x) := (4(x1 − x3 ), σ + x2 /4) =
1 4
0 σ x+ =: Aσ,0 x + bσ,0 , 0 0
(90)
−4 0 0 x+ =: Aσ,1 x + bσ,1 , σ 0 0
(91)
for every x ∈ R4 . Note that for every σ ∈ {−1, 0, 1}, m ∈ {−1, 0, 1} and x, y ∈ R, we have fσ,m (x, y, f (x), f (y)) = sm (wσ (x, y)) = u3m+σ+5 (x, y). Now, define
A−1,−1 A := ... ,
(92)
b−1,−1 b := ... b1,1
A1,1
and Ψ := (2, 4, 18; A1 , b1 , A × A2 , Ab2 + b). Note that indeed, L(Ψ) = 2, W(Ψ) = 18 and Ω(Ψ) ⊆ ±{0, 1/4, 1, 2, 3, 4}. Moreover, for every x, y ∈ R+ , we have Ψ(x, y) = A × A2 ρ(A1 (x, y) + b1 ) + Ab2 + b = A(A2 ρ(A1 (x, y) + b1 ) + b2 ) + b (88)
= A(x, y, f (x), f (y)) + b
(89),(90),(91)
=
(f−1,−1 (x, y, f (x), f (y)), . . . , f1,1 (x, y, f (x), f (y)))
(92)
= (u1 (x, y), . . . , u9 (x, y)) = u(x, y).
B.2.3
Simulation of the transition and command functions
We next build a network that encodes the currently scanned symbol together with the current state. Lemma B.10. Let n ∈ N. There exists a right-selector neural network Ψ with W(Ψ) = 3n, L(Ψ) = 3 and Ω(Ψ) ⊆ ±{0, 1, 2, 3} such that Ψ(1n,q , η(τ )) = 13n,3(q−1)+Rτ +2 , for every q ∈ {1, . . . , n} and τ ∈ ΓZ . Proof. First, let use consider −1 A2 := 1 0
1 −1 0
0 −1 1
0 1 , −1
1 b2 := 0 . 0
Let g : R → R3 be the function defined by g(x) := A2 ρ(A1f x + b1f ) + b2 , where A1f and b1f are as in the proof of Lemma B.8. Let f : R → R be the function defined by (64). 1. For x ≤ −2/3, we have g(x) = (1, 0, 0) = 13,1 = 13,2+f (x) . 2. For −1/3 < x < −1/3, we have g(x) = (0, 1, 0) = 13,2 = 13,2+f (x) . 3. For x ≥ 2/3, we have g(x) = (0, 0, 1) = 13,3 = 13,2+f (x) . 27
Therefore, in particular,
g(η̃(τ + )) = 13,2+f (η̃(τ + )) = 13,2+Rτ , In 0 0 1 Z (n+4)×(n+1) 1 for every τ ∈ Γ . Now, let à := , b̃ := 1 ∈ Rn+4 , Ã2 ∈ R3n×(n+4) be ∈R 0 A1f bf defined by 2 Ã1 b − 13 T 1n,ℓ b2 − 13 Ã2 A2 ∈ R3×(n+4) , and b̃2 = . ∈ R3n . Ã2 := . where Ãℓ := 1Tn,ℓ .. .. T 1n,ℓ b2 − 1 3 Ãn Define the neural network Ψ := (n + 1, n + 4, 3n, 3n; Ã1 , b̃1 , Ã2 , b̃2 , I3n , 03n ). Note that indeed, Ψ is a right-selector, L(Ψ) = 3, W(Ψ) = 3n and Ω(Ψ) ⊆ ±{0, 1, 2, 3}. Moreover, for every q, ℓ ∈ {1, . . . , n}, τ ∈ ΓZ , we have T 1n,ℓ ρ Ãℓ ρ(Ã1 (1n,q , η̃(τ + )) + b̃1 ) + b̃2 − 13 = ρ 1Tn,ℓ ρ(In 1n,q + 0) + A2 ρ(A1f 1n,q η(τ ) + b1f ) + b2 − 13 1Tn,ℓ T 1n,ℓ = ρ 1Tn,ℓ 1n,q + g(η̃(τ + )) − 13 1Tn,ℓ = ρ (δq,ℓ 13 + 13,2+Rτ − 13 ) = δq,ℓ 13,2+Rτ .
Therefore, for every q ∈ {1, . . . , n} and τ ∈ ΓZ , we have Ψ(1n,q , η̃(τ + )) = Ã2 ρ(Ã1 (1n,q , η̃(τ + )) + b̃1 ) + b̃2 = (δq,1 13,2+Rτ , . . . , δq,n 13,2+Rτ ) = 13n,3(q−1)+2+Rτ . Finally, by Lemma B.7 applied to m = 1, d = n + 2, Ψ1 = Ψ and ι1 : {1, . . . , n + 1} → {1, . . . , n + 2}, i 7→ i, there exists a neural network Ψ̃ with W(Ψ̃) = 3n, L(Ψ̃) = 3 and Ω(Ψ̃) = Ω(Ψ) ∪ {0} = ±{0, 1, 2, 3} such that Ψ̃(1n,q , η(τ )) = Ψ(πι (1n,q , η(τ ))) = Ψ(1n,q , η̃(τ + )) = 13n,3(q−1)+2+Rτ , for every q ∈ {1, . . . , n} and τ ∈ ΓZ . This concludes the proof. Given a set S ⊆ A, we denote by χS : A → {0, 1} the characteristic function of S, defined by χS (a) := 1 if a ∈ S and χS (a) := 0 otherwise. The pattern network allows us to realize the state, tape, and command selectors by ordinary affine layers. Lemma B.11. Let M := (n, k, d, δ, κ) be a TMNU, M > 0, and let the functions ∆q : Rn+1 → Rn , ∆τ : Rn+1 → R9 , and ∆κ : Rn+1 → RνM be defined as in Lemma B.2. Then, there exist a neural network ΨM such that W(ΨM ) ≤ max{3n, n + νM + 9}, L(ΨM ) = 3 and Ω(ΨM ) ⊆ ±{0, 1, 2, 3, M } and ∆q (1n,q , τ ) ΨM (1n,q , η(τ )) = ∆τ (1n,q , τ ) − 19 , M (∆κ (1n,q , τ ) − 1νM ) for every q ∈ {1, . . . , n} and τ ∈ ΓZ . Proof. Let M := (n, k, d, δ, κ) be a TMNU, and M > 0. Given A, B two sets and a function f : A → B, we denote by graph(f ) := {(a, b) ∈ A × B : f (a) = b} the graph of f . We define the matrices W M,q ∈ Rn×3n , W M,τ ∈ R9×3n , and W M,κ ∈ RνM ×3n by M,q ′ W3q+σ−1,q ′ = χgraph(δ q ) ((q, σ), q ),
M,τ W3q+σ−1,k = χgraph(δτ ) ((q, σ), k),
M,κ W3q+σ−1,ℓ = χgraph(κ̃) ((q, σ), ℓ),
for all q, q ′ ∈ {1, . . . , n}, σ ∈ {−1, 0, 1}, k ∈ {1, . . . , 9} and ℓ ∈ {1, . . . , νM }. Note that, for instance, W M,q 13n,3q+σ−1 = 1n,δq (q,σ) ,
q ∈ {1, . . . , n},
σ ∈ Γ,
and the same holds for W M,τ and W M,κ with respect to δ τ and κ̃, respectively. Define the neural network W M,q 0n ΨM := 3n, n; W M,τ , −19 . M W M,κ −M 1νM 28
Note that L(ΨM ) = 1, W(ΨM ) = max{3n, n + νM + 9} and Ω(ΨM ) ⊆ ±{0, 1, M }. Moreover, for every q ∈ {1, . . . , n} and σ ∈ Γ, we have M,q 1n,δq (q,σ) W 0n 19,δτ (q,σ) − 19 . ΨM (13n,3q+σ−1 ) = W M,τ 13n,3q+σ−1 + −19 = W M,κ −M 1νM M 1νM ,κ̃(q,σ) − 1νM By Lemma B.10, there exists a right-selector neural network Ψ′ such that L(Ψ′ ) = 3, W(Ψ′ ) = 3n and Ω(Ψ′ ) ⊆ ±{0, 1, 2, 3}, and Ψ′ (1n,q , η(τ )) = 13n,3q+Rτ −1 . Hence, by Lemma B.6 applied to Ψ′ and ΨM , there exists a neural network Ψ̃M such that L(Ψ̃M ) = 3, W(Ψ̃M ) ≤ max{3n, n + 9 + νM } and Ω(Ψ̃M ) ⊆ ±{0, 1, 2, 3, M }, such that 1n,δq (q,Rτ ) ∆q (1n,q , τ ) 19,δτ (q,Rτ ) − 19 = ∆τ (1n,q , τ ) − 19 . Ψ̃M (1n,q , η(τ )) = Ψ(Ψ′ (1n,q , η(τ ))) = M (∆κ (1n,q , τ ) − 1νM ) M 1νM ,κ̃(q,Rτ ) − 1νM This concludes the proof. B.2.4
Neural state management
This part builds a ReLU neural network that computes all possible outputs of the command function κ of a TMNU M on a given neural state ω. The following lemma packages all candidate neural-state updates into one network output. Lemma B.12. Let M := (n, d, δ, κ) be a TMNU, and fM = (f1 , . . . , fνM ) as in Definition B.1. Then, there exists a neural network Ψ satisfying L(Ψ) = 2, W(Ψ) = 2νM d, Ω(Ψ) ⊆ ±{0, 1} ∪ ±Ω(M) and Ψ(ω) = ρ (fM (ω), −fM (ω)) , for every ω ∈ Rd . Proof. Let M := (n, d, δ, κ) be a TMNU, and consider f1 , . . . , fνM to be the elements of FM ordered as f1 <L f2 <L · · · <L fνM . Let i ∈ {1, . . . , νM }. If fi is an affine function, then there exist Ai ∈ Rd×d and bi ∈ Rd such that fi (ω) = Ai ω + bi for every ω ∈ Rd . Therefore, we have ρ(fi (ω)) = ρ(Ai ω + bi )
and
ρ(−fi (ω)) = ρ(−Ai ω − bi ) =: ρ(A′i ω + b′i ),
for every ω ∈ Rd . If fi is a function of the form fi = ρSi for some Si ⊆ {1, . . . , d}, then we have ρ(fi (ω)) = ρ(ρSi (ω)) = ρ(ω) =: ρ(Ai ω + bi ) and
and
ρ(−fi (ω)) = ρ(−ρSi (ω)) =: (z1 , . . . , zd ),
where zℓ = 0 = ρ(0 · ω + 0) if ℓ ∈ Si and zℓ = ρ(−ωℓ ) = ρ(−1 · ωℓ + 0) if ℓ ∈ / Si , for every ℓ ∈ {1, . . . , d}. Accordingly, we define A′i ∈ {−1, 0}d×d and b′i := 0d such that ρ(−fi (ω)) = ρ(A′i ω + b′i ) for every ω ∈ Rd . Hence, by defining the matrices ′ ′ A1 A1 b1 b1 .. .. .. .. ′ ′ A := . , A := . , b := . , and b := . , A′νM
Aν M and the neural network Ψ :=
bνM
b′νM
A b 2d, 2νM d; ′ , ′ , A b
we have L(Ψ) = 2, W(Ψ) = 2νM d, Ω(Ψ) ⊆ ±{0, 1} ∪ ±Ω(M) and Ψ(ω) = ρ (fM (ω), −fM (ω)) , for every ω ∈ Rd . This concludes the proof. B.2.5
Finalization of the proof
The final selector needed in the network construction is itself realized by a small ReLU network.
29
Lemma B.13. Let n, d ∈ N, and let gn,d : Rn(d+1) → Rd be as in Lemma B.3. Then, there exists two leftseparator neural networks Ψ1 and Ψ2 satisfying L(Ψ1 ) = L(Ψ2 ) = 2, W(Ψ1 ) = n(d+1), W(Ψ2 ) = n(2d+1), Ω(Ψ1 ), Ω(Ψ2 ) ⊆ ±{0, 1}, such that Ψ1 (x, z) = gn,d (x, z) and Ψ(x, z, z ′ ) = gn,d (x, z) − gn,d (x, z ′ ) for every x ∈ Rn+ and z, z ′ ∈ Rnd . Proof. The Lemma follows directly from the definition of gn,d as gn,d (x, z1 , . . . , zd ) :=
n X
ρ(zi + xi 1d ),
x ∈ Rn , z1 , . . . , zn ∈ Rd ,
(93)
i=1
as there exists two matrices A1 ∈ {0, 1}nd×n(d+1) and A2 ∈ {0, 1}d×nd such that gn,d (x, z) = A1 ρ(A2 (x, z)) for every x ∈ Rn and z ∈ Rnd , and two matrices A′1 ∈ {0, 1}nd×n(2d+1) and A′2 ∈ {−1, 0, 1}d×n(2d+1) such that gn,d (x, z) − gn,d (x, z ′ ) = A′1 ρ(A′2 (x, z, z ′ )) for every x ∈ Rn and z, z ′ ∈ Rnd . We can now combine the tape, command, and selection subnetworks into a neural network for one full TMNU step. Theorem B.14. Let M := (n, d, δ, κ) be a TMNU and C > 0. Then, there exists a ReLU neural network Ψ ∈ Nn+2+d,n+2+d such that L(Ψ) = 4, W(Ψ) ≤ max{3n, n + νM + 9} + 2dνM + 18,
Ω(Ψ) ⊆ ±{0, 1/4, 1, 2, 3, 4, ∥M∥C } ∪ ±Ω(M)
and Ψ(γM (c)) = γM (M (c)),
c ∈ BM (C).
Proof. Let M := ∥M∥C , ΨM be the neural network given by Lemma B.11, Ψ1 be the neural network given by Lemma B.9, and Ψ2 be the neural network given by Lemma B.12. Consider also the neural networks I18,2 and I2dνM ,2 given by Lemma B.5, and note that they are left- and right-selectors. Therefore, by Lemma B.6 applied to Ψ1 and I18,2 , there exists a neural network Ψ̃1 such that L(Ψ̃1 ) = 3, W(Ψ̃1 ) = 18 and Ω(Ψ̃1 ) ⊆ ±{0, 1/4, 1, 2, 3, 4}, and Ψ̃1 (η(τ )) = I18,2 (Ψ1 (η(τ ))) = ρ(u(η(τ ))) = u(η(τ )),
τ ∈ ΓZ .
Moreover, by Lemma B.6 applied to Ψ2 and I2dνM ,2 , there exists a neural network Ψ̃2 such that L(Ψ̃2 ) = 3, W(Ψ̃2 ) = 2dνM and Ω(Ψ̃2 ) ⊆ ±{0, 1} ∪ ±Ω(M), and Ψ̃2 (x) = I2dνM ,2 (Ψ2 (x)) = ρ (ρ (fM (x), −fM (x))) = ρ (fM (x), −fM (x)) , for every x ∈ R2dνM . Now, by Lemma B.7 applied to d := n + 2 + d, m = 3, Ψ1 = ΨM , Ψ2 = Ψ̃1 , Ψ3 = Ψ̃2 and ι1 : {1, . . . , n + 2} → {1, . . . , n + 2 + d}, i 7→ i, ι2 : {1, 2} → {1, . . . , n + 2 + d}, i 7→ i + n and ι3 : {1, . . . , d} → {1, . . . , n + 2 + d}, i 7→ i + n + 2, there exists a neural network Ψ̃ satisfying L(Ψ̃) = 3, W(Ψ̃) ≤ max{3n, n + νM + 9} + 2dνM + 18, and Ω(Ψ̃) ⊆ ±{0, 1/4, 1, 2, 3, 4, M } ∪ ±Ω(M), such that Ψ̃(x) = (ΨM (πι1 (x)), Ψ̃1 (πι2 (x)), Ψ̃2 (πι3 (x))) = (ΨM (x1:n+2 ), Ψ̃1 (xn+1:n+2 ), Ψ̃2 (xn+3:n+2+d )), for every x ∈ Rn+2+d . In particular, for x = γM (c) ∈ Rn+2+d for some c ∈ CM , we have ∆q (1n,q , τ ) ∆τ (1n,q , τ ) − 19 M (∆κ (1n,q , τ ) − 1νM ) . Ψ̃(γM (c)) = (ΨM (1n,q , η(τ )), Ψ̃1 (η(τ )), Ψ̃2 (ω)) = u(η(τ )) ρ(fM (ω)) ρ(−fM (ω))
(94)
Now, let Ψ′1 and Ψ′2 be the left-separator neural networks given by Lemma B.13 applied to n = 18 and d = n + 2 + d, respectively. Consider also the neural network In,2 given by Lemma B.5, and note that it is a left-separator. By Lemma B.7 applied to d := n + 9 + νM + 18 + 2dνM , m = 3, Ψ1 = In,2 , Ψ1 = Ψ+ , Ψ2 = Ψ, ι1 : {1, . . . , n} → {1, . . . , n + 9 + νM + 18 + 2dνM }, i 7→ i, ( i+n if i ∈ {1, . . . , 9}, ι2 : {1, . . . , 27} → {1, . . . , n + 9 + νM + 18 + 2dνM }, i 7→ i + n + νM if i ∈ {10, . . . , 27},
30
and ( ι3 : {1, . . . , (2d + 1)νM } → {1, . . . , n + 9 + νM + 18 + 2dνM }, i 7→
i+n+9 i + n + 27 + νM
if i ≤ νM , if i > νM ,
there exists a left-selector neural network Ψ satisfying L(Ψ′ ) = 2, W(Ψ′ ) = n + 27 + (2d + 1)νM and Ω(Ψ′ ) ⊆ ±{0, 1} ∪ ±Ω(M), such that Ψ′ (x) = (In,2 (πι1 (x)), Ψ′1 (πι2 (x)), Ψ′2 (πι3 (x))) for every x ∈ Rn+27+(2d+1)νM . Now, by Lemma B.6 applied to Ψ̃ and Ψ′ , there exists a neural network Ψ̂ such that L(Ψ̂) = 4, W(Ψ̂) = max{3n, n+νM +9}+2dνM +18 and Ω(Ψ̂) ⊆ ±{0, 1/4, 1, 2, 3, 4, M }∪±Ω(M), such that Ψ̂(x) = Ψ′ (Ψ̃(x)) = (ρ(Ψ̃(x)1:n ), Ψ′1 (πι2 (Ψ̃(x))), Ψ′2 (πι2 (Ψ̃(x)))), for every x ∈ Rn+2+d . In particular, for x = γM (c) for some c ∈ CM , we have ρ(Ψ̃(x)1:n ) = ρ(∆q (1n,q , τ )) = Fq (γM (c)), Ψ′1 (πι2 (Ψ̃(x))) = g9,2 (∆τ (1n,q , τ ) − 19 , u(η(τ ))) = Fτ (γM (c)), and Ψ′2 (πι3 (Ψ̃(x))) = gνM ,d (M (∆κ (1n,q , τ ) − 1νM ), ρ(fM (ω))) − gνM ,d (M (∆κ (1n,q , τ ) − 1νM ), ρ(−fM (ω))) = Fω,C (γM (c)), where Fq , Fτ and Fω,C are as the proof of Theorem B.4. Hence, for every c ∈ BM (C), we have (a)
Ψ̂(γM (c))(Fq (γM (c)), Fτ (γM (c)), Fω,C (γM (c))) = FM (γM (c)) = γM (M (c)), where (a) follows from the proof of Theorem B.4. This concludes the proof
B.3
Construction of an RNN that simulates a TMNU
In this appendix, we formally show that we can simulate the iteration of a ReLU neural network by an RNN. We introduce the following notation and terminology for RNNs. Let R := (d, m, d′ ; Ah , bh , Ax , Ao , bo ) be an RNN. (a) We define
Ω(R) := Ω(Ax ) ∪ Ω(Ah ) ∪ Ω(Ao ) ∪ Ω(bh ) ∪ Ω(bo ).
(95)
(b) We say that R is a left-selector RNN if Ax is a left-selector matrix and bh = 0, and that R is a right-selector RNN if Ao is a right-selector matrix and bo = 0. B.3.1
Technical Lemmata for RNNs
In this part, we prove two useful lemmata that allow us to construct RNNs that perform linear transformations of the input and output of a given RNN. We first record that a selector can be appended to the output without changing the hidden dynamics. ′
Lemma B.15. Let n ∈ N, R be an RNN, and A ∈ Rn×d be a left-selector matrix. Then, there exists an RNN R̃ satisfying m(R̃) = m(R) and Ω(R̃) ⊆ Ω(R) ∪ {0} such that for every x ∈ Rn and t ∈ N, we have R̃Dx[t] = A(RDx[t]). Proof. Let R := (d, m, d′ ; Ah , bh , Ax , Ao , bo ). We define R̃ := (d, m, n; Ah , bh , Ax , Ão , b̃o ), where Ão := AAo and b̃o := Abo . Note that, indeed, we have m(R̃) = m = m(R) and Ω(R̃) ⊆ Ω(R) ∪ {0}. Now, let x ∈ Rn and t ∈ N. Note that R̃Dx[t] = Ão (HDx[t]) + b̃o = AAo (HDx[t]) + Abo = A(RDx[t]). This concludes the proof.
31
The companion input transformation is slightly more delicate, because it must be absorbed into the recurrent initialization. Lemma B.16. Let n ∈ N, R be a left-selector RNN, A ∈ Rd×n , and b ∈ Rd . Then, there exists an RNN R̃ satisfying m(R̃) = m(R) + 1 and Ω(R̃) = Ω(R) ∪ Ω(A) ∪ ±Ω(b), such that for every x ∈ Rn and t ∈ N, we have R̃Dx[t] = RD(Ax + b)[t]. Proof. Let (d, m, d′ ; Ah , bh , Ax , Ao , bo ) := R. We define R̃ := (n, m + 1, d′ ; Ãh , b̃h , Ãx , Ão , bo ) by Ah −Ax b b + Ax b Ax A Ãh := ∈ Rm+1×m+1 , b̃h := h , Ãx := , and Ão := (Ao , 0). 0 0 1 0 Note that, indeed, m(R̃) = m + 1 = m(R) + 1, and that since R is a left-selector, we have {0, 1} = Ω(Ax ), Ω(−Ax b) ⊆ Ω(b), Ω(Ax A) ⊆ Ω(A), and Ω(bh + Ax b) ⊆ −Ω(b), so that Ω(R̃) = Ω(R) ∪ Ω(A) ∪ ±Ω(b). Now, let x ∈ Rn and define ht := H̃Dx[t]1:m and dt := H̃Dx[t]m+1 for every t ∈ N0 ∪ −1, where H̃ is the hidden state operator of R̃. In particular, note that h−1 = 0, d−1 = 0. Then, note that ( ht = ρ(Ãh (ht−1 , dt−1 ) + b̃h + Ãx Dx[t]) = ρ(Ah ht−1 − Ax bdt−1 + bh + Ax b + Ax ADx[t]) (96) dt = ρ(0 · dt−1 + 1) = 1 for every t ∈ N0 . Therefore, for every t ∈ N0 , we have ( ρ(Ah ht−1 + bh + Ax b + Ax ADx[t]), ht = ρ(Ah ht−1 + bh + Ax ADx[t]),
if t = 0 if t > 0
= ρ(Ah ht−1 + bh + Ax D(Ax + b)[t]) = HD(Ax + b)[t],
(97) (98)
where (a) follows from the fact that dt−1 = 1 for every t ∈ N. Finally, for every t ∈ N, we have R̃Dx[t] = Ão (ht , dt ) + bo = Ao ht + bo = Ao HD(Ax + b)[t] + bo = RD(Ax + b)[t]. This concludes the proof. B.3.2
Simulation of the iterations of a neural network by an RNN
This part is very technical, and is separated in three steps: given a ReLU neural network Ψ of depth L, we first show that we can design an RNN that outputs the positive part of the iterates of Ψ at time steps that are multiples of L, and outputs zero at other time steps. Then, we show how to modify such an RNN to output the positive part of the iterates of Ψ at every time step. Finally, we show how to modify such an RNN to output the iterates of Ψ. The next lemma corresponds to the first step of this construction. The first construction stores the layers of one network evaluation across L recurrent steps. Lemma B.17. Let Ψ ∈ Nn,n be a ReLU neural network, and let L := L(Ψ). Then, there exists m ∈ N satisfying n + L ≤ m ≤ (W(Ψ) + 1)L and AΨ ∈ Rm×m such that Ω(AΨ ) = Ω(Ψ) ∪ {0, 1} and for every x ∈ Rn , the sequence (ht ∈ Rm )t∈N defined by h0 := (x, 0, 1L,1 ),
ht+1 := ρ(AΨ ht ) for every t ∈ N0
(99)
satisfies π1:n ht = (ρ ◦ Ψ)t//L (x)δt mod L,0 ,
t ∈ N0 ,
(100)
where // denotes the integer division. Proof. Let n ∈ N, Ψ := (N0 , N1 , . . . , NL ; A1 , b1 , A2 , b2 , . . . , AL , bL ) be a ReLU neural network, such that PL−1 N0 = NL = n andΨ ≥ 0. We let N := n + ℓ=1 Nℓ and m := N + L, and we define the matrices 0 0 ... 0 AL 0 0 ... 0 bL A1 0 . . . b1 0 . . . 0 0 0 0 0 A2 . . . 0 b2 . . . N ×N 0 0 0 0 , B := A := ∈ RN ×L , ∈R .. .. . . . . . . . . .. .. .. .. .. .. .. . .. . 0 0 . . . AL−1 0 0 0 . . . bL−1 0
32
0 1 C := 0 .. . 0
0 0 1 .. .
0
... ... ... .. . ...
0 0 0 .. .
1
1 0 0 ∈ RL×L , .. . 0
and
AΨ :=
A 0
B ∈ Rm×m . C
Note that, indeed, we have n+L≤m=N +L=n+
L−1 X
Nℓ + L ≤ LW(Ψ) + L = (W(Ψ) + 1)L,
ℓ=1
and that Ω(AΨ ) = Ω(Ψ) ∪ {0, 1}. Now, let x ∈ Rn and let (hx,t ∈ Rm )t∈N be defined by hx,0 := (x, 0, 1L,1 ) and hx,t+1 := ρ(AΨ hx,t ) for every t ∈ N0 . We split the sequence (hx,t )t∈N into subsequences (hℓx,t ∈ RNℓ )t∈N and (cℓt ∈ R)t∈N for ℓ ∈ {0, . . . , L − 1} such that L−1 0 hx,t = (h0x,t , h1x,t , . . . , hL−1 ) for every t ∈ N. x,t , ct , . . . , ct
Note that, in particular, h0x,0 = x, and that
h1x,0 = 0,
...,
hL−1 x,0 = 0,
c00 = 1,
L L−1 h0x,t+1 = ρ(AL hL−1 ), x,t + b ct
and
c0t+1 = ρ(cL−1 ), t
c10 = 0,
...,
cL−1 = 0, 0
ℓ ℓ−1 hℓx,t+1 = ρ(Aℓ hℓ−1 ), x,t + b ct (ℓ−1)
cℓt+1 = ρ(ct
),
(101) (102) (103)
for every ℓ ∈ {1, . . . , L − 1} and t ∈ N0 . We divide the proof into several claims. Claim. For every t ∈ N, ℓ ∈ {0, . . . , L − 1}, we have cℓt = δℓ,t mod L . Proof of the claim. We proceed by induction on t. For the base case, note that for every ℓ ∈ {0, . . . , L − 1}, we have cℓ0 = 1L,1 (ℓ) = δℓ,0 . Now, let t ∈ N and assume that for every ℓ ∈ {0, . . . , L − 1}, we have cℓt = δℓ,t mod L . Then, by (103), we have c0t+1 = ρ(cL−1 ) = ρ(δL−1,t mod L ) = δ0,(t+1) mod L , and for t (ℓ−1)
every ℓ ∈ {1, . . . , L − 1}, we have cℓt+1 = ρ(ct proof of the claim.
) = ρ(δℓ−1,t mod L ) = δℓ,(t+1) mod L . This concludes the
Claim. For every t ∈ {0, . . . , L − 1}, we have h0x,t = xδt,0 and hℓx,t = ρ(Ψℓ (x))δt,ℓ for every ℓ ∈ {1, . . . , L − 1}. Proof of the claim. We proceed by induction on t. For the base case, note that by (101), we have hℓx,0 = xδ0,ℓ = ρ(Ψ0 (x))δ0,ℓ , for every ℓ ∈ {0, . . . , L − 1}. Now, let t ∈ {0, . . . , L − 2} and assume that for every ℓ ∈ {0, . . . , L − 1}, we have hℓx,t = Ψℓ (x)δt,ℓ . Then, by (102), we have L L−1 h0x,t+1 = ρ(AL hL−1 ) = ρ(AL ΨL−1 (x)δt,L−1 + bL δt,L−1 ) x,t + b ct
= ρ((AL ΨL−1 (x) + bL )δt,L−1 ) = ρ(AL ΨL−1 (x) + bL )δt,L−1 = 0 = xδt+1,0 , and for every ℓ ∈ {1, . . . , L − 1}, we have ℓ ℓ−1 hℓx,t+1 = ρ(Aℓ hℓ−1 ) = ρ(Aℓ Ψℓ−1 (x)δt,ℓ−1 + bℓ δt,ℓ−1 ) x,t + b ct
= ρ((Aℓ Ψℓ−1 (x) + bℓ )δt,ℓ−1 ) = ρ(Aℓ Ψℓ−1 (x) + bℓ )δt,ℓ−1 = ρ(Ψℓ (x))δt+1,ℓ . This concludes the proof of the claim. Claim. For every k ∈ N, we have hx,kL = ((ρ ◦ Ψ)k (x), 0, 1L,1 ). Proof of the claim. First note that (a)
and
L L−1 L L h0x,L = ρ(AL hL−1 x,L−1 + b cL−1 ) = ρ(A ΨL−1 (x) + b ) = ρ(ΨL (x)) = ρ(Ψ(x)),
(104)
(a)
(105)
ℓ ℓ−1 ℓ ℓ hℓx,L = ρ(Aℓ hℓ−1 x,L−1 + b cL−1 ) = ρ(A Ψℓ−1 (x) + b )δL,ℓ = 0,
33
for every ℓ ∈ {1, . . . , L − 1}, where (a) follows from the preceding two claims. Therefore, L−1 0 hx,L = (h0x,L , h1x,L , . . . , hL−1 ) = (ρ ◦ Ψ(x), 0, 1L,1 ). x,L , cL , . . . , cL
(106)
We now proceed by induction on k. For the base case, note that h0 = (x, 0, 1L,1 ) = ((ρ ◦ Ψ)0 (x), 0, 1L,1 ). Now, let k ∈ N and assume that hx,kL = ((ρ ◦ Ψ)k (x), 0, 1L,1 ). Therefore, we have hx,(k+1)L = h(ρ◦Ψ)k (x),L = ((ρ ◦ Ψ)((ρ ◦ Ψ)k (x)), 0, 1L,1 ) = ((ρ ◦ Ψ)k+1 (x), 0, 1L,1 ). This concludes the proof of the claim. Claim. For every k ∈ N, r ∈ {1, . . . , L − 1}, we have h0x,kL+r = (ρ ◦ Ψ)k+1 (x)δr,0 . Proof of the claim. For r = 0, the preceding claim implies that for every k ∈ N, we have h0x,kL = (ρ ◦ Ψ)k (x) = (ρ ◦ Ψ)k+1 (x)δ0,0 . Now, suppose r ∈ {1, . . . , L − 1}. Then, by the preceding claim, h0x,kL+r = h0(ρ◦Ψ)k+1 (x),r = (ρ ◦ Ψ)0 ((ρ ◦ Ψ)k+1 (x))δr,0 = 0 = (ρ ◦ Ψ)k+1 (x)δr,0 .
(107)
Now, let t ∈ N0 . Note that t = (t//L)L + (t mod L), so that by the preceding claim, we have π1:n hx,t = h0x,t = h0x,(t//L)L+(t mod L) = (ρ ◦ Ψ)t//L (x)δt mod L,0 .
(108)
This concludes the proof. We now show how to modify the RNN constructed in Lemma B.17 to output the positive part of the iterates of Ψ at every time step, and not only at time steps that are multiples of L. The next lemma corresponds to this second step of the construction. The following stabilization step keeps the last completed iterate available between two multiples of the depth. Lemma B.18. Let Ψ ∈ Nn,n be a ReLU neural network, and let L := L(Ψ). Then, there exists m̃ ∈ N and ÃΨ ∈ Rm̃×m̃ satisfying (L + 2)n + L ≤ m̃ ≤ (2L + 2)(W(Ψ) + 1) and Ω(ÃΨ ) = Ω(Ψ) ∪ {−1, 0, 1}, such that for every x ∈ Rn , the sequence (ht ∈ Rm̃ )t∈N defined by h0 := (0n(L+1) , x, 0, 1L,1 ),
ht+1 := ρ(ÃΨ ht ) for every t ∈ N0
(109)
satisfies π1:n ht = (ρ ◦ Ψ)(t−1)//L (x),
(110)
t ∈ N,
where // denotes the integer division. Proof. Let m ∈ N and AΨ ∈ Rm×m be as in Lemma 5.3. We define In 0 In 0 −In 0 In 0 0 0 ∈ Rm̃×m̃ . ÃΨ := 0 In(L−1) 0 0 0 0 0 AΨ Note that, indeed, we have (L + 2)n + L ≤ (L + 1)n + m = m̃ ≤ (L + 1)n + (W(Ψ) + 1)L ≤ (L + 2)(W(Ψ) + 1), and that Ω(ÃΨ ) = Ω(AΨ ) ∪ {−1, 0, 1} = Ω(Ψ) ∪ {−1, 0, 1}. Now, let x ∈ Rn and let (ht ∈ Rm )t∈N0 be defined as in (109). We split the sequence (ht )t∈N into subsequences (h∗t ∈ Rn )t∈N0 , (hℓt ∈ Rn )t∈N0 for ℓ ∈ {0, . . . , L − 1}, and (h̃t ∈ Rm )t∈N0 such that ht = (h∗t , h0t , . . . , hL−1 , h̃t ) for every t ∈ N0 . t Note that, in particular, h∗0 = 0,
h00 = 0,
h10 = 0,
...,
hL−1 = 0, 0
h̃0 = (x, 0, 1L,1 ),
(111)
and that h∗t+1 = ρ(h∗t − h0t + π1:n h̃t ),
h0t+1 = ρ(π1:n h̃t ),
34
hℓt+1 = ρ(hℓ−1 ), t
and h̃t+1 = ρ(AΨ h̃t ),
(112)
for every t ∈ N0 and ℓ ∈ {1, . . . , L − 1}. In particular, by Lemma B.17, we have π1:n h̃t = (ρ ◦ Ψ)t//L (x)δt mod L,0 ≥ 0 for every t ∈ N.
(113)
We divide the proof into several claims. Claim. For every ℓ ∈ {0, . . . , L − 1}, we have ( π1:n h̃t−ℓ−1 ℓ ht = 0
if t > ℓ, if t ≤ ℓ.
Proof. We first treat the case ℓ = 0. Note that by (111), we have h00 = 0, and that by (112), we have h0t = ρ(π1:n h̃t−1 ) = π1:n h̃t−1 ,
(114)
for every t > 0, where the second equality follows π1:n h̃t−1 ≤ 0, which is given by (113). Now, ℓ ∈ {1, . . . , L − 1}. First, let t ∈ {0, . . . , ℓ}. Then, by (112) applied t times, we have (a)
ℓ−t hℓt = hℓ−t = 0, t−t = h0
where (a) follows from (111). Now, let t > ℓ. Then, by (112) applied ℓ times, we have (a)
0 hℓt = hℓ−ℓ t−ℓ = ht−ℓ = π1:n h̃t−ℓ−1
where (a) follows from (114). This concludes the proof of the claim. Claim.
(P
t−1
h∗t =
if t ≤ L, if t > L.
π1:n h̃j j=t−L π1:n h̃j
j=0 Pt−1
(115)
Proof. We make the proof by induction on t. For the base case, note that by (111), we have h∗0 = 0, so that (115) holds. Now, let t ∈ N0 and assume that (115) holds. Note that in particular, we have h∗t ≥ 0. First, assume that t ∈ {0, . . . , L − 1}. Then, by (112), we have (a)
(b)
h∗t+1 = ρ(h∗t − h0t + π1:n h̃t ) = ρ(h∗t + π1:n h̃t ) = h∗t + π1:n h̃t ,
(116)
where (a) follows from the preceding claim and (b) is by h∗t ≥ 0 and π1:n h̃t ≥ 0, which is given by (113). Therefore, be the induction hypothesis, we have h∗t+1 = h∗t + π1:n h̃t =
t−1 X
π1:n h̃j + π1:n h̃t =
j=0
t X
π1:n h̃j ,
j=0
so that (115) holds for t + 1. Now, assume that t ≥ L. Then, by (112), we have (a)
(b)
h∗t+1 = ρ(h∗t − h0t + π1:n h̃t ) = ρ(h∗t − π1:n h̃t−L + π1:n h̃t ) = h∗t − π1:n h̃t−L + π1:n h̃t ,
(117)
where (a) follows from the preceding claim and (b) is by h∗t ≥ 0, π1:n h̃t ≥ 0, and π1:n h̃t−L ≤ 0, which is given by (113). Therefore, by the induction hypothesis, we have h∗t+1 = h∗t − π1:n h̃t−L + π1:n h̃t =
t−1 X
π1:n h̃j − π1:n h̃t−L + π1:n h̃t =
j=t−L
t X
π1:n h̃j ,
j=t+1−L
so that (115) holds for t + 1. This concludes the proof of the claim. Now, let t ∈ N. Note that the preceding claim and (113) can be reformulated as h∗t =
t−1 X j=(t−L)∨0
(113)
π1:n h̃j =
t−1 X
(ρ ◦ Ψ)j//L (x)δj mod L,0 .
j=(t−L)∨0
Note that since the set St := {(t − L) ∨ 0, . . . , t − 1} has at most L elements, there exists at most one element j ∈ St such that j mod L = 0. Such an element is given by j = ((t − 1)//L)L. Therefore, we have h∗t = (ρ ◦ Ψ)(((t−1)//L)L)//L (x)δ(((t−1)//L)L) mod L,0 = (ρ ◦ Ψ)(t−1)//L (x).
35
Finally,
π1:n ht = h∗t = (ρ ◦ Ψ)(t−1)//L (x).
This concludes the proof. The following technical lemma allows us to split a neural network into its positive and negative parts, which will be useful in the proof of the final simulation result. This splitting lets the RNN recover signed iterates while still using ReLU states. Lemma B.19. Let Ψ ∈ Nn,n be a ReLU neural network. Then, there exists a neural network Ψ′ ∈ N2n,2n satisfying L(Ψ′ ) = L(Ψ), W(Ψ′ ) = max{W(Ψ), 2n}, and Ω(Ψ′ ) = ±Ω(Ψ) such that for every x ∈ Rn , we have Ψ′ (x, 0) = Ψ′ (ρ(x), ρ(−x)) = (Ψ(x), −Ψ(x)). Proof. Let Ψ := (N0 , . . . , NL , A1 , . . . , bL ), with N0 = NL =: n. We define Ψ′ := (N0′ , . . . , NL′ , A′1 , . . . , b′L ) by N0 = NL := 2n, Nℓ′ := Nℓ for every ℓ ∈ {1, . . . , L − 1}, A′ℓ = Aℓ for every ℓ ∈ {2, . . . , L − 1}, b′ℓ := bℓ for every ℓ ∈ {1, . . . , L − 1}, AL bL A′1 := A1 −A1 , A′L := , and b′L := . −AL −bL Note that, indeed, we have L(Ψ′ ) = L(Ψ), W(Ψ′ ) = max{2n, maxℓ∈{1,...,L−1} Nℓ } = max{W(Ψ), 2n}, and Ω(Ψ′ ) = ±Ω(Ψ). Now, let x ∈ Rn . Note that for every ℓ ∈ {1, . . . , L − 1}, we have Ψ1 (x, 0) = A′1 (x, 0) + b′1 = A1 x + b1 = Ψ1 (x), and
Ψ1 (ρ(x), ρ(−x)) = A1 (ρ(x) − ρ(−x)) + b1 = A1 x + b1 = Ψ1 (x).
Therefore, we can show by induction that ΨL−1 (x, 0) = ΨL−1 (ρ(x), ρ(−x)) = ΨL−1 (x). Finally, we have ΨL (x, 0) = ΨL (ρ(x), ρ(−x)) = A′L (x, 0) + b′L = (AL x + bL , −AL x − bL ) = (ΨL (x), −ΨL (x)).
We finally show how to modify the RNN constructed in Lemma B.18 to output the iterates of Ψ at every time step, and not only at time steps that are multiples of L. The next lemma corresponds to this third and final step of the construction. We deliberately make an RNN that takes as input a vector of the form (02n(L+1) , x, −x, 0, 1L,1 ) instead of plain x to avoid having to use negative weights in the output layer of the RNN, which would not be allowed in a left-selector RNN, and will compromise the final simulation result. However, we provide a version of the final simulation result in which the RNN takes as input plain x at the end of this section, see Lemma B.21. The next result is the signed-iterate simulation in the left-selector form needed later. Lemma B.20. Let Ψ ∈ Nn,n be a ReLU neural network. Then, there exists a left-selector RNN R satisfying m(R) ≤ (2L(Ψ) + 2)(2n ∨ W(Ψ) + 1) and Ω(R) = ±Ω(Ψ) ∪ ±{0, 1} such that for every x ∈ Rn and t ∈ N, we have RDy[t] = Ψt//L (x), where y := (02n(L+1) , x, −x, 0, 1L,1 ) ∈ Rm(R) . Proof. Let Ψ′ : R2n → R2n be as in the preceding lemma, and let m̃ ∈ N ÃΨ′ ∈ Rm̃×m̃ be as in Lemma B.18 applied to Ψ′ . We define R := (m̃, m̃, n; ÃΨ′ , 0, Im̃ , Ao , 0), where Ao = π1:n − πn+1:2n = In −In 0 ∈ Rn×m̃ . Note that, indeed, we have m(R) = m̃ ≤ (2L(Ψ′ )+2)(W(Ψ′ )+1) = (2L(Ψ)+2)(2n∨W(Ψ)+1) and Ω(R) = Ω(ÃΨ′ ) ∪ {0, 1} = Ω(Ψ′ ) ∪ ±{0, 1} ⊆ ±Ω(Ψ) ∪ ±{0, 1}. Now, let x ∈ Rn , y := (02n(L+1) , x, −x, 0, 1L,1 ) ∈ Rm(R) , and consider the sequence (ht ∈ Rm̃ )t∈N0 defined by h0 := ρ(y) and ht+1 := ρ(ÃΨ′ ht ) for every t ∈ N0 . Note that by Lemma B.18, we have π1:2n ht = (ρ ◦ Ψ′ )(t−1)//L (ρ(x, −x)). Therefore, for every t ∈ N, we have Ao ht = π1:n ht − πn+1:2n ht = π1:n (ρ ◦ Ψ′ )(t−1)//L (ρ(x, −x)) − πn+1:2n (ρ ◦ Ψ′ )(t−1)//L (ρ(x, −x)) (a)
= π1:n (ρ ◦ Ψ)(t−1)//L (x) − πn+1:2n (ρ ◦ (−Ψ))(t−1)//L (x) = Ψ(t−1)//L (x), 36
where (a) follows from Lemma B.19. We now show that for every t ∈ N, we have HDy[t] = ht+1 . We make the proof by induction on t. For the base case, note that by Definition 2.1, we have HDy[0] = ρ(Im̃ y) = ρ(y) = h0 . Now, for the induction step, let t ∈ N0 and assume that HDy[t] = ht+1 . Then, by Definition 2.1, we have HDy[t + 1] = ρ(ÃΨ′ ht+1 + Im̃ Dy[t + 1]) = ρ(ÃΨ′ ht+1 ) = ht+2 . Therefore, for every t ∈ N, we have HDy[t] = ht+1 , so that RDy[t] = Ao ht+1 = Ψt//L (x). This concludes the proof. We close this part with the following Theorem, that modifies the RNN constructed in Lemma B.20 to take as input plain x instead of a vector of the form (02n(L+1) , x, −x, 0, 1L,1 ). This will not be used directly in the proof of the main TMNU simulation result, because the constructed RNN is not a left-selector, but is informative as a standalone result. This gives the same iteration simulation in the more familiar plain-input format. Theorem B.21. Let Ψ ∈ Nn,n be a ReLU neural network. Then, there exists an RNN R satisfying m(R) ≤ (2L(Ψ) + 2)(2n ∨ W(Ψ) + 1) + 2n + L + 1 and Ω(R) = ±Ω(Ψ) ∪ ±{0, 1} such that for every x ∈ Rn and t ∈ N, we have RDx[t] = Ψt//L (x). Proof. Let R′ = (m̃, m̃, n; ÃΨ′ , 0, Im̃ , Ao , 0) be as in the preceding lemma, such that for every x ∈ Rn , we have R′ D(02n(L+1) , x, −x, 0, 1L,1 )[t] = Ψt//L (x). Note that for every x ∈ Rn , we have (02n(L+1) , x, −x, 0, 1L,1 ) = Ax + b, where A ∈ Rm̃×n , b ∈ Rm̃ . Therefore, by Lemma B.16, there exists an RNN R satisfying m(R) = m̃ + n + L + 1 ≤ (2L(Ψ) + 2)(2n ∨ W(Ψ) + 1) + 2n + L + 1 and Ω(R) = Ω(R′ ) ∪ ±{0, 1} = ±Ω(Ψ) ∪ ±{0, 1} such that for every x ∈ Rn and t ∈ N, we have RDx[t] = R′ D(Ax + b)[t] = Ψt//L (x). This concludes the proof. B.3.3
Construction of an RNN that simulates a TMNU
In this part, we apply the preceding results to show that we can simulate the iteration of a TMNU by an RNN. The next lemma corresponds to the first step of the construction, in which we show that we can simulate the iteration of a TMNU by the iteration of a neural network, and then apply Lemma B.20 to show that we can simulate the iteration of a TMNU by an RNN. We now obtain the advertised RNN simulation theorem for bounded TMNU trajectories. Theorem B.22. Let M be a TMNU and u ∈ {0, 1}N such that M has uniformly C-bounded trajectories at u. Then, there exists an RNN R satisfying m(R) ≤ 10(max{3n, n + νM + 9} + 2dνM + 19), and Ω(R) = {0, 1/4, 1, 2, 3, 4, ∥M∥C } ∪ ±Ω(M) ∪ ±{η(u)}, such that RDx[t + 1] = Mu x[t//4],
t ∈ N0 ,
x ∈ [−1, 1].
(118)
Proof. By Theorem B.14, there exists a ReLU neural network Ψ ∈ Nn+2+d,n+2+d satisfying L(Ψ) = 4, W(Ψ) ≤ max{3n, n + νM + 9} + 2dνM + 18, Ω(Ψ) ⊆ ±{0, 1/4, 1, 2, 3, 4, ∥M∥C } ∪ ±Ω(M), such that for every c ∈ BM (C), ΨM,C (γM (c)) = γM (M(c)). (119) Then, by Lemma B.20, there exists a left-selector RNN R satisfying m(R) ≤ (2L(ΨM,C ) + 2)((2(n + 2 + d)) ∨ W(ΨM,C ) + 1) ≤ 10(max{3n, n + νM + 9} + 2dνM + 19)
37
and Ω(R) = ±Ω(Ψ) ∪ {0, 1} ⊆ ±{0, 1/4, 1, 2, 3, 4, ∥M∥C } ∪ ±Ω(M), such that for every z ∈ Rn , RDyz [t] = Ψt//4 (z),
t ∈ N,
(120)
where yz := (02N (L+1) , z, −z, 0, 1L,1 ) ∈ Rm(R) , N := n + 2 + d, and L := L(Ψ) = 4. Now, fix u ∈ {0, 1}N and assume that M has uniformly C-bounded trajectories at u. Then, for every x ∈ [−1, 1], we have cx := (1; |u; x, 0, . . . , 0) ∈ BM (C), so that for every t ∈ N, we have RDyγM (cx ) [t] = Ψt//4 (γM (cx )) = γM (Mt//4 (cx )).
(121)
Now, note that for every x ∈ [−1, 1], yγM (cx ) = (02N (L+1) , γM (cx ), −γM (cx ), 0, 1L,1 ) = (02N (L+1) , 1n,1 , η(u), x, 0d−1 , −1n,1 , −η(u), −x, 0d−1 , 0, 1L,1 ) = (02N (L+1) , 0n+2 , 1, 0d+n+1 , −1, 0)x + (02N (L+1) , 1n,1 , η(u), 0d , −1n,1 , −η(u), 0, 1L,1 ) =: Ax + b. Then, by Lemma B.16, there exists an RNN R̃ satisfying m(R̃) = m(R) + 1 and Ω(R̃) = Ω(R) ∪ Ω(A) ∪ ±Ω(b) ⊆ ±{η(u)} ⊆ {0, 1/4, 1, 2, 3, 4, ∥M∥C } ∪ ±Ω(M) ∪ ±{η(u)}, such that for every x ∈ [−1, 1] and t ∈ N, we have R̃Dx[t] = RDyγM (cx ) [t]. Hence, for every x ∈ [−1, 1] and t ∈ N, we have R̃Dx[t] = RDyγM (cx ) [t] = γM (Mt//4 (cx )). (122) Moreover, for every x ∈ [−1, 1] and t ∈ N, we have Mu x[t] = πd πω Mt (cx ) = πn+2+d γM (Mt (cx )) =: A′ γM (Mt (cx )),
(123)
where A′ := πn+2+d is a left-selector matrix. Hence, by Lemma B.15, there exists an RNN R′ satisfying m(R′ ) = m(R̃) and Ω(R′ ) ⊆ Ω(R̃) ∪ {0} ⊆ {0, 1/4, 1, 2, 3, 4, ∥M∥C } ∪ ±Ω(M) ∪ ±{η(u)} such that for every x ∈ [−1, 1] and t ∈ N, we have R′ Dx[t] = A′ (R̃Dx[t]) = A′ γM (Mt//4 (cx )) = Mu x[t//4].
(124)
This concludes the proof.
C
Detailed TMNU constructions
In this section, we build a TMNU that can approximate any continuous function f : [−1, 1] → R, in the sense of paradigm (24). The construction is made progressively, by first designing simple TMNUs that can approximate some simple functions, and then by using these simple TMNUs as subroutines to design more complex TMNUs that can approximate more complex functions, until we reach the point where we can design a TMNU that can approximate any continuous function f : [−1, 1] → R. Throughout this section, we will specify the number of states and neural dimension of the TMNUs explicitly, and the transition and command functions implicitly, by describing what the TMNU does in each state and for each symbol read by the head. Sometimes, we will not specify the transition and command functions completely, but only partially, by describing what the TMNU does in some states and for some symbols read by the head, and leaving the rest of the transition and command functions unspecified. For such unspecified state-symbol pairs, we will assume that the TMNU simply transitions to the halting state and does not update the tape or the neural state, i.e., that δ(q, σ) = (n, σ, 0) and κ(q, σ) = id for every unspecified state-symbol pair (q, σ). Moreover, for some state-symbol pairs, we will only specify the actions that modify the tape and the neural state. Specifically, if for some state symbol pair (q, σ), the command function applies the identity, i.e., does not update the neural state, then we will simply not specify the command function for this state-symbol pair, and we will assume that κ(q, σ) = id. Similarly, if for some state-symbol pair (q, σ), the head does not move right or left, we will simply not specify the transition function for this state-symbol pair, and we will assume that δ(q, σ) = (q ′ , σ ′ , 0) for some q ′ ∈ {1, . . . , n} and σ ′ ∈ Γ. This way of describing the TMNUs is more intuitive, and is sufficient to specify TMNUs without ambiguity. Once a TMNU is specified, we derive some properties of its evolution. Specifically, we will be interested to show that given some pair of configurations c, c′ of a TMNU M, there exists some t ∈ N0 , t ≥ 1 such that Mt (c) = c′ , and whether the computation remains bounded during the computation, i.e., whether ∥c∥tM := max ∥Ms (c)∥ ≤ C 0≤s≤t
38
(125)
for some C > 0. The next Lemma establishes that in order to study whether Mt (c) = c′ , and ∥c∥tM ≤ C, we can split the evolution from c to c′ into several sub-evolutions, and study each of these sub-evolutions separately. This is a very useful property, because it allows us to design TMNUs in a modular way, by designing subroutines that can be used as building blocks to design more complex TMNUs. This bookkeeping lemma is the basic concatenation rule used throughout the construction. Lemma C.1. (Chaining) Let M be a TMNU, c1 , c2 , c3 ∈ CM , t1 , t2 ∈ N, and C1 , C2 > 0 such that Mt1 (c1 ) = c2 ,
Mt2 (c2 ) = c3 ,
1 ∥c1 ∥tM ≤ C1 ,
2 ∥c2 ∥tM ≤ C2 .
(126)
Then, Mt1 +t2 (c1 ) = c3 ,
1 +t2 ∥c1 ∥tM ≤ C1 ∨ C2 .
and
(127)
(c1 ) = M (M (c1 )) = M (c2 ) = c3 . Moreover, since Proof. Since M (c1 ) = c2 , we have M t1 t2 c1 ∈ BM (C1 ) and c2 ∈ BM (C2 ), we have ∥Ms (c1 )∥ ≤ C1 ≤ C1 ∨ C2 for every s = 0, . . . , t1 , and ∥Ms (c2 )∥ ≤ C2 for every s = 0, . . . , t2 . Therefore, for every s = t1 + 1, . . . , t1 + t2 , we have t1
t1 +t2
t2
t1
t2
∥Ms (c1 )∥ = ∥Ms−t1 (Mt1 (c1 ))∥ = ∥Ms−t1 (c2 )∥ ≤ C2 ≤ C1 ∨ C2 .
C.1
(128)
Subroutines for TMNU constructions
In this section, we introduce the notion of subroutine for TMNUs, that will allow us to design more complex TMNUs by combining simpler TMNUs as building blocks. The idea is that a TMNU M can be designed in such a way that, when it is in some subset of its states, it behaves exactly as some other TMNU N, on a subset of its neural dimensions, while leaving the remaining neural dimensions unchanged. For example, assume that the TMNU M has 5 states and 3 neural dimensions, and the TMNU N has 3 states and 2 neural dimensions. Then, we will say that N is a subroutine of M with state correspondence (2, 3, 4) and neural dimension correspondence (1, 2), or in condensed form, with correspondence (2, 3, 4; 1, 2), if in state 2 and 3, the TMNU M updates its tape and neural dimensions 1 and 2 exactly as N would in states 1 and 2, respectively, and that when N halts, i.e., reaches state 3, M transitions to state 4. Such a behavior can be expressed formally by imposing some relation between the transition and command functions of both TMNUs. The following definition makes this relation precise. Definition C.1. (Subroutine) Let M := (nM , dM , δM , κM ) and N := (nN , dN , δN , κN ) be two TMNUs, such that nN ≤ nM and dN ≤ dM , and let ιq : {1, . . . , nN } ,→ {1, . . . , nM } and ιω : {1, . . . , dN } ,→ {1, . . . , dM } be two injective functions. We say that N is a subroutine of M with correspondance (ιq , ιω ) if for every q ∈ {1, . . . , nN − 1}, σ ∈ Γ and ω ∈ RdM , we have q σ m δM (ιq (q), σ) = (ιq ◦ δN (q, σ), δN (q, σ), δN (q, σ)),
πιω κM (ιq (q), σ)(ω) = κN (q, σ)(πιω ω),
and
πι⊥ω κM (ιq (q), σ)(ω) = πι⊥ω ω,
(129) (130)
where πιω : RdM → RdN is the projection defined by πιω ω = (ωi )i∈ιω ({1,...,dN }) , and πι⊥ω : RdM → RdM −dN is the orthogonal projection defined by πι⊥ω ω = (ωi )i∈ι / ω ({1,...,dN }) . We now introduce two pieces of notation that make the bookkeeping in subroutine arguments more transparent. Let N be a subroutine of M with correspondence (ιq , ιω ). If c = (q; τ ; ω) is a configuration of M with q ∈ ιq ({1, . . . , nN }), we define its subroutine shadow by Shιq ,ιω (c) := ι−1 (131) q (q); τ ; πιω ω . Conversely, if c′ = (q ′ ; τ ′ ; ω ′ ) is a configuration of N and h⊥ ∈ RdM −dN , we define the lift of c′ with frozen complementary neural state h⊥ as the configuration ⊥
Lifthιq ,ιω (c′ ) := (q; τ ; ω), where
q := ιq (q ′ ),
τ := τ ′ ,
πιω ω := ω ′ ,
and
(132) πι⊥ω ω := h⊥ .
(133)
The subroutine lemma states that the large machine follows the shadow computation exactly until the subroutine halts. Lemma C.2. Let M, N be two TMNUs such that N is a subroutine of M with correspondance (ιq , ιω ). Let c := (q; τ ; ω) be a configuration of M such that q ∈ ιq ({1, . . . , nN − 1}), and define c′ := Shιq ,ιω (c),
and 39
h⊥ := πι⊥ω ω.
(134)
Then, for all t ∈ {0, . . . , TN (c′ )},
⊥ Mt (c) = Lifthιq ,ιω Nt (c′ ) ,
(135)
where TN (c′ ) is the halting time of c′ in N defined by TN (c′ ) := inf{t ∈ N0 : πq Nt (c′ ) = nN }. In particular, for all t ∈ {0, . . . , TN (c′ )}, we have ∥c∥tM ≤ ∥c′ ∥tN ∨ ∥c∥, (136) Proof. Write Mt (c) = (qt ; τt ; ωt ) and Nt (c′ ) = (qt′ ; τt′ ; ωt′ ). We prove (135) by induction on t. The case t = 0 follows directly from the definitions of Sh and Lift. Assume that (135) holds for some t < TN (c′ ). Equivalently, we have qt = ιq (qt′ ),
τt = τt′ ,
πιω ωt = ωt′ ,
and
πι⊥ω ωt = h⊥ .
Since t < TN (c′ ), the state qt′ is not the halting state of N. Therefore, the subroutine identities (129) and (130) apply to qt′ , and yield ′ qt+1 = ιq (qt+1 ),
′ τt+1 = τt+1 ,
′ πιω ωt+1 = ωt+1 ,
and
πι⊥ω ωt+1 = h⊥ .
This is precisely (135) at time t + 1, and the induction is complete. Finally, by (135), for every s ∈ {0, . . . , t}, ∥Ms (c)∥ ≤ ∥Ns (c′ )∥ ∨ ∥h⊥ ∥∞ ≤ ∥c′ ∥tN ∨ ∥c∥. Taking the maximum over s = 0, . . . , t gives (136). We now initiate the construction of TMNUs that can approximate any continuous function f : [−1, 1] → R. The general philosophy guiding the incoming constructions is the following. At its roots, our construction exploits the fact that given a continuous function f : [−1, 1] → R, there exists a sequence of polynomials (Pi )i∈N0 with coefficients having an arbitrarily long but finite binary representation that converges to f . We will show that there exists a TMNU that, given that its tape is initialized as containing some encoding ui ∈ {0, 1}∗ of the coefficients of Pi for some i ∈ N0 , and that its neural state is initialized as containing some input x ∈ [−1, 1], can approximate Pi (x), and hence f (x). Then, we will use this TMNU as a subroutine of a larger TMNU that, given that its tape is initialized as containing an infinite sequence of bits u := u0 u1 u2 . . . that is the result of the concatenation of the encodings ui of the coefficients of Pi for every i ∈ N0 , and that its neural state is initialized as containing some input x ∈ [−1, 1], can approximate f (x) by successively approximating the polynomials Pi (x) for every i ∈ N0 . In other words, the TMNU we design will be such that, if some infinite binary sequence u is written on its tape, will recognize some finite prefix u0 of u as being some instruction to be executed, will execute this instruction, and delete the prefix u0 from the tape, and then will repeat this process with the remaining infinite binary sequence on the tape. This way, if u = u0 u1 u2 . . . encodes the coefficients of the polynomials Pi for every i ∈ N0 , then the TMNU will be able to successively approximate Pi (x) for every i ∈ N0 , and therefore to approximate f (x). With this philosophy in mind, we will show that every elementary TMNU that we design will be able to read some finite prefix p of the infinite sequence u written on its tape, recognize p as being some instruction to be executed, execute this instruction, and delete the prefix p from the tape. This will be reflected in all the statements we will establish regarding the properties of the TMNUs we will design. In the following, we will first give four examples of simple TMNUs that can be used as building blocks to design more complex TMNUs, and then we will show how to use these simple TMNUs as subroutines to design a TMNU that can approximate any continuous function f : [−1, 1] → R.
C.2
Elementary TMNUs
As a first elementary example of the philosophy described above, we define a TMNU that reads the first bit of the infinite binary sequence u written on the tape, and depending on this bit, either multiplies the neural state by −1 or leaves it unchanged, and then deletes this bit from the tape. This TMNU can be seen as a TMNU that reads some finite prefix u of the infinite sequence u written on the tape, recognizes u as being some instruction to be executed, executes this instruction, and deletes the prefix u from the tape, where in this case the instruction is "if the first bit of u is 1, then multiply the neural state by −1, otherwise do nothing". Definition C.2. We let ± be the TMNU with neural dimension 1 and 2 states, defined by the following procedure. • State 1: Let b ∈ {□, 0, 1} under scan. Then, write □, move right, update the neural state as ω ← (−1)b · ω, and go to State 2. • State 2: Halt. 40
Its behavior is immediate from the definition and will be used as a one-step subroutine. Lemma C.3. Let b ∈ {0, 1}, v ∈ {0, 1}# and x ∈ R. Define the configurations c := (1; |bv; x) and c′ := (2; |v; (−1)b x). Then, 1 ± (c) = c′ , and ∥c∥± = |x|. (137) 1 Proof. By definition of ± , we have ± (c) = c′ . Moreover, we have ∥c∥± = ∥c∥ ∨ ∥c′ ∥ = |x|, which concludes the proof.
As a second example of this philosophy, we design a TMNU that, given that its tape is initialized as containing some finite binary sequence u = 1k 0 followed by some infinite binary sequence u, and that its neural state is initialized as containing some input x ∈ [−1, 1], will delete u while multiplying x by 2k . Essentially, this TMNU acts as follows. As long as it reads a 1 on the tape, it multiplies the neural state by 2 and deletes this 1 from the tape. When it reads a 0 on the tape, it deletes this 0 from the tape and halts. Therefore, if the tape is initialized as containing some finite binary sequence u = 1k 0 followed by some infinite binary sequence u, and if the neural state is initialized as containing some input x ∈ [−1, 1], then after k + 1 steps, the tape will contain only the infinite binary sequence u, and the neural state will contain 2k x. This TMNU is precisely defined as follows. Definition C.3. We let S be the TMNU with neural dimension 1 and 2 states, defined by the following procedure. • State 1. Let b ∈ {□, 0, 1} under scan. i. If b ∈ {□, 0}, write □, move right, and go to State 2. ii. If b = 1, write □, move right, update the neural state as ω ← 2 · ω, and go back to State 1. • State 2. Halt. We now establish that S can be used to multiply its input by 2k for some k ∈ N, by reading some finite binary sequence u = 1k 0 on the tape, and deleting it. Note that the proof of this result relies on the Chaining Lemma (Lemma C.1), which allows us to split the evolution from some configuration c to some configuration c′ into several sub-evolutions, and to study each of these sub-evolutions separately. The formal statement keeps track of both the final configuration and the trajectory bound. Lemma C.4. Let k ∈ N0 , v ∈ {0, 1}# and x ∈ R. Define the configurations c := (1; |1k 0v, |; x) and c′ := (2; |v, |; 2k x). Then, Sk+1 (c) = c′ , ∥c∥k+1 ≤ 2k |x|. (138) S In the above Lemma, the tape in configuration c contains the finite binary sequence 1k 0 followed by some (potentially) infinite binary sequence v, and the neural state in configuration c contains some input x ∈ R. The Lemma states that after k + 1 steps, the tape will contain only the infinite binary sequence v, and the neural state will contain 2k x. In other words, the TMNU S has read the finite binary sequence 1k 0 on the tape, recognized it as being an instruction to be executed, executed this instruction by multiplying the input x by 2k , and deleted the finite binary sequence 1k 0 from the tape. This is a very simple example of how a TMNU can read some finite prefix of an infinite binary sequence written on its tape, recognize it as being some instruction to be executed, execute this instruction, and delete this prefix from the tape. We now give the proof of the above Lemma. Proof. Let k ∈ N0 and x ∈ R. We define the configurations c0 , c1 , . . . , ck+1 of S by ct := (1; |1k−t 0v, |; 2t x), and
t ∈ {0, . . . , k},
ck+1 := (2; |v, |; 2k x).
Claim. For all t ∈ {0, . . . , k}, we have S(ct ) = ct+1 and ∥ct ∥1S ≤ 2k |x|. Proof. Let t ∈ {0, . . . , k − 1}. In configuration ct , the machine is in State 1 and reads b = 1 on tape 1, so after one step, the machine is in configuration ct+1 . Moreover, we have ∥ct ∥1S ≤ 2t+1 |x| ≤ 2k |x|. Now, let t = k. In configuration ck , the machine is in State 1 and reads b = 0 on tape 1, so after one step, the machine is in configuration ck+1 . Moreover, we have ∥ck ∥1S ≤ 2k |x|. This concludes the proof of the claim. Now, since c0 = c and ck+1 = c′ , the claim implies that Sk+1 (c) = c′ , and, moreover, ∥c∥k+1 = max ∥ct ∥1S ≤ 2k |x|. S t=0,...,k
This concludes the proof. 41
(139)
We now design a TMNU that multiplies a real number x ∈ R by some dyadic number a ∈ [0, 1), by reading some finite binary sequence u on the tape that encodes a in binary form. We define δ[0,1) : {0, 1}∗ → [0, 1] by ℓ(u) X δ[0,1) (u) = ui 2−i , u ∈ {0, 1}∗ (140) i=1
We denote by D+ 1 the output set of the function δ[0,1) , i.e., ∗ D+ ⊆ [0, 1]. 1 := δ[0,1) (u) : u ∈ {0, 1}
(141)
n Note that D+ 1 is dense in [0, 1], because for every x ∈ [0, 1] and every n ∈ N, there exists u ∈ {0, 1} such −n that |x − δ[0,1) (u)| ≤ 2 . We also define kind of inverse encoding by
u(a) = arg min{|u| : u ∈ {0, 1}∗ , δ[0,1) (u) = a},
(142)
for every a ∈ D+ 1. Continuing with the philosophy introduced above, the TMNU we design to multiply x by some a ∈ D+ 1 needs to be able to identify some prefix of an infinite binary sequence u written on the tape as encoding the number a, and to delete this prefix from the tape while multiplying x by a. In order to nonambiguously identify some prefix of u as encoding a, we will use the following prefix-free encoding of binary sequences, that we denote by · : {0, 1}∗ → {0, 1}∗ . We first specify the prefix code used to delimit finite dyadic instructions. Definition C.4. (Prefix-free encoding) For u ∈ {0, 1}∗ , we let u ∈ {0, 1}2ℓ(u)+1 be defined as u2k = uk ,
u2k−1 = 1, ∀k ∈ {1, . . . , ℓ (u)},
and u2ℓ(u)+1 = 0. The TMNU we design has a neural state in dimension 2. Initially, the neural state contains (x, 0) for some x ∈ R, and the tape content is of the form uv for some u ∈ {0, 1}∗ encoding a number a ∈ D+ 1 and some v ∈ {0, 1}# . Then, the TMNU will exploit the prefix-free encoding to identify the prefix u of the tape content as encoding the number a. Now, remark that ax can be rewritten as ax =
ℓ(u) X
uk 2
−k
x=
k=1
ℓ(u) X
uk yk ,
(143)
k=1
where yk := 2−k x for every k ∈ {1, . . . , ℓ (u)}. Note that, in particular, we have yk+1 = yk /2, for every k ∈ {1, . . . , ℓ (u) − 1}. Therefore, the TMNU will compute successively each term yk on the first coordinate of the neural state, and will add yk to the second coordinate of the neural state if uk = 1. This way, after reading the whole prefix u, the first coordinate of the neural state will contain yℓ(u) = 2−ℓ(u) x, and the Pℓ(u) second coordinate of the neural state will contain k=1 uk yk = ax. Then, the TMNU will delete the prefix u from the tape, and will halt with the first coordinate of the neural state containing 2−ℓ(u) x and the second coordinate of the neural state containing ax. The precise definition of this TMNU is given as follows. Definition C.5. We let Hom+ be the TMNU with neural dimension 2 and 3 states, defined by the following procedure. • State 1. Let b ∈ {□, 0, 1} be the symbol under scan. i. If b ∈ {0, □}, update the neural state as ω ← (ω2 , 0), and go to State 3. ii. If b = 1, write □, move right, update the neural state as ω ← 12 ω1 , ω2 , and go to State 2. • State 2. Let b ∈ {□, 0, 1} be the symbol under scan. Then, write □, move right, update the neural state as ω ← (ω1 , ω2 + bω1 ) , and go to State 1. • State 3. Halt. We now cast the discussion above into the following formal Lemma, that establishes that Hom+ can be used to multiply a real number x ∈ R by some dyadic number a ∈ D+ 1 , by reading some finite binary sequence u on the tape that encodes a in binary form, and deleting this prefix from the tape. The estimate also records that the auxiliary accumulation never exceeds the input magnitude. (a) Lemma C.5. Let a ∈ D+ , v ∈ {0, 1}# , and x ∈ R. Define the configurations c := (1; |uv; x, 0) 1 , u := u ′ and c := (3; |v; ax, 0). Then, ℓ(u) (Hom+ )ℓ(u) (c) = c′ , ∥c∥Hom+ ≤ |x|. (144)
42
(a) Proof. Let a ∈ D+ , n := ℓ (u), v ∈ {0, 1}# , and x ∈ R. We define a sequence (ct )0≤t≤2n+1 of 1 , u := u + configurations of Hom by = (1; |u2t+1:2n+1 v; 2−t x, at x) , for all t = 0, . . . , n, c2t (145) c2t+1 = 2; |u2(t+1):2n+1 v; 2−(t+1) x, at x , for all t = 0, . . . , n − 1, c2n+1 = (3; |v; ax, 0),
where at := δ[0,1) (u1:t ) for every t ∈ {0, . . . , n}. Claim. For all t = 0, . . . , 2n, we have Hom+ (ct ) = ct+1 and ∥ct ∥1Hom+ ≤ |x|. Proof of the Claim. We divide the proof of the claim into three cases. 1. Let 0 ≤ t ≤ n − 1, and assume that the machine is in configuration c2t . We have c2t = 1; |u2t+1:2n+1 v; 2−t x, at x , therefore the machine is in State 1 and reads b := u2t+1 = 1 on the tape. By Definition of State 1 of Hom+ (Definition C.5), the machine writes □ on the tape, moves the head one cell to the right, updates the neural state to (ω1 /2, ω2 ), and goes to State 2. Hence, after one step, the machine is in configuration 2; |u2t+2:2n+1 v; 2−(t+1) x, at x = c2t+1 . Moreover, we have ∥c2t ∥1Hom+ ≤ |x|. 2. Let 0 ≤ t ≤ n − 1. Assume that the machine is in configuration c2t+1 . We have c2t+1 = 2; |u2(t+1):2n+1 v; 2−(t+1) x, at x , therefore the machine is in State 2 and reads b := u2(t+1) = ut+1 on the tape. By Definition of State 2 of Hom+ (Definition C.5), the machine writes □ on the tape, moves the head one cell to the right, updates the neural state to (ω1 , ω2 + b · ω1 ), where ω2 + b · ω1 = at x + ut+1 · 2−(t+1) x = at+1 x, and goes to State 1. Hence, after one step, the machine is in configuration 1; |u2t+3:2n+1 v; 2−(t+1) x, at+1 x = c2t+2 . Moreover, we have ∥c2t+1 ∥1Hom+ ≤ |x|. 3. Case t = 2n. Assume that the machine is in configuration c2n := 1; |u2n+1:2n+1 v; 2−n x, an x = 1; |u2n+1 v; 2−n x, ax , so the machine is in State 1, and reads the symbol b = u2n+1 = 0. By Definition of State 1 of Hom+ (Definition C.5), the machine updates the neural state to (ax, 0), and goes to State 3. This implies that, after one time step, the machine is in configuration (3; |v; ax, 0) = c2n+1 . Moreover, we have ∥c2n ∥1Hom+ ≤ |x|. This concludes the proof of the claim. Now, since c = c0 and c′ = c2n+1 , the claim implies that
and, moreover,
(Hom+ )2n+1 (c) = c′ ,
(146)
∥c∥2n+1 = max ∥ct ∥1Hom+ ≤ |x|. Hom+
(147)
0≤t≤2n
Since ℓ (u) = 2n + 1, this concludes the proof. As a final example of the philosophy of building TMNUs that identify a finite prefix of an infinite sequence written on their tape as an instruction to be executed, we define a family of TMNUs that simulate RNNs. Specifically, for every RNN R, we define a TMNU MR such that given that 1n 0u is initially written on the tape and the neural state contains some x ∈ Rd (where d is the input dimension of R), the TMNU erases 1n 0 from its tape, and produces RDx[n] on its neural state. This allows us, in particular, to import any results already present in the literature for RNNs to TMNUs. In particular, this 43
allows to show that TMNUs can approximate the product of two real numbers, by simulating the RNN defined in [14, Theorem 11]. We define formally the TMNU that simulates an RNN. Definition C.6. Let R := (d, m, d′ ; Ah , bh , Ax , Ao , bo ) be an RNN. We define the TMNU MR with neural dimension d := m and 4 states, defined by the following procedure. • State 1. Update the neural state as ω ← Ax ω1:d + bh , and go to State 2. • State 2. Update the neural state as ω ← ρ(ω) and go to State 3. • State 3. Let b ∈ {□, 0, 1} be the symbol under scan. i. If b ∈ {0, □}, write □, move right, update the neural state as Ao b ω← ω+ o , 0 0 and go to State 4. ii. If b = 1, write □, move right, update the neural state as ω ← Ah ω + bh and go to State 2. • State 4. Halt. The simulation property of this machine is recorded next. Lemma C.6. Let R := (d, m, d′ ; Ah , bh , Ax , Ao , bo ) be an RNN, and let MR be the TMNU defined in Definition C.6. Let n ∈ N, v ∈ {0, 1}# , and x ∈ Rd . Define the configurations c := (1; |1n 0v; x, 0, . . . , 0) and c′ := (4; |v; RDx[n], 0, . . . , 0). Then, M2n+3 (c) = c′ , R
(148)
and ∥c∥2n+3 MR ≤ ∥x∥∞ ∨ ∥Ax x + bh ∥∞ ∨ ∥RDx[n]∥∞ ∨ max ∥HDx[t]∥∞ ∨ max ∥Ah HDx[t] + bh ∥∞ , (149) 0≤t≤n
0≤t≤n
where H is the hidden state operator of R as defined in Definition 2.1. Proof. Let R := (d, m, d′ ; Ah , bh , Ax , Ao , bo ) be an RNN, n ∈ N0 , v ∈ {0, 1}# , and x ∈ Rd . We define a sequence of configurations (ct )0≤t≤2n+3 of MR by = (1; |1n 0v; x, 0, . . . , 0), c0 c1 = (2; |1n 0v; Ax x + bh ), (150) c2t = (3; |1n−t+1 0v; HD(x)[t − 1]), for all t = 1, . . . , n + 1, n−t c2t+1 = (2; |1 0v; Ah HD(x)[t − 1] + bh ), for all t = 1, . . . , n, c = (4; |v; RD(x)[n], 0, . . . , 0). 2n+3 We also let B := ∥x∥∞ ∨ ∥Ax x + bh ∥∞ ∨ ∥RDx[n]∥∞ ∨ max ∥HDx[t]∥∞ ∨ max ∥Ah HDx[t] + bh ∥∞ . 0≤t≤n
0≤t≤n
Claim. For every t ∈ {0, . . . , 2n + 2}, we have MR (ct ) = ct+1 . Proof of the claim. First note that MR (c0 ) = c1 by State 1 of MR (Definition C.6), and that MR (c1 ) = c2 by State 2 of MR . Moreover, in configuration c2n+2 = (3; |0v; HDx[n]), the machine is in State 3 and reads a symbol 0 on its tape. Therefore, MR (c2n+2 ) = c2n+3 . Now, let t ∈ {1, . . . , n}. We have c2t = (3; |1n−t+1 0v; HDx[t − 1]), so the machine is in State 3 and reads the symbol b = 1 on the tape. By State 3 of MR , we have MR (c2t ) = c2t+1 , since c2t+1 = (2; |1n−t 0v; Ah HDx[t − 1] + bh ). Moreover, we have c2t+1 = (2; |1n−t 0v; Ah HDx[t − 1] + bh ), so the machine is in State 2, and by State 2 of MR , we have MR (c2t+1 ) = c2(t+1) , since c2(t+1) = (3; |1n−t 0v; HDx[t]). This concludes the proof of the claim. Since c = c0 and c′ = c2n+3 , the claim implies that M2n+3 (c) = c′ . R
44
(151)
Moreover, by denoting ωt the neural state of configuration ct , we have ∥c∥2n+3 MR =
max
0≤t≤2n+2
∥ct ∥1MR ≤
max
0≤t≤2n+3
∥ωt ∥∞
≤ max{∥x∥∞ , ∥Ax x + bh ∥∞ , ∥RDx[n]∥∞ , max ∥HDx[t]∥∞ , max ∥Ah HDx[t] + bh ∥∞ } 0≤t≤n
0≤t≤n
= B, by definition of B. This concludes the proof of the Lemma. Now, we can use the following result from the literature on RNNs to design a TMNU that can approximate the multiplication of two inputs x and y. Lemma C.7. [14, Theorem 11] There exists an RNN R× := (2, 14, 1; Ah , bh , Ax , Ao , bo ), such that for every x ∈ [−1, 1]2 , R× Dx[n] − (x1 · x2 ) ≤ 2−2n−1 ,
and
H× Dx[n] ∞ , Ah H× Dx[n] + bh ∞ , R× Dx[n] ≤ 1,
for every n ∈ N, and Ax x + bh ∈ [−1, 1]14 . We turn this RNN primitive into a TMNU by applying the simulation construction above. Definition C.7. We let × be the TMNU defined as the TMNU MR× of Definition C.6, where R× is the RNN of Lemma C.7. The resulting machine inherits the approximation and boundedness properties of the RNN multiplier. Lemma C.8. Let n ∈ N0 , v ∈ {0, 1}# , and x, y ∈ [−1, 1]. Define the configurations c := (1; |1n 0v; x, y, 012 ) and c′ := (4; |v; R× D(x, y)[n], 013 ). Then, × 2n+3 (c) = c′ ,
2n+3 ∥c∥× ≤ 1.
(152)
Proof. Let R× := (2, 14, 1; Ah , bh , Ax , Ao , bo ) be the RNN of Lemma C.7, and let H× be its hidden state operator. By Lemma C.6, we have × 2n+3 (c) = c′ and 2n+3 ∥c∥× ≤ |x| ∨ |y| ∨ ∥Ax (x, y) + bh ∥∞ ∨ |R× D(x, y)[n]|
∨ max ∥H× D(x, y)[t]∥∞ ∨ max ∥Ah H× D(x, y)[t] + bh ∥∞ . 0≤t≤n
0≤t≤n
Since (x, y) ∈ [−1, 1]2 , we have |x| ∨ |y| ≤ 1. Moreover, Lemma C.7 implies that ∥Ax (x, y) + bh ∥∞ ≤ 1, |R× D(x, y)[n]| ≤ 1, and, by applying the same lemma for each t ∈ {0, . . . , n}, max ∥H× D(x, y)[t]∥∞ ∨ max ∥Ah H× D(x, y)[t] + bh ∥∞ ≤ 1.
0≤t≤n
0≤t≤n
This concludes the proof. We now have defined four TMNUs that allows to perform some simple operations: multiplication by −1, multiplication by 2k for some k ∈ N0 , multiplication by some a ∈ D+ 1 , and approximation of the multiplication of two inputs. These TMNUs can be combined to design more complex TMNUs that perform more complex operations. Specifically, by combining these operations we can design a TMNU that approximate any polynomial, and consequently, and any continuous function. In the next section, we introduce the notion of subroutine for TMNUs, that will allow us to design more complex TMNUs by combining simpler TMNUs as building blocks. Now equipped with this tool for combining TMNUs, we are ready to design more complex TMNUs that can perform more complex operations, by combining simpler TMNUs as building blocks. As a first example, we can combine the TMNU± and the TMNU Hom+ to design a TMNU that can multiply its + input by some a ∈ D1 := D+ 1 ∪ −D1 . For every a ∈ D1 , we let ( 0, if a ≥ 0, s(a) = 1, if a < 0, and define v (a) := s(a) u(|a|) . We now introduce the TMNU Hom that can multiply its input by some a ∈ D1 by reading some finite binary sequence u on the tape that encodes a in binary form, and deleting this prefix from the tape. Definition C.8. We let Hom be the TMNU with neural dimension 2 and 4 states, defined by the following procedure. 45
• State 1. Use ± as a subroutine with correspondence (1, 2; 1). • State 2-3. Use Hom+ as a subroutine with correspondence (2, 3, 4; 1, 2). • State 4. Halt. By using the Subroutine Lemma C.2, we can easily verify that Hom can be used to multiply an input x by some a ∈ D1 , by reading some finite binary sequence u on the tape that encodes a in binary form, and deleting this prefix from the tape, as follows. Lemma C.9. Let a ∈ D1 , u := v (a) , v ∈ {0, 1}# , and x ∈ R. Define the configurations c := (1; |uv; x, 0) and c′ := (4; |v; ax, 0). Then, ℓ(u) Homℓ(u) (c) = c′ , ∥c∥Hom ≤ |x|. (153) Proof. Let a ∈ D1 , u := v (a) , v ∈ {0, 1}# , and x ∈ R. Define the configurations c0 := (1; |uv; x, 0),
c1 := (2; |u2:ℓ(u) v; (−1)u1 x, 0),
c2 := (4; |v; ax, 0).
We first show that Hom(c0 ) = c1 and ∥c0 ∥1Hom ≤ |x|. By definition of State 1 of Hom (Definition C.8), the machine uses the TMNU ± as a subroutine with correspondence (1, 2; 1). Let c′0 := (1; |uv; x),
c′1 := (2; |u2:ℓ(u) v; (−1)u1 x)
be configurations of ± . Then, Sh(1,2),(1) (c0 ) = c′0 ,
Lift0(1,2),(1) (c′1 ) = c1 .
1 By Lemma C.3, we have ± (c′0 ) = c′1 and ∥c′0 ∥± ≤ |x|. Therefore, by Lemma C.2, we obtain
±(c′0 )) = c1 Hom(c0 ) = Lift0(1,2),(1) (± and
1 ∥c0 ∥1Hom ≤ ∥c′0 ∥± ∨ ∥c0 ∥ ≤ |x|. ℓ(u)−1
We now show that Homℓ(u)−1 (c1 ) = c2 and ∥c1 ∥Hom ≤ |x|. By definition of States 2 and 3 of Hom (Definition C.8), the machine uses the TMNU Hom+ as a subroutine with correspondence (2, 3, 4; 1, 2). Let c̃1 := (1; |u2:ℓ(u) v; (−1)u1 x, 0), c̃2 := (3; |v; |a|(−1)u1 x, 0) = (3; |v; ax, 0) be configurations of Hom+ . Then, Sh(2,3,4),(1,2) (c1 ) = c̃1 ,
Lift∅ (2,3,4),(1,2) (c̃2 ) = c2 .
Since u2:ℓ(u) = u(|a|) and ℓ u2:ℓ(u) = ℓ (u) − 1, Lemma C.5 gives (Hom+ )ℓ(u)−1 (c̃1 ) = c̃2 ,
ℓ(u)−1
∥c̃1 ∥Hom+ ≤ |x|.
Therefore, by Lemma C.2, we obtain + ℓ(u)−1 Homℓ(u)−1 (c1 ) = Lift∅ (Hom ) (c̃ ) = c2 1 (2,3,4),(1,2) and
ℓ(u)−1
∥c1 ∥Hom
ℓ(u)−1
≤ ∥c̃1 ∥Hom+ ∨ ∥c1 ∥ ≤ |x|.
Combining the two stages, and using c = c0 and c′ = c2 , we obtain Homℓ(u) (c) = c′ , and
ℓ(u)
ℓ(u)−1
∥c∥Hom = ∥c0 ∥1Hom ∨ ∥c1 ∥Hom
≤ |x|.
In the incoming section, we use the Subroutine Lemma C.2 in combination with the elementary TMNUs defined in previous section in order to build a TMNU that approximate any polynomial with dyadic coefficients.
46
C.3
Approximation of polynomials with dyadic coefficients
In this section, we show how to use the previously defined TMNUs in order to approximate polynomials with dyadic coefficients. Specifically, we will design a TMNU that realises the following approximation of a polynomial P = a0 + a1 x + . . . + ad xd , with d ∈ N0 and a0 , . . . , ad ∈ D := Z + D1 . For d, n ∈ N0 , we define the function Gd,n : [−1, 1] → [−1, 1] by G0,n (x) = 1,
Gd+1,n (x) = R× D(Gd,n (x), x)[n],
d ∈ N, n ∈ N0 , x ∈ [−1, 1],
and further, for n ∈ N0 , the function FP,n : [−1, 1] → R by FP,n (x) =
d X
ai Gi,n (x).
i=0
The function FP,n is precisely the function that we will realize with a TMNU, by computing succesively G0,n (x), G1,n (x), . . . , Gd,n (x), and then combining these values with linear transformations to obtain FP,n (x). In Appendix D are shown the following bounds on the magnitude and on the approximation rate of Gd,n and FP,n , for n, d ∈ N0 . In this Lemma, we use the quantities ∥P ∥0 := d + 1,
and
∥P ∥1 := |a0 | + |a1 | + · · · + |ad |,
(154)
where P = a0 + a1 x + . . . + ad xd is a polynomial with d ∈ N0 and a0 , . . . , ad ∈ R. Lemma C.10. Let d, n ∈ N0 and P be a polynomial. Then, for all x ∈ [−1, 1], we have |Gd,n (x)| ≤ 1, |FP,n (x)| ≤ ∥P ∥1 , Gd,n (x) − xd ≤ d2−2n−1 ,
and
|FP,n (x) − P (x)| ≤ ∥P ∥1 ∥P ∥0 2−2n−1 .
We now explain how we design a TMNU that computes FP,n . The idea is that the TMNU starts with its tape containing an encoding of the coefficients a0 , . . . ad ∈ D of P , and an encoding of n ∈ N0 , and with its neural state containing (x, 1, 0) = (x, G0,n (x), 0) for some x ∈ [−1, 1], that is transformed successively into (x, G1,n (x), a0 G0,n (x)), (x, G2,n (x), a0 G0,n (x) + a1 G1,n (x)), . . . , (x, Gd,n (x), a0 G0,n (x) + a1 G1,n (x) + · · · + ad−1 Gd−1,n (x)), and finally into (x, Gd,n (x), FP,n (x)). We will specifically design a Pi−1 TMNU π that implements one step of this procedure, that is, that transforms (x, Gi,n (x), j=0 aj Gj,n (x)) Pi into (x, Gi+1,n (x), j=0 aj Gj,n (x)), for any i ∈ {0, . . . , d − 1}. Then, we will use π as a subroutine of a larger TMNU that implements the whole procedure. Before proceeding, we want to add a small correction to the preceding description of the procedure. In view of the simulation of the TMNU we build by an RNN, we need to make sure that the neural state of the TMNU remains bounded below some constant, as explained at the beginning of Section 6. In the procedure above, the neural state is bounded by ∥P ∥1 , so this TMNU can indeed be simulated by an RNN. However, ultimately, this TMNU will be used as a subroutine of a larger TMNU that approximates continuous functions by means of a sequence (Pi )i∈N0 of approximating polynomials, that may satisfy ∥Pi ∥1 → ∞. To cope with this issue, we will make a slight modification of the procedure above, that i→∞
guarantees that the neural state of the TMNU remains bounded by ∥P ∥L∞ ([−1,1]) +∥P ∥0 ∥P ∥1 2−2n−1 . This way, when approximating a sequence of polynomials (Pi )i∈N0 that converges uniformly to some continuous function, we can always choose n large enough so that ∥Pi ∥0 ∥Pi ∥1 2−2n−1 ≤ 1, and moreover, the uniform convergence of (Pi )i∈N0 implies that there exists some constant M > 0 such that ∥Pi ∥L∞ ([−1,1]) ≤ M for all i ∈ N0 , hence the neural state of the TMNU remains bounded by M + 1 when approximating the sequence (Pi )i∈N0 , and can therefore be simulated by an RNN. We now describe how to modify the above procedure. For every polynomial P ̸= 0, define kP := 1 + ⌊log2 ∥P ∥1 ⌋ ∨ 0, and P ∗ := 2−kP P . Note that ∥P ∗ ∥∞ ≤ ∥P ∗ ∥1 < 1. Then, instead of computing FP,n (x), we will compute FP ∗ ,n (x), and then multiply the result by 2kP to obtain FP,n (x). Therefore, during the process of computing FP ∗ ,n (x), the neural state of the TMNU remains bounded by ∥P ∗ ∥1 < 1, and at the end of the process, the TMNU multiplies the result by 2kP to obtain FP,n (x), so the neural state remains bounded below |2kP FP ∗ ,n (x)| = |FP,n (x)| ≤ ∥P ∥L∞ ([−1,1]) + ∥P ∥0 ∥P ∥1 2−2n−1 , by Lemma C.10. We now give the formal definition of the TMNU π described above. This machine performs one coefficient update and one monomial update. Definition C.9. We define the TMNU π with neural dimension 16 and 9 states, defined by the following procedure. • State 1. Update the neural state as ω ← (ω1 , ω2 , ω3 , ω2 , 012 ), and go to State 2. • State 2-4. Use Hom (Def. C.8) as subroutine with correspondence (2, 3, 4, 5; 4, 5). • State 5. Update the neural state as ω ← (ω1 , ω2 , ω3 + ω4 , ω1 , 012 ), and go to State 6. 47
• State 6-8. Use × (Def. C.7) as subroutine with correspondence (6, 7, 8, 9; 2, 4, . . . , 16). • State 9. Halt. We now show that P π, in essence, implements one stepP of the procedure described above, that is, it i−1 i transforms (x, Gi,n (x), j=0 aj Gj,n (x)) into (x, Gi+1,n (x), j=0 aj Gj,n (x)), for any i ∈ {0, . . . , d − 1}, by reading some finite binary sequence u on the tape that encodes ai in binary form, and deleting this prefix from the tape, and by reading some finite binary sequence 1n 0 on the tape that encodes n, and deleting this prefix from the tape. In fact, what we prove is slightly more general, as we show that π transforms (x, Gi,n (x), z) into (x, Gi+1,n (x), z + ai Gi,n (x)), for any x ∈ [−1, 1], z ∈ R, i, n ∈ N0 . Lemma C.11. Let n, i ∈ N0 , a ∈ D1 , u := v (a) , and v ∈ {0, 1}# , x ∈ [−1, 1], and z ∈ R. Define T := ℓ (u)+2n+5 and the configurations c := (1; |u1n 0v; x, Gi,n (x), z, 013 ) and c′ := (9; |v; x, Gi+1,n (x), z + aGi,n (x), 013 ). Then, π T (c) = c′ , ∥c∥Tπ ≤ 1 ∨ (|z| + |a|). (155) Proof. Let n, i ∈ N0 , a ∈ D1 , u := v (a) , v ∈ {0, 1}# , x ∈ [−1, 1], z ∈ R, and define y := Gi,n (x). We set THom := ℓ (u) ,
T× := 2n + 3,
and
T := THom + T× + 2 = ℓ (u) + 2n + 5.
We define the configurations c0 , c1 , c2 , c3 , c4 of π by n c0 = (1; |u1 0v; x, y, z, 013 ), c1 = (2; |u1n 0v; x, y, z, y, 012 ), c2 = (5; |1n 0v; x, y, z, ay, 012 ), c3 = (6; |1n 0v; x, y, z + ay, x, 012 ), c = (9; |v; x, G 4 i+1,n (x), z + ay, 013 ).
(156)
Claim. The following four statements hold. (a) π(c0 ) = c1 and ∥c0 ∥1π ≤ 1 ∨ |z|. (b) π THom (c1 ) = c2 and ∥c1 ∥TπHom ≤ 1 ∨ |z|. (c) π(c2 ) = c3 and ∥c2 ∥1π ≤ 1 ∨ (|z| + |a|). T
(d) π T× (c3 ) = c4 and ∥c3 ∥π× ≤ 1 ∨ (|z| + |a|). Proof. We prove each statement in turn. (a) Assume that the machine is in configuration c0 . The machine is in State 1, so after one step, the machine is in configuration c1 . Moreover, we have ∥c0 ∥1π ≤ ∥(x, y, z, y, 012 )∥∞ ≤ 1 ∨ |z|, where we used |x| ≤ 1 and |y| = |Gi,n (x)| ≤ 1 by Lemma C.10. (b) Let c′1 := (1; |u1n 0v; y, 0), and c′2 := (4; |1n 0v; ay, 0). Then, Sh(2,3,4,5),(4,5) (c1 ) = c′1 ,
(x,y,z,0
)
11 Lift(2,3,4,5),(4,5) (c′2 ) = c2 .
Hom By Lemma C.9, we have HomTHom (c′1 ) = c′2 and ∥c′1 ∥THom ≤ |y| ≤ 1. Since Hom is a subroutine of π with correspondance (2, 3, 4, 5; 4, 5) by Definition C.9, Lemma C.2 yields (x,y,z,011 ) π THom (c1 ) = Lift(2,3,4,5),(4,5) HomTHom (c′1 ) = c2 ,
and
Hom ∥c1 ∥TπHom ≤ ∥c′1 ∥THom ∨ ∥c1 ∥ ≤ 1 ∨ |z|.
(c) Assume that the machine is in configuration c2 . The machine is in State 5, so after one step, the machine is in configuration c3 . Moreover, we have ∥c2 ∥1π ≤ ∥(x, y, z, ay, 012 )∥∞ ∨ ∥(x, y, z + ay, x, 012 )∥∞ ≤ 1 ∨ (|z| + |a|). (d) Let c′3 := (1; |1n 0v; y, x, 012 ), and c′4 := (4; |v; R× D(y, x)[n], 013 ) = (4; |v; Gi+1,n (x), 013 ). Then, Sh(6,7,8,9),(2,4,...,16) (c3 ) = c′3 , 48
(x,z+ay)
Lift(6,7,8,9),(2,4,...,16) (c′4 ) = c4 .
T
By Lemma C.8, we have × T× (c′3 ) = c′4 and ∥c′3 ∥×× ≤ 1. Since × is a subroutine of π with correspondence (6, 7, 8, 9; 2, 4, . . . , 16) by Definition C.9, Lemma C.2 yields (x,z+ay) π T× (c3 ) = Lift(6,7,8,9),(2,4,...,16) × T× (c′3 ) = c4 , and T
(a)
T
∥c3 ∥π× ≤ ∥c′3 ∥×× ∨ ∥c3 ∥ ≤ 1 ∨ (|z| + |a|), where (a) is by |x| ≤ 1, |Gi+1,n (x)| ≤ 1 and |y| = |Gi,n (x)| ≤ 1, where |Gj,n (x)| ≤ 1 for j = i, i + 1 by Lemma C.10. This concludes the proof of the claim. Now, by invoking repeatedly the Chaining Lemma (Lemma C.1) together with the claim, and using c = c0 and c′ = c4 , we obtain π T (c) = c′ , and
∥c∥Tπ ≤ 1 ∨ (|z| + |a|).
This concludes the proof. We finally design a TMNU that uses the TMNU π as a subroutine to compute FP,n (x), for any polynomial P with dyadic coefficients, any n ∈ N0 , and any x ∈ [−1, 1]. In order to implement this TMNU in a way that its neural state remains bounded by ∥P ∥L∞ ([−1,1]) + ∥P ∥0 ∥P ∥1 2−2n−1 , we will first compute FP ∗ ,n (x), where P ∗ := 2−kP P is the rescaled version of P defined above, and then we will multiply the result by 2kP to obtain FP,n (x). This TMNU will parse an encoding of the coefficients of P ∗ , of n and kP on its tape, that we formally define as follows. Definition C.10. Let P : x 7→ a0 x0 + a1 x1 + · · · + ad xd be a polynomial with coefficients a0 , . . . , ad ∈ D, and let n ∈ N0 . We define ∗
∗
∗
∗
uP := v (a0 ) · 1 · v (a1 ) · 1 · v (a2 ) · · · 1 · v (ad ) · 0 · 1kP 0, and
∗
∗
∗
∗
uP,n := v (a0 ) 1n 0 · 1 · v (a1 ) 1n 0 · 1 · v (a2 ) 1n 0 · · · 1 · v (ad ) 1n 0 · 0 · 1kP 0, where a∗0 , . . . , a∗d ∈ D1 are the coefficients of the polynomial P ∗ . We now formally introduce the TMNU that computes FP,n (x), for any polynomial P with dyadic coefficients, any n ∈ N0 , and any x ∈ [−1, 1]. This machine iterates π across all coefficients and then rescales the result. Definition C.11. We let Π be the TMNU with neural dimension 16 and 11 states, defined by the following procedure. • State 1-8. Use π (Def. C.9) as subroutine with correspondence (1, . . . , 9; 1, . . . , 16). • State 9. Let b ∈ {□, 0, 1} under scan. i. If b ∈ {0, □}, write □, move right, update the neural state as ω ← (ω1 , ω3 , 014 ), and go to State 10. ii. If b = 1, write □, move right and go to State 1. • State 10. Use S (Def. C.5) as subroutine with correspondence (10, 11; 2). • State 11. Halt. The next lemma verifies the complete polynomial-evaluation cycle and its bound. Lemma C.12. Let n ∈ N0 , P ∈ D[X], v ∈ {0, 1}# , and x ∈ [−1, 1]. Define the configurations c := (1; |uP,n v; x, 1, 014 ), and c′ := (11; |v; x, FP,n (x), 014 ). Define T := ℓ (uP ) + ∥P ∥0 (2n + 5). Then, ΠT (c) = c′ ,
∥c∥TΠ ≤ 1 ∨ |FP,n (x)|.
(157)
Proof. Let n ∈ N0 , P a polynomial of degree d ∈ N with coefficients a0 , . . . , ad ∈ D, v ∈ {0, 1}# , and x ∈ [−1, 1]. Let b0 , . . . , bd ∈ D1 be the coefficients of P ∗ , and define the polynomials P0∗ = 0, and ∗ P1∗ , . . . , Pd+1 by Pi∗ : x 7→ b0 x0 + b1 x1 + · · · + bi−1 xi−1 , i ∈ {1, . . . , d + 1}. 49
Define also and u
(d+1)
u(i) := v (bi ) 1n 0 · 1 · v (bi+1 ) 1n 0 · · · 1 · v (bd ) 1n 0, = ϵ. Note that u
(0)
·0·1
kP
i ∈ {0, . . . , d},
0 = uP,n . For t ∈ {0, . . . , d}, set Tt := ℓ v (bt ) + 2n + 5,
and set TS := kP + 1 and T := ℓ (uP ) + ∥P ∥0 (2n + 5). We define the configurations ct of Π for t ∈ {0, . . . , 2d + 3} by c2t c2t+1 c2d+1 c 2d+2 c2d+3
:= 1; |u(t) 01kP 0v, |; x, Gt,n (x), FPt∗ ,n (x), 013 ,
t ∈{0, . . . , d},
∗ ,n (x), 013 := 9; |1u(t+1) 01kP 0v, |; x, Gt+1,n (x), FPt+1 , := 9; |01kP 0v, |; x, Gd+1,n (x), FP ∗ ,n (x), 013 , := 10; |1kP 0v; x, FP ∗ ,n (x), 014 , := (11; |v; x, FP,n (x), 014 ) .
t ∈ {0, . . . , d − 1}, (158)
Claim. The following four statements hold. (a) For all t ∈ {0, . . . , d}, ΠTt (c2t ) = c2t+1 and ∥c2t ∥TΠt ≤ 1. (b) For all t ∈ {0, . . . , d − 1}, Π(c2t+1 ) = c2t+2 and ∥c2t+1 ∥1Π ≤ 1. (c) Π(c2d+1 ) = c2d+2 and ∥c2d+1 ∥1Π ≤ 1. (d) ΠTS (c2d+2 ) = c2d+3 and ∥c2d+2 ∥TΠS ≤ 1 ∨ |FP,n (x)|. Proof of the Claim. We prove each statement in turn. (a) Let t ∈ {0, . . . , d}. Note that we can write c2t and c2t+1 as c2t = (1; |v (bt ) 1n 0ṽ; x, Gt,n (x), z, 013 ), and
c2t+1 = (9; |ṽ; x, Gt+1,n (x), z + bt Gt,n (x), 013 ),
where z := FPt∗ ,n (x), and ṽ := 1u(t+1) 01kP 0v if t < d, and ṽ := 01kP 0v if t = d. Therefore, by Lemma C.11, we have π Tt (c2t ) = c2t+1 and (a)
∗ ∥c2t ∥Tπt ≤ 1 ∨ (|z| + |bt |) ≤ 1 ∨ (∥Pt∗ ∥1 + |bt |) ≤ 1 ∨ ∥Pt+1 ∥1 ≤ 1 ∨ ∥P ∗ ∥1 ≤ 1,
(159)
where (a) follows by |z| = |FPt∗ ,n (x)| ≤ ∥Pt∗ ∥1 (Lemma C.10). Since π is a subroutine of Π with correspondence (1, . . . , 9; 1, . . . , 16) by Definition C.11, and since Sh(1,...,9),(1,...,16) (c2t ) = c2t and Lift∅ (1,...,9),(1,...,16) (c2t+1 ) = c2t+1 , Lemma C.2 gives ΠTt (c2t ) = c2t+1 ,
∥c2t ∥TΠt ≤ ∥c2t ∥Tπt ∨ ∥c2t ∥ ≤ 1.
(b) Let t ∈ {0, . . . , d − 1}. In configuration c2t+1 , the machine is in State 9 and reads the symbol 1, so after one step, the machine is in configuration c2(t+1) . Moreover, we have ∥c2t+1 ∥1Π ≤ ∥c2t+1 ∥ ∨ ∥c2t+2 ∥ ≤ 1. (c) In configuration c2d+1 , the machine is in State 9 and reads the symbol 0, so after one step, the machine is in configuration c2d+2 . Moreover, we have ∥c2d+1 ∥1Π ≤ ∥c2d+1 ∥ ∨ ∥c2d+2 ∥ ≤ 1. (d) Define the configurations c′2d+2 := (1; |1kP 0v; FP ∗ ,n (x)), and c′2d+3 := (2; |v; 2kP FP ∗ ,n (x)). By Lemma C.4, we have STS (c′2d+2 ) = c′2d+3 ,
∥c′2d+2 ∥TS S ≤ |2kP FP ∗ ,n (x)| = |FP,n (x)|.
Since S is a subroutine of Π with correspondence (10, 11; 2) by Definition C.11, and since Sh(10,11),(2) (c2d+2 ) = c′2d+2 ,
50
(x,0
)
14 Lift(10,11),(2) (c′2d+3 ) = c2d+3 ,
Lemma C.2 gives ΠTS (c2d+2 ) = c2d+3 ,
∥c2d+2 ∥TΠS ≤ ∥c′2d+2 ∥TS S ∨ ∥c2d+2 ∥ ≤ 1 ∨ |FP,n (x)|.
This concludes the proof of the claim. By invoking repeatedly the Chaining Lemma C.1 together with the claim, and using c = c0 and c′ = c2d+3 , we obtain ΠT (c) = c′ , where T =
d X
Tt + d + 1 + TS
t=0
=
d X ℓ v (bt ) + (d + 1)(2n + 5) + d + 1 + kP + 1 t=0
= ℓ (uP ) + ∥P ∥0 (2n + 5). Moreover,
C.4
∥c∥TΠ ≤ 1 ∨ |FP,n (x)|.
Approximation of continuous functions by TMNUs
The final piece is to construct a TMNU that, given the encoding of a sequence P := (Pi )i∈N0 ∈ D[X] of polynomials with dyadic coefficients, a sequence of natural numbers n := (ni )i∈N0 initially written on its tape, and its neural state initially containing some x ∈ [−1, 1], computes successively FP0 ,n0 , FP1 ,n1 , FP2 ,n2 , . . .. More precisely, we encode the sequence P and n on the tape of the machine as follows. Definition C.12. Let P := (Pi )i∈N0 ∈ D[X] be a sequence of polynomials with dyadic coefficients, and n := (ni )i∈N0 be a sequence of natural numbers. We define the encoding of P and n by ⟨P, n⟩ := uP0 ,n0 uP1 ,n1 uP2 ,n2 · · · . For every k ∈ N0 , we let P≥k := (Pi+k )i∈N0 and n≥k := (ni+k )i∈N0 . Note that ⟨P≥k , n≥k ⟩ = uPk ,nk uPk+1 ,nk+1 · · · . This TMNU has neural dimension of 17, and uses the 16 first dimensions to simulate the TMNU Π, and the last dimension is designed to store the value of the last computation of Π. The formal definition of such a machine is given as follows. Definition C.13. We let C be the TMNU with neural dimension 17 and 12 states defined by the following procedure. • State 1. Update the neural state as ω ← (ω1 , 1, 014 , ω2 ), and go to State 2. • State 2-11. Use Π (Def. C.11) as subroutine with correspondence (2, . . . , 11, 1; 1, . . . , 16). • State 12. Halt. The following lemma describes one full cycle of this infinite polynomial-evaluation machine. Lemma C.13. Let P := (Pi )i∈N0 be a sequence of polynomials, n := (ni )i∈N0 be a sequence of natural numbers, and x ∈ [−1, 1]. For all i ∈ N0 , define the configuration ci := 1; |⟨P≥i , n≥i ⟩; x, FPi−1 ,ni−1 (x), 014 , FPi−2 ,ni−2 (x) , with the convention that FP−1 ,n−1 (x) := FP−2 ,n−2 (x) := 0. Note that, in particular, c0 := (1; |⟨P, n⟩; x, 016 ). Define Ti := ℓ (uPi ) + ∥Pi ∥0 (2ni + 5) + 1. (160) Then, CTi (ci ) = ci+1 ,
∥ci ∥TCi ≤ 1 ∨ sup |FPj ,nj (x)|. j∈N0
51
(161)
In particular, by denoting Ni :=
Pi−1
j=0 Tj , we have C
C⟨P,n⟩ x[t] = FPi−1 ,ni−1 (x),
Ni
(c0 ) = ci , and
for all t ∈ {Ni + 1, . . . , Ni+1 }, i ∈ N0 .
(162)
Proof. Let P := (Pi )i∈N0 be a sequence of polynomials, n := (ni )i∈N0 be a sequence of natural numbers, and x ∈ [−1, 1]. For all i ∈ N0 , define the configuration ci := 1; |⟨P≥i , n≥i ⟩; x, FPi−1 ,ni−1 (x), 014 , FPi−2 ,ni−2 (x) . We also define the configuration c̃i by c̃i := 2; |⟨P≥i , n≥i ⟩; x, 1, 014 , FPi−1 ,ni−1 (x) . Fix i ∈ N0 , and set
Si := ℓ (uPi ) + ∥Pi ∥0 (2ni + 5),
and
Ti := Si + 1.
If C is in configuration ci , then by definition it goes to configuration c̃i in one transition. Therefore, C(ci ) = c̃i , and ∥ci ∥1C ≤ ∥(x, FPi−1 ,ni−1 (x), 014 , FPi−2 ,ni−2 (x))∥∞ ∨ ∥(x, 1, 014 , FPi−1 ,ni−1 (x))∥∞ (a)
≤ 1 ∨ |FPi−2 ,ni−2 (x)| ∨ |FPi−1 ,ni−1 (x)|,
where (a) follows from |x| ≤ 1. We now analyze the Π subroutine started from c̃i . Define the configurations c̃′i and c′i+1 by c̃′i := (1; |⟨P≥i , n≥i ⟩; x, 1, 014 ) , and
c′i+1 := (11; |⟨P≥i+1 , n≥i+1 ⟩; x, FPi ,ni (x), 014 ) .
Note that c̃′i can be rewritten as c̃′i = (1; |uPi ,ni ⟨P≥i+1 , n≥i+1 ⟩; x, 1, 014 ). Therefore, by Lemma C.12, we have ΠSi (c̃′i ) = c′i+1 , ∥c̃′i ∥SΠi ≤ 1 ∨ |FPi ,ni (x)|. Since Π is a subroutine of C with correspondence (2, . . . , 11, 1; 1, . . . , 16) by Definition C.13, and since Sh(2,...,11,1),(1,...,16) (c̃i ) = c̃′i , Lemma C.2 yields
FP
(x)
i−1 i−1 Lift(2,...,11,1),(1,...,16) (c′i+1 ) = ci+1 , ,n
CSi (c̃i ) = ci+1 ,
and, for all t ∈ {0, . . . , Si },
π17 πω Ct (c̃i ) = FPi−1 ,ni−1 (x).
(163)
Moreover, by Lemma C.2, we have ∥c̃i ∥SCi ≤ ∥c̃′i ∥SΠi ∨ ∥c̃i ∥ ≤ 1 ∨ |FPi−1 ,ni−1 (x)| ∨ |FPi ,ni (x)|. By the Chaining Lemma C.1, we obtain and
CTi (ci ) = ci+1 ,
∥ci ∥TCi ≤ 1 ∨ |FPi−2 ,ni−2 (x)| ∨ |FPi−1 ,ni−1 (x)| ∨ |FPi ,ni (x)| ≤ 1 ∨ sup |FPj ,nj (x)|. j∈N0
We now move to the proof of the last statement. For i ∈ N0 , define Ni := CNi (c0 ) = ci ,
Pi−1
j=0 Tj . By induction,
for all i ∈ N0 .
(164)
Now, fix i ∈ N0 and t ∈ {Ni + 1, . . . , Ni+1 }. Then s := t − Ni − 1 belongs to {0, . . . , Si }, and (164)
(a)
Ct (c0 ) = Cs (C(CNi (c0 ))) = Cs (C(ci )) = Cs (c̃i ),
52
where (a) is by C(ci ) = c̃i . Therefore, by Definition 4.5, (a)
C⟨P,n⟩ x[t] = π17 πω Ct (c0 ) = π17 πω Cs (c̃i ) = FPi−1 ,ni−1 (x), where (a) is by (163). This concludes the proof of the last statement. We finally close this section by showing that the TMNU C can approximate uniformly any continuous function f : [−1, 1] → R, and is uniformly bounded over its computations. This gives the qualitative approximation theorem at the TMNU level. Theorem C.14. For every continuous function f : [−1, 1] → R, there exists u ∈ {0, 1}N such that lim ∥Cu (x)[t] − f (x)∥L∞ ([−1,1]) = 0,
t→∞
and (1; |u; x, 016 ) ∈ BC∞ 1 + ∥f ∥L∞ ([−1,1]) . Proof. Let f : [−1, 1] → R be a continuous function. By the Weierstrass approximation theorem and density of dyadic numbers in R, there exists a sequence of polynomials with dyadic coefficients (Pi )i∈N0 ∈ D[X] such that ∥f − Pi ∥L∞ ([−1,1]) ≤ 2−i−1 for all i ∈ N0 . Further, we define n := (ni )i∈N0 := (i + ⌈(kPi + log2 ∥Pi ∥0 )/2⌉)i∈N0 . Note that (a)
|FPi ,ni − Pi (x)| ≤ ∥Pi ∥1 ∥Pi ∥0 2−2ni −1 ≤ ∥Pi ∥1 ∥Pi ∥0 2−2i−kPi −log2 ∥Pi ∥0 −1 ≤ 2−2i−1 ,
(165)
where (a) follows by Lemma C.10. By Lemma C.13, we have that for every i ∈ N0 and t ∈ {Ni + 1, . . . , Ni+1 }, (165)
|C⟨P,n⟩ x[t] − f (x)| ≤ |C⟨P,n⟩ x[t] − Pi (x)| + |Pi (x) − f (x)| ≤ |FPi ,ni − Pi (x)| + 2−i−1 ≤ 2−i . Upon noting that t ≥ Ni + 1 ⇐⇒ t ∈ ∪j≥i {Nj + 1, . . . , Nj+1 }, we get that for every i ∈ N0 , t ≥ Ni + 1, |C⟨P,n⟩ x[t] − f (x)| ≤ sup 2−j = 2−i , for all x ∈ [−1, 1]. j≥i
For the second statement, set Cf := 1 + ∥f ∥L∞ ([−1,1]) . First note that, for every x ∈ [−1, 1], 1 ∨ sup |FPi ,ni (x)| ≤ 1 ∨ sup (|FPi ,ni (x) − Pi (x)| + |Pi (x) − f (x)| + |f (x)|) i∈N0
i∈N0
≤ 1 ∨ sup (2−2i−1 + 2−i−1 + |f (x)|) ≤ Cf . i∈N0
Let c0 := (1; |⟨P, n⟩; x, 016 ). With the notation of Lemma C.13, we have CNi (c0 ) = ci and ∥ci ∥TCi ≤ Cf for every i ∈ N0 . Hence, for every t ∈ N0 , either t = 0, in which case ∥Ct (c0 )∥ = ∥c0 ∥ ≤ 1 ≤ Cf , or there exist i ∈ N0 and s ∈ {1, . . . , Ti } such that t = Ni + s, and then ∥Ct (c0 )∥ = ∥Cs (ci )∥ ≤ ∥ci ∥TCi ≤ Cf . Therefore, c0 ∈ BC∞ (Cf ). This concludes the proof. Note that in the above proof, we do not know a priori the properties sequence of approximating polynomials given by the Weierstrass approximation theorem, therefore, the speed of convergence of the TMNU C to f is not known. In the remainder of this section, we analyze the convergence rate of C to f when f is a polynomial with arbitrary real coefficients. We first extend the definitions given for dyadic polynomials to polynomials with real coefficients. Specifically, given P := a0 + a1 X + a2 X 2 + . . . + ad X d ∈ R[X], we Pd define ||P ∥0 := d + 1, ∥P ∥1 := i=0 |ai |, kP := ⌊log2 ∥P ∥1 ⌋ + 1 and P ∗ := 2−kP P . Note that ∥P ∥1 < 2kP , and hence ∥P ∗ ∥1 < 1. For a ∈ [0, 1] and n ∈ N0 , we define the dyadic approximation of a at precision n by a⇂n := s(a) Note that
⌊2n |a|⌋ ∈ 2−n Z ⊆ D. 2n
|a − 2−n | < |a⇂n | ≤ |a|.
(166)
Given P = a0 + a1 X + a2 X 2 + . . . + ad X d ∈ R[X] and n ∈ N0 , we define the dyadic approximation of P
53
at precision n by P⇂n :=
Pd
j=0 (aj )⇂n X
j
∈ 2−n Z[X]. Note that
∥P⇂n −P ∥L∞ ([−1,1]) ≤ ∥P ∥1 (d + 1)2−n , ∥P⇂n ∥1 ≤ ∥P ∥1
and
∥P⇂n ∥0 ≤ ∥P ∥0 .
(167)
Before proving polynomial rates, we need a simple bound on the length of the polynomial encoding. Lemma C.15. Let n ∈ N0 , and P ∈ 2−n Z[X] be a polynomial of degree d. Then, we have ℓ (uP ) ≤ (d + 1)(2(n + kP ) + 2) + 1 + kP . P Proof. Let n ∈ N0 , and P ∈ 2−n Z[X] be a polynomial of degree d. Then, P ∗ = 2−kP P ∈ 2−n−k Z[X] ∩ + −n−kP −n−kP (|a|) D1 [X]. Now, for every a ∈ 2 Z ∩ D1 , we have |a| ∈ 2 Z ∩ D1 , and hence ℓ u ≤ n + kP , which implies that ℓ v (a) = 2ℓ u(|a|) + 2 ≤ 2(n + kP ) + 2. (168)
Therefore, by definition of uP , we have ℓ (uP ) =
d X ℓ v (bj ) + 2(d + 1) + 1 + kP ≤ (d + 1)(2(n + kP ) + 2) + 1 + kP . j=0
We now are ready to prove the main result of this section, which gives an upper bound on the convergence rate of C to a polynomial P with real coefficients. Theorem C.16. Let P ∈ R[X] be a polynomial of degree d. Then, there exists u ∈ {0, 1}N such that for all x ∈ [−1, 1] and t ≥ τP , − t |Cu x[t] − P (x)| ≤ 2 · 2 tP , where tP := 18(d + 1) and τP := tP (6kP + d + 11) + 1, and (1; |u; x, 016 ) ∈ BC∞ 1 + ∥P ∥L∞ ([−1,1]) ,
for all x ∈ [−1, 1].
Proof. Let P ∈ R[X] be a polynomial of degree d. Define n := (ni )i∈N0 by ni := 2i+1 +kP +⌈log2 (d+1)⌉+1 and P := (Pi )i∈N0 by Pi := P⇂ni . Set u := ⟨P, n⟩. Note that |FPi−1 ,ni−1 (x) − P (x)| ≤ |FPi−1 ,ni−1 (x) − Pi−1 (x)| + |Pi−1 (x) − P (x)| (a)
≤ ∥Pi−1 ∥1 ∥Pi−1 ∥0 2−2ni−1 −1 + ∥P ∥1 (d + 1)2−ni−1
≤ 2∥P ∥1 (d + 1)2−ni−1 ≤ 2−2
i
(169)
for every i ∈ N0 and x ∈ [−1, 1], where (a) follows by Lemma C.10. By Lemma C.13, there exists a nondecreasing sequence (Ni )i∈N0 ∈ N such that i
|Cu x[t] − P (x)| = |FPi−1 ,ni−1 (x) − P (x)| ≤ 2−2 , for all t ∈ {Ni + 1, . . . , Ni+1 }, i ∈ N0 , (170) S which upon noting that t ≥ Ni + 1 ⇐⇒ t ∈ j≥i {Nj + 1, . . . , Nj+1 }, delivers that for all t ≥ Ni + 1, j
i
|Cu x[t] − P (x)| ≤ sup 2−2 = 2−2 . j≥i
Moreover, Lemma C.13 also delivers that for all i ∈ N0 , Ni ≤
i−1 X
ℓ uPj + ∥Pj ∥0 (2nj + 5) + 1
j=0
where by noting that Pj ∈ 2j Z and applying Lemma C.15, we get ℓ uPj ≤ (d + 1)(2(nj + kPj ) + 2) + 1 + kPj ,
54
and hence, by injecting nj ≤ 2j+1 + kP + d + 2, and noting that kPj ≤ kP and ∥Pj ∥0 ≤ d + 1, we get Ni ≤
i−1 X
(d + 1) 4 2j+1 + kP + d + 2 + kP + 7 + 2 + kP
j=0 i−1 X ≤ (d + 1) 8 · 2j + 6kP + d + 11 ≤ 9(d + 1)2i , j=0
for i ≥ αP := log2 (6kP + d + 11). Now, we let t̃P := 9(d + 1), and note that for every t ≥ τP := t̃P 2αP +1 + 1 = 18(d + 1)(6kP + d + 11) + 1, we have ⌊log2 ((t − 1)/t̃P )⌋ ≥ ⌊αP + 1⌋ ≥ αP , and hence N⌊log2 ((t−1)/t̃P )⌋ + 1 ≤ t̃P 2⌊log2 ((t−1)/t̃P )⌋ + 1 ≤ t̃P 2log2 ((t−1)/t̃P ) + 1 = t, so that, by (169), ⌊log2 ((t−1)/t̃P )⌋
|Cu x[t] − P (x)| ≤ 2−2
≤ 2−2
log2 ((t−1)/t̃P )−1
− 2t−1 t̃
=2
P
− tt
=2
P
1
· 2 tP ≤ 2 · 2
− tt
P
.
Finally, set CP := 1 + ∥P ∥L∞ ([−1,1]) and fix x ∈ [−1, 1]. By (169), applied with i + 1 in place of i, we have sup |FPi ,ni (x) − P (x)| ≤ 1. i∈N0
Hence,
1 ∨ sup |FPi ,ni (x)| ≤ 1 ∨ (1 + |P (x)|) ≤ CP . i∈N0
Let c0 := (1; |u; x, 016 ). With the notation of Lemma C.13, we have CNi (c0 ) = ci and ∥ci ∥TCi ≤ CP for every i ∈ N0 . Therefore, for every t ∈ N0 , either t = 0, in which case ∥Ct (c0 )∥ = ∥c0 ∥ ≤ 1 ≤ CP , or there exist i ∈ N0 and s ∈ {1, . . . , Ti } such that t = Ni + s, and then ∥Ct (c0 )∥ = ∥Cs (ci )∥ ≤ ∥ci ∥TCi ≤ CP . This proves that c0 ∈ BC∞ (CP ).
C.5
Native evaluation of Chebyshev expansions
In this section, we show how to implement a native evaluation of Chebyshev expansions using the TMNU × as a subroutine. The main idea is to use the recurrence relation of Chebyshev polynomials to evaluate the expansion without explicitly computing the polynomials themselves. Namely, we use the recurrence relation T0 (x) = 1, T1 (x) = x, Tn+1 (x) = 2xTn (x) − Tn−1 (x), n ∈ N0 , (171) to compute the Chebyshev polynomials iteratively. This allows us to evaluate the Chebyshev partial Pn (f ) (f ) (f ) sums Sn (x) = j=0 cj Tj (x) efficiently, using the coefficients cj obtained from the Chebyshev series representation of the function f . Before proceeding, we introduce some notation to simplify the presentation.For n ∈ N0 , write Multn (x, y) := R× D(x, y)[n],
(x, y) ∈ [−1, 1]2 ,
and define the approximate normalized Chebyshev sequence by Q0,n (x) :=
1 , 2
Q1,n (x) :=
x , 2
Qk+1,n (x) := 2 Multn (x, Qk,n (x)) − Qk−1,n (x).
(172)
The normalization by 1/2 leaves room for the numerical error while calling × . The next machine implements one step of this normalized recurrence. Definition C.14. We let TStep be the TMNU with neural dimension 17 and 6 states defined by the following procedure. • State 1. Update the neural state as ω ← (ω1 , ω2 , ω3 , ω1 , ω3 , 012 ), and go to State 2. 55
• States 2–5. Use × as a subroutine with correspondence (2, 3, 4, 5; 4, . . . , 17). • State 5. Update the neural state as ω ← (ω1 , ω3 , 2ω4 − ω2 , 014 ), and go to State 6. • State 6. Halt. We first verify the effect and boundedness of a single Chebyshev step. Lemma C.17. Let n ∈ N0 , v ∈ {0, 1}# , x, a, b ∈ [−1, 1], and define eb := 2 Multn (x, b) − a. For the configurations c := (1; |1n 0v; x, a, b, 014 ), and T := 2n + 5, we have Proof. Let
TStepT (c) = c′ , T× := 2n + 3,
c′ := (6; |v; x, b, eb, 014 ), ∥c∥TTStep ≤ 3.
T := T× + 2 = 2n + 5.
We define the following four configurations of TStep: c0 := (1; |1n 0v; x, a, b, 014 ), c := (2; |1n 0v; x, a, b, x, b, 0 ), 1 12 c := (5; |v; x, a, b, Mult (x, b), 013 ), 2 n c3 := (6; |v; x, b, 2 Multn (x, b) − a, 014 ).
(173)
Note that c0 = c and c3 = c′ . We prove separately the three stages of the computation. Claim. The following statements hold: (a) TStep(c0 ) = c1 and ∥c0 ∥1TStep ≤ 1. T
× (b) TStepT× (c1 ) = c2 and ∥c1 ∥TStep ≤ 1.
(c) TStep(c2 ) = c3 and ∥c2 ∥1TStep ≤ 3. Proof of the Claim. We prove each statement in turn. (a) In configuration c0 , the machine is in State 1. Therefore, by Definition C.14, it preserves the first three neural coordinates, copies x and b into coordinates 4 and 5, sets all remaining coordinates to zero, and goes to State 2. Hence, TStep(c0 ) = c1 . Moreover, since x, a, b ∈ [−1, 1], ∥c0 ∥1TStep ≤ ∥(x, a, b, 014 )∥∞ ∨ ∥(x, a, b, x, b, 012 )∥∞ ≤ 1. (b) Define the configurations of × c′1 := (1; |1n 0v; x, b, 012 ),
c′2 := (4; |v; Multn (x, b), 013 ).
By Lemma C.8, × T× (c′1 ) = c′2 ,
T
∥c′1 ∥×× ≤ 1.
By Definition C.14, × is a subroutine of TStep with correspondence (2, 3, 4, 5; 4, . . . , 17). Furthermore, Sh(2,3,4,5),(4,...,17) (c1 ) = c′1 , and
(x,a,b)
Lift(2,3,4,5),(4,...,17) (c′2 ) = c2 .
Therefore, Lemma C.2 yields (x,a,b) TStepT× (c1 ) = Lift(2,3,4,5),(4,...,17) × T× (c′1 ) = c2 , 56
and
T
T
× ∥c1 ∥TStep ≤ ∥c′1 ∥×× ∨ ∥c1 ∥ ≤ 1.
(c) In configuration c2 , the machine is in State 5. By Definition C.14, it preserves x, moves b into the second coordinate, places 2 Multn (x, b) − a in the third coordinate, sets the remaining coordinates to zero, and goes to State 6. Thus, TStep(c2 ) = c3 . Lemma C.7 gives | Multn (x, b)| ≤ 1. Consequently, ∥c2 ∥1TStep ≤ ∥c2 ∥ ∨ ∥c3 ∥ ≤ 1 ∨ |2 Multn (x, b) − a| ≤ 3. This concludes the proof of the claim. By repeatedly applying the Chaining Lemma C.1 to the three stages of the claim, we obtain TStepT (c) = c′ , where T = 1 + T× + 1 = 2n + 5. Moreover, T
× ∥c∥TTStep ≤ ∥c0 ∥1TStep ∨ ∥c1 ∥TStep ∨ ∥c2 ∥1TStep ≤ 3.
The next estimate translates the approximate multiplication error into a recurrence error. Lemma C.18. Let n ∈ N0 , k ∈ N, x, a, b ∈ [−1, 1], and define eb := 2 Multn (x, b) − a. Define also the incoming errors E− := a −
Tk−1 (x) , 2
E := b −
Tk (x) , 2
and the local multiplication error εn (x, b) := Multn (x, b) − xb. Then,
eb − Tk+1 (x) = 2xE − E− + 2εn (x, b), 2
Proof. By definition,
|εn (x, b)| ≤ 2−2n−1 .
Multn (x, b) = xb + εn (x, b).
Since x, b ∈ [−1, 1], Lemma C.7 gives |εn (x, b)| = | Multn (x, b) − xb| ≤ 2−2n−1 . Using a=
Tk−1 (x) + E− , 2
b=
Tk (x) + E, 2
we obtain eb = 2 Multn (x, b) − a Tk−1 (x) = 2(xb + εn (x, b)) − + E− 2 Tk (x) Tk−1 (x) = 2x + E + 2εn (x, b) − + E− 2 2 2xTk (x) − Tk−1 (x) = + 2xE − E− + 2εn (x, b) 2 Tk+1 (x) = + 2xE − E− + 2εn (x, b). 2 Rearranging proves (174). 57
(174)
Iterating the previous recurrence gives a uniform stability bound for the normalized Chebyshev values. The key is to remark that the error propagation follows the same recurrence as the Chebyshev polynomials of the second kind, defined by the recurrence U0 (x) = 1,
U1 (x) = 2x,
Ur+1 (x) = 2xUr (x) − Ur−1 (x),
r ∈ N,
(175)
for x ∈ [−1, 1]. We show the following classical boundedness property of the Chebyshev polynomials of the second kind. Lemma C.19. For r ∈ N,
sup |Ur (x)| = r + 1. x∈[−1,1]
Proof. The proof follows by first showing a useful formula for Ur (x), and then using it to find the maximum value of |Ur (x)| on the interval [−1, 1]. We show directly from the recurrence that, for every θ ∈ [0, π], Ur (cos(θ)) =
r X
ei(r−2ℓ)θ .
ℓ=0
For r = 0, this says U0 (cos(θ)) = 1, which is true. For r = 1, this says U1 (cos(θ)) = eiθ + e−iθ = 2 cos(θ), which is also true. For the inductive step, assume the formula holds at the two consecutive indices r and r − 1. Then the recurrence (175) gives Ur+1 (cos(θ)) = 2 cos(θ)Ur (cos(θ)) − Ur−1 (cos(θ)) = (eiθ + e−iθ )
r X
ei(r−2ℓ)θ −
ℓ=0
=
r X
=
ei(r−1−2ℓ)θ
ℓ=0
ei(r+1−2ℓ)θ +
ℓ=0 r X
r−1 X
r X
ei(r−1−2ℓ)θ −
ℓ=0
r−1 X
ei(r−1−2ℓ)θ
ℓ=0
ei(r+1−2ℓ)θ + e−i(r+1)θ
ℓ=0
=
r+1 X
ei(r+1−2ℓ)θ .
ℓ=0
This completes the induction. Now, now that sup |Ur (x)| = sup |Ur (cos(θ))| = sup x∈[−1,1]
θ∈[0,π]
r X
ei(r−2ℓ)θ ≤
θ∈[0,π] ℓ=0
r X
|ei(r−2ℓ)θ | = r + 1.
ℓ=0
We are now ready to prove the stability of the normalized Chebyshev sequence. Lemma C.20. Let k, n ∈ N0 satisfy
k ≤ 2n .
Then, |Qk,n (x)| ≤ 1,
Qk,n (x) −
(176)
Tk (x) ≤ k 2 2−2n−1 , 2
x ∈ [−1, 1].
Proof. Fix n ∈ N0 . For every k ∈ N0 , define Ek,n (x) := Qk,n (x) −
Tk (x) , 2
and the local multiplication error εk,n (x) := Multn (x, Qk,n (x)) − xQk,n (x). We claim that, for every k ∈ {0, . . . , 2n }, |Qk,n (x)| ≤ 1,
Ek,n (x) = 2
k−1 X j=1
58
Uk−j−1 (x)εj,n (x),
(177)
for x ∈ [−1, 1], where Ur , r ∈ N0 are the Chebyshev polynomials of the second kind defined by (175). We prove the claim by induction on k. For k = 0, 1, the sum is empty, and (171) and (172) yield E0,n (x) = E1,n (x) = 0 and |Q0,n (x)| = |Q1,n (x)| ≤ 1/2. This verifies the claim at the base cases. Now assume that (177) holds at the two consecutive indices k − 1 and k such that k < 2n . Then, by Lemma C.18, Ek+1,n (x) = 2xEk,n (x) − Ek−1,n (x) + 2εk,n (x) = 4x
k−1 X
Uk−j−1 (x)εj,n (x) − 2
j=1
=2
k−2 X
k−2 X
Uk−j−2 (x)εj,n (x) + 2εk,n (x)
j=1
(2xUk−j−1 (x) − Uk−j−2 (x)) εj,n (x) + 4xU0 (x)εk−1,n (x) + 2εk,n (x)
j=1
=2
k−2 X
Uk−j (x)εj,n (x) + 2U1 (x)εk−1,n (x) + 2U0 (x)εk,n (x)
j=1
=2
k X
Uk−j (x)εj,n (x).
j=1
Now, since by Lemma C.19, |Ur (x)| ≤ r + 1 on [−1, 1], we have |Ek,n (x)| ≤ 2−2n
k−1 X
(k − j) ≤ k 2 2−2n−1 .
j=1
Moreover, since |Tk (x)| ≤ 1 on [−1, 1], |Qk,n (x)| ≤
(a) 1 Tk (x) 1 1 + |Ek,n (x)| ≤ + k 2 2−2n−1 ≤ + = 1, 2 2 2 2
where (a) is by the assumption k ≤ 2n . This closes the induction and proves the claim. Finally, the error bound follows from the claim and the bound on |Ur (x)|. For a dyadic vector a = (a0 , . . . , ad ) ∈ Dd+1 , define Ha,n (x) := 2
d X
ak Qk,n (x).
(178)
k=0
If a ̸= 0, let
ka := 1 + (⌊log2 (2∥a∥1 )⌋ ∨ 0) ,
a∗ := 2−ka a,
(179)
and set k0 := 0 and 0 := 0. Thus, 2∥a ∥1 < 1. Define the encoding ∗
∗
∗
∗
∗
(a0 ) n uT 1 0 1 v (a1 ) 1n 0 1 · · · 1 v (ad ) 1n 0 0 1ka 0. a,n := v
The summation machine now loops over the encoded coefficients and calls TStep after each accumulation. Definition C.15. We let TSum be the TMNU with neural dimension 18 and 13 states defined by the following procedure. • State 1. Update the neural state as ω ← (ω1 , ω2 , ω3 , ω4 , 2ω2 , 013 ), and go to State 2. • States 2–4. Use Hom as a subroutine with correspondence (2, 3, 4, 5; 5, 6). • State 5. Update the neural state as ω ← (ω1 , ω2 , ω3 , ω4 + ω5 , 014 ), and go to State 6. • States 6–10. Use TStep as a subroutine with correspondence (6, . . . , 11; 1, 2, 3, 5, . . . , 18).
59
• State 11. If the scanned symbol is 1, erase it, move right, and go to State 1. If it is 0, erase it, move right, update the neural state as ω ← (ω1 , ω4 , 016 ), and go to State 12. • State 12. Use S as a subroutine with correspondence (12, 13; 2). • State 13. Halt. The next lemma verifies one complete encoded Chebyshev-sum evaluation. Lemma C.21. Let a = (a0 , . . . , ad ) ∈ Dd+1 , n ∈ N0 such that d + 1 ≤ 2n , v ∈ {0, 1}# , and x ∈ [−1, 1]. Define d X ∗ Ta,n := ℓ v (ak ) + 2n + 8 + ka + 1. (180) k=0
For 1 x c := (1; |uT a,n v; x, 2 , 2 , 0, 014 ),
c′ := (13; |v; x, Ha,n (x), 016 ),
we have
T
TSumTa,n (c) = c′ ,
a,n ∥c∥TSum ≤ 3 ∨ |Ha,n (x)|.
Proof. For every k ∈ {0, . . . , d + 1}, define zk := 2
k−1 X
a∗j Qj,n (x),
j=0
with the convention z0 := 0. Since by Lemma C.20, |Qk,n (x)| ≤ 1 for every k ∈ {0, . . . , d + 1}, we have |zk | ≤ 2∥a∗ ∥1 < 1,
k ∈ {0, . . . , d + 1}.
(181)
For every k ∈ {0, . . . , d}, let wk denote the suffix of the encoding beginning with the coefficient a∗k , namely ∗ ∗ wk := v (ak ) 1n 0 1 · · · 1 v (ad ) 1n 0 0 1ka 0v, and set wd+1 := 1ka 0v. We also define ( sk := Thus,
1wk+1 , 0wd+1 ,
k ∈ {0, . . . , d − 1}, k = d.
∗
wk = v (ak ) 1n 0sk ,
w0 = uT a,n v.
For every k ∈ {0, . . . , d}, define ∗ THom,k := ℓ v (ak ) ,
TTStep := 2n + 5,
and the following configurations of TSum: ck,0 := (1; |wk ; x, Qk,n (x), Qk+1,n (x), zk , 014 ), ck,1 := (2; |wk ; x, Qk,n (x), Qk+1,n (x), zk , 2Qk,n (x), 013 ), ck,2 := (5; |1n 0sk ; x, Qk,n (x), Qk+1,n (x), zk , 2a∗k Qk,n (x), 013 ), ck,3 := (6; |1n 0sk ; x, Qk,n (x), Qk+1,n (x), zk+1 , 014 ), c := (11; |sk ; x, Qk+1,n (x), Qk+2,n (x), zk+1 , 014 ). k,4 Finally, define cS := (12; |wd+1 ; x, zd+1 , 016 ),
cout := (13; |v; x, Ha,n (x), 016 ),
Note that c0,0 = c. Moreover, 2ka zd+1 = 2
d X
aj Qj,n (x) = Ha,n (x),
j=0
and therefore cout = c′ . 60
TS := ka + 1.
(182)
Claim. For every k ∈ {0, . . . , d}, the following statements hold: (a) TSum(ck,0 ) = ck,1 and ∥ck,0 ∥1TSum ≤ 2. T
Hom,k (b) TSumTHom,k (ck,1 ) = ck,2 and ∥ck,1 ∥TSum ≤ 2.
(c) TSum(ck,2 ) = ck,3 and ∥ck,2 ∥1TSum ≤ 2. T
TStep (d) TSumTTStep (ck,3 ) = ck,4 and ∥ck,3 ∥TSum ≤ 3.
(e) If k < d, then TSum(ck,4 ) = ck+1,0 and ∥ck,4 ∥1TSum ≤ 1. Moreover, TSum(cd,4 ) = cS ,
∥cd,4 ∥1TSum ≤ 1.
Finally, TSumTS (cS ) = cout ,
S ∥cS ∥TTSum ≤ 1 ∨ |Ha,n (x)|.
Proof of the Claim. Fix k ∈ {0, . . . , d}. (a) In configuration ck,0 , the machine is in State 1. Therefore, Definition C.15 gives TSum(ck,0 ) = ck,1 . Moreover, ∥ck,0 ∥1TSum ≤ 2, since |x|, |Qk,n (x)|, |Qk+1,n (x)| ≤ 1, |zk | < 1, and |2Qk,n (x)| ≤ 2. (b) Define the configurations of Hom c′k,1 := (1; |wk ; 2Qk,n (x), 0),
c′k,2 := (4; |1n 0sk ; 2a∗k Qk,n (x), 0).
Lemma C.9 gives T
HomTHom,k (c′k,1 ) = c′k,2 ,
Hom,k ∥c′k,1 ∥Hom ≤ 2|Qk,n (x)| ≤ 2.
Since Hom is a subroutine of TSum with correspondence (2, 3, 4, 5; 5, 6), and since Sh(2,3,4,5),(5,6) (ck,1 ) = c′k,1 ,
(x,Q
(x),Q
k,n k+1,n Lift(2,3,4,5),(5,6)
Lemma C.2 yields TSumTHom,k (ck,1 ) = ck,2 ,
(x),zk ,012 )
(c′k,2 ) = ck,2 ,
T
Hom,k ∥ck,1 ∥TSum ≤ 2.
(c) In configuration ck,2 , the machine is in State 5. Definition C.15 therefore gives TSum(ck,2 ) = ck,3 , since zk + 2a∗k Qk,n (x) = zk+1 . Furthermore, (181) and |2a∗k Qk,n (x)| ≤ 2|a∗k | ≤ 2 imply ∥ck,2 ∥1TSum ≤ 2. (d) Define the configurations of TStep c′k,3 := (1; |1n 0sk ; x, Qk,n (x), Qk+1,n (x), 014 ), and
c′k,4 := (6; |sk ; x, Qk+1,n (x), Qk+2,n (x), 014 ).
Lemma C.17 gives TStepTTStep (c′k,3 ) = c′k,4 ,
T
TStep ∥c′k,3 ∥TStep ≤ 3.
Since TStep is a subroutine of TSum with correspondence (6, . . . , 11; 1, 2, 3, 5, . . . , 18), and since Sh(6,...,11),(1,2,3,5,...,18) (ck,3 ) = c′k,3 , z
k+1 Lift(6,...,11),(1,2,3,5,...,18) (c′k,4 ) = ck,4 ,
Lemma C.2 yields TSumTTStep (ck,3 ) = ck,4 ,
T
TStep ∥ck,3 ∥TSum ≤ 3.
(e) If k < d, then sk = 1wk+1 . Thus, in configuration ck,4 , the machine is in State 11 and reads the symbol 1. It erases this symbol, moves right, and goes to State 1, so TSum(ck,4 ) = ck+1,0 . The neural state is unchanged and bounded by one, hence ∥ck,4 ∥1TSum ≤ 1.
61
Finally, sd = 0wd+1 . Therefore, in configuration cd,4 , the machine is in State 11 and reads the symbol 0. It erases this symbol, moves right, updates its neural state to (x, zd+1 , 016 ), and goes to State 12. Hence, TSum(cd,4 ) = cS , ∥cd,4 ∥1TSum ≤ 1. Now, define the configurations of S c′S := (1; |wd+1 ; x, zd+1 , 016 ), By Lemma C.4,
STS (c′S ) = c′out ,
c′out := (2; |v; x, Ha,n (x), 016 ). ∥c′S ∥TS S ≤ |Ha,n (x)|.
Since S is a subroutine of TSum with correspondence (12, 13; 2), and since
Lemma C.2 gives
(x,0
)
Sh(12,13),(2) (cS ) = c′S ,
16 Lift(12,13),(2) (c′out ) = cout ,
TSumTS (cS ) = cout ,
S ∥cS ∥TTSum ≤ 1 ∨ |Ha,n (x)|.
This concludes the proof of the claim. Repeated applications of the Chaining Lemma C.1, together with the claim now give TSumTa,n (c) = c′ ,
T
a,n ≤ 3 ∨ |Ha,n (x)|. ∥c∥TSum
Indeed, Ta,n =
d X
(1 + THom,k + 1 + TTStep + 1) + TS
k=0 d X ∗ = ℓ v (ak ) + 2n + 8 + ka + 1. k=0
For sequences A = (ai )i∈N0 of finite dyadic vectors and n = (ni )i∈N0 , define T T ⟨A, n⟩T := uT a0 ,n0 ua1 ,n1 ua2 ,n2 · · · .
The continuous Chebyshev machine repeatedly executes these finite sum instructions. Definition C.16. We let TC be the TMNU with neural dimension 19 and 14 states defined by the following procedure. • State 1. Update the neural state as ω ← (ω1 , 12 , ω21 , 015 , ω2 ), and go to State 2. • States 2–13. Use TSum as a subroutine with correspondence (2, . . . , 13, 1; 1, . . . , 18). • State 14. Halt. The following lemma records one cycle and the resulting output convention. Lemma C.22. Let A = (ai )i∈N0 be a sequence of finite dyadic vectors, let n = (ni )i∈N0 , and let x ∈ [−1, 1]. For every i ∈ N0 , write ai = (ai,0 , . . . , ai,di ), and assume that di + 1 ≤ 2ni . Define the tails A≥i := (ai+j )j∈N0 ,
n≥i := (ni+j )j∈N0 ,
and use the conventions Ha−1 ,n−1 (x) := Ha−2 ,n−2 (x) := 0. For every i ∈ N0 , define the configuration ci := 1; |⟨A≥i , n≥i ⟩T ; x, Hai−1 ,ni−1 (x), 016 , Hai−2 ,ni−2 (x) , 62
and set Ti := Tai ,ni + 1, where Tai ,ni is as in (180). Then, for every i ∈ N0 , TCTi (ci ) = ci+1 ,
(183)
i ∥ci ∥TTC ≤ 3 ∨ |Hai−2 ,ni−2 (x)| ∨ |Hai−1 ,ni−1 (x)| ∨ |Hai ,ni (x)|.
(184)
and In particular, c0 = (1; |⟨A, n⟩T ; x, 018 ). By denoting Ni :=
i−1 X
Tj ,
j=0
we have TCNi (c0 ) = ci for every i ∈ N0 , and TC⟨A,n⟩T x[t] = Hai−1 ,ni−1 (x), Proof. Fix i ∈ N0 , and define
hj := Haj ,nj (x),
t ∈ {Ni + 1, . . . , Ni+1 }.
(185)
j ∈ N0 ,
together with the conventions h−1 := h−2 := 0. We also define the intermediate configuration 1 x e ci := 2; |⟨A≥i , n≥i ⟩T ; x, , , 015 , hi−1 . 2 2 We prove separately the initialization transition and the TSum subroutine stage. Claim. The following statements hold: (a) TC(ci ) = e ci and
∥ci ∥1TC ≤ 1 ∨ |hi−2 | ∨ |hi−1 |.
(b) TCTai ,ni (e ci ) = ci+1 , and
Ta ,ni
∥e ci ∥TCi
≤ 3 ∨ |hi−1 | ∨ |hi |,
where Tai ,ni is as in (180). Moreover, for every s ∈ {0, . . . , Tai ,ni }, π19 πω TCs (e ci ) = hi−1 .
(186)
Proof of the Claim. We prove both statements in turn. (a) In configuration ci , the machine is in State 1. Therefore, by Definition C.16, it preserves x, initializes the first two normalized Chebyshev values as 1/2 and x/2, sets the accumulator and the remaining working coordinates to zero, copies hi−1 into coordinate 19, and goes to State 2. Hence, TC(ci ) = e ci . Since |x| ≤ 1, we also have ∥ci ∥1TC ≤ ∥(x, hi−1 , 016 , hi−2 )∥∞ 1 x x, , , 015 , hi−1 ∨ 2 2 ∞ ≤ 1 ∨ |hi−2 | ∨ |hi−1 |. (b) Define the configurations of TSum e c′i := and Since
1 x 1; |⟨A≥i , n≥i ⟩T ; x, , , 015 , 2 2
c′i+1 := (13; |⟨A≥i+1 , n≥i+1 ⟩T ; x, hi , 016 ) . ⟨A≥i , n≥i ⟩T = uT ai ,ni ⟨A≥i+1 , n≥i+1 ⟩T , 63
Lemma C.21 gives
TSumTai ,ni (e c′i ) = c′i+1 ,
and
Ta ,n
i i ∥e c′i ∥TSum ≤ 3 ∨ |hi |.
By Definition C.16, TSum is a subroutine of TC with correspondence (2, . . . , 13, 1; 1, . . . , 18). Furthermore, Sh(2,...,13,1),(1,...,18) (e ci ) = e c′i , and
h
i−1 Lift(2,...,13,1),(1,...,18) (c′i+1 ) = ci+1 .
Therefore, Lemma C.2 yields and
TCTai ,ni (e ci ) = ci+1 , Ta ,ni
∥e ci ∥TCi
Ta ,n
i i ≤ ∥e c′i ∥TSum ∨ ∥e ci ∥ ≤ 3 ∨ |hi−1 | ∨ |hi |.
Since coordinate 19 is not used by the TSum subroutine, the same application of Lemma C.2 gives (186). This concludes the proof of the claim. Since Ti = 1 + Tai ,ni , the Chaining Lemma C.1 and the claim give TCTi (ci ) = ci+1 , and
i ∥ci ∥TTC ≤ 3 ∨ |hi−2 | ∨ |hi−1 | ∨ |hi |.
This proves (183) and (184). We now prove the last statements. By induction on i, TCNi (c0 ) = ci ,
i ∈ N0 .
Fix i ∈ N0 and t ∈ {Ni + 1, . . . , Ni+1 }. Then s := t − Ni − 1 ∈ {0, . . . , Tai ,ni }. Using TCNi (c0 ) = ci and TC(ci ) = e ci , we obtain TCt (c0 ) = TCs (e ci ). Therefore, by Definition 4.5 and (186), TC⟨A,n⟩T x[t] = π19 πω TCs (e ci ) = hi−1 . This proves (185). We can now balance degree and precision to obtain the native Chebyshev rate. Theorem C.23 (Native Chebyshev tradeoff). Let f : [−1, 1] → R be a Dini-continuous continuous function. Let η : [1, ∞) → N be nondecreasing, with η(t) → ∞ and t/η(t) nondecreasing. Suppose that η(t) log2 (η(t) + 1) ≤ At,
t ≥ 1,
(187)
for some A > 0. Define Bf := 2∥f ∥L∞ ([−1,1]) + 1,
γf := ⌈log2 (Bf )⌉ ,
and τ := τf,A := 4 10 + γf + A(80 + 6γf ) .
(188)
Then, there exists u ∈ {0, 1} such that N
∞ (1; |u; x, 018 ) ∈ BTC 3 + S(f, 0) + ∥f ∥L∞ ([−1,1]) ,
x ∈ [−1, 1],
and, for every x ∈ [−1, 1] and t ≥ τ , t/τ
|TCu x[t] − f (x)| ≤ 2 · 2− η(t/τ ) + S (f, η(t/τ )) , 64
(189)
where S(f, d) is as in (14). Proof. First note that since f is DiniLipschitz continuous, S(f, d) → 0 as d → ∞ and S(f, d) < ∞ for every d ∈ N0 . For every i ∈ N0 , set i 2 qi + 3ℓi + γf i di := η(2 ), qi := , ℓi := ⌈log2 (di + 1)⌉ , pi := qi + ℓi + 2, ni := , (190) di 2 and for k ∈ {0, . . . , di }, choose ai,k ∈ 2−pi Z such that (f )
|ai,k − ck | ≤ 2−pi , (f )
where ck
(191)
is the k-th Chebyshev coefficient of f , and set ai := (ai,0 , . . . , ai,di ),
A := (ai )i∈N0 ,
n := (ni )i∈N0 ,
u := ⟨A, n⟩T .
The proof is effected by estimating the error and duration of each cycle of the computation. (a) We first estimate the error of cycle i ∈ N0 . Note that di + 1 ≤ 2ℓi ≤ 2ni , so Lemma C.20 applies. Therefore, we get (f ) Hai ,ni (x) − Sdi (x)
= 2
di X
di X (f ) ai,k Qni ,k (x) − ck Tk (x)
k=0
≤ 2
di X
k=0
ai,k Qni ,k (x) − 2
k=0
≤2
di X
di di di X X X (f ) (f ) (f ) ck Qni ,k (x) + 2 ck Qni ,k (x) − ck Tk (x) k=0
(f )
|ai,k − ck | + 2
k=0
di X
k=0 (f )
|ck | Qni ,k (x) −
k=0
(a)
k=0
Tk (x) 2
(190)
≤ 2di 2−pi + Bf d2i 2−2ni −1 ≤ 2−qi −1 + 2−qi −1 = 2−qi ,
where (a) follows from Lemma C.20 and (191). Hence, |Hai ,ni (x) − f (x)| ≤ 2−qi + S(f, di ),
x ∈ [−1, 1].
(192)
(b) We now derive an explicit bound on the duration of cycle i. First note that ∥ai ∥1 ≤
di X
(f ) (f ) (|ai,k − ck | + |ck |)
(191)
≤ (di + 1) 2
−pi
+
k=0
max
k∈{0,...,di }
(f ) |ck |
(11)
≤ (di + 1) 1 + 2∥f ∥L∞ ([−1,1]) = (di + 1)Bf .
(193)
From (193) and the definition (179) of kai , we have kai ≤ ℓi + γf + 2.
(194)
Since ai,k ∈ 2−pi Z, we have a∗i,k ∈ 2−pi −kai Z ∩ D1 . Therefore, ∗ (168) ℓ v (ai,k ) ≤ 2(pi + kai ) + 2.
(195)
Let Ti := Tai ,ni + 1 be the duration of cycle i in TC, where Tai ,ni is as in (180). By (194), and (195), (180)
Ti ≤ 1 +
di X ∗ ℓ v (ai,k ) + 2ni + 8 + kai + 1 k=0
(195)
≤ (di + 1)(2pi + 2kai + 2ni + 10) + kai + 2
(194)
≤ (di + 1)(2pi + 2(ℓi + γf + 2) + 2ni + 10) + ℓi + γf + 2 + 2
(190)
≤ (di + 1) (3qi + 7ℓi + 3γf + 22) + ℓi + γf + 4.
65
(196)
We bound each term in (196). First, since di = η(2i ) ≥ 1, (187) gives di log2 (di + 1) ≤ A2i ,
di ≤
A2i A2i ≤ = A2i , log2 (di + 1) log2 (2)
di + 1 ≤ 2di ≤ 2A2i .
(197)
Moreover, by (190), 2i + 1, di
qi ≤
ℓi ≤ log2 (di + 1) + 1 ≤ 2 log2 (di + 1) ≤ 2A2i .
(198)
which yields the two product bounds (di + 1)qi ≤ 2di
i 2 + 1 = 2 · 2i + 2di ≤ (2 + 2A)2i , di
(di + 1)ℓi ≤ 4di log2 (di + 1) ≤ 4A2i .
(199) (200)
Substituting (197), (198), (199) and (200) into (196), and using γf + 4 ≤ (γf + 4)2i , gives Ti ≤ 3(2 + 2A) + 7(4A) + (3γf + 22)(2A) + 2A + γf + 4 2i τ = 10 + γf + A(80 + 6γf ) 2i =: 2i . 4
(201)
Hence, with the notation of Lemma C.22, Ni + 1 =
i−1 X
Tj + 1 ≤
j=0
τ τ i (2 − 1) + 1 ≤ 2i , 4 4
(202)
where the last inequality uses τ /4 ≥ 10/4 ≥ 1. We now express the error in terms of machine time. Fix t ≥ τ , and set i := ⌊log2 (4t/τ )⌋ . Then
t τ i 2 ≤ t, and 2i−1 ≥ . 4 τ In particular, t ∈ {Nj + 1, . . . , Nj+1 } for some j ≥ i, so by Lemma C.22, i ≥ 2,
(203)
Ni + 1 ≤
(192)
|TCu x[t] − f (x)| = |Haj−1 ,nj−1 (x) − f (x)| ≤ 2−qj−1 + S(f, dj−1 ) ≤ sup
2−qk + S(f, dk )
(204)
k≥i−1
= sup
−
2
2k η(2k )
+ S(f, η(2k ))
(a) i−1 − 2 ≤ 2 η(2i−1) + S(f, η(2i−1 )),
(205)
k≥i−1
where (a) follows from the fact that η(t) and t/η(t) are nondecreasing, and t 7→ 2−t and t 7→ S(f, t) are nonincreasing. By (203), i−1 2 t/τ i−1 di−1 = η(2 ) ≥ η(t/τ ), and qi−1 ≥ ≥ . η(2i−1 ) η(t/τ ) Substituting these inequalities into (204) proves (189). Finally, by (192), for every i ∈ N0 and x ∈ [−1, 1], |Hai ,ni (x)| ≤ |Hai ,ni (x) − f (x)| + |f (x)| ≤ 1 + S(f, di ) + ∥f ∥L∞ ([−1,1]) ≤ 1 + S(f, 0) + ∥f ∥L∞ ([−1,1]) Lemma C.22 therefore gives ∞ (1; |u; x, 018 ) ∈ BTC 3 + S(f, 0) + ∥f ∥L∞ ([−1,1]) .
C.6
Magnitude estimates for the TMNUs used in the constructions
We close the appendix by recording the estimates on the quantity ∥M∥C that enters Theorem 5.4. Recall from Section 5 that, if FM = {f1 , . . . , fνM } denotes the set of command maps of a TMNU M with neural dimension d, then ∥M∥C := max sup ∥fj (ω)∥∞ . 1≤j≤νM ω∈[−C,C]d
66
We first introduce a global magnitude for command maps. If W : Rd → Rd is affine, say W (ω) = Aω +b, set d X m(W ) := max |Aij | + |bi | . 1≤i≤d
j=1
If W = ρS ∈ Rd , set m(W ) := 1. Finally, for a TMNU M, define m(M) := 1 ∨ max m(f ). f ∈FM
This global command magnitude controls the bounded-region quantity used in the simulation theorem. Lemma C.24. Let M be a TMNU. Then, for every C > 0, ∥M∥C ≤ m(M)(C ∨ 1). Proof. Let f ∈ FM and ω ∈ [−C, C]d . If f = ρS for some S ⊆ {1, . . . , d}, then ∥f (ω)∥∞ ≤ ∥ω∥∞ ≤ C ≤ C ∨ 1. If f (ω) = Aω + b is affine, then d X ∥f (ω)∥∞ ≤ max |Aij | C + |bi | ≤ m(f )(C ∨ 1). 1≤i≤d
j=1
Taking the maximum over f ∈ FM proves the claim. The next estimate explains how this magnitude behaves when a machine is built from a subroutine. Lemma C.25. Let N be a subroutine of M with correspondence (ιq , ιω ), and define the set of states of M that do not simulate non-halting states of N by Qrem := {1, . . . , nM } \ ιq ({1, . . . , nN − 1}). Then, with the convention that the maximum over an empty set is 0, m(M) ≤ m(N) ∨ 1 ∨
max
q∈Qrem , σ∈Γ
m(κM (q, σ)) .
Proof. For a command map f : RdN → RdN , let fe : RdM → RdM be the map obtained by applying f to the coordinates indexed by ιω and by leaving all complementary coordinates unchanged, i.e., πιω fe(ω) = f (πιω ω), Note that
πι⊥ω fe(ω) = πι⊥ω ω.
(206)
m(fe) ≤ m(f ) ∨ 1.
(207)
Now fix q ∈ {1, . . . , nN − 1} and σ ∈ Γ, and set f := κN (q, σ). Let fe : RdM → RdM be the extension of f defined by (206). By the subroutine identity (130), κM (ιq (q), σ) = fe. Hence, by (207),
m(κM (ιq (q), σ)) ≤ m(κN (q, σ)) ∨ 1 ≤ m(N) ∨ 1.
The remaining command maps of M are exactly those indexed by q ∈ Qrem and σ ∈ Γ. Taking the maximum over all command maps of M, and recalling the outer maximum with 1 in the definition of m(M), gives the desired estimate. We finish by tabulating the command magnitudes for every machine used above. Lemma C.26. The TMNUs used in the monomial and native Chebyshev constructions satisfy
M ± nM 2 dM 1 νM 2 m(M) 1
S 2 1 2 2
Hom+ 3 2 4 2
× 4 14 5 ≤5
Hom π 4 9 2 16 5 11 ≤2 ≤5 67
Π 11 16 13 ≤5
C TStep 12 6 17 17 14 7 ≤5 ≤5
TSum TC 13 14 18 19 15 16 ≤5 ≤ 5.
In particular, for every C > 0, ∥C∥C ≤ 5(C ∨ 1),
∥TC∥C ≤ 5(C ∨ 1).
Proof. The rows for nM and dM are read directly from Definitions C.2, C.3, C.5, C.7, C.8, C.9, C.11, C.13, C.14, C.15, and C.16. For ×, Definition C.7 identifies it with the TMNU associated to the multiplication RNN of Lemma C.7, which has hidden dimension 14; the TMNU construction of Definition C.6 has 4 states. We next count the distinct command maps. By inspection of Definitions C.2, C.3, and C.5, the elementary machines have ν± = 2, νS = 2, νHom+ = 4. Moreover, Definition C.6 gives five command maps for a simulated RNN, and therefore ν× = 5. When a machine is built by inserting subroutines, we count the union of the extended command maps of the subroutines and the additional explicit command maps; the identity command is counted only once. Inspection of the coordinates on which the non-identity parts act shows that there are no further coincidences between the command maps listed below. Thus νHom = ν± + νHom+ − 1 = 5. The machine π adds two explicit commands to the Hom and × subroutines, whence νπ = νHom + ν× − 1 + 2 = 11. Similarly, Π adds one explicit command to the π and S subroutines, and C adds one explicit command to the Π subroutine. Hence νΠ = 13, νC = 14. On the native Chebyshev side, TStep adds two explicit commands to the × subroutine, so νTStep = 5 + 2 = 7. The machine TSum combines the Hom, TStep, and S subroutines and adds three explicit commands: the initialization command, the accumulation command, and the exit command before the final scaling subroutine. Therefore νTSum = νHom + (νTStep − 1) + (νS − 1) + 3 = 15. Finally, TC adds one explicit initialization command to the TSum subroutine, and hence νTC = 16. It remains to prove the command-magnitude row. We inspect the command maps in the order in which the machines are constructed. Throughout, the command maps that are not explicitly listed are identities, or frozen complementary coordinates coming from subroutines, and therefore have magnitude 1. When a machine contains several disjoint subroutine blocks, we apply Lemma C.25 successively to these blocks and then take the maximum with the magnitudes of the remaining explicit commands. ±) = 1. • For ± , the only nontrivial command is ω 7→ ±ω, hence m(± • For S, the nontrivial command is ω 7→ 2ω, while all other commands are the identity. Hence m(S) = 2. • For Hom+ , the possible affine commands are (ω1 , ω2 ) 7→ (ω2 , 0), and
(ω1 , ω2 ) 7→ ( 12 ω1 , ω2 ),
(ω1 , ω2 ) 7→ (ω1 , ω2 + bω1 ),
b ∈ {□, 0, 1}.
Since □ = −1, these maps have magnitudes at most 1, 1, and 2, respectively. Thus m(Hom+ ) = 2. • For × , we inspect of the RNN multiplication construction used in Lemma C.7. This RNN was ×) ≤ 5. introduced in [14, Theorem 11]. This yields m(× • The machine Hom uses ± and Hom+ as subroutines and has no additional nontrivial command map. Lemma C.25 therefore gives ±) ∨ m(Hom+ ) ∨ 1 = 2. m(Hom) ≤ m(±
68
• The explicit commands of π are and
ω 7→ (ω1 , ω2 , ω3 , ω2 , 012 ) ω 7→ (ω1 , ω2 , ω3 + ω4 , ω1 , 012 ),
with magnitudes 1 and 2. Its subroutines are Hom and ×, hence ×) ≤ 5. m(π) ≤ 2 ∨ m(Hom) ∨ m(× • The only explicit nontrivial command of Π outside its subroutines is ω 7→ (ω1 , ω3 , 014 ), which has magnitude 1. Its subroutines are π and S, so m(Π) ≤ 1 ∨ m(π) ∨ m(S) ≤ 5. • The explicit initialization command of C is ω 7→ (ω1 , 1, 014 , ω2 ), which has magnitude 1. Since the only subroutine is Π, m(C) ≤ 1 ∨ m(Π) ≤ 5. • For TStep, the explicit commands are ω 7→ (ω1 , ω2 , ω3 , ω1 , ω3 , 012 ) and
ω 7→ (ω1 , ω3 , 2ω4 − ω2 , 014 ),
with magnitudes 1 and 3. The only subroutine is × , hence m(TStep) ≤ 5. • For TSum, the explicit commands outside subroutines have magnitudes at most 2: the initialization command has the row 2ω2 , the accumulation command has the row ω4 + ω5 , and the exit command ω 7→ (ω1 , ω4 , 016 ) has magnitude 1. Its subroutines are Hom, TStep, and S, so m(TSum) ≤ 2 ∨ m(Hom) ∨ m(TStep) ∨ m(S) ≤ 5. • Finally, the explicit initialization command of TC is ω 7→ (ω1 , 12 , ω21 , 015 , ω2 ), which has magnitude at most 1. The only subroutine is TSum, and therefore m(TC) ≤ 5. The two displayed bounds for ∥ · ∥C are now immediate from Lemma C.24.
D
Technical results on approximation of polynomials with RNNs
We begin with the monomial estimate produced by repeated approximate multiplication. Lemma D.1. For all d, n ∈ N0 and x ∈ [−1, 1], we have Gd,n (x) − xd ≤ d2−2n−1 . and |Gd,n (x)| ≤ 1. Proof. We prove the first statement by induction on d. (a) Base case d = 0. For all n ∈ N0 and x ∈ [−1, 1], we have |G0,n (x) − x0 | = |1 − 1| = 0 ≤ 0 · 2−2n−1 .
69
(b) Inductive step. Assume that the statement holds for some d ∈ N0 . We prove that it also holds for d + 1. Let n ∈ N0 and x ∈ [−1, 1]. We have |Gd+1,n (x) − xd+1 | = |R× D(Gd,n (x), x)[n] − Gd,n (x)x + Gd,n (x)x − xd+1 | ≤ |R× D(Gd,n (x), x)[n] − Gd,n (x)x| + |Gd,n (x)x − xd+1 | ≤ 2−2n−1 + |Gd,n (x) − xd ||x| ≤ 2−2n−1 + d2−2n−1 · 1 = (d + 1)2−2n−1 , where we used the induction hypothesis in the last inequality. This concludes the proof of the first statement. The second statement is a direct consequence of Lemma C.7. Summing the monomial estimates gives the corresponding bound for dyadic polynomials. Lemma D.2. Let P : x 7→ a0 x0 + a1 x1 + · · · + ad xd be a polynomial with coefficients a0 , . . . , ad ∈ D. For all n ∈ N0 and x ∈ [−1, 1], we have |FP,n (x)| ≤ ∥P ∥1 . and |FP,n (x) − P (x)| ≤ ∥P ∥1 ∥P ∥0 2−2n−1 . Proof. Let P : x 7→ a0 x0 + a1 x1 + · · · + ad xd be a polynomial with coefficients a0 , . . . , ad ∈ D. Let also n ∈ N0 and x ∈ [−1, 1]. We have |FP,n (x)| =
d X
ai Gi,n (x) ≤
i=0
d X
|ai ||Gi,n (x)| ≤
i=0
d X
|ai | = ∥P ∥1 ,
i=0
where we used Lemma D.1 in the second inequality. Moreover, we have |FP,n (x) − P (x)| =
d X
ai Gi,n (x) −
i=0
≤
d X
d X
ai xi
i=0
|ai ||Gi,n (x) − xi |
i=0
≤
d X
|ai |i2−2n−1
i=0
≤ ∥P ∥1 ∥P ∥0 2−2n−1 , where we used Lemma D.1 in the second inequality, and the fact that i ≤ d ≤ ∥P ∥0 for all i ∈ {0, . . . , d} in the last inequality. This concludes the proof.
E
Lower bounds on convergence speed
This appendix gathers lower-bound estimates showing that the runtime dependence in the RNN approximation paradigm cannot be removed in general.
E.1
Fixed-time lower bound for the squaring function
We shall use the following elementary terminology. A continuous piecewise-affine function g : [−1, 1] → R has at most N breakpoints if there exist −1 = x0 < x1 < · · · < xN < xN +1 = 1 such that g is affine on each interval [xj , xj+1 ]. We begin by recording that a fixed-time RNN realization is piecewise affine with controlled breakpoints. Lemma E.1. Let R = (1, m, 1; Ah , bh , Ax , Ao , bo ) be a scalar-input scalar-output RNN, and let t0 ∈ N0 . Then the map x ∈ [−1, 1] 7−→ (RDx)[t0 ] is continuous piecewise-affine and has at most (m + 1)t0 +1 − 1 breakpoints. 70
Proof. For t ∈ N0 , write
ht (x) := (HDx)[t].
We prove by induction that each coordinate of ht is affine on every interval of a partition of [−1, 1] with at most (m + 1)t+1 intervals. For t = 0, we have h0 (x) = ρ(Ax x + bh ). The m scalar affine functions appearing before the ReLU have at most m zeros in total. These zeros partition [−1, 1] into at most m + 1 intervals, and on each such interval every coordinate of h0 is affine. Assume that the claim holds at time t − 1. On each interval of the corresponding partition, each coordinate of ht−1 is affine. Therefore each coordinate of Ah ht−1 (x) + bh is affine on that interval and has at most one zero there. Adding the zeros of these m affine functions refines the interval into at most m + 1 subintervals. Thus the number of intervals is multiplied by at most m + 1, and on every resulting interval ht (x) = ρ(Ah ht−1 (x) + bh ) is affine. This proves that ht0 is affine on at most (m + 1)t0 +1 intervals. Since (RDx)[t0 ] = Ao ht0 (x) + bo , the same partition works for the output map, which therefore has at most (m + 1)t0 +1 − 1 breakpoints. The lower-bound argument needs a simple counting estimate for approximating a quadratic by piecewise-affine functions. Lemma E.2. Let ε > 0, and let g : [−1, 1] → R be continuous piecewise-affine. If sup |g(x) − x2 | ≤ ε, x∈[−1,1]
then g has at least √12ε − 1 breakpoints. Proof. We first record a local obstruction. Let I = [a, b] ⊂ [−1, 1] have length ℓ = b − a, and let λ be affine on I. Denote by m = (a + b)/2 the midpoint of I and put r = ℓ/2. Define e(y) := (m + y)2 − λ(m + y),
y ∈ [−r, r].
Since λ is affine, there exist α, β ∈ R such that e(y) = y 2 + αy + β. In particular,
e(0) = β,
yielding
e(r) = r2 + αr + β,
e(−r) = r2 − αr + β,
e(r) + e(−r) − e(0) = r2 . 2
If E := supx∈I |x2 − λ(x)|, then |e(−r)|, |e(0)|, |e(r)| ≤ E. Therefore r2 =
e(r) + e(−r) |e(r)| + |e(−r)| − e(0) ≤ + |e(0)| ≤ 2E. 2 2
Since r = ℓ/2, we obtain E ≥ r2 /2 = ℓ2 /8. Hence every affine λ satisfies sup |x2 − λ(x)| ≥ x∈I
ℓ2 . 8
Suppose now that g has N breakpoints. Then [−1, 1] is decomposed into N + 1 intervals I0 , . . . , IN on which g is affine. Fix one of these intervals, say Ij , and denote its length by ℓj . Since g is affine on Ij , the local estimate above can be applied with λ = g|Ij . Hence ℓ2j ≤ sup |x2 − g(x)| ≤ sup |x2 − g(x)| ≤ ε. 8 x∈Ij x∈[−1,1] Therefore ℓj ≤
√
8ε. Since this holds for every j ∈ {0, . . . , N } and since the lengths of the intervals sum
71
to 2, we get
1 2 N +1≥ √ = √ , 8ε 2ε
as claimed. Combining Lemmas E.1 and E.2 shows the following. If an RNN R with scalar input and scalar output satisfies sup |(RDx)[t0 ] − x2 | ≤ ε x∈[−1,1]
for some ε > 0, then
1 (m(R) + 1)t0 +1 − 1 ≥ √ − 1, 2ε
and hence
1 2ε
m(R) ≥
E.2
2(t 1+1) 0
− 1.
Minimax errors over compact classes
We now introduce the notation used for minimax lower bounds. For m ∈ N and B ≥ 0, let Rm,B := {R = (1, m′ , 1; Ah , bh , Ax , Ao , bo ) : m′ ≤ m, M(R) ≤ B} . Thus Rm,B is the class of scalar-input scalar-output RNNs whose hidden state size is at most m and whose weights have magnitude at most B. For t ∈ N0 and R ∈ Rm,B , we define Φt (R) : [−1, 1] → R,
Φt (R)(x) := (RDx)[t].
Equivalently, the set of functions realized at time t by RNNs in Rm,B is At (m, B) := {Φt (R) : R ∈ Rm,B } ⊂ C([−1, 1]). Let X ⊂ C([−1, 1]) be compact for the uniform norm ∥f ∥∞ :=
sup |f (x)|. x∈[−1,1]
The fixed-time minimax error of the class Rm,B over X is defined by Et (X ; m, B) := sup
inf
sup |(RDx)[t] − f (x)| .
f ∈X R∈Rm,B x∈[−1,1]
In terms of the realized function class At (m, B), this is simply Et (X ; m, B) = sup
inf
f ∈X g∈At (m,B)
∥g − f ∥∞ .
The lower bounds below will estimate this quantity from below as a function of the time t, the hidden state size m, and the weight magnitude B. We next recall the metric-entropy language used to formulate these lower bounds. Definition E.1 (Covering number and metric entropy). Let Y ⊂ C([−1, 1]) be a compact set and ε > 0. We define the ε-covering number of Y by N [ N (ε, Y) := inf N ∈ N : ∃g1 , . . . , gN ∈ C([−1, 1]), Y ⊂ B ∞ (gj , ε) , j=1
where B ∞ (g, ε) := {h ∈ C([−1, 1]) : ∥h − g∥∞ ≤ ε} . The metric entropy of Y at scale ε is H(ε, Y) := log N (ε, Y). We next define the parameter-to-realization map. Definition E.2 (Parameter-to-realization map). Let m ∈ N and t ∈ N0 , and set pm := m2 + 3m + 1. 72
We identify Rpm with
Rm×m × Rm × Rm×1 × R1×m × R.
For θ = (Ah , bh , Ax , Ao , bo ) ∈ Rpm , let Rθ := (1, m, 1; Ah , bh , Ax , Ao , bo ), and define Ft,m : Rpm → C([−1, 1]),
Ft,m (θ)(x) := Φt (Rθ )(x).
Note that Definition E.2 gives, for every B ≥ 0, At (m, B) = Ft,m ([−B, B]pm ) . The following lemma gives Lipschitz control of this realization map with respect to the parameters. Lemma E.3. Let m ∈ N, B ≥ 1, and t ∈ N0 . Define Λt,m,B := 8B 2 m2 (2mB)2t . 2
Then Ft,m is Λt,m,B -Lipschitz from ([−B, B]m +3m+1 , ∥ · ∥∞ ) to (C([−1, 1]), ∥ · ∥∞ ). 2
Proof. Let θ, θ′ ∈ [−B, B]m +3m+1 be two parameter vectors, and let δ := ∥θ − θ′ ∥∞ . We write R, R′ for the corresponding RNNs and hs (x) := (HDx)[s], Set St,m,B :=
h′s (x) := (H′ Dx)[s],
t X (mB)j
and
s ∈ N0 .
Ht,m,B := 2BSt,m,B .
j=0
First we bound the size of the hidden states. First note that ∥h0 (x)∥∞ = ∥ρ(Ah h−1 (x) + Ax (Dx)[0] + bh )∥∞ = ∥ρ(Ax x + bh )∥∞ ≤ ∥Ax x∥∞ + ∥bh ∥∞ ≤ B|x| + B ≤ 2B, for every x ∈ [−1, 1]. Similarly, ∥h′0 (x)∥∞ ≤ 2B. Moreover, for every s ∈ {1, . . . , t} and x ∈ [−1, 1] we have ∥hs (x)∥∞ = ∥ρ(Ah hs−1 (x) + Ax (Dx)[s] + bh )∥∞ = ∥ρ(Ah hs−1 (x) + bh )∥∞ ≤ ∥Ah hs−1 (x)∥∞ + ∥bh ∥∞ ≤ mB∥hs−1 (x)∥∞ + B ≤ mB∥hs−1 (x)∥∞ + 2B. It follows by induction that ∥hs (x)∥∞ ≤ 2B
s X (mB)j ≤ Ht,m,B ,
s ∈ {0, . . . , t},
(208)
j=0
and we have the same bound for h′s (x). We now estimate the sensitivity of the hidden state to the parameters. Put Ds := sup ∥hs (x) − h′s (x)∥∞ . x∈[−1,1]
Using the 1-Lipschitz property of ρ, for every s ∈ {0, . . . , t} and every x ∈ [−1, 1], we have ∥hs (x) − h′s (x)∥∞ ≤ ∥Ah hs−1 (x) + Ax (Dx)[s] + bh − A′h h′s−1 (x) − A′x (Dx)[s] − b′h ∥∞ ≤ ∥A′h (hs−1 (x) − h′s−1 (x))∥∞ + ∥(Ah − A′h )hs−1 (x)∥∞ + ∥(Ax − A′x )(Dx)[s]∥∞ + ∥bh − b′h ∥∞ ≤ mBDs−1 + mHt,m,B δ + δ + δ. Taking the supremum over x ∈ [−1, 1] gives Ds ≤ mBDs−1 + (mHt,m,B + 2)δ,
73
with D−1 = 0. Therefore Dt ≤ (mHt,m,B + 2)δ
t X
(mB)j .
(209)
j=0
Finally, for the output layer, |(RDx)[t] − (R′ Dx)[t]| ≤ Bm∥ht (x) − h′t (x)∥∞ + mHt,m,B δ + δ. Combining this with (209) gives
t X ∥Ft,m (θ) − Ft,m (θ′ )∥∞ ≤ 1 + Bm(mHt,m,B + 2) (mB)j + mHt,m,B δ ≤ Λt,m,B δ. j=0
This Lipschitz control converts parameter dimension into a covering-number bound. Lemma E.4. Let m ∈ N, B ≥ 1, t ∈ N0 , and ε > 0. Then N (ε, At (m, B)) ≤
1+
2BΛt,m,B ε
m2 +3m+1 .
Proof. Let p := m2 + 3m + 1. The cube [−B, B]p can be covered, for the ℓ∞ norm, by at most p 2BΛt,m,B 1+ ε balls of radius ε/Λt,m,B . By Lemma E.3, the image under Ft,m of each such ball is contained in a ball of radius ε in C([−1, 1]). Since At (m, B) ⊂ Ft,m ([−B, B]p ) by Definition E.2, this yields the claimed covering bound for At (m, B). The next comparison turns small uniform approximation error into an entropy inequality. Lemma E.5. Let X ⊂ C([−1, 1]) be compact, let m ∈ N, B ≥ 1, and t ∈ N0 . Then Et (X ; m, B) ≥ inf {ε > 0 : N (2ε, X ) ≤ N (ε, At (m, B))} . Proof. Let η > Et (X ; m, B). By definition of Et (X ; m, B), for every f ∈ X there exists gf ∈ At (m, B) such that ∥f − gf ∥∞ < η. Let g1 , . . . , gM be an η-covering of At (m, B), with M = N (η, At (m, B)). For each f ∈ X , choose j ∈ {1, . . . , M } such that ∥gf − gj ∥∞ ≤ η. Then ∥f − gj ∥∞ ≤ ∥f − gf ∥∞ + ∥gf − gj ∥∞ < 2η. Hence {g1 , . . . , gM } is a 2η-covering of X , and so N (2η, X ) ≤ N (η, At (m, B)) . Therefore for every η > Et (X ; m, B), we also have η ∈ {ε > 0 : N (2ε, X ) ≤ N (ε, At (m, B))} . Taking the infimum over η > Et (X ; m, B) gives the claim. Combining the two entropy estimates gives a necessary condition for minimax approximation. Corollary E.6. Let X ⊂ C([−1, 1]) be compact, let m ∈ N, B ≥ 1, and t ∈ N0 . Then ( 5m2 ) 1 (2mB)2t+7 Et (X ; m, B) ≥ inf ε ∈ (0, 1] : N (ε, X ) ≤ 2 ε Proof. Let
p := m2 + 3m + 1
74
and
( S :=
ε ∈ (0, 1] : N (ε, X ) ≤
(2mB)2t+7 ε
5m2 ) .
By Lemma E.4, for every ε > 0, p 2BΛt,m,B 1+ . ε
N (ε, At (m, B)) ≤ Since m ≥ 1 and B ≥ 1, we have and
p = m2 + 3m + 1 ≤ 5m2
2BΛt,m,B = 16B 3 m2 (2mB)2t ≤ (2mB)2t+5 .
Therefore N (ε, At (m, B)) ≤
Hence, if ρ ∈ (0, 1] and N (ρ, X ) ≤ N
ρ
(2mB)2t+6 ρ/2
5m2
then N (ρ, X ) ≤
(2mB)2t+5 1+ ε
2
5m2 .
, At (m, B) , ≤
(2mB)2t+7 ρ
5m2 .
Hence
o n ρ , At (m, B) ⊂ S. ρ ∈ (0, 1] : N (ρ, X ) ≤ N 2 If Et (X ; m, B) ≥ 1/2, then the claim is immediate. Assume Et (X ; m, B) < 1/2. The proof of Lemma E.5 shows that every η > Et (X ; m, B) satisfies N (2η, X ) ≤ N (η, At (m, B)) . Thus every 2η with η ∈ (2Et (X ; m, B), 1/2] belongs to S, and so inf S ≤ 2Et (X ; m, B). This proves the claim. We now spell out what Corollary E.6 gives for the function classes used in the main text. Throughout this subsection, we fix m ∈ N, B ≥ 1, and put At,m,B := 5m2 (2t + 7) log(2mB),
Cm := 5m2 .
With this notation, the condition appearing in Corollary E.6 can be written as log N (ε, X ) ≤ At,m,B + Cm log(ε−1 ).
(210)
Thus the minimax lower bound is obtained by inverting the metric entropy of X . The following theorem gives a convenient rate-level inversion, avoiding the exact transcendental formulas. Theorem E.7 (Entropy inversion). Let X ⊂ C([−1, 1]) be compact. Let h : (0, 1] → (0, ∞) be nonincreasing and assume that log(ε−1 ) = o(h(ε))
log N (ε, X ) ≥ h(ε),
and
ε ∈ (0, 1].
(211)
Then there exist constants K > 0 and t0 ∈ N, depending only on m, B and h, such that, for every t ≥ t0 , Et (X ; m, B) ≥
1 −1 h (K(t + 1)), 2
where h−1 (T ) := inf {ε ∈ (0, 1] : h(ε) ≤ T }. Proof. Since log(ε−1 ) = o(h(ε)), there exists ε1 ∈ (0, 1] such that Cm log(ε−1 ) ≤
1 h(ε), 2
75
ε ∈ (0, ε1 ].
(212)
Let ε ∈ (0, ε1 ] satisfy (210). Combining (210), (211) and (212), we obtain 1 h(ε) ≤ At,m,B + Cm log(ε−1 ) ≤ At,m,B + h(ε), 2 hence Choose K > 0 such that
h(ε) ≤ 2At,m,B . 2At,m,B ≤ K(t + 1),
t ∈ N0 .
Then every ε ∈ (0, ε1 ] satisfying (210) also satisfies ε ≥ h−1 (K(t + 1)). Since h−1 (K(t + 1)) → 0 as t → ∞, there exists t0 ∈ N such that h−1 (K(t + 1)) ≤ ε1 for every t ≥ t0 . For such t, the same lower bound is automatic for admissible scales ε > ε1 . Therefore ( 5m2 ) (2mB)2t+7 inf ε ∈ (0, 1] : N (ε, X ) ≤ ≥ h−1 (K(t + 1)). ε Corollary E.6 proves the claim.
76