Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks

Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks Shivang Rawat1,∗

Mirko Morello1

Flaviano Morone1,2

David J. Heeger1,2

1

arXiv:2609.04134v1 [cs.LG] 3 Sep 2026

2 ∗

Telepath New York University

shivang@telepath.com

Abstract Temporal integration gives continuous-time recurrent networks memory, but in deep stacks it also delays bottom-up signals and attenuates top-down errors. We develop Recursive Quadrature Filters (RQFs), biologically motivated complex-valued temporal filters that are a special case of diagonal state-space models (SSMs), and ask whether this failure mode can be addressed by making each layer’s bottom-up input prospective. Starting from an energy model, we derive the RQF dynamics and show that each RQF is a band-pass filter whose learnable parameters control its tuning frequency and bandwidth. We then make each layer’s bottom-up input prospective using a parameter-free two-tap update that leaves the recurrent transition and parallel scan unchanged. We extend this correction to general diagonal SSMs and show that it mitigates depth-dependent gradient attenuation when temporal gradients are truncated, i.e., spatial-only backpropagation. We evaluate the intervention in RQFs, S5, and ORGaNICs (a nonlinear gated RNN) trained using full backpropagation through time (BPTT) and spatial-only backpropagation. Under full BPTT, prospective variants match or outperform their non-prospective controls in every model and configuration. A non-residual width-32 six-layer RQF reaches 96.09 ± 0.23% accuracy on raw-audio Speech Commands with 31.9k parameters; a width-64 sixlayer RQF reaches 83.56 ± 2.11% on the 16,384-step Path-X task. These results identify RQFs as a parameter-efficient recurrent substrate and prospective-input coding as an input-side correction for deep continuous-time recurrent networks.

1

Introduction

Biological neural circuits face a fundamental trade-off between memory and reactivity. A biological neuron is well-modeled as a leaky integrator whose membrane time constant τ confers temporal context but also delays and attenuates fast input changes [1]. When stacked into an L-layer feedforward circuit, this filter cascades, so the effective relaxation time grows roughly as Lτ [2]. The same trade-off is built into recurrent theories of cortical function [3], in which slow recurrent dynamics let the network settle into a representation that combines feedforward drive with prior expectation. This lag also creates a learning problem: when a top-down teaching signal arrives at the output, the activity it should instruct in early layers is already roughly Lτ out of date. Signals that vary on timescales comparable to τ are therefore poorly served by the standard leaky cascade. This paper contributes a recurrent substrate and an input-side correction. The substrate is the Recursive Quadrature Filter (RQF), a linear dynamical system inspired by neuroscience that is a special case of a diagonal state-space model, so it trains on sequences of tens of thousands of steps with standard machine-learning machinery. The correction drives each layer with a prospective rather than an Preprint.

instantaneous bottom-up signal; it adds no parameters and applies to existing diagonal SSMs as readily as to RQFs. The biological motivation for this correction comes from prospective firing: under sinusoidally modulated somatic current injection, cortical pyramidal cells can fire at a phase that leads their input, and biophysical models explain this as a firing-rate advance relative to membrane voltage [4, 5]. Apical dendritic stimulation can likewise modulate the soma with little integration lag through dendritic resonance mechanisms [6, 5]. Related prospective responses have been documented in retinal ganglion-cell populations, cerebellar Purkinje cells, and human muscle-spindle afferents [7– 9]. Such phase-leading behavior can be modeled by applying the prospective (or look-ahead) operator [2, 10, 11] χτ ≡ 1 + τ

d dt

(1)

to the underlying voltage or firing rate. For a first-order membrane filter, χτ is its exact differential inverse, so applying it cancels the filter’s lag. When interpreted as a time-advance operator, however, it is accurate only to first order: expanding the unit time-shift in τ gives u(t + τ ) = u(t) + τ u̇(t) + O(τ 2 ) = χτ (u(t)) + O(τ 2 ). Thus, χτ advances a smooth signal by one membrane time constant to first order in τ . We apply this prospective operator in networks of RQFs, a class of biologically motivated complexvalued temporal filters [12]. We derive RQFs from an energy-based theory of cortical function [3], in which a state parameter tunes a compromise between feedforward sensory drive and an internally generated prior. Taking the prior to be the prospective image of the state under a phase-rotation model collapses the resulting dynamics into a scalar complex-valued band-pass filter with a learnable phase-prior frequency ω and a learnable bandwidth parameter γ (Section 2). It has been suggested that the brain relies on a set of canonical neural microcircuits, across brain regions and modalities, implementing hierarchies of spatial band-pass filtering, output nonlinearities, and spatial pooling that extract and combine features [13–19], much as convolutional neural networks do. RQFs extend this paradigm to extract and combine features over time. Temporal band-pass filters like RQFs have been used to model temporal-frequency selectivity and direction selectivity of neurons in primary visual cortex [12]. The encoding of sound in the cochlea and the auditory nerve can likewise be modeled as a bank of band-pass filters [20]. We prove that scalar RQFs are a special case of single-channel diagonal complex state-space models (SSMs), so an RQF layer is a constrained diagonal SSM with biologically interpretable poles and inherits the parallel-scan and FFT-convolutional training algorithms used for SSMs [21, 22]. This distinguishes RQFs from recent prospective-coding cortical models: Neuronal least-action [10] (NLA) and latent equilibrium [2] use prospective signals for local biological learning, while generalized latent equilibrium [11] (GLE) uses prospective and retrospective operators for online credit assignment in a feedforward architecture. RQFs instead provide a recurrent temporal-filtering primitive compatible with long-sequence SSM training. We show, both theoretically and empirically, that deep RQF networks improve when each layer’s bottom-up input drive is prospective. Prospective and instantaneous-input RQFs share the same prospective phase-rotation prior via recurrence; they differ only in whether the bottom-up drive is look-ahead-coded. This input-side change is local to each layer, leaves the RQF parameters and state transition unchanged, admits a two-tap discretization applicable to diagonal SSMs (Section 3), and directly affects credit assignment under spatial-only backpropagation. Spatial-only backpropagation treats every earlier recurrent state as a stop-gradient variable and differentiates only through the within-step spatial chain across layers. Under this regime, instantaneous inputs introduce an explicit (h/τ )L−ℓ attenuation (where h is the discretization timestep), while prospective-input coding changes the discretization prefactor of each spatial hop from O(h/τ ) to O(1) as h/τ → 0. Contributions. • Architecture (machine learning). We introduce RQFs; although motivated by neuroscience, here we develop them for machine-learning applications. We prove that RQFs are a special case of diagonal SSMs (Section 2), making RQF layers compatible with the parallel-scan and FFT-convolution algorithms used for diagonal SSMs and bringing tasks such as the 16,384-step Path-X within reach. 2

• Prospective input (machine learning). We replace each layer’s instantaneous bottom-up drive with a prospective one and derive the corresponding discretization for any diagonal SSM (Section 3). The implementation adds no parameters, leaves the state transition and parallel scan untouched, and costs only a second input tap; under full BPTT, it yields higher mean test accuracy than the matched control in every configuration. • Theory (computational neuroscience and machine learning). We prove that under spatialonly backpropagation, instantaneous bottom-up inputs incur a geometric discretizationinduced depth attenuation that prospective-input coding removes (Section 3 and Appendix C). • Experiments (machine learning). We evaluate prospective input on Speech Commands and Path-X with RQF stacks, S5, and the nonlinear ORGaNICs substrate [23], under full BPTT and spatial-only backpropagation (Section 4). Scope of contributions. The RQF substrate and the input-side correction are evaluated as machinelearning contributions under standard BPTT; the cortical theory provides their motivation. The third contribution concerns spatial-only backpropagation, in which the recurrence is never differentiated through time. We adopt that regime because it is the one described by our derivation and isolates the effect of the input drive; it is a diagnostic, neither a prerequisite for the machine-learning contribution nor a recommended training procedure. Spatial-only backpropagation is temporally local, but exact error propagation across layers still uses conjugate-transposed forward weights and therefore retains the weight-transport problem. We consequently describe RQFs as a biologically motivated circuit abstraction but do not suggest that the complete training procedure is biologically plausible.

2

Recursive Quadrature Filters (RQFs)

The substrate for RQFs is a leaky integrator whose state relaxes toward a convex mixture of the external input and an internally predicted recurrent state: τ ẏ = −y + [γ z + (1 − γ) yb] ,

(2)

where y(t) ∈ C is the filter state; z(t) ∈ C is the input drive; τ > 0 is the base time constant; yb is the recurrent prediction; and γ ∈ (0, 1) controls the balance between input tracking and recurrence. This convex blend mirrors the cortical energy model of Heeger [3]; Appendix A.1 derives Eq. (2) from that energy functional, with γ as the state parameter and yb as the prior drive. The recurrent prediction yb in Eq. (2) requires a model of how y evolves locally. We take this model to be a phase-rotation prior: the undamped complex oscillator ẏ = ιωy, the minimal generator on C of a single phase-prior frequency ω. This choice follows the internal-model principle: a system that prospectively tracks a structured signal should contain a model of that signal’s dynamics [24]. Applying the prospective operator χτ from Eq. (1) to this prior gives yb ≜ χτ (y) ẏ=ιωy = (1 + ιωτ ) y,

(3)

where χτ (y(t)) is causal because it uses only the instantaneous activity and its local time derivative. Substituting Eq. (3) into Eq. (2) yields the Recursive Quadrature Filter (RQF), a scalar complexvalued temporal filter governed by τ ẏ = −y + γ z + (1 − γ)(1 + ιωτ ) y,

(4)

where ω > 0 is the learnable phase-prior frequency and γ is the learnable bandwidth parameter. The state is complex-valued even when z is real, with its real and imaginary parts forming an inphase/quadrature pair. Defining α ≜ γ/τ , ω0 ≜ (1 − γ)ω, and λ ≜ −α + ι ω0 collapses Eq. (4) into the compact scalar recurrence ẏ = λy + αz, so the RQF is a stable single-pole complex filter for any γ ∈ (0, 1) (Fig. 1). The prospective factor (1 + ιωτ )y sets the local recurrent prediction, whereas the homogeneous state evolves as y(t + ∆) = eλ∆ y(t) when z = 0. Appendix A.2 derives the transfer function (revealing the band-pass property), quality factor, damped-oscillator form, and the effect of cascading RQFs for fine-tuning the bandwidth. Multi-layer architecture. A layer of an RQF network bundles nℓ scalar filters operating in parallel, (ℓ) (ℓ) each with its own phase-prior frequency ωi and bandwidth γi but sharing a common time constant 3

a

b

c

e

d

Figure 1: The Recursive Quadrature Filter (RQF): pole, impulse response, and frequency response. An RQF is a scalar complex-valued temporal filter with learnable phase-prior frequency ω and bandwidth parameter γ at a fixed base time constant τ , governed by Eq. (4). The derived quantities are the decay rate α = γ/τ , resonant frequency ω0 = (1 − γ)ω, effective time constant τ0 = τ /γ, and pole λ = −α + ιω0 . (a) Free response of y in the complex plane, y(t) = y(0) eλt . (b) The pole lies in the stable region Re(λ) < 0 for any γ ∈ (0, 1). (c) Notation used throughout the paper. (d) Impulse response y(t) = α eλt to a Dirac drive z(t) = δ(t), with in-phase and quadrature components oscillating inside the envelope |y(t)| = α e−t/τ0 (γ = 0.1, ωτ = 0.65). (e) Magnitude response for several values of γ, showing the transition from sharp resonance to broad integration. τ . Collecting the parameters into vectors ω (ℓ) , γ (ℓ) ∈ Rnℓ and writing the layer state as y(ℓ) ∈ Cnℓ , the layer dynamics are the element-wise vectorization of Eq. (4): τ ẏ(ℓ) = −y(ℓ) + γ (ℓ) ⊙ z(ℓ) + (1 − γ (ℓ) ) ⊙ (1 + ιτ ω (ℓ) ) ⊙ y(ℓ) , (ℓ)

(5)

nℓ

where z ∈ C is the bottom-up input drive to the layer; ⊙ denotes the element-wise product; and 1 ∈ Rnℓ is the all-ones vector. Defining α(ℓ) ≜ γ (ℓ) /τ and λ(ℓ) ≜ −α(ℓ) + ι (1 − γ (ℓ) ) ⊙ ω (ℓ) , Eq. (5) collapses to the compact diagonal complex ODE ẏ(ℓ) = λ(ℓ) ⊙ y(ℓ) + α(ℓ) ⊙ z(ℓ) . To build a deep network, we stack L layers. The input drive to layer ℓ is   z(ℓ) = W(ℓ) x(ℓ) , x(ℓ) = ρ y(ℓ−1) ,

(6)

(7)

where ρ : Cnℓ−1 → Cnℓ−1 is an activation function (also called an output nonlinearity); x(ℓ) ∈ Cnℓ−1 is the layer input to layer ℓ, with the boundary x(1) ≜ rin ; W(ℓ) ∈ Cnℓ ×nℓ−1 is a complex inter-layer weight matrix; and z(ℓ) ∈ Cnℓ is the post-weight bottom-up input drive. The activation function ρ is not required to be holomorphic: nothing in the layer dynamics or in the discretization of Section 3 relies on complex differentiability, and Wirtinger calculus supplies gradients for any real-differentiable map of (Re(y), Im(y)). The default is the pointwise family [ρ(y)]i = ρ0 (yi ), with the canonical choice being the split-ReLU ρ0 (y) = max{0, Re(y)} + ι max{0, Im(y)}, 4

(8)

a

b

Figure 2: Prospective-input coding in multi-layer RQFs. The prospective-input variant replaces the bottom-up post-nonlinearity signal entering layer ℓ with its τ -step look-ahead χτ (ρ(y(ℓ−1) )) defined in Eq. (15); the instantaneous-input baseline feeds ρ(y(ℓ−1) ) directly into W(ℓ) . (a) The state y(ℓ−1) is passed through the activation ρ, advanced by the prospective operator χτ , and mixed by the inter-layer weight W(ℓ) to form the bottom-up drive z(ℓ) . (b) Action of χτ on a sinusoidal signal z(t) (black). The look-ahead χτ (z(t)) = z(t) + τ ż(t) (pink, dashed) advances z by approximately τ by extrapolating along the local tangent. which applies a real-valued rectifier independently to the in-phase and quadrature components and is non-holomorphic by construction. The RQF experiments use no normalization, although layer normalization, filter normalization or other pointwise or channel-coupling nonlinearities are admissible. Discretization. For the implementation, we discretize the continuous-time layer dynamics in Eq. (6) with a zero-order hold (ZOH) on the bottom-up drive z(ℓ) . ZOH treats z(ℓ) as piecewise constant (ℓ) on each integration interval [tn , tn+1 ], with the constant value denoted by zn+1 , and integrates the resulting forced linear ODE in closed form. It is the default discretization for diagonal SSMs, adopted by DSS, S4D, S5, and Mamba [22, 21, 25, 26], because it (i) is unconditionally stable for any pole in the open left half-plane, meaning that high-Q neurons impose no extra step-size constraint; (ii) integrates the linear part exactly, retaining the full eλh response at finite h; and (iii) agrees with forward Euler to first order in h/τ , with an O((h/τ )2 ) difference. ZOH discretization of Eq. (6) yields the diagonal complex affine recurrence (ℓ)

(ℓ)

(ℓ)

(ℓ)

yn+1 = Āh ⊙ yn(ℓ) + B̄h ⊙ zn+1 ,

(9)

where

 α(ℓ)  (ℓ) ⊙ Ā − 1 , (10) h λ(ℓ) (ℓ) (ℓ−1) and where zn+1 = W(ℓ) ρ(yn+1 ) samples the bottom-up input drive in Eq. (7) at tn+1 . The (ℓ) (ℓ) coefficients Āh and B̄h depend only on the parameters (γ (ℓ) , ω (ℓ) ), the base time constant τ , and the step size h, and can be precomputed once per parameter update and reused across all time steps. The full derivation is given in Appendix B.1. (ℓ)

Āh

≜ eλ

(ℓ)

h

,

(ℓ)

B̄h

≜

RQFs are a special case of diagonal state-space models. A continuous-time linear state-space model (SSM) maps an input signal x(t) ∈ RU through a latent state y(t) ∈ RP to an output o(t) ∈ RM via ẏ(t) = A y(t) + B x(t), o(t) = C y(t) + D x(t), (11) with state matrix A ∈ RP ×P , input matrix B ∈ RP ×U , output matrix C ∈ RM ×P , and feedthrough D ∈ RM ×U [27, 25]. All SSMs share this template but differ in how the recurrent core is parameterized and how the feature channels are coupled. S4 applies a bank of single-input single-output (SISO) 5

SSMs across feature channels [27]; DSS and S4D replace S4’s structured state matrix with diagonal variants [22, 21]; S5 uses one multi-input multi-output (MIMO) SSM whose effective recurrent matrix is complex-valued and diagonal [25]; the LRU uses a discrete diagonal linear recurrence with dense input and output projections [28]; and Mamba retains a diagonal state transition while making the discretized parameters input-dependent [26]. In the fixed-parameter diagonal case, the recurrent core can be written directly in a complex diagonal form, e˙ (t) = Λ y e (t) + B x(t), y P ×P

P ×U

with Λ ∈ C diagonal, B ∈ C ek+1 = Āh y ek + B̄h xk+1 , with y

Āh ≜ eΛh ,

e (t) + D x(t), o(t) = C y M ×P

, and C ∈ C

(12)

. ZOH discretization at step h gives

B̄h ≜ Λ−1 Āh − I B.

(13)

The temporal modes are controlled by the entries of Λ: an eigenvalue λp ∈ C in the open left half-plane produces a damped oscillation at angular frequency Im(λp ) with decay rate |Re(λp )|. The RQF layer is the following constrained special case of Eq. (12), obtained by substituting the layer drive from Eq. (7) into Eq. (6): ẏ(ℓ) = diag(λ(ℓ) ) y(ℓ) + diag(α(ℓ) ) W(ℓ) x(ℓ) .

(14)

Thus, the SSM input x(t) corresponds to the RQF layer input x(ℓ) , and the dense input projection is the inter-layer weight W(ℓ) . In the canonical SSM notation of Eq. (12), the full continuous-time input matrix is B = diag(α(ℓ) )W(ℓ) ; the diagonal factor is not an extra SSM parameter but the RQF input gain already present in Eq. (6). The RQF-specific constraint is that α(ℓ) = γ (ℓ) /τ and λ(ℓ) = −α(ℓ) + ι(1 − γ (ℓ) ) ⊙ ω (ℓ) , which identifies the channel input gain with the inverse (ℓ) decay timescale 1/τ0 . The inclusion is one-way: a general diagonal SSM need not satisfy this pole–input-gain coupling and therefore need not be an RQF. The remaining SSM machinery transfers without modification: any output projection C ∈ CM ×nℓ and feedthrough D ∈ CM ×U may be appended to read out from y(ℓ) , and the per-block residual connections used in S4, S5, and Mamba stacks are admissible at the layer level. Unless stated otherwise, we set C = I and D = 0 throughout, so each layer’s output is its own complex state y(ℓ) and inter-layer mixing is delegated to W(ℓ+1) ; residual connections are treated as an explicit configuration choice in our experiments (Section 4).

3

Prospective-input coding in multi-layer RQFs

We use the terms instantaneous-input and prospective-input to refer only to the bottom-up input drive from layer ℓ − 1 to layer ℓ. Both variants retain the RQF recurrent prediction yb = (1 + ιωτ )y from Section 2; the comparison changes the input path, not the RQF dynamics. With the notation of Eq. (7), the instantaneous choice uses x(ℓ) = ρ(y(ℓ−1) ), while the prospective-input alternative applies the look-ahead operator before the inter-layer weights act (Fig. 2b):       e(ℓ) ≜ χτ x(ℓ) = ρ y(ℓ−1) + τ ρ̇ y(ℓ−1) , e(ℓ) , x z(ℓ) = W(ℓ) x (15) χ where the dot denotes the time derivative. Since W(ℓ) is fixed over the integration interval, applying χτ to x(ℓ) and then multiplying by W(ℓ) is equivalent in the forward pass to applying χτ to z(ℓ) ; the discretization below uses this compact input drive notation.
Discretization. Applying the ZOH update to the prospective input drive χτ z(ℓ) gives the following recurrence; Appendix B.2 provides the derivation: i h (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) yn+1 = Āh ⊙ yn(ℓ) + B̄h + γ (ℓ) ⊙ Āh ⊙ zn+1 − γ (ℓ) ⊙ Āh ⊙ z(ℓ) (16) n , (ℓ)

(ℓ)

(ℓ)

(ℓ−1)

where zk = W(ℓ) xk and xk = ρ(yk ). In standard signal-processing terms, Eq. (16) is a first-order IIR filter with a two-tap feedforward path: the recurrence on y(ℓ) has a single feedback (ℓ) (ℓ) (ℓ) pole Āh , while the compact drive enters through weighted taps at zn+1 and zn . This form permits a concrete comparison with the exponential-trapezoidal rule of Mamba-3 [29]. Writing vt = Bt xt , that rule uses ht = αt ⊙ ht−1 + (1 − λt )∆t αt ⊙ vt−1 + λt ∆t vt , 6

λt ∈ [0, 1],

(17)

Table 1: RQF on Speech Commands. Test accuracy for non-residual width-64 stacks trained with (a) full BPTT and (b) spatial-only backpropagation. Raw audio Depth Params Prospective

MFCC Non-pros.

(a) Full BPTT 2 52.3k 94.94±0.10 94.40±0.17 4 68.9k 96.27±0.16 95.77±0.22 6 85.6k 96.35±0.20 96.13±0.12 (b) Spatial-only backpropagation 2 N/A N/A N/A 4 68.9k 84.70±3.17 81.70±1.96 6 85.6k 82.92±5.42 75.34±1.18

Params Prospective

Non-pros.

53.5k 70.2k 86.8k

96.42±0.26 96.30±0.33 96.65±0.24 96.41±0.14 96.70±0.19 96.49±0.07

53.5k 70.2k 86.8k

94.89±0.20 94.20±0.28 92.63±0.85 89.04±0.83 91.55±0.76 76.38±1.57

so its endpoint weights form a convex blend. Prospective coding instead discretizes B[x(t) + τ ẋ(t)] and gives the signed previous-input tap −τ Āh Bxn (Appendix B.3). For a matched scalar real LTI system with ∆t = h, matching even this previous-input coefficient in Eq. (17) would require λt = 1 + τ /h > 1. The constructions therefore share a two-tap form but are not equivalent, and prospective input is not a special case of Mamba-3’s admissible interpolation. Prospective input adds no trainable parameters to an RQF layer and leaves its diagonal state transition and associative scan unchanged. For a sequence of length T , layer ℓ maps nℓ−1 inputs to nℓ RQF (ℓ) (ℓ) channels and computes the dense drive zt = W(ℓ) xt in O(T nℓ nℓ−1 ) work under either input rule. Prospective-input coding adds O(T nℓ ) element-wise computation and one lagged read of the already materialized drive, but no additional full-sequence activation storage; streaming execution needs only an O(nℓ ) rolling buffer for the previous drive. The diagonal RQF scan remains O(T nℓ ) with O(log T ) parallel depth. Thus, the asymptotic complexity of both inference and full BPTT is unchanged, although the second tap adds a small lower-order operation. Why prospective-input coding improves the spatial gradient path. Spatial-only backpropagation freezes the temporal recurrence at the previous step and leaves only the within-step chain through depth. An instantaneous-input Euler step sends each inter-layer error through gnp = (h/τ )γ, whereas prospective input uses g+ = (1 + h/τ )γ. Therefore, as h/τ → 0 at fixed depth and fixed parameters, the discretization prefactor multiplying each spatial hop changes from O(h/τ ) to O(1) and the explicit (h/τ )L−ℓ factor disappears. This statement concerns that prefactor only: the common product of activation slopes, weight norms, and damping gains may still attenuate with depth, and the upper bounds in Appendix C do not guarantee an O(1) complete gradient. The implemented ZOH update has the same small-h leading-order coefficients. Spatial-only backpropagation is local in time but still uses W† to propagate exact adjacent-layer errors, so it retains the weight-transport problem and is used here only to isolate the analyzed path.

4

Experiments1

RQFs are band-pass filters by construction (Appendix A.2), so speech provides a natural first test of whether prospective-input coding improves learning through the RQF cascade. We use the 10-class subset of Google Speech Commands v0.02 [30], comprising one-second utterances of the keywords yes, no, up, down, left, right, on, off, stop, and go. We consider two representations of each utterance: the raw waveform, supplied as a sequence of 16,000 scalar samples, and 20 Mel-frequency cepstral coefficients (MFCCs), supplied as a sequence of 161 frames. Thus, the two representations pose the same classification problem at very different sequence lengths and levels of temporal abstraction. We contrast two regimes for assigning gradient credit through the recurrence. Full backpropagation through time (BPTT) differentiates through the complete recurrent trajectory. Spatial-only backpropagation instead detaches every previous recurrent state and retains only the within-step chain through depth. We use the latter as a mechanistic diagnostic, rather than as the recommended optimizer, 1 Code for the experiments is available at https://github.com/Sequel-Institute/prospective-rqf.

7

Table 2: αP-S5 versus native S5, full BPTT. Test accuracy on Speech Commands for residual S5 blocks with 64 units per layer. Raw audio Depth Params

αP-S5

MFCC Native S5

Params

αP-S5

Native S5

48.9k 95.55±0.39 94.94±0.42 50.1k 96.52±0.20 95.89±0.38 74.2k 96.06±0.35 95.96±0.27 75.4k 96.40±0.17 96.31±0.25 99.5k 96.37±0.31 96.21±0.23 100.7k 96.41±0.20 96.36±0.14

2 4 6

Table 3: Discrete ORGaNICs on MFCC Speech Commands. Residual width-64 models. Full BPTT Depth Params Prospective 2 4 6

Non-pros.

Spatial-only Prospective

Non-pros.

58.3k 93.28±0.37 92.10±0.21 79.88±1.28 81.49±1.46 124.3k 93.21±0.59 91.91±0.53 82.61±1.21 81.67±0.88 190.3k 91.92±0.19 91.92±0.11 83.43±1.73 80.56±1.41

because it removes temporal gradient credit and directly exposes the spatial attenuation analyzed in Section 3. The cleanest test of this analysis is the RQF stack itself. We therefore begin with non-residual RQF models of depth L ∈ {2, 4, 6} and width n ∈ {32, 64}, using split-ReLU between layers and no normalization or gating. Avoiding residual connections leaves no bypass around the recurrent substrate, so changes in spatial credit flow are attributed to the input drive. Within every RQF comparison, the prospective and non-prospective variants match in architecture, optimizer, schedule, seeds, and parameter count; the look-ahead scale and base time constant are both τ = 5h. All tables report test accuracy as the mean ± sample standard deviation over five seeds, and Appendix E gives the complete parameterization and training protocol. Under full BPTT, prospective-input coding raises mean test accuracy in all matched depth, width, and feature comparisons (Tables 1 and 5). The smallest raw-audio models reach 94.59 ± 0.28%, 95.66 ± 0.23%, and 96.09 ± 0.23% at two, four, and six layers, respectively, while using only 23.5k–31.9k parameters (Table 5). The spatial-only runs reveal the predicted depth dependence more clearly (Table 1): on MFCC inputs, the prospective advantage grows from 0.69 to 3.59 to 15.17 points as depth increases. The direct gradient measurements connect this behavioral result to the analysis. Under spatial-only backpropagation, prospective inputs produce larger spatial per-hop gains and early-layer weightgradient norms than instantaneous inputs. Thus, the advantage of prospective input is evident when learning depends on the spatial path isolated by the theory (Appendix D.2; Tables 7 and 8). We next test whether the prospective construction transfers to a conventional residual SSM by comparing αP-S5 with native S5 [25]. We define αP-S5 by scaling each row of the learned S5 input e matrix with the corresponding native-clock pole decay rate αp = −Re(λp ), i.e., Bc = diag(α)B. Prospective input uses the shared physical look-ahead horizon τ = 5h, matching the RQF models. Appendix E.3 and Eq. (E.11) give the complete architecture and discretization. The mean test accuracy of αP-S5 is higher at every tested depth, width, and input type (Tables 2 and 6); because αP-S5 also rescales the input matrix, this comparison tests the complete construction rather than the second tap alone. These experiments also place the minimal RQF architecture in context. At width 32, the RQF is more accurate than αP-S5 at every raw-audio depth while using fewer parameters; at six layers, it reaches 96.09 ± 0.23% with 31.9k parameters, compared with 95.06 ± 0.99% with 40.9k for αP-S5 (Tables 5 and 6). To determine whether the effect depends on a linear RQF or SSM recurrence, we also apply the input correction to discrete residual ORGaNICs, a nonlinear recurrent substrate with dynamic, recurrent normalization [23, 31]. Prospective-input coding improves the full BPTT mean at two and four layers and ties at six layers. Under spatial-only backpropagation, the prospective model improves at four and six layers (Table 3). The correction therefore transfers beyond the linear RQF core. 8

Speech is well matched to a bank of tempo- Table 4: LRA Path-X, full BPTT. Width-64 RQFs ral band-pass filters, but it does not establish on the 16,384-step task (chance: 50%). whether the same minimal RQF architecture Depth Prospective Non-pros. can learn structure that is neither acoustic nor 4 82.19±0.88 81.14±1.00 naturally localized in frequency. We therefore 6 83.56±2.11 81.63±0.80 turn to Path-X from Long Range Arena [32], a binary classification task formed by flattening a 128 × 128 image into a 16,384-step sequence whose relevant endpoints can be separated across the full input. We retain the forward-only, non-residual RQF model, set the width to 64, and train with full BPTT using a Path-X-specific optimization schedule (Appendix E.4). The four-layer prospective and non-prospective models reach 82.19 ± 0.88% and 81.14 ± 1.00%, respectively; at six layers, they reach 83.56 ± 2.11% and 81.63 ± 0.80% (Table 4). Therefore, without bidirectionality, normalization, gating, or an expanded state, the minimal RQF stack learns a fully non-local 16,384-step task more than 30 points above chance. Making the input prospective leads to a more accurate model.

5

Related work

Prospective coding in recurrent neural circuits. Prospective coding has been used to address timing mismatch and credit assignment in biologically motivated learning rules. Latent equilibrium uses the look-ahead variable u + τ u̇ to recover fast feedforward computation and local error propagation [2]. Neuronal least-action formulates a least-action principle over future-discounted voltages, yielding prospective firing rates and prospective somato-dendritic mismatch errors [10]. Generalized latent equilibrium separates retrospective membrane integration from prospective output dynamics, yielding local spatio-temporal credit assignment [11]. Recent work has also identified cellular mechanisms for prospective and retrospective firing [5] and framed prospective neurons as a solution to teaching-signal synchronization [33]. Our work complements these circuit theories by showing that prospectivity can also serve as an input-side architectural addition for deep recurrent networks. Normalization, ORGaNICs, and quadrature dynamics. The RQF construction also belongs to a line of recurrent cortical circuit models in which dynamics, prediction, and normalization are coupled. Heeger [3] modeled neural activity as a compromise between feedforward drive, feedback drive, and prior drive, with prediction over time implemented by recursive quadrature pairs. ORGaNICs extend this idea to gated recurrent neural integrator circuits with dynamic normalization [23], a canonical neural computation [18]. Most prior ORGaNICs work has focused on stability and normalization [31, 34, 35]; RQFs were also combined with ORGaNICs to model temporal-frequency selectivity and direction selectivity of neurons in primary visual cortex [12]. The RQF is the corresponding single-channel complex primitive, whose complex state carries the quadrature pair. Credit assignment and trainability. The trainability problem we study is related to, but distinct from, the classical vanishing-gradient problem in RNNs, which arises from products of recurrent Jacobians through time [36]. Online alternatives such as RTRL avoid reverse-time unrolling but introduce large sensitivity states [37]; recent work makes this more tractable by exploiting independent recurrent modules across layers [38]. Equilibrium propagation, Lagrangian extensions, and Hamiltonian echo methods instead recover gradient information from the response or time reversal of physical dynamical systems [39–42]. Our learning mode of spatial-only backpropagation removes reverse-time temporal credit but does not solve the separate weight-transport problem across layers. State-space models and discretization upgrades. SSMs have made diagonal recurrent substrates practical for long sequences. S4 introduced efficient structured state-space layers [27], while DSS and S4D showed that complex diagonal state matrices can retain much of this performance with simpler implementations [22, 21]. S5 and LRU refine this family through MIMO scans and carefully parameterized linear recurrences [25, 28]. Mamba addresses a different limitation by making discretized parameters input-dependent [26]. We deliberately keep the parameters input-independent so that the prospective-input intervention and its gradient scaling can be isolated. Mamba-3 also uses two input taps, but its convex endpoint interpolation is analytically distinct from the signed prospective difference (Section 3) [29]. 9

6

Discussion

Takeaways for machine learning. Prospective-input coding is a two-tap change to a diagonal SSM’s input path that adds no parameters and preserves the parallel scan. Prospective RQFs yield higher mean test accuracy than their matched controls in all configurations trained with full BPTT; the prospective αP-S5 construction likewise exceeds native S5 in all comparisons. Separately, RQFs are a special case of diagonal SSMs with clear advantages: a stack of these biologically motivated filters, carrying no normalization, no gating, and no residual path, is more accurate than S5 at every depth on raw audio while using fewer parameters. Takeaways for computational neuroscience. Prospective firing has been measured in biological neurons, and the look-ahead operator that models it exactly inverts a first-order membrane low-pass filter. Our results give that inverse a specific role: carrying a teaching signal through a deep cascade of leaky integrators. The evidence is mechanistic rather than only behavioral. Under spatial-only backpropagation in a deep recurrent network, we measure the error gain surviving each backward hop across layers, and a prospective drive preserves several times more of it than an instantaneous one. The consequence is that temporally local learning reaches a new scale: a six-layer stack that never differentiates its recurrence through time reaches 83% on 16,000-step raw audio, far longer than the sequences on which the local rules of NLA and GLE have been demonstrated. Limitations. The prospective-input correction is not directly applicable to discrete-time gated RNNs such as LSTMs and GRUs [43, 44]. The operator χτ = 1 + τ d/dt is defined on a continuous-time trajectory and uses τ as an intrinsic integration timescale, whereas standard LSTMs and GRUs are specified by per-step algebraic update equations. A discrete architecture can be embedded in many possible continuous flows, but without such a substrate, there is no canonical derivative or effective time constant to invert. Our proof isolates the linear RQF substrate under spatial-only backpropagation. Nonlinear activations, residual connections, normalization layers, input-dependent parameters, and imperfect choices of the look-ahead scale can all change the size of the empirical gain. Finally, we did not attempt to maximize performance in this paper; future work should learn or adapt the effective time constants, use multi-timescale look-ahead operators, optimize the pole initialization, and test residual and normalization choices. Selective RQFs and future work. Mamba makes state-space parameters input-dependent, whereas the present controlled setting uses fixed γ and ω. A selective extension could use γt = γmin + (γmax − γmin ) sigmoid(gγ (xt )), ωt = ωmin + (ωmax − ωmin ) sigmoid(gω (xt )),

(18)

while retaining a selective associative scan. Associative recall, state tracking, and language modeling would test whether prospective input complements selectivity. Spatial-only backpropagation isolates within-step spatial credit assignment but deliberately discards credit through recurrent time. Extending the prospective-input mechanism to local learning rules that retain useful temporal credit is therefore an open problem. Temporal credit assignment in RNNs is difficult because gradients propagated through time can vanish or explode [36]; deriving local alternatives has consequently been a long-standing goal [37, 38, 10, 11]. Biological synapses do not receive gradients produced by unrolling an entire recurrent computation, yet the brain learns from temporally extended experience. The goal is to use neuroscience principles to identify recurrent dynamical models whose learning rules use only local information (local errors and activity) while retaining useful credit assignment through time.

Acknowledgments and Disclosure of Funding We thank Stefano Martiniani, Jenny Listman, Kyle Stratis, Ruben Coen-Cagli, and Wayne Mackey for helpful discussions.

References [1] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014. 10

[2] Paul Haider, Benjamin Ellenberger, Laura Kriener, Jakob Jordan, Walter Senn, and Mihai A Petrovici. Latent equilibrium: a unified learning theory for arbitrarily fast computation with arbitrarily slow neurons. Advances in neural information processing systems, 34:17839–17851, 2021. [3] David J Heeger. Theory of cortical function. Proceedings of the National Academy of Sciences, 114(8):1773–1782, 2017. [4] Harold Köndgen, Caroline Geisler, Stefano Fusi, Xiao-Jing Wang, Hans-Rudolf Lüscher, and Michele Giugliano. The dynamical response properties of neocortical neurons to temporally modulated noisy inputs in vitro. Cerebral cortex, 18(9):2086–2097, 2008. [5] Simon Brandt, Mihai Alexandru Petrovici, Walter Senn, Katharina Anna Wilmes, and Federico Benitez. Prospective and retrospective coding in cortical neurons. arXiv preprint arXiv:2405.14810, 2024. [6] Daniel Ulrich. Dendritic resonance in rat neocortical pyramidal cells. Journal of neurophysiology, 87(6):2753–2759, 2002. [7] Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a sensory population. Proceedings of the National Academy of Sciences, 112(22):6908–6913, 2015. [8] Srdjan Ostojic, Germán Szapiro, Eric Schwartz, Boris Barbour, Nicolas Brunel, and Vincent Hakim. Neuronal morphology generates high-frequency firing resonance. Journal of Neuroscience, 35(18):7056–7068, 2015. [9] Michael Dimitriou and Benoni B Edin. Human muscle spindles act as forward sensory models. Current Biology, 20(19):1763–1767, 2010. [10] Walter Senn, Dominik Dold, Akos F Kungl, Benjamin Ellenberger, Jakob Jordan, Yoshua Bengio, João Sacramento, and Mihai A Petrovici. A neuronal least-action principle for real-time learning in cortical circuits. ELife, 12:RP89674, 2024. [11] Benjamin Ellenberger, Paul Haider, Federico Benitez, Jakob Jordan, Kevin Max, Ismael Jaras, Laura Kriener, and Mihai A Petrovici. Backpropagation through space, time and the brain. Nature Communications, 17(1):66, 2026. doi: 10.1038/s41467-025-66666-z. [12] David J Heeger and Klavdia O Zemlianova. A recurrent circuit implements normalization, simulating the dynamics of v1 activity. Proceedings of the National Academy of Sciences, 117 (36):22494–22505, 2020. [13] Rodney J Douglas, Christof Koch, Misha Mahowald, Kevan AC Martin, and Humbert H Suarez. Recurrent excitation in neocortical circuits. Science, 269(5226):981–985, 1995. [14] David J Heeger, Eero P Simoncelli, and J Anthony Movshon. Computational models of cortical visual processing. Proceedings of the National Academy of Sciences, 93(2):623–627, 1996. [15] Eero P Simoncelli and David J Heeger. A model of neuronal responses in visual area mt. Vision research, 38(5):743–761, 1998. [16] Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11):1019–1025, 1999. [17] Maximilian Riesenhuber and Tomaso Poggio. Neural mechanisms of object recognition. Current opinion in neurobiology, 12(2):162–168, 2002. [18] Matteo Carandini and David J Heeger. Normalization as a canonical neural computation. Nature reviews neuroscience, 13(1):51–62, 2012. [19] Timothy D Oleskiw, Justin D Lieber, Eero P Simoncelli, and J Anthony Movshon. Foundations of visual form selectivity for neurons in macaque areas v1 and v2. bioRxiv, 2024. doi: 10.1101/2024.03.04.583307. [20] Georg Von Békésy. Experiments in hearing. McGraw Hill, 1960. 11

[21] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in neural information processing systems, 35:35971–35983, 2022. [22] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in neural information processing systems, 35:22982–22994, 2022. [23] David J Heeger and Wayne E Mackey. Oscillatory recurrent gated neural integrator circuits (organics), a unifying theoretical framework for neural dynamics. Proceedings of the National Academy of Sciences, 116(45):22783–22794, 2019. [24] Bruce A Francis and Walter Murray Wonham. The internal model principle of control theory. Automatica, 12(5):457–465, 1976. [25] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933, 2022. [26] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [27] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. [28] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International conference on machine learning, pages 26670–26698. PMLR, 2023. [29] Aakash Lahoti, Kevin Y Li, Berlin Chen, Caitlin Wang, Aviv Bick, J Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved sequence modeling using state space principles. arXiv preprint arXiv:2603.15569, 2026. [30] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018. [31] Shivang Rawat, David J Heeger, and Stefano Martiniani. Unconditional stability of a recurrent neural circuit implementing divisive normalization. Advances in Neural Information Processing Systems, 37:14712–14750, 2024. [32] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020. [33] Nicolas Zucchet, Qianqian Feng, Axel Laborieux, Friedemann Zenke, Walter Senn, and João Sacramento. Teaching signal synchronization in deep neural networks with prospective neurons. arXiv preprint arXiv:2511.14917, 2025. [34] Flaviano Morone, Shivang Rawat, David J Heeger, and Stefano Martiniani. Stabilization of recurrent neural networks through divisive normalization. Proceedings of the National Academy of Sciences, 123(30):e2601841123, 2026. doi: 10.1073/pnas.2601841123. [35] Asit Pal, Shivang Rawat, David J Heeger, and Stefano Martiniani. Hierarchical neural circuit theory of normalization and inter-areal communication. eLife, Aug 2026. doi: 10.7554/eLife. 111297.1. [36] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318. Pmlr, 2013. [37] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280, 1989. [38] Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and Joao Sacramento. Online learning of long-range dependencies. Advances in Neural Information Processing Systems, 36: 10477–10493, 2023. 12

[39] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017. [40] Serge Massar. Equilibrium propagation for learning in lagrangian dynamical systems. Physical Review E, 112(3):035304, 2025. [41] Victor Lopez-Pastor and Florian Marquardt. Self-learning machines based on hamiltonian echo backpropagation. Physical Review X, 13(3):031020, 2023. [42] Guillaume Pourcel and Maxence Ernoult. Learning long range dependencies through time reversal symmetry breaking. arXiv preprint arXiv:2506.05259, 2025. [43] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [44] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.

13

A

Details on Recursive Quadrature Filters (RQFs)

This appendix collects the technical material supporting the construction in Section 2. We first recover the leaky-integrator substrate of Eq. (2) by modeling y as a dynamical process that minimizes a single-channel slice of the cortical energy function of Heeger [3] over time, and then collect the elementary linear-dynamical-systems facts that motivate the band-pass interpretation, the special-case relationship to SSMs, and the equivalence to a damped harmonic oscillator. A.1

Energy-based derivation of RQFs

The starting point is Eq. 1 of Heeger [3], which includes a feedforward residual and a prior residual at every layer:   X (i) (i) X (i)
2
X (i)
2 (i) (i) (i) . EH = αH λH yj − z j + 1 − λH yj − ybj (A.1) i

j

j

We select one response coordinate, omit the terms contributed by the next layer, and condition on the supplied feedforward target z and prior target yb. We extend this scalar compromise to two real quadrature coordinates with shared parameters and encode them as y = y1 + ιy2 (and likewise for z and yb). The sum of the paired real squared residuals is then a complex modulus. Although Heeger’s αH (t) and λH (t) may vary, the RQF construction holds them fixed during relaxation, takes αH to be common to a quadrature pair, and identifies a learned per-channel γ = λH (the subscript distinguishes Heeger’s symbols from the RQF decay rate α and pole λ used elsewhere). A channel-dependent αH would instead induce channel-dependent effective base time constants. Let τH denote Heeger’s relaxation time. Gradient descent on the selected quadrature pair gives τH ẏ = −2αH [γ(y − z) + (1 − γ)(y − yb)] .

(A.2)

Defining τ ≜ τH /(2αH ) absorbs the common scale. Equivalently, the normalized pair energy is ∂E = γ(y − z) + (1 − γ)(y − yb), ∂y †

E = γ|y − z|2 + (1 − γ)|y − yb|2 ,

(A.3)

and, with z and yb held fixed, its Wirtinger-gradient dynamics τ ẏ = −∂E/∂y † are τ ẏ = −y + γz + (1 − γ)b y.

(A.4)

Along this flow, dE ∂E ∂E 2 ∂E = ẏ + † ẏ † = − dt ∂y ∂y τ ∂y †

2

≤ 0.

Thus, the energy is nonincreasing and decreases strictly away from stationary points, while the dynamics recover Eq. (2). The recurrent prior (b y ) comes from the quadrature predictor in Eq. 3 of Heeger [3]. For angular frequency Ω, its real-coordinate form is      yb1 (t) cos(Ω∆t) − sin(Ω∆t) y1 (t − ∆t) = , (A.5) yb2 (t) sin(Ω∆t) cos(Ω∆t) y2 (t − ∆t) or yb(t) = eιΩ∆t y(t − ∆t): the state sampled ∆t earlier is rotated forward to predict the present. The RQF applies the same rotation to the present state to predict one look-ahead horizon ahead, yb = eιωτ y with ω = Ω and ∆t = τ , and keeps the first-order term 
 eιωτ y = 1 + ιωτ + O (ωτ )2 y, yb ≜ χτ (y)|ẏ=ιωy = (1 + ιωτ )y. (A.6) Substitution into Eq. (A.4) yields τ ẏ = −y + γz + (1 − γ)(1 + ιωτ )y, which recovers the RQF in Eq. (4). 14

A.2

State-space form, oscillator equivalence, and cascade bandwidth

Two-dimensional real state-space form. We write the scalar complex-valued equation in Eq. (4) as a real two-dimensional coupled dynamical system to expose an equivalence that is invisible in the complex form: the RQF is a damped harmonic oscillator. Decomposing the complex state y = u + ιv and the input z = zR + ιzI into their real and imaginary parts, Eq. (4) is equivalent to the two-dimensional real linear system u̇ = −αu − ω0 v + αzR ,

v̇ = ω0 u − αv + αzI ,

(A.7)

or, inmatrix form, ẋ = Ax + Bz with x = (u, v)⊤ , z = (zR , zI )⊤ , A = −αI + ω0 J, B = αI, and J = 01 −1 0 , the standard planar rotation generator. The matrix A therefore implements an isotropic exponential decay at rate α combined with a planar rotation at angular velocity ω0 . The eigenvalues follow from det(A − λI) = (λ + α)2 + ω02 = 0, giving the complex conjugate pair λ1,2 = −α ± ιω0 ; both lie strictly in the open left half-plane for any γ ∈ (0, 1), so the filter is unconditionally stable. The free response is x(t) = e−αt R(ω0 t) x(0) with R(θ) a planar rotation, and the impulse response of the scalar complex form is the damped complex exponential h(t) = α e−αt (cos(ω0 t) + ι sin(ω0 t)) for t ≥ 0, the same primitive that diagonal complex SSMs convolve with their input sequence. Frequency response and quality factor. The band-pass interpretation follows directly from the scalar form ẏ = λy + αz. With a zero initial condition, the transfer function from z to y is H(s) = α/(s − λ) = α/(s + α − ιω0 ), so its magnitude on the imaginary axis is α |H(ιΩ)| = p , (A.8) 2 α + (Ω − ω0 )2 which is a Lorentzian peaked at unity gain when Ω = ω0 and has full half-power bandwidth ∆Ω3dB = 2α. The two learnable parameters set the two axes of this response: ω sets the phase-prior frequency and, through ω0 = (1 − γ)ω, the resonance location, while γ sets the bandwidth through α = γ/τ . Combining them gives the band-pass quality factor Q =

ω0 (1 − γ) ω τ = . 2α 2γ

(A.9)

Small γ therefore yields a narrow band-pass filter centered near the phase-prior frequency ω, whereas large γ broadens the gain toward first-order low-pass integration. Equivalence with a damped harmonic oscillator. A scalar damped harmonic oscillator with natural (undamped) frequency ωn and damping ratio ζ obeys the standard second-order equation ẍ + 2ζωn ẋ + ωn2 x = F (t),

(A.10)

where F (t) is an external forcing; the damping ratio ζ controls the qualitative regime, with the system being undamped when ζ = 0, underdamped when 0 < ζ < 1 (the impulse response is a decaying sinusoid), critically damped when ζ = 1, and overdamped when ζ > 1. To put the RQF in this form, we differentiate the first equation in Eq. (A.7), substitute v̇ from the second, and eliminate the residual v using the first equation again. The result is a single second-order ODE in u,
ü + 2α u̇ + (α2 + ω02 ) u = α żR + αzR − αω0 zI , (A.11) which has exactly the form of Eq. (A.10) with the external forcing
F (t) = α żR (t) + α zR (t) − αω0 zI (t),

(A.12)

built from the in-phase and quadrature components of z: a proportional-plus-derivative response α(żR + αzR ) to the real part, plus a static coupling −αω0 zI to the imaginary part. Factoring α out of the real-part term and introducing the effective time constant τ0 ≡ 1/α = τ /γ exhibits the in-phase drive as a prospective signal on the timescale τ0 : α(żR + αzR ) = α2 (zR + τ0 żR ) = α2 χτ0 (zR ). The forcing can therefore be written compactly as F (t) = α2 χτ0 (zR (t)) − αω0 zI (t).

(A.13)

Matching coefficients between Eqs. (A.10) and (A.11) reads off the oscillator parameters as q α α ωn = ζ = p = , (A.14) α2 + ω02 = |λ|, 2 2 |λ| α + ω0 15

and substituting the RQF parameterization α = γ/τ and ω0 = (1 − γ)ω gives the closed-form map from the learnable parameters (ω, γ, τ ) to the oscillator parameters, 1p 2 γ ωn = γ + (1 − γ)2 ω 2 τ 2 , ζ = p . (A.15) τ γ 2 + (1 − γ)2 ω 2 τ 2 For any γ ∈ (0, 1) and ωτ > 0, the inequality ζ < 1 holds strictly, so an RQF isp always underdamped: its impulse response is a sinusoid at the damped oscillation frequency ω0 = ωn 1 − ζ 2 contained in an e−αt envelope, and its magnitude response is the Lorentzian in Eq. (A.8). The standard oscillator quality factor Qosc = 1/(2ζ) = ωn /(2α) and thepband-pass quality factor Q = ω0 /(2α) from Eq. (A.9) therefore differ by the factor ωn /ω0 = 1/ 1 − ζ 2 , which approaches unity in the high-Q (low-ζ) regime, and the two coincide as ζ → 0. Cascade bandwidth. Here, we summarize the effect of cascading RQFs, which could be an architectural knob for adjusting per-channel selectivity. A single RQF has a Lorentzian magnitude response with full half-power bandwidth BW1 = 2α set by Eq. (A.8). Sharper per-channel selectivity at fixed α can be obtained by cascading N identical RQFs within a layer, i.e. in series with no nonlinearity between them. With N identical RQFs in series, the transfer function is HN (s) = H(s)N , so by Eq. (A.8) the cascade magnitude squared is |HN (ιΩ)|2 =

α2N α2 + (Ω − ω0 )2

N ,

(A.16)

which peaks at unity at Ω = ω0 for every N , since stacking unity-gain filters preserves unity gain at resonance. Setting |HN (ιΩ)|2 = 1/2 and solving for the frequency offset,
N
α2 + (Ω − ω0 )2 = 2 α2N ⇐⇒ (Ω − ω0 )2 = 21/N − 1 α2 , (A.17) √ gives the full half-power bandwidth BWN = 2α 21/N − 1 and therefore the cascade bandwidth ratio r p ln 2 0.833 BWN 1/N (A.18) 2 − 1 −−−−→ ≈ √ , = N →∞ BW1 N N 2 where the asymptotic form follows from 21/N = 1 + (ln 2)/N + O(1/N √ ). A four-fold within-layer 1/4 cascade, for example, would narrow the per-channel bandwidth to 2 − 1 ≈ 0.435 of a singlestage filter at fixed α while requiring N × the computation on the linear path and no extra learnable parameters (the cascade shares the channel’s (ω0 , α)). We do not exploit this knob in the present work; every layer uses N = 1, but it remains an inexpensive design choice.

B

Discretization

This appendix gives the full derivations of the ZOH discrete update in Eq. (9), stated in Section 2, and the impulse-ZOH update for prospective-input coding that produces the two-tap input path referenced in Section 3. As in the main text, z(ℓ) denotes the compact post-weight drive used for forward discretization; Appendix C expands z(ℓ) = W(ℓ) x(ℓ) when deriving learning rules. B.1

Instantaneous input

Starting from the per-layer ODE ẏ(ℓ) = λ(ℓ) ⊙ y(ℓ) + α(ℓ) ⊙ z(ℓ) in Eq. (6), the element-wise decoupling reduces the vector ODE to nℓ independent scalar problems, each admitting the variationof-constants solution over [tn , tn + h]: Z h (ℓ) (ℓ) y(ℓ) (tn + h) = eλ h ⊙ y(ℓ) (tn ) + α(ℓ) ⊙ eλ (h−s) ⊙ z(ℓ) (tn + s) ds, (B.1) 0

where the exponential is evaluated element-wise because λ(ℓ) is a vector. The zero-order hold scheme (ℓ) (ℓ−1) replaces z(ℓ) (tn + s) on (0, h] by the right-endpoint sample zn+1 = W(ℓ) ρ(yn+1 ), pulls it out of the integral, and evaluates the remaining kernel integral via u = h − s: Z h Z h (ℓ) (ℓ) (ℓ) eλ h − 1 eλ (h−s) ds = eλ u du = , (B.2) λ(ℓ) 0 0 16

(ℓ)

(ℓ)

with division taken element-wise (well-defined because Re(λi ) = −αi < 0). Substituting the (ℓ) (ℓ) (ℓ) (ℓ) coefficients Āh ≜ eλ h and B̄h ≜ (α(ℓ) /λ(ℓ) ) ⊙ (Āh − 1) from Eq. (10) yields the diagonal complex affine recurrence of Eq. (9). Both coefficients depend only on the per-neuron parameters (γ (ℓ) , ω (ℓ) ), the base time constant τ , and the step size h, and can therefore be precomputed once per parameter update and reused across all time steps. The homogeneous recurrence is unconditionally stable, since |eλh | = e−αh < 1 for any α > 0, and the ZOH update differs from forward Euler by O((h/τ )2 ) for h ≪ τ . B.2

Prospective input

For the prospective-input variant, applying χτ to the layer input x(ℓ) before W(ℓ) is equivalent in the forward pass to driving Eq. (6) by χτ (z(ℓ) ) = z(ℓ) + τ ż(ℓ) in place of z(ℓ) . The variation-of-constants integral of Eq. (B.1) therefore picks up an additional τ ż(ℓ) contribution. We model z(ℓ) on [tn , tn +h] (ℓ) as a right-continuous piecewise-constant signal that holds value zn+1 throughout the interval and (ℓ) (ℓ) that jumps from zn to zn+1 at the left endpoint s = 0; this is the same hold convention used for the instantaneous-input ZOH update of Appendix B.1, made explicit at the boundary because the prospective term involves ż(ℓ) . Under this convention, ż(ℓ) is a Dirac impulse at s = 0 of magnitude equal to the jump,   (ℓ) ż(ℓ) (tn + s) = zn+1 − z(ℓ) δ(s), (B.3) n (ℓ)

and the look-ahead χτ (z(ℓ) ) splits additively into a constant part zn+1 on [0, h] and an impulsive (ℓ) (ℓ) part τ (zn+1 − zn )δ(s) at s = 0. Each part contributes in closed form to the integral of Eq. (B.1). (ℓ) (ℓ) The constant part reproduces the calculation of Appendix B.1 and contributes B̄h ⊙ zn+1 . The (ℓ) (ℓ) (ℓ) impulsive part is sampled by the kernel at s = 0, where eλ (h−s) = eλ h = Āh (we use the Rh standard convention that the boundary impulse at s = 0 is included in 0 ). Multiplying the impulse strength by this kernel value and by the prefactor α(ℓ) from Eq. (B.1) gives     (ℓ) (ℓ) (ℓ) (ℓ) α(ℓ) τ ⊙ Āh ⊙ zn+1 − z(ℓ) = γ (ℓ) ⊙ Āh ⊙ zn+1 − z(ℓ) , (B.4) n n where the second equality uses α(ℓ) τ = γ (ℓ) from α(ℓ) = γ (ℓ) /τ . Summing the two contributions and grouping by input sample yields the impulse-ZOH update of Eq. (16) in the main text. No finite-difference approximation has been introduced: the update is exact under the right-continuous piecewise-constant model of z(ℓ) . B.3

Two-tap form for diagonal SSMs

The preceding derivation does not rely on the RQF parameterization. To transfer the prospective-input correction to the diagonal SSM core of Eq. (12), we keep the SSM parameters fixed and replace the input signal entering the continuous-time state equation by its look-ahead: e˙ (t) = Λ y e (t) + B χτ (x(t)) = Λ y e (t) + B[x(t) + τ ẋ(t)] , y

e (t) + D x(t), (B.5) o(t) = C y

where τ is the look-ahead scale in χτ and (Λ, B, C, D) are exactly the SSM parameters from Eq. (12). Thus, the intervention is not a reparameterization of the SSM input matrix: B remains the continuous-time input matrix, and the only change is that the state receives χτ (x) instead of x. The feedthrough path is written with x(t) because the prospective-input correction concerns the recurrent state input; applying the same look-ahead to the direct path would be a separate architectural choice. Under the same right-continuous hold convention used in Appendix B.2, x(tn + s) takes value xn+1 on (0, h] and has a jump from xn to xn+1 at s = 0. Therefore, ẋ(tn + s) = (xn+1 − xn )δ(s), and the variation-of-constants formula gives en+1 = Āh y en + B̄h xn+1 + τ Āh B (xn+1 − xn ), y

(B.6)

where Āh = eΛh and B̄h = Λ−1 (Āh − I)B are the standard ZOH coefficients from Eq. (13). Grouping terms by input sample yields the two-tap input path en+1 = Āh y en + B̄+ xn+1 + B̄− xn , y

B̄+ ≜ B̄h + τ Āh B, 17

B̄− ≜ −τ Āh B. (B.7)

Eq. (B.7) is the direct SSM analogue of Eq. (16). The recurrent dynamics are unchanged: the state is advanced by the same diagonal transition Āh , while the input enters through a two-tap feedforward path on the current and previous samples. For S4D, this amounts to replacing the one-tap ZOH input coefficient B̄h by the pair (B̄+ , B̄− ), whose values are determined by the original continuous-time input matrix B, the transition Āh , and the look-ahead scale τ ; the pole parameterization, state transition, output projection, and feedthrough term are otherwise unchanged. The same coefficient replacement applies to any diagonal state-space model whose continuous-time core has the form of Eq. (12).

C

Spatial-only backpropagation gradient derivation: full proof

This appendix gives the gradient calculation supporting Section 3. We consider an arbitrary realvalued loss local in time,   (L) (L)† Lk+1 = Φk+1 yk+1 , yk+1 , (C.1) where Φk+1 may include labels, readouts, or other quantities sampled at the same time step but does (ℓ) not backpropagate through earlier recurrent states. Spatial-only backpropagation freezes yk for (0) (L) every layer ℓ and differentiates only through the within-step spatial chain yk+1 → · · · → yk+1 . Unlike the forward-discretization derivation, we expand the compact drive as z(ℓ) = W(ℓ) x(ℓ) (ℓ) (ℓ−1) (ℓ) (ℓ−1) because the gradients must pass through W(ℓ) . We write xk+1 = ρ(yk+1 ) and xk = ρ(yk ), with x(1) supplied by the external input. Because ρ may be non-holomorphic, the exact chain rule is most transparent in real coordinates. For a complex vector and matrix, respectively, define     Re(v) Re(W) −Im(W) R(v) ≜ , R(W) ≜ , (C.2) Im(v) Im(W) Re(W) so that R(Wv) = R(W)R(v). For a real vector g, let D(g) ≜ R(diag(g)), and define the real activation Jacobian and layer error by Jρ (y) ≜

∂R(ρ(y)) , ∂R(y)

(ℓ)

δR,k+1 ≜ ∇R(y(ℓ) ) Lk+1 .

(C.3)

k+1

For split-ReLU, Jρ is diagonal, with the real and imaginary activity masks on its two diagonal blocks, and ∥Jρ ∥2 ≤ 1 (using any standard subgradient at zero). Moreover, ∥R(W)∥2 = ∥W∥2 and ∥D(g)∥2 = ∥g∥∞ . For clarity, we use forward Euler in the derivation because it exposes the factor of h/τ directly. The ZOH implementation has the same leading-order spatial gradient coefficients in the (ℓ) (ℓ) small-step limit: Āh = 1 + λ(ℓ) h + O(h2 ) and B̄h = (h/τ )γ (ℓ) + O(h2 ), while the impulse-ZOH (ℓ) (ℓ) (ℓ) (ℓ) prospective coefficients satisfy B̄h + γ ⊙ Āh = γ (ℓ) + O(h) and −γ (ℓ) ⊙ Āh = −γ (ℓ) + O(h). Thus, the Euler calculation below proves the leading-order ZOH scaling as h/τ → 0. C.1

Spatial-only backward pass, instantaneous input

Forward Euler discretization of the instantaneous-input RQF layer is   h (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) γ , yk+1 = A(ℓ) ⊙ yk + gnp ⊙ W(ℓ) xk+1 , gnp ≜ τ

(C.4)

(ℓ)

where A(ℓ) = 1 + hλ(ℓ) . Since yk is frozen, the only live path from layer ℓ to layer ℓ + 1 runs (ℓ+1) (ℓ) through xk+1 = ρ(yk+1 ) and then through the matrix W(ℓ+1) . Applying the chain rule gives, for ℓ < L,  ⊤  ⊤  ⊤ (ℓ) (ℓ) (ℓ+1) (ℓ+1) δR,k+1 = Jρ yk+1 R W(ℓ+1) D gnp δR,k+1 . (C.5) Consequently, in the Euclidean norm, (ℓ)

δR,k+1

≤ 2

L  Y

  (m−1) Jρ yk+1

m=ℓ+1

18

W(m) 2

2

(m) gnp

 ∞

(L)

δR,k+1

. 2

(C.6)

When the weight norms, activation-Jacobian norms, and entries of γ (m) are O(1), Eq. (C.6) contains the explicit factor (h/τ )L−ℓ . This is the discretization-induced depth attenuation: it is separate from any ordinary conditioning effects caused by the weights or nonlinearities. To state the parameter gradient without ambiguity about Wirtinger factors, let C invert vector realification, C([u⊤ , v⊤ ]⊤ ) = (ℓ) (ℓ) u + ιv, and set δC,k+1 = C(δR,k+1 ). Define the packed complex gradient ∂L . ∂W∗ The corresponding local gradient contribution has the outer-product form   (ℓ) (ℓ)† (ℓ) ∇pack L = g ⊙ δ k+1 np C,k+1 xk+1 . W(ℓ) ∇pack W L ≜ ∇Re(W) L + ι ∇Im(W) L = 2

C.2

(C.7)

(C.8)

Spatial-only backward pass, prospective

For the prospective case, the look-ahead is applied to the layer input x(ℓ) before multiplication by W(ℓ) . Using a backward difference at tk+1 gives (ℓ)

(ℓ)

(ℓ)

ek+1 = xk+1 + τ x

(ℓ)

xk+1 − xk h

=

h + τ (ℓ) τ (ℓ) xk+1 − xk . h h

(C.9)

The forward Euler step is therefore     (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) yk+1 = A(ℓ) ⊙ yk + g+ ⊙ W(ℓ) xk+1 + g− ⊙ W(ℓ) xk , with (ℓ)

g+ ≜

h + τ (ℓ) γ , τ

(ℓ)

g− ≜ −γ (ℓ) .

(C.10)

(C.11)

(ℓ)

The previous input xk is frozen as a state variable for spatial error propagation, so it does not create (ℓ−1) (ℓ) a live path from yk+1 to yk+1 . The current weight matrix multiplying that frozen input is still (ℓ+1)

trainable. For the inter-layer error recursion, only the current-input coefficient g+  ⊤  ⊤  ⊤ (ℓ) (ℓ) (ℓ+1) (ℓ+1) δR,k+1 = Jρ yk+1 R W(ℓ+1) D g+ δR,k+1 .

appears: (C.12)

The corresponding norm bound is (ℓ)

δR,k+1

≤ 2

L  Y m=ℓ+1

  (m−1) Jρ yk+1

2



(m)

W(m) 2

g+

∞

(L)

δR,k+1

. 2

(C.13)

(m)

Since ∥g+ ∥∞ = (1 + h/τ )∥γ (m) ∥∞ , the explicit small factor (h/τ )L−ℓ is absent. More precisely, at fixed depth and fixed parameters, the discretization prefactor on each hop is O(1) rather than O(h/τ ) as h/τ → 0. The bounds in Eqs. (C.6) and (C.13) are upper bounds and provide no lower bound: the remaining product of activation-Jacobian norms, weight norms, and γ can still attenuate with depth. We therefore make no claim that the complete prospective gradient is O(1) or depth-independent. The local weight gradient contains both input taps:     (ℓ) (ℓ) (ℓ)† (ℓ) (ℓ) (ℓ)† ∇pack L = g+ ⊙ δC,k+1 xk+1 + g− ⊙ δC,k+1 xk . (C.14) W(ℓ) k+1 Eq. (C.14) is local in time up to the one-step input memory introduced by the two-tap path, and local (ℓ) in space once the adjacent real-coordinate error δR,k+1 is available. Exact spatial backpropagation supplies that error through R(W(ℓ+1) )⊤ and the activation Jacobian in Eqs. (C.5) and (C.12).

D

Experimental results

D.1

Full BPTT results with 32 filters per layer

Tables 5 and 6 report the width-32 results for RQF and αP-S5, respectively. 19

Table 5: RQF, full BPTT. Non-residual stacks with 32 filters per layer on Speech Commands. Raw audio Depth Params Prospective 2 4 6

MFCC Non-pros.

Params Prospective

Non-pros.

23.5k 94.59±0.28 93.96±0.28 24.1k 96.38±0.13 96.17±0.22 27.7k 95.66±0.23 95.29±0.22 28.3k 96.33±0.23 96.12±0.31 31.9k 96.09±0.23 95.27±0.22 32.5k 96.35±0.17 95.73±0.27

Table 6: αP-S5 versus native S5, full BPTT. Residual S5 blocks with 32 units per layer on Speech Commands; the complete αP construction changes the continuous-time input gain, adds the second tap, and uses the pole clipping described in Appendix E.3. Raw audio αP-S5

Depth Params 2 4 6

D.2

MFCC Native S5

Params

αP-S5

Native S5

27.9k 93.46±0.93 92.34±0.50 28.6k 96.04±0.32 95.36±0.24 34.4k 94.74±0.59 93.31±1.09 35.1k 96.31±0.32 95.84±0.32 40.9k 95.06±0.99 94.69±0.66 41.5k 96.44±0.28 95.84±0.18

Direct measurements of the spatial gradient path

These measurements test how much of the error signal survives as it propagates backward through the layers. We use a six-layer, 64-channel raw-audio RQF under spatial-only backpropagation at the paper’s operating point, h/τ = 0.2. Table 7 reports the weight-gradient norms: the two gradients are similar at layer 6, which is closest to the loss, but the instantaneous gradient decays much faster toward earlier layers. At layer 1, the prospective gradient is 10,290× larger. Table 7: Direct weight-gradient norms. Frobenius norms ∥∇pack L∥F at h/τ = 0.2. W(ℓ) Layer ℓ 1 2 3 4 5 6

Prospective

Instantaneous

−7

−11

7.69 × 10 5.71 × 10−6 1.03 × 10−5 1.90 × 10−5 3.08 × 10−5 5.42 × 10−5

7.47 × 10 6.01 × 10−9 5.61 × 10−8 5.62 × 10−7 5.04 × 10−6 4.87 × 10−5

Ratio 10,290× 950× 185× 34× 6.1× 1.1×

Table 8 sweeps the step size. We measure the per-hop error gain g as follows: with qℓ the RMS norm of ∂L/∂x(ℓ) , we fit g = (q2 /q6 )1/4 over the four hops from layer 6 to layer 2. The table compares the ratio gpros /ginst with the exact-ZOH current-tap prediction |B̄h + γ ⊙ Āh |/|B̄h | for the same filter bank. As h/τ decreases, gpros remains between 0.585 and 0.624, whereas ginst falls approximately in proportion to h/τ . The measured ratio follows the predicted increase across the sweep. Table 8: Per-hop error gain versus step size. gpros and ginst are the measured per-hop error gains for prospective and instantaneous input; prediction and measurement use the same six-layer, 64-channel filter bank. h/τ

Predicted ratio

gpros

ginst

Measured ratio

1.00 0.50 0.20 0.10 0.05

1.96 2.95 5.93 10.91 20.89

0.619 0.594 0.585 0.598 0.624

0.361 0.220 0.104 0.0565 0.0305

1.72 2.70 5.65 10.59 20.43

20

E

Architecture and hyperparameter details

E.1

RQF architecture

A bias-free real projection maps the input dimension din ∈ {1, 20} to width n ∈ {32, 64}. This is followed by L ∈ {2, 4, 6} square complex RQF layers of width n, with split-ReLU between layers and no normalization, gating, dropout, bidirectionality, or residual connections. At every step, the classifier concatenates the real and imaginary parts of the final state and applies a real MLP 2n → 256 → 10 with ReLU. Predictions use logits averaged across time. Parameter counts refer to real-valued trainable degrees of freedom, so a complex parameter counts twice. With C = 10 classes, the count is Nparams = din n + 2Ln2 + 2Ln + 512n + 2826. (E.1) Raw waveforms have shape (16,000, 1) and use h = 1/16,000 with fixed τ = 5h = 3.125 × 10−4 . MFCC inputs have shape (161, 20) and use their effective 160 Hz frame clock, h = 1/160 and τ = 5h = 0.03125. Path-X uses h = 1/16,384 and τ = 5h. No separate look-ahead scale is learned. Each RQF channel has complex state si , dimensionless frequency θi = τ ωi , bandwidth parameter γi , and continuous-time dynamics dsi γi = λi si + ui (t), dt τ

λi = −

γi + ι ωi (1 − γi ), τ

(E.2)

where ui (t) is the post-weight drive; si and ui are the per-channel entries of y(ℓ) and z(ℓ) in the main text. All runs use the ZOH recurrence in Eq. (E.3); prospective input replaces its single input coefficient with the two-tap coefficients in Eq. (E.4), with the delayed input initialized to zero. The scalar instantaneous recurrence is si,n+1 = Āh,i si,n + B̄h,i ui,n+1 ,

Āh,i = eλi h ,

B̄h,i =

γi Āh,i − 1 . τ λi

(E.3)

The prospective-input recurrence keeps the same state coefficient and replaces the input contribution by the two-tap drive
si,n+1 = Āh,i si,n + B̄h,i + γi Āh,i ui,n+1 − γi Āh,i ui,n , (E.4) with the previous input buffer initialized to zero. This is the scalar-channel version of Eq. (16). RQF parameterization and initialization. The learned leaves are log θi and log γi : θi = τ ωi = elog θi ,

γi = elog γi .

(E.5)

For a sequence with model-coordinate duration Tseq , initialization is log θi ∼ U(log θmin , log θmax ) ,

θmin =

2πτ , Tseq

θmax = 2πF,

γi =

θi . 1 + θi

(E.6)

All runs use F = 1. Speech Commands uses Tseq = 1 in the model clock for each input representation, while Path-X uses Tseq = 2π. After every optimizer step, θi is clamped to [θmin , 2π] and γi to [10−6 , 1 − 10−6 ]. Weight and readout initialization. Let nin be the complex fan-in, Gf = 1/2 the nominal RQF filter power gain, and Gρ the input activation’s second-moment gain. Each real and imaginary component of a complex weight is sampled independently as follows: 2 Re(Wij ), Im(Wij ) ∼ N (0, σW ),

σW = (2nin Gf Gρ )

−1/2

.

(E.7) √ Thus,pGρ = 1 for a direct standardized input and Gρ = 1/2 after split-ReLU, giving σW = 1/ nin and 2/nin , respectively. The√separate√real input projection retains the default PyTorch linear initialization, uniform on [−1/ din , 1/ din ], and its bias is removed. The first classifier layer √ √ likewise retains its default initialization, with weights and biases uniform on [−1/ 2n, 1/ 2n]. The 256 → C logit weights are normal with standard deviation 10−3 and zero bias. The reference implementation evaluates the recurrence in complex64 with autocasting disabled; optimized scan kernels preserve the same coefficients. 21

E.2

ORGaNICs architecture

The discrete ORGaNICs control follows the sequential model of Rawat et al. [31]. We fix the recurrent matrix in every layer to Wr = I, learn the intrinsic time constants, retain the gain and normalization dynamics, and stack L ∈ {2, 4, 6} layers of width 64. The reported parameter counts include the learned intrinsic time constants. Shape-matched layers use identity residual additions; the first 20 → 64 MFCC layer cannot use one. Layer normalization is disabled. For layer ℓ, let x(ℓ) (t) ∈ Rdℓ denote its input, with x(1) equal to the 20-dimensional MFCC sequence. Each layer has 64-dimensional excitatory state y(ℓ) , inhibitory normalization state a(ℓ) , input gain (ℓ) b(ℓ) , and semisaturation gain b0 . We use the rectified-input variant, defining [u]+ = max(u, 0) element-wise and h i z(ℓ) (t) =

(ℓ) (ℓ) Wzx x (t)

,

+

(ℓ) (ℓ) (ℓ) z(ℓ) χ (t) = z (t) + δχ τy ⊙ ż (t).

(E.8)

(ℓ)

Here, z(ℓ) (t) is the input drive and zχ (t) is the drive passed to the ORGaNICs input-gain pathway. The binary variable δχ selects the input-drive condition: δχ = 0 gives the instantaneous-input (ℓ) baseline, whereas δχ = 1 gives the prospective-input variant χτ (z(ℓ) (t)) with look-ahead scale τy . The gain modulators evolve dynamically toward the following instantaneous targets:   (ℓ) (ℓ) (ℓ) B(ℓ) (t) = σsig Wxb x(ℓ) (t) + Wyb y(ℓ) (t) + Wab a(ℓ) (t) ,   (E.9) (ℓ) (ℓ) (ℓ) (ℓ) B0 (t) = σsig Wx0 x(ℓ) (t) + Wy0 y(ℓ) (t) + Wa0 a(ℓ) (t) , (ℓ)

where σsig is the logistic sigmoid; B(ℓ) is the target for the input gain b(ℓ) ; and B0 is the target for (ℓ) the semisaturation gain b0 . b (ℓ) = [Wr y(ℓ) ]+ = [y(ℓ) ]+ . For the rectified-recurrence variant, define the recurrent prediction y The continuous-time ORGaNICs layer is then described by q    (ℓ) b (ℓ) , τy(ℓ) ⊙ ẏ(ℓ) = −y(ℓ) + b(ℓ) ⊙ z(ℓ) + 1 − a ⊙y χ + h  2 i2 h i  (ℓ) (ℓ) τa(ℓ) ⊙ ȧ(ℓ) = −a(ℓ) + σ02 b0 + Way y(ℓ) ⊙ a(ℓ) , (E.10) + + (ℓ)

τb

(ℓ)

⊙ ḃ(ℓ) = −b(ℓ) + B(ℓ) , (ℓ)

(ℓ)

(ℓ)

τb0 ⊙ ḃ0 = −b0 + B0 . All products, powers, divisions by time constants, and square roots in Eq. (E.10) are element-wise except for multiplication by learned matrices. In the implementation, Eq. (E.10) is integrated with an ordered forward-Euler step: the gain states are updated first, and those updated gains enter the principal and normalization updates. Approximating ż by a backward difference, the Euler step multiplies the prospective term b ⊙ τy ⊙ (zt − zt−1 )/∆t by ηy = ∆t/τy , so the two-tap prospective correction reduces to b ⊙ (zt − zt−1 ) in addition to the ordinary Euler drive, with the previous (ℓ) input-drive buffer initialized as z−1 = 0. The gain dynamics for b0 and b are otherwise identical in the prospective and instantaneous variants. (ℓ)

The normalization matrix Way is a learned full-rank positive matrix represented in log space; its initialization is 50% sparse off-diagonal with unit diagonal. √ Wzx and the six gain-target matrices in Eq. (E.9) use Kaiming-uniform initialization with a = 5. The semisaturation parameter is fixed to σ0 = 1. The four time-constant vectors are learned through bounded positive sigmoid maps; their unconstrained leaves are standard normal at initialization, with 0 < ∆t/τy , ∆t/τa < 0.05 and 0 < ∆t/τb , ∆t/τb0 < 0.1. The principal state y(ℓ) (0) is a normalized random positive vector; (ℓ) a(ℓ) (0) ∼ U(0, 1) is not normalized, b0 (0) = 1, and b(ℓ) (0) ∼ U(0, 1). A learned linear 64 → 10 readout produces per-step logits, which are averaged for evaluation. The input correction is the only difference between the members of each prospective/non-prospective pair. E.3

αP-S5 architecture and parameterization

The S5 controls use the native forward-only residual S5 architecture [25]. A real input projection maps din ∈ {1, 20} to n ∈ {32, 64} units per layer. The L ∈ {2, 4, 6} pre-normalized residual 22

blocks use this value of n, conjugate symmetry, eight HiPPO blocks, ZOH discretization, and a half-GLU pointwise map. The readout normalizes the final n-dimensional block state and projects it to 64 dimensions; that representation is mean-pooled and classified by a 64 → 256 → 10 GELU MLP with dropout 0.1. The reported runs use batch pre-normalization and are unidirectional. Let λ collect the diagonal poles in S5’s native clock, let Λ = diag(λ), and let ∆ collect the learned per-mode steps. For a physical input interval h, mode p has effective continuous-time pole e in the native clock. λeff,p = (∆p /h)λp . We define αp = −Re(λp ) > 0 and set Bc = diag(α)B e p: , so the α gain The corresponding physical input row is Beff,p: = (∆p /h)Bc,p: = −Re(λeff,p )B remains tied to the physical pole decay rate and is not learned separately. Prospective input uses the same shared physical horizon as the RQF experiments, τ = 5h. Substituting Beff,p: = (∆p /h)Bc,p: into the impulse-ZOH correction of Eq. (B.6) gives τ Āp Beff,p: = 5∆p Āp Bc,p: . Thus, the factor h cancels when the update is expressed in S5’s native clock, and the implementation computes the following coefficients:
Ā = diag eλ⊙∆ , B̄ = Λ−1 (Ā − I)Bc , B̄+ = B̄ + 5 diag(∆)ĀBc ,

(E.11)

B̄− = −5 diag(∆)ĀBc . It then advances ht+1 = Āht + B̄+ xt+1 + B̄− xt . The two taps span adjacent input samples; 5∆p is the native-clock impulse coefficient produced by the shared 5h horizon, not a learned or modedependent physical look-ahead. The delayed input starts at zero, and the direct D ⊙ xt feedthrough remains instantaneous. Within αP-S5, the two-tap operation adds no trainable parameters and leaves the recurrent transition, residual path, output projection, and scan structure unchanged. The poles and basis transformation use the standard HiPPO-DPLR initialization. Before transforma√ e and C e are LeCun-normal with standard deviation 1/ Plocal , and Di ∼ N (0, 1). tion, entries of B −3 −1 Log steps are initialized uniformly so that ∆p ∈ [10 , 10 ]. In the reference αP-S5 configuration, real pole parts are clipped to at most −10−4 before forming α, whereas the native one-tap S5 configuration does not apply this clipping and uses the unscaled learned input matrix. Tables 2 and 6 therefore compare the complete αP-S5 construction with native S5; they do not isolate the second tap while holding the continuous-time input map and pole clipping fixed. E.4

Optimizer and training

All Speech Commands runs use the 10-word subset of v0.02 with a stratified 70/15/15 split (split seed 0) and feature-wise standardization from the training split. MFCC extraction uses 20 coefficients, a 200-sample FFT window, 64 mel bands, and a 100-sample hop, producing 161 frames. A checkpoint is saved each epoch, and test accuracy is read from the checkpoint with the highest validation accuracy. Prospective/non-prospective pairs use identical training settings and seeds. Neither Speech Commands nor Path-X uses data augmentation. The Speech Commands v0.02 archive includes a CC BY 4.0 license. Path-X was obtained from the official LRA release; the accompanying code repository is Apache 2.0, but the downloaded data archive does not include separate dataset-license terms. RQF. RQF Speech Commands models use AdamW with a batch size of 32; the reported runs use seeds 42–46. Complex mixing weights, the input projection, and the readout use a learning rate of 10−3 and weight decay 10−4 ; RQF pole parameters use a learning rate of 10−4 and zero weight decay. The global gradient norm is clipped at 1.0. Training runs for at most 300 epochs with cosine annealing to 10−6 and stops after 20 epochs without improved validation accuracy. Full BPTT retains the complete temporal graph and applies cross entropy with label smoothing 0.1 to the mean-pooled logits. Spatial-only backpropagation detaches the recurrent state after every step and applies the same smoothed cross entropy per time step; validation and test predictions average logits across time. ORGaNICs. ORGaNICs uses AdamW with a learning rate of 10−3 and weight decay 10−4 , cross entropy with label smoothing 0.1, and gradient clipping at 1.0. The learning rate follows cosine annealing to 10−6 over a maximum of 300 epochs, with early stopping after 20 epochs without 23

improved validation accuracy. Full BPTT runs use the mean-pooled prediction loss; spatial-only backpropagation detaches every recurrent state after each step and applies the loss at each time step. αP-S5. S5 uses the same AdamW optimizer, base batch size, label smoothing, clipping, cosine schedule, early stopping, and checkpoint selection as the RQF Speech Commands runs; the reported e C, e and the log steps, use the base 10−3 runs use seeds 0–4. All S5 parameters, including Λ, B, learning rate and 10−4 weight decay. The native-clock gain α = −Re(diag(Λ)) introduces no optimizer state because it is derived from the poles. Path-X. We use the 200,000-example pathfinder128/curv_contour_length_14 task with a seeded 80/10/10 split (split seed 42). Pixels are divided by 255 and standardized by the training-split scalar mean and standard deviation. The four- and six-layer width-64 RQFs use Adam with ϵ = 10−4 , zero weight decay, a learning rate of 3 × 10−3 for all parameter groups, a batch size of 64, cross entropy at each time step, and gradient clipping at 1.0. Training runs for 300 epochs with cosine annealing to zero; evaluation averages logits over time and reports the test result at the best-validation epoch. The five runs use seeds 0–4 for initialization and data-shuffling order while retaining the same data split. These runs execute the complex recurrence as a real-pair scan in bfloat16 rather than through the complex64 Speech reference path. Path-X uses the same recurrence, log-space parameterization, split-ReLU, and real–imaginary MLP readout as the Speech model, but its first complex matrix maps the scalar pixel input directly to width 64; it does not use the separate real input projection used by the Speech model. The optimizer schedule and the initialization duration Tseq were chosen for Path-X rather than carried over from the Speech Commands runs.

24

Related documents

• Pretraining Recurrent Networks without Recurrence#259457
• Rethinking Speech Codecs: From Compression to Autoregressive Generative Modeling#927539
• ChronoSSM: Training for Temporally Aware Representations in Autoregressive State Space Models#444599
• Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall#1117895
• Switching Linear Attention#1144001