Adaptive Speech-to-Spike Encoding for Spiking Neural Networks Taharim Rahman Anon ID 1 , Jakaria Islam Emon ID 1 1
PI LLC, Sapporo, Hokkaido, Japan
[email protected], [email protected]
arXiv:2606.19039v1 [cs.NE] 17 Jun 2026
put representations, often requiring larger models to extract discriminative features.
Abstract The mismatch between continuous acoustic signals and discrete event-driven processing remains a fundamental bottleneck for neuromorphic speech processing. Current systems typically rely on fixed spike encoders, forcing downstream Spiking Neural Networks (SNNs) to compensate for non-adaptive input representations. To address this, we present a learnable residual speech-to-spike encoder jointly trained end-to-end with a Recurrent Leaky Integrate-and-Fire (R-LIF) backbone. We validate this approach on the Google Speech Commands v2 (GSCv2) benchmark, achieving up to 94.97% accuracy. Notably, the learned encoder remains highly parameter-efficient with a compact 35k-parameter variant that reaches 89.8%, matching or exceeding prior baselines that require an order of magnitude more parameters. Our encoder-focused analysis, including linear probing and gradient-residual inspection, indicates that the encoder does not target faithful signal reconstruction but instead learns task-aligned spike representations that enhance class separability. Finally, we benchmark bio-inspired, hardware-friendly credit assignment by comparing Direct Feedback Alignment (DFA) with surrogate-gradient BPTT under identical architectures and training conditions. We find that DFA reaches 91.5% accuracy, quantifying the performance trade-off of bio-inspired learning rules for modern neuromorphic audio. Index Terms: spiking neural networks, keyword spotting, neuromorphic computing, learnable residual speech-to-spike encoding, local learning
1. Introduction Neuromorphic computing offers a compelling paradigm for processing temporal signals at the extreme edge, promising high energy efficiency through sparse, event-driven processing. However, mapping continuous-time auditory signals to discrete Spiking Neural Networks (SNNs) remains a fundamental challenge. Unlike the visual domain, where dynamic vision sensors (DVS) [1] provide a native asynchronous event stream, audio acquisition typically relies on standard microphones that output dense, high-bandwidth waveforms. In practice, fixed stepforward speech-to-spike encoders rely on static, manually chosen thresholds. Consequently, even advanced SNN architectures with convolutional [2, 3] or recurrent [4] backbones remain constrained by this fixed, heuristic encoder [5]. This fixed structure forces the network to compensate for suboptimal inThis paper was accepted at Interspeech 2026. This version is a preprint.
Parallel to these architectural constraints is the challenge of learning with local, hardware-compatible update rules. While Backpropagation Through Time (BPTT) with surrogategradient remains the gold standard for accuracy [6], it relies on computationally expensive error propagation and symmetric weights that are difficult to implement on neuromorphic hardware [7]. Bio-inspired alternatives such as Direct Feedback Alignment (DFA) [8, 9] avoid the weight transport problem and enable parallel layer updates, including recent extensions operating directly on spike trains [10]. Yet the performance tradeoff between these feedback mechanisms and surrogate BPTT on neuromorphic speech tasks remains underexplored. Motivated by these limitations, we introduce a fully differentiable residual speech-to-spike encoding mechanism that replaces fixed thresholds with learnable parameters. We validate this approach on the Google Speech Commands v2 (GSCv2) dataset [11], a stringent benchmark where traditional deep learning baselines set a high bar [12, 13]. We summarize our contributions as follows: 1. We introduce a learnable residual speech-to-spike encoder, jointly optimized with a recurrent LIF (R-LIF) backbone. This yields up to 94.97% accuracy with an average encoder spike rate of 6.56%, enabling competitive performance at small model sizes down to 35k parameters. 2. We provide an encoder-side analysis using temporal probes and gradient statistics and demonstrate that the learnable encoder does not aim to faithfully reconstruct the input log-mel spectrogram; instead, it constructs a task-aligned spike representation that significantly enhances the linear separability of classes compared to fixed baselines. 3. We benchmark credit-assignment mechanisms by comparing surrogate-gradient BPTT with DFA. Under this comparison, DFA reaches 91.5% accuracy (vs. 94.97 for BPTT) for spiking keyword spotting on the GSC dataset, clarifying both the potential and current limitations of bio-inspired learning rules. The rest of the paper is organized as follows: Section 2 details the learnable residual encoder and R-LIF backbone. Section 3 outlines the experimental setup and training protocols. Section 4 presents performance benchmarks, providing an interpretability analysis of the proposed encoder and evaluating learning rule trade-offs. Finally, Section 5 concludes with future directions.
Phase II: Fine Residual Encoding. After applying coarse correction, we compute the time-step residual as, rt = xt − x̂t . (2) The fine error is then et = rt − r̂t−1 and encoded with the (2) finer step δ : (2,+)
st
(2)
− δ (2) ,
(4)
(2) −et − δ (2) .
(5)
= H et
(2,−) st =H
The fine trace is updated similarly: (2,+)
r̂t = r̂t−1 + δ (2) st
Figure 1: Overview of the proposed learnable step-forward speech-to-spike (S2S) encoder with a spiking backbone. The encoder converts log-mel features into signed spike trains using learned step sizes δ 1 (coarse) and δ 2 (fine residual). The resulting event streams are processed by an R-LIF backbone, followed by spike readout and a lightweight MLP head.
2. Methods As illustrated in Fig. 1, the proposed architecture integrates a differentiable speech-to-spike front end with a recurrent spiking classifier. The pipeline processes an input log-mel spectrogram X ∈ RC×T , where xc,t denotes the log-mel magnitude at band c and time t. At the encoder stage, the dense input is converted into a sparse binary spike tensor S by the learnable step-forward encoder (Sec. 2.1). The encoder emits positive and negative spikes for both coarse and fine streams, producing a 4C-channel event representation. Event streams are processed by a multi-layer Recurrent LIF backbone (Sec. 2.2) to capture temporal dependencies and a lightweight nonlinear readout head maps the aggregated spike activity to class probability. The full pipeline is differentiable, allowing the encoder parameters to be optimized jointly with the backbone weights. 2.1. Learnable Step-Forward Encoding The encoder transforms the sequence of log-mel feature values xc,t for each frequency band into spike trains. For clarity, we omit the frequency-band index c in the equations below. The encoder maintains a coarse trace x̂t with a larger step size δ (1) and a fine residual trace r̂t with a smaller step size δ (2) . For each frequency band, these traces are initialized to zero and are not learned parameters. The step-size parameters ρ and κ are global scalar variables shared across all frequency bands. Phase I: Coarse Encoding. We first calculate the tracking error (1) relative to the coarse trace: et = xt −x̂t−1 . Spikes are emitted if the error magnitude exceeds the coarse step: (1,+) (1) st = H et − δ (1) , (1) (1,−) (1) (1) st = H −et − δ . (2) where H(·) is the Heaviside step function. The coarse trace is updated via a Step-Forward rule: (1,+) (1,−) x̂t = x̂t−1 + δ (1) st − st . (3)
(2,−)
− st
.
(6)
Final spike output is the concatenation of these four streams: (1,±) (2,±) st = st , st resulting in 4C channels per time step. Optimization and Hierarchy. To ensure a valid hierarchy between step sizes (δ (2) < δ (1) ), we parameterize the step sizes using trainable scalar variables ρ, κ ∈ R that are shared across all frequency bands: δ (1) = softplus(ρ) + ϵ,
δ (2) = δ (1) · σ(κ),
(7)
where σ(·) is the logistic sigmoid and ϵ = 10−4 is a small constant ensuring strictly positive step sizes. Since H(·) is nondifferentiable, we use a surrogate gradient for backpropagation [14]. Specifically, we employ a straight-through estimator with a sigmoid backward shape in the encoder. 2.2. Recurrent LIF Backbone and Nonlinear Readout The spiking backbone consists of a stack of R-LIF layers. At layer ℓ, let sℓt ∈ {0, 1}dℓ be the spike output and uℓt ∈ Rdℓ the membrane potential. Given the input sℓ−1 , the dynamics are t governed by: ℓ ℓ uℓt = β ℓ ⊙ vt−1 + W ℓ sℓ−1 + Wrec sℓt−1 + bℓ , t sℓt = H uℓt − θℓ ,
vtℓ = uℓt − θℓ ⊙ sℓt ,
(8) (9) (10)
where β ℓ is a learnable leak, θℓ is a learnable threshold, and vtℓ denotes the post-reset membrane state. Eq. (10) implements a subtractive (soft) reset. For the backbone, we differentiate spikes using the fast-sigmoid surrogate [6]. We summarize the final-layer spike train via time averaging, T
s̄ =
1 X L st , T t=1
(11)
and map s̄ to class logits using a lightweight two-layer MLP: logits = W2 ϕ Dropout(W1 s̄ + b1 ) + b2 , (12) where ϕ(·) is ReLU. As this head runs once per utterance, it adds minimal compute while improving separability compared to a purely linear readout. 2.3. Training Objective and Learning Algorithms The training objective combines cross-entropy with an activity penalty to induce sparsity: L = LCE + λspk
T X 1 X sℓt . T dℓ t=1 1 ℓ
(13)
where sℓt ∈ {0, 1}dℓ denotes the spike vector at layer ℓ and time t, and ∥sℓt ∥1 counts the number of spikes at that layer and time step. Here, dℓ is the number of neurons in layer ℓ, and T is the number of discrete simulation steps. We compare two credit assignment strategies to optimize this objective: Surrogate-Gradient BPTT. we trained the network with BPTT [6], propagating gradients through the unrolled temporal graph. The discrete spiking thresholds are handled using the surrogate derivative of H(·). Direct Feedback Alignment (DFA). To assess hardwareefficient training, we alternatively employ DFA [8]. This decouples the backward pass by projecting a global error signal e to hidden layers via fixed random matrices B ℓ . The global error e = ∂L/∂z denotes the gradient of the loss with respect to the output logits z, and has dimension equal to the number of classes. The local error signal is:
spike budget more effectively than static, non-adaptive threshold settings. Table 1: Comparison of fixed vs. learnable encoding under an identical R-LIF backbone. Fixed Step Size refers to the hand tuned Step-Forward quantization step ∆ (threshold) used in the baseline encoder.
Metric
Step-Forward Learnable (Fixed Step Size) S2S (Ours)
Test Acc. (%) Spikes / utterance Input Sparsity (%)
90.70 2982 90.7
94.97 2119 93.4
4.2. Comparison with State-of-the-Art δtℓ = (B ℓ e) ⊙ H̃ ′ (uℓt − θℓ ).
(14)
This avoids the symmetric weight transport required by BPTT while still using a local surrogate gradient for the spiking nonlinearity.
3. Experimental Setup We evaluate on the Google Speech Commands v2 (GSC-v2) dataset using the standard 35-class protocol [11]. Audio is sampled at 16 kHz. We extract 80-bin log-mel spectrograms using a 25 ms analysis window and a 10 ms hop, and apply log compression (log(1 + x)) to the mel power spectrum before passing features to the spike encoder. The encoder maps 80 input bins to 320 spike channels (coarse± and fine±). We evaluate three backbone scales with identical R-LIF dynamics: Large (∼1.8M parameters), Small (∼0.7M parameters), and a highly constrained Tiny model (∼35k parameters). We compare the proposed learnable encoder against two baselines: (i) the same R-LIF backbone driven by a fixed Step-Forward encoder employing non-learned thresholds, and (ii) state-of-the-art spiking KWS systems from prior work. All models are trained with AdamW and cosine learningrate decay[15]. We apply standard regularization, including dropout and label smoothing, together with the spike-rate penalty described in Sec. 2. To maximize performance under tight parameter constraints, we use knowledge distillation, the compact student model minimizes the KL divergence between its temperature-scaled probabilities and those of a pre-trained teacher model, in addition to the standard cross-entropy loss.
4. Results and Analysis 4.1. Efficacy of Learnable Encoding We first isolate the contribution of the proposed encoder by comparing it against a fixed Step-Forward baseline under an identical R-LIF backbone. As shown in Table 1, proposed learnable encoder yields a substantial gain in test accuracy, improving performance from 90.70% to 94.97%. This improvement is accompanied by a substantial reduction in input event activity (2982 → 2119 spikes per utterance). Consistent with this trend, the measured input sparsity increases from 90.7% to 93.4%, indicating that the learned thresholds reduce unnecessary events while preserving task-relevant information. Together, these results indicate that end-to-end optimization can regulate the
Table 2 benchmarks our system against recent spiking KWS baselines on GSC-v2. In the Large configuration, our model reaches 94.97% accuracy, outperforming prior spiking pipelines such as Speech2Spikes [16] , SIDC-KWS[17], ED-sKWS [18], and approaching the performance of delay-learning architectures such as DCLS-Delays [19]. In the highly parameterconstrained configuration, the Tiny model (∼35k parameters) retains 89.8% accuracy, matching or exceeding baselines that rely on substantially larger backbones. These results highlight that improving the front end encoding can significantly reduce the parameter budget required to reach a given accuracy. Table 2: Benchmarking against spiking KWS systems on GSCv2 (35 classes). Model
Params (K)
Acc. (%)
Ours (Large) Ours (Small) Ours (Tiny)
1820 699 35
94.97 92.64 89.80
DCLS-Delays [19] SIDC-KWS [17] ED-sKWS [18] SRNN+ALIF [4] Speech2Spikes [16]
2500 403 307 222 410
95.3 94.7 93.1 92.5 89.5
4.3. Interpretability: Separability vs. Reconstruction Does the encoder primarily preserve signal fidelity, or does it shape features for classification? To probe this, we freeze the trained encoder and train a classifier directly on its output spike representation. The learnable encoder yields 71.63% test accuracy compared to 63.72% for the fixed encoder, a gap of 8.24 percentage points, indicating improved class separability at the representation level. In addition, we analyze encoder training signals by inspecting per-band gradients and reconstruction residuals. The observed gradient patterns are not explained solely by reconstruction distortion. Specifically, The correlation between gradient magnitude and per-band spike activity is negligible (r ≈ −0.00018), confirming that the encoder is not simply optimizing firing rate. In contrast, the correlation between gradient magnitude and reconstruction error is clearly positive (r ≈ 0.22), indicating that regions where the encoder deviates more strongly from the original mel representation tend to receive larger training signals, suggesting that the learned quan-
Table 3: Compute proxy on GSC-v2 for the Large model (45 nm estimates). Compute Proxy Global Sparsity (%) Dense Ops (Nominal) Active Ops (Event-driven) Estimated Active Energy
95.5 82.9 M 3.9 M 16 µJ
6. Acknowledgments
tization behavior is shaped by the downstream discriminative objective rather than signal fidelity alone. 4.4. Sparsity and Energy Proxy
Taharim Rahman Anon contributed to this work during her internship at PI LLC. The authors thank PI LLC (Sapporo, Hokkaido, Japan) for providing the GPU resources that supported the experiments in this study.
7. Generative AI Use Disclosure
For the Large model, event activity is evaluated and energy is estimated using a hardware-agnostic compute proxy. We adopted the Synaptic operation count (SynOps) that reflects the number of synaptic operations effectively activated by spikes[20]. We emphasize that this provides a coarse estimate intended for relative comparison rather than a precise measurement of on-chip energy. Activity is reported by collecting the network firing rate, and the event-driven operations are estimated as: SynOps = DenseOps × SpikingRate.
a fundamental trade-off between hardware-friendly local learning and optimal performance. Future work includes tightening hardware-aware energy estimates through more fine-grained, layer-level compute proxies, and exploring sparsity-constrained local learning rules that aim to close the DFA–BPTT accuracy gap without increasing spike activity.
(15)
Here, DenseOps denotes the nominal dense operation count under full activation, and SpikingRate is the measured average firing rate. Accumulate operations (AC) for SNN synaptic events are also computed. These operation counts are then converted into Joules using Horowitz-style 45 nm energy estimates[21]. As shown in Table 3, the Large model achieves 95.5% global sparsity, reducing operations from 82.9 M dense to 3.9 M active SynOps, with an estimated active energy of 16 µJ. 4.5. Benchmarking Learning Rules Finally, we compare surrogate-gradient BPTT against DFA on the same architecture, learnable encoder and training conditions, building on prior demonstrations that DFA [9, 22] can support learning in spiking neural networks while avoiding symmetric weight transport. This controlled comparison establishes a clear accuracy–hardware-efficiency trade-off between surrogate-gradient BPTT and DFA. While DFA simplifies credit assignment and avoids symmetric weight transport, it achieves lower accuracy (91.5%) than surrogate-gradien BPTT (94.97%) under identical conditions. This performance gap highlights both the promise and current limitations of local learning rules for neuromorphic speech applications and motivates further research to close the remaining accuracy gap without sacrificing hardware efficiency.
5. Conclusion We introduce a learnable residual speech-to-spike encoder that replaces fixed Step-Forward thresholds with trainable coarse and fine step sizes and jointly optimized with an R-LIF backbone. The proposed approach improves classification accuracy while reducing input spike activity in neuromorphic keyword spotting. We further provided a controlled benchmark of learning rules, showing that Direct Feedback Alignment can train the same spiking keyword spotting model to high accuracy but remains below surrogate-gradient BPTT. This result highlights
In this paper, we have utilized ChatGPT (OpenAI: GPT-5.2) only to assist with minor editing and polishing of the manuscript after the core scientific content and main ideas of the work had been developed and written by the authors. The specific assistance of AI tools includes editing and formatting equations into LaTeX, including grammar, spelling, and overall readability to ensure the textual consistency. No generative AI tool was used to generate the primary research contributions, technical ideas, experimental design, or results. All authors reviewed and take full responsibility for the originality, accuracy, and integrity of the paper.
8. References [1] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128×128 120 db 15 µs latency asynchronous temporal contrast vision sensor,” IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566–576, Feb. 2008. [2] E. Yılmaz, Ö. B. Gevrek, J. Wu, Y. Chen, X. Meng, and H. Li, “Deep convolutional spiking neural networks for keyword spotting,” in Interspeech 2020, 2020, pp. 2557–2561. [3] M. Wang, H. Zhang, Y. Wang, X.-D. Zhang, C. Xu, Q. Wang, Z.T. Li, J. Lv, Y. Wang, and Y. Tian, “Global-local convolution with spiking neural networks for energy-efficient keyword spotting,” in Interspeech 2024, 2024, pp. 4523–4527. [4] A. Bittar and P. N. Garner, “A surrogate gradient spiking baseline for speech command recognition,” Frontiers in Neuroscience, vol. 16, p. 865897, 2022. [Online]. Available: https://www.frontiersin.org/journals/neuroscience/articles/ 10.3389/fnins.2022.865897/full [5] D. Auge, J. Hille, E. Mueller, and A. Knoll, “A survey of encoding techniques for signal processing in spiking neural networks,” Neural Processing Letters, vol. 53, 07 2021. [6] F. Zenke and T. P. Vogels, “The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks,” Neural Computation, vol. 33, no. 4, pp. 899– 925, 2021. [7] A. F. M. V. Renner, F. C. Sheldon, A. V. Zlotnik, L. Tao, and A. T. Sornborger, “The backpropagation algorithm implemented on spiking neuromorphic hardware,” Nature Communications, vol. 15, no. 1, 11 2024. [Online]. Available: https://www.osti. gov/biblio/2476747 [8] A. Nøkland, “Direct feedback alignment provides learning in deep neural networks,” in Advances in Neural Information Processing Systems, 2016, pp. 1037–1045. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2016/file/ d490d7b4576290fa60eb31b5fc917ad1-Paper.pdf [9] T. P. Lillicrap, D. Cownden, D. B. Tweed, and C. J. Akerman, “Random synaptic feedback weights support error backpropagation for deep learning,” Nature Communications, vol. 7, p. 13276, 2016. [Online]. Available: https://www.nature. com/articles/ncomms13276
[10] J. Lee, R. Zhang, W. Zhang, Y. Liu, and P. Li, “Spike-train level direct feedback alignment: Sidestepping backpropagation for onchip training of spiking neural nets,” Frontiers in Neuroscience, vol. 14, p. 143, 2020. [11] P. Warden, “Speech commands: A dataset for limitedvocabulary speech recognition,” 2018. [Online]. Available: https://arxiv.org/abs/1804.03209 [12] Y. Gong, Y.-A. Chung, and J. R. Glass, “AST: Audio Spectrogram Transformer,” in Interspeech 2021, 2021, pp. 571–575. [13] A. Berg, M. O’Connor, and M. T. Cruz, “Keyword transformer: A self-attention model for keyword spotting,” in Interspeech 2021, 2021, pp. 4249–4253. [14] E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradientbased optimization to spiking neural networks,” IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51–63, 2019. [15] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019. [Online]. Available: https: //openreview.net/forum?id=Bkg6RiCqY7 [16] M. Stewart, K. Cygnar, T. Hamilton, F. Leduc-Primeau, K. Thibodeau, C. Thakur, and E. Sparks, “Speech2spikes: Efficient audio encoding pipeline for real-time neuromorphic processors,” in Neuro-Inspired Computational Elements Conference (NICE 2023). New York, NY, USA: Association for Computing Machinery, 2023, pp. 1–6. [17] J. G. Lim and S. E. Kim, “SIDC-KWS: Efficient Spiking Inception-Dilated Conformer with Self-Attention for Keyword Spotting,” in Interspeech 2025, 2025, pp. 2665–2669. [18] Z. Song, Q. Liu, Q. Yang, Y. Peng, and H. Li, “ED-sKWS: Early-Decision Spiking Neural Networks for Rapid, and EnergyEfficient Keyword Spotting,” in Proc. Interspeech 2024, 2024, pp. 4528–4532. [Online]. Available: https://www.isca-archive. org/interspeech 2024/song24c interspeech.html [19] I. Hammouamri, I. Khalfaoui Hassani, and T. Masquelier, “Learning delays in spiking neural networks using dilated convolutions with learnable spacings,” in International Conference on Representation Learning (ICLR), B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., 2024, pp. 17 890–17 903. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2024/file/ 4df1cc5a7528b7197ad8ae76ff30107a-Paper-Conference.pdf [20] M. Sorbaro, Q. Liu, M. Bortone, and S. Sheik, “Optimizing the energy consumption of spiking neural networks for neuromorphic applications,” Frontiers in Neuroscience, vol. Volume 14 - 2020, 2020. [Online]. Available: https://www.frontiersin.org/journals/ neuroscience/articles/10.3389/fnins.2020.00662 [21] M. Horowitz, “1.1 computing’s energy problem (and what we can do about it),” in 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 2014, pp. 10–14. [22] Y. Zhang, K. Inoue, M. Nakajima, T. Hashimoto, Y. Kuniyoshi, and K. Nakajima, “Training spiking neural networks via augmented direct feedback alignment,” arXiv preprint, 09 2024.