A Neural Transformer-Based Framework for Waveform-to-Token-to-Speech Generation

We propose a novel transformer-based architecture that directly maps continuous waveform signals into discrete token sequences, and subsequently into intelligible speech. Our framework integrates a neural waveform-to-token encoder with a transformer-based language model for sequence generation, followed by a token-to-speech decoder for acoustic realization. Unlike traditional speech recognition or text-to-speech pipelines, our model unifies acoustic, symbolic, and generative components into a single transformer-based paradigm, enabling controlled charge-like modulation of information flow and interpretable latent structures. We validate our design using MATLAB/Simulink co-simulation, demonstrating both the theoretical feasibility and practical applicability of this approach.