FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS Harshit Singh1 , Ayush Pratap Singh2 , Nityanand Mathur3 1
University Of Maryland, 2 TU Darmstadt, 3 Smallest AI [email protected]
arXiv:2606.20518v1 [cs.AI] 18 Jun 2026
Abstract Flow-matching text-to-speech systems achieve remarkable zero-shot quality but remain static after deployment: pronunciation errors on out-of-vocabulary proper nouns persist unless the model is retrained. We introduce FlowEdit, a lifelong adaptation framework for frozen flow-matching TTS that learns pronunciation corrections as latent conditioning edits rather than weight updates. When corrective feedback is provided, FlowEdit optimizes a token-level perturbation in the text embedding space, then stores the correction in a Modern Hopfield Network serving as content-addressable episodic memory. At inference, corrections are retrieved via soft attention with a similarity gate, enabling fuzzy morphological matching. On our curated benchmark of 312 multilingual proper nouns across 18 language families, FlowEdit reduces target-word Phoneme Error Rate by 92.7% relative to the zero-shot baseline while maintaining identical general-speech quality. Corrections complete in approximately 15 seconds on a single GPU. Index Terms: speech synthesis, text-to-speech, pronunciation correction, flow matching, continual learning
1. Introduction State-of-the-art text-to-speech (TTS) models like F5-TTS [1], Matcha-TTS [2], and VALL-E [3] deliver impressive zero-shot quality. However, these systems remain static once deployed. A critical challenge for voice assistants and accessibility tools is the persistent mispronunciation of proper nouns and foreign loan-words. Once an error is hardcoded into a frozen model, it persists indefinitely without costly retraining. Existing remedies scale poorly. Grapheme-to-phoneme (G2P) dictionaries fail on polyglot names lacking standard rules. Fine-tuning risks catastrophic forgetting [4] and voice drift. Null-space weight editing [5] risks cumulative interference as edits accumulate. We propose FlowEdit, a non-destructive alternative inspired by speech therapy rather than surgery. Our key insight is that the differentiable nature of conditional flow-matching models enables latent input optimization. Instead of updating the massive weight matrices θ of the diffusion transformer (DiT), FlowEdit corrects pronunciation by optimizing a token-level perturbation vector δ added to the text conditioning signal. To ensure these corrections are remembered across sessions without degrading the base model, we securely store the optimized latent edits in a Modern Hopfield Network [6]. This acts as a content-addressable episodic memory that seamlessly interfaces with the frozen TTS backbone. During inference, corrections are retrieved via soft attention combined with a similarity gate, enabling fuzzy morphological matching (e.g., retrieving a correction for the root “Linux” when synthesizing “Linux’s”). Contributions: (i) We introduce latent optimization for pro-
nunciation correction in frozen flow-matching TTS, optimizing speech trajectories without weight updates. (ii) We design a Hopfield Refiner with gated retrieval for non-destructive, lifelong episodic memory. (iii) FlowEdit achieves a 92.7% relative reduction in target-word Phoneme Error Rate (PER) with mathematically guaranteed zero forgetting of general speech. (iv) We curate P OLYGLOT-N OUNS, a challenging benchmark for personalized pronunciation adaptation.
2. Related Work Neural TTS Architectures. The TTS landscape has evolved from autoregressive vocoders like WaveNet [7] and Tacotron [8, 9] to flow-based and diffusion models [10, 11, 12]. Recent systems like F5-TTS [1], Matcha-TTS [2], VALL-E [3], and others achieve remarkable zero-shot quality. Despite this progress, none offer mechanisms for post-deployment pronunciation correction. Pronunciation Correction in TTS. Traditional systems handle pronunciation mapping via grapheme-to-phoneme (G2P) models and pronunciation lexicons. However, modern endto-end models often predict spectrograms directly from raw text or byte-pair encodings (BPE), deliberately bypassing explicit phoneme bottlenecks to improve prosody and naturalness. This makes targeted phoneme injection difficult. Some approaches force alignment to user-provided phonemes, but this demands linguistic expertise (e.g., IPA symbols) from endusers. FlowEdit addresses this gap by correcting pronunciations purely from audio feedback, requiring no linguistic knowledge. Model Editing and Continual Learning. Techniques for modifying pre-trained model behavior without full retraining are well-studied in NLP. ROME [13] and MEMIT [14] directly patch feed-forward network weights to edit factual associations. LoRA [15] enables parameter-efficient adaptation via low-rank weight decompositions. In continual learning, methods like Elastic Weight Consolidation (EWC) [16] aim to mitigate catastrophic forgetting [4] through regularization. Complementary strategies include Progressive Neural Networks [17], which avoid forgetting via lateral connections to frozen column networks, and PackNet [18], which iteratively prunes and re-trains subnetworks for each task. Experience replay methods [19] maintain a buffer of prior-task examples to regularize updates. While effective in classification settings, these approaches require access to prior training data or architectural expansion—neither of which is feasible for frozen, deployed TTS systems. Recently, SonoEdit [5] adapted null-space weight editing specifically for LLM-based speech models. However, all weight-editing approaches risk unintended collateral damage to the model’s learned manifold as edits accumulate over time. By offloading corrections to an external Hopfield mem-
ory and editing only the input latent space, FlowEdit completely sidesteps both catastrophic forgetting and parameter drift.
3. Methodology 3.1. Preliminaries: Conditional Flow Matching F5-TTS [1] employs a Diffusion Transformer (DiT) [20, 21] with 22 layers and embedding dimension d=1024 to estimate a vector field vt (x, t; θ) that transforms Gaussian noise p0 to speech mel-spectrograms p1 , conditioned on text representations c. Training minimizes the Conditional Flow Matching objective [11]. Throughout, we color-code notation: frozen model components, Hopfield memory, and learnable/agent quantities. LCFM (θ) = Et,x1 ,x0 ∥vt (ψt (x0 ), t) − (x1 − x0 )∥2 . (1) where ψt (x0 ) = (1−t)x0 + tx1 is the optimal transport interpolant. At inference, synthesis proceeds by integrating the learned ODE from t=0 to t=1: Z 1 x1 = x 0 + vt (xt , t; θ) dt, (2) 0
which we discretize via fixed-step Euler integration with N =32 steps. Crucially, this forward pass is fully differentiable with respect to the conditioning input c, enabling gradient-based optimization of the text embeddings without modifying θ. 3.2. The FlowEdit Framework FlowEdit operates through an interactive loop (Figure 1) that learns, stores, and retrieves pronunciations. Stage 1: Detection and Grounding. The user provides a corrective signal: a reference audio yref paired with the target text (e.g., “Siobhan” pronounced correctly). We use WhisperLarge-v3 [22] forced alignment to localize the temporal boundaries of the target word, extracting the target token indices I. We expand this by one token on each side to absorb tokenizer boundary errors. Stage 2: Latent Input Optimization. We freeze all DiT parameters θ and introduce a learnable perturbation δ ∈ RS×d initialized at zero for sequence length S: ∗
2
δ = arg min ∥ Mel(gθ (c + δ)) − Mel(yref )∥ δ
+ λ∥δ∥22 .
(3)
where c = E(x) denotes text-encoder embeddings, gθ denotes synthesis through the frozen DiT and ODE solver (Eq. 2), and λ=0.001 is a regularization weight. We mask non-target positions (δ j = 0∀j ∈ / I). To obtain ∇δ L without storing all intermediate ODE states, we employ the adjoint sensitivity method [12]. Defining the adjoint state a(t) = ∂L/∂xt , the gradient with respect to the conditioning input is recovered by solving a reverse-time ODE: Z 0 dL ∂vt ∂vt = − a(t)⊤ dt, ȧ(t) = −a(t)⊤ , (4) dδ ∂c ∂x t 1 initialized at a(1) = ∇x1 L. This yields memory-efficient gradients with constant (rather than O(N )) memory cost, independent of the number of Euler steps. Optimization utilizes Adam [23] with gradient clipping (∥∇δ ∥∞ ≤ 1.0) and a cosine-annealed learning rate (η0 =0.01 → η50 =0.001). Data augmentation (time-stretching and gain scaling in mel space) is applied to the reference to promote robust latents. Optimization runs for 50 steps. Correction wall-clock time scales approximately linearly with the number of ODE solver steps N :
Figure 1: FlowEdit architecture. Left: Inference pipeline. Input text is encoded by a frozen Text Encoder and refined by the Hopfield Refiner, which retrieves stored corrections via soft attention. The refined embeddings are decoded by the frozen DiT. Right: Correction loop. User reference audio triggers forced alignment, optimization of δ ∗ , and a memory write to the Hopfield memory. at N ∈ {16, 32, 64, 128}, correction takes {8, 15, 28, 54} seconds on an A100, respectively. Since the adjoint method (Eq. 4) requires only constant memory independent of N , memory cost remains flat at ∼3.2 GB across all step counts. All reported results use N =32 as the quality–speed optimum. Stage 3: Associative Memory via Hopfield Networks. To ensure corrections persist, we store them in a Hopfield Refiner inserted after the text encoder. Each finalized correction writes a key–value pair: Ki = pool(cI ),
Vi = pool(δI∗ ),
(5)
where pool averages over the corrected token span. Retrieval follows the Modern Hopfield update [6]: √ Mem(Q) = softmax(β QK ⊤ ) V , β = 1/ d. (6) Queries and keys are L2-normalized. A similarity gate suppresses irrelevant retrievals for out-of-domain words: ĉ = c + σ(max(β QK ⊤ j ) − τ ) ⊙ Mem(Q). j
(7)
where σ is the sigmoid function and τ ≈5.0 is a learned threshold scalar mapping cosine similarity to a gating factor. This formulation yields a critical property: fuzzy morphological matching. Because Softmax connects similar inputs, a correction for “Linux” can partially address the query vector for “Linux’s” or “Linuxed”, avoiding the strict 1:1 match constraints of dictionary lookups. Memory Management and Complexity. Deduplication prevents memory explosion (cosine similarities > 0.95 trigger exponential moving average updates instead of new insertions). Memory is bounded via LRU pruning at a user-defined budget Mmax . Retrieval is O(M d) per token—negligible relative to the cost of a DiT. For homograph disambiguation (e.g., a “bass” fish vs. “bass”
guitar), FlowEdit utilizes context-conditioned keys by taking a Gaussian-weighted average of surrounding text embeddings within a window of ±3 tokens.
Table 1: Main results on P OLYGLOT-N OUNS (312 words, 1,560 utterances). PER values report mean ± 95% CI over 3 runs. Best in bold, second-best underlined. PERtarget ↓ (wav2vec 2.0)
PERgen ↓
MCD↓
Human Eval↑
Time
Zero-shot Lexicon Fine-tuning LoRA Prompting
42.5±1.2 18.7±0.9 8.2±0.5 11.8±0.9 18.3±1.1
4.1 4.1 15.3 6.7 4.2
6.82 5.61 4.10 4.65 5.31
72.1 68.3 74.8 71.4 69.8
— Manual ∼20m ∼8m ∼5m
FlowEdit
3.1±0.3
4.1
3.22
78.6
∼15s
Method
4. Experiments 4.1. Experimental Setup We use F5-TTS [1] (335M params, d=1024) with a HiFiGAN [24] vocoder as the frozen backbone. Only the percorrection δ vectors and Hopfield memory are learned. All runs use a single A100-80GB. We evaluate on P OLYGLOT-N OUNS, our curated set of 312 proper nouns across 18 language families, each paired with 5 carrier sentences from native speakers (1,560 clips total). The benchmark is stratified by: language family, word length (1–4 syllables), and phoneme complexity (low/medium/high as rated by a native-speaker panel of 6 annotators, Cohen’s κ = 0.71). General-speech forgetting is measured on 500 held-out LibriTTS-R [25] utterances drawn from the LibriSpeech [26] ecosystem. Baselines: F5-TTS zeroshot, eSpeak-NG lexicon override, full fine-tuning (500 steps, lr= 10−5 ), LoRA (r=16, 500 steps) [15], and prompt tuning (8 prefix tokens). To avoid circularity with our Whisperbased alignment, we evaluate PERtarget using two independent systems: (i) a wav2vec 2.0-based phoneme recognizer finetuned on CommonVoice 13.0, and (ii) human transcription by native-speaker annotators for a 60-word stratified subset. All reported PER figures use wav2vec 2.0; human evaluation confirms scores within ±0.4% absolute. We report target-word PER (PERtarget , wav2vec 2.0), general PER (PERgen , LibriTTSR), MCD, human evaluation scores (24 listeners, hidden anchor), and A100 wall-clock time. 4.2. Main Results Table 1 details the primary evaluation metrics. FlowEdit achieves the strongest overall performance profile. It realizes a PERtarget of 3.1%, constituting a 92.7% relative reduction from the zero-shot baseline. Notably, it outperforms full finetuning (8.2%) because FlowEdit strictly isolates and optimizes the acoustic trajectory for the exact tokens in question, minimizing optimization interference. Results are consistent across both evaluators: human transcription of the 60-word subset yields a FlowEdit PER of 3.4%, confirming that Whisper-alignment during optimization does not inflate automatic metric scores. Crucially, FlowEdit exhibits exactly zero forgetting. General PER on LibriTTS-R remains 4.1%, statically tied to the baseline, as model logic outside of Hopfield-activated tokens is completely unmodified. In contrast, fine-tuning severely shifts the manifold, raising PERgen to 15.3%. Efficiency: Convergence occurs in ∼50 steps (∼15 seconds), making FlowEdit 80× faster than fine-tuning workflows. Human evaluation confirms annotators prefer FlowEdit due to the lack of artifacts. Perlanguage analysis. FlowEdit consistently dominates across families (Table 3). Celtic and Vietnamese (baseline 58–61%) reduce to 4.9–5.3%; Slavic (38.6%) to 2.9%. Correction magnitude is uniform (∆PER ≈ 90–93%), showing latent optimization generalizes well. Reference audio >1.5 s yields optimal results; performance plateaus beyond 3 s.
0 Tonal languages (Mandarin, Vietnamese) show higher residual PER due to F0 reconstruction limitations discussed in §4.5.
Table 2: Ablation of FlowEdit components on P OLYGLOTN OUNS. Variant FlowEdit (full) w/o memory (single-use) w/o gating (σ=1) w/ hard NN lookup 25 optimization steps λ = 0.0001 λ = 0.01
PERtarget ↓
MCD↓
PERgen ↓
3.1 6.9 3.1 3.8 6.8 3.9 4.6
3.22 3.85 3.24 3.35 3.95 3.41 3.18
4.1 4.1 5.8 4.4 4.1 4.1 4.1
4.3. Ablation Studies and Convergence Table 2 identifies the impact of architectural choices. Memory persistence is essential: isolating optimizations without the write-back Hopfield mechanism degrades correction longevity (6.9% PER). Note that the single-use variant re-optimizes δ from scratch at every inference call without caching; the degraded PER (6.9%) thus reflects initialization variance and the absence of EMA-smoothed memory consolidation across the 3 evaluation runs, not a within-session effect. Gating prevents bleeding: removing σ reduces similarity discrimination, activating latents on unrelated words and lifting baseline PERgen to 5.8%. Varying the threshold scalar τ ∈ {3.0, 5.0, 7.0} yields PERgen of {6.9, 5.8, 4.3}% in the no-gate condition and {4.1, 4.1, 4.2}% with gating enabled, confirming that the learned τ ≈5.0 is near-optimal and robust. Soft Hopfield rules: Using strict 1-nearest-neighbor indexing limits the efficacy of fuzzy matching (3.8% PER), validating soft-attention over the Hopfield manifold. Regularization strength: λ=0.001 (default) balances phonetic accuracy and embedding smoothness; under-regularization (λ=0.0001) yields slightly noisier latents without PER gain. Figure 2 profiles optimization dynamics. Convergence follows a two-phase pattern: rapid descent (steps 1–15, ∥∇δ L∥ drops 85%) capturing coarse phonetics, then refinement (steps 15–50, gradient norms <0.02) for spectral details. Behavior is consistent across all 312 words. 4.4. Continual Editing and Morphological Transfer Table 4 characterizes lifelong learning. Fine-tuning suffers gradient interference over 200 edits (drift 6.8), while FlowEdit sustains near-zero drift (0.1) via external memory. FlowEdit’s soft attention handles morphological variants (8.4% PER) far better than rigid lexicons (36.4%). 4.5. Compute Scaling and Speaker Transfer FlowEdit scales across hardware: 42 s on NVIDIA L4, 38 s on RTX 3090. Retrieval overhead is <35 ms for M ≤ 500 corrections. Speaker-agnostic transfer. Since corrections operate in the
Target-Word PER PER Target-Word PER Target-Word
10 10
6
5 5
5
4 4
4
10
3 3
010 10
10
25 25
25
Optimization Steps Steps Optimization Optimization Steps
50 50
50
General (Forgetting) General PERPER (Forgetting) General PER (Forgetting)
FlowEditFlowEdit FlowEdit 17.517.5 17.5 Fine-tuning Fine-tuning Fine-tuning LoRALoRA LoRA Prompt tuning tuning 15.015.0 15.0 Prompt tuningPrompt
10.010.0 10.0
3
2 2 2 0 0
FlowEdit FlowEdit FlowEdit Fine-tuning Fine-tuning Fine-tuning LoRALoRA LoRA Prompt tuning tuning Prompt tuning Prompt
12.512.5 12.5
PER gen (%) ↓
20
0 0 0 0 0
6 6
MCD ↓ MCD ↓
20 20
7
PER gen (%) ↓ PER gen (%) ↓
30
7
MCD ↓
30 30
PER target (%) ↓
40
PER target (%) ↓ PER target (%) ↓
40 40
Mel-Cepstral Distortion Mel-Cepstral Distortion Mel-Cepstral Distortion
FlowEditFlowEdit FlowEdit 7 Fine-tuning Fine-tuning Fine-tuning LoRALoRA LoRA Prompt Prompt tuningtuning Prompt tuning
010 10
10
25 25
25
Optimization Steps Steps Optimization Optimization Steps
50 50
7.5 7.5
7.5
5.0 5.0
5.0
2.5 2.5
2.5
0.0 0.0 0.0 50 0 0
0 10 10
10
25 25
25
Optimization StepsSteps Optimization Optimization Steps
50 50
50
Figure 2: Optimization convergence of FlowEdit. PERtarget (↓) falls smoothly and rapidly within 50 iterations, while general speech PER (↓) remains identically static at the zero-shot baseline of 4.1%. Weight-editing baselines (dashed lines) display inferior final settling points.
Table 3: Per-language-family PER breakdown. FlowEdit consistently dominates across all families. Language Family
Baseline PER
FlowEdit PER
58.3 61.1 47.2 38.6 44.3 41.8 36.9 29.4 27.1
4.9 5.3 9.1 2.9 3.7 3.2 2.6 1.8 1.5
Celtic Vietnamese Mandarin (tonal) Slavic Sino-Tibetan Arabic Dravidian Germanic (non-English) Romance
Table 4: Long-horizon stability (200 sequential edits) and morphological transfer (application to unseen inflected variants). Method Fine-tune Lexicon FlowEdit
Continual Editing
Morphological Transfer PER↓
Drift@200↓
Retent.↑
Possessive
Compound
Overall
6.8 0.0 0.1
89.4% — 96.8%
— 35.2 6.2
— 38.9 10.8
— 36.4 8.4
speaker-agnostic text embedding space, a correction learned from one speaker transfers effectively across voices with minimal degradation. We validate this across a panel of 12 speakers (6F/6M, 4 accent groups: General American, British RP, Indian English, Australian English) drawn from VCTK [27]. For each of the 312 test words, we apply a correction learned from a single source speaker to all 11 remaining speakers. Mean crossspeaker PER is 3.6% (vs. 3.1% source-speaker), with no accent group exceeding 4.2%. This confirms that corrections in text embedding space are effectively speaker-agnostic, enabling a single shared correction bank to serve all users in a multispeaker deployment without per-voice retraining. Memory capacity ceiling. Retrieval quality remains stable up to M =500 stored corrections (PERtarget increase <0.2%). Beyond M =1000, softmax attention begins to dilute retrieval precision, raising PER by ∼0.6%; LRU pruning of stale entries mitigates this in practice. To characterize large-scale deployment, we simulate M ∈ {500, 1k, 5k, 10k} by populating memory with synthetic embeddings drawn from the empirical δ ∗ distribution. PERtarget degrades gracefully: 3.1%, 3.7%, 5.2%, 6.8% respectively. At M =10k, retrieval latency rises to 112 ms per utterance—still below perceptibility thresholds for streaming TTS. For deployments exceeding M =5k, we recom-
mend partitioning memory by domain or language family, reducing effective M per shard and restoring sub-4% PER. This hierarchical sharding strategy will be formalized in future work. Failure modes. Residual errors concentrate on two cases: (i) monosyllabic words with single-phoneme targets (e.g., the name “Xi”), where the optimization has minimal temporal context to anchor the correction, and (ii) tonal languages. Mandarin and Vietnamese exhibit higher residual PER (9.1% and 5.3% respectively, Table 3) because the mel reconstruction loss Lmel underweights fundamental frequency (F0) relative to spectral envelope. To characterize this, we compute F0 RMSE between FlowEdit output and reference for tonal vs. non-tonal words: tonal words yield 18.3 Hz RMSE vs. 4.7 Hz for non-tonal, confirming that pitch trajectory errors are the dominant residual failure mode. As a preliminary remedy, we evaluate an augmented loss L = Lmel + αLF0 with α = 0.3, where LF0 is the RMSE between CREPE-estimated [28] pitch tracks. This reduces Mandarin PER from 9.1% to 6.4% and Vietnamese from 5.3% to 4.1% at the cost of 3 additional optimization seconds. We leave full integration as future work. 4.6. Latent Edit Geometry and Robustness Edit vector interpretability. PCA over the 312 δ ∗ vectors reveals three phonetic clusters: (i) vowel remappings, (ii) consonant insertions/deletions, and (iii) stress-shift corrections. Pairwise cosine similarity averages 0.07 (std 0.04), confirming nearorthogonality. Robustness: PER stays within 0.4% of baseline at SNR ≥ 15 dB; 32 kbps Opus compression has negligible impact (PER 3.2%).
5. Conclusion FlowEdit enables lifelong pronunciation adaptation through gradient-based latent optimization backed by Hopfield memory. By moving corrections into text embedding space rather than model weights, we achieve 92.7% PER reduction with mathematically guaranteed zero forgetting, stability across 200 sequential edits, and speaker-agnostic transfer—all in 15 seconds per correction. This non-destructive paradigm makes FlowEdit immediately deployable in production systems where user-specific refinement is essential yet retraining is infeasible. Future work. Integrating phoneme-conditioned generation and expanding memory to capture domain-specific prosody will extend FlowEdit’s capabilities as a blueprint for responsive TTS deployment.
6. Use of Generative AI Disclosure In preparing this manuscript, the authors used generative AI tools for language refinement (rephrasing and improving the clarity of author-written text) and as a coding assistant (helping write and debug software for experiments and analysis). All research contributions, including the methodology, experimental design, results, and scientific claims, are the authors’ own. The authors reviewed and verified all AI-assisted text and code, and take full responsibility for the content of this paper.
7. References [1] Y. Chen, Z. Chen, Z. Zhang et al., “F5-TTS: A fairytaled and faithful flow-matching based text-to-speech system,” in Proc. ICLR, 2025. [2] S. Mehta et al., “Matcha-TTS: A fast TTS architecture with conditional flow matching,” in Proc. ICASSP, 2024. [3] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., “VALL-E: Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023. [4] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks, vol. 113, pp. 54–71, 2019. [5] A. P. Singh, H. Singh, N. Mathur, A. Mandloi, and S. Kamath, “Sonoedit: Null-space constrained knowledge editing for pronunciation correction in llm-based tts,” arXiv preprint arXiv:2601.17086, 2026. [6] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve et al., “Hopfield networks is all you need,” in Proc. ICLR, 2021. [7] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. [8] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al., “Tacotron: Towards end-to-end speech synthesis,” in Proc. Interspeech, 2017. [9] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. J. Skerry-Ryan et al., “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP, 2018. [10] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Proc. NeurIPS, 2020. [11] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, and M. Nickel, “Flow matching for generative modeling,” in Proc. ICLR, 2023. [12] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, “Neural ordinary differential equations,” in Proc. NeurIPS, 2018. [13] K. Meng, D. Bau, A. Mitchell, and C. Finn, “Locating and editing factual associations in GPT,” in Proc. NeurIPS, 2022. [14] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, “Mass-editing memory in a transformer,” in Proc. ICLR, 2023. [15] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. [16] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. GrabskaBarwinska et al., “Overcoming catastrophic forgetting in neural networks,” in Proc. NAS, 2017. [17] A. A. Rusu et al., “Progressive neural networks,” in arXiv preprint arXiv:1606.04671, 2016. [18] A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” in Proc. CVPR, 2018. [19] D. Rolnick et al., “Experience replay for continual learning,” in Proc. NeurIPS, 2019.
[20] W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proc. ICCV, 2023. [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. NeurIPS, 2017. [22] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023. [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Proc. ICLR, 2015. [24] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Proc. NeurIPS, 2020. [25] Y. Koizumi et al., “LibriTTS-R: A restored multi-speaker text-tospeech corpus,” in Proc. Interspeech, 2023. [26] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, 2015. [27] C. Veaux, J. Yamagishi, and K. MacDonald, “CSTR VCTK corpus,” 2017. [28] J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “CREPE: A convolutional representation for pitch estimation,” in Proc. ICASSP, 2018.