Conceptio › Archive › arXiv CS

arXiv CSopen access

Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching

arXiv:2604.15009v1 [cs.AI] 16 Apr 2026

Aihua Li Duke University [email protected]

Abstract Flow matching retains the generation quality of diffusion models while enabling substantially faster inference, making it a compelling paradigm for generative modeling. However, when applied to language modeling, it exhibits fundamental limitations in representing complex latent distributions with irregular geometries, such as anisotropy and multimodality. To address these challenges, we propose a mixture-of-experts flow matching (MoE-FM) framework, which captures complex global transport geometries in latent space by decomposing them into locally specialized vector fields. Building on MoE-FM, we develop a non-autoregressive (NAR) language modeling approach, named YAN, instantiated with both Transformer and Mamba architectures. Across multiple downstream tasks, YAN achieves generation quality on par with both autoregressive (AR) and diffusion-based NAR language models, while requiring as few as three sampling steps. This yields a 40× speedup over AR baselines and up to a 103 × speedup over diffusion language models, demonstrating substantial efficiency advantages for language modeling.

Introduction

Along with the remarkable success of autoregressive (AR) large language models in generating high-quality text (Radford et al., 2019; Grattafiori et al., 2024; Bai et al., 2023; Bi et al., 2024), their inference latency has long been a subject of concern. AR models generate tokens sequentially in a left-to-right manner, requiring one forward pass per token generation (Gu et al., 2018; Ghazvininejad et al., 2019; Kaiser et al., 2018). Aiming to parallelize decoding and speed up inference, diffusionbased non-autoregressive (NAR) language models have emerged as a popular alternative in recent years, inspired by the remarkable performance of diffusion models in computer vision (Ho et al., 2020; Dhariwal and Nichol, 2021). However, in the context of language generation, these methods still exhibit a fundamental performance gap compared to well-established AR models at comparable scales (Huang et al., 2022; Feng et al., 2025; Gu and Kong, 2021; Liu et al., 2021). In practice, to achieve competitive quality, existing diffusion methods typically require hundreds or thousands of inference steps to iteratively refine generated tokens (Feng et al., 2025). This, in turn, offsets the theoretical efficiency benefits of parallel decoding. To improve the quality-efficiency trade-off of current NAR language models, we investigate flow matching, a generative modeling paradigm that has demonstrated substantial efficiency advantages (Lipman et al., 2022, 2024; Liu et al., 2022; Yang et al., 2024). Flow matching generates samples by integrating a deterministic ordinary differential equation (ODE), which can be trained to follow relatively straight trajectories, thereby bypassing the iterative denoising procedures of diffusion models that contribute to high inference latency. Beyond efficiency, flow matching has also achieved strong generation quality in image and video synthesis (Esser et al., 2024; Davtyan et al., 2023). Despite this promise, its application to language modeling remains largely unexplored. Preprint.

（Source） x+

⋯

z! x(’*+

t=0

t = n/T

t = (n + 1)/T

x(’

Encoder

Vector Fields NN

Gating

Frozen Gating

（Expert Assignment）

（Vector Field）

Decoder Euler Update

z$%& （Source Latent）

MLP

Res-FFN

Transformer / Mamba

⋯

y(*+

（Output）

MoE-FM Integrator Cross Attention

Figure 1: Overview of the YAN non-autoregressive language model.

However, when instantiated for language modeling, we identify a fundamental limitation of flow matching in modeling complex latent distributions. Prior studies have shown that text representations exhibit highly irregular geometries, including multimodality, anisotropy, and fragmented manifolds (Cai et al., 2021; Gao et al., 2019; Ethayarajh, 2019; Rajaee and Pilehvar, 2021a,b). Under such conditions, vanilla flow matching with a single global vector field proves insufficient to faithfully capture the underlying transport structure, particularly under limited training scales and a small number of sampling steps. To enhance the representational capacity of flow-based language models, we propose mixture-ofexperts flow matching (MoE-FM; Fig. 2). MoE-FM models the conditional target vector field through a mixture-of-experts formulation, where multiple expert vector fields are combined via data-dependent soft routing. This approach effectively decomposes global transport and encourages specialization in distinct local transport geometries. Building on this enhanced flow matching formulation, we introduce a new NAR language modeling paradigm, which leverages MoE-FM in the latent space, aiming to learn token representations that are sufficiently expressive to support efficient sequence decoding with few parallelizable layers. We refer to the proposed model as YAN—Flow Until You Almost Know (Fig. 1). Our main contributions are summarized as follows: • We improve the representational fidelity and sampling efficiency of flow matching for language modeling by developing mixture-of-experts flow matching (Fig. 2). This leads to higher-quality recovery of text latent representations compared to vanilla flow matching, and induces straighter ODE trajectories that allow accurate generation with fewer integration steps. • We introduce YAN, a non-autoregressive language modeling framework that employs latent mixture-of-experts flow matching (Fig. 1). We instantiate YAN with both Transformer (Vaswani et al., 2017) and Mamba (Gu and Dao, 2024) architectures, and train models at the 200M-parameter scale. • We evaluate YAN across a range of downstream language modeling tasks, including text infilling and completion, and show that it achieves improved generation quality relative to baseline methods (Tab. 1). • We analyze the inference efficiency of YAN and show that it achieves high-quality longdocument infilling with as few as three Euler sampling steps. This results in a 40–50× speedup over autoregressive baselines and an order-of-magnitude 103 × speedup over diffusion-based language models (Fig. 4).

Background and Problem Statement

2.1

Flow Matching

Flow matching (FM) (Lipman et al., 2022, 2024; Holderrieth and Erives, 2025) is a class of generative models that synthesizes samples from a target distribution p1 by learning a time-dependent vector field that transports an initial distribution p0 to p1 . Formally, let u : Rm × [0, 1] → Rm , (zt , t) 7→ ut (zt ) denote a vector field over Rm defining the ordinary differential equation (ODE) dzt = ut (zt )dt, 2

Figure 2: Comparison of vanilla flow matching (VFM) and mixture-of-experts flow matching (MoE-FM) on synthetic datasets. Results on (a) grid and (b) half-moon data show that MoE-FM more accurately recovers data distributions with irregular geometries, including disconnected, curved modes and nonlinear low-dimensional manifolds. Moreover, MoE-FM learns straighter transport trajectories from noise to data, thereby improving generative performance with fewer sampling steps.

whose solution is a trajectory {zt }0≤t≤1 ⊂ Rm . The goal of FM is to learn a target vector field u∗ such that, if the initial state satisfies z0 ∼ p0 , then the terminal state satisfies z1 ∼ p1 , with intermediate states zt following a prescribed probability path {pt }0≤t≤1 , i.e., zt ∼ pt for all t ∈ (0, 1). Inspired by the forward process of denoising diffusion models (Ho et al., 2020; Song et al., 2021a; Holderrieth and Erives, 2025), FM adopts the Gaussian conditional probability path defined as pt|1 (z | z1 ) = N (z; tz1 , (1 − t)2 I) with z0 ∼ N (0, I). This is equivalent to imposing a linear interpolation zt = tz1 + (1 − t)z0 , yielding the target conditional vector field u∗t (zt | z1 ) = (z1 − zt )/(1 − t). FM learns this target conditional vector field by regressing it with a parameterized vector field uψ t under an ℓ2 loss ∗ 2 LCFM (ψ) = Et,z1 ,zt |z1 ∥uψ (1) t (zt ) − ut (zt | z1 )∥ , where the expectation is taken over t ∼ U(0, 1), z1 ∼ p1 , and the intermediate conditional distribution zt | z1 ∼ pt|1 . The validity of this optimization follows from the fact that the conditional objective (1) is equivalent, up to a constant independent of the parameter ψ, to the marginal objective LFM (ψ) = ∗ 2 Et,zt ∥uψ t (zt ) − ut (zt )∥ , where t ∼ U(0, 1) and zt ∼ pt . In other words, optimizing the conditional objective (1) equivalently recovers the target marginal vector field (Lipman et al., 2022). Moreover, the Gaussian conditional formulation is commonly used in practice, as it leads to the simplified objective 2 LVFM (ψ) = Et,z1 ,zt |z1 ∥uψ (2) t (zt ) − (z1 − z0 )∥ , which can be efficiently estimated via Monte Carlo sampling. As indicated by Eq. (2), under Gaussian linear interpolation, the target conditional vector field induces straight-line trajectories connecting the initial noise z0 to the target endpoint z1 . Sampling along such deterministic ODE trajectories enables faster inference than diffusion models, which iteratively denoise the samples along stochastic trajectories (Lipman et al., 2022; Liu et al., 2022; Song et al., 2021b). 2.2

Non-Autoregressive Language Modeling

The language modeling objective of this work is formulated as a standard sequence-to-sequence task, which aims to learn the distribution of a target token sequence y = (y1 , . . . , yL ) given a source sequence x = (x1 , . . . , xL′ ), with each token drawn from a vocabulary V = {1, . . . , V }. Motivated by the efficiency advantages of flow matching, we aim to leverage flow matching to push the efficiency limits of current NAR language models (Nie et al., 2025; Sahoo et al., 2024; Gong et al., 2025; Ye et al., 2025). We adopt a latent variable NAR formulation Z p(y | x) = p(y | z, x)p(z | x)dz (3) with a latent variable z ∈ Rm (Shu et al., 2020; Gu et al., 2018; Gu and Kong, 2021; Yuan et al., 2022), and apply continuous flow matching in the latent space. Compared to discrete flow matching 3

(Gat et al., 2024), the continuous approach typically exhibits better optimization stability and has shown higher quality in prior work (Cheng et al., 2025).

Flow Matching for Language Modeling

We begin by showing the limitations of vanilla flow matching when applied to latent language modeling (Sec. 3.1), and then develop a mixture-of-experts approach (Sec. 3.2). 3.1

Limitations of Vanilla Flow Matching

In our preliminary experiments, the vanilla flow matching (VFM) trained with objective (2) underperforms in learning the token latent distribution introduced in Sec. 4, particularly under finite training scale. Intuitively, this distribution is highly anisotropic, concentrated on a degenerate manifold, and exhibits isolated modes and clustering effects (Cai et al., 2021; Gao et al., 2019; Ethayarajh, 2019; Rajaee and Pilehvar, 2021a,b). These geometries pose a challenging setting for VFM. To illustrate this issue, Fig. 2 presents two examples—multimodal grid data and half-moon data—demonstrating the performance of VFM under irregular geometries involving disconnected, curved modes and nonlinear low-dimensional manifolds. Samples are generated using an Euler ODE solver. As shown, VFM produces samples that are blurred across modes, and this further degrades as the number of sampling steps decreases. Moreover, the learned trajectories are highly curved, despite the target vector field being designed to be straight. Similar issues have been reported in recent works (Samaddar et al., 2025; Guo and Schwing, 2025). From a theoretical standpoint, this limitation can be attributed to the fact that VFM approximates the true distribution of the target vector field qdata (u∗ | zt , t) with a Gaussian distribution, qψVFM (u∗ | zt , t) = N (u∗ ; uψ t (zt ), I), by minimizing the Kullback-Leibler (KL) divergence (Kullback and Leibler, 1951) KL qdata ∥qψVFM = Eu∗ ∼qdata (u∗ |zt ,t) log qdata (u∗ | zt , t) − log qψVFM (u∗ | zt , t) . Hereafter, for notational simplicity, u∗ = z1 − z0 denotes the sample-level conditional target vector field in Eq. (2). This objective is equivalent to minimizing the ℓ2 objective (2), whose solution is characterized by the following proposition: Proposition 3.1. ûVFM (zt , t) = E[z1 − z0 | zt , t] is the conditional minimizer of the VFM objective (2). See proof in Appendix A.1. Since the Gaussian approximation is unimodal and, in the presence of multimodality, its solution follows the average direction of z1 − z0 , it tends to underfit multimodal vector fields, especially under few-step sampling and limited training data. Motivated by the intuition that multimodality can be better captured by employing multiple vector fields associated with different modes, we incorporate a mixture-of-experts mechanism (MoE; Jacobs et al., 1991; Jordan and Jacobs, 1994) and propose mixture-of-experts flow matching in the following section. 3.2

Mixture-of-Experts Flow Matching

Mixture-of-experts flow matching (MoE-FM) aims to improve generative performance and inference efficiency by decomposing global transport into multiple locally specialized vector fields that model heterogeneous transport geometries. Specifically, it introduces K expert vector fields ψ m m uψ k : R × [0, 1] → R , (zt , t) 7→ uk,t (zt ),

for k = 1, . . . , K, together with a learnable gating network that outputs K routing probabilities ψ π ψ : Rm × [0, 1] → ∆K−1 , (zt , t) 7→ (π1ψ , . . . , πK ),

where ∆K−1 denotes the (K − 1)-dimensional probability simplex, meaning that πkψ ≥ 0 and PK ψ ∗ k=1 πk = 1. The true conditional distribution qdata (u | zt , t) is approximated by an MoE model qψMoE-FM (u∗ | zt , t) =

K X

2 πkψ (zt , t)N (u∗ ; uψ k,t (zt ), σ I),

k=1

(4)

by minimizing the KL divergence KL qdata ∥qψMoE-FM , where σ ≥ 0 is a pre-specified parameter discussed below. This is equivalent to minimizing the negative log-likelihood (NLL) loss K n o X 1 ψ ψ 2 LMoE-FM (ψ) = Et,z1 ,zt |z1 − log πk (zt , t) × exp − 2 ∥uk,t (zt ) − (z1 − z0 )∥ . (5) 2σ k=1

Properties and Benefits. To characterize the behavior of MoE-FM, the following theorem describes the optimal expert vector fields and routing functions. Theorem 3.2. The MoE-FM objective (5) admits the following conditional optima: π̂kMoE-FM (zt , t) = E[γkψ (zt , t, z1 − z0 ) | zt , t], ûMoE-FM (zt , t) = k

E[γkψ (zt , t, z1 − z0 )(z1 − z0 ) | zt , t] E[γkψ (zt , t, z1 − z0 ) | zt , t]

for k = 1, . . . , K. Here, γkψ (zt , t, z1 − z0 ) denotes the k-th expert responsibility, defined as πkψ κσ z1 − z0 , uψ k,t (zt ) ψ γk (zt , t, z1 − z0 ) = PK , ψ ψ k′ =1 πk′ κσ z1 − z0 , uk′ ,t (zt ) where κσ (v, v ′ ) = exp −∥v − v ′ ∥2 /(2σ 2 ) is the radial basis function kernel. The proof is provided in Appendix A.2. These results show that the expert responsibilities γkψ implement a soft gating mechanism that assigns each velocity target u∗ = z1 − z0 to experts according to proximity in the vector field space. Each expert vector field ûMoE-FM then estimates a k local transport direction via γkψ -weighted averaging. Consequently, different experts specialize in distinct local flow geometries. As shown in Fig. 2, MoE-FM effectively allocates different expert vector fields to model distinct regions of the space. This produces higher-quality samples that more faithfully recover the data distribution compared to VFM. Moreover, the interpolating trajectories are substantially straighter, enabling accurate reconstruction of multimodal samples with very few sampling steps (e.g., four steps). Special Cases. (1) When K = 1, MoE-FM reduces to the VFM method, in which σ is independent of ψ and therefore does not affect the training objective. (2) The parameter σ ≥ 0 controls the softness of expert assignments. As σ → 0, the routing converges to a hard assignment rule using nearest-neighbor assignment. As σ → ∞, the likelihood becomes non-identifiable in the expert assignments, meaning that all assignments are equally likely. (see Appendix A.3). Sampling. We adopt a trajectory-level frozen routing strategy during sampling (Fig. 1). Specifically, we sample a discrete expert assignment e ∼ Cat(π ψ (z0 , 0)) at time t = 0, and then keep the assignment fixed throughout the trajectory. The final state is then obtained by integrating the corresponding time-dependent vector field uψ e,t (zt ). This yields a trajectory-level conditional generation process, in which each trajectory is governed by a single expert vector field. Compared to time-varying routing strategies, frozen routing avoids frequent expert switching during ODE integration, resulting in much more stable and computationally efficient integration. Moreover, it preserves trajectory-level geometric consistency by preventing continuous interpolation across different transport fields. In our language modeling formulation, we apply the MoE mechanism at the token level, allowing token-specific trajectories.

NAR Language Modeling via YAN

We propose a flow-based NAR language modeling paradigm (Sec. 4.1) and describe its training strategy (Sec. 4.2). The overall pipeline is illustrated in Fig. 1. 4.1

Mathematical Formulation

YAN is a latent variable language model in the class of (3), parameterized as Z pθ,ψ (y | x) = pθ (y | z)pψ (z | x)dz, 5

(6)

Figure 3: Latent representation distributions under different regularization schemes, projected onto the first two principal components and colored by selected high-frequency tokens: (a) unregularized cross-entropy, (b) MMD, (c) MMD + scale regularization, and (d) MMD + scale regularization with noisy perturbation. Percentages in parentheses denote the variance explained. Reduced variance concentration in the leading components indicates more isotropic latent spaces, with no dominant direction of variation.

where pθ (y | z) is a decoder that generates the target sequence y from a continuous latent representation z ∈ RL×d , and pψ (z | x) is a conditional latent generator given the source sequence ∗ ∗ x. Additionally, YAN introduces an encoder, Eϕ : V L → RL ×d , mapping discrete sequences of ∗ arbitrary length L∗ to a continuous space RL ×d . In YAN, the latent variable z is intended to capture the global semantic content and token-level dependency structure of the target sequence y. Conditioning on such a representation is designed to reduce token dependencies and, ideally, render tokens y1:L conditionally independent, i.e., pθ (y | z) =

L Y

pθ (yl | z).

l=1

Under this assumption, decoding can be performed in a fully parallel manner where each token is generated independently as yl | z ∼ Cat (ξθ,l (z)) ,

l = 1, . . . , L,

(7)

V −1 and {ξθ,l (z)}L denote token-wise probability vectors. The conditional latent generator l=1 ⊂ ∆ pψ (z | x) is learned using the mixture-of-experts flow matching proposed in Sec. 3. The encoder serves two purposes. First, it contextualizes the source sequence via zsrc = Eϕ (x). Second, it constructs a target latent representation ztgt = Eϕ (y), which defines the endpoint that the latent generator is trained to flow toward. This design enables self-supervised training of the latent flow. In the absence of such a target, an alternative is to introduce a pretrained teacher for distillation; however, in our preliminary experiments, we observe no empirical benefit from doing so, consistent with findings in other work (Li et al., 2022).

In summary, YAN aims to learn a latent representation z that is sufficiently expressive for the target sequence y to be decoded using a very small number of parallelizable layers, while employing flow matching as an efficient conditional latent generator. Motivated by this, we name the model YAN—Flow Until You Almost Know. 4.2

Two-Stage Training

Under the latent variable formulation (6), direct maximum likelihood training is intractable due to the marginalization over the latent variable. As an alternative, we adopt a two-stage training strategy. 6

Stage 1: Train the Encoder and Decoder. YAN employs an asymmetric autoencoder comprised of a high-capacity encoder and a lightweight decoder. Let ỹ ∈ V L̃ denote a text sequence from distribution p̃data and z̃ = Eϕ (ỹ) be its encoded latent representation. The first training stage learns to perform reconstructions of ỹ using this autoencoder under objective LRegularizedAE (Θ) = λCE LCE (Θ) + λMMD LMMD (Θ) + λScale LScale (Θ).

(8)

for pre-specified tuning parameters λCE , λMMD , λScale ≥ 0, where Θ = (θ, ϕ, ψ) denotes the collection of learnable parameters for notational simplicity. Here, the cross-entropy (CE) reconstruction loss, LCE (Θ) = Eỹ∼p̃data −

L̃ X

log pθ (ỹl | z̃) ,

(9)

l=1

is augmented by two latent regularizers. The first regularizer is the Maximum Mean Discrepancy (MMD; Gretton et al., 2012; Dziugaite et al., 2015), a measure of the discrepancy between the marginal distribution P of z̃ and the standard Gaussian distribution Q = N (0, I), defined as MMD2κ (P, Q) = Ex,x′ ∼P [κ(x, x′ )] + Ey,y′ ∼Q [κ(y, y ′ )] − 2Ex∼P,y∼Q [κ(x, y)], P ′ ′ ′ where κ(x, x )′ 2= 2 s∈S κs (x, x ) is a sum of radial basis function kernels κs (x, x ) = exp −∥x − x ∥ /(2s ) over a range of bandwidths s ∈ S. Replacing the expectations with the corresponding empirical estimates yields the MMD term used in training, 2

\ LMMD (Θ) = M MDκ (P, Q).

(10)

See Appendix A.4 for details. Our use of the MMD regularizer here is motivated by the Wasserstein autoencoder (WAE; Tolstikhin et al., 2018). Although closely related to the variational autoencoder (VAE; Kingma and Welling, 2013), WAE outperforms VAE by encouraging the latent representations to match the Gaussian distribution marginally rather than conditionally, thereby improving reconstruction quality (Tolstikhin et al., 2018). We also apply an ℓ2 penalty on the scale of z̃, LScale (Θ) = ∥z̃∥2 , to discourage latent representations from drifting excessively far from the origin. Theoretically, these regularization choices can be viewed as promoting latent isotropy while preserving local semantic structure, a property shown to benefit downstream performance and facilitate effective utilization of the latent space (Kudrjashov et al., 2024; Tyshchuk et al., 2023). Following the same principle, we further inject small Gaussian perturbations into the encoder outputs and let the decoder reconstruct from these perturbed representations, thereby expanding the region associated with each latent code. Fig. 3 visualizes the learned latent representation distributions under different regularization schemes. As shown, imposing MMD and scale regularization encourages the distribution to concentrate toward the origin and exhibit more isotropic, Gaussian-like behavior. Stage 2: Train the Latent Flow. The second stage trains the MoE-FM via the following objective LYAN (Θ) = αMoE-FM LMoE-FM (Θ) + αCE LCE (Θ),

(11)

where LMoE-FM is the negative log-likelihood loss defined in Eq. (5). In this stage, the latent flow is trained with initial point z0 ∼ N (0, I) and target endpoint ztgt = Eϕ (y), conditioned on the source latent representation zsrc = Eϕ (x). After obtaining the final latent state ẑ1 via the frozen routing sampling with an Euler solver, LCE is computed as the cross-entropy loss between the ground-truth sequence and the sequence decoded from ẑ1 . We provide additional analysis of the training in Appendix A.5.

Experiments

5.1

Experimental Setup

Our experiments evaluate the language generation and understanding capabilities of YAN, as well as its inference efficiency, particularly on long-form documents. In language model evaluation, while perplexity is a common evaluation metric, we do not use it in our setting for two reasons. First, perplexity has known limitations in measuring long-range 7

Table 1: Quality and efficiency results. For metrics with multiple values, results are reported following the order specified in the Metric column. T ∗ denotes the optimal sampling step that yields the best performance on each dataset, and the reported metric values correspond to T ∗ . R-1, R-L, and BS-F1 denote ROUGE-1, ROUGE-L, and BERTScore-F1, respectively. Bold and underlined values indicate the best and second-best results, respectively. YAN-M and YAN-TRF denote YAN with Mamba and Transformer architectures, respectively. Model Info

GPT-2

BART

Size T∗

124M -

139M -

68.9/61.8/206 42.8/85.1 28.5/79.7 93.8 98.9 90.1 48.0/41.0 47.7/15.7

81.8/80.6/211 46.5/90.3 21.3/70.6 91.2 94.7 88.0 78.9/76.7 78.3/74.8

Dataset

Metric

NarrativeQA SimpleStories ROCStories AG News DBpedia SST-2 SQuAD bAbI

R-1/R-L/TPS EM/BS-F1 EM/BS-F1 Accuracy Accuracy Accuracy F1/BS-F1 F1/BS-F1

LLaDA

YAN-M

8B 210M 1k/2/4/6/8/6/8/6 4/4/4/4/4/6/4/4

18.5/15.8/14 21.6/20.6 5.1/-17.5 92.1 95.3 90.7 88.8/87.2 99.7/99.6

YAN-TRF 200M 4/4/4/3/4/3/4/4

94.6/93.6/18.1k 94.9/93.9/20.6k 59.7/91.1 65.5/93.9 26.1/77.0 31.0/82.7 95.1 97.2 99.5 99.1 87.4 91.0 70.8/71.3 80.4/78.2 86.4/85.3 88.5/87.8

understanding and does not always align with human-like language processing (Hu et al., 2024; Kuribayashi et al., 2021). Second, theoretically, NAR language models—including the latent variable formulation in this work—generally do not admit tractable likelihoods. Instead, likelihoods must be approximated via estimation procedures or variational lower bounds (Sahoo et al., 2024; Lou et al., 2024), making cross-method perplexity comparisons unreliable. Accordingly, we adopt metrics that directly assess generation quality across multiple downstream tasks, in line with prior NAR work (Li et al., 2022; Gong et al., 2025). Architecture. We parameterize YAN using both Transformer (Vaswani et al., 2017) and Mamba (Gu and Dao, 2024) architectures, as shown in Fig. 1. The Transformer accepts both state and time as inputs, similar to Diffusion Transformers (Peebles and Xie, 2023), and incorporates cross-attention to the source input. It operates bidirectionally without causal masking. Mamba does not support cross-attention by design; therefore, we condition on the source input by combining Mamba-based self token mixing with an explicit cross-attention layer. Under computational constraints, we train YAN at the 200M-parameter scale. Detailed hyperparameter settings are provided in Appendix B.3. Training. We pretrain YAN on a dataset consisting of FineWiki (75%) (Penedo, 2025) and FineWeb (25%) (Penedo et al., 2024). FineWiki contains full-length Wikipedia articles and FineWeb is a large corpus of English web text. See Appendix B.1 for detailed dataset statistics and descriptions. Tasks and Metrics. We consider four downstream tasks. (1) Text infilling evaluates the ability to leverage bidirectional global context to generate coherent text for missing spans. We use NarrativeQA (Kočiský et al., 2018) and report ROUGE (Lin, 2004) and tokens per second (TPS). (2) Last-word completion evaluates next-token prediction given left-to-right context. We evaluate on ROCStories (Mostafazadeh et al., 2016) and SimpleStories (Finke et al., 2025) using exact match for accuracy and BERTScore-F1 (Zhang et al., 2020) for semantic similarity. (3) Question answering (QA) assesses reading comprehension and answer extraction from supporting passages. We evaluate on bAbI (Dodge et al., 2015; Weston et al., 2015) and SQuAD (Rajpurkar et al., 2016) using F1 score and BERTScore-F1. (4) Classification is a standard benchmark task for language understanding (Wang et al., 2018). We use the AG News, DBpedia (Zhang et al., 2015), and SST-2 (Socher et al., 2013) datasets and report classification accuracy. Additional dataset and evaluation details are provided in Appendix B.2. We further evaluate diversity using Dist-2 (Li et al., 2016) for bigram diversity, SelfBLEU (Zhu et al., 2018) for sentence-level similarity, and semantic distance (SemDist), computed as the average cosine distance between Sentence-BERT embeddings (Reimers and Gurevych, 2019). Baselines. For NAR methods, we include LLaDA (8B-Base) (Nie et al., 2025), which represents the state of the art in performance and scale among diffusion language models, including DiffuSeq (91M) (Gong et al., 2023), Plaid (1B) (Gulrajani and Hashimoto, 2023), and MDLM (110M) (Sahoo et al., 2024). We also evaluated DiffuLLaMA (7B) (Gong et al., 2025), an NAR diffusion model annealed from pretrained LLaMA (Grattafiori et al., 2024); however, its performance is consistently inferior to 8

Figure 4: Inference efficiency comparison. Top: Generation quality versus the sampling step T on the text infilling task. Bottom: Inference time across sequence lengths, reported as a ratio relative to YAN with T = 3 (GPT-2 is truncated due to its maximum context length of 512). In both plots, YAN refers to the Transformer-based YAN model. YAN achieves high-quality generation with as few as three sampling steps, yielding a 40–50× speedup over AR baselines. In contrast, the diffusion language model LLaDA requires approximately one inference step per token to reach acceptable generation quality (i.e., T = 1000), resulting in approximately 103 × higher inference latency for long documents (the T = 1000 curve is omitted for clarity).

that of LLaDA, and results are therefore omitted. For AR methods, we include BART (139M) (Lewis et al., 2020), whose source-target structure is similar to YAN, whereas LLaDA treats the source text as a prefix concatenated to the target. We also include GPT-2 (124M) (Radford et al., 2019), a decoder-only AR baseline frequently used in prior NAR work (e.g., Sahoo et al., 2024; Gong et al., 2023; Gong et al., 2025). These AR models are smaller than YAN, thereby controlling for model size when evaluating inference efficiency. Since YAN is not designed for general-purpose zero-shot language tasks in its current scope, we fine-tune it on each downstream task, following common practice for models with comparable capacity (Gong et al., 2023; Sahoo et al., 2024). For fairness, baseline models are also fine-tuned according to the procedures described in their original papers (see Appendix B.4 for details). To ensure a consistent comparison of efficiency, we perform inference with a batch size of 1 and use oracle sequence lengths for all methods, thereby eliminating the impact of padding on efficiency measurements. We adopt greedy sampling for all methods. All evaluations are conducted on a single NVIDIA H200 GPU. 5.2

Language Modeling Capabilities

At the current model scale, YAN demonstrates solid language modeling capabilities, showing strong performance in both text generation and understanding, as shown in Tab. 1. Language Understanding Performance. YAN consistently outperforms baseline models on classification tasks, achieving near-perfect accuracy on AG News and DBpedia. On the more challenging SST-2 binary sentiment dataset, YAN continues to deliver the strongest performance, followed by LLaDA. In QA tasks, LLaDA attains the highest quality scores, with YAN ranking second, partially 9

due to the larger training scale of LLaDA, which provides richer world knowledge. Nonetheless, the competitive performance of both LLaDA and YAN across QA benchmarks indicates that NAR models exhibit stronger global text comprehension compared to AR models, particularly outperforming decoder-only architectures such as GPT-2. Text Generation Quality. YAN achieves the highest generation quality on both text infilling and last-word completion tasks. In text infilling, YAN outperforms BART, which is trained as a denoising autoencoder. This advantage can be partially attributed to the longer context length used by YAN, which facilitates the modeling of long-range dependencies in the NarrativeQA dataset. For last-word completion, YAN surpasses GPT-2 and attains the highest accuracy on SimpleStories, which is longer than ROCStories and thus provides richer contextual information. Transformer versus Mamba. While YAN exhibits competent language modeling capabilities under both architectures, the Transformer-based variant generally outperforms its Mamba-based counterpart. This performance gap suggests that the self-attention mechanism—in particular, bidirectional attention without causal masking in our NAR setting—is more effective at capturing contextual information and modeling long-range dependencies. These empirical results are consistent with prior analyses showing that Mamba architectures tend to underperform Transformers on memory-intensive tasks, including information retrieval and long-context understanding (Waleffe et al., 2024; Jelassi et al., 2024; Ben-Kish et al., 2025). 5.3

Inference Efficiency Analysis

We observe a substantial inference efficiency advantage of YAN over both AR and diffusion-based NAR models. Fig. 4 analyzes inference efficiency as a function of the sampling step T and the generated sequence length on the infilling task, leading to the following key findings. YAN achieves high-quality long-document generation with as few as three Euler sampling steps. This behavior is also observed across other considered downstream tasks with shorter target sequences, as shown in Tab. 1 and in more detailed results provided in Appendix C.1. This stands in sharp contrast to LLaDA, which requires approximately one inference step per token to reach acceptable generation quality. Moreover, increasing the sampling steps beyond this low-step regime does not yield further performance improvements and is therefore unnecessary. YAN achieves a 40–50× inference speedup over AR baselines and a speedup on the order of 103 × over the diffusion language model. The former comparison is made against GPT-2 and BART, which have smaller parameter scales than YAN, indicating that the observed efficiency advantage is not driven by model capacity but instead arises from the parallel decoding with the MoE-FM generator. The latter comparison is against LLaDA with T = 1000, which is required to attain its highest generation quality, as shown in Fig. 4 (see Appendix C.2 for additional results).

Figure 5: Trade-off between quality and diversity.

5.4

Diversity Analysis

While the sampling path follows deterministic ODE trajectories, YAN is a valid generative model with stochasticity in the generated text. In this case, diversity becomes an important metric for preventing mode collapse. However, the evaluation of generation diversity is traded off against generation quality in open-ended language generation tasks (Zhang et al., 2021). Fig. 5 illustrates the trade-off between quality and diversity on the infilling task under different sampling steps. As shown, 10

configurations that prioritize quality typically correspond to lower diversity. Nevertheless, Fig. 5 also shows configurations whose diversity exceeds that of the baselines, indicating that diversity can be effectively preserved and adjusted through the choice of configuration, such as the number of sampling steps.

Conclusions

We presented mixture-of-experts flow matching that enhances the latent representational capacity of vanilla flow matching when applied to text. Building on this formulation, we proposed YAN, a non-autoregressive language model using latent flows. YAN achieves high-quality generation with substantially fewer decoding steps, leading to faster inference compared to autoregressive and diffusion-based language models. Given its effectiveness at the current model scale, a promising direction for future work is to scale this approach to larger models and datasets. Moreover, while our current implementation adopts a dense mixture-of-experts formulation, exploring sparse expert routing as the model scales may further improve inference efficiency while maintaining generation quality.

References K. Amara, R. Sevastjanova, and M. El-Assady. Syntaxshap: Syntax-aware explainability method for text generation. In Findings of the Association for Computational Linguistics ACL 2024, pages 4551–4566, 2024. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. A. Ben-Kish, I. Zimerman, S. Abu-Hussein, N. Cohen, A. Globerson, L. Wolf, and R. Giryes. Decimamba: Exploring the length extrapolation potential of mamba. In The Thirteenth International Conference on Learning Representations, 2025. X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004. X. Cai, J. Huang, Y. Bian, and K. Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations, 2021. C. Cheng, J. Li, J. Fan, and G. Liu. α-flow: A unified framework for continuous-state discrete flow matching models. arXiv preprint arXiv:2504.10283, 2025. A. Davtyan, S. Sameni, and P. Favaro. Efficient video prediction via sparsely conditioned flow matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23263–23274, 2023. P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, A. Miller, A. Szlam, and J. Weston. Evaluating prerequisite qualities for learning end-to-end dialog systems. arXiv preprint arXiv:1511.06931, 2015. G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906, 2015. P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024. 11

K. Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, 2019. G. Feng, Y. Geng, J. Guan, W. Wu, L. Wang, and D. He. Theoretical benefit and limitation of diffusion language model. arXiv preprint arXiv:2502.09622, 2025. L. Finke, C. Sreedhara, T. Dooms, M. Allen, E. Zhang, J. D. Rodriguez, N. Nabeshima, T. Marshall, and D. Braun. Parameterized synthetic text generation with simplestories. arXiv preprint arXiv:2504.09184, 2025. J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y. Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019. I. Gat, T. Remez, N. Shaul, F. Kreuk, R. T. Chen, G. Synnaeve, Y. Adi, and Y. Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37:133345–133385, 2024. M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324, 2019. A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex. Openwebtext corpus. http://Skylion007. github.io/OpenWebTextCorpus, 2019. S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2023. S. Gong, S. Agarwal, Y. Zhang, J. Ye, L. Zheng, M. Li, C. An, P. Zhao, W. Bi, J. Han, et al. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The journal of machine learning research, 13(1):723–773, 2012. A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling, 2024. J. Gu and X. Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 120–133, 2021. J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations, 2018. I. Gulrajani and T. B. Hashimoto. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems, volume 36, pages 16693–16715. Curran Associates, Inc., 2023. P. Guo and A. Schwing. Variational rectified flow matching. In Forty-second International Conference on Machine Learning, 2025. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020. P. Holderrieth and E. Erives. An introduction to flow matching and diffusion models. arXiv preprint arXiv:2506.02070, 2025. Y. Hu, Q. Huang, M. Tao, C. Zhang, and Y. Feng. Can perplexity reflect large language model’s ability in long text understanding? arXiv preprint arXiv:2405.06105, 2024. 12

F. Huang, T. Tao, H. Zhou, L. Li, and M. Huang. On the learning of non-autoregressive transformers. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9356–9376. PMLR, 17–23 Jul 2022. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. S. Jelassi, D. Brandfonbrener, S. M. Kakade, and E. Malach. Repeat after me: Transformers are better than state space models at copying. In International Conference on Machine Learning, pages 21502–21521. PMLR, 2024. M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994. L. Kaiser, S. Bengio, A. Roy, A. Vaswani, N. Parmar, J. Uszkoreit, and N. Shazeer. Fast decoding in sequence models using discrete latent variables. In International Conference on Machine Learning, pages 2390–2399. PMLR, 2018. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. T. Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl_a_00023. S. Kudrjashov, O. Karpik, and E. Klyshinsky. Shrink the longest: improving latent space isotropy with simplicial geometry. In International Conference on Analysis of Images, Social Networks and Texts, pages 120–130. Springer, 2024. S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951. T. Kuribayashi, Y. Oseki, T. Ito, R. Yoshida, M. Asahara, and K. Inui. Lower perplexity is not always human-like. arXiv preprint arXiv:2106.01229, 2021. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7871–7880, 2020. J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 110–119, 2016. X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems, volume 35, pages 4328– 4343. Curran Associates, Inc., 2022. C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Chen, D. Lopez-Paz, H. Ben-Hamu, and I. Gat. Flow matching guide and code. arXiv preprint arXiv:2412.06264, 2024. X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. Y. Liu, Y. Wan, J.-G. Zhang, W. Zhao, and P. S. Yu. Enriching non-autoregressive transformer with syntactic and semanticstructures for neural machine translation. arXiv preprint arXiv:2101.08942, 2021. 13

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, pages 32819–32848. PMLR, 2024. A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu. K. Misra, A. Ettinger, and J. Rayz. Exploring bert’s sensitivity to lexical cues using tests from semantic priming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4625–4635, 2020. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, 2016. S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. G.

Penedo. Finewiki, 2025. URL https://huggingface.co/datasets/ Source: Wikimedia Enterprise Snapshot API HuggingFaceFW/finewiki. (https://api.enterprise.wikimedia.com/v2/snapshots). Text licensed under CC BY-SA 4.0 with attribution to Wikipedia contributors.

G. Penedo, H. Kydlíček, A. Lozhkov, M. Mitchell, C. A. Raffel, L. Von Werra, T. Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. S. Rajaee and M. T. Pilehvar. A cluster-based approach for improving isotropy in contextual embedding space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 575–584, 2021a. S. Rajaee and M. T. Pilehvar. How does fine-tuning affect the geometry of embedding space: A case study on isotropy. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3042–3049, 2021b. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016. doi: 10.18653/v1/D16-1264. N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. W. Rudin. Real and complex analysis. McGraw-Hill, Inc., 1987. S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, volume 37, pages 130136–130184, 2024. doi: 10.52202/079017-4135. A. Samaddar, Y. Sun, V. Nilsson, and S. Madireddy. Efficient flow matching using latent variables. arXiv preprint arXiv:2505.04486, 2025. R. Shu, J. Lee, H. Nakayama, and K. Cho. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. In Proceedings of the aaai conference on artificial intelligence, volume 34, pages 8846–8853, 2020. 14

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013. J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018. K. Tyshchuk, P. Karpikova, A. Spiridonov, A. Prutianova, A. Razzhigaev, and A. Panchenko. On isotropy of multimodal embeddings. Information, 14(7), 2023. ISSN 2078-2489. doi: 10.3390/ info14070392. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. R. Waleffe, W. Byeon, D. Riach, B. Norick, V. Korthikanti, T. Dao, A. Gu, A. Hatamizadeh, S. Singh, D. Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353–355, 2018. J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. Van Merriënboer, A. Joulin, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. L. Yang, Z. Zhang, Z. Zhang, X. Liu, M. Xu, W. Zhang, C. Meng, S. Ermon, and B. Cui. Consistency flow matching: Defining straight flows with velocity consistency. arXiv preprint arXiv:2407.02398, 2024. J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. H. Yuan, Z. Yuan, C. Tan, F. Huang, and S. Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers. arXiv preprint arXiv:2212.10325, 2022. H. Zhang, D. Duckworth, D. Ippolito, and A. Neelakantan. Trading off diversity and quality in natural language generation. In Proceedings of the workshop on Human Evaluation of NLP Systems (HumEval), pages 25–33, 2021. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015. H. Zheng, S. Gong, R. Zhang, T. Chen, J. Gu, M. Zhou, N. Jaitly, and Y. Zhang. Continuously augmented discrete diffusion model for categorical generative modeling. arXiv preprint arXiv:2510.01329, 2025. Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097–1100, 2018.

Mathematical Details

A.1

Proof of Proposition 3.1

The following proposition restates Proposition 3.1, with subscripts explicitly indicating the random variables with respect to which the expectations are taken. Proposition A.1. The vanilla flow matching objective (2) 2 LVFM (ψ) = Et,z0 ,z1 ,zt ∥uψ t (zt ) − (z1 − z0 )∥

is conditionally minimized by ûVFM (zt , t) = Eu∗ [u∗ | zt , t] = Ez0 ,z1 [z1 − z0 | zt , t]. where u∗ = z1 − z0 . Proof. Write û = ûVFM (zt , t) for brevity. Apply decomposition ∥uψ − u∗ ∥2 = ∥uψ − û + û − u∗ ∥2 = ∥uψ − û∥2 + ∥û − u∗ ∥2 + 2(uψ − û)⊤ (û − u∗ ), where the cross term vanishes in expectation since Eu∗ (uψ − û)⊤ (û − u∗ ) | zt , t = (uψ − û)⊤ Eu∗ [û − u∗ | zt , t] = (uψ − û)⊤ (û − Eu∗ [u∗ | zt , t]) = 0. Then, the objective becomes LVFM (ψ) = Et,u∗ ,zt ∥uψ − u∗ ∥2 = Eu∗ ∥uψ − û∥2 | zt , t + Eu∗ ∥û − u∗ ∥2 | zt , t + 2Eu∗ (uψ − û)⊤ (û − u∗ ) | zt , t | {z } | {z } | {z } =∥uψ −û∥2

independent of ψ

= ∥u − û∥ + C, where C is a constant independent of ψ. This is minimized at uψ = û. A.2

Proof of Theorem 3.2

Lemma A.2. Introduce an expert assignment random variable g such that the MoE distribution of u∗ in Equation (4), K X p(u∗ | zt , t) = πkψ (zt , t)φσ u∗ ; uψ k,t (zt ) , k=1

admits the equivalent conditional representation 2 g | zt , t ∼ Cat Πψ (zt , t) , u∗ | g = k, zt , t ∼ N (uψ k,t (zt ), σ I), ψ for k ∈ {1, . . . , K}. Here, Πψ (zt , t) = (π1ψ , . . . , πK ) ∈ ∆K−1 denotes the vector of routings ψ ψ πk = πk (zt , t), and 1 2 −m/2 2 φσ (x; µ) = (2πσ ) exp − 2 ∥x − µ∥ 2σ

denotes the m-dimensional Gaussian density function. Then, given an observation of u∗ , the posterior distribution of g is g | zt , t, u∗ ∼ Cat Γψ (zt , t, u∗ ) , ψ where Γψ (zt , t, u∗ ) = (γ1ψ , . . . , γK ) ∈ ∆K−1 denotes the vector of responsibilities πkψ φσ u∗ ; uψ k,t (zt ) ψ ψ ∗ γk = γk (zt , t, u ) = PK , k ∈ {1, . . . , K}. ψ ψ ∗ k′ =1 πk′ φσ u ; uk′ ,t (zt )

(12)

Proof. By Bayes’ theorem, for each k ∈ {1, . . . , K}, Pr(g = k | zt , t)p(u∗ | zt , t, g = k) γkψ = Pr(g = k | zt , t, u∗ ) = PK , ′ ∗ ′ k′ =1 Pr(g = k | zt , t)p(u | zt , t, g = k ) which simplifies to the stated expression. Lemma A.3. Define S(u∗ ; η) =

K X

πkψ (zt , t)φσ u∗ ; uψ k,t (zt ) ,

ℓ(η) = Eu∗ [− log S(u∗ ; η) | zt , t],

k=1 ψ ψ where u = z1 − z0 and η = (π1ψ , . . . , πK , uψ 1 , . . . , uK ) denotes the collection of parameters given ψ ψ ψ ψ (zt , t), with (π1 , . . . , πK ) ∈ ∆K−1 and uk = uk,t (zt ) ∈ Rm . Then, conditional on (zt , t), ∗

∇η ℓ(η) = Eu∗ [−∇η log S(u∗ ; η) | zt , t], where the gradient with respect to π is taken on the simplex ∆K−1 , provided that the following regularity conditions hold: ψ (A1) There exists ϵ ∈ (0, 1/K) such that (π1ψ , . . . , πK ) ∈ ∆K−1 = (π1 , . . . , πK ) : πk ≥ ϵ PK ϵ, k=1 πk = 1 . (A2) There exists B < ∞ such that ∥uψ k (zt , t)∥ ≤ B for all k = 1, . . . , K; (A3) Eu∗ [∥u∗ ∥ | zt , t] < ∞. Proof. Write f (u∗ ; η) = − log S(u∗ ; η). ∞ (i) Differentiability. Since the Gaussian density φσ (u∗ ; µψ with respect to µψ k ) is C k , and S(u∗ ; η) is linear in πkψ for all k, it follows that f (u∗ ; η) is differentiable in η for all u∗ .

(ii) Gradients. For k = 1, . . . , K, ∇uψ f (u∗ ; η) =

ψ ∗ πkψ φσ u∗ ; uψ k,t (zt ) uk − u

S(u∗ ; η)

σ2

∗ γkψ (uψ k −u ) , σ2

φσ u ; uψ γkψ k,t (zt ) ∗ ∇πψ f (u ; η) = − = − , k S(u∗ ; η) πkψ ψ where γkψ is defined in (12). Since (π1ψ , . . . , πK ) ∈ ∆K−1 , the gradient with respect to π is ∗

interpreted as the gradient on the simplex, i.e., the orthogonal projection of the Euclidean gradient onto the tangent space K X Tπ ∆K−1 = v ∈ RK : vk = 0 . k=1

Consequently, the simplex gradient is K γkψ 1 X γkψ′ ∗ ∇∆ f (u ; η) = − + . ψ πk K ′ πkψ′ πkψ k =1

(iii) Dominating bound. Since 0 ≤ γkψ ≤ 1, ∇∆ f (u∗ ; η) ≤ πψ k

2 ϵ

by assumption (A1), and ∗ ∗ ∥uψ ∥uψ ∥u∗ ∥ + B k −u ∥ k ∥ + ∥u ∥ ≤ ≤ k σ2 σ2 σ2 by assumption (A2). Thus, there exists constants C0 , C1 < ∞ such that, for all η, u∗ , ∥∇η f (u∗ ; η)∥ ≤ C0 + C1 ∥u∗ ∥, and the right-hand side is integrable with respect to u∗ by assumption (A3).

∥∇uψ f (u∗ ; η)∥ ≤

Combining (i)-(iii),the result follows from the Dominated Convergence Theorem (Rudin, 1987). With Lemmas A.2 and A.3 in place, the following result completes the proof of Theorem 3.2. Theorem A.4. The MoE-FM objective (5) K n o X 1 ψ ψ 2 LMoE-FM (ψ) = Et,z0 ,z1 ,zt − log πk (zt , t) exp − 2 ∥uk,t (zt ) − (z1 − z0 )∥ 2σ k=1

is conditionally minimized by π̂kMoE-FM (zt , t) = Ez0 ,z1 [γkψ (zt , t, z1 − z0 ) | zt , t], ûMoE-FM (zt , t) = k

Ez0 ,z1 [γkψ (zt , t, z1 − z0 )(z1 − z0 ) | zt , t] Ez0 ,z1 [γkψ (zt , t, z1 − z0 ) | zt , t]

for k ∈ {1, . . . , K}, where γkψ is the responsibility defined in (12). Proof. By the Law of Iterated Expectations, LMoE-FM (ψ) = Ezt ,t Ez0 ,z1 (− log S(z1 − z0 , η) | zt , t) = Ezt ,t [ℓ(η)] where η, S(·), and ℓ(·) are defined in Lemma A.3. Now we show that π̂k = π̂kMoE-FM (zt , t) and ûk = ûMoE-FM (zt , t) minimize ℓ(η). k (i) By Lemma A.3 and its proof, ∗ 2 ∇uψ ℓ(η) = Eu∗ ∇uψ f (u∗ ; η) | zt , t = Eu∗ γkψ (uψ k − u )/σ | zt , t . k

Let ∇uψ ℓ(η) = 0, then we have minimizer k

ûk =

Eu∗ [γkψ u∗ | zt , t] Eu∗ [γkψ | zt , t]

(ii) Define Lagrangian (Boyd and Vandenberghe, 2004) X K πkψ − 1 J = ℓ(η) + λ k=1

for multiplier λ ∈ R. Let ∇πψ J = Eu∗ k

Since

γkψ

− ψ | zt , t + λ = 0, πk

k = 1, . . . , K.

PK ψ ψ k=1 πk = 1 and k=1 γk = 1, we have λ = 1 and minimizer π̂k = Eu∗ [γkψ | zt , t].

A.3 A.3.1

Two Extrema of σ σ→0

Proposition A.5. When σ → 0, π̂kMoE-FM (zt , t) → Eu∗ 1{k ∈ M(u∗ )}πkM,ψ (u∗ ) | zt , t , Eu∗ 1{k ∈ M(u∗ )}πkM,ψ (u∗ )u∗ | zt , t MoE-FM ûk (zt , t) → , Eu∗ 1{k ∈ M(u∗ )}πkM,ψ (u∗ ) | zt , t ∗ 2 for k = 1, . . . , K, where M(u∗ ) = arg min1≤k≤K ∥uψ k − u ∥ is the set of minimizing indices and M,ψ ∗ ψ P ψ πk (u ) = πk / k′ ∈M(u∗ ) πk′ is the scaled routing probability on set M(u∗ ).

∗ 2 Proof. Define dk = ∥uψ k − u ∥ and dmin = min1≤k≤K dk . The responsibilities (12) can be expressed as πkψ exp − 2σ1 2 (dk − dmin ) ψ γk = PK , k ∈ {1, . . . , K}. ψ 1 ′ k′ =1 πk′ exp − 2σ 2 (dk − dmin )

Since dk ≥ dmin for all k, then, as σ → 0, the denominator is n o X ψ X ψ X ψ 1 πk ′ + πk′ exp − 2 (dk′ − dmin ) → πk ′ , 2σ ′ ′ ′ k ∈M

k ∈M

k ∈M /

while the numerator πkψ exp

o 1 − 2 (dk − dmin ) → 2σ

(

πkψ , k ∈ M, 0, k∈ / M.

Thus, as σ → 0 γkψ → 1{k ∈ M} P

πkψ

ψ k′ ∈M πk′

= 1{k ∈ M}πkM,ψ .

This leads to the stated expressions under regularity assumptions in Lemma A.3.

Corollary A.6. Assume singleton M(u∗ ) = {k ∗ (u∗ )}. Then, as σ → 0, π̂kMoE-FM (zt , t) → Pr k ∗ (u∗ ) = k | zt , t , ûMoE-FM (zt , t) → Eu∗ [u∗ | k ∗ (u∗ ) = k, zt , t]. k for k = 1, . . . , K. Interpretations. As σ → 0, the optimal routing π̂kMoE-FM (zt , t) converges to the conditional prob∗ 2 ability that expert k is selected by the hard assignment rule k ∗ (u∗ ) = arg min1≤k≤K ∥uψ k −u ∥ . MoE-FM Meanwhile, each expert vector field ûk (zt , t) converges to the conditional expectation of the velocity target given that it is assigned to expert k. Therefore, MoE-FM reduces to a hard-assignment MoE formulation that performs conditional vector field estimation. A.3.2

σ→∞

As σ → ∞, the MoE-FM likelihood (5) LMoE-FM (ψ) converges to a constant independent of ψ, being uninformative in the parameter ψ. Also, the responsibilities (12) converge to the routings. In other words, observing u∗ provides no guidance for optimization, and the expert assignment is non-identifiable under the MoE-FM objective—no assignment is preferred over another by the objective. A.4

A Full Expression of MMD Regularizer

Let {xi }ni=1 ∼ P and {yj }m j=1 ∼ Q be independent and identically distributed samples drawn from distributions P and Q, respectively. We use the unbiased empirical estimator of the squared Maximum Mean Discrepancy (MMD): 2

\ M MDκ (P, Q) =

n m X X 1 1 2 XX κ(xi , xj ) + κ(yi , yj ) − κ(xi , yj ), n(n − 1) m(m − 1) nm i=1 j=1 i̸=j

i̸=j

′ ′ ′ where κ(x, x )′ 2= 2 s∈S κs (x, x ) is a sum of radial basis function kernels κs (x, x ) = exp −∥x − x ∥ /(2s ) and S = {0.2, 0.5, 1.0, 2.0, 5.0} denotes a set of kernel bandwidths.

A.5

A discussion of the YAN objective

The second stage of the YAN model trains the flow matching under the loss function (11), i.e., LYAN (Θ) = αMoE-FM LMoE-FM (Θ) + αCE LCE (Θ). Although it might seem counterintuitive to include the CE loss in training the flow, as CE is not required by the flow formulation, an interesting observation from our implementation is that the CE term plays a crucial role in effectively training the latent flow. With LMoE-FM alone, the learned flow suffers from misalignment issues, where the latent distribution is well modeled but decodes to incorrect tokens. We interpret this behavior as arising from the token-label-agnostic nature of the NLL objective, whereas incorporating the CE loss anchors the latent flow to the decoding objective. Indeed, CE supervision is common in NAR modeling, despite being motivated by different considerations (e.g., Gong et al., 2025; Zheng et al., 2025).

Implementation Details

B.1

Dataset Statistics

Table 2: Summary of datasets used for pre-training and fine-tuning, including dataset size and per-sample word/token length statistics. Datasets FineWiki + FineWeb NarrativeQA ROCStories SimpleStories bAbI SQuAD AG News DBpedia SST-2

Words

Size 8.7M 46.8k 98.2k 2.1M 11k 98.2k 127.6k 630k 68.2k

Tokens

Min

Mean

Max

Min

Mean

Max

7 212 19 42 17 34 17 9 5

501.16 600.74 43.96 225.77 38.05 139.53 43.75 54.69 12.89

1770 1048 71 606 65 599 162 226 49

20 267 26 55 32 56 31 23 15

769.67 765.88 53.19 280.65 59.33 191.53 64.47 86.48 24.94

2048 1406 88 754 92 787 227 303 69

Tab. 2 summarizes the datasets used for pre-training and fine-tuning, reporting total sample size and per-sample word and token length statistics (min / mean / max). Word lengths are measured as the number of words per sample, and token lengths are computed after tokenization. For each dataset, we take a 90%-5%-5% split for training, validation, and testing. Tab. 3 shows the total number of tokens observed during pre-training, estimated by Total Tokens = Training Steps × Global Batch Size × Average Tokens per Sample. Table 3: Training steps and tokens. Model

Training Steps

Global Batch Size

Total Tokens

820k 600k

32 32

20.2B 14.8B

YAN-Transformer YAN-Mamba

B.2

Datasets and Evaluation Details

FineWiki (Penedo, 2025) and FineWeb (Penedo et al., 2024) are high-quality large-scale corpora that have been used in recent NAR work for pre-training (e.g., Gong et al., 2025). In particular, FineWeb improves upon the commonly used OpenWebText dataset (Gokaslan et al., 2019). We use its subset, FineWeb-Edu, which contains educational content with dense factual and conceptual knowledge (Lozhkov et al., 2024). NarrativeQA (Kočiský et al., 2018) consists of stories paired with corresponding questions and answers for evaluating document-level understanding. It provides long documents with rich long20

range dependencies and lexical diversity; therefore, we leverage it for text infilling evaluation by randomly masking 5%–10% of tokens, with masking lengths uniformly sampled from 1, 2, 3. ROCStories (Mostafazadeh et al., 2016) comprises daily-life short stories. We use it for the last-word completion task, following Misra et al. (2020) and Amara et al. (2024). At a larger scale and with longer contexts, we similarly use the SimpleStories dataset (Finke et al., 2025), which contains stories generated by GPT-4o-mini. For the last-word completion task, we treat the text excluding the final token as the source input and the final token as the target output. bAbI (Dodge et al., 2015; Weston et al., 2015) is a reading comprehension dataset consisting of various question–answering tasks, including counting, lists/sets, and argument relations. SQuAD (Rajpurkar et al., 2016) is another reading comprehension dataset in which the answer to each question is a span extracted from the corresponding passage. AG News and DBpedia (Zhang et al., 2015) contain news articles and Wikipedia articles, respectively, categorized into 4 and 14 classes for classification evaluation. SST-2 (Socher et al., 2013) is designed for sentence-level sentiment classification and is included in the GLUE benchmark (Wang et al., 2018). For question-answering and classification tasks, we treat the context passage (and question, when applicable) as the source input and the answer or class label as the target output.

Figure 6: Effect of sampling steps on generation quality of YAN (Transformer) across tasks.

Figure 7: Effect of sampling steps on generation quality of YAN (Mamba) across tasks.

B.3

Model Optimization and Hyperparameters

Transformer. The encoder of YAN uses 6 transformer blocks (Vaswani et al., 2017). The vector field network uses 8 transformer blocks, with cross attention to the encoder at the 1st, 2nd, 3rd, 5th layers. The gating network is a 2-layer MLP. The decoder network consists of two linear layers with a 21

Figure 8: Effect of sampling steps on generation quality of LLaDA across tasks.

skip connection (i.e., residual feed-forward network). During training, ODE integration is performed using four Euler steps. The maximum context length is 2048, and the hidden dimension of the model is 512. We use 6 experts, with σ = 0.1. Mamba. The Mamba-based YAN follows a similar architecture to its Transformer-based counterpart, except that within each Transformer block, the self-attention mechanism is replaced by a bidirectional Mamba token mixer. Specifically, two independently parameterized Mamba modules are applied to the input sequence in forward and reverse temporal order, and their outputs are concatenated and linearly projected back to the model dimension. The encoder uses 4 such blocks, and the vector field network uses 6 such blocks. Tokenizer. YAN uses the LLaMA 3 tokenizer (Grattafiori et al., 2024) with a vocabulary size of 128256. Since YAN handles variable-length inputs via right padding, we additionally include a special token <|padding|> with tokenid=128256. To support the infilling task, we further add a special token <|mask|> with tokenid=128257. The resulting vocabulary size is 128258. For all other baseline models evaluated on downstream tasks, we use their original tokenizers. Optimization. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with β1 = 0.9, β2 = 0.999, and a weight decay of 0.01. The learning rate is set to 1e−4 for pre-training and 1e−5 for fine-tuning. We apply a linear warmup schedule for the first 1000 steps starting from 0, after which the learning rate remains constant at the target value. Training is performed on a single node using 8 NVIDIA H200 GPUs. B.4

Fine-Tuning

BART is a sequence-to-sequence model that treats the source and target texts in the same way as YAN. For LLaDA and GPT-2, the source is treated as a prefix concatenated to the target. Across all models, we use oracle sequence lengths for both training and inference, excluding the impact of different length-handling strategies.

Additional Results

C.1

Sensitivity to Sampling Steps

As shown in Fig. 6 and Fig. 7, a sampling step of T = 3 is sufficient for YAN to achieve acceptable generation quality with both Transformer and Mamba architectures. Increasing T beyond 6 generally degrades generation quality, suggesting that excessive sampling has adverse effects. This behavior contrasts sharply with the diffusion language model LLaDA (Nie et al., 2025), for which increasing the number of sampling steps typically improves generation quality, as illustrated in Fig. 8. 22

Figure 9: Inference time across generated sequence lengths, reported as a ratio to YAN (Transformer) with T = 3.

C.2

Inference Time of LLaDA

Fig. 9 shows the inference time curves of LLaDA with sampling steps T = 500 and T = 1000, complementing Fig. 4. It demonstrates the inference latency on the order of 103 × higher than that of YAN. C.3

Generation Examples

Fig. 10 and Fig. 11 present selected examples generated by YAN for three downstream tasks: question answering, last-word completion, and text infilling. Each example is generated using an Euler ODE solver with four steps.

Example 3

Example 2

Example 1

(a) Last-Word Completion Source

Ground Truth Target

Generation

A tv network shot a new show. They started to advertise it. A lot of viewers watched it. It had high ratings. It won an ____.

award

gold

wealth

Ground Truth Target

Generation

garden

increase in jet aircraft traffic

Beside a bubbling brook, a man found a magical fish that could grant any wish. He asked for wealth and received bags of gold. But with the gold came greed, and soon he lost his friends. They turned away, not wanting his riches. The man felt empty and alone. He learned that true wealth is not about money but the people we cherish. The fish had given him everything but happiness. With a heavy heart, he returned the fish to the brook, choosing love over ____. Pat liked soft cheeses. Goat cheese was his favorite. He would spread it on crackers and put it on pizza. Pat ate so much goat cheese that he felt sick. He waited a week before he ate any ____.

(b) Question Answering

Example 2

Example 1

Source

John went to the garden. John journeyed to the bathroom. Daniel went to the hallway. Daniel journeyed to the garden. Mary went back to the bedroom. Sandra travelled to the bedroom. ## QUESTION ## Where is Daniel? Airport construction boomed during the 1960s with the increase in jet aircraft traffic. Runways were extended out to 3,000 m (9,800 ft). The fields were constructed out of reinforced concrete using a slip-form machine that produces a continual slab with no disruptions along the length. The early 1960s also saw the introduction of jet bridge systems to modern airport terminals, an innovation which eliminated outdoor passenger boarding. These systems became commonplace in the United States by the 1970s. ## QUESTION ## Why did airport construction boom during the 1960s?

Figure 10: Last-word completion and question answering examples of YAN

Source

Ground Truth Target

Generation

The film follows three friends who have been in a rut in their lives: Adam Yates (John Cusack) is dumped by his girlfriend; Nick WebberAgnew (Craig Robinson) is a henpecked husband with a dead-end job at a dog spa; and Lou Dorchen (Rob Corddry) is a party animal in his 40s. When Lou is hospitalized for carbon monoxide poisoning, Adam and Nick sympathetically take him and Adam's shut-in 20-year-old nephew Jacob (Clark Duke) to a ski resort at Kodiak Valley, where the three had some good times in the past. During a night of heavy drinking in the hotel room's hot tub, they spill the contents of a drink called Chernobly on the console. The next day, they go skiing, but after too many strange occurrences (people dressed in 1980s fashion, music videos on MTV), they realize they have traveled back to 1986. Not only that, but they have also assumed their younger bodies: they see each other as their normal age, but in their reflections and to other people, they appear as they did back then, except Jacob, who appears as himself but occasionally flickers. A mysterious hot tub repairman (Chevy Chase) appears and warns them not to change anything as it might affect the machine. In order to minimize the butterfly effect, the guys plan to re-enact their experiences. Adam has to break up with his girlfriend Jenny (Lyndsy Fonseca) and get stabbed in the eye with a fork; Lou must pick a fight and get beaten up by Blaine (Sebastian Stan), a ski patrol bully; and Nick must …

The film follows three friends who have been in a rut in their lives. Adam Yates (John Cusack) is dumped by his girlfriend; Nick WebberAgnew (Craig Robinson) is a henpecked husband with a dead-end job at a dog spa; and Lou Dorchen (Rob Corddry) is a party animal in his 40s. When Lou is hospitalized for carbon monoxide poisoning, Adam and Nick sympathetically take him and Adam's shut-in 20-year-old nephew Jacob (Clark Duke) to a ski resort at Kodiak Valley, where the three had some good times in the past. During a night of heavy drinking in the hotel room's hot tub, they spill the contents of a drink called Chernobly on the console. The next day, they go skiing, but after too many strange occurrences (people dressed in 1980s fashion, music videos on MTV), they realize they have traveled back to 1986. Not only that, but they have also assumed their normal bodies: they see each other as their normal age, but in their reflections and to other people, they appear as they did back then, except Jacob, who appears as himself but occasionally flickers. A mysterious hot tub repairman (Chevy Chase) appears and warns them not to change anything as it might affect the machine. In order to minimize the butterfly effect, the guys plan to re-enact their relationship. Adam has to break up with his girlfriend Jenny (Lyndsy Fonsyse) and get get into the eye with a fork; Lou must pick a fight to get beaten up by Blaine (Sebastian Stan), a ski patrol bully; and Nick must …

Mask and ground truth

Exact match

Syntactically coherent

Syntactically wrong

Figure 11: A text infilling example of YAN.

Record · ID 19027 · SHA-256 b46186a9aeac896a
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.