What Does a Discrete Diffusion Model Learn?

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

What Does a Discrete Diffusion Model Learn?

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

What Does a Discrete Diffusion Model Learn? Rodrigo Casado Noguerales∗1 , Bernhard Schölkopf1,2,3 , Thomas Hofmann1 , and Aran Raoufi1 1

arXiv:2607.05381v1 [cs.LG] 6 Jul 2026

ETH Zurich Max Planck Institute for Intelligent Systems, Tübingen 3 ELLIS Institute Tübingen

July 7, 2026

Abstract What does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural network in the wrong coordinate changes the process being trained and sampled. Starting with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process, boundary terms included, we prove the Oracle Distance theorem: the negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one, not merely a bound. Its unique optimizer is therefore the conditional expectation of the true reverse jump rate given the current noisy state, and its irreducible cost is the rate at which the forward process Zt destroys information about the clean data Z0 , d − dt I(Z0 ; Zt ), so every noising process shares the same best achievable negative ELBO: the data entropy. For sequences with token-factorizing noise, the oracle projection yields three exact coordinates for the optimizer: denoiser, cavity (bridge plug-in), and score, with closed-form conversions among them. This framework identifies which law each loss in the literature actually optimizes, recovering MDM, UDM, SEDD, and GIDD as special cases; explains why denoiser and cavity coincide for masked diffusion but not for uniform diffusion; proves that a denoiser parameterization makes the uniform ELBO diverge at initialization while the bridge plug-in stays finite; and calibrates ELBO implementations exactly at initialization. Every identity is verified numerically, without approximation, on an exactly solvable model.

∗

Correspondence: [email protected]

Contents 1 Introduction

2 A general theory of discrete diffusion 4 2.1 The general theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Discrete diffusion for sequential data . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Background and related work

4 A continuous-time discrete diffusion framework 4.1 Variational formulation of diffusion models . . . . . . . . . . . . . . . . . . . . . . . 4.2 CTMC formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 The CTMC ELBO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Importance sampling, time clocks, and the empirical ELBO . . . . . . . . . . . . . .

19 19 21 24 29

5 What negative ELBO minimization really optimizes 5.1 ELBO as distance to the oracle reverse process . . . . . . . . . . . . . . . . . . . . . 5.2 The ELBO as a projection problem and the Pythagorean decomposition . . . . . . . 5.3 The oracle ELBO is the rate of information loss . . . . . . . . . . . . . . . . . . . . . 5.4 The universal NELBO floor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 An alternative proof of the Oracle Distance theorem . . . . . . . . . . . . . . . . . . 5.6 A Bregman-divergence and generator-matching view . . . . . . . . . . . . . . . . . .

31 31 33 35 36 37 39

6 Sequence modeling 6.1 Token-factorizable noising processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Three exact token-wise factorizations of the reverse rate . . . . . . . . . . . . . . . . 6.3 Parameterizations in the literature . . . . . . . . . . . . . . . . . . . . . . . . . . . .

41 41 42 45

7 Masked, uniform, and GIDD noising processes 7.1 Common CTMC form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Masked diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Uniform diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Generalized interpolating discrete diffusion . . . . . . . . . . . . . . . . . . . . . . .

49 49 50 52 56

8 Numerical verification

9 Conclusion

Introduction

Discrete diffusion models offer a natural route to generative modeling on language and other categorical data: corrupt a sequence directly on its alphabet, then learn to run the corruption backward [AJH+ 21, HNJ+ 21]. Continuous-time Markov chain (CTMC) formulations make this precise, replacing a chosen finite sampling grid by a genuine jump process, so that a generative model is determined by the reverse jump rates it induces [CBDB+ 22]. This paper starts from a simple observation. In the discrete-diffusion literature the learned reverse process is described through several apparently different objects (denoisers, score ratios, bridge plug-ins, leave-one-out predictors, masked-token losses, finite-step reverse kernels [LME24, SAS+ 24, vRFD+ 25]), often treated as alternative names for the same model. They are not. A neural network’s output is not a reverse process until one specifies how it is converted into jump rates, and reading the same output in the wrong coordinate changes the continuous-time Markov chain being optimized and sampled from. This distinction is not purely academic. In uniform diffusion, the standard bridge-plug-in objective does not train an ordinary denoiser that predicts the clean token from the full noisy sequence; it trains a cavity law, a predictor that uses the noisy context but does not condition on the local noisy token it is about to revise. Masked diffusion hides this because, at a masked position, the local observation carries no information about the clean token, so denoiser and cavity coincide. Uniform and hybrid diffusion have no such cancellation, and for them the conversion between coordinates is an analytic part of the model, not an implementation detail. The central question is therefore: what does negative-ELBO minimization actually optimize? Our answer is path-based. The continuous-time ELBO is not a loose surrogate for likelihood but an exact path-law divergence between two reverse jump processes, the oracle process determined by the forward noising law and the learned process induced by the model, an identity we call the Oracle Distance (Theorem 3). Training therefore makes the model’s entire reverse trajectory, not only its endpoint distribution, match the oracle’s. At the infinitesimal level, training observes the clean data and so has access to the cleanconditioned reverse bridge; sampling does not, and the learned rate may depend only on the noisy state currently visited. The ELBO therefore performs a projection: it averages the cleanconditioned bridge rate over the posterior information available at the noisy state, and the result is exactly the marginal reverse rate needed for generation, the jump-process analogue of the conditional-mean principle behind regression and flow matching. This projection also explains the value of the objective: once the fixed data-entropy floor and endpoint terms are accounted for, the irreducible part of what remains is exactly the information the forward process destroys, accumulated over diffusion time. Every noising process pays the same total oracle price, and they differ only in when and where they spend it, and in how the chosen parameterization conditions the resulting learning problem. For token-factorizing noising processes the projection yields an explicit dictionary: the same reverse rate can be written in denoiser, cavity, or score coordinates, each valid but only after the correct conversion. This resolves several puzzles at once: why masked diffusion reduces to ordinary denoising cross-entropy, why the bridge plug-in targets a cavity law in uniform diffusion, why a denoiser parameterization can be ill-conditioned near the clean endpoint, and how scorebased discrete diffusion fits into the same continuous-time object. The same lens then separates the two error sources of a trained model: the reverse-rate error that the ELBO measures, and the factorization error that only sampling reveals. 3

Contributions. This paper gives a self-contained, general theory of continuous-time discrete diffusion; §2 surveys the core results and insights and we recommend starting there, deferring the full derivations to §§4–7 and related work to §3. Concretely, our main contributions are: • A general CTMC ELBO. Building on [CBDB+ 22], we derive the continuous-time discretediffusion ELBO for finite-state jump processes, with its reconstruction and terminal-prior boundary terms, the support and regularity conditions under which it is well defined, and two complementary derivations (an infinitesimal-KL limit and a Girsanov argument). The result is a mechanical recipe for the training objective of any forward noising process. • The ELBO as path divergence: the Oracle Distance. After the fixed entropy and endpoint terms are separated, the negative ELBO is exactly the reverse-time path divergence from the oracle reverse process to the learned one (Theorem 3). The objective is therefore a distance between reverse dynamics, not merely a likelihood bound. • The ELBO as a reverse-rate projection. The population optimizer is the posterior average of the clean-conditioned reverse bridge rate, viewed from the information available at the current noisy state. This yields a Pythagorean decomposition into irreducible information loss and model mismatch, and identifies the universal oracle floor, the data entropy. • One reverse rate, three coordinates. For product CTMCs we express the same optimal reverse rate in denoiser, cavity, and score coordinates, with closed-form conversions among them. This identifies which law is optimized in masked, uniform, score-based, and hybrid discrete diffusion; in particular, the standard bridge plug-in is optimized by the cavity law, not the denoiser. • Practical consequences. The dictionary explains why masked diffusion admits ordinary denoising cross-entropy, why denoiser parameterizations are ill-conditioned for uniform and full-support hybrid diffusion, and why the bridge plug-in targets a cavity law; it calibrates ELBO implementations at initialization, including the boundary terms usually dropped in cross-method comparisons; and it separates a trained model’s reverse-rate error, which the ELBO measures, from the factorization error of parallel-token sampling, which the ELBO cannot see.

A general theory of discrete diffusion

This section is an exposition of the main results, conceptual intuitions, and practical insights of the paper; the full derivations live in §§4–7 for the interested reader.

2.1

The general theory

Diffusion models turn sampling from a complicated data distribution into the problem of reversing a simple noising process. One chooses a forward stochastic process (Zt )t∈[0,T ] whose marginals qt := Law(Zt ) interpolate between the data law q0 = pdata and a terminal law qT from which sampling is easy; generation then reduces to learning the reverse dynamics: sample ZT ∼ qT , run an approximate reverse process back to time zero, and hope that the resulting law is close to q0 . In discrete diffusion the noising process corrupts data directly on its finite alphabet, so the object to reverse is a discrete Markov evolution rather than a Gaussian perturbation in Euclidean space. 4

From VAEs to the diffusion ELBO. Historically, the training objective behind diffusion models comes from the variational autoencoder [KW14, RMW14]. For a latent-variable model pθ (x, z), with data x and latent z, and encoder law q(z | x), Jensen’s inequality gives θ pθ (x, z) p (x, z) θ ≥ Eq(z|x) log = Eq(z|x) [log pθ (x | z)] − KL q(· | x) pθz , log p (x) = log Eq(z|x) q(z | x) q(z | x) whose RHS is the usual variational bound: the evidence lower bound (ELBO), a reconstruction term minus a prior-matching term (§4.1). Diffusion models use the same variational principle, but the latent variable is no longer a single random variable: it is the whole noising trajectory. In discrete time, fix clean data z0 and let theQlatent variable be the Markov chain z1:N = (z1 , . . . , zN ), with forward law q1:N |0 (z1:N | z0 ) = N k=1 qk|k−1 (zk | zk−1 ). If the learned reverse θ model has terminal prior pN , final reconstruction kernel pθ0|1 (z0 | z1 ), and reverse kernels pθk−1|k , so Q θ that the full model law is pθ0:N (z0:N ) = pθN (zN ) N k=1 pk−1|k (zk−1 | zk ), the VAE bound becomes ELBON (θ; z0 ) = Eq1|0 [log pθ0|1 (z0 | z1 )] − KL qN |0 (· | z0 ) pθN −

N X

Eqk|0 KL qk−1|k,0 (· | zk , z0 ) pθk−1|k (· | zk ) .

k=2

In particular, the discrete-time ELBO already has the structure that will persist in continuous time: a reconstruction term at the left endpoint, a prior term at the right endpoint, and a sum of local discrepancies between the true reverse bridge kernels (the clean-conditioned kernels qk−1|k,0 (· | zk , z0 )) and learned reverse kernels. Proposition 1 (§4.1) gives the exact finite-chain statement used as the discrete skeleton for the CTMC derivation. Continuous time and the reverse jump rate. Passing from a discrete chain to a continuoustime Markov chain (CTMC) removes the arbitrary choice of a discretization level N , allows time points to be sampled continuously, and makes time changes or importance-sampling clocks part of the mathematical formulation rather than implementation details. It also exposes the infinitesimal object that the model is trying to learn: the reverse jump rate. This is useful both practically, because models can be trained from randomly sampled times and without fixing the sampling grid, and mathematically, because the ELBO becomes the path-space relative entropy and has an exact local expression in terms of jump rates. To derive this continuous-time ELBO, let us introduce first some notation. A CTMC process (Zt )t∈[0,T ] is characterized through its generator Qt , which governs the firstorder probabilities of jumping from the current state to another or staying: ( X h Qt (x, y) + o(h), y ̸= x, P(Zt+h = y | Zt = x) = Qt (x, x) := − Qt (x, y). 1 + h Qt (x, x) + o(h), y = x, y̸=x We write qt = Law(Zt ) and qt|0 (· | z0 ) = Law(Zt | Z0 = z0 ). The clean-conditioned reverse rate and the marginal reverse rate are q (y | z0 ) b t (z, y | z0 ) = Qt (y, z) t|0 Q , qt|0 (z | z0 )

b t (z, y) = Qt (y, z) qt (y) , Q qt (z) 5

y ̸= z.

(1)

In diffusion models, the object ultimately learned is a reverse process. A neural network may be read as a denoiser, a score ratio, or a bridge plug-in, but none of these is a generative process b θt (zt , y): functions of the noisy state until it is converted into reverse jump rates, which we denote Q zt , never of the hidden clean data z0 . The paper later compares these parameterizations in detail, bθ . but at the level of the ELBO they all enter through the rates Q t The continuous-time ELBO.

For scalars a, b ≥ 0, define Φ(a, b) := a log

a − a + b, b

(2)

with the usual conventions at the boundary. This local rate divergence is the Bregman divergence of the negative-entropy potential F (u) = u log u − u, and it describes the KL rate between two CTMC processes. In this paper we prove that, under the support and regularity assumptions of Theorem 1 and up to the two endpoint comparison terms, the continuous-time ELBO on [0, T ] has the path-integral form   Z T X b t (zt , y | z0 ), Q b θt (zt , y)  dt. Eqt|0 (zt |z0 )  log pθ0 (z0 ) ≥ ELBO[0,T ] (θ; z0 ) = − Φ Q 0

y̸=zt

CTMC path-space ELBOs of this kind have appeared in the continuous-time discrete diffusion literature, for example in the formulations of [CBDB+ 22, LME24, SHW+ 24, SAS+ 24, vRFD+ 25]. Our derivation is included not merely to recover the known training objective, but to keep the terms in a form suitable for studying the objective itself; Theorem 1 (§4.3) is the precise finite-window statement. There are two reasons to revisit the derivation. First, many presentations discard modelindependent terms of the ELBO because they do not affect optimization. This is natural when the goal is only to obtain a loss function, but it hides the information-theoretic structure of the objective. In this paper we keep these terms and show that they are not decorative: they identify the irreducible cost of the ELBO with the information destroyed by the forward process, and they lead to a path-space decomposition of what the reverse model is actually learning. Second, practical implementations rarely optimize the ideal full-interval objective without modification. Endpoint behavior at t = 0, T can be singular or numerically unstable, posterior ratios may have high variance, reconstruction and terminal-prior choices may be separated from the path loss, and Monte Carlo training often uses time windows or importance-sampling clocks. For these reasons one often truncates the interval to [t1 , t2 ] ⊂ (0, T ). Once this is done, however, the effect on the ELBO is no longer transparent: the boundary terms that were harmless in the full-interval or perfectly matched setting become essential for interpreting the objective as a likelihood bound. In this paper we prove that, under the light conditions of Theorem 1, the finite-window CTMC likelihood admits the bound h i ELBO[t1 ,t2 ] (θ; z0 ) := Eqt |0 (zt1 |z0 ) log pθ0|t1 (z0 | zt1 ) − KL qt2 |0 (· | z0 ) pθt2 1   Z t2 (ELBO) X b t (zt , y | z0 ), Q b θ (zt , y)  dt, − Eqt|0 (zt |z0 )  Φ Q t t1

y̸=zt

where pθ0|t1 is the reconstruction decoder and pθt2 is the terminal prior. In implementations, the same neural model that specifies the reverse rates often also provides this reconstruction head. bθ This form of the ELBO is particularly useful because its inputs are the model reverse rate Q b | z0 ). The latter is easy to compute from the forward and the clean-conditioned reverse rate Q(· b process itself: Qt (z, y | z0 ) = Qt (y, z) qt|0 (y | z0 )/qt|0 (z | z0 ) requires only the conditional forward kernels qt|0 and no knowledge of the data law pdata beyond samples z0 . The object we ultimately b t (z, y) = Qt (y, z)qt (y)/qt (z), because only the reverse want, however, is the marginal reverse rate Q CTMC with these rates transports qT back to q0 = pdata . The difficulties interpreting the ELBO. The perceived drawback of the ELBO is that it is an inequality: it seemingly replaces the intractable likelihood by a computable bound, as reflected in its name. Indeed, taking expectation in (ELBO) with respect to Z0 ∼ q0 = pdata and subtracting the data entropy H(q0 ) = H(pdata ) from both sides gives a KL upper bound against pθ0 , the model’s law at time zero, that is, the law of its generated samples: KL pdata pθ0 ≤ −H(q0 ) − Eq0 ELBO[t1 ,t2 ] (θ; z0 ) . A metric which is a bound is hard to interpret. If two papers report different ELBO values, and the reported number is only an upper bound on a KL divergence with an unknown gap, it is not obvious whether a better value reflects a better generative law or only a tighter variational approximation. This is especially acute in domains such as text and music, where direct metrics on generated samples are difficult to define. bθ There are also two structural questions specific to diffusion models. First, the ELBO trains Q b t (· | z0 ), Q b θt ), where the first argument is the clean-conditioned through the local divergence Φ(Q b t . What, then, is reverse rate, whereas generation requires the marginal reverse CTMC with rate Q the population optimizer of the objective? Second, what is the best achievable value of the ELBO itself? In this paper, we answer these questions by identifying the ELBO with a path-space relative entropy, plus constants with direct information-theoretic meaning. The Oracle Distance: NELBO = entropy + path KL to the oracle. The answer compares whole trajectories rather than time marginals. An endpoint law and a family of jump rates determine the law of the entire trajectory, its path law (§4.2), and two path laws matter here: ⋆ P[0,T ] , the law of the noising trajectories themselves, read in reverse time as started at qT and run b t ; and P θ , the model’s, started at its terminal prior backward with the marginal reverse rates Q [0,T ]

b θ . Indeed, direction is a property of the chosen factorization, an endpoint and run backward with Q t law and rates, not of the law itself: a process Markov in forward time is Markov in reverse time, with different rates, and P ⋆ simply is the noising law. In the clean full-interval setting, where the model starts at the correct terminal law qT and the endpoint terms vanish, we prove in §§4–5 that ⋆ θ −Eq0 ELBO[0,T ] (θ; z0 ) = H(q0 ) + KL P[0,T (3) ] P[0,T ] . The significance is that the ELBO is not just an arbitrary tractable bound: after adding the entropy constant, it is exactly the path-space KL between the true and the learned reverse dynamics. We call this expression of the ELBO the Oracle Distance identity (Corollary 1, §5.1). In particular, ⋆ θ KL pdata pθ0 ≤ KL P[0,T ] P[0,T ] = −H(q0 ) − Eq0 ELBO[0,T ] (θ; z0 ) , 7

because marginalization cannot increase relative entropy. This explains why optimizing the ELBO is powerful: it makes the learned reverse path law follow the true reverse path law. It also shows the limitation of the metric: the path-law KL can be substantially larger than the endpoint KL between generated samples and data, so the ELBO may penalize pathwise discrepancies that are not fully visible at the final sample level (we expand on this point later in §2.2). The same identity has a clipped-window version (Theorem 3, §5.1): training on [t1 , t2 ] ⊂ (0, T ) with a model terminal prior pθt2 and a model final decoder pθ0|t1 gives −Eq0 ELBO[t1 ,t2 ] (θ; z0 ) = H(q0 ) + KL P[t⋆1 ,t2 ] P[tθ1 ,t2 ] i h + Eqt1 KL q0|t1 (· | zt1 ) pθ0|t1 (· | zt1 ) .

(4)

The reconstruction mismatch at t1 sits outside of the path KL, while the usual terminal-prior θ mismatch KL qt2 pt2 sits inside it: by the finite-state Girsanov formula (Theorem 2, §4.3), a KL between path laws always contains the KL between their laws at the starting time, here t2 , plus an integral of local rate divergences, hP Z t2 i ⋆ θ θ b t (zt , y), Q b θt (zt , y) dt. KL P[t1 ,t2 ] P[t1 ,t2 ] = KL qt2 pt2 + Eqt Φ Q y̸=zt t1

In this sense, the theorem reveals what the CTMC ELBO actually is. The optimizer is a reverse-rate projection. The optimizer is now an almost trivial corollary: the path KL in (3) or the clipped-window version (4) is non-negative and vanishes exactly when the model runs the true marginal reverse rates, so among all rates measurable with respect to the b t (zt , y): noisy state Zt only, the optimum is exactly the true reverse marginal rate Q h i b t (Zt , y | Z0 ) Zt = zt = Q b θ⋆ (zt , y) = E Q b t (zt , y), Q y ̸= zt . (5) t

Qcθt (zt , y)

10 0 10 −1 10 −2

y=x 10 −2 10 −1 10 0 Qct (zt , y) = q(z0 | zt ) [Qct (zt , y | z0 )]

b θ (zt , y) from a trained uniform-diffusion denoiser neural network, Figure 1: The model rates Q t h i b t (zt , y) = Eq(z |z ) Q b t (zt , y | z0 ) . sampled across many (zt , y, t), average to the oracle rate Q 0 t 8

The middle expression, which equals the marginal rate by a short Bayes computation (Theorem 5, §5.2), is what gives the optimum its meaning: although the training target is clean-conditioned, its projection onto the information available to the model is exactly the unconditional reverse rate needed for generation (Figure 1). The irreducible cost is information loss. The entropy constant in (3) is not mere bookbt , keeping; it has a precise information-theoretic origin. Evaluated at the oracle reverse rate Q the per-time ELBO cost that remains is exactly the rate at which the forward process destroys information about the data,   X b t (Zt , y | Z0 ), Q b t (Zt , y)  = d H(Z0 | Zt ) = − d I(Z0 ; Zt ) ≥ 0, Φ Q Jt⋆ := EZ0 ,Zt  dt dt y̸=Zt

the CTMC counterpart of the I-MMSE relation between estimation and information (Theorem 4, §5.1). Integrated over the schedule this telescopes to the total information the data carries, the data entropy H(Z0 ) = H(pdata ) (Figure 2). A Pythagorean split. The two previous facts combine into a proof of the Oracle Distance (3). For any model rate, the per-time trained cost separates exactly, with no cross term, into what the oracle pays plus how far the model’s rates are from the oracle’s (a Bregman property of Φ;

(a) Same total 2.5

H(q0 )/L = 1.25

1.2

uniform

= dtd H(Z0 | Zt )

2.0

H(Z0 | Zt )/L

1.0

masked

1.5

0.8 uniform

0.6

0.5

1.0 total area

masked

0.4

0.0

clean

0.2

0.4

0.6

diffusion time t

0.8

1.0

noisy

0.0

H(q0 )/L

Z 1

0.5

0.2 0.0

1.0

GIDD λ

1.4

(b) Opposite schedule

J t dt

GIDD λ

0.0

clean

0.2

0.4

0.6

diffusion time t

0.8

1.0

0.0

noisy

Figure 2: Every noising process destroys exactly H(q0 ) information, at its own rate, computed at the exact oracle on the GIDD family (§8) that interpolates masked and uniform diffusion through the parameter λ. (a) Information destroyed by time t, H(Z0 | Zt )/L, follows different paths to the d common ceiling H(q0 )/L (Theorem 4, §5.1). (b) Its rate Jt⋆ = dt H(Z0 | Zt ): uniform destroys information earlier than masked, with equal area (inset).

Proposition 2, §5.1): hP h i i b t (Zt , y | Z0 ), Q b θ (Zt , y) = J ⋆ + EZt P b t (Zt , y), Q b θ (Zt , y) . EZ0 ,Zt Φ Q Φ Q t t t y̸=Zt y̸=Zt |{z} {z } oracle {z } | | cost trained cost

model–oracle rate divergence

Integrating over time, the oracle cost telescopes to the data entropy H(q0 ) by the information-loss identity above, while the model–oracle divergence accumulates into the path KL: this is the identity. A universal entropy floor. The same result also identifies the intrinsic floor of the ELBO. Since the path-law KL in (3) is non-negative and vanishes exactly at the true marginal reverse dynamics, for a model class expressive enough to realize them, we get inf −Eq0 ELBO[0,T ] (θ; z0 ) = H(pdata ).

(6)

10 0

empirical theory

1.0

10 −1

0.5

10 −2

10 −2 0

2000

4000

6000

8000

training step

2000

4000

6000

8000

GIDD λ (masked → uniform)

denoiser cavity score

(b) GIDD family

10 0

NELBO − H(q0 )

(a) excess = path KL

0.0

training step

path density J t

floor J t model J θt

2 1 0

0.0 clean

0.2

0.4

diffusion time t

0.6

0.8

1.0 noisy

Figure 3: The NELBO is an exact distance to the oracle (Corollary 1, §5.1). Panels (a,b) plot the per-token excess NELBO(θ) − H(q0 ): it vanishes over training for every parameterization (panel (a); UDM denoiser, cavity, and score networks from §2.2) and every noising process (panel (b); GIDD family with cavity network). (c) The Pythagorean split over diffusion time t: the per-time cost is the oracle rate Jt⋆ plus the model–oracle divergence, whose integral is the path KL. 10

Thus the best achievable value is not an artifact of the particular noising process or parameterization; it is the data entropy (Figure 3). Lemma 3 and Corollary 3 (§5.4) give the formal statements, and Figure 7 shows the boundary-term decomposition summing to this floor. The ELBO has a universal entropy floor, and the excess above that floor measures pathwise mismatch to the true reverse process. Training clocks and the information-uniform time. In practice the path integral in (ELBO) is estimated by Monte Carlo over t and zt ∼ qt|0 (· | z0 ), optionally with an importance density pIS (t) for variance reduction; choosing pIS is the same as choosing the clock on which the reverse chain is run (§4.4). Beyond the variance-optimal choice, one clock is intrinsic to the process: since the d oracle cost rate is Jt⋆ = dt H(Z0 | Zt ) (Theorem 4), the clock τ (t) = H(Z0 | Zt )/H(q0 ) drains information linearly, I(Z0 ; Zτ ) = (1 − τ ) H(q0 ), and straightens the otherwise process-dependent curves of Figure 2a onto a single diagonal.

2.2

Discrete diffusion for sequential data

Discrete diffusion is most often applied to sequence modeling, broadly any multi-dimensional data such as text, images, music, or graphs. However, a practitioner immediately meets three apparently different ways to parameterize the model: predict the clean token directly (a denoiser ); the bridge plug-in trick, widely assumed to be an equivalent way of training that same denoiser (it is not); or predict a ratio of noisy conditionals (a concrete score). Folklore holds that some of these mysteriously misbehave, and indeed the uniform-diffusion ELBO can diverge at initialization. This section resolves the confusion with the central guiding principle we just introduced in the previous section: all three are coordinates of one object, the true reverse rate that the ELBO optimizes. Choosing a parameterization is just choosing which coordinates of that object to learn. Using this one fact we give an answer to several open practical questions: what is the dictionary that converts between the three parameterizations, why a denoiser model fed to a cavity sampler silently trains a different law under uniform but not masked diffusion, what it would take to improve on masked diffusion, and it also provides calibration checks that run at initialization, among others. Sequences as product spaces. Sequential data lives on a product space (§6), whose (high-order) positional correlations are exactly the structure generative models aim to capture. We index the L positions by 1 ≤ i ≤ L and work on the finite product state space X := X1 × · · · × XL . To exploit this structure, the standard choice is a noising process that factors across positions (Theorem 6, §6.1): each position is corrupted independently, so for times s < t the forward kernel is L Y i qt|s (y | x) := qt|s (y i | xi ), x, y ∈ X , i=1 i the forward where xi is the i-th coordinate, x−i := (xj )j̸=i the remaining coordinates, and qt|s kernel on Xi . Equivalently, by the Kolmogorov forward equation, the generator factors additively,

Qt (x, y) =

L X

(Qt )i (x, y),

(Qt )i (x, y) = 1{x−i = y −i } Rti (xi , y i ),

i=1

(7)

so jumps occur one coordinate at a time, at a rate Rti . Nothing forces the per-position processes to be equal, though they usually are. This factorization into token-wise jump rates is what makes both training and the standard marginal-based sampling tractable, as we now show. Like the left-to-right factorization of autoregressive models, this choice trades a (hopefully mild) loss of expressivity for tractability. From sequence rates to token rates. The payoff of factorization is that the whole reverse process is fixed by token marginals alone, never full-sequence laws. To see it, we compute the true reverse rate of Theorem 5 for the clean-conditioned positional rates, averaging over q(z0 | zt ), and cancelling the non-i factors, to get " i # h i qt|0 (y i | z0i ) i i i b t )i (zt , y) = E (Q b t )i (Zt , y | Z0 ) Zt = zt = R (y , z ) E i (Q . (8) t t q(z0 |zt ) i (z i | z i ) qt|0 t 0 The reverse rates therefore depend on the data only through the clean-token marginals q(z0i | zt ), which is our first key insight for sequence modeling and is formalized as the denoiser coordinate in Theorem 7 (§6). The literature typically posits a token-level parameterization on the strength of factorizability; deriving it through the projection result instead shows what that factorization buys (the dependence on data through those clean-token marginals alone) and thereby which token-level choices affect the optimized law and which are immaterial. Existing parameterizations and open questions. The marginal q(z0i | zt ) is the familiar denoiser, which we denote by πi⋆ : the i-th clean-token law given the current noisy sequence, πi⋆ (z0i | zt , t) := q(z0i | zt ). The standard recipe, known as the mean parameterization, fits a neural network head πiθ ≈ πi⋆ , b θt , and feeds those to the ELBO of Theorem 1; it is commonly plugs it into (8) to obtain rates Q read as the “average reverse rate” under the denoiser’s uncertainty. Reverse sampling then factors the step through the same head, pθs|t (zs | zt ) =

L Y

pθ (zsi | zt ),

h i i pθ (zsi | zt ) = Eπθ (z i |zt ,t) qs|t,0 (zsi | zti , z0i ) , i

(9)

i=1

even though the true law qs|t cannot factor over positions: if it did, the clean-data law q(z0 ) = q(z0 | zT ) (by terminal mixing, Assumption 1) would factor too, giving a pure unigram model. This recipe works for masked diffusion but fails for uniform diffusion, where the ELBO diverges at initialization. Uniform-diffusion (UDM) and generalized-interpolating (GIDD) works instead i is averaged over the adopt the bridge plug-in parameterization, in which the forward kernel qt|0 network’s output distribution before forming the rate (1), and surprisingly this keeps the ELBO finite and allows smooth training. A third route, that of concrete scores, instead reads the head as ratios of noisy conditional laws. These options raise the central questions we answer next: how are the three related, are they interchangeable, why does one of them sometimes fail, and what reverse CTMC does each actually induce?

Three coordinates, one object. The answer lies in the ELBO’s unique optimizer, the unconditional reverse rate of Theorem 5, because we can rewrite (8) in terms of true laws other than the denoiser. The bridge plug-in turns out to target the cavity law µ⋆i , namely the i-th clean-token law given the context zt−i but ignoring its own noisy observation zti , µ⋆i (z0i | zt−i , t) := q(z0i | zt−i ), while the concrete score s⋆i targets a ratio of noisy conditional laws, s⋆i (zti , y i | zt−i , t) :=

qt (y i , zt−i ) , qt (zt )

where (y i , zt−i ) denotes the sequence zt with its i-th token replaced by y i . Our second contribution makes this picture exact by identifying three exact and equivalent expressions for the token-wise b t )i , which together form a coordinate dictionary, true reverse rates (Q " i # qt|0 (y i | ·) i i i b t )i (zt , y) = Rt (y , zt ) Eπ⋆ (·|z ,t) (Q (48), denoiser t i (z i | ·) i qt|0 t h i i (y i | ·) Eµ⋆ (·|z −i ,t) qt|0 i (49), cavity = Rti (y i , zti ) i t h i (z i | ·) Eµ⋆ (·|z −i ,t) qt|0 t t

= Rti (y i , zti ) s⋆i (zti , y i | zt−i , t)

(50), score.

Denoiser πi⋆ (z0i | zt ) := q(z0i | zt ) MDLM, D3PM, MD4

compose via cavity

local Bayes update i (z i | ·) πi⋆ ∝ µ⋆i qt|0 t

True reverse rate b t )i (zt , y) (Q h

b t )i (Zt , y | Z0 ) Zt = zt = E (Q

Cavity ⋆ µi (z0i | zt−i ) := q(z0i | zt−i ) UDLM, GIDD, Duo

ratio of cavity-averaged kernels h i i (y i | z i ) Ezi ∼µ⋆ qt|0 0 0 i h i s⋆i = i (z i | z i ) Ezi ∼µ⋆ qt|0 t 0 0

i linear in µ⋆i : invert qt|0 and normalize to go back

Score s⋆i (zti , y i | zt−i ) qt (y i , zt−i ) := qt (zt ) SEDD, RADD, TCSM

Figure 4: The reverse rate in three coordinates: each vertex lists a coordinate of the reverse rate, the law it learns (omitting the time conditioning from the notation), and the methods it recovers; each arrow names its conversion formula, with arguments suppressed (Proposition 3, §6.1). 13

i (y i | ·)/q i (z i | ·) over the To read these off: the denoiser line averages the forward-kernel ratio qt|0 t|0 h t i i ⋆ i (· | z i ) ; and clean token z0 ∼ πi ; the cavity line takes a ratio of the cavity-averaged kernels Eµ⋆i qt|0 0 the score is already that reweighting and enters raw. All three agree at the oracle laws, which is exactly the dictionary. But averaging the ratio (denoiser) and taking the ratio of averages (cavity) are different operations on a learned head, which is why the denoiser/cavity distinction bites for uniform yet vanishes for masked diffusion (next paragraph). Fitting a head to each yields a different parameterization, and each is a complete specification of the generative process, since substituting a coordinate into the rate fixes the reverse jump rates and, by the Kolmogorov forward equation, those rates in turn determine the reverse CTMC in full. Figure 4 gathers the three coordinates: the oracle law each one learns, the literature methods it recovers, and the closed-form conversions between them. Because all three coordinates target the same oracle object, namely the unique optimum of the

1.25

den.

2.02

7.24

cav.

1.25

head

cav.

2.42

1.25

6.41 2

score

1.25

score

2.62

1.33

1.25 0

(d) correct conversion

(e) naive misread

3.51

3.51 3.51

den.

14.77

34.90

8.44

3.51

score

9.94

3.51 0

den.

cav. score read as

den.

cav. score read as

0.0

0.5

GIDD λ (f) zero at masked

1.0

33.39

4.60

0.2

score

0.5

0.0

cav.

0.4

0.0

cav. score read as

gen-PPL penalty

cav.

den.

head

den.

cav. score read as

|gen-PPL−true|

den.

1.0

8 6

0.5

4 2 0.0

0 0.0

0.5

GIDD λ

GIDD λ (masked → uniform)

1.25

0.6

NELBO penalty

(b) naive misread NELBO−H(q0 )/L

den.

(a) correct conversion

GIDD λ (masked → uniform)

1.0

Figure 5: Reading the exact reverse rate held in coordinate A (row) as coordinate B (column). Top, per-token NELBO: (a) with the closed-form conversion (Figure 4, Table 2) every cell equals the floor H(q0 )/L; (b) the naive misread penalizes every off-diagonal. Bottom, generative perplexity (ancestral sampling, exact oracle laws): (d) every correct conversion reaches oracle performance; (e) off-diagonals penalized. (c,f ) Sweeping the GIDD weight λ, the denoiser/cavity penalty grows continuously from zero at masked, where the two coordinates coincide. 14

ELBO, each head πiθ , µθi , sθi is optimized exactly at πi⋆ , µ⋆i , s⋆i , uniquely up to identifiability of the head-to-rate map. Why masked diffusion is special. The link between πi⋆ and µ⋆i is simple: the denoiser is the i (z i | z i ), cavity reweighted by the local forward kernel qt|0 t 0 i (zti | z0i ). πi⋆ (z0i | zt , t) ∝ µ⋆i (z0i | zt−i , t) qt|0 i (z i | z i ) is exactly the Bayes update for observing the local token z i on Thus, the forward kernel qt|0 t t 0

top of the context zt−i . This splits the mutual information between z0i and zt into two parts: what the context zt−i reveals (the cavity µ⋆i , which must be learned), and what the local token zti adds (the forward kernel weight, available analytically). A denoiser head must reproduce both of these, which is a concrete source of its training difficulty. Masked diffusion escapes the distinction entirely, i (m | ·) constant, so because at a masked token the local observation is uninformative and thus qt|0 that πi⋆ = µ⋆i , that is, the denoiser and cavity coincide in masked diffusion (Corollary 4, §7). We show the importance of this distinction in Figure 5. Two sources of modeling bias. The framework cleanly separates the two sources of error in a discrete-diffusion model. The first is reverse-rate error, the gap between the model rate and the true reverse jump rate, which is exactly what the ELBO measures, by the Oracle Distance Theorem 3. The second is sampling factorization error and is invisible to the ELBO. The trajectory drift noted above is not a third source: it is the same reverse-rate error, incurred now at the model’s own self-generated states rather than at the true (teacher-forced) states the ELBO scores; in the wellspecified, sufficient-capacity limit the learned rate is correct at every state and no drift occurs, so drift is a generalization gap rather than an objective mismatch (§9). Factorization error, by 40

masked uniform GIDD

gen. perplexity

20 15 10 8 6 5 4 3.5

true PPL 10 0

10 1

10 2

sampling steps (NFE) Figure 6: Sampling with the exact oracle denoiser and the factorized ancestral samplers of §7 zeroes the reverse-rate error, so the excess of the generative perplexity is pure sampling error, which even the oracle rates incur. It vanishes only as the sampling budget grows, masked being penalized more due to its inability to self-correct. 15

path (t2 =0.6) path (t2 =0.8) path (t2 =0.95) boundary prediction log V

3 2 1 0

16 32 64

vocabulary V

1.50

6 4 masked

recon.

path

prior

H(q0 )/L

1.25

2 2

slope (V−1)/V V = 32 V = 64

V=4 V=8 V = 16

path NELBO

per-token NELBO

(b) Denoiser 1/V diverges

nats / token

(a) Cavity 1/V: NELBO = log V

1.00 0.75 0.50 0.25 0.00

masked

uniform

GIDD

log(1/βt1 )

Figure 7: (a) A cavity head µθi ≡ 1/V has per-token NELBO log V once boundary terms are kept (Proposition 4). (b) The same head read as a denoiser makes the uniform NELBO diverge, linearly in log(1/βt1 ) with slope V V−1 (Proposition 6), while masked stays finite. (c) The oracle NELBO splits as reconstruction + path + terminal prior, summing to the same H(q0 ) for any noising process (Corollary 3; the window [0.1, 0.9] exaggerates the boundary terms). contrast, persists even when the reverse-rate error is exactly zero. The literature uses various small-time sampling schemes (Euler, τ -leaping, Tweedie), but rarely makes the sequence-to-token derivation explicit, even though the sampling gap is exactly the factorization bias. All such schemes b t )i (zt , y) can be treated as rest on the same fact, namely that over a short step [s, t] the rates (Q constant up to O(t − s) terms, and under that approximation the unconditional reverse rate factors exactly, so the factorized ancestral sampler is justified. Figure 6 isolates this factorization error: even an oracle with zero reverse-rate error degrades at few-step sampling. Calibration at initialization. The framework also yields calibration identities at initialization, for an uninformative head 1/V . First, a cavity 1/V head has per-token NELBO log V (Proposition 4, §6), which provides a useful debugging tool, whereas a denoiser 1/V head instead makes the uniform-diffusion NELBO diverge (Proposition 6, §7); both identities are confirmed numerically in Figure 7. This divergence also pinpoints when denoisers are hard to train: whenever placing mass i (y i | z i )/q i (z i | z i ), on the “wrong” token near the clean endpoint t1 creates singular quotients qt|0 0 0 t|0 t which is precisely UDM’s failure mode. Practical takeaways.

The framework reduces to a few concrete rules for the practitioner.

• Match the neural network head during sampling to its training loss, or convert. A head is optimized by the law its loss targets; feed a cavity head into a denoiser sampler and one is missing the analytic Bayes update (42). • Avoid a raw denoiser head for uniform and GIDD. Its ELBO diverges at initialization (Proposition 6); use the cavity (bridge plug-in) coordinate, which stays finite and well-calibrated. 16

• Use the log V identity as a calibration and debugging tool. A cavity 1/V head has per-token NELBO log V independently of the diffusion process and the training endpoints [t1 , t2 ], after including the boundary terms. • Keep the boundary terms. The reconstruction and terminal-prior KL terms are required for ELBOs to be comparable across processes; dropping them, as is common, silently breaks such comparisons. Instantiations: masked, uniform, GIDD. Section §7 instantiates the framework on the three standard processes. We do so by showing that GIDD’s ELBO is really a generic cavity ELBO, valid for any source-independent rate, with a closed-form integrand given by a KL plus an Itakura–Saito term (Proposition 5). Specializing it recovers the masked, uniform, and GIDD integrands, and the score coordinate likewise re-derives the results of SEDD and RADD in a few steps; the cavity optimality of the UDM bridge plug-in observed by [GJS+ 26] follows as one special case (see §3). Table 5 collects the resulting formulas, including the prior boundary terms that are usually dropped in the literature but are needed to compare ELBOs across methods.

Background and related work

Discrete diffusion and its continuous-time form. Discrete diffusion originates with multinomial and structured categorical corruption processes [HNJ+ 21, AJH+ 21], the discrete analogue of the Gaussian diffusion line [SDWMG15, HJA20], and now scales to large language models [NZY+ 25]. Campbell et al. [CBDB+ 22] cast discrete diffusion as a continuous-time Markov chain (CTMC) with exact forward and reverse rates and a continuous-time variational bound, later extended by score-based [SYD+ 23, LME24] and flow-based [GRS+ 24, CYB+ 24] formulations, and generalized to broad classes of Markov processes by denoising Markov models [BSDB+ 24] and Generator Matching [HHY+ 25]. Section §4 is a self-contained re-derivation of the CTMC ELBO building on [CBDB+ 22], with the reconstruction and terminal-prior boundary terms and the rate-support and regularity conditions made explicit, and with two derivations (an infinitesimal-KL argument and a Girsanov argument, Theorem 2). Existing discrete formulations absorb the boundary terms into model-independent constants or send them to zero with T → ∞ [CBDB+ 22, LME24, SAS+ 24, JSJN25], which makes reported ELBOs incomparable across noising processes as implemented; we keep them explicit throughout, which is what enables the cross-process calibration results of §6. To our knowledge, no discrete-diffusion work states the exact identity, with boundary terms, for arbitrary model rates. Other insights from our framework, such as the equivalence between importance sampling and the process’ time parameterization, connect to likelihood-faithful versus reweighted diffusion objectives [KSPH21, ND21, ST25, RNB+ 26]. Reverse-process parameterizations. The learned reverse process is parameterized in several ways: as a denoiser [AJH+ 21, HNJ+ 21, CBDB+ 22, SAS+ 24, SHW+ 24], as a concrete score [MCSE22, SYD+ 23, LME24, ONX+ 25, ZZZ+ 25], or as a leave-one-out or cavity law [vRFD+ 25, GJS+ 26, SSP+ 25, SDG+ 25, SLY+ 26], with the masked ELBO reducing to weighted cross-entropy [SHW+ 24, SAS+ 24, ZYYK24]. Two recent frameworks unify subsets of these: Generator Matching [HHY+ 25] regresses a marginal Markov-process generator, and Target Concrete Score Matching [ZZZ+ 25] unifies training objectives through the target concrete score, recovering several existing masked

and score-based objectives as special cases. Generator Matching in particular already contains, at the level of matching objectives, two ingredients we use: the marginal generator as a posterior average of conditional generators, and Bregman divergences as the loss class whose conditional and marginal forms share gradients (their Propositions 1–2). What it does not provide that we do is the likelihood side: the exact CTMC NELBO, boundary terms included, is such a Bregman objective for Φ, so the projection optimum is the maximum-ELBO reverse process, the excess above the oracle is a path KL, and the irreducible part is the information-destruction rate. Section §6 is complementary at the parameterization level: rather than a single objective, we give an explicit coordinate dictionary of the reverse rate for all denoiser, cavity, and score parameterizations, with exact conversion formulas, and identify which coordinate each literature loss optimizes. Prior work often blurs the position- versus sequence-level distinction and reads one network in another’s sampler without converting, the score line [LME24, ONX+ 25] and the independent uniform-diffusion analysis of [GJS+ 26] being the closest exceptions. The denoiser-versus-leave-one-out distinction. Concurrently and independently, Gourevitch et al. [GJS+ 26] have recently observed for the special case of uniform diffusion (UDM) that the bridge plug-in is optimized by a leave-one-out (cavity) law rather than the standard denoiser, deriving denoiser/leave-one-out/score conversions for UDM that they use to improve UDM training and sampling. Our work obtains this result by a different route that places it within a general product-CTMC treatment. The Oracle Distance Theorem 3 and the reverse-rate projection it induces characterize the denoiser, cavity, and score optima, and their conversion formulas, for every token-factorizable noising process from a single projection principle. Our framework also explains exactly when and why the coordinates differ and when a denoiser parameterization makes the ELBO ill-posed, with explicit divergence rates for UDM (Proposition 6) that give a theoretical account of the leave-one-out advantage that Gourevitch et al. document empirically. Furthermore, their empirical gains on text modelling provide strong evidence that distinctions such as the exact reverse-rate process and oracle law being optimized are far from purely academic and have a measurable effect on discrete diffusion modeling. d Information-theoretic view. The irreducible ELBO cost we identify with dt H(Z0 | Zt ) = d − dt I(Z0 ; Zt ) is the CTMC instance of estimation–information identities: the Gaussian I-MMSE relation [GSV05], its jump-rate analogue whose natural loss is exactly our jump divergence Φ [AW12], the information-theoretic reading of the continuous diffusion ELBO [KBVS23], and the discrete IMDSE/I-MDCE relations of [JSJN25], which they state at the oracle in the score coordinate for general rate matrices and in the cross-entropy coordinate for masked diffusion, with T → ∞. Our identities extend the latter along three axes: from the oracle to arbitrary model rates (the excess being exactly a path KL), to every coordinate of any token-factorizable process, and to finite windows with explicit boundary terms. The conditional-mean optimality and Pythagorean decomposition are special cases of [BGW05], the jump-rate counterpart of the denoiser/cavity/score identities in continuous diffusion [Vin11]. Because squared error is itself the Bregman divergence of F (u) = 12 ∥u∥2 and Φ that of F (u) = u log u − u, these identities are not merely parallel: the Gaussian I-MMSE and our CTMC case are two instances of one Bregman representation of the information loss.

A continuous-time discrete diffusion framework

The proofs in this and the following sections follow one recurring pattern: identify the exact reverse Markov process, then ask what part of it a model can represent. The variational step to deduce the ELBO bound is standard after realizing that a diffusion model is a VAE whose latent is the noising trajectory; in continuous time the discrete sum of bridge-kernel KLs becomes a path b t (zt , y | z0 ), Q b θ (zt , y)), plus a reconstruction term at integral of local jump-rate divergences Φ(Q t the lower endpoint and a terminal-prior term at the upper endpoint. This is the CTMC ELBO of Section §4.3, obtained either as a vanishing-mesh limit of the discrete-time ELBO or directly as the relative entropy between jump processes using Girsanov’s formula. The true bridge rate still depends on the hidden clean data Z0 , whereas the learned reverse CTMC sees only the noisy state Zt . The ELBO therefore poses a local projection problem, whose minimizer among Zt -measurable rates is the posterior mean of the clean-conditioned rate: the marginal time-reversal of the forward process. Its oracle per-time ELBO cost is exactly the inford mation the forward process destroys, dt H(Z0 | Zt ), and the Bregman–Pythagoras identity for Φ splits any model’s path cost into this irreducible term plus its mismatch to the oracle rate. With the two boundary terms, this yields the main result of Section §5: after subtracting the fixed data entropy H(q0 ) and up to the two endpoint terms, the negative ELBO (NELBO) is exactly the path-law KL from the oracle reverse CTMC to the learned one. The final sequence-modeling sections §§6–7 specialize this identity. For product noising processes, the forward generator changes one token at a time, so the oracle reverse rate decomposes into one-token jumps, expressible in three coordinates: a denoiser, a cavity law, or a concrete score.

4.1

Variational formulation of diffusion models

We begin with the variational identity underlying VAEs. Let pθ (z0 , z1 ) be a generative model for a finite data distribution z0 ∼ q0 on X and a latent variable z1 ∈ Z. Let q(z1 | z0 ) be an encoder satisfying q(z1 | z0 ) ≪ pθ (z0 , ·) for every z0 . The standard ELBO derivation starts from the marginal log-likelihood and applies Jensen’s inequality for the concave function log: θ pθ (z0 , z1 ) p (z0 , z1 ) ≥ Eq(z1 |z0 ) log := ELBO(θ; z0 ). (10) log pθ0 (z0 ) = log Eq(z1 |z0 ) q(z1 | z0 ) q(z1 | z0 ) The right-hand side is the evidence lower bound, which can be split as h i ELBO(θ; z0 ) = Eq(z1 |z0 ) log pθ (z0 | z1 ) − KL q(· | z0 ) pθz1 . The two terms are the reconstruction log-likelihood and prior KL, respectively. Alternatively, the ELBO can be derived via the data processing inequality q(z0 , z1 ) log q0 (z0 ) − ELBO(θ; z0 ) = Eq1|0 (z1 |z0 ) log θ p (z0 , z1 ) = KL q(· | z0 ) pθ (· | z0 ) + log q0 (z0 ) − log pθ0 (z0 ) ≥ log q0 (z0 ) − log pθ0 (z0 ).

In particular, this shows that the gap in the ELBO inequality is exactly a KL divergence: log pθ0 (z0 ) − ELBO(θ; z0 ) = KL q(· | z0 ) pθ (· | z0 ) ≥ 0. It measures the discrepancy between the encoding and model posterior latent laws. Diffusion models are VAEs whose latent variable is the noising trajectory. Fix z0 ∈ X and let z0:N be a discrete-time Markov trajectory with N ≥ 1, initial law q0 , and forward kernels qt|t−1 . Then N Y q1:N |0 (z1:N | z0 ) = qt|t−1 (zt | zt−1 ). t=1

The learned reverse-time model factors as pθ (z0:N ) = pθN (zN )

N Y

pθt−1|t (zt−1 | zt ),

t=1

and is also Markov in forward time. In the VAE interpretation, z1:N is the latent variable and pθN is its terminal prior. Proposition 1 (Discrete-time diffusion ELBO). Assume the KL terms below are well defined. Then h i ELBO(θ; z0 ) = Eq1|0 (z1 |z0 ) log pθ0|1 (z0 | z1 ) − KL qN |0 (· | z0 ) pθN −

N X

i θ Eqt|0 (zt |z0 ) KL qt−1|t,0 (· | zt , z0 ) pt−1|t (· | zt ) .

(11)

t=2

Moreover, log pθ0 (z0 ) − ELBO(θ; z0 ) = KL q1:N |0 (· | z0 ) pθ1:N |0 (· | z0 ) ≥ 0. Proof. We start with the VAE ELBO equality with latent variable z1:N and use the reverse-model factorization above together with the reverse factorization q1:N |0 (z1:N | z0 ) = qN |0 (zN | z0 )

N Y

qt−1|t,0 (zt−1 | zt , z0 )

t=2

to obtain pθ (z0:N ) ELBO(θ; z0 ) = Eq1:N |0 (z1:N |z0 ) log q1:N |0 (z1:N | z0 ) " # Q θ pθN (zN ) N t=1 pt−1|t (zt−1 | zt ) = Eq1:N |0 (z1:N |z0 ) log . Q qN |0 (zN | z0 ) N t=2 qt−1|t,0 (zt−1 | zt , z0 )

Splitting the logarithm into boundary and transition terms gives (11). The gap identity is the corresponding VAE identity with latent variable z1:N .

4.2

CTMC formulation

We now specialize the trajectory latent to a continuous-time Markov chain; see [Nor97, JS03, Bré81] for standard textbook references. Notation. A single letter q or p denotes the finite-dimensional marginals, kernels, and bridges of a process, with subscripts recording the relevant times: qt is the time-t marginal, qt|s the transition kernel for times s < t, qt|0 (· | z0 ) the forward kernel from clean data, and qs|t,0 (· | zt , z0 ) the bridge. We often identify a random variable with its value and drop the event when it is clear, writing q(z0 | zt ) for q(Z0 = z0 | Zt = zt ). Jump rates take Q, R, and sometimes a, b: Qt is the forward b t denotes time reversal. generator, a hat Q We abbreviate “almost every” and “almost surely” for a probability measure µ as µ-a.e. and µ-a.s., for properties that hold on a set of probability 1. For two laws µ, ν, absolute continuity µ ≪ ν means ν(A) = 0 ⇒ µ(A) = 0. On finite state spaces this is simply support inclusion supp µ ⊆ supp ν, and the associated relative density (the Radon–Nikodym derivative) dµ/dν is the P likelihood ratio. We use H(Z) = − z p(z) log p(z) for the entropy of a discrete variable Z and I(X; Y ) = KL(pX,Y ∥ pX pY ) = H(X) − H(X | Y ) for the mutual information between X and Y . CTMC processes and their generators. Let X be finite and 0 < T , possibly T = ∞. A continuous-time discrete diffusion is a continuous-time Markov chain (CTMC) (Zt )t∈[0,T ] on X whose law is fixed by a time-dependent rate matrix (generator) Qt , with X Qt (x, y) ≥ 0 (x ̸= y), Qt (x, x) = − Qt (x, y), y̸=x

so that each row sums to zero. The off-diagonal entry Qt (x, y) is the instantaneous rate of the jump x → y: the transition kernel qt|s (y | x) = P(Zt = y | Zs = x) satisfies, as h ↓ 0, qt+h|t (y | x) = δ(x, y) + h Qt (x, y) + o(h). While the chain is in a given state x, the time until the next jump R u is exponentially distributed with rate −Qt (x, x), i.e., P(Zt = x, ∀t ∈ [s, u] | Zs = x) = exp( s Qt (x, x) dt). Conditional on a jump from x occurring at time t, the probability of destination y ̸= x is proportional to Qt (x, y), i.e., P(Zt+ = y | Zt = x, Zt+ ̸= x) = Qt (x, y)/(−Qt (x, x)). Sample paths are right-continuous and piecewise constant. Kolmogorov equations. Arranged as matrices with normalized rows, the kernels obey Chapman– Kolmogorov qu|s = qt|s qu|t and the forward and backward linear ODEs ∂t qt|s = qt|s Qt ,

∂s qt|s = −Qs qt|s ,

qs|s = I.

P In particular, with q0 given, the marginal qt solves ∂t qt = qt Qt , that is, ∂t qt (y) = x qt (x)Qt (x, y). By the uniqueness of solutions for ODEs with initial condition, the law of the chain is uniquely determined by q0 and Qt .

Path laws and trajectory likelihoods. Three standard facts, used in the Girsanov derivation of §4.3, describe the process at the path level; see [Nor97, Bré81, JS03]. The first is well-posedness: a law at the time from which the process is run (the left endpoint for a forward chain, the right endpoint for a reverse-time chain), together with bounded, measurable rates on the window [s, u], determines a unique CTMC path law, a genuine probability measure on the space of right-continuous, piecewise-constant trajectories, just as q0 and Qt determine the marginals through the Kolmogorov equations. Every process in this paper is specified exactly this way, by an endpoint law and a jump-rate family indexed by time, and this fact is what lets us speak of the path law so defined. Path laws take an uppercase P , subscripted by the window as in P[s,u] , and integration against them is written P (dz[s,u] ); finite-space factors keep plain pmf notation, so a joint law over a data point and a trajectory reads P (z0 , dz[s,u] ). The second fact is the trajectory likelihood. On a finite state space with bounded rates on the compact window [s, u], the chain makes finitely many jumps almost surely, so a path is described by its jump coordinates: the initial state x0 , the number of jumps n, the ordered jump times s < r1 < · · · < rn ≤ u, and the post-jump states x1 , . . . , xn . With respect to the reference measure on these jump coordinates, the CTMC has density qs (x0 ) exp

Z u

n Y Qt (Zt , Zt ) dt Qrk (xk−1 , xk ),

k=1

(12)

that is, the initial law qs (x0 ), the no-jump survival factor accumulated from the exit rate −Qt (Zt , Zt ) along the piecewise-constant path, and the rate Qrk (xk−1 , xk ) of each jump. This density is not a canonical object on the path space: it is taken relative to the jump-coordinate reference measure, and only ratios of such densities enter below, for which that reference measure cancels. xy Third, consider the compensated Rjump sum Mtxy := Ntxy − Λxy t , where Nt counts the number t xy of x → y jumps in [s, t] and Λt := s Qr (Zr , y)1(Zr = x) dr is the so-called compensator. Then Mtxy is a mean-zero martingale and Λxy t is the expected number of x → y jumps in [s, t] given the past. In particular, for any f : X × X → R, we have     Z u X X Qt (Zt , y) f (Zt , y) dt , (13) E f (Zr− , Zr ) 1{Zr− ̸= Zr } = E  s

s<r≤u

y̸=Zt

that is, the expected number of x → y jumps in [t, t + dt] is qt (x) Qt (x, y) dt. Time reversal.

The time-reversed process is also a CTMC. When qt (x) > 0, its generator is b t (x, y) = Qt (y, x) qt (y) , Q qt (x)

x ̸= y,

(14)

as follows from the Markov property and Bayes’ rule. In a generative model we do not have access to the marginal qt , but we can condition on the clean data. Conditioning the reversal on the endpoint z0 (replacing qt by qt|0 (· | z0 )), and using that Qt (y, zt | z0 ) = Qt (y, zt ) by the Markov property, gives the clean-conditioned reverse rate b t (zt , y | z0 ) = Qt (y, zt ) Q

qt|0 (y | z0 ) , qt|0 (zt | z0 )

y ̸= zt ,

(15)

where zt is the current noisy state at time t and y is the state reached by moving infinitesimally backward in time. In terms of small-time transition kernels, b t (zt , y | z0 ) + o(h), qt−h|t,0 (y | zt , z0 ) = δ(zt , y) + h Q

h ↓ 0.

Together with the forward rates Qt , these reverse rates specify every path law used in this paper, which we now name, starting from the clean-conditioned one, the natural object in diffusion. Definition 1 (Noising and denoising path laws). Fix a window [t1 , t2 ] ⊂ (0, T ) and let (Zt )t∈{0}∪[t1 ,t2 ] be the noising process with Z0 ∼ q0 . The clean-conditioned noising path law is P[t⋆1 ,t2 ]|0 (· | z0 ) := Law (Zt )t∈[t1 ,t2 ] | Z0 = z0 , a probability measure on the space of paths over [t1 , t2 ].1 Prefixing the data law z0 ∼ q0 gives the ⋆ joint law P0,[t defined by 1 ,t2 ] ⋆ (z0 , dz[t1 ,t2 ] ) = q0 (z0 ) P[t⋆1 ,t2 ]|0 (dz[t1 ,t2 ] | z0 ), P0,[t 1 ,t2 ]

and marginalizing it over z0 gives the marginal noising path law P[t⋆1 ,t2 ] := Law (Zt )t∈[t1 ,t2 ] . The model (denoising) path law P[tθ1 ,t2 ] is constructed from a terminal prior and the model b θ (zt , y). Attaching the data through a backward rates: initialized at pθt2 and run backward via Q t θ reconstruction kernel p0|t1 (z0 | zt1 ) extends it to the model joint θ P0,[t (z0 , dz[t1 ,t2 ] ) = P[tθ1 ,t2 ] (dz[t1 ,t2 ] ) pθ0|t1 (z0 | zt1 ). 1 ,t2 ]

All of these are well defined by the well-posedness discussed above and, by the same Kolmogorov uniqueness, each noising law is equivalently characterized by running the corresponding rates from either endpoint of the interval, conditionally on Z0 or not, forward or backward. For instance, the clean-conditioned law P[t⋆1 ,t2 ]|0 run backward is the CTMC started at qt2 |0 (· | z0 ) with the cleanb t (· | z0 ) of (15); likewise, the joint law P ⋆ conditioned reverse rates Q read entirely in reverse 0,[t1 ,t2 ]

b t , and attaches the data through time starts at qt2 , runs backward with the marginal reverse rates Q q0|t1 (· | zt1 ), using that Z0 is conditionally independent of Z[t1 ,t2 ] given Zt1 by the Markov property. θ b θt , pθ ), In particular, the joint model law P0,[t is uniquely determined by the triple (pθt2 , Q 0|t1 1 ,t2 ] θ b where Qt may depend on the noisy state Zt but not on the unknown clean data Z0 . To conclude, we introduce here the terminal mixing assumption, on which the practical interest of diffusion rests: the noising process destroys all information about the clean data as t ↑ T , which enables sampling from qT without access to the data. However, from the mathematical point of view, this is not a necessary assumption for most of the results in this paper. Assumption 1 (Terminal mixing). As t ↑ T (including T = ∞), the forward process forgets the data: qt|0 (· | z0 ) → qT for every z0 ∈ supp q0 , where the terminal law qT does not depend on z0 . In particular, the joint law of (Z0 , Zt ) converges to the independent product q0 ⊗ qT and I(Z0 ; Zt ) → 0. 1

Formally, the Skorokhod space D([t1 , t2 ], X ) of càdlàg (right-continuous with left limits) paths from [t1 , t2 ] to the finite state space X , equipped with the σ-field generated by the coordinate maps z[t1 ,t2 ] 7→ zt . Since this σ-field is generated by cylinder events, a path law is uniquely determined by its finite-dimensional marginals, and these are in turn determined by an endpoint law and the jump rates through the Kolmogorov equations; existence follows from the usual construction of the chain from its jump times and jump chain. This is the rigorous content of the well-posedness fact of §4.2, and how the path laws of this definition are constructed.

4.3

The CTMC ELBO

To go from the discrete-time ELBO of Proposition 1 to continuous time, we need to introduce the following definition and hypothesis. Definition 2 (Local rate divergence). For a, b ≥ 0, define  a   b − a + a log b , a > 0, b > 0, Φ(a, b) := b, a = 0, b ≥ 0,   +∞, a > 0, b = 0. This is the scalar relative entropy rate between two Poisson jump intensities. It is non-negative and vanishes if and only if a = b. Assumption 2 (Rate-support condition). We say the learned reverse rates satisfy the rate-support condition if the true reverse rates are absolutely continuous with respect to the learned rates, that is, b t (zt , · | z0 ) ≪ Q b θt (zt , ·), for dt dq(zt , z0 )-a.e. t, zt , z0 . Q This is equivalent to b t (zt , · | z0 ) ⊆ supp Q b θt (zt , ·), supp Q that is, the learned process can jump wherever the true reverse process can. We now state the continuous-time ELBO on an interior compact window [t1 , t2 ] ⊂ (0, T ), thus avoiding potentially singular endpoints as is commonly done in practice. We provide two alternative derivations: first, a proof based on the infinitesimal expansion of the KL terms of the discrete-time ELBO (Proposition 1), formalizing the typical informal arguments of the literature; second, a more general derivation that relies on Girsanov’s formula and avoids the stronger assumptions required by the infinitesimal-KL approach. Theorem 1 (CTMC ELBO). Let X be finite and let the true and learned reverse CTMCs have b t (· | z0 ) and Q b θt on [t1 , t2 ] ⊂ (0, T ). Define the per-time jump divergence bounded, measurable rates Q X b t (zt , y | z0 ), Q b θt (zt , y) . Jtθ (z0 , zt ) := Φ Q (16) y̸=zt

θ and that the reconstruction term below is well defined. Let P0,[t (z0 , ·) be the fixed-z0 section of 1 ,t2 ] θ the model joint law, a finite path measure of total mass p0 (z0 ). With z[t1 ,t2 ] as latent variable, the ELBO is defined at the path-measure level by2 θ ELBO[t1 ,t2 ] (θ; z0 ) := − KL P[t⋆1 ,t2 ]|0 (· | z0 ) P0,[t (z , ·) , (17) 0 1 ,t2 ] 2

θ Strictly, KL(· ∥ ·) here is an abuse of notation: its second argument P0,[t (z0 , ·) is not a probability measure 1 ,t2 ] θ but a fixed-z0 slice of the jointlaw, of total mass p0 (z0 ). The symbol is to be read simply as the mean log-ratio dP ⋆

(·|z0 )

EP[t⋆ ,t ]|0 (·|z0 ) log dP[tθ 1 ,t2 ]|0 (z ,·) , which is well defined under the stated absolute-continuity hypotheses (the Radon– 1

0,[t1 ,t2 ]

Nikodym derivative being a ratio of the kind in (12)).

and admits the expanded form i h ELBO[t1 ,t2 ] (θ; z0 ) = Eqt |0 (zt1 |z0 ) log pθ0|t1 (z0 | zt1 ) − KL qt2 |0 (· | z0 ) pθt2 1 | {z } | {z } reconstruction

Z t2 − t

terminal prior

(18)

i Eqt|0 (zt |z0 ) Jtθ (z0 , zt ) dt . {z }

path integral of jump divergence

Moreover, the ELBO gap is the relative entropy between the clean-conditioned path laws, log pθ0 (z0 ) − ELBO[t1 ,t2 ] (θ; z0 ) = KL P[t⋆1 ,t2 ]|0 (· | z0 ) P[tθ1 ,t2 ]|0 (· | z0 ) ≥ 0. Our first derivation is based on the following lemma on the infinitesimal expansion of a one-step KL between jump kernels, turning each KL term into the local rate divergence Φ. Lemma 1 (Φ as the differential KL of jump kernels). Let at , bt be piecewise continuous CTMC a rates on a finite space X and a compact time interval [t, t + ∆], ∆ > 0. Let qt,t+∆ , pbt,t+∆ denote a their transition kernels from t to t + ∆. Assume that the jump graph of qt is a fixed set E and that, for every t and constants 0 < c < C < ∞, c ≤ at (x, y), bt (x, y) ≤ C, ∀(x, y) ∈ E;

at (x, y) = 0, ∀(x, y) ∈ / E;

0 ≤ bt (x, y) ≤ C. (19)

Then, uniformly in t and x, X a KL qt,t+∆ (· | x) pbt,t+∆ (· | x) = ∆ Φ at (x, y), bt (x, y) + o(∆). y̸=x

Proof. The proof proceeds by computing and controlling the o(∆)-expansion of the KL divergence for each possible jump x 7→ y ∈ X depending on whether (x, y) ∈ E or not. For jumps x ̸= y, (x, y) ∈ E, integrating the uniformly continuous rates directly gives a qt,t+∆ (y | x) = ∆at (x, y) + o(∆),

pbt,t+∆ (y | x) = ∆bt (x, y) + o(∆).

Because both rates are bounded below and above on E, we have a qt,t+∆ (y | x) log

a qt,t+∆ (y | x)

pbt,t+∆ (y | x)

= ∆at (x, y) log

at (x, y) + o(∆), bt (x, y)

(x, y) ∈ E.

(20)

It remains to control x ̸= y with (x, y) ∈ / E. If y is unreachable from x in the graph E, then a qt,t+∆ (y | x) = 0 by the second part of (19). Otherwise, let d ≥ 2 be the length of a shortest E-path

from x to y and let N∆ be the total number of jumps of the process q in the interval [t, t + ∆]. The uniform rate bounds and the finiteness of X give a common bound on the total exit rate from any state. Thus, we can bound N∆ by a Poisson process with that same rate, to get Pr(N∆ ≥ k) = O(∆k ),

k ≥ 1,

uniformly in t and the initial state. Following one shortest path from x to y and making no other jumps has probability at least a constant times ∆d under both processes, whereas any single-jump 25

a qt,t+∆ (y | x)

= O(∆d | log ∆|) = O(∆2 | log ∆|).

pbt,t+∆ (y | x)

Thus all transitions outside the graph E contribute o(∆) in total: X

a qt,t+∆ (y | x) log

y̸=x,(x,y)∈E /

a qt,t+∆ (y | x)

pbt,t+∆ (y | x)

= o(∆).

(21)

Finally, returning to P x after a jump requires P at least another jump, with probability O(∆2 ). Thus, denoting At (x) := y̸=x at (x, y), Bt (x) := y̸=x bt (x, y), self-transition probabilities satisfy a qt,t+∆ (x | x) = 1 − ∆At (x) + o(∆),

pbt,t+∆ (x | x) = 1 − ∆Bt (x) + o(∆).

The self-transition KL contribution is therefore 1 − ∆At (x) + o(∆) = ∆(Bt (x) − At (x)) + o(∆). 1 − ∆Bt (x) + o(∆)

(1 − ∆At (x) + o(∆)) log

(22)

To conclude, adding the self-term (22) to the transitions in E according to (20) and the o(∆) remainder (21) from the terms outside E gives, uniformly in t and x, X X at (x, y) a KL qt,t+∆ (· | x) pbt,t+∆ (· | x) = ∆ bt (x, y) − at (x, y) + ∆ + o(∆) at (x, y) log bt (x, y) y̸=x (x,y)∈E X =∆ Φ at (x, y), bt (x, y) + o(∆). y̸=x

The additional assumptions of Lemma 1, namely, the fixed jump-graph E, piecewise continuity, and uniform boundedness conditions, are only required for the infinitesimal-KL argument of the first proof below. Nonetheless, they hold for masked, uniform, and fixed-support GIDD processes. Proof 1 of Theorem 1 (infinitesimal KL). First impose the stronger conditions of Lemma 1: piecewise continuity and, on each continuity piece, the true jump graph is fixed, and both rates are uniformly bounded above and away from zero on the true graph. Apply the discrete-time ELBO of Proposition 1 to a finite partition (τk )k of [t1 , t2 ], with t1 = τ0 < τ1 < · · · < τN = t2 ,

∆k := τk+1 − τk ,

|∆| := max ∆k . 0≤k<N

This gives h i ELBO(τk )k (θ; z0 ) = Eqt |0 (zt1 |z0 ) log pθ0|t1 (z0 | zt1 ) − KL qt2 |0 (· | z0 ) pθt2 1

−

N −1 X

Eqτ

k+1

h i θ KL q (· | z , z ) p (· | z ) . τk+1 0 τk+1 τk |τk+1 ,0 τk |τk+1 |0 (zτk+1 |z0 )

k=0

The two kernels in each summand are the exact transitions of the true and learned reverse CTMCs over an interval of length ∆k . Applying Lemma 1 in the backward time orientation gives X b θ (x, y) + rk (x), b τ (x, y | z0 ), Q KL qτk |τk+1 ,0 (· | x, z0 ) pθτk |τk+1 (· | x) = ∆k Φ Q τk+1 k+1 y̸=x

where the lemma’s uniform remainders satisfy |rk (x)| ≤ ∆k η(∆k ) for some function η(δ) → 0 independent of x and k. Taking the expectation over zτk+1 ∼ qτk+1 |0 (· | z0 ) and summing gives N −1 X

∆k Eqτ

k+1

−1 i NX h θ Jτk+1 (z0 , zτk+1 ) + E[rk ]. |0 (zτk+1 |z0 )

k=0

The first sum is a Riemann sum of the piecewise continuous integrand t 7→ Eqt|0 (zt |z0 ) Jtθ (z0 , zt ) Rt and converges to t12 Eqt|0 (zt |z0 ) Jtθ (z0 , zt ) dt as |∆| → 0 by the integrability assumption. The second is controlled uniformly, N −1 X k=0

E[rk ] ≤

N −1 X k=0

∆k η(∆k ) ≤

−1 NX sup η(δ) ∆k = (t2 − t1 ) sup η(δ) −−−−→ 0, δ≤|∆|

k=0

δ≤|∆|

|∆|→0

yielding (18) as the ∆ → 0 limit of the discrete-time ELBOs.3 Finally, the same skeleton-limit argument applied to the discrete-time VAE gap identity (latent z[t1 ,t2 ] ) yields the path-level gap identity above. Our second derivation rests on the following change-of-measure (Girsanov) formula for finitestate jump processes, where we use the path-law properties and notation introduced in §4.2. Theorem 2 (Girsanov formula and relative entropy for finite-state CTMCs). Let X be finite and [s, u] a compact interval. Let P, P θ be the path laws of two CTMCs specified by initial laws ps , pθs at time s and bounded, measurable rates at , bt , satisfying the support conditions ps ≪ pθs and at (x, ·) ≪ bt (x, ·) for every x and a.e. t. Then P ≪ P θ , and the path-law relative entropy between the laws is   Z u X EZt ∼P  KL P P θ = KL ps pθs + Φ at (Zt , y), bt (Zt , y)  dt, (23) s

y̸=Zt

where both sides are finite if and only if the integral is. Note the structure of (23): the KL between two CTMC path laws is not a pure rate mismatch, but it also carries the KL between the two laws at the starting time. P P Proof. Write At (x) = y̸=x at (x, y) and Bt (x) = y̸=x bt (x, y) and apply the trajectory likeliθ hood (12) to P and P . Under the rate-support condition at ≪ bt , the jump-coordinate reference 3

Strictly speaking, the argument above computes the limit of the ELBOs of the finite time-skeletons (Zt )t∈πn . To conclude that this limit is the path-level ELBO (17), take nested partitions πn whose union is dense in [t1 , t2 ]: the skeletons then generate the whole path σ-field, and the skeleton KLs increase to the path KL [Gra11, Lemma 5.4.1].

measure cancels in the density ratio, which is therefore well defined P -a.s. so that P ≪ P θ , and in particular the ratio up to τ ∈ [s, u] is given by Z τ Y ar (Zr− , Zr ) dP ps (Zs ) B (Z ) − A (Z ) = exp , t t t t dt θ θ br (Zr− , Zr ) dP [s,τ ] ps (Zs ) s s<r≤τ : Zr− ̸=Zr

(for further details we refer to [Bré81, JS03]). Taking log of this ratio and integrating against P , the jump compensator (13) with f = log(at /bt ) gives     Z u X X a (Z , Z ) a (Z , y) − r r  t t r  dt, EP  log EP  at (Zt , y) log = br (Zr− , Zr ) b (Z t t , y) s s<r≤u

y̸=Zt

hP i Ru while the holding factor contributes s EP dt. Adding the initial y̸=Zt bt (Zt , y) − at (Zt , y) θ relative entropy KL ps ps and collecting terms through Φ(a, b) = b−a+a log(a/b) yields (23). Proof 2 of Theorem 1 (Girsanov). We start from the path-measure definition (17). The reverse Markov property factorizes the model z0 -slice as θ P0,[t (z0 , dz[t1 ,t2 ] ) = pθ0|t1 (z0 | zt1 ) P[tθ1 ,t2 ] (dz[t1 ,t2 ] ). 1 ,t2 ]

Substituting this factorization into (17) and applying the Radon–Nikodym chain rule to the resulting density gives h i − ELBO[t1 ,t2 ] (θ; z0 ) = −Eqt |0 (·|z0 ) log pθ0|t1 (z0 | Zt1 ) + KL P[t⋆1 ,t2 ]|0 (· | z0 ) P[tθ1 ,t2 ] , 1

KL qt2 |0 (· | z0 ) pθt2 +

Z t2 t1

 Eqt|0 (zt |z0 ) 

 X

b t (zt , y | z0 ), Q b θt (zt , y)  dt, Φ Q

y̸=zt

directly yielding (18), with equality at the path-law level by Girsanov’s formula. θ For the gap identity, the finite measure P0,[t (z0 , ·) has total mass pθ0 (z0 ), so its normalization 1 ,t2 ] is the model posterior path law P[tθ1 ,t2 ]|0 (· | z0 ) :=

θ P0,[t (z0 , ·) 1 ,t2 ]

pθ0 (z0 )

θ Substituting P0,[t (z0 , ·) = pθ0 (z0 ) P[tθ1 ,t2 ]|0 (· | z0 ) into (17) splits off the constant log pθ0 (z0 ) and 1 ,t2 ] leaves log pθ0 (z0 ) − ELBO[t1 ,t2 ] (θ; z0 ) = KL P[t⋆1 ,t2 ]|0 (· | z0 ) P[tθ1 ,t2 ]|0 (· | z0 ) .

The ELBO recipe.

Equation (18) gives a three-step recipe for any discrete-diffusion process:

1. Forward law. Choose the noising kernel qt|0 (· | z0 ) with generator Qt . b t (zt , y | z0 ) using (15), and choose 2. Reverse rates. Form the clean-conditioned reverse rate Q θ b t (zt , y) that has access to the noisy state but not the clean data. a model rate Q bt , Q b θ ) over the admissible jumps y ̸= zt , integrate over 3. Assemble. Sum the per-jump Φ(Q t time and the noisy state zt ∼ qt|0 (· | z0 ), and append the reconstruction and prior boundary terms. Models are trained by maximizing the ELBO, equivalently by minimizing its negative. We write the per-data and data-averaged negative ELBO NELBO[t1 ,t2 ] (θ; z0 ) := − ELBO[t1 ,t2 ] (θ; z0 ), NELBO[t1 ,t2 ] (θ) := Eq0 NELBO[t1 ,t2 ] (θ; z0 ) . As this is the quantity practitioners track and optimize, we phrase our results in terms of the NELBO throughout: variational bounds are stated as ELBOs, but the trained cost, calibration values, and floors below are their negatives.

4.4

Importance sampling, time clocks, and the empirical ELBO

The bound (18) becomes a training objective by estimating its time integral, optionally with an importance sampling (IS) schedule pIS (t) for variance reduction. Empirical estimator.

Drawing t ∼ pIS on [t1 , t2 ] and zt ∼ qt|0 (· | z0 ) gives the unbiased estimator

\ [t ,t ] (θ; z0 ) = Et∼pIS , q NELBO 1 2 t|0

Jtθ (z0 , zt ) + KL qt2 |0 (· | z0 ) pθt2 + rec(θ), c pIS (t)

(24)

where 1/pIS (t) is the change of measure required to keep the IS estimate unbiased and rec(θ) c is a cross-entropy-like term given by the empirical average of − log pθ0|t1 (z0 | zt1 ) over sampled (z0 , zt1 ). In particular, this term accommodates any final ad hoc sampling step from t1 to 0, such as the standard collapsing or resampling to clean tokens used in practice. The terminal-prior KL can be computed either analytically or approximately from the forward terminal state and the choice of sampling prior. Retaining the two boundary terms keeps the per-process estimates complete and allows direct comparison of ELBOs across methods by tracking the boundary terms usually neglected in the literature as “constant”. Importance sampling as the Markov process clock. Choosing an importance sampler for the time integral is equivalent to choosing the clock on which the Markov process is parameterized. To see it, let τ = τ (t) be a strictly increasing C 1 change of time with dτ /dt > 0, and let t = t(τ ) denote its inverse. The reparameterized chain Zτ′ := Zt(τ ) has transition kernel and generator qτ′ 2 |τ1 (y | x) := qt(τ2 )|t(τ1 ) (y | x),

Q′τ (x, y) =

dt Q (x, y). dτ t(τ )

For our finite-state setting this can be seen from the chain rule in the definition of the generator. Consequently, writing τ0 = τ (0) and t = t(τ ), the reverse rates transform in the same way, qτ′ |τ0 (y) qt|0 (y) dt dt b ′ ′ b Qτ (z, y) = Qτ (y, z) ′ = Qt (z, y), Qt (y, z) = qτ |τ0 (z) dτ qt|0 (z) dτ Rt and similarly for the learned and clean-conditioned rates. Estimating t12 Eqt|0 Jtθ dt with importance density pIS (t) corresponds to sampling uniformly according to the clock Z t pIS (u) du,

τ (t) := τ1 +

dτ = pIS (t) dt,

dt 1 = . dτ pIS (t)

Since Φ(ca, cb) = cΦ(a, b) for all c > 0, the Jacobian factor can be absorbed into both reverse rates:   Z t2 Z τ2 h i X dt b t (zt , y | z0 ), Q b θ (zt , y)  dτ Eqt|0 Jtθ dt = Φ Q Eqt|0  t dτ t1 τ1 y̸=zt   Z τ2 Z τ2 h i X ′ ′θ b b   = Eq′ Φ Qτ (zτ , y | z0 ), Qτ (zτ , y) dτ = Eq′ Jτ′θ dτ. τ1

τ |τ0

τ1

y̸=zτ

τ |τ0

Thus, choosing the importance sampler pIS is precisely choosing the clock on which the reverse Markov chain is run. In the separable generator case where Qt = dλ dt M with λ(t) a strictly increasing scalar function of time, one may further choose the distinguished clock τ = λ(t), in which case the forward process becomes time-homogeneous with generator M . It is worth noting that importance sampling gives an unbiased estimator of the ELBO, whereas deliberately reweighting the per-time integrand changes the training objective. Suitable reweightings, such as monotone ones, can themselves define valid and potentially tighter variational bounds [ST25, KSPH21] and are commonly used to trade likelihood for sample quality. Variance-optimal importance sampling. For one sample t ∼ pIS , the Monte Carlo estimator attains minimal variance at the optimal importance density [RC04], q p⋆IS (t) ∝ Eq0 qt|0 Jtθ (z0 , zt )2 . This second-moment rule is widely used in practice for adaptive timestep sampling in DDPMs, while variational diffusion models directly learn the noise schedule to reduce the variance of the VLB estimator [ND21, KSPH21]. In the next section, Theorem 4 identifies the oracle mean integrand Jt⋆ with the instantaneous d H(Z0 | Zt ). Thus pIS (t) ∝ Jt⋆ defines an information-uniform rate of information loss, Jt⋆ = dt clock: it equalizes the expected oracle information loss per unit clock, but is not generally varianceoptimal. Nonetheless, recent continuous-diffusion work uses this conditional-entropy-rate profile to adapt the training noise distribution [RNB+ 26].

What negative ELBO minimization really optimizes

5.1

ELBO as distance to the oracle reverse process

We begin with our main result on how to interpret the ELBO: after subtracting the data entropy, the negative ELBO (NELBO) splits into two mismatch terms, the reconstruction error at the lower endpoint and the path-law KL from the oracle to the model CTMC. The NELBO is therefore more than a variational upper bound on the negative log-likelihood: it is the path divergence to the oracle reverse dynamics, which we call the Oracle Distance theorem. Recall from Definition 1 the noising and model path laws P[t⋆1 ,t2 ] and P[tθ1 ,t2 ] on [t1 , t2 ] ⊂ (0, T ), b t and Q b θt . in their backward descriptions started at qt and pθt and run with the rates Q 2

Theorem 3 (Oracle Distance). Consider [t1 , t2 ] ⊂ (0, T ) with bounded, measurable reverse rates b t (zt , ·) ≪ Q b θ (zt , ·) a.e., as b t and Q b θt . Impose the support conditions q0|t ≪ pθ , qt ≪ pθ , and Q Q t2 t 2 1 0|t1 well as integrability of the jump-divergence   Z t2 X b t (zt , y), Q b θt (zt , y)  dt < ∞. Eq0 qt|0  Φ Q t1

y̸=zt

For the path laws P[t⋆1 ,t2 ] , P[tθ1 ,t2 ] above, h i NELBO[t1 ,t2 ] (θ) − H(q0 ) = Eqt1 KL q0|t1 (· | zt1 ) pθ0|t1 (· | zt1 ) + KL P[t⋆1 ,t2 ] P[tθ1 ,t2 ] ,

(25)

where the path-law KL is, by Theorem 2, hP Z t2 i ⋆ θ θ b bθ KL P[t1 ,t2 ] P[t1 ,t2 ] = KL qt2 pt2 + dt. Eqt y̸=zt Φ Qt (zt , y), Qt (zt , y)

(26)

b t (zt , ·) ≪ Q b θt (zt , ·) is a consequence of Q b t (zt , · | z0 ) ≪ Q b θt (zt , ·), Note that the support condition Q since the marginal reverse rate is a conditional expectation of the clean-conditioned rate. Simibt , Q b θt ) is evaluated against the unconditional reverse rates, and larly, the jump-rate divergence Φ(Q b t (· | z0 ), Q b θ ) either using the Pythagorean its integrability follows from the integrability of Φ(Q t decomposition (Proposition 2) or Jensen’s inequality and Φ’s convexity in its first argument. The proof combines two ingredients of independent interest, which we state here and prove in §5.2 and §5.3. The first is a Pythagorean decomposition of the ELBO integrand around the marginal reverse rate: the model pays the oracle’s cost plus its own mismatch to the oracle, with no cross term. Proposition 2 (Pythagorean decomposition of the ELBO integrand). For any Zt -measurable model b θ such that the model mismatch term below is finite, we have reverse rate Q t h i h i Eq0 qt|0 Jtθ (z0 , zt ) = Eq0 qt|0 Jtθ⋆ (z0 , zt ) + | {z } oracle cost

Eqt |

i b t (zt , y), Q b θt (zt , y) , Φ Q y̸=zt {z }

(27)

model mismatch

b θ⋆ = Q b t , which §5.2 where Jtθ⋆ is the integrand (16) evaluated at the marginal reverse rate Q t identifies as its unique Zt -measurable optimizer. 31

The second ingredient evaluates the oracle cost, which we denote by   h i X b t (Zt , y | Z0 ), Q b t (Zt , y)  . Jt⋆ := Eq0 qt|0 Jtθ⋆ (z0 , zt ) = EZ0 ,Zt  Φ Q

(28)

y̸=Zt

We now prove that the oracle cost Jt⋆ is exactly the rate at which the noising process destroys information about the clean data, provided the following mild regularity assumptions (which the masked, uniform, and GIDD processes of §7 satisfy; for GIDD requiring that the interpolating law has fixed support). Assumption 3 (Regularity and fixed support.). In the compact window [t1 , t2 ] ⊂ (0, T ), we have piecewise continuous and bounded rates Qt such that the piecewise supports of qt and qt|0 (· | z0 ) are independent of t for each z0 ∈ supp q0 . Theorem 4 (NELBO oracle integrand = information loss rate). Under the regularity and fixedsupport assumptions above, d d H(Z0 | Zt ) = − I(Z0 ; Zt ) ≥ 0, dt dt

(29)

Jt⋆ dt = H(Z0 | Zt2 ) − H(Z0 | Zt1 ) = I(Z0 ; Zt1 ) − I(Z0 ; Zt2 ).

(30)

Jt⋆ = and consequently, Z t2 t1

The regularity assumption is needed only for this second ingredient, not for Theorem 3 itself; §5.5 gives an alternative proof of the latter that avoids it. Granting the two ingredients, the proof is a short assembly. Proof of Theorem 3. Averaging the negative CTMC ELBO (18) over Z0 ∼ q0 expresses the data-averaged negative ELBO, NELBO[t1 ,t2 ] (θ), as the sum of a reconstruction term, a path term, and a terminal-prior term: h i Z t2 h i h i θ EZ0 ,Zt1 − log p0|t1 (Z0 | Zt1 ) + EZ0 ,Zt Jtθ (Z0 , Zt ) dt + EZ0 KL qt2 |0 (· | Z0 ) pθt2 , t1

and Theorem 4 integrates the oracle cost into the information lost over the window, Z t2 Jt⋆ dt = H(Z0 | Zt2 ) − H(Z0 | Zt1 ). t1

Terminal-prior term. Splitting the log ratio through the marginal qt2 , we get " # h i | Z ) q (Z ) (Z q 0 t t |0 t t 2 + Eqt2 log θ2 2 EZ0 KL qt2 |0 (· | Z0 ) pθt2 = EZ0 ,Zt2 log 2 qt2 (Zt2 ) pt2 (Zt2 ) = I(Z0 ; Zt2 ) + KL qt2 pθt2 = H(Z0 ) − H(Z0 | Zt2 ) + KL qt2 pθt2 , where we used the mutual information identities I(Z0 ; Zt2 ) = H(Z0 ) − H(Z0 | Zt2 ) = H(Zt2 ) − H(Zt2 | Z0 ). Assembly. Summing the three terms in the order reconstruction, path, and terminal prior, gives i h + EZt1 KL q0|t1 (· | Zt1 ) pθ0|t1 (· | Zt1 ) NELBO[t1 ,t2 ] (θ) = H(Z0 | Zt1 ) Z t2 hP i b t (Zt , y), Q b θ (Zt , y) dt +H(Z0 | Zt2 ) −H(Z0 | Zt1 ) + Q EZt Φ t y̸=Zt t1 +H(Z0 ) −H(Z0 | Zt2 ) + KL qt2 pθt2 . The conditional entropies telescope into the data entropy H(Z0 ) = H(q0 ), yielding h i NELBO[t1 ,t2 ] (θ) − H(q0 ) = EZt1 KL q0|t1 (· | Zt1 ) pθ0|t1 (· | Zt1 ) Z t2 hP i θ b t (Zt , y), Q b θ (Zt , y) dt, + KL qt2 pt2 + EZt Q Φ t y̸=Zt t1

which is (25) when combined with the expansion of KL P[t⋆1 ,t2 ] P[tθ1 ,t2 ] from (26). When both boundary terms vanish, the identity gives a clean expression for the NELBO: exactly the jump-rate divergence between the oracle and the model reverse processes (Figure 3). Corollary 1. In thesetting of Theorem 3, suppose the reconstruction term θ at t1 in (25) and the θ terminal KL qt2 pt2 vanish as t1 ↓ 0, t2 ↑ T , and that Eq0 qt|0 Jt (z0 , zt ) is integrable on [0, T ]. Then Z T hP i b t (zt , y), Q b θ (zt , y) dt. NELBO[0,T ] (θ) − H(q0 ) = Eqt Φ Q (31) t y̸=zt 0

5.2

The ELBO as a projection problem and the Pythagorean decomposition

The model rate cannot depend on the unknown clean data Z0 . The local CTMC ELBO therefore poses a projection problem: given a random target rate depending on (Z0 , Zt ), choose the best rate that only has access to Zt . Its solution is governed by an exact Pythagorean theorem for Φ, which identifies the optimal projection and splits the trained cost into oracle cost plus model mismatch. Lemma 2 (Bregman–Pythagoras identity for Φ). Let A ≥ 0 be such that E[A log A] < ∞, G an observable, and B = b(G) ≥ 0 a G-measurable predictor for which the displayed expectations are finite. With B ⋆ := E[A | G], E[Φ(A, B)] = E[Φ(A, B ⋆ )] + E[Φ(B ⋆ , B)]. 33

Proof. Condition on G and write m = B ⋆ = E[A | G] and b = B. First suppose m, b > 0. Expanding Φ(a, b) = b − a + a log ab and using E[A | G] = m (with the convention 0 log 0 = 0) gives E[Φ(A, b) | G] = E b − A + A log Ab G = b − m + E[A log A | G] − m log b, A E[Φ(A, m) | G] = E m − A + A log m G = E[A log A | G] − m log m, m Φ(m, b) = b − m + m log = b − m + m log m − m log b. b Adding the last two displays reproduces the first, giving the conditional identity E[Φ(A, B) | G] = E[Φ(A, B ⋆ ) | G] + Φ(B ⋆ , B). The boundary cases follow by continuity with the conventions in the definition of Φ. Taking the expectation over G yields the claim. Since E[Φ(B ⋆ , B)] ≥ 0 vanishes if and only if B = B ⋆ a.e., the identity exhibits the conditional mean B ⋆ = E[A | G] as the unique minimizer of E[Φ(A, B)] over non-negative G-measurable predictors B. (The same conclusion follows directly from convexity: b 7→ E[Φ(A, b) | G = g] is strictly convex with derivative 1 − E[A | G = g] /b.) Applied to the reverse rates, with the conditional mean computed by Bayes’ rule, this yields the population optimizer of the ELBO. Theorem 5 (Reverse-rate projection). Fix a time t and a jump destination y, and let (Z0 , Zt ) ∼ b θt measurable with respect to the noisy state Zt q0 qt|0 on the event {Zt ̸= y}. Among model rates Q h i b t (Zt , y | Z0 ), Q b θt (Zt , y) is minimized uniquely up to null only, the expected rate divergence E Φ Q sets by h i b θ⋆ (zt , y) := Eq (z |z ) Q b t (zt , y | z0 ) = Q b t (zt , y). Q t 0|t 0 t

(32)

Proof. By Lemma 2 and the discussion following it, with G = Zt and target b t (Zt , y | Z0 ) = Qt (y, Zt ) A(Z0 , Zt ) := Q

qt|0 (y | Z0 ) , qt|0 (Zt | Z0 )

the ELBO-optimal Zt -measurable rate is the conditional mean E[A(Z0 , Zt ) | Zt = zt ], given by h i X q0 (z0 )qt|0 (zt | z0 ) qt|0 (y | z0 ) b t (Zt , y | Z0 ) Zt = zt = Qt (y, zt ) E Q qt (zt ) qt|0 (zt | z0 ) z0 X 1 = Qt (y, zt ) q0 (z0 ) qt|0 (y | z0 ) qt (zt ) z 0

= Qt (y, zt )

qt (y) qt (zt )

b t (zt , y). =Q

The oracle dynamics are therefore those of the true marginal reversal, whose jump rates are the posterior average of the clean-conditioned rates (32) (Figure 1). Applying Lemma 2 jump by jump around this optimizer proves the first ingredient. 34

Proof of Proposition 2. Fix t and a candidate reverse jump destination y, and apply Lemma 2 with b θ (Zt , y). By Theorem 5 the b t (Zt , y | Z0 ), and model rate B = Q observable G = Zt , target A = Q t ⋆ b conditional mean is B = E[A | Zt = zt ] = Qt (zt , y), so h h i i b t (zt , y | z0 ), Q b θ (zt , y) = Eq q Φ Q b t (zt , y | z0 ), Q b t (zt , y) Eq0 qt|0 Φ Q t 0 t|0 h i b t (zt , y), Q b θt (zt , y) , + Eqt Φ Q the mismatch expectation collapsing to one over zt ∼ qt because both arguments are Zt -measurable. Summing over the admissible jumps y ̸= zt concludes the proof.

5.3

The oracle ELBO is the rate of information loss

The second ingredient identifies the irreducible part of the ELBO, the oracle cost of Proposition 2: even the oracle reverse rate must pay for the information that the forward noising process destroys about the clean data, and the ELBO tracks exactly this (Figure 2). We now prove Theorem 4, stated in §5.1. Proof of Theorem 4. Terms with zero clean-conditioned rate contribute zero under the 0 log 0 = 0 convention. On every positive-Φ term, the fixed-support assumption makes the following ratios well defined. b t (· | Z0 ), Q b t ), the quotient of reverse rates is, by the conditional and unconditional Within Φ(Q reverse rate formulas, qt|0 (y | z0 ) qt|0 (z | z0 ) qt|0 (y | z0 ) qt (z) q0|t (z0 | y) = = , qt (y) qt|0 (z | z0 ) qt (y) q0|t (z0 | z) Qt (y, z) qt (z)

Qt (y, z)

b t (z, y | z0 ) Q = b t (z, y) Q

where the last equality is Bayes’ rule. In the expansion Φ(a, the first two i h b) = b − a + a log(a/b), b b terms cancel after conditioning on Zt because Qt (z, y) = E Qt (z, y | Z0 ) Zt = z , as in Theorem 5. Thus, we get   X b b t (Zt , y | Z0 ) log Qt (Zt , y | Z0 )  Jt⋆ = EZ0 ,Zt  Q b t (Zt , y) Q y̸=Zt

XXX z0

b t (z, y | z0 ) log q0 (z0 ) qt|0 (z | z0 ) Q

y̸=z

q0|t (z0 | y) . q0|t (z0 | z)

Using the reverse rate identity (15) to make the substitution b t (z, y | z0 ) = qt|0 (y | z0 ) Qt (y, z), qt|0 (z | z0 ) Q we get, after adding the diagonal generator terms whose logarithms vanish, that Jt⋆ =

q0 (z0 ) qt|0 (y | z0 ) Qt (y, z) log

z0 ,y,z

q0|t (z0 | y) . q0|t (z0 | z)

(33)

Reciprocally, we now differentiate the conditional entropy X q0 (z0 ) qt|0 (z | z0 ) log q0|t (z0 | z). H(Z0 | Zt ) = − z0 ,z

Because the state space and supports are finite and constant on the interior window, differentiation may pass through the sum. The derivative of the log-density terms cancels by normalization: X X X q0|t (z0 | z) = 0, q0 (z0 ) qt|0 (z | z0 ) ∂t log q0|t (z0 | z) = qt (z) ∂t z0 ,z

so that

X d q0 (z0 ) ∂t qt|0 (z | z0 ) log q0|t (z0 | z). H(Z0 | Zt ) = − dt z0 ,z P Inserting the Kolmogorov forward equation ∂t qt|0 (z | z0 ) = y qt|0 (y | z0 )Qt (y, z), X d H(Z0 | Zt ) = − q0 (z0 ) qt|0 (y | z0 ) Qt (y, z) log q0|t (z0 | z). dt z ,y,z 0

Since z Qt (y, z) = 0, we may add the vanishing term y) = 0, obtaining

z0 ,y,z q0 (z0 )qt|0 (y | z0 )Qt (y, z) log q0|t (z0 |

X q0|t (z0 | y) d H(Z0 | Zt ) = q0 (z0 ) qt|0 (y | z0 ) Qt (y, z) log . dt q (z | z) 0 0|t z ,y,z

(34)

d Comparing (33) and (34) proves the entropy identity. Since H(q0 ) is constant, dt H(Z0 | Zt ) = d − dt I(Z0 ; Zt ). Non-negativity follows from Φ ≥ 0. The fixed-support assumption also makes the conditional entropy absolutely continuous on compact windows, so integration proves (30).

Corollary 2 (Monotonicity of retained information). The map t 7→ I(Z0 ; Zt ) is nonincreasing. Indeed, for s < t the Markov chain Z0 → Zs → Zt and the data-processing inequality give d I(Z0 ; Zt ) ≤ I(Z0 ; Zs ). Alternatively, wherever differentiable, − dt I(Z0 ; Zt ) = Jt⋆ ≥ 0. Rt In particular, the oracle path term t12 Jt⋆ dt is exactly the mutual information lost over that window, I(Z0 ; Zt1 ) − I(Z0 ; Zt2 ).

5.4

The universal NELBO floor

Theorem 3 already explains why the data entropy is the universal NELBO floor: after subtracting H(q0 ), the objective is a sum of relative entropies and is therefore non-negative. We nonetheless record the direct variational argument below, since it does not rely on the information-loss regularity assumptions and it makes the equality case explicit. At the oracle the ELBO is tight and the NELBO attains the data entropy H(q0 ), independently of the noising process. Lemma 3. For every noising process law P ⋆ and denoising model process law P θ on [t1 , t2 ] ⊂ (0, T ) satisfying the conditions of Theorem 1, we have h i NELBO[t1 ,t2 ] (θ) ≥ Eq0 − log pθ0 (z0 ) ≥ H(q0 ). (35) 36

Proof. The first inequality is the ELBO gap of Theorem 1; the second is Gibbs’ inequality h i Eq0 − log pθ0 (z0 ) = H(q0 ) + KL q0 pθ0 ≥ H(q0 ).

Corollary 3 (Universal oracle NELBO floor). Fix [t1 , t2 ] ⊂ (0, T ). Suppose there exists a model b θ⋆ = Q b t on [t1 , t2 ], θ⋆ that realizes the oracle prior pθt2⋆ = qt2 , the oracle marginal reverse rate Q t ⋆ = q0|t1 . Then, it exactly recovers the clean-data marginal and the oracle reconstruction kernel pθ0|t 1 pθ0⋆ = q0 . In particular, the ELBO is tight, ELBO[t1 ,t2 ] (θ⋆ ; z0 ) = log pθ0⋆ (z0 ) = log q0 (z0 ),

(36)

and the expected NELBO attains the data entropy floor, inf NELBO[t1 ,t2 ] (θ) = NELBO[t1 ,t2 ] (θ⋆ ) = H(q0 ). θ

(37)

The same conclusion holds at t2 = T by taking the limit t2 ↑ T . ⋆ Proof. Evaluate Theorem 3 at θ⋆ . Its reconstruction term vanishes because pθ0|t = q0|t1 , and 1

its path KL vanishes because P[tθ1⋆,t2 ] = P[t⋆1 ,t2 ] : both are initialized at qt2 and run with the b t . Hence NELBO[t ,t ] (θ⋆ ) = H(q0 ), which with the floor of Lemma 3 marginal reverse rate Q 1 2 gives inf θ NELBO[t1 ,t2 ] (θ) = H(q0 ). Lemma 3 also writes NELBO[t1 ,t2 ] (θ⋆ ) − H(q0 ) as a sum of two non-negative gaps, the expected ELBO gap and KL(q0 ∥ pθ0⋆ ). Since this sum is zero, both vanish a.e., so pθ0⋆ = q0 and the ELBO is tight for q0 -a.e. z0 , proving (36). Finally, by the terminal mixing and convergence Assumption 1, the joint law of (Z0 , Zt ) converges on the finite state space, where entropy and mutual information are continuous and bounded, so the same conclusion passes to t2 ↑ T , including T = ∞.

5.5

An alternative proof of the Oracle Distance theorem

The assembly proof of §5.1 relies on the regularity and fixed-support conditions of Assumption 3, which the theorem itself does not need. We close the section with a proof that avoids them: it needs only the reverse-time factorization of the joint noising law, the KL chain rule, and Girsanov’s formula, at the price of manipulating laws on the path space rather than rates. The comparison between the two proofs is also instructive: the KL chain rule plays below the role that the Pythagorean split played at the rate level, and since Φ is the KL rate between Poisson jump intensities, Proposition 2 may be read as the infinitesimal, jump-by-jump version of the chain rule. Two reminders from §4.2 before the proof. First, the path-laws (Definition 1): the noising process defines the clean-conditioned law P[t⋆1 ,t2 ]|0 (· | z0 ), a kernel from clean data to trajectories; ⋆ prefixing z0 ∼ q0 gives the joint P0,[t , and marginalizing z0 out gives P[t⋆1 ,t2 ] , while the model 1 ,t2 ] θ b θt , pθ ). Second, law P[tθ1 ,t2 ] and its joint P0,[t are built in reverse time from the triple (pθt2 , Q 0|t1 1 ,t2 ] the descriptions: by Kolmogorov uniqueness, each law is equally specified from either endpoint of the window, and the backward description of the joint noising law is what makes the proof below work. To avoid the nuances of path densities, we integrate against these laws directly, writing P (dz[t1 ,t2 ] ) in the path variable while keeping plain pmf notation for the finite data factor z0 (§4.2), 37

dP with Radon–Nikodym derivatives dP ′ in place of density ratios. In particular, the joint noising and model laws are, as per their defining factorizations, ⋆ (z0 , dz[t1 ,t2 ] ) = q0 (z0 ) P[t⋆1 ,t2 ]|0 (dz[t1 ,t2 ] | z0 ), P0,[t 1 ,t2 ] θ P0,[t (z0 , dz[t1 ,t2 ] ) = P[tθ1 ,t2 ] (dz[t1 ,t2 ] ) pθ0|t1 (z0 | zt1 ). 1 ,t2 ]

Direct proof of Theorem 3. The support hypotheses ensure that every term below is well defined: P[t⋆1 ,t2 ] ≪ P[tθ1 ,t2 ] by Theorem 2, and together with q0|t1 ≪ pθ0|t1 from the hypotheses, we get ⋆ θ P0,[t ≪ P0,[t . This then ensures that the Radon–Nikodym derivatives below are well defined. 1 ,t2 ] 1 ,t2 ] First, negating the ELBO definition (17) and averaging over z0 ∼ q0 expresses the average NELBO as a Radon–Nikodym identity thanks to the first joint factorization above, giving Z NELBO[t1 ,t2 ] (θ) =

log

dP[t⋆1 ,t2 ]|0 (· | z0 ) θ dP0,[t (z0 , ·) 1 ,t2 ]

⋆ (z0 , dz[t1 ,t2 ] ). (z[t1 ,t2 ] ) P0,[t 1 ,t2 ]

⋆ Since q0 is the z0 -marginal of P0,[t and H(q0 ) = −Eq0 log q0 (Z0 ), subtracting H(q0 ) inserts 1 ,t2 ] the factor q0 (z0 ) into the numerator. By the same factorization, the integrand becomes the joint Radon–Nikodym derivative, so

Z NELBO[t1 ,t2 ] (θ) − H(q0 ) =

log

⋆ dP0,[t 1 ,t2 ] θ dP0,[t 1 ,t2 ]

⋆ ⋆ θ dP0,[t = KL P P 0,[t1 ,t2 ] 0,[t1 ,t2 ] . 1 ,t2 ]

Next, by the Markov property, the conditional law of Z0 given the whole window trajectory Z[t1 ,t2 ] depends on that trajectory only through its left endpoint Zt1 : Law(Z0 | Z[t1 ,t2 ] ) = q0|t1 (· | Zt1 ). Hence the joint noising law also factors as ⋆ P0,[t (z0 , dz[t1 ,t2 ] ) = P[t⋆1 ,t2 ] (dz[t1 ,t2 ] ) q0|t1 (z0 | zt1 ). 1 ,t2 ]

(38)

With both joint laws factored in the same order, the Radon–Nikodym derivative factorizes, ⋆ dP0,[t 1 ,t2 ] θ dP0,[t 1 ,t2 ]

(z0 , z[t1 ,t2 ] ) =

dP[t⋆1 ,t2 ] dP[tθ1 ,t2 ]

(z[t1 ,t2 ] )

q0|t1 (z0 | zt1 ) pθ0|t1 (z0 | zt1 )

⋆ P0,[t -a.s. 1 ,t2 ]

⋆ Taking logs and integrating against P0,[t splits the joint divergence into a window term and an 1 ,t2 ] endpoint-kernel term, h i ⋆ θ ⋆ θ θ KL P0,[t P = KL P P + E KL q (· | z ) p (· | z ) , t t q 0|t t 1 1 ,t ] 0,[t ,t ] [t ,t ] [t ,t ] 0|t 1 1 2 1 2 1 2 1 2 1 1

the last expectation being over zt1 ∼ qt1 , the time-t1 marginal of P[t⋆1 ,t2 ] ; this proves (25). Finally, as in the previous proof, the path KL is the sum of the terminal prior KL and the integrated jump-rate divergence by Theorem 2, giving (26).

Remark 1 (The arrow of time and independent mechanisms). These results have an intriguing connection to the independent mechanisms principle from causal inference [SJP+ 12]. At the pathlaw level there is no asymmetry between the two time directions (a process is Markov in forward time if and only if it is Markov in reverse time). In the forward direction, the description factorizes into two independent mechanisms: an initial condition (the data law q0 ) and a dynamical law (the generator Qt ), chosen without knowledge of each other — the noising mechanism contains no information about the data. Backward, this independence is destroyed: by (14), the reverse generator b t (x, y) = Qt (y, x) qt (y) requires the marginals qt and hence the data law. Janzing et al. [JCS16] Q qt (x) derive the thermodynamic arrow of time from precisely this independence of initial state and dynamics. A similar logic — asymmetry through restriction of the model class — underlies causal direction detection in time series: a process representable by mechanisms driven by independent noise in one direction generically admits no such representation in the other, which is what renders the direction identifiable [PJGS09, PJS13]. Our view on discrete diffusion is an instance of this asymmetry: the forward process belongs to the small class of data-independent, position-factorized mechanisms, whereas its reversal, whenever q0 is correlated across positions, exists only in the larger class of data-dependent generators — and the purpose of training is to supply that dependence. The present framework explicitly quantifies the breakdown: the mutual information I(Z0 ; Zt ) measures how much data-dependence the reverse generator must carry at time t, and Theorem 4 identifies d the oracle integrand Jt⋆ = − dt I(Z0 ; Zt ) as the rate at which the forward mechanism’s independence leaks into the required reverse mechanism. The violation of independence under reversal thus has a conserved total — the universal floor H(pdata ) of Corollary 3, the total mechanism-dependence any reverse process must reconstruct — while the choice of noising process controls only when along the trajectory it is paid (Figure 2).

5.6

A Bregman-divergence and generator-matching view

These results are instances of standard Bregman identities. A strictly convex, differentiable potential F generates DF (a, b) = F (a) − F (b) − F ′ (b)(a − b). The local rate divergence Φ is generated by the negative-entropy potential F (u) = u log u − u,

F ′ (u) = log u.

For any Bregman divergence DF , the conditional mean B ⋆ = E[A | G] minimizes E[DF (A, b(G))] over G-measurable predictors and satisfies E[DF (A, B)] = E[DF (A, B ⋆ )] + E[DF (B ⋆ , B)] [BGW05]. Lemma 2 specializes this identity to F (u) = u log u − u; Theorem 5 and Proposition 2 apply it with b t (zt , y | Z0 ) and G = Zt . The optimal rate (32) is therefore the marginal generator obtained A=Q by posterior-averaging the clean-conditioned generators, as in Generator Matching [HHY+ 25]. It is also the jump-process analogue of flow matching [GRS+ 24]: the regressed object is a reverse intensity and the divergence is Φ rather than the squared Euclidean norm. The following remark makes this correspondence exact. Remark 2 (The continuous-diffusion analogue). Nothing in the Oracle Distance is specific to finite state spaces. Let the noising process be a Markov diffusion dZt = ft (Zt ) dt + σt (Zt ) dWt on Rd , with Z0 ∼ q0 , (Wt )t a standard Brownian motion, and at := σt σtT . Under standard smoothness and nondegeneracy assumptions, the reverse-time process is again a diffusion with the same diffusion matrix and, written at forward time t, the reversed drift P bbt (z) = −ft (z) + ∇· at (z) + at (z) ∇ log qt (z), (∇· at )i = j ∂j (at )ij ; 39

conditioning on the clean data replaces the marginal by the forward kernel, bbt (z | z0 ) = −ft (z) + ∇· bt , at (z) + at (z) ∇ log qt|0 (z | z0 ). The score ∇ log qt thus plays the role of the marginal reverse rate Q and ∇ log qt|0 that of the computable clean-conditioned target. The two ingredients then translate verbatim. For reverse processes sharing the diffusion matrix, Girsanov’s theorem plays the role of Theorem 2, with a quadratic drift energy in place of Φ: ⋆ θ KL P[0,T ] P[0,T ] = KL qT

pθT

1 + 2

Z T

h Eqt

i bbt − bbθ T a−1 bbt − bbθ (Zt ) dt. t t t

The projection of Theorem 5 becomes a weighted L2 projection, whose optimizer among Zt measurable drifts is again the conditional mean, h i bbθ⋆ (z) = E bbt (Zt | Z0 ) Zt = z = bbt (z), t the drift form of Fisher’s identity ∇ log qt (z) = E ∇ log qt|0 (z | Z0 ) Zt = z . With matched endpoints, the population identity then reads exactly as Corollary 1, 1 NELBO[0,T ] (θ) = H(q0 ) + 2

Z T

h Eqt

i 2 dt, σt−1 bbt − bbθt (Zt )

with H(q0 ) now the differential entropy. Training on a window [t1 , t2 ] ⊂ (0, T ) with a terminal prior pθt2 and a reconstruction decoder pθ0|t1 gives, exactly as in Theorem 3, h i NELBO[t1 ,t2 ] (θ) − H(q0 ) = Eqt1 KL q0|t1 (· | zt1 ) pθ0|t1 (· | zt1 ) + KL P[t⋆1 ,t2 ] P[tθ1 ,t2 ] , with the terminal mismatch KL qt2 pθt2 again sitting inside the path KL. Table 1 collects the dictionary.

finite-state CTMC

continuous diffusion

Forward infinitesimal object

generator Qt (x, y)

generator ft · ∇ + 12 at :∇2

True reverse object

b t (zt , y) reverse rate Q

reverse drift bbt (z), or score ∇ log qt (z)

Tractable conditional target

b t (zt , y | z0 ) Q

bbt (z | z0 ), or ∇ log qt|0 (z | z0 )

Path-KL divergence

Φ(a, b) = a log ab − a + b

1 T −1 2 (u − v) at (u − v)

Path-KL formula

Girsanov’s theorem

Population optimizer

Girsanov formula and relative entropy (Theorem 2) h i b t (zt , y | Z0 ) Zt = zt E Q

NELBO meaning

data entropy + reverse path KL, plus the two boundary mismatches

h i E bbt (z | Z0 ) Zt = z

Table 1: The discrete/continuous diffusion dictionary: the same path structure throughout, with jump rates and the rate divergence Φ replaced by reverse drifts and the quadratic Girsanov energy.

Sequence modeling

In this section, Q denotes sequence-level rates and R single-token rates. Forward noising factors over positions, but correlations in the clean-data law mean that the unconditional noisy marginal qt generally does not. Only the conditional kernel qt|0 (· | z0 ) factors.

6.1

Token-factorizable noising processes

We first state the CTMC result behind the token-wise formulas. Let Xi be finite and X := X1 × · · · × XL . We recall that a time-inhomogeneous CTMC on any given finite X has generator Qt and transition kernel qt|s (y | x) if and only if they satisfy the forward Kolmogorov equation X ∂t qt|s (y | x) = qt|s (r | x)Qt (r, y), qs|s (y | x) = 1{x = y}. r∈X

Theorem 6 (Product CTMCs). Suppose coordinate 1 ≤ i ≤ L has generator Rti on Xi , with i . Define the product kernel transition kernel qt|s qt|s (y | x) :=

L Y

i qt|s (y i | xi ),

x, y ∈ X .

i=1

Then, its unique infinitesimal generator is given by Qt (x, y) =

L X

(Qt )i (x, y),

(Qt )i (x, y) = 1{x−i = y −i } Rti (xi , y i ),

i=1

where x−i denotes the vector of all coordinates of x excluding the i-th. Conversely, any process with generator of the displayed form has this product transition kernel, i.e., it factors over positions. Proof. First, Qt is a generator: if x ̸= y, then Qt (x, y) ≥ 0, since the only possible nonzero terms are off-diagonal entries of the coordinate generators. Moreover, X

Qt (x, y) =

L X X

1{x−i = y −i } Rti (xi , y i ) =

i=1 y∈X

y∈X

L X X

Rti (xi , u) = 0.

i=1 u∈Xi

Thus the diagonal entries are the negative row sums of the off-diagonal entries. Second, it remains to check the Kolmogorov equation. Since each coordinate kernel satisfies the Kolmogorov equation for Xi , X i i ∂t qt|s (y i | xi ) = qt|s (u | xi )Rti (u, y i ), u∈Xi

differentiating the full kernel product gives ∂t qt|s (y | x) =

L X i=1

i ∂t qt|s (y i | xi )

L X Y j X Y j i qt|s (y j | xj ) = qt|s (u | xi )Rti (u, y i ) qt|s (y j | xj ). i=1 u∈Xi

j̸=i

Reciprocally, from the definitions of qt|s and of Qt , we have X

L L XY X j j j qt|s (z | x)Qt (z, y) = qt|s (z | x ) 1{z −i = y −i }Rti (z i , y i ) z∈X j=1

z∈X

L X X

i=1

1{z

−i

L Y j =y } qt|s (z j | xj )Rti (z i , y i ) −i

i=1 z∈X

L X X

j=1 i (u | xi ) qt|s

i=1 u∈Xi

Y j qt|s (y j | xj )Rti (u, y i ), j̸=i

which is exactly the derivative computed above. The initial condition also holds: qs|s (y | x) =

L Y

1{xi = y i } = 1{x = y}.

i=1

Therefore qt|s solves the Kolmogorov forward equation with generator Qt . By uniqueness of solutions to finite-dimensional linear ODEs, this is the transition kernel associated with Qt . For the converse, any process with a generator of the displayed sum form solves the same forward Kolmogorov equation; its solution is unique, and the product kernel above is one such solution, so the transition kernel necessarily factors over positions. Almost all instances of discrete diffusion modeling on sequences [HNJ+ 21, AJH+ 21, CBDB+ 22, LME24, SHW+ 24, vRFD+ 25, SSP+ 25, SDG+ 25, SLY+ 26] are product CTMCs like the ones described above, i.e., the forward noising process is token-wise independent. Since the clean-data law q0 is generally correlated across positions, neither the unconditional marginals qt nor the unconditional backward kernels factor. But if we stay clean-conditioned, then the reverse process does factor: for s < t, Bayes’ rule gives L q i (z i | z i )q i (z i | z i ) L Y qs|0 (zs | z0 )qt|s (zt | zs ) Y s 0 t|s t s|0 s i qs|t,0 (zs | zt , z0 ) = = = qs|t,0 (zsi | zti , z0i ). i (z i | z i ) qt|0 (zt | z0 ) qt|0 t 0 i=1

i=1

Thus, we can clean-conditioned backward Qapplyi the product-CTMC theorem to the factorized b t )i (· | z0 ). For a b t (· | z0 ) = PL (Q q to obtain the backward bridge rates Q process qs|t,0 = L i=1 s|t,0 i=1 jump from the noisy sequence zt to the sequence y with y i ̸= zti , the corresponding true token-wise backward bridge rate is q i (y i | z0i ) b t )i (zt , y | z0 ) = 1{z −i = y −i }Rti (y i , zti ) t|0 . ai (t, z0 , zt , y) := (Q t i (z i | z i ) qt|0 t 0 These will be the target rates appearing inside the token-wise CTMC ELBO loss.

6.2

Three exact token-wise factorizations of the reverse rate

For the remainder of these notes, let z0 = (z01 , . . . , z0L ) ∈ X , 42

z0i ∈ Xi ,

(39)

where z0i is the clean token at position i, and eℓ denotes the ℓ-th element of the per-position vocabulary Xi (so ℓ ranges over 1 ≤ ℓ ≤ |Xi |). The forward kernel is assumed to factor over tokens, but the data distribution q0 need not. The following are three exact factorizations of the oracle reverse rate and are thus the starting point for the token-wise ELBO. Fix a sequence position i and a candidate sequence y with y i ̸= zti . The clean-token posterior at position i is πi⋆ (eℓ | zt , t) := q(Z0i = eℓ | Zt = zt ). (40) This is the standard denoiser target. Bayes’ rule isolates the analytic i-th-token forward kernel from the remaining sequence: P Q j j j i = e ,Z = z ) i =e q0 (z0 ) q(Z z :z j qt|0 (zt | z0 ) 0 t t ℓ ℓ 0 0 πi⋆ (eℓ | zt , t) = = P Q j j j q(Zt = zt ) z q0 (z0 ) j q (zt | z0 ) 0

t|0

P j j j −i Q i (z i | e ) qt|0 ℓ t j̸=i qt|0 (zt | z0 ) z0−i q0 (eℓ , z0 ) , =P P j j j −i Q i (z i | e ) q q (z | z ) −i q0 (er , z0 ) r t t 0 j̸=i t|0 r t|0 z 0

where (eℓ , z0−i ) is the sequence with i-th token eℓ and the remaining tokens from z0 . It turns out that the common factor given by the sum over z0−i can be interpreted in simple terms, motivating the following definition. Define the cavity law µ⋆i (eℓ | zt−i , t) := q(Z0i = eℓ | Zt−i = zt−i ).

(41)

It conditions on the noisy context zt−i but not the local noisy token zti . Expanding by Bayes’ rule, P −i −i q(Z0i = eℓ , Zt−i = zt−i ) z0 :z0i =eℓ q(Z0 = z0 , Zt = zt ) −i ⋆ µi (eℓ | zt , t) = = P −i −i q(Zt−i = zt−i ) z0 q(Z0 = z0 , Zt = zt ) P j j j −i Q j̸=i qt|0 (zt | z0 ) z0−i q0 (eℓ , z0 ) . =P P j j j −i Q j̸=i qt|0 (zt | z0 ) r z −i q0 (er , z0 ) 0

Thus, the denoiser is the cavity law reweighted by the forward kernel on the local token: i πi⋆ (eℓ | zt , t) ∝ µ⋆i (eℓ | zt−i , t) qt|0 (zti | eℓ ).

(42)

We may interpret this relation as the update that the denoiser given only the noisy context zt−i (the cavity law) obtains after observing as well the local noisy token zti . That required update turns out to be given by the local forward kernel. Equivalently, for any f , i h i (z i | e ) Eµ⋆ (eℓ |z −i ,t) f (eℓ ) · qt|0 ℓ t t i h i . (43) Eπi⋆ (eℓ |zt ,t) [f (eℓ )] = i (z i | e ) Eµ⋆ (eℓ |z −i ,t) qt|0 ℓ t i

ℓ

Define lastly the i-th concrete score as s⋆i (zti , y i | zt−i , t) :=

q(Zti = y i | Zt−i = zt−i ) qt (y i , zt−i ) , = qt (zt ) q(Zti = zti | Zt−i = zt−i )

(45)

where the second equality uses Bayes’ rule to cancel the common Zt−i = zt−i conditioning. It turns out that the three quantities πi⋆ , µ⋆i , and s⋆i subsume the different ELBO parameterizations in the literature, and they each are sufficient to express the unconditional reverse rate in terms of forward laws. To see this, we compute the ELBO-optimal rate for each token position i, which, by Theorem 5, is the average of the reverse conditional rate (39): h i b t )i (zt , y) = Eq (z |z ) (Q b t )i (zt , y | z0 ) (Q 0|t 0 t −i −i i i i qt|0 (y | z0 ) = Eq0|t (z0 |zt ) 1{zt = y } Rt (y , zt ) qt|0 (zt | z0 ) "Q j j j # j qt|0 (y | z0 ) = 1{zt−i = y −i } Rti (y i , zti ) Eq0|t (z0 |zt ) Q j j j j qt|0 (zt | z0 ) = 1{zt−i = y −i }Rti (y i , zti ) The last two steps use qt|0 =

i (y i | e ) qt|0 ℓ ⋆ πi (eℓ | zt , t) i . i qt|0 (zt | eℓ ) ℓ

j −i = z −i , which cancel all j ̸= i factors. t j qt|0 and y

Remark 3 (Support of the denoiser law). In the last step of the previous calculation and throughout the remainder of the paper, we use the convention that terms with both numerator πi⋆ (eℓ | i (z i | e ) = 0 contribute zero. In reality, Bayes’ rule excludes such zt , t) = 0 and denominator qt|0 ℓ t terms: they would only arise from incorrectly applying Bayes’ rule at a token zti outside of the support of Zti , that is, a.s. unreachable from clean data via the forward kernel. In particular, if we define the reverse support i Si (zti , t) := {ℓ : qt|0 (zti | eℓ ) > 0}, (46) we have supp πi⋆ (· | zt , t) ⊆ Si (zti , t).

(47)

Thus, the i-th optimal rate depends on the clean-data posterior only through its i-th marginal πi⋆ or equivalently µ⋆i or s⋆i , enabling per-token rate parameterizations. This is precisely what allows factorized, per-token modeling of the reverse rate, making discrete diffusion tractable. To conclude, we now see how the token-wise unconditional reverse rates can be written in terms of πi⋆ , µ⋆i , or s⋆i via i (y i | e ) qt|0 ℓ πi⋆ (eℓ | zt , t) i i qt|0 (zt | eℓ ) ℓ h i i (y i | e ) Eµ⋆ (eℓ |z −i ,t) qt|0 ℓ t h i = 1{zt−i = y −i } Rti (y i , zti ) i i (z i | e ) Eµ⋆ (eℓ |z −i ,t) qt|0 ℓ t

b t )i (zt , y) = 1{z −i = y −i } Ri (y i , z i ) (Q t t t

(49)

= 1{zt−i = y −i } Rti (y i , zti ) s⋆i (zti , y i | zt−i , t). 44

(48)

(50)

The second equality uses (43); the third uses (44). These are exact factorized forms of the unconditional reverse rate, in the denoiser, cavity, and score coordinates respectively. Each leads to a different expression of the reverse rates by choosing a neural network head to model the respective law, and thus, by the Kolmogorov forward equation, each determines a unique reverse CTMC. In particular, the coordinates differ only in the algebra of the neural network head-to-rate map, given by the three displayed equations above.

6.3

Parameterizations in the literature

The literature models the reverse process in one of the three coordinates of Theorem 7; the population optimum of each is the matching law πi⋆ , µ⋆i , or s⋆i . For each we record the model reverse rate aθi that enters the CTMC ELBO and the finite-step ancestral sampler it induces (whose parallel-token error is isolated in Figure 6). The samplers share one template: a tractable reverse step uses the token-factorized ansatz L Y pθ (zsi | zt ), (51) pθs|t (zs | zt ) := i=1

whose factors are the exact coordinate marginals q(zsi | zt ) at the oracle and approximate only through the forced product over positions; as s ↑ t simultaneous coordinate changes have probability O((t − s)2 ), so all three coordinates induce the same reverse-CTMC generator. Denoiser [AJH+ 21, HNJ+ 21, CBDB+ 22, SAS+ 24, SHW+ 24]. A head πiθ ≈ πi⋆ gives the reverse rate i (y i | e ) X qt|0 ℓ −i −i i i i θ θ πi (eℓ | zt , t) i ai,denoiser (t, zt , y) := 1{zt = y } Rt (y , zt ) . (52) i qt|0 (zt | eℓ ) ℓ

πiθ (eℓ | zt , t)

ℓ

i (y | e ) qs|0 ℓ i (z i | e ) qt|0 ℓ t

(53)

Continuing the discussion from Remark 3, we again need the convention that 0/0 := 0. However, for a learned denoiser it is not guaranteed that supp πiθ (· | zt , t) ⊆ Si (zti , t). Contrary to this, the cavity rate (55) never divides by per-token likelihood, and out-of-support i (z i | e ) = 0. Thus, a cavity head is tokens vanish on their own thanks to the Bayes weight qt|0 ℓ t automatically confined to Si , whereas a denoiser head must learn to vanish off Si (or be masked to it by hand) to keep the rate finite, already pointing to the practical difficulty of training denoiser heads in discrete diffusion. This also reflects the importance of fixing the jump support: when Si (zti , t) does not vary on [t1 , t2 ], the out-of-support tokens can be excluded once and for all. In masked diffusion the support is the whole vocabulary at a masked position and the single clean token at a revealed one (which has no reverse jump), so the restriction is automatic; in uniform diffusion it is the entire vocabulary for every t > 0 and collapses only at t = 0, so a denoiser head must concentrate on the true token exactly as t ↓ 0, which is precisely where an uninformative one diverges (Proposition 6). 45

ℓ

(54) and hence the reverse rate aθi,cavity (t, zt , y) := 1{zt−i = y −i } Rti (y i , zti )

mθi (y i | zt , t) . mθi (zti | zt , t)

(55)

mθi,t→s (y | zt , t) mθi (zti | zt , t)

(56)

Although often presented as a clean-token denoiser, this head is ELBO-optimized by the cavity law µ⋆i , not by πi⋆ . Score [MCSE22, SYD+ 23, LME24, ONX+ 25, ZZZ+ 25]. A head sθi ≈ s⋆i gives the reverse rate aθi,score (t, zt , y) := 1{zt−i = y −i } Rti (y i , zti ) sθi (zti , y i | zt−i , t), (57) whose path loss is exactly SEDD’s diffusion-weighted denoising score entropy [LME24], with the forward rate Rti a forced (not auxiliary) weight; we derive this, with its masked RADD specialization, in §7. For a finite step, a score head is converted to a denoiser or cavity head (Proposition 3) and sampled as one of those. Converting between coordinates. Each head attains the oracle rate when it matches its target πi⋆ , µ⋆i , or s⋆i , so all three reach the same ELBO optimum under sufficient capacity; off the optimum they are interconvertible wherever the relevant likelihood is positive. This equivalence, and the penalty for reading a head in the wrong coordinate without converting, are confirmed numerically in Figure 5 and Table 6. i (z i | e ) and let q i ≡ [ q i (w | e ) ] Proposition 3 (Coordinate conversions). Write wℓ := qt|0 ℓ ℓ ℓ,w be t t|0 t|0 the single-token forward channel; throughout, distributions are row vectors acting on kernels from the left. The denoiser, cavity, and score heads are interconverted by

µθi (eℓ ) ∝

πiθ (eℓ ) , wℓ

πiθ (eℓ ) ∝ µθi (eℓ ) wℓ ,

sθi =

mθi (y i ) , mθi (zti )

i mθi = µθi qt|0 ,

(58)

with mθi (y) ∝ sθi (zti , y) when starting from a score. The denoiser↔cavity maps require wℓ > 0; the i to have full row rank and returns a probability vector iff µθ ≥ 0. score→cavity map requires qt|0 i Proof. The cavity-to-denoiser map is the local Bayes update (42), πiθ ∝ µθi wℓ ; inverting it gives µθi ∝ πiθ /wℓ wherever wℓ > 0. By (45) the score is the ratio mθi (y i )/mθi (zti ) of the cavity-induced i , so either distribution head determines it. Conversely the score fixes mθ up marginal mθi = µθi qt|0 i θ θ i θ to scale, mi (y) ∝ si (zt , y) (normalized to a probability vector), and µi is then the unique solution i whenever q i has full row rank. of the linear system mθi = µθi qt|0 t|0 46

The maps into the score are always available, while recovering a distribution coordinate from a score needs an invertible channel and a non-negative solution. Where some wℓ = 0, the visible states of masked diffusion, the denoiser cannot recover the corresponding cavity mass. We instantiate all of these conversions in closed form for the interpolating family in Corollary 5. from \ to

denoiser πiθ

cavity µθi

denoiser πiθ

—

∝ πiθ (eℓ )/wℓ (58)∗

score sθi P θ i i ℓ πi (eℓ ) qt|0 (y | eℓ )/wℓ

cavity µθi

∝ µθi (eℓ ) wℓ (42)

—

mθi (y i )/mθi (zti )

score sθi

via µθi , then (42)

i solve† mθi = µθi qt|0

—

i (z i | e ) and mθ (y i ) = Table 2: Conversions between coordinates (Proposition 3), with wℓ := qt|0 ℓ t i P θ i (y i | e ). ∗ Wherever w > 0. † Recover mθ (·) ∝ sθ (z i , ·) by normalization, then solve µ (e ) q ℓ ℓ i i t ℓ i ℓ t|0 i , arranged as a matrix with normalized the linear system; the inverse is unique if and only if qt|0 rows, has full row rank.

The section’s main theorem records the conclusion: one optimizer, three exact coordinates. Theorem 7 (The reverse rate in three coordinates). For a product-CTMC noising process the b t (Theorem 5). On a path ELBO (18) is uniquely minimized by the unconditional reverse rate Q −i −i i i one-token jump zt → y (y = zt , y ̸= zt ), through the denoiser, cavity, and score laws πi⋆ , µ⋆i , s⋆i of (40), (41), (45), it admits the three equal forms (all carrying 1{zt−i = y −i }) # " i qt|0 (y i | eℓ ) i i i b t )i (zt , y) = Rt (y , zt ) Eπ⋆ (e |z ,t) (48), denoiser (Q ℓ t i (z i | e ) i qt|0 ℓ t h i i (y i | e ) Eµ⋆ (eℓ |z −i ,t) qt|0 ℓ t h i (49), cavity = Rti (y i , zti ) i i (z i | e ) Eµ⋆ (eℓ |z −i ,t) qt|0 ℓ t i

= Rti (y i , zti ) s⋆i (zti , y i | zt−i , t)

(50), score.

Table 3 collects the dictionary: the learned target, the induced model rate, the literature methods each coordinate corresponds to, and the failure mode each invites. The practical upshot is a single rule, use a head in the coordinate its loss optimizes, or convert analytically, which the rest of the paper makes concrete: a denoiser head fed into the cavity ansatz (56) silently optimizes the cavity law µ⋆i rather than the denoiser πi⋆ ; the two are related by the closed-form Bayes update (58). We conclude the section with a calibration result for the cavity ELBO which can be of extreme practical utility, since it allows end-to-end testing of the ELBO implementation at initialization. Proposition 4 (Uninformative cavity NELBO). Fix [t1 , t2 ] ⊂ (0, T ), assume all positions share a common clean vocabulary Xi = Y with V = |Y|, and let the cavity law be uniform on Y: µθi (eℓ | zt−i , t) ≡

1 . V

Let punif-fwd be the uniform law on Y L , let punif-fwd be its forward image under qt|0 , and choose the t 0 θ unif-fwd θ unif-fwd boundary laws pt2 = pt2 , p0|t1 = p0|t1 . Then, for every sequence z0 ∈ Y L , NELBO[t1 ,t2 ] (θ; z0 ) = L log V.

(59)

Proof. It suffices to treat one position. The Y-uniform cavity head induces the noisy marginal X qt|0 (y | eℓ ) = punif-fwd (y), mθ (y | zt , t) = V1 t ℓ

so by (55) the model rate equals the oracle reverse rate (32) of the noising process punif-fwd (the forward kernel started from punif-fwd ). With the matching boundary laws pθt2 = punif-fwd and pθ0|t1 = t2 0 punif-fwd , Corollary 3 applied to punif-fwd makes the ELBO tight 0|t1 NELBO[t1 ,t2 ] (θ; eℓ ) = − log punif-fwd (eℓ ) = log V, 0 and the L positions factor to give NELBO[t1 ,t2 ] (θ; z0 ) = L log V . This gives a calibration check only when the boundary laws are included as stated: an exactly uniform cavity head then has per-token NELBO log V . With different reconstruction or terminal priors, the path-rate calibration still holds but the full NELBO need not equal L log V . If the forward process has erased all information about z0 by t2 , symmetry makes the usual terminal law qt2 equal to punif-fwd for an i.i.d. token-wise kernel. t2 Denoiser

Cavity

Score

Symbol

πi⋆ (eℓ | zt , t)

µ⋆i (eℓ | zt−i , t)

Definition

q(Z0i = eℓ | zt )

Reverse rate† b θ )i (zt , y) (Q t

Sampling‡ pθ (zsi | zt )

i (z i | e ) qt|0 ℓ t

q(Z0i = eℓ | zt−i ) P θ i i ℓ µi (eℓ )qt|0 (y | eℓ ) P θ i i ℓ µi (eℓ ) qt|0 (zt | eℓ ) P θ i i ℓ µi (eℓ )qs|0 (zs | eℓ ) P θ i i ℓ µi (eℓ ) qt|0 (zt | eℓ )

s⋆i (zti , y i | zt−i , t) qt (y i , zt−i ) qt (zt )

Recovers

MDLM, D3PM, MD4 [AJH+ 21, HNJ+ 21, CBDB+ 22, SAS+ 24, SHW+ 24]

UDLM, GIDD, Duo [vRFD+ 25, GJS+ 26, SSP+ 25, SDG+ 25, SLY+ 26]

θ ℓ πi (eℓ )

i (y i | e ) qt|0 ℓ i (z i | e ) qt|0 ℓ t i (z i | e ) qs|0 ℓ s

sθi (zti , y i ) via denoiser/cavity SEDD, RADD, TCSM [MCSE22, SYD+ 23, LME24, ONX+ 25, ZZZ+ 25]

Table 3: The reverse rate in three coordinates (Theorem 7) and the matching ancestralsampling step. Reverse rates (52)–(57) and sampling steps (53),(56) are shown with arguments θ (· | zt , t) suppressed on Si (zti , t). The cavity entries use the cavity-induced noisy Pandθ πi supported θ i marginal mi (·) = ℓ µi (eℓ ) qt|0 (· | eℓ ) and its cross-time form mθi,t→s (54). † Reverse rates omit the common prefactor 1{zt−i = y −i } Rti (y i , zti ); ‡ sampling steps pθ (zsi = y | zt ) omit the common factor i (z i | y). qt|s t 48

Masked, uniform, and GIDD noising processes

The noisy state space Z includes a mask state m for masked and GIDD hybrid diffusion, whereas uniform diffusion uses the clean vocabulary Y; all positions here share this common vocabulary, of size V = |Y|. All processes below use a differentiable, decreasing signal schedule with α0 = 1 and α1 = 0. Write α′ βt := 1 − αt , λt := − t . αt

7.1

Common CTMC form

All processes’ forward kernels in this section factor over positions following the general CTMC product structure of Theorem 6. We use the true reverse rate ai from (39) and the denoiser and cavity model rates from (52) and (55), respectively. Below, we suppress unchanged sequence coordinates and write the candidate token y i as y. Leaving the boundary terms from §4 implicit, the CTMC NELBO path integrand is Jtθ (z0 , zt ) =

L X

Φ ai (t, z0 , zt , y), aθi (t, zt , y) ,

(60)

i=1 y∈Ni (zt )

with the neighborhood of z ∈ Z L at position i defined by Ni (z) := {y ∈ Z L : y −i = z −i , y i ̸= z i }. Here aθi denotes either model-rate coordinate above. Proposition 5 (Cavity ELBO for source-independent jump rates). Consider a product CTMC for which the off-diagonal rates at position i depend only on time and destination but not on the source token: Rti (y i , xi ) = γi (t, xi ), xi ̸= y i . Since the forward jump enters the reverse rate with its destination equal to the current noisy token, the relevant value below is γi (t, zti ). Fix z0 , zt , and define the true and model noisy marginals by i di (y i ) := qt|0 (y i | z0i ),

mi (y i ) := mθi (y i | zt , t).

b t (zt , · | z0 ) ≪ Q b θ (zt , ·), the Assuming di (zti ), mi (zti ) > 0 and the usual rate-support condition Q t per-position cavity-coordinate contribution to the negative ELBO integrand is cavity Ji,t =

γi (t, zti ) i i KL(d ∥ m ) + D d (z ) m (z ) , i i i i IS t t di (zti )

where DIS (p ∥ q) :=

(61)

p p − log − 1 q q

is the Itakura–Saito divergence. Consequently, the sequence-level cavity-coordinate integrand is Jtcavity (z0 , zt ) =

L X γi (t, z i ) i=1

t di (zti )

KL(di ∥ mi ) + DIS di (zti ) mi (zti ) .

(62)

Proof. Every token y i ̸= zti determines a unique sequence y ∈ Ni (zt ) with y −i = zt−i and y i in place of zti . Let x := zti and simplify the notation further by writing m(y) := mi (y) and d(y) := di (y). Equations (39) and (55), together with positive homogeneity of Φ and the source-independence assumption, give X d(y) m(y) cavity . Ji,t = γi (t, x) Φ , d(x) m(x) y̸=x P P Since y̸=x d(y) = 1 − d(x) and y̸=x m(y) = 1 − m(x) for d and m are probability vectors, we get X d(y) m(y) 1 − m(x) 1 − d(x) 1 X d(y)m(x) Φ = , − + d(y) log d(x) m(x) m(x) d(x) d(x) d(x)m(y) y̸=x y̸=x   X 1  d(x) d(y) d(x)  = −1+ d(y) log − (1 − d(x)) log d(x) m(x) m(y) m(x) y̸=x   d(y) 1 X d(x) d(x) = + − log − 1 d(y) log d(x) m(y) m(x) m(x) y̸=x

1 [KL(d ∥ m) + DIS (d(x) ∥ m(x))] . d(x)

Multiplying by γi (t, x) proves (61).

7.2

Masked diffusion

Let Z = Y ∪ {m}, where m is absorbing. The single-position marginal is i qt|0 (y | z0i ) = αt 1{y = z0i } + βt 1{y = m}.

(63)

As in the previous section, we will denote clean tokens by eℓ , where ℓ = 1, . . . , V and V = |Y|. For 0 ≤ s < t ≤ 1, the discrete transition kernel is αt αt i qt|s (y | x) = 1{y = x} + 1 − 1{y = m}, x, y ∈ Z. (64) αs αs Taking the continuous-time limit, the CTMC has only one kind of forward jump: Rt (eℓ , m) = λt . CTMC ELBO

(65)

The only nonzero true reverse jumps go from the mask to the clean token:

ai (t, z0 , zt , eℓ ) = λt

α′ αt 1{eℓ = z0i , zti = m} = − t 1{eℓ = z0i , zti = m}. βt βt

(66)

In particular, if zti ̸= m, there is no off-diagonal true reverse jump at that position. The induced noisy marginal satisfies mθi (m | zt , t) = βt ,

mθi (eℓ | zt , t) = αt µθi (eℓ | zt−i , t),

so the model reverse rate from m to eℓ is aθi,cavity (t, zt , eℓ ) = −

αt′ θ µ (eℓ | zt−i , t) 1{zti = m}. βt i 50

(67)

Corollary 4 (Masked: denoiser equals cavity). For masked diffusion the denoiser and cavity laws coincide on every masked position: if zti = m then πi⋆ (· | zt , t) = µ⋆i (· | zt−i , t).

(68)

Proof. At a masked position qt|0 (m | eℓ ) = βt does not depend on ℓ, so the local Bayes update (42) multiplies µ⋆i by a constant and renormalizes; it is therefore the identity, and πi⋆ = µ⋆i . The two coordinates then give the same masked reverse rate: a denoiser head πiθ yields aθi,denoiser (t, zt , eℓ ) = λt

V −1 X

πiθ (er | zt , t)

r=0

qt|0 (eℓ | er ) α′ = − t πiθ (eℓ | zt , t)1{zti = m}, qt|0 (m | er ) βt

which is (67) with µθi = πiθ . At an unmasked position zti = er the denoiser is the point mass δr rather than the cavity law, but the masked CTMC has no off-diagonal reverse jump out of an unmasked token, so the distinction never enters the ELBO or the sampler. At a masked position, the local jump divergence reduces exactly to cross-entropy: X α′ α′ α′ X Φ − t 1{eℓ = z0i }, − t µθi (eℓ | zt−i , t) = − t Φ 1{eℓ = z0i }, µθi (eℓ | zt−i , t) βt βt βt ℓ ℓ i αt′ h =− − log µθi (z0i | zt−i , t) . βt We conclude mask Ji,t (z0 , zt ) = −

h i αt′ 1{zti = m} − log µθi (z0i | zt−i , t) , βt

(69)

for both cavity and denoiser parameterizations. In particular, only masked positions contribute to the masked-diffusion CTMC ELBO. SEDD and RADD as special cases score by

In score coordinates, denoting the true clean-conditioned σi :=

i (y i | z i ) qt|0 0 i (z i | z i ) qt|0 t 0

and using the positive homogeneity of Φ, the per-jump path cost is Φ ai , aθi,score = Rti (y i , zti ) Φ(σi , sθi ) = Rti (y i , zti ) sθi − σi log sθi + σi (log σi − 1) .

(70)

After summing and averaging, this recovers SEDD’s denoising score entropy [LME24]. The last term σi (log σi − 1) is θ-independent, Rti gives SEDD’s weights, and Theorem 5 gives the projectedscore optimum qt (y i , zt−i ) . s⋆i = E[σi | Zt ] = qt (zt ) For masked diffusion the only active channel is m → eℓ with weight λt , and the masked concrete score factors as s⋆i (m, eℓ | zt−i , t) = (αt /βt ) µ⋆i (eℓ | zt−i , t), so the only learnable part is the cavity law, equal to πi⋆ on masked positions and, for fixed visible context, time-independent. This also exactly recovers RADD’s reparameterization [ONX+ 25]. 51

Prior KL term For a single position with clean token z0i , the natural masked prior is the deterministic mask law pnatural = δm . t2 Using (63) at a given position i,

i i KLMDM natural (t2 ) := KL qt2 |0 (· | z0 ) δm

( 0, = +∞,

αt2 = 0, αt2 > 0.

(71)

If instead the fixed prior is the law obtained by drawing a clean token uniformly and then applying the masked forward kernel to time t2 , then αt X punif-fwd = βt2 δm + 2 δeℓ . t2 V ℓ

The mask mass cancels in the KL, and only the clean token contributes: i i unif-fwd (· | z ) p = αt2 log V. KLMDM (t ) := KL q 0 t2 unif-fwd 2 t2 |0

(72)

This uniform-forwarded per-token prior gives a finite truncation correction for t2 < 1 and vanishes as t2 ↑ 1. By contrast, the deterministic mask prior has finite KL only at the fully masked endpoint. Sampling For a step from t = τj to s = τj−1 , evaluate the network at (zt , t). A visible token zti = eℓ remains eℓ . If zti = m, then pθ (Zsi = m | zt ) =

βs , βt

pθ (Zsi = eℓ | zt ) =

αs − α t θ µi (eℓ | zt , t), βt

(73)

where µθi can be replaced with πiθ since they coincide on masked positions. Once unmasked, a token cannot be revised. At the final (nonzero) level t1 , keep visible tokens and decode any remaining masks: ( δeℓ (·), zti1 = eℓ , i ẑ0 ∼ Cat πiθ (· | zt1 , t1 ) , zti1 = m.

7.3

Uniform diffusion

Uniform diffusion uses Z = Y and mixes the clean token with the uniform law 1/V : i qt|0 (er | eℓ ) = αt 1{er = eℓ } +

βt . V

(74)

For 0 ≤ s < t ≤ 1, αt i 1{er = eℓ } + qt|s (er | eℓ ) = αs

αt 1− αs

1 . V

(75)

The continuous-time limit gives, for r ̸= ℓ, Rt (er , eℓ ) =

λt , V

(76)

and Rt (er , er ) = −λt (V − 1)/V . Thus the process proposes uniform replacements at a rate λt . 52

CTMC ELBO is

For a current token zti and a proposed earlier token eℓ ̸= zti , the true reverse rate ai (t, z0 , zt , eℓ ) =

λt αt 1{eℓ = z0i } + βt /V . V αt 1{zti = z0i } + βt /V

(77)

The induced noisy marginal is mθi (eℓ | zt , t) = αt µθi (eℓ | zt−i , t) +

βt , V

so the cavity model rate for the zti → eℓ jump is aθi,cavity (t, zt , eℓ ) =

λt αt µθi (eℓ | zt−i , t) + βt /V , V αt µθi (zti | zt−i , t) + βt /V

eℓ ̸= zti .

(78)

If instead the network output is interpreted as a denoiser clean-token law πiθ (· | zt , t), the denoiser rate is αt 1{eℓ = er } + βt /V λt X θ πi (er | zt , t) , eℓ ̸= zti . (79) aθi,denoiser (t, zt , eℓ ) = V r αt 1{zti = er } + βt /V For the cavity coordinate, the true and model noisy marginals in Proposition 5 become di (eℓ ) = qt|0 (eℓ | z0i ) = mi (eℓ ) =

βt + αt 1{eℓ = z0i }, V

βt + αt µθi (eℓ | zt−i , t). V

For the current token zti , Proposition 5 applies with γi (t, eℓ ) = λt /V : UDM,cavity Ji,t =

αt 1{eℓ = z0i } + βVt βt i α 1{e = z } + log t ℓ 0 V αt 1{zti = z0i } + βVt eℓ ∈Y αt µθi (eℓ | zt−i , t) + βVt αt 1{zti = z0i } + βVt αt 1{zti = z0i } + βVt − log −1 + αt µθi (zti | zt−i , t) + βVt αt µθi (zti | zt−i , t) + βVt λt

(80)

Unlike masked diffusion, every observed token may be a corrupted version of any clean token, so every position can contribute to the loss. For the denoiser form, define gℓ⋆ :=

αt 1{eℓ = z0i } + βt /V , αt 1{zti = z0i } + βt /V

gℓθ :=

πiθ (er | zt , t)

αt 1{eℓ = er } + βt /V . αt 1{zti = er } + βt /V

Then UDM,denoiser Ji,t =

λt X Φ(gℓ⋆ , gℓθ ). V i eℓ ̸=zt

(81)

Prior KL term The fixed prior is the fully random token law, which is also the stationary law of the uniform diffusion kernel and thus coincides with the uniform forward law, pnatural (eℓ ) = punif-fwd (eℓ ) = t2 t2

1 , V

ℓ = 1, . . . , V.

Using (74), the per-token prior KL is unif KLunif natural (t2 ) = KLunif-fwd (t2 ) =

βt αt2 + 2 V

log (V αt2 + βt2 ) +

(V − 1)βt2 log βt2 . V

(82)

As t2 ↑ 1, KLunif natural (t2 ) → 0. Sampling For a step from t = τj to s = τj−1 , evaluate the network at (zt , t); unlike masked diffusion, every position may be resampled. The per-position law is the cavity law (56) with the uniform kernel (75), pθ (Zsi = eℓ | zt ) = qt|s (zti | eℓ )

αs µθi (eℓ | zt−i , t) + βs /V , αt µθi (zti | zt−i , t) + βt /V

(83)

the denominator being the noisy marginal mθi (zti | zt , t); a denoiser head enters the same step after the local Bayes conversion µθi ∝ πiθ /wℓ (58). At the final level t1 , decode each position from its clean-token posterior: ẑ0i ∼ Cat πiθ (· | zt1 , t1 ) . In contrast to the uniform cavity calibration (Proposition 4), the same uninformative head used as a denoiser makes the ELBO diverge. Proposition 6 (Uninformative denoiser ELBO diverges for uniform diffusion). For uniform diffusion (with α0 = 1), the uninformative denoiser head πiθ ≡ 1/V has, as t ↓ 0, h i V − 1 −α′ UDM,denoiser t (1 + o(1)), Eqt|0 Ji,t = V βt which is non-integrable at t = 0, so that, if V ≥ 2, the NELBO diverges as t1 ↓ 0 like NELBO[t1 ,t2 ] (θ) =

(V − 1)L 1 log + O(1). V βt1

Proof. We fix one position with clean token z0i , and study the true and model reverse rates ai (t, z0 , zt , eℓ ) and aθi (t, zt , eℓ ) using (77) and (79). On the one hand, the true rates are controlled by the ratio of forward kernels  βt /V i i   α +β /V = O(βt ) → 0, eℓ ̸= z0 = zt , λt αt 1{eℓ = z0i } + βt /V λt  αtt +βtt /V ai (t, z0 , zt , eℓ ) = = = Vβαt t + 1, eℓ = z0i ̸= zti , βt /V V αt 1{zti = z0i } + βt /V V   1, otherwise.

As αt → 1 and βt → 0, if the current token is “correct” zti = z0i , then all jumps away from it decay like βt . If the current token is “wrong” zti ̸= z0i , then the dominant jump is the one that corrects the wrong state by jumping to the actual clean token eℓ = z0i , whose rate diverges like −αt′ λt V α t i +1 = 1 + o(1) , ai (t, z0 , zt , z0 ) = V βt βt using λt αt = −αt′ . On the other hand, the model rate for πiθ ≡ 1/V is, up to the same factor λt /V , the average 1 X αt 1{eℓ = er } + βt /V . V r αt 1{zti = er } + βt /V By the same case-analysis with er in place of z0i and since eℓ ̸= zti (we can’t have self-jumps to the current token), we always have exactly one r such that eℓ = er ̸= zti , giving a divergent term αt /βt , while all the others remain bounded. This means that for every eℓ , an uninformative denoiser rate always contains a divergent term, and thus −α′ λt αt t aθi,denoiser (t, zt , eℓ ) = + O(1) = + O(−αt′ ). V βt V βt We now conclude by considering first the J terms where zti = z0i . On those, the trueto-model rate ratio is ai /aθi = [βt /(V αt + βt )]/[αt /βt + O(1)] = O(βt2 ), and thus Φ(ai , aθi ) = aθi 1 + O βt2 | log βt | . Together with the model-rate expansion above, this shows that each of the V − 1 destinations contributes its model rate up to an O(−αt′ ) remainder. Since q(zti = z0i | z0i ) = 1 − (V − 1)βt /V = 1 + O(βt ), we get h i −αt′ UDM,denoiser Eqt|0 Ji,t 1{zti = z0i } = (V − 1) + O(−αt′ ). V βt It remains to check the complementary event where zti ̸= z0i . The rate estimates above give −αt′ ai ai −αt′ −αt′ ai θ θ θ ai = O , ai = Θ =⇒ Φ(ai , ai ) = ai 1 − θ + θ log θ = O . βt βt βt ai ai ai Thus its V − 1 terms are each of order O(−αt′ /βt ), while the event itself has probability q(zti ̸= z0i | z0i ) = (V − 1)βt /V = O(βt ). Consequently, h i UDM,denoiser Eqt|0 Ji,t 1{zti ̸= z0i } = O(−αt′ ), so the event probability cancels the rate singularity. Putting the two contributions together gives h i V − 1 −α′ V − 1 −αt′ UDM,denoiser t + O(−αt′ ) = (1 + o(1)). Eqt|0 Ji,t = V βt V βt Integrating with −αt′ dt = dβt gives the per-position path term V V−1 log(1/βt1 ) + O(1). Summing over L positions gives the factor L. The downfall of the uninformative denoiser is that its jump support allows “wrong” jumps from clean tokens even near t = 0 and those kernel ratios diverge like 1/βt . In contrast, the cavity law probabilities receive a matching O(βt ) term from the Bayes factor to downweight those wrong jumps near the clean endpoint, thus keeping the cavity ELBO finite (illustrated in Figure 7). 55

7.4

Generalized interpolating discrete diffusion

Generalized Interpolating Discrete Diffusion (GIDD) [vRFD+ 25] additionally chooses a differentiable mixing distribution πt on Z (a row vector). Its single-position marginal is i = αt I + βt 1πt . qt|0

i (y | eℓ ) = αt 1{y = eℓ } + βt πt (y), qt|0

(84)

i by For 0 ≤ s < t ≤ 1 with αs > 0, define qt|s i i i . qt|s = qs|0 qt|0

Solving this identity within the same rank-one family gives i qt|s = αt|s I + βt|s 1πt|s ,

αt|s :=

αt , αs

βt|s πt|s := βt πt −

αt βs πs . αs

(85)

Direct multiplication verifies the Chapman–Kolmogorov equations. Expanding at t = s + ∆ gives the destination-rate vector and off-diagonal generator ht (y) := βt πt′ (y) + λt πt (y), Rt (x, y) = ht (y), y ̸= x. (86) P The diagonal enforces zero row sums. Since y ht (y) = λt , this agrees with the expansion of (85). Valid schedules require ht (y) ≥ 0 for every off-diagonal destination. Consistent with the standing interior assumptions, we additionally take the supports of πt and ht to be fixed on (0, 1); strictly positive entries are then bounded away from zero on compact interior windows. CTMC ELBO

For GIDD, the true and cavity reverse rates take the form ai (t, z0 , zt , y) = ht (zti )

qt|0 (y | z0i ) , qt|0 (zti | z0i )

and aθi,cavity (t, zt , y) = ht (zti )

y ̸= zti ,

mθi (y | zt , t) , mθi (zti | zt , t)

(87)

(88)

with noisy marginals mθi (y | zt , t) ≡ mi (y) = αt

µθi (eℓ | zt−i , t)1{y = eℓ } + βt πt (y).

ℓ

If πiθ (· | zt , t) is a denoiser clean-token law, then for y ∈ Z \ {zti }, aθi,denoiser (t, zt , y) = ht (zti )

πiθ (eℓ | zt , t)

ℓ

αt 1{y = eℓ } + βt πt (y) . αt 1{zti = eℓ } + βt πt (zti )

The true noisy marginal in Proposition 5 is di (y) = qt|0 (y | z0i ) = αt 1{y = z0i } + βt πt (y).

(89)

Since (86) has destination-dependent rate γi (t, y) = ht (y), the cavity-coordinate per-position NELBO is ht (zti ) GIDD,cavity KL(di ∥ mi ) + DIS di (zti ) mi (zti ) . Ji,t = (90) i di (zt ) For the denoiser-coordinate form, define, for y ̸= zti , gi⋆ (y) :=

di (y) αt 1{y = z0i } + βt πt (y) = di (zti ) αt 1{zti = z0i } + βt πt (zti )

and giθ (y) :=

πiθ (eℓ | zt , t)

ℓ

αt 1{y = eℓ } + βt πt (y) . αt 1{zti = eℓ } + βt πt (zti )

Then GIDD,denoiser Ji,t = ht (zti )

Φ gi⋆ (y), giθ (y) .

(91)

y̸=zti

Prior KL term distribution

The natural fixed prior for GIDD at the upper endpoint is the interpolating pnatural = πt2 . t2

For a clean token z0i and an arbitrary position i (by symmetry, all positions are equivalent), (84) gives the endpoint law. If πt2 (z0i ) > 0, the per-token prior KL is αt2 + βt2 πt2 (z0i ) i i KLGIDD + βt2 (1 − πt2 (z0i )) log βt2 . natural (t2 ; z0 ) = αt2 + βt2 πt2 (z0 ) log πt2 (z0i )

(92)

If πt2 (z0i ) = 0 while αt2 > 0, this KL is +∞ by support mismatch. The uniform-forwarded prior α punif-fwd (y) = Vt2 1{y ∈ Y} + βt2 πt2 (y) (a uniform clean token pushed through qt2 |0 ) instead gives, t2 since the noise floor βt2 πt2 cancels off the clean vocabulary, a per-token KL i KLGIDD unif-fwd (t2 ; z0 ) =

X y∈Y

αt 1{y = z0i } + βt2 πt2 (y) , αt2 1{y = z0i } + βt2 πt2 (y) log 2 αt2 /V + βt2 πt2 (y)

(93)

finite whenever αt2 > 0. The masked (αt2 log V ) and uniform (πt stationary, so KLunif-fwd = KLnatural ) cases are recovered by πt = δm and πt uniform. Remark 4 (GIDD uninformative denoiser ELBO). The divergence of Proposition 6 is not special to uniform diffusion. Fix a clean token x with πt (x) > 0 near t = 0 and consider the revealed event zti = z0i = x, whose probability tends to one. For any clean destination y ̸= x, the eℓ = y term of an uninformative denoiser head gives giθ (y) ≥

1 αt + βt πt (y) , V βt πt (x)

gi⋆ (y) =

βt πt (y) = O(βt ). αt + βt πt (x)

Hence Φ(gi⋆ (y), giθ (y)) = giθ (y)(1 + o(1)), exactly the gℓ⋆ = O(βt ) versus gℓθ = Θ(1/βt ) mechanism used in the proof of Proposition 6. This one destination already contributes at order ht (x)/(βt πt (x)), which has the exact primitive ht (x) π ′ (x) λt d βt πt (x) = t + = log . βt πt (x) πt (x) βt dt αt 57

Since βt πt (x)/αt → 0, its integral diverges as the lower endpoint tends to zero; for uniform πt , this is the same endpoint logarithm computed explicitly in Proposition 6. Thus any clean token to which πt assigns positive mass (and which occurs with positive data probability) produces the divergence, including uniform and full-support hybrids. i (y | e ) = α 1{y = Corollary 5 (Coordinate conversions for the interpolating family). For qt|0 t ℓ i eℓ } + βt πt (y) the matrix qt|0 (rows as distributions) is invertible on the clean vocabulary, and the score-to-cavity map is given by the affine transformation

µθi (eℓ ) =

mθi (eℓ ) − βt πt (eℓ ) , αt

sθ (z i , y) mθi (y) = P i θ t i ′ , y ′ si (zt , y )

(94)

a probability vector iff mθi (eℓ ) ≥ βt πt (eℓ ) for every ℓ. All conversions are then explicit (Table 4). P i (e | e ) = α µθ (e )+ Proof. The noise floor βt πt (y) is source-independent, so mθi (eℓ ) = r µθi (er )qt|0 r t i ℓ ℓ βt πt (eℓ ) decouples across ℓ; solving for µθi (eℓ ) gives (94), with the noisy marginal recovered from a i = α I + β 1π has full row rank for α > 0. score as in Proposition 3. The matrix qt|0 t t t t Uniform diffusion gives µθi (eℓ ) = (mθi (eℓ ) − βt /V )/αt , and masked diffusion gives µθi (eℓ ) = mθi (eℓ )/αt = (βt /αt ) sθi (m, eℓ ), recovering RADD [ONX+ 25]. This is the discrete analogue of empirical-Bayes deconvolution: the prior (cavity) is recovered by inverting the noising channel. from \ to

denoiser πiθ (eℓ )

cavity µθi (eℓ )

score sθi (zti , y)

denoiser πiθ

—

∝ πiθ (eℓ )/wℓ

mθi (y)/mθi (zti )

cavity µθi

∝ µθi (eℓ ) wℓ

—

mθi (y)/mθi (zti )

score sθi

∝ µθi (eℓ ) wℓ (via s → µ)

mθi (eℓ ) − βt πt (eℓ ) αt

—

Table 4: Conversions for the interpolating family (Corollary 5), with current state zti , local likelihood wℓ = αt 1{eℓ = zti } + βt πt (zti ), and cavity-induced marginal mθi (y) = αt µθi (y) + βt πt (y) (taken ∝ sθi (zti , y) when starting from a score). Specializations: masked (πt = δm ) gives µθi (eℓ ) = (βt /αt ) sθi (m, eℓ ) (RADD); uniform gives µθi (eℓ ) = (mθi (eℓ ) − βt /V )/αt . Table 5 collects the three processes as instances of the common form of §6: each is fixed by a single-token generator, and the recipe of §4.3 then determines its reverse rate, NELBO integrand, and boundary terms.

Masked Forward rate Reverse rate b θ )i (zt , y) (Q t Sampling pθ (zsi | zt ) Cavity integrand cavity Ji,t

Uniform

λt Rt (ey , m) = λt Rt (ey , ex ) = Rt (x, y) = ht (y) V ′ θ i i α λt αt µθi (yi )+βt /V i ) αt µi (y )+βt πt (y ) − t 1{zti = m}µθi (y i ) h (z t θ i θ i i t αt µi (zt )+βt πt (zt ) βt V αt µi (zt )+βt /V α − α s t θ i  µi (zs ), zsi ∈ Y  θ i i βt i (z i | z i ) αs µi (zs ) + βs πs (zs ) q t s t|s β  αt µθi (zti ) + βt πt (zti )  s, zsi = m βt αt′ ht (zti ) θ i i i i 1{zt = m} log µi (z0 ) KL(di ∥ mi ) + DIS di (zt ) mi (zt ) βt di (zti )

Natural prior per-token KL

0 if αt2 = 0, else +∞

Unif. fwd. per-token KL

αt2 log V

di (z0i ) log = natural

1/V cavity NELBO 1/V denoiser NELBO

GIDD

di (z0i ) + βt2 (1−πt2 (z0i )) log βt2 πt2 (z0i ) KL qt2 |0 punif-fwd (93) t2

L log V L log V (π = µ)

(V −1)L log β1t + O(1) V 1

(divergent, Prop. 6)

diverges if πt puts mass on data-supported clean tokens (Rmk. 4)

Table 5: The three main discrete diffusion processes as instances of our framework, with di (·) = i (· | z i ), m = mθ (· | z , t) (Proposition 5), and h (y) = β π ′ (y) + λ π (y) the GIDD off-diagonal qt|0 i t t t t t t 0 i jump rate of (86); cavity quantities suppress the context, µθi (·) = µθi (· | zt−i , t). Two terminal priors are recorded: the natural one ((71), (82), (92)) and the uniform-data-forwarded one (72).

uniform: read as

GIDD: read as

head coord

den.

cav.

score

den.

cav.

score

denoiser cavity score

1.25(0) 2.43(0) 2.63(0)

2.03(1) 1.25(0) 1.33(0)

7.28(1) 6.45(1) 1.25(0)

1.25(0) 2.57(0) 2.72(0)

1.91(1) 1.25(0) 1.31(0)

3.92(1) 3.46(1) 1.25(0)

Table 6: Oracle per-token NELBO values (H(q0 )/L = 1.25; mean±std over 6 seeds, 2.03(1) = 2.03 ± 0.01). Each entry reads the exact reverse rate held in the row coordinate as the column coordinate without the conversion of Table 2: the diagonal is the native value, but every off-diagonal is penalized unless the conversion is applied (Theorem 7, Figure 5a).

Numerical verification

Every quantity is computed on an exactly-solvable toy model, so oracle laws, entropies, and pathKLs are available in closed form. The data law pdata = q0 is a homogeneous bigram chain on V = 8 tokens of length L = 8 (transition matrix drawn once from a fixed seed, a row-softmax of N (0, 1) entries at temperature 0.3), with H(q0 ) = 10.00 nats, i.e. 1.25/token. We instantiate masked, uniform, and GIDD noising with the linear schedule αt = 1 − t on the window [t1 , t2 ] = [0.02, 0.98], GIDD interpolating masked and uniform through a mixing weight λ controlling the terminal distribution via π1 = (1 − λ)δm + λ V1 (λ = 12 except where swept; not to be confused with the schedule rate λt of §7). Every floor, calibration, conversion, and sampling step is oracle-exact; only Figure 1 and Figure 3 involve training, fitting a small DiT (hidden 128, 4 heads, 3 blocks; Adam at 10−3 , batch 256, 8000 steps). Generative perplexities are exact q0 -NLLs of 2500 ancestral samples over 256 steps. Each plot verifies one result of the theory: the reverse-rate projection, a trained denoiser head landing on the oracle marginal rate (Figure 1); the shared information floor H(q0 ) across processes (Figure 2); the Oracle Distance, every model’s excess NELBO decaying to that floor (Figure 3); the coordinate dictionary and its convert-or-pay penalty (Figure 5, Table 6); the parallel-sampling factorization error, isolated at the oracle denoiser (Figure 6); and the initialization calibration, the log V cavity value and the diverging denoiser (Figure 7).

Conclusion

We have given a self-contained, rigorous account of the continuous-time discrete-diffusion ELBO, with explicit boundary terms, and used it to answer the question of the title at two levels. At the level of the objective, the NELBO is not merely a likelihood bound: it equals the data entropy plus the path KL from the oracle reverse process to the learned one (the Oracle Distance, Theorem 3); d its irreducible part is the rate of information loss dt H(Z0 | Zt ), and its best achievable value is the data entropy H(pdata ), for every noising process. At the level of the learned object, a discrete diffusion model learns the marginal reverse jump rate, the posterior average of the clean-conditioned rate, and the denoiser, cavity, and score parameterizations are three coordinates of this one object (Table 3), related by exact conversions. This viewpoint turns scattered constructions into instances of one recipe, with concrete consequences: closed-form per-process ELBOs whose boundary terms make reported values comparable across noising processes (§7, Table 5), the log V cavity calibration

(Proposition 4), the denoiser-coordinate divergence that explains why the denoiser head is harder to train under uniform and GIDD noising (Proposition 6), and the reason masked diffusion is special: there the denoiser and cavity laws coincide (Corollary 4), and the distinction disappears. On an exactly-solvable model, each of these identities (the shared floor, the Oracle Distance decay, the reverse-rate projection, and the convert-or-pay penalties) is confirmed numerically without approximation. Our analysis characterizes the teacher-forced objective: the reverse rate is regressed at noisy states zt ∼ qt|0 drawn from the true forward process, not at the states a model visits when sampling from its own trajectory. This is a consistent target. In the well-specified, infinite-capacity, infinitedata limit the learned rate matches the true reverse rate at every state, so rollout never leaves qt and no drift occurs. Sampling drift is therefore a finite-capacity generalization gap (the learned rate extrapolates poorly to self-generated, low-qt states) rather than a defect of the objective, a more precise statement than the usual reading of exposure bias as objective mismatch. The clean results here, namely the Bregman projection optimum, the exact distance to the oracle, and the entropy floor, all presuppose teacher-forced states; objectives or samplers that instead target self-generated states gain robustness to drift but lose these exact characterizations. Two limitations point to the natural next steps. First, our validation is on a controlled, exactlysolvable model, chosen so that every oracle quantity is available in closed form; confirming that the coordinate-conversion gains persist at model scale is the natural next step, and the concurrent uniform-diffusion results of [GJS+ 26] already provide such evidence at that scale for the uniform special case. Second, the reverse-rate/factorization split (§2.2) isolates two complementary and separately addressable error sources: reverse-rate error at off-distribution states, reducible through drift-aware training or added capacity and data; and factorization error at finite step count, reducible through corrector or self-correcting samplers. Scaling the convert-or-pay comparison across the full denoiser/cavity/score family, and across noising processes, is a concrete program our dictionary makes precise. The single operational rule it leaves the practitioner is simple: learn a head in the coordinate its loss optimizes, or convert between coordinates analytically before you sample.

References [AJH+ 21]

Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34, pages 17981–17993, 2021. URL: https://openreview.net/forum?id=h7-XixPCAL.

[AW12]

Rami Atar and Tsachy Weissman. Mutual information, relative entropy, and estimation in the Poisson channel. IEEE Transactions on Information Theory, 58(3):1302– 1318, 2012. doi:10.1109/TIT.2011.2172572.

[BGW05]

Arindam Banerjee, Xin Guo, and Hui Wang. On the optimality of conditional expectation as a Bregman predictor. IEEE Transactions on Information Theory, 51(7):2664–2669, 2005. doi:10.1109/TIT.2005.850145.

[Bré81]

Pierre Brémaud. Point Processes and Queues: Martingale Dynamics. Springer Series in Statistics. Springer, 1981.

[BSDB+ 24]

Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising Markov models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(2):286–301, 2024. doi:10. 1093/jrsssb/qkae005.

[CBDB+ 22] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, volume 35, 2022. URL: https://openreview.net/forum?id=DmT862YAieY. [CYB+ 24]

Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 5453–5512. PMLR, 2024. URL: https://openreview.net/forum? id=kQwSbv0BR4.

[GJS+ 26]

Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, and Alain Durmus. Uniform diffusion models revisited: Leave-one-out denoiser and absorbing state reformulation, 2026. URL: https://arxiv.org/abs/ 2605.22765, arXiv:2605.22765.

[Gra11]

Robert M. Gray. Entropy and Information Theory. Springer, 2 edition, 2011. doi: 10.1007/978-1-4419-7970-4.

[GRS+ 24]

Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In Advances in Neural Information Processing Systems, volume 37, 2024. URL: https://openreview.net/ forum?id=GTDKo3Sv9p.

[GSV05]

Dongning Guo, Shlomo Shamai, and Sergio Verdú. Mutual information and minimum mean-square error in Gaussian channels. IEEE Transactions on Information Theory, 51(4):1261–1282, 2005. doi:10.1109/TIT.2005.844072.

[HHY+ 25]

Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary Markov processes. In International Conference on Learning Representations (ICLR), 2025. Oral presentation. URL: https://openreview.net/ forum?id=RuP17cJtZo.

[HJA20]

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6840–6851, 2020. URL: https://openreview.net/forum?id=St1giarCHLP.

[HNJ+ 21]

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, volume 34, pages 12454–12465, 2021. URL: https://openreview.net/forum?id=6nbpPqUCIi7.

[JCS16]

Dominik Janzing, Rafael Chaves, and Bernhard Schölkopf. Algorithmic independence of initial condition and dynamical law in thermodynamics and causal inference. New Journal of Physics, 18(9):093052, 2016. doi:10.1088/1367-2630/18/9/093052.

[JS03]

Jean Jacod and Albert N. Shiryaev. Limit Theorems for Stochastic Processes, volume 288 of Grundlehren der mathematischen Wissenschaften. Springer, 2 edition, 2003. doi:10.1007/978-3-662-05265-5.

[JSJN25]

Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, and Albert No. Information-theoretic discrete diffusion. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL: https://openreview.net/forum?id=B2iPEX5A9c, arXiv:2510.24088.

[KBVS23]

Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretic diffusion. In International Conference on Learning Representations (ICLR), 2023. URL: https://openreview.net/forum?id=UvmDCdSPDOW.

[KSPH21]

Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, volume 34, pages 21696–21707, 2021. URL: https://openreview.net/forum?id=2LdBqxc1Yv.

[KW14]

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2014. URL: https: //arxiv.org/abs/1312.6114.

[LME24]

Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 32819–32848. PMLR, 2024. URL: https: //openreview.net/forum?id=CNicRIVIPA.

[MCSE22]

Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, volume 35, 2022. URL: https://openreview.net/forum?id= _RL7wtHkPJK.

[ND21]

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162–8171. PMLR, 2021. URL: https://proceedings.mlr.press/v139/nichol21a.html.

[Nor97]

J. R. Norris. Markov Chains. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1997. doi:10.1017/CBO9780511810633.

[NZY+ 25]

Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Oral presentation. URL: https://openreview.net/forum?id=KnqiC0znVF, arXiv:2502.09992.

[ONX+ 25]

Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations (ICLR), 2025. URL: https://openreview.net/forum?id=sMyXP8Tanm.

[PJGS09]

Jonas Peters, Dominik Janzing, Arthur Gretton, and Bernhard Schölkopf. Detecting the direction of causal time series. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML), pages 801–808. ACM, 2009. doi:10.1145/ 1553374.1553477.

[PJS13]

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Causal inference on time series using restricted structural equation models. In Advances in Neural Information Processing Systems (NeurIPS), volume 26, pages 154–162, 2013. URL: https://proceedings.neurips.cc/paper_files/paper/2013/hash/ 47d1e990583c9c67424d369f3414728e-Abstract.html.

[RC04]

Christian P. Robert and George Casella. Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer, 2 edition, 2004. doi:10.1007/978-1-4757-4145-2.

[RMW14]

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICML), volume 32 of Proceedings of Machine Learning Research, pages 1278–1286. PMLR, 2014. URL: https://proceedings.mlr.press/v32/rezende14.html.

[RNB+ 26]

Gabriel Raya, Bac Nguyen, Georgios Batzolis, Yuhta Takida, Dejan Stancevic, Naoki Murata, Chieh-Hsin Lai, Yuki Mitsufuji, and Luca Ambrogioni. Noise scheduling as information-guided allocation in diffusion training, 2026. URL: https://arxiv.org/ abs/2602.18647, arXiv:2602.18647, doi:10.48550/arXiv.2602.18647.

[SAS+ 24]

Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, volume 37, 2024. URL: https://openreview.net/forum?id=L4uaAR4ArM.

[SDG+ 25]

Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T. Chiu, and Volodymyr Kuleshov. The diffusion duality. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 52584–52619. PMLR, 2025. URL: https://openreview.net/forum?id=9P9Y8FOSOk.

[SDWMG15] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning (ICML), volume 37 of Proceedings of Machine Learning Research, pages 2256–2265. PMLR, 2015. URL: https://proceedings.mlr.press/v37/sohl-dickstein15.html. [SHW+ 24]

Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, volume 37, pages 103131–103167, 2024. URL: https://openreview.net/forum?id=xcqSOfHt4g.

[SJP+ 12]

Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris M. Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning (ICML), pages 1255–1262. Omnipress, 2012. URL: https://icml.cc/2012/papers/625.pdf.

[SLY+ 26]

Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, and Ante Jukic. Scaling beyond masked diffusion language models. In International Conference on Machine Learning (ICML), 2026. To appear. URL: https://arxiv.org/abs/2602.15014, arXiv:2602.15014.

[SSP+ 25]

Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P. de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. In International Conference on Learning Representations (ICLR), 2025. URL: https://openreview.net/forum?id=i5MrJ6g5G1.

[ST25]

Jiaxin Shi and Michalis K. Titsias. Demystifying diffusion objectives: Reweighted losses are better variational bounds, 2025. URL: https://arxiv.org/abs/2511. 19664, arXiv:2511.19664.

[SYD+ 23]

Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In International Conference on Learning Representations (ICLR), 2023. URL: https://openreview.net/forum?id= BYWWwSY2G5s.

[Vin11]

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–1674, 2011. doi:10.1162/neco_a_00142. 65

[vRFD+ 25]

Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 61810–61843. PMLR, 2025. URL: https://openreview.net/forum?id=rvZv7sDPV9.

[ZYYK24]

Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In Conference on Language Modeling (COLM), 2024. URL: https://openreview.net/forum?id=PEQFHRUFca, arXiv:2302.05737.

[ZZZ+ 25]

Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua M. Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 76716–76753. PMLR, 2025. URL: https://openreview.net/forum? id=ZMrdvSm7xi.

Record · ID 343460 · SHA-256 c692c514bac65b3c
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.