Online Class Posteriors for Interpretable and Controllable Discrete Diffusion

Under review as a conference paper at ICLR 2027

O NLINE C LASS P OSTERIORS FOR I NTERPRETABLE AND C ONTROLLABLE D ISCRETE D IFFUSION Felix Koulischer1,* Gianni Van De Velde1, * François Remy 2 1 Chris Develder Thomas Demeester1 1

2

Ghent University - imec

Parallia Healthcare

arXiv:2611.03454v1 [cs.LG] 2 Nov 2026

A BSTRACT Discrete diffusion models remain far less understood than their continuous counterparts, despite recent breakthroughs across modalities, most notably in text. Leveraging tools from continuous diffusion, we formulate a theoretical framework to track the class-conditional posterior in an online manner by combining conditional and unconditional model predictions. In masked language modeling, this posterior offers highly interpretable dynamics: tokens aligned with a class boost the posterior, while contradictory ones suppress it. We also show that the posterior can be used for logical guidance operations (AND, OR, NOT) in discrete diffusion. These operations enable precise control over generation, such as avoiding specific classes. In addition, we discuss a mismatch in the formulation of classifier-free guidance between discrete and continuous settings: discrete models mix the data distributions, while continuous models mix the marginals. We validate our methods and discuss their potential applications on synthetic data, with all quantities analytically tractable, as well as on MNIST, text, and antibody sequences.

1

I NTRODUCTION

Diffusion models are the default generative paradigm for continuous data, from images to protein structures (Sohl-Dickstein et al., 2015; Ho et al., 2020; Podell et al., 2024; Abramson et al., 2024). They learn the so-called score function ∇ log pt (xt ), which gradually transports noise towards data. Discrete Diffusion Models (DDMs) transfer this idea by operating directly on the discrete state space (Austin et al., 2021; Shi et al., 2024; Campbell et al., 2022), as an alternative to embedding tokens into a continuous latent space and reusing continuous machinery (Dieleman et al., 2022). Like their continuous counterparts, DDMs progressively corrupt data, either by masking tokens or by replacing them with tokens drawn uniformly from the vocabulary. In the discrete setting, the score function is replaced by a rate function Rt (x, y), the instantaneous probability of token y transitioning to token x. Integrating this rate over diffusion time defines a continuous time Markov chain. This process, like in the continuous case, carries the state from noise to data (Campbell et al., 2022; Gat et al., 2024). Emerging evidence in language and DNA modeling suggests that DDMs could rival autoregressive models, thanks to their bidirectional context and multi-token generation (Nie et al., 2025; Stark et al., 2024; Sahoo et al., 2026). Yet, while the literature on continuous diffusion models has extensively explored noise schedules (Karras et al., 2022; Stancevic et al., 2025), training dynamics (Karras et al., 2024b), and inference-time guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2021), the algorithmic tools for discrete diffusion, and in particular for guiding it, remain far less developed. We address this gap by transferring to the discrete setting two guidance tools that have no discrete counterpart yet: the online tracking of the class posterior during generation, and the composition of conditional generative processes through logical operators. Classifier-free guidance (CFG) (Dhariwal & Nichol, 2021; Ho & Salimans, 2021) applies a constant push towards a condition throughout the denoising process, regardless of whether the considered sample needs it. Recent work in continuous diffusion has shown that guidance can instead adapt to *

Joint First Authors: felix.koulischer@ugent.be, gianni.vandevelde@ugent.be

1

Under review as a conference paper at ICLR 2027

the state of the trajectory, only being used if a trajectory truly requires it (Koulischer et al., 2025b;a; Galashov et al., 2025). Additionally, guidance can be used for composing several conditional generative processes, for instance to generate samples satisfying either or both of two conditions. This requires weighting each process according to how likely the current state is to satisfy its corresponding condition (Skreta et al., 2025; Alesiani et al., 2026). All of these constructions rely on the same quantity, the class posterior pt (c | xt ), which estimates the probability that the noisy state xt will eventually satisfy condition c. In this work, we focus on how conditional and unconditional model predictions, and their corresponding rate functions, can be combined to estimate this posterior and construct different conditional generative processes. We show that the class posterior can be tracked during discrete diffusion directly from the conditional and unconditional predictions, for both uniform and masking noise. The estimation is exact when analytical quantities are available and exhibits interpretable token-level dynamics: because the state changes one token at a time, transitions that support a class increase its posterior, while transitions supporting a different class decrease it. This provides a direct interpretation of the posterior that has no continuous analogue. In masked language modeling, for example, it reveals which generated tokens move a sample towards or away from a class. Building on this posterior estimation, we extend CFG to the logical AND, OR, and NOT operators for discrete rate functions, drawing on recent advances in the continuous domain (Alesiani et al., 2026; Skreta et al., 2025; Koulischer et al., 2025b). These operators combine conditional rate functions through posterior-dependent weighting: the OR gate arbitrates between the conditions according to their posterior probabilities, while the NOT gate detects the undesired condition and activates a repulsive term only when necessary. We verify these constructions analytically and validate them on discrete image and antibody sequence generation. Our derivation also reveals that standard discrete CFG (Schiff et al., 2025; Nisonoff et al., 2025) differs from its continuous counterpart in a way that has gone unnoticed. Continuous CFG mixes marginals multiplicatively, which is realized as an additive mixture of predictions, whereas discrete CFG instead mixes data distributions multiplicatively. These operations are not equivalent because the noising process does not commute with multiplicative mixing (Bradley & Nakkiran, 2024). Our contributions are threefold: (i) we cast the online class posterior to the discrete domain and demonstrate its interpretable dynamics in language generation; (ii) we derive posterior-weighted OR and NOT operators for discrete rate functions, verify these on an analytical example, and validate on discrete image generation and antibody sequence generation; and (iii) we identify and formalize the multiplicative/additive mismatch between continuous and discrete CFG.

2

BACKGROUND AND R ELATED W ORK

2.1

D ISCRETE D IFFUSION M ODELS

L We consider sequences xt = (x1t , . . . , xL t ) ∈ V of L discrete tokens, with t ∈ [0, 1] running from noise (t = 0) to data (t = 1). Let qt (xt | x1 ) denote the forward noising kernel for a single token. We consider the mixture path
qt (xt | x1 ) = κ(t)δx1 (xt ) + 1 − κ(t) σ(xt ), κ(0) = 0, κ(1) = 1, (1)

where σ is a base noise distribution. The corresponding sequence-level corruption process factorizes over tokens, with classical choices including masking, in which tokens converge to an absorbing mask token, and uniform noising, in which tokens are replaced by samples from a uniform distribution (Austin et al., 2021). The forward process defines a continuous-time Markov chain (CTMC) with marginal distribution pt . Let Rt (x, y) denote the instantaneous rate of transitioning from state y to state x. With this convention, the marginal distribution evolves according to the master equation dpt (x) X = Rt (x, y)pt (y) ≡ Rt pt (2) dt y∈V

with standard notation for the rate matrix Rt . Equivalently, for a small time step h > 0, the transition kernel, which can be used to construct a sampler, is given by Pt,t+h (x | y) = δxy + hRt (x, y) + o(h), 2

(3)

Under review as a conference paper at ICLR 2027

approximated

0

0.5 0.0 1.0 0.5 0.0 0.00

0.25

0.50 t

0.75

1.00

Uniform: p(c|xt)

1.0

1.00 0.75 0.50 0.25 0.00

Masking: p(c|xt)

Masking: p(c|xt)

Uniform: p(c|xt)

exact

1.00 0.75 0.50 0.25 0.00 0.00

1

2

(a) Synthetic data

0.25

3

4

5

6

7

0.50

8

9

prior

0.75

t

1.00

(b) MNIST

Figure 1: Illustration of the online posterior evaluation. In (a), evaluated on an analytically tractable setting in which data points consist of two tokens represented by blue and red with each class having an 80% majority of one of the two tokens, trajectories are shown above the posteriors. In (b), the posterior is evaluated on MNIST digit generation. On top for uniform, below for masked diffusion. The increase in the posterior occurs simultaneously with the emergence of the digit.

Typically, a neural network is trained to approximate the denoiser pθ (x1 | xt ) using a time-weighted cross-entropy loss (Shi et al., 2024). This posterior can subsequently be converted into an instantaneous rate function. Importantly, the rate function is not unique: multiple generators can induce the same prescribed marginal path. One such construction is Discrete Flow Matching (Shaul et al., 2025), which, for the marginal path above, defines; Rθ,t (x, y) =

κ̇(t) [pθ (x1 = x | xt = y) − δxy ] . 1 − κ(t)

(4)

A different valid construction is given by the Markovian ancestral sampler (Austin et al., 2021). In the language domain, Discrete Diffusion Models, or Diffusion Language Models (DLMs), already come with a range of decoding strategies for improving generation. LLaDA (Nie et al., 2025), a commonly used DLM, uses a fixed decoding schedule and confidence unmasking. The schedule fixes how many tokens are unmasked at each diffusion step, while confidence unmasking determines which ones: rather than choosing at random, it unmasks the tokens with the highest confidence. 2.2

D IFFUSION G UIDANCE

Guidance in continuous diffusion models: Sampling on a class label or prompt c alone often does not yield satisfactory samples. In continuous diffusion models, guidance corrects this by tilting the noisy marginal with an exponentiated classifier term (Dhariwal & Nichol, 2021), p̃t (xt |c) ∝ pt,θ (xt ) pt,θ (c|xt )λ ,

(5)

where λ > 1 controls the strength of the conditioning signal. Rewriting pt,θ (c|xt ) via Bayes’ rule replaces the classifier with a ratio of a conditional and an unconditional model, p̃t (xt |c) ∝ pt,θ (xt )1−λ pt,θ (xt |c)λ , so that both terms can be obtained from a single jointly-trained network (Ho & Salimans, 2021). Taking the gradient of the log of the tilted marginal gives the familiar rules of Classifier Guidance and Classifier-Free Guidance, respectively: ∇ log p̃t (xt |c) = ∇ log pt (xt ) + λ∇ log pt (c|xt )

(6)

= ∇ log pt (xt ) + λ ∇ log pt (xt |c) − ∇ log pt (xt ) , an additive combination of the conditional and unconditional score predictions. 3

(7)

Under review as a conference paper at ICLR 2027

Guidance in discrete diffusion models: The discrete setting admits a similar decomposition, although not equivalent. Guidance was introduced for discrete diffusion with the same goal of sharpening an under-confident conditional signal (Nisonoff et al., 2025; Schiff et al., 2025), realized by combining the logits of the predictions instead of the predictions themselves:
log p̃(x1 |xt , c) = log p(x1 |xt ) + λ log p(x1 |xt , c) − log p(x1 |xt ) , (8) a multiplicative mixture of the predicted token distributions, in contrast to the additive mixture of scores used above. Recent progress on guidance: Both formulations have since been refined. In the continuous case it was proposed to schedule the guidance scale λ over time rather than hold it fixed, aiming to target the noise levels that benefit most from it (Kynkäänniemi et al., 2024; Koulischer et al., 2025a; Galashov et al., 2025). Recent work has also shown that deliberately weakening the unconditional model pushes samples away from low-quality regions of the data space as well as from unconditionality (Karras et al., 2024a). Analogous ideas, including time-dependent schedules, have since begun to be carried over to the discrete formulation as well (Rojas et al., 2026). More recently, guidance has been reframed as one instance of a broader family of operators for combining diffusion processes, including AND (both conditions), OR (either condition) and NOT (avoid a condition) (Skreta et al., 2025; Koulischer et al., 2025b; Alesiani et al., 2026). The bottleneck of the latter two is the requirement of a class-conditional posterior pt (c|xt ) needed to weight this combination, which can be trained separately (Alesiani et al., 2026) or approximated from existing conditional/unconditional models (Koulischer et al., 2025b; Skreta et al., 2025). Discrete diffusion, however, has lacked both ingredients: the logical operators and a posterior to weight them. In this work, we close this gap, formulating the full family of operators for the discrete setting and showing that the posterior can be tracked during inference with high interpretability.

3

P OSTERIOR E STIMATION AND C OMBINING M ULTIPLE P ROCESSES

This section describes our key methodological contributions for discrete diffusion models. Section 3.1 shows how the class-conditional posterior can be estimated during inference by combining conditional and unconditional model predictions. Building on this, Section 3.2 introduces the three logical operators AND, OR and NOT, beginning with the notation they require and an important note on the (non-)commutativity of mixing and noising operations, which, for the AND operator, reveals that discrete and continuous CFG are not equivalent (Bradley & Nakkiran, 2024). 3.1

O NLINE P OSTERIOR E STIMATION

The conditional posterior pt (c|xt ), which evaluates how likely a noisy state xt ends up satisfying the condition c, is often mistakenly considered intractable during inference. In the continuous case, its tractability when sampling using a reversible Markov chain has been demonstrated (Koulischer et al., 2025b; Skreta et al., 2025), but to the best of our knowledge, this has not been explored for discrete diffusion models. We here present how p(c | xt ) can be tracked iteratively through inference and what specifics arise in the discrete case as opposed to the continuous case. The idea is to track the posterior p(c | x0:t ) throughout inference using the conditional and unconditional models’ predictions, with x0:t a shorthand notation for all states leading up to t: log p(c | x0:t ) = log p(c | x0:t−1 ) + log p(xt | xt−1 , c) − log p(xt | xt−1 ). In practice, we follow Koulischer et al. (2025b) and introduce two hyperparameters τ and δ:
log p(c | x0:t ) = log p(c | x0:t−1 ) + τ log p(xt | xt−1 , c) − log p(xt | xt−1 ) + δ.

(9)

(10)

If a Markovian sampler is used, the current state is a sufficient statistic for the class: p(c|xt ) = p(c | x0:t ). In the masking case, this identity always holds since a token never changes once unmasked and xt therefore retains every revealed token. In the uniform case, the Markov property only holds when sampling using an ancestral sampler specifically constructed to be Markovian. However, in practice we find that using the transition kernel predicted by a discrete flow matching sampler is more stable, which is why we use it in the results section.1 1

For more detail, see Appendices B.3 and C.2.

4

Under review as a conference paper at ICLR 2027

A key advantage of the discrete case, compared to its continuous analogue, is that one can see which token flips caused the posterior to change, therefore allowing the user to understand what might have caused the model to opt for one class or another. This is particularly interesting in masked language modeling, where the posterior reveals which words commit the sample to a class, long before denoising completes. 3.2

L OGICAL OPERATORS : AND , OR , NOT

We now show how logical mixture operators Λ ∈ {AND, OR, NOT} over conditional distributions i i Λ {p(x|ci )}N i=1 , defined by their rate matrices Rt or predictors p (x1 |xt ), combine into a joint rate Rt or predictor pΛ (x1 |xt ). Crucially, an operator can be applied at two levels: (i) before noising, thus mixing the data distributions (which we show is what discrete CFG does (Schiff et al., 2025)), or (ii) after, which amounts to mixing the marginals (as in continuous CFG (Ho & Salimans, 2021)). Because the noising convolution commutes with additive but not multiplicative mixing, the two levels (i)–(ii) coincide for the additive OR and NOT operators, but differ for the multiplicative AND. Hence, we provide separate derivations for AND, but jointly present them for OR and NOT (see Appendix B.1 for a detailed treatment). AND gate. (data-level mixture)

A data-level mixture targets the mixture distribution exactly. For QN i AND , this is the multiplicative mixture pΛ (x1 |xt ) ∝ i=1 p (x1 |ci ), with as predictor: N Y pΛ 1 (x1 ) p (x1 |xt ) = p(xt |x1 ) Λ ∝ p(xt |x1 ) pi (x1 |ci ) pt (xt ) i=1 Λ

∝ p(xt |x1 )1−N

N Y

pi (x1 |xt , c)

(11)

(12)

i=1

Thus the predicted distributions should be multiplied with each other, which is commonly implemented as adding logits together: log p(x1 |xt ) ∝

N X

log pi (x1 |xt , c)

(13)

i=1

From this predictor a rate function can be constructed using eq. (4). This sum of logits strongly resembles how CFG is implemented in the discrete diffusion literature, i.e., through eq. (8). AND gate. (marginal-level mixture) On the other hand, an approach reminiscent of the continuous diffusion literature can be applied: considering a multiplicative mixture of the marginals defined as QN 2 pΛ t (xt ) ∝ i=1 pt (xt |ci ). The joint rate function can be directly derived through: N X  X dpt (xt |ci ) Y dpΛ t (xt ) = p(xt |cj ) = Rti pΛ t (xt ) dt dt i i

(14)

j̸=i

Hence, the rate matrices of the joint process satisfy: RtΛ =

N X

Rti .

(15)

i=1

The shape of this combination is also intuitive: as long as one of the conditions is not satisfied, the corresponding rate will try to strongly modify the outcome. Only once xt satisfies both conditions will both rate functions remain reasonably small. This is analogous to continuous CFG, which similarly combines two infinitesimal generators, with the score functions being replaced by the discrete rate functions. 2

Note the change in subscript from 1 to t .

5

Under review as a conference paper at ICLR 2027

gate. (both) For the additive operation behind the OR gate, pΛ t (xt ) ≡ pt (xt ) = Λ i=1 p(ci )pt (xt |ci ), with mixture weights given by the priors p(ci ), pt is simply the unconditional marginal induced by the partition ci N i=1 . The joint rate function can again be computed directly: ! X X dpΛ dpt (xt |ci ) pt (xt |ci ) i t (xt ) = p(ci ) = p(ci ) Λ Rt pΛ (16) t (xt ) dt dt p (x ) t i i OR

PN

Recognizing the resulting Bayesian posterior gives: RtΛ =

N X

p(ci | xt ) Rti .

(17)

i=1

Thus, the global rate is obtained by averaging the conditional rates according to the posterior probability of each component given the current noisy state. This perspective closely mirrors the construction underlying Flow Matching (Lipman et al., 2023; Gat et al., 2024): rather than learning the global dynamics directly, one can define simple conditional dynamics and learn how to identify their contribution from the current state. The global dynamics are then obtained by posterior-weighted averaging of the conditional dynamics. In particular, taking each datapoint as a separate condition yields datapoint-specific rates that can be combined through p(ci |xt ) to recover the global rate. Our construction therefore extends this conditional-to-global principle from datapoint-level conditioning to arbitrary partitions of the data. Beyond this theoretical connection, the OR gate is not strictly necessary for class-conditional generation, since sampling a condition according to its prior ci ∼ p(c), and subsequently using the corresponding conditional model, yields the same desired distribution. NOT gate. (both) Removing, or avoiding, a class from generation can be expressed as subtracting its conditional distribution from the unconditional one, pΛ t (xt ) ∝ pt (xt ) − p(c) pt (xt |c). The joint rate is then given by:   Rt pt (xt ) − p(c) Rtc pt (xt |c) pt (c|xt ) dpΛ t (xt ) c = = Rt − (Rt − Rt ) pΛ (18) t (xt ) dt 1 − p(c) 1 − pt (c|xt )

And therefore: RtΛ = Rt −

p(c | xt ) Rtc − Rt . 1 − p(c | xt )

(19)

p(c|xt ) This equation resembles a classifier-free guidance equation, in which 1−p(c|x can be seen as a t) dynamic guidance scale that is small if the undesired class is unlikely to be generated and that becomes asymptotically large if that class becomes likely. Again, this is entirely analogous to what is found in the continuous case, with as single difference that the continuous generators, or scorefunctions, are replaced by their discrete analogues, the rate functions (Koulischer et al., 2025b).

4

R ESULTS

Section 4.1 first validates the posterior approximation on (i) synthetic data for which ground-truth quantities are available before being tested on MNIST images (LeCun & Cortes, 2010), (ii) text, and (iii) antibody sequence generation (Shuai et al., 2023). Next, in Section 4.2, we evaluate the NOT gate for image and antibody sequence generation. 4.1

O NLINE P OSTERIOR E STIMATION

Synthetic data. Our synthetic dataset consists of binary sequences of length 10, with red and blue tokens. Each sequence is assigned to one of two classes based on a majority token: one token (red or blue) is sampled with probability pmaj , and the other with 1 − pmaj . Figure 1a shows that the inferred posterior matches the analytically tractable ground-truth quantity computed using Bayes’ rule with the known marginal likelihoods. Importantly, the posterior inference remains exact across both masking and uniform noise processes,3 despite their substantially different transition dynamics. 3

Uniform noise requires the Markovian ancestral transition kernel. Details are provided in Appendix C.2.

6

Under review as a conference paper at ICLR 2027

Wine Class 5 'ginger'

large jumps

1.0 0.9

'red' 'red'

0

'wine'

AUC

Δlog(pc )

Other classes

'cabernet'

'wine'

−5

0.8

Text, masking MNIST, masking MNIST, uniform Antibody, masking Antibody, uniform

0.7 0.6

'syrup'

−10 0.0

0.2

0.4

0.6

0.5

0.8

1.0

0.0

0.2

t

(a) Illustrative posterior trajectories

0.4

t

0.6

0.8

1.0

(b) AUC during denoising

Figure 2: In (a), the progress of the posterior approximation is shown throughout generation of four sentences on “beverage” with as class “wine”. Highlighted words show large jumps in the posterior, corresponding to those given in Table 1. In (b), one-vs-rest AUC is evaluated throughout generation, using a modality-specific external classifier to determine the final class.

Image generation. For the image modality, we train conditional models on MNIST using both masking and uniform noising schemes (training details and ablations in Appendix D.1). During sampling, the posterior over the 10 MNIST classes is estimated online by applying a softmax to the class-specific log-posterior quantities computed with eq. (10). To evaluate the posterior throughout denoising, trajectories are classified using a pretrained ViT applied to the final state. At each noise level t, we evaluate the posterior scores in a one-vs-rest classification setting and report the AUC averaged over the ten classes, measuring how well the online posterior discerns the class eventually expressed by the generated sample. As shown in Fig. 2b, the AUC increases throughout denoising, indicating that the posterior becomes progressively more informative as the sample develops class-specific structure. Representative trajectories and their corresponding posterior estimates are shown in Fig. 1b, with additional examples in Appendix D.3. In both noise processes, the posterior increases as class-specific structure emerges; compared to masking, the increase tends to occur later under uniform noising, caused by the ability of uniform diffusion to revise previously generated tokens and therefore defer commitment to a particular class. Text generation. Here, evaluation is based on the Cognac (Chen et al., 2022) benchmark, with as task generating a Wikipedia-style sentence around a certain topic. The two models are obtained by prompting LLaDA-8B-Instruct (Nie et al., 2025) with two different prompts. The unconditional model is asked to generate about a category (e.g., “beverage”), while the conditional model is prompted to generate about a sub-category (e.g., “wine”) which we refer to as the classes. Full details of the experimental setup are in Appendix E.1. Importantly, the online posterior estimation allows analyzing which unmasking events result in the generation of a sentence of a particular sub-category. Table 1 quantifies this, by showing the tokens that have the strongest impact on the

Topic

Sub-topic

1st

(class c)

2nd

3rd

Token

∆ log pc

Token

∆ log pc

Token

∆ log pc

wine syrup

+2.3 -5.5

cabernet ginger

+1.3 -3.9

red espresso

+1.1 -3.2

beverage

wine

+ −

sweet

candy

+ candy − marzipan

+2.8 -2.1

lollipop cookies

+0.8 -0.7

are sweet

+0.5 -0.6

+ −

+2.4 -6.8

Swiss cream

+1.4 -0.5

cheese product

+0.5 -0.5

dairy product cheese

cheddar yogurt

Table 1: Tokens pushing most towards (+) and away from (−) the class, estimated by our posterior estimation. (Sub)words in gray provide context for the token shown in black. Each gray context corresponds to the actual context for that token in all samples containing that token. 7

Under review as a conference paper at ICLR 2027

1.0 0.8 0.6 0.4 0.2 0.0

target OR(0,2) OR(6,9)

0.8 Probability

Probability

1.0

target: 1/9 NOT(1)

0.6 0.4 0.2

0

1

2

3

4 5 6 Predicted class

7

8

0.0

9

(a) Samples under NOT gate

0

1

2

3

4 5 6 Predicted class

7

8

9

(b) Samples under OR gate

Figure 3: Illustration of the NOT and OR gates on MNIST under a uniform noise model. (a) The NOT gate, showing how unconditional samples containing the undesired number “1” are transformed into other, predominantly round numbers such as “0” and “9”. (b) The OR gate evaluated on two distinct pairs, with corresponding samples shown above. posterior across 1000 samples for different tasks. Words with strong positive impact on the posterior are either a word that belongs to the class (e.g., ‘cheddar’ for class cheese), or words that strongly implicate a class word (e.g., ‘Swiss’ for class cheese). Conversely, words with strong negative impact on the posterior often exemplify subcategories of the topic that are unrelated to the constraint (e.g., ‘yogurt’). In Figure 2a, four trajectories are shown, illustrating how the posterior moves strongly in the right direction after words are generated exemplifying the class or the complement of a class. Antibody sequence generation. For antibody sequence generation, we train a 50M-parameter Transformer based on the architecture of Fournier et al. (2024) on the OAS dataset (Olsen et al., 2022; Shuai et al., 2023), restricted to human antibodies. Antibody sequences are partitioned by V-gene type as classified by IgBLAST (Ye et al., 2013). The V-gene largely determines the antibody framework, providing a strong conditioning signal. Our posterior estimation accurately identifies the V-gene from the partially generated sequence, achieving near-perfect AUC as shown in Figure 2b. Furthermore, the posterior becomes informative after only a small number of token flips, allowing the conditioning variable to be identified early in generation. Further details on this modality are provided in Appendix F. 4.2

L OGICAL O PERATORS

Of the three gates, the NOT gate is most interesting, combining two rate functions into an otherwise unavailable novel rate function. Note that in text, functionality of the NOT gate can be achieved adapting the prompt by stating which class to avoid, a baseline our method matches as shown in Appendix E.3. Next, we consider the more interesting cases of applying NOT for image and antibody

FID ( ↓ )

10

NOT: 0

Negative CFG

NOT gate: Dynamic Negative Guidance (Ours)

NOT: 1

NOT: 2

0.00 0.02 0.04 0.06 0.08 0.10

0.00 0.02 0.04 0.06 0.08 0.10

8 6 4 0.00 0.02 0.04 0.06 0.08 0.10

Violation Rate ( ↓ )

Violation Rate ( ↓ )

Violation Rate ( ↓ )

Figure 4: Pareto curves comparing the quality of the generated samples to the safety of the approach, respectively through the FID and the violation rate. Both negative CFG, as well as the theoretically justified dynamic NOT gate display a trade-off between quality and safety, with the dynamic approach outperforming the static one, albeit at the cost of an additional hyperparameter. 8

Under review as a conference paper at ICLR 2027

Method

Setting

Majority

Unconditional Negative CFG NOT gate

majority-calibrated minority-calibrated both

Minority

Viol. ↓

FPD ↓

Viol. ↓

FPD ↓

0.278 0.013 0.186 0.010

0.174 0.104 0.084 0.074

0.032 0.002 0.017 0.016

0.007 0.326 0.020 0.010

Table 2: NOT gate performance when removing majority and minority V-gene families. Static guidance is calibrated to match the dynamic method’s violation rate in the corresponding regime, then transferred across regimes. Gray entries show performance outside the calibration regime. The full Pareto front is given in Appendix F.5. sequence generation. For validation of the theory, rather than intended for practical application, we also evaluate the OR gate in the image modality. Image generation. The NOT gate is evaluated using the Pareto front of safety and quality, measured respectively by the violation rate against a ViT classifier, and FID on 4096 samples (Heusel et al., 2017). In Figure 4, the dynamic approach is compared to static CFG-style negative logit guidance, which samples from p̃(x) ∝ p1 (x1 )p1 (c | x1 )−λ . A grid search for the optimal hyperparameter is performed for the two approaches, with more details in Appendix D.4. Both methods exhibit the expected trade-off between violation rate and image quality. The dynamic NOT gate consistently achieves a better trade-off than negative CFG, at the cost of the additional hyperparameter τ . Figure 3a shows the distribution resulting from excluding the class “1” and how it impacts samples generated from the same seed using the unconditional model. The OR gate is evaluated similarly by restricting the posterior to the two desired conditions, visualized in Figure 3b. Antibody sequence generation. In this setting, we evaluate how the NOT gate removes a V-gene family while preserving quality, measured respectively using violation rate and Fréchet Protein Distance (FPD) to the desired population in ESM2 embedding space (Lin et al., 2023). The dynamic NOT gate is compared against unconditional generation and static negative CFG. A key challenge in this setting is the strong imbalance between V-gene families: nearly 95% of sequences belong to just three of the seven families. Removing a majority class requires substantially stronger guidance than removing a minority class. The dynamic formulation accounts for this imbalance as the initial guidance strength scales with the class prior. In Table 2, we choose a setting such that both majority and minority classes are removed with only small FPD increases.4 For comparison, negative CFG is calibrated separately to achieve a comparable violation rate for majority and minority classes. Majority-calibrated guidance is overly strong for minority classes, causing a substantial increase in FPD, while minority-calibrated guidance is too weak for majority classes. The dynamic NOT gate does not suffer the same flaw thanks to its prior-weighted initialization.

C ONCLUSION We have shown that the class-conditional posterior in discrete diffusion can be tracked online, at no cost beyond the conditional and unconditional predictions already required for guidance. In diffusion language modeling this posterior is highly interpretable: individual tokens can be seen to commit a sample toward a class or away from it during generation. Building on the posterior, we derived the AND, OR, and NOT operators over discrete rate functions, providing a principled means of composing conditional processes. The NOT gate is of particular practical value, enabling generation that steers away from a class in modalities such as images and antibody sequences, where prompting a model to avoid a concept is not possible. As the construction is agnostic to modality, noising process, data imbalances and schedules, it applies unchanged across the settings we consider. The derivation additionally revealed a previously unremarked discrepancy between classifier-free guidance in the continuous domain, an additive mixture of scores, and the discrete domain, a multiplicative mixture of predictions. Limitations are provided in Appendix A. 4

See Appendix F.5 for the full Pareto curves and guidance sweeps.

9

Under review as a conference paper at ICLR 2027

ACKNOWLEDGMENTS This research was partly funded by the Research Foundation - Flanders (FWO-Vlaanderen) under grant G0C2723N and by the Flemish Government (AI Research Program). We would like to thank Leander Gabriel, Florian Handke and Johannes Deleu for the interesting and insightful discussions that have led up to some of this work, particularly in understanding the posterior estimation algorithm. AI DISCLOSURE In this work, we used generative AI tools to implement methods used in this paper. Additionally, we used generative AI tools to plot results, create or edit software code, and edit the paper to improve readability. We have not used generative AI tools for other purposes. We have reviewed all AIassisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

R EFERENCES Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung, Michael O’Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvilė Žemgulytė, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex Bridgland, Alexey Cherepanov, Miles Congreve, Alexander I. Cowen-Rivers, Andrew Cowie, Michael Figurnov, Fabian B. Fuchs, Hannah Gladman, Rishub Jain, Yousuf A. Khan, Caroline M. R. Low, Kuba Perlin, Anna Potapenko, Pascal Savy, Sukhdeep Singh, Adrian Stecula, Ashok Thillaisundaram, Catherine Tong, Sergei Yakneen, Ellen D. Zhong, ichal Zielinski, Augustin Žı́dek, Victor Bapst, Pushmeet Kohli, Max Jaderberg, Demis assabis, and John M. Jumper. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):493–500, 06 2024. URL https://doi.org/10.1038/s41586-024-07487-w. Francesco Alesiani, Jonathan H Warrell, Tanja Bien, Henrik Christiansen, Matheus Ferraz, and Mathias Niepert. Logical guidance for the exact composition of diffusion models. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/ forum?id=OAM1jJsMGp. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021. Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is a predictor-corrector, 2024. URL https://arxiv.org/abs/2408.09000. Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022. Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. Controllable Text Generation with Language Constraints, 2022. Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021. Sander Dieleman, Laurent Sartran, Arnaud Dauphin, Alessia Mancuso, Matthew Rebain, Mikołaj Bińkowski, Natascha Kursawe, Jack W Rae, and Oriol Casagrande. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: //openreview.net/forum?id=YicbFdNTTy. 10

Under review as a conference paper at ICLR 2027

Quentin Fournier, Robert M. Vernon, Almer van der Sloot, Benjamin Schulz, Sarath Chandar, and Christopher James Langmead. Protein language models: Is scaling necessary? bioRxiv, 2024. URL https://www.biorxiv.org/content/early/2024/09/23/ 2024.09.23.614603. Alexandre Galashov, Ashwini Pokle, Arnaud Doucet, Arthur Gretton, Mauricio Delbracio, and Valentin De Bortoli. Learn to guide your diffusion model, 2025. URL https://arxiv. org/abs/2510.00815. Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017. Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview. net/forum?id=qw8AKxfYbI. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020. Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusionbased generative models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022. Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum? id=bg6fVPVs3s. Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proc. CVPR, 2024b. Felix Koulischer, Florian Handke, Johannes Deleu, Thomas Demeester, and Luca Ambrogioni. Feedback guidance of diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL https://openreview.net/forum?id= 8ySOcf7UpM. Félix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, and Luca Ambrogioni. Dynamic negative guidance of diffusion models. In The Thirteenth International Conference on Learning Representations, 2025b. Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=nAIhvNy15T. Yann LeCun and Corinna Cortes. Mnist handwritten digit database. ATT Labs, 2010. URL http: //yann.lecun.com/exdb/mnist/. Marie-Paule Lefranc, Véronique Giudicelli, Patrice Duroux, Joumana Jabado-Michaloud, Géraldine Folch, Safa Aouinti, Emilie Carillon, Amélie Houles, Typhaine Paysan-Lafosse, Saida HadiSaljoqi, Souphatta Sasorith, Gérard Lefranc, and Sofia Kossida. Imgt, the international immunogenetics information system® 25 years on. Nucleic Acids Research, 2015. Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 2023. URL https://www.science. org/doi/abs/10.1126/science.ade2574. 11

Under review as a conference paper at ICLR 2027

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t. George A. Miller. WordNet: A lexical database for English. Commun. ACM, 38:39–41, 1995. ISSN 0001-0782. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview. net/forum?id=KnqiC0znVF. Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id= XsgHl54yO7. Tobias H Olsen, Fergus Boyles, and Charlotte M Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science, 01 2022. William S. Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2022. URL https://api. semanticscholar.org/CorpusID:254854389. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=di52zR8xgf. Kevin Rojas, Ye He, Chieh-Hsin Lai, Yuhta Takida, Yuki Mitsufuji, and Molei Tao. Improving classifier-free guidance in masked diffusion: Low-dim theoretical insights with high-dim impact. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=mMK9pvQJxf. Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, and Ante Jukić. Scaling beyond masked diffusion language models. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/ forum?id=xKJG1CNtQM. Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander M Rush, Thomas PIERROT, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id= i5MrJ6g5G1. Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kineticoptimal perspective. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net. Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024. Richard W Shuai, Jeffrey A Ruffolo, and Jeffrey J Gray. Iglm: Infilling language modeling for antibody sequence design. Cell Systems, 2023. Marta Skreta, Lazar Atanackovic, Avishek Joey Bose, Alexander Tong, and Kirill Neklyudov. The superposition of diffusion models using the itô density estimator. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=2o58Mbqkd2. 12

Under review as a conference paper at ICLR 2027

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, 2015. URL https://proceedings.mlr.press/v37/ sohl-dickstein15.html. Dejan Stancevic, Florian Handke, and Luca Ambrogioni. Entropic time schedulers for generative diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=EfDIApcjgI. Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. In Proceedings of the 41st International Conference on Machine Learning, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report, 2025. Jian Ye, Ning Ma, Thomas L. Madden, and James M. Ostell. Igblast: an immunoglobulin variable domain sequence analysis tool. Nucleic Acids Research, 07 2013. URL https://doi.org/ 10.1093/nar/gkt382.

13

Under review as a conference paper at ICLR 2027

A

L IMITATIONS AND FUTURE WORK

Due to the broad scope of the current paper, spanning several modalities and noising schemes, a detailed analysis of how diffusion-language-modeling choices such as confidence unmasking, block decoding, and remasking interact with the posterior is absent. We leave this to future work. A more fundamental limitation is that the posterior is updated only after the state has changed, so a token that strongly determines a class affects the estimate only once it has been unmasked. For the NOT gate this is a genuine shortcoming: ideally the posterior would reshape the transition probabilities before such tokens are generated, preventing the unwanted class rather than reacting to it. We view this anticipatory formulation as a promising direction.

B

A DDITIONAL DERIVATIONS

In this appendix additional useful derivations are given. In Appendix B.1, the difference between how the common classifier-freee guidance method works in the discrete and continuous domains is discussed. Thereafter, in Appendix B.2, a detailed derivation of all logical operators introduced in the main body is given. Then, in Appendix B.3 an important note regarding under which samplers our online posterior estimate is equivalent to the ground-truth marginal computed posterior. B.1

C ONTINUOUS VS DISCRETE CFG

In the continuous diffusion literature guidance is seen as a multiplicative corruption/sharpening at the marginal level which results in an additive mixture of model predictions or score functions. In the discrete diffusion literature however it is common to see this corruption as a multiplicative tweaking of the predicitons themselves which results in an additive mixture of the predicted logits. This difference is not well known yet quite significant for our work as here we start with the same assumptions as in the continuous case, namely corruptions on the marginal level, and derive the rate function that would yield the discrete analogue of continuous CFG, which we show to be an additive mixture of the rate functions. Let us frame this in more detail mathematically. Classifier-free guidance starts of from a sharpening of the marginals: p(xt |c)λ pCFG (xt ) = p(xt ) (20) p(xt )λ When looking at the infinitesimal generator of that distribution, which in the continuous case is its score function, one finds:   ∇ log pCFG (xt ) = ∇ log p(xt ) + λ ∇ log p(xt |c) − ∇ log p(xt ) (21) Using tweedies formula the score function depends linearly on the denoiser prediction x̂0 = E[x0 |xt ] such that in practice CFG can be implemented as a linear combination of the denoisers prediction:   x̂CFG (22) 0 (xt ) = x̂0 (xt ) + λ x̂0 (xt |c) − x̂0 (xt ) In our derivation of the and operation we start from the exact same assumption and find that for a multiplicative mixture of the marginals one again needs to add their infinitesimal generators, which in the discrete case are given by the rate matrix Rt instead of the score functions ∇ log p(xt ). This time, the rate functions linearly depend on the models token predictions p(x0 |xt ), hence these should be added together to obtain CFG. In the discrete diffusion literature, guidance is derived from a different assumption, namely a multiplicative mixture of the model predictions. This is erroneously described as the assumption used in the continuous case,5 but beyond that this is a perfectly fine modification of the generative process. More precisely, the assumption commonly used in the discrete case is the following: pD-CFG (x0 |xt , c) = p(x0 |xt )

p(x0 |xt , c)λ p(x0 |xt )λ

(23)

5 The error is most visible in the notation used for what they define as a score function ∇ log p(xs |xt , c), which is incorrect. The score function is the gradient of the log likelihood of the marginal, namely ∇ log p(xt |c), and not of the transition kernel.

14

Under review as a conference paper at ICLR 2027

Typically models are trained to predict logits, that is log probabilities, in which the decomposition becomes:   log pD-CFG (x0 |xt , c) = log p(x0 |xt ) + λ log p(x0 |xt , c) − log p(x0 |xt ) (24) The main advantage of this modification is that by changing the distribution at the logits level, there is no risk to obtain an invalid distribution, because a softmax normalization will always be applied afterwards. In the following section we will try to unravel what the differences are between these two assumptions and what possible pitfalls might be. Let us first try to see how one assumption transfers into the second one: p(x0 |xt , c)λ p(x0 |xt )λ  λ p(xt |x0 ,c) p(x |c) 0 p(x |c) p(xt |x0 ) t = p(x0 )  λ p(xt ) p(xt |x0 ) p(x0 ) p(xt )

pD-CFG (x0 |xt , c) = p(x0 |xt )

(25)

(26)

p(xt |x0 ,c)λ

p(x0 |c)λ p(xt |x0 ) p(xt |x0 )λ = p(x0 ) λ p(x0 )λ p(xt ) p(xt |c)λ

(27)

p(xt )

Realizing that due to the conditional independence of the noising process we have p(xt |x0 ) = λ t |c) p(xt |x0 , c), and if we further note that pCFG (xt ) = p(xt ) p(x we find: p(xt )λ p(x0 |c)λ p(xt |x0 ) p(x0 )λ p(xt ) p(xt |c)λλ

(28)

p(xt |x0 ) (x0 |c) CFG p (xt |c)

(29)

pD-CFG (x0 |xt , c) = p(x0 )

p(xt )

CFG

=p

(30) Now, one might think that we can invoke the conditional independence of the noising process to the sharpened distribution (i.e., setting p(xt |x0 ) = pCFG (xt |x0 , c) to obtain through Bayes that pD-CFG (x0 |xt , c)P= pCFG (x0 |xt , c). This is however erroneous. Silently doing this would require ′ CFG pCFG (xt |c) = (x0 |c),6 i.e., that the tilted marginals correspond to the noised x′0 p(xt |x0 )p version of the tilted data distribution. When applying a multiplicative mixture at the distributional level this is not true, as the convolution behind the noising operation does not commute with the multiplicative mixing operation. In fact a similar result is the reason that CFG is heavily criticized in the continuous case, the sharpening of the marginals does not result in samples that correspond to the expected sharpened data distribution. So we find that the difference between the two process lies in the non-commutativity of the noising and mixing operations, or mathematically by the non-commutativity of the convolution and the multiplication. Typical continuous CFG corresponds to sharpening all the marginals individually, while discrete CFG yields the denoiser that retrieves the sharpened data distribution. The difference between the two orders of mixing and noising are further illustrated in Figure 5. To prove this second fact let’s derive which prediction pλ (x0 |xt ), or rate function, would yield a λ 0 |c) . For this let us write out the λ-sharpened data distribution pλ (x0 |c) = pCFG (x0 |c) = p(x0 ) p(x p(x0 )λ true predictor of that distribution namely: p(xt |x0 )pλ (x0 |c) ′ λ ′ x′ p(xt |x0 )p (x0 |c)

pλ (x0 |xt , c) = P

(31)

0

6 This needs to hold to apply Bayes’ rule as a valid joint distribution should be obtained. Alternatively the sum over x0 in eq. (30) should yield 1.

15

Under review as a conference paper at ICLR 2027

mixing (D-CFG)

noising (D-CFG)

noising (C-CFG)

mixing (C-CFG)

p1

p2

D-CFG: mix, then noise

C-CFG: noise, then mix

Figure 5: Illustration of the non-commutativity of noising and mixing in the case of a multiplicative mixture. Continuous CFG first noises the data distribution and mixes the marginals, whereas discrete CFG first mixes the distributions and then only noises.

Or by flipping the conditioning using Bayes and only looking at the x0 dependence as the rest is simply a normalization constant that will be taken care off by the softmax:
1−λ
λ pλ (x0 |xt , c) ∝x0 p(xt |x0 )p(x0 ) p(xt |x0 , c)p(x0 |c)
1−λ
λ = p(x0 |xt )p(xt ) p(x0 |xt , c)p(xt |c) 1−λ

= p(x0 |xt )

λ

p(x0 |xt , c) p(xt )

1−λ

∝x0 p(x0 |xt )

1−λ

λ

p(xt |c)

λ

p(x0 |xt , c)

(32) (33) (34) (35) (36)

Hence we find that the exact denoiser, or generator, that will yield samples from the tilted data distribution is precisely the one implemented in the discrete version of CFG. Hence this proves that by adding the logits together, the generator sampling from the sharpened data distribution is obtained. B.2

D ETAILED DERIVATION OF L OGICAL O PERATORS : AND , OR , NOT

In this section, the derivations leading up to the different logical operators outlined in the main paper is given in more detailed. The same ordering as in the main body is kept, starting with the derivation for the AND operator, then the OR operator and finally the NOT operator. For all cases the resulting process is analyzed when mixing the data distributions as well as when mixing the marginals themselves. Naturally as outlined in the main body for the OR and NOT operators both result in the same process due to the commutativity of their specific additive mixing operations with noising. AND -gate. (data-level mixture) In this case we consider a multiplicative mixture on the data QN distribution level, i.e., pΛ 0 (x0 ) ∝ i=1 p0 (x0 |ci ). The key challenge is to express the predictor that leads to the mixture distribution, denoted by pΛ (x0 |xt ), as a function of the assumed known conditional predictors p0 (x0 |ci ). Once this mixture predictor is known, we construct the according rate function using Eq.(4). The mixture denoiser can be computed by rewriting it as a data weighted

16

Under review as a conference paper at ICLR 2027

function of the noising kernel through Bayes: pΛ (x0 ) pΛ (x0 |xt ) = p(xt |x0 ) 0Λ pt (xt ) ∝ p(xt |x0 )

N Y

(37)

pΛ 0 (x0 |ci )

(38)

i=1

∝ p(xt |x0 )1−N

N Y

p(xt |x0 )pΛ 0 (x0 |ci )

(39)

pΛ (x0 |xt , c)

(40)

i=1

∝ p(xt |x0 )1−N

N Y i=1

Or thus the predictions distributions should be multiplied with each other, this is more commonly implemented as adding logits together: log p(x0 |xt ) ∝

N X

log p(x0 |xt , c)

(41)

i=1

From this predictor a rate function can be constructed through Eq. (4). This is precisely how CFG is commonly implemented in the discrete diffusion literature, i.e., such as in eq (8). AND gate. (marginal-level mixture) On the other hand, for the multiplicative mixing an approach reminiscent of the continuous diffusion literature can be applied, namely considering a multiplicative QN mixture of the marginals defined as pΛ t (xt ) ∝ i=1 pt (xt |ci ). Here it’s simplest to directly derive the joint rate function through: N

X dpt (xt |ci ) Y dpΛ t (xt ) = p(xt |cj ) dt dt i

(42)

j̸=i

=

X

Ri pt (xt |ci )

i

=

p(xt |cj )

(43)

j̸=i

X

N Y

Ri

i

=

N Y

p(xt |cj )

(44)

j

X

 Ri pΛ t (xt )

(45)

i

Hence the rate matrices of the joint process satisfies: RtΛ =

N X

Rti .

(46)

i=1

By construction RtΛ inherits the required properties of a valid rate matrix, namely non-negative offdiagonal entries and rows summing to zero, so the resulting process remains valid. The shape of this combination is also intuitive: as long as one of the two conditions is not satisfied, the corresponding rate will try to strongly modify the outcome. Only once xt satisfies both conditions will both rate functions remain reasonably small. This is entirely analogous to how CFG is implemented in the continuous diffusion literature, namely as an addition of two infinitesimal generators; in the latter a score function, and in this case a rate function. OR-gate. (data-level mixture) In this context, mixing multiple distributions together in data space PN happens through a sum over the parts, i.e. pΛ 1 (x1 ) = i=1 p(ci )p1 (x1 |ci ). Looking at how the denoiser for this distribution can be expressed as a function of the conditional denoisers p1 (x1 |xt , ci ),

17

Under review as a conference paper at ICLR 2027

one has: pΛ 1 (x1 ) pΛ 1 (x1 |xt ) = p(xt |x1 ) Λ pt (xt ) N X

p1 (x1 |ci ) pΛ t (xt )

(48)

p(ci )

p(xt |x1 )p1 (x1 |ci ) pΛ t (xt )

(49)

p(ci )

pt (xt |ci ) p1 (x1 |xt , ci ) pΛ t (xt )

(50)

= p(xt |x1 )

i=1

=

N X i=1

=

N X i=1

=

N X

(47)

p(ci )

p(ci |xt )p1 (x1 |xt , ci ).

(51)

i=1

Hence, the denoiser of the mixture distribution is given by a posterior-weighted average of the conditional denoisers. The corresponding rate function can then be constructed from this mixture denoiser using Eq. (4). OR-gate. (marginal level mixture) For the additive operation behind the OR gate pΛ t (x) = PN p(c ) p (x |c ) which can be interpreted as an unconditional model p (x) in this setting, with i t t i t i=1 as mixture weights the priors p(ci ), one can again derive the joint rate function directly:

X dpt (xt |ci ) dpΛ t (xt ) = p(ci ) dt dt i X = Rti p(ci )pt (xt |ci )

(52) (53)

i

pt (xt |ci ) i  Rt pt (xt ) p(xt ) i X  p(ci |xt )Rti pΛ = t (xt ) =

X

p(ci )

(54) (55)

i

Where in the second line we multiply and divide by the mixture marginal pΛ t (xt ). In other words, RtΛ =

N X

p(ci | xt ) Rti .

(56)

i=1

Importantly, the posterior quantity is interpreted here as a classifier restricted to the specified conditions. This is precisely the mechanism that allows a dataset-specific predictor to be learned while training only on instances from that dataset. A predictor approximating the datapoint-specific distributions p(x0 | xt ), with each datapoint viewed as a class, can be averaged over the corresponding deterministic datapoint-specific rates to recover a global rate, an idea closely related to the construction used in Flow Matching (Lipman et al., 2023; Gat et al., 2024). The results show that this construction extends to arbitrary partitions of the datapoints. Beyond this theoretical connection, the OR gate is not strictly necessary for class-conditional generation, since sampling a condition according to its prior, and subsequently using the corresponding conditional model, yields the same desired distribution. NOT-gate. (data-level mixture) For the NOT gate, i.e., removing a particular part from a larger distribution, the resulting distribution corresponds to the complement of that part. In particular, removing condition ci from the unconditional distribution gives

pΛ 1 (x1 ) = p1 (x1 |¬ci ) =

1 p1 (x1 ) − p(ci )p1 (x1 |ci ) . 1 − p(ci ) 18

(57)

Under review as a conference paper at ICLR 2027

Importantly, because the addition commutes with the convolution based noising kernel, the same identity holds at the marginal level:
1 pΛ pt (xt ) − p(ci )pt (xt |ci ) . (58) t (xt ) = pt (xt |¬ci ) = 1 − p(ci ) Deriving the predictor for this distribution gives: pΛ 1 (x1 ) pΛ (59) 1 (x1 |xt ) = p(xt |x1 ) Λ pt (xt )
p(xt |x1 ) p1 (x1 ) − p(ci )p1 (x1 |ci ) = (60) pt (xt ) − p(ci )pt (xt |ci ) pt (xt )p1 (x1 |xt ) − p(ci )pt (xt |ci )p1 (x1 |xt , ci ) = (61) pt (xt ) − p(ci )pt (xt |ci ) p1 (x1 |xt ) − p(ci |xt )p1 (x1 |xt , ci ) = (62) 1 − p(ci |xt ) Hence, the denoiser for the NOT operation can be written as a posterior-dependent modification of the unconditional denoiser:  p(ci |xt )  p (x |x , c ) − p (x |x ) . (63) pΛ (x |x ) = p (x |x ) − 1 1 t i 1 1 t 1 t 1 1 t 1 1 − p(ci |xt ) The corresponding rate function can then be constructed from this predictor using Eq. (4). NOT-gate. (marginal-level mixture) The removal, or avoidance, of a certain class only given it’s class-specific rate can be mathematically expressed as the subtraction of a conditional distribution from its unconditional distribution pΛ t (xt |ci ) ∝ pt (xt ) − p(ci ) pt (xt |ci ). The joint rate can again be derived as:7   1 dpΛ t (xt ) = Rt pt (xt ) − p(c)Rti pt (xt |c) (64) dt 1 − p(ci ) 

 1 = Rt pt (xt ) − p(ci )pt (xt |ci ) − p(ci )pt (xt |c) Rti − Rt (65) 1 − p(ci ) 
 pt (ci |xt ) Rtc − Rt pΛ (66) = Rt − t (xt ) 1 − pt (ci |xt ) We therefore find:
p(c | xt ) RtΛ = Rt − R c − Rt . (67) 1 − p(c | xt ) t This equation strongly resembles a classifier-free guidance equation, in which the posterior ratio can be seen as a dynamic guidance scale that is small if the undesired class if unlikely to be generated and that becomes asymptotically large if that class becomes likely to be generated. Again, this is entirely analogous to what is found in the continuous case, with as single difference that the continuous generators, or score-functions, are replaced by their discrete analogues, the rate functions Koulischer et al. (2025b).

B.3

H ISTORY INDEPENDENCE OF THE POSTERIOR

An important point is that the posterior tracked throughout inference is the likelihood conditioned on the whole denoising trajectory and not only its least noisy state xt . This is a consequence of the way the posterior estimation is constructed through the Markov chain: p(x0 , ..., xt |c) (68) p(c|x0 , ..xt ) = p(c) p(x0 , ..., xt ) p(x0 , ..., xt−1 |c) p(xt |xt−1 , c) = p(c) (69) p(x0 , ..., xt−1 ) p(xt |xt−1 , c) p(xt |xt−1 , c) = p(c|x0 , ..., xt−1 ) (70) p(xt |xt−1 , c) (71) 7

The same remark as for the OR gate holds here.

19

Under review as a conference paper at ICLR 2027

Which corresponds to the log posterior update scheme described: log p(c|x0 , ..xt ) = log p(c|x0 , ..., xt−1 |c) + log p(xt |xt−1 , c) − log p(xt |xt−1 , c)

(72) (73)

Crucially however, thanks to the conditional independence of the noising process, the following holds: p(x0 , ..., xt |c) p(c|x0 , ..xt ) = p(c) (74) p(x0 , ..., xt ) p(xt |c) p(x0 , ..., xt−1 |xt , c) = p(c) (75) p(xt ) p(x0 , ..., xt−1 |xt ) = p(c|xt ) (76) (77) Here the important part is that the dependence on the whole trajectory drops out due to the fact that conditional and unconditional sequences get noised in the same way i.e., the noising is independent on the condition. However, for this to hold the sampler needs to be memoryless in the sense that when the denoising process is reversed it should give rise to the noising process itself. This subtlety is not satisfied by all samplers, such as the ODE sampler of DFM that we use in this paper. An example of a moment where such a subtlety is relevant is when using the uniform model. Without a purely stochastic ancestral sampler the quantity tracked through inference is not exactly equal to the marginal computed posterior, as shown in Figure 6 . We find however that the DFM sampler works best in practice on trained models, such as the ones trained on MNIST or OAS. An advantage of the masking noising scheme is that this equality is always satisfied, meaning that the posterior tracking will always give rise to the true marginal computed posterior. This is because in the case of a masked model the least noisy state truly holds all the information regarding the final state when compared to it’s history. In essence the older states contain the exact same words but with a higher masking rate.

C

A DDITIONAL DETAILS REGARDING SYNTHETIC DATA EVALUATION

To verify our theoretical framework, in particular the online posterior estimation and the logical operators, we first work in a synthetic regime in which everything is analytically tractable. Appendix C.1 describes the constructed dataset and how the conditional and unconditional predictors are obtained in closed form, the latter both by exhaustive enumeration and as a mixture of the conditionals. Appendix C.2 compares the online posterior estimate to the exact ground-truth quantity. Appendix C.3 validates the logical operators AND, OR and NOT, alongside an illustration of the differences between continuous and discrete CFG. All experiments are run for both masking and uniform noise. C.1

DATASET CONSTRUCTION

We use binary sequences of length L = 20 over the tokens “A” and “B” visualized in red and blue, with either two or four classes specifiable. Classes are defined by the majority likelihood of one token in a certain region. For two classes the majority token dominates the entire sequence, while for four classes it dominates only the first or the second half, with the other half remaining uniform. The four-class set-up is mainly used to obtain non-mutually-exclusive classes, such that logical operations like AND can be verified. This is summarized in Table 3. NB A Within a class all token positions are independent, so that p1 (x1 |c) = pN with NA and A (1 − pA ) NB the number of “A” and “B” tokens. With this set-up every sequence has a non-zero likelihood under every class: even a sequence consisting solely of “B” can be sampled by the “A” model. The corruption also acts independently per position, so the noisy marginals factorize as well. With κt the noise schedule and V = 2 data tokens, uniform noise gives t pt (xjt = A|c) = κt pA + 1−κ V ,

t pt (xjt = B|c) = κt (1 − pA ) + 1−κ V ,

(78)

while for masking a position is A with probability κt pA , B with probability κt (1 − pA ) and MASK with probability 1 − κt . The conditional marginal pt (xt |c) is the product over positions. This 20

Under review as a conference paper at ICLR 2027

class A B A1 B1 A2 B2

p1 (xj1 = A|c) for j ∈ H1

p1 (xj1 = A|c) for j ∈ H2

pA 1 − pA pA 1 − pA 1/2 1/2

1/2 1/2 pA 1 − pA

Table 3: Synthetic data construction. The sequences are split into two halves H1 , H2 of 10 tokens each, filled in with a majority token whose strength is set by a tunable parameter pA . For two classes both halves share the same majority likelihood. For four classes only one half is dominated by a token while the other is uniform, which increases the overlap between classes and makes logical operations such as AND more interesting. also gives the conditional predictors: under masking a masked position simply predicts token “A” with pA and token “B” with 1 − pA . Under uniform noise the observed token xjt is not a reliable copy of the clean token, since with probability 1 − κt it was replaced by a uniformly random one, so the conditional predictor weighs the class prior against how well each candidate explains the observation: p(xj1 = a|xjt , c) ∝ p1 (xj1 = a|c) pt (xjt |xj1 = a) with pt (xjt = v|xj1 = u) = κt ⊮[v = j j t u]+ 1−κ V , where p1 (x1 = A|c) is the probability listed in Table 3 and p1 (x1 = B|c) its complement. The normalization is exactly the marginal of Eq. 78, and the prediction interpolates between the class prior p1 (xj1 |c) at κt = 0 (pure noise, no information) and the observed token at κt = 1 (no noise). Now to obtain an unconditional model that correctly samples from this dataset, two options can 20 be used. The first is a summation over the mixture dataset: construct all 2P ≈ 1M sequences, assign each the average likelihood under the conditional models, p1 (x1 ) = c p(c) p1 (x1 |c), and loop over them. The noisy marginal and the unconditional denoiser are then obtained by explicit summation, with pt (xt |x1 ) the corruption kernel: P X x1 : xj1 =a p1 (x1 ) pt (xt |x1 ) j . (79) pt (xt ) = p1 (x1 ) pt (xt |x1 ), p(x1 = a|xt ) = pt (xt ) x 1

The second, far simpler option is to define the unconditional model as the mixture of all conP j ditionals, i.e., an OR gate over the classes, p(xj1 |xt ) = c p(c|xt ) p(x1 |xt , c). The sole difficulty is the Pposterior p(c|xt ), which is perfectly tractable through Bayes’ rule and Eq. 78, since pt (xt ) = c p(c) pt (xt |c). Both constructions are exactly equivalent, which we also confirm numerically using the enumeration. In all further experiments we use the OR-gate version, as it does not require an exhaustive summation over the 2L sequences. C.2

VALIDATION OF POSTERIOR ESTIMATION

The online posterior estimation can be compared to the exact quantity, given by the ratio of the conditional and unconditional marginals of Eq. 78: pt (c|xt ) = p(c)

pt (xt |c) pt (xt )

(80)

The posterior can also be tracked with the log-posterior update derived in the main body: log pt (c|x0 , ..., xt ) = log pt−1 (c|x0 , ..., xt−1 ) + log pt−1 (xt |xt−1 , c) − log pt−1 (xt |xt−1 ) (81) This equation still allows for some freedom, namely in the choice of transition likelihoods, or sampler. In this work we mainly focus on two specific cases the kinetic optimal discrete flow matching sampler and the stochastic ancestral sampler. Also, note that it is perfectly sound to use one choice for sampling and another for tracking the posterior. As shown in Appendix B.3, if a memoryless stochastic sampler is used to compute the transition likelihoods, the following holds pt (c|x0 , ..., xt ) = pt (c|xt ): the entire posterior information is contained in the least noisy point of the trajectory, i.e., the current state. For masking this always holds, as tokens remain unmasked once revealed, and the flow matching and ancestral samplers coincide. For uniform noise it holds only if the denoising kernel is derived as the reverse of the corruption kernel, i.e., for an ancestral sampler. 21

Under review as a conference paper at ICLR 2027

approximated (Ancestral evaluator)

exact

1.0

Ancestral: p(c|xt)

Ancestral: p(c|xt)

exact

0.5 0.0

1.0 0.5 0.0 1.0

DFM: p(c|xt)

DFM: p(c|xt)

1.0

approximated (DFM evaluator)

0.5 0.0 0.00

0.25

0.50 t

0.75

0.5 0.0 0.00

1.00

(a) Using ancestral transition likelihoods

0.25

0.50 t

0.75

1.00

(b) Using dfm transition likelihoods

Figure 6: Illustration of the online posterior evaluation under different samplers and evaluators. In (a), samples are taken from an ancestral sampler at the top and a dfm sampler at the bottom and the posterior likelihoods are updated using the ancestral transition likelihoods. In this case the tracked posterior is equal to the ground truth posterior. In (b), the exact same samples are shown but with this time the DFM transition likelihoods used for the posterior estimation, in which case the tracked quantity is not equal to the exact one anymore. Figure 1a shows that the tracked posterior is exact for both noise types when the ancestral sampler is used. In Figure 6a we show that if the posterior likelihood is computed using the transition likelihoods predicted by the ancestral sampler, the tracked quantity is equal to the ground truth quantity regardless of whether an ancestral or dfm sampler is used to generate the sample. On the other hand in Figure 6b, we show that if the transition likelihoods are computed using the flow matching sampler the tracked and exact quantities are no longer equal, i.e., there is a real difference between pt (c|x0 , ..., xt ) and pt (c|xt ). These do however remain strongly correlated, with for instance the dashed and full curves going up and down together, simply with the wrong scale. An important note is that in practice, on real trained models, we find that the flow matching sampler yields to more stable and accurate posterior estimation. We hypothesize that this discrepancy between theory and practice is also caused by the fact that the deterministic ODE shows smoother less stochastic trajectories, such that the posterior information is easier to discern for the trained model. A precise understanding of this phenomenon is however left as future work. C.3

VALIDATION OF THE LOGICAL OPERATORS

To validate the logical operators we compare the generated samples to the exact target distribution of each gate. Comparing full sequences would require far too many samples, so we instead count the number of tokens “A” in the first and in the second half, (n1 , n2 ). As all classes are exchangeable within a half, each target (the AND-law, and the mixtures underlying OR and NOT) depends on a sequence only through these counts, such that its exact distribution T (n1 , n2 ) is easily computed. We then measure the KL divergence between the empirical distribution of (n1 , n2 ) over M generated samples and T . We do this not only for the final samples, but also for the noisy states xt at intermediate noise levels, such that Fig. 7 shows how the generated distribution approaches the target as denoising progresses. For the AND gate it is important to also keep the forward kernel in the predictor. The exact data-level predictor for the AND gate is log p(x1 |xt ) =

N X

log p(x1 |xt , ci ) + (1 − N ) log pt (xt |x1 ) + const,

(82)

i=1

and not simply the sum of the logits of the conditional denoisers, as each of these already contains the forward kernel pt (xt |x1 ) once, while the target should contain it only once. For masking the 22

Under review as a conference paper at ICLR 2027

masking

AND(A1, B2)

uniform

exact process

OR(A1, B1)

NOT(A1)

KL over (n1, n2)

101 100 10−1 10−2 0.2

0.4

t

0.6

0.8

1.0

0.2

0.4

t

0.6

0.8

1.0

0.2

0.4

t

0.6

0.8

1.0

Figure 7: Evolution of the KL divergence between the distribution of (n1 , n2 ) of the noisy state xt and that of the clean target, for AND(A1 , B2 ), OR(A1 , B1 ) and NOT(A1 ), under masking (full) and uniform (dotted) noise. In all cases the derived gates sample from the desired target distribution, illustrated by the KL distance between generated and reference samples going to zero.

additional term does not depend on x1 and drops out of the softmax. For uniform noise however pt (xjt |xj1 ) favors the observed token, such that the term does depend on x1 and has to be kept. As it only involves the forward kernel it is known exactly, without needing p1 or the model. Importantly for a CFG-like mixture of conditional and unconditional distributions this factor does not exist as there, the base distribution is multiplied with a ratio of conditional and unconditional distributions. As such, this factor appears for both numerator and denominator and cancels out cleanly. The other two gates, OR and NOT, are weighted averages of the conditional denoisers and require no such correction.

D

A DDITIONAL DETAILS REGARDING IMAGE GENERATION

In this section additional details and results on the image modality are provided. First, Appendix D.1 describes the architecture and the training hyperparameters. Second, Appendix D.2 shows ablations on the model performance, namely FID, precision, recall and C2ST as a function of the guidance scale and of the number of sampling steps. Then, Appendix D.3 the posterior estimations is analyzed in detail. Finally, Appendix D.4 contains an in depth discussion of the results obtained using the NOT gate.

D.1

T RAINING DETAILS

Our image models are based on the DiT architecture (Peebles & Xie, 2022), adapted to discrete tokens. Every pixel value is a token from a vocabulary of 256 values (plus a mask token for masking diffusion), such that an MNIST image is a sequence of 784 tokens. The tokens are embedded, processed by transformer blocks and decoded into a categorical distribution over the 256 pixel values for every pixel. The noise level t and the class label are provided through adaptive layer normalization (AdaLN-Zero), where the label embedding has an additional entry for the unconditional case. The model is trained to predict the clean image with a cross-entropy loss, which for masking is only computed on the masked tokens and for uniform noise on all tokens. Noise levels are sampled in a stratified way, using a linear noise schedule. Conditional models are trained with label dropout, while unconditional models are trained separately by always dropping the label. As is typical for diffusion models, an exponential moving average (EMA) of the weights is used for sampling. The hyperparameters, listed in Table 4, were set to standard values (Shi et al., 2024) and not tuned per noise type. 23

Under review as a conference paper at ICLR 2027

Architecture Parameters Depth / hidden size / heads Sequence length / vocabulary Optimizer Learning rate Betas / weight decay Warmup steps / schedule Batch size Training steps Gradient clipping EMA decay Label dropout Noise schedule Noise level sampling Loss weighting

DiT with AdaLN-Zero 7.6M 6 / 256 / 4 784 / 256 (+ mask token) AdamW 4 · 10−4 (0.9, 0.999) / 0.01 1000 / cosine decay 512 200k 1.0 0.9999 0.15 linear stratified in [0, 1] none

Table 4: Hyperparameters of the MNIST models. The unconditional models are trained separately with label dropout 1.0, i.e., without ever seeing the label. The cosine decay is defined over a horizon longer than the training run, so that the learning rate only decays to about 3 · 10−4 at the end of training. Figure 8 shows the validation loss per noise level during training. In both cases the loss decreases with t, as less information has to be inferred at lower noise levels. The highest noise levels converge first, presumably because they are dominated by uncertainty that cannot be reduced (many images are consistent with a nearly empty input), while the low noise levels keep improving until the end of training. The two noise types are not directly comparable, as masking averages over the masked tokens only. At low noise the loss is only over masked tokens, so the loss stays high, whereas for uniform noise most tokens are already noise-free and thus easy to guess, so the loss goes to zero.

0.0-0.1 0.1-0.2

0.2-0.3 0.3-0.4

Noise level t 0.4-0.5 0.5-0.6

0.6-0.7 0.7-0.8

Validation loss

Masking

0.8-0.9 0.9-1.0

Uniform

100 10 1 0

25

50

75 100 125 150 175 200 0

Training steps (k)

25

50

75 100 125 150 175 200

Training steps (k)

Figure 8: Validation loss during training of the MNIST models, averaged over bins of noise levels t ∈ [0, 0.1], [0.1, 0.2], . . . , for masking (left) and uniform noise (right). D.2

M ODEL ABLATIONS

We evaluate the generated images with three metrics, all computed on InceptionV3 features of 4096 generated samples compared to the training set. The Fréchet inception distance (FID) is the Fréchet distance between Gaussian fits to the generated and the real features, and measures overall quality and diversity (lower is better). Precision and recall are k-nearest-neighbor based: precision is the fraction of generated samples that lie within the manifold of the real data (fidelity), recall the fraction of real samples that lie within the manifold of the generated data (diversity). The C2ST is the heldout accuracy of a logistic regression classifier that separates real from generated features, where 0.5 means the two are indistinguishable. Figure 9 shows the effect of the guidance scale on these metrics. 24

Under review as a conference paper at ICLR 2027

Conditional models (scale 1) are clearly better than unconditional ones (scale 0), and unconditional models that are trained separately (stars) are better than the unconditional mode of the jointly trained model.

FID ( )

20 15 10 5 0

1

2

3

Guidance Scale

1.0

Uniform Cosine

Separately trained unconditional Linear Sin 1.0

C2ST Accuracy ( )

Precision & Recall ( )

Masking 0.8 0.6 0.4 0.2

0.0 0

Precision Recall

1

2

0.6 0.4 0.2 0.0 0

3

Guidance Scale

0.8

1

2

Guidance Scale

3

Figure 9: Sweep over the guidance scale for the MNIST-trained model. CFG is implemented as usual in the discrete case, i.e., a linear combination of logits. During inference 128 steps of the specified schedulers were taken. The stars correspond to the separately trained unconditional model.

4

8

16

32

64 128

Number of Steps

Masking Cosine 1.0

Linear

Uniform

Sin

0.8 0.6 0.4 0.2 0.0 4

Precision Recall

8

16

32

C2ST Accuracy ( )

250 200 150 100 50 0

Precision & Recall ( )

FID ( )

Figures 10 and 11 show the dependence on the number of sampling steps. The quality increases with the number of steps in all settings. For masking there is a clear ranking of the sampling schedules, with the cosine schedule, which is slow early and fast late, performing best. This is expected: early in the denoising process the tokens have to be placed carefully, while late steps are easy as most of the image is already fixed. For uniform noise the schedule matters less, thanks to the ability to correct earlier mistakes. The differences between schedules shrink as the number of steps grows, as a schedule is only a smarter way to use a given compute budget and not a better model.

64 128

Number of Steps

1.0 0.8 0.6 0.4 0.2 0.0 4

8

16

32

64 128

Number of Steps

Figure 10: Sweep over the number of sampling steps for the unconditional MNIST-trained model. Masking and uniform models are evaluated under three schedulers; sine (fast at first and then slow), linear (constant sped) and cosine (slow at first and then fast). From left to right, the FID, the Precision and Recall and the C2ST accuracy. 25

Under review as a conference paper at ICLR 2027

FID ( )

200 150 100 50 0

4

8

16

32

64 128

Number of Steps

Linear

Uniform

Sin

0.8 0.6 0.4 0.2 0.0 4

Precision Recall

8

16

32

C2ST Accuracy ( )

Precision & Recall ( )

250

Masking Cosine 1.0

64 128

Number of Steps

1.0 0.8 0.6 0.4 0.2 0.0 4

8

16

32

64 128

Number of Steps

Figure 11: Sweep over the number of sampling steps for the conditional MNIST-trained model. Masking and uniform models are evaluated under three schedulers; sine (fast at first and then slow), linear (constant sped) and cosine (slow at first and then fast). From left to right, the FID, the Precision and Recall and the C2ST accuracy.

Finally, the uniform model outperforms the masking model, especially for a low number of sampling steps. This comparison should be taken with care, as the training hyperparameters were set to standard values and not tuned per method, so they may happen to suit the uniform model better. As the cosine scheduler (slow at first, fast at the end) outperforms all others all experiments use it during sampling. Furthermore, for maximal accuracy, all experiments were performed at 128 steps. All metrics in this modality are computed on 4096 samples. D.3

P OSTERIOR E STIMATION

In the context of image generation, the posterior can be properly normalized by marginalizing over all classes. In practice, we achieve this by tracking the log-posterior for each class and subsequently applying a softmax over all classes. This results in a normalized and directly interpretable posterior distribution. Moreover, considering all classes jointly provides a clearer classification signal, as predictions are compared against one another rather than only against an unconditional prediction. Examples similar to the one presented in the main body in Figure 1b, including an example in which the trajectory is misclassified, are shown in Figure 6. In all examples, the posterior concentrates on the final class more rapidly under masking noise than under uniform noise. To quantify the effectiveness of the online posterior estimation, we consider two main metrics: onevs-rest AUC and classification accuracy. Both metrics are evaluated as a function of denoising time, highlighting the main advantage of our approach: the posterior provides a classification signal before generation is complete. Ground-truth labels for the generated samples are obtained using a pretrained ViT classifier (Dosovitskiy et al., 2021). The accuracy is computed as the agreement rate between the top posterior prediction and the class assigned by the ViT. For the AUC, we use the macro-averaged one-vs-rest (OvR) extension of the ROC AUC to the multiclass setting. For each class c, samples assigned class c by the ViT are treated as positives and all remaining samples as negatives, while the posterior probability pt (c | xt ) is used as the ranking score to compute the corresponding binary ROC AUC. The per-class AUCs are then averaged uniformly over the 10 classes. An AUC of 0.5 corresponds to chance level, which is also the expected value at t = 0, when no class-specific evidence has yet been accumulated. The evolution of both metrics throughout denoising, for the three modalities evaluated in this work, is shown in Fig. 13. For a given noise schedule, the masking variant consistently accumulates class information more rapidly, which is expected from its inability to modify tokens once they have been unmasked. It is important to emphasize that we do not claim that our posterior provides a perfect classifier in this setting. Rather, its primary advantage is that it provides a proxy for the final class before generation is complete. As shown by both the AUC and accuracy curves, the classification signal largely saturates after approximately half of the generation process. This saturation is likely related to the out-of-distribution nature of the conditional models at low noise levels. As denoising progresses, the conditional marginals increasingly separate, such that evaluations of a model conditioned on one class become less meaningful for states belonging to another class. For example, 26

Under review as a conference paper at ICLR 2027

2

3

4

0.25

5

6

7

8

0.50

9

prior

0.75

t

0

Uniform: p(c|xt)

1.00 0.75 0.50 0.25 0.00 0.00

1

1.00 0.75 0.50 0.25 0.00

Masking: p(c|xt)

Uniform: p(c|xt)

1.00 0.75 0.50 0.25 0.00

Masking: p(c|xt)

0

1.00 0.75 0.50 0.25 0.00 0.00

1.00

1

3

0.25

4

5

6

7

0.50

t

4

5

6

7

8

0.50

9

prior

0.75

t

1.00

(b) Successful example

8

9

prior

0.75

0

1.00

(c) Successful example

Uniform: p(c|xt)

Masking: p(c|xt)

1.00 0.75 0.50 0.25 0.00 0.00

2

1.00 0.75 0.50 0.25 0.00

Masking: p(c|xt)

Uniform: p(c|xt)

1.00 0.75 0.50 0.25 0.00

1

3

0.25

(a) Successful example 0

2

1.00 0.75 0.50 0.25 0.00 0.00

1

2

0.25

3

4

5

6

0.50

t

7

8

0.75

9

prior

1.00

(d) Unsuccessful example

Figure 12: Illustration of the online posterior evaluation. In (a-c), trajectories for which the online posterior estimate recovers the final class for both masking and uniform noise are shown. In (d), an example is shown for which the masking model does not recover the correct class label. All shown trajectories use τ = 0.1, a value taken from a calibration run on the NOT gate.

a model trained exclusively on “2”s cannot be expected to accurately predict the transition probabilities required to transform a partially denoised “4” into what it knows. This limitation occurs primarily once the sample has already been resolved, at which point the posterior has little additional utility. The posterior most informative precisely during the earlier stages of denoising, when class information is still emerging and an early estimate is most useful.

D.4

L OGICAL OPERATORS : NOT GATE

For the NOT gate, avoiding the generation of a given digit is a trade-off between image quality and safety. We track quality with the FID with respect to reference data that excludes the unwanted digit, and safety with the violation rate, i.e., the fraction of generated images that a ViT classifier assigns to the unwanted digit. As a static baseline we use negative prompting, a standard method from continuous diffusion, which is CFG with a negative guidance scale −λ. To stay consistent with the discrete literature, CFG is implemented as usual by an addition of logits. Figure 14 shows FID and violation rate for a sweep over λ. A very small scale already suffices to push away from the unwanted digit, while larger scales quickly harm the image quality. 27

1.0

1.0

0.9

0.8

0.8

Accuracy

AUC

Under review as a conference paper at ICLR 2027

Text, masking MNIST, masking MNIST, uniform Antibody, masking Antibody, uniform

0.7 0.6 0.2

0.4

t

0.6

0.8

Text, masking MNIST, masking MNIST, uniform Antibody, masking Antibody, uniform

0.4 0.2

0.5 0.0

0.6

0.0

1.0

0.0

0.2

(a) AUC

0.4

t

0.6

0.8

1.0

(b) Accuracy

Figure 13: Evaluation of the online posterior estimation for different modalities during denoising. In (a), the OvR AUC. In (b), the accuracy.

25

NOT(1) 0.10

20

0.08

Violation Rate ( )

FID ( )

NOT(0)

15 10 5 0.0

0.2

0.4

0.6

0.8

Constant Guidance Scale

0.06 0.04 0.02 0.00

1.0

NOT(2)

0.0

0.2

0.4

0.6

0.8

Constant Guidance Scale

1.0

Figure 14: Static negative guidance (negative prompting): FID and violation rate as a function of the constant guidance scale λ, for NOT(0), NOT(1) and NOT(2).

Our method has two hyperparameters: the initial guidance scale s0 , which mathematically is a parameter of the prior, and the inverse temperature τ with which the posterior updates are rescaled. In this setting the results were of sufficient quality that the offset parameter δ was not needed. We perform a grid search over both hyperparameters and analyze both metrics (Figure 15). A larger s0 increases the strength of the guidance, whereas a larger τ makes it fade faster. For small τ (≤ 0.05) the trade-off is almost independent of τ , while for larger values the guidance fades too quickly and more violations remain. We therefore fix τ = 0.01 in all further experiments. 28

Under review as a conference paper at ICLR 2027

Violation Rate ( )

8.0

5.14

4.78

4.60

4.62

4.83

4.93

5.32

5.37

0.1

4.93

4.52

4.47

4.65

5.07

5.45

6.09

6.44

0.05

4.80

4.32

4.41

4.98

5.50

5.80

6.62

7.37

0.06

0.05

0.03

0.02

0.02

0.02

0.02

0.01

0.1

0.06

0.05

0.03

0.02

0.01

0.01

0.01

0.00

0.05

0.06

0.04

0.02

0.01

0.01

0.00

0.00

0.00

0.03

0.03

0.06

0.04

0.02

0.01

0.00

0.00

0.00

0.00

0.02

0.01

0.05

0.04

0.02

0.01

0.00

0.00

0.00

0.00

0.1

0.2

0.4

0.6

0.8

1

1.4

2.2

0.2

0.09

0.08

0.06

0.05

0.05

0.04

0.04

0.03

0.1

0.09

0.07

0.05

0.04

0.03

0.02

0.02

0.01

0.05

0.08

0.06

0.03

0.02

0.01

0.01

0.01

0.00

0.03

0.08

0.05

0.02

0.01

0.01

0.00

0.00

0.00

0.01

0.07

0.05

0.02

0.01

0.00

0.00

0.00

0.00

0.1

0.2

0.4

0.6

0.8

1

1.4

2.2

0.2

0.07

0.05

0.03

0.03

0.02

0.02

0.01

0.01

0.1

0.06

0.05

0.03

0.02

0.01

0.01

0.01

0.00

0.05

0.06

0.04

0.02

0.01

0.01

0.01

0.00

0.00

0.03

0.06

0.04

0.02

0.01

0.01

0.00

0.00

0.00

0.01

0.06

0.04

0.02

0.01

0.01

0.00

0.00

0.00

0.1

0.2

0.4

0.6

0.8

1

1.4

2.2

0.05

7.0 6.5 6.0 5.5

0.03

4.64

4.20

4.11

5.02

5.64

6.10

7.27

7.85

0.01

4.44

3.96

4.26

4.97

5.80

6.72

7.37

8.01

0.1

0.2

0.4

0.6

0.8

1

1.4

2.2

0.2

7.28

6.55

5.45

5.22

4.87

4.86

4.85

4.80

0.1

6.79

5.78

4.81

4.57

4.56

4.77

5.05

5.60

Initial Guidance Scale

5.0 4.5 4.0

FID ( )

0.05

20.0

6.21

5.03

4.31

4.68

5.19

5.86

Initial Guidance Scale

7.34

9.52

17.5 15.0

0.03

5.79

4.83

4.37

5.77

6.97

8.33

11.55

15.72

0.01

5.34

4.38

4.87

7.07

9.70

12.56

17.65

24.59

0.1

0.2

0.4

0.6

0.8

1

1.4

2.2

4.76

4.51

3.88

3.86

4.05

10.0 7.5 5.0

0.2

3.99

3.86

Initial Guidance Scale

0.1

4.57

4.22

3.95

3.83

3.77

3.88

4.09

4.50

0.05

4.38

4.04

3.67

3.70

3.68

3.96

4.25

5.27

0.03

4.26

3.83

3.51

3.46

3.78

4.11

4.98

6.07

0.01

4.20

3.63

3.29

3.50

3.98

4.51

5.61

7.23

0.1

0.2

0.4

0.6

0.8

1

1.4

2.2

Inverse Temperature

Inverse Temperature

0.06

0.00

Violation Rate ( )

7.0 6.5 6.0 5.5 5.0 4.5

3.5

0.06 0.05 0.04

4.0

Initial Guidance Scale

0.08

0.02

FID ( )

4.05

0.01

0.04

12.5

Initial Guidance Scale

0.04

Violation Rate ( )

22.5

Inverse Temperature

Inverse Temperature

0.06

0.2

7.5

Inverse Temperature

Inverse Temperature

FID ( ) 0.2

Initial Guidance Scale

0.03 0.02 0.01

Figure 15: FID and violation rate of the dynamic NOT gate as a function of the initial guidance scale and the inverse temperature τ , for NOT(0), NOT(1) and NOT(2) (top to bottom). Amber is best, blue is worst.

Figure 16 shows FID and violation rate as a function of s0 . The initial guidance scale is an intuitive way of adding more guidance, and results in a trade-off profile very similar to that of the static baseline. 29

Under review as a conference paper at ICLR 2027

25

NOT(1) 0.10

20

0.08

Violation Rate ( )

FID ( )

NOT(0)

15 10 5 0.0

0.5

1.0

1.5

Initial Guidance Scale

NOT(2) 10% violation rate

0.06 0.04 0.02 0.00 0.0

2.0

0.5

1.0

1.5

Initial Guidance Scale

2.0

Figure 16: Dynamic negative guidance with τ = 0.01: FID and violation rate as a function of the initial guidance scale, for NOT(0), NOT(1) and NOT(2). To compare both methods, Figure 17 (a copy of the figure in the main body) plots FID against violation rate. The dynamic approach outperforms the static baseline, at the cost of one additional hyperparameter. NOT: 0

FID ( ↓ )

10

Negative CFG

NOT gate: Dynamic Negative Guidance (Ours)

NOT: 1

NOT: 2

0.00 0.02 0.04 0.06 0.08 0.10

0.00 0.02 0.04 0.06 0.08 0.10

8 6 4 0.00 0.02 0.04 0.06 0.08 0.10

Violation Rate ( ↓ )

Violation Rate ( ↓ )

Violation Rate ( ↓ )

Figure 17: FID against violation rate for dynamic negative guidance (full) and the static baseline (dashed), for NOT(0), NOT(1) and NOT(2). Interestingly, the profile of the guidance scale is very consistent (Figure 18). The guidance is mainly needed at the start of the generation and then decays roughly exponentially, where the trajectory only determines how fast. This suggests once more that for discrete image generation the first pixels are decisive, not only for the quality, as reflected by the choice of a cosine scheduler, but also for the class. NOT(0)

Guidance scale

1.0

NOT(1)

NOT(2)

Single trajectory Mean

0.8 0.6 0.4 0.2 0.0

0.0

0.2

0.4

t

0.6

0.8

1.0

0.0

0.2

0.4

t

0.6

0.8

1.0

0.0

0.2

0.4

t

0.6

0.8

1.0

Figure 18: Guidance scale of the dynamic NOT gate over the sampling process (τ = 0.01, initial guidance scale 1): single trajectories (black) and their mean. All results are shown for the uniform model. The masking model shows qualitatively similar behavior. 30

Under review as a conference paper at ICLR 2027

E

A DDITIONAL DETAILS REGARDING TEXT EVALUATION

In this appendix additional details and results on the text modality are provided. First, Appendix E.1 describes the experimental setup for the text modality. Second, Appendix E.2 analyzes the posterior estimation in more detail. Finally, Appendix E.3 contains an in depth discussion of the results obtained using the NOT gate. E.1

E XPERIMENTAL SETUP DETAILS FOR TEXT EVALUATION

In our analysis on the text modality, we use the Cognac (Chen et al., 2022) dataset and benchmark. For the posterior evaluation, we use the dataset as a set of prompts to verify whether the answer is in a certain class. For the NOT gate evaluation, we use the benchmark, where the (guided) model needs to generate text about one topic (e.g., beverages), but it cannot generate anything related to a subcategory (e.g., wine), the class. An example of a prompt in this dataset is given in Figure 19. There is an instruction to generate about a specific topic, some examples and finally some formatting instructions. We extended the formatting instructions, as the DLM otherwise failed often to format correctly. Note that this is the prompt unconditional to the class, so it does not mention that class. Generate examples that are under the category of beverage. == red wine is a type of wine made from dark colored grape varieties. == == pinot noir (french: [pino nwaK]) is a red wine grape variety of the species vitis vinifera. == == turkish coffee is a style of coffee prepared in a cezve using very finely ground coffee beans without filtering. == Now keep generating. Only output the continuation in == ... == format, no other text. Your continuation must be entirely original.

Figure 19: Neutral, unconditional prompt Answer evaluation. The evaluation uses WordNet (Miller, 1995) to evaluate the generated text. It leads to three possible outcomes: “Class”, “Other classes” and “Invalid”. First, it checks whether the model generated something related to the topic at all, by finding literal matches of WordNet leaf hyponyms (i.e., words that fall under the topic). This method showed some limitations, as literal string matching on the semantically incomplete WordNet had quite some false negatives. To alleviate this issue, we introduce LLM as a judge with the LLM being Qwen3 8B (Yang et al., 2025). We find that the combination of both (so either a WordNet match or LLM as a judge match) works best for a fair evaluation. This is because WordNet catches niche words, while the LLM catches non-literal matches. As an example, we take the dev-set ID 31, with topic snack food and class (subtopic) sandwich. WordNet has only four leaves that would be counted on topic, but not in the class: chip, corn chip, potato skin, tortilla chip. While catching niche snacks such as potato skin, it is clear that the category is broader. A generated sample “== A chocolate bar is a popular snack food, typically made from cocoa and sugar. ==” was counted off topic by WordNet, but on topic by the LLM judge. As such, both methods truly complement each other. To check whether the model not only generated something on topic, but also in the specified class (subtopic), again WordNet is used with string matching. For the class detection, we found much less false negatives and adding LLM as a judge the same way increased the false positives strongly. Hence, we keep the WordNet only approach for class detection. These two checks create three possible outcomes for generated text. The first one is “Class”, which means the text is on topic and the specified class was generated. The next is “Other classes”, where the text is on topic, but not the specified class. The last option is “Invalid”, where the model fails to generate something on topic at all. As the DLMs also showed some failure modes (such as not respecting the == == format), we count degenerate output also under the class of Invalid, no matter the Wordnet matches. Prompting the model. As DLM, we choose the popular LLaDA 8B model (Nie et al., 2025). This model is prompted in three different ways. The unconditional prompt, as shown in Figure 19, does 31

Under review as a conference paper at ICLR 2027

not mention the constraint at all. The negative prompt adds an instruction to surely mention the class, as shown in Figure 20. We call this the negative prompt, because for the NOT gate, we will guide away from this prompt and thus negatively guide on it. The positive prompt has the opposite instruction, where it is asked to not mention the subcategory. We would like to highlight that classical CFG, also in text, intends to strengthen the adherence to the entire prompt, by contrasting with the unconditional prompt filled with only [MASK] tokens. However, for our guidance setup, we argue that the unconditional model is the one that sees the neutral prompt. Just like in image generation, the model already “knows” about the task that it should generate natural images, or like our MNIST model knows it should generate numbers between 0-9, the diffusion model that generates inside the data distribution for our task, is the one that is prompted with a neutral prompt that explains the task at hand. As such, we do not use pMASK as the basis of all models, but rather the neutrally prompted pneutral prompt . As this is our base model, our conditional models also build further on this prompt, only adding to the base prompt. Posterior evaluation setup. For the posterior evaluation, we combine the three prompts in the main paper. That is: the neutral prompt is used to generate the text, while the posterior of the class is calculated on the negative versus the positive prompt. We use the positive prompt here as it gives an even better posterior signal than other prompt combinations, as shown in Appendix E.2. For the NOT gate in text, we explore two different guidance setups and two baselines, with all setups being shown in Table 5. The first setup is that of the NOT gate in probability space, as outlined in the theory of Section 3.2. This model only uses the neutral and negative prompt, so it does not use the “power” of the prompted NOT. In the second setup, negative CFG, we naively apply the well known logit-space CFG, but with a negative sign to guide away from the negative prompt. As baselines, we have the unconditional model, which is unaware of any constraints. Any samples that are already correct under this model, were at no risk of violation in the first place, As such the unconditional model sets a worst case lower bound. The second baseline model is the positive prompted model. This model is interesting to compare with, as it could be argued that this captures a very rough upper bound of how much the model “knows” not to generate. NOT gate (prob-space) Negative CFG (logit-space) Unconditional Positive prompted

p = pneutral prompt − λt · (pneg prompt − pneutral prompt ) l = lneutral prompt − λ · (lneg prompt − lneutral prompt ) p = pneutral prompt p = ppos prompt

Table 5: Contrastive signals used in this paper.

Generate examples that are under the category of beverage. == red wine is a type of wine made from dark colored grape varieties. == == pinot noir (french: [pino nwaK]) is a red wine grape variety of the species vitis vinifera. == == turkish coffee is a style of coffee prepared in a cezve using very finely ground coffee beans without filtering. == Now keep generating and include things in the category of wine. Only output the continuation in == ... == format, no other text. Your continuation must be entirely original.

Figure 20: Negative prompt

E.2

P OSTERIOR ESTIMATION

In this appendix, the posterior is analyzed deeper in the following ways: we analyze the impact of the prompt types to compare during posterior estimation, we determine classification accuracy in a few-class scenario and we give more examples on high impact tokens for some prompts. 32

Under review as a conference paper at ICLR 2027

Our posterior approximation can be applied to any two models that give probabilities on the token being generated. This allows for three different couples of models in the case of “NOT” classification: negative vs positive prompt, negative vs neutral and neutral vs positive prompt. The posterior approximations for each of these runs are given in Figure 21. We note some interesting behavior, for example in Figure 21a, the posterior approximation does not move much for violated samples, but it goes down much for correct samples. We interpret it as follows: when the unconditional model was going to violate anyways, adding the negative part of the prompt does not change much. However, if the unconditional prompt wants to generate a correct sample, the negative part of the prompt adds a strong signal. Conversely, we see the opposite in Figure 21b, where the violations increase the posterior approximation and the correct samples stay rather flat. Adding both signals together by directly comparing negative vs positive, we see in Figure 21c, that violations go up and correct samples go down. Do note in the typical NOT gate context (in other modalities), only the negative vs neutral signal is present and this is also the one used in NOT gate experiments. However, for the sake of finding the best posterior approximation, we find that negative vs positive is best at classifying samples, as shown in Figure 22. In more detail: negative vs positive yields the strongest classifier via the posterior approximation, with an ROC of 0.94. The other approximations perform significantly worse with p=0.05 in a bootstrap test. In conclusion, the best posterior approximation for classification is the negative vs positive setup. Class (644)

Other classes (1220)

Class (644)

Invalid (86)

Invalid (86)

Class (644)

0

−10

0

0.2

0.4

0.6

0.8

1.0

−20 0.0

0.2

0.4

t

0

0.6

0.8

1.0

−20 0.0

0.2

0.4

0.6

0.8

1.0

t

t

(a) Negative vs neutral

Invalid (86)

−10

−10

−20 0.0

Other classes (1220)

10

Δlog(pc )

Δlog(pc )

10

Δlog(pc )

Other classes (1220)

10

(c) Negative vs positive

(b) Neutral vs positive

Figure 21: Posterior trajectories with different prompt types.

AUC = 0.869

Random

Random

AUC = 0.939 1.0

0.8

0.8

0.8

0.6 0.4 0.2 0.0

True Positive Rate

1.0

True Positive Rate

True Positive Rate

AUC = 0.911 1.0

0.6 0.4 0.2 0.0

0.0

0.2

0.4

0.6

0.8

1.0

Random

0.6 0.4 0.2 0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

False Positive Rate

False Positive Rate

False Positive Rate

(a) Negative vs neutral

(b) Neutral vs positive

(c) Negative vs positive

1.0

Figure 22: ROC curves for classification of correct vs violated using different prompt types. While it is hard in free-text generation to define a fixed set of non-overlapping classes, we attempt to construct such a few-class setup, in order to determine the classification accuracy in such tasks. To this end, we take 1000 samples from the unconditional model per prompt and we select four prompts with reasonable variety (i.e., not the same topic being generated 99% of the time). Then, the majority topics are selected per prompt, and these are defined as the classes. We use the softmax over the posteriors to determine class probability and classify the sample as the highest-probability class. Figure 23 shows a single sample being generated, while Table 6 shows that the posterior estimations yield more than 95% accuracy at the end of generation, while Figure 13b shows the accuracy during generation. 33

Under review as a conference paper at ICLR 2027

In Table 7 we extend the list of intuitive tokens tokens with high impact on the posterior from the main paper.

wine

1.0

brandy

a

the

tequila

wine

variety red

0.8

pc

0.6 0.4 0.2 0.0 0.0

0.2

0.4

0.6

0.8

1.0

t Figure 23: The posterior of multiple classes during generation of the prompt with ID 76 in the development set of Cognac. The answer of the model is “== Cabernet Sauvignon is a type of red wine made from the Cabernet Sauvignon grape variety. ==”. Some tokens are highlighted with a dashed line at the time step of generation. The blue line, which represents the wine class, clearly jumps on the words “wine”, “variety” and “red” being generated. On the other hand, the posteriors do not move much upon “a” or “the” being generated.

ID 76 276 353 418

Topic alcohol beverage liquor domestic animal

Classes (%) wine 57%, brandy 18%, tequila 26% wine 55%, beer 28%, liqueur 17% gin 65%, brandy 23%, rum 12% cat 16%, dog 84%

Accuracy 99.0% 95.1% 95.2% 95.6%

Table 6: Accuracy of the posterior estimation, classifying which (manually defined) subtopic was generated using argmax of the posterior. The IDs refer to the Cognac dataset’s development set. 34

Under review as a conference paper at ICLR 2027

Topic

Sub-topic

1st

(class c)

2nd

3rd

Token

∆ log pc

Token

∆ log pc

Token

∆ log pc

sweet

candy

+ −

candy marzipan

+2.8 -2.1

lollipop cookies

+0.8 -0.7

are sweet

+0.5 -0.6

dairy product

cheese

+ −

cheddar yogurt

+2.4 -6.8

Swiss cream

+1.4 -0.5

cheese product

+0.5 -0.5

dairy product

milk

+ −

milk whey

+0.8 -0.7

with cheese

+0.4 -0.7

whole liquid

+0.4 -0.6

alcohol

wine

+ −

wine absinthe

+2.9 -5.5

red tequila

+0.6 -4.6

Cabernet liqueur

+0.5 -4.2

beverage

wine

+ −

wine syrup

+2.3 -5.5

cabernet ginger

+1.3 -3.9

red espresso

+1.1 -3.2

liquor

gin

+ −

gin vodka

+1.5 -5.3

pink liqueur almonds

+0.8 -4.8

floral liqueur coffee

+0.8 -3.8

domestic animal

dog

+ −

dog cat

+4.0 -3.4

-sized coon

+2.0 -0.5

coat breed

+0.9 -0.2

Table 7: Tokens pushing most towards (+) and away from (−) the specified class, as estimated by our posterior estimation. (Sub)words in gray are context to put the token (black) in context for the reader. Each of there gray contexts were the actual context for that token for 100% of the samples with that token. E.3

L OGICAL OPERATORS : NOT GATE

The results of the NOT gate in language on the Cognac benchmark are shown in Table 8. Firstly, note that the unconditional model already answers correctly, i.e., any class other than the undesired class, 70% of the times. This means that the model never intended to violate the constraint anyway in many cases. With this in mind, the positive prompted model, i.e., the one prompted not to violate the constraint, scores only 79%. Interestingly, both negative CFG and the NOT gate yield very similar results. So while the NOT gate is more grounded in the theory, we would recommend negative CFG to practitioners, as there are not three hyperparameters (grid search shown in Figure 24) but just one hyperparameter to tune. The results of the NOT gate are also very similar to the positive prompted model. This validates that the not gate can work as well as prompting the model. In the case of language, this is rather silly, as we can just prompt the model, but this comparison serves more as a proof of concept in the one modality where we can actually compare with a prompted model. So these results are complementary to the other modalities, where simple prompting is not available. Method Unconditional Prompted “NOT” Negative CFG (logit-space) NOT gate (prob-space)

Correct (Other classes) 0.70 ±0.03 0.79 ±0.03 0.81 ±0.02 0.79 ±0.03

Violated (Class) 0.23 ±0.03 0.10 ±0.02 0.06 ±0.01 0.08 ±0.02

Invalid 0.07 ±0.02 0.11 ±0.02 0.13 ±0.02 0.14 ±0.02

Table 8: Results of the NOT on the Cognac benchmark

F

A DDITIONAL DETAILS REGARDING ANTIBODY SEQUENCE GENERATION

In this section additional details and results on antibody sequence generation are provided. First, Appendix F.1 introduces the task and the dataset for readers without a background in biology. Second, Appendix F.2 describes the architecture and the training hyperparameters. Third, Appendix F.3 presents ablations on the model performance and introduces the metrics used. Finally, Appendix F.5 covers the posterior estimation and the results obtained with the NOT gate. The structure deliberately 35

Under review as a conference paper at ICLR 2027

0.72

0.71

0.62

0.53

0.5

0.68

0.73

0.77

0.73

0.63

0.53

0.2

0.67

0.78

0.82

0.76

0.59

0.48

0.1

0.67

0.80

0.80

0.75

0.55

0.45

0.05

0.68

0.79

0.80

0.73

0.53

0.44

0

0.69

0.78

0.78

0.74

0.54

0.44

0.1

0.5

1

2

4

8

16

0.54

0.45

0.46

0.45

0.45

0.44

16

0.44

0.46

0.46

0.46

0.46

0.47

8

0.80

0.53

0.46

0.45

0.45

0.44

8

0.49

0.44

0.44

0.46

0.48

0.52

4

0.76

0.79

0.71

0.50

0.47

0.45

4

0.75

0.70

0.70

0.71

0.73

0.71

2

0.67

0.78

0.82

0.76

0.59

0.48

2

0.78

0.80

0.80

0.82

0.77

0.72

1

0.65

0.72

0.77

0.78

0.76

0.58

1

0.78

0.81

0.78

0.77

0.74

0.70

0.5

0.66

0.68

0.73

0.76

0.75

0.72

0.5

0.76

0.77

0.75

0.73

0.70

0.69

0

0.66

0.69

0.68

0.73

0.72

0.72

0

0.73

0.71

0.70

0.68

0.68

0.67

0.1

0.5

1

2

4

8

0

0.05

0.1

0.2

0.5

1

Initial Guidance Scale

Initial Guidance Scale

Inverse Temperature τ

0.80

Correct

0.73

Delta Δ

0.68

Delta Δ

Inverse Temperature τ

0.85 1

0.75

0.70

0.65

0.60

Figure 24: Grid search on the development set of the Cognac benchmark. The optimal hyperparameters that were used for the final experiment are indicated with a bold box. We found that the text modality needs the δ from Equation (10). We use ∆ to indicate the cumulative δ over all steps. The first heatmap shows that τ is a little less sensitive to tune than the initial guidance scale. The second heatmap shows a downwards diagonal ridge, which seems to indicate that high initial guidance combined with high delta can make the guidance scale explode and yield poor results. The third heatmap again shows less sensitivity for τ , and larger sensitivity for ∆. mirrors that of the appendix on images (Sections D.1 and D.2), which precedes this one, so that a reader familiar with it can focus on the differences, mainly the metrics used to evaluate the generated sequences. F.1

DATASET AND BACKGROUND

This work is aimed at a broad machine learning audience, which might not be familiar with proteins, let alone antibodies. In this section we therefore give the reader the minimum needed to understand the specificities of the task. Interested readers are referred to the protein language model literature, for instance Lin et al. (2023); Fournier et al. (2024), and to Shuai et al. (2023) for antibodies in particular. Antibodies. Antibodies are Y-shaped proteins that are a key part of our immune system. Like every protein, an antibody is defined by its sequence of amino acids, of which 20 are commonly used and each is denoted by a letter. An antibody consists of two identical heavy chains and two identical light chains. At the tip of each chain sits a variable domain of roughly 120 residues, which recognises the target of the antibody, the antigen. This domain is built from four framework regions (FR1-4), which form a conserved structural scaffold, interleaved with three complementarity-determining regions (CDR1-3). The CDRs are hypervariable loops that make direct contact with the antigen and therefore determine what the antibody binds to. The CDR3 of the heavy chain (CDRH3) is the most diverse of them. During an immune response, the CDRs are fine-tuned towards the antigen in a process of somatic hypermutation and selection called affinity maturation. V genes. The variable domain of the heavy chain is encoded by recombining a V, a D and a J gene segment. The V gene encodes the largest part of the domain, namely the framework regions FR1-3 and CDR1-2, whereas CDR3 lies at the junction of the three segments and is therefore the most variable. The human heavy-chain V genes are commonly grouped into seven families, IGHV1 to IGHV7, whose members are highly similar in sequence. As the germline-encoded parts of a sequence are only mildly altered by mutations, the family of an antibody can be determined very reliably by aligning it to a database of germline genes. This makes the V-gene family a convenient class label for a conditional generative model: it is discrete and it can be assigned to every sequence. It should also be noted that the distribution of V-genes in humans is highly unbalanced, with IGHV3 accounting for 44% of human antibodies, whereas IGHV6 and IGHV7 make up only about 0.6% and 0.5%, respectively (see Table 9). Dataset. The Observed Antibody Space (OAS) (Olsen et al., 2022) is the largest and most widely used database of antibody sequences. As our goal is not to obtain a state-of-the-art antibody model, but to demonstrate the flexibility of our approach, we use an older partition of the database, namely 36

Under review as a conference paper at ICLR 2027

the OAS95 set of 559M sequences clustered at 95% identity, that was used to train IgLM (Shuai et al., 2023). IgLM is an autoregressive antibody language model, which we use to evaluate whether the generated sequences look reasonable (Appendix F.3). To verify our theoretical claims, however, a conditional dataset is required. We therefore keep the human heavy chains of OAS95 that are complete variable domains (starting with the typical first residues of FR1 and without ambiguous residues), and classify them by V-gene family using the publicly available IgBLAST tool (Ye et al., 2013) with the IMGT database of human germline genes (Lefranc et al., 2015). Exact duplicates are removed. This leaves 55.4M sequences, of which 2% are held out for validation. The sequences are 122 residues long on average, and 99% of them are between 113 and 132 residues long. As the family is mainly determined by the germline-encoded regions, this classification is essentially unambiguous.

V-gene family Share of sequences (%)

IGHV1 24.9

IGHV2 2.5

IGHV3 44.1

IGHV4 25.3

IGHV5 2.2

IGHV6 0.6

IGHV7 0.5

Table 9: Class balance of the antibody dataset (human heavy chains of OAS95, measured on the held-out validation split of 1.1M sequences). The conditional model is trained on these seven classes.

F.2

T RAINING DETAILS

Architecture. The antibody models are based on the AMPLIFY protein language model (Fournier et al., 2024), a bidirectional Transformer encoder with rotary position embeddings and SwiGLU feed-forward layers. As for images, we add adaptive layer normalization (AdaLN-Zero) as conditioning mechanism (Peebles & Xie, 2022): the noise level t and the V-gene family are embedded and summed into a single conditioning vector that modulates every block, where the label embedding has an additional entry for the unconditional case. The vocabulary has 27 tokens: 21 residue tokens (the 20 amino acids and the ambiguity code B) and 6 special tokens, of which one is the mask token used for masking diffusion. Every sequence is enclosed by a begin and an end token and padded to a fixed length of 256 tokens, well above the longest sequences. These special tokens are never corrupted and are excluded from the loss. As a consequence, the model does not have to determine the length of a sequence itself: when sampling, the padding pattern of a randomly chosen real training sequence is used as starting point, so that the lengths follow the empirical length distribution independently of the class. The resulting models have 50M parameters, of which 33.6M belong to the AMPLIFY backbone and 16.6M to the AdaLN modulation layers.

Training. As for images, the model predicts the clean sequence with a cross-entropy loss, which for masking is computed on the masked residues only and for uniform noise on all residues. We use a linear noise schedule with stratified sampling of the noise level. Conditional models are trained with label dropout, so that the same network also acts as unconditional model when the label is replaced by the unconditional token. In contrast to the images, no separately trained unconditional model is used, and no exponential moving average of the weights is kept. The hyperparameters in Table 10 were not tuned per noise type. Apart from the noise type, the masking and uniform models only differ in their number of training steps, 200k and 400k, which corresponds to roughly 0.9 and 1.9 epochs of the 54.3M training sequences. 37

Under review as a conference paper at ICLR 2027

Architecture Parameters Depth / hidden size / heads Sequence length / vocabulary Optimizer Learning rate Betas / weight decay Warmup steps / schedule Batch size Training steps Gradient clipping Numerical precision EMA Label dropout Noise schedule Noise level sampling Loss weighting

AMPLIFY with AdaLN-Zero 50M 12 / 480 / 20 256 / 27 (incl. mask token) AdamW 1 · 10−4 (0.9, 0.95) / 0.01 1000 / cosine decay 256 (2 GPUs × 128) 200k (masking) / 400k (uniform) 1.0 bf16 none 0.15 linear stratified in [0, 1] none

Table 10: Hyperparameters of the antibody models.

Training curves. Training progress is shown per noise level in Figure 25. As expected, lower noise levels (closer to t = 1) show a lower loss, particularly with uniform noise. This is beause when training a masked model the loss is only computed on the still masked tokes, while for the uniform model the loss is always computed on all tokens. In the latter, most tokens at low noise levels are already correct and should simply be returned back as output, a luxury the masking model can never have. This distinction also renders the comparison of loss curves between masking and uniform cases irrelevant. An interesting note in this setting is that the loss decreases very rapidly in the beginning and then only progresses very slowly. We hypothesize that this is due to the dual nature of antibody sequences, very easy framework regions, which are learnt fast and then highly unpredictable CDRs on which progress is slow. For the V-gene conditional task discussed in this paper the first part is most relevant, which is why our models are never evaluated on CDRs.

0.0-0.1 0.1-0.2

0.2-0.3 0.3-0.4

Noise level t 0.4-0.5 0.5-0.6

0.6-0.7 0.7-0.8

Validation loss

Masking

0.8-0.9 0.9-1.0

Uniform

100

10 1 0

25

50

75 100 125 150 175 200 0

Training steps (k)

25

50

75 100 125 150 175 200

Training steps (k)

Figure 25: Validation loss during training of the antibody models, averaged over bins of noise levels t ∈ [0, 0.1], [0.1, 0.2], . . . , for masking (left) and uniform noise (right).

F.3

M ODEL ABLATIONS

Metrics. The image metrics (FID, precision, recall and C2ST) rely on features of a pretrained image classifier. For antibodies we replace them by their protein counterparts, and add a measure of how natural the sequences are. All metrics are computed on 4096 generated sequences, whose class labels are drawn from the empirical class frequencies of the training set (Table 9), such that rare families remain rare, and whose lengths are set as explained above. 38

Under review as a conference paper at ICLR 2027

• IgLM log-likelihood. IgLM (Shuai et al., 2023) is an autoregressive language model trained on natural antibodies, conditioned on the chain type and the species (here heavy and human). We report the average log-likelihood per residue of the generated sequences under IgLM, which measures how antibody-like they are (higher is better). For reference, real training sequences reach −0.57 and random amino acid sequences of the same lengths −3.80. Figure 26 shows the gap between generated and real sequences, which should go to zero. • FPD. The Fréchet Protein Distance is the protein counterpart of the FID: the Fréchet distance between Gaussian fits to the ESM-2 embeddings (Lin et al., 2023) of generated and real sequences. It measures quality and diversity together (lower is better). We compare to 20000 random training sequences. • C2ST. The classifier two-sample test is the held-out accuracy of a logistic regression that separates real from generated embeddings (4096 of each, 80/20 split), where 0.5 is the best possible value, as then generated samples are indistinguishable from real ones. Additionally, we report the conditioning accuracy: the fraction of generated sequences that IgBLAST, the same tool that produced the training labels, assigns to the requested family. This is the counterpart of the classifier-based accuracy used for images. These metrics are computed for both noise types as a function of the number of sampling steps. This is shown in Figure 26 where the unconditional model is compared to a conditional model with class labels from the true data priors. As expected the quality of all samples increases as the number of inference steps increases. At low number of sampling steps the conditional model outperforms the unconditional one, which can be attributed to the less diverse distribution the model is sampling from. In essence the conditioning signal is often a nearly deterministic signal for portions of the sequence, by using a conditional model, these sections will never be confused, which is not true for the unconditional model, particularly if multiple tokens are flipped simultaneously. Another interesting note is that all metrics seem to saturate at 32 steps, again indicating the strong determinism of the task at hand. In this regime, at every step around 4 tokens are flipped per step and still this is sufficient for perfect samples.

10 1 10 2 4

8 16 32 64 128 256

Number of sampling steps

Conditional 0.0 0.2 0.4 0.6 0.8 1.0 1.2

Uniform

1.0

C2ST Accuracy ( 0.5)

FPD ( )

100

IgLM gen ref log-likelihood

Unconditional

4

8 16 32 64 128 256

Number of sampling steps

Masking

0.8 0.6 0.4 0.2 0.0 4

8 16 32 64 128 256

Number of sampling steps

Figure 26: Sweep over the number of sampling steps for the uniform-noise antibody model, for the unconditional (label dropped) and the conditional model, with a linear sampling schedule and 4096 sequences per point. The uniform model in full and the masking model in dashed lines. Shown are, from left to right, the FPD, the gap between the IgLM log-likelihood of the generated and of real sequences, and the C2ST accuracy. Class labels for the conditional model are drawn from the empirical class frequencies. All results are shown for the uniform-noise model, sampled with the linear schedule that was also used for training. F.4

P OSTERIOR ESTIMATION

In this context the V-gene classes are so well separated that the online posterior’s task is relatively easy, a few token flips is usually enough to determine the class fully. This is highlighted by the 39

Under review as a conference paper at ICLR 2027

perfect accuracy and AUC scores the method obtains, not only at the end point, but very early on Figure 13. We see this as a significant proof, that when a strong classification signal is available, the proposed online posterior distinguishes it reliably. F.5

L OGICAL OPERATORS : NOT GATE

Metrics For the NOT gate we use similar metrics to those used to perform the ablations on the model in Appendix F.3. In particular, the IgLM log-likelihood is used as a quality metric to verify sequence quality on an individual level. To measure distributional coverage, the FPD is again used with ESM2 embeddings, this time computed between generated samples and reference data which do not include the undesired class. The idea is that should a sequence of the unwanted class be generated, the FPD measured with respect to that set should increase as well. Further, as always, we measure safety or removal capacity of the method with a simple violation rate. This value is obtained by classifying samples using IgBLAST and computing the fraction of samples from the undesired class. All metrics are computed on 4096 samples. Methods To highlight the severe class imbalances present in the data, samples are first generated unconditionally. This gives a first worst-case scenario baseline. Then similarly to the image case, a static logit negative guidance version implemented. This is essentially standard discrete CFG with a negative guidance scale, that therefore pushes away from a certain condition instead of reinforcing it. Finally, the dynamic negative guidance approach, i.e. the NOT gate, is ablated. A key point of this method is that a properly scaled prior-weighted initial guidance scale is chosen. In other words, for a single hyperparameter setting a majority class receives more guidance than a minority class. Challenge Due to the severe class imbalance present in the data it should be intuitively clear that finding one setting that works best for all cases is no easy task. Majority class simply require much more guidance to be removed properly when compared to minority classes. This is particularly true in static approaches, such as standard CFG, where a fixed guidance scale is chosen for all classes regardless of their prevalence. Our dynamic approach and its prior-weighted initial guidance scale does not suffer the same flaw. By design the guidance scale start off at a multiple its theoretical p(ci ) value, i.e., at r 1−p(c with the factor r defined by the value of the initial guidance scale s0 through i) r = (N − 1)s0 , see Appendix G for more details. This means that a single choice of hyperparameter will automatically apply more guidance to majority classes than to minority ones simply because the initial value is prior-weighted. This is an extra quality of our NOT gate. A single hyperparameter setting can be set, and work equally well at removing any class regardless of how imbalanced the dataset really is. Results As outlined above, finding a single static guidance scale for negative CFG, that is able to remove all classes regardless of their prevalence is impossible. This is shown in Figure 27. On the right the violation rate, decreasing for all classes as the negative guidance scale is increased, is shown. On the left, the FPD is shown, here a very different story emerges. Majority classes follow the usual trade-off, first decreasing and then increasing indicating that guidance is starting to become too strong. For the minority on the other hand every guidance scale value seems to be too strong in terms of FPD. This makes sense: the FPD is a distributional metric and in this static case guidance affects all trajectories just as much, and does not only apply to the targeted trajectories. One cannot hope the method to require the same hyperparameter for minority and majority classes. On the other hand, our NOT gate with dynamic negative guidance, that has a prior-weighted initial guidance is shown in Figure 28. Here, we again can observe that the violation rate goes down as more guidance is applied, but crucially because guidance is not applied equally across classes, minority classes suffer much less from guidance. We view this prior weighted initial point as a key advantage of our theoretically grounded approach. Similarly to how the two are compared in images, a Pareto curve comparing safety (violation rate) and diversity/quality (FPD) is shown in Figure 29. To highlight the differences between majority and minority classes the two Pareto fronts are plotted. Similarly to what was obtained on images, the dynamic approach outperforms the static one. Even more so if a single global setting (one fixed set of hyperparameters) is desired. 40

Under review as a conference paper at ICLR 2027

IGHV1

Suppressed family NOT(⋅) IGHV3 IGHV4

IGHV2

0.6

IGHV5

IGHV6

IGHV7

0.4

Violation rate

0.5

FPD

0.4 0.3

0.3

0.2

0.2 0.1

0.1 0.0 0.0

0.2

0.4

0.6

0.8

0.0

1.0

0.0

Negative guidance

0.2

0.4

0.6

0.8

1.0

Negative guidance

Figure 27: Negative CFG with static negative logit guidance. On the left, the Frechet Protein Distance measuring distributional coverage and sample quality, and on the right, the violation rate. Notice the strong differences between majority (IGHV1-4) and minority classes (IGHV5-7).

IGHV1

IGHV2

Suppressed family NOT(⋅) IGHV3 IGHV4

IGHV5

IGHV6

IGHV7

4

6

0.40

0.10

0.35

Violation rate

FPD

0.08 0.06 0.04

0.30 0.25 0.20 0.15 0.10

0.02

0.05 0.00

0

2

4

0.00

6

Initial guidance scale

0

2

Initial guidance scale

Figure 28: NOT gate with dynamic negative guidance. On the left, the Frechet Protein Distance measuring distributional coverage and sample quality, and on the rate the violation rate. Here, thanks to the prior weighted initial guidance scale, minority classes are much less harmed when applying guidance.

41

Under review as a conference paper at ICLR 2027

0.40

Majority (IGHV1/3/4) Minority (IGHV2/5/6/7) Dynamic, prior-weighted Static logit guidance Unconditional

0.35 0.30

FPD

0.25 0.20 0.15 0.10 0.05 0.00 0.00

0.05

0.10

0.15

0.20

0.25

Violation rate

Figure 29: Pareto front between quality (FPD) and safety (violation rate) for majority and minority classes. Negative CFG with static logit guidance is compared to the NOT gate with dynamic negative guidance.

G

P OSTERIOR ESTIMATION HYPERPARAMETERS

The posterior estimation, when used as formulated by the theory, tends to be stochastic and overconfident, which is why prior work in the continuous domain introduced the two regularizing hyperparameters τ and δ (Koulischer et al., 2025b). We also use an initial guidance scale to determine the strength of guidance with which generation starts. By setting τ < 1 in Equation (10), strong jumps in the posterior evaluation are avoided. All grid searches across modalities resulted in a value of τ that satisfies this intuition. The second hyperparameter δ gives a constant push upwards to the posterior. This is most important when interfering during generation with the NOT gate, as there the precise calibration of the posterior value is key in contrast to the AUC/Accuracy experiments that are mainly impacted by the correct ranking of classes. In practice we find that the posterior most often decreases, particularly for the NOT gate, this is shown for text in Figure 21a, even before guidance and Figure 18 after guidance for images. We argue this is because at best the conditional model agrees with the neutral model, yielding an update log p(xt | xt−1 , c) − log p(xt | xt−1 ) around 0. At worst the models disagree and we pick the token that the unconditional agrees most with, which means log p(xt | xt−1 ) is relatively high and the difference will be strongly negative. So the posterior going up is a very rare case in practice, though it is clear intuitively that when the conditional model agrees, the posterior is going up. Thus, the δ gives a permanent upward push to actually increase the posterior when the models agree. The third hyperparameter is the initial guidance scale s0 , that works on the prior probability p(c | x0 ), which is again important for the NOT gate. The idea of the initial guidance scale is to control the baseline strength of guidance, with three different cases merged in one formula. For this, we scale the priors in guidance scale-space, as if they were uniform: The guidance scale for a uniform p(ci ) , which is N 1−1 , and thus we would need a scaling factor r of s0 ·(N − 1) prior would start at 1−p(c i) in order to achieve initial guidance of s0 . For all three cases, we take the real prior, convert it into p(ci ) guidance scale-space with 1−p(c , multiply with r and invert back to probability-space. The first i) case is that of uniform priors, such as MNIST. In that case, the whole formula just boils down do using exactly s0 as the strength for all classes. In the second case, we consider the prior to be unknown, for example in the text modality. As the prior is unknown and we want to start with a fixed guidance strength, the approach reduces to the uniform case with a flat guidance scale s0 . The 42

Under review as a conference paper at ICLR 2027

third case is that of known but non-uniform priors. There, we again convert the prior to the guidance p(ci ) scale-space: 1−p(c , we scale it by a factor r and then invert back into probability space. This way i) we scale all guidance strength equally, and the strength per class is still informed by its prior, rather than using a flat scale. We chose this parametrization as we found it easy to work with and sweep, but there is no fundamental reason to choose this parametrization over another.

H

R EPRODUCTION DETAILS

We make our code available on https://github.com/Gianni-Van-de-Velde/Online-Class-Posteriorsfor-Interpretable-and-Controllable-Discrete-Diffusion. The current document is still under review and, as a consequence, both the text as the code are prone to minor chanes. All text and image experiments ran on NVIDIA A40 GPUs and 4 virtual cores of a Intel Xeon Silver 4310 CPU. For antibodies the experiments where performed on NVIDIA GeForce RTX 4090 GPUs.

43

Related documents

• ILRR: Inference-Time Steering Method for Masked Diffusion Language Models#1048565
• Diffusion Language Models for Speech Recognition#14019
• Learning by Creating and Justifying Transfer Frames#239451
• Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance#1355151
• Discrete Diffusion Language Models Are Training-Free Multi-Label Classifiers#609086