On the Redundancy of Timestep Embeddings in Diffusion Models José A. Chávez
arXiv:2606.20416v1 [cs.LG] 18 Jun 2026
Independent Researcher, Lima, Peru [email protected]
Abstract. Diffusion models rely heavily on explicit timestep embeddings to modulate the denoising process across various noise scales. In this work, we challenge the necessity of these temporal signals by analyzing their impact on U-Net and Diffusion Transformer architectures. Beyond empirical evidence, we provide a theoretical framework demonstrating that, under certain conditions, the global minimizer of the diffusion training objective can be achieved without explicit timestep conditioning. Our findings reveal a surprising robustness when timestep embeddings are completely removed. Extensive ablation studies on the CelebA and CIFAR-10 datasets show that these time-agnostic models can maintain high structural fidelity and even surpass their conditioned counterparts in competitive metrics, including FID, precision, and recall. Our analysis suggests these architectures can implicitly infer noise scales from the corrupted input under specific assumptions, rendering explicit temporal conditioning redundant. This study challenges long-standing temporal conditioning paradigms and paves the way for more efficient and structurally focused generative architectures. Keywords: timestep embeddings
1
Introduction
Diffusion Models [23] have shown significant success in generative modeling, synthesizing high-fidelity data through a gradual denoising process [2, 6, 19, 21]. These models rely heavily on timestep embeddings, which serve to encode temporal information to adjust the noise removal level at each step. This information is typically injected via feature map addition or modulation across the model architecture. Recently, transformer architectures have emerged as a powerful alternative to convolutional backbones in diffusion models. Diffusion Transformers (DiT) [17], based on the Visual Transformer architecture [4], provide several key advantages over traditional U-Net architectures. Chief among these is scalability; DiT demonstrates superior performance as model size and computational power increase, showing strong correlations between GFLOPs and sample quality. This allows for superior performance in high-resolution image generation and largescale datasets. Furthermore, transformers excel at modeling long-range depen-
2
J. Chávez
U-Net*
U-Net
DiT*
DiT
dencies due to their self-attention mechanism, enabling better coherence in image generation by capturing relationships across distant regions of an image. DiTs outperform traditional U-Net backbones on ImageNet [3] at 512 × 512 and 256 × 256 resolution.
Fig. 1: Unconditional face generation on the CelebA dataset (128 × 128). We compare the visual fidelity of standard architectures against our time-agnostic variants. The first and third rows show samples from standard DiT and U-Net backbones with explicit timestep conditioning. The second and fourth rows (marked with *) display samples from our time-agnostic counterparts.
Regardless of the architectural backbone chosen—be it a U-Net or a DiT— current implementations fundamentally depend on a mechanism to inject the temporal signal or the noise scale. Although the importance of timestep embeddings is widely accepted, there has been limited investigation into their actual ne-
On the Redundancy of Timestep Embeddings in Diffusion Models
3
cessity. In this work, we challenge the conventional wisdom by exploring whether these temporal signals are truly indispensable for diffusion model architectures to effectively learn the denoising process. We conduct a rigorous ablation study on both U-Net and DiT architectures, systematically removing explicit timestep conditioning to evaluate the impact on sample quality and structural fidelity. Our empirical results reveal a surprising robustness in these time-agnostic models, which can maintain high performance even without explicit temporal signals. Our contributions are as follows: 1. We conduct a rigorous ablation study on the effect of timestep embeddings across both convolutional (U-Net) and transformer-based (DiT) diffusion architectures. 2. We provide a comprehensive theoretical analysis that timestep signals can be functionally redundant under certain conditions, as the model can implicitly recover the noise scale directly from the corrupted input.
2
Background
Denoising diffusion probabilistic models (DDPM) [6] are latent variable models R of the form pθ (x0 ) := pθ (x0:T )dx1:T , where x1 , . . . , xT are latent variables in the same sample space as x0 ∼ q(x0 ). The forward process is defined as a Markov chain that gradually adds Gaussian noise to the data:
\label {q_formula} q(\mathbf {x}_{1:T}\mid \mathbf {x}_0):=\prod ^{T}_{t=1}q(\mathbf {x}_{t}\mid \mathbf {x}_{t-1}),
(1)
√ where the true posterior q(xt | x0 ) := N (xt ; ᾱt x0 , (1− ᾱt )I) is parametrized by a time-dependent scalar ᾱt ∈ {ᾱ1 , . . . , ᾱT }. After a re-parameterization we can write: \label {x_t} \mathbf {x}_t:=\sqrt {\bar {\alpha }_t}\mathbf {x}_0+\sqrt {1-\bar {\alpha }_t}\boldsymbol {\epsilon },
(2)
where ϵ ∼ N (0, I).The reverse process is defined as a Markov chain:
\label {p_formula} p_\theta (\mathbf {x}_{0:T}):=p_\theta (\mathbf {x}_T)\prod ^{T}_{t=0}p_\theta (\mathbf {x}_{t-1}\mid \mathbf {x}_t).
2.1
(3)
The Training Process
In the standard DDPM framework [6], a neural network ϵθ processes two inputs: the corrupted image xt and the discrete timestep t ∈ {0, 1, . . . , T }. The objective function is to predict the ground-truth noise ϵ added to the original image x0 in Eq. (2) by minimizing the following loss function:
\mathcal {L}(\theta )&=\mathbb {E}_{\mathbf {x}_t,t}[\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t,t}[||\boldsymbol {\epsilon }-\boldsymbol {\epsilon }_\theta (\mathbf {x}_t,t)||^2\mid \mathbf {x}_t,t]].
(5)
The global minimizer can be obtained by analyzing the inner expectation for each fixed xt and t. If we define y := ϵθ (xt , t) and ϕ(xt , t) := Eϵ|xt ,t [||ϵ − y|xt , t||2 ], we can obtain the global minimizer y∗ of ϕ(xt , t) by setting the gradient of ϕ(xt , t) with respect to y to zero: \label {L_global_minimizer} \mathbf {0}&=-2\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t,t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t,t] + 2\mathbf {y}^* \nonumber \\ \therefore \boldsymbol {\epsilon }^{*}_\theta (\mathbf {x}_t,t)&=\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t,t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t,t]. (6) Now, to understand the role of the timestep t in the training process, we need to analyze how it is injected into the architecture. In practice, DDPM injects the timestep as a positional encoding (pe): \label {time_signal} \mathbf {h}:= \mathbf {h}+ \text {ffn}(\text {pe}(t)),
(7)
where ffn is a feedforward network (ffn) and h is the input of each convolutional layer in the U-Net architecture. This process is repeated for each convolutional layer in DDPM [6]. DiT [17] instead, integrates the temporal conditioning mechanisms through an Adaptive Layer Normalization (AdaIN) [18]. 2.2
The Sampling Process
During the sampling process, given the noise predicted by ϵθ , DDPM uses it to predict the uncorrupted image x0 : \label {x_0} \mathbf {x}_{0}(\theta )=\frac {1}{\sqrt {\bar {\alpha }_t}}\left (\mathbf {x}_t-\sqrt {1-\bar {\alpha }_t}\boldsymbol {\epsilon }_\theta (\mathbf {x}_t,t)\right ).
(8)
As stated in DDPM, xt−1 can be predicted using x0 and xt :
\label {x_t_minus_1_extended} \mathbf {x}_{t-1}=\frac {\sqrt {\bar {\alpha }_{t-1}}(1-\alpha _t)}{1-\bar {\alpha }_t}\mathbf {x_0}(\theta )+\frac {\sqrt {\alpha _t}(1-\bar {\alpha }_{t-1})}{1-\bar {\alpha }_t}\mathbf {x}_t+\sqrt {1-\alpha _t}\mathbf {r}, Similarly, DDIM [24] computes first x0 to estimate xt−1 .
(9)
On the Redundancy of Timestep Embeddings in Diffusion Models
3
5
The Redundancy of Timesteps in the Loss Function
Given that samples x0 from the train data distribution q(x0 ) and Eq. (2) are fulfilled, we can express Eq. (4) as: \label {l_theta_extended} \mathcal {L}(\theta )=\mathbb {E}_{\mathbf {x}_0,\boldsymbol {\epsilon },t}||\boldsymbol {\epsilon }-\boldsymbol {\epsilon }_\theta (\sqrt {\bar {\alpha }_t}\mathbf {x}_0+\sqrt {1-\bar {\alpha }_t}\boldsymbol {\epsilon },t)||^2.
(10)
Then, using Tweedie’s Formula, we can predict the true posterior mean of xt : \label {tweedie} \mathbb {E}[\sqrt {\bar {\alpha }_t}\mathbf {x}_0\mid \mathbf {x}_t]&=\mathbf {x}_t+(1-\bar {\alpha }_t)\nabla _{\mathbf {x}_t}\log p(\mathbf {x}_t)
(11)
Here, ∇xt log p(xt ) represents the score function [8]. Following the derivation presented in [15], we can use Eq. (2) to formalize the connection between the ground-truth noise and the score function: \label {score_noise} \mathbf {x}_t-\sqrt {1-\bar {\alpha }_t}\boldsymbol {\epsilon }&=\mathbf {x}_t+(1-\bar {\alpha }_t)\nabla _{\mathbf {x}_t}\log p(\mathbf {x}_t) \\ \therefore \boldsymbol {\epsilon }&=-\sqrt {1-\bar {\alpha }_t}\nabla _{\mathbf {x}_t}\log p(\mathbf {x}_t). (13) Thus, the loss function in DDPM [6] is equivalent to the loss function used in noise conditional score networks [25]. Intuitively, the score function gives a measure of how to move xt in the data space to maximize its log-probability. Thus, √ Eq. (10) tries to predict the opposite direction of the score function scaled by 1 − ᾱt . In other words, the network ϵθ is trained to predict the noise that drives xt away from the data manifold. √ √ Theorem 1. Let xt = ᾱt x0 + 1 − ᾱt ϵ be a corrupted sample with an implicit noise scale ᾱt ∈ (0, 1), where x0 ∼ q(x0 ) and ϵ ∼ N (0, I). Let ϵθ be a neural network trained to minimize the time-agnostic loss function \mathcal {L}_{TA}(\theta )=\mathbb {E}_{\mathbf {x}_0,\boldsymbol {\epsilon },t}[||\boldsymbol {\epsilon }-\boldsymbol {\epsilon }_\theta (\sqrt {\bar {\alpha }_t}\mathbf {x}_0+\sqrt {1-\bar {\alpha }_t}\boldsymbol {\epsilon })||^2] Then: 1. The global minimizer ϵ∗θ of LT A (θ) satisfies ϵ∗θ = E[ϵ | xt ]. 2. The global minimizer can be determined by the implicit noise scale ᾱt and the score function ∇xt log p(xt ). 3. If there exists a measurable map µ : Rd → (0, 1) such that µ(xt ) = ᾱt , then ϵ∗θ = E[ϵ | xt ] = E[ϵ | xt , t]. Proof. By the law of total expectation, we can express LT A (θ) as: \mathcal {L}_{TA}(\theta )&=\mathbb {E}_{\mathbf {x}_t}[\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t}[||\boldsymbol {\epsilon }-\boldsymbol {\epsilon }_\theta (\mathbf {x}_t)||^2\mid \mathbf {x}_t]].
6
J. Chávez
Similar to the derivation of the global minimizer in Eq. (6), we have: \boldsymbol {\epsilon }^{*}_\theta (\mathbf {x}_t)&=\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t] Now, we take the conditional expectation with respect to xt on both sides of Eq. (2):
\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t]&=\frac {\mathbf {x}_t-\sqrt {\bar {\alpha }_t}\mathbb {E}[\mathbf {x}_0\mid \mathbf {x}_t]}{\sqrt {1-\bar {\alpha }_t}} \\
Then, we use Eq. (11) to express the global minimizer ϵ∗θ of LT A (θ) in terms of the score function: \boldsymbol {\epsilon }^{*}_\theta (\mathbf {x}_t)&=-\sqrt {1-\bar {\alpha }_t}\nabla _{\mathbf {x}_t}\log p(\mathbf {x}_t). Since µ(xt ) = ᾱt , we can define the conditional distribution p(ᾱt | xt ) as a Dirac delta distribution centered at µ(xt ) because ᾱt is deterministic given xt . Consequently, the conditional expectation can be written as:
\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t]&=\int \mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t,\bar {\alpha }_t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t,\bar {\alpha }_t]\delta (\bar {\alpha }_t-\mu (\mathbf {x}_t))d\bar {\alpha }_t \\ &=\mathbb {E}_{\boldsymbol {\epsilon }\mid \mathbf {x}_t,\bar {\alpha }_t}[\boldsymbol {\epsilon }\mid \mathbf {x}_t,\bar {\alpha }_t]\bigg |_{\bar {\alpha }_t=\mu (\mathbf {x}_t)}
Thus Eϵ|xt [ϵ | xt ] = Eϵ|xt ,ᾱt [ϵ | xt , t] as there exists a bijection between ᾱt and the timestep t. ⊔ ⊓ Theorem 1 shows that the global minimizer of the time-agnostic loss function LT A (θ) is proportional to the score function ∇xt log p(xt ). Moreover, if there exists a measurable mapping µ : Rd → (0, 1) such that µ(xt ) = ᾱt , the global minimizer of LT A (θ) is equivalent to the global minimizer of the standard loss function L(θ) in Eq. (6). Thus, the time signal t can be disregarded by the network as long as it can implicitly infer ᾱt from the corrupted input xt . Now, we need to prove the existence of the measurable mapping µ : Rd → (0, 1) such that µ(xt ) = ᾱt in order to show that the global minimizer of LT A (θ) is equivalent to the global minimizer of L(θ). √ √ Theorem 2. Let xt = ᾱt x0 + 1 − ᾱt ϵ be a corrupted sample on an unknown scale ᾱt ∈ (0, 1), where i x0 ∼ q(x0 ) and ϵ ∼ N (0, I) are independent, and xt is observable. ii x0 , ϵ ∈ Rd . 2 iii E[ ||xd0 || ] is bounded.
On the Redundancy of Timestep Embeddings in Diffusion Models 2
7
P
If ||xd0 || − → C for some positive constant C ̸= 1. Then, there exists a sequence P
of measurable mappings {µd : Rd → (0, 1)}∞ → ᾱt as d → ∞. d=1 such that µd (xt ) − 2
Proof. We consider the statistic Hd (xt ) := ||xdt || and express it in terms of x0 and ϵ:
H_d(\mathbf {x}_t)&=\bar {\alpha }_t\frac {||\mathbf {x}_0||^2}{d}+(1-\bar {\alpha }_t)\frac {||\boldsymbol {\epsilon }||^2}{d}+2\sqrt {\bar {\alpha }_t(1-\bar {\alpha }_t)}\frac {\mathbf {x}^T_0\boldsymbol {\epsilon }}{d} 2
P
Since ||ϵ||2 ∼ χ2 (d), we have ||ϵ|| − → 1 as d → ∞. Moreover, x0 and ϵ are d independent, thus E[xT0 ϵ] = 0, and we can express E[(xT0 ϵ)2 | x0 ] as follows: \mathbb {E}[(\mathbf {x}^T_0\boldsymbol {\epsilon })^2\mid \mathbf {x}_0] &= ||\mathbf {x}_0||^2 \\ \therefore \mathbb {E}[(\mathbf {x}^T_0\boldsymbol {\epsilon })^2] &= \mathbb {E}[\mathbb {E}[(\mathbf {x}^T_0\boldsymbol {\epsilon })^2\mid \mathbf {x}_0]] = \mathbb {E}[||\mathbf {x}_0||^2]
Now, given a δ > 0, by Cheybyshev’s inequality:
P\left (\left |\frac {\mathbf {x}^T_0\boldsymbol {\epsilon }}{d}\right |>\delta \right )\leq \frac {1}{d}\mathbb {E}\left [\frac {||\mathbf {x}_0||^2}{d}\right ]/\delta ^2
T 2 x0 ϵ Since E[ ||xd0 || ] is bounded, we have P > δ → 0 as d → ∞. Finally, d we have the following convergence in probability:
H_d(\mathbf {x}_t)&\xrightarrow {P} \bar {\alpha }_t (C-1) + 1 Since C is a positive constant with C ̸= 1, H converges to a deterministic function of ᾱt . Define the affine function h : (0, 1) → R as follows: h(\bar {\alpha }_t):=\bar {\alpha }_t (C-1) + 1. Observe that h is a strictly monotone function and therefore invertible. Consequently, for any δ > 0, we can express the probability of the event
|h^{-1}(H_d(\mathbf {x}_t))-h^{-1}(h(\bar {\alpha }_t))|>\delta as follows:
P(|h^{-1}(H_d(\mathbf {x}_t))-h^{-1}(h(\bar {\alpha }_t))|>\delta ) &= P(|H_d(\mathbf {x}_t)-h(\bar {\alpha }_t)|>\delta |{C-1}|). Therefore, by assumptions of the theorem, we have:
h^{-1}(H_d(\mathbf {x}_t))\xrightarrow {P} h^{-1}(h(\bar {\alpha }_t))=\bar {\alpha }_t. P
If we define µd (xt ) := h−1 (Hd (xt )), we obtain µd (xt ) − → ᾱt as d → ∞
⊔ ⊓
In practical settings, image data are digitally represented with bounded intensity values (e.g., normalized to [−1, 1] or [0, 1]). Consequently, the components of x0 are bounded, which immediately implies the existence of uniformly bounded fourth moments. In particular, there exists a constant M > 0 such that E x40,i ≤ M for all i ∈ {1, . . . , d}. Moreover, since ||x0 ||2 ≤ m2 d for some finite i h 2 constant m > 0, it follows that E ||xd0 || is also bounded in standard image datasets. While natural images exhibit significant local correlations, the dependencies between individual pixels tend to decay rapidly with spatial distance. Thus, under the assumption of a weak dependence condition such that: \sum _{i<j} |\text {Cov}(\mathbf {x}_{0,i}^2, \mathbf {x}_{0,j}^2)| = O(d),
Then,
\text {Var}\left (\sum ^d_{i=1} \mathbf {x}^2_{0,i}\right ) &= \sum ^d_{i=1} \text {Var}(\mathbf {x}^2_{0,i}) + 2\sum _{i<j} \text {Cov}(\mathbf {x}^2_{0,i}, \mathbf {x}^2_{0,j}) \\ &\leq dM + O(d) = O(d).
2 Pd Let C := E[ ||xd0 || ] = d1 i=1 E[x20,i ] the average of expected second moment of the individual data components of x0 . By Cheybyshev’s inequality, we can 2 establish the convergence in probability of ||xd0 || to C. For any δ > 0:
P\left (\left |\frac {||\mathbf {x}_0||^2}{d}-C\right |>\delta \right ) &\leq \frac {1}{d^2\delta ^2}O(d) \\ \therefore P\left (\left |\frac {||\mathbf {x}_0||^2}{d}-C\right |>\delta \right ) &\to 0 \text { as } d\to \infty .
If the data distribution is normalized such that its individual components have zero mean and unit variance, then C = 1. In this regime, the existence of the mapping µd does not guarantee the recovery of ᾱt from xt as Hd (xt ) will
On the Redundancy of Timestep Embeddings in Diffusion Models
9
converge to a constant function. Consequently, this particular mechanism does not guarantee redundancy of the time signal in the loss function. We can interpret Theorem 2 as follows: in high-dimensional spaces, the norm of the corrupted sample xt concentrates around a deterministic function of ᾱt whenever the empirical second moment of the components of x0 concentrates around a constant C ̸= 1. Thus, by measuring the norm of xt , we can recover ᾱt with high probability. The mapping µd is a theoretical construct to establish existence; in practice, a network may infer the timestep implicitly without needing explicit computation of Hd .
4
Related Work
Diffusion models [23] adopt an idea from physics, where a sample is gradually converted into a well-known distribution (e.g., Gaussian) by adding noise in a forward process. Then, a reverse process is learned to restore the original data from the noise. In DDPM [6], the reverse process is parameterized by a neural network ϵθ that takes as input the noisy image xt and the temporal signal t to predict the noise ϵ used to corrupt the original image x0 . To inject the scalar t into the model, DDPM uses positional encodings followed by a feedforward network. In contrast, DiT [17] uses an AdaIN [7] to inject the time signal into the model. Various works have explored improving the scheduling of noise scales in diffusion models [10,16], but the question of whether the time signal itself is necessary has not been thoroughly investigated. Our work fills this gap by providing both theoretical insight and empirical evidence on the redundancy of timestep embeddings in diffusion models. Denoising Autoencoders (DAE) [29] and Generalized DAEs (GDAE) [1] established early on that generative density modeling can be achieved through iterative corruption-reversal processes without strictly requiring explicit scale or corruption inputs, as exemplified by the Walkback algorithm. Furthermore, Denoising Score Matching (DSM) [28] proved that the DAE objective is equivalent to estimating the score function ∇xt log p(xt ) under a specific noise distribution. This fundamental connection directly underpins Noise Conditional Score Networks (NCSN) [25]. Rooted in these classical foundations, several recent studies have begun to question the strict necessity of explicit timestep conditioning [19, 22, 27, 30], proposing instead modern noise-agnostic or blind denoising frameworks. For instance, [26] establishes general error bounds for time-agnostic generative modeling, while [20] investigates the geometry of diffusion via the marginal energy landscape. Concurrently, [9] leverages the mathematical “blessings of dimensionality” under low intrinsic dimensions or single-data-point assumptions to bypass explicit conditioning. While these foundational and recent works explore the broad feasibility of noise-agnostic architectures, they predominantly rely on global concentration statistics and fall short in regimes where global features become uninformative.
J. Chávez
U-Net*
U-Net
DiT*
DiT
10
Fig. 2: Qualitative comparison of conditional-class generation on CIFAR-10. The first and third rows show samples from standard DiT and U-Net backbones with explicit timestep conditioning. The second and fourth rows (marked with *) display samples from our time-agnostic counterparts.
In contrast, our work provides a distinct, complementary perspective through an architectural and local statistical lens. We explicitly analyze normalized latent spaces—a regime where global statistics fail (C ≈ 1)—and prove that noise P scale identification remains viable under a weak dependence condition ( i<j |Cov(x20,i , x20,j )| = O(d)) that rigorously accounts for the structural, local nature of spatial image data.
On the Redundancy of Timestep Embeddings in Diffusion Models
11
Table 1: Evaluation of unconditional image generation CelebA dataset 128 × 128 [14]. Numbers are reported for 50000 samples. *: models trained without the timestep embeddings. Model
FID↓
Precision↑
Recall↑
Time (s)↓
DiT DiT*
70.04 70.48
0.68 0.65
0.13 0.14
1.558 ± 0.046 1.324 ± 0.042
U-Net U-Net*
69.82 63.63
0.727 0.729
0.10 0.12
1.115 ± 0.047 1.089 ± 0.042
Table 2: Evaluation of conditional-class image generation on CIFAR-10. Numbers are reported for 50000 samples. *: models trained without the timestep embeddings.
5
Model
FID↓
Precision↑
Recall↑
Time (s)↓
DiT DiT*
51.32 34.74
0.68 0.75
0.47 0.49
8.553 ± 0.222 8.498 ± 0.420
U-Net U-Net*
52.67 41.18
0.66 0.68
0.45 0.57
7.105 ± 0.400 7.048 ± 0.368
Experiments
In this section, we conduct an analysis of timestep embeddings in both U-Net and DiT architectures. We evaluate the impact of removing timestep embeddings on the quality of generated samples using standard metrics such as FID, Precision, and Recall. We also provide a qualitative comparison of the generated images to visually assess the differences in fidelity and diversity between models with and without timestep conditioning. Experimental Setup We adopt the common setup for measuring the Fréchet Inception Distance (FID) [5], following the standard procedure in the literature [6,11]. For sample diversity and fidelity, we also report Precision and Recall metrics [13]. The setup for Precision and Recall is the same as in [13]. We train on the CelebA dataset with samples at 128 × 128 resolution, and we train on the CIFAR-10 dataset with samples at 32 × 32 resolution. For each dataset, we train two configurations for each architecture: one with timestep embeddings and another without them. For sampling time, we use the clock time when generating 100 batches of 100 samples each, and report the average time per batch along with the standard deviation across batches on a RTX 4070 GPU. Training Details The U-Net architecture is based on DDPM [6], while the DiT architecture follows the DiT-S/2 configuration of [17] with 252 hidden units. For the CelebA dataset, we first train a VAE [12] to learn a lower-dimensional latent representation of the data. Then we train both U-Net and DiT in this latent
12
J. Chávez
space. For the CIFAR-10 dataset, we directly train both architectures in the data space. The diffusion model in CelebA is conducted in the latent space of the VAE, at 16 × 16 resolution. While in CIFAR-10, it is performed in the data space, at 32 × 32 resolution. 5.1
Quantitative Results
The quantitative results are summarized in Tab. 1 and Tab. 2. In the CelebA dataset, DiT* (DiT without timestep embeddings) achieves an FID of 70.48, which is slightly lower than the 70.04 obtained by standard DiT. In terms of Precision and Recall, DiT* shows a minor decrease in Precision but an improvement in Recall. For U-Net, removal of timestep embeddings leads to a significant improvement in FID from 69.82 to 63.63, as well as increases in both Precision and Recall. In the CIFAR-10 dataset, the impact of removing timestep embeddings is even more pronounced. DiT* achieves a substantial improvement in FID from 51.32 to 34.74, along with increases in both Precision and Recall. Similarly, UNet* shows a significant improvement in FID from 52.67 to 41.18, Precision from 0.66 to 0.68, and Recall from 0.45 to 0.57. In summary, the quantitative results indicate that removing timestep embeddings can lead to similar performance in terms of FID, Precision, and Recall in the CelebA dataset, while it can lead to significant improvements in these metrics in the CIFAR-10 dataset. 5.2
Qualitative Analysis
The qualitative comparison in Fig. 1 reveals that for the CelebA dataset, both DiT and U-Net generate visually coherent faces with and without timestep embeddings. However, samples from default DiT and U-Net (with timestep embeddings) exhibit fewer artifacts. In contrast, DiT* and U-Net* show a greater diversity in terms of facial features, color distribution, and pose, which may contribute to improved FID scores. In the CIFAR-10 dataset, the differences in Fig. 2 are more difficult to discern visually, but the samples of DiT* and U-Net* appear to have more varied textures and colors, reflecting the redundancy of the temporal signal in guiding the generation process in a high-diversity dataset. 5.3
Robustness and Performance Trade-offs
In terms of computational performance, Tab. 1 shows that removal of timestep embeddings leads to a reduction in inference time for both DiT and U-Net. This suggests that, when the diffusion process is performed in the latent space at 16 × 16 resolution, the temporal conditioning mechanism introduces additional computational overhead, and its removal can lead to more efficient sampling without compromising the quality of generated samples. In the CIFAR10 dataset, Tab. 2 shows a less pronounced reduction in inference time when removing timestep embeddings, which may be due to the fact that the diffusion process is performed in the data space at 32 × 32 resolution, where the
On the Redundancy of Timestep Embeddings in Diffusion Models
13
computational overhead of temporal conditioning may be less significant. The performance trade-offs between visual fidelity, diversity, and computational efficiency highlight the potential benefits of re-evaluating the necessity of timestep embeddings in diffusion models.
6
Discussion
Tab. 1 shows a slight decrease in FID for DiT* compared to DiT, accompanied by a minor decrease in Precision and slight increase in Recall. This suggests that timestep embeddings may have a limited role in guiding the denoising process. In contrast, the improvement in FID for U-Net* relative to U-Net indicates that timestep embeddings may be less beneficial—and potentially even detrimental— for convolutional architectures on this dataset. Qualitative analysis in Fig. 1 supports this interpretation, as U-Net* samples exhibit more diversity in facial features and poses, which may contribute to improved FID scores. In the CIFAR-10 dataset, substantial improvements in FID for both DiT* and U-Net* compared to their standard counterparts indicate that timestep embeddings may not provide additional benefit for conditional image generation in a high-diversity dataset. The increases in both Precision and Recall further indicate that removing timestep embeddings allows the models to generate samples that are not only more diverse but also more representative of the true data distribution. Qualitative analysis supports these interpretations. Samples from DiT* and U-Net* appear to preserve structural details and varied textures. Although the visual differences may be subtle as CIFAR-10 is more diverse and has lower resolution than CelebA, the improved FID scores imply that the models without timestep embeddings are better at capturing the underlying data distribution, leading to an overall improvement in sample quality. Furthermore, the sampling time results indicate that removing timestep embeddings leads to a reduction in inference time for both DiT and U-Net across both datasets. This suggests that the temporal conditioning mechanism introduces additional computational overhead. The performance trade-offs between visual fidelity, diversity, and computational efficiency highlight the potential benefits of re-evaluating the necessity of timestep embeddings in diffusion models. Theorems 1 and 2 provide a theoretical framework for analyzing the potential redundancy of timestep embeddings in diffusion models. In particular, the existence of a measurable mapping µ capable of recovering ᾱt from the corrupted input. This suggests that, under suitable conditions, a neural network may implicitly infer the noise level without the timestep signal. This occurs when the per-component second moment of the data distribution concentrates around a constant C ̸= 1. In the case of CIFAR-10, diffusion is performed directly in the data space, where C is not necessarily equal to one. This may partially explain the performance improvements observed when removing timestep embeddings. In contrast, in the CelebA dataset, the diffusion process is conducted in the latent space of a VAE, where samples are normalized to have approximately unit variance across dimensions. In this regime, the mechanism described in Theo-
14
J. Chávez
rem 2 may be weakened, which could account for the more limited impact of removing timestep embeddings, especially for the DiT architecture. The structural resilience of the time-agnostic U-Net* compared to its DiT* counterpart, as evidenced in Tab. 1, can be explained through the lens of Local Identifiability. While Theorem 2 establishes that global noise-scale recovery is theoretically constrained in normalized latent spaces where Cglobal ≈ 1, natural spatial inhomogeneity ensures that localized sub-structures frequently deviate from unity (Clocal ̸= 1). The capacity to exploit these local variance fluctuations depends heavily on the architecture’s inductive bias and sampling density. A Vision Transformer backbone, such as the DiT, partitions the input into N sparse, non-overlapping patches, making it statistically more likely to average out spatial sub-structures. Conversely, a convolutional U-Net employing stride-1 convolutions processes the spatial grid as a highly dense set of ≈ p2 N (p > 1) overlapping receptive fields. This sliding-window mechanism dramatically maximizes the empirical probability of encountering localized regions where the noise scale ᾱt remains identifiable. Consequently, this structural sampling disparity enables CNN-based backbones to maintain robust time-agnostic convergence even within normalized latent manifolds, whereas non-overlapping attention grids suffer a noticeable performance degradation. This architectural insight underscores the critical interplay between model design and the theoretical conditions for timestep redundancy, suggesting that convolutional architectures may be inherently better suited to leverage local identifiability in high-dimensional generative modeling tasks.
7
Conclusion
This work has investigated the role of timestep embeddings in diffusion models, providing both theoretical insights and empirical evidence for their redundancy in high-dimensional regimes. Our theoretical framework demonstrates that the temporal signal is not strictly necessary to guide the denoising process, as the noise scale ᾱt remains asymptotically identifiable through the corrupted input xt under certain conditions. Quantitative and qualitative results across the CelebA and CIFAR-10 datasets support these findings, showing that time-agnostic models can maintain—and in some cases, surpass—the performance of their conditioned counterparts while reducing computational overhead. While our study is focused on low-to-medium resolution dataset, the scalability to higher resolution synthesis and more complex datasets remains to be verified. Future work should explore these frontiers to determine if timestep embeddings become necessary as data complexity and resolution increase.
References 1. Bengio, Y., Yao, L., Alain, G., Vincent, P.: Generalized denoising auto-encoders as generative models. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1. p. 899–907. NIPS’13, Curran Associates Inc., Red Hook, NY, USA (2013)
On the Redundancy of Timestep Embeddings in Diffusion Models
15
2. Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 22563–22575 (June 2023) 3. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009. 5206848 4. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021), https://openreview. net/forum?id=YicbFdNTTy 5. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6629–6640. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017) 6. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 6840–6851. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf 7. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 1510–1519 (2017). https://doi.org/10.1109/ICCV.2017.167 8. Hyvärinen, A.: Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research 6(24), 695–709 (2005), http://jmlr.org/ papers/v6/hyvarinen05a.html 9. Kadkhodaie, Z., Pooladian, A.A., Chewi, S., Simoncelli, E.: Blind denoising diffusion models and the blessings of dimensionality. arXiv preprint arXiv:2602.09639 (2026) 10. Karras, T., Aittala, M., Laine, S., Aila, T.: Elucidating the design space of diffusionbased generative models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022) 11. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4396–4405 (2019). https://doi.org/10.1109/ CVPR.2019.00453 12. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013) 13. Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved precision and recall metric for assessing generative models. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper_files/paper/2019/file/ 0234c510bc6d908b28c70ff313743079-Paper.pdf 14. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (December 2015)
16
J. Chávez
15. Luo, C.: Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970 (2022) 16. Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International conference on machine learning. pp. 8162–8171. PMLR (2021) 17. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4195– 4205 (October 2023) 18. Perez, E., Strub, F., de Vries, H., Dumoulin, V., Courville, A.: Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence 32(1) (Apr 2018). https://doi.org/10.1609/aaai.v32i1. 11671, https://ojs.aaai.org/index.php/AAAI/article/view/11671 19. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684– 10695 (June 2022) 20. Sahraee-Ardakan, M., Delbracio, M., Milanfar, P.: The geometry of noise: Why diffusion models don’t need noise conditioning. arXiv preprint arXiv:2602.18428 (2026) 21. Sauer, A., Lorenz, D., Blattmann, A., Rombach, R.: Adversarial diffusion distillation (2023), https://arxiv.org/abs/2311.17042 22. Sauer, A., Lorenz, D., Blattmann, A., Rombach, R.: Adversarial diffusion distillation. In: Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXXVI. p. 87–103. SpringerVerlag, Berlin, Heidelberg (2024). https://doi.org/10.1007/978-3-031-730160_6, https://doi.org/10.1007/978-3-031-73016-0_6 23. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 2256–2265. PMLR, Lille, France (07–09 Jul 2015), https://proceedings.mlr.press/v37/sohl- dickstein15. html 24. Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020) 25. Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper_files/ paper/2019/file/3001ef257407d5a371a96dcd947c7d93-Paper.pdf 26. Sun, Q., Jiang, Z., Zhao, H., He, K.: Is noise conditioning necessary for denoising generative models? arXiv preprint arXiv:2502.13129 (2025) 27. Tian, K., Jiang, Y., Yuan, Z., PENG, B., Wang, L.: Visual autoregressive modeling: Scalable image generation via next-scale prediction. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024), https: //openreview.net/forum?id=gojL67CfS8 28. Vincent, P.: A connection between score matching and denoising autoencoders. Neural Computation 23(7), 1661–1674 (2011). https://doi.org/10.1162/NECO_ a_00142 29. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning. p. 1096–1103. ICML ’08, Association for
On the Redundancy of Timestep Embeddings in Diffusion Models
17
Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/ 1390156.1390294, https://doi.org/10.1145/1390156.1390294 30. Xiao, S., Wang, Y., Zhou, J., Yuan, H., Xing, X., Yan, R., Li, C., Wang, S., Huang, T., Liu, Z.: Omnigen: Unified image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13294– 13304 (June 2025)