arXiv:2605.10790v1 [cs.LG] 11 May 2026
Elucidating Representation Degradation Problem in Diffusion Model Training
Zhipeng Yao 1 ⋆ Dazhou Li 3 ⋆ Zitong Zhang 1 Durude Mahee 1 Fan Zhu 1 Wenbin Zhang 4 Xinwei He 5 Yeying Jin 2 † Rui Yu 1 † 1 University of Louisville 2 National University of Singapore 4 Florida International University 3 Shenyang University of Chemical Technology 5 Huazhong Agricultural University § https://github.com/LilYau350/Elucidated-Representation-Diffusion
Abstract Diffusion models have achieved remarkable success, yet their training remains inefficient due to a severe optimization bottleneck, which we term Representation Degradation. As noise levels increase, the outputs of the trained model exhibit progressive structural distortion, which can destabilize training and impair generation quality. Our analysis suggests that this instability is driven by mismatched target recoverability, which is associated with Neural Tangent Kernel (NTK) spectral weakening and effective low-rank behavior. To address this, we propose Elucidated Representation Diffusion (ERD), a plug-and-play framework that dynamically reallocates optimization effort according to effective recoverability. By stabilizing representation learning without external supervision, ERD accelerates convergence and achieves strong empirical performance across diffusion backbones.
1
Introduction
Diffusion models [50, 15, 52] have become a dominant paradigm for generative modeling [34, 17, 28, 67]. As the field increasingly shifts toward scalable Transformer-based architectures [43, 2], improving training efficiency has become a central challenge. Existing methods often accelerate training by heuristically reallocating the optimization effort across noise levels [11, 23, 55, 65, 10, 66], but they remain largely agnostic to the quality of the internal features learned. More recently, Representation Alignment (REPA) [62] has shown that aligning diffusion features with pre-trained priors can substantially accelerate training. This empirical success raises a fundamental question: “what intrinsic bottleneck makes standard diffusion training inefficient in the first place?” To answer this question, we empirically examine how the output of the trained model evolves across noise levels. We find that although the predicted geometry aligns well with the target distribution at low noise levels, it becomes increasingly distorted and eventually collapses under high noise. As illustrated in Fig. 1, the predicted distribution, when projected onto a 2D plane, preserves meaningful structure at low noise but loses it severely at high noise. We refer to this phenomenon as Representation Degradation [7]. These observations reveal an intrinsic instability in diffusion training, demonstrating that as noise increases, the model becomes progressively less able to preserve semantically meaningful structure. To explain this phenomenon, we analyze diffusion training in the Neural Tangent Kernel (NTK) regime as a continuous-time, noise-conditioned regression process [19, 25]. Our analysis shows that optimization at each noise level is jointly shaped by three factors: the allocation of optimization effort, the local kernel geometry induced by corrupted inputs, and the recoverability of the target itself. As noise increases, corrupted inputs lose informative structure, which weakens the local Preprint. ⋆ Equal contribution. † Equal advising. Contact: [email protected], [email protected]
Given xt input network, where t ∈ 0 → T 0.8
1.0 0.5
0.6
2
1.0
0.4
1
0.5
0
0.0
1
0.5
0.2 0.0
0.0 0.2
0.5
0.4
2
0.6
1.0 1.0
0.5
0.0
0.5
1.0
0.5
0.0
3
0.5
(a) xθ -prediction.
1.0
2
1
0
1
2
1.0
0.5 0.0
0.5
1.0
(b) ϵθ -prediction.
Figure 1: We visualized pre-trained model predictions as forward diffusion progressively corrupts the inputs. Fig. 1a shows that the image distribution becomes increasingly stretched and flattened, while Fig. 1b shows the predicted noise shifting from an isotropic to an anisotropic distribution. NTK spectrum and reduces learnability along semantically meaningful directions. Only a small fraction of the target remains effectively recoverable within finite training time, while standard training may still allocate substantial optimization effort to these weakly recoverable regions [3]. This mismatch is consistent with spectral degeneration [44], effective low-rank behavior, and contributes to representation degradation [21] in the extreme-noise regime. These findings identify representation degradation as an intrinsic optimization bottleneck rather than a purely architectural limitation. To address it, we propose Elucidated Representation Diffusion (ERD), a training framework that reallocates optimization effort according to target recoverability without relying on external alignment networks. Specifically, ERD adopts a target-adaptive weighting rule that emphasizes noise levels with stronger recoverable signal and downweights regions that receive excessive optimization mass despite limited learnability. In this way, ERD improves optimization stability, mitigates representation degradation, and accelerates convergence. Our main contributions are summarized as follows: • Representation Degradation: We identify Representation Degradation in diffusion training, showing that model outputs undergo substantial structural distortion and collapse as noise increases, reflecting an instability in the learned representations. • Optimization Analysis: We develop an NTK-based framework showing how allocation, local kernel geometry, and target recoverability jointly induce spectral degeneration and effective lowrank collapse in high-noise regimes. • Elucidated Representation Diffusion: We propose an architecture-agnostic training framework that derives a target-adaptive weighting rule from recoverable signal at each noise level, improving optimization stability and accelerating diffusion training without external priors.
2
Preliminaries
In this section, we introduce the diffusion formulation in the log-signal-to-noise ratio (log-SNR) domain and the continuous-time Neural Tangent Kernel framework used in subsequent analysis. 2.1
Diffusion Models in the Log-SNR Domain
Let x0 ∼ q(x0 ) be a data sample in Rd . The forward diffusion process generates latent variables through the transition q(xt | x0 ) = N (xt ; αt x0 , σt2 I). We reparameterize the process by the kernel α2
log-SNR λ ≜ log σ2t . We assume that the diffusion schedule is chosen so that the map t 7→ λt t is monotone on the interval of interest, allowing t and λ to be used interchangeably. Under this parameterization, the corrupted variable can be written as xλ = α(λ)x0 + σ(λ)ϵ, where ϵ ∼ N (0, I).
We consider a generalized target of the form yλ = cx (λ)x0 + cϵ (λ)ϵ, which subsumes standard parameterizations such as xθ -prediction, ϵθ -prediction, and vθ -prediction. Following the continuoustime variational formulation of diffusion training [26, 25], we consider the corresponding weighted training objective Z λmax 1 LELBO (θ) = M (λ) Ex0 ,ϵ ∥fθ (xλ , λ) − yλ ∥22 dλ, (1) 2 λmin 2
dt where fθ is parameterized by θ ∈ RP , and M (λ) ≜ w(λ)p(λ) dλ denotes the effective allocation measure induced by the loss weighting, the sampling distribution over noise levels, and the schedule Jacobian. Throughout the paper, M (λ) serves as a compact description of how optimization effort is distributed across the log-SNR domain.
2.2
Continuous-Time Dynamics and the Neural Tangent Kernel
We study Eq. (1) from a continuous-time optimization perspective. Let τ ≥ 0 denote training time. ) Under gradient flow, the parameters evolve according to dθ(τ dτ = −∇θ LELBO (θ). Because diffusion models use a shared-parameter architecture across all noise levels, optimization at one noise level generally interacts with optimization at others. This motivates describing the training process on the joint input–noise space rather than treating each noise level independently. For this purpose, we introduce the matrix-valued joint Neural Tangent Kernel (NTK) ⊤ Θ (x, λ), (x′ , λ′ ) = ∇θ fθ (x, λ) ∇θ fθ (x′ , λ′ ) ∈ Rd×d , (2) where ∇θ fθ (x, λ) ∈ Rd×P denotes the Jacobian of the network output with respect to the parameters. This joint kernel acts as a linear operator on Rd and captures both coupling across output dimensions and cross-noise interactions induced by parameter sharing. In particular, it provides a convenient framework for describing how training signals at different noise levels interact along a shared optimization trajectory.
In the overparameterized regime [19], the NTK perspective offers a useful approximation for analyzing training dynamics. In the following sections, we will build on this viewpoint to study how optimization allocation, input corruption, and target structure jointly influence diffusion training across noise levels.
3
Elucidating Representation Degradation
To explain standard diffusion training inefficiency, we study overparameterized continuous-time dynamics. By connecting NTK dynamics with the global ELBO, we formalize target-dependent recoverability mismatch: optimization effort may be over-allocated to weakly recoverable signals. Under our surrogate dynamics, this mismatch weakens contraction and introduces Bayes-noise forcing, providing a mechanism for representation degradation. Proofs are in Sections A to D. 3.1
Neural Tangent Kernel Dynamics in Diffusion Models
We first derive continuous-time dynamics on the joint input–noise space. Under the assumptions in Section A, let τ denote training time andm the width scaling parameter. The time-dependent joint NTK is denoted by Θτ,m (x, λ), (x′ , λ′ ) , and the residual by eτ,m (x, λ) ≜ fτ,m (x, λ) − yλ . Proposition 3.1 (Finite-Width Joint Dynamics). Under the population gradient flow associated with the joint regression objective, the residual eτ,m (x, λ) follows the dynamics: ∂ eτ,m (x, λ) = −Eλ′ ∼ρ, x′ ∼qλ′ Θτ,m (x, λ), (x′ , λ′ ) eτ,m (x′ , λ′ ) . ∂τ
(3)
Thus, residual evolution at noise level λ is governed by a global expectation over all training noise levels λ′ . Through the joint NTK, optimization signals from different noise regimes interact along the same parameter trajectory, so updates from one regime can influence residuals elsewhere. This makes explicit that fixed-noise training cannot, in general, be treated as an isolated subsystem under shared-parameter diffusion training. 3.2
Evidence Lower Bound Dynamics
We next lift the joint NTK dynamics to the global ELBO. For fixed λ, let fλ⋆ (x) ≜ E[yλ | xλ = x] denote the Bayes predictor. Under the diffusion parameterization and the assumptions in Section A, the decomposition below is exact at the population level. 3
θ -prediction
xθ -prediction
vθ -prediction
uθ -prediction
100
MSE loss
100 10−1 10−1
10−2
10−3
10−2
Empirical Bayes Floor 0.0
0.2
0.4
100
0.6
Diffusion time t
0.8
1.0
Empirical Bayes Floor 0.0
0.2
0.4
0.6
Diffusion time t
0.8
1.0
Empirical Bayes Floor 0.0
0.2
0.4
0.6
Diffusion time t
0.8
1.0
100
Empirical Bayes Floor 0.0
0.2
0.4
0.6
Diffusion time t
0.8
1.0
Figure 2: Empirical MSE versus the Bayes optimal error (Bayes floor) across diffusion time t for different prediction targets1 . The Bayes floor is given by the conditional expectation E[y | xt ]. The gap between the empirical loss and the Bayes floor reflects the learnability of each parameterization. Lemma 3.2 (ELBO Bayes Decomposition). For each fixed log-SNR level λ, the local weighted ELBO density orthogonal decomposition ℓτ,m (λ) = ℓ⋆λ + ℓ̄τ,m (λ), where ℓ⋆λ ≜ ⋆ admits an 2exact 1 1 2 M (λ)E ∥fλ (xλ ) − yλ ∥2 is the irreducible Bayes floor, and ℓ̄τ,m (λ) ≜ 2 M (λ)∥fτ,m (·, λ) − fλ⋆ (·)∥2L2 (qλ ) is the optimizable excess density. Thus only ℓ̄τ,m (λ) is optimizable, while ℓ⋆λ is parameter-independent. Equivalently, the population MSE decomposes into the irreducible Bayes error E[∥yt − E[yt | xt ]∥22 ] and an optimizable excess term. As shown in Fig. 2, the Bayes floor lower-bounds the achievable prediction error, and the gap to the empirical loss reflects the remaining optimizable component. In poorly recoverable regimes, the optimizable excess can be small relative to the irreducible term, so the local ELBO becomes dominated by target-dependent Bayes error; under the sample-path surrogate below, this irreducible component appears as the forcing term ξτ . To characterize the excess dynamics, we adopt the continuous-time sample-path surrogate of SGD from Section C, with trajectory τ 7→ (xλτ , λτ ) and Bayes fluctuation ξτ ≜ fλ⋆τ (xλτ ) − yλτ . Let rτ,m (x, λ) ≜ fτ,m (x, λ) − fλ⋆ (x) denote the Bayes-centered residual. Theorem 3.3 (Local Excess Dynamics). Under the continuous-time surrogate dynamics, the local excess density satisfies the exact differential identity: D E d ℓ̄τ,m (λ) = −M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 dτ L (qλ ) D E − M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ . (4) L2 (qλ )
The first term contracts the Bayes-centered residual, whereas the second injects Bayes forcing from the irreducible target component through the shared-parameter trajectory. To obtain an explicit long-time bound, we impose the pathwise coercivity condition D E M (λ) ru,m (·, λ), M (λu )Θu,m (·, λ), (xλu , λu ) ru,m (xλu , λu ) ≥ µu (λ) ℓ̄u,m (λ). (5) L2 (qλ )
Here µu (λ) acts asR an effective contraction rate under the surrogate dynamics. For β > 0, define τ Γτ,s (λ) ≜ exp − s (µu (λ) − β) du and Fs (λ) ≜ M (λs )Θs,m (·, λ), (xλs , λs ) ξs . Integrating Eq. (4) under Eq. (5) yields: Theorem 3.4 (Non-asymptotic Degradation Bound). For any evaluation time τ ≥ 0, the optimizable excess density is bounded by: Z M (λ) τ 2 ℓ̄τ,m (λ) ≤ Γτ,0 (λ)ℓ̄0,m (λ) + Γτ,s (λ) Fs (λ) L2 (q ) ds. (6) λ 2β 0 Remark 3.5 (Rate interpretation). The bound in Theorem 3.4 yields a continuous-time rate interpretation. If, for a fixed evaluation level λ, the effective contraction rate satisfies µu (λ) ≥ µ(λ) > 0 and the Bayes-forcing term is uniformly bounded as ∥Fs (λ)∥2L2 (qλ ) ≤ B(λ), then for any β ∈ (0, µ(λ)), M (λ)B(λ) ℓ̄τ,m (λ) ≤ e−(µ(λ)−β)τ ℓ̄0,m (λ) + 1 − e−(µ(λ)−β)τ . (7) 2β(µ(λ) − β) 1 ϵ and x denote the noise and data, v denotes the velocity [47], and u denotes the flow matching vector field [36].
4
θ -prediction
xθ -prediction
Degradation
101
Degradation
Degradation
100
100
100
10−1
10−1
10−1
10−2
10−2
10−2
uθ -prediction 101
Degradation 0.8
100
0.6
10−1
0.4
10−2
0.2
Noise Level (Time t)
Noise Norm
vθ -prediction
101
101
10−3 Healthy
Healthy 10−3
10−3 10−3
10−2
10−1
100
Signal Norm
101
10−3
10−2
10−1
100
Signal Norm
101
Healthy
10−3 10−3
10−2
10−1
100
Signal Norm
101
Healthy 10−3
10−2
10−1
100
Signal Norm
101
Figure 3: Signal–noise decomposition across diffusion time t. The horizontal axis shows the recoverable signal norm ∥E[yλ | xt ]∥2 , and the vertical axis shows the irreducible Bayes-noise norm ∥yλ − E[yλ | xt ]∥2 . Larger vertical values indicate stronger Bayes-noise contamination. Thus the transient excess contracts at rate O(e−(µ(λ)−β)τ ) toward a forcing-controlled neighborhood of size O(M (λ)B(λ)/[β(µ(λ) − β)]). Reaching accuracy ε above this neighborhood requires training time O (µ(λ) − β)−1 log(ℓ̄0,m (λ)/ε) . Hence recoverability mismatch slows optimization via small µ(λ) and enlarges the residual neighborhood via large M (λ) or B(λ). This rate view shows that persistent Bayes forcing can control the remaining error even after the initial excess contracts, motivating the recoverability analysis below. 3.3
Recoverability Mismatch and Representation Degradation
The balance between contraction and forcing varies across both noise levels and prediction targets, yielding recoverability mismatch: M (λ) may remain large where fλ⋆ (xλ ) = E[yλ | xλ ] captures only a small recoverable component of the target. Since representation quality is closely tied to posterior estimation quality [35], such mismatch can degrade learned features. For targets whose recoverable component depends on the data signal, the extreme-noise regime makes xλ weakly informative about that component, so the local ELBO may become dominated by target-dependent Bayes error. Combining this regime with Theorem 3.4 gives: Theorem 3.6 (Degradation under Recoverability Mismatch). Under the sample-path surrogate, the pathwise coercivity condition underlying Theorem 3.4, and the target-dependent recoverability-loss regime described above, the local ELBO density is bounded by three competing components: Z M (λ) τ 2 ℓτ,m (λ) ≤ ℓ⋆λ + Γτ,0 (λ)ℓ̄0,m (λ) + (8) Γτ,s (λ) Fs (λ) L2 (q ) ds . λ |{z} 2β | {z } 0 {z } | Target-dependent Weakened Contraction Continuous Cross-Noise Contamination
Bayes Floor
This decomposes the local ELBO into an irreducible Bayes floor, a contracted initial excess term, and cross-noise forcing from shared-parameter training. Thus degradation is not a single-noise phenomenon; it is mediated by cross-noise interactions through Fs (λ). This bound shows why poorly recoverable regimes are hard to optimize: Bayes error can dominate, contraction weakens, and cross-noise forcing persists through the joint NTK. We next connect this loss-level mechanism to representation learning. Under fθ (x, λ) = Wθ hθ (x, λ), the same Bayes-noise component enters the representation-parameter gradient. Proposition 3.7 (Bayes-Noise Contamination of Representation Gradients). The stochastic gradient with respect to the representation parameters ϑ exactly decomposes as: b = M (λ) ∇ϑ hθ (xλ , λ) ⊤ W ⊤ rθ (xλ , λ) + M (λ) ∇ϑ hθ (xλ , λ) ⊤ W ⊤ ξλ , ∇ϑ ℓ(θ) (9) θ θ where rθ is the recoverable Bayes-centered residual (signal), and ξλ is the irreducible Bayes noise. When ∥rθ (xλ , λ)∥2 ≪ ∥ξλ ∥2 , the ξλ -term dominates Eq. (9). As shown in Fig. 3, severe mismatch regimes, e.g., high noise for x0 -prediction or low noise for ϵ-prediction, exhibit Bayes-noise norms far larger than recoverable signal norms. Thus, gradients there can be dominated by irreducible noise, allowing large M (λ) to interfere with representations learned in more recoverable regimes. 5
NTK Matrix at high noise level
104
1.0 0.8 0.6 0.4 0.2
NTK Eigenvalues
NTK Matrix at low noise level
∙1 ∙2 ∙3
103
102 0.0
0.0
0.2
0.4
0.6
0.8
1.0
Diffusion Time t
(a) NTK heatmaps
(b) NTK spectrum
Figure 4: NTK analysis across diffusion noise levels. Fig. 4a Joint NTK heatmaps visualize crossnoise coupling induced by shared parameters. Fig. 4b The NTK spectrum characterizes mode-wise contraction strength and reveals weakened high-noise optimization modes. Finally, we refine the contraction behavior mode by mode using a local fixed-noise spectral surrogate at evaluation level λ. The trajectory is assumed to remain in the NTK regime, and the frozen fixed-noise kernel on L2 (qλ ; Rd ) admits an orthonormal eigensystem {κλj }j≥1 . Theorem 3.8 (Spectral Local ELBO Bound). Under a fixed-noise surrogate approximation, the local ELBO density satisfies the following mode-wise bound for any constant γ > 0: Z M (λ) X −(2M (λ)κλj −γ)τ λ 2 M (λ) X τ −(2M (λ)κλj −γ)(τ −s) λ 2 ⋆ e ℓτ,m (λ) ≤ ℓλ + |a0,j | + e |ηs,j | ds (10) 2 2γ 0 j≥1
j≥1
λ where aλ0,j and ηs,j are the modal coefficients of the initial residual and Bayes forcing ξs , respectively.
This bound gives a mode-wise view of the same mechanism: modes with 2M (λ)κλj > γ contract rapidly, while small κλj weakens contraction and increases sensitivity to forcing. As shown in Fig. 4, the empirical NTK spectrum weakens at high noise, and the joint NTK heatmaps reveal cross-noise coupling. Together with Theorem 3.7, these observations suggest that weakened spectral contraction and Bayes-noise-dominated gradients jointly contribute to representation degradation. The analysis above motivates reallocating optimization effort away from poorly recoverable regimes, which leads to the target-adaptive weighting rule introduced next. 3.4
Elucidated Representation Diffusion
Motivated by recoverability mismatch, we propose Elucidated Representation Diffusion (ERD), a target-adaptive weighting framework that reallocates optimization effort according to recoverable signal. ERD is designed to reduce Bayes-noise-dominated updates without external priors or architectural modifications. Using the generalized parameterization from Section 2.1, assume x0 ⊥ ϵ, E[x0 ] = 0, Cov(x0 ) = I, and ϵ ∼ N (0, I). Existing efficient training methods often reweight losses using target energy, marginal SNR, or timestep allocation [3, 11, 61, 65, 55]. However, these criteria do not explicitly measure how much of each target component is expressed in the corrupted input. Since x0 and ϵ enter xλ with amplitudes α(λ) and σ(λ), the learnability of yλ depends on both target energy and input-side expression. We define the channel-scaled effective target as yeλ ≜ cx (λ)α(λ)x0 +cϵ (λ)σ(λ)ϵ. The recoverability score ωy (λ) is its expected root-mean-square amplitude. Since x0 and ϵ are independent and normalized, we obtain r q 1 2 2 2 ωy (λ) = E[∥e yλ ∥2 ] = (cx (λ)α(λ)) + (cϵ (λ)σ(λ)) . (11) d Here ωy (λ) is a tractable component-wise proxy for target recoverability, rather than an exact posterior measure such as ∥E[yλ | xλ ]∥. Unlike SNR- or target-energy-based weighting, ERD weights a target by the amplitude of components actually expressed in the corrupted input, so targets with similar energy can receive different weights when their recoverable components differ. 6
Figure 5: Selected 256×256 samples. We use a CFG scale of 4.0 and 50 EDM Heun steps. Under the standard continuous-time configuration where log-SNR is sampled uniformly and scheduledependent factors are absorbed into the base measure, the effective allocation M (λ) is controlled by the loss weight w(λ). ERD sets wy⋆ (λ) ∝ ωy (λ) and normalizes it to preserve the average loss scale. For non-uniform base allocations, the same principle applies after accounting for the timestep sampler and schedule Jacobian. The RMS form is sign-invariant: it measures the absolute recoverable contribution of each independent channel and avoids spurious cancellation between components with opposite signs. Substituting the target-specific coefficients cx (λ) and cϵ (λ) into Eq. (11) yields a unified weighting rule for canonical diffusion and flow-matching objectives. Applying wy⋆ (λ) reduces optimization mass where the effective recoverable signal is weak. This targets the mismatch in Theorem 3.7: when the recoverable residual is small relative to irreducible Bayes noise, reducing M (λ) suppresses noise-dominated representation updates. ERD therefore alleviates recoverability mismatch and improves training efficiency without external supervision or architectural overhead.
4
Experiments
This section details the experimental setup for image generation, including datasets, architectures, and training configurations. We present ablation studies and method comparisons to demonstrate our approach’s adaptability to different models and targets. Lastly, system-level evaluations assess the overall effectiveness of our method. 4.1
Experimental Setup
Datasets. We evaluate image generation on ImageNet [5] and CelebA [37]. For ImageNet, which contains over 1.3M images from 1,000 classes, we use the 256×256 version and adopt Latent Diffusion Model (LDM) training [45] for efficiency, encoding images into 32×32×4 latent representations with the Stable Diffusion VQ-VAE encoder [54]. For CelebA, which contains over 200K celebrity face images, we follow [52] to crop and resize images to 64×64 and train models in pixel space for unsupervised generation. Training. Most experiments use DiT [43] and U-ViT [2] as backbones optimized via Adam optimizer [27, 38], while ablations also include UNet [46] under the ADM framework [6]. Unless otherwise specified, training is conducted in the LDM latent space [45] with VQ-VAE encoding [54]; we use the DDPM framework [15] with T = 1000 diffusion steps and apply Exponential Moving Average (EMA) with decay 0.9999 [15]. Evaluation. We evaluate EMA models by generating 50,000 images, i.e., FID-50K. Performance is measured by FID [13] and sFID [41], where lower is better, and Inception Score (IS) [48], where higher is better. We also report Precision and Recall [29] for fidelity and coverage. For classconditional generation, we use classifier-free sampling [14]; all image sampling adopts the Heun sampler from EDM [23]. 7
Table 1: Training strategy ablation of DiT [43] in latent space [45] without classifier-free guidance (CFG).
Table 2: FID results on CelebA 64×64 from architectural ablations centered on UNet and DiT frameworks.
Training Strategy
FID-50k↓
Method
Baseline Improvement DiT-S/2 [43] + Ours
69.35 62.92
Training Trick DiT-S/2 [43] + Adam [27] β2 = 0.95 [1, 60] + Cosine schedule [42] + Ours
69.35 68.77 65.68 57.54
4.2
#Params
FID↓
Ablation on U-Net DDIM [51] Soft Truncation [51] UNet (Ours)
79M 62M 59M
3.26 1.90 1.53
Ablation on Transformer U-ViT-S [2] DiT-S DiT-S (Ours)
44M 33M 33M
2.87 3.00 2.37
Ablation Study
We conduct ablation studies to evaluate the effects of model architecture and training strategy on image generation performance. For training-strategy ablations, we train DiT in latent space on ImageNet for 400K steps with a batch size of 256 and a fixed learning rate of 10−4 , keeping other configurations consistent with the pixel-space setup. For architecture ablations, we train UNet (following the architecture configurations in [11]) and DiT in pixel space on CelebA with a batch size of 128. We optimize both models using Adam and a cosine noise schedule [42] over 500K training iterations. Specifically, we set the learning rate to 10−3 with betas (0.9, 0.95) for DiT, and the learning rate to 10−4 with betas (0.9, 0.999) for UNet [11]. Tab. 1 shows consistent improvements across different training strategies, while Tab. 2 shows our method remains effective across architectures.
100
120 100
80
80
60
60 100K
200K
300K
Training Iteration (a) ϵθ -prediction
400K
Baseline Min-SNR Ours
110
FID-50K
FID-50K
120
120
Baseline Min-SNR Ours
140
FID-50K
Baseline Min-SNR P2 Ours
140
100 90 80 70 60
100K
200K
300K
Training Iteration (b) xθ -prediction
400K
100K
200K
300K
Training Iteration (c) vθ -prediction
400K
Figure 6: Comparing different loss weighting designs by predicting ϵθ , xθ and vθ on DiT-S. 4.3
Comparison with Other Methods
Referring to Tab. 1, we configured the cosine learning rate scheduler with the Adam optimizer’s betas set to (0.9, 0.95), and conducted the training with a batch size of 256 over 400K steps, ensuring robust convergence and optimization. We evaluated different prediction objectives, ϵθ , xθ , and vθ [47], by comparing the Baseline, Min-SNR [11], P2 [3], and our proposed weighting method for training DiT-S on ImageNet. As illustrated in Fig. 6, our method consistently achieves the lowest FID-50K scores across all prediction objectives, demonstrating superior performance over other weighting methods throughout the training iterations. 4.4
System Level Comparison
We thoroughly evaluate our method on both DiT and U-ViT architectures using the ImageNet 256×256 dataset. In Tab. 3, we report FID scores without classifier-free guidance (CFG). Under the same training schedule (400K iterations), our method consistently outperforms the original DiT and U-ViT, as well as the SiT baseline, across all model scales. We also evaluate system-level performance under CFG, as shown in Tab. 4. Compared with strong baselines including LDM [45], U-ViT-H/2 [2], and SD-DiT [68], our method achieves highly competitive performance, significantly narrowing the gap with the recent REPA [62]. 8
Table 3: FID comparisons with DiTs, SiTs and U-ViT on ImageNet 256×256. We do not use classifier-free guidance (CFG). ↓ denotes lower values are better. Iter. indicates the training iteration.
5
Table 4: System-level comparison on ImageNet 256×256 with CFG. ↓ and ↑ indicate whether lower or higher values are better, respectively. Results that include additional CFG scheduling are marked with an asterisk (*), where the guidance interval from [30] is applied for.
Model
#Params
Iter.
FID↓
DiT-B/2 SiT-B/2 Ours
130M 130M 130M
400K 400K 400K
43.4 33.0 32.3
DiT-L/2 SiT-L/2 Ours
458M 458M 458M
400K 400K 400K
23.3 18.8 17.2
DiT-XL/2 SiT-XL Ours
675M 675M 675M
400K 400K 400K
19.5 17.2 15.0
U-ViT-M/2 Ours
131M 131M
400K 400K
27.7 19.9
DiT-XL/2 [43] SiT-XL/2 [40] REPA* [62]
U-ViT-L/2 Ours
287M 287M
400K 400K
20.4 15.0
U-ViT-H/2 Min-SNR Ours
501M 501M 501M
400K 400K 400K
13.7 11.7 8.9
Model
Epochs
FID↓
sFID↓
IS↑
Pre.↑
Rec.↑
Pixel diffusion ADM-U [6] VDM++[26] Simple diffusion [18] CDM [16]
400 560 800 2160
3.94 2.40 2.77 4.88
6.14 -
186.7 225.3 211.8 158.7
0.82 -
0.52 -
Latent diffusion, U-Net LDM-4 [45]
200
3.60
-
247.7
0.87
0.48
Latent diffusion, Transformer MaskDiT [64] 1600 SD-DiT [68] 480 Min-SNR* [11] 1400
2.28 3.23 1.57
5.67 -
276.6 -
0.80 -
0.61 -
2.27 2.06 1.42
4.60 4.50 4.70
278.2 270.3 305.7
0.83 0.82 0.80
0.57 0.59 0.65
Latent diffusion, Transformer + U-Net hybrid DiffiT* [12] 1.73 MDTv2-XL/2* [8] 1080 1.58 4.52
276.5 314.7
0.80 0.79
0.62 0.65
U-ViT-H/2 [2] Ours Ours*
263.9 282.5 315.4
0.82 0.82 0.81
0.57 0.60 0.62
1400 1400 800
400 800 800
2.29 1.89 1.45
5.68 4.48 4.51
Related Work
Diffusion Model Training Acceleration. Prior work improves diffusion training efficiency mainly through resource allocation across noise levels [3, 11, 61, 59, 58, 66, 65, 55, 42, 10]. Other approaches modify training dynamics or the diffusion process itself [33, 57]. Despite their effectiveness, these methods are largely heuristic and do not explain the intrinsic optimization difficulty. Representation Alignment in Diffusion Models. Recent works suggest that improving representation quality can substantially accelerate diffusion training, through representation alignment and related pretraining or latent modeling paradigms [62, 32, 60, 56, 20, 63, 49, 4]. These results indicate that diffusion training efficiency depends not only on noise-level allocation, but also on learned internal representations. However, existing methods mainly exploit such benefits through external priors, auxiliary objectives, or modified latent frameworks, without explaining the intrinsic role of representations in standard diffusion training. In contrast, our work reveals that standard diffusion training inherently suffers from representation degradation. Theoretical Analyses of Diffusion and NTK Dynamics. Prior theory has studied diffusion models from variational, dynamical, and score-based perspectives [26, 39, 25, 24, 9]. In parallel, NTK theory and studies on spectral bias and collapse [19, 31, 44, 22] provide tools for characterizing optimization geometry and representation collapse. However, these lines of work have not been connected to explain why extreme-noise diffusion training leads to degraded representations. Our work bridges this gap by linking diffusion optimization to noise-dependent NTK dynamics and showing that extreme corruption induces spectral degeneration and representation collapse.
6
Limitations & Conclusion
Limitations. Our analysis relies on continuous-time optimization, NTK-regime, and sample-path surrogate assumptions, which may not fully capture finite-width and highly nonlinear diffusion training dynamics. In addition, ERD uses an effective-amplitude score as a tractable proxy for recoverability, rather than directly estimating the exact posterior recoverable signal. Conclusion. Representation degradation arises when weak recoverability, enlarged Bayes floors, and spectral weakening bias diffusion training toward irreducible noise. ERD mitigates this issue by reallocating optimization effort according to effective target recoverability, improving training efficiency and generation quality across architectures and prediction targets. 9
Acknowledgment This research was supported by N VIDIA grants and utilized N VIDIA GPUs and software, including N VIDIA A100 GPUs on Brev. The authors gratefully acknowledge the N VIDIA Academic Grant Program for providing computational resources that enabled this work.
References [1] Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Zhen Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. Advances in Neural Information Processing Systems, 36:34278–34294, 2023. 8 [2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669–22679, 2023. 1, 7, 8, 9, 33, 35 [3] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022. 2, 6, 8, 9 [4] Xiangxiang Chu, Renda Li, and Yong Wang. Usp: Unified self-supervised pretraining for image generation and understanding. arXiv preprint arXiv:2503.06132, 2025. 9 [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009. 7 [6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. 7, 9, 33 [7] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019. 1 [8] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389, 2023. 9 [9] Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. arXiv preprint arXiv:2401.15604, 2024. 9 [10] Tiankai Hang and Shuyang Gu. arXiv:2407.03297, 2024. 1, 9
Improved noise schedule for diffusion training.
arXiv preprint
[11] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7441–7451, 2023. 1, 6, 8, 9 [12] Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. In European Conference on Computer Vision, pages 37–55. Springer, 2024. 9 [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 7, 33 [14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 7 [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 7, 28 [16] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1–33, 2022. 9 [17] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 1
10
[18] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213–13232. PMLR, 2023. 9 [19] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018. 1, 3, 9 [20] Dengyang Jiang, Mengmeng Wang, Liuzhuozheng Li, Lei Zhang, Haoyu Wang, Wei Wei, Guang Dai, Yanning Zhang, and Jingdong Wang. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves. arXiv preprint arXiv:2505.02831, 2025. 9, 35 [21] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021. 2 [22] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. 9 [23] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 1, 7, 33 [24] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24174–24184, 2024. 9, 33 [25] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36:65484–65516, 2023. 1, 2, 9, 28 [26] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021. 2, 9, 28 [27] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 7, 8, 33 [28] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020. 1 [29] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019. 7, 34 [30] Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724, 2024. 9, 35 [31] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019. 9 [32] Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504.10483, 2025. 9 [33] Wenhao Li, Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, and Chang Xu. Not all steps are equal: Efficient generation with progressive diffusion models. arXiv preprint arXiv:2312.13307, 2023. 9 [34] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328–4343, 2022. 1 [35] Xiao Li, Zekai Zhang, Xiang Li, Siyi Chen, Zhihui Zhu, Peng Wang, and Qing Qu. Understanding representation dynamics of diffusion models via low-dimensional modeling. arXiv preprint arXiv:2502.05743, 2025. 5 [36] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 4, 28 [37] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015. 7
11
[38] Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. 7, 33
Decoupled weight decay regularization.
arXiv preprint
[39] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022. 9 [40] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23–40. Springer, 2024. 9, 28, 34, 35 [41] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021. 7, 33 [42] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021. 8, 9, 33 [43] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023. 1, 7, 8, 9, 33, 34, 35 [44] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pages 5301–5310. PMLR, 2019. 2, 9 [45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 7, 8, 9, 33 [46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015. 7 [47] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 4, 8, 28 [48] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 7, 34 [49] Minglei Shi, Haolin Wang, Wenzhao Zheng, Ziyang Yuan, Xiaoshi Wu, Xintao Wang, Pengfei Wan, Jie Zhou, and Jiwen Lu. Latent diffusion model without variational autoencoder. arXiv preprint arXiv:2510.15301, 2025. 9 [50] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015. 1 [51] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 8 [52] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 1, 7, 28 [53] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016. 33 [54] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 7 [55] Kai Wang, Yukun Zhou, Mingjia Shi, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. A closer look at time steps is worthy of triple speed-up for diffusion model training. arXiv preprint arXiv:2405.17403, 2024. 1, 6, 9 [56] Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Jian Yang, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think. arXiv preprint arXiv:2507.01467, 2025. 9 [57] Zike Wu, Pan Zhou, Kenji Kawaguchi, and Hanwang Zhang. Fast diffusion model, 2023. 9
12
[58] Tianshuo Xu, Peng Mi, Ruilin Wang, and Yingcong Chen. Towards faster training of diffusion models: An inspiration of a consistency phenomenon. arXiv preprint arXiv:2404.07946, 2024. 9 [59] Jingfeng Yao, Wang Cheng, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffusion transformers training without architecture modification. arXiv preprint arXiv:2410.10356, 2024. 9 [60] Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15703–15712, 2025. 8, 9 [61] Hu Yu, Li Shen, Jie Huang, Man Zhou, Hongsheng Li, and Feng Zhao. Debias the training of diffusion models. arXiv preprint arXiv:2310.08442, 2023. 6, 9 [62] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024. 1, 8, 9 [63] Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025. 9 [64] Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09305, 2023. 9 [65] Tianyi Zheng, Cong Geng, Peng-Tao Jiang, Ben Wan, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Non-uniform timestep sampling: Towards faster diffusion model training. In ACM Multimedia 2024, 2024. 1, 6, 9 [66] Tianyi Zheng, Peng-Tao Jiang, Ben Wan, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Beta-tuned timestep diffusion model. In European Conference on Computer Vision, 2024. 1, 9 [67] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5826–5835, 2021. 1 [68] Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao, Zhenglong Sun, Tao Mei, and Chang Wen Chen. Sd-dit: Unleashing the power of self-supervised discrimination in diffusion transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8435–8445, 2024. 8, 9
13
Elucidating Representation Degradation Problem in Diffusion Model Training Appendix Contents A Foundations and Problem Formulation
14
B Neural Tangent Kernel Dynamics in Diffusion Models
15
C Evidence Lower Bound Dynamics
16
D Recoverability Mismatch and Representation Degradation
21
E Unified Effective-Amplitude-Aware Loss Weight
26
F Toy Experiments on 2D GMM Dataset
28
F.1
Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
F.2
Gradient Contamination and the Bayes Floor Gap
. . . . . . . . . . . . . . . . . . . . . . .
29
F.3
Representation Collapse in Feature Space. . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
F.4
Neural Tangent Kernel Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
G Experimental Supplement
A
32
G.1 Further Implementation Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
G.2 Evaluation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
G.3 Detailed Quantitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
G.4 Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
Foundations and Problem Formulation
In this section, we introduce the mathematical definitions and foundational assumptions used throughout the appendix for analyzing Neural Tangent Kernel dynamics and optimization behavior in diffusion models. We also organize the later results so that the subsequent conclusions are derived explicitly from the preceding ones, rather than appearing as isolated statements. Definition A.1 (Forward process and matrix-valued joint Neural Tangent Kernel). Let the data distribution be q(x0 ) on Rd . The forward diffusion process yields a family of latent variables xλ with conditional law q(xλ | x0 ) = N xλ ; α(λ)x0 , σ(λ)2 I , (12) where the log signal-to-noise ratio is defined by α(λ)2 λ = log . (13) σ(λ)2 We assume that the diffusion schedule is chosen so that the map t 7→ λt is monotone on the domain of interest. Hence t and λ are interchangeable parameterizations of the same forward corruption process, and we may equivalently write the model as fθ,m (x, t) or fθ,m (x, λ) with λ = λt . Let fθ,m (x, λ) : Rd × [λmin , λmax ] → Rd be a shared-parameter neural network estimator parameterized by θ ∈ RP , where m denotes the width scaling parameter. For vector-valued outputs, we adopt the operator-valued Neural Tangent Kernel formulation. The matrix-valued joint Neural Tangent Kernel on the input–noise space is defined by ⊤ Θθ,m (x, λ), (x′ , λ′ ) = ∇θ fθ,m (x, λ) ∇θ fθ,m (x′ , λ′ ) ∈ Rd×d , (14) 14
where ∇θ fθ,m (x, λ) ∈ Rd×P denotes the Jacobian of the network output with respect to the parameters. The kernel Θθ,m (x, λ), (x′ , λ′ ) acts as a linear operator on Rd and captures both coupling across output dimensions and coupling across noise levels induced by parameter sharing. As a local special case, the fixed-noise kernel is obtained by restricting the joint kernel to λ′ = λ, namely Θθ,m,λ (x, x′ ) ≜ Θθ,m (x, λ), (x′ , λ) . (15) Assumption A.2 (Optimization setup). To analyze the continuous-time training dynamics, we impose the following conditions: • Shared-parameter architecture: The same parameter vector θ is used across all noise levels, so the training dynamics are governed by the joint kernel Θθ,m (x, λ), (x′ , λ′ ) on the input–noise space. • Initialization and activation: The network is initialized with a standard width-scaled random initialization, and the activation function is assumed to be Lipschitz continuous and at least twice differentiable. • Continuous-time dynamics: The optimization is modeled in continuous training time τ by gradient flow, corresponding to the small-step-size limit of gradient descent. • Integrability and regularity: All expectations and conditional expectations appearing below are well-defined, the relevant second moments are finite, measurable versions of conditional expectations exist, and differentiation may be interchanged with expectation wherever used. • Nonnegative allocation weight: The effective allocation factor introduced later satisfies M (λ) ≥ 0 on the entire log-SNR interval.
B
Neural Tangent Kernel Dynamics in Diffusion Models
We first establish the shared-parameter finite-width training dynamics on the joint input–noise space. This result serves as the starting point for all later ELBO-level consequences. Let m denote the width parameter of the network, or more generally the common scaling parameter controlling the hidden-layer widths in the architecture, and write the model as fθ,m (x, λ). For each fixed finite m, consider the joint regression objective h i 1 2 Lm (θ) = Eλ∼ρ, x∼qλ ∥fθ,m (x, λ) − y(x, λ)∥2 , (16) 2 where ρ(λ) denotes the sampling distribution over noise levels and y(x, λ) ∈ Rd is the training target. For a gradient-flow trajectory θm (τ ), define fτ,m (x, λ) ≜ fθm (τ ),m (x, λ) and eτ,m (x, λ) ≜ fτ,m (x, λ) − y(x, λ), and write the time-dependent joint matrix-valued Neural Tangent Kernel as ⊤ Θτ,m (x, λ), (x′ , λ′ ) ≜ ∇θ fτ,m (x, λ) ∇θ fτ,m (x′ , λ′ ) ∈ Rd×d . (17) Proposition B.1 (Finite-width training dynamics with shared parameters). Under Theorem A.2, for each finite width parameter m, if θm (τ ) evolves by gradient flow for the objective Eq. (16), then the residual field satisfies ∂ eτ,m (x, λ) = −Eλ′ ∼ρ, x′ ∼qλ′ Θτ,m (x, λ), (x′ , λ′ ) eτ,m (x′ , λ′ ) . ∂τ
(18)
Proof. Under the regularity conditions in Theorem A.2, differentiation may be interchanged with expectation. Differentiating Eq. (16) with respect to θ gives h i 1 2 ∇θ Lm (θ) = ∇θ Eλ∼ρ, x∼qλ ∥fθ,m (x, λ) − y(x, λ)∥2 2 1 2 = Eλ∼ρ, x∼qλ ∇θ ∥fθ,m (x, λ) − y(x, λ)∥2 2 h ⊤ i = Eλ∼ρ, x∼qλ ∇θ fθ,m (x, λ) fθ,m (x, λ) − y(x, λ) . (19) 15
Using gradient flow, we obtain dθm (τ ) = −∇θ Lm (θm (τ )) dτ h
⊤
i fτ,m (x′ , λ′ ) − y(x′ , λ′ ) h i ⊤ = −Eλ′ ∼ρ, x′ ∼qλ′ ∇θ fτ,m (x′ , λ′ ) eτ,m (x′ , λ′ ) . = −Eλ′ ∼ρ, x′ ∼qλ′
∇θ fτ,m (x′ , λ′ )
(20)
Fixing (x, λ) and applying the chain rule along the trajectory θm (τ ), we have dθm (τ ) ∂ fτ,m (x, λ) = ∇θ fτ,m (x, λ) ∂τ dτ h i ⊤ = −∇θ fτ,m (x, λ)Eλ′ ∼ρ, x′ ∼qλ′ ∇θ fτ,m (x′ , λ′ ) eτ,m (x′ , λ′ ) i h ⊤ = −Eλ′ ∼ρ, x′ ∼qλ′ ∇θ fτ,m (x, λ) ∇θ fτ,m (x′ , λ′ ) eτ,m (x′ , λ′ ) = −Eλ′ ∼ρ, x′ ∼qλ′ Θτ,m (x, λ), (x′ , λ′ ) eτ,m (x′ , λ′ ) .
(21)
Since eτ,m (x, λ) = fτ,m (x, λ) − y(x, λ) and y(x, λ) is independent of τ , it follows that ∂ ∂ eτ,m (x, λ) = fτ,m (x, λ) ∂τ ∂τ = −Eλ′ ∼ρ, x′ ∼qλ′ Θτ,m (x, λ), (x′ , λ′ ) eτ,m (x′ , λ′ ) ,
■
which is exactly Eq. (18).
C
(22)
Evidence Lower Bound Dynamics
We now lift the joint Neural Tangent Kernel dynamics in Theorem B.1 to the global ELBO objective. The key organizational principle of this section is as follows. First, we decompose the local ELBO density into an irreducible Bayes floor and an optimizable excess term. Second, we derive an exact continuous-time excess-dynamics identity relative to a formal sample-path surrogate of stochastic gradient descent. Third, all later quantitative bounds in this section are derived explicitly from that excess-dynamics identity. We work in the log-SNR domain λ ∈ [λmin , λmax ]. For a finite-width network fθ,m (x, λ) with width scaling parameter m, let the forward-corrupted input and the generalized diffusion target be parameterized by xλ = α(λ)x0 + σ(λ)ϵ and yλ = cx (λ)x0 + cϵ (λ)ϵ, respectively, where x0 ∼ q(x0 ) and ϵ ∼ N (0, I) are independent. This parameterization subsumes standard diffusion targets such as noise prediction and data prediction. We express the continuous-time ELBO objective as Z λmax h i 1 2 LELBO,m (θ) = M (λ) Ex0 ∼q, ϵ∼N (0,I) ∥fθ,m (xλ , λ) − yλ ∥2 dλ, (23) 2 λmin dt where M (λ) ≜ w(λ)p(λ) dλ denotes the effective allocation measure induced by the loss weighting, sampling distribution, and schedule Jacobian. By assumption, M (λ) ≥ 0 on the interval of interest.
Recall the trajectory output fτ,m defined above. Under the diffusion parameterization, the instantaneous prediction residual is eτ,m (xλ , λ) ≜ fτ,m (xλ , λ) − yλ . The corresponding local weighted ELBO density is defined by 1 ℓτ,m (λ) ≜ M (λ)Ex0 ∼q, ϵ∼N (0,I) ∥eτ,m (xλ , λ)∥22 , (24) 2 so that Z λmax LELBO,m (θm (τ )) = ℓτ,m (λ) dλ. (25) λmin
For each fixed log-SNR level λ, define the Bayes-optimal predictor by fλ⋆ (x) ≜ E[yλ | xλ = x]. Correspondingly, define the parameter-indexed Bayes-centered residual and the trajectory-indexed 16
Bayes-centered residual by rθ,m (x, λ) ≜ fθ,m (x, λ) − fλ⋆ (x) and rτ,m (x, λ) ≜ fτ,m (x, λ) − fλ⋆ (x), the local weighted excess ELBO density by ℓ̄τ,m (λ) ≜
1 M (λ)∥rτ,m (·, λ)∥2L2 (qλ ) , 2
(26)
and the local Bayes floor by ℓ⋆λ ≜
1 M (λ)Ex0 ∼q, ϵ∼N (0,I) ∥fλ⋆ (xλ ) − yλ ∥22 . 2
(27)
The next lemma is the basic decomposition from which all later local ELBO statements will be derived. Lemma C.1 (Bayes decomposition of the local ELBO density). For each fixed log-SNR level λ, the local weighted ELBO density admits the exact orthogonal decomposition ℓτ,m (λ) = ℓ⋆λ + ℓ̄τ,m (λ).
(28)
ℓτ,m (λ) ≥ ℓ⋆λ .
(29)
In particular,
Proof. For each fixed λ, decompose the prediction residual around the Bayes predictor: eτ,m (xλ , λ) = fτ,m (xλ , λ) − yλ = fτ,m (xλ , λ) − fλ⋆ (xλ ) + fλ⋆ (xλ ) − yλ = fτ,m (xλ , λ) − fλ⋆ (xλ ) + fλ⋆ (xλ ) − yλ . {z } | {z } | rτ,m (xλ ,λ)
(30)
ξλ
Substituting Eq. (30) into Eq. (24) yields 1 M (λ)E ∥rτ,m (xλ , λ) + ξλ ∥22 2 1 = M (λ)E ∥rτ,m (xλ , λ)∥22 + ∥ξλ ∥22 + 2⟨rτ,m (xλ , λ), ξλ ⟩ 2 1 1 = M (λ)E∥rτ,m (xλ , λ)∥22 + M (λ)E∥ξλ ∥22 + M (λ)E⟨rτ,m (xλ , λ), ξλ ⟩. 2 2
ℓτ,m (λ) =
(31)
It remains to show that the cross-term vanishes. Since rτ,m (xλ , λ) is a measurable function of xλ alone and E[ξλ | xλ ] = E[fλ⋆ (xλ ) − yλ | xλ ] = fλ⋆ (xλ ) − E[yλ | xλ ] = 0,
(32)
the law of total expectation gives E⟨rτ,m (xλ , λ), ξλ ⟩ = E[⟨rτ,m (xλ , λ), E[ξλ | xλ ]⟩] = 0.
(33)
Therefore 1 1 M (λ)E∥fλ⋆ (xλ ) − yλ ∥22 + M (λ)∥fτ,m (·, λ) − fλ⋆ (·)∥2L2 (qλ ) 2 2 = ℓ⋆λ + ℓ̄τ,m (λ),
ℓτ,m (λ) =
which proves Eq. (28). Since ℓ̄τ,m (λ) ≥ 0, the lower bound Eq. (29) follows immediately.
(34) ■
The preceding decomposition is a population identity. We next isolate the corresponding decomposition at the level of the sampled stochastic gradient, which will later be inserted into the continuous-time surrogate dynamics. 17
Lemma C.2 (Bayes decomposition of the stochastic ELBO gradient). For a sampled triple (x0 , ϵ, λ), define the stochastic local ELBO loss by 1 ℓbm (θ; x0 , ϵ, λ) ≜ M (λ)∥fθ,m (xλ , λ) − yλ ∥22 . (35) 2 Then the corresponding stochastic gradient admits the exact decomposition ⊤ ∇θ ℓbm (θ; x0 , ϵ, λ) = M (λ) ∇θ fθ,m (xλ , λ) fθ,m (xλ , λ) − fλ⋆ (xλ ) ⊤ ⋆ + M (λ) ∇θ fθ,m (xλ , λ) fλ (xλ ) − yλ . (36) Proof. Differentiate Eq. (35) with respect to θ: 1 2 b M (λ)∥fθ,m (xλ , λ) − yλ ∥2 ∇θ ℓm (θ; x0 , ϵ, λ) = ∇θ 2 ⊤ = M (λ) ∇θ fθ,m (xλ , λ) fθ,m (xλ , λ) − yλ . Now decompose the instantaneous prediction residual around the Bayes predictor: fθ,m (xλ , λ) − yλ = fθ,m (xλ , λ) − fλ⋆ (xλ ) + fλ⋆ (xλ ) − yλ . Substituting Eq. (38) into Eq. (37) yields ⊤ ∇θ ℓbm (θ; x0 , ϵ, λ) = M (λ) ∇θ fθ,m (xλ , λ) fθ,m (xλ , λ) − fλ⋆ (xλ ) ⊤ ⋆ + M (λ) ∇θ fθ,m (xλ , λ) fλ (xλ ) − yλ , which is exactly Eq. (36).
(37) (38)
(39) ■
The next step introduces the continuous-time surrogate that connects the stochastic gradient decomposition above to a function-space Neural Tangent Kernel dynamics. The following statement should be interpreted as a formal sample-path interpolation of stochastic gradient descent, rather than as a rigorous stochastic-approximation limit theorem. Lemma C.3 (Formal continuous-time sample-path surrogate for SGD). Consider the stochastic gradient descent update (k) θk+1 = θk − η ∇θ ℓbm (θk ; x , ϵ(k) , λ(k) ). (40) (k)
0 (k) (k) (k) )x0 + σ(λ )ϵ and ξ (k) = fλ⋆(k) (xλ(k) ) − yλ(k) . Let τ = kη
At iteration k, let xλ(k) = α(λ and consider the piecewise-linear interpolation of the iterates. In the small-step-size regime, we adopt the formal sample-path surrogate ⊤ dθm (τ ) = −M (λτ ) ∇θ fτ,m (xλτ , λτ ) fτ,m (xλτ , λτ ) − fλ⋆τ (xλτ ) dτ ⊤ − M (λτ ) ∇θ fτ,m (xλτ , λτ ) ξτ , (41) where ξτ ≜ fλ⋆τ (xλτ ) − yλτ . All subsequent identities in this subsection are exact relative to the surrogate dynamics Eq. (41). Proof. Starting from Eq. (40), divide by η to obtain the difference quotient θk+1 − θk (k) = −∇θ ℓbm (θk ; x0 , ϵ(k) , λ(k) ). η (k)
(42)
Applying Eq. (36) at the iterate θk and the sampled triple (x0 , ϵ(k) , λ(k) ) gives ⊤ θk+1 − θk = −M (λ(k) ) ∇θ fθk ,m (xλ(k) , λ(k) ) fθk ,m (xλ(k) , λ(k) ) − fλ⋆(k) (xλ(k) ) η ⊤ − M (λ(k) ) ∇θ fθk ,m (xλ(k) , λ(k) ) ξ (k) . (43) Introducing τ = kη and replacing the sampled sequence by a continuously refreshed signal (x0,τ , ϵτ , λτ ) yields the surrogate pathwise relation ⊤ dθm (τ ) = −M (λτ ) ∇θ fτ,m (xλτ , λτ ) fτ,m (xλτ , λτ ) − fλ⋆τ (xλτ ) dτ ⊤ − M (λτ ) ∇θ fτ,m (xλτ , λτ ) ξτ , (44) which is exactly Eq. (41). ■ 18
The following theorem is the central result of this section. All subsequent upper bounds and degradation estimates will be obtained by explicitly invoking it. Theorem C.4 (Exact local excess dynamics under the sample-path surrogate). Under the preceding definitions and relative to the surrogate dynamics Eq. (41), the Bayes-centered residual satisfies ∂ rτ,m (x, λ) = −M (λτ )Θτ,m (x, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) ∂τ − M (λτ )Θτ,m (x, λ), (xλτ , λτ ) ξτ .
(45)
Consequently, the local excess density satisfies the exact differential identity D E d ℓ̄τ,m (λ) = −M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 dτ L (qλ ) D E − M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ . (46) L2 (qλ )
Proof. Fix an evaluation pair (x, λ). By the chain rule along the trajectory θm (τ ), dθm (τ ) ∂ fτ,m (x, λ) = ∇θ fτ,m (x, λ) . ∂τ dτ Substituting Eq. (41) into Eq. (47), we obtain
(47)
⊤ ∂ fτ,m (x, λ) = −∇θ fτ,m (x, λ)M (λτ ) ∇θ fτ,m (xλτ , λτ ) rτ,m (xλτ , λτ ) ∂τ ⊤ − ∇θ fτ,m (x, λ)M (λτ ) ∇θ fτ,m (xλτ , λτ ) ξτ = −M (λτ )Θτ,m (x, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) − M (λτ )Θτ,m (x, λ), (xλτ , λτ ) ξτ .
(48)
Since fλ⋆ (x) is independent of τ , differentiating rτ,m (x, λ) = fτ,m (x, λ) − fλ⋆ (x) gives ∂ ∂ rτ,m (x, λ) = fτ,m (x, λ), ∂τ ∂τ and substituting Eq. (48) proves Eq. (45).
(49)
Next differentiate Eq. (26) with respect to τ : d d 1 2 ℓ̄τ,m (λ) = M (λ)∥rτ,m (·, λ)∥L2 (qλ ) dτ dτ 2 E D ∂ rτ,m (·, λ) . = M (λ) rτ,m (·, λ), ∂τ L2 (qλ )
(50)
Substituting Eq. (45) into Eq. (50) yields D E d ℓ̄τ,m (λ) = −M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 dτ L (qλ ) D E − M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ , (51) L2 (qλ )
■
which is exactly Eq. (46).
The next result is the first quantitative consequence of Theorem C.4. It controls the local excess dynamics by separating the residual-coupling term from the Bayes-forcing term. Proposition C.5 (Quantitative upper bound for local ELBO excess dynamics). For every β > 0, the local excess density satisfies D E d ℓ̄τ,m (λ) ≤ −M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 dτ L (qλ ) 2 M (λ) + β ℓ̄τ,m (λ) + M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ 2 . (52) 2β L (qλ ) 19
Moreover, if the trajectory remains in the Neural Tangent Kernel regime so that Θτ,m (x, λ), (x′ , λ′ ) ≈ Θ0,m (x, λ), (x′ , λ′ ) ,
(53)
then the corresponding frozen-kernel approximation satisfies D E d ℓ̄τ,m (λ) ≲ −M (λ) rτ,m (·, λ), M (λτ )Θ0,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) dτ L2 (qλ ) 2 M (λ) . (54) M (λτ )Θ0,m (·, λ), (xλτ , λτ ) ξτ + β ℓ̄τ,m (λ) + 2β L2 (qλ ) Proof. Starting from the exact identity Eq. (46) in Theorem C.4, apply Cauchy–Schwarz to the forcing term: D E rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ 2 L (qλ )
≤ ∥rτ,m (·, λ)∥L2 (qλ ) M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ
L2 (qλ )
.
(55)
1 2 Applying Young’s inequality ab ≤ β2 a2 + 2β b yields D E rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ
L2 (qλ )
≤
2 β 1 ∥rτ,m (·, λ)∥2L2 (qλ ) + M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ . 2 2β L2 (qλ )
(56)
Using the identity ∥rτ,m (·, λ)∥2L2 (qλ ) = 2M (λ)−1 ℓ̄τ,m (λ),
(57)
we obtain from Eq. (46) D E d ℓ̄τ,m (λ) ≤ −M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) dτ L2 (qλ ) 2 βM (λ) M (λ) + ∥rτ,m (·, λ)∥2L2 (qλ ) + M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ 2 2 2β L (qλ ) D E = −M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 L (qλ )
2 M (λ) M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ 2 , + β ℓ̄τ,m (λ) + 2β L (qλ )
(58)
which proves Eq. (52). Under the frozen-kernel approximation Eq. (53), replacing Θτ,m by Θ0,m yields Eq. (54). ■ The following pathwise coercivity condition is the additional assumption that allows the preceding differential inequality to be integrated explicitly. It should be interpreted as a sufficient sample-path lower bound on the residual-coupling term and is not implied solely by positive semidefiniteness of the population Neural Tangent Kernel operator. Theorem C.6 (Non-asymptotic integral bound for the local ELBO excess). Fix λ ∈ [λmin , λmax ] and β > 0. Suppose that along the trajectory there exists a measurable function µτ (λ) such that D E M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 ≥ µτ (λ) ℓ̄τ,m (λ) (59) L (qλ )
for all τ ≥ 0. Then for every τ ≥ 0, Z τ ℓ̄τ,m (λ) ≤ exp − µs (λ) − β ds ℓ̄0,m (λ) 0 Z τ Z τ 2 M (λ) + exp − µu (λ) − β du M (λs )Θs,m (·, λ), (xλs , λs ) ξs 2 ds. 2β L (qλ ) 0 s (60) 20
In particular, if µτ (λ) ≥ µ(λ) > β for all τ ≥ 0, then ℓ̄τ,m (λ) ≤ e−(µ(λ)−β)τ ℓ̄0,m (λ) Z τ 2 M (λ) ds. M (λs )Θs,m (·, λ), (xλs , λs ) ξs + e−(µ(λ)−β)(τ −s) 2β L2 (qλ ) 0
(61)
Consequently, lim sup ℓ̄τ,m (λ) ≤ lim sup τ →∞
τ →∞
Z τ
e−(µ(λ)−β)(τ −s)
0
2 M (λ) ds. M (λs )Θs,m (·, λ), (xλs , λs ) ξs 2β L2 (qλ ) (62)
Proof. Under the coercivity condition Eq. (59), the differential inequality in Eq. (52) from Theorem C.5 yields 2 M (λ) d . (63) M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ ℓ̄τ,m (λ) ≤ − µτ (λ) − β ℓ̄τ,m (λ) + dτ 2β L2 (qλ ) Rτ Define the integrating-factor phase Φ(τ ) ≜ 0 µs (λ) − β ds. Multiplying Eq. (63) by eΦ(τ ) gives 2 d Φ(τ ) M (λ) M (λτ )Θτ,m (·, λ), (xλτ , λτ ) ξτ 2 e ℓ̄τ,m (λ) ≤ eΦ(τ ) . (64) dτ 2β L (qλ ) Integrating from 0 to τ yields eΦ(τ ) ℓ̄τ,m (λ) − ℓ̄0,m (λ) ≤
Z τ 0
eΦ(s)
2 M (λ) M (λs )Θs,m (·, λ), (xλs , λs ) ξs ds. 2β L2 (qλ )
(65)
Multiplying both sides by e−Φ(τ ) and using Z τ e−Φ(τ ) eΦ(s) = exp − µu (λ) − β du ,
(66)
s
we obtain Z τ ℓ̄τ,m (λ) ≤ exp − µs (λ) − β ds ℓ̄0,m (λ) 0 Z τ Z τ 2 M (λ) M (λs )Θs,m (·, λ), (xλs , λs ) ξs 2 ds, + exp − µu (λ) − β du 2β L (qλ ) 0 s (67) which proves Eq. (60). The bound Eq. (61) follows immediately from the lower bound µτ (λ) ≥ µ(λ) > β. Finally, taking lim supτ →∞ in Eq. (61) yields Eq. (62). ■
D
Recoverability Mismatch and Representation Degradation
We now combine the preceding ELBO-level consequences with the degeneration of the corrupted input distribution in the extreme-noise regime. The goal of this section is to make explicit that the degradation theorem below is not an isolated statement: it is derived by combining the Bayes decomposition in Theorem C.1, the non-asymptotic excess bound in Theorem C.6, and the input degeneration phenomenon established next. To analyze the asymptotic extreme-noise regime, extend the schedule beyond the finite training interval and consider the idealized limit λ → −∞. Assume further that the forward schedule admits a finite terminal noise level σmax in this limit. Lemma D.1 (Input distribution degeneration at the high-noise limit). Assume in addition that the data distribution has finite second moment Ex0 ∼q ∥x0 ∥22 < ∞. Let the input to the network be xλ = α(λ)x0 + σ(λ)ϵ, where x0 ∼ q(x0 ) and ϵ ∼ N (0, I). As the forward process approaches the extreme-noise regime and λ → −∞, the marginal probability measure of the input qλ (xλ ) converges 2 weakly to the pure isotropic Gaussian distribution N (0, σmax I). More precisely, the 2-Wasserstein distance satisfies 2 W2 qλ , N (0, σmax I) → 0 as λ → −∞. (68) 21
Proof. By the definition of the 2-Wasserstein metric W2 , we upper bound the transport distance by choosing the natural joint coupling induced by the shared noise variable ϵ. Under this coupling, we obtain Z Z 2 2 2 W2 qλ , N (0, σmax I) ≤ ∥(α(λ)x0 + σ(λ)ϵ) − σmax ϵ∥2 q(x0 )p(ϵ) dx0 dϵ Rd Rd h i 2 = Ex0 ,ϵ ∥α(λ)x0 + (σ(λ) − σmax )ϵ∥2 = Ex0 ,ϵ α(λ)2 ∥x0 ∥22 + 2α(λ)(σ(λ) − σmax )⟨x0 , ϵ⟩ + (σ(λ) − σmax )2 ∥ϵ∥22 = α(λ)2 Ex0 ∥x0 ∥22 + 2α(λ)(σ(λ) − σmax )E[⟨x0 , ϵ⟩] + (σ(λ) − σmax )2 Eϵ ∥ϵ∥22 = α(λ)2 Ex0 ∥x0 ∥22 + (σ(λ) − σmax )2 d, (69)
where E[⟨x0 , ϵ⟩] = 0 by independence and E[∥ϵ∥22 ] = d for ϵ ∼ N (0, I). Since the extended forward schedule satisfies limλ→−∞ α(λ) = 0 and limλ→−∞ σ(λ) = σmax , the right-hand side of Eq. (69) converges to 0. Hence 2 lim W22 qλ , N (0, σmax I) = 0, (70) λ→−∞
2 which implies Eq. (68). Therefore qλ converges weakly to N (0, σmax I) in the limit.
■
We now combine the Bayes decomposition in Theorem C.1, the excess bound in Theorem C.6, and the input degeneration phenomenon in Theorem D.1. This yields a quantitative characterization of degradation under recoverability mismatch. Theorem D.2 (Quantitative degradation under recoverability mismatch). Fix λ ∈ [λmin , λmax ] and β > 0. Suppose that along the trajectory there exists a measurable function µτ (λ) such that D E M (λ) rτ,m (·, λ), M (λτ )Θτ,m (·, λ), (xλτ , λτ ) rτ,m (xλτ , λτ ) 2 ≥ µτ (λ) ℓ̄τ,m (λ) (71) L (qλ )
for all τ ≥ 0. Then the local ELBO density admits the bound Z τ ⋆ ℓτ,m (λ) ≤ ℓλ + exp − µs (λ) − β ds ℓ̄0,m (λ) 0 Z τ Z τ 2 M (λ) µu (λ) − β du + exp − M (λs )Θs,m (·, λ), (xλs , λs ) ξs 2 ds. 2β L (qλ ) 0 s (72) ⋆ In particular, in the high-noise regime where the Bayes floor ℓλ increases while the effective contraction µτ (λ) may decrease, the local ELBO is dominated by an enlarged irreducible component together with a weaker contraction of the optimizable excess. Moreover, the forcing term shows that this degradation is mediated by cross-noise kernel interactions induced by shared parameters, and hence the high-noise regime is not an autonomous single-noise subsystem. Proof. By the Bayes decomposition established in Theorem C.1, we have ℓτ,m (λ) = ℓ⋆λ + ℓ̄τ,m (λ). (73) By the non-asymptotic integral bound in Theorem C.6, which is itself derived from the exact excess dynamics in Theorem C.4, the coercivity assumption Eq. (71) yields Z τ ℓ̄τ,m (λ) ≤ exp − µs (λ) − β ds ℓ̄0,m (λ) 0 Z τ Z τ 2 M (λ) + exp − µu (λ) − β du M (λs )Θs,m (·, λ), (xλs , λs ) ξs 2 ds. 2β L (qλ ) 0 s (74) Substituting Eq. (74) into Eq. (73), we obtain ℓτ,m (λ) = ℓ⋆λ + ℓ̄τ,m (λ) Z τ ⋆ ≤ ℓλ + exp − µs (λ) − β ds ℓ̄0,m (λ) 0 Z τ Z τ 2 M (λ) + exp − µu (λ) − β du M (λs )Θs,m (·, λ), (xλs , λs ) ξs 2 ds, 2β L (qλ ) 0 s (75) 22
which is exactly Eq. (72). It remains to explain why this bound captures recoverability mismatch. By Theorem D.1, in the idealized extreme-noise regime the input distribution approaches a pure isotropic Gaussian law. Therefore the corrupted input retains progressively less data-dependent structure, so the recoverable component of the target becomes weaker. When interpreted together with the Bayes decomposition Eq. (28), this means that the irreducible Bayes floor ℓ⋆λ becomes comparatively more dominant as the noise level increases. At the same time, the non-asymptotic excess bound shows that the optimizable excess is controlled by a competition between contraction through µτ (λ) and forcing through the cross-noise kernel term. Hence the high-noise regime is affected simultaneously by a larger irreducible component, a weaker effective contraction of the optimizable excess, and continued cross-noise forcing through shared parameters. This proves the claimed quantitative degradation statement. ■ The preceding theorem states the degradation mechanism at the loss level. We next show that the same Bayes-forcing mechanism appears directly in the gradient acting on learned representations. Proposition D.3 (Quantitative Bayes-noise contamination of representation gradients). Suppose the network admits a representation-readout factorization fθ,m (x, λ) = Wθ,m hθ,m (x, λ), m
(76)
d×m
where hθ,m (x, λ) ∈ R is the representation and Wθ,m ∈ R is the readout. Let ϑ ⊆ θ denote parameters that affect hθ,m but leave Wθ,m fixed. Then for a sampled triple (x0 , ϵ, λ), ⊤ ⊤ ∇ϑ ℓbm (θ; x0 , ϵ, λ) = M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m rθ,m (xλ , λ) ⊤ ⊤ + M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m ξλ . (77) Consequently, the gradient norm satisfies ∇ϑ ℓbm (θ; x0 , ϵ, λ) 2 ≤ M (λ) ∇ϑ hθ,m (xλ , λ) F ∥Wθ,m ∥op ∥rθ,m (xλ , λ)∥2 + ∥ξλ ∥2 .
(78)
Therefore, in regimes where the recoverable residual signal diminishes while the irreducible Bayes fluctuation remains non-negligible, the representation gradient becomes increasingly dominated by Bayes-noise contamination. Proof. For the sampled local loss ℓbm (θ; x0 , ϵ, λ) ≜ 12 M (λ)∥fθ,m (xλ , λ) − yλ ∥22 , differentiate with respect to ϑ. Since M (λ) is independent of θ, we obtain ⊤ ∇ϑ ℓbm (θ; x0 , ϵ, λ) = M (λ) ∇ϑ fθ,m (xλ , λ) fθ,m (xλ , λ) − yλ . (79) Because ϑ affects hθ,m but leaves Wθ,m fixed, the chain rule gives ∇ϑ fθ,m (xλ , λ) = ∇ϑ Wθ,m hθ,m (xλ , λ) = Wθ,m ∇ϑ hθ,m (xλ , λ).
(80)
Substituting Eq. (80) into Eq. (79), we obtain ⊤ ⊤ ∇ϑ ℓbm (θ; x0 , ϵ, λ) = M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m fθ,m (xλ , λ) − yλ .
(81)
Now decompose the instantaneous prediction residual around the Bayes predictor: fθ,m (xλ , λ) − yλ = fθ,m (xλ , λ) − fλ⋆ (xλ ) + fλ⋆ (xλ ) − yλ = rθ,m (xλ , λ) + ξλ .
(82)
Substituting Eq. (82) into Eq. (81) yields ⊤ ⊤ ∇ϑ ℓbm (θ; x0 , ϵ, λ) = M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m rθ,m (xλ , λ) + ξλ ⊤ ⊤ = M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m rθ,m (xλ , λ) ⊤ ⊤ + M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m ξλ , which proves Eq. (77). 23
(83)
To prove the norm bound, apply the triangle inequality to Eq. (77): ⊤ ⊤ ∇ϑ ℓbm (θ; x0 , ϵ, λ) 2 ≤ M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m rθ,m (xλ , λ) 2 ⊤ ⊤ + M (λ) ∇ϑ hθ,m (xλ , λ) Wθ,m ξλ . 2
(84)
Using the operator bound ⊤ ⊤ ∇ϑ hθ,m (xλ , λ) Wθ,m v
2
≤ ∇ϑ hθ,m (xλ , λ) F ∥Wθ,m ∥op ∥v∥2
(85)
for any v ∈ Rd , and applying Eq. (85) first with v = rθ,m (xλ , λ) and then with v = ξλ , we obtain ∇ϑ ℓbm (θ; x0 , ϵ, λ) 2 ≤ M (λ) ∇ϑ hθ,m (xλ , λ) F ∥Wθ,m ∥op ∥rθ,m (xλ , λ)∥2
+ M (λ) ∇ϑ hθ,m (xλ , λ) F ∥Wθ,m ∥op ∥ξλ ∥2 = M (λ) ∇ϑ hθ,m (xλ , λ) F ∥Wθ,m ∥op ∥rθ,m (xλ , λ)∥2 + ∥ξλ ∥2 ,
(86)
which is exactly Eq. (78). The final interpretation follows directly from the decomposition together with the degradation picture in Theorem D.2: when the recoverable component becomes small relative to the irreducible Bayes fluctuation, the representation gradient is increasingly dominated by the Bayes-noise term. ■ Together, theorems D.2 and D.3 establish a unified quantitative picture: input degeneration reduces recoverable signal, the ELBO develops a non-uniform irreducible floor with weakened contraction, and the same Bayes-induced noise propagates through the shared-parameter architecture to directly contaminate representation gradients. Finally, under an additional fixed-noise surrogate closure, the preceding local excess mechanism admits a modal refinement in the eigenbasis of the frozen fixed-noise kernel operator. This last theorem should be read as a local spectral refinement of the earlier degradation mechanism, not as a direct spectral decomposition of the full joint-noise dynamics. Theorem D.4 (Spectral local ELBO bound under a fixed-noise surrogate). Fix λ ∈ [λmin , λmax ]. Assume that the frozen-kernel approximation Eq. (53) holds on the time interval of interest and that, for the purpose of local spectral analysis at the fixed noise level λ, the Bayes-centered residual satisfies Z ∂ rτ,m (·, λ) = −M (λ) Θ0,m (·, λ), (x′ , λ) rτ,m (x′ , λ) qλ (x′ ) dx′ ∂τ − M (λτ )Θ0,m (·, λ), (xλτ , λτ ) ξτ . (87) Assume moreover that −M (λτ )Θ0,m (·, λ), (xλτ , λτ ) ξτ ∈ L2 (qλ ; Rd )
(88)
for all τ ≥ 0. Assume further that there exists a complete orthonormal system {ϕλj }j≥1 ⊂ L2 (qλ ; Rd ) with nonnegative eigenvalues {κλj }j≥1 such that Z Θ0,m (x, λ), (x′ , λ) ϕλj (x′ ) qλ (x′ ) dx′ = κλj ϕλj (x), j ≥ 1. (89) Write, in L2 (qλ ; Rd ), rτ,m (x, λ) =
X
aλτ,j ϕλj (x),
aλτ,j = ⟨rτ,m (·, λ), ϕλj ⟩L2 (qλ ) ,
j≥1
(90)
and define λ ητ,j =
D
E − M (λτ )Θ0,m (·, λ), (xλτ , λτ ) ξτ , ϕλj 2
L (qλ )
24
.
(91)
Then for every γ > 0, M (λ) X −(2M (λ)κλj −γ)τ λ 2 e |a0,j | 2 j≥1 Z M (λ) X τ −(2M (λ)κλj −γ)(τ −s) λ 2 + e |ηs,j | ds. 2γ 0
ℓτ,m (λ) ≤ ℓ⋆λ +
(92)
j≥1
In particular, for modes satisfying 2M (λ)κλj > γ, the corresponding contribution decays exponentially, whereas modes associated with smaller eigenvalues are more weakly contracted and hence more susceptible to persistent Bayes forcing. Proof. We begin from the fixed-noise surrogate dynamics Eq. (87). By the definition of the modal coefficients in Eq. (90), we have aλτ,j = ⟨rτ,m (·, λ), ϕλj ⟩L2 (qλ ) .
(93)
Differentiating Eq. (93) with respect to τ yields D∂ E d λ aτ,j = rτ,m (·, λ), ϕλj . dτ ∂τ L2 (qλ ) Substituting Eq. (87) into Eq. (94), we obtain Z D E d λ aτ,j = − M (λ) Θ0,m (·, λ), (x′ , λ) rτ,m (x′ , λ) qλ (x′ ) dx′ , ϕλj 2 dτ L (qλ ) D E λ + − M (λτ )Θ0,m (·, λ), (xλτ , λτ ) ξτ , ϕj L2 (qλ ) DZ E λ = −M (λ) Θ0,m (·, λ), (x′ , λ) rτ,m (x′ , λ) qλ (x′ ) dx′ , ϕλj + ητ,j . L2 (qλ )
(94)
(95)
Using the residual expansion Eq. (90), we compute Z Z X λ λ ′ Θ0,m (x, λ), (x′ , λ) rτ,m (x′ , λ) qλ (x′ ) dx′ = Θ0,m (x, λ), (x′ , λ) aτ,k ϕk (x ) qλ (x′ ) dx′ k≥1
=
X
aλτ,k
Z
Θ0,m (x, λ), (x′ , λ) ϕλk (x′ ) qλ (x′ ) dx′
k≥1
=
X
aλτ,k κλk ϕλk (x),
(96)
k≥1
where the last line follows from Eq. (89). Taking the L2 (qλ ) inner product of Eq. (96) with ϕλj and using orthonormality, we get DZ E DX E λ λ λ λ Θ0,m (·, λ), (x′ , λ) rτ,m (x′ , λ) qλ (x′ ) dx′ , ϕλj = a κ ϕ , ϕ τ,k k k j 2 2 L (qλ )
k≥1
=
X k≥1
L (qλ )
aλτ,k κλk ⟨ϕλk , ϕλj ⟩L2 (qλ )
= aλτ,j κλj . Substituting Eq. (97) into Eq. (95), we conclude that d λ λ a = −M (λ)κλj aλτ,j + ητ,j . dτ τ,j
(97)
(98)
We next derive a mode-wise energy bound. Differentiating |aλτ,j |2 and applying Eq. (98), we obtain d λ 2 d |aτ,j | = 2aλτ,j aλτ,j dτ dτ
λ = 2aλτ,j − M (λ)κλj aλτ,j + ητ,j
λ = −2M (λ)κλj |aλτ,j |2 + 2aλτ,j ητ,j .
25
(99)
Applying Young’s inequality with parameter γ > 0, we have λ λ 2 2|aλτ,j ητ,j | ≤ γ|aλτ,j |2 + γ −1 |ητ,j | .
(100)
Combining Eq. (99) and Eq. (100) gives d λ 2 λ 2 |aτ,j | ≤ − 2M (λ)κλj − γ |aλτ,j |2 + γ −1 |ητ,j | . dτ
(101)
Applying the integrating-factor argument to Eq. (101), we obtain Z τ λ λ λ 2 |aλτ,j |2 ≤ e−(2M (λ)κj −γ)τ |aλ0,j |2 + e−(2M (λ)κj −γ)(τ −s) γ −1 |ηs,j | ds.
(102)
0
Since {ϕλj }j≥1 is a complete orthonormal system and the expansion Eq. (90) holds in L2 (qλ ; Rd ), Parseval’s identity gives X ∥rτ,m (·, λ)∥2L2 (qλ ) = |aλτ,j |2 . (103) j≥1
Recalling that ℓ̄τ,m (λ) =
1 M (λ)∥rτ,m (·, λ)∥2L2 (qλ ) , 2
(104)
we obtain from Eq. (102) and Eq. (103) that ℓ̄τ,m (λ) =
M (λ) X λ 2 |aτ,j | 2 j≥1
M (λ) X −(2M (λ)κλj −γ)τ λ 2 M (λ) X e |a0,j | + ≤ 2 2γ j≥1
Z τ
j≥1
0
λ
λ 2 e−(2M (λ)κj −γ)(τ −s) |ηs,j | ds.
(105) Finally, by the Bayes decomposition in Theorem C.1, ℓτ,m (λ) = ℓ⋆λ + ℓ̄τ,m (λ).
■
Substituting Eq. (105) into Eq. (106) yields Eq. (92).
E
(106)
Unified Effective-Amplitude-Aware Loss Weight
The preceding analysis shows that optimization imbalance arises when different target parameterizations expose different amounts of usable signal across noise levels. This suggests that the allocation rule should not be tied merely to the total target energy, nor to a sign-sensitive linear alignment quantity, but rather to the effective amplitude with which each independent source component is expressed through the corrupted input. We now formalize this principle and derive a unified prototype weight that applies to the canonical diffusion and flow-matching targets in a single formula. Proposition E.1 (Unified effective-amplitude-aware weight). Consider the generalized target parameterization xλ = α(λ)x0 + σ(λ)ϵ,
yλ = cx (λ)x0 + cϵ (λ)ϵ,
(107)
where x0 ⊥ ϵ, E[x0 ] = 0, Cov(x0 ) = I, and ϵ ∼ N (0, I). Define the effective allocation by My (λ) = wy (λ)p(λ) Under fixed p(λ) and fixed
dt . dλ
(108)
dt dλ , define the effective-amplitude score by
r ωy (λ) =
i 1 h 2 E ∥cx (λ)α(λ)x0 + cϵ (λ)σ(λ)ϵ∥2 . d 26
(109)
Then the prototype weight is wy⋆ (λ) ∝ ωy (λ) =
q
2 2 cx (λ)α(λ) + cϵ (λ)σ(λ) .
Consequently, by substitution into Eq. (110), the canonical targets satisfy α(λ), yλ = x0 , σ(λ), yλ = ϵ, wy⋆ (λ) ∝ p 2 2 α(λ) + σ(λ) , yλ = uλ = ϵ − x0 , α(λ)σ(λ), yλ = vλ = α(λ)ϵ − σ(λ)x0 ,
(110)
(111)
where uλ = ϵ − x0 and vλ = α(λ)ϵ − σ(λ)x0 , up to a common normalization factor. Proof. The guiding principle is to measure the effective scale with which each independent source component contributes through the corrupted input xλ . Since xλ = α(λ)x0 + σ(λ)ϵ,
(112)
the x0 -channel is expressed with amplitude α(λ) and the ϵ-channel is expressed with amplitude σ(λ). Therefore, for the generalized target yλ = cx (λ)x0 + cϵ (λ)ϵ,
(113)
the corresponding channel-scaled effective target is yeλ ≜ cx (λ)α(λ)x0 + cϵ (λ)σ(λ)ϵ.
(114)
By definition, r
1 E[∥e yλ ∥22 ] d r i 1 h 2 E ∥cx (λ)α(λ)x0 + cϵ (λ)σ(λ)ϵ∥2 . = d
ωy (λ) =
(115)
Expanding the squared norm gives i 1 h 2 ωy (λ)2 = E ∥cx (λ)α(λ)x0 + cϵ (λ)σ(λ)ϵ∥2 d 1 = E cx (λ)2 α(λ)2 ∥x0 ∥22 + cϵ (λ)2 σ(λ)2 ∥ϵ∥22 + 2cx (λ)cϵ (λ)α(λ)σ(λ)⟨x0 , ϵ⟩ . (116) d Using x0 ⊥ ϵ, E[x0 ] = 0, Cov(x0 ) = I, and ϵ ∼ N (0, I), we have E⟨x0 , ϵ⟩ = 0, E∥x0 ∥22 = Tr Cov(x0 ) = Tr(I) = d, E∥ϵ∥22 = Tr Cov(ϵ) = Tr(I) = d.
(117)
Substituting Eq. (117) into Eq. (116) yields 1 cx (λ)2 α(λ)2 d + cϵ (λ)2 σ(λ)2 d d = cx (λ)2 α(λ)2 + cϵ (λ)2 σ(λ)2 .
ωy (λ)2 =
(118)
Therefore, ωy (λ) =
q
2 2 cx (λ)α(λ) + cϵ (λ)σ(λ) ,
(119)
which proves Eq. (110). The canonical-target formulas in Eq. (111) follow immediately by substituting the corresponding coefficient pairs (cx , cϵ ) into Eq. (110). ■ 27
Remark E.2 (Reduction from allocation operator to weight). Under the standard training configuration considered in this work, the log-SNR variable is sampled uniformly, so p(λ) = const., and dt the continuous-time schedule t(λ) is parameterized linearly with respect to λ, so dλ = const.. Therefore, My (λ) = wy (λ)p(λ)
dt ∝ wy (λ). dλ
(120)
Consequently, in this regime, the allocation principle reduces directly to a target-adaptive loss weighting rule, up to an overall normalization constant. Remark E.3 (Interpretation). The prototype score Ay (λ) is not based on a sign-sensitive linear alignment quantity such as cx (λ)α(λ) + cϵ (λ)σ(λ). Instead, it measures the effective RMS amplitude contributed by the two independent source channels after scaling by the amplitudes with which those channels appear in the corrupted input. As a result, the rule is invariant to sign flips of the target coefficients and does not spuriously collapse for targets such as velocity, where the two channels contribute with opposite signs.
F
Toy Experiments on 2D GMM Dataset
To empirically validate our theoretical findings on Representation Degradation, NTK spectral decay, and gradient contamination, we design controlled experiments on a 2D Gaussian Mixture Model (GMM). This simplified, structured setting allows us to visually and quantitatively analyze representation dynamics, empirical NTK, and phase space trajectories across noise levels, avoiding the confounding factors of complex architectures. F.1
Experimental Setup
Dataset Construction. We synthesize a 2D GMM with 4 symmetric clusters at µ ∈ {(±2.0, ±2.0)} as our initial data distribution q(x0 ). Samples are generated by uniformly selecting a center and injecting Gaussian noise (σ0 = 0.3), ensuring topologically distinct clusters with intra-cluster variance. Diffusion Processes and Schedules. For clean data x0 , the perturbed data marginal at continuous time t ∼ U(0, 1) is xt = αt x0 + σt ϵ, where ϵ ∼ N (0, I). To demonstrate the universality of our findings across diverse Signal-to-Noise Ratio (SNR) trajectories [26, 25], we evaluate three schedule interpolants: • Linear (Flow Matching) [36]: αt = 1 − t, σt = t. • Variance Preserving (VP) [52]: Standard Score SDE schedule mapped to t ∈ [0, 1], with a linear noise schedule β(t) = βmin + t(βmax − βmin ), where βmin = 0.1 and βmax = 20.0. • Generalized Variance Preserving (GVP) [40]: A spherical interpolant defined by αt = cos(tπ/2) and σt = sin(tπ/2). Prediction Target Formulations. To verify that gradient contamination and degradation are universal regardless of output parameterization, we evaluate four prediction targets. The network fθ (xt , t) predicts either the noise ϵ (standard DDPM [15]), clean data x0 , velocity [47] vt = αt ϵ − σt x0 , or flow matching vector field [36] ut = ϵ − x0 .
Network Architecture. The prediction network is a time-conditioned MLP mapping 2D inputs (xt , t) to 2D outputs. Continuous time t is projected into a 64-dimensional sinusoidal positional embedding and concatenated with xt . The feature extractor is a 3-layer MLP (hidden dimension m = 256, SiLU activations), followed by a linear readout head.
Training Details. We train separate models across the four target formulations (ϵ, x0 , v, u) using the Mean Squared Error (MSE) objective. Models are optimized with Adam (learning rate 1 × 10−3 , batch size 512) for 2, 000 iterations to ensure convergence. Alongside global models trained over t ∈ [0, 1], we train piecewise independent models on restricted time bins (e.g., t ∈ [0.0, 0.2]) to isolate local learning dynamics and establish empirical Bayes baselines. 28
F.2
Gradient Contamination and the Bayes Floor Gap
We first empirically validate the local ELBO decomposition and the subsequent degradation mechanisms established in Section C and Section D. The Bayes Floor Gap and Loss Domination. Recall from Theorem C.1 that the local loss decomposes into an irreducible Bayes floor ℓ⋆λ and an optimizable excess ℓ̄τ,m (λ). To validate Theorem D.2, Fig. 7 (top rows) reveals a striking loss domination phenomenon in the globally trained model: the overall optimization process is hijacked by regions exhibiting the largest absolute loss. Counterintuitively, the excess gap is minimal where the Bayes floor is large, but massive where the floor is inherently low (e.g., t → 1 for ϵθ ). Crucially, partitioning the trajectory into independent piecewise bins (bottom rows) significantly mitigates this bottleneck, allowing local models to closely approximate the Bayes floor. Isolating optimization to narrower noise scales relieves the network from the burden of reconciling conflicting gradients. This clearly corroborates our derivation: the massive excess in theoretically simpler regimes is not a capacity deficit, but a fundamental consequence of weakened effective contraction and cross-noise interference under global parameter sharing. Global Shared Model Trained on Full Range t ∈ [0, 1]
100
Global Shared Model Trained on Full Range t ∈ [0, 1] 100
MSE loss
10−2
10
LINEAR (Empirical) LINEAR (Bayes Floor)
0.0
MSE loss
VP (Empirical) VP (Bayes Floor)
GVP (Empirical) GVP (Bayes Floor)
0.2
Piecewise Bin t ∈ [0.0, 0.2]
0.4
Piecewise Bin t ∈ [0.2, 0.4]
6 × 10−1
0
0.6
Piecewise Bin t ∈ [0.4, 0.6]
0.8
1.0
Piecewise Bin t ∈ [0.6, 0.8]
Piecewise Bin t ∈ [0.8, 1.0]
10−1
10−2
2 × 10−1 3 × 10−1
3 × 10−1
0.1
Diffusion time t
0.2
0.2
0.3
Diffusion time t
0.4
10−1 0.4
10−3
10−2 0.5
Diffusion time t
0.6
0.6
0.7
Diffusion time t
0.8
0.8
0.9
Diffusion time t
0.0
0.2
Piecewise Bin t ∈ [0.0, 0.2]
Piecewise Bin t ∈ [0.2, 0.4]
10−1
10−1
3 × 10−1
4 × 10−1
4 × 10−1
LINEAR (Empirical) LINEAR (Bayes Floor)
10−4
Diffusion time t
4 × 10−1 6 × 10−1
0.0
10−2
10−3
−3
10
10−1
MSE loss
MSE loss
10−1
10
Piecewise Bin t ∈ [0.4, 0.6]
GVP (Empirical) GVP (Bayes Floor)
0.8
1.0
Piecewise Bin t ∈ [0.6, 0.8]
Piecewise Bin t ∈ [0.8, 1.0]
4 × 100
4 × 100
3 × 100
2 × 100
10−1
0.1
Diffusion time t
0.2
0.2
0.3
Diffusion time t
0.4
10−1 0.4
3 × 100
100 0.5
Diffusion time t
0.6
0.6
0.7
Diffusion time t
0.8
0.8
0.9
Diffusion time t
1.0
(b) Empirical Loss vs. Bayes Floor (xθ ) Global Shared Model Trained on Full Range t ∈ [0, 1]
MSE loss
Global Shared Model Trained on Full Range t ∈ [0, 1]
MSE loss
VP (Empirical) VP (Bayes Floor)
0.6
100
−3
(a) Empirical Loss vs. Bayes Floor (ϵθ )
Diffusion time t
100
10−2
10−4 0.0
1.0
0.4
100
100 LINEAR (Empirical) LINEAR (Bayes Floor)
0.0
0.2
Piecewise Bin t ∈ [0.0, 0.2]
Piecewise Bin t ∈ [0.2, 0.4]
Diffusion time t
0.6
Piecewise Bin t ∈ [0.4, 0.6]
100
1.0
Piecewise Bin t ∈ [0.6, 0.8]
Piecewise Bin t ∈ [0.8, 1.0]
4 × 100
0.0 Piecewise Bin t ∈ [0.0, 0.2]
3 × 100
100 0.1
Diffusion time t
0.2
0.2
0.3
Diffusion time t
0.4
0.4
0.5
Diffusion time t
0.6
0.6
0.2
4 × 100
2 × 100
4 × 10−1
0.0
LINEAR (Empirical) LINEAR (Bayes Floor)
0.8
3 × 100
100 6 × 10−1
3 × 10−1
GVP (Empirical) GVP (Bayes Floor)
MSE loss
MSE loss
100
0.4
VP (Empirical) VP (Bayes Floor)
0.7
0.8
0.8
0.9
Diffusion time t
1.0
VP (Empirical) VP (Bayes Floor)
0.6
Piecewise Bin t ∈ [0.4, 0.6]
2 × 100
GVP (Empirical) GVP (Bayes Floor)
0.8
1.0
Piecewise Bin t ∈ [0.6, 0.8]
Piecewise Bin t ∈ [0.8, 1.0]
5 × 100
5 × 100 4.8 × 100
4 × 100
4.6 × 100
8 × 10−1
100
7 × 10−1
0.0
(c) Empirical Loss vs. Bayes Floor (vθ )
Piecewise Bin t ∈ [0.2, 0.4]
4 × 100
Diffusion time t
3 × 100
100 9 × 10−1
6 × 10−1
Diffusion time t
0.4
4.4 × 100
100
3 × 100
4.2 × 100
6 × 10−1
0.1
Diffusion time t
0.2
0.2
0.3
Diffusion time t
0.4
0.4
0.5
Diffusion time t
0.6
0.6
0.7
Diffusion time t
0.8
0.8
0.9
Diffusion time t
1.0
(d) Empirical Loss vs. Bayes Floor (uθ )
Figure 7: Empirical Loss and Analytical Bayes Floor across Different Targets. We evaluate the globally trained shared models (top row of each subfigure) and piecewise independent models (bottom row of each subfigure) across the full diffusion time t ∈ [0, 1]. Phase Space Trajectories and Gradient Contamination. To trace this optimization bottleneck, we analyze the learning signals. Theorem D.3 establishes that as recoverable signal diminishes, parameter gradients become increasingly dominated by the irreducible Bayes-noise term ξλ . To empirically visualize this, we map local prediction targets into a 2D phase space spanned by their recoverable signal and unrecoverable noise norms. As Fig. 8 illustrates, degradation trajectories are strictly target-dependent. Severe contamination crossing into the noise-dominated regime occurs exclusively at low noise levels (t → 0) for ϵθ , but at high noise levels (t → 1) for xθ . Mixed targets like vθ and uθ exhibit more balanced trajectories, avoiding single-pole degradation. This provides direct empirical evidence for our framework. The network is bombarded by pure Bayesnoise gradients exactly where a target’s irreducible Bayes floor is highest. Under global parameter sharing, these contaminated gradients propagate throughout the architecture, fundamentally disrupting the network’s capacity to minimize excess risk even in simpler regimes. 29
Noise Norm
10
−2
Healthy 10−3 10−3
10−2
10−1
100
100
10−1
10−1
10−2
10−2
10−3 10−3
101
Signal Norm
Healthy 10−2
10−1
100
0.8 0.6 0.4 0.2
Healthy
10−3 10−3
101
Signal Norm
Linear (Flow Matching) 101
Degradation
10−2
10−1
100
10
100
10−1
10−1
10−1
10−2
10−2
10−2 10−3
Healthy 10−3
10−2
10−1
100 10−1
Degradation
0.8
100
0.6 10−1
0.4
10−2
10−2
10−2
Healthy
10−3 10−3
10−2
10−1
100
10−3 10−3
10−2
10−1
100
10−2
101
Signal Norm
0.2
Healthy
Healthy
101
Signal Norm
10−2
10−1
100
0.8 0.6 0.4
10−3
Healthy 10−3
101
Signal Norm
10−2
10−1
100
0.2
101
Signal Norm
10−1
100
Degradation
101
Generalized VP (GVP)
Degradation
101
Degradation 0.8
100
100
100
10−1
10−1
10−1
10−2
10−2
10−2 10−3
Healthy 10−3
101
Signal Norm
Variance Preserving (VP) 101
10−2
10−1
100
(c) vθ -prediction phase space
10−3 10−2
10−1
100
0.4
10−3
Healthy 10−3
101
Signal Norm
0.6
Healthy 10−3
101
Signal Norm
10−2
10−1
100
Signal Norm
0.2
Noise Level (Time t)
100
10−1
101
Degradation
10−3
101
Linear (Flow Matching)
Noise Norm
101
Degradation
Healthy
10−3
Degradation
(b) xθ -prediction phase space
Generalized VP (GVP) Noise Level (Time t)
Noise Norm
101
Variance Preserving (VP)
100
Signal Norm
(a) ϵθ -prediction phase space Linear (Flow Matching)
Generalized VP (GVP) 101
Degradation
0
101
Signal Norm
Variance Preserving (VP) 101
Degradation
100
Noise Level (Time t)
10−1
Generalized VP (GVP) 101
Degradation
100
Noise Level (Time t)
10
Variance Preserving (VP) 101
Degradation
0
Noise Norm
Linear (Flow Matching) 101
101
(d) uθ -prediction phase space
Figure 8: Phase Space Trajectories of Gradient Contamination. The samples universally migrate from a healthy signal-dominated regime to a severely degraded noise-dominated regime as t increases. The diagonal dashed line represents the 50% contamination boundary. F.3
Representation Collapse in Feature Space.
In Theorem D.4, we theoretically proved that under the fixed-noise surrogate, the shared-parameter architecture suffers from a spectral decay, wherein modes associated with the fixed-noise kernel are weakly contracted and susceptible to persistent Bayes forcing. To geometrically elucidate the consequence of this theoretical spectral decay, we track the temporal evolution of the deep internal representations. We extract the 256-dimensional hidden representations hθ,m (xt , t) immediately preceding the readout head at four uniformly sampled diffusion times t ∈ {0.1, 0.4, 0.7, 0.9}. To ensure a geometrically rigorous comparison, we fit a PCA solely on the healthy low-noise representations at t = 0.1 and project all subsequent higher-noise representations onto this exact same principal subspace. As vividly illustrated in Fig. 9, the representation topology undergoes a continuous and catastrophic collapse. At t = 0.1, the representations perfectly preserve the structural boundaries. However, as t → 0.9, these distinct clusters physically shrink and aggressively migrate towards the origin, collapsing into a tightly entangled unstructured singular manifold. This geometric visualization provides direct empirical substantiation of the spectral collapse predicted by Theorem D.4. t = 0.1 20
t = 0.4
Linear
t = 0.7
t = 0.1
t = 0.9
20
20
20
10
10
10
6
t = 0.4
Linear
4
0
0
0
−10
−10
−10
−20 20
−20
0
−20
20
20
VP
PC 2
10
−20
0
−20
20
20
10
−20
0
−20
20
20
10
0
0
0
0
−10
−10
−10
−20
0
20
−20
GVP
−20
0
20
−20
−20
0
10
−20
0
20
10
−20
2
0 −2
−4
−4
−5
0
0
0
0
−10
−10
−10
−20
20
0
20
PC 1
−5
t = 0.1
−20
0
20
PC 1
t = 0.4
PC 2
0
−6
5
0
0
0
−5
−5
0
−10
10
GVP
0
−10
10
0
−10
10
5.0
5.0
5.0
2.5
2.5
2.5
−20
0
0.0
0.0
0.0
0.0
−2.5
−2.5
−2.5
−2.5
−5.0
−5.0
−5.0
−5.0
−5
20
PC 1
0
−5
5
PC 1
0
−5
5
PC 1
0
PC 1
−5
t = 0.7
t = 0.9
t = 0.1
10
10
10
5
5
5
5
0
0
t = 0.4
Linear
0
t = 0.7 10
10
5
5
5
0
0
−5
−5
−5
−5
−5
−10
−10
−10
−10
−10
−10
0
10
6
−10
0
10
6
−10
0
−20
10
4
4
4
20
2
2
2
2
10
PC 2
4
0
−20
20
VP
0
−20
20
0
−20
20
20
20
20
10
10
10
0
0
0
0
0
0
0
0
−2
−2
−2
−2
−10
−10
−10
−10
−4
−4
−4
−4
−20
−20
−20
−20
−6 15
−5
0
5
−6 15
GVP
−5
0
5
−6 15
−5
0
5
−6 15
−5
0
−20
5
10
10
10
10
10
5
5
5
5
5
0
0
0
PC 2
PC 2
6
0
0
−20
20
GVP
0
0
−20
20
0
−20
20
10
10
10
5
5
5
0
0
−5
−5
−5
−5
−5
−5
−5
−10
−10
−10
−10
−10
−10
−10
−10
0
PC 1
10
−15
−10
0
PC 1
10
−15
−10
0
PC 1
5
10
−15
−10
0
PC 1
−10
10
(c) vθ -prediction
0
PC 1
10
−10
0
PC 1
10
0
20
0
20
0
10
0
−5 −10 −15
0
PC 1
0
−5 −10
10
10
t = 0.9
10
−5
0
5
0
5
−10 −10
0
(b) xθ -prediction
−5
VP
−5
5
−10 6
PC 2
0
−5
10
0
−5
0
−10
PC 2
Linear
5
−6
5
5
(a) ϵθ -prediction 10
0
5
−5 20
PC 2
PC 2
0
0
−6
5
VP
0
2
0 −2
10
−10
PC 1
2
0
−4
2.5
−20
4
−2
5
−20
6
4
0
5.0
10
t = 0.9
6
4
−4 −6
20
t = 0.7
6
−2
10
−10 −20
2
PC 2
0 −10
PC 2
PC 2
10
−10
0
PC 1
10
−10
PC 1
(d) uθ -prediction
Figure 9: Representation Collapse in Global Shared Models. The deep hidden representations hθ,m (xt , t) are tracked across four increasing diffusion times. Projected onto the PCA basis fitted at t = 0.1, the representations catastrophically collapse from cleanly separable data clusters into an entangled, unstructured mass as t → 1. 30
F.4
Neural Tangent Kernel Analysis
To unequivocally establish that the capacity degeneration formalized in Theorem D.2 and Theorem D.4 is a universal phenomenon intrinsic to global parameter sharing, rather than an artifact of a specific output parameterization, we provide the comprehensive analysis across all four prediction targets (ϵ, x0 , v, and u) in this section. Eigenvalue Spectral Decay. To explicitly quantify the weakened effective contraction in Theorem D.4, we compute the top eigenvalues (κ1 , κ2 , κ3 ) of the Neural Tangent Kernel over normalized time. As Fig. 10 demonstrates, the spectrum exhibits severe yet highly target-dependent attenuation. For xθ and uθ , eigenvalues suffer a massive monotonic drop spanning several orders of magnitude. Conversely, the ϵθ spectrum exhibits a U-shape and increases at high noise levels to indicate retained optimization capacity. Within our theoretical framework, these vanishing eigenvalues translate directly to a diminished contraction rate. The network loses its optimization driving force in specific degraded regimes, leaving learning dynamics vulnerable to persistent Bayes forcing and mathematically confirming our geometric observations.
3 × 10
Variance Preserving (VP)
Generalized VP (GVP)
105
∙1 ∙2 ∙3
4
6 × 104
4 × 104 3 × 104
2 × 104
2 × 104
10 0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
4
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
Linear (Flow Matching) NTK Eigenvalues
NTK Eigenvalues
Linear (Flow Matching) 4 × 104
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
4 × 103
3 × 103
3 × 103
103
0.0
2 × 103
∙1 ∙2 ∙3
2 × 103
1.0
0.2
0.4
0.6
0.8
1.0
Diffusion Time t
Generalized VP (GVP)
104 103 6 × 103
4 × 103
4 × 103
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
Linear (Flow Matching) NTK Eigenvalues
NTK Eigenvalues
104 6 × 103
102 0.0
103 0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
(b) xθ -prediction
Variance Preserving (VP)
∙1 ∙2 ∙3
Generalized VP (GVP) 6 × 103 4 × 103
(a) ϵθ -prediction Linear (Flow Matching)
Variance Preserving (VP) 104
6 × 103
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
Variance Preserving (VP)
105
∙1 ∙2 ∙3
Generalized VP (GVP)
3 × 104 2 × 104
104
104
104
6 × 103 4 × 103
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
(c) vθ -prediction
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
(d) uθ -prediction
Figure 10: Eigenvalue Spectral Decay. The aggressive attenuation of the top-3 eigenvalues is a universal bottleneck, empirically supporting the bounds in Theorem D.4. Effective Rank Collapse. To further assess the global structural integrity of the Neural Tangent Kernel spectrum, we evaluate the effective rank based on Shannon entropy. While tracking individual top eigenvalues reveals absolute magnitude decay, the effective rank quantifies the overall dimensionality and richness of the learned functional space. As illustrated in Fig. 11, the effective rank undergoes a severe plummet toward near singularity limits at the extremes of the diffusion process, particularly as t → 1. This widespread rank collapse dictates that the entire eigenvalue distribution becomes heavily skewed and highly correlated. The network fundamentally loses its functional degrees of freedom, forcing its representational manifold to collapse into a constrained low-dimensional subspace. This global spectral degeneration offers definitive mathematical confirmation of the representation collapse induced by persistent Bayes noise contamination. Variance Preserving (VP)
Generalized VP (GVP)
4.0 3.5
4.00
3.5
3.75
3.0
3.50
3.0
2.5
3.25 0.0
0.2
0.4
0.6
0.8
Diffusion Time t
2.0 1.0 0.0
2.5 0.2
0.4
0.6
0.8
Diffusion Time t
1.0
Linear (Flow Matching) Effective Rank
Effective Rank
Linear (Flow Matching) 4.25
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
4.5
Variance Preserving (VP)
3.5 4.0
3.50
5.0
6
4.5
5
4.0
2.5
Generalized VP (GVP)
0.0
3.5
4
3.25 0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
4.0
3.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
3.5 0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
(b) xθ -prediction
4.00 3.75
4.5
4.0
0.0
0.0
Linear (Flow Matching) Effective Rank
Effective Rank
Linear (Flow Matching) 4.25
Generalized VP (GVP)
4.5
(a) ϵθ -prediction 4.50
Variance Preserving (VP)
5.0
5.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
4.0 3.5
5.0
4.0
4.5
3.5
4.0 3.5
3.0 0.2
0.4
0.6
0.8
Diffusion Time t
1.0
Generalized VP (GVP)
4.5
3.0 0.0
(c) vθ -prediction
Variance Preserving (VP)
4.5
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
0.0
0.2
0.4
0.6
0.8
Diffusion Time t
1.0
(d) uθ -prediction
Figure 11: Effective Rank Collapse. The Shannon entropy-based effective rank severely plummets to near-singularity limits as t → 1. 31
NTK Homogenization. A direct spatial consequence of spectral collapse appears in Fig. 12 via normalized NTK Gram matrices. Across all formulations the network maintains a discriminative block-diagonal feature space at low noise levels (t = 0.05). In this healthy state inputs from the same cluster share aligned gradients while distinct clusters remain orthogonal. The space homogenizes into an entangled uniform matrix at high noise (t = 0.95) confirming predicted capacity loss. Geometrically this implies the kernel degenerates toward a rank-one projection discarding sample-specific topology. Homogenization reveals the network processes inputs identically and loses data-conditional discrimination. Consequently the network applies near-identical updates to distinct data modes. Global parameter sharing makes these correlated updates wash out orthogonal boundaries established in cleaner regimes. Gradient space entanglement uncovers the mechanism driving cross-noise interference and validates degradation bounds governed by Bayes forcing. Linear (Flow Matching)
Variance Preserving (VP)
Generalized VP (GVP)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
1.0 0.8 0.6 0.4 0.2 0.0
(a) ϵθ -prediction formulation. Linear (Flow Matching)
Variance Preserving (VP)
Generalized VP (GVP)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
1.0 0.8 0.6 0.4 0.2 0.0
(b) xθ -prediction formulation. Linear (Flow Matching)
Variance Preserving (VP)
Generalized VP (GVP)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
1.0 0.8 0.6 0.4 0.2 0.0
(c) vθ -prediction formulation. Linear (Flow Matching)
Variance Preserving (VP)
Generalized VP (GVP)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
Low Noise (t = 0:05) High Noise (t = 0:95)
1.0 0.8 0.6 0.4 0.2 0.0
(d) uθ -prediction formulation.
Figure 12: NTK Gram Matrix Evolution. The structural collapse from cleanly separated data clusters to an entangled homogeneous state occurs across all parameterizations.
G
Experimental Supplement
G.1
Further Implementation Details.
In this section, we provide additional implementation details, including training configurations, architecture choices, and sampling procedures. A complete list of hyperparameters is provided in Table 5. Here Convolution represents whether to add a 3×3 convolutional block before output. 32
Basic setting. All experiments were conducted with 8 NVIDIA A100 GPUs (80GB) with a global batch size of 256. To accelerate training, we employ mixed-precision training (fp16). We adopt the AdamW optimizer [27, 38] with a constant learning rate of 1 × 10−4 without weight decay. Following common practice in diffusion-based image generation, we pre-compute latent representations from raw images using the Stable Diffusion VAE [45], and all experiments are performed in the latent space. As a result, no data augmentation is applied, which we find has negligible impact on performance, consistent with observations in EDM2 [24]. For image decoding, we use the stabilityai/sd-vae-ft-ema decoder. Diffusion setting. We adopt a discrete-time formulation where the forward process is defined over t ∈ [0, 1000), utilizing the cosine noise schedule following IDDPM [42]. During training, to enable classifier-free guidance (CFG) at inference time, we randomly drop out the class conditioning labels with a probability of puncond = 0.1, replacing them with a learned null token. Following standard practice to stabilize optimization and improve sample quality, we maintain an Exponential Moving Average (EMA) of the model weights with a decay rate of 0.9999 for all evaluations. For sampling, we use the deterministic EDM Heun sampler [23] with N = 50 integration steps. Backbone architecture. To validate the universality of our method, we use DiT [43] and U-ViT [2] as the primary backbone networks. Compared with the homogeneous sequence processing of DiT, U-ViT adopts a specialized symmetric encoder–decoder architecture. This topology incorporates a designated middle block and long-range skip connections seamlessly bridging corresponding encoder and decoder layers, which significantly improves multi-scale feature interaction across different depths. Following the canonical design, hierarchical encoder features are sequentially concatenated with the corresponding decoder features and then projected by a dense linear layer. Table 5: Hyperparameter setup. Related result
Table 3
Table 4
Architecture Model Input dim. Num. layers Hidden dim. Num. heads Convolution Params (M)
DiT-B/2 32×32×4 12 768 12 ∅ 130
DiT-L/2 32×32×4 24 1,024 16 ∅ 458
DiT-XL/2 32×32×4 28 1,152 16 ∅ 675
U-ViT-M/2 32×32×4 17 768 12 ✓ 131
U-ViT-L/2 32×32×4 21 1,024 16 ✓ 287
U-ViT-H/2 32×32×4 29 1,152 16 ✗ 501
U-ViT-H/2 32×32×4 29 1,152 16 ✗ 501
Optimization Training iteration / epoch Batch size Optimizer Learning rate (β1 , β2 )
400K 256 AdamW 1e-4 (0.9, 0.999)
400K 256 AdamW 1e-4 (0.9, 0.999)
400K 256 AdamW 1e-4 (0.9, 0.999)
400K 256 AdamW 1e-4 (0.99, 0.99)
400K 256 AdamW 1e-4 (0.99, 0.99)
400K 256 AdamW 1e-4 (0.99, 0.99)
800 256 AdamW 1e-4 (0.99, 0.99)
Diffusion settings Noise schedule Training objective Sampler Sampling steps Guidance scale
cosine ϵ-prediction EDM Heun 50 -
cosine ϵ-prediction EDM Heun 50 -
cosine ϵθ -prediction EDM Heun 50 -
cosine ϵθ -prediction EDM Heun 50 -
cosine ϵθ -prediction EDM Heun 50 -
cosine ϵθ -prediction EDM Heun 50 -
cosine ϵθ -prediction EDM Heun 50 1.35
G.2
Evaluation Details
We adopt the same evaluation protocol as ADM [6], and use the identical reference batches provided in their official implementation.2 For efficiency, we enable mixed-precision (fp16) during sample generation, and empirically observe no noticeable degradation compared to full fp32 precision. Below, we briefly describe the evaluation metrics used throughout our experiments. • Fréchet Inception Distance (FID) [13] quantifies the discrepancy between real and generated image distributions in a learned feature space. The features are extracted using an Inception-v3 network [53], and the distance is computed under the assumption that both distributions follow multivariate Gaussian statistics. • Spatial FID (sFID) [41] extends FID by leveraging intermediate spatial feature maps of Inceptionv3, allowing it to better reflect spatial consistency in generated images. 2 https://github.com/openai/guided-diffusion/tree/main/evaluations
33
• Inception Score (IS) [48] is also based on the Inception-v3 classifier, but evaluates samples using the predicted class logits. Concretely, it computes the Kullback–Leibler divergence between the marginal label distribution and the conditional label distribution obtained after softmax normalization. • Precision and Recall [29] assess the quality and coverage of generated samples, respectively. Precision reflects the proportion of generated images that are visually realistic, while recall measures how well the generated distribution captures the diversity of the training data manifold. G.3
Detailed Quantitative Results
Tables 6 and 7 present the quantitative comparisons on DiT and U-ViT backbones, respectively. Table 6: Detailed results on DiT models of different sizes, without classifier-free guidance.
Model
#Params
Iter.
FID↓
sFID↓
IS↑
Prec.↑
Rec.↑
SiT-B/2 [40] DiT-B/2 [43] + Ours + Ours + Ours + Ours
130M 130M 130M 130M 130M 130M
400K 400K 100K 200K 300K 400K
33.0 43.4 60.0 43.6 36.8 32.3
6.5 8.4 7.6 7.3 7.0
43.7 21.8 32.6 40.5 46.6
0.53 0.40 0.47 0.51 0.53
0.63 0.59 0.64 0.66 0.66
SiT-L/2 [40] DiT-L/2 [43] + Ours + Ours + Ours + Ours
458M 458M 458M 458M 458M 458M
400K 400K 100K 200K 300K 400K
18.8 23.3 39.9 25.2 20.0 17.2
5.3 7.7 7.0 6.8 6.5
72.0 33.9 55.8 69.6 78.9
0.64 0.50 0.58 0.61 0.62
0.64 0.64 0.65 0.66 0.67
SiT-XL/2 [40] DiT-XL/2 [43] + Ours + Ours + Ours + Ours
675M 675M 675M 675M 675M 675M
400K 400K 100K 200K 300K 400K
17.2 19.5 35.9 21.7 17.3 15.0
7.5 6.6 6.4 6.4
37.4 62.6 77.4 86.4
0.53 0.60 0.62 0.64
0.64 0.65 0.67 0.66
Table 7: Detailed results on U-ViT models of different sizes, without classifier-free guidance.
Model
#Params
Iter.
FID↓
sFID↓
IS↑
Prec.↑
Rec.↑
U-ViT-M/2 + Ours + Ours + Ours + Ours
131M 131M 131M 131M 131M
400K 100K 200K 300K 400K
27.7 38.4 26.7 22.3 19.9
7.0 8.8 8.1 7.5 7.0
50.9 37.0 54.3 64.1 70.8
0.55 0.52 0.57 0.59 0.60
0.66 0.63 0.65 0.66 0.66
U-ViT-L/2 + Ours + Ours + Ours + Ours
287M 287M 287M 287M 287M
400K 100K 200K 300K 400K
20.4 31.9 20.1 16.6 15.0
6.6 7.4 7.2 6.8 6.7
68.0 44.3 67.3 79.2 86.3
0.60 0.56 0.62 0.63 0.64
0.67 0.63 0.64 0.66 0.65
Min-SNR U-ViT-H/2 + Ours + Ours + Ours + Ours
501M 501M 501M 501M 501M 501M
400K 400K 100K 200K 300K 400K
11.7 13.7 21.6 12.0 9.8 8.9
6.6 5.6 6.3 5.6 5.5 5.4
99.7 87.9 60.6 92.9 106.8 114.3
0.66 0.65 0.63 0.67 0.68 0.68
0.65 0.65 0.60 0.64 0.64 0.66
To analyze training efficiency, we visualize the convergence curves on different models in Fig. 13. 34
45
50
48.2 45.0 43.6
43.4
40
400K
25.2 22.3 20.0
100K
200K 300K Training Iteration
(a) DiT-B/2
30.9
30 20
27.7
26.7
25
31.2
30
26.3
25
15
400K
31.9
30
28.5
25
23.8 21.7
19.9
200K 300K Training Iteration
400K
15
20.2 17.3
100K
200K 300K Training Iteration
33.1
30
19.5 17.2 15.0
400K
U-ViT Min-SNR Ours
29.5
25 20
19.5
15
15.9 14.0
23.1 20.4
20.1
20
22.3
100K
35.9
26.9
35 36.4
DiT SiT Ours
20
18.8 17.2
FID-50K
35
23.3
U-ViT Ours
40 FID-50K
FID-50K
38.4
38.7
(c) DiT-XL/2
44.7
45
U-ViT Ours
45 40
44.5
40
(b) DiT-L/2
51.7
50
28.4
28.1
20
33.0 32.3
200K 300K Training Iteration
30
45
35
36.5
35
25
38.0 36.8
100K
DiT SiT Ours
42.5 39.9
40 57.5
FID-50K
FID-50K
62.2 60.0
60
48.0
DiT SiT Ours
FID-50K
75.8
70
16.6
100K
200K 300K Training Iteration
(d) U-ViT-M/2
15.0
400K
10
100K
(e) U-ViT-L/2
15.2 12.7 11.6
200K 300K Training Iteration
13.7 11.7 10.4
400K
(f) U-ViT-H/2
Figure 13: Convergence curves on ImageNet 256×256. Intermediate FID values of DiT are approximated from the training curves [40, 20] for visualization only. We also provide our U-ViT-H/2 at 4M iteration with classifier-free guidance with different class-free guidance scales. Moreover, we provide the results with the guidance interval [30]. Table 8: Results with different classifier-free guidance scale w.
Model
#Params
Iter.
w
FID↓
sFID↓
IS↑
Prec.↑
Rec.↑
DiT-XL/2 [43] SiT-XL/2 [40] U-ViT-H/2 [2] Ours Ours Ours Ours Ours
675M 675M 501M 501M 501M 501M 501M 501M
7M 7M 2M 4M 4M 4M 4M 4M
1.500 1.500 1.500 1.300 1.350 1.400 1.450 1.500
2.27 2.06 2.29 2.08 2.00 1.94 1.91 1.89
4.60 4.50 5.68 4.62 4.57 4.53 4.50 4.48
278.24 270.3 263.88 255.6 266.9 273.8 279.3 282.5
0.83 0.82 0.82 0.79 0.80 0.81 0.82 0.82
0.57 0.59 0.57 0.64 0.63 0.62 0.61 0.60
Table 9: Different classifier-free guidance and guidance interval [30].
Model
#Params
Iter.
Interval
w
FID↓
sFID↓
IS↑
Prec.↑
Rec.↑
Ours Ours Ours Ours Ours Ours Ours
501M 501M 501M 501M 501M 501M 501M
4M 4M 4M 4M 4M 4M 4M
[0, 0.9] [0, 0.85] [0, 0.8] [0, 0.8] [0, 0.8] [0, 0.75] [0, 0.7]
2.10 2.10 2.00 2.10 2.20 2.10 2.10
1.71 1.56 1.48 1.45 1.49 1.52 1.60
4.43 4.47 4.54 4.51 4.53 4.66 4.83
326.8 319.5 304.8 312.4 311.7 300.3 289.6
0.83 0.82 0.81 0.81 0.82 0.80 0.79
0.59 0.60 0.63 0.62 0.61 0.64 0.65
G.4
Qualitative Results
35
Figure 14: We use classifier-free guidance with w = 4.0. Class label = “loggerhead sea turtle” (33).
Figure 15: We use classifier-free guidance with w = 4.0. Class label = “macaw” (88).
Figure 16: We use classifier-free guidance with w = 4.0. Class label = “sulphur-crested cockatoo” (89).
Figure 17: We use classifier-free guidance with w = 4.0. Class label = “golden retriever” (207).
36
Figure 18: We use classifier-free guidance with w = 4.0. Class label = “husky” (250).
Figure 19: We use classifier-free guidance with w = 4.0. Class label = “arctic wolf” (270).
Figure 20: We use classifier-free guidance with w = 4.0. Class label = “arctic fox” (279).
Figure 21: We use classifier-free guidance with w = 4.0. Class label = “lion” (291).
37
Figure 22: We use classifier-free guidance with w = 4.0. Class label = “otter” (360).
Figure 23: We use classifier-free guidance with w = 4.0. Class label = “red panda” (387).
Figure 24: We use classifier-free guidance with w = 4.0. Class label = “panda” (388).
Figure 25: We use classifier-free guidance with w = 4.0. Class label = “acoustic guitar” (402).
38
Figure 26: We use classifier-free guidance with w = 4.0. Class label = “balloon” (417).
Figure 27: We use classifier-free guidance with w = 4.0. Class label = “baseball” (429).
Figure 28: We use classifier-free guidance with w = 4.0. Class label = “dog sled” (537).
Figure 29: We use classifier-free guidance with w = 4.0. Class label = “fire truck” (555).
39
Figure 30: We use classifier-free guidance with w = 4.0. Class label = “laptop” (620).
Figure 31: We use classifier-free guidance with w = 4.0. Class label = “space shuttle” (812).
Figure 32: We use classifier-free guidance with w = 4.0. Class label = “ice cream” (928).
Figure 33: We use classifier-free guidance with w = 4.0. Class label = “cheeseburger” (933).
40
Figure 34: We use classifier-free guidance with w = 4.0. Class label = “cliff drop-off” (972).
Figure 35: We use classifier-free guidance with w = 4.0. Class label = “coral reef” (973).
Figure 36: We use classifier-free guidance with w = 4.0. Class label = “lake shore” (975).
Figure 37: We use classifier-free guidance with w = 4.0. Class label = “volcano” (980).
41