Posterior Collapse as Automatic Spectral Pruning Johannes Hirn1 1
Image Processing Laboratory (IPL), Universitat de València, Paterna, València 46980, Spain∗
arXiv:2605.22691v1 [cs.LG] 21 May 2026
We show that posterior collapse in 𝛽-VAEs implements automatic spectral pruning. A latent mode collapses if its contribution to reconstruction is below the cutoff set by 𝛽. Equilibrium solutions with different 𝛽 thus reveal a cascade of collapses as latent modes decouple from least to most useful. We derive this as a consequence of the loss via a Landau stability analysis. We define a latentrescaling-invariant order parameter that ranks active latent modes and whose collapse thresholds identify which effective variables to inspect first. In the linear Gaussian case, the collapse spectrum, utility spectrum, and normalized PCA spectrum coincide, and each collapse follows a mean-field law. We test these predictions on the WorldClim dataset. I.
INTRODUCTION
Variational autoencoders (VAEs) learn compressed probabilistic representations by balancing data fidelity against the divergence from a reference latent prior [1, 2]. 𝛽-VAEs make this rate–distortion tradeoff explicit by introducing the regularization strength 𝛽 as a control parameter multiplying the relative entropy term [3, 4]. In practical applications, it has often been observed that many latent coordinates revert toward the prior, stop carrying meaningful information about the input, and are effectively ignored by the decoder. Most of the VAE literature treats this posterior collapse as a failure mode, attributing it to decoder expressivity, bad local minima, amortization lag, or a mismatch between optimization dynamics and the variational objective; many successful approaches have therefore been proposed to delay or soften it [5–9]. Even in the linear case, where the geometry is much cleaner, collapse is usually considered a pathology [10–12]. In this paper, we show that posterior collapse is an equilibrium property of the loss itself. Optimization effects can of course modify access to that equilibrium: learning-rate schedules, annealing protocols, early stopping, and decoder parameterization can make collapse more severe, delay it, or obscure the equilibrium regime selected by the loss. Yet, posterior collapse is not inherently good or bad: whether one seeks to avoid, delay, or exploit collapse depends on the downstream goal. When the latent representation itself is the object of interest, as in feature learning, dimensionality reduction, and the interpretability of effective variables, posterior collapse can be a feature instead of a bug. We therefore study posterior collapse as a measurable response to the regularization strength. Rate–distortion analyses of VAEs, including the information-theoretic ELBO view of Alemi et al., already emphasize the global tradeoff between total distortion 𝐷 and total rate 𝑅
∗ [email protected]; https://orcid.org/0000-0003-0267-2479
[6]. At that level, changing the regularization strength 𝛽 moves the model along a smooth 𝐷(𝑅) curve. Our refinement is to resolve that smooth curve into mode-wise collapse events. Rather than asking only where the model lies in the (𝐷, 𝑅) plane, we ask which modes remain active at each point along this curve and how much distortion each mode removes. If the latent representation is truncated to its first 𝑘−1 ordered modes, the marginal reduction in distortion obtained by adding the 𝑘-th mode defines a utility spectrum. Our scans expose a mechanism hidden by the global rate-distortion curve: as we increase regularization, latent coordinates do not simply degrade together in a featureless continuum or vanish at unpredictable values of the regularization strength. They collapse in sequence, each at its characteristic threshold. The ordered set of these thresholds defines a collapse spectrum. The central claim established below is that the collapse spectrum and the utility spectrum coincide mode by mode. Instead of dismissing collapse as a binary failure event, we are thus led to consider ordered thresholds as quantitative observables. This perspective places VAEs in direct contact with the information bottleneck (IB) program. The original IB formulation of Tishby, Pereira, and Bialek casts representation learning as a competition between compression and information [13]. The Gaussian IB solution of Chechik, Globerson, Tishby, and Weiss makes the spectral structure especially explicit: different bottleneck dimensions appear sequentially through a cascade controlled by the spectrum of a quadratic operator [14]. The present paper makes this decomposition explicit in linear Gaussian VAEs: posterior collapse reveals the mode-wise tradeoffs and instabilities hidden inside the smooth global rate–distortion curve. This baseline linear case avoids basis rotations induced by nonlinear feature learning and suppresses architecture-dependent mixing, while preserving the essential rate–distortion competition. The connection between linear Gaussian VAEs and PCA is of course well known. The goal here is not to rediscover PCA as an endpoint, but to show how the variational objective reaches that endpoint mode by mode. When scanned toward stronger regularization, the
2 equilibrium family is a sequence of collapses; scanned in the opposite direction, the same sequence corresponds to PCA modes activating one by one, thus improving the reconstruction. In this sense, spectral pruning is analogous to cutting off PCA modes by their individual eigenvalues, not by imposing a threshold on the cumulative explained variance. In the linear Gaussian VAE, normalization conventions can be fixed unambiguously. This is important because raw 𝛽 values are not universal: they depend on input scaling, loss normalization, and reconstruction convention. Once the control parameter is expressed in normalized distortion units, however, collapse thresholds and utilities can be compared directly with normalized PCA eigenvalues. All empirical figures in the present paper are for linear VAEs trained on WorldClim: a gridded global climatology of 19 bioclimatic variables at 10 arc-minute resolution. We use geography-aware train/validation/test splits to reduce spatial leakage. The final figures were computed on the held-out test split after the analysis protocol was fixed. The paper is organized as follows. Section II defines the collapse-spectroscopy protocol: normalized scan units, latent order parameters, collapse spectra, utility spectra, and the signal-fraction ranking convention. Section III gives the single-mode Landau derivation and identifies the scale-invariant signal fraction as the order parameter. Section IV shows that, in the linear Gaussian case, the collapse spectrum, utility spectrum, and PCA spectrum are the same object. Section V interprets the result as automatic spectral pruning. Section VI concludes by explaining what this linear baseline establishes. Section VII outlines the broader spectral-ranking program beyond the linear Gaussian case. II.
The loss used throughout is the 𝛽-VAE objective (1)
where the reconstruction distortion is the expected negative log-likelihood, 𝐷 = −⟨log 𝑝𝜃 (𝑥 ∣ 𝑧)⟩,
(2)
and the latent regularization term is given by the Kullback-Leibler divergence to a normal prior 𝑝(𝑧) = 𝒩(0, 𝐼) 𝑅 = KL[𝑞𝜙 (𝑧 ∣ 𝑥) ‖ 𝑝(𝑧)].
2 𝑝𝜃 (𝑥 ∣ 𝑧) = 𝒩(𝑥𝜃̂ (𝑧), 𝜎dec 𝐼) .
(4)
For this fixed-variance Gaussian likelihood, the distortion can be rewritten (up to an additive constant) as a sum of squares
𝐷=
1 ⟨‖𝑥 − 𝑥𝜃̂ (𝑧)‖2 ⟩. 2 2𝜎dec
(5)
We now introduce our normalizations, using the total data variance 𝑉 ≡ ‖𝑥 − 𝑥‖̄ 2 = ∑ 𝜆𝑘 ,
(6)
𝑘
where 𝜆𝑘 are the PCA eigenvalues of the centered data. We report squared reconstruction errors in units of this variance, 2
⟨‖𝑥 − 𝑥𝜃̂ (𝑧)‖ ⟩ 𝐷̃ ≡ , 𝑉
(7)
which implies that the optimized fully collapsed reconstruction has unit normalized distortion 𝐷∣̃
𝑞𝜙 (𝑧∣𝑥)=𝑝(𝑧)
= 1.
Pulling out a common prefactor, we recast the original loss in a way that makes explicit the real tradeoff between normalized distortion and information
COLLAPSE SPECTROSCOPY
ℒ = 𝐷 + 𝛽𝑅,
mean. Angled brackets denote posterior averages over latent noise conditional on a data point: ⟨𝑓(𝑧, 𝑥)⟩ is the posterior expectation of 𝑓 at fixed 𝑥 under the posterior. In the experiments and derivations below, the decoder likelihood is Gaussian with fixed isotropic variance,
(3)
Throughout this paper, overbars denote averages over the data distribution; in particular, 𝑥̄ denotes the data
ℒ=
𝑉 (𝐷̃ + 2𝑇 𝑅) , 2 2𝜎dec
(8)
where we have also introduced a new notation for the effective control parameter
𝑇 ≡
2 𝛽𝜎dec . 𝑉
(9)
Hereafter, we refer to 𝑇 as an effective temperature; we justify this terminology in Appendix B. The same control parameter can be described in three equivalent ways, depending on context: as the raw regularization parameter 𝛽 once the decoder variance convention is fixed, as the normalized information price 𝑇 in rate–distortion units, or as the utility cutoff below which a latent mode collapses in the mode-wise spectral picture.
3 Together, Eqs. (9) and (8) show that, in this fixed2 variance Gaussian setting, 𝛽 and 𝜎dec are not indepen2 dent control axes: only the product 𝛽𝜎dec controls the balance between the quadratic reconstruction term and the rate, and must still be compared to the total data variance 𝑉 . This normalization is important in practice. Statements such as “𝛽 = 1 gives 𝐾 active latent dimensions” are not intrinsic properties of the data or the model family: they depend on input scaling, loss normalization, and reconstruction convention. By using normalized units, one obtains a control parameter 𝑇 that can be meaningfully compared across datasets. For instance, in the extreme case where a single spectral mode carries essentially all the variance, that mode would collapse at 𝑇 ≃ 1. For any 𝑇 ≥ 1, all normalized PCA thresholds lie below the information price, so the fully collapsed distortion is
Thus 𝑀𝑘2 and the posterior SNR induce the same ranking of latent dimensions. This SNR-like relevance criterion follows the active-unit heuristics used in some interpretability analyses of VAE representation learning [15– 17]. Section III explains why the ratio 𝑀𝑘2 (13) is selected by the one-mode loss. The collapse threshold 𝑇𝑘 of ranked mode 𝑘 is extracted from the boundary between the active and collapsed branches. The primary fit uses the one-mode law to extract a threshold 𝑇𝑘
𝑀𝑘2 (𝑇 ) = [1 −
𝑇 ] . 𝑇𝑘 +
(16)
Because the collapse threshold is measured mode by mode after ranking the latent coordinates, the result is not a single crossover value of 𝛽 but an ordered spectrum
̃ 𝐷(𝑇 ≥ 1) = 1. For each latent coordinate 𝑘, we monitor four observables on held-out data, 𝜇𝑘 (𝑥)2 ,
𝜎𝑘 (𝑥)2 ,
log 𝜎𝑘 (𝑥)2 ,
KL𝑘 (𝑥) =
1 [𝜇 (𝑥)2 + 𝜎𝑘 (𝑥)2 2 𝑘
𝑅𝑘 (𝑇 ) = KL𝑘 (𝑥), (10) with the divergence between two Gaussians given by
{𝑇1 , 𝑇2 , …}.
This collapse spectrum is the first of the two spectra in the paper. The second spectrum is obtained from truncated reconstructions. Let 𝐷̃ 𝑘 denote the normalized distortion achieved by keeping the first 𝑘 ranked latent modes and pruning the rest. Then the marginal utility of the 𝑘-th mode is
(11)
− log 𝜎𝑘 (𝑥)2 − 1].
Δ𝐷̃ 𝑘 = 𝐷̃ 𝑘−1 − 𝐷̃ 𝑘 .
For the standard Gaussian prior, the KL identity above relates these four observables algebraically. The Jensen gap between the log of the mean variance and the mean log-variance 𝐽𝑘 (𝑇 ) = log 𝜎𝑘 (𝑥)2 − log 𝜎𝑘 (𝑥)2 .
(12)
is used later as a diagnostic of posterior-variance heterogeneity. Our default ranking and fitting observable is the posterior signal fraction
𝑀𝑘2 (𝑇 ) =
𝜇𝑘 (𝑥)2 𝜇𝑘 (𝑥)2 + 𝜎𝑘 (𝑥)2
.
(17)
(18)
These utilities define a ranked spectrum of distortion reductions. In a weaker picture, collapse thresholds and utilities could have been merely correlated: modes that collapse later might simply also tend to be more useful. For the linear Gaussian baseline, the stronger claim derived below is that the two spectra coincide: 𝑇𝑘 = Δ𝐷̃ 𝑘 .
(19)
We refer to the relation (19) as the mode utility– threshold duality.
(13) III.
SINGLE-MODE LANDAU DERIVATION
Equivalently, we can rewrite this signal fraction as 𝑀𝑘2 (𝑇 ) =
SNR𝑘 (𝑇 ) 1 + SNR𝑘 (𝑇 )
(14)
in terms of the signal-to-noise ratio
SNR𝑘 (𝑇 ) =
𝜇𝑘 (𝑥)2 𝜎𝑘 (𝑥)2
.
The simplest setting for studying posterior collapse is a linear VAE restricted to a single scalar data mode. Consider a centered scalar mode 𝑥 with variance 𝜆 = 𝑥2 .
(15)
(20)
For a one-mode linear VAE with Gaussian posterior and linear decoder,
4
𝑞(𝑧 ∣ 𝑥) = 𝒩(𝑎𝑥, 𝜎𝑧2 ), 𝑥̂ = 𝑤𝑧, 𝑝(𝑧) = 𝒩(0, 1).
(21)
the one-mode objective can first be written in terms of averaged posterior observables as ℒ=
1 ⟨(𝑥 − 𝑤𝑧)2 ⟩ 2 2𝜎dec 𝛽 + [𝜇(𝑥)2 + 𝜎𝑧2 (𝑥) − log 𝜎𝑧2 (𝑥) − 1] . 2
(22)
For the constant-variance linear parametrization in Eq. (21), Eq. (22) becomes ℒ=
1 [(1 − 𝑤𝑎)2 𝜆 + 𝑤2 𝜎𝑧2 ] 2 2𝜎dec 𝛽 + [𝑎2 𝜆 + 𝜎𝑧2 − log 𝜎𝑧2 − 1] . 2
ℒ⋆ 𝐷0 = (1 − 𝑀 2 ) + 𝜏 [𝐴2 − log(𝐴2 ) − log(1 − 𝑀 2 ) − 1] , (29)
ℓ⋆ (𝑀 2 , 𝐴2 ) ≡
with
𝜏=
2 𝛽𝜎dec . 𝜆
(30)
The reduced loss in Eq. (29) is the starting point for the branch analysis below.
(23)
This constant-variance step is an ansatz: its accuracy can be tested via the Jensen gap 𝐽𝑘 (𝑇 ) plotted in Appendix C. The original optimization has three scalar variables, (𝑎, 𝜎𝑧2 , 𝑤). We can minimize in those variables, or in any equivalent reparametrization. A useful reparametrization is to separate a scale-invariant signal fraction 𝑀2 =
where the star subscript indicates that the decoder has been set to the value 𝑤⋆ that minimizes the loss at fixed encoder statistics. Dividing by the collapsed one-mode 2 distortion scale 𝐷0 = 𝜆/(2𝜎dec ) gives
A. Branch analysis
The collapsed branch is the solution in which the posterior equals the prior, 𝜇(𝑥)2 = 0, 𝜎𝑧2 (𝑥) = 1, and its Gaussian distortion is the collapsed baseline
𝐷0 =
𝜆 . 2 2𝜎dec
(31)
The active branch is obtained by minimizing first with respect to the signal fraction:
𝜇(𝑥)2 𝜇(𝑥)2 + 𝜎𝑧2 (𝑥)
,
(24)
from the absolute posterior scale 𝐴2 = 𝜇(𝑥)2 + 𝜎𝑧2 (𝑥).
(25)
The key step is the decoder minimization. At fixed posterior statistics, stationarity with respect to 𝑤 fixes
𝜕ℓ⋆ 𝜏 = −1 + , 𝜕𝑀 2 1 − 𝑀2
(32)
𝑀2 = 1 − 𝜏
(33)
yielding
provided the condition 𝑤⋆ =
𝑎𝜆 𝜇(𝑥)2 + 𝜎𝑧2 (𝑥)
.
(26) 𝜏 < 𝜏𝑐 ≡ 1,
Substituting this decoder back into the reconstruction term yields 1 𝜆 ⟨(𝑥 − 𝑤⋆ 𝑧)2 ⟩ = 2 (1 − 𝑀 2 ). 2 2𝜎dec 2𝜎dec
(27)
Thus, for the constant-variance one-mode solution, eliminating the decoder gives the reduced objective ℒ⋆ (𝑀 2 , 𝐴2 ) =
𝜆 (1 − 𝑀 2 ) 2 2𝜎dec 𝛽 + [𝐴2 − log(𝐴2 ) − log(1 − 𝑀 2 ) − 1] , 2 (28)
(34)
is satisfied, i.e. the reduced temperature is smaller than the collapse threshold 𝜏𝑐 = 1. Equivalently, in raw 𝛽 2 units this collapse threshold is 𝛽𝑐 = 𝜆/𝜎dec by Eq. (30). For 𝜏 ≥ 1, the active branch ceases to exist and the collapsed branch remains, so 𝑀 2 = 0. The Landau form is obtained by expanding the loss near 𝑀 2 = 0: 𝜏 (𝑀 2 )2 + 𝑂((𝑀 2 )3 ) . 2 (35) The coefficient of 𝑀 2 in Eq. (35) changes sign at the collapse point in Eq. (34), and the exact minimizer gives the linear law (33) for 𝑀 2 . Thus the one-mode VAE has ℓ⋆ (𝑀 2 , 𝐴2 ) = ℓ⋆ (0, 𝐴2 ) + (𝜏 − 1) 𝑀 2 +
5 the branch structure of a Landau theory without requiring the usual phenomenological free-energy ansatz. In Landau language, 𝜏 − 1 is the mass term for the latent order parameter. At high 𝑇 it is positive, and the collapsed branch with 𝜇(𝑥) = 0 is stable. At low 𝑇 , this coefficient becomes negative, the collapsed branch becomes unstable, and an active branch appears in which the posterior mean 𝜇(𝑥) = ⟨𝑧⟩ condenses into an inputdependent collective coordinate. Both the raw posterior mean-square 𝜇(𝑥)2 and the scale-invariant signal fraction 𝑀 2 (𝑇 ) measure the inputdependent displacement of the posterior along a latent coordinate. An analogy in disordered systems is the Edwards–Anderson order parameter for a 𝑍2 reflection symmetry 𝑧 → −𝑧: the data-averaged mean may vanish, but the squared local mean remains nonzero.
B. Scale-Invariant Order Parameter
The reason the signal fraction is the robust order parameter can be seen from the decoder-eliminated reconstruction term:
C.
Latent Scale and Auxiliary Observables
The same reduced objective also shows explicitly where the latent-scale condition comes from: 1 𝜕ℓ⋆ = 𝜏 (1 − 2 ) . 2 𝜕𝐴 𝐴
For 𝛽 > 0, exact stationarity with respect to the remaining latent scale fixes 𝐴2 = 𝜇(𝑥)2 + 𝜎𝑧2 (𝑥) = 1.
(36)
The optimized reconstruction depends only on the signal fraction 𝑀 2 , not on the absolute latent scale 𝐴2 . Under a latent rescaling, 𝜇(𝑥) → 𝑠𝜇(𝑥), 𝜎𝑧2 (𝑥) → 𝑠2 𝜎𝑧2 (𝑥), 𝑤 → 𝑤/𝑠.
(37)
the raw second moments and 𝐴2 rescale, but 𝑀 2 and the optimized reconstruction do not. In practice, however, the scale-fixing contribution is proportional to 𝑇 , and the scale constraint gets weaker at low temperature, where the empirical result may deviate from the one-mode canonical solution. The raw observables 𝜇2 , 𝜎𝑧2 , and 𝐴2 can therefore drift from their canonical values, while the scale-invariant collapse law for 𝑀 2 remains stable. The local coordinate 𝜏 is useful because the isolated one-mode collapse threshold is always 𝜏𝑐 = 1. The global scan coordinate used in the experiments is instead 𝑇 = 2 𝛽𝜎dec /𝑉 . For a quadratic (PCA) mode with eigenvalue 𝜆 from a dataset with total variance 𝑉 , the same collapse threshold is therefore
𝑇𝑐 =
2 𝛽𝑐 𝜎dec 𝜆 = . 𝑉 𝑉
(38)
(40)
In this canonical one-mode scale, 𝜇(𝑥)2 = 𝑀 2 and 𝜎𝑧2 (𝑥) = 1 − 𝑀 2 . Empirically, however, the raw learned coordinate can have 𝐴2𝑘 (𝑇 ) = 𝜇𝑘 (𝑥)2 + 𝜎𝑘 (𝑥)2 measurably different from one over a scan. This is why we use 𝑀𝑘2 as the main observable order parameter, whereas the raw 𝜇𝑘 (𝑥)2 can deviate from the theoretical predictions. In the canonical one-mode solution the same branch gives 𝜎𝑧2 (𝑥) = 𝜏 =
𝜆 1 ⟨(𝑥 − 𝑤⋆ 𝑧)2 ⟩ = 2 (1 − 𝑀 2 ). 2 2𝜎dec 2𝜎dec
(39)
𝑇 , 𝑇𝑐
log 𝜎𝑧2 (𝑥) = log 𝜏 = log 𝑇 − log 𝑇𝑐 , 1 1 1 𝑇 𝑅 = log = log 𝑐 . 2 𝜏 2 𝑇
(41)
so the posterior log-variance is linear in log 𝑇 with slope +1, while 𝑅 is linear in − log 𝑇 with slope 1/2. This canonical branch is a constant-variance ansatz for the posterior noise; the implemented linear encoder is slightly more general because its log-variance head is affine in the input. In the empirical analysis, 𝜎𝑘 (𝑥)2 , log 𝜎𝑘 (𝑥)2 , and 𝑅𝑘 are therefore compared as separately saved observables. Appendix C shows the deviations in 𝐴2𝑘 (𝑇 ) and in the Jensen gap 𝐽𝑘 (𝑇 ). The active-branch distortion is 𝐷 = 𝐷0 𝜏 = 𝐷0
𝑇 . 𝑇𝑐
(42)
Equivalently, after normalizing by the collapsed onemode distortion 𝐷0 , the active-branch distortion is 𝐷/𝐷0 = 𝜏 = 𝑇 /𝑇𝑐 . The same coefficient that controls the Landau sign change therefore also controls the distortion recovered by keeping the mode. Here that coefficient is the variance 𝜆 of the scalar data direction. The one-mode calculation also has a second use: it gives the local Landau form in Eq. (35) for a single latent direction near collapse, including in a nonlinear VAE once a local branch and basis have been chosen. If the decoder-eliminated reconstruction term is expanded to quadratic order around that branch, the resulting local utility operator plays the role that the data covariance plays in the linear Gaussian problem.
6 The important difference is that, beyond the linear case, this operator is learned, branch-dependent, and can change as the active set changes with 𝑇 . Thus, for a nonlinear VAE, the one-mode derivation still applies locally on the active side of collapse: it predicts the branch structure and identifies the signal fraction as the scale-invariant observable. It suggests that the same local quadratic coefficient controls infinitesimal utility and collapse, but the finite utilities and numerical collapse thresholds have to be measured from the collapse scan itself. In the linear case, however, PCA provides predictions to compare against because it diagonalizes the quadratic fluctuation operator globally. We turn to this calibration in the next section. IV.
𝛽𝑐,𝑘 =
As for the utility of adding the 𝑘-th ranked mode to a truncated reconstruction, it is governed by the same eigenvalue. The collapsed contribution of one eigendirection is the one-mode baseline in Eq. (31). On the active branch, Eq. (42) gives 𝐷 = 𝐷0 𝜏 , so at zero information price the mode can remove that entire collapsed contribution. In likelihood units, the reduction in distortion is therefore
Δ𝐷𝑘 =
Let the centered data covariance be
Δ𝐷̃ 𝑘 =
𝐶 = 𝑈 diag(𝜆1 , … , 𝜆𝑑 ) 𝑈 ⊤ , (43)
𝑗
Rotating the inputs and the linear decoder by 𝑈 leaves the Gaussian reconstruction loss unchanged. In this principal-component basis the quadratic data-fit term is diagonal, and the stationary linear decoder can be chosen to align its active columns with PCA eigendirections, up to rotations within degenerate eigenspaces. The full loss is then a sum of independent one-mode problems, one for each eigenvalue 𝜆𝑘 , plus unused collapsed directions. The only remaining distinction is between the local onemode coordinate and the global scan coordinate. For the 𝑘-th eigendirection,
𝜏𝑘 =
2 𝛽𝜎dec 𝑇 = . 𝜆𝑘 𝜆𝑘 /𝑉
(44)
Thus the one-mode collapse threshold 𝜏𝑘 = 1 becomes 𝑇𝑘 = 𝜆𝑘 /𝑉 .
(46)
𝜆𝑘 , 2 2𝜎dec
(47)
and after rewriting in normalized squared-error units,
PCA CALIBRATION
𝑉 = tr 𝐶 = ∑ 𝜆𝑗 .
𝜆𝑘 . 2 𝜎dec
(45)
This is the precise sense in which the single-mode Landau branch becomes a PCA spectrum: PCA supplies the orthogonal directions in which the quadratic problem factorizes, while the one-mode solution tells us how each direction collapses as the information price is varied. The Landau coefficient, the collapse threshold, and the reconstruction utility are therefore all set by the same quadratic scale. The exact stationary solution gives the collapse behavior: the signal fraction 𝑀𝑘2 collapses linearly as we approach the threshold from below. The raw posterior mean-square is then 𝜇𝑘 (𝑥)2 = 𝐴2𝑘 𝑀𝑘2 . In the original likelihood parameters the raw-𝛽 collapse thresholds are
Δ𝐷𝑘 𝜆 = 𝑘. 2 𝑉 𝑉 /(2𝜎dec )
(48)
Thus 𝜆 𝑇𝑘 = Δ𝐷̃ 𝑘 = 𝑘 . 𝑉
(49)
The calibration in Eq. (49) states our main result directly: the collapse threshold, reconstruction utility, and PCA explained-variance ratio are the same normalized spectral weight for ranked mode 𝑘. The local behavior of the order parameters is then fixed by the one-mode laws above: the signal fraction 𝑀𝑘2 is linear in the distance to collapse, the canonical log-variance is linear in log 𝑇 with slope +1, and the canonical rate is linear in − log 𝑇 with slope 1/2. This gives a direct empirical test of the utility– threshold duality. In the WorldClim experiments, we check whether the collapse thresholds extracted from the scan match the utilities measured by truncated reconstructions, and compare truncated reconstruction with PCA cumulative explained variance. Training and datasplit details are collected in Appendix A. Our WorldClim linear-Gaussian results are organized around three main figures and one appendix figure containing two diagnostics. 1. Fig. 1 displays the raw posterior mean-square 𝜇𝑘 (𝑥)2 , the scale-invariant signal fraction 𝑀𝑘2 (𝑇 ), and the posterior log-variance log 𝜎𝑘 (𝑥)2 on a common scan axis. The collapse thresholds are extracted from the fixed-exponent fits in the 𝑀𝑘2 panel. 2. Fig. 2 displays the truncated normalized distortion (using posterior means to remove sampling noise) as a function of the scan coordinate and as a function of retained rank, with the PCA reference included in the rank plot.
7 3. Fig. 3 displays reconstruction utility against the corresponding collapse thresholds, testing the utility–threshold duality itself. 4. Fig. 4 in Appendix C diagnoses the behavior of the posterior scale 𝐴2𝑘 (𝑇 ) and the posterior-variance Jensen gap 𝐽𝑘 (𝑇 ). Unless otherwise noted, points and curves in these figures are colored by ranked mode.
2
1.5
1
0.5
0 1
0.8
0.6
0.4
0.2
0 0
−2
−4
−6
−8
−10
The collapse scan is shown in Fig. 1. The top panel shows the raw posterior mean-square 𝜇𝑘 (𝑥)2 . The middle panel shows the scale-invariant signal fraction 𝑀𝑘2 (𝑇 ), with fixed-exponent collapse fits overlaid. Least-squares fits are used on all points with 𝑀𝑘2 > 0.1 to extract the collapse thresholds used in Fig. 3. The bottom panel shows the posterior log-variance log 𝜎𝑘 (𝑥)2 . On the canonical one-mode branch, this quantity is linear in log 𝑇 with slope +1. The top and bottom panels show visible noise for 𝑇 ≲ 10−5 ; the middle panel hides most of this noise because 𝑀𝑘2 saturates at one. These order-parameter panels also address a practical interpretability question: which latent should one inspect first? A priori, one could answer this using almost any latent observable, or by directly selecting coordinates according to reconstruction utility, for example through a greedy-forward ranking. Fig. 1 shows why the raw posterior mean-square 𝜇𝑘 (𝑥)2 is not a good standalone answer: it displays level crossings due to latent-scale drift (see Fig. 4). In this linear experiment, the posterior log-variance also gives clean ranked curves, yet we use the signal fraction as the default ranking observable because we expect it to be more robust in future non-linear VAEs given that it is scale-invariant and directly selected by the one-mode loss. Fig. 2 makes the connection between the collapse scan and the rank-resolved distortion curve explicit. The left panel shows the truncated normalized distortion 𝐷̃ 𝑘 (𝑇 ) for successive truncation ranks. Its horizontal guides mark the best observed distortion level for each rank and help read plateaus and rank-to-rank improvements. The right panel shows the corresponding best distortion as a function of rank together with the PCA reference. This cumulative view makes the spectral tail clear: the linear VAE normalized distortion plateaus near 10−7 after rank 18, while PCA rank 18 already reaches normalized distortion below 10−12 , and is therefore outside the range of the log plot. Fig. 3 compares reconstruction utilities with collapse thresholds extracted from the fixed-exponent 𝑀𝑘2 fits in Fig. 1. Each point in Fig. 3 represents one ranked latent mode. The diagonal guide 𝑦 = 𝑥 highlights the mode utility–threshold duality: in the ideal Gaussian baseline these quantities coincide exactly; empirically, the points lie close to the diagonal. Utilities are extracted directly from truncated reconstructions and remain fairly stable through rank 18. Collapse thresholds, however, depend on the 𝑀𝑘2 fits and become inaccurate at rank 17 and higher.
−12
V. AUTOMATIC SPECTRAL PRUNING 10
−7
10
−6
10
−5
10
−4
10
−3
10
−2
10
−1
Figure 1: Order-parameter collapse scan.
1
The VAE utility and collapse spectra are not separate phenomena: in the linear Gaussian case they are different consequences of the same quadratic spectral structure, and in nonlinear VAEs the same relation is expected to hold locally for the quadratic operator controlling each
8
1 10 10 10 10 10 10 10
1
−1
10
−2
10
−3
10
−4
10
−5
10
−6
10
−7
10 10
−6
10
−4
10
−2
1
−1
−2
−3
−4
−5
−6
PCA
−7
22 20 18 16 14 12 10
8
6
4
2
0
Rank
Figure 2: Truncated distortion and rank pruning.
1
3
ytilitu noitcurtsnoceR
10
10
10
10
The posterior mean-square 𝜇𝑘 (𝑥)2 plotted in Fig. 1 is an obvious first diagnostic. Several observables could in principle be used to rank latents or extract collapse thresholds. The single-mode derivation above shows that the posterior signal fraction
1
2
−1 4 5
7
−2
6
8 9
𝑀𝑘2 =
10
11
−3 12
14
13
15
−4
16
17
10
−5 18
10
−5
10
−4
10
−3
10
−2
10
−1
1
Collapse threshold
𝜇𝑘 (𝑥)2 𝜇𝑘 (𝑥)2 + 𝜎𝑘 (𝑥)2
(50)
is selected by the one-mode loss and is invariant under latent rescalings. It is therefore insensitive to latentscale drift, making it a robust observable for extracting collapse thresholds and ranking latent dimensions. Since 𝑀𝑘2 is monotone in the aggregate posterior signal-to-noise ratio, this also justifies previous uses of SNR to rank latents. In Landau language, the measured signal fraction 𝑀𝑘2 is the order parameter. Near collapse, the local reduced loss has the branch form
Figure 3: Reconstruction utility vs collapse threshold. 𝑇 𝑇 − 1) 𝑀𝑘2 + (𝑀𝑘2 )2 + 𝑂((𝑀𝑘2 )3 ) , 𝑇𝑘 2𝑇𝑘 (51) so the collapsed state is stable for 𝑇 > 𝑇𝑘 and the active branch appears continuously for 𝑇 < 𝑇𝑘 . In the VAE, this is not a phenomenological assumption: it is the Taylor expansion of the exact one-mode loss after the decoder has been minimized. This Landau picture should not be overextended. A recent treatment has emphasized posterior collapse as a phase transition in variational autoencoders [18]. Our ℓ𝑘 (𝑀𝑘2 ) = ℓ𝑘 (0) + (
collapse branch. This is the sense in which posterior collapse acts as spectral pruning: the collapse cascade reveals the same hierarchy as the reconstruction utilities. The normalized scan coordinate acts as a moving utility cutoff: mode 𝑘 remains active if 𝑇 < Δ𝐷̃ 𝑘 and collapses if 𝑇 > Δ𝐷̃ 𝑘 . The criterion is marginal and modewise, not a rule based on the cumulative explained variance retained by all earlier modes.
9 thermodynamic analogy is deliberately limited; see Appendix B. The objective minimized here is not obtained by marginalizing over microscopic degrees of freedom. It is an analytic objective optimized over model parameters and within a chosen Gaussian family of latent posteriors. This is why we do not claim a thermodynamic secondorder phase transition or critical behavior. The Landau form should be read as a stability analysis of the crossover between collapsed and active regimes for each latent channel: as 𝑇 is lowered, an input-dependent posterior mean condenses out of Gaussian prior noise, with 𝑀𝑘2 measuring the strength of this input dependence. In this language, the latent variable is the candidate collective coordinate, while 𝑀𝑘2 is the order parameter measuring whether that coordinate remains input-dependent. The scan therefore does more than select coordinates: it measures which collective coordinates remain active at a given utility cutoff, orders them by reconstruction relevance, and counts how many are needed to represent the data at a prescribed accuracy. In the linear VAE these collective coordinates are PCA-like linear combinations. In a nonlinear VAE the same diagnostic can be applied to emergent nonlinear variables, although their utility scales must be measured from the scan rather than predicted from PCA eigenvalues.
VI. CONCLUSIONS
In this paper, we have shown that posterior collapse in VAEs is an objective-level pruning mechanism, not fundamentally a training failure. Increasing the regularization strength 𝛽 removes modes in order of increasing utility. In the linear Gaussian VAE, a 𝛽 scan resolves the global rate–distortion curve into a ranked cascade of collapses. For each latent dimension, the posterior signal fraction is the scale-invariant order parameter for the collapse crossover, while truncated reconstructions give the corresponding utility spectrum. In this calibrated setting, the collapse spectrum, utility spectrum, and normalized PCA spectrum coincide once 𝛽 is normalized in distortion units. This agreement turns the linear result into a calibrated null model. Beyond this baseline, nonlinear decoders, non-Gaussian likelihoods, finite training and optimization effects, and nearly degenerate regimes can rotate the latent basis, mix modes, or renormalize the meanfield collapse law. In those settings the collapse spectrum will deviate from PCA: departures become measurable signals of nonlinear representation learning rather than ambiguities of loss normalization. The normalization itself is also of practical use for non2 linear 𝛽-VAEs. In the common convention 𝜎dec = 1, the nominal setting 𝛽 = 1 corresponds to 𝑇 = 1/𝑉 , not to 𝑇 = 1. How many latents will remain uncollapsed at this operating point is then anyone’s guess: for standardized data with many input degrees of freedom, 𝑉 can easily be tens or hundreds, so 𝛽 = 1 may already lie at 𝑇 ≪ 1,
in which case (too) many latents may remain active. In other cases, only few latents will remain uncollapsed. We suggest a practical protocol for choosing 𝛽: choose an initial normalized information price 𝑇 , then convert back to 𝛽 for the chosen reconstruction convention. For instance, setting 𝑇 = 0.01 retains modes whose marginal normalized utility is above the 1% scale, while lowerutility modes collapse. One can then scan near that point to reach the target distortion, using the signal fraction to count and rank active latent dimensions. This avoids a “prune first, ask questions later” strategy: if one fixes a small latent budget from the start, the achieved distortion is already constrained by the architecture and cannot reveal what the full model could have reached at the same operating point.
VII.
OUTLOOK
This paper is the first step toward an effective-variable approach to spectral interpretability. Reinterpreting posterior collapse as spectral pruning turns the VAE into a practical probe: by scanning the regularization strength, one extracts a spectral ranking of learned variables ordered by reconstruction utility. The collapse scan does not by itself interpret a latent variable; it identifies which learned variables are most relevant, and therefore which ones should be inspected first. This provides a way to test low-dimensionality rather than assume it. Instead of postulating that the data lie on a small set of latent factors, the collapse scan measures how many variables are needed to reach a chosen reconstruction tolerance. A manifold-like description appears only when the cumulative utility spectrum saturates rapidly; otherwise the data require more modes, as in the linear case. In this sense, the relevant object is not simply a latent space, but a set of utility-ordered latent variables. This suggests a limited analogy with thermodynamic and renormalization group ideas. In physical systems, macroscopic behavior is often controlled to a good approximation by a small number of emergent variables or relevant operators. Additional effective variables or operators enter only when higher precision is required. The present work gives the simplest VAE analogue of that idea: posterior collapse identifies which latent coordinates are worth retaining at a given utility cutoff. Nonlinear models, coarse-graining, and dataset-size scaling provide the natural next steps for testing whether such effective variables remain stable, mix, or become irrelevant under changes of representation and scale. In particular, nonlinear decoders can concentrate reconstruction utility into fewer effective variables, producing a sparser active spectrum at the same distortion tolerance. Collapse scans provide a direct way to measure this utility concentration and to test whether the leading variables become more interpretable or disentangled at particular normalized information prices.
10 Appendix A: WorldClim Experiment Details
The WorldClim experiments use the 19 bioclimatic variables from WorldClim at 10 arc-minute resolution. We keep grid cells that are valid in all 19 layers, giving 808,053 available samples. Inputs are standardized per feature using the training split only. For the 32-latent LinearVAE run used in the manuscript figures, the data are split with the spatialblock protocol described in the code: equal-area blocks of side length 500 km, area-weighted sampling, split seed 0, and the non-training blocks split between validation and test. This gives 499,599 training samples, 153,897 validation samples, and 154,027 test samples, corresponding to 1952, 602, and 602 batches at batch size 256. Each scan point is trained in equilibrium mode with an Adam optimizer, learning rate 3 × 10−4 , and a maximum budget of 500,000 optimizer updates. With 1952 training batches, this corresponds to about 256.1 epochs. The relative patience parameter is 0.1 of the update budget, so early stopping can occur after 50,000 updates, or about 25.6 epochs, without a validation-loss improvement; otherwise training runs to the update budget. In the implementation we follow the usual 𝛽-VAE convention and set the decoder variance scale to one, 2 i.e. 𝜎dec = 1. All reported scan coordinates are converted 2 back to 𝑇 = 𝛽𝜎dec /𝑉 . The signal-fraction threshold used in the scan diagnostics is 𝑀𝑘2 > 0.1. Appendix B: Relative Entropy and Effective Temperature
The regularization term in a VAE is the data-averaged KL divergence to the latent prior, 𝑅 = KL[𝑞(𝑧 ∣ 𝑥) ‖ 𝑝(𝑧)].
(B1)
It is useful to distinguish carefully between absolute entropy, KL divergence, and entropy-like quantities built from a reference distribution. On a finite discrete state space, Shannon entropy is 𝐻[𝑞] = − ∑ 𝑞𝑖 log 𝑞𝑖 .
𝑖
(B4)
𝑖
so the relation is no longer a pure constant shift of 𝐻[𝑞]. This is exactly the prior-relative situation relevant for VAEs. In continuous space the situation is more subtle: differential entropy,
ℎ[𝑞] = − ∫ 𝑞(𝑧) log 𝑞(𝑧) 𝑑𝑧,
(B5)
is not invariant under changes of coordinates and depends implicitly on the reference measure. The invariant object is therefore not absolute entropy but the KL divergence to a chosen reference density,
KL(𝑞‖𝑝) = ∫ 𝑞(𝑧) log
𝑞(𝑧) 𝑑𝑧 ≥ 0. 𝑝(𝑧)
(B6)
The issue is that there is no canonical normalizable uniform density on an unbounded continuous space. In this sense, the latent prior 𝑝(𝑧) is not just a convenient regularizer; it defines the reference relative to which disorder is measured. The sign convention is worth fixing explicitly. In information theory, “relative entropy” often means the KL divergence itself, KL(𝑞‖𝑝) ≥ 0. To match the thermodynamic convention in which entropy enters as 𝐸 − 𝑇 𝑆, it is convenient to define relative entropy with the opposite sign, 𝑆rel [𝑞|𝑝] ≡ −KL(𝑞‖𝑝) ≤ 0,
(B7)
so that the maximum value 𝑆rel = 0 is reached when the mode posterior is equal to the prior, i.e. upon collapse. In this sign convention, free-energy-like expressions take the familiar form “energy minus temperature times entropy.” The VAE objective is usually written with the divergence convention, ℒ = 𝐷 + 𝛽𝑅,
If the reference distribution is uniform, 𝑢𝑖 = 1/𝑁 , then 𝑞𝑖 = −𝐻[𝑞] + log 𝑁 . 𝑢𝑖
KL(𝑞‖𝑝) = −𝐻[𝑞] − ∑ 𝑞𝑖 log 𝑝𝑖 ,
(B2)
𝑖
KL(𝑞‖𝑢) = ∑ 𝑞𝑖 log
For a non-uniform reference 𝑝, however, one has
(B3)
Thus, in the discrete uniform-reference case, the KL divergence to the uniform reference differs from negative absolute entropy only by an additive constant. This is why one can often switch between “entropy” and “relative entropy” without much damage in finite statistical physics.
𝑅 = KL[𝑞(𝑧 ∣ 𝑥) ‖ 𝑝(𝑧)].
(B8)
so the plus sign in front of 𝛽𝑅 is not a contradiction with thermodynamic notation. It simply reflects the fact that the regularizer is written as a nonnegative divergence. If one instead defines the averaged prior-relative entropy 𝑆rel ≡ −𝑅, the same objective becomes ℒ = 𝐷 − 𝛽𝑆rel .
(B9)
This is the form that most closely resembles 𝐸 − 𝑇 𝑆.
11 The analogy, however, is structural rather than literal. In equilibrium statistical mechanics, free energy is obtained by summing or integrating over microscopic states, and the Gibbs distribution appears as the exact minimizer of 𝐸 − 𝑇 𝑆 at fixed temperature. In a VAE, by contrast, one optimizes model parameters and approximate posteriors within a chosen variational family. The thermodynamic language is therefore useful as a variational analogy: it identifies the tradeoff controlled by 𝛽, or by 𝑇 after normalization, but it should not be read as an exact marginalization over physical microstates. Dividing the loss by a positive constant (as one could have done in Eq. (8)) does not change its minimizers, so the location of the optimum is unchanged at the level of exact optimization. However, a literal Gibbs distribution built from a rescaled loss would change, because it would assign different relative statistical weight to configurations away from the minimum. In practical training, finite step size, minibatch noise, optimizer state, and stopping criteria can still make different numerical normalizations behave differently, despite their identical argmins. What survives this translation is the useful part of the analogy. The negative log-likelihood plays the role of the energy, the negative KL divergence to the prior plays the role of an entropy relative to a reference state, and the normalized information price 𝑇 controls the tradeoff between the two.
Thus, if 𝐴2𝑘 = 1, the rate and log-variance carry the same information in different units. Departures of 𝐴2𝑘 (𝑇 ) from one diagnose posterior-scale drift, not a failure of the scale-invariant signal-fraction law. The posteriorvariance Jensen gap 𝐽𝑘 (𝑇 ) = log 𝜎𝑘 (𝑥)2 − log 𝜎𝑘 (𝑥)2 is a separate check of posterior-variance heterogeneity across samples. Both diagnostics are shown for the WorldClim run in Fig. 4.
[1] D. P. Kingma and M. Welling, Auto-encoding variational bayes, in 2nd International Conference on Learning Representations (ICLR) (2014). [2] D. P. Kingma and M. Welling, An introduction to variational autoencoders, Foundations and Trends® in Machine Learning 12, 307 (2019). [3] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner, betavae: Learning basic visual concepts with a constrained variational framework, in 5th International Conference on Learning Representations (ICLR) (OpenReview.net, 2017). [4] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, Understanding disentangling in 𝑏𝑒𝑡𝑎-vae, arXiv preprint (2018), arXiv:1804.03599. [5] S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio, Generating sentences from a continuous space, in Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning (2016) pp. 10–21. [6] A. A. Alemi, B. Poole, I. Fischer, J. V. Dillon, R. A. Saurous, and K. Murphy, Fixing a broken elbo, arXiv preprint (2018), arXiv:1711.00464. [7] J. He, D. Spokoyny, G. Neubig, and T. Berg-Kirkpatrick, Lagging inference networks and posterior collapse in variational autoencoders, in 7th International Conference on Learning Representations (ICLR) (OpenReview.net, 2019). [8] B. Dai, Z. Wang, and D. Wipf, The usual suspects? re-
assessing blame for VAE posterior collapse, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119 (PMLR, 2020) pp. 2313–2322. [9] O. Rybkin, K. Daniilidis, and S. Levine, Simple and effective vae training with calibrated decoders, arXiv preprint (2021), arXiv:2006.13202. [10] J. Lucas, G. Tucker, R. B. Grosse, and M. Norouzi, Don’t blame the ELBO! a linear VAE perspective on posterior collapse, in Advances in Neural Information Processing Systems 32 (2019). [11] Y. Ichikawa and K. Hukushima, Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing, in Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (PMLR, 2024) pp. 1936–1944. [12] Y. Ichikawa and K. Hukushima, High-dimensional asymptotics of vaes: Threshold of posterior collapse and dataset-size dependence of rate-distortion curve, Journal of Statistical Mechanics: Theory and Experiment 2025, 073402 (2025). [13] N. Tishby, F. C. Pereira, and W. Bialek, The information bottleneck method, arXiv preprint (2000), arXiv:physics/0004057. [14] G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, Information bottleneck for gaussian variables, Journal of Machine Learning Research 6, 165 (2005). [15] G. Barenboim, L. Del Debbio, J. Hirn, and V. Sanz, Exploring how a generative AI interprets music, Neural Computing and Applications 36, 17007 (2024).
Appendix C: Latent Scale Diagnostics
Section III uses the canonical constant-variance onemode branch. Our linear encoder is slightly more flexible: its log-variance head is affine in the input. We therefore keep 𝜇𝑘 (𝑥)2 , 𝜎𝑘 (𝑥)2 , log 𝜎𝑘 (𝑥)2 , and 𝑅𝑘 as separate saved observables, with no Jensen substitution of log 𝜎𝑘 (𝑥)2 for log 𝜎𝑘 (𝑥)2 . The saved per-coordinate KL rate is not an independent collapse observable: it is related to posterior logvariance via the latent-scale drift
1 1 𝑅𝑘 (𝑇 ) = − log 𝜎𝑘 (𝑥)2 + [𝐴2𝑘 (𝑇 ) − 1] . 2 2
(C1)
12 [16] V. Sanz, Learning symmetries in datasets, Applied Sciences 16, 1930 (2026). [17] V. Sanz, Artificial intelligence and symmetries: Learning, encoding, and discovering structure in physical data, arXiv preprint (2026), arXiv:2602.02351. [18] Z. Li, F. Zhang, Z. Zhang, and Y. Chen, Posterior collapse as a phase transition in variational autoencoders, Physica A: Statistical Mechanics and its Applications 683, 131228 (2026).
1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0
0.001
0.0008
0.0006
0.0004
0.0002
0
10
−6
10
−4
10
−2
1
Figure 4: WorldClim posterior-scale and Jensen-gap diagnostic for ranked latent modes.