Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models Peter Claphama,∗ , Lisa Bonhemeb and Marek Grzesa a School of Computing, University of Kent, Canterbury, United Kingdom
arXiv:2605.15965v1 [cs.LG] 15 May 2026
b International Agency for Research on Cancer (IARC/WHO), Genomic Epidemiology Branch, Lyon, France
ARTICLE INFO
ABSTRACT
Keywords: variational autoencoders posterior collapse polarised regime entropy mutual information
Variational Autoencoders (VAEs) often exhibit a polarised regime in which latent variables separate into active, passive, and mixed subsets. Existing criteria for identifying active dimensions depend on a Gaussian prior, limiting their applicability to variational models and specific priors. We propose a simple information-theoretic classification of the polarised regime based on the entropy of the mean representation. We show theoretically how this entropy couples to KL minimisation through entropy– variance bounds, and we relate the resulting criterion to Bonheme’s active/passive conditions. We also clarify a key limitation: entropy of the mean alone cannot reliably distinguish active from mixed dimensions without additional signals from the variance representation. Empirically, we evaluate the entropy criterion on 𝛽-VAEs, identifiable VAEs, Least-Volume Autoencoders, and L2-regularised autoencoders, and find that it consistently recovers a polarised regime when such a regime is present across the model classes studied. Finally, we show that passive dimensions can yield small but consistent improvements on downstream tasks when latent codes are appropriately normalised, suggesting that collapse is often a matter of scale rather than absolute information removal.
1. Introduction Variational Autoencoders (VAEs) are powerful tools for unsupervised representation learning (Kingma and Welling, 2014). These models aim to learn a low dimensional yet informative representation of the input data. It is crucial for the representation, whether intended for generation or other downstream tasks, to capture maximum information about the underlying data distribution. One particular coding strategy ends up becoming prominent in such models, that is the polarised regime (Dai and Wipf, 2019; Rolínek, Zietlow and Martius, 2019). In a polarised regime, the overall representation is split into three subsets: active, passive and mixed. Active variables encode information that captures the maximum variation in the data. Passive variables, on the other hand, encode no (or very little) information. This variable type occurs when the model adheres too tightly to the prior distribution (Lucas, Tucker, Grosse and Norouzi, 2019; Wang and Ziyin, 2022) or when it has more latent dimensions than needed (Dai and Wipf, 2019; Rolínek et al., 2019). Often, these factors result in variables encoding noise. Finally, mixed variables exhibit characteristics of both active and passive variables. These variables capture information that may explain some important variation, but may sometimes not be useful. One intuitive way to understand the three distinct subsets is to consider them as switches. Active variables are always ‘on’ and used by the decoder. Passive variables are always ‘off’ and ignored by the decoder. Mixed variables switch between ‘off’ and ‘on’ depending on the input data. ∗ Corresponding author
[email protected] (P. Clapham)
P. Clapham et al.
Identifying active versus passive variables is of significant importance. If the model collapses to only passive variables in a phenomenon known as posterior collapse (Lucas et al., 2019), there is an impact on generative performance and latent interpretability (Bowman, Vilnis, Vinyals, Dai, Józefowicz and Bengio, 2016). On the other hand, some selective collapse may be desirable to focus on salient features in downstream tasks. Since one may wish to discard uninformative dimensions, passive variables are a natural first choice. However, recent work has identified a problem with this approach to the polarised regime. Passive variables have been found to correlate with other variables (Locatello, Bauer, Lucic, Gelly, Schölkopf and Bachem, 2019; Bonheme and Grzes, 2023a). This indicates that passive variables may carry some information, making discarding them problematic. Traditional methods to detect active latent dimensions rely on prior-specific heuristics, such as a KL divergence threshold (Lucas et al., 2019) from a Gaussian prior, or more recent methods that measure it more directly (Locatello et al., 2019). These methods tie the definition of posterior collapse tightly to a specific prior and standard VAE setup. Some work has set out to expand the definition to more models, such as in the case of iVAEs, but this just introduced a new specific definition (Bonheme and Grzes, 2023b). There is a need for a general, principled criterion to characterise latent variables that does not depend on any prior assumptions. We propose an entropy-based framework using information theory (in particular, self-information or entropy) as a universal measure of a latent variable’s information or “activity”. The idea is to quantify how much information
Page 1 of 13
Entropy-Based Polarised Regime
each latent dimension carries, thereby defining active versus passive variables in a prior-agnostic way. The contributions of the paper are as follows:
(2)
𝐳 = 𝝁 + 𝝈 ⊙ 𝝐,
• We propose an information-theoretic characterisation of the polarised regime based on the entropy of the mean representation, and show how it relates to KLand prior-based definitions.
where 𝝐 ∼ (0, 𝐈) is a random noise variable drawn from a standard normal distribution, and ⊙ denotes the Hadamard product. VAEs are trained using variational inference, optimising the Evidence Lower Bound (ELBO) objective:
• We analyse theoretically how entropy, variance and KL are coupled in VAEs, identifying conditions under which high entropy implies high KL for active units.
[ ] ( ) VAE = 𝔼𝑞𝜙 (𝐳|𝐱) log 𝑝𝜃 (𝐱|𝐳) −𝐷KL 𝑞𝜙 (𝐳|𝐱) || 𝑝(𝐳) , (3)
• We empirically evaluate the entropy criterion on 𝛽-VAEs, iVAEs, LV-AEs (and L2-AEs as a nonvariational baseline), showing that it recovers a polarised regime across viable architectures. • We study the effect of selecting latent dimensions by entropy on downstream logistic-regression tasks, and show that including passive units can yield small but consistent gains when normalised. This work initially set out to examine whether passive latent variables in VAEs retain more information than is typically assumed. While our empirical results reveal that passive variables provide only slight utility in downstream tasks, this investigation led to a broader outcome: the development of an entropy-based, information-theoretic criterion for characterising latent activity. Unlike KL-based criteria tied to Gaussian priors, the proposed approach is modelagnostic and applies naturally to deterministic architectures. We demonstrate that entropy reliably separates active and passive variables not only in VAEs, but also in iVAEs and LV-AEs (Khemakhem, Kingma and Hyvärinen, 2019; Chen and Fuge, 2024). This broader applicability highlights entropy as a principled tool for assessing latent representations across a wide class of unsupervised learning models.
2. Preliminaries 2.1. Variational Autoencoders VAEs, shown in Figure 1 (Kingma and Welling, 2014), are neural network architectures designed for unsupervised learning of latent representations. Unlike traditional autoencoders, VAEs map input data to a probability distribution in the latent space rather than a deterministic vector. This is generally achieved by encoding the data into a Gaussian distribution characterised by its mean (𝝁) and covariance (𝚺) parameters: 𝑞𝜙 (𝐳|𝐱) = (𝝁, 𝚺).
(1)
In practice, VAEs commonly assume a diagonal covariance 𝚺 = diag(𝝈 2 ), so the encoder outputs vectors 𝝁 and 𝝈 2 . The sampled representation (𝐳) is obtained by sampling from this distribution using the reparameterisation trick, ensuring differentiability during training P. Clapham et al.
where 𝑝(𝐳) is the prior distribution over the latent space, usually selected to be a standard Gaussian distribution.
x
μ Encoder
z
Decoder
x̂
σ
Figure 1: Illustration of the architecture of a Variational Autoencoder (VAE).
2.2. Polarised Regime Understanding the polarised regime is crucial for interpreting the behaviour and limitations of VAEs in representing data. As discussed in Section 1, in a polarised regime the latent space is split into three categories: active, passive and mixed. • Active variables: These variables encode the information that captures the maximum variation in the data. These variations are the most informative when it comes to reconstructing the data. This maximises the log-likelihood, the first term in Eq. 3. • Passive variables: These variables encode very little information about the data as they have collapsed to the prior distribution. This minimises the KLD, the second term in Eq. 3. • Mixed variables: These variables encode information that explains some variation in the data that may not always be present. They exhibit the characteristics of both active and passive variables. A polarised regime can form when the hyper-parameter 𝛽 > 1 (Higgins, Matthey, Pal, Burgess, Glorot, Botvinick, Mohamed and Lerchner, 2017). This constraint raises the loss when features stray from the prior. As a result, the model is compelled to only ‘activate’ features that are the most informative, such that the accuracy can be maximised. In order to satisfy the KLD, other weakly informative features are collapsed to passive variables. Ideally, these features would be data noise. However, in more heavily regularised models subject to posterior collapse, this could include ground-truth features. Page 2 of 13
Entropy-Based Polarised Regime
In some cases, an informative feature may occur in some but not all data examples. Occasionally when this is true, a latent variable may appear as active only when an informative feature is present in the data. This satisfies the reconstruction error. Conversely, the same variable may appear as passive when the corresponding feature is absent, thereby fulfilling the KLD constraint. Past efforts have been made to understand and classify this phenomenon. Initially, researchers named it ‘KL vanishing’ as when variables collapse to the passive type, their contribution to the KLD portion of the loss vanishes (Higgins et al., 2017). Improvements to this were made, with Lucas et al. (2019) referring to the KL vanishing of individual variables. This motivated the idea of ‘collapsed variables’ defined by a sufficiently low KL. This was iterated on by Rolínek et al. (2019), where the individual variable collapse was described as the polarised regime. Henceforth, posterior collapse refers to the total collapse of all variables. The polarised regime refers to the selective collapse of a subset of variables. The most recent definition of this was presented by Bonheme and Grzes (2023a). They give a definition based on the mean representation (𝝁) and the variance representation (𝝈). For a representation 𝝈, 𝝈̄ refers to its mean over the data.
information theory. Specifically, the mutual information between 𝑿 and 𝒁. Unlike information measures such as correlation, which is restricted to capturing linear dependencies, mutual information can identify both linear and non-linear dependencies. Additionally, we considered the entropy of 𝒁 to measure the disorder or unpredictability of a variable, providing insight into its complexity. This is the expected surprise of a variable 𝒁. Finally, we examined variance for a commonly used measure. Despite its limitations, especially with asymmetrical distributions, variance can be useful. For instance, in PCA, maximising variance minimises reconstruction error (Jolliffe, 2002). One way of presenting the relationships between the joint entropy (H(𝑿, 𝒀 )), conditional entropy (H(𝒀 |𝑿) for instance), and mutual information (MI(𝑿; 𝒀 )) is with a Venn diagram such as in Figure 2. This Venn diagram represents the decomposition of mutual information, where the joint entropy is the two circles combined: MI(𝑿; 𝒀 ) = H(𝑿, 𝒀 ) − H(𝒀 |𝑿) − H(𝑿|𝒀 ).
(4)
• Active: 𝝈̄ ≪ 1 ̄ ≪ 1, Var(𝝁) ≪ 1 • Passive: 𝝈̄ ≈ 1, Var(𝝈) ≪ 1 & |𝝁|
H(X)
H(Y)
• Mixed: ∀𝐱, 𝐳 ∈ Active ∩ Passive H(X|Y)
Mixed variables become hard to define in practice because it requires separating all elements of 𝐱 into distinct active and passive subsets for a latent dimension 𝐳𝑖 . To resolve the computational complexity it was assumed that, while the polarised regime holds, all variables that are not active or passive will be mixed. While this assumption holds under mild regularisation, Clapham and Grzes (2023) shows variables that fall out of the definition of ‘active’, ‘passive’ or ‘mixed’. This occurs under strong regularisation. A notable drawback of this definition is that it depends on the specific statistics of the standard VAE’s encoder distribution, namely the prior over the mean representation 𝝁 and the variance representation 𝝈. For other model architectures, a polarised regime cannot be well defined using this definition. The polarised regime is convenient to consider when selecting variables for downstream tasks. In some way, the model has selected the most informative variables for you. Therefore it would be convenient to just discard those that are passive. However, as this work shows, that approach could be refined.
2.3. Statistical Quantities This work considered three quantities to capture the information of latent variables: mutual information, entropy, and variance. To capture the non-linear information shared between the input matrix 𝑿 and the latent representation 𝒁, we used P. Clapham et al.
MI(X;Y)
H(Y|X)
Figure 2: The overlap between the marginal entropies H(𝐗) and H(𝐘) (Cover and Thomas, 2012).
2.4. Entropy Approximation Throughout the theoretical analysis (Section 4), we distinguish between Shannon entropy H(⋅) for discrete variables and differential entropy h(⋅) for continuous variables. In empirical sections, however, we use the notation H(⋅) generically to denote whichever entropy functional is being approximated. All latent variables and mean representations considered in this work are continuous-valued, so any estimator employed approximates the differential entropy of the induced empirical distribution. This notational simplification does not affect the comparisons drawn between entropy, variance, and KL divergence, since the entropy–variance relationships established in Section 4 apply directly to differential entropy. It is also important to clarify the scope of the entropy estimates used throughout this work. We do not interpret the estimated entropy values as calibrated measures of information content for individual latent dimensions. Instead, the quantities of interest are the relative ordering and separation Page 3 of 13
Entropy-Based Polarised Regime
between dimensions when analysing posterior collapse. Although the entropy functional is approximated, the estimators employed are designed to preserve qualitative structure and rank ordering across dimensions. Consequently, our analysis relies on comparative differences in entropy rather than absolute magnitude. In this work we primarily estimate entropy using the matrix-based Rényi 𝛼 entropy estimator (Rényi, 1961; Yu and Príncipe, 2018). The parameter 𝛼 > 0, 𝛼 ≠ 1, controls the order of the entropy functional. In the limit 𝛼 → 1, Rényi entropy converges to Shannon entropy, and for continuous variables this corresponds to differential entropy. This approach operates directly on Gram matrices constructed from samples of the random variable and avoids explicit density estimation, making it suitable for highdimensional latent representations. Given a normalised Gram matrix 𝑨 constructed from samples of the random variable 𝑿, the entropy estimate is [𝑁 ] ∑ [ ] 1 1 𝛼 𝛼 log tr(𝑨 ) = log 𝜆𝑖 (𝑨) , 𝑆𝛼 (𝑨) = 1−𝛼 1−𝛼 𝑖=1 (5) where 𝜆𝑖 (𝑨) denotes the 𝑖-th eigenvalue of 𝑨. If 𝑩 denotes the Gram matrix constructed from samples of 𝒁, the joint entropy estimator is ( 𝑆𝛼 (𝑨, 𝑩) = 𝑆𝛼
𝑨⊙𝑩 tr(𝑨 ⊙ 𝑩)
) .
(6)
Given estimates of the marginal and joint entropies, mutual information is approximated via the Shannon decomposition 𝐼𝛼 (𝑿; 𝒁) = 𝑆𝛼 (𝑨) + 𝑆𝛼 (𝑩) − 𝑆𝛼 (𝑨, 𝑩).
(7)
This estimator provides a practical approximation of mutual information for high-dimensional representations, with empirical studies showing good performance under appropriate hyper-parameter choices (Lee and Jo, 2021). Several alternative estimators of differential entropy were also evaluated. These include histogram-based estimators, 𝑘-Nearest Neighbour estimators such as the Kozachenko– Leonenko method, and parametric density approaches using Gaussian mixture models with Monte Carlo estimation. While histogram estimators are simple to implement, they are sensitive to bin width and suffer from the curse of dimensionality. kNN estimators are non-parametric and asymptotically consistent but become computationally expensive for large sample sizes. Parametric approaches offer flexibility but introduce model selection and optimisation overhead. A comparison of these estimators on latent variables is shown in Figure 4. In practice, the matrix-based Rényi estimator provided a favourable trade-off between computational efficiency and stability of the entropy estimates. In P. Clapham et al.
addition, it enables efficient computation of joint entropy between variables of different dimensionalities, facilitating estimation of I(𝑿; 𝒁). Consequently, the Rényi estimator was adopted for the majority of experiments reported in this work.
3. Information-Theoretic Criterion The polarised regime introduced in Section 2.2 describes how latent dimensions separate into active and passive modes depending on whether they carry substantial information about the data. Prior work formalises this behaviour through the KL divergence, or structural assumptions specific to variational autoencoders. In contrast, we propose an information-theoretic characterisation that is independent of the model architecture and, crucially, independent of any assumed prior distribution over latent variables. Our starting point is the mean representation 𝝁(𝐱) produced by the encoder. Since 𝝁(𝐱) is a deterministic function of the random variable 𝐱 ∼ , it provides a natural object for quantifying the amount of information transmitted through each latent dimension. In future references, 𝐱 will be omitted in line with prior work. In the case of deterministic models, the latent representation 𝐳 is necessarily the mean representation. We consider the entropy H(𝝁𝑖 ) of the empirical marginal distribution of the 𝑖th dimension 𝝁𝑖 over the dataset. Intuitively, if 𝝁𝑖 varies substantially across datapoints, then the latent dimension 𝑖 encodes a significant amount of information; if it remains nearly constant, it is uninformative. This motivates the following simple criterion: active(𝑖) ⟺ H(𝝁𝑖 ) > 𝜏,
passive(𝑖) ⟺ H(𝝁𝑖 ) ≤ 𝜏, (8)
where 𝜏 is a threshold selected to separate high-entropy and low-entropy dimensions. The mixed case, in which a latent dimension exhibits both active and passive behaviour across datapoints, corresponds to dimensions with high entropy for one subset of the data, and dimensions with low entropy for another subset. Detecting such cases is challenging practice with the consequences interpreted in Section 7. Although the criterion in equation 8 is simple, its implications are notable. It does not rely on the analytical KL divergence, on a Gaussian prior, or on the variational structure of the model, and can therefore be applied to a wide range of latent-variable architectures, including purely deterministic autoencoders. Indeed, entropy-based activity measures information actually expressed in the representation, rather than deviation from a chosen prior. Moreover, as shown in Section 4, entropy-based activity aligns closely with the theoretical conditions that give rise to Bonheme’s criteria for the polarised regime: entropy is small exactly when KL minimisation collapses 𝝁𝑖 and 𝝈𝑖2 across datapoints, and entropy is large when substantial information must be encoded. Finally, Section 6 evaluates the entropy criterion across 𝛽-VAEs, iVAEs, LV-AEs, and L2-AEs, demonstrating that Page 4 of 13
Entropy-Based Polarised Regime
it reliably recovers the structure of the polarised regime and remains robust in settings where existing notions of activity are not directly applicable.
Since 𝔼[𝐗2 ] = Var(𝐗) + (𝔼[𝐗])2 ≥ Var(𝐗) as (𝔼[𝐗])2 ≥ 0, we obtain the following bound for the mean representation 𝝁𝑖 :
4. Theoretical Analysis of the Polarised Regime
1 2h(𝝁𝑖 ) 𝑒 . (10) 2𝜋𝑒 This bound shows that the mean term in the KL divergence is monotonically lower-bounded by the entropy of the mean representation. Consequently, a higher h(𝝁𝑖 ) raises the lower bound of the KL divergence, establishing a degree of congruence between the entropy-based and KL-based characterisations of the polarised regime. In particular, highentropy mean representations correspond to higher KL divergence, consistent with the notion of active variables. It is worth noting that the KL divergence can remain large due to contributions from the variance terms even when the mean representation carries little information. Takida, Liao, Uesaka, Takahashi and Mitsufuji (2021) observed a related phenomenon: posterior collapse can occur despite nonnegligible KL divergence when the latent variance is fixed and only the mean representation is learned. In their analysis collapse is characterised through a vanishing mutual infor̂ between the input and reconstructed output. mation I(𝑿; 𝑿) However, because this quantity depends on the decoder, it conflates representation collapse with decoder smoothness. In this work we instead consider quantities defined directly on the representation 𝝁, such as I(𝑿; 𝝁) and H(𝝁), which isolate collapse of the latent representation itself.
As discussed in Section 2.2, there are three competing definitions of the polarised regime including ours. The first is the KLD between the prior and the posterior for 𝑧, given in closed-form by ( ) 1 𝔼[𝝁2𝑖 ]+𝔼[𝝈𝑖2 ]−𝔼[log 𝝈𝑖2 ]−1 . 𝐷𝐾𝐿 (𝑞(𝐳|𝐱) ‖ 𝑝(𝐳)) = 2 (9) where a variable is passive when the KL is sufficiently low, typically some threshold 𝜖, for a sufficiently large subset of the dataset 𝛿 (Lucas et al., 2019). The second definition, this time explicitly for the polarised regime, is given in Section 2.2. Third, our novel entropy definition is given in Equation 8. In this section, we draw closer comparisons between the theoretical definitions introduced earlier and the limitations that may arise when applying our entropy-based criterion in practice. First, in Section 4.1 we show entropy aligns with KL via entropy-variance bounds. Then in Section 4.2 we show that entropy partially recovers Bonheme’s mean condition. Next, in Section 4.3 we show how KL minimisation recovers Bonheme’s passive case. Finally, in Section 4.4 we clarify the role of variance and explain why entropy remains preferable.
4.1. Entropy & KLD To begin, we compare our entropy criterion to the KL divergence. The closed-form expression for the KL divergence (Eq. 9) comprises three key terms: the first depends on the mean representation, while the latter two depend on the variance representation. Our criterion, by contrast, involves only the mean representation; discussion of the variance representation follows after we have compared the three definitions. To connect entropy to the KL mean term 𝔼[𝐗2 ], we exploit classical entropy-variance inequalities. Shannon (1948) defined the entropy power of a random variable as
𝑁(𝐗) =
1 2h(𝐗) 𝑒 , 2𝜋𝑒
where h(𝐗) denotes its differential entropy. For any realvalued random variable with finite variance, the entropy power inequality guarantees 𝑁(𝐗) ≤ Var(𝐗) with equality iff 𝐗 is Gaussian. The entropy-variance inequality follows, where ( ) 1 h(𝐗) ≤ log 2𝜋𝑒Var(𝐗) . 2 P. Clapham et al.
𝔼[𝝁2𝑖 ] ≥ Var(𝝁𝑖 ) ≥
4.2. Entropy & Bonheme Bonheme’s classification is formulated directly in terms of the empirical distributions of the mean and variance representations across datapoints. The distinction between active, passive, and mixed variables is therefore structural rather than information-theoretic. An active variable is characterised by a variance representation concentrated near zero (𝜎𝑖2 ≈ 0), implying that samples 𝑧𝑖 ∼ (𝜇𝑖 , 𝜎𝑖2 ) contain negligible stochastic noise. A passive variable, by contrast, has a tightly concentrated variance representation (Var(𝝈𝑖 ) ≪ 1) centred near one, and a mean representation that is likewise tightly concentrated (Var(𝝁𝑖 ) ≪ 1) around zero. In this regime the posterior collapses toward the prior, corresponding to over-fitting to the prior distribution. A mixed variable alternates between these behaviours across datapoints. To visualise these regimes, we show representative histograms of the mean and variance representations for each case. • Active variables exhibit broad mean distributions together with variance mass concentrated near zero. • Passive variables instead exhibit sharply concentrated mean distributions near zero and variance mass concentrated near one. • Mixed variables combine these behaviours across datapoints, with variance mass alternating between regions near zero and near one. Page 5 of 13
Entropy-Based Polarised Regime 500
500
400
400
300
300
200
200
100
100
0 3
2
1
0
1
00.5
2
500
500
400
400
300
300
200
200
100
100
0 0.010 0.005 0.000 0.005 0.010
00.5
2000
2500
1500
2000
H(𝑿) = −(𝑞 ln 𝑞 + 𝑝 ln 𝑝), 0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
1500
1000
1000
500 0 4
is approximately Bernoulli-distributed, taking value 1 with probability 𝑝 (passive) and 0 with probability 𝑞 = 1 − 𝑝 (active). The corresponding entropy is
500
2
0
2
4
00.5
Figure 3: Representative mean (left) and variance (right) distributions for active (top), passive (middle), and mixed (bottom) latent variables.
We now clarify the relationship between Bonheme’s criteria and the entropy-based criterion introduced in Section 3. Our earlier analysis established that H(𝝁𝑖 ) is tightly linked to Var(𝝁𝑖 ) via entropy–variance inequalities. In particular, the entropy of the mean representation measures the extent to which that representation varies across the dataset. In this sense, H(𝝁𝑖 ) captures precisely the statistical quantity that Bonheme encodes through Var(𝝁𝑖 ). The entropy-based criterion can therefore be viewed as an information-theoretic reformulation of Bonheme’s mean condition within the polarised regime. However, Bonheme’s definition relies critically on the variance representation. In that framework, the variance acts as a switch: a dimension is active when 𝜎𝑖2 ≈ 0 and passive when 𝜎𝑖2 ≈ 1, with mixed variables switching between these states across datapoints. This provides a direct and interpretable mechanism for identifying whether a latent dimension is effectively “on” or “off” for a given input. One might attempt to extend our entropy-based approach by computing the entropy of the variance representation itself. Yet this does not distinguish active from passive variables. In both regimes the variance representation is tightly concentrated (near 0 for active variables and near 1 for passive variables), and hence has low entropy in either case. Entropy of the variance therefore fails to separate the two polarised extremes. Where entropy of the variance does provide signal is in the mixed case. Across the dataset, a mixed variable alternates between values close to 0 (active) and values close to 1 (passive). In this idealised setting the variance representation P. Clapham et al.
(11)
which is maximised when 𝑝 = 𝑞 = 12 . Thus, while entropy of the variance representation cannot separate active from passive variables, it naturally highlights mixed variables by assigning them strictly higher entropy than either polarised regime.
4.3. KLD & Bonheme To find equivalence between the KL definition and Bonheme’s definition, consider the effect of minimising the KLD. This is exactly the setting where passive variables emerge. Recall Equation 9. The means of the mean representation and variance representation are given by the minima of the KL term, 0 and 1 respectively. The variance contribution of the KL is given by 𝑓 = 𝝈𝑖2 −log 𝝈𝑖2 −1. For this component to be equal to 0, 𝝈 2 must be equal to 1 for all values. An equivalent argument for the mean representation is trivial. Taken together, we arrive exactly at Bonheme’s definition of a passive variable. An active variable, by contrast, is one whose KL remains sufficiently large, as with the entropy criterion. To detect mixed variables, we examine how the pointwise KL behaves across datapoints. For an individual datapoint, a low KL indicates the variable is passive and a high KL indicates the variable is active. A latent dimension is mixed when some datapoints yield low KL (passive) and others yield high KL (active). This definition depends on the presence of a polarised regime, and breaks down when that regime no longer holds. Bonheme’s criteria can identify when the polarised regime fails, when a variable doesn’t satisfy the conditions, but in such cases it no longer provides a basis for variable selection. The entropy-based criterion remains useful across modes, including those where the polarised regime does not hold. Even if there is no polarised regime, per-dimension information can be of use.
4.4. Variance & Entropy The preceding analysis raises an important question: if entropy and variance provide comparable characterisations of latent activity, is entropy necessary in practice? Since variance is computationally cheaper to estimate, additional analysis is required to justify the use of entropy. We therefore begin by examining the empirical relationship between entropy and variance. Differential entropy has a well-known correspondence with variance for many common distributions, and similar behaviour is observed empirically in real data. This graph illustrates that relationship, as well as introducing an intriguing picture of the polarised regime. Observing along the x-axis, passive variables are very closely clustered (along the y-axis asymptote). Meanwhile, mixed Page 6 of 13
Entropy-Based Polarised Regime 1.5
passive mixed active
1.0
H[ ] (nats)
0.5 0.0 0.5 1.0 1.5 2.0 0.0
0.2
0.4
Var[ ]
0.6
0.8
1.0
Figure 4: Per dimension differential entropy/variance for smallNORB 𝑛𝑧 = 64, 𝛽 = 4. Variables are classified by Bonheme’s criteria and coloured accordingly
However, entropy is not reducible to variance alone. Unlike variance, entropy remains sensitive to distributional structure beyond second moments. In particular, distributions with identical variance may exhibit different entropy when their mass is distributed differently. This distinction will become relevant in Section 6, where empirical results demonstrate regimes in which variance remains fixed while entropy varies. We therefore conclude that entropy provides a prioragnostic quantity aligned with KL-based definitions in the polarised regime, while retaining sensitivity to distributional effects not captured by variance alone. This fact becomes crucial in settings with fixed second moments.
5. Models Used in Experiments and active variables are rather dispersed (along the x-axis asymptote). This indicates that using the variance of the mean representation may serve to separate between active (or mixed) variables strongly, while the entropy may serve to separate between passive variables. However, one can also observe that both of these definitions allows one to draw a line between active and mixed variables. This difference is rather small with entropy, larger with variance. Neither measure resolves it cleanly. So why use entropy in the first place? We created synthetic data to simulate a mixed distribution while fixing the variance. It is given by 𝐳 ∼ (1 − 𝜋) (0, 𝜖 2 ) + 𝜋 (0, 𝜎 2 ). 𝜎 2 is selected to ensure Var(𝐳) is constant based on the mixture parameter 𝜋, with 𝜖 = 0.05. This is a continuous spike-andslab model. The results can be found in Figure 5. They show that, while the variance is constant, the entropy appropriately varies between the two extremes (passive for low 𝜋, active for high 𝜋). 1.5
entropy estimate (nats)
1.0 0.5
5.1. 𝛽-VAEs 𝛽-VAEs (Higgins et al., 2017) introduce a hyper-parameter (𝛽) to the ELBO objective: ( ) [ ] VAE = 𝔼𝑞𝜙 (𝐳|𝐱) log 𝑝𝜃 (𝐱|𝐳) − 𝛽𝐷KL 𝑞𝜙 (𝐳|𝐱) || 𝑝(𝐳) . (12) By adjusting 𝛽, the trade-off between reconstruction accuracy and the Kullback-Leibler divergence (KLD) term can be controlled. These two terms can be described as the ‘expectation’ and ‘surprise’ respectively. This hyperparameter is incorporated to modulate the importance of the KLD term, increasing or decreasing the regularisation strength. Research has shown that 𝛽 > 1 can lead to the formation of a polarised regime in the latent space (Rolínek et al., 2019), where active units become disentangled (Bengio, Courville and Vincent, 2013; Locatello et al., 2019). Increasing 𝛽 further leads to posterior collapse (Bowman et al., 2016) where the latent representation fully collapses to the prior.
5.2. Identifiable VAEs (iVAEs)
0.0 0.5 H_hist H_knn H_gmm_mc H_gauss(var)
1.0 1.5 10 3
10 2
10 1
100
Figure 5: Differential entropy of 𝐳 for a number of different approximation techniques: histogram, k-nearest neighbours, Gaussian Mixture Model Monte Carlo.
Taken together, these comparisons establish precise relationships between the three activity criteria. KL minimisation enforces Bonheme’s passive conditions exactly, recovering collapse when the mean and variance representations converge to the prior. Entropy, while not explicitly defined relative to a prior, is coupled to KL through entropy–variance inequalities and therefore inherits its ordering properties with respect to activity. P. Clapham et al.
Identifiable VAEs (iVAEs) (Khemakhem et al., 2019) are a variant of the VAE designed to tackle the problem of VAE’s non-identifiable latent space. They do so by incorporating auxiliary information, 𝐮, from the dataset. This could be something as simple as the data class. When this is done, the model is said to be identifiable up to a linear invertible transformation. To train, an adjusted ELBO is used [ ] ( ) iVAE = 𝔼𝑞𝜙 (𝐳|𝐱,𝐮) log 𝑝𝜃 (𝐱|𝐳) −𝐷KL 𝑞𝜙 (𝐳|𝐱, 𝐮) || 𝑝(𝐳|𝐮) , (13) where 𝑝(𝐳|𝐮) is a conditional prior that depends on the auxiliary information 𝐮. This prior, while still typically Gaussian, differs from the non-conditional Gaussian prior used in standard VAEs. To overcome this, an adjusted criterion for the polarised regime had to be used (Bonheme and Grzes, 2023b). Page 7 of 13
Entropy-Based Polarised Regime
However, Wang and Cunningham (2021) proved that posterior collapse cannot occur in identifiable VAEs. It is important to note that identifiability is enforced in a different manner in that work. Rather than incorporating auxiliary information, it uses a VAE with a reversible encoder. However it proves that, regardless of architecture, if a latent space has the mathematical property of identifiability then it cannot collapse. Exactly how these conflicting results interact has not yet been the subject of study, and are covered in Sections 6.4 and 7.2.
5.3. Least-Volume Autoencoders (LV-AEs) Least-Volume AEs (LV-AEs) (Chen and Fuge, 2024) are a variant of the autoencoder which introduces a regularisation term that penalises the volume of the latent space. The notion of volume is derived from viewing the latent space as a product space, where the overall volume is given by the product of per-dimension variance. Without further constraints, however, the autoencoder could trivially minimise the volume by reducing the whole space down to a single point, requiring the decoder to compensate with high magnitudes. In order to prevent this, the decoder is constrained by enforcing the Lipschitz continuity of its layers. Under this constraint, the optimisation naturally produces a disentangled latent space in which some dimensions collapse to nearly zero variance while others retain meaningful variation. Thus, despite being a deterministic model, the LV-AE exhibits a polarised regime reminiscent of that observed in 𝛽-VAEs. In practice, the LV-AE is trained with the regularised loss: √ ( ) ( ) vol = MSE 𝐱, 𝐱′ − 𝜆 𝑚 Π𝑖 Var(𝐳𝑖 ) + 𝜂 ,
(14)
where 𝜂 is a small constant added to avoid collapse where Var(𝐳𝑖 ) ≈ 0, 𝜆 controls the strength of the volume penalty, and 𝑚 is the latent dimensionality.
5.4. L2-Regularised Autoencoders (L2-AEs) The L2-Regularised Autoencoder (L2-AE) extends the standard deterministic autoencoder by adding an explicit penalty on the magnitude of the latent code. Given an encoder 𝐳 = 𝑓𝜃 (𝐱) and decoder 𝐱′ = 𝑔𝜙 (𝐳), the model is trained using a regularised reconstruction loss, L2 (𝐱, 𝐱′ ) = ‖𝐱 − 𝐱′ ‖22 + 𝜆 ‖𝐳‖22 ,
(15)
where 𝜆 controls the strength of the latent penalty. Unlike variational models, the L2-AE does not impose a prior distribution over 𝐳 and does not employ a KL divergence term. Instead, the latent structure is shaped solely by the reconstruction objective and the L2 penalty. L2 regularisation is a very soft penalty, so it is not expected that these will have a strong effect on the resultant latent space. This model serves as a benchmark for comparison in Section 6.5. P. Clapham et al.
6. Empirical Evaluation This section presents the empirical results of our study. We begin by examining the coherence of three measures of latent activity: entropy, variance, and mutual information, and show that they evolve consistently throughout training. We then use the entropy of the mean representation to characterise the polarised regime, analysing both the training dynamics and the final entropy distribution across latent dimensions. Next, we apply this method to a range of architectures, including Least-Volume AEs and identifiable VAEs, to assess its generality beyond the classical VAE setting. Finally, we evaluate the utility of active and passive variables for downstream prediction tasks using logistic regression.
6.1. Experimental Setup For the following experiments we trained model encoders with 5 convolutional layers and 5 transposed convolutional layers in the decoder. An additional linear layer was used for the mean and variance representations for variational models. iVAEs require additional linear parameters for its conditional priors. 𝛽-VAEs were trained with 𝛽 taking values [1, 2, 4, 8, 16]. iVAEs were trained with 𝛽: [5, 3, 1, 0.5, 0.1, 0.05, 0.01]. LV-AEs were trained with 𝜆 taking values [1𝑒−4 , 1.5𝑒−4 , 2𝑒−4 , 2.5𝑒−4 , 3𝑒−4 ] and 𝜂 = 1. The standard benchmark datasets, as used by Locatello et al. (2019), were used: • MNIST (LeCun, Bottou, Bengio and Haffner, 1998) • smallNORB (LeCun, Huang and Bottou, 2004) • d-Sprites (Matthey, Higgins, Hassabis and Lerchner, 2017)
6.2. Information Measure Coherence Figure 6 shows that entropy, variance, and mutual information MI(𝐗; 𝝁) follow similar trajectories during training for a typical active latent variable with 𝛽 = 4.0. All three quantities increase steadily as the encoder discovers stable structure before plateauing at convergence. While the overall trajectory is consistent between variance and the other two, there is slight numerical difference. Since the mean representation 𝝁 = 𝑓 (𝑿) is a deterministic function of the input, the mutual information between 𝑿 and 𝝁 satisfies 𝐼(𝑿; 𝝁) = 𝐻(𝝁) in the discrete case. In continuous settings, the corresponding differential entropy plays an analogous role, up to estimator bias. Empirically, entropy remains almost fully contained within H(𝑿) (Figure 7), and varying 𝛽 changes the overall scale of this containment without altering its qualitative shape (Figure 8). The small residual term H(𝝁|𝑿) is attributed to numerical errors in our approximation. In all experiments this residual remains small and does not change the qualitative ordering of latent dimensions by entropy, which is the property required by our activity criterion. We note an increase in mutual information at 𝛽 = 2.0. While this conflicts with some publications making the case that
Page 8 of 13
test train
0
50
100 Epoch
150
200
1.6 1.5 1.4 1.3 1.2 1.1 1.0
test train
MI(z; x)
0.9 0.8 0.7 0.6 0.5 0.4 0.3
H( )
Var( )
Entropy-Based Polarised Regime
0
50
100 Epoch
150
200
1.6 1.5 1.4 1.3 1.2 1.1 1.0
test train
0
50
100 Epoch
150
200
Figure 6: Left variance, middle entropy, right mutual information. Values for a typical active variable 𝛽 = 4.0.
1.6
increasing 𝛽 must reduce mutual information, this finding supports those found by Dai, Wang and Wipf (2020). Since the three measures behave consistently in all cases, we use the entropy H(𝝁) as the primary measure for identifying active and passive variables for the remainder of this paper. It does not suffer the same limitations as variance, while being mathematically equivalent to the mutual information but without the additional computational strain. H(μ)
test train
H( )
1.4 1.2 1.0 0.8
0
50
100 Epoch
150
(a) Active variable 0.4
H(X)
test train
0.3
H( )
MI(X;μ) H(X|μ)
0.2 0.1 0.0
Figure 7: Venn diagram of quantities from smallNORB 𝛽 = 4.0. Note the exclusion of 𝐻(𝝁|𝑿).
50
100 Epoch
150
200
(b) Passive variable
H(X| ) MI(X; ) H( |X)
0.8 0.6 0.4 0.2 1.0
2.0
4.0
8.0
A common variation of collapse is shown in Figure 10, where the entropy initially rises and then falls. This supports the interpretation that some dimensions are useful during training yet ultimately prove redundant for reconstruction at convergence, so are regularised out of existence.
16.0
Figure 8: Venn diagram represented as a stacked graph, varying 𝛽. H(𝑿) stays constant, while H(𝝁) decreases for larger 𝛽. Results are scaled by the joint entropy H(𝑿, 𝝁), adding to 1.
test train
0.4
H( )
Stacked Entropy
0
Figure 9: Typical active & passive variables on smallNORB at 𝛽 = 2.0.
1.0
0.0
200
0.3 0.2 0.1
6.3. Polarised Regime Using the threshold 𝐻(𝝁) > 𝜏, we observe a clear separation between active and passive variables throughout training. Representative examples from smallNORB with 𝛽 = 2.0 are shown in Figure 9. Active variables begin with moderate entropy which increases as training progresses, stabilising at a high value (Figure 9a). Passive variables begin low before collapsing rapidly to near-zero entropy (Figure 9b). The early fluctuations are due to random initialisation and the early stages of optimisation before the latents stabilise. P. Clapham et al.
0.0
0
50
100 Epoch
150
200
Figure 10: Alternative passive variable on smallNORB, 𝛽 = 2.0.
After convergence, the marginal entropy distribution shows a sharp division between a small set of high-entropy active dimensions and a cluster of near-zero passive dimensions (Figure 11). This separation is consistent across seeds and datasets, and shows a strong distinction between active variables and passive variables. Page 9 of 13
Entropy-Based Polarised Regime 1.6
active passive
1.2
0.12
H(zi)
0.14
1.0
H(zi)
active passive
0.16
1.4
0.8
0.10 0.08 0.06
0.6
0.04
0.4
0.02 0.00
0.2
1
2
3
4
5
6
7
8
9
10
7
8
9
10
8
9
10
Element
0.0
1
2
3
4
5
6
7
8
9
10
(a) 𝛽 = 5
Element
active passive
1.4
Figure 11: Marginal entropies of 𝝁 for smallNORB at 𝛽 = 2.0.
1.2
H(zi)
1.0
6.4. Generalisability
0.8 0.6 0.4 0.2 0.0
1
2
3
4
5
6
Element
(b) 𝛽 = 0.5 active passive
0.7 0.6 0.5
H(zi)
Applying the entropy criterion to LV-AEs reveals the same polarised structure (Figure 12). For very small 𝜆, several dimensions retain high entropy, while larger 𝜆 causes complete collapse. Only a narrow range between 1𝑒−4 and 5𝑒−4 avoids degeneration. For too small 𝜆, collapse is avoided entirely. For too large 𝜆, all latent dimensions collapse. The magnitudes are interesting here. For noncollapsed units, there is very high entropy. Conversely, collapsed units retain nothing.
0.4 0.3 0.2
6
active passive
5
0.1 0.0
1
2
3
4
H(zi)
5
6
7
Element
4
(c) 𝛽 = 0.05
3 2
Figure 13: Typical variable distributions for trained iVAEs.
1 0
1
2
3
4
5
6
7
8
9
10
Element
This satisfies the arguments made by Wang and Cunningham (2021). While some dimensions appear to have very low entropy, as is common in selective collapse, it is clear the entropy is non-zero. This contrasts with Figures 11 - 12.
(a) 𝜆 = 1.5𝑒−4 6
active passive
5
H(zi)
4 3
6.5. Downstream Tasks
2 1 0
1
2
3
4
5
6
7
8
9
10
Element
(b) 𝜆 = 3𝑒−4 Figure 12: LV-AEs at two points of collapse.
iVAEs also exhibit a polarised regime (Figure 13). Large values of 𝛽 force all latents to collapse, small values distribute entropy evenly, and intermediate values yield a clean active/passive division. Total collapse is avoided in all settings, but we still observe selective collapse consistent with a polarised regime, aligning with the results found by Bonheme and Grzes (2023b). However, Wang and Cunningham (2021) showed that posterior collapse cannot occur in identifiable VAEs. Our results help reconcile these two perspectives. While there is a clear polarised regime in sufficiently regularised models, when a unit collapses it still retains some entropy. P. Clapham et al.
To evaluate the practical utility of the learned variables, we train logistic regressors on the top 𝑛 variables ranked by entropy. For all the following figures, the left graph represents the average accuracy on a regressor using a normalised latent code and the right graph using the raw inputs. Figure 14 summarises the results for VAEs on smallNORB. When using raw latent vectors, accuracy peaks once all active variables are included. As passive variables are orders of magnitude smaller than active variables, the accuracy curve flattens despite their inclusion. After normalisation, however, accuracy continues to improve as passive variables are added. This can be seen to correct the magnitude problem, allowing the classifier to exploit any small residual variation. Similar patterns appear for the iVAE (Figure 15) and LVAE (Figure 16). L2-regularised autoencoders (Figure 17), on the other hand, exhibit an almost linear relationship as latent variables are added. This is expected since autoencoders are generally known to recover the solution of PCA (Baldi and Hornik, 1989), which does not use regularisation. Page 10 of 13
Entropy-Based Polarised Regime Beta values 4.0 8.0 16.0 0.60 0.55 0.50 0.45 0.40 0.35 0.30 0.25 0.20 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Number of latent features Number of latent features 2.0
Accuracy
Accuracy
1.0
0.60 0.55 0.50 0.45 0.40 0.35 0.30 0.25 0.20
Figure 14: VAE regression results using the top 𝑛 dimensions on smallNORB for various 𝛽.
5
3
values 0.5
1
0.1
0.05
0.01
0.9
0.8
0.8
0.7
0.7 Accuracy
Accuracy
0.9
0.6 0.5 0.4
0.6 0.5 0.4 0.3
0.3 1 2 3 4 5 6 7 8 9 10 Number of latent features
1 2 3 4 5 6 7 8 9 10 Number of latent features
Figure 15: iVAE regression performance on MNIST.
0.0001
λ values 0.0002
0.00015
0.00025
0.0003
0.9 0.8 0.8 0.7
7.1. Empirical Confirmation of Theoretical Results The empirical results in Section 6 directly confirm the relationships derived in Section 4. In particular, the observed separation between high-entropy and near-zero-entropy latent dimensions mirrors the theoretical coupling between entropy, variance, and KL minimisation. As predicted by the entropy–variance bounds, dimensions with large empirical spread exhibit high entropy and correspond to large KL contributions, while collapsed dimensions exhibit near-zero entropy consistent with convergence toward the prior. Moreover, the monotonic relationship between entropy and variance shown in Figure 4 provides empirical support for the entropy–variance inequality discussed in Section 4.1. Although no universal equality holds outside Gaussian settings, the ordering induced by entropy aligns closely with variance in practice. This confirms that entropy is not merely a heuristic proxy, but an empirically faithful reflection of the structural properties underlying the polarised regime. Similarly, the clean recovery of active/passive separation across 𝛽-VAEs supports the theoretical equivalence between KL minimisation and Bonheme’s passive condition established in Section 4.3. Dimensions with minimal KL contributions are precisely those with collapsed mean representations and near-zero entropy.
Accuracy
Accuracy
0.7 0.6 0.5
0.6
7.2. Extensions Beyond the Theoretical Analysis
0.5 0.4
0.4 0.3 0.3 0.2 0.2
1
2
3 4 5 6 7 8 Number of latent features
9
10
1
2
3 4 5 6 7 8 Number of latent features
9
10
Figure 16: LV-AE regression results on MNIST.
0.65 0.60 0.55 0.50 0.45 0.40 0.35 0.30 0.25
2.0
Beta values 4.0
8.0
16.0
0.6 Accuracy
Accuracy
1.0
0.5 0.4 0.3
1 2 3 4 5 6 7 8 9 10 Number of latent features
1 2 3 4 5 6 7 8 9 10 Number of latent features
Figure 17: L2-AE regression results on smallNORB.
In regimes where entropy distributions do not show a clear polarised regime, the separation between active and passive variables becomes ambiguous. This may occur in weakly regularised models, where all latent dimensions retain moderate entropy and no dimensions are passive. In such cases, a pattern much more like Figure 17 occurs where there is no selective collapse.
7. Discussion
While Section 4 established equivalence relationships within the classical VAE framework, the experiments extend these results in three important directions. First, the emergence of a polarised regime in LV-AEs demonstrates that probabilistic priors are not a necessary ingredient. The theoretical analysis in Section 4 relied on KLbased reasoning for VAEs; however, Section 6.4 shows that geometric volume regularisation alone is sufficient to induce entropy-based separation. This suggests that polarisation is a more general phenomenon of regularised representation learning rather than a purely variational artefact. Second, the behaviour of iVAEs provides empirical nuance to the theoretical claim that identifiable models cannot exhibit posterior collapse. While total collapse is indeed avoided, selective suppression of dimensions still occurs. The entropy criterion reveals that these suppressed units retain small but non-zero entropy, reconciling empirical behaviour with identifiability theory. This extension clarifies that identifiability constrains global collapse but does not eliminate selective polarisation. Third, the synthetic spike-and-slab experiments (Figure 5) validate the theoretical distinction between entropy and variance. Section 4 argued that entropy captures distributional structure beyond second moments; empirically, we observe precisely this effect when variance is held constant but entropy varies with mixture weight. This confirms that entropy provides strictly richer information than variance in non-Gaussian settings.
The following section elaborates on the results of the previous empirical section. P. Clapham et al.
Page 11 of 13
Entropy-Based Polarised Regime
7.3. Novel Observations The training dynamics reveal “attempted actives”, latent dimensions whose entropy initially rises before collapsing at convergence. This behaviour suggests that some variables are transiently useful during representation formation but ultimately suppressed by regularisation pressure. This dynamic phenomenon is not captured by static definitions of the polarised regime and provides empirical support for the hypothesis provided by Bonheme and Grzes (2023a) that latent variables are learned in order of reconstruction relevance, consistent with the interpretation that posterior collapse may partly reflect competitive allocation of representational capacity. The downstream experiments show that passive variables, despite having near-zero entropy, retain small but consistent predictive value when appropriately normalised. This challenges the common assumption that collapsed variables are entirely devoid of information. Instead, collapse appears to be a matter of scale rather than absolute information removal. Across architectures, latent dimensions consistently separate into high- and low-entropy subsets. This suggests that the polarised regime arises naturally from regularised optimisation, where representational capacity is allocated to dimensions with the greatest reconstruction benefit. The emergence of a polarised regime in deterministic architectures such as LV-AEs suggests that selective collapse is not unique to variational objectives but may arise more generally from regularised representation learning. This further emphasises the need for agnostic criteria.
7.4. Limitations While the proposed entropy-based criterion provides a simple, prior-agnostic criterion for identifying active and passive latent variables, it also exhibits several important limitations. Most fundamentally, the criterion relies on the existence of a well-defined polarised regime. When the latent space does not separate cleanly into active and passive subsets, entropy alone cannot provide a sharp classification. In particular, mixed variables remain difficult to resolve: although the entropy of the variance representation can highlight switching behaviour in principle, the entropy of the mean representation cannot distinguish whether a variable is active or mixed as there will be high uncertainty in the mean representation in both modes. The threshold 𝜏 used to classify activity is also heuristic. Although the bimodality of entropy distributions tends to provide a clear value in polarised regimes, there is currently no principled, universally optimal choice of 𝜏 that transfers across datasets, architectures, and training objectives. This makes the choice of 𝜏 reliant on each practitioner’s experimental setup. From a modelling perspective, the empirical evaluation is restricted to image datasets and convolutional autoencoder architectures. While the entropy criterion is agnostic to architectural details in principle, it remains an open question
P. Clapham et al.
whether the same behaviour will emerge in sequential models (Bowman et al., 2016), diffusion-based architectures, etc. The broad applicability of the proposed criterion also makes exhaustive empirical evaluation difficult. While the method is architecture-agnostic in principle, the present study is necessarily limited to a subset of model classes and datasets. Finally, although passive variables were shown to contribute modestly to downstream tasks after normalisation, the gains are small. This limits the practical utility of selectively including variables by entropy, and suggests that the primary value of the entropy criterion lies in diagnostic analysis and representation understanding rather than in performance optimisation. Together, these limitations clarify that entropy-based activity should be understood as a principled descriptive tool for analysing latent structure, rather than as a complete solution to latent variable selection in all regimes.
8. Conclusion We introduced an entropy-based criterion for latent activity based on the mean representation of the encoder. Unlike KL-based thresholds tied to Gaussian priors, the proposed criterion applies across both variational and deterministic architectures, enabling a more general characterisation of the polarised regime. Theoretical analysis showed that entropy is closely related to existing notions of active variables. In particular we demonstrated that entropy is coupled to KL minimisation through entropy–variance inequalities, aligning it with existing definitions of active and passive dimensions while remaining sensitive to distributional structure beyond second moments. This places entropy in a useful middle ground: it is more broadly applicable than prior-specific heuristics, yet still grounded in the mechanisms that give rise to the polarised regime. Empirically, we found that entropy behaves coherently with mutual information and variance, and reliably recovers the polarised regime when such structure is present. This held not only for 𝛽-VAEs, but also for identifiable VAEs and deterministic autoencoder variants, including LVAEs. These results suggest that the polarised regime is not confined to standard variational objectives, but can emerge more generally from regularised representation learning. We further observed that dimensions with very low entropy can retain small but consistent downstream utility when latent codes are appropriately normalised, indicating that collapse is often a matter of scale rather than complete information removal. At the same time, the proposed criterion has clear limitations. Most notably, entropy of the mean representation alone cannot cleanly distinguish active from mixed variables, and the threshold used to separate active and passive dimensions remains heuristic. More broadly, the present experiments were restricted to image-based architectures and a limited family of latent-variable models. Future work should therefore aim to develop more principled criteria Page 12 of 13
Entropy-Based Polarised Regime
for mixed-variable detection, and to test whether the same information-theoretic picture persists in broader generative settings. Taken together, these results suggest that the polarised regime should be understood not merely as a pathology of Gaussian variational inference, but as a broader phenomenon of regularised representation learning. From this perspective, entropy is not just an alternative diagnostic, but a more general language for describing when and how latent dimensions become informative or are suppressed.
Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this work, the author(s) used ChatGPT to assist with drafting and editing text. After using this tool, the author(s) reviewed and edited the content as needed and take full responsibility for the content of the publication.
Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Funding This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) through a Doctoral Training Partnership.
Disclaimer Where authors are identified as personnel of the International Agency for Research on Cancer/World Health Organization, the authors alone are responsible for the views expressed in this article and they do not necessarily represent the decisions, policy or views of the International Agency for Research on Cancer/World Health Organization.
Data availability The datasets used in this study are publicly available from their original sources.
References Baldi, P., Hornik, K., 1989. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks 2, 53–58. Bengio, Y., Courville, A., Vincent, P., 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1798–1828. Bonheme, L., Grzes, M., 2023a. Be more active! understanding the differences between mean and sampled representations of variational autoencoders. Journal of Machine Learning Research 24, 1–30. URL: http://jmlr.org/papers/v24/21-1145.html.
P. Clapham et al.
Bonheme, L., Grzes, M., 2023b. The polarised regime of identifiable variational autoencoders, in: The First Tiny Papers Track at the International Conference on Learning Representations 2023. URL: https: //openreview.net/forum?id=iSkcAjBqUHU. Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Józefowicz, R., Bengio, S., 2016. Generating Sentences from a Continuous Space, in: CoNLL. Chen, Q., Fuge, M., 2024. Compressing latent space via least volume, in: ICLR. Clapham, P., Grzes, M., 2023. Posterior collapse in variational gradient origin networks. 2023 International Conference on Machine Learning and Applications (ICMLA) , 980–987. Cover, T.M., Thomas, J.A., 2012. Elements of Information Theory. 2nd ed., Wiley-Interscience, Hoboken, New Jersey. Dai, B., Wang, Z., Wipf, D., 2020. The Usual Suspects? Reassessing Blame for VAE Posterior Collapse, in: ICML. Dai, B., Wipf, D., 2019. Diagnosing and enhancing VAE models, in: International Conference on Learning Representations. URL: https: //openreview.net/forum?id=B1e0X3C9tQ. Higgins, I., Matthey, L., Pal, A., Burgess, C.P., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A., 2017. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, in: ICLR. Jolliffe, I.T., 2002. Principal Component Analysis. 2nd ed., Springer, New York. doi:10.1007/b98835. Khemakhem, I., Kingma, D.P., Hyvärinen, A., 2019. Variational autoencoders and nonlinear ica: A unifying framework, in: International Conference on Artificial Intelligence and Statistics. URL: https://api. semanticscholar.org/CorpusID:195874364. Kingma, D.P., Welling, M., 2014. Auto-encoding variational bayes, in: International Conference on Learning Representations (ICLR). LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324. LeCun, Y., Huang, F.J., Bottou, L., 2004. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. 2, II–104 Vol.2. Lee, S., Jo, J., 2021. Information flows of diverse autoencoders. Entropy 23, 862. URL: http://dx.doi.org/10.3390/e23070862, doi:10.3390/ e23070862. Locatello, F., Bauer, S., Lucic, M., Gelly, S., Schölkopf, B., Bachem, O., 2019. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. ArXiv abs/1811.12359. Lucas, J., Tucker, G., Grosse, R., Norouzi, M., 2019. Understanding Posterior Collapse in Generative Latent Variable Models, in: DGS@ICLR. Matthey, L., Higgins, I., Hassabis, D., Lerchner, A., 2017. dSprites: Disentanglement testing Sprites dataset. https://github.com/deepmind/dsprites-dataset/. Rolínek, M., Zietlow, D., Martius, G., 2019. Variational autoencoders pursue pca directions (by accident), in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12398–12407. doi:10.1109/CVPR.2019.01269. Rényi, A., 1961. On measures of entropy and information, in: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, University of California Press. pp. 547–561. Shannon, C.E., 1948. A mathematical theory of communication. Bell System Technical Journal 27, 379–423, 623–656. Takida, Y., Liao, W.H., Uesaka, T., Takahashi, S., Mitsufuji, Y., 2021. Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE. ArXiv abs/2102.08663. Wang, Y., Cunningham, J.P., 2021. Posterior collapse and latent variable non-identifiability, in: Third Symposium on Advances in Approximate Bayesian Inference. URL: https://openreview.net/forum?id= Kr8zEOm-lO. Wang, Z., Ziyin, L., 2022. Posterior Collapse of a Linear Latent Variable Model. ArXiv abs/2205.04009. Yu, S., Príncipe, J.C., 2018. Understanding autoencoders with information theoretic concepts. Neural networks : the official journal of the International Neural Network Society 117, 104–123.
Page 13 of 13