Disentanglement with Holographic Reduced Representations
arXiv:2606.09725v1 [cs.LG] 8 Jun 2026
Jhonny J. Velasquez Olivera Virginia Tech [email protected]
Christo K. Thomas Worcester Polytechnic Institute [email protected]
Walid Saad Virginia Tech [email protected]
Abstract The process of disentanglement, that focuses on separating the factors of variation in a dataset using neural networks, is a long standing challenge in machine learning. Prior solutions to this problem included the design of variational autoencoders and generative adversarial networks that use concepts from variational inference and information-theoretic constraints in their loss functions, respectively. In contrast to these prior works that rely on continuous representations in their models, we propose a new design that naturally views disentangled representations as symbolic structures due to the compositional nature of the relationship between the concepts that make up a sample from a distribution. However, learning discrete symbolic structures using traditional neural network architectures while maintaining differentiability is challenging, often requiring complex architectures to accomplish. To this end, in this paper, we propose an unsupervised learning algorithm that makes use of holographic reduced representations (HRR) for disentanglement using neural networks. We find that the unbinding operation, defined over HRR vectors, yields a suitable inductive bias for the separation of factors, and yields competitive results compared to other baselines, as measured using latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs (also called slots) and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts the representation can reliably encode, providing a quantitative account of the inductive bias toward disentanglement. The disentangled representations produced in this process differ from other autoencoder based models in that the individual latent units are vectors themselves which are summed together, differing from the paradigm of latent units behaving as scalar dimensions of low dimensional vectors. We show that this type of HRR representation is more robust to noise than other disentangled representations and can maintain good reconstruction performance across a range of SNRs, while simultaneously gracefully degrading in the recoverable semantic content. Our results thus show that the use of vector symbolic architectures (VSA), such as HRRs, may hold a promising potential for representation learning, paving the way toward exploring new ways in which the symbolic benefits of VSAs can be used to represent data with neural networks.
1
Introduction
The pursuit of learning efficient representations for data has been of interest to the deep learning field for many years, and led to the emergence of the field of representation learning [1]. Understanding representations of data within neural networks can shed light into how to construct better inductive biases to train them, and provide new insights into how neural networks interact with the data they use. Disentangled representation learning is a popular approach to the more general representation Preprint.
learning problem whose focus is on how to disentangle the generative factors that make up a dataset. Perhaps the earliest work on this was a more heavily regularized variational autoencoder (VAE) [2], called the β-VAE [3]. Following the β-VAE, a line of work emerged that attempted to further improve on its disentanglement abilities leading to works such as the total correlation (TC) VAE [4], and the Factor-VAE [5], each of which build on the joint reconstruction and Kullback Leibler (KL) divergence objectives of the original VAE. Alongside these models, a parallel line of work proposed more comprehensive metrics for evaluating disentanglement [6, 7]. Most prior art in disentanglement has followed the VAE framework, with the exception of InfoGAN, an information-theoretic extension of generative adversarial networks (GANs) [8, 9]. Recently, there has also been evidence to show that disentanglement can arise using discrete representations [10], building on previous evidence that neural networks can learn discrete representations, in general [11]. However, disentanglement does not arise on its own. Without inductive biases on both the data and the model, unsupervised disentanglement is impossible [12]. This points to the inductive bias of each method as the main driver of its disentanglement quality. Vector symbolic architectures (VSAs) offer a structurally different inductive bias, namely a compositional vector space, but their potential for unsupervised disentanglement remains largely unexplored. A prior work explored this intersection and was able to disentangle generative factors using VSAs [13]. However, the training procedure required carefully constructed data pairs that differ in only one generative factor at a time, limiting the solution of [13] to datasets that span a sufficient number of factor combinations. A central challenge to using VSAs in representation learning is the difficulty of mapping highdimensional data to the structured vector spaces that VSAs provide. Mapping data to vectors using VSAs is traditionally done using hand designed methods that attempt to capture the structures apparent in the raw data [14–17]. For example, one method of mapping images to a VSA vector involves representing the image as a sum of many vectors, each representing pixel intensity and position [17]. While such methods are interpretable, they can only encode structure that the designer anticipates in advance, leaving little room for the model to discover new representations from data. On the other hand, neural networks have been shown to be able to learn strong representations of data that can be used for downstream tasks such as classification [18–25]. However, using neural networks to learn VSA representations directly from data remains a relatively unexplored field. Some works have trained neural networks to map data into VSAs [26–29], but these typically rely on labeled training data [27], generalize poorly, and leave little room for the model to discover its own structure. Despite the rich literature on disentanglement and the long history of VSAs as a representation framework, there is no existing method that learns HRR-structured latent representations from data without supervision, and no existing theory that explains why an HRR structure should produce disentanglement. This paper closes this gap. The main contribution of this paper is to demonstrate, both empirically and theoretically, that holographic reduced representations (HRRs) provide an effective inductive bias for unsupervised disentanglement, achieving competitive performance on standard benchmarks while admitting a quantitative information-theoretic account of why the bias works. In particular, our key contributions include: • We show that training a neural network with a reconstruction objective, paired with a latent reconstruction process and an HRR structural regularizer, produces disentangled representations of the generative factors of a dataset without the use of any labels, unlike prior work on learning VSA representations from data [13]. • We derive a theoretical upper bound on the mutual information between the input and the HRR-structured latent, quantifying how many distinct symbolic concepts the representation can reliably encode. This bound provides a principled basis for choosing the architectural parameters HRR vector dimension, number of symbols, and the latent codebook size. • Experimental results show that the resulting representations achieve the best aggregate scores on five of six standard disentanglement metrics (InfoMEC and DCI) across four datasets, with relative improvements of roughly 5 − 10% over the next-best baseline on the core disentanglement metrics, while exhibiting noise robustness comparable to VQVAE and considerably better than other disentanglement methods. HRR is the only model that achieves both superior disentanglement and strong noise robustness on the studied benchmarks. Qualitative analyses of latent component swapping further confirm that the learned representation aligns with individual generative factors. 2
2
Preliminaries of HRR vectors
HRRs are a VSA that represents discrete entities, called symbols, as high-dimensional random vectors, and represents structured combinations of those entities through algebraic operations on the vectors. Based on [30], an HRR vector x ∈ Rd is typically initialized with entries sampled independently as xi ∼ N (0, 1/d), so that distinct randomly initialized vectors are approximately orthogonal for large d. HRRs define two main operations, binding and bundling. Binding, denoted ⊗, associates two vectors through circular convolution, producing a representation that is approximately dissimilar to either input. In practice, this operation can be computed efficiently as multiplication in the frequency domain. Bundling, denoted ⊕, superimposes vectors through addition and is commonly used to represent sets or compositions of symbols. A useful property of HRRs is approximate unbinding. Given a bound representation z = x ⊗ y, one can recover a noisy estimate of x by applying an approximate inverse of y : x̂ = y† ⊗ z ≈ x. The approximation error is often called unbinding noise. Here, “inverse” follows the terminology of the HRR literature rather than denoting an exact algebraic inverse; we defer further details to the appendix and refer readers to prior work on HRR inverses [27], and the original HRR paper [30].
3
HRRs for disentanglement
Disentangled representations aim to capture the structure within a dataset. Although continuous representations, and more recently discrete representations [10, 31], have been shown to separate some of this structure, we will show that symbolic representations may be better suited for this task. Symbolic structures are useful for representing data with compositional structure, where entities, attributes, and relations combine in meaningful ways. For example, consider an image of a dog in a backyard. Although the image is represented at the pixel level, we naturally interpret the scene in terms of entities, attributes, and relations. There is a dog, the dog is in a backyard, and the dog has attributes such as breed, age, color, and size. This interpretation can be organized as a simple symbolic structure, where the root node captures the main entity, dog, and its associated attributes are represented as role-filler pairs such as breed: Chihuahua or size: small. In a VSA, this kind of structure can be encoded by binding each role vector to its corresponding filler vector and bundling the resulting pairs into a single distributed representation. While this example uses only one level of role-filler binding, more complex hierarchies can be expressed through recursive composition [30]. The flat latent representations used in prior continuous and discrete disentanglement work [10, 31] do not naturally express this kind of compositional structure. Here, we present a proof of concept that a neural network can be trained to produce such structured vector representations directly from data. HRRs allow us to impose such a structure on a neural network’s latent space using a hetero-associative memory, where pairs of vectors are stored such that one element of the pair can be retrieved using the other as a cue. We define an HRR structured representation as a vector in the form z = (s1 ⊗ v1 ) ⊕ · · · ⊕ (sm ⊗ vm ). Each si and vi is an HRR vector, where ⊗ and ⊕ are the binding and bundling operations over HRRs. We refer to the si as symbol vectors, which act as the cues to the memory and to vi as value vectors, which act as the items stored in the memory. We refer to the number of symbol-value pairs m as the symbolic length of the representation. The encoder is trained to produce z from the data x. Ideally, the model learns to associate each si with the most important generative factors of the data. For example, on images of chairs, s1 could represent chair type (sofa, stool, recliner, etc.), s2 chair pose, and s3 chair size; the corresponding vi would then encode the specific value of each factor (e.g., v1 for “sofa"). Each bound Lm pair si ⊗ vi thus represents the realization of one generative factor, and the bundled sum z = i=1 si ⊗ vi forms a neuro-symbolic representation of x. The symbolic length m also controls how finely the model can separate generative factors. A larger m provides more symbol-value pairs in which to place distinct factors, while a smaller m forces the model to group factors together. 3.1
Model architecture
Fundamentally, our model is a CNN based autoencoder that learns to map the data, x, directly to the latent representation, z, with an HRR unbinding and vector quantization step. Along with the autoencoder we initialize a set of m symbol vectors, S = {s1 , s2 , . . . , sm }, and a codebook of 3
k ≫ m value vectors, C = {c1 , c2 , . . . , ck }. The vectors in S are randomly initialized from the HRR distribution described in Section 2 and remain frozen throughout training. The codebook C is also initialized randomly but not from the HRR distribution as the codebook is optimized throughout the training process. The encoder first predicts the latent representation, z, from the data. From the latent representation, we retrieve the vectors in the hetero-associative memory through the unbinding operation defined in Section 2 by binding the latent once with the inverse of each of the vectors in S. This retrieval produces the noisy value vectors {ṽ1 , ṽ2 , . . . , ṽm }, which each get processed by a small feedforward neural network, referred to as a denoising network, to produce {v̄1 , v̄2 , . . . , v̄m }. From here, we do vector quantization as described in [11] to map each v̄i to its nearest Euclidean neighbor in the codebook C, resulting in the final set of value vectors {v̂1 , . . . , v̂m }, where each v̂i ∈ C. We will write V̄ ∈ Rm×d and C ∈ Rk×d for the matrices whose rows are the denoised retrievals and the codebook entries, respectively. We bind each v̂i with the corresponding symbol vector that it originally corresponded with to construct ẑ = (s1 ⊗ v̂1 ) ⊕ · · · ⊕ (sm ⊗ v̂m ). The decoder then uses ẑ to reconstruct the original input x. 3.2
Regularization and optimization
For a matrix X ∈ Rn×d with rows xi , we define the structural regularizer !2 n 1X 2 2 2 R(X; τn , τv ) = ∥xi ∥ − τn + (mean(X)) + (Var(X) − τv ) , n i=1
(1)
where mean and Var are computed globally over all entries. This regularizer pushes a matrix toward zero mean, target per-row squared norm τn , and target per-entry variance τv , matching the statistical properties expected of HRR vectors and their bundled compositions (Appendix C). We apply R to three different matrices, with targets chosen so that each matches the relevant HRR distribution, as shown in Table 1. Table 1: Specifications of the three structural regularizers. Term
X
n
τn
τv
Llatent Lvalue Lcodebook
⊤
1 m K
m 1 1
m/d 1/d 1/d
z V̄ C
The targets τv = 1/d for V̄ and C reproduce the per-component variance of HRR vectors (Section 2); the targets for z (τn = m, τv = m/d) match the expected squared norm and per-component variance of a sum of m bound HRR pairs (Appendix C). Because the vector quantization step is non-differentiable, we use the straight-through estimator [32] to propagate gradients through it. Loss function The model is trained with an unsupervised auto-encoding objective, using the AdamW optimization algorithm [33]. The total loss combines a binary cross-entropy reconstruction loss Lrecon , the vector quantization (VQ) and commitment losses from the original VQ-VAE [11], and the three structural regularizers defined above. Following [11], we define LVQ = ∥sg[v̄] − c||2 , Lcommit = ∥v̄ − sg[c]||2 , where sg[·] is the stop gradient operator, v̄ is the denoised retrieval, and c is the nearest Euclidean neighbor in the codebook V. LVQ allows gradients to reach the codebook, while Lcommit allows gradients to reach the encoder. Both terms ensure the codebook and the encoder work together to converge on mutually optimal codebook entries and latent predictions, respectively. The total objective is (2) L = Lrecon + λvq LVQ + λc Lcommit + λlat Llatent + λval Lvalue + λcb Lcodebook , {z } | {z } | regularization
reconstruction
where all λ ≥ 0 are scalar weights. The hyperparameters used for all runs are listed in Table 5. 3.3
Capacity of the HRR latent channel
Having specified the model and its training objective, we now turn to an information-theoretic analysis of the HRR latent channel, which clarifies how much information the symbolic latent ẑ can carry 4
about the input x and motivates the design choices made in our experiments. In what follows, we refer to the i-th symbol-value pair (si , vi ) as the i-th slot. The unbinding analysis in Appendix C shows that, when retrieving the value vector bound to slot i, the result decomposes as X (k) ṽi = (1 + ξ) vi + η i + λi , (3) k̸=i (k) with intrinsic noise η i and cross-talk λi from the other m − 1 bound pairs. Treating ξ → 0 in
expectation, the per-component signal power is 1/d and the per-component noise power is m/d, giving an SNR of 1/m per dimension. Each retrieval thus passes through d parallel AWGN subchannels, each with the same SNR. We use this to bound the total information the symbolic latent ẑ can carry about the input x. Pm Proposition 1. Let ẑ = i=1 si ⊗ v̂i be the symbolic latent, where each v̂i is the quantized value selected from a codebook of size k. Under the unbinding noise model of Appendix C, the mutual information between input and latent satisfies 1 I(x; ẑ) ≤ min m · d2 log 1 + m , m log k . (4)
Proof. The proof proceeds in four steps. The data-processing inequality reduces I(x; ẑ) to a sum of per-slot mutual informations once approximate slot independence (Proposition 2) is established. Each per-slot term is then bounded by (a) the capacity of the d-dimensional AWGN unbinding channel, d 2 log(1 + SNR) with SNR = 1/m, and (b) the codebook entropy log k. Taking the minimum of the two gives the bound. See Appendix D for the full proof. Slot–dimension tradeoff
The continuous part of the bound in (4) is md log(1 + 1/m), 2
or equivalently d2 log(1 + 1/m) per slot. Thus, increasing the number of slots m provides more slots, but also lowers the effective per-slot SNR because each retrieved slot receives cross-talk from the other slots. For large m, log(1 + 1/m) ≈ 1/m, so the total continuous capacity approaches d2 . In this regime, adding more slots does not substantially increase the total amount of information that can be carried by the HRR latent, rather, it primarily divides the available capacity across more noisier retrieval channels. This formalizes the empirical observation that unbinding performance degrades as m increases at fixed d (Appendix E). It also motivates using a modest latent-to-source overcomplete ratio in our experiments, as the number of slots should be large enough to cover the generative factors, but not so large that each slot’s effective retrieval channel becomes dominated by cross-talk noise. This motivates the 1.5× latent-to-source ratio used in Section 4.1. Codebook sizing The second term, m log k, is the maximum discrete rate available after quantization. When d log k < log(1 + 1/m), 2 the codebook resolution is the active bottleneck, and increasing k can increase the representational rate. Once d log k ≥ log(1 + 1/m), 2 the HRR retrieval channel becomes the bottleneck, so further increasing k cannot increase the leading-order capacity bound. The codebook perplexity curves in Figure 6 are consistent with this interpretation. Harder datasets such as Isaac3D and MPI3D-C drive higher codebook perplexity, suggesting that these datasets require more discrete states per slot and are therefore more likely to operate near the codebook-resolution constraint. 3.4
Approximate slot independence
We write q(ṽ1 , . . . , ṽm | x) for the joint distribution of the unbound retrievals conditional on an input x, where the randomness is induced by the HRR initialization of the symbol vectors si ∈ S. 5
Although these symbols are fixed after model initialization, treating them as random in the analysis allows us to quantify the typical dependence induced by HRR cross-talk. The proof of Proposition 1 uses the fact that the retrieved slots become approximately independent (k) in high dimension. This dependence is not exactly zero at finite d: the cross-talk term λi in (3) † couples slot i to slot k through products of the form si ⊗ sk ⊗ vk . However, the resulting off-diagonal covariance blocks are small. In particular, Appendix D.2.2 shows that for i ̸= j, 1 (vj ⊗ vi )a+b , d so the off-diagonal covariance is generated by a circular convolution of independent HRR vectors. Consequently, the aggregate off-diagonal covariance vanishes with d. Proposition 2 (Approximate slot independence). Let q(ṽ1 , . . . , ṽm | x) be the joint distribution of unbound retrievals under the HRR noise model of Appendix C, and let qi (ṽi | x) be its marginals. With posterior variance floor t2 > 0, 2 Qm m DKL q(Ṽ1:m | x) (5) i=1 qi (ṽi | x) = O t4 d , (Σij )ab =
which vanishes as d → ∞ at fixed m and fixed t > 0. Proof. The chain rule identifies the KL divergence above with the total correlation among the retrieved slots, Pm Q DKL q(Ṽ1:m | x) i qi (ṽi | x) = i=2 I(ṽi ; Ṽ<i | x). We bound this total correlation directly rather than bounding each conditional mutual information separately. Under the Gaussian approximation to the joint retrieval law, the KL between the joint distribution and the product of its marginals has a closed-form log-determinant expression. Writing Σ̄ for the block-diagonal covariance obtained by removing the off-diagonal covariance blocks, and ∆ = Σ − Σ̄ for the off-diagonal part, the trace term cancels because ∆ has zero diagonal blocks. After block-diagonal whitening, the KL reduces to a log-determinant involving M = (Σ̄ + t2 I)−1/2 ∆(Σ̄ + t2 I)−1/2 . The posterior variance floor gives ∥(Σ̄ + t2 I)−1/2 ∥op ≤ t−1 , while the HRR off-diagonal block estimates give ∥∆∥2F = O(m2 /d). Therefore ∥M∥2F = O(m2 /(t4 d)). A second-order expansion of − log det(I + M), together with the small-operator-norm condition on M, then yields the stated bound. See Appendix D.1 for the full derivation. Together, Propositions 1 and 2 give a partial answer to the question of why the HRR structure favors disentangled representations. Two mechanisms act on the encoder. First, slots are statistically near-independent under unbinding (Proposition 2), so the encoder can optimize each slot largely independently of the others, rather than coping with cross-slot noise correlations. Second, attaining the capacity bound of Proposition 1 requires non-redundant slot usage by placing distinct generative factors in distinct slots. Quantization then discretizes each slot’s content and pushes the encoder toward a finite, factor-aligned set of codes.
4
Experimental results and analysis
In the experiments, we will quantify the disentanglement performance of our proposed HRR model compared to previous models proposed for disentanglement, show that different slots in our representation correspond to different generative factors, and show the robustness to noise of our HRR representation. All together, these results demonstrate that HRR-inspired representations can achieve good disentanglement quality while offering a robustness to noise not seen in prior disentanglement works. 4.1
Disentanglement performance
We qualitatively measure our model’s disentanglement performance using the InfoMEC metrics, proposed in a prior work [10], because it uses information-theoretic metrics, mutual information, to 6
Table 2: Disentanglement scores over n = 5 seeds. InfoMEC cells show (InfoM InfoE InfoC); bold marks the best per column using the CI-overlap criterion. Aggregated is the mean across all datasets. Model
Aggregated
Shapes3D
Falcor3D
Isaac3D
MPI3D-C
(0.54 0.56 0.46) (0.63 0.62 0.57) (0.70 0.86 0.52) (0.66 0.81 0.49) (0.73 0.89 0.55)
(0.37 0.23 0.29) (0.41 0.42 0.39) (0.49 0.82 0.53) (0.56 0.78 0.48) (0.51 0.83 0.53)
(0.28 0.25 0.86) (0.29 0.26 0.86) (0.64 0.36 0.91) (0.59 0.31 0.92) (0.68 0.36 0.92)
(0.29 0.28 0.79) (0.12 0.12 0.73) (0.49 0.34 0.73) (0.51 0.29 0.78) (0.55 0.34 0.77)
InfoMEC := (InfoM InfoE InfoC) ↑ β-VAE β-TCVAE VQ-VAE QLAE HRR (ours)
(0.54 0.57 0.45) (0.56 0.63 0.51) (0.57 0.88 0.47) (0.65 0.84 0.49) (0.68 0.88 0.54)
(0.59 0.81 0.47) (0.61 0.83 0.51) (0.58 1.00 0.44) (0.77 1.00 0.52) (0.85 1.00 0.61)
(0.64 0.68 0.59) (0.59 0.65 0.55) (0.50 0.86 0.40) (0.61 0.79 0.46) (0.63 0.81 0.45)
β-VAE β-TCVAE VQ-VAE QLAE HRR (ours)
(0.31 0.28 0.85) (0.31 0.29 0.85) (0.62 0.34 0.86) (0.59 0.31 0.88) (0.67 0.37 0.89)
(0.45 0.38 0.97) (0.40 0.33 0.95) (0.71 0.35 0.96) (0.76 0.35 0.99) (0.88 0.41 1.00)
(0.23 0.22 0.77) (0.44 0.43 0.86) (0.63 0.33 0.83) (0.50 0.28 0.84) (0.58 0.37 0.85)
DCI := (D I C) ↑
measure disentanglement quality as opposed to using feature importance from learned classifiers and regression models, as done in DCI, an older disentanglement metric [7]. However we still report the DCI scores to complement our InfoMEC scores. For consistency we use the same baselines that were proposed [10]. Specifically, our baselines include, the β-VAE [3], the TC-VAE [4], the VQ-VAE [11], and QLAE [10]. Consistent with [10], we use Shapes3D [34], MPI3D [35] (specifically the complex real world shape variant), and NVIDIA’s Falcor3D and Isaac3D datasets [36]. Consistent with prior works[10], we sweep a single hyperparameter for each (model, dataset) pair and use the best performing hyperparameter over two seeds. For more details on the sweep, we refer to Table 6 of Appendix E. We note the best performing hyperparameter and re-run each model on each dataset with its best hyperparameter 5 times from which we construct a confidence interval. As is consistent with other disentanglement works, we train on the entire dataset without holding out a validation or test set [12]. The purpose of this is that, in disentanglement, we are mostly concerned with how the model can learn to disentangle the different latent factors, not combating overfitting. For each run, we sample a random subset of the dataset (without removing it from the training data) that we use to evaluate the disentanglement metrics. To ensure fairness, each model uses the same encoder and decoder architecture, and the same optimization parameters, consistent with the work of [12]. For each dataset, each model is initialized with 1.5× latent units as there are sources. More details, such as model architecture and optimization hyperparameters can be found in Table 3 and Table 4 in Appendix E. Details on the bolding criteria for the results can also be found in Appendix E. As seen in Table 2, our HRR based disentanglement method outperforms most baselines across the InfoM metric, only surpassed by QLAE on the MPI3D-C dataset, with competitive performance on Falcor3D with β-VAE. Our model is also able to match or exceed baselines on InfoE and InfoC scores all datasets except on Falcor3D for the InfoE and InfoC metrics, where the VQ-VAE and β-TCVAE reach the highest scores, respectively. However, in aggregate, our method outperforms, or meets, other baselines across all metrics, providing a 4.6% improvement in InfoM, and 5.9% improvement in InfoC, over the second highest scores in the respective categories. The pattern across metrics is consistent with the analysis of Section 3.3. HRR’s strongest gains are on InfoM (modularity), where the per-slot independence induced by unbinding (Proposition 2) directly biases the encoder toward placing distinct factors in distinct slots. The smaller gains on InfoE and InfoC, particularly on Falcor3D where our proposed model is outperformed by VQ-VAE and β-TCVAE, indicate that explicitness and compactness are governed less by the slot structure and more by the choice of decoder and codebook size. 4.2
Latent robustness to noise when decoding
Another quantity that we measure is the robustness to noise of the different latent representations. This provides insight into how much a latent representation could be corrupted by external noise while still being able to be accurately decoded. We posit that robust representations will play an important role in the upcoming era of physical AI, as we deploy machine learning algorithms into the real world. Research on representations has largely taken place in controlled settings, and has ignored the challenges associated with real world use. Noisy representations will be an important 7
Latent noise robustness (Gaussian): PSNR vs. SNR Shapes3D
Falcor3D
35 25 20
PSNR (dB)
30
20 15
15 10
10 5 20
10
0
SNR (dB)
10
20
5
MPI3D-C 35
40
25
PSNR (dB)
PSNR (dB)
Isaac3D 45
30
40
30
35
PSNR (dB)
45
30 25 20
10
10
HRR (ours)
10
0
SNR (dB) QLAE
10
20
20 15
15 20
25
20
-TCVAE
10
0
SNR (dB) -VAE
10
20
20
10
0
SNR (dB)
10
20
VQ-VAE
Figure 1: Average performance of each model, as measured by PSNR, under varying levels of Gaussian noise intensity. The gap between HRR and VQ-VAE is small across most datasets, and both models perform considerably better than the other baselines. Shaded regions represent a 95% confidence interval. challenge to overcome, and should be learned in such a way that they can tolerate corruption. For instance, two embodied agents wishing to collaborate should be able to understand each other despite corruption in a message, especially in critical scenarios. Robust representations are also of interest in latent-based world models, since preserving the semantic content of internal representations is crucial for their performance. A more robust representation could allow for more error toleration when making predictions in latent space since the result may still be able to be decoded accurately. To this end, we measure the reconstruction peak-signal-to-noise-ratio (PSNR) of representations under zero mean Gaussian noise, at varying signal-to-noise (SNR) ratios. We apply noise to the latent representation from +20 dB SNR down to -20 dB SNR and measure the PSNR between the resulting reconstruction and the input. As seen in Fig. 1, the proposed HRR model and VQ-VAE models both outperform the other baselines, with the former performing the best. This is because VQ-VAE’s optimization allows for codebook entries to spread apart in Euclidean space. A noise perturbation must therefore be large enough to push a quantized vector closer to a different codebook entry before the discrete code changes, leaving the recovered code unchanged for moderate noise levels. A visualization of the reconstruction performance can be seen in Fig. 2, in which we see that the HRR and VQ-VAE both are able to maintain their semantic content under most noise intensities. We include more results for other datasets in Appendix I.
8
Latent noise robustness Original
Clean Recon
+20dB
+10dB
+5dB
Gaussian +0dB
-5dB
-10dB
-20dB
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 2: Each row is a visualization of how reconstruction quality degrades as noise intensity increases for each model. For each model, the seed with the highest InfoM score was used for evaluation. The HRR model and VQ-VAE maintain their semantic content the best. Signal power is estimated over the entire validation set. 4.3
Latent component swapping
Because of the vector quantization process that assigns value vectors to each slot in our latent variables, traditional latent traversals (where scalar dimensions are varied through an interval) cannot be used as a fair qualitative method of assessment for our method. Instead, we opt to show the results of what happens when you swap latent units between two representations from the dataset, i.e, inject a latent factor from one representation into the other. For truly disentangled representations, it would be reasonable to expect a degree of modularity (in the qualitative sense, not the quantitative one used as a metric) between the representations, such that the latent units are compositional. Fig. 3 compares latent component swaps for HRR and other baselines. We make the following observations here. First, HRR’s swaps consistently modify a single generative factor at a time while leaving others intact. Where QLAE’s swaps occasionally produce visible artifacts in unrelated factors, HRR maintains the unswapped attributes more cleanly. Second, HRR preserves global scene semantics across nearly all swap positions, while QLAE occasionally alters the orientation of objects during a swap. The concentration of changes within a single generative factor follows from the approximate slot independence established in Proposition 2. If the per-slot retrievals were tightly coupled, swapping one slot would propagate changes across others, but HRR’s slot-by-slot swapping remain contained. From Fig. 3, we can make two key observations. First, these results provide qualitative support for Proposition 2. If the per-slot retrievals were tightly coupled, swapping one component would propagate changes across the others, but HRR’s swaps largely preserve unrelated factors. Second, the qualitative gap with QLAE is informative because both methods quantize their latents but only HRR imposes a symbolic binding structure, suggesting that the binding-and-bundling architecture may be important to enable single-factor edits. 4.4
Latent interpolation
A natural extension of component swapping is to swap slots cumulatively rather than individually. Starting from a source image’s latent z1 , we progressively replace slots with the corresponding slots from a target image’s latent z2 . At step i, slots 1 through i have been swapped from z2 . For a disentangled representation, the decoded sequence should walk from the source image to the target one factor at a time, with each step changing only the most recently swapped factor. Figure 4 visualizes this for the HRR-based model and all the baselines, evaluated on the same image. Cumulative swapping is a more stringent test of slot independence than the single-component swap experiment of Section 4.3. In this scenario, errors from earlier swaps could distort later ones if slots were entangled, but each step in the HRR-based model’s sequence changes one factor cleanly, providing qualitative evidence consistent with Proposition 2. We note that this is the natural notion of interpolation for a quantized symbolic latent, since smooth blending of codebook entries is not meaningful and a discrete walk through factor configurations is what compositional structure should support. 9
Factor Injection x
x
slot 0
slot 1
slot 2
Shapes3D
slot 3
slot 4
slot 5
slot 6
slot 7
slot 8
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 3: Latent component swaps performance for highest performing model relative to InfoM score. The slot 0 column represents taking the first latent unit from x2 , using it for the first latent unit for x1 , and decoding the result. Slot 1 does the same, but for the second latent unit, and so on. Each slot is replaced with its original value after visualization, so we expect each decoded result to manipulate either one, or no, aspects of the image. x
0 swaps
1 swap
2 swaps
3 swaps
Interpolation
Shapes3D
4 swaps
5 swaps
6 swaps
7 swaps
8 swaps
9 swaps
x
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 4: Latent interpolation swaps components progressively until the representation is fully transformed into that of the second image. We see that in the interpolation, the HRR-based model does not introduce any extraneous generative factors that were not present in the original two images. For instance, no decoded results for the HRR introduce a light blue wall, as the QLAE and β-VAE do.
5
Discussion
In this paper, we have proposed using HRRs for disentanglement and shown that the resulting representations match or exceed strong baselines on the InfoMEC and DCI metrics across four standard datasets while being more robust to additive latent noise than every disentanglement-focused baseline. To the best of our knowledge, this is the first work to use HRRs for disentanglement without supervision, and the first to train a neural network to produce HRR-structured latents directly from data. Sections 3.3 and 3.4 provide a partial mechanism for why the HRR structure favors disentanglement. Approximate slot independence under the unbinding channel (Proposition 2) lets the encoder optimize each slot near-independently, while the per-slot capacity bound (Proposition 1) penalizes redundant slot usage. Together with the quantization inductive bias identified in [10], these two pressures favor placing distinct generative factors in distinct slots. These results are a good foundation for explaining the disentanglement that emerges from the proposed HRR-based model. Tighter constants, a formal connection to identifiability, and an empirical study of the d/m ratio are natural next steps that would further validate these results. Beyond disentanglement, our results indicate that VSA-inspired representations are worth investigating more broadly in the deep learning field. The robustness to latent noise observed here may be particularly relevant for emerging use cases, such as physical-AI. In physical settings, representations must tolerate the noise that is fundamental in sensors and wireless channels. Robustness will be an important factor in ensuring representations remain useful despite corruption. Future work could explore whether the symbolic structure of HRRs translates into compositional generalization on factor combinations unseen during training, which prior work has noted as a weakness of standard disentanglement methods [37]. 10
References [1] Y. Bengio, A. Courville, and P. Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, August 2013. ISSN 0162-8828, 2160-9292. doi: 10.1109/TPAMI.2013.50. [2] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, December 2022. [3] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017. [4] Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating Sources of Disentanglement in Variational Autoencoders. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. [5] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649–2658. PMLR, July 2018. [6] Karl Ridgeway and Michael C Mozer. Learning Deep Disentangled Embeddings With the F-Statistic Loss. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. [7] Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. [8] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. [9] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. [10] Kyle Hsu, Will Dorrell, James C. R. Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023. [11] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. [12] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research, 21(209):1–62, 2020. [13] Alexandr Korchemnyi, Alexey K. Kovalev, and Aleksandr I. Panov. Symbolic disentangled representations for images, 2024. [14] Denis Kleyko, Dmitri A. Rachkovskij, Evgeny Osipov, and Abbas Rahimi. A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations. ACM Comput. Surv., 55(6):130:1–130:40, December 2022. ISSN 0360-0300. doi: 10.1145/3538531. [15] Laura Smets, Werner Van Leekwijck, Ing Jyh Tsang, and Steven Latré. An encoding framework for binarized images using hyperdimensional computing. Frontiers in Big Data, 7, June 2024. ISSN 2624-909X. doi: 10.3389/fdata.2024.1371518. 11
[16] Mohsen Imani, Chenyu Huang, Deqian Kong, and Tajana Rosing. Hierarchical hyperdimensional computing for energy efficient classification. In Proceedings of the 55th Annual Design Automation Conference, Dac ’18, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 978-1-4503-5700-5. doi: 10.1145/3195970.3196060. [17] Alpha Renner, Lazar Supic, Andreea Danielescu, Giacomo Indiveri, Bruno A. Olshausen, Yulia Sandamirskaya, Friedrich T. Sommer, and E. Paxon Frady. Neuromorphic visual scene understanding with resonator networks. Nature Machine Intelligence, 6(6):641–652, June 2024. ISSN 2522-5839. doi: 10.1038/s42256-024-00848-0. URL https://www.nature. com/articles/s42256-024-00848-0. [18] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597–1607. PMLR, November 2020. [19] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. [20] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630–9640, October 2021. doi: 10.1109/ICCV48922.2021.00951. [21] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-Supervised Learning from Images with a JointEmbedding Predictive Architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, June 2023. doi: 10.1109/CVPR52729.2023. 01499. [22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271–21284. Curran Associates, Inc., 2020. [23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988, June 2022. doi: 10.1109/CVPR52688. 2022.01553. [24] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space, September 2013. URL http://arxiv.org/abs/1301. 3781. arXiv:1301.3781 [cs]. [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, July 2021. [26] Michael Hersche, Mustafa Zeqiri, Luca Benini, Abu Sebastian, and Abbas Rahimi. A neurovector-symbolic architecture for solving Raven’s progressive matrices. Nature Machine Intelligence, 5(4):363–375, March 2023. ISSN 2522-5839. doi: 10.1038/s42256-023-00630-8. [27] Ashwinkumar Ganesan, Hang Gao, Sunil Gandhi, Edward Raff, Tim Oates, James Holt, and Mark McLean. Learning with Holographic Reduced Representations. In Advances in Neural Information Processing Systems, volume 34, pages 25606–25620. Curran Associates, Inc., 2021. 12
[28] Mohammad Mahmudul Alam, Edward Raff, and Tim Oates. Towards generalization in subitizing with neuro-symbolic loss using holographic reduced representations. In Neurosymbolic learning and reasoning in the era of large language models, 2023. URL https: //openreview.net/forum?id=AOAP8sLYdt. [29] Mohammad Mahmudul Alam, Edward Raff, Stella Biderman, Tim Oates, and James Holt. Recasting Self-Attention with Holographic Reduced Representations. In Proceedings of the 40th International Conference on Machine Learning, pages 490–507. PMLR, July 2023. [30] T.A. Plate. Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3): 623–641, May 1995. ISSN 1941-0093. doi: 10.1109/72.377968. [31] Kyle Hsu, Jubayer Ibn Hamid, Kaylee Burns, Chelsea Finn, and Jiajun Wu. Tripod: Three complementary inductive biases for disentangled representation learning. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, Vienna, Austria, 2024. JMLR.org. [32] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. [33] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. [34] Chris Burgess and Hyunjik Kim. 3D shapes dataset, 2018. [35] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett, editors, Advances in neural information processing systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ d97d404b6119214e4a7018391195240a-Paper.pdf. [36] NVlabs/High-res-disentanglement-datasets. NVIDIA Research Projects, November 2025. [37] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021. [38] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh International Conference on Learning Representations, 2023. [39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
13
A
Additional details on model design
The denoising network described in Section 3.1 is implemented as a single hidden layer feed forward neural network, with a hidden layer size that is 2× the size of the input and that uses leaky relu activation. The codebook design was inspired from a prior work in slot attention [38]. Specifically, we maintain k "seed" vectors of dimension d, same as z, and generate the codebook by processing each seed vector through a single hidden layer feed forward neural network with a hidden layer size that is the same size as the input and also uses leaky relu activation. We take the output of this neural network to be the approximate-HRR codebook vectors which we regularize. A step that we took to improve training stability was the inclusion of multiplication constant at the output of the encoder and input to the decoder. Given the structural inductive biases imposed on the encoder, directly producing a zero-mean latent with m/d per-component variance would push the model towards having small weights, potentially leading p to training instability and vanishing gradients. Thus, we multiply the output of the encoder by m/d to ensure a higher variance output by the encoder. Similarly, since the reconstructed latent will also have approximately m/d variance, p we multiply the input to the decoder by d/m to transform it to have approximately unit variance for the decoder’s p input. We treat the output of the encoder the same way as the output of the encoder, and apply a 1/d scaling to the produced codebook vectors. Although we observe in Figure 5 of Appendix C.2 that the empirical variance during training is not exactly as assumed, we found that including the normalization constant improved training stability, but other choices may also have the same effect.
B
Algorithm details
This section covers more details on our training algorithm that we presented in Section 3. Algorithm 1 describes the latent reconstruction process that occurs at the bottleneck of the autoencoder. Algorithm 2 references Algorithm 1 in the broader training loop. Algorithm 1 HRR-VQ: Slot Retrieval, Denoising, and Quantization Require: Normalized latent z ∈ Rd , frozen symbol vectors S = {s1 , . . . , sm }, codebook C ∈ Rk×d with rows c1 , . . . , ck , denoising network h : Rd → Rd Ensure: Quantized latent ẑ, quantized value vectors {v̂i }, denoised retrievals {v̄i }, assignment indices {ji } 1: for i = 1, . . . , m do 2: ṽi ← s−1 ▷ HRR unbind: retrieve noisy value vector for slot i i ⊗z 3: end for 4: for i = 1, . . . , m do 5: v̄i ← h(ṽi ) ▷ Denoising network 6: ji ← arg minj ∥v̄i − cj ∥2 ▷ Nearest-codebook-vector lookup 7: v̂i ← cji ▷ Quantized value vector 8: vist ← v̄i + sg(v̂i − v̄i ) ▷ Straight-through estimator 9: end for Pm st 10: ẑ ← ▷ Rebind and bundle all slots i=1 si ⊗ vi 11: return ẑ, {v̂i }, {v̄i }, {ji }
14
Algorithm 2 Pseudocode for optimizing the HRR Autoencoder Require: Dataset D, batch size b, AdamW hyperparameters (α, β1 , β2 , wd), loss weights λ := (λr , λvq , β, λz , λc ), number of symbols m, codebook size k, latent dimension d 1: initialize CNN encoder f : X → Rd , CNN decoder g : Rd → X , denoising network h : Rd → Rd 2: initialize seed matrix S ∈ Rk×d , codebook generator φ : Rd → Rd i.i.d. 3: sample and freeze S = {si }m si ∼ N (0, d1 Id ) ▷ Random HRR symbol vectors; never i=1 , updated 4: while (f, g, h, S, φ) has not converged do 5: for n = 1, . . . , b do 6: x∼D p 7: z ← f (x) · pm/d ▷ Encode and normalize latent variance ▷ Generate codebook from seed matrix 8: C ← φ(S) · 1/d 9: ẑ, {v̂i }, {v̄ }, {j } ← HRR-VQ(z, S, C, h) ▷ Algorithm 1 i i p 10: x̂ ← g(ẑ · d/m) ▷ Denormalize and decode 11: Lrecon ← λr · BCE(x̂, x) P 2 1 12: Lvq ← λvq · m ▷ Codebook toward denoised retrievals i v̂i − sg(v̄i ) P 2 1 13: Lcommit ← β · m i sg(v̂i ) − v̄i ▷ Encoder commits to codebook 2 2 2 14: Rz ← E[z]2 + Var(z) − m + ∥z∥ − m d 2 2 1 P 2 2 ∥v̄ ∥ − 1 15: Rv ← E[v̄] + Var(v̄) − d1 + m i i P 2 1 1 2 2 16: Rc ← E[C] + Var(C) − d + k j ∥cj ∥2 − 1 Lreg ← λz (Rz + Rv ) + λc · Rc L(n) ← Lrecon + Lvq + Lcommit + Lreg end for P (f, g, h) ← AdamW ∇f,g,h 1b n L(n) , (f, g, h), α, β1 , β2 , wd P 21: (S, φ) ← AdamW ∇S,φ 1b n L(n) , (S, φ), 2α, β1 , β2 , wd ▷ Codebook LR = 2α 22: end while 17:
18: 19: 20:
C
Properties and structure of the HRR
In the following sections, we show that retrieval through the unbinding operation over HRRs introduces two terms that describe noise from different sources within the representation, and show that the particular variance used to initialize the HRR is not an essential component to preserving per-component SNR. C.1
Distribution of a retrieval
It is possible to derive the distribution of a retrieval from the unbinding operation. This can be done through the circular correlation of unbinding. A version of this example was originally shown in [30] and also revisited in [27]. Under the circular correlation definition of binding (circular convolution with the inverse would be equivalent), x0 (c20 + c21 + c22 ) + x1 c0 c2 + x2 c0 c1 + x1 c0 c1 +x2 c1 c2 + x1 c1 c2 + x2 c0 c2 2 2 2 x1 (c0 + c1 + c2 ) + x0 c0 c1 + x2 c0 c2 + x0 c0 c2 † c ⊗ (c ⊗ x) = +x2 c1 c2 + x0 c1 c2 + x2 c0 c1 x (c2 + c2 + c2 ) + x c c + x c c + x c c 2 0 0 0 1 1 1 2 0 1 2 1 2 +x1 c0 c2 + x0 c0 c2 + x1 c0 c1 # x0 (1 + ξ) + η0 = x1 (1 + ξ) + η1 = (1 + ξ) x + η. x2 (1 + ξ) + η2 "
15
Here, ξ = (c20 + · · · + c2d ) − 1. Through the central limit theorem and by assuming independence between ci and xi , we can show that ξ ∼ N (0, 2/d), and ηi = N (0, (d − 1)/d2 ). For the average case, the 1 + ξ term can be taken to 1, since E ∥c∥2 = 1, which leaves the final result to be our intended retrieval target plus a noise term. These results can be extended to our case for retrieval from a sum of bound vectors. We add another pair to the quantity from the previous example, c† ⊗ [(c ⊗ x) + (d ⊗ y)]. For compactness, denote wi = (c ⊗ x)i , and vi = (d ⊗ y)i . The retrieval can be expressed as below. " # c0 (w0 + v0 ) + c1 (w1 + v1 ) + c2 (w2 + v2 ) ··· c ⊗ [(c ⊗ x) + (d ⊗ y)] = ··· c0 (c0 x0 + c1 x2 + c2 x1 + d0 y0 + d1 y2 + d2 y1 ) +c1 (c0 x1 + c1 x0 + c2 x2 + d0 y1 + d1 y0 + d2 y2 = +c2 (c0 x2 + c1 x1 + c2 x0 + d0 y2 + d1 y1 + d2 y0 ··· ··· 2 2 (c0 + c1 + c22 ) x0 +c0 c1 x2 + c0 c2 x1 + c0 d0 y0 + c0 d1 y2 + c0 d2 y1 +c0 c1 x1 + c1 c2 x2 + c1 d0 y1 + c1 d1 y0 + c1 d2 y2 = +c0 c2 x2 + c1 c2 x1 + c2 d0 y2 + c2 d1 y1 + c2 d2 y0 ··· ··· # " x0 (1 + ξ) + η0 + λ0 = x1 (1 + ξ) + η1 + λ1 x2 (1 + ξ) + η2 + λ2 = (1 + ξ)x + η + λ. †
The final expression, for the m = 2 case, results in the same expression for the m = 1 case, with an extra term for the cross talk noise due to unbinding. Each λi has d2 terms, each like cj dk yl . Assuming independence between terms, cj dk yl ∼ N (0, 1/d3 ). Since there are d2 terms, λi = N (0, 1/d). In general, for the retrieval of a vector from a latent z with m bound terms, x̂ = (c(i) )† ⊗ z = (1 + ξ)x + | {z } retrieval
η |{z}
intrinsic noise
+
m X
λ(k) ,
k̸=i
| {z }
cross-talk noise
where c(i) is the cue associated with a target we are trying to retrieve, and λ(k) is the cross talk term due to the k’th bound pair in the representation. Thus, we can interpret the retrieval as being composed of 3 main parts, the intended retrieval target, the intrinsic noise, and the cross talk noise. Since our signal and noise terms are both zero mean, we can quantify the per component signal power as S = 1/d, and the per component noise power as m−1 1 m N = d−1 = m d2 + d d − d2 . For large d, we have N ≈ d . Then the per component SNR for a retrieval is SNR = C.2
1/d 1 = . m/d m
Importance of the HRR initialization
The analysis in Appendix C.1 shows that the zero-mean requirement for the HRR vectors ensures that all the noise terms are also zero-mean. The per component variance of 1/d ensures that the expected squared norm of HRR vectors is one. This allows us to ensure that the norm of a representation does not grow unbounded as it is bound with more vectors. This was a desirable property for the original intended purpose of the HRR, which was to be able to create representations of arbitrary 16
depth [30]. For our case, this requirement is not strictly necessary since we only use HRR vectors to create bundles of m bound pairs, not for binding arbitrary amounts. If we were to instead consider the case in which HRR vector components were instead sampled from N (0, c), where c > 0. Then we can find that the binding variance becomes Var((a ⊗ b)i ) = d · c2 . For retrieval, the per component cross talk variances becomes Var(λi ) = md2 c3 , and the intrinsic noise becomes Var(ηi ) = d(d − 1)c3 ≈ d2 · c3 . The per component signal power becomes S = Var((c20 +· · ·+c2d )·xi ) = (2dc2 +c2 d2 )·c ≈ c2 d2 ·c, for d ≫ c, and thus the SNR is SNR =
d2 c3 1 d2 c3 = = , (m − 1)d2 c3 + d2 c3 md2 c3 m
thus showing that the per component SNR actually remains unchanged with respect to the variance of the atomic HRR vectors. This shows that the 1/d variance requirement from the original HRR implementation may not be necessary for our purposes, and thus any reasonable choice, c for the per component variance may also work. However, the original choice of variance is retained for consistency with the original HRR implementation and other works using the HRR. The results in Fig. 5 support this finding. The model learns to keep the zero mean requirement as it is essential for the unbinding process to work. The variances drift from the 1/d constant associated with the HRR, but this finding supports the analysis above, as the 1/d requirement is not strict. This leaves room for the model to converge on a separate variance that is more optimal for that particular dataset, which we believe is what the figure reflects.
17
HRR training statistics (mean ± 95% CI) Codebook vector mean over training
0.003 0.002 0.001 0.000 0.001 0.002 0
0.35
0.0150 0.0125 0.0100 0.0075 0.0050 0.0025 0.0000
20000 40000 60000 80000 100000 120000
Training Step Latent z variance over training
0
0.04
20000 40000 60000 80000 100000 120000
Training Step Latent z mean over training
0.03
0.30
Latent z Mean
Latent z Variance
Codebook variance over training 0.0175
0.004
Codebook Variance
Codebook Mean
0.005
0.25 0.20 0.15 0.10 0.05
0.02 0.01 0.00 0.01 0.02
0.00 0
20000 40000 60000 80000 100000 120000
Training Step
Shapes3D
0
Falcor3D
20000 40000 60000 80000 100000 120000
Isaac3D
Training Step MPI3D-C
Figure 5: Mean and variance over training iterations, for both the codebook vectors and the latent representations. Most statistics converge on stable values as training progresses.
VQ perplexity over training Shapes3D Falcor3D Isaac3D MPI3D-C
VQ Perplexity
200 150 100 50 0
0
20000
40000
60000
80000
Training Step
100000
120000
Figure 6: Codebook usage over training iterations, measured by perplexity. We can see that the model learns to utilize the codebook vectors, with the harder datasets having higher codebook usage.
18
D
Proofs of theoretical results
Next, we prove Propositions 1 and 2 from Sections 3.3 and 3.4. In this section, expectations are with respect to the HRR initialization distribution (each component i.i.d. N (0, 1/d)) and the data distribution, as appropriate. We write Ṽ1:m for the tuple (ṽ1 , . . . , ṽm ) and Ṽ<i for (ṽ1 , . . . , ṽi−1 ). D.1
Proof of proposition 2 (approximate slot independence)
By the chain rule of probability, m X q(Ṽ1:m | x) qi (ṽi | Ṽ<i , x) = . log Q log q (ṽ | x) qi (ṽi | x) i i i i=2
(6)
The term where i = 1 is identically zero since the conditioning set is empty. Taking expectations under q(Ṽ1:m | x), h i Pm Q Pm qi (ṽi |Ṽ<i ,x) DKL q ∥ i qi = = I ṽ ; Ṽ | x . (7) i <i i=2 Eq log i=2 qi (ṽi |x) We bound the right-hand side of(7) by treating the total correlation as a single aggregate quantity, rather than bounding each term I ṽi ; Ṽ<i | x individually. As we discuss in Section D.4, bounding each term in isolation leads to expressions involving the smallest eigenvalue of a conditioning block, which is not accessible from the operator norms we can establish. Instead, we work with the closedform KL representation of the full sum and bound it through a single log-determinant expression. This approach requires an understanding of the block covariance structure of the joint retrieval Ṽ1:m , which we develop next. D.2
Covariance Block Structure
The mean of the joint retrieval distribution q(Ṽ1:m | x) is (v1 (x), . . . , vm (x)). The block covariance matrix Σ ∈ Rmd×md , with block entries Σ11 Σ21 Σ= ... Σm1
Σ12 Σ22 .. . ···
··· ··· .. . ···
Σ1m Σ2m , .. . Σmm
where any entry Σij = E ṽi ṽjT ∈ Rd×d . Now, recall that the structure of a retrieval can be expressed as X ṽi = (1 + ξi )vi + ηi + λ(ik) , k̸=i
where λ(ik) represents the cross-talk introduced from s†i ⊗ (sk ⊗ vk ). From the definition of the covariance, we have h i Cov ṽi , ṽj | Ṽ1:m , x = E (ṽi − vi )((ṽj − vj )T | Ṽ1:m , x h i = E ϵi ϵT j Ṽ1:m , x T X X = E ξi vi + ηi + λ(ik) ξj vj + ηj + λ(jr) Ṽ1:m , x ℓ̸=i
19
r̸=j
D.2.1
Diagonal Blocks
We begin by analyzing the diagonal blocks of the block covariance matrix, Σ. Specifically, we’d like to find the structure of an entry in Σii , defined as
(Σii )ab = E ξi vi + ηi +
X ℓ̸=i
λ(iℓ) · ξi vi + ηi +
X
λ(ir) Ṽ1:m , x .
r̸=i
a
b
We begin by computing the expected values of the terms in the resulting product. 1. E [(ξi vi )a · (ξi vi )b ] = (vi )a (vi )b E ξi2 = (vi )a (vi )b · d2 2. E [(ηi )a (ηi )b ] ̸= 0 3. E [(ξi vi )a (ηi )b ] = (vi )a E [ξi (ηi )b ] = 0 h i 4. E (ηi )a (λ(iℓ) )b = E (ηi )a (s†i ⊗ sℓ ⊗ vℓ )b Observe that (ηi )a will contain traces of elements of si , so the unbinding operation will introduce terms with squared elements of si . However, these terms will still have elements of sℓ to the first power, which in expectation, from the HRR distribution, will result in h i E (ηi )a (λ(iℓ) )b = 0.
5. E (ξi vi )a (λ(iℓ) )b Applying the same reasoning as in 4), entries from sℓ will remain in the product to first power, resulting in h i E (ξi vi )a (λ(iℓ) )b = 0.
6. E (λ(iℓ) )a (λ(ir) )b ̸= 0 For ℓ ̸= r, the expected value will evaluate to zero due to the elements of sℓ and sr appearing to first degree, however, the case where ℓ = r will not be zero since squared elements of the symbols will appear. From the analysis above, we see that the covariance, Σii , can be expressed as X h (iℓ) (iℓ) T i Σii = E (ξi vi + ηi )(ξi vi + ηi )T + E λ (λ ) , ℓ̸=i
as the other cross-terms cancel. Then, applying the definitions of circular convolution and circular (i) correlation, we can express an entry from a single cross-talk contribution. Letting sk denote the k’th 20
entry of si , we have " d−1 d−1 ! h i X X (i) (iℓ) (iℓ) (ℓ) E (λ )a (λ )b = E sk sm (vℓ )k+a−m ·
!# (i) (ℓ) sk′ sm′ (vℓ )k′ +b−m′
k′ =0 m′ =0
k=0 m=0
=E
d−1 X d−1 X
"d−1 d−1 d−1 d−1 XX X X
# (i) (i) (ℓ) sk s(ℓ) m sk′ sm′ (vℓ )k+a−m (vℓ )k′ +b−m′
k=0 m=0 k′ =0 m′ =0 d−1 X d−1 X d−1 X d−1 X
=
i h (i) (i) (ℓ) E sk s(ℓ) m sk′ sm′ (vℓ )k+a−m (vℓ )k′ +b−m′
k=0 m=0 k′ =0 m′ =0 d−1 X d−1 X
=
h i (i) (i) (ℓ) E sk s(ℓ) m sk sm (vℓ )k+a−m (vℓ )k+b−m
k=0 m=0 d−1 d−1
1 XX (vℓ )k+a−m (vℓ )k+b−m d2 m=0
=
k=0
d−1 d−1
1 XX (vℓ )a+n (vℓ )b+n d2 n=0
=
k=0
d−1 X
1 (vℓ )a+n (vℓ )b+n , d n=0
=
where n = k − m mod d. Thus, the contribution of the cross-talk term can be written as d−1 i X1X X h E (λ(iℓ) )a (λ(iℓ) )b = (vℓ )a+n (vℓ )b+n . d n=0 ℓ̸=i
ℓ̸=i
Evaluating the non-crosstalk contributions, we have that E [(ξi vi + ηi )a (ξi vi + ηi )b ] = E [(ξi vi )a (ξi vi )b + (ηi )a (ηi )b ] = (vi )a (vi )b E ξi2 + E [(ηi )a (ηi )b ] 2 = (vi )a (vi )b · + E [(ηi )a (ηi )b ] . d We can derive an expression for (ηi )a directly from the unbinding definition, (ṽi )a =
d−1 X d−1 X
(i)
sk s(i) m (vi )k+a−m
k=0 m=0
=
d−1 X
s(i) s(i) (vi )k+a−k + k k
k=0
=
d−1 X
m̸=k
(i) sk
2
(vi )a +
k=0
=
(i) sk s(i) m (vi )k+a−m
X
X
(i) sk s(i) m (vi )k+a−m
m̸=k (i)
s1
2
(i)
+ s2
2
d−1 X d−1 2 X (i) (i) + · · · + sd (vi )a + sk s(i) m (vi )k+a−m k=0 m̸=k
= ∥si ∥2 (vi )a +
d−1 X d−1 X
(i)
sk s(i) m (vi )k+a−m .
k=0 m̸=k
21
So we have that
E [(ηi )a (ηi )b ] = E
d−1 X d−1 X
(i) · sk s(i) m (vi )k+a−m
= E
(i) (i) sk′ sm′ (vi )k′ +b−m′
k′ =0 m′ ̸=k′
k=0 m̸=k
d−1 X d−1 X
d−1 X d−1 X d−1 X d−1 X
(i) (i) (i) sk s(i) m sk′ sm′ (vi )k+a−m (vi )k′ +b−m′
k=0 m̸=k k′ =0 m′ ̸=k′
=
d−1 X d−1 X d−1 X d−1 X
h i (i) (i) (i) E sk s(i) s s ′ ′ m k m (vi )k+a−m (vi )k′ +b−m′
k=0 m̸=k k′ =0 m′ ̸=k′
Observe that the expectation is non-zero only when k = k ′ , m = m′ and when k = m′ , m = k ′ , thus,
E [(ηi )a (ηi )b ] =
d−1 X d−1 X
h i (i) (i) (i) E sk s(i) m sk sm (vi )k+a−m (vi )k+b−m
k=0 m̸=k
+
d−1 X d−1 X
h i (i) (i) (i) E sk s(i) (vi )k+a−m (vi )m+b−k m sm sk
k=0 m̸=k d−1 X d−1 X
d−1 d−1
1 XX (vi )k+a−m (vi )m+b−k d2 k=0 m̸=k k=0 m̸=k d−1 d−1 d−1 X d−1 X 1 X X = 2 (vi )k+a−m (vi )k+b−m + (vi )k+a−m (vi )m+b−k d k=0 m̸=k k=0 m̸=k X 1 X = (vi )a+n (vi )b+n + (vi )a+n (vi )b−n . d =
1 d2
n̸=0
(vi )k+a−m (vi )k+b−m +
n̸=0
Finally, we can state
d−1 2 X1X (Σii )ab = (vi )a (vi )b · + (vℓ )a+n (vℓ )b+n d} d n=0 | {z ℓ̸=i | {z } E[(ξi vi )a (ξi vi )b ] P (iℓ) ) (λ(iℓ) ) a b] ℓ̸=i E[(λ X 1 X + (vi )a+n (vi )b+n + (vi )a+n (vi )b−n . d n̸=0 n̸=0 | {z } E[(ηi )a (ηi )b ]
22
Observe that when a = b, and when d ≫ 0, we’ll have, for the average case d−1 X X1X X 1 2 (vi )2a+n + (vℓ )2a+n + (vi )a+n (vi )b−n (Σii )aa = (vi )2a · + d d n=0 d n̸=0 ℓ̸=i n̸=0 X 1 2 2 ∥vℓ ∥ + ∥vi ∥ − (vi )a + (vi )a+n (vi )b−n | {z } d | d {z } n̸=0 ℓ̸=i ≈1 ≈1−1/d {z } |
2 1 = (vi )2a · + | {z } d 1/d
X
2
≈0
2 m−1 1 1 ≈ 2+ + − 2 d d d d 1 m = 2+ d d m ≈ . d The approximations defined under the braces come from the fact that elements of vi are sampled from the HRR distribution, N (0, 1/d). In the alternative case, for a ̸= b, we have (Σii )ab ≈ 0 due to the products of terms that have zero mean. Thus, we have the approximation Σii ≈
D.2.2
m I. d
Off-Diagonal Blocks
For the off-diagonal blocks, we are interested in
(Σij )ab = E ξi vi + ηi +
X ℓ̸=i
λ(iℓ) · ξj vj + ηj + a
X r̸=j
λ(jr) Ṽ1:m , x . b
By going through cases again, we make the following observations. 1. E [(ξi vi )a · (ξj vj )b ] = (vi )a (vj )b E [ξi ξj ] = 0 2. E [(ηi )a (ηj )b ] = 0 3. E [(ξi vi )a (ηj )b ] = (vi )a E [ξi (ηj )b ] = 0 4. E (ηi )a (λ(jr) )b = 0 We have already seen that for the case when i = j, the expectation is zero. Additionally, when i ̸= j, the expectation is still zero, as entries of si , sj , and sr , all appear at first power. 5. E (ξi vi )a (λ(jr) )b = 0 Applying the same logic as 4), we see that this expectation is also zero. 6. E (λ(iℓ) )a (λ(jr) )b For this case, the expectation will be non-zero only for the case when ℓ = j and r = i. 23
Examining case six closer, similarly as before, we can write (Σij )ab =
d−1 X d−1 X d−1 X d−1 X
h i (i) (j) (i) E sk s(j) m sk′ sm′ (vj )k+a−m (vi )k′ +b−m′
k=0 m=0 k′ =0 m′ =0
=
d−1 X d−1 X
h i (i) (j) (i) E sk s(j) (vj )k+a−m (vi )m+b−k m sm sk
k=0 m=0 d−1 d−1
=
1 XX (vj )k+a−m (vi )m+b−k d2 m=0 k=0
=
d−1 X
1 (vj )a+n (vi )b−n . d n=0
This establishes that entries of Σij are at least O( d1 ). Then, observe that from the definition of circular convolution, we can rewrite the final equality as d−1
(Σij )ab =
1X (vj )a+n (vi )b−n d n=0
1 (vj ⊗ vi )a+b . d We can then express the squared Frobenius norm of Σij as =
∥Σij ∥2F =
d X d X
(Σij )2ab
a=1 b=1 d−1 d−1
1 XX |(vj ⊗ vi )a+b |2 = 2 d a=0 b=0
d−1 X
=
1 |(vj ⊗ vi )c |2 ·d d2 c=0
=
1X |(vj ⊗ vi )c |2 d c=0
=
1 ∥(vj ⊗ vi )∥2 . d
d−1
Since each entry of (vj ⊗ vi ) has variance d1 , we can say that (vj ⊗ vi )c = O( √1d ). From this, we can establish that ∥(vj ⊗ vi )∥2 =
d−1 X
|(vj ⊗ vi )c |2
c=0
=
d−1 X c=0
O
1 d
= O (1) . Thus, we have that ∥Σij ∥2F =
1 ∥(vj ⊗ vi )∥2 , d| {z } O(1)
which implies that ∥Σij ∥2F is O
1 d
. 24
D.2.3
Sharpened Off-Diagonal Operator Norm
In general, for a matrix A, we define the operator norm as ∥A∥op = max∥x∦=0 ∥Ax∥ ∥x∥ . We also establish √ a sharper bound on ∥Σij ∥op . While the Frobenius bound gives ∥Σij ∥op ≤ ∥Σij ∥F = O(1/ d), the circulant structure of Σij yields a tighter estimate. Recall (Σij )ab = d1 (vj ⊗ vi )a+b , i.e. Σij = d1 H(vj ⊗ vi ) where H(g)ab = g(a+b) mod d . The matrix H(g) is a circulant composed with the index-reversal permutation, H(g) = P Circ(g) with P orthogonal, so its singular values coincide with those of Circ(g), namely the moduli {|ĝk |} of the DFT of g. Let g = v j ⊗ vi ,
gℓ = (vj ⊗ vi )ℓ ,
with all subscripts interpreted modulo d. Then (Σij )ab =
1 ga+b , d
so g0 g1 1 g2 Σij = d ... gd−1
g1 g2 g3 .. . g0
··· ··· ··· .. . ···
g2 g3 g4 .. . g1
gd−1 g0 g1 . .. . gd−2
Equivalently, Σij =
1 H(g), d
H(g)ab = g(a+b) mod d .
Using the convention Circ(g)ab = g(b−a) mod d , we have
g0
g1 g0
gd−1 g d−2 Circ(g) = . .. g1
gd−1 .. . g2
g2 g1 g0 .. . g3
··· ··· ··· .. . ···
gd−1 gd−2 gd−3 . .. . g0
Now define the index-reversal permutation matrix P by Pab = 1{b ≡ −a
(mod d)}.
1 0 0 P = . .. 0
··· ··· 1 .. . ···
Explicitly, 0 0 0 .. . 1
0 0 ... .. . 0
0 1 0 . .. . 0
Therefore, H(g) = P Circ(g), and hence
1 P Circ(vj ⊗ vi ). d Using that binding multiplies DFT coefficients pointwise, Σij =
1 ∥Σij ∥op = d1 max (v\ j ⊗ vi )k = d max |v̂j,k | |v̂i,k |. k
k
For v ∼ the HRR distribution, each transform coefficient is complex Gaussian with E|v̂k |2 = 1, so with high probability over the codebook maxk |v̂k |2 = O(log d), giving ∥Σij ∥op = O logd d . 25
D.3
A Posterior Precision Floor
The log-determinant reduction developed below requires the joint covariance Σ to be nonsingular. We first show that, for the idealized noiseless retrieval model, Σ is in fact singular. The key point is that the retrieved slots follow an exact deterministic linear constraint. Recall that the latent structure that we use takes the form m X M= sk ⊗ vk , k=1
and the retrieval of slot i is
ṽi = s†i ⊗ M. We analyze this relation in the Fourier domain, where circular convolution and correlation diagonalize. For a real vector u ∈ Rd , write its discrete Fourier transform as ûω =
d−1 X
un e−2πiωn/d ,
ω = 0, . . . , d − 1.
n=0
The circular convolution theorem gives \ (a ⊗ b)ω = âω b̂ω . Similarly, for the unbinding operation (c† ⊗ y)j =
d−1 X
ck yk+j ,
k=0
a direct computation gives † ⊗ y) = ĉ ŷ . (c\ ω ω ω
Therefore, cω = M
m X
ŝk,ω v̂k,ω .
k=1
Define the shared Fourier coefficient cω = Zω := M
m X
ŝk,ω v̂k,ω .
k=1
Then the retrieved slot satisfies b̃ i,ω = ŝi,ω Zω . v Thus every slot’s retrieval at frequency ω is obtained from the same scalar Zω , modulated by that slot’s own symbol coefficient. Now weighing each retrieved coefficient by the conjugate of the corresponding target coefficient and summing over slots, we get m m X X b̃ i,ω = v̂i,ω v v̂i,ω ŝi,ω Zω . i=1
i=1
Factoring out Zω , we obtain m X
b̃ i,ω = Zω v̂i,ω v
i=1
m X
ŝi,ω v̂i,ω = Zω
i=1
m X
ŝi,ω v̂i,ω .
i=1
By the definition of Zω , this becomes m X
b̃ i,ω = Zω Zω = |Zω |2 . v̂i,ω v
i=1
The right-hand side is real-valued for every realization of the symbols. Hence ! m X b̃ i,ω = 0 v̂i,ω v Im i=1
26
identically. Conditioned on the codebook {vi }m i=1 , the coefficients v̂i,ω are fixed. Also, the Fourier transform is a fixed invertible linear map on the real coordinates of each retrieved vector. Therefore ! m X b̃ i,ω Lω (Ṽ1:m ) := Im v̂i,ω v i=1
is a fixed real-linear functional of the joint retrieval vector Ṽ1:m = (ṽ1 , . . . , ṽm ) ∈ Rmd . The identity above says that Lω (Ṽ1:m ) = 0
almost surely.
Equivalently, there exists a real vector aω ∈ Rmd , depending only on the fixed codebook and on the Fourier frequency, such that a⊤ almost surely. ω Ṽ1:m = 0 Thus ⊤ Var a⊤ ω Ṽ1:m = aω Σaω = 0. Because Σ is a covariance matrix, it is positive semidefinite. Hence a⊤ ω Σaω = 0 implies Σaω = 0. Therefore aω is a nontrivial null direction of Σ, so Σ is not full rank and det Σ = 0. Under the joint Gaussian model, this makes log det(Σ̄† Σ) diverge to −∞, so the log-determinant expression for the total correlation becomes infinite. This singularity is therefore a property of the idealized noiseless retrieval model, not of the log-determinant reduction itself. The resolution is that the variational model does not produce deterministic slots. Each posterior factor qi (ṽi | x) is a non-degenerate density, since the encoder assigns the slot a finite posterior precision. We model this posterior uncertainty as independent additive Gaussian noise across slots and coordinates, with per-coordinate variance τ 2 > 0. Under this regularization, the joint covariance becomes Στ = Σ + τ 2 Imd , and the block-diagonal marginal covariance becomes Σ̄τ = Σ̄ + τ 2 Imd . The off-diagonal blocks Σij , i ̸= j, are unchanged, while every degenerate direction receives variance τ 2 . Since Σ ⪰ 0, we have Στ = Σ + τ 2 Imd ≻ 0. Thus the regularized covariance is nonsingular, and the log-determinant expression is well-defined. Henceforth, in the total-correlation bound, we replace Σ̄ by Σ̄τ = Σ̄ + τ 2 Imd . A useful consequence is that, because Σ̄ ⪰ 0, λmin (Σ̄ + τ 2 Imd ) ≥ τ 2 . Therefore ∥(Σ̄ + τ 2 Imd )−1/2 ∥op = p
1 λmin
(Σ̄ + τ 2 I
md )
≤ τ −1 .
(8)
This gives a posterior precision floor for the whitening operation. Its operator norm is controlled by τ alone, and the conditioning of the noiseless diagonal blocks Σii never enters. 27
D.4
Reduction to the Log-Determinant
Remark 1. A natural approach to bounding (7) would be to work with each I ṽi ; Ṽ<i | x individually. For jointly Gaussian variables, each such term equals − 12 log det(I − Ri ), where −1/2 −1/2 Ri = Σii Σi,<i Σ−1 is the canonical correlation matrix. Bounding Ri via the <i,<i Σ<i,i Σii submultiplicative chain −1/2 2 ∥op ∥Σi,<i ∥2op ∥Σ†<i,<i ∥op
∥Ri ∥op ≤ ∥Σii
would require ∥Σ†<i,<i ∥op = 1/λmin (Σ<i,<i ), which is the smallest eigenvalue of Σ<i,<i and is not controlled by the operator norms we can establish. Moreover, even granting such a bound, the submultiplicative product discards the alignment between the ill-conditioned directions of Σ†<i,<i and the directions in which Σi,<i carries mass, so the resulting estimate would not be tight. We therefore do not bound the Ri individually. Since we are ultimately interested in the aggregate quantity established in (7), m X Q I ṽi ; Ṽ<i | x = DKL q ∥ i qi , i=2
we bound this total correlation directly. Under the joint Gaussian model with mean (v1 (x), . . . , vm (x)) and covariance Σ, this KL admits the closed form Q DKL q ∥ i qi = 21 tr Σ̄† Σ − md − log det Σ̄† Σ , Q where Σ̄ = blkdiag(Σ11 , . . . , Σmm ) is the covariance of the product of marginals i qi . Write ∆ = Σ − Σ̄ for the part of Σ consisting of its off-diagonal blocks Σij (i ̸= j); by construction ∆ has vanishing diagonal blocks. Then Σ̄† Σ = I + Σ̄† ∆, and because Σ̄† is block-diagonal while ∆ has no diagonal blocks, X tr Σ̄† Σ = md + tr Σ̄† ∆ = md + tr Σ†ii ∆ii = md, i
so the trace term cancels exactly. Introducing the symmetric matrix M = Σ̄−1/2 ∆ Σ̄−1/2 , which satisfies det(I + Σ̄† ∆) = det(I + M ) by similarity and tr(M ) = tr(Σ̄† ∆) = 0, we are left with m X X I ṽi ; Ṽ<i | x = − 12 log det(I + M ) = − 12 log(1 + µk ), (9) i=2
k −1/2
−1/2
where {µk } are the eigenvalues of M . Since I + M = Σ̄ Σ Σ̄ ⪰ 0, each eigenvalue satisfies µk ≥ −1. Note that this reduction requires only the marginal whitening Σ̄−1/2 , and does not invert the conditioning block Σ<i,<i . D.5
Bounding the Total Correlation
P We now bound (9) with Σ̄ regularized as above. The eigenvalues of M obey k µk = tr(M ) = 0 and µk > −1. For each k, the second-order Taylor expansion of log(1 + ·) with Lagrange remainder gives µ2k log(1 + µk ) = µk − , ζk between 0 and µk . 2(1 + ζk )2 In either sign of µk one has 1 + ζk > 1 + µmin ≥ 1 − ∥M ∥op , so provided ∥M ∥op < 1, log(1 + µk ) > µk −
28
µ2k . 2 (1 − ∥M ∥op )2
Summing over k and using
P
k µk = 0 and
2 2 k µk = ∥M ∥F ,
P
m X X log(1 + µk ) ≤ I ṽi ; Ṽ<i | x = − 21 i=2
k
∥M ∥2F . 4 (1 − ∥M ∥op )2
(10)
We now need to bound the two quantities on the right, ∥M ∥2F , and ∥M ∥op . Writing W = (Σ̄ + τ 2 I)−1/2 so that M = W ∆W , the submultiplicativity of the Frobenius norm under operator-norm multiplication together with (8) gives ∥M ∥F = ∥W ∆W ∥F ≤ ∥W ∥2op ∥∆∥F ≤ τ −2 ∥∆∥F . P Since ∆ collects the off-diagonal blocks, ∥∆∥2F = i̸=j ∥Σij ∥2F , and substituting the established ∥Σij ∥2F = d1 ∥vj ⊗ vi ∥2 with ∥vj ⊗ vi ∥2 = O(1), 2 1 X 1 X m ∥M ∥2F ≤ 4 ∥Σij ∥2F = 4 ∥vj ⊗ vi ∥2 = O 4 . (11) τ τ d τ d i̸=j
i̸=j
For the operator norm, the same reasoning gives ∥M ∥op ≤ τ −2 ∥∆∥op . Applying the triangle inequality and Cauchy-Schwarz to ∆, which collects all off-diagonal blocks Σij , i ̸= j, r sX 2 d ∥Σij ∥2op = m(m − 1) O logd2 d = O m log ∥∆∥op ≤ , d i̸=j
where we use the sharpened estimate ∥Σij ∥op = O(log d/d) from the preceding section. Therefore ∥M ∥op = O m log d/(τ 2 d) . The regularity condition ∥M ∥op < 1 of (10) holds once d ≳ m log d/τ 2 , in which regime (1 − ∥M ∥op )−2 = 1 + o(1). Combining (10) and (11), we obtain the final estimate. Conditioned on the codebook, with per-slot posterior precision τ 2 and d ≳ m log d/τ 2 , 2 m X m I ṽi ; Ṽ<i | x = O 4 . τ d i=2 The total correlation among the retrieved slots thus vanishes as d−1 , grows quadratically in the number of slots, and vanishes with the posterior precision through τ −4 . D.6
Proof of proposition 1 (HRR latent capacity)
We first bound I(x; ẑ) by a sum over slots, then bound each term by both an additive white gaussian noise (AWGN) capacity and a codebook entropy, and take the minimum. Step 1: reduction to a sum The quantized symbolic latent ẑ =
P
i si ⊗ v̂i is a deterministic
function of (v̂1 , . . . , v̂m ) given the (frozen) symbol vectors. Hence I(x; ẑ) ≤ I x; V̂1:m by the data-processing inequality. By the chain rule and the identity I x; v̂i | V̂<i − I(x; v̂i ) = I v̂i ; V̂<i | x − I v̂i ; V̂<i , 2 m m m X X X m I x; V̂1:m ≤ I(x; v̂i ) + I v̂i ; V̂<i | x = I(x; v̂i ) + O 4 , t d i=1 i=2 i=1 where we dropped the term I v̂i ; V̂<i . Since quantization is applied slotwise, (v̂1 , . . . , v̂m ) is a post-processing of (ṽ1 , . . . , ṽm ). Therefore the conditional total correlation among the quantized slots is no larger than that of the continuous retrievals, so Proposition 2 applies to the second sum. 29
Step 2: per-slot AWGN bound Conditional on the encoder output, slot i is observed through the unbinding channel ṽi = vi (x)+ϵi where ϵi has covariance (m/d)Id (Appendix C). The signal vi (x) has expected squared norm at most 1 (by the HRR distribution and the value regularizer Lvalue which targets τn = 1). The capacity of a d-dimensional AWGN channel with average input power P and per-dimension noise variance σ 2 is d2 log(1 + P/(dσ 2 )). Substituting P = 1 and σ 2 = m/d gives 1 per-slot capacity d2 log(1 + m ). Quantization can only reduce mutual information (data-processing), d 1 so I(x; v̂i ) ≤ I(x; ṽi ) ≤ 2 log(1 + m ). Step 3: per-slot codebook bound Since v̂i takes values in a finite codebook of size k, I(x; v̂i ) ≤ H(v̂i ) ≤ log k. Each term satisfies I(x; v̂i ) ≤ min d2 log(1 + 1/m), log k , so I(x; ẑ) ≤ m · min d2 log(1 + 1/m), log k = min m · d2 log(1 + 1/m), m log k ,
Step 4: combining
□
which is the claim.
Remark on tightness The AWGN bound is tight when the encoder produces values vi (x) that are Gaussian-distributed with covariance proportional to identity (the capacity-achieving input). In practice, the combination of the variance regularizer Lvalue (which targets variance 1/d per entry, matching the HRR distribution) and quantization toward a learned codebook drives the encoder toward this regime. The codebook bound m log k is tight when codebook entries are used uniformly. Design corollary
For a target representation rate R in nats, Proposition 1 requires m·
d log(1 + 1/m) ≥ R 2
and
m log k ≥ R.
Equivalently, 2R and k ≥ eR/m . m log(1 + 1/m) The first condition captures the continuous HRR retrieval bottleneck, while the second captures the discrete codebook bottleneck. For large m, since log(1 + 1/m) ≈ 1/m, the dimension condition becomes approximately d ≳ 2R. d≥
Thus, increasing m lowers the effective per-slot SNR, but also distributes the target rate across more slots. These effects approximately balance in the large-m regime, causing the continuous HRR capacity to saturate near d/2. In contrast, the codebook requirement k ≥ eR/m becomes easier to satisfy as m increases, since each slot needs to represent a smaller fraction of the total rate. Together, these conditions suggest to choose m to be large enough to cover the expected number of generative factors, d to be large enough to support the desired total continuous rate, and k large enough to avoid a discrete quantization bottleneck.
E
Experimental setup
Choice of baselines Our baselines are chosen to position HRR against two reference points: established VAE-based disentanglement methods [3, 4] and methods that share our use of vector quantization [11, 10]. Hyperparameters for models As described in Section 4.1, to ensure fair comparisons we ran a sweep for each model and dataset over a single hyperparameter, over two seeds. We then computed the average InfoM score across both seeds for each hyperparameter, and took the hyperparameter that had the highest average InfoM scores. The values that were swept over can be found in Table 6, and are consistent with prior works [10]. The results of the sweep, the respective hyperparameters used for the main results in Table 2, can be found in Table 7. Our HRR model uses a single set of hyperparameters across all datasets as we found that tuning hyperparameters did not provide much of an improvement in disentanglement performance. The 30
exact values can be found in Table 5. An interesting observation early in our investigation is that performance was better when the weight of the regularization on the latent produced by the encoder, λlat , was low. We found similar performance keeping in a range between 0 and 1e − 5. Although we keep it at zero, and thus has no influence on the optimization of our model, we still include λlat as a design parameter for future works. Although higher weight on λlat produced latent representations closer to a true expected HRR vector, it tended to trade off disentanglement performance. We hypothesize that giving the encoder more freedom in producing its representations outweighs the benefit of the structure obtained by imposing it. However, even without regularizing the latent, Fig. 5 shows us that the encoder still learns to produce latent representations that are zero mean and that have relatively constant per-dimension variance. Computational details for experiments All models were trained using PyTorch [39] on an internal SLURM cluster. Each training job used a single GPU. Due to limited availability on the shared cluster, runs were scheduled on whichever compatible GPU resources were available, primarily NVIDIA L40S, A30, and V100 GPUs. GPU type affected wall-clock runtime, but all models were trained using the same codebase, datasets, training protocol, evaluation pipeline, and random-seed structure. On an NVIDIA L40S, a typical HRR training run, as well as the baseline training runs under the same experimental protocol, required slightly over one hour, although runs could take considerably longer on A30 or V100 GPUs. The main results reported in Table 2 required approximately 7 aggregate GPU-days. The codebook-size and latent-dimension comparisons in Appendix G required approximately another 7 aggregate GPU-days. The hyperparameter sweeps used to select baseline configurations required approximately 4 aggregate GPU-days. In total, the experiments reported in this paper required approximately 18 aggregate GPU-days. The full research project required additional compute due to preliminary experiments, failed runs, debugging, and exploratory model variants that are not included in the reported results. After training, we used an NVIDIA GeForce RTX 3090 for inference on trained models and for constructing final figures and visualizations. Computation of disentanglement metrics with HRRs Since our final representation, ẑ, is composed of discrete components from the codebook, we use the codebook indices produced by the vector quantization process to compute the disentanglement metrics, as was done for VQ-VAE and QLAE paper [10]. Common experiment variables All models share the same CNN architecture, inspired from the model used in the QLAE paper [10]. A detailed description of the autoencoder architecture is in Table 3. Additionally, all models share the same optimization configuration, detailed in Table 4. The HRR model uses one set of hyperparameters for all datasets, which are listed in Table 5. Confidence intervals and bold font criteria The results in Table 2 summarize performance over five independent seeds, specifically the sample mean of the scores for each respective metric. For each model, dataset, and metric, we report the sample mean together with a 95% confidence interval for the mean, in Table 12. The interval is centered at the sample mean and uses a Student-t critical value with n − 1 degrees of freedom: s x̄ ± t0.975,n−1 √ , n where x̄ is the sample mean, s is the sample standard deviation computed with Bessel’s correction, and n = 5 is the number of seeds. For bolding, we use a CI-overlap criterion. In each column, we first identify the entry with the highest upper confidence bound. We then use that entry’s lower confidence bound as a threshold: any entry whose lower confidence bound meets or exceeds this threshold is also bolded.
31
Layer
In
Out
k
s
Activation
Conv 1 Conv 2 Conv 3 Conv 4
Conv2d† Conv2d† Conv2d† Conv2d†
C 64 128 256
64 128 256 512
4 4 4 4
2 2 2 2
LReLU(0.2) LReLU(0.2) LReLU(0.2) LReLU(0.2)
Flatten FC 1 FC 2
— Linear Linear
— 8,192 512
8,192 512 dz
— — —
— — —
— ReLU —
Canvas Style
Learnable param. h0 Linear
— dz
512×4×4 512
— —
— —
— ReLU
Up 1 Up 2 Up 3 Up 4
StyleConvT‡ StyleConvT‡ StyleConvT‡ StyleConvT‡
512 512 256 128
512 256 128 64
4 4 4 4
2 2 2 2
LReLU(0.2) LReLU(0.2) LReLU(0.2) LReLU(0.2)
Out
Conv2d
64
C
3
1
—
Stage Encoder
Decoder
Table 3: Architecture and shared training hyperparameters for the CNN autoencoder. dz denotes the latent dimension and C the number of image channels. All convolutional layers omit bias terms. † Followed by InstanceNorm2d with learnable affine parameters (γ, β); padding p = 1. ‡ StyleConvT: ConvTranspose2d (p = 1) → LReLU(0.2) → InstanceNorm2d (no affine) → AdaIN, where the per-
channel scale and bias are predicted from the style vector w via a linear layer.
Hyperparameter
Value
Optimizer Max. gradient steps Learning rate η Batch size Gradient clip β1 β2 ε
AdamW 125,000 3 × 10−4 128 0.5 0.9 0.999 1 × 10−8
Table 4: Optimizer hyperparameters used for all runs
32
Hyperparameter
Symbol
Value
Latent dimension Number of symbols Codebook size
dz m k
512 9 512
Commitment cost Latent reg. weight Codebook reg. weight VQ loss weight Reconstruction weight
β λlat λcb λvq λrec
0.6 0 1 × 10−3 0.25 1.0
Learning rate Codebook Learning Rate Weight decay
η ηcb —
3 × 10−4 2η 0
Table 5: Model hyperparameters for the proposed HRR autoencoder. Used for all experiments
method
hyperparameter
values
β-VAE [3] β-TCVAE [4] VQ-VAE [11] QLAE [10]
β = λKL β = λtotal correlation weight decay weight decay
[0.1, 0.3, 1, 3, 10] [0.1, 0.3, 1, 3, 10] [0.001, 0.01, 0.1, 1] [0.001, 0.01, 0.1, 1]
Table 6: Regularization hyperparameter tuning for each baseline
model
Shapes3D
Falcor3D
Isaac3D
MPI3D-C
β-VAE β-TCVAE VQ-VAE QLAE
β = 10 β=3 λwd = .001 λwd = 1
β = 10 β=3 λwd = .01 λwd = 1
β=3 β=3 λwd = .001 λwd = 1
β = 10 β=3 λwd = 1 λwd = 1
Table 7: Selected regularization hyperparameters for each model and dataset. Values were chosen from the corresponding hyperparameter sweeps.
33
F
Limitations
Empirical comparisons Our empirical comparison is limited by the set of baselines that could be evaluated under a shared experimental protocol. In particular, we considered including Tripod [31], a recent follow-up to QLAE [10] that combines multiple inductive biases for disentanglement and reports strong InfoMEC and DCI performance. However, Tripod differs from the other baselines not only in its objective, but also in its architectural assumptions and computational requirements. One component of its objective involves estimating Hessian-based regularization terms during training, which substantially increased training time in our setting. In addition, the architecture proposed for Tripod uses residual connections and differed from the CNN backbone used for the other methods in our comparison. We attempted to evaluate Tripod in two ways: using the architecture proposed by the original authors, and adapting its Hessian-based regularization to the CNN backbone used in our experiments. The former was prohibitively expensive under our available compute budget, while the latter did not yield stable results consistent with those reported in the original paper. Across five seeds, we observed large variation in InfoM, with some runs reaching high scores and others performing substantially worse. Because we could not determine whether this instability reflected an implementation issue, sensitivity to architectural changes, or hyperparameter dependence, we chose not to include Tripod as a baseline. We believe this is the more conservative choice, since reporting an unreliable reproduction could unfairly understate the performance of Tripod. As a result, our claims should be interpreted as comparisons against the baselines evaluated under our shared-backbone protocol, rather than as a definitive comparison against all recent disentanglement methods. Our main contribution is the investigation of VSA-inspired representations as an inductive bias for unsupervised disentanglement, rather than establishing absolute state-of-the-art disentanglement performance across all possible architectures and objectives. Latent-to-source ratio The choice of using 1.5× latents to sources ratio differs from the 2× ratio that was used in [10]. The reason for this is that unbinding performance degrades with m in our model since the unbinding noise becomes worse the more symbol-value pairs are bound together. Thus, we use a 1.5× ratio to maintain the difficulty of the problem without penalizing performance of the HRR-based model. Increased autoencoder parameter count with HRRs We point out that due to the larger size of the latent vector produced by our model, the autoencoder inevitably ends up having more parameters due to the larger projection matrices at the bottleneck. In our experiments, we found that this caused the autoencoder for our HRR model to have around 15.4M parameters, while the baselines had around 14.9M parameters. Although this causes the model architectures to not be completely identical, we argue that this is a feature of our model design, as the representations of prior disentanglement works are limited to keeping vector dimensionality close to the number of ground truth factors. Thus they are not able to take advantage of increasing model capacity without also hurting disentanglement performance by creating more latent units than necessary. Broader Impacts This work is primarily methodological and studies VSA-inspired inductive biases for unsupervised disentanglement on benchmark datasets. A potential positive impact of this work is that more structured latent representations may improve the interpretability, modularity, and controllability of learned representations. Such properties could be useful in scientific modeling, representation analysis, controllable generation, and downstream systems where understanding the factors encoded by a model is important. At the same time, improvements in disentangled representation learning may also have negative downstream uses. More controllable latent representations can make it easier to manipulate generated or reconstructed content along semantically meaningful axes. In generative modeling contexts, this could contribute to misuse such as deceptive media editing, impersonation, or other forms of content manipulation. More interpretable representations could also be misused in surveillance or profiling settings if applied to sensitive human data. 34
G
Effect of HRR latent dimension and codebook size
Although we chose a single configuration for our HRR model for results, we run sweeps over each dataset for 9 combinations of different latent dimension and codebook sizes. The results for each disentanglement metric, modularity, explicitness, and compactness, are included in Figs. 7 8 9, respectively. InfoM 0.85
0.587 ±0.396
0.766 512 ±0.289
0.824 ±0.077
0.816 ±0.102
0.754 1024 ±0.128
0.687 ±0.224
0.853 ±0.027
128
256
512
0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.45
Codebook Size (k)
0.628 256 ±0.147
0.618 ±0.171
0.608 512 ±0.081
0.639 ±0.038
0.633 ±0.154
0.607 1024 ±0.214
0.615 ±0.283
0.668 ±0.114
128
256
Isaac3D
0.85
0.602 ±0.265
512
0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.45
Codebook Size (k)
0.558 256 ±0.444
0.593 ±0.348
0.717 512 ±0.047
0.730 ±0.001
0.722 ±0.139
0.718 1024 ±0.136
0.757 ±0.014
0.765 ±0.057
128
256
MPI3D-C
0.85
0.615 ±0.026
512
0.80
Latent Dimensionality
0.786 ±0.116
Falcor3D
Latent Dimensionality
0.745 256 ±0.074
Latent Dimensionality
Latent Dimensionality
Shapes3D
0.75 0.70 0.65 0.60 0.55 0.50 0.45
Codebook Size (k)
0.85
0.559 256 ±0.041
0.533 ±0.095
0.497 ±0.069
0.502 512 ±0.123
0.502 ±0.229
0.548 ±0.064
0.442 1024 ±0.053
0.470 ±0.150
0.511 ±0.039
128
256
512
0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.45
Codebook Size (k)
Figure 7: InfoM performance over different combinations of codebook size and latent dimension InfoE
0.998 1024 ±0.005
128
0.998 ±0.006 1.000 ±0.000
256
0.992 ±0.032 1.000 ±0.000 1.000 ±0.000
512
Codebook Size (k)
0.95 0.90 0.85 0.80 0.75
0.800 256 ±0.037 0.791 512 ±0.026 0.782 1024 ±0.045
128
0.802 ±0.055 0.801 ±0.047 0.784 ±0.060
256
Isaac3D 0.824 ±0.068 0.821 ±0.052 0.822 ±0.024
512
0.95 0.90 0.85 0.80 0.75
Codebook Size (k)
0.889 256 ±0.061 0.871 512 ±0.070 0.877 1024 ±0.042
128
0.908 ±0.045 0.891 ±0.016 0.849 ±0.109
256
MPI3D-C 0.911 ±0.030 0.872 ±0.081 0.865 ±0.047
512
Codebook Size (k)
Latent Dimensionality
0.999 512 ±0.001
1.000 ±0.000
Latent Dimensionality
1.000 256 ±0.000
Falcor3D Latent Dimensionality
Latent Dimensionality
Shapes3D
0.95 0.90 0.85 0.80 0.75
0.818 256 ±0.038
0.845 ±0.019
0.840 ±0.047
0.838 512 ±0.016
0.823 ±0.058
0.830 ±0.038
0.741 1024 ±0.110
0.834 ±0.100
0.843 ±0.023
128
256
512
0.95 0.90 0.85 0.80 0.75
Codebook Size (k)
Figure 8: InfoE performance over different combinations of codebook size and latent dimension
35
InfoC
0.616 ±0.020
0.65
0.414 ±0.205 0.586 ±0.125
0.60 0.55
0.523 256 ±0.126 0.424 512 ±0.064
0.493 ±0.173 0.490 ±0.067
0.442 ±0.159
0.50
0.557 1024 ±0.121
0.502 ±0.214
0.651 ±0.047
128
256
512
Codebook Size (k)
0.45
Isaac3D
0.65
0.525 ±0.183
0.60 0.55
0.456 256 ±0.295 0.585 512 ±0.045
0.506 ±0.341 0.624 ±0.003
0.597 ±0.113
0.50
0.408 1024 ±0.205
0.427 ±0.132
0.477 ±0.056
128
256
512
0.45
Codebook Size (k)
MPI3D-C
0.65
0.474 ±0.059
Latent Dimensionality
0.582 512 ±0.164
0.540 ±0.183
Falcor3D
Latent Dimensionality
0.490 256 ±0.089
Latent Dimensionality
Latent Dimensionality
Shapes3D
0.60 0.55
0.65
0.532 256 ±0.070
0.531 ±0.126
0.503 ±0.075
0.554 512 ±0.193
0.532 ±0.193
0.591 ±0.056
0.496 1024 ±0.038
0.522 ±0.258
0.586 ±0.149
128
256
512
0.50
0.568 1024 ±0.177
0.577 ±0.104
0.575 ±0.050
128
256
512
Codebook Size (k)
0.45
0.60 0.55 0.50 0.45
Codebook Size (k)
Figure 9: InfoC performance over different combinations of codebook size and latent dimension
36
G.1
Negative results for the HRR model
In the design of our model we ablated a few design choices but found that they either had no benefit, or ended up hurting the model’s performance. We list the most notable of those below. 1. Initializing the symbol vectors, S, with an improved scheme proposed in a prior work [27]. 2. Setting the codebook, C, to be static. 3. Setting the symbol vectors, S to be learnable. 4. Removing the pre-processing network described in Section 3.1. 5. Directly keeping approximate-HRR vectors in the codebook, as opposed to the parameterized approach described in the previous paragraph.
H
Complete disentanglement results
The full disentanglement results for each dataset, with the mean and 95% confidence intervals described in Appendix E, can be found in Table 12.
37
model β-VAE β-TCVAE VQ-VAE QLAE HRR (ours)
InfoM ↑
InfoE ↑
InfoC ↑
D↑
I↑
C↑
PSNR (dB) ↑
0.59 ± 0.04 0.6 ± 0.1 0.6 ± 0.1 0.8 ± 0.1 0.85 ± 0.05
0.8 ± 0.1 0.8 ± 0.1 1.000 ± 0.005 0.998 ± 0.005 0.999 ± 0.005
0.47 ± 0.04 0.51 ± 0.04 0.44 ± 0.06 0.52 ± 0.07 0.61 ± 0.04
0.64 ± 0.09 0.6 ± 0.2 0.74 ± 0.07 0.76 ± 0.08 0.88 ± 0.02
0.997 ± 0.005 0.98 ± 0.03 0.98 ± 0.01 0.993 ± 0.006 0.995 ± 0.007
0.55 ± 0.06 0.5 ± 0.2 0.35 ± 0.02 0.35 ± 0.04 0.41 ± 0.02
38.9 ± 0.3 40.8 ± 0.8 44.8 ± 0.3 36 ± 1 41 ± 4
Table 8: Full results for Shapes3D model β-VAE β-TCVAE VQ-VAE QLAE HRR (ours)
InfoM ↑
InfoE ↑
InfoC ↑
D↑
I↑
C↑
PSNR (dB) ↑
0.64 ± 0.03 0.6 ± 0.2 0.50 ± 0.07 0.61 ± 0.07 0.63 ± 0.05
0.68 ± 0.02 0.65 ± 0.02 0.86 ± 0.03 0.79 ± 0.01 0.81 ± 0.01
0.59 ± 0.07 0.6 ± 0.2 0.40 ± 0.03 0.46 ± 0.07 0.45 ± 0.07
0.40 ± 0.04 0.3 ± 0.2 0.64 ± 0.04 0.50 ± 0.03 0.58 ± 0.05
0.83 ± 0.01 0.79 ± 0.06 0.81 ± 0.02 0.84 ± 0.02 0.85 ± 0.02
0.39 ± 0.04 0.3 ± 0.2 0.340 ± 0.007 0.28 ± 0.01 0.37 ± 0.01
29.8 ± 0.2 31.0 ± 0.1 31.4 ± 0.2 28.3 ± 0.4 29.0 ± 0.6
Table 9: Full results for Falcor3D model β-VAE β-TCVAE VQ-VAE QLAE HRR (ours)
InfoM ↑
InfoE ↑
InfoC ↑
D↑
I↑
C↑
PSNR (dB) ↑
0.54 ± 0.06 0.63 ± 0.06 0.70 ± 0.05 0.66 ± 0.06 0.73 ± 0.06
0.56 ± 0.03 0.62 ± 0.06 0.86 ± 0.08 0.81 ± 0.05 0.89 ± 0.02
0.46 ± 0.06 0.6 ± 0.1 0.5 ± 0.1 0.49 ± 0.05 0.55 ± 0.05
0.31 ± 0.05 0.35 ± 0.04 0.64 ± 0.09 0.59 ± 0.04 0.680 ± 0.006
0.84 ± 0.03 0.87 ± 0.01 0.91 ± 0.07 0.92 ± 0.03 0.92 ± 0.02
0.30 ± 0.04 0.34 ± 0.04 0.36 ± 0.06 0.31 ± 0.02 0.364 ± 0.005
37.0 ± 0.4 37.2 ± 0.4 45 ± 2 38 ± 4 43.4 ± 0.7
Table 10: Full results for Isaac3D model β-VAE β-TCVAE VQ-VAE QLAE HRR (ours)
InfoM ↑
InfoE ↑
InfoC ↑
D↑
I↑
C↑
PSNR (dB) ↑
0.37 ± 0.05 0.41 ± 0.01 0.5 ± 0.1 0.56 ± 0.05 0.5 ± 0.1
0.23 ± 0.03 0.42 ± 0.06 0.82 ± 0.03 0.78 ± 0.04 0.83 ± 0.02
0.29 ± 0.09 0.39 ± 0.06 0.53 ± 0.05 0.48 ± 0.02 0.53 ± 0.06
0.08 ± 0.03 0.27 ± 0.02 0.49 ± 0.08 0.51 ± 0.04 0.55 ± 0.04
0.57 ± 0.02 0.75 ± 0.01 0.73 ± 0.07 0.78 ± 0.03 0.769 ± 0.007
0.10 ± 0.03 0.26 ± 0.02 0.34 ± 0.02 0.29 ± 0.02 0.34 ± 0.02
29.5 ± 0.3 35.9 ± 0.1 34.9 ± 0.5 34.2 ± 0.3 37 ± 1
Table 11: Full results for MPI3D-C Table 12: Full InfoMEC results. Bold marks the best per column using the CI-overlap criterion. ↑: higher is better.
I
Additional noise results
We include further results for the noise robustness experiment for the remaining datasets. Figs. 10, 11, 12 show the results on Isaac3D, Falcor3D, and MPI3D-C, respectively.
38
Latent noise robustness Original
Clean Recon
+20dB
+10dB
+5dB
Gaussian +0dB
-5dB
-10dB
-20dB
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 10: Each row is a visualization of how reconstruction quality degrades as noise intensity increases for each model. For each model, the seed with the highest InfoM score was used for evaluation.
Latent noise robustness Original
Clean Recon
+20dB
+10dB
+5dB
Gaussian +0dB
-5dB
-10dB
-20dB
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 11: Each row is a visualization of how reconstruction quality degrades as noise intensity increases for each model. For each model, the seed with the highest InfoM score was used for evaluation.
39
Latent noise robustness Original
Clean Recon
+20dB
+10dB
+5dB
Gaussian +0dB
-5dB
-10dB
-20dB
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 12: Each row is a visualization of how reconstruction quality degrades as noise intensity increases for each model. For each model, the seed with the highest InfoM score was used for evaluation.
40
J
Additional latent component swaps
We include further results for the latent component swap experiment for the remaining datasets. Figs. 13, 14, 15 show the results on Isaac3D, Falcor3D, and MPI3D-C, respectively. x
x
slot 0
slot 1
slot 2
slot 3
slot 4
Factor Injection slot 5
Isaac3D slot 6
slot 7
slot 8
slot 9
slot 10
slot 11
slot 12
slot 13
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 13: Latent component swaps performance for highest performing model relative to InfoM score. The slot 0 column represents taking the first latent unit from x2 , using it for the first latent unit for x1 , and decoding the result. Slot 1 does the same, but for the second latent unit, and so on. Each slot is replaced with its original value after visualization, so we expect each decoded result to manipulate either one, or no, aspects of the image.
x
x
slot 0
slot 1
slot 2
Factor Injection
Falcor3D
slot 3
slot 4
slot 5
slot 6
slot 7
slot 8
slot 9
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 14: Latent component swaps performance for highest performing model relative to InfoM score. The slot 0 column represents taking the first latent unit from x2 , using it for the first latent unit for x1 , and decoding the result. Slot 1 does the same, but for the second latent unit, and so on. Each slot is replaced with its original value after visualization, so we expect each decoded result to manipulate either one, or no, aspects of the image.
41
x
x
slot 0
slot 1
slot 2
Factor Injection
MPI3D-C
slot 3
slot 4
slot 5
slot 6
slot 7
slot 8
slot 9
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 15: Latent component swaps performance for highest performing model relative to InfoM score. The slot 0 column represents taking the first latent unit from x2 , using it for the first latent unit for x1 , and decoding the result. Slot 1 does the same, but for the second latent unit, and so on. Each slot is replaced with its original value after visualization, so we expect each decoded result to manipulate either one, or no, aspects of the image.
42
K
Additional latent interpolations
We include further results for the latent interpolation experiment for the remaining datasets. Figs. 16, 17, 18 show the results on Isaac3D, Falcor3D, and MPI3D-C, respectively. x
0 swaps
1 swap
2 swaps
3 swaps
4 swaps
5 swaps
6 swaps
Interpolation Isaac3D 7 swaps 8 swaps
9 swaps
10 swaps
11 swaps
12 swaps
13 swaps
14 swaps
x
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 16: Latent interpolation swaps components progressively until the representation is fully transformed into that of the second image. Interpolation x
0 swaps
1 swap
2 swaps
3 swaps
4 swaps
Falcor3D
5 swaps
6 swaps
7 swaps
8 swaps
9 swaps
10 swaps
x
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 17: Latent interpolation swaps components progressively until the representation is fully transformed into that of the second image. Interpolation x
0 swaps
1 swap
2 swaps
3 swaps
4 swaps
MPI3D-C
5 swaps
6 swaps
7 swaps
8 swaps
9 swaps
10 swaps
x
HRR (ours)
QLAE
-TCVAE
-VAE
VQ-VAE
Figure 18: Latent interpolation swaps components progressively until the representation is fully transformed into that of the second image.
43