The Standard Deviation in VICReg Buys a Safety Margin

The Standard Deviation in VICReg Buys a Safety Margin Hawar Dizayee

hawardizayee@gmail.com

Independent Researcher

arXiv:2610.32630v1 [cs.LG] 23 Oct 2026

Abstract VICReg prevents collapse with a hinge on the standard deviation of each embedding dimension, and its authors argue from the gradient that a hinge on the variance would fail. To our knowledge, this claim has not been tested experimentally. We test it in a controlled setting (ResNet-18, CIFAR-10 and CIFAR-100), changing only the hinge and sweeping its weight µ over five orders of magnitude. At the default weight (µ = 1), the two hinges reach the same linear-probe accuracy on CIFAR-10 (about 89.5%). Both collapse below a threshold weight, but the standard-deviation hinge’s threshold is 35–130× lower. CIFAR-100 shows a consistent gap. In training, the variance hinge’s gradient fades as the feature variance shrinks, while the standard-deviation hinge’s does not until that variance is very small. The standard-deviation hinge therefore buys a safety margin: it keeps training at weights far below the point where the variance hinge collapses.

1

Introduction

VICReg (Bardes et al., 2022) keeps a joint-embedding model from collapsing with three loss terms: an invariance term that pulls the two views of an image together, a covariance term that decorrelates embedding dimensions, and a variance term that keeps every dimension spread out across the batch. The variance term is a hinge on the standard deviation of each dimension. Bardes et al. (2022, Section 4.1) state that using the standard deviation rather than the variance is crucial: with a hinge on the variance, the gradient vanishes as the embeddings shrink, and training collapses. This argument is made from the form of the gradient; to our knowledge it has not been tested directly. The EB-JEPA library (Terver et al., 2026), which we build on, uses the same standard-deviation hinge. We test the claim: we change only the hinge (standard deviation or variance) and sweep its weight µ over five orders of magnitude. At EB-JEPA’s default weight (µ = 1) the two hinges reach the same accuracy (about 89.5% on CIFAR-10). Both collapse when the weight is low enough, but the variance hinge collapses first, at a weight 35 to 130 times higher than the standard-deviation hinge. Our contributions are: • A controlled test of the standard-deviation-versus-variance claim in VICReg, changing only the hinge and sweeping its weight over five orders of magnitude, with three seeds at every weight and a CIFAR-100 check (Section 4). • The finding that both hinges have a collapse threshold, that the two thresholds sit 35–130× apart, and that the hinges tie at the default weight (Section 4). • An account of the mechanism from the measured gradients during training, including why the standard-deviation hinge also collapses (Section 5). • Code to reproduce every run, publicly available at https://github.com/hawardizayee/ vicreg-hinge. 1

2

Background

The VICReg loss. VICReg (Bardes et al., 2022) trains an encoder followed by a projector on two augmented views of each image. Let Z, Z ′ ∈ RN ×D be the projector outputs of a batch of N images for the two views, with rows zi and entries zij . We call each of the D embedding dimensions a feature and write z̄j for the batch mean of feature j. The loss has three terms. The invariance term pulls the two views of an image together: N 1 X ′ s(Z, Z ) = ∥zi − zi′ ∥2 . (1) N D i=1 The variance term keeps each feature from becoming constant over the batch. VICReg puts a hinge on the standard deviation of each feature: D N   p 1 X 1 X max 0, γ − Varj + ϵ , where Varj = (zij − z̄j )2 . (2) vstd (Z) = D j=1 N − 1 i=1 The covariance term decorrelates the features: X 1 [C(Z)]2j,k , c(Z) = D(D − 1)

where

j̸=k

The total loss is

N 1 X (zi − z̄)(zi − z̄)⊤ . C(Z) = N − 1 i=1

L = s(Z, Z ′ ) + µ [v(Z) + v(Z ′ )] + ν [c(Z) + c(Z ′ )] ,

(3)

(4)

with γ = 1 and ϵ = 10 unless stated otherwise. We write the three terms as the EB-JEPA implementation computes them, with the invariance weight fixed to 1; they differ from the definitions in the VICReg paper by constant factors (Appendix A). −4

The variance hinge.

The alternative we test puts the hinge on the variance itself: D

vvar (Z) =

1 X max(0, γ − Varj ) . D j=1

(5)

With γ = 1 both hinges are zero for a feature of unit√variance. For a constant feature the variance hinge equals 1 and the standard-deviation hinge equals 1 − ϵ. They differ in their gradient (Section 5, Eq. 6). The claim we test. Bardes et al. (2022) state that using the standard deviation and not the variance in the hinge is crucial (their Section 4.1). Their argument is about the gradient: with the variance, the gradient with respect to a value goes to zero as the value approaches the batch mean, so the term stops pushing the embeddings apart and they collapse.

3

Setup

Code and model. All runs use a standalone reimplementation of the image_jepa example of the EBJEPA library (Terver et al., 2026; Meta FAIR, 2025), with three flags added: loss.hinge=std|var, which picks the variance hinge (Section 2), loss.eps, which sets the ϵ of the standard-deviation hinge, and a dataset flag for CIFAR-100. The encoder is a ResNet-18 (He et al., 2016) with the usual CIFAR stem, followed by a projector of three linear layers (hidden and output width 1024), with batch normalization and ReLU after the first two. The loss is applied to the projector outputs. The code is publicly available at https://github.com/hawardizayee/vicreg-hinge. Data and training. We train on CIFAR-10 (Krizhevsky, 2009) for 300 epochs with batch size 256, using LARS (You et al., 2017) (learning rate 0.3, momentum 0.9, weight decay 10−4 , η = 0.02) with 10 warmup epochs and cosine decay to 0, in bfloat16 mixed precision. Augmentations are the EB-JEPA defaults (random resized crop, color jitter, grayscale, solarization, horizontal flip), applied independently to each view. We repeat part of the grid on CIFAR-100, normalized with CIFAR-100’s own mean and standard deviation. Full details are in Appendix B. 2

Table 1: Linear-probe accuracy (%) on CIFAR-10 at epoch 300 for the variance (var) and standard-deviation (std) hinge across the variance-term weight µ. Mean over 3 seeds (1, 42, 1000), with the standard deviation in small type. Collapsed runs end between 16% and 23% (chance 10%), the level of the run with no variance term; Table 6 gives their values. ‡ training diverged (loss became NaN; probe at chance). – : not run. µ 0 (no var term) 0.001 0.003 0.01 0.1 0.2 0.3 0.35 0.4 0.5 0.9 1 10 100

var hinge std hinge 18.79 ± 3.06 – collapsed – collapsed – 76.57 ± 0.21 collapsed 82.93 ± 0.61 collapsed – collapsed 88.15 ± 0.17 collapsed – 84.77 ± 0.33 – 87.17 ± 0.16 89.48 ± 0.25 89.43 ± 0.09 – 89.53 ± 0.13 89.58 ± 0.15 86.58 ± 0.05 87.65 ± 0.17 div.‡ div.‡

Loss weights. The invariance term has weight 1, as in EB-JEPA, and the covariance weight is 100. Both are fixed in every run of the main grid. The only loss weight we change is the variance weight µ. These weights cannot be compared directly to the 25 / 25 / 1 of the VICReg paper (Bardes et al., 2022); Appendix A converts them. What we vary. Runs in the main grid differ in exactly three things: the hinge (std or var), µ, and the seed. Tables 1 and 2 list every weight of the grid for each hinge; the weights are denser near each collapse threshold, and the grid includes runs with no variance term (µ = 0). Every point, on CIFAR-10 and on CIFAR-100, uses three seeds (42, 1, 1000). Three controls step outside this grid and are described where they are reported: full fp32, VICReg’s covariance weight, and a larger ϵ. Evaluation. We report the accuracy of an online linear probe trained on the encoder features during pre-training, following the EB-JEPA protocol. The probe trains on detached encoder features. The number we report is validation accuracy after the last epoch. A run is collapsed when this accuracy ends near the no-variance-term baseline (16–23% on CIFAR-10, against 10% chance), and diverged when the loss becomes NaN. Collapsed runs stay above the 10% chance level because collapse is not exact: the encoder outputs of different images still differ slightly, and the probe can use those differences. Training-dynamics monitor. For Section 5, we fix a monitor batch: the first 256 training images, each with two augmented views drawn once before training. After every epoch, we pass both views through the network in evaluation mode, in fp32 and without tracking gradients, and save the encoder and projector outputs. The feature-scale and gradient analyses are both computed afterward from these saved outputs. Compute.

4

Each run takes about 65 minutes on one NVIDIA RTX PRO 4500.

Results

Figure 1 and Table 1 give the linear-probe accuracy at the end of training for every weight µ and both hinges. At no weight do the three seeds disagree on the outcome. The hinges tie at the default weight. At µ = 1 the var hinge reaches 89.53 ± 0.13 and the std hinge 89.58 ± 0.15 on CIFAR-10. The gap of 0.05 points is smaller than the spread over seeds. On CIFAR-100 3

std threshold µ 0.003–0.01

100

var threshold µ 0.35–0.4

80

)%( ycarucca

60

40 20 diverged (NaN loss)

0

0.001

0.003

0.01

0.03

0.1

0.3

µ

1

3

10

30

100

Figure 1: CIFAR-10 linear-probe accuracy after 300 epochs versus the variance-term weight µ, for the variance hinge (var; violet circles, solid line) and the standard-deviation hinge (std; green squares, dashed line). Each point is the mean over 3 seeds; error bars show the standard deviation and are often smaller than the marker. Shaded bands mark each hinge’s collapse threshold (std: 0.003–0.01; var: 0.35–0.4). At µ = 100 training diverged for both hinges (NaN loss; the probe is at chance, 10%). Where the two hinges nearly coincide (µ = 1, 10 and 100) the std marker is shifted slightly to the right for visibility. the std hinge is half a point ahead (63.82 ± 0.27 against 63.33 ± 0.24; Table 2), about twice the spread over seeds. A hinge on the variance is therefore not a failing design in itself: at the default weight it trains to within half a point of the hinge on the standard deviation on both datasets. Both hinges have a collapse threshold, far apart. The var threshold lies between 0.35 and 0.4 and the std threshold between 0.003 and 0.01, which is 35 to 130 times lower. Close to a threshold: lower accuracy and a late start. Lowering µ costs accuracy for both hinges before it causes collapse, the var hinge sooner than the std hinge (2.3 points behind at µ = 0.5; Tables 1 and 2), and the effective rank of the final representations falls with it (Appendix D). Training also starts later. Figure 2 (left and middle) shows the two sides of each threshold over the first 100 epochs: just above a threshold, the surviving runs cannot be told apart from the collapsing ones until about epoch 35 for the var hinge (µ = 0.4) and about epoch 50 for the std hinge (µ = 0.01). We call the first epoch at which a run’s accuracy exceeds 35%, a level no collapsed run reaches, its escape delay. It grows as µ approaches a threshold from above, for both hinges (Figure 2, right), so a run at a low weight can look collapsed for tens of epochs and still train. CIFAR-100. On CIFAR-100 both thresholds are consistent with those on CIFAR-10 (chance 1%; Table 2). The var hinge collapses at µ = 0.1 (3.26±0.35) and µ = 0.3 (3.23±0.29) and survives at µ = 0.4 (56.34±0.55), so its threshold lies between 0.3 and 0.4. The std hinge collapses at µ = 0.003 (3.37 ± 0.42) and survives at µ = 0.01 (35.74 ± 0.29), so its threshold lies between 0.003 and 0.01. Just above its threshold the std hinge is far below its accuracy at µ = 0.1 (54.52 ± 0.54), a larger loss than on CIFAR-10. 4

80

var hinge

80

µ = 0.4

60

60

µ = 0.01

40

µ = 0.35 collapses

20 0 0

never

50

epoch

100

µ = 0.003 collapses

20 0 0

50

100

epoch

60

hcope

)%( ycarucca

40

std hinge

40

first epoch with accuracy above 35%

std hinge var hinge

20 0

0.001

0.01

µ

0.1

1

Figure 2: The collapse threshold over the first 100 epochs of training (CIFAR-10). Colors as in Figure 1. Left and middle: linear-probe accuracy per epoch on the two sides of each hinge’s threshold. Lines are the mean over 3 seeds and the shaded band spans the lowest to the highest seed; the paler curve is the weight that collapses. Right: escape delay, the first epoch at which accuracy exceeds 35%, against µ (mean over 3 seeds). A cross marks a weight at which all three seeds collapse and never reach 35%. Table 2: Linear-probe accuracy (%) on CIFAR-100 at epoch 300, in the format of Table 1. Collapsed runs end between 2% and 4% (chance 1%); Table 6 gives their values. – : not run. µ 0.003 0.01 0.1 0.3 0.4 0.5 1

var hinge – – collapsed collapsed 56.34 ± 0.55 58.90 ± 0.25 63.33 ± 0.24

std hinge collapsed 35.74 ± 0.29 54.52 ± 0.54 – – – 63.82 ± 0.27

Controls. Training uses mixed precision, and the hinge computes a batch variance at every step, so we checked that neither threshold is a precision effect. Rerun in full fp32 (seed 42), the var hinge still collapses at µ = 0.3 (15.62) and µ = 0.35 (15.79), and the std hinge still collapses at µ = 0.003 (15.83) and survives at µ = 0.01 (76.66). We also reran the std hinge with ϵ = 10−2 (seed 42): it collapses at µ = 0.05 and survives at µ = 0.1, so its threshold moves up roughly tenfold, as Section 5 predicts. At VICReg’s default ratios (µ = 0.5, ν = 41 in our units) both hinges train and the std hinge is 0.57 points ahead (Appendix A). Above µ = 1 both hinges lose accuracy at µ = 10, the var hinge about a point more, and both diverge at µ = 100 (consistent with Table 2 of Terver et al. 2026). This high-weight regime is a different failure from low-µ collapse and is outside this study.

5

Mechanism

Push and pull. The invariance term pulls the two views of an image together, and the variance term pushes each feature away from its batch mean. For a feature j below the target γ, the gradients of the two hinges, before the weight µ, are 2 ∂vvar =− (zij − z̄j ), ∂zij D(N − 1)

∂vstd 1 z − z̄j p ij =− , ∂zij D(N − 1) Varj + ϵ 5

(6)

var hinge · pale = the runs collapse

std hinge · pale = the runs collapse 1

0.3

0.3

0.1

0.1

0.03

0.03

gniniart revo llup / hsup

1

0.1

0.2

0.3

0.35

µ

0.4

0.5

0.9

1

0.001

0.003

0.01

0.1

µ

0.3

0.5

1

Figure 3: Push/pull over training (CIFAR-10), on a logarithmic axis: the norm of the gradient of the weighted variance term divided by the norm of the gradient of the invariance term, both taken with respect to the projector outputs of the monitor batch. One box per weight µ, holding every epoch of the 3 seeds (900 values). The box spans the middle half of the values, the line is the median and the whiskers reach the lowest and the highest value. Pale boxes are weights that collapse; colors as in Figure 1. ′ and the gradient of the invariance term is ∂s/∂zij = N2D (zij − zij ). At every epoch we take the gradient of the weighted variance term and the gradient of the invariance term with respect to the projector outputs of the monitor batch, and divide the norm of the first by the norm of the second, each taken over all entries of both views. We call this ratio push/pull (Figure 3).

The two hinges push differently. The var hinge pushes in proportion to the deviation zij − z̄j , so its push shrinks with the feature. This is the argument of VICReg. The pull shrinks with the features as well, so for the var hingeppush/pull hardly changes as a run collapses: it is set by µ. The std hinge divides the deviation by Sj √ = Varj + ϵ, so its push keeps its size as the feature shrinks and push/pull rises, until Sj reaches its floor ϵ = 0.01. For both hinges the ratio falls as µ decreases, except for the var hinge at µ ≥ 0.9, discussed below (Figure 3). Why the std hinge’s threshold is lower. Because its push does not shrink, the std hinge gains √ on the pull as√a feature shrinks, and even a weak hinge can win. This holds until the spread falls below ϵ. There √ Sj ≈ ϵ, and the std hinge of Eq. 6 becomes a var hinge with a push 1/(2 ϵ) = 50 times larger. The std hinge’s threshold therefore depends on ϵ. A one-seed control with ϵ = 10−2 agrees: the factor falls to 5 and the threshold moves up about tenfold (Section 4). A low ratio late in training is not collapse. The tall var boxes at µ ≥ 0.9 have a different cause. There most features reach the target variance of 1 during training. The hinge is zero for those features, so the push falls: at µ = 1 (seed 42) only 15% of the features are still below the target at the end of training. Push/pull describes the tug-of-war only while the hinge is active for nearly all features, which holds for every collapsing run.

6

Related Work

Joint-embedding methods must keep their embeddings from collapsing to a constant, and they do so in different ways. BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) rely on an asymmetric architecture with a predictor and a stop-gradient. Barlow Twins (Zbontar et al., 2021) drives the cross-correlation matrix of the two views toward the identity, and VICReg (Bardes et al., 2022) regularizes the variance and the 6

covariance of each view directly. Collapse can also be partial: Jing et al. (2022) show that the embeddings can end up spanning a lower-dimensional subspace, and RankMe (Garrido et al., 2023a), which we report in Appendix D, measures the effective rank of the embeddings. Analyses of VICReg relate it to contrastive methods (Garrido et al., 2023b) and to the optimization of mutual information (Shwartz-Ziv et al., 2023). Bardes et al. (2022) argue for the standard deviation from the form of the gradient, and their ablations vary the loss weights, not the hinge. The EB-JEPA paper (Terver et al., 2026) sweeps the variance weight on CIFAR-10, again with the standard-deviation hinge and only at weights of 1 and above, and notes that VICReg needs more careful tuning than SIGReg (Balestriero & LeCun, 2025). We found no work that trains with a hinge on the variance, or that brackets the weight at which either hinge collapses. The closest theoretical result is that VICReg does not in general recover full-rank representations (Chapman et al., 2024); it concerns the objective, not the hinge. LeJEPA (Balestriero & LeCun, 2025) replaces the variance and covariance terms with SIGReg, which has a single trade-off weight. VISReg (Wu et al., 2026) keeps a standard-deviation term for scale, with the same gradient argument as VICReg. We tested neither.

7

Limitations

The thresholds we report hold for one setup (ResNet-18, CIFAR, ϵ = 10−4 , fixed invariance and covariance weights). The standard-deviation hinge’s threshold depends on ϵ, which we varied once and on one seed. Both thresholds are also specific to the loss scaling of EB-JEPA (Appendix A). Our claim is the pattern, not the values of µ at which it occurs. All accuracies come from an online linear probe at a fixed budget of 300 epochs; we did not run an offline linear evaluation or a k-nearest-neighbor evaluation. Push/pull compares the sizes of two gradients on one batch, not their directions or the resulting weight updates, so it is a diagnostic consistent with the gradient argument, not a proof of it.

8

Conclusion

VICReg’s authors argue that the hinge must act on the standard deviation, because a hinge on the variance loses its gradient as the embeddings shrink. In our setting, the experiments support the gradient argument but narrow what follows from it. A hinge on the variance trains at the default weight, and both hinges collapse below a threshold weight, with the standard-deviation hinge’s threshold 35 to 130 times lower. What the standard deviation buys is a safety margin: a larger distance between the default weight and the collapse threshold. The measured gradients are consistent with this: the variance hinge’s push √ fades as the features shrink, while the standard-deviation hinge’s push holds until the spread falls below ϵ.

References Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022. James Chapman, Lennie Wells, and Ana Lawry Aguila. Unconstrained stochastic CCA: Unifying multiview and self-supervised learning. In International Conference on Learning Representations, 2024. Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. Facebook Research. VICReg: Official implementation. https://github.com/facebookresearch/vicreg, 2022. Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning, 2023a. 7

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In International Conference on Learning Representations, 2023b. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to selfsupervised learning. In Advances in Neural Information Processing Systems, 2020. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Meta FAIR. EB-JEPA: Energy-based joint-embedding predictive architectures. https://github.com/ facebookresearch/eb_jepa, 2025. Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, and Yann LeCun. An information theory perspective on variance-invariance-covariance regularization. In Advances in Neural Information Processing Systems, 2023. Basile Terver, Randall Balestriero, Megi Dervishi, David Fan, Quentin Garrido, Tushar Nagarajan, Koustuv Sinha, Wancong Zhang, Mike Rabbat, Yann LeCun, and Amir Bar. A lightweight library for energy-based joint-embedding predictive architectures. arXiv preprint arXiv:2602.03604, 2026. Haiyu Wu, Randall Balestriero, and Morgan Levine. VISReg: Variance-invariance-sketching regularization for JEPA training. arXiv preprint arXiv:2606.02572, 2026. Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.

8

A

Relation to the weights of the VICReg paper

The weights in Eq. 4 are those of the EB-JEPA implementation. They are not on the scale of the (λ, µ, ν) of Bardes et al. (2022), because the terms are normalized differently. Table 3 gives each term as written in the VICReg paper, as computed by the official VICReg code (Facebook Research, 2022), and as computed by EB-JEPA (Meta FAIR, 2025). Table 3: The three loss terms as written in the VICReg paper, as computed by the official VICReg code, and as computed by EB-JEPA. D is the embedding size. Term Invariance Variance Covariance

VICReg paper summed over dimensions, averaged over images the two views added squared off-diagonals summed, divided by D

Official VICReg code averaged over both

EB-JEPA as the code

the two views averaged as the paper

the two views added averaged over the D(D − 1) off-diagonals

The paper and its code differ in the first two rows. The weights (25, 25, 1) are the defaults of the code, so we convert against the code. Dividing the code’s loss by λ and comparing it with Eq. 4 term by term gives µours = µ/(2λ) and νours = ν (D − 1)/λ, with D = 1024 here. We convert only the ratios to the invariance weight; the overall scale of the loss, which also sets its balance against weight decay, is left as it is. Table 4: Weights of the VICReg paper and the same weights in our units, for D = 1024. VICReg (λ, µ, ν) (25, 25, 1), the default (10, 10, 1) about (10, 20, 1), not in their table

Ours (µ, ν) (0.5, 41) (0.5, 102) (1, 100), our default

In every row of their Table 7 (arXiv version) that trains, the variance weight equals the invariance weight, which is µ = 0.5 in our units. EB-JEPA’s default, µ = 1, is twice that. We ran both hinges at VICReg’s default ratios, µ = 0.5 and ν = 41, with three seeds. Both train: the std hinge reaches 89.16 ± 0.13 and the var hinge 88.59 ± 0.19, with the std hinge ahead on every seed. At µ = 0.5 and ν = 100, which has to within 2% the ratios of their row (10, 10, 1), the gap is larger: 89.48 ± 0.25 against 87.17 ± 0.16 (Table 1). We did not measure the var threshold at ν = 41. The conversion matches the loss only. VICReg’s experiments use ImageNet, a ResNet-50, an 8192-wide projector and a batch size of 2048, so equal weights do not make the two settings equivalent, and the values of µ we report should not be read as thresholds for VICReg’s own setup.

B

Implementation details

Table 5 lists the settings shared by all runs of the main grid. They are those of the image_jepa example of EB-JEPA (Meta FAIR, 2025), which our code reimplements, except for the projector width (1024 here, 2048 by default in EB-JEPA). For this configuration (batch size 256, 1024×1024 projector, standard-deviation hinge, µ = 1, ν = 100) the EB-JEPA repository reports 90.12%; our reimplementation reaches 89.58 ± 0.15 over three seeds (Table 1).

C

Accuracy at every weight

Table 6 gives the accuracy at every weight µ, including the values of the collapsed runs, which Tables 1 and 2 only mark as collapsed. 9

Table 5: Settings shared by all runs of the main grid. The loss weights are given in Section 3. Encoder Projector Loss Optimizer

Learning rate Batch size, epochs Precision Augmentations

Linear probe

Software

ResNet-18; first convolution 3 × 3 with stride 1 and padding 2, no max-pooling, no classification layer; output size 512 three linear layers with hidden width 1024 and output width 1024; batch normalization and ReLU after the first two computed on the projector outputs of the two views, under mixed precision LARS, momentum 0.9, weight decay 10−4 , η = 0.02 with learning-rate clipping; biases and normalization parameters are excluded from the LARS scaling and from weight decay 0.3, set once per epoch: linear warmup from 3 × 10−5 over 10 epochs, then cosine decay to 0 256 (last incomplete batch dropped), 300 bfloat16 automatic mixed precision per view: random resized crop to 32 × 32 (scale 0.2–1), color jitter (brightness 0.4, contrast 0.4, saturation 0.2, hue 0.1; probability 0.8), grayscale (0.2), solarization (0.1), horizontal flip (0.5), then normalization with the dataset mean and standard deviation one linear layer on the detached encoder output of the first view, cross-entropy loss, trained by the same optimizer and schedule as the model; evaluated on the 10,000 validation images (CIFAR’s test split) without augmentation PyTorch 2.8.0

Table 6: Linear-probe accuracy (%) at epoch 300 for every weight µ, with the values of the runs that Tables 1 and 2 mark as collapsed. Mean over 3 seeds (1, 42, 1000), with the standard deviation in small type. Italic: collapsed run (CIFAR-10 chance 10%, CIFAR-100 chance 1%). ‡ training diverged (loss became NaN; probe at chance). – : not run. µ 0 (no var term) 0.001 0.003 0.01 0.1 0.2 0.3 0.35 0.4 0.5 0.9 1 10 100

CIFAR-10 var hinge std hinge 18.79 ± 3.06 – 16.97 ± 0.29 – 18.58 ± 1.52 – 76.57 ± 0.21 17.95 ± 0.76 82.93 ± 0.61 18.20 ± 1.57 – 18.34 ± 1.47 88.15 ± 0.17 17.64 ± 1.45 – 84.77 ± 0.33 – 87.17 ± 0.16 89.48 ± 0.25 89.43 ± 0.09 – 89.53 ± 0.13 89.58 ± 0.15 86.58 ± 0.05 87.65 ± 0.17 div.‡ div.‡

10

CIFAR-100 var hinge std hinge – – – – – 3.37 ± 0.42 – 35.74 ± 0.29 3.26 ± 0.35 54.52 ± 0.54 – – 3.23 ± 0.29 – – – 56.34 ± 0.55 – 58.90 ± 0.25 – – – 63.33 ± 0.24 63.82 ± 0.27 – – – –

D

Effective rank of the final representations

Figure 4 shows the effective rank of the final representations. For both hinges it falls as µ approaches the collapse threshold from above, and it falls further in the projector output than in the encoder output.

500 400

var hinge std hinge encoder output projector output

eMknaR

300 200 100 0

0.01

0.1

µ

1

Figure 4: RankMe (Garrido et al., 2023a) of the final model against the weight µ (CIFAR-10): the exponential of the entropy of the normalized singular values of the embeddings of the 10,000 validation images, after subtracting their mean. Solid lines are the encoder output (512 dimensions, so at most 512) and dashed lines the projector output (1024 dimensions, at most 1024). Each point is the mean of the 3 seeds, which lie within 7 of each other at every point; colors as in Figure 1. Weights at which the runs collapse (var hinge µ ≤ 0.35, std hinge µ ≤ 0.003) are left out: their embeddings are at numerical resolution and have no meaningful rank.

11

Related documents

• The Standard Deviation in VICReg Buys a Safety Margin#1448146
• A comparison of covariate adjustment approaches under model misspecification in individually randomized trials#231913
• Marginal Girsanov Reweighting: Stable Variance Reduction for Long-Time-Scale Dynamics from Biased Simulation#661896
• Models in the analysis of variance. With discussion#1305759
• Unit-level aggregation breaks coverage in causal foundation models#1386740