Complexity of Linear Regions in Self-supervised Deep ReLU Networks Mufhumudzi Muthivhi, Terence L. van Zyl University of Johannesburg, Johannesburg, South Africa
Abstract
supcon
14000
There has been growing interest in studying the complexity of Rectified Linear Unit (ReLU) based activation networks. Recent work investigates the evolution of the number of piecewise-linear partitions (linear regions) that are formed during training. However, current research is limited to examining the complexity of models trained in a supervised way. Self-Supervised Learning (SSL) differs in that it directly optimises the representation space using a loss function to enhance the model’s performance across multiple downstream tasks. This study investigates the local distribution of linear regions produced by SSL models. We demonstrate that the evolution of linear regions correlates with the representation quality by utilising SplineCam to extract two-dimensional polytopes near the data distribution. We track the number, area, eccentricity, and boundaries of regions throughout training. The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets, MNIST and FashionMNIST. The analysis of the experimental results shows that selfsupervised methods create substantially fewer regions to achieve comparable accuracy to supervised models. Contrastive methods rapidly expand regions over time, whereas self-distillation methods tend to consolidate by merging neighbouring regions. Lastly, we can detect representation collapse early within the geometric space of linear regions. Our analysis suggests that polytopal metrics can serve as reliable indicators of representation quality and model performance.
12000 Number of Regions
arXiv:2604.24393v1 [cs.LG] 27 Apr 2026
{mmuthivhi, tvanzyl}@uj.ac.za
10000
classifier
moco
8000
triplet
6000
byol simclr simsiam
4000 2000 0.5
0.6
0.7 Accuracy
0.8
0.9
Figure 1. Self-supervised methods (SimCLR, BYOL, SimSiam) produce fewer linear regions than their supervised counterparts (Classifier, Triplet, SupCon) while achieving comparable accuracy. Both contrastive and self-distillation approaches leverage positive pairings, resulting in the rapid merging of linear regions due to shared activation patterns. The opacity reflects the progression of training over epochs. Low-opacity points depict results achieved early in training, while darker points represent results achieved later.
space. Understanding the structure and distribution of regions near the data distribution has become an active area of research. Practitioners count the number of regions as a measure of network complexity [27, 33, 34]. Humayun et al. propose SplineCam for interpretation and visualisation of the exact number of regions from a two-dimensional input space [20]. Most current work focuses on analysing the linear regions generated by supervised classifier networks [15, 22]. Some studies have explored the local geometry of generative models [19, 21]. More recent work examines the complexity of input space partitioning in spiking neural networks [29]. However, the geometric properties of Self-Supervised Learning (SSL) representations remain largely underexplored. SSL has shown remarkable perfor-
1. Introduction Continuous Piecewise Linear (CPWL) neural networks partition the input space into a finite set of convex polytopes, referred to as linear regions [15]. Within each region, the network applies a single affine transformation. Regions are formed by applying a nonlinear activation function over the output values of the affine transformation. The set of all linear regions produced by a neural network defines a piecewise-linear tessellation of the input 1
Accepted to CVPR Findings 2026
mance on several downstream tasks [2, 9, 35]. It intrinsically generates supervision signals by augmenting input data. This work investigates the local distribution of linear regions produced by SSL methods. We transform a highdimensional input space into a two-dimensional space and partition it to obtain its linear regions. Specifically, we consider contrastive and self-distillation methods [5, 6, 13, 17]. Self-supervised contrastive learning captures the similarity among instances. They minimise the distance between positive pairs within the embedding space while simultaneously trying to keep negative pairs apart. However, they require large batch sizes or memory banks to store sufficient negative samples, thereby preventing overfitting. Selfdistillation methods forgo the need for negatives by using a predictor network to predict the representation of views from the same image. Several specific conditions have been proposed to prevent representations from collapsing into degenerate solutions [23, 36, 40]. We hypothesise that different SSL objectives produce unique geometric properties in the partitioned input space. In contrastive methods, positive pairs encourage samples to share activation patterns, essentially merging regions together. Negative pairs exert a repulsive force that tears regions apart. In contrast, self-distillation methods do not use negative pairs; thus, regions merge rapidly. This effect reflects on the heightened risk of representation collapse for self-distillation methods. Figure 1 contrasts the performance of supervised and self-supervised methods. All the SSL methods produce fewer regions and achieve accuracy comparable to that of supervised variants in terms of performance. Furthermore, results show that the contrastive objective produces a larger number of isotropic small regions that fit the in-distribution data well. Self-distillation encourages fewer, larger, and more anisotropic partitions, which correlates with improved generalisation performance. We contribute to the existing literature by: • track the number, area, eccentricity, and boundaries of regions throughout training from a projected highdimensional input space, • compare the evolution of regions from supervised, contrastive and self-distillation methods, • simulates the effects of representation collapse within the polytopal space, • provides a qualitative analysis of linear regions and their density within the projected high-dimensional input space.
Linear (CPWL) nonlinearity such as a Rectified Linear Unit (ReLU), leaky-ReLU or MaxOut operation [12]. CPWL functions allow practitioners to assess the complexity of a neural network by counting the number of linear partitions (linear regions) formed by a network. Montúfar et al. show that the number of linear regions grows exponentially with depth [27]. Raghu et al. relate the number of regions to expressivity and generalization [32]. Novak et al. demonstrate that neural networks have strong robustness near the training data distribution, especially with data augmentation and ReLU activations [30]. Serra et al. provides tighter bounds on the number of regions compared to previous studies [34]. Several works utilise CPWL functions to investigate properties in generative models, Recurrent Neural Networks, or spiking neural networks, including the understanding of classifiers, out-of-distribution detection, and the interpretation and visualisation of regions emerging from supervised classifiers [4, 10, 15, 19, 20, 22, 24, 29].
2.2. Self-supervised Learning Recently, Self-supervised Learning (SSL) methods have achieved performance comparable to that of their supervised counterparts across a range of downstream tasks [35]. SSL utilises a formulated pretext-task to generate several pseudo targets from the input data [14]. An objective function learns a representation space that is invariant to augmentations made to the input data. Early work utilised contrastive learning to associate positive pairs with each other while pushing negative pairs apart [5, 17]. Self-distillation methods maximise the similarity between two augmented views of the same image [6, 13]. One limitation is that they do not explicitly include a repulsive force (negative examples) to prevent the representations from collapsing into degenerate solutions. Hence, the majority of work in this area focuses on devising implicit architectural constraints and optimisation strategies, such as a prediction head, stop-gradient operation, or a redundancy-reduction objective [2, 3, 39].
2.3. Representation Learning Self-supervised representations have been shown to convey stronger transferability and generalisation characteristics [2, 8, 9, 35]. Practitioners assess the quality of representations by freezing the encoder and extracting a feature vector for each image [37]. Garrido et al. propose to analyse the singular value distribution of the learned embeddings to assess representation quality [11]. Chen and He [6] uses the mean of the standard deviation of the L2-normalised feature vector to determine the level of representation collapse. Jha et al. [23] suggests that minimising the magnitude of the expected representation over all data samples while simultaneously maximising the magnitude of individual samples over different data augmentations helps prevent represen-
2. Related Work 2.1. Continuous Piecewise-Linear Functions A neural network consists of one or more linear transformations followed by a non-linear operation. Current deep networks predominantly use a Continuous Piecewise2
Accepted to CVPR Findings 2026
Neural Network
Linear Region
Activation Pattern Input Layer Layer 1
Layer 2
Layer 3
Layer 4
Output Layer
Figure 2. Illustration of how a ReLU network partitions the input space into linear regions. The orange neurons are active. The distinct activation pattern corresponds to a single linear region highlighted in blue on the right plot. Each region applies a unique affine transformation to the partitioned values.
tation collapse. This work extends these efforts by introducing a polytopal analysis of learned representations. By quantifying the number, geometric shape, and evolution of linear regions within the network’s activation space, we provide a geometric view of the encoder’s features across supervised, contrastive, and self-distillation methods.
3. Methodology
High Volume
High Eccentricity
High Number of Boundaries
Low Volume
Low Eccentricity
Low Number of Boundaries
Figure 3. Illustration of the geometric properties of linear regions. Each region is characterised by its volume, eccentricity, and boundary count.
3.1. Definitions Let f denote a Continuous Piecewise Linear (CPWL) neural network composed of layers with nonlinear activation functions σ. For an input x ∈ Rd and parameters θ = {W, b}, a single-layer transformation can be written as f (x; θ) = σ(W x + b),
be expressed as a union of linear regions: Rd ≈
(1)
Ri ,
(3)
i=1
where Ri denotes the i-th region and NR is the total number of distinct regions formed by the activation boundaries. The number and structure of these regions serve as a measure of the network’s complexity [15, 20].
where W ∈ Rm×d and b ∈ Rm represent the weight matrix and bias vector, respectively. For ReLU-based networks, the activation function is defined as σ(z) = max(0, z),
N R [
(2)
3.2. Geometric Properties
d
which creates a partitioning of the input space R into a finite set of convex polytopes [12]. Each region corresponds to a unique activation pattern of the hidden units, such that within each region, f behaves as an affine transformation. Figure 2 depicts a unique activation pattern produced by a neural network and its corresponding region mapped to a two-dimensional input grid. Formally, the input space can
We consider the volume, eccentricity, and boundaries of each polytope to characterise the geometric properties of the learned representation space [15]. Volume quantifies the extent to which the input space is subjected to a single affine transformation. A region with a larger volume suggests a greater contribution to the final representation of x. Eccentricity captures the degree of anisotropy. Circular 3
Accepted to CVPR Findings 2026
Table 1. Number and properties of regions generated by a backbone neural network with a width 64 and depth of 5 trained over the moons dataset. Model SupervisedClassifier SupervisedContrastive Self-supervisedContrastive Self-distillation
Number Of Regions
Mean Area (×10−4 )
Mean Eccentricity
Mean Boundaries
9232
6.70
0.793
3.52
9679
6.42
0.774
3.52
4354
13.8
0.717
3.53
3602
15.5
0.705
3.55
regions have a lower eccentricity, ensuring that the learnt representations remain farther from activation boundaries. Elongated regions with high eccentricity are often located near the activation boundaries. Small changes in input can trigger shifts in the active set of neurons. Lastly, we refer to the vertices of the polytope as the region’s boundaries. More boundaries suggest a complex partitioning of the input space. Highly partitioned input space could lead to less stability since a perturbation of the data point x can cause it to jump between several regions. Figure 3 provides an illustration of the effects of each geometric property on data point x.
Supervised Classifier
Supervised Contrastive
Self-supervised Contrastive
Self-distillation
Volume: In two dimensions, we compute the volume of a region Ri with vertices {(a0 , b0 ), . . . , (an−1 , bn−1 )} using the shoelace formula: n−1
1 X ak bk+1 − bk ak+1 , Vi = 2
Figure 4. Density of linear regions produced near the moon’s data distribution (left). The regions are generated by a backbone neural network with a width 64 and a depth of 5.
(4)
k=0
such that the last set of points is equal to the first (an , bn ) = (a0 , b0 ).
3.3. 2-dimensional Input Space
Eccentricity: To characterise the anisotropy of each region, we compute an eccentricity measure based on its axisaligned bounding box. Let
We first provide an intuitive visualisation of our framework by using a two-dimensional input space that sits near the data distribution. To examine how the network partitions the input space, we define a uniform sampling grid D = {(ak , bk ) | ak , bk ∈ [0, 1]}.
∆x = max(ak )−min(ak ),
∆y = max(bk )−min(bk ), (5) given the major max(∆a , ∆b ) and minor min(∆a , ∆b ) axis lengths, we describe the eccentricity as s 2 min(∆a , ∆b ) Ei = 1 − , (6) max(∆a , ∆b )
(7)
Each point (ak , bk ) is propagated through the network to record the activation pattern across all ReLU operations. The subset of points in D that produce the same activation pattern belongs to the same linear region Ri . Thus, the input domain can be approximated as D≈
with Ei = 0 for degenerate (single-point) regions.
N R [
Ri ,
(8)
i=1
which is the union of linear regions. By dividing the mesh according to the activation patterns, we obtain a piecewiselinear tessellation of the input space, as illustrated in Figure 4. We train a neural network f on the two-dimensional
Boundaries: The number of boundaries Bi = n is simply the total number of unique edges for a two-dimensional polytope. 4
Accepted to CVPR Findings 2026
Table 2. Total number of parameters (in thousands) for the encoder, projection, and prediction heads across datasets. Backbone
Supervised Classifier Triplet Loss SupCon Self-Supervised Contrastive SimCLR MoCo Self-distillation SimSiam BYOL
66.9 for each
MNIST
FashionMNIST
166 for each
Model
Projection
learning extends contrastive learning by pulling together representations of samples from the same class while treating samples from different classes within the mini-batch as negatives [25]. We also study three more self-supervised methods. Simple Framework for Contrastive Learning (SimCLR) is also a contrastive learning objective that does not use labels [5]. SimCLR applies augmentations to a single image to derive positive pairs and samples negatives from the mini-batch. Momentum Contrast (MoCo) uses a queue of negative samples generated by a momentum encoder [17]. Similarity Siamese (SimSiam) and Bootstrap Your Own Latent (BYOL) are both self-distillation methods that do not make use of negative samples [6, 13]. SimSiam uses a dual parallel encoder with a prediction head and a stop gradient operation to prevent the model from collapsing. BYOL further applies two separate identical encoders. A frozen encoder has a moving average over the weights to update the target encoder.
Prediction
MNIST
FashionMNIST
MNIST
FashionMNIST
0.6 – -
18 – -
– – –
– – –
6.3 6.2
25 24.8
– –
-
10.5 12.6
41.5 49.7
1.5 4.3
4.2 6.7
Moons dataset and visualise how the model develops linear regions that partition the input domain. Table 1 presents the number of regions and geometric properties of the linear regions produced by a supervised classifier, supervised contrastive, a self-supervised contrastive and a self-distillation model. The self-distillation model has the largest volume of linear regions on average, and the classifier exhibits the highest eccentricity and the largest number of regions. This result correlates visually with the illustration in Figure 4
4.2. Datasets Following prior work, we train and present results over two benchmark datasets [15, 16, 20]. MNIST consists of 70,000 grayscale images of handwritten digits [26]. FashionMNIST is a more challenging alternative containing images of clothing items [38]. All images are grayscale with 60,000 samples for training and 10,000 for testing.
3.4. High-dimensional Input Space For a higher-dimensional input space, we adopt a projection-based strategy to partition the input space. Specifically, following the experimental setup of Novak et al. and Hanin and Rolnick, we visualise the local structure of linear regions by projecting a small neighbourhood of the data distribution onto a two-dimensional plane within the d-dimensional input space [15, 30]. The plane is defined by three data points from distinct classes (zero, one, and two) and centred at the circumcenter of these examples. We employ SplineCam to automatically extract and visualise the continuous piecewise-linear partitions [20]. SplineCam utilises spline-based interpolation to map activation boundaries.
4.3. Implementation Details The supervised loss functions are adopted from PyTorch Metric Learning, and the self-supervised models are taken from Lightly [1, 28]. Each model runs for 100 epochs. We apply random resized cropping, horizontal flipping, greyscaling and Gaussian blur to our augmentation strategy. All training configurations and settings are listed in the supplementary material. We use a single NVIDIA GeForce GTX 5070 Ti GPU to accelerate training.
5. Results
4. Experimental Setup
Table 3. Accuracy and linear regions properties over the MNIST dataset.
4.1. Models For each supervised and self-supervised variant, we use the same encoder architecture depending on the dataset used. Table 2 presents the number of parameters for each method. We adopt the same scaling factors for the projection and prediction heads as specified in the original proposed implementations. We investigate three supervised methods. The classifier uses a linear layer to compute a linear combination of the activations from the previous encoder layer [31]. Triplet loss associates an image with another positive image belonging to the same class while pushing the negative image away [18]. Supervised Contrastive (SupCon)
Models Accuracy supervised contrastive Classifier 0.8800 Triplet loss 0.8531 SupCon 0.8557 self-supervised contrastive SimCLR 0.8351 MoCo 0.6641 self-distilliation Simsiam 0.7915 BYOL 0.6772
5 Accepted to CVPR Findings 2026
Regions
Volume
Eccentricity
Boundaries
12817 7365 11349
0.1164 0.1789 0.1042
0.7521 0.7563 0.7435
3.534 3.523 3.549
4297 8731
0.3362 0.1644
0.7632 0.7698
3.526 3.544
3653 3730
0.3986 0.3924
0.7725 0.7989
3.567 3.539
1.0 0.6 0.4
0.6
0
20
40 60 Epochs
80
supcon triplet moco byol simclr simsiam
0.4
simclr simsiam byol
0.2 0.0
0.8
Number of Regions
Number of Regions
0.8
classifier
1.0
classifier supcon moco triplet
0.2 0.0
100
0
(a) MNIST
20
40 60 Epochs
80
100
(b) FashionMNIST
Figure 5. Evolution of linear regions over 100 epochs generated by a backbone encoder. The supervised methods (Classifier, SupCon and MoCo) observe a slight decrease in the number of regions, followed by a rapid increase as training progresses. In contrast, self-supervised methods experience an early decline before gradually increasing. On average, contrastive self-supervised methods (SimCLR and MoCo) expand their region counts more rapidly than the self-distillation methods (SimSiam and BYOL). The use of negative samples in contrastive methods promotes partitioning of the input space over time. The absence of a repulsive force in self-distillation approaches results in slower growth and even stagnation of region complexity.
Table 4. Accuracy and linear regions properties over the FashionMNIST dataset.
gions throughout training.
5.2. Accuracy and Regions Table 3 and Table 4 present the number of regions produced by the encoder and its accuracy on the validation set over the MNIST and FashionMNIST datasets, respectively. Supervised methods require high region counts to achieve a moderately high accuracy rate. SupCon requires 7,200 regions to achieve an accuracy of 83.69%, whereas SimCLR requires only 4,297 regions at an accuracy of 83.53%. Although the SSL methods converge at lower accuracy, they require substantially fewer regions to achieve comparable accuracy. The reduction in complexity is a desirable property, as prior work shows that models with fewer regions exhibit improved robustness and generalization [7]. Moreover, Humayun et al. demonstrates that the delayed generalisation phenomenon, known as grokking, also emerges at reduced complexity [22]. SSL methods produce an almost linear relationship between the number of regions and to accuracy achieved. After merging regions during initialisation, simply splitting the same regions correctly aligns with the data distribution.
Models Accuracy supervised contrastive Classifier 0.8012 Triplet loss 0.8204 SupCon 0.8072 self-supervised contrastive SimCLR 0.7568 MoCo 0.7091 self-distilliation Simsiam 0.7523 BYOL 0.7644
Regions
Volume
Eccentricity
Boundaries
78178 41413 40988
0.0235 0.0413 0.0371
0.7339 0.7386 0.7440
3.549 3.547 3.542
8080 35996
0.2139 0.0475
0.7378 0.7399
3.542 3.550
10731 20946
0.1602 0.0898
0.7430 0.7280
3.549 3.555
5.1. Number Of Regions Figure 5 depicts the evolution of linear regions of each model from the first to the 100th epoch. Supervised and self-supervised learning methods exhibit distinct learning progressions. The supervised methods exhibit a rapid rise in the number of regions. Within the first 10 epochs, SupCon regions increase 137% and 63% for the MNIST and FashionMNIST datasets, respectively. Around the 20th epoch, a slight and steady decrease in regions follows. An early plateau suggests a merging of regions to accurately fit the data distribution. The self-supervised methods experience an inverse relationship. The number of regions after initialisation plateaued at 47%, 62%, and 62% for SimCLR, SimSiam, and BYOL, respectively, on the MNIST dataset. A similar relationship is observed for the FashionMNIST dataset. The MoCo and SimCLR follow a smoother trajectory of steady region growth throughout training. Simsiam and BYOL increase regions slightly, then slowly merge re-
5.3. Geometry Of Regions The SSL methods produce regions that are, on average, about 2.42 and 3.76 times the volume of the supervised methods for MNIST and FashionMNIST, respectively. The final mean eccentricity and number of boundaries are around less than 0.8 and 3.6 across each of the models. Figure 6 illustrates the evolution of the three geometric properties from initialisation to the 1st, 50th, and 99th epoch on the MNIST dataset. The eccentricity and number of boundaries exhibit somewhat skewed distributions, suggesting that the mean may not provide a fully representative 6
Accepted to CVPR Findings 2026
0.8 simclr
0.4 0.2 0.0
0
10
20
0.8
30
40
random Epoch 1 Epoch 50 Epoch 99
simsiam
0.6 0.4 0.2 0.0
0
5
10
Density
count
2000
0 0.00 0.25 0.50 0.75 1.00
6 random Epoch 1 Epoch 50 Epoch 99
0.6
1
15
0 3
3
3000
2
2000
4
5
6
7
8
9
count
4
4000
1
1000
0 0.00 0.25 0.50 0.75 1.00 6 4
0 3
4
5
6
7
8
4
5
6
7
8
3000 2000
count
2
Number of Boundaries 6000
2
Density
0
random Epoch 1 Epoch 50 Epoch 99
Eccentricity
Density
supcon
0.8 0.6 0.4 0.2 0.0
Volume
2
1000
0 0.00 0.25 0.50 0.75 1.00
0 3
Figure 6. Evolution of region area, eccentricity, and boundary count distributions across epochs for SupCon, SimCLR, and SimSiam on MNIST dataset.
characterisation. However, SimCLR and SimSiam slightly alter the tails of their eccentricity distributions. This progression indicates that a subset of the regions becomes more elongated as training progresses. The self-distillation methods produce, on average, a higher final mean eccentricity than the SSL contrastive methods.
step, until it collapses.
5.5. Discussion The analysis of the results reveals a clear distinction in the regional evolution between supervised and SSL methods. SSL exhibits restrained region growth while supervised methods continuously partition its input space. As a result, SSL methods require substantially fewer regions to achieve comparable accuracy performance. The reduced region count in SSL corresponds to larger average region volumes. Specifically, contrastive SSL methods exhibit lower eccentricity compared to self-distillation methods. The contrastive objective produces a greater number of regions than self-distillation methods and is more isotropic. Lastly, a comparison of the representation standard deviation and the number of regions across training steps indicates that collapse becomes evident earlier in the geometric space. This is a result of regions merging rapidly with no repulsive force in the representation space to discourage shared activation patterns.
5.4. Representation Collapse We show that the polytopal space can indicate representation collapse early within training. To simulate collapse, we remove the prediction head from the SimSiam framework. Figure 7 jointly tracks the number of linear regions and the standard deviation of feature representations over 700 steps. The representation standard deviation reaches its highest value of 0.02 at 100 steps, then fluctuates for the next 300 steps. The standard deviation only starts to gradually decrease at the 400th step. In contrast, the number of regions rapidly decreases by the 100th step. By approximately 150 steps, the network retains only about 5% of the regions present at initialisation. After this point, the number of active regions stabilises between 5% and 20% of the original count. Figure 8 depicts the partitioned input space at initialization, 100th step and at the end of training. The network transitions from a highly partitioned structure at initialisation to progressively larger regions at the 100th
6. Conclusion The study investigates the local distribution of linear regions produced by Self-supervised learning methods (SSL). Furthermore, we compared how supervised, contrastive, and 7
Accepted to CVPR Findings 2026
0.30
0.20 0.15 0.10 0.05 0.00
7. Acknowledgment
0.0200 0.0175 0.0150 0.0125 0.0100 0.0075 0.0050 0.0025 0.0000
We extend our gratitude for the support given by the Center for Artificial Intelligence (CAIR), Institute for Intelligent Systems and the University of Johannesburg, South Africa.
Representation Std
Number of Regions
0.25
0
100
200
Collapse
300
400 Steps
No Collapse
500
600
700
References [1] Lightly AI. Lightly: A computer vision framework for selfsupervised learning, 2025. 5 [2] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912– 9924, 2020. 2 [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 2 [4] Josue Casco-Rodriguez, Tyler Burley, CJ Barberan, Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Visualizing linear rnns through unrolling. In Latinx in AI@ NeurIPS 2024, 2024. 2 [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR, 2020. 2, 5 [6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750–15758, 2021. 2, 5 [7] Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu networks via maximization of linear regions. In the 22nd International Conference on Artificial Intelligence and Statistics, pages 2057–2066. PMLR, 2019. 6 [8] Akshay Raj Dhamija, Touqeer Ahmad, Jonathan Schwan, Mohsen Jafarzadeh, Chunchun Li, and Terrance E Boult. Self-supervised features improve open-world learning. arXiv preprint arXiv:2102.07848, 2021. 2 [9] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5414–5423, 2021. 2 [10] Matteo Gamba, Adrian Chmielewski-Anders, Josephine Sullivan, Hossein Azizpour, and Marten Bjorkman. Are all linear regions created equal? In International Conference on Artificial Intelligence and Statistics, pages 6573–6590. PMLR, 2022. 2 [11] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International conference on machine learning, pages 10929–10974. PMLR, 2023. 2 [12] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and
Representation Std
Figure 7. Relationship between the number of linear regions and representation standard deviation during training for Simsiam without a prediction head to simulate collapse. The number of regions decreases rapidly earlier in training compared to the standard deviation of the representation.
Initialization
Step 100
Collapse
Figure 8. Illustration of region evolution during representation collapse. The network’s activation space transitions from a highly partitioned structure at initialisation to regions with high volume by step 100, where most linear regions have merged. This trajectory only worsens until collapse.
self-distillation methods progressively learn within the geometric space. Our findings reveal that the geometric space remains a promising approach for assessing representation quality. Future research could investigate techniques to directly optimise the polytopal space. A novel regularisation strategy could encourage fewer regions with greater volume and controlled eccentricity to improve the robustness and transferability of the model. Out-of-Distribution detection benefits from identifying the location of in-distribution data in the representation space. In the geometric space, a highly partitioned input space may indicate overfitting, revealing regions densely aligned with in-distribution data. Lastly, in continual learning, researchers could leverage self-distillation methods to create as few regions as possible while avoiding collapse within the current task. When new tasks are introduced, contrastive self-supervised methods can then expand the polytopal space by generating additional regions to accommodate new data. 8
Accepted to CVPR Findings 2026
[25] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020. 5 [26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 2002. 5 [27] Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014. 1, 2 [28] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Pytorch metric learning. arXiv preprint arXiv:2008.09164, 2020. 5 [29] Duc Anh Nguyen, Ernesto Araya, Adalbert Fono, and Gitta Kutyniok. Time to spike? understanding the representational power of spiking neural networks in discrete time. arXiv preprint arXiv:2505.18023, 2025. 1, 2 [30] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08760, 2018. 2, 5, 1 [31] Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv preprint arXiv:1312.6098, 2013. 5 [32] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning, pages 2847–2854. PMLR, 2017. 2 [33] David Rolnick and Max Tegmark. The power of deeper networks for expressing natural functions. arXiv preprint arXiv:1705.05502, 2017. 1 [34] Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. In International conference on machine learning, pages 4558–4566. PMLR, 2018. 1, 2 [35] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. 2 [36] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268–10278. PMLR, 2021. 2 [37] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733–3742, 2018. 2 [38] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashionmnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. 5 [39] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via
statistics, pages 315–323. JMLR Workshop and Conference Proceedings, 2011. 2, 3 [13] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020. 2, 5 [14] Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9052–9071, 2024. 2 [15] Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, pages 2596–2604. PMLR, 2019. 1, 2, 3, 5 [16] Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems, 32, 2019. 5 [17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020. 2, 5 [18] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737, 2017. 5 [19] Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Polarity sampling: Quality and diversity control of pre-trained generative networks via singular values. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10641–10650, 2022. 1, 2 [20] Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, and Richard G Baraniuk. Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3789–3798, 2023. 1, 2, 3, 5 [21] Ahmed Imtiaz Humayun, Ibtihel Amara, Cristina Vasconcelos, Deepak Ramachandran, Candice Schumann, Junfeng He, Katherine Heller, Golnoosh Farnadi, Negar Rostamzadeh, and Mohammad Havaei. What secrets do your manifolds hold? understanding the local geometry of generative models. arXiv preprint arXiv:2408.08307, 2024. 1 [22] Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why. arXiv preprint arXiv:2402.15555, 2024. 1, 2, 6 [23] Abhishek Jha, Matthew B Blaschko, Yuki M Asano, and Tinne Tuytelaars. The common stability mechanism behind most self-supervised learning approaches. arXiv preprint arXiv:2402.14957, 2024. 2 [24] Xu Ji, Razvan Pascanu, R Devon Hjelm, Balaji Lakshminarayanan, and Andrea Vedaldi. Test sample accuracy scales with training sample density in neural networks. In Conference on Lifelong Learning Agents, pages 629–646. PMLR, 2022. 2
9 Accepted to CVPR Findings 2026
redundancy reduction. In International conference on machine learning, pages 12310–12320. PMLR, 2021. 2 [40] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X Pham, Chang D Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. arXiv preprint arXiv:2203.16262, 2022. 2
10 Accepted to CVPR Findings 2026
Complexity of Linear Regions in Self-supervised Deep ReLU Networks Supplementary Material Introduction
80000
This supplementary section provides additional details about our experimental setup. We outline the full hyperparameter configurations used for each model across both benchmark datasets. We further elaborate on the highdimensional projection strategy employed. We also expand on the high-dimensional projection strategy adopted. Finally, we include additional illustrations of the results from the FashionMNIST dataset and include supplementary visualisations of the geometric structure of linear regions at the final training epoch for all models.
70000
Number of Regions
50000
30000
10000 0.5
Table 5. Training hyperparameter configurations for each model on MNIST dataset.
Batch Size
Learning Rate
Weight Decay
Temperature
0.01 0.02 0.01 0.01 0.008 0.04 0.04
0.0
100 for each
256 256 256 256 64 256 256
0.05 0.05 0.2 0.2 -
1.0e-5 for each
Epochs
Weight Decay
Temperature
0.0
100 for each
0.02 0.04 0.01 0.01 0.01 0.04 0.04
0.05 0.05 0.2 0.2 -
1.0e-5 for each
Learning Rate
256 256 256 256 64 256 256
0.8
0.9
where β0 , β1 ∈ R are the coefficients that satisfy the linear system 1 a M = b (10) b 2
Models Batch Size
0.7 Accuracy
3. an affine mapping that transforms high-dimensional coordinates into the two-dimensional input space. Let v1 = x1 − x0 and v2 = x2 − x0 . The circumcenter C of the triangle formed by x0 , x1 and x2 can be expressed as: C = x0 + β 0 v1 + β 1 v2 (9)
Table 6. Training hyperparameter configurations for each model on FashionMNIST dataset.
Epochs
0.6
Figure 9. Number of regions and accuracy achieved by supervised and self-supervised methods on FashionMNIST dataset. The opacity reflects the progression of training over epochs. Lowopacity points depict results achieved early in training, while darker points represent results achieved later.
Models
Classifier Triplet loss SupCon SimCLR MoCo Simsiam BYOL
moco supcon triplet byol simclr simsiam
40000
20000
Hyperparameters
Classifier Triplet loss SupCon SimCLR MoCo Simsiam BYOL
classifier
60000
such that T v v M = 1T 1 v1 v2
1 ∥v1 ∥2 v1T v2 , b= v2T v2 2 ∥v2 ∥2
(11)
solving the linear system gives the circumcenter. Next we use the Gram–Schmidt process to obtain the orthonormal basis vectors. Given the center vectors w1 = x1 − w1 C and w2 = x2 − C. If u1 = ∥w is the normalized vector 1∥ then: ⊤ w2 u1 w2⊥ = w2 − u1 (12) 1 u⊤ 1u
High-dimensional Projection We describe the process used to project a two-dimensional point to a high-dimensional space [30, 33]. Let x0 , x1 and x2 ∈ Rd be three sampled input images. We construct: 1. the circumcenter of the triangle produced by the three points, 2. the orthonormal basis vector of the plane,
w⊥
points in the perpendicular direction. Let u2 = w2⊥ then 2
the orthonomal basis vector is defined as {u1 , u2 } ⊆ Rd . 1
Accepted to CVPR Findings 2026
random Epoch 1 Epoch 50 Epoch 99
simclr
10.0 7.5 5.0 2.5 0.0 0.0
2.5
5.0
7.5 10.0 12.5 random Epoch 1 Epoch 50 Epoch 99
simsiam
3 2 1 0 0.00
0.25
0.50
0.75
1.00
count
0 0.00 0.25 0.50 0.75 1.00
0 3
4
5
6
7
8
9 10
4
5
6
7
8
9 10
10000
2
count
0 0.0 0.2 0.4 0.6 0.8 1.0
10000
1
1 0 0.00 0.25 0.50 0.75 1.00 3
5000 0 3
10000
2
count
1
Number of Boundaries 20000
2
Density
2
Eccentricity
3
Density
supcon
3
random Epoch 1 Epoch 50 Epoch 99
Density
Volume
1 0 0.00 0.25 0.50 0.75 1.00
5000 0 3
4
5
6
7
8
9
Figure 10. Evolution of region area, eccentricity, and boundary count distributions across epochs for SupCon, SimCLR, and SimSiam on FashionMNIST dataset.
The input x can be obtained from the point [a, b] in the twodimensional space by setting x(a, b) = C + au1 + bu2
(13)
a C b 1
(14)
similarly x = u1
u2
such that T = u1 u2 C is the transformation matrix from a two-dimensional space to a high-dimensional space.
2 Accepted to CVPR Findings 2026
classifier
triplet
supcon
simclr
moco
simsiam
byol
Figure 11. Final geometry of regions at the 100th epoch for each model on the MNIST dataset. Supervised methods (Classifier, Triplet and SupCon) produce highly dense and smaller regions. Self-supervised methods (SimCLR, MoCo, SimSiam, and BYOL) produce less dense regions with larger volumes. Contrastive self-supervised methods (MoCo and SimCLR) have linear regions that are more dense than the self-distillation methods (Simsiam and BYOL). The use of negatives as a repulsive force within the embedding space partitions the geometric space further.
3 Accepted to CVPR Findings 2026
classifier
triplet
supcon
simclr
moco
simsiam
byol
Figure 12. Final geometry of regions at the 100th epoch for each model on the FashionMNIST dataset. The same qualitative trends observed on MNIST are reproduced here. Regions consolidate in self-distillation methods and finer partitioning in supervised and selfsupervised contrastive models.
4 Accepted to CVPR Findings 2026