arXiv:2604.18390v1 [cs.LG] 20 Apr 2026
Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus Esteban Rodríguez-Betancourt
Edgar Casasola-Murillo
Posgrado en Computación e Informática Universidad de Costa Rica [email protected]
Escuela de Ciencias de la Computación Universidad de Costa Rica [email protected]
Abstract—In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood. In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of selfdistillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with nontrivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup. Index Terms—Self-supervised learning; Representation learning; Self-distillation; Feature extraction
I. I NTRODUCTION Representation Learning consists in learning useful, general purpose data representations, that capture the structure and semantics of the input space [1], [2]. Self-supervised learning (SSL) methods have demonstrated that they are well suited to learn useful representations, bypassing the need for labeled datasets. These methods have been applied successfully to diverse domains such as text and vision representations. While self-supervised learning methods are undeniably successful, they usually rely on components that are not trivial to devise. For instance, a family of methods such as BYOL [3] and DINO [4] rely on self-distilling representations from a randomly initialized network called teacher, while this teacher is updated using EMA mechanism. While the mechanisms clearly works, we lack a theoretical understanding of how they work [5]. In this work, we investigate the effects on learning representations of self-distillation from a group of untrained networks. We observe non-trivial improvements in downstream tasks such as CIFAR-10 classification. However, those gains are dependent of the learning rate and model architecture. In order to isolate any possible learning to just the selfdistillation dynamics, we opted to strip away many components usually found in self-distillation mechanisms, such as pretext tasks that produce a view asymmetry, loss function mutations, student/teacher asymmetry, projector/predictor layers.
Even after removing all those components, we were able to get learned representations that were better than our random baseline, for downstream tasks such as CIFAR-10 [6] classification. Additionally, this learning mechanism remains stable and avoids representational collapse. Under this simple setup, we studied the effect of several hyperparameters, such as learning rate, loss function, number of peers and teachers and differences based on network architecture. Additionally, we did an early exploration of what may have been learned by the networks under this setup. II. C ONCEPTS AND R ELATED WORK A. Representation Learning and Self-Supervision Representation learning aims to extract features from data that capture semantic structure and generalize well to downstream tasks [1], [2]. In the absence of labeled data, selfsupervised learning (SSL) has become a widely used strategy, particularly in domains such as computer vision and natural language processing. SSL methods learn from the data itself by designing pretext tasks or objectives that encourage the model to encode meaningful information. Notable examples include SimCLR with contrastive losses [7], and Barlow Twins for redundancy reduction [8]. B. Self-Distillation in SSL A successful branch of SSL focuses on self-distillation, where one model is trained to predict the output of another. In BYOL [3] and SimSiam [9], a student network learns to align its representations with those of a teacher network that receives a different view of the same input. These methods typically rely on architectural asymmetry, including a predictor head and a stop-gradient operation, to avoid trivial solutions. DINO [4] replaces the predictor with a softmax-based loss that uses centering and sharpening to prevent collapse, and employs an EMA-updated teacher to stabilize training. C. Preventing Collapse in Non-Contrastive SSL A typical issue in self-distilled self-supervised learning models is that the network can end up collapsing most of the inputs into the same embedding. This issue has been observed in multiples methods and many different solutions have been proposed. In SimSiam [9], authors noticed that stopping the gradient from to the “teacher” side was fundamental to stop
the collapse. DINO [4] relied on loss centering and scaling to prevent the collapse into the same representation. Other authors opt to tackle directly the collapse. For instance, SimDINO [10] introduced an explicit expansion component into the loss based on coding rate regularization. D. Random Teachers and Self-Distillation without Training Recent work by Sarnthein et al. [11] explores a surprising setting where a fixed randomly initialized teacher can still guide a student to learn transferable representations. Their results show that even learning to mimic a frozen untrained teacher is sufficient for learning meaningful features. They also highlight how proximity between teacher and student initialization affects learning dynamics. This behavior has been observed previously, for example, it is mentioned as a core motivation behind BYOL [3]. Like Sarnthein et al., we explore the question of what is learned under a minimal setup without pretext tasks and other mechanisms of complete SSL methods. However, in this work, rather than using a fixed teacher, we use a peer-to-peer group of randomly initialized networks. So, we can study the effect of learning from a moving randomly initialized network, without introducing the asymmetric effect of an EMA teacher. III. M ETHODOLOGY Our setup is inspired by DINO [4], however, we stripped away most of the components to focus exclusively on the effects on learning of self distillation. In DINOHerd, we start with a group of untrained, randomly initialized neural networks. The specific architecture of the network is not relevant, as long as it is reasonable for the task at hand and it returns embeddings we can compare with a loss function. For each batch, we randomly choose one student and T teachers. For each teacher and student, we generate the corresponding output for the given batch. We stress each network sees exactly the same view of the data, the teachers do not receive a different augmentation than the student. Finally, we use a loss function to determine the difference between the student and the teachers, making sure to only back propagate the error through the student, not the teachers. PyTorch pseudocode is shown in Listing 1 and a schematic of our training setup is shown at Figure 1. A. Teacher dynamics In our framework, the teacher role is assigned randomly to N networks. This assignment lasts for only a single batch. This creates an environment where no peer has a fixed special role during the whole training: each peer will be a student or a teacher at any moment during the training. This dynamic simplifies our setup, as we do not need to implement any different logic for updating the weights of the teacher. As is typical with other similar methods, we use a stop-gradient operator on the teacher branch during backpropagation; only the current student weights are updated per batch. We explored the effect of choosing more than one teacher per batch. The results of those experiments are shown in Subsection V-A.
Neural Network Peers
ra og
nt
die
N Net 8765 Net Net Net Net Net Net Net4321
Teacher
\\
Teacher output
training loss xi
Student
Input
Randomly selected per batch
Student output
Fig. 1. Schematic of DINOHerd setup with a single teacher and student chosen at random per batch.
Listing 1. PyTorch pseudocode for DINOHerd # N: Number of peers 2 # T: Number of teachers 3 peers = [Network() for _ in range(N)] 4 optimizers = [make_optimizer(p) for p in peers] 1
5
for x in dataloader: # Sample student and teachers from pool 8 idxs = random.sample(range(len(peers)), T + 1) 9 student_idx = idxs[0] 6 7
10 11 12
student = peers[student_idx] teachers = [peers[i] for i in idxs[1:]]
13 14 15
# Forward pass for student student_out = student(x)
16 17 18 19 20
# Forward pass for teachers (frozen) with torch.no_grad(): teacher_outs = [teacher(x) for teacher in teachers]
21 22 23
24
# Average loss between student/teachers losses = [loss_fn(student_out, teacher_out) for teacher_out in teacher_outs] loss = torch.mean(torch.stack(losses))
25 26 27 28 29
# Backprop through student only optimizers[student_idx].zero_grad() loss.backward() optimizers[student_idx].step()
B. Loss function and learning rate During development, we experimented with traditional loss functions such as cosine loss and mean squared error (MSE). However, we observed that lower learning rates consistently improved the stability and quality of the learned representations. A lower learning rate is equivalent to scaling the gradient, which we show as follows: Learning rate and gradient scaling equivalence: Consider a learning rate defined as η = α·C. Let θt be the weights of a neural network at training step t. The SGD update rule is θt+1 = θt − η · ∇L(θt ) . Substituting the learning rate η, we obtain: θt+1 = θt − α · C · ∇L(θt ) . Then, by the linearity of the gradient operator with respect to scalar multiplication, we get: θt+1 = θt − C · ∇ (α · L(θt )) .
C. Training
Conv2D 64 to 128
BatchNorm
LeakyReLU
MaxPool2D (2 × 2)
Conv2D 256 to 256
BatchNorm
LeakyReLU
MaxPool2D (2 × 2)
TABLE I CIFAR-10 CLASSIFICATION RESULTS (16 PEERS , 10 EPOCHS ). Network
KNN (K=5) F1 (%)
Linear F1 (%)
MLP F1 (%)
Baseline Trained
37.22 41.58
16.97 43.43
15.71 49.26
55 50 45 40 35 30 25 20 15
0
200 400 600 800 1000 1200 1400 Batch
Fig. 3. Comparison of linear probe accuracy of 8 peers on CIFAR-10.
Feature Map
LeakyReLU
BatchNorm BatchNorm
LeakyReLU
Conv2D 3 to 64 Conv2D 128 to 256
Input Image
For both training and evaluating our models, we used CIFAR-10 dataset [6]. It consists of 60000 32x32 color images of ten different classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. The only augmentation applied to the dataset was a 50% probability horizontal flip. Unlike other self-supervised and self-distilled setups, in our case the teacher and the student always receive exactly the same view of the data. We used a simple convolutional neural network, as shown in Figure 2. For evaluation, we trained a KNN (K=5), a linear probe and a MLP on top of the frozen backbone.
512, being trained on a single NVIDIA GeForce RTX 3060 with 12GB of VRAM. The resulting model had a file size of 3.9MB.
Accuracy (%)
Since the learning rate effectively acts as a multiplicative scaling factor on the loss, we began to reinterpret it as a form of implicit temperature control on the loss landscape (conceptually akin to the temperature scaling used in DINO [4]). A lower loss scale reduces the influence of smaller gradients, allowing only the most salient error signals to drive updates. This observation motivated the design of our salient loss, which focuses training on the single most divergent feature dimension per sample. Rather taking all the dimensions into account, our salient loss only considers the dimension with the maximum distance (max((A − B)2 )). Interestingly, introducing this custom loss did not changed much the learning dynamics, as explained in subsection V-C. Although the salient loss provides a focused gradient signal, we found that increasing the learning rate led to worse representations. The best results were obtained using small learning rates with the Adam or AdamW optimizers in combination with the salient loss. This suggests that while saliency helps guide learning, it must be coupled with gradual updates to maintain alignment across the peer networks.
Fig. 2. Diagram of the neural network used for learning vision representations
IV. M AIN R ESULTS A. Vision Results We trained a single vision model on CIFAR-10 dataset [6] for 10 epochs using a learning rate of 1e-8 and 16 peers. The training took a total of 2:14 minutes using a batch size of
After training, we evaluated the learned representations using a KNN classifier (K = 5), a linear probe and a small MLP. Both the linear probe and the MLP were trained for 20 epochs using SGD and a learning rate of 0.01. Table I summarizes the results compared to an untrained baseline. As the table shows, the training process improved the classification results, specially when using linear or MLP probes. To further understand the learning dynamics within the peers, we trained a smaller setup for a single epoch, with just eight randomly initialized models, using a learning rate of 1e8 and a batch size of 32. Then, every 50 batches we trained a linear probe on CIFAR-10 to evaluate the quality of the learned features. As shown in Figure 3, the untrained networks start with low quality representations (mostly below 20% linear probe accuracy) and in around 500 batches most networks stabilized. Also, the plot shows that the learning dynamics improves all the networks consistently, even if CIFAR-10 classification is never used as a training objective.
V. H YPERPARAMETER E XPLORATION In this section, we will describe the results of performing several variations on our training framework. All the ablation studies were done using our vision model trained on CIFAR10.
55 50 45 40 35 30 25 20 15
50
Accuracy (%)
45 40 35 30 25 20 15
1 teacher 2 teachers 0
1000
Batch
2000
3000
Fig. 5. Comparison of accuracy varying the number of teachers.
60 0
500 1000 1500 2000 2500 3000 Batch 2 peers 32 peers 4 peers 64 peers 8 peers 128 peers 16 peers
Fig. 4. Comparison of linear probe accuracy varying the number of peers.
B. Individual vs Ensemble Performance When using a linear probe over the concatenated outputs of multiple peers, we observed a slight improvement of the downstream task performance of the models. As shown in the Figure 6, the single model has lower accuracy than the ensembles of two or more models. This demonstrate that each peer is learning different features, that could be useful for classification. However, there are also diminishing gains on adding additional models outputs to the concatenated embeddings. In this case, the accuracy was evaluated evaluated on CIFAR-10 every 50 batches, with batch size 32 and learning rate 1e-8. In all cases, a 16 peers setup was used.
50 Accuracy (%)
Accuracy (%)
A. Effect of Number of Peers and Teachers Count When evaluating the impact of the number of peers and teachers on the learned representations, we found that, aside from making the learning slower, increasing the number of peers produce eventually similar results, as shown in Figure 4. Similarly, we did not find important differences between using one or two teachers, as shown in Figure 5. For both cases, the accuracy was evaluated every 200 batches for two epochs, using salient loss, learning rate of 1e − 8 and a batch size of 32. For Figure 5 16 peers were used.
55
40 30 20 0
500
1000 1500 Batch Single Model 2 Models Ensemble 4 Models Ensemble 8 Models Ensemble 16 Models Ensemble
Fig. 6. Comparison of the ensemble of a varying number of peer outputs.
We compared the learning dynamics of different loss functions (MSE, salient loss) and network architectures (Simple CNN, ResNet18 [12], VGG11 [13], DenseNet121 [14]) on CIFAR-10 using our setup. In this case, we used two peers, one teacher, a learning rate of 1e-8, and a batch size of 32, and evaluated CIFAR-10 performance every 50 batches. As shown in Figure 7, it seems that in our setup, smaller networks get bigger gains than bigger ones. Our simple CNN got the highest accuracy around 50%, followed by DenseNet121 (around 38%), then Resnet18 (around 31%) and finally VGG11 (around 20%). There are no differences on the loss function, which is surprising given that salient loss is dropping all but one of the output dimensions. It suggests that the alignment is a very focused process, that does not require abrupt changes on the latent space.
45.0 42.5 40.0 37.5 35.0 32.5 30.0 27.5
50 Accuracy (%)
that even a randomly initialized network preserves some kind of distances related with actual data features.
Accuracy (%)
C. Impact of the Loss Function and Network Architecture
25.0
40 30 20
0
500
Batch LR=1e-02 LR=1e-04
1000
1500
LR=1e-06
Fig. 8. Comparison of KNN (K=5) accuracy varying the learning rate.
0
500
Batch
1000
1500
simple_cnn (salient) resnet18 (salient) vgg11 (salient) densenet121 (salient) simple_cnn (mse) resnet18 (mse) vgg11 (mse) densenet121 (mse) Fig. 7. Comparison of linear probe accuracy varying network architecture and loss function.
D. Learning Rate Sensitivity As shown in Figure 8 and Figure 9, we found that a lower learning rate produces more useful learned representations, both with KNN or a linear probe. Other interesting aspect to highlight is that the initial representations of the random network are more amenable for KNN classification, meaning
VI. D ISCUSSION In our setup, we found that it is possible to get non-trivial accuracy gains from just self-distilling knowledge between peers in a group of randomly initialized networks. To further investigate what is being learned, we compared the distances between samples as measured by the initial untrained neural network and by the trained network. For this, we took CIFAR-10 images and compared its embeddings with a shuffled list of the same embeddings, both for the untrained network and the trained one (keeping the same shuffling). As shown in Figure 10, we observe a global shift toward increased separation, particularly among initially close samples. This emergent expansion behavior may partially explain why some non-contrastive self-supervised methods, such as BYOL and DINO, can succeed even in the absence of explicit negative samples. However, a full understanding of this mechanism and its relationship to semantic learning will require further study. VII. C ONCLUSIONS AND F UTURE W ORK In this work, we studied the effect of self-distillation on learning. Our findings suggest that self-distillation alone can yield improved representations. However, these effects are variable, depending on the network architecture or the learning rate.
Interestingly, we found that the distances in the learned representations are usually bigger than in the initial network. The nature of this expansion needs further analysis, which we leave for future work.
55 50
R EFERENCES
Accuracy (%)
45 40 35 30 25 20 15
0
500
Batch LR=1e-02 LR=1e-04
1000
1500
LR=1e-06
Fig. 9. Comparison of linear probe accuracy varying the learning rate.
Cosine Distances - CIFAR 10 0.35
y=x
0.25
Number of Pairs
Random Network
0.30
1600 1400 1200 1000 800 600 400 200
0.20 0.15 0.10 0.05 0.0
0.1
0.2 0.3 0.4 Trained Network
Fig. 10. Comparison of the cosine distances between the CIFAR-10 embeddings generated by the untrained network and the ones produced by the trained network.
[1] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, 2013. [2] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. The MIT Press, 2016. [3] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent - a new approach to self-supervised learning,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 21 271–21 284. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf [4] M. Caron, H. Touvron, I. Misra, H. Jegou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 9630–9640. [5] X. Wang, X. Chen, S. S. Du, and Y. Tian, “Towards demystifying representation learning with non-contrastive self-supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2110.04947 [6] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Toronto, Ontario, Tech. Rep. 0, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf [7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in Proceedings of the 37th International Conference on Machine Learning, ser. ICML’20. JMLR.org, 2020. [8] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” in International conference on machine learning. PMLR, 2021, pp. 12 310–12 320. [9] X. Chen and K. He, “Exploring simple siamese representation learning,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 15 745–15 753. [10] Z. Wu, J. Zhang, D. Pai, X. Wang, C. Singh, J. Yang, J. Gao, and Y. Ma, “Simplifying DINO via coding rate regularization,” 2025. [Online]. Available: https://arxiv.org/abs/2502.10385 [11] F. Sarnthein, G. Bachmann, S. Anagnostidis, and T. Hofmann, “Random teachers are good teachers,” in International Conference on Machine Learning, 2023. [Online]. Available: https://arxiv.org/abs/2302.12091 [12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. [13] S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” in 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015, pp. 730–734. [14] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2261–2269.