Official Review · ICLR 2025

This paper introduces the "Pan for Gold" hypothesis, which challenges the traditional view that structured labels and well-defined loss functions are essential for deep learning models to learn meaningful representations and generalize well. The authors propose that generalization emerges naturally through the stochasticity inherent in SGD when training overparameterized models, even with unstructured (randomized) labels. They suggest that SGD acts like panning for gold, where valuable features are naturally sifted out from noise without relying on human-imposed structures. To support this hypothesis, the authors conduct experiments where models are trained on datasets with completely randomized labels. Surprisingly, these models still learn meaningful features, as evidenced by improved performance over random initialization. They analyze this phenomenon using the NTK framework and observe a "swing phenomenon," where model outputs fluctuate significantly during early training stages. Based on these observations, they introduce the PUL algorithm. They demonstrate PUL's effectiveness in tasks like unsupervised domain adaptation and object discovery. Additionally, they suggest that PUL mitigates issues like massive activation in vision transformers, aiding in model quantization. * **Challenging Conventional Wisdom:** The paper attempts to question the traditional beliefs about the necessity of structured labels and loss functions in deep learning, which is an interesting and bold endeavor. * **Novel Hypothesis Introduction:** The "Pan for Gold" hypothesis is a creative metaphor that could inspire new ways of thinking about generalization in deep learning. * **Exploration of Unstructured Labels:** Investigating the effects of training with unstructured labels is an intresting approach that could uncover overlooked aspects of model training dynamics. * **Lack of Theoretical Rigor:** The paper makes strong claims without providing a solid theoretical foundation. The mathematical analysis is superficial and does not rigorously justify the "Pan for Gold" hypothesis or explain why unstructured labels should lead to better generalization. * **Insufficient Empirical Evidence:** The experimental evaluation is limited and inadequate to support the bold claims made. Experiments are conducted on small datasets like MNIST, CIFAR-10 and SVHN, which are not representative of modern large-scale tasks. The performance improvements reported are marginal and could be due to experimental noise. * **No Comparison with Baselines:** The paper fails to compare the proposed PUL algorithm with established baselines or state-of-the-art methods in the respective tasks. Without such comparisons, it's impossible to assess the significance of the results or attribute improvements to the proposed method. * **Overgeneralization of Findings:** The authors make sweeping generalizations about deep learning based on limited and specific experiments. The claim that generalization em