There Will Be a Scientific Theory of Deep Learning
arXiv:2604.21691v1 [stat.ML] 23 Apr 2026
Jamie Simon∗ UC Berkeley and Imbue
Daniel Kunin UC Berkeley
Alexander Atanasov Harvard University
Enric Boix-Adserà University of Pennsylvania
Blake Bordelon Harvard University
Jeremy Cohen Flatiron Institute
Nikhil Ghosh Flatiron Institute
Florentin Guth NYU & Flatiron Institute
Arthur Jacot New York University
Mason Kamb Stanford University
Dhruva Karkada UC Berkeley
Eric J. Michaud Astera Institute
Berkan Ottlik University of Pennsylvania
Joseph Turnbull UC Berkeley
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: 1. solvable idealized settings that provide intuition for learning dynamics in realistic systems; 2. tractable limits that reveal insights into fundamental learning phenomena; 3. simple mathematical laws that capture important macroscopic observables; 4. theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and 5. universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We assert that learning mechanics should be a mathematical theory, grounded in first-principles calculations that closely predict empirics, reliant on well-tested approximations and assumptions, aiming for broad impact across the machine learning stack once it reaches maturity. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic and mutually supportive relationship between learning mechanics and the developing discipline of mechanistic interpretability. Where mechanistic interpretability aims to be the biology of deep learning, learning mechanics should aspire to be its physics, mirroring the complementary relationship between biology and physics in the natural sciences. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.
∗ Correspondence to [email protected].
1
1
Introduction
Deep learning is famously a black-box learning method, the most powerful, most inscrutable, and now most technologically important member of the machine learning pantheon. Properly trained, neural networks learn to perform a wide array of tasks with superhuman performance, but we have no unified scientific framework that explains why or how. Motivated by both scientific curiosity and the promise of practical engineering benefit, the effort to put rigorous mathematical and scientific backing behind this applied discipline has spanned decades. Despite some progress, however, our understanding remains primitive: neural networks are still trained using methods discovered largely through trial and error rather than first principles, and theory plays little role in the day-to-day practice of deep learning. The challenge has only compounded as practice has advanced, and in the era of large language models and diffusion models, the mysteries are arguably deeper than they were one or two decades ago. Will we ever understand? This paper makes the case that, yes, there will be a scientific theory of deep learning; that we can see pieces of this theory starting to emerge; and that this theory will take the form of a mechanics of the learning process. The questions driving deep learning theory have changed over time, and to understand where the field is going, it is useful to first look back at how we got here. Deep learning theory is as old as machine learning itself, with roots in the McCulloch–Pitts neuron and the perceptron in the middle of the last century. The earliest theoretical questions in machine learning were about expressivity: what functions can simple models represent, and how can they be learned from data? As learning came to be understood as a statistical problem, and simple learning systems found practical success, the theoretical focus shifted to ask: when does learning from finite samples generalize? This gave rise to classical learning theory, including statistical and computational/PAC learning theory. Paired with classical optimization theory, these frameworks gave clean end-to-end guarantees of the optimization and generalization of simple learning systems. In parallel, a classical tradition of the statistical physics of machine learning developed satisfying theories of the average-case behavior of simple models. While these classical theories built a strong foundation for understanding learning, the rise of deep learning through multilayer networks, backpropagation, and increasing scale in both data and compute exposed limitations in their explanatory power. Neural networks are complex, nonconvex, and overparameterized (in contrast with the simple, convex, parsimonious models for which classical learning theory excels), and they optimize and generalize better than these classical approaches can guarantee or explain. Furthermore, it became clear that neural networks were not merely fitting data or achieving low training error, they were learning structured internal representations and displaying striking regularities across tasks and scales. The classical questions of performance and efficiency remained important, but answering them would first require understanding a new host of phenomena shaped both by the dynamics of neural networks through training and the structure of the data they are trained on. This marked a transition in which deep learning theory changed in character from a largely mathematical study of what is possible to a truly scientific effort to describe, explain, and ultimately predict the behavior of complex empirical systems. New scientific endeavors often start with an empirical tension in which nature presents something interesting we cannot predict or explain with existing tools, and although neural networks are artificial computational systems, this same scientific tension is present here. We should thus approach this task as scientists, embracing empirics, seeking unifying principles, and identifying recurring motifs. We should also expect the path forward to look more like the development of a scientific field than the development of a mathematical one. The purpose of this paper is to convince the reader that this scientific tension is gradually giving way to a scientific theory which resolves it. In Section 2, we pull together major strands of ongoing research and identify five lines of evidence that such a theory is emerging: 1. there are a growing number of analytically solvable settings in which learning is fully captured by simple mathematics, including but not limited to deep linear networks and kernel methods (Section 2.1); 2. there exist useful limits, including limits of infinite width and depth, that provide insight into fundamental learning behaviors (Section 2.2); 3. in many cases, simple empirical laws suffice to capture meaningful macroscopic statistics, including test-time performance and loss-landscape sharpness (Section 2.3); 2
4. many of the hyperparameters governing optimization can be disentangled and understood, leaving behind a simpler effective dynamical system (Section 2.4); and 5. as applied deep learning has scaled up and converged on best practices, universal phenomena have increasingly appeared across settings and tasks (Section 2.5). These lines of research broadly share several overarching characteristics: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics of learning; and they emphasize accurate average-case predictions over rigorous worst-case bounds. In this sense, the emerging scientific theory of neural networks appears to have much in common with theories in physics such as classical mechanics, continuum mechanics, statistical mechanics, and quantum mechanics. We argue that this emerging theory is best understood as a mechanics of learning.
1.1
What’s in a mechanics?
Mechanics is the branch of physics studying how forces acting on objects determine their movement through space and time. Neural network learning can be thought of in this way: much as an object moves continuously through physical space, learning involves a model moving through parameter space via discrete updates. In the physical sciences, forces come from interactions between components of a system. Similarly, the process of deep learning is shaped by interactions between the parameters, dataset, task, and learning rule. In physics, these forces are mediated by fields; in deep learning, they are mediated by gradients. In physics, systems settle into equilibria at local minima of a potential determined by internal interactions and external constraints; analogously, neural networks converge to local minima of a loss landscape shaped by their architecture and training data. While the systems under study are very different, since the key problems of both are essentially about movement and interaction, we might expect some features of the resulting sciences to be shared. These analogies are not just speculation: we can see these similarities reflected in the lines of research listed above. All branches of mechanics (and especially classical mechanics) develop a library of analytically solvable settings to gain intuition; so too does learning mechanics. All branches of mechanics use limits as simplifying tools; so too does learning mechanics. Continuum and statistical mechanics, the branches which most directly deal with large numbers of interacting components, describe zoomed-out summary statistics rather than the motion of every particle; this has also proven a useful approach in dealing with the complexity of deep learning. Every physical system has one or more system parameters (characteristic scales, coupling constants, etc.) affecting its behavior, and some techniques for treating these are essentially the same as those used to study hyperparameters in deep learning. Finally, physics is full of cases in which the same phenomena show up in very different settings, and similarly we see universal behavior emerging across deep learning systems. All considered, the emerging science shares deep similarities with established branches of mechanics. By analogy to classical, continuum, statistical, and quantum mechanics, we suggest the intended theory be called learning mechanics. Seven desiderata for learning mechanics. We should be clear at the outset what we want from a mechanics of learning. Assessing how mature branches of mechanics were motivated, developed, and succeeded, we can see what sort of goals to aim for. Here are seven desiderata for this research program: 1. Learning mechanics should be fundamental, proceeding logically from a first-principles description of neural network training. Interim assumptions about network weights, dynamics, and performance will be useful tools, but they should ultimately be explained from first principles. 2. Learning mechanics should be mathematical, making unambiguous quantitative statements about important properties of neural networks. No mechanics is a qualitative science; neither will be learning mechanics. 3. Learning mechanics should be predictive, making claims supported by simple, repeatable empirical measurements. We have excellent experimental control of our system, and every major development should be unambiguously verified in experiment. 4. Learning mechanics should be comprehensive, describing aspects of neural networks’ training process, hidden representations, and final weights in a single picture. It is worth emphasizing that this theory will not — and should not — aim to describe everything. A map at the full resolution of the world would be the size of the
3
world and thus of little use. What we seek instead is a theory that operates at the right level of resolution — one that sacrifices detail in favor of insight. 5. Learning mechanics should be intuitive, being simple, illuminating, and satisfying in its demystification of deep learning. Like physics, learning mechanics should strive for simple insight over technical complexity. 6. Learning mechanics should be useful, serving as the scientific foundation for applied deep learning as physics does for other forms of engineering. Concrete goals should include greatly reducing the need for hyperparameter tuning, giving predictive tools for dataset design, and providing rigorous foundations for AI safety work. 7. Finally, learning mechanics should be humble, being solid in what it describes and explicit about what it cannot. Every branch of physical science has a regime of applicability outside of which it breaks down, and these boundaries are taught together with the science so that it may be used reliably. We anticipate the mechanics of learning applicable to realistic deep learning will break down in many small-scale, handcrafted, or otherwise special cases, and this is the price we will pay for the right simple picture in the regimes we care about. A mechanics of learning with these virtues — one that is fundamental, mathematical, predictive, comprehensive, intuitive, useful, and humble — would be transformative, paradigm-setting. We expect such a theory would resolve important open questions that have long remained out of reach, as we discuss in Section 5.
1.2
Why learning mechanics matters
Building learning mechanics will not be easy. It will require sustained effort, both intellectual and institutional. It is therefore worth being clear about why such a project matters. The reasons to seek a mechanics of learning fall into three broad categories: scientific, practical, and safety-related. The scientific reasons concern what such a theory could teach us about intelligence and the natural world. The striking engineering success of large neural networks suggests that they exploit deep principles of learning and representation that we do not yet understand. This has historical precedent: technology has often preceded scientific theory, as was the case with steam engines’ role in motivating thermodynamics, which went on to explain much more than engine efficiency. A similar story played out in flight: the development of airplanes through trial and error and inspiration from the natural world helped motivate aerodynamic theory, which in turn enabled both better aircraft design and a deeper understanding of how birds themselves fly. In our case, the principles that govern learning in artificial neural networks may also shed light on our own biological intelligence, with potentially important implications for neuroscience and cognitive science. The practical reasons concern the design and development of real-world AI systems. A mature theory of deep learning could guide model design, optimization, scaling, and deployment, replacing trial and error with more reliable principles. Theory has already begun to play this role in a limited but growing number of cases, including empirical scaling laws (Section 2.3), mathematical prescriptions for hyperparameter scaling (Section 2.4), and theoretically-motivated optimizers and methods for data attribution (Section 4). A deeper, more complete theory will give more such guidance and make it sharper and more predictive. The safety reasons concern our ability to describe, characterize, and govern increasingly powerful AI systems. Some form of regulation will likely be necessary, but it is difficult to regulate a technology that we cannot clearly describe. A theory that identifies the relevant variables, mechanisms, and organizing principles of large models could help provide the clarity needed for reliability, oversight, and control. One avenue by which fundamental theory might aid in AI safety is by supporting mechanistic interpretability, a point to which we return in Section 3.
1.3
Plan for this paper
This paper is structured as follows. In Section 2, we present our five lines of evidence that a scientific theory of deep learning is beginning to emerge. We motivate each line of evidence with an intuitive explanation and highlight examples of research successes that illustrate the underlying principle. In Section 3, we discuss the relationship between learning mechanics and other perspectives on the science of deep learning, including a possible symbiotic relationship between learning mechanics and mechanistic interpretability. In Section 4, we review and address common arguments that fundamental theory will not be possible. In Section 5, we give a portrait of ten important open directions in learning mechanics, from predicting scaling laws to eliminating hyperparameters, where we expect 4
to see major progress in the coming years. Finally, in Section 6, we offer some advice for young researchers looking to get involved in this scientific project and extend a hand with some introductory resources. We write this paper for a broad audience. We hope the veteran scientist of deep learning will find something valuable in our synthesis of useful approaches and results, and feel galvanized by our depiction of an emerging science. We hope to convince the deep learning practitioner that theory is on a path to fulfilling its longstanding promise of practical utility and to encourage them to experiment with their systems with an eye for science. We hope to convince the AI safety or mechanistic interpretability researcher that white-box theory is difficult yet possible — that a first-principles study of dynamics can help put solid foundations beneath their important work, and that our communities should work together (see Section 3 for our vision of symbiosis). Lastly, we hope to make it easier for young students and newcomers to the field to get involved. This is an exciting and important area of work, and while it requires some mathematical maturity to get started, it is our belief that the barrier to entry could be much lower. Various deep intuitions about this science have been percolating inside the theory community for a while, and this paper is an attempt to state them clearly. We hope to make it easier for folks with the requisite background to quickly get up to speed and contribute.
2
Evidence of an emerging mechanics of learning
A great cause for optimism that a mechanics of learning is possible is the fact that the essential ingredients of deep learning are both explicit and measurable. A deep learning system is characterized by the following components: Architecture:
a neural network f (x; θ) specified as a composition of simple linear and nonlinear transformations.
Data:
a dataset D = {(xi , yi )}ni=1 consisting of samples from an unknown data-generating distribution (x, y) ∼ Pdata .
Task:
an objective L(θ) measuring the performance of the network f (x; θ) on the dataset D.
Learning rule:
a gradient-based update equation, e.g. θ (t+1) = θ (t) − η∇L(θ (t) ), together with a parameter (0)
initialization, e.g. θi
2 ∼ N (0, αinit ) and optimization hyperparameters, e.g. learning rate η.
Nothing about the learning process is hidden. Unlike many complex systems where the equations governing dynamics must be inferred from observations, deep learning directly exposes its “equations of motion.” Moreover, these dynamics are extraordinarily measurable: every weight, activation, gradient, and loss value can be recorded, along with arbitrary statistics derived from them. As a result, deep learning experiments are unusually easy to design, replicate, and interrogate, making it more straightforward to discover empirical regularities and rigorously test theoretical predictions. Few fast-moving scientific domains offer comparable transparency in their governing equations or comparable freedom in what can be measured. What, then, stands in the way of a scientific theory of deep learning? The central challenge is not opacity, but complexity. While we have direct access to the architecture, data, task, and learning rule, the interaction of these components gives rise to learning dynamics that are nonlinear, coupled, and high-dimensional. These dynamics depend in subtle ways on the choice of hyperparameters. And even though we can inspect every training sample, data distributions are complex and have defied simple characterization. Nevertheless, we argue that this complexity conceals underlying regularities, and that deep learning will indeed admit a scientific theory. In what follows, we present five broad observations that serve as evidence for an emerging mechanics of learning. Each of these admits direct analogies to tools and ideas in other disciplines of mechanics. These are summarized in Table 1.
5
Section
Approach
Examples in deep learning
Examples from physics
2.1
solvable settings
deep linear networks, kernel regression, multi-index models
harmonic oscillator, hydrogen atom, Ising model
2.2
simplifying limits
lazy vs. rich learning, width, depth → ∞, small initialization
thermodynamic limit (n, V → ∞), classical limit (ℏ → 0), hydrodynamic limit (k, ω → 0)
2.3
simple empirical laws
neural scaling laws edge of stability neural feature ansatz
the laws of Kepler, Snell, Boyle, Hooke, Newton, Faraday, Ohm, Poiseuille, Planck, Hubble, etc.
2.4
study of system parameters
step size as sharpness regularization, µP and width-scaling
scaling analysis, nondimensionalization, chaotic vs. ordered regimes
2.5
universal phenomena
common inductive biases and representations across models
critical phenomena, renormalization group flow
Table 1: Useful tools and ideas in the emerging science of deep learning closely resemble important tools and ideas from physics, particularly classical mechanics, continuum mechanics, statistical mechanics, and quantum mechanics. Extrapolating, this suggests that there will be a mechanics of learning which offers a unifying first-principles theory of the training process, hidden representations, final weights, and test-time performance of neural networks.
2.1
Analytically solvable settings exist
A reliable way to build scientific understanding in complex systems is to study pared-down yet representative settings in which quantitative calculations are possible. For example, physics uses representative solvable settings like the harmonic oscillator and the hydrogen atom as sources of intuition for much broader classes of system. Deep learning appears to be particularly amenable to this approach: scientists have identified a rich landscape of minimal models where the learning dynamics simplify and many quantities of interest become solvable. These analytically tractable cornerstones are useful because they reveal phenomena and mechanisms to look for when we turn to realistic deep learning.1 One particularly fruitful simplification is linearization. Here we discuss two distinct instantiations of this idea: linearization in the data, where f (x; θ) becomes linear in x, and linearization in the parameters, where f (x; θ) becomes linear in θ. Linearization in the data. A deep linear network is obtained by removing all nonlinearities from a neural network’s architecture, yielding a model that is linear in its inputs x but remains highly nonlinear in its parameters θ: f (x; θ) = WL WL−1 · · · W1 x,
where θ := {Wℓ }L ℓ=1 , each Wℓ is a linear transformation, and L ≥ 2. (1)
Deep linear networks have a long history of study because, despite their simplicity, they retain many hallmark behaviors of deep learning [Nam et al., 2025]. These include saddle-point-dominated loss landscapes [Baldi and Hornik, 1989], dynamics with sharp phase transitions and separation of timescales [Gissin et al., 2019, Atanasov et al., 2021], edge-of-stability oscillations with gradient descent [Even et al., 2023], and strong initialization-dependent inductive biases [Woodworth et al., 2020, Kunin et al., 2024]. Analysis of these networks is typically carried out with the gradient flow learning rule — the continuous-time limit of gradient descent — under simplifying assumptions on 1 A complementary view is that any eventual complete theory of deep learning must encompass these simplified settings. Their solutions may provide conceptual scaffolding, serving as nucleation sites from which a more general theory crystallizes.
6
Gradient flow dynamics of singular values of W2 W1
experiment theory
100
60 40 20 0
2000
4000
t
6000
8000
experiment theory
0.8 0.6 0.4 0.2
0
ReLU NTK learning binarizations of CIFAR-5m
1.0
test MSE
80
si (t)
1.2
0.0 101
10000
(a) Linearization in the data
plane vs. frog car vs. ship dog vs. all others deer vs. horse 102
n training samples
103
(b) Linearization in the parameters
Figure 1: Linearization yields exact solutions that match experiments. (a) Canonical work by Saxe et al. [2014] showed that, under a task-aligned initialization θ (0) and whitened inputs x ∼ N (0, I), the gradient flow learning dynamics of deep linear networks decouple into independent solvable Bernoulli ODEs. This leads to sequential learning of singular modes, with larger-singular-value modes emerging first. Panel (a) reproduces Fig. 3 of Saxe et al. [2014]. (b) Linearizing a nonlinear network by truncating nonlinear terms in its Taylor expansion around initialization reduces least-squares training to kernel ridge regression with the neural tangent kernel (NTK). This analysis connects the network’s architecture to its inductive bias through the NTK eigenstructure, enabling accurate predictions for the test performance of these networks. Panel (b) is based on Fig. 2 of Simon et al. [2023a]. the data distribution and with carefully chosen initializations [Fukumizu, 1998, Saxe et al., 2014, Tarmoun et al., 2021, Dominé et al., 2025]. In these regimes, the learning dynamics can often be solved exactly or reduced to low-dimensional dynamical systems. Across many such analyses, a consistent lesson emerges: learning exhibits a greedy low-rank bias, acquiring some components of the task before others. Canonical work by Saxe et al. [2014] first showed how deep linear networks learn singular vectors of the input–output correlation sequentially during training, with learning prioritized toward modes associated with the largest singular values, as shown in Figure 1. This bias has been hypothesized to benefit generalization by separating the signal from the noise [Lampinen and Ganguli, 2018], and closely mirrors behavior observed in nonlinear networks, where simpler functions are often learned before more complex ones [Kalimeris et al., 2019, Simon et al., 2023b]. Moreover, a range of factors — including small initializations [Gidel et al., 2019, Li et al., 2021a, Jacot et al., 2021, Pesme and Flammarion, 2023], increased depth [Gunasekar et al., 2018, Arora et al., 2018, 2019b], stronger mini-batch noise [Pesme et al., 2021, Chen et al., 2024], and explicit ℓ2 regularization [Ziyin et al., 2022, Wang and Jacot, 2024] — have all been shown to further strengthen this greedy learning bias. Linearization in the parameters. A linearized network is obtained by truncating the nonlinear terms in a network’s Taylor expansion around its initial parameters. This yields a model that is linear in its parameters θ but remains highly nonlinear in the data x: flin (x; θ) = f (x; θ0 ) + ∇θ f (x; θ0 )⊤ (θ − θ0 ),
where ∇θ f (·; θ0 ) is the gradient at initialization.
(2)
This is not some contrived construction: in fact, there are settings in which a model is well-approximated throughout training by its linearization, i.e., ∀t, f (x; θt ) ≈ flin (x; θt ). For example, any neural network architecture can be driven into the linearized regime by taking suitable limits [Jacot et al., 2018, Lee et al., 2019, Chizat et al., 2019, Liu et al., 2020], as discussed in Section 2.2. Additionally, recent evidence suggests that language model fine-tuning occurs in a near-linearized regime [Malladi et al., 2023, Ren and Sutherland, 2025]. Since a linearized network is linear in its parameters, its learning dynamics are identical to those of linear regression, with one key difference: while the dynamics of linear regression are driven by the Gram kernel, KGram (x, x′ ) = x⊤ x′ , linearized networks are described by the neural tangent kernel (NTK), KNTK (x, x′ ) := ∇θ f (x; θ0 )⊤ ∇θ f (x′ ; θ0 ). When the task is least squares regression and training uses small-step gradient descent, the dynamics are analytically tractable and the final predictor is given by kernel ridge regression with the NTK [Jacot et al., 2018]. 7
This setting yields insight into a variety of deep learning phenomena. For example, since the details of the network architecture influence the mathematical structure of the NTK through the fixed feature map ∇θ f (·; θ0 ), one learns how the linearized model’s inductive bias follows from its architecture [Arora et al., 2019c, Geifman et al., 2020]. Furthermore, one may accurately predict the model’s expected generalization error on arbitrary targets f ⋆ by accounting for the structure of the input data [Jacot et al., 2020, Canatar et al., 2021, Loureiro et al., 2021, Hastie et al., 2022, Wei et al., 2022, Simon et al., 2023a], as shown in Figure 1. Applying this framework to realistic data distributions uncovers the origin of the typical models’ tendency to learn simple and generalizing functions [Basri et al., 2020, Karkada et al., 2025]. Linearized models also capture relevant phenomena such as double descent [Belkin et al., 2019, Advani et al., 2020] and scaling laws [Caponnetto and de Vito, 2007, Pillaud-Vivien et al., 2018, Cui et al., 2023, Atanasov et al., 2024]. However, despite these theoretical merits, linearized networks are unrealistic in a few critical ways. Most notably, they do not capture the strong feature-learning capabilities that generic neural networks exhibit, often leading to overly pessimistic predictions for sample complexity [Ghorbani et al., 2020, Vyas et al., 2022]. Moreover, by reducing training to a tractable linear problem, these models sidestep the intrinsically nonconvex optimization phenomena of deep learning. To describe these and other aspects of deep learning, one must look beyond linearization. Beyond linearization. An important frontier for theory lies in developing analytically tractable toy models that remain genuinely nonlinear in both the data and the parameters (see Open Direction 1). In these settings, the influence of the data distribution becomes more complex, making it difficult to obtain a unified and general framework. However, a growing body of work is progressing in this direction by isolating specific nonlinear mechanisms and making them solvable under assumptions on the data. One line of work studies Gaussian inputs and structured targets (e.g., single- and multi-index models). Fully nonlinear neural networks provably outperform kernel methods using fewer samples because they exploit the structure in the target function to learn relevant features [Abbe et al., 2022, Damian et al., 2022b, Bietti et al., 2022, Ba et al., 2022, Dandi et al., 2023]. Complementarily, methods from statistical physics enable computing exact asymptotics for Bayes-optimal inference and learning dynamics in these models [Barbier et al., 2019, Aubin et al., 2018, Mignacco et al., 2020]. A related setting is two-layer neural networks with quadratic activation functions, where recent results have characterized the exact asymptotics, training dynamics, and scaling laws [Erba et al., 2025, Ben Arous et al., 2025, Defilippis et al., 2025, Ren et al., 2025]. Several other lines of research isolate distinct nonlinear phenomena: the convergence of homogeneous networks trained on logistic losses to max-margin solutions [Soudry et al., 2018a, Lyu and Li, 2020], the reduction of training dynamics to low-dimensional summary statistics in teacher-student models [Saad and Solla, 1995, Goldt et al., 2019, Ben Arous et al., 2022, Veiga et al., 2022, Zavatone-Veth et al., 2025], memorization in associative memory models [Nichani et al., 2025], learned algorithmic structure in modular arithmetic tasks [Morwani et al., 2023, Gromov, 2023, Kunin et al., 2025], nonlinear solvable models of attention [Zhang et al., 2025, Boncoraglio et al., 2025], and improved scaling laws from nonlinear feature learning [Bordelon et al., 2025]. Taken together, these approaches illustrate both the promise and the limitations of current nonlinear toy models: each captures a slice of fully nonlinear learning dynamics, yet no unified framework has emerged. We view this space as an open and rapidly evolving area, and return to these challenges in our discussion of open problems in Section 5.
2.2
Insightful limits reveal fundamental behavior
Modern deep learning systems are enormous: they regularly involve hundreds of interacting architectural components comprised of hundreds of billions of parameters and trained on trillions of tokens. With so many interacting degrees of freedom, constructing detailed microscopic theories that track individual parameters in practical systems seems all but hopeless. Fortunately, complex systems often simplify when approximated as effectively infinite in size, revealing simple mathematical structure that remains informative even for the original finite system. This strategy is well established in statistical and chemical physics: for example, the ideal gas law, P V = nRT , is derived in a limit of infinite number of particles (often termed the thermodynamic limit) yet accurately describes real parcels of gas of finite volume. Limits are a central mathematical tool for managing the complexity of deep learning, and their recurring success in doing so provides strong evidence for an emerging theory.
8
Figure 2: Large and small network output multipliers are sufficient to induce lazy and rich training dynamics. Pn We train a shallow student network fˆ(x) = α ai ReLU(wi⊤ x) with width n = 200 to match a teacher network i=1 n P3 f ∗ (x) = i=1 a∗i ReLU((wi∗ )⊤ x) on two-dimensional input data. We plot the training trajectories of the student weights wi (color denotes sgn(ai )) against the teacher feature directions. Left: with α = 0.1, the dynamics are rich: the student weights grow significantly and cluster in angle around the teacher feature directions. Right: with α = 30, the dynamics are lazy: the student weights move negligibly during training, even though the loss drops. Experiment reproduced from Chizat et al. [2019]. Here we discuss the limit of infinite width in detail. We conclude by mentioning other limits and offering some unifying ideas. The infinite width limit and the lazy/rich dichotomy. The dynamics of a deep neural network often simplify when one takes the number of neurons in each hidden layer to infinity. Such a limit generally leads to so-called mean-field behavior in which we only need to describe the evolution of the neuron population as a whole (as e.g. a probability distribution) and we can ignore what each individual neuron is doing. However, achieving this limit requires shrinking the initialization scale as width increases to prevent activations in deeper layers from diverging. The key subtlety in taking the infinite width limit is that the rate at which we suppress these initial weights strongly influences the resulting training dynamics, leading to one of two qualitatively distinct limiting behaviors. The lazy, kernel, or linearized regime. The first forays into the land of infinite width studied only a network’s statistics at initialization, not its training dynamics [Neal, 1996, Poole et al., 2016]. These works found that, in order for the inputs to hidden neurons to neither vanish nor explode as width increases, the parameter size at initialization has to decay as [width]−1/2 . This is not a surprise: it is just the well-known LeCun initialization rule [Lecun et al., 1998], which can be easily derived from the central limit theorem. Later works that tried naively training the parameters of these infinite-width networks found the surprising fact that the network’s weights and hidden representations change only negligibly, yet these small changes accumulate to produce substantial changes in the output function. As a result, the training dynamics are linear in the parameters in the sense discussed in Section 2.1, and the evolution of the target function may be expressed entirely in terms of the NTK [Jacot et al., 2018, Lee et al., 2019]. While a network in this limit is wonderfully analytically tractable, the fact that its hidden representations do evolve only negligibly means that it fails to exhibit feature learning. While the definition of feature learning is much debated (see Open Direction 4), all agree that at minimum it requires the network’s hidden activations on a given data sample to change from their values at initialization, which does not happen in this limit. This suggests the NTK infinite-width limit is not the right one to study. Networks in this linearized regime were later termed “lazy” by Chizat et al. [2019]. The rich, active, or feature-learning regime. In answer to this, several authors identified an alternative infinite width limit in which training does induce feature learning. The key insight was essentially to downscale the final-layer weights by a factor of [width]−1 , rather than the earlier [width]−1/2 , thereby forcing the network weights to change
9
more to compensate.2 While this makes the function trivial at initialization (at infinite width it is uniformly zero), it can still grow nontrivially during training, changing by an order-one amount upon each gradient step. This downscale-the-network-output idea first appeared in the shallow “mean-field networks” of Mei et al. [2019], Rotskoff and Vanden-Eijnden [2018], and Chizat and Bach [2018]. Geiger et al. [2020] and Yang and Hu [2021] found that this idea also works for networks of arbitrary depth, bundling the resulting hyperparameter scaling factors together into the celebrated “Maximal Update Parameterization” discussed in Section 2.4. It is now widely accepted that infinite-width neural networks can learn features. Wide networks in this “rich” regime display a huge range of interesting behaviors that their lazy counterparts do not. The most significant is certainly that the hidden features of these networks change over time, adapting to the structure in the input data, altering the internal geometry of hidden representations over the course of training [Bordelon and Pehlevan, 2022]. Subpopulations of neurons specialize, learning to attend to different features latent in the data [Aubin et al., 2018, Goldt et al., 2019, Ren et al., 2025]. For instance, in tasks where the optimal predictions involve low-dimensional subspaces of high dimensional data, the distribution over first layer weights evolves to amplify weights in the subspace of interest [Mei et al., 2018, Abbe et al., 2022, Moniri et al., 2023, Cui et al., 2024, Defilippis et al., 2025, Erba et al., 2025, Montanari and Wang, 2026]. When the scale of the initialization is made even smaller, they often show the greedy low-rank bias discussed in Section 2.1, acquiring some components of the task before others [Saxe, 2015, Atanasov et al., 2021, 2025].3 The lazy–rich dichotomy, and its dependence on initialization scale, emerged as a central finding of infinite-width analyses. Subsequent work has shown that analogous behavior appears even at finite width: scaling down the network output promotes feature learning, pushing models toward the rich regime, whereas increasing the output scale tends to linearize training dynamics and induce lazy behavior [Chizat et al., 2019]. This sensitivity to initialization scale connects to a broader literature on inductive bias, where seemingly small changes to the learning setup can steer training toward fundamentally different solution classes [Maennel et al., 2018, Woodworth et al., 2020]. Figure 2 illustrates how the same finite network, trained with different output scalings, can exhibit either lazy or rich learning dynamics. The infinite depth limit and other hyperparameter limits. As with infinite width, one can arrive at a stable infinite depth limit of a deep residual network by downscaling the contribution of each layer so the residual stream does not blow up. Here, too, there are different limiting behaviors depending on the size of this downscaling factor: suppressing each layer by a factor of [depth]−1 results in limiting dynamics in which the residual stream changes smoothly over depth [Bordelon et al., 2024b, Chizat, 2025, Chaintron et al., 2026] (reminiscent of Neural ODEs [Chen et al., 2018]) while suppressing each layer by a factor of [depth]−1/2 results in limiting dynamics in which the residual stream diffuses as if driven by a stochastic differential equation [Bordelon et al., 2023, Yang et al., 2023b]. Networks in these two limits converge to qualitatively different solutions in realistic architectures such as transformers [Dey et al., 2025]. It is not yet clear which is the more important limit to study. Some deep learning architectures admit size limits other than those of large width or large depth. Instead of increasing size or total number of distinct feedforward layers, one can also analyze the infinite limits of recurrent architectures using similar mean-field ideas [Clark et al., 2026, Bauer et al., 2026]. State-of-the-art transformer models include more expressive constituent blocks such as multi-head self-attention layers and mixture-of-expert multi-layer perceptrons. These layers have multiple scaling directions including head count, head size, and context length for attention [Hron et al., 2020, Bordelon et al., 2024b] and expert count, expert size, and sparsity for mixture-of-expert models [Małaśnicki et al., 2025, Jiang et al., 2026]. Clarifying the interplay of different infinite limits in these models is important to making contact with modern practice and to disentangle various hyperparameters related to initialization and optimization (see Section 2.4). 2 The lazy vs. rich dichotomy is conceptually similar to elastic vs. plastic deformation in materials. A material will deform linearly in response to a small force, and its internal atomic structure will not change. In response to a larger force, it will deform nonlinearly, and its internal structure changes. 3 There is also a well-developed line of work studying the signatures of feature learning in large-width networks from a Bayesian perspective. Naively, infinite-width networks have simple Bayesian statistics given by Gaussian processes [Lee et al., 2017], which is analogous to the “lazy” limit of conventionally-trained networks. This view treats this Gaussian process limit as a solvable reference point [Cohen et al., 2021b, Lavie et al., 2024] and then reintroduces finite width, using mean-field and variational techniques to characterize aspects of feature adaptation to data [Cohen et al., 2021b, Seroussi et al., 2023, Rubin et al., 2023, 2025b,a]. One may also induce feature learning by rescaling the total likelihood (see e.g. Yang et al. [2023a]), which is analogous to the final-layer downscaling which gives the rich limit in conventional training.
10
Lastly, most optimization hyperparameters have an associated limit. As the batch size approaches infinity, we reach population gradient descent. As we take learning rate to zero, we recover gradient flow. If we add an infinitesimal weight decay and take training time to infinity, we first optimize the loss to convergence, then perform parameter norm minimization conditioned on the final value of the loss. We discuss how to understand the corrections induced by having finite values for some of these hyperparameters in Section 2.4. Joint scaling limits. Sometimes scaling limits in multiple variables (ν1 , ν2 ) play nicely, in the sense that lim gives the same result as lim
lim
ν1 →∞ ν2 →∞
lim . For example, the infinite width and depth limits in residual networks usually
ν2 →∞ ν1 →∞
commute in this way, so long as one takes a sensible parameterization [Hayou and Yang, 2023]. However, in many theoretical machine learning settings, different scaling dimensions do not commute, and the limiting behavior could depend on a limiting ratio ν2 /ν1 . Such joint/proportional scaling limits are common in random matrix theory: for example, consider the SVD of a random matrix with P rows and N columns with N, P → ∞ with P/N held constant. In machine learning theory, neural networks trained with random data can often be described by a joint scaling limit [data] [data] where both the dataset size and parameter count are taken to infinity, but one or more of the ratios [input dim] , [width] , or [data] [parameters] is a finite value [Seung et al., 1992, Saad and Solla, 1995, Zdeborová and Krzakala, 2016, Li and Sompolinsky,
2021, Maillard et al., 2024, Martin et al., 2024, Barbier et al., 2025]. This joint scaling is likely necessary in the study of compute-optimal neural scaling laws where the training horizon (i.e. dataset size) is scaled linearly with the total parameters [Hoffmann et al., 2022] and to theoretically characterize hyperparameter transfer phenomena [Bordelon and Pehlevan, 2025]. These joint (data & model size) limits are potentially important as infinite parameter limits at fixed dataset size are capable of perfect interpolation and do not capture scaling law behaviors across model sizes (see Section 2.3). Other well-studied joint scaling quantities include the ratio [width] [depth] in non-residual networks [Hanin [learning rate] and Nica, 2019, Li et al., 2022, Noci et al., 2023, Hanin and Jiang, 2025], the ratio [output multiplier] in the rich regime rate] [Atanasov et al., 2025], and the “SGD noise temperature” [learning [batch size] [Mandt et al., 2017, Jastrzebski et al., 2017].
The Discretization Hypothesis. Overall, the widespread use of limits to manage the complexity of deep learning reflects a recurring theme across scientific disciplines: appropriate asymptotic perspectives often render otherwise intractable systems analytically tractable. Many theorists share a heuristic belief that most practical neural networks can be understood as noisy, finite approximations to models of infinite size.4 By analogy, one numerically solves a partial differential equation by discretizing over space and time, and the finer the discretization, the smaller the numerical error from the desired continuum process. This is very possibly also true of deep neural networks, with width and depth taking the place of space and time. Other finite hyperparameters, such as the learning rate, batch size, and dataset size, might also be understood in this way. We might call this belief the Discretization Hypothesis. While it has yet to be made precise or proven (see Open Direction 5), this hypothesis has implicitly underpinned much important work, and little in the analytical study of large models makes sense without it. The Discretization Hypothesis amounts to the statement that finite-size corrections from limits typically worsen performance while saving costs in data, time, memory, and compute. Showing that these finite-size effects deliver a general benefit that cannot be achieved any other way would falsify this hypothesis.
2.3
Simple empirical laws capture meaningful macroscopic statistics
Deep learning is highly measurable: it is easy to track a vast array of quantities before, during, and after training. While any quantity can be measured, the most lawful are typically aggregate, macroscopic statistics over many weights and samples. For instance, the train and test losses are aggregates over many samples. These quantities are occasionally described by simple empirical laws relating one to another. Such laws have already played an important role in shaping both our understanding and practice of deep learning. This pattern has ample precedent in the quantitative sciences. Many important physical and chemical laws were first discovered as empirical regularities and only later understood in terms of deeper principles, including laws due to 4Works studying finite-size corrections to infinite limits include [Hanin and Nica, 2019, Roberts et al., 2022, Zavatone-Veth et al., 2021, Segadlo et al., 2022, Bordelon and Pehlevan, 2023, Glasgow et al., 2025].
11
Test Loss
Kepler, Snell, Boyle, Hooke, Faraday, Ohm, Poiseuille, and Planck. Given how often scientific fields have developed in this way, it seems likely that deep learning will continue to yield empirical laws as its science matures. Here, we highlight a handful of examples and conclude with takeaways for theorists.
Compute
Dataset Size
Parameters
PF-days, non-embedding
tokens
non-embedding
Figure 3: The loss of large neural networks decays according to predictable neural scaling laws. These neural scaling laws take the form of power laws (linear on log-log plots) in compute, dataset size, and parameter count. Reproduced from [Kaplan et al., 2020]. Neural scaling laws. The single most important measurement of any machine learning system is its test loss. Given the complexity of large deep learning systems, one might expect the test loss to be a complex, unknowable function of the system’s hyperparameters. This is not so: studies of neural scaling laws [Kaplan et al., 2020, Hestness et al., 2017] demonstrate that, within an architectural family, the final loss follows a predictable power law function governed by only three scalar variables: compute, the amount of data, and the network’s size. These power laws are shown in Figure 3. Why does test loss decay as a power law in these variables, and what determines the scaling law exponent? We still do not know! While scaling laws are often attributed to structure in the data, with candidate explanations in terms of the dimensionality of the data manifold [Sharma and Kaplan, 2022, Bahri et al., 2024], feature superposition [Liu et al., 2025], and power laws latent in task structure [Cui et al., 2021, Bordelon et al., 2024a, Michaud et al., 2023, Ren et al., 2025, Defilippis et al., 2025], they may also depend on details of the architecture and optimizer [Barkeshli et al., 2026]. At present, no framework can robustly predict the observed exponents a priori from dataset and architectural properties across realistic settings (see Open Direction 7), though recent progress has begun to move in this direction [Cagnetta et al., 2026]. The fact that test loss is so predictable strongly suggests that a simple underlying explanation remains to be found. Weight dynamics at the edge of stability. Because every model is the result of a training process, we would like to understand the dynamics and trajectory of a model’s weights during training. While there are simple cases where these dynamics are exactly solvable (see Section 2.1), this is usually well out of reach. The loss landscape dictates the network’s dynamics, but a direct visualization of the loss, as is done in Li et al. [2018], suggests an immensely complicated landscape that is unlikely to have lawful regularities. Nonetheless, some robust patterns in the coarse, aggregate properties of weight trajectories have been found. One of these is the sharpness of the network loss surface, defined as the largest eigenvalue of the Hessian with respect to the parameters. When a typical network is trained using full-batch gradient descent with learning rate η, the sharpness undergoes two distinct phases: a gradual increase (termed progressive sharpening) followed by a plateau near 2/η (Cohen et al. [2021a]; see Figure 4), called the edge of stability. Having identified these regularities, we can begin to understand them. Progressive sharpening provably occurs in deep linear networks [Even et al., 2023, Yoo et al., 2025], yet a quantitative explanation suitable to realistic nonlinear networks remains to be found (see Open Direction 8). More is understood about why the sharpness stabilizes at 2/η. Particularly, 2/η is the maximum stable sharpness achieved in convex optimization–any sharpness larger than 2/η would cause parameter oscillations of increasing magnitude. In more general cases, Damian et al. [2022a] showed 12
100 50 0
500 1000 1500 2000 2500 iteration
2 1
train loss
ResNet on CIFAR-10 = 2/25 = 2/50 = 2/100 = 2/200
1 0
300 200 100 0
60 40 20 0
0
2000
4000 6000 iteration
8000
= 2/5 = 2/10 = 2/20 = 2/40
2
0 sharpness
1
VGG on CIFAR-10
3 train loss
= 2/20 = 2/50 = 2/80 = 2/110
2
sharpness
sharpness
train loss
Fully-connected net on CIFAR-10 5k subset
0
2000
4000 iteration
6000
Figure 4: Gradient descent occurs near the edge of stability. Three architectures are trained with full-batch gradient descent on CIFAR-10 with varying learning rate η. Plots show the train loss (top row) and Hessian sharpness (bottom row). For each step size η, observe that the sharpness rises to 2/η (dashed horizontal lines) and hovers at or just above this value. Reproduced from Cohen et al. [2021a]. how coarse properties of the third-order loss curvature can cause the (second-order) sharpness to stabilize at 2/η. Follow-up work reveals that loss dynamics at the edge of stability can be decomposed as smooth, time-averaged, gradient flow dynamics plus oscillations in unstable directions [Cohen et al., 2025]. These works make quantitative predictions about the parameter trajectory which closely match experiment. Coarse properties of hidden representations and weights. There are a handful of other cases in which coarse properties of neural networks’ hidden representations and weights are known to obey simple equations. We will briefly mention three of these. Neural collapse. Consider a neural network classifier trained to choose among C classes. Papyan et al. [2020] found that, at the end of training, the final-hidden-layer representations of samples from each class tend to cluster tightly around their class mean. Furthermore, the C class mean vectors form a regular simplex. Later theoretical work has explained this geometric arrangement as the natural energy-minimizing configuration when (a) the loss used is cross-entropy and (b) a small amount of weight decay is applied [Zhu et al., 2021].5 The neural feature ansatz. At the other end of the network, there are some robust regularities known about the first-layer weights. Radhakrishnan et al. [2024] show that, after training, the Gram matrix of the the first-layer weights W1⊤ W1 aligns with the average gradient outer product: W1⊤ W1 ∝ Ex∼Pdata ∇x f (x; θ)∇x f (x; θ)⊤ , (3) where ∇x f (x; θ) denotes the Jacobian of the network with respect to x. While this rule is heuristic and inexact, it often makes strikingly accurate predictions for quantities like the top eigenvectors of W1⊤ W1 . Similar heuristics hold at deeper layers. At time of writing, there are only partial theoretical explanations for this phenomenon; see Ziyin et al. [2024], Boix-Adsera et al. [2025]. Gradient flow conservation laws. A striking regularity identified in linear networks is that the difference between ⊤ the covariance and Gram matrices of consecutive layers Wℓ Wℓ⊤ − Wℓ+1 Wℓ+1 is conserved under gradient flow [Saxe et al., 2014, Du et al., 2018, Arora et al., 2019a]. What initially appeared to be a curiosity of linear networks was later shown to follow from continuous symmetries of the parameterization — an instance of the Noether principle — and thus could be used to identify similar conserved quantities in nonlinear networks [Kunin et al., 2021, Tanaka and Kunin, 2021, Marcotte et al., 2024a,b]. For instance, the rescaling symmetries in networks with homogeneous nonlinearities (e.g., ReLU), the scale symmetries preceding normalization layers (e.g., batch normalization), the translation symmetries in the logits preceding a softmax, and the rotation symmetries between key and query matrices in attention all lead to symmetry-specific statistics of the parameters that are conserved under gradient flow and weakly broken by SGD in predictable ways. 5 This parallels how gradient descent on separable logistic regression converges in direction to the max-margin separator [Soudry et al., 2018b].
13
Takeaways for theorists. Theory can be built “bottom-up,” starting from first-principles math as in Sections 2.1 and 2.2, or “top-down,” starting from empirical observations and attempting to explain them. In this section we have highlighted a few notable examples of top-down theories. We expect more to come. The measurability of deep learning makes observation and empiricism a particularly fruitful approach, since experimentation can be iterated on quickly, while revealing mathematically simple relations and structure in trained models. Of course, some caution is necessary: most macroscopic statistics don’t obey a simple and general mathematical law — or at least don’t seem to until plotted against the right quantity — and so the challenge is to find those that do. We encourage theorists of deep learning to proactively use experiments to look for lawful regularities in neural networks.
2.4
Hyperparameters can be disentangled and understood
Training a deep learning system involves many numerical knobs, termed “hyperparameters.” These include optimization hyperparameters such as the learning rate, batch size, momentum, and initialization variance, as well as architecture hyperparameters such as width, and depth. The large number of hyperparameters in deep learning presents a challenge not only for practitioners, who must tune them carefully in order to achieve optimal performance, but also for researchers, who must grapple with many confounding factors when trying to interpret the outcome of scientific experiments. It is only in the last few years that the theory community has come to realize that hyperparameters can be disentangled and understood, and that the resulting mathematics is often both useful for practitioners and clarifying for theorists. This study of hyperparameters bears similarities to the study of the constant parameters governing the behavior of a physical dynamical system. For example, in a fluid flowing through a pipe, a dimensionless number called the Reynolds number computed from the pipe diameter and the fluid’s speed, density, and viscosity determines whether flow is laminar or turbulent. While solving for the trajectory of the turbulent fluid is extremely difficult, it is nonetheless very helpful to be able to quickly predict whether flow will be turbulent at all — and how things change if you scale up the pipe diameter or increase the fluid flow. Similarly, while solving the optimization dynamics of a neural network is very difficult, it is often very helpful to quickly obtain a coarse picture of how things change if you change one or more hyperparameters. In this section we highlight two lines of work in which hyperparameters have been found to admit explanatory theory. Understanding optimization hyperparameters. Stochastic gradient descent has two hyperparameters: learning rate and batch size. The algorithm’s dynamics are often invariant under a simultaneous rescaling of both. That is, if one doubles both the learning rate and batch size, and halves the number of optimizer steps (or equivalently, keeps fixed the number of training examples processed), then the trajectory stays nearly the same. This so-called linear scaling rule [Goyal et al., 2017] is useful for transferring a learning rate that was tuned for one batch size to a different one. A line of theoretical work has clarified this rule of thumb by interpreting SGD as a discretization of an underlying stochastic differential equation (SDE), a perspective that predicts the linear scaling rule [Mandt et al., 2017, Jastrzebski et al., 2017, Chaudhari and Soatto, 2018, Li et al., 2019, 2021b]. Malladi et al. [2022] extended this line of work from SGD to adaptive optimizers, for which they argued that the learning rate should scale with the square root of the batch size. This invariance perspective explains how to adjust hyperparameters across batch sizes, but not how to choose the batch size itself. That choice involves an inherent tradeoff between two resources: serial time (the number of sequential training steps) and overall compute (the total amount of computation, often closely tied to cost) [Ma et al., 2018, Jain et al., 2018, McCandlish et al., 2018, Shallue et al., 2019]. For a practitioner who cares only about serial time and not at all about cost, the optimal batch size is the full dataset. Conversely, for a practitioner who cares only about cost and not at all about serial time, the optimal batch size is 1. In reality, no practitioner falls exactly in either bucket; a practitioner might care more about one resource than the other, but is generally willing to accept some slack in return for a better deal on the second resource. A frequently discussed concept is that of the critical batch size, a batch size which trades off between these two concerns. McCandlish et al. [2018] proposed a simple model of this tradeoff under which the Pareto frontier between serial time and compute takes the form of a hyperbola. Optimization hyperparameters in deep learning affect not just the speed and cost of training but also the trajectory that training follows. This in turn affects various properties of the learned network, including generalization performance [Keskar et al., 2016, Schulman and Lab, 2025] and compressibility [Catalan-Tatjer et al., 2025, Barsbey et al., 2025]. A
14
NNs) with billions of parameters. We show that, in the mal Update Parametrization (µP), many optimal HPs el size changes. This leads to a new HP tuning paradigm trize the target model in µP, tune the HP indirectly on a fruitful line of work has sought to explain these effects through the hypothesis that many implicit effects of optimizer hot transfer them to the full-sized model, i.e., without hyperparameters can be understood as implicit regularization of loss function curvature. Empirical studies initially observed thatµTransfer first-order optimizers the curvature (i.e. and Hessian)ResNet. of the loss function, with larger learning t all. We verify onregularize Transformer rates and smaller batch sizes yielding stronger regularization strengths [Keskar et al., 2016, Jastrzebski et al., 2017, rring pretraining from a model of in13M parameters, 2020, Cohen et HPs al., 2021a]. Meanwhile, theoretical works simplified settings showed that this effect can be explained by Taylor-expanding the objective to third order, as such a calculation reveals that oscillating or fluctuating dynamics numbers ofautomatically BERT-large (350M parameters), a total induce curvature regularization [Blanc et al., 2020, Liwith et al., 2021c, Damian et al., 2021, Wen et al., 2022, Li et al., 2025]. Building on this body of work, Cohen et al. [2025] recently showed that for several optimizers in the pretraining BERT-large once; 2) by transferring from full-batch setting, the whole training trajectory on realistic neural nets is well-modeled by a curvature-penalized gradient flow, where the role of of the hyperparameters is to modulate both the form and strength of the curvature rform published numbers the 6.7B GPT-3 model, with penalty. As a result, we now have a mathematical understanding of the learning rate in full-batch gradient descent, tal pretraining cost. Pytorch implementation and are mostly free toA instead study the simpler dynamics of gradient flowof plusour a loss curvature penalty. Other analyses have developed analogous characterizations for stochastic dynamics in more specialized settings [Pesme github.com/microsoft/mup and installable via pip et al., 2021, Chen et al., 2024]. Fully extending this characterization to stochastic and adaptive optimizers would give 6
7
us a common language for reasoning about the implicit effects of optimization hyperparameters on the training trajectory. It then remains to understand how these modifications to the training trajectory influence properties of the learned network (see Open Direction 8). Standard Practice Standard Practice
Our Work μP
7.0 6.5
Training Loss
tical to deep Width lt in subpar y. Many pubpare to one optimum stable HP tuning. optimum shifts training exlog LearningRate log LearningRate , since stateFigure 5: The theory of 1: network parameterization permits learning rate transfer across widths.on Transformers Figure Training loss against learning rate f parameters of varying widths trained on WikiText-2 under standard parameterization (left) and µP (right). Under standard parameterization, the optimal learning decreases asd model width trained increases. Under µP, byAdam. contrast, the optimal Transformers ofrate varying with tune. model learning rate remains nearly constant across widths, making it possible to predict the learning rate for wide networks Conventionally contrast with our technique, from experiments on narrower, cheaper and models.in Reproduced from Yang et al. [2022]. t neural netdifferent widths do not share the same optimal hyrent infiniteDisentangling architecture hyperparameters from optimization hyperparameters. There has been a highly perparameter; wider networks do not always persuccessful line of work aimed at disentangling architecture hyperparameters such as width, depth, and output mal Update multiplierform (see the lazy/rich dichotomy in narrower Section 2.2), from ones; optimization such as the learning rate better thanPrograms inhyperparameters fact they underand initialization variance. The Tensor framework [Yang and Hu, 2021, Yang and Littwin, 2023] makes this summarized the same-width networks ourη =technique separationperform explicit, writing hyperparameters such as the learning rate in in the form η · [width] , separating a eature learnscale-independent coefficient η from a width dependent factor with exponent c. This line of work then asks: how aftersuchtuning learning rate (see dashed can we seteven these exponents that we retain interesting training behavior at infinite width? Aline). remarkable insight es that each from this analysis is that all non-trivial and non-explosive scalings give one of two limiting behaviors, analogous See Sections 3 and 4 for experimental setup. to the rich/lazy dichotomy in Section 2.2: in the Neural Tangent Parameterization (NTP), features are frozen during during trainAn additional, but apparently weaker, effect is captured by an implicit regularization of the gradient norm [Barrett and Dherin, 2020, Smith et al., 2021]. st, while the This perspective is reminiscent of the Itô’s correction in stochastic calculus: after a nonlinear transformation, noise can contribute an additional deterministic drift. Likewise, stochastic or oscillatory dynamics may be described an effective flow on a modified loss. res activations are of unit order atoptimization initialization, it byactually causes uring training [57] essentially due to an imbalance of per-layer 15 6.0 5.5
128 256 512 1024 2048 4096 8192
5.0 4.5 4.0 3.5
20
18
16 2
14
12
10
20
18
16
14
12
0
0
6 7
t AI Residency Program.
10
2
c
(a) Universality across architectures
(b) Universality across data modalities
Figure 6: Universality across architectures and data modalities. (a): Different diffusion model architectures (from top to bottom: DDPM, a consistency model—both based on UNet—and U-ViT) converge to the same learned distribution and produce identical images when given the same input seed. Adapted from Zhang et al. [2024] (b): As language models performance (horizontal axis) increases, their internal representations become increasingly similar to that of vision models, and more so for larger models (from yellow to purple lines). Adapted from Huh et al. [2024]. training, and in the Maximal Update Parameterization (µP), features evolve. Since feature learning is essential for most tasks, this analysis tells us that µP is the scaling to use, resolving how hyperparameters should scale with model width. This understanding enables hyperparameter transfer: we can tune hyperparameters on small proxy models and then transfer them to large, production-size models, where they remain near-optimal when both models are sufficiently wide ([Yang et al., 2022]; Figure 5). At the same time, the theory underpinning this result is asymptotic and does not fully account for its empirical effectiveness. In practice, models are trained at widths far smaller than the dataset size, and the usefulness of transfer depends on how quickly optimal hyperparameters stabilize with width. Noci et al. [2024] and Ghosh et al. [2025] and Hayou [2025] take steps toward closing this gap, providing evidence that a small set of spectral statistics stabilizes rapidly across widths under µP and approximately governs the optimal hyperparameters. This scaling-centric approach to hyperparameters was later extended to depth scaling [Yang et al., 2023b, Bordelon et al., 2023, Dey et al., 2025], and leveraging this approach with other scaling dimensions remains an important future direction (see Open Direction 6).
2.5
Universal phenomena appear across settings and tasks
Deep learning is not a single recipe followed exactly every time: different systems use very different architectures, datasets, training algorithms, and objectives, with ingredients combined in creative ways. This versatility has enabled successes on many tasks and modalities including vision, language, speech, time series, protein sequences, and games, but the resulting model diversity makes it less clear how to approach the development of scientific theory. Do these diverse settings share deep commonalities we might hope to capture scientifically? Here, we review a growing body of evidence that there are indeed universal phenomena at play in these diverse settings. This is good news for theory: when many different complex systems exhibit the same universal behavior, it suggests that a simple underlying explanation may exist. We highlight this universality through three different viewpoints: (1) different architectures reach comparably good performance on many tasks; (2) different datasets share similar statistical properties; and (3) the learned representations and weights across different architectures and datasets are surprisingly alike. This roughly echoes examples of universality in which disparate physical systems share deep commonalities or display similar behavior at large scales.8 We end by highlighting a few theoretical successes in modeling universal phenomena. 8 Universal behavior across physical systems can often be understood with the renormalization group, a technique which formalizes the idea that, as one examines a system from a more and more zoomed-out perspective, most details “wash out” and only a handful of aggregate effects remain important. We note that another apt analogy for universality in deep learning, this one from biology, is convergent evolution: species that “solve similar problems” tend to “find similar solutions” after many generations.
16
Universal inductive biases. Performance on a given task is often robust to variations in architectures, training algorithms, and objectives, in the sense that many alternate choices still lead to models that can solve the task. A well-known example is the choice between convolutional networks and transformers in computer vision tasks, which after much debate have been shown to obtain similar performance when matching compute, data size, and training recipes [Liu et al., 2022, Smith et al., 2023]. In diffusion models, this similarity has been further shown to hold at the level of input-output mappings, with transformers and UNets generating near-identical images when fed with the same noise samples [Zhang et al., 2024], as shown in Figure 6. These results strongly indicate that different architectures share similar inductive biases despite their apparent differences. As a partial explanation, recent work has shown that assuming inductive biases towards locality and adaptivity to geometric structures leads to accurate quantitative predictions about the behavior of diffusion generative models [Kadkhodaie et al., 2024, Kamb and Ganguli, 2025, Niedoba et al., 2025]. Universal structure in data. The no-free-lunch theorem states that generalization on completely arbitrary data with a common learning strategy is not possible [Wolpert, 1996]. Therefore, deep learning must rely on particular features of the data present across all datasets and modalities on which it succeeds. For instance, many classes of images and audio signals share power-law spectral properties, sparsity patterns, and multiscale structures, and can be analyzed with general-purpose wavelet bases [Olshausen and Field, 1996, Mallat, 1999] A similar phenomenon in text data is the ubiquity of Zipf’s law (word frequencies obey a power-law distribution) that holds over many natural and artificial languages [Li, 2002, Piantadosi, 2014]. Hierarchical, compositional structure is also routinely used to model both images and text, which can sometimes be related through a common model [Cagnetta et al., 2024, Sclocchi et al., 2025, Cagnetta et al., 2025]. These shared statistical properties are a partial explanation for the ability of a single learning algorithm (say, a transformer trained with SGD) to tackle seemingly unrelated datasets, leaving only the finer-grained differences between them to be learned. Universality in representations. Going deeper in the internals of the network, it has been observed that representations learned by different networks can be similar across random initializations, widths, and architecture [Raghu et al., 2017, Kornblith et al., 2019, Bansal et al., 2021, Huh et al., 2024, Moschella et al., 2022]. It has been shown that networks trained to solve different tasks learn similar representations across training datasets (ImageNet and Places-365, Lenc and Vedaldi [2015]), objectives (supervised or self-supervised, [Bansal et al., 2021]), and modalities (vision or language, [Huh et al., 2024]). Furthermore, this similarity grows as model size and performance increase, hinting that neural activations converge towards a universal (“Platonic”) representation [Bansal et al., 2021, Huh et al., 2024], as shown in Figure 6. In simplified settings such as random feature representations, this convergence is a consequence of the law of large numbers applied to the feature kernels [Rahimi and Recht, 2007, Guth et al., 2024]; in deep linear networks, it can be proven to arise from the implicit regularization of SGD [Ziyin and Chuang, 2025]; in more diverse settings, recent evidence suggests that the universality of representations may ultimately trace its origins to universal structure in data [Huh et al., 2024, Karkada et al., 2026]. Recent advances in identifiability theory [Hyvärinen et al., 2024] also have shown that representational convergence happens at the global optimum of unsupervised [Klindt et al., 2020], self-supervised [Zimmermann et al., 2021] and supervised [Reizinger et al., 2024] objective functions under a suitable data generating process [Reizinger et al., 2025]. Several works have also shown empirically that this similarity can extend to the level of individual neurons [Li et al., 2015, Dravid et al., 2023, Khosla et al., 2024]. In some cases, similar representations have been found in both artificial neural networks and biological neural networks [Olshausen and Field, 1996, Yamins et al., 2014, McIntosh et al., 2016], though the extent of this correspondence remains controversial [Bowers et al., 2023]. While a global trend towards similarity is emerging, it should be noted that the range of settings in which this convergence is observed, and its extent, are not fully known (see Open Direction 10). In particular, recent work has shown that this apparent convergence to universal representations depends crucially on the chosen comparison metric across similarities [Gröger et al., 2026]). A growing literature is devoted to understanding which representation similarity metrics one should choose in different circumstances [Sucholutsky et al., 2023, Klabunde et al., 2025] and highlighting the cases where they can be unified [Harvey et al., 2024, Williams, 2024]. If the mechanisms learned by large models are indeed universal, this is very encouraging for theory: behavior shared across many systems should depend primarily on the features common to all such systems, and thus admit a description simpler than any particular model in isolation. Moreover, if the internal structure of trained neural networks primarily reflects the structure of data, then in studying neural networks we may ultimately be studying 17
the structure of data and its generating processes (see Open Direction 2). In particular, since language data comes directly from humans, understanding its structure may teach us something new and fundamental about ourselves.
3
Relation to other perspectives
There are several ongoing approaches to developing explanatory scientific theory of deep learning, each adopting a different perspective and using different sets of tools. We believe that these perspectives are essentially all complementary: all either directly seek a mechanics of learning or would symbiotically benefit from one. The statistical perspective. The rich tradition of classical learning theory remains influential today.9 Bartlett et al. [2021] offer a lucid summary of its central framing: any statistical prediction method must balance expressivity (to represent the richness of real data), complexity control (to make the most of finite training data), and computational efficiency (to yield practical algorithms). It is apparent that deep learning is sufficiently expressive, but it is not clear how a good function is selected from this enormous function class, nor why simple gradient methods suffice to train such complex beasts. The modern statistical viewpoint suggests two answers: deep learning has an implicit inductive bias towards simple, well-generalizing functions [Wilson, 2025], and despite their nonconvexity, the very high dimensionality (overparameterization) of neural networks makes optimization easy. These questions are good ones, and we believe these answers are basically correct. The challenge is now to make them precise in the case of neural networks. It is clear that doing so will require taking a close look at the nature of the training process. Only once we have done so will we be able to back out how this implicit bias arises and why gradient methods suffice for optimization. We do not believe these answers will be generic statements, but instead critically rely on important properties of deep learning and of natural data. The statistical perspective thus leads naturally to a serious scientific study of the mechanics of training. The information-theoretic perspective. A closely-related approach seeks to explain deep learning in terms of information-theoretic ideas. In this view, learning is a process of extracting information from datasets, and a learning system works when it extracts information useful for prediction while discarding irrelevant information. This perspective hopes to understand learning as compression of the dataset into either the model’s parameters or its hidden representations, with good generalization resulting when this compression is successful [Shwartz-Ziv and Tishby, 2017, Xu and Raginsky, 2017]. We find this perspective insightful, and it seems likely to us that a picture of this nature will hold. As with the statistical perspective, a major remaining question is how to make this view concrete and actionable: how do the architecture and training process of deep learning interact to actually implement this compression, and what factors make it more or less successful? Doing this, too, will require taking a close look at the nature of the training process, the architecture, the data, and their interactions. The information-theoretic perspective thus also leads naturally to a serious scientific study of the mechanics of training. Physics of deep learning. This community descends from the older physics of machine learning lineage [Hopfield, 1982, Amit et al., 1985, Gardner, 1988] and essentially seeks satisfying average-case theories of neural network learning [Zdeborová, 2020, Bahri et al., 2020, Michaud, 2024, Ringel et al., 2025].10 The close relationship between physics and machine learning was recognized by the 2024 Nobel Prize in Physics. This approach is in line with (and has largely shaped) the perspective presented in this paper, and the project of this community is arguably the development of a mechanics of learning. The challenge then is to clarify important problems and coordinate effort for efficient progress. Perspectives from neuroscience. Several approaches to developing a science of the brain suggest approaches to developing a science of deep learning. One approach starts from hypotheses about neural systems — for example, that their computation amounts to some form of approximate probabilistic inference — and seeks to make deductions 9 The Simons Institute for the Theory of Computing has provided an important substrate for developing this statistical perspective of deep learning through collaborations and seminars. 10 In the modern day, this community is mediated in part by recurring events at the Kavli Institute for Theoretical Physics, the Aspen Center for Theoretical Physics, and the Les Houches School of Physics, and organizations such as the NSF AI Institute for Artificial Intelligence and Fundamental Interactions and the Simons collaboration on the physics of learning and neural computation.
18
and predictions from this hypothesis [Dayan et al., 1995, Friston, 2010]. Some of these predictions seem to hold suspiciously well in deep learning: see, for example, the case of edge-selective cells in the visual cortex [Olshausen and Field, 1996] and edge-selective receptive fields in convolutional networks [e.g. Zeiler and Fergus, 2014]. Another approach termed systems neuroscience seeks to directly decompose subsets of the brain into interpretable circuits and reverse-engineer the structure of their learned representations [Chung and Abbott, 2021, Bernardi et al., 2020, Kriegeskorte et al., 2008]. This approach resembles mechanistic interpretability, which has adopted some of its methods and intuitions. We expect and encourage this dialogue to continue, and it seems plausible that some of these high-level hypotheses about the brain — e.g., that the brain admits at least a partial decomposition into interpretable circuits and that local circuits implicitly solve inference tasks — will turn out to be true of deep learning. The reasons these facts are true, if indeed they are, is surely bound up in the dynamical way learning actually happens. A study of the mechanics of learning is thus important to the continued exploration of these ideas. Developmental interpretability/singular learning theory. This approach, which grew out of the mechanistic interpretability community, seeks first-principles predictive theories of neural network learning based on the singular learning theory framework of Watanabe [2009], emphasizing a Bayesian perspective and aiming to understand training as a process of sequential phase transitions mediated by the geometry of the loss landscape [Hoogland et al., 2023]. We see this community as seeking the same goal we suggest here — a fundamental mechanics of learning, and a rigorous foundation for interpretability — but with a toolkit that differs from the other listed perspectives. There is potential for fruitful cross-pollination and tool-sharing between these different approaches. Science of deep learning. It has long been appreciated by practitioners that machine learning is largely a practice of trial and error and that it may be possible and beneficial to systematize it [Langley, 1988, Gal, 2015, Rahimi, 2017, Baraniuk et al., 2020]. Indeed, much of the rapid empirical progress of the last decade resulted from systematic organization around agreed-upon benchmark tasks [Donoho, 2024]. Nonetheless, the training and application of large models remains more alchemy than science. We believe that a fundamental mechanics of the learning process is the foundation on which this science will finally be built.
3.1
Learning mechanics ⇄ mechanistic interpretability
We discuss mechanistic interpretability specially because there is a unique opportunity for cooperation. Mechanistic interpretability aims to understand trained neural networks by identifying the internal mechanisms — features, circuits, and learned algorithms — that give rise to their behavior. At its core, this approach is guided by the belief that neural networks admit a human-understandable, mechanistic description that can be uncovered through careful empirical reverse engineering.11 This approach has already borne fruit: many visually striking or interpretable mechanisms have been discovered in large models to date [Olah et al., 2020, Templeton et al., 2024, Engels et al., 2024, Gurnee et al., 2025, Lindsey et al., 2025].12 This is a complementary perspective to our own and presents a wonderful opportunity for symbiosis. At time of writing, mechanistic interpretability remains largely a qualitative science, more reliant on human-judged empirics than on compact mathematical principles or simple governing laws. This is quite natural: semantically-meaningful functions resist mathematical characterization.13 On the other hand, a mechanics of learning would be quantitative by definition, but by the same token will be too low-level to answer important questions of semantic meaning on its own. These approaches study the same system — i.e., deep learning — at different levels of abstraction, and so of 11 The mechanistic interpretability community does not yet share a formal definition of what constitutes a “mechanistic description,” though see Geiger et al. [2025] for a recently proposed causal framing. Informally, many researchers proceed under a set of working assumptions: (1) that neural networks encode the state of internal computational variables in their activations, often referred to as “features”; (2) that successive layers transform and combine these features in structured “circuits”; and (3) that, taken together, these circuits implement algorithms that admit some level of human-understandable description. 12 Mechanistic interpretability is deeply associated with Anthropic and the AI safety and Effective Altruism communities, though it is increasingly pursued in academic labs. We also note that mechanistic interpretability has recently split into an ambitious camp that hopes to develop full, explanatory scientific theory and a pragmatic camp which is mostly interested in targeted interventions for particular cases. See also Saphra and Wiegreffe [2024] for a discussion of the origin of the term “mechanistic interpretability” and the dynamics of its community. 13 For example, try writing down a function that can classify dogs vs. cats from image pixel values. The difficulty of expressing such functions in mathematics is why we invented deep learning in the first place!
19
course they can (and should) work together for mutual gain. Calls for rigorous foundations for interpretability have been steadily growing [Sharkey et al., 2025, Joshi et al., 2026, Greenspan et al., 2026], and this is one thing learning mechanics can and should seek to help provide. In turn, mechanistic interpretability offers learning mechanics a rich and growing tableau of empirical phenomena ripe for the development of explanatory mathematical theory. Learning mechanics → mechanistic interpretability. We emphasize two complementary avenues through which learning mechanics can support mechanistic interpretability: formalizing core assumptions and explaining how mechanisms develop through training. Formalizing core assumptions. Learning mechanics can make explicit, formalize, and, where necessary, challenge the core and often implicit assumptions that guide interpretability research. These include: • linear representability — that features correspond to meaningful directions in activation space [Mikolov et al., 2013, Park et al., 2023a, Nanda et al., 2023b, Marks and Tegmark, 2023, Jiang et al., 2024, Csordás et al., 2024]; • locality — features and circuits are localizable to particular subsets of model components[Meng et al., 2022, Wang et al., 2022, Conmy et al., 2023, Arora et al., 2025]; • sparsity — that individual features and circuits are activated or functionally relevant on only a small fraction of inputs [Cunningham et al., 2023, Bricken et al., 2023]; and • compositionality — that complex network representations and computations arise from the composition of simpler, modular sub-mechanisms [Thorpe, 1989, Smolensky, 1990, Lepori et al., 2023, Schug et al., 2023, Ramesh et al., 2023]. These core assumptions underpin the identification, isolation, and analysis of the internal mechanisms of trained neural networks in mechanistic interpretability research. A mathematical theory of learning offers a way to clarify the regimes in which these assumptions hold, the conditions under which they fail, and the sense in which they can be derived from training dynamics and data statistics (see Open Direction 4). Explaining how mechanisms develop through training. Mechanistic interpretability has generally prioritized describing what mechanisms trained neural networks have learned, and there remains a rich opportunity for work which aims to explain how and why such mechanisms form in the first place. There is already substantial interest within parts of the interpretability community in this dynamical/theoretical perspective, including work on the formation of induction heads [Elhage et al., 2021, Olsson et al., 2022], grokking and progress measures [Nanda et al., 2023a], sudden phase transitions in circuit formation [Elhage et al., 2022, Chen et al., 2023, Gopalani et al., 2024, Park et al., 2024], and the research program of developmental interpretability [Hoogland et al., 2023, 2025], discussed earlier. Our goal is not to replace these efforts but to encourage deeper engagement between mechanistic interpretability and the broader landscape of mathematically grounded ideas and tools in learning mechanics. Echoing Saphra [2022], we hope that learning mechanics can play a role analogous to evolution in biology: just as “nothing in biology makes sense except in the light of evolution,” the internal mechanisms of trained networks may be most naturally understood in the light of the processes that give rise to them. Learning mechanics ← mechanistic interpretability. Conversely, learning mechanics has been deeply influenced by the empirical discoveries of mechanistic interpretability, which often identify concrete phenomena that invite first-principles explanation. Mechanistic interpretability places the structure of data at the center of its analyses, revealing settings in which the relationship between input structure and learned mechanisms is especially clear [Nanda et al., 2023a, Shai et al., 2024]. By contrast, much of classical deep learning theory has relied on highly simplified data models, leaving a gap between theoretical predictions and behaviors observed in practice. In this way, mechanistic interpretability helps bridge this gap by providing learning mechanics with concrete, well-defined targets for theoretical modeling. Several such observations have already proven influential in stimulating work in learning mechanics, including the emergence of induction heads for in-context learning [Bietti et al., 2023, Reddy, 2023, Nichani et al., 2024], the role of Fourier features in algebraic tasks [Morwani et al., 2023, Kunin et al., 2025, Marchetti et al., 2026], and the geometry of features arising from the structure of correlations in the data [Engels et al., 2024, Prieto et al., 2025, Karkada et al., 2026]. Just as the development of physics was often driven by empirical discoveries in adjacent fields, we expect progress in learning mechanics to be driven by theorists who take seriously empirical phenomena, including those uncovered by the mechanistic interpretability community, and seek to explain them. 20
4
Reasons for skepticism and responses
We have made a case that an ambitious mathematical theory of deep learning is possible and that developing this theory is a worthwhile endeavor. This is far from a universal view, and so we now address common counterarguments that a theory of deep learning is either not possible or not a goal worthy of our effort. Competent researchers have been trying to develop a theory of deep learning for decades, and we don’t have one. Surely if there was a theory, we would have already found it. It is true that machine learning theory is a field with a long history, and certain avenues for developing theory have been thoroughly explored. Why should now be different? There are several reasons for optimism. First, the practical success of deep learning is comparatively recent, and we have a wealth of new empirical systems to study and mine for explainable phenomena. Some of these phenomena, like the apparent convergence to universal representations discussed in Section 2.5, were only revealed by the last few years of model scaling. These developments have turned the search for a theory of deep learning from a mathematics into an empirical science (and one with no lack of interesting things to measure). We now have much better means to ask questions and check our answers in a tight feedback loop. Second, the field is much bigger: empirical successes have attracted researchers from physics, mathematics, neuroscience, and other adjacent fields, and so we have more and more diverse minds on the case. Third, it is worth noting that the development of major sciences has usually taken at least several decades, so we should not be too discouraged that we do not yet have all the answers. The objects currently understood from theory are very primitive compared to e.g. LLMs. Surely firstprinciples understanding of large models is too heavy a lift. Indeed, we expect that building up to LLMs will be a heavy lift and take considerable time. The near-term hope is instead that some understanding of the basic building blocks of deep learning will prove useful even without a constructive theory that explains the whole model. We can see this happening already in isolated pockets, including empirical scaling laws (Section 2.3), mathematical prescriptions for hyperparameter scaling (Section 2.4), neural-tangent-kernel-based methods for data attribution [Park et al., 2023b], and theoretically motivated optimizers [Gupta et al., 2018, Jordan et al., 2024]. These “local theories” of small pieces of the deep learning stack are useful for hyperparameter scaling in large models, even though they are in no way comprehensive theories of the model! One might hope for similarly useful “local theories” that treat subjects like training instabilities, dataset selection and attribution, or the effect of normalization layers. It is also important to stress that the identification of the right basic objects in a field of science often makes it possible to ask applied questions in a more sensible way. Consider, for example, how the understanding that all matter is made of atoms underlies virtually all other basic science, and how knowledge of electromagnetism permits optical and radiological tools in countless applied disciplines. As discussed in Section 3.1, we hope that learning mechanics can offer tools that adjacent fields such as mechanistic interpretability can apply to better carry out their work. In this way, rigorous work on primitive objects can aid the applied science of large models even without a rigorous theory that builds all the way up. What matters is a model’s high-level behavior. Microscopic theories are too zoomed in to see this. Models’ high-level behavior is indeed important. How does this fit in with the lower-level sciences of deep learning? We argue that deep learning may be studied at the level of physics, biology, or psychology, with this last including the study of the model’s capabilities, personality [Betley et al., 2026], and goals. It seems likely that study at all levels will be necessary. Learning mechanics (the physics of deep learning) is the farthest from model psychology, with mechanistic interpretability (the biology) lying in the middle and connecting the two.14 We don’t need a theory of deep learning, we need a theory of data. We think we need both: we need a theory of the structure in data and a theory of how a parameterized model learns it. We touch on the necessity of developing a useful theory of data in Section 2.5 and Open Direction 2. These are both part of the project of developing a mechanics of learning. 14We note that these three levels of study of deep learning are roughly analogous to Marr’s levels of analysis of a computational system: the physical implementation of the computation, how the computation is performed algorithmically, and what is being computed [Marr, 2010].
21
AI will understand itself before we do. Why try to build theory? This is a present concern for human intellectual endeavors across the board. Our response here has three parts. First, theory is already useful, and will continue to be more impactful as it develops, so this scientific work is likely to make a near-term impact. Second, it seems unlikely that AI working in isolation will suddenly and separately “solve deep learning theory.” It seems more likely that breakthrough progress in a transitory period will come from human scientists using or working with AI, and expert humans will remain in the loop. Third, if one’s goal is AI safety, some human oversight of AI systems will be necessary (unless one trusts the AIs to fully police themselves), and having a human-parseable theory of deep learning gives us a foot in the door.
5
Open directions in learning mechanics
It is important for any field, at any stage of development, to have a sense of its important open questions and goals. In this section, we present a curated list of open directions which we expect can be solved by a theory of the mechanics of learning in the next decade. These directions are loosely ordered by their connection to the lines of evidence introduced in Section 2. We hope this helps sharpen a shared research agenda. For a longer catalog and a forum for community discussion, see learningmechanics.pub/openquestions. Open Direction 1: What are simple, solvable models of genuinely deep, nonlinear learning? As discussed in Section 2.1, deep linear networks and kernel methods are the two main workhorse solvable models of learning mechanics. The first captures nonlinear dynamics of the parameters, and the second learns nonlinear functions of the data. While a few special cases of solvable models with both forms of nonlinearity are known, no unified framework has emerged. Can we get the best of both worlds while maintaining some level of generality? Is there a class of solvable model that captures both deep, nonlinear dynamics and nonlinear function learning? Can such models illuminate new things about feature learning, the role of depth, optimization phenomena (e.g. progressive sharpening), and architectural innovations (e.g. normalization layers, residual streams, self-attention, and gated nonlinearities)? Can it be usefully applied to modern learning paradigms like self-supervised learning, reinforcement learning, and denoising diffusion? Open Direction 2: What would a theory capable of capturing natural data look like? Deep neural networks find and exploit structure in natural data. This means that the structure of the data must somehow enter into our theories. What is this structure, and how do we find it? Despite the complexity of data, in many cases models appear to derive their learning signal from a small set of sufficient statistics. What are these minimal data statistics, and how do they enter into a predictive theory of what the model learns? Are these statistics different for different models and at different stages of training? Can we describe the relevant structure in a dataset in terms of a model with free parameters found via an empirical fit? Open Direction 3: Does deep learning implicitly minimize some notion of functional complexity? Deep networks trained by conventional optimizers are widely believed to have some sort of bias towards learning simple functions. This idea has surfaced many times under different names (e.g. implicit regularization, maximum margin bias, simplicity bias, and spectral bias), but has only been characterized precisely in highly specific settings, and a general picture has not been found. Do deep neural networks broadly seek to minimize some precise notion of complexity among functions with low loss? If so, what is the appropriate notion of complexity — Kolmogorov, circuit, weight norm, or something else? In what settings or limits is this minimization exact, and when is it only approximate? Do the sparse features and circuitry studied by mechanistic interpretability naturally emerge as the solution to this minimization problem? Open Direction 4: How do we formally define the features learned by neural networks? Mechanistic interpretability seeks to identify and disentangle the features, circuits, and mechanisms learned by neural networks. Can these concepts be given precise mathematical definitions grounded in first principles? What formal structures naturally emerge from such a definition? Can we use these notions to evaluate and formalize central assumptions of mechanistic interpretability, including linear representability, locality, sparsity, and compositionality, as 22
discussed in Section 3? How do these ideas connect with the less semantically-meaningful — but more precise — rich vs. lazy picture of feature learning discussed in Section 2.2? Open Direction 5: Are finite neural networks properly understood as approximations to infinite limits? In Section 2.2, we articulated the Discretization Hypothesis, which states that finite neural networks are simply discretized approximations to infinite networks, analogous to how a spatiotemporal discretization is used to numerically approximate the solution to a differential equation. For network width, the limiting continuous object is the measure of neuron activity in hidden layers, while finite depth in a residual network can be viewed as a discretization of a neural SDE or ODE. Small step sizes can render stochastic optimization algorithms approximately equivalent to some kind of flow. In this view, increasing model size (and decreasing learning rate while commensurately increasing step count) serve essentially to improve model performance by decreasing discretization error, at the price of additional computation. Is this the right way to understand width, depth, learning rate, and other finite hyperparameters in deep learning? What does the limiting continuum system look like? Open Direction 6: Can we understand and eliminate all hyperparameters? In Sections 2.2 and 2.4, we outlined a research program in which hyperparameters are systematically analyzed, disentangled, and in some cases removed by taking appropriate limits. How far can this program go? Can we reach zero hyperparameters, or are some hyperparameters irreducible? If we eliminate all hyperparameters, what remains? Open Direction 7: Can we predict scaling law exponents a priori? As discussed in Section 2.3, large models exhibit robust power-law scaling of loss with respect to model size, data, and compute. The observed exponents are nontrivial: they do not appear to be simple fractions which might result from elementary dimensionality arguments. It is widely believed that these values are driven largely by structure latent in the dataset, but may also depend on details of the architecture and optimizer. While many explanations for scaling laws have been proposed, a decisive test of any such theory is its ability to predict these exponents quantitatively from first principles. At present, no framework can robustly do so across realistic settings. Can we develop a theory of scaling laws that both explains why power laws arise and predicts their exponents a priori? What measurements of the dataset, architecture, and optimization are required to do so? Open Direction 8: How does loss curvature interplay with architecture, features, and generalization? As discussed in Sections 2.3 and 2.4, a significant feature of deep learning optimization is that the optimizer implicitly regularizes the curvature (i.e. Hessian) along its trajectory, by steering towards regions of the loss landscape with lower curvature. While progress has been made on formalizing this effect using curvature-penalized gradient flows, it remains unclear how these curvature dynamics relate to other concerns in deep learning theory. Why does the curvature tend to rise in the absence of any such implicit regularization, and can this “progressive sharpening” be attributed to certain properties of the architecture or data distribution? How does the implicit curvature regularization affect the features that are learned? Why does it sometimes lead to improved generalization? Open Direction 9: What makes for a good optimizer in deep learning? It remains fundamentally unclear why some deep learning optimizers work better than others. Why do adaptive methods, such as Adam and Muon, consistently outperform simpler alternatives like SGD when training large language models? How does adaptive preconditioning in these optimizers interact with a network’s architecture and loss landscape to lead to faster, more stable training? Can we identify fundamental principles that explain the success of modern optimizers, predict when they will fail, and guide the design of new ones? Open Direction 10: In what sense do large models trained differently learn similar representations? In Section 2.5, we discussed evidence that large models trained from different random seeds — and sometimes even with different widths, architectures, data, or objectives — tend to learn similar internal representations. A precise version of this claim would be very powerful: understanding how representation learning is universal would give us confidence that theory developed for one model and setting transfers to many others. The central difficulty here is methodological: how do we assess “similarity”? There is no single way to compare high-dimensional representations — metrics based on kernel alignment, nearest-neighbors, model stitching, and more 23
compare different aspects of representation geometry. Which ones are stable across training regimes? What is the appropriate metric that quantifies this similarity? What is the largest range of experimental settings under which convergence is observed — what are the representation universality classes?
6
How to get involved in the development of learning mechanics
It is always difficult to start doing research in a new field. Consequently, we would like to make it as easy as possible for newcomers to get started. In this section, we extend a hand with some encouragement and advice. There is no specific academic background required to do useful work in this field. Well-regarded researchers in deep learning theory come from backgrounds in physics, mathematics, computer science, neuroscience, statistics, and more. Moreover, knowing another field well is useful, and established ideas from other fields can be applied to deep learning in some form or another, as the diversity of perspectives on deep learning theory attests (Section 3). Good things grow from cross-pollination. A firm grasp of undergraduate mathematics, a familiarity with deep learning, and a desire to learn are the only definite prerequisites.
6.1
Tenets for getting started
If you want to join this field, you are more than welcome. While there is no single correct way to craft theory, there are plenty of pitfalls that many of us encountered when starting out. To help avoid some of them, we have compiled a shortlist of guiding principles for doing research in this field. These tenets are not intended to maximize your number of citations in the short term, and following them may involve some swimming against the current of academia. Instead, they are intended to maximize your impact in the long term and your ability to integrate and contribute to the community. 1. Do experiments frequently. As discussed in Section 2.3, deep learning is a field where the cost of doing experiments is relatively low, with a fast turnaround time. Use that to your advantage! Experiments serve to check assumptions, inform theoretical models, reveal the limitations of a theory, peer beyond the cases that a theory covers, and surface interesting phenomena to study in the first place. Try to include experiments in every paper, and make them as simple and revealing as possible. 2. Simplicity and insight matter more than technical complexity. If you want to do work that is useful for others, they need to understand it, not merely be impressed by it. Take the time to simplify your findings, identify the underlying intuition, and check with simple experiments. A useful idea for thinking about a type of problem is generally more valuable than a difficult theorem or a solution to any particular problem, so emphasize these useful ideas when you present your work. This will make your results more accessible and easier for others to extend and apply. Your conference reviewers may disagree with this philosophy — the conference system tends to reward technical complexity and undervalue simplicity — but it will make your work more impactful in the long run. 3. Value scientific understanding over state-of-the-art performance. Applied deep learning is a field of engineering whose progress is measured by benchmarks. For scientists of deep learning, the game is different: your contribution is gauged by your contribution to collective understanding. It is easy to feel pressure to tack on some engineering benefit in a scientific paper to make the paper seem more relevant and timely, but doing so usually dilutes the paper’s scientific contribution without really affecting practice. Of course, fundamental science should eventually improve state-of-the-art performance, and when this naturally falls out of the science, it is a powerful way to demonstrate what has been understood.15 When it does not, though, there is no need to force it: set benchmarks aside for the time being and seek understanding on its own terms. 4. Don’t try to do it alone. Deep learning is a field with a lot of history and many known results, and guidance from a live human will help you. You can get pretty far from reading material online and talking to AI, but it doesn’t replace human collaboration and mentorship. You should seek out other people interested in this area, ask for their feedback, and ask to work with them. A weak corollary to this is: when one exists, watch the talk version of a paper in addition to reading it. A great deal more of the nuance of a project is conveyed through in a live presentation. The Physics Meets ML and Physics of Learning series are good places to look for talks on what we here call learning mechanics. 15 For a good example of this, see Yang et al. [2022].
24
5. Try a few different problems before going deep into one. Deep learning theory has so many open questions that you probably won’t know where to start. That’s okay — just jump in, and feel free to change problems a few times early on.16 Knowing multiple areas is essential to having high-level ideas anyways, and working in an area is the best way to learn it. 6. Invest in fundamental tools and techniques. Compared to more established fields of science, mathematics, and engineering, deep learning is very young. These other fields have identified deep ideas and developed powerful tools applicable to broad classes of problem. Learning these fundamental tools comes in handy when similar problems arise in deep learning. For example, statistical physics and random matrix theory have powerful tools for thinking about high-dimensional interacting systems, and these tools are of great use in learning mechanics when one takes infinite limits. Many of the central ideas of classical optimization theory make regular appearances in the study of neural network optimization. Other concepts from statistical signal processing, such as wavelet decompositions, graphical models, and information theory, have been comparatively less leveraged so far, but may prove to be useful and complementary tools of learning mechanics. We will put as much useful introductory material as we can on learningmechanics.pub, and we encourage discussion in the comments there. We also encourage taking a crack at the open directions in Section 5. Work hard, have fun, and best of luck — we hope to see a great deal more fundamental science of deep learning in the next few years!
Acknowledgements We are grateful for feedback on this paper from many people from several overlapping groups: researchers working on the theory of machine learning, researchers working on AI safety and mechanistic interpretability, practitioners and applied deep learning scientists, neuroscientists, and physicists. This includes Alberto Bietti, Alex Infanger, Alex Williams, Amil Dravid, Anthony Thomas, Avrajit Ghosh, Bin Yu, Bruno Loureiro, Chandan Singh, Clémentine Dominé, David Berman, David Klindt, Denny Wu, Ev Gunter, Honam Wong, Itay Lavie, Jacob Yates, Jacob Zavatone-Veth, Jeff Gore, Jesse Hoogland, Jiechao Feng, Jingfeng Wu, Kaden Tro, Lauren Greenspan, Lenka Zdeborová, Lily Stelling, Lukas Bongartz, Nina Miolane, Noa Rubin, Peter Bartlett, Raymond Fan, Samyak Jain, Soufiane Hayou, Sultan Daniels, Wanyu Lei, Yasaman Bahri, and Zohar Ringel.
16 Most authors of this paper did this, as you can see from our respective research records!
25
References Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pages 4782–4887. PMLR, 2022. Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428–446, 2020. Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Spin-glass models of neural networks. Physical Review A, 32(2):1007, 1985. Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, and Sarah Schwettmann. Language model circuits are sparse in the neuron basis, 2025. Blog post: https://transluce.org/neuron-circuits. Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International conference on machine learning, pages 244–253. PMLR, 2018. Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks, 2019a. Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019b. Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019c. Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2021. Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592, 2024. Alexander Atanasov, Alexandru Meterez, James B Simon, and Cengiz Pehlevan. The optimization landscape of sgd across the feature learning strength. In The Thirteenth International Conference on Learning Representations, 2025. Benjamin Aubin, Antoine Maillard, Florent Krzakala, Nicolas Macris, Lenka Zdeborová, et al. The committee machine: Computational to statistical gaps in learning a two-layers neural network. Advances in Neural Information Processing Systems, 31, 2018. Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35:37932–37946, 2022. Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual review of condensed matter physics, 11(1):501–528, 2020. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27), June 2024. ISSN 1091-6490. doi: 10.1073/pnas.2311878121. Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53–58, 1989. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225–236, 2021. Richard Baraniuk, David Donoho, and Matan Gavish. The science of deep learning. Proceedings of the National Academy of Sciences, 117(48):30029–30032, 2020. Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences, 116(12): 5451–5460, 2019.
26
Jean Barbier, Francesco Camilli, Minh-Toan Nguyen, Mauro Pastore, and Rudy Skerk. Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation. arXiv preprint arXiv:2510.24616, 2025. Maissam Barkeshli, Alberto Alfarano, and Andrey Gromov. On the origin of neural scaling laws: from random graphs to natural language. arXiv preprint arXiv:2601.10684, 2026. David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009.11162, 2020. Melih Barsbey, Lucas Prieto, Stefanos Zafeiriou, and Tolga Birdal. Large learning rates simultaneously achieve robustness to spurious correlations and compressibility. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2055–2066, 2025. Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30: 87–201, 2021. Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International conference on machine learning, pages 685–694. PMLR, 2020. Jan P Bauer, Kirsten Fischer, Moritz Helias, and Agostina Palmigiano. A unified theory of feature learning in rnns and dnns. arXiv preprint arXiv:2602.15593, 2026. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019. Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling. Advances in neural information processing systems, 35:25349–25362, 2022. Gérard Ben Arous, Murat A Erdogdu, Nuri Mert Vural, and Denny Wu. Learning quadratic neural networks in high dimensions: Sgd dynamics and scaling laws. arXiv preprint arXiv:2508.03688, 2025. Silvia Bernardi, Marcus K Benna, Mattia Rigotti, Jérôme Munuera, Stefano Fusi, and C Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183(4):954–967, 2020. Jan Betley, Niels Warncke, Anna Sztyber-Betley, Daniel Tan, Xuchan Bao, Martı́n Soto, Megha Srivastava, Nathan Labenz, and Owain Evans. Training large language models on narrow tasks can lead to broad misalignment. Nature, 649(8097):584–589, 2026. Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. Advances in neural information processing systems, 35:9768–9783, 2022. Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36:1560–1588, 2023. Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory, pages 483–513. PMLR, 2020. Enric Boix-Adsera, Neil Mallinar, James B. Simon, and Mikhail Belkin. The features at convergence theorem: a first-principles alternative to the neural feature ansatz for how networks learn representations, 2025. Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, and Lenka Zdeborová. Singlehead attention in high dimensions: A theory of generalization, weights spectra, and scaling laws. In Workshop on Scientific Methods for Understanding Deep Learning, 2025. Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35:32240–32256, 2022. Blake Bordelon and Cengiz Pehlevan. Dynamics of finite width kernel and prediction fluctuations in mean field neural networks. Advances in Neural Information Processing Systems, 36:9707–9750, 2023. Blake Bordelon and Cengiz Pehlevan. Deep linear network training dynamics from random initialization: Data, width, depth, and hyperparameter transfer. arXiv preprint arXiv:2502.02531, 2025.
27
Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. arXiv preprint arXiv:2309.16620, 2023. Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024a. Blake Bordelon, Hamza Chaudhry, and Cengiz Pehlevan. Infinite limits of multi-head transformer dynamics. Advances in Neural Information Processing Systems, 37:35824–35878, 2024b. Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. In The Thirteenth International Conference on Learning Representations, 2025. Jeffrey S Bowers, Gaurav Malhotra, Marin Dujmović, Milton Llera Montero, Christian Tsvetkov, Valerio Biscione, Guillermo Puebla, Federico Adolfi, John E Hummel, Rachel F Heaton, et al. Deep problems with neural network models of human vision. Behavioral and Brain Sciences, 46:e385, 2023. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemanticfeatures/index.html. Francesco Cagnetta, Leonardo Petrini, Umberto M Tomasini, Alessandro Favero, and Matthieu Wyart. How deep neural networks learn compositional data: The random hierarchy model. Physical Review X, 14(3):031001, 2024. Francesco Cagnetta, Hyunmo Kang, and Matthieu Wyart. Learning curves theory for hierarchically compositional data with power-law distributed features. arXiv preprint arXiv:2505.07067, 2025. Francesco Cagnetta, Allan Raventós, Surya Ganguli, and Matthieu Wyart. Deriving neural scaling laws from the statistics of natural language. arXiv preprint arXiv:2602.07488, 2026. Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):2914, 2021. Andrea Caponnetto and Ernesto de Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007. Albert Catalan-Tatjer, Niccolò Ajroldi, and Jonas Geiping. Training dynamics impact post-training quantization robustness. arXiv preprint arXiv:2510.06213, 2025. Louis-Pierre Chaintron, Lénaı̈c Chizat, and Javier Maas. Resnets of all shapes and sizes: Convergence of training dynamics in the large-scale limit. arXiv preprint arXiv:2603.18168, 2026. Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1–10. IEEE, 2018. Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. arXiv preprint arXiv:2309.07311, 2023. Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. Advances in Neural Information Processing Systems, 36, 2024. Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. Lénaı̈c Chizat. The hidden width of deep resnets: Tight error bounds and phase diagrams. arXiv preprint arXiv:2509.10167, 2025. Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018.
28
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019. SueYeon Chung and Larry F Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology, 70:137–144, 2021. David G Clark, Blake Bordelon, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Structure, disorder, and dynamics in task-trained recurrent neural circuits. bioRxiv, pages 2026–03, 2026. Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021a. Jeremy M. Cohen, Alex Damian, Ameet Talwalkar, J. Zico Kolter, and Jason D. Lee. Understanding optimization in deep learning with central flows, 2025. Omry Cohen, Or Malka, and Zohar Ringel. Learning curves for overparametrized deep neural networks: A field theory perspective. Physical Review Research, 3(2):023034, 2021b. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36: 16318–16352, 2023. Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408.10920, 2024. Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131–10143, 2021. Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology, 4(3):035033, 2023. Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue M Lu, Lenka Zdeborová, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. arXiv preprint arXiv:2402.04980, 2024. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449–27461, 2021. Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594, 2022a. Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory, pages 5413–5452. PMLR, 2022b. Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. arXiv preprint arXiv:2305.18270, 2023. Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889–904, 1995. Leonardo Defilippis, Yizhou Xu, Julius Girardin, Emanuele Troiani, Vittorio Erba, Lenka Zdeborová, Bruno Loureiro, and Florent Krzakala. Scaling laws and spectra of shallow neural networks in the feature learning regime. arXiv preprint arXiv:2509.24882, 2025. Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: Completep enables compute-efficient deep transformers. arXiv preprint arXiv:2505.01618, 2025.
29
Clémentine Carla Juliette Dominé, Nicolas Anguita, Alexandra Maria Proca, Lukas Braun, Daniel Kunin, Pedro AM Mediano, and Andrew M Saxe. From lazy to rich: Exact learning dynamics in deep linear networks. In The Thirteenth International Conference on Learning Representations, 2025. David Donoho. Data science at the singularity. Harvard Data Science Review, 6(1), 2024. Amil Dravid, Yossi Gandelsman, Alexei A Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1934–1943, 2023. Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. arXiv preprint arXiv:2405.14860, 2024. Vittorio Erba, Emanuele Troiani, Lenka Zdeborová, and Florent Krzakala. The nuclear route: Sharp asymptotics of erm in overparameterized quadratic networks. arXiv preprint arXiv:2505.17958, 2025. Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s) gd over diagonal linear networks: Implicit bias, large stepsizes and edge of stability. Advances in Neural Information Processing Systems, 36:29406–29448, 2023. Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 11(2):127–138, 2010. Kenji Fukumizu. Effect of batch learning in multilayer neural networks. Gen, 1(04):1E–03, 1998. Yarin Gal. The science of deep learning. https://www.cs.ox.ac.uk/people/yarin.gal/website/ blog5058.html, 2015. Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A: Mathematical and general, 21(1):257–270, 1988. Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the laplace and neural tangent kernels. Advances in Neural Information Processing Systems, 33:1451–1461, 2020. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, et al. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83):1–64, 2025. Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020. Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33:14820–14830, 2020. Nikhil Ghosh, Denny Wu, and Alberto Bietti. Understanding the mechanisms of fast hyperparameter transfer. arXiv preprint arXiv:2512.22768, 2025. Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. Advances in Neural Information Processing Systems, 32, 2019. Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely. The implicit bias of depth: How incremental learning drives generalization. arXiv preprint arXiv:1909.12051, 2019.
30
Margalit Glasgow, Denny Wu, and Joan Bruna. Propagation of chaos in one-hidden-layer neural networks beyond logarithmic time. arXiv preprint arXiv:2504.13110, 2025. Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019. Pulkit Gopalani, Ekdeep S Lubana, and Wei Hu. Abrupt learning in transformers: A case study on matrix completion. Advances in Neural Information Processing Systems, 37:55053–55085, 2024. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. Lauren Greenspan, David Berman, Aryeh Brill, Ro Jefferson, Artemy Kolchinsky, Jennifer Lin, Andrew Mack, Anindita Maiti, Fernando E Rosas, Alexander Stapleton, et al. Towards worst-case guarantees with scale-aware interpretability. arXiv preprint arXiv:2602.05184, 2026. Fabian Gröger, Shuo Wen, and Maria Brbić. Revisiting the platonic representation hypothesis: An aristotelian view. arXiv preprint arXiv:2602.14486, 2026. Andrey Gromov. Grokking modular arithmetic. arXiv preprint arXiv:2301.02679, 2023. Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018. Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, and Joshua Batson. When models manipulate manifolds: The geometry of a counting task. Transformer Circuits Thread, 2025. URL https: //transformer-circuits.pub/2025/linebreaks/index.html. Florentin Guth, Brice Ménard, Gaspar Rochette, and Stéphane Mallat. A rainbow in deep network black boxes. Journal of Machine Learning Research, 25(350):1–59, 2024. Boris Hanin and Tianze Jiang. Global universality of singular values in products of many large random matrices. arXiv preprint arXiv:2503.07872, 2025. Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. arXiv preprint arXiv:1909.05989, 2019. Sarah E Harvey, Brett W Larsen, and Alex H Williams. Duality of bures and shape distances with implications for comparing neural representations. In Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, pages 11–26. PMLR, 2024. Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949–986, 2022. Soufiane Hayou. A proof of learning rate transfer under mu p. arXiv preprint arXiv:2511.01734, 2025. Soufiane Hayou and Greg Yang. Width and depth limits commute in residual networks. In International Conference on Machine Learning, pages 12700–12723. PMLR, 2023. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, and Stan van Wingerden. Towards developmental interpretability. LessWrong, July 2023. URL https://www.lesswrong.com/posts/ TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability. Accessed: 2026-02-03.
31
Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape degeneracy and stagewise development in transformers. Transactions on Machine Learning Research, 2025. John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982. Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, pages 4376–4386. PMLR, 2020. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Forty-first International Conference on Machine Learning, 2024. Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics, 76(1):1–33, 2024. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018. Arthur Jacot, Berfin Şimşek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Kernel alignment risk estimator: Risk prediction from training data. Advances in neural information processing systems, 33:15568–15578, 2020. Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933, 2021. Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research, 18(223):1–42, 2018. Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017. Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09572, 2020. Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, and Boris Hanin. Hyperparameter transfer with mixture-of-expert layers. arXiv preprint arXiv:2601.20205, 2026. Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint arXiv:2403.03867, 2024. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/ posts/muon/. Shruti Joshi, Aaron Mueller, David Klindt, Wieland Brendel, Patrik Reizinger, and Dhanya Sridhar. Causality is key for interpretability claims to generalise. arXiv preprint arXiv:2602.16698, 2026. Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024. Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019. Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. In Forty-second International Conference on Machine Learning, 2025. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
32
Dhruva Karkada, Joseph Turnbull, Yuxi Liu, and James B Simon. Predicting kernel regression learning curves from only raw data statistics. arXiv preprint arXiv:2510.14878, 2025. Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri. Symmetry in language statistics shapes the geometry of model representations. arXiv preprint arXiv:2602.15029, 2026. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016. Meenakshi Khosla, Alex H Williams, Josh McDermott, and Nancy Kanwisher. Privileged representational axes in biological and artificial neural networks. bioRxiv, pages 2024–06, 2024. Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 57(9):1–52, 2025. David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.10930, 2020. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMlR, 2019. Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008. Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021. Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems, 37:81157–81203, 2024. Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R DeWeese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A theory of feature learning in two-layer neural networks. arXiv preprint arXiv:2506.06489, 2025. Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint arXiv:1809.10374, 2018. Pat Langley. Machine learning as an experimental science. Machine Learning, 3(1):5–8, 1988. Itay Lavie, Guy Gur-Ari, and Zohar Ringel. Towards understanding inductive bias in transformers: A view from infinity. arXiv preprint arXiv:2402.05173, 2024. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791. Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165, 2017. Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019. Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991–999, 2015. Michael Lepori, Thomas Serre, and Ellie Pavlick. Break it down: Evidence for structural compositionality in neural networks. Advances in Neural Information Processing Systems, 36:42623–42660, 2023.
33
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. Mufan Li, Mihai Nica, and Dan Roy. The neural covariance sde: Shaped infinite depth-and-width networks at initialization. Advances in Neural Information Processing Systems, 35:10795–10808, 2022. Qianxiao Li, Cheng Tai, et al. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40):1–47, 2019. Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Physical Review X, 11(3):031059, 2021. Wentian Li. Zipf’s law everywhere. Glottometrics, 5(2002):14–21, 2002. Xinghan Li, Haodong Wen, and Kaifeng Lyu. Adam reduces a unique form of sharpness: Theoretical insights near the minimizer manifold. arXiv preprint arXiv:2511.02773, 2025. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Feature Extraction: Modern Questions and Challenges, pages 196–212. PMLR, 2015. Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2021a. Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes), 2021b. Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?–a mathematical framework. arXiv preprint arXiv:2110.06914, 2021c. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/ 2025/attribution-graphs/biology.html. Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33:15954–15964, 2020. Yizhou Liu, Ziming Liu, and Jeff Gore. Superposition yields robust neural scaling, 2025. Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022. Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems, 34:18137–18151, 2021. Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020. Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning, pages 3325–3334. PMLR, 2018. Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprint arXiv:1803.08367, 2018.
34
Antoine Maillard, Emanuele Troiani, Simon Martin, Lenka Zdeborová, and Florent Krzakala. Bayes-optimal learning of an extensive-width neural network from quadratically many samples. Advances in Neural Information Processing Systems, 37:82085–82132, 2024. Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń, Maciej Pióro, Jan Ludziejewski, Maciej Stefaniak, Michał Krutul, Sebastian Jaszczur, Marek Cygan, Kamil Adamczewski, and Jakub Krajewski. µ-parametrization for mixture of experts. arXiv preprint arXiv:2508.09752, 2025. Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 7697–7711. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paperfiles/paper/2022/file/ 32ac710102f0620d0f28d5d05a44fe08-Paper-Conference.pdf. Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610–23641. PMLR, 2023. Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999. Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1–35, 2017. Giovanni Luca Marchetti, Daniel Kunin, Adele Myers, Francisco Acosta, and Nina Miolane. Sequential group composition: A window into the mechanics of deep learning. arXiv preprint arXiv:2602.03655, 2026. Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: Conservation laws for gradient flows, 2024a. Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Keep the momentum: Conservation laws beyond euclidean gradient flows, 2024b. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023. David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. Simon Martin, Francis Bach, and Giulio Biroli. On the impact of overparameterization on the training of a shallow neural network in high dimensions. In International Conference on Artificial Intelligence and Statistics, pages 3655–3663. PMLR, 2024. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. Advances in neural information processing systems, 29, 2016. Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665–E7671, 2018. Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimensionfree bounds and kernel limit. In Conference on learning theory, pages 2388–2464. PMLR, 2019. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36:28699–28722, 2023. Eric J Michaud. A physics of systems that learn, 2024. physics-of-learning.pdf.
35
URL https://ericjmichaud.com/
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540–9550, 2020. Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751, 2013. Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. arXiv preprint arXiv:2310.07891, 2023. Andrea Montanari and Zihao Wang. Phase transitions for feature learning in neural networks. arXiv preprint arXiv:2602.01434, 2026. Depen Morwani, Benjamin L Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. arXiv preprint arXiv:2311.07568, 2023. Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022. Yoonsoo Nam, Seok Hyeong Lee, Clementine CJ Domine, Yeachan Park, Charles London, Wonyl Choi, Niclas Goring, and Seungjai Lee. Position: Solve layerwise linear models first to understand neural dynamical phenomena (neural collapse, emergence, lazy/rich regime, and grokking). arXiv preprint arXiv:2502.21009, 2025. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023a. Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16–30, 2023b. Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29–53. Springer, 1996. Eshaan Nichani, Alex Damian, and Jason D Lee. How transformers learn causal structure with gradient descent. In Proceedings of the 41st International Conference on Machine Learning, pages 38018–38070, 2024. Eshaan Nichani, Jason D Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. In The Thirteenth International Conference on Learning Representations, 2025. Matthew Niedoba, Berend Zwartsenberg, Kevin Patrick Murphy, and Frank Wood. Towards a mechanistic explanation of diffusion model generalization. In Forty-second International Conference on Machine Learning, 2025. Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 36:54250–54281, 2023. Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvieto. Super consistency of neural network landscapes and learning rate transfer. Advances in Neural Information Processing Systems, 37:102696–102743, 2024. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020. Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.
36
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020. doi: 10.1073/pnas. 2015509117. Core F Park, Maya Okawa, Andrew Lee, Hidenori Tanaka, and Ekdeep S Lubana. Emergence of hidden capabilities: Exploring learning dynamics in concept space. Advances in Neural Information Processing Systems, 37:84698–84729, 2024. Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023a. Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023b. Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. Advances in Neural Information Processing Systems, 36:7475–7505, 2023. Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34:29218–29230, 2021. Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112–1130, 2014. Loucas Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31, 2018. Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems (NeurIPS), pages 3360–3368. Curran Associates, Inc., 2016. Lucas Prieto, Edward Stevinson, Melih Barsbey, Tolga Birdal, and Pedro AM Mediano. Correlations in the data lead to semantically rich feature geometry under superposition. In Mechanistic Interpretability Workshop at NeurIPS 2025, 2025. Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science, 383(6690):1461–1467, 2024. doi: 10.1126/science.adi5639. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017. Ali Rahimi. Let’s take machine learning from alchemy to electricity: Test-of-time award presentation. Test-of-Time Award talk, 2017. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007. Rahul Ramesh, Ekdeep Singh Lubana, Mikail Khona, Robert P Dick, and Hidenori Tanaka. Compositional capabilities of autoregressive transformers: A study on synthetic, interpretable tasks. arXiv preprint arXiv:2311.12997, 2023. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. arXiv preprint arXiv:2312.03002, 2023. Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-entropy is all you need to invert the data generating process. arXiv preprint arXiv:2410.21869, 2024. Patrik Reizinger, Randall Balestriero, David Klindt, and Wieland Brendel. Position: An empirically grounded identifiability theory will accelerate self-supervised learning research. arXiv preprint arXiv:2504.13101, 2025. Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations, 2025.
37
Yunwei Ren, Eshaan Nichani, Denny Wu, and Jason D. Lee. Emergence and scaling laws in sgd learning of shallow neural networks, 2025. Zohar Ringel, Noa Rubin, Edo Mor, Moritz Helias, and Inbar Seroussi. Applications of statistical field theory in deep learning. arXiv preprint arXiv:2502.18553, 2025. Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2022. Grant Rotskoff and Eric Vanden-Eijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. Advances in neural information processing systems, 31, 2018. Noa Rubin, Inbar Seroussi, and Zohar Ringel. Grokking as a first order phase transition in two layer networks. arXiv preprint arXiv:2310.03789, 2023. Noa Rubin, Orit Davidovich, and Zohar Ringel. Mitigating the curse of detail: Scaling arguments for feature learning and sample complexity. arXiv preprint arXiv:2512.04165, 2025a. Noa Rubin, Kirsten Fischer, Javed Lindner, David Dahmen, Inbar Seroussi, Zohar Ringel, Michael Krämer, and Moritz Helias. From kernels to features: A multi-scale adaptive theory of feature learning. arXiv preprint arXiv:2502.03210, 2025b. David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337–4340, 1995. Naomi Saphra. Interpretability creationism. Blog post, June 2022. URL https://nsaphra.net/post/ creationism/. Accessed: 2026-02-03. Naomi Saphra and Sarah Wiegreffe. Mechanistic? In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 480–498, 2024. Andrew Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the International Conference on Learning Represenatations 2014. International Conference on Learning Represenatations 2014, 2014. Andrew Michael Saxe. Deep linear neural networks: A theory of learning in the brain and mind. Stanford University, 2015. Simon Schug, Seijin Kobayashi, Yassir Akram, Maciej Wołczyk, Alexandra Proca, Johannes Von Oswald, Razvan Pascanu, Joao Sacramento, and Angelika Steger. Discovering modular solutions that generalize compositionally. arXiv preprint arXiv:2312.15001, 2023. John Schulman and Thinking Machines Lab. Lora without regret. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250929. https://thinkingmachines.ai/blog/lora/. Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences, 122(1):e2408799121, 2025. Kai Segadlo, Bastian Epping, Alexander Van Meegen, David Dahmen, Michael Krämer, and Moritz Helias. Unified field theoretical approach to deep and recurrent neuronal networks. Journal of Statistical Mechanics: Theory and Experiment, 2022(10):103401, 2022. Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns. Nature Communications, 14(1):908, 2023. Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8):6056, 1992. Adam Shai, Lucas Teixeira, Alexander Oldenziel, Sarah Marzen, and Paul Riechers. Transformers represent belief state geometry in their residual stream. Advances in Neural Information Processing Systems, 37:75012–75034, 2024.
38
Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20 (112):1–49, 2019. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496, 2025. Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1–34, 2022. Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017. James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R DeWeese. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. Transactions on Machine Learning Research, 2023a. James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pages 31852–31876. PMLR, 2023b. Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021. Samuel L Smith, Andrew Brock, Leonard Berrada, and Soham De. Convnets match vision transformers at scale. arXiv preprint arXiv:2310.16764, 2023. Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159–216, 1990. Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1–57, 2018a. Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878, 2018b. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Iris Groen, Jascha Achterberg, et al. Getting aligned on representational alignment. arXiv preprint arXiv:2310.13018, 2023. Hidenori Tanaka and Daniel Kunin. Noether’s learning dynamics: Role of symmetry breaking in neural networks. Advances in Neural Information Processing Systems, 34:25646–25660, 2021. Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pages 10153–10161. PMLR, 2021. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/ scaling-monosemanticity/index.html. Simon Thorpe. Local vs. distributed coding. Intellectica, 8(2):3–40, 1989. Rodrigo Veiga, Ludovic Stephan, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks. Advances in Neural Information Processing Systems, 35:23244–23255, 2022.
39
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding generalization in deep learning. arXiv preprint arXiv:2206.10012, 2022. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. Zihan Wang and Arthur Jacot. Implicit bias of sgd in l 2-regularized linear dnns: One-way jumps from high to low rank. In The Twelfth International Conference on Learning Representations, 2024. Sumio Watanabe. Algebraic geometry and statistical learning theory, volume 25. Cambridge university press, 2009. Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In International conference on machine learning, pages 23549–23588. PMLR, 2022. Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-aware minimization minimize sharpness? arXiv preprint arXiv:2211.05729, 2022. Alex H Williams. Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis. In Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, pages 10–23. PMLR, 2024. Andrew Gordon Wilson. Position: Deep learning is not so mysterious or different. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. David H Wolpert. The lack of a priori distinctions between learning algorithms. Neural computation, 8(7):1341–1390, 1996. Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635–3673. PMLR, 2020. Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in neural information processing systems, 30, 2017. Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performanceoptimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619–8624, 2014. Adam X Yang, Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. A theory of representation learning gives a deep generalisation of kernel methods. In International Conference on Machine Learning, pages 39380–39415. PMLR, 2023a. Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 11727–11737. PMLR, 2021. Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-width limit. arXiv preprint arXiv:2308.01814, 2023. Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022. Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244, 2023b. Geonhui Yoo, Minhak Song, and Chulhee Yun. Understanding sharpness dynamics in nn training with a minimalist example: The effects of dataset difficulty, depth, stochasticity, and more. In International Conference on Machine Learning, pages 72574–72617. PMLR, 2025. Jacob Zavatone-Veth, Abdulkadir Canatar, Ben Ruben, and Cengiz Pehlevan. Asymptotics of representation learning in finite bayesian neural networks. Advances in neural information processing systems, 34:24765–24777, 2021.
40
Jacob A Zavatone-Veth, Blake Bordelon, and Cengiz Pehlevan. Summary statistics of learning link changing neural representations to behavior. Frontiers in Neural Circuits, 19:1618351, 2025. Lenka Zdeborová. Understanding deep learning is also a job for physicists. Nature Physics, 16(6):602–604, 2020. Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms. Advances in Physics, 65(5):453–552, 2016. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818–833. Springer, 2014. Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. In Forty-first International Conference on Machine Learning, 2024. Yedi Zhang, Aaditya K Singh, Peter E Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention. arXiv preprint arXiv:2501.16265, 2025. Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820–29834, 2021. Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International conference on machine learning, pages 12979–12990. PMLR, 2021. Liu Ziyin and Isaac Chuang. Proof of a perfect platonic representation hypothesis. arXiv preprint arXiv:2507.01098, 2025. Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of a deep linear network. Advances in Neural Information Processing Systems, 35:24446–24458, 2022. Liu Ziyin, Isaac Chuang, Tomer Galanti, and Tomaso Poggio. Formation of representations in neural networks. arXiv preprint arXiv:2410.03006, 2024.
41