Variational Bounds for Perceptron Learning from Structured Data

Variational Bounds for Perceptron Learning from Structured Data Francesco Camilli∗, Pierluigi Contucci, Federica Gerace, and Emanuele Mingione Department of Mathematics, Alma Mater Studiorum – Università di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy

arXiv:2608.04882v1 [cs.LG] 5 Aug 2026

August 6, 2026

Abstract We introduce a variational approach to a finite-temperature continuous-spin perceptron trained on a Gaussian mixture. The model allows for a broad class of concave utilities and log-concave separable prior measures on the spins. By combining the interpolation method with log-concavity and concentration estimates, we derive lower and upper minimax variational bounds for the limiting quenched pressure. Remarkably, the two bounds differ only in the order of optimization of two variational parameters, while all remaining extrema are controlled by the concave–convex structure of the variational potential. Whenever the two optimizations commute, the two bounds match and identify the solution of the model. The same potential yields the fixed-point equations as stationarity conditions and provides a unified route to the computation of the ground-state energy, training loss, and generalization error.

1

Introduction

The perceptron is one of the earliest models of artificial learning and remains a fundamental testing ground for the mathematical theory of high-dimensional classification. Introduced by Rosenblatt as a model of pattern recognition [1], it was soon connected to the geometry of linear separability [2]. Its statistical-mechanics formulation, initiated by Gardner and Derrida [3, 4], revealed that even this elementary neural architecture displays nontrivial collective behavior when the number of patterns and the number of weights grow proportionally. The perceptron has since become a canonical model at the interface of spin glasses, statistics, and machine learning, and has played a central role in the statistical-mechanical theory of learning from examples [5, 6, 7] This proportional regime has recently attracted renewed attention in high-dimensional statistics. Precise asymptotic results are now available for logistic regression, regularized M -estimators, generalized linear models, and convex empirical-risk minimization [8, 9, ∗

Corresponding author: francesco.camilli2@unibo.it

1

10, 11, 12]. Related AMP-based results characterize the zero-temperature asymptotics of convex teacher–student generalized linear models with rotationally invariant design matrices, under suitable uniqueness and concentration assumptions [13]. For Gaussianmixture data, in particular, the roles of the loss, regularization, signal strength, and sampling ratio have been characterized sharply [14, 15, 16, 17]. Related works have studied universality beyond independent Gaussian covariates and the extent to which Gaussian mixtures describe learning curves on structured or real data [18, 19], including estimators obtained by sampling from the associated finite-temperature Gibbs measure [20]. These results illustrate the usefulness of simple random-data models for isolating phenomena that are difficult to detect through worst-case generalization bounds. From the spin-glass perspective, the rigorous analysis of the perceptron was developed through the cavity method. The cavity formulation of Gardner’s computation was developed by Mézard, who also derived the associated TAP equations [21]. Talagrand proved the Gardner–Derrida formula at sufficiently high temperature for a broad class of binary perceptron models [22] and for the Gaussian perceptron, he established overlap concentration in the high-temperature, low-density regime [23]. Shcherbina and Tirozzi proved the Gardner formula for a continuous-spin model [24]. These works obtain sharp thermodynamic results and identify the relevant scalar self-consistency equations. Complementary approaches based on Gordon’s comparison theorem and the convex Gaussian min–max theorem provide scalar descriptions of several convex learning problems, especially at zero temperature [9, 11, 16]. However, a direct finite-temperature variational formulation was not available in the present non-Bayes-optimal continuous-spin setting, with a general separable log-concave prior spin measure. For discrete spins, recent progress includes a new proof of the Gardner formula for Ising perceptrons at small densities, based on moment methods conditioned on approximate-message-passing iterates [25]. In a closely related direction, Sáenz and Sur characterize finite-dimensional posterior marginals in high-dimensional generalized linear models, proving through leave-one-out arguments that they converge to Gaussian tilts of the prior [26]. In this paper we consider a perceptron trained on a mixture of two Gaussian clouds whose centers are aligned with a random direction. The weights are distributed according to a Gibbs measure determined by a concave utility, a log-concave single-site prior measure. It can thus be thought as a mismatched inference problem, where the inferential model, the perceptron, does not match the data generating process, a Gaussian mixture. Nevertheless, the perceptron manages to reconstruct the hyperparameters of the generating process, thus solving an inverse problem. In this work we study the thermodynamic properties, specifically the limit of the log partition function, or quenched pressure, associated with the Gibbs measure of this inverse problem. We derive nested minimax variational bounds for the limiting quenched pressure, both at positive and zero temperature. In both cases, the two bounds differ only in the order of the min and the max computation. All the remaining extrema can be exchanged through the concave–convex structure of the potential. Whenever the two outer optimizations commute, the thermodynamic limit therefore exists and is given by a single variational principle. The variational nature of the result is a central feature of our approach. The usual fixed-point equations are recovered as stationarity conditions of one scalar potential, rather than introduced as a system of self-consistency relations. The potential retains global information: it provides a rule for comparing different stationary points and places the pressure, the ground-state energy, the training loss, and the generalization error within 2

the same framework. It also transfers the possible obstruction to an exact formula from uniqueness of the fixed-point system to the exchangeability of two explicitly identified optimizations. In the Gaussian-prior and random-label limit, our equations reduce to those obtained by the classical cavity analysis (see e.g. [27]). Our proof uses an evolution of Guerra and Toninelli’s interpolation technique [28, 29], called adaptive interpolation, introduced by Barbier and Macris [30, 31]. The method is very effective for Bayes-optimal inference problems, where Nishimori identities [32] and correlation inqualities [33, 34] provide order parameter concentrations [35]. Here these tools are unavailable. The argument instead relies on log-concavity, Brascamp–Lieb estimates [36], and adaptive choices of the interpolation path. Related results establish multioverlap concentration and strong replica symmetry for broad classes of disordered log-concave Gibbs measures [37]. This continues a line of work showing that adaptive interpolation can also be useful outside the Bayes-optimal setting [38, 39]. We mention that the problem lacks the usual convexity à la Guerra, that in many cases (e.g. [31] and [40] as examples) complements the standard thermodynamic convexity of generating functionals leading to matching bounds. This complementary behavior is in some sense recovered through log-concavity, that imposes another type of correlation inequalities, implying in turn new concavity properties on the log partition function. Besides providing a new route to the perceptron free energy, this formulation may offer a useful starting point for more structured architectures. Extending cavity computations from a single perceptron to multilayer systems requires controlling several coupled families of order parameters and cavity removals. A variational interpolation, in contrast, may allow these parameters to be organized directly within a global potential. Recent progress on the information-theoretic analysis of fully trained multilayer networks shows that such models are becoming accessible in controlled high-dimensional regimes [41, 42]. We leave the discussion of this perspective to the conclusions. The paper is organized as follows. In Section 2, we define the model and state the variational bounds, the stationarity equations, and the formulas for the training loss, generalization error, and ground-state energy. Section 3 contains the adaptive interpolation argument, the convexity analysis, and the zero-temperature and perturbative limits. In Section 4, we investigate the matching of the variational bounds for representative loss functions. The appendices collect the ODE, moment, log-concavity, and concentration estimates used in the proofs.

2

Definitions and main results

Let N, M be two positive integers. We introduce the patterns, or inputs, as a set of M i.i.d. random N -dimensional vectors, that can be grouped in the M × N matrix G = (gµ )µ≤M = (gµ,i )i≤N µ≤M .

(2.1)

The law of the inputs is determined by the following relation λ gµ = √ yµ θ + Zµ N

(2.2)

where λ ≥ 0, yµ ∼ (δ−1 + δ1 )/2, Zµ ∼ N (0, 1N ) and θ ∈ RN , with its components R R i.i.d. θi ∼ Pθ with Pθ satisfying Poincarè inequality ( f 2 dPθ ≤ cθ (f ′ )2 dPθ for a ceneterd i.i.d.

i.i.d.

3

function f , with Poincaré constant cθ ) with unit second moment. The variable yµ is called the label of the µ-th input, while θ is called centroid. We denote the set of input-label couples as D = {(gµ , yµ )µ≤M }. Equation (2.2) defines a Gaussian mixture, with two isotropic Gaussian clouds centered √ at ±θ/ N . If λ = 0 there is no dependency between inputs and labels, which would set us in the perceptron with random labels setting, where the two clouds are perfectly superposed. The scaling of the noise part Zµ and the signal part containing θ is such that separating the two clouds with a hyperplane is nor impossible for some λ > 0, nor always trivial. It is feasible admitting a certain number of errors. In thermodynamic terms, we shall see this is equivalent to impose standard scalings on the Hamiltonian of the problem. For y ∈ {−1, 1} let uy , ϕ : R → R such that uy , ϕ ∈ C 2 (R) and H1) uy is concave and non affine and ∃ C > 0 such that −C(1 + s2 ) ≤ uy (s) ≤ 0 ,

|u′y (s)| ≤ C(1 + |s|)

(2.3)

H2) ϕ is nonnegative, convex and L-Lipschitz; H3) max(∥u′′y ∥∞ , ∥ϕ′′ ∥∞ ) ≤ C. H4) |u+1 (s) − u−1 (s)| ≤ C(1 + |s|),

s ∈ R.

(2.4)

Remark 1. Relating to machine learning and optimization literature, the utility function is the so-called loss function up to a sign: ℓ(x, y) = −uy (x). The loss functions used for classification purposes, such as the logistic loss of Section 4, verifies all the above hypotheses. The class H1-H4 is big enough to include also the quadratic case ℓ(x, y) = (x − y)2 /(2∆) with ∆ a properly tuned parameter, and the smoothed hinge loss. Given a realization of D we define the random Hamiltonian on the configurations space R ∋ σ = (σi )i≤N as  X  X κ gµ · σ − ϕ(σi ) − ∥σ∥2 (2.5) −HN (σ) = uyµ √ 2 N i≤N µ≤M N

where κ > 0. The variables in σ will be referred to as spins or weights. Given β > 0 the (random) Gibbs measure induced by the Hamiltonian H is the measure on RN with density νN (σ) =

e−βHN (σ) ZN (β, λ)

(2.6)

over the Lebesgue measure, where ZN is the normalization called partition function. Observe that, if uy is affine, then the Hamiltonian decouples over spin sites, thus making the model explicitly integrable. The notation ⟨·⟩ denotes the averages over independent (conditionally on the quenched disorder) samples from νN (σ)dσ, called replicas. For instance, for a function f of two replicas: Z ⟨f ⟩N = dx dy νN (x)νN (y)f (x, y) . (2.7) RN ×RN

4

Our goal is the computation of the quenched pressure of the model pN (β, λ) =

1 ED log ZN (β, λ) N

(2.8)

M → α. N

(2.9)

in the propotional limit N, M → ∞ with

This scaling regime is the natural one for studying the linear separability and storage capacity of random patterns [2, 3, 4]. We now introduce our variational potential. Definition 1 (Variational potential). For r, δ ≥ 0, ρ ≥ q ≥ 0 and m, h ∈ R define √

√

Ψβu (ρ, q, m) = Ey,v log Eξ e βuy (λym+ q v+ ρ−q ξ) Z Λϕ (r, δ, h) = Ez,θ log dσ e−βHϕ (σ;r,δ,h) ,

(2.10) (2.11)

where Hϕ (σ; r, δ, h) = ϕ(σ) +

√ κ+δ 2 σ − (z r + λhθ)σ 2

(2.12)

and ξ, v, z are independent, N (0, 1), θ ∼ Pθ , and y ∼ (δ−1 + δ1 )/2. The variational potential is Φu (ρ, q, m, r, δ, h) =

βδ ρ β 2 r(ρ − q) − + α Ψβu (ρ, q, m) + Λϕ (r, δ, h) − λβmh. 2 2

(2.13)

For later convenience we also introduced the reduced potential Φ⋆u (ρ, r) = sup inf Φu (ρ, q, m, r, δ, h) q,m δ,h

(2.14)

with δ ∈ [0, αC], h2 ∈ [0, αr], q ∈ [max(0, ρ − (βκ)−1 ), ρ], m2 ∈ [0, ρ]. The parameters ρ, q and m have the usual meanings of self-overlap, replica overlap, and alignment with the centroid, respectively. The parameters r, δ and h are their conjugate scalar-channel variables. Their precise relations are given by the stationarity equations in Proposition 1 below. We are finally in place to state our main result: Theorem 1 (Lower and upper bounds for the quenched pressure). For any β, α > 0, λ ≥ 0 and under the hypothesis H1 − H4 one has lim inf pN (β, λ) ≥ sup inf Φ⋆u (ρ, r) ,

(2.15)

lim sup pN (β, λ) ≤ inf sup Φ⋆u (ρ, r) .

(2.16)

N →∞

ρ≥0 r≥0

r≥0 ρ≥0

N →∞

In particular, whenever the two outer optimizations over ρ and r can be exchanged, the thermodynamic limit of the pressure exists and is given by the common variational value. The theorem does not require uniqueness of the full stationary-point system. 5

Exact high-dimensional asymptotics for related regularized classification problems have also been obtained through Gaussian comparison and approximate-message-passing methods [11, 17]. The proof of Theorem 1 uses the celebrated interpolation technique, originally introduced by Guerra and Toninelli for the Sherrington-Kirkpatrick model [28, 29], in its adaptive version, due to Barbier and Macris [30, 31]. Leaving out the Bayes-optimal setting analysed in [12], from which this paper also draws inspiration, forcing interpolation to work on such a general perceptron is the main novelty of our work. In fact, not only we are out of optimal inferential settings (see [12]) which introduce many crucial symmetries, called Nishimori identities [32], but we also allow for a rather general class of prior measure on the spins. The robustness w.r.t. this class of priors is inherited by the interpolation method. As a consequence of Nishimori identities, one can also prove useful correlation inequalities [34, 33] that in some cases are “all the convexity you need” to prove Guerra-type bounds, namely by discarding terms of definite sign [40, 43, 44]. Here though, since they are not available, the useful structural convexity is provided by log-concavity. We shall indeed see that a repeated use of Prékopa-Leindler theorem (see [36] for instance), asserting that the convolution of log-concave distributions is still log-concave, yields important properties of the variational potential, see Proposition 4 and Lemma 2, that enable us to obtain a bound in the direction opposite to the usual convexity of the thermodynamic generating functionals. The two bounds we have in Theorem 1 differ only for an exchange of supρ and inf r . The other optimizations actually commute as a consequence of Sion’s theorem [45] and are guaranteed to have unique solutions. We shall indeed see that Φu is jointly convex in δ, h and jointly concave in q, m, all of which are bounded inside convex and compact sets. Hence optimizations over r and ρ are the only source of possible mismatches of the two bounds. Unfortunately, we were not able to prove the matching under sufficiently general criteria, but for functions uy that are of interest for optimization and machine learning we provide numerical evidence that the matching occurs for several β, α, κ in Section 4. The possibility of exchanging these two optimizations has a similar role to that of the uniqueness requirement of Talagrand [27] and [46] on the solutions of the fixed point equation system contained in the Proposition below. Proposition 1 (Stationarity conditions). The critical points of Φ satisfy the following equations: 2 ρ = − ∂δ Λϕ (r, δ, h) β (h) λβm = ∂h Λϕ (r, δ, h) 2α (q) r = − 2 ∂q Ψβu (ρ, q, m) β (m) λβh = α∂m Ψβu (ρ, q, m) 2 (r) ρ − q = 2 ∂r Λϕ (r, δ, h) β 2α (ρ) δ = βr − ∂ρ Ψβu (ρ, q, m) β

(δ)

6

(2.17)

Introduce the notations dσe−βHϕ (σ;r,δ,h) (·) ⟨·⟩ϕ = R dτ e−βHϕ (τ ;r,δ,h) R

⟨·⟩βu =

Eξ eβuy (ν) (·) , Eξ eβuy (ν)

ν := λym +

(2.18) √ √ qv + ρ − qξ .

(2.19)

Then the stationarity equations rewrite as ρ = E⟨σ 2 ⟩ϕ ,

m = Eθ⟨σ⟩ϕ , q = E⟨σ⟩2ϕ h D u′′ (ν) E i ′ 2 ′ 2 ′ 2 , r = αE⟨u (ν)⟩βu , δ = βα E⟨u (ν)⟩βu − E (u (ν)) + β βu

h = αEy⟨u′y (ν)⟩βu . (2.20)

Remark 2. The above set of equations and the variational potential match the results obtained by Talagrand [27] for the case of Gaussian spin prior (ϕ ≡ 0) and random labels (λ = 0). Indeed, when λ = 0 both m and h disappear from the variational potential. r − δ = r̄ in Talagrand’s notation and for Gaussian spin prior ⟨·⟩ϕ is actually a Gaussian measure, so expressions for ρ and q in (2.20) can be explicitly computed yielding the cavity equations. From our master Theorem 1, with some further mild assumptions, we are able to derive formulae for the so-called traning loss and, more importantly, generalization error of the perceptron. Both observables are obtained by adding a scalar perturbation to the Hamiltonian, identifying the corresponding derivative of the pressure, and then differentiating the limiting variational formula. Corollary 1 (Training loss). Suppose u is such that for uγ = γu the two bounds in Theorem 1 match for all γ ∈ (1 − ϵ, 1 + ϵ) for a fixed ϵ > 0, thus identifying the thermodynamic √ √ limit. Define ν ∗ = λym∗ + q ∗ v + ρ∗ − q ∗ ξ, with ρ∗ , q ∗ , m∗ and r∗ , δ ∗ , h∗ unique solvers of the variational principle for u1 = u. Then, the training loss has the following asymptotics ∗ M  g · σ E Eξ uy (ν ∗ )eβuy (ν ) 1 DX µ E u yµ √ =E . −εt := lim N →∞ M Eξ eβuy (ν ∗ ) N µ=1

(2.21)

Corollary 2 (Generalization error). Let α′ > 0, y ′ ∼ δ−12+δ1 and g ′ = √λN y ′ θ + Z ′ , with Z ′ ∼ N (0, 1N ) all independent of the quenched disorder, and a function Wy : R −→ R with ∥Wy ∥∞ ,∥Wy′ ∥∞ , ∥Wy′′ ∥∞ ≤ Cw for a.e. y. Introduce Φũ,γ (ρ, q, m, r, δ, h) :=

βδρ β 2 r(ρ − q) − + αΨβu (ρ, q, m) + α′ Ψβ ũγ (ρ, q, m) 2 2 + Λϕ (r, δ, h) − λβmh , (2.22)

and suppose there exists an η > 0 such that for all γ ∈ [0, η) and t ∈ [−η, η], ũy,γ = γ(uy + tWy ) is concave and the optimizations in (2.15) and (2.16) yield the same result for Φũ,γ . Let ρ∗ , q ∗ , m∗ be as in Corollary 1. Then εW g :=

D  g ′ · σ E √ √ = EEξ Wy′ (λy ′ m∗ + q ∗ v ′ + ρ∗ − q ∗ ξ) . lim E Wy′ √ N →∞ N 7

(2.23)

In the above, α′ has to be interpreted as resulting from additional M ′ observations of input-output couples. The introduction of these fresh samples is necessary to generate the generalization error (2.23) from derivatives of the quenched pressure and of its limit. The influence of these new examples on the Gibbs measure is then forced to disappear by sending γ → 0+ in such derivatives. The additional assumptions we need for these corollaries are “local” validity of variational principles for small variations of the function uy . These are needed because the averages of interest are generated through derivatives, which require existence of a proper variational formula in a whole neighborhood of uy , though as small as you want. From Theorem 1 one can compute a properly rescaled β → ∞ limit, i.e. the 0temperature limit, that yields the ground state energy of the perceptron. Proposition 2 (Ground state energy). Introduce the rescaled potential Φ̃u (X, ρ, m, r, δ, h) =

δρ rX − + αΨ̃u (ρ, X, m) + Λ̃ϕ (r, δ, h) − λmh 2 2

(2.24)

with √ ξ2 i √ Ψ̃u (ρ, X, m) = Ey,v sup uy (λym + ρv + Xξ) − 2 ξ h√ κ + δ 2i Λ̃ϕ (r, δ, h) = Ez sup ( rz + λθh)σ − ϕ(σ) − σ . 2 σ h

(2.25) (2.26)

Denote the average ground state energy by HN (σ) 1 = lim − pN (β, λ) , σ β→∞ N β

eN = E inf

(2.27)

where the average is w.r.t. the quenched randomness in the Hamiltonian. Then lim inf −eN ≥ sup inf

sup

sup

inf

inf Φ̃u (X, ρ, m, r, δ, h) ,

(2.28)

lim sup −eN ≤ inf sup

sup

sup

inf

inf Φ̃u (X, ρ, m, r, δ, h) .

(2.29)

N →∞

N →∞

ρ≥0 r≥0 X∈[0,κ−1 ] m2 ≤ρ δ∈[0,αC] h2 ≤αr

r≥0 ρ≥0 X∈[0,κ−1 ] m2 ≤ρ δ∈[0,αC] h2 ≤αr

The proof of Proposition 2 closely mimics ideas from physics. It involves a reparameterization of the variational bounds (2.15) and (2.16). The key is that, thanks to log-concavity, the new parameter X ≈ β(ρ − q) remains bounded even when β → ∞. In fact, the two parameters ρ, the “self overlap”, and q, the “overlap”, collapse onto one another when β diverges. Remark 3. Concerning Corollary 2, using the class of functions we chose for Wy , we can approximate arbitrarily well, via a density argument, also the usual generalization error, which amounts to Wy (x) = (y−sign(x))2 /4. Then, letting β → ∞ in (2.23), and assuming the β-rescaled variational bounds in Proposition 2 match, one can prove the generalization error approaches  1  p p 1 εg = 1 − E y ′ sign(λy ′ m̃ + ρ̃v ′ ) = 1 − E sign(λm̃ + ρ̃v ′ ) 2 2 R∞ s2 with m̃ and ρ̃ solving the variational principle of Proposition 2. Calling Q(x) = x √ds2π e− 2 one readily gets  λm̃  εg = Q √ (2.30) ρ̃ 8

which matches the result of [16], which uses Gordon’s minimax theorem [47, 48] to reduce the original high-dimensional optimization problem to a scalar auxiliary problem. At β → ∞ the Gibbs distribution concentrates on the minimum of the Hamiltonian function, which is convex. m̃ is thus the expected alignment of this minimum point with the direction of the centroid θ and is directly connected to the generalization performance (2.30). This is telling us that what the perceptron is learning through its weights is actually the direction θ, that is orthogonal to the hyperplane that separates at best the two clouds of points. Despite (2.30) appears in the same way as [16], our results apply to a general class of prior measures on the spins, or equivalently, regularizations. The choice of the regularization is thus hidden in the values ρ̃ and m̃ solving the β → ∞ variational principle. Furthermore, approaches based on Gordon’s minimax theorem, require uniqueness of a system of fixed point equations, thus suffering from the same limitations of the cavity approaches [27, 46]. We, on the other hand, get that very system from a variational principle, where the limitation is transferred to the exchangeability of two optimizations.

3

Proofs

Roadmap of the proof The proof is organized around the exact interpolation identity of Proposition 3, the so called sum rule. The interpolating Hamiltonian (3.1) connects the original perceptron at t = 0 to a collection of independent scalar channels at t = 1, or gas of spins, whose pressure is computed in Lemma 1. Differentiating along the interpolation path and performing Gaussian integration by parts yields the sum rule (3.15), in which the error term (3.16) is written as a sum of products between fluctuations of the order parameters and discrepancies between their Gibbs averages and the velocities of the interpolation path. The upper and lower bounds are then obtained through two complementary adaptive choices. For the upper bound, we fix ṙ(t) = r, choose (δ̇, ḣ) through the scalar minimization in (δ, h), and evolve (ρ, q, m) according to the Gibbs averages of (Q11 , Q12 , M1 ), as in (3.52). For the lower bound, we instead fix ρ̇(t) = ρ, choose (q̇, ṁ) through the scalar maximization in (q, m), and evolve (r, δ, h) according to the Gibbs averages of (S12 , S12 − S11 , H1 ), as in (3.57). Proposition 4 provides the convex– concave structure needed to apply Jensen’s and minimax arguments, while Lemma 2 guarantees that the lower-bound path remains in the physical region r, δ ≥ 0. The moment estimates and concentration results collected in the appendices ensure global existence of the two interpolation paths and, through Remark 5, make ΩN vanish in the thermodynamic limit. The two constructions therefore produce bounds that differ only in the order of the outer optimizations over ρ and r. The remaining parts of the section derive the zero-temperature result by introducing X = β(ρ − q) and applying uniform Laplace asymptotics, and obtain the training and generalization observables by differentiating suitable scalar perturbations of the pressure.

3.1

Adaptive interpolation

Our proof relies on the interpolation method introduced in [30], which builds on the celebrated Guerra interpolation, successfully employed for the Sherrington-Kirkpatrick model [29]. For perceptron-like models, our adaptive interpolation is inspired by Talagrand’s cavity computation [27] for the Shcherbina Tirozzi model [24].

9

We introduce the interpolating Hamiltonian: −HNt (σ, ξ) =

X

uyµ (Stµ ) +

µ≤M

X p κ + δ(t) r(t) z · σ − ϕ(σi ) − ∥σ∥2 + λh(t)θ · σ (3.1) 2 i≤N

where r Stµ =

p p 1−t θ·σ Zµ · σ + (1 − t)λyµ + λyµ m(t) + vµ q(t) + ξµ ρ(t) − q(t). N N

(3.2)

i.i.d.

with vµ , ξµ ∼ N (0, 1), and r(t), δ(t), h(t), q(t), m(t), ρ(t) are generic functions of t ∈ [0, 1], which, by construction, have to satisfy the following conditions: i) q(t), r(t), δ(t) ≥ 0; ii) q(t) ≤ ρ(t); iii) q(0) = ρ(0) = r(0) = δ(0) = m(0) = h(0) = 0. In the following we denote αN = M/N . From the above Hamiltonian, the definitions of interpolating partition function, Gibbs measure density and pressure respectively follow: Z t t ZN (β, λ) = Eξ dN σe−βHN (σ,ξ) (3.3) t

e−βHN (σ,ξ) νt (σ, ξ) = t ZN (β, λ) Z 1 t t pN (β, λ) = E log Eξ dN σe−βHN (σ,ξ) . N

(3.4) (3.5)

The outer quenched expectation in (3.5) averages over the variables (Zµ , vµ , yµ )µ≤M , z, θ. For later convenience we define the following quantities M
2 1 X ′′ [βuyµ (Stµ ) + βu′yµ (Stµ ) ], S11 = N µ=1

σ (a) · σ (b) , a, b = 1, 2 N M θ · σ (a) 1 X (a) Ma = , Ha = yµ u′yµ (Stµ ) , N N µ=1

M

β2 X ′ (1) (2) S12 = uyµ (Stµ )u′yµ (Stµ ) N µ=1

Qab =

(3.6) (3.7)

a = 1, 2

(3.8)

where superscripts like (a) denote replica indices, i.e. independent samples from (3.4) conditionally on the quenched disorder. Lemma 1 (Endpoints). At the endpoints of the interpolation path it holds: p0N (β, λ) = pN (β, λ) p1N (β, λ) = αN Ψβu (ρ(1), q(1), m(1)) + Λϕ (r(1), δ(1), h(1)) with Ψβu and Λϕ as in Definition 1. 10

(3.9) (3.10)

Proof. To prove Lemma 1, it suffices to evaluate HNt (σ, ξ) at the endpoints t = 0 and t = 1. The case t = 0 is straightforward. Indeed, condition iii) immediately implies that HN0 (σ, ξ) = HN (σ), from which (3.9) follows directly once we use eq. (3.5). At t = 1, the quenched and annealed random variables in the interpolating Hamiltonian decouple. Consequently,

−HN1 (σ) =

M X

uyµ (S1µ ) + z · σ

µ=1

X p κ + δ(1) r(1) − ϕ(σi ) − ||σ||2 + λh(1)θ · σ 2 i≤N

(3.11)

where: S1µ = λyµ m(1) + vµ

p

q(1) + ξµ

p ρ(1) − q(1)

(3.12)

It then follows from eq. (3.5), that the interpolating pressure decomposes into two independent contributions: the first depends only on the pattern index, while the second depends only on the spin variables. More precisely,   √ √ P 1 q(1)+ξµ ρ(1)−q(1) β M µ=1 uyµ λyµ m(1)+vµ Ey,v log Eξ e N Z  √  P κ+δ(1) 1 β z·σ r(1)− i≤N ϕ(σi )− 2 ||σ||2 +λh(1)θ·σ N + Ez,θ log d σe N

p1N (β, λ) =

(3.13)

The first contribution factorizes over the samples, whereas the second factorizes over the spin coordinates. Using αN = M/N , we obtain   √ √ βuy λym(1)+v q(1)+ξ ρ(1)−q(1) 1 pN (β, λ) = αN Ey,v log Eξ e + Z   √ (3.14) κ+δ(1) −β ϕ(σ)+ 2 σ 2 −(z r(1)+λh(1)θ)σ + Ez,θ log dσe This is precisely the decomposition into the variational potentials Ψβu and Λϕ introduced in Definition 1, thereby yielding (3.10). Proposition 3 (Sum rule). Given Lemma 1, the pressure is: pN (β, λ) = αN Ψβu (ρ(1), q(1), m(1)) + Λϕ (r(1), δ(1), h(1)) Z 1 h i δ̇(t)ρ̇(t) − β ṙ(t)(ρ̇(t) − q̇(t)) +β dt − λḣ(t)ṁ(t) + ΩN 2 0

(3.15)

where the remainder is Z D E 1 1 ΩN = dt E (ρ̇(t) − Q11 ) (β(β ṙ(t) − δ̇(t)) − S11 ) − (q̇(t) − Q12 ) (β 2 ṙ(t) − S12 ) 2 0 t Z 1 D E + βλ dt E (ḣ(t) − H1 )(ṁ(t) − M1 ) . (3.16) t

0

Proof. The proof of Proposition 3 is based on the sum rule: Z 1 0 1 pN (β, λ) = pN (β, λ) − ṗtN (β, λ) dt 0

11

(3.17)

From Lemma 1, the endpoints of the interpolation path can be expressed explicitly as Z 1 pN (β, λ) = αN Ψβu (ρ(1), q(1), m(1)) + Λϕ (r(1), δ(1), h(1)) −

ṗtN (β, λ) dt

(3.18)

0

Therefore, it remains to compute the integral of the derivative of the interpolating pressure along the interpolation path. Differentiating eq. (3.5) yields ṗtN (β, λ) =

E 1 D E − β ḢNt (σ, ξ) N t

(3.19)

where ⟨·⟩t denotes expectation with respect to the Gibbs measure associated with the interpolating Hamiltonian. Differentiating the interpolating Hamiltonian in eq. (3.1) gives

−β ḢNt (σ, ξ) = β

M X

δ̇(t) ṙ(t) −β u′yµ (Stµ ) Ṡtµ + βz · σ p ||σ||2 + βλḣ(t)θ · σ 2 2 r(t) µ=1

(3.20)

where, by differentiating (3.2), 1 θ·σ q̇(t) ρ̇(t) − q̇(t) Ṡtµ = − p σ · Zµ − λyµ + λyµ ṁ(t) + vµ p + ξµ p (3.21) N 2 (1 − t)N 2 q(t) 2 ρ(t) − q(t) Substituting (3.20) and (3.21) into (3.19), we obtain an explicit expression for the derivative of the interpolating pressure, composed of several terms: ṗtN (β, λ) = I + II + III + IV + V + V I + V II + V III

(3.22)

We analyze each contribution separately. The first term couples the pattern noise with the spin variables: M E β DX ′ σ · Zµ I=− E uyµ (Stµ ) p N 2 (1 − t)N t µ=1

(3.23)

At this stage, we apply Stein’s lemma, Ex [xf (x)] = Ex [f ′ (x)], with x = Zµ . Differentiating with respect to all the components Zµ we get: " M DX  2E 1 ′′ ′ 2 ||σ|| I=− + E βuyµ (Stµ ) + (βuyµ (Stµ )) 2N N t µ=1 (3.24) # M DX     σ (1) · σ (2) E (1) (2) − β 2 u′yµ Stµ u′yµ Stµ . N t µ=1 The second and third contributions are treated analogously. The only difference is that the quenched Gaussian variables are vµ and zi instead of zµ , M E β DX ′ q̇(t) II = E uyµ (Stµ ) p vµ N t 2 q(t) µ=1

12

E β D ṙ(t) III = E p z·σ N t 2 r(t)

(3.25)

Applying Stein’s lemma once again, now with x = vµ and x = zi respectively, yields " M  E DX 1 II = E βu′′yµ (Stµ ) + (βu′yµ (Stµ ))2 q̇(t) + 2N t µ=1 # M  DX     E (1) (2) (3.26) − β 2 u′ S u′ S q̇(t) tµ

yµ

yµ

tµ

µ=1



1 III = E β 2 2

D ||σ||2 E N

− t

D σ (1) · σ N

 (2) E

t

 ṙ(t)

t

The fourth contribution is slightly different, as it involves the annealed random variable ξµ rather than a quenched one: M E ρ̇(t) − q̇(t) β DX ′ p u (Stµ ) ξµ IV = E N t 2 ρ(t) − q(t) µ=1

(3.27)

Stein’s lemma can be applied once again, now with x = ξµ . In contrast to the previous cases, however, the derivative acts only on the Gibbs weight and on prefactors of ξµ , since the partition function is independent of ξµ . Differentiating with respect to ξµ therefore yields M  Ei 1 X hD  ′′ ′ 2 E βuyµ (Stµ ) + (βuyµ (Stµ )) (ρ̇(t) − q̇(t)) (3.28) IV = 2N µ=1 t In each of the preceding contributions, Stein’s lemma introduces overlap terms between distinct spin configurations, thereby considerably simplifying the resulting expressions. The remaining contributions, however, are already naturally expressed in terms of the overlap with the centroid or the norm of the weight vector. Consequently, no integration by parts is required, and these terms are left unchanged. More precisely, M θ · σE 1 DX ′ βuyµ (Stµ ) λyµ V =− E N N t µ=1

M E 1 DX ′ βuyµ (Stµ )λyµ ṁ(t) VI = E N t µ=1

(3.29) D E D E δ̇(t) 1 1 V III = E βλ(θ · σ)ḣ(t) V II = − E β||σ||2 N 2 t N t Substituting all these terms into (3.22), using the definitions (3.6)-(3.8), and integrating the derivative of the interpolating pressure along the interpolation path as in (3.18), we obtain (3.15) and (3.16). This completes the proof. Remark 4. Observe that in the proof of the sum rule we did not rely on the knowledge of the prior measure of the spins, which distinguishes our approach from the previous ones and thus enlarges the class of perceptron models we can tackle. Remark 5. The form of the remainder ΩN is in a particularly convenient form. In fact, in order to deal effectively with it we just need to prove concentration w.r.t. the quenched measure of the order parameters Q11 , Q12 , M1 and boundedness of the second moments of S11 , S12 , H1 . These properties hold and are proved in Propositions 6, 7 in the Appendix. We can thus split the expectations in (3.16) simply using Cauchy-Schwartz inequality paying with a negligible error that, as we shall see, is uniform in t and can thus be discarded. Since these results are obtained with methods analogous to those of [46] and [27], the proofs of these bounds and concentrations are deferred to Appendix D 13

3.2

Convexities

Here we list some further preliminary results that shall be used throughout. We start with some crucial convexity properties of the variational potential: Proposition 4 (Convexity properties of Φ). Assume u concave and non-affine and λ > 0. Φ has the following properties: i) Λϕ and Φ are strictly convex in (δ, h); ii) Ψβu and Φ are strictly concave in (q, m). Proof. (i). Recall the definitions (2.12) and (2.18), and introduce Z Fϕ (r, δ, h) = log dσe−βHϕ (σ;r,δ,h) . With this notation we have   1 V[σ 2 ] − λθ [⟨σ 3 ⟩ϕ − ⟨σ 2 ⟩ϕ ⟨σ⟩ϕ ] −2 4 2 β Hessδ,h Fϕ = − λθ [⟨σ 3 ⟩ϕ − ⟨σ 2 ⟩ϕ ⟨σ⟩ϕ ] λ2 θ2 V[σ] . 2

(3.30)

(3.31)

where variances and covariance will be intended w.r.t. ⟨·⟩ϕ . Its determinant reads det β −2 Hessδ,h Fϕ =

 λ2 θ2  V[σ]V[σ 2 ] − Cov[σ, σ 2 ]2 . 4

(3.32)

Whenever θ ̸= 0 the above is always non-negative thanks to Cauchy-Schwartz. Moreover, it is degenerate if and only if σ 2 − ⟨σ 2 ⟩ϕ = a(σ − ⟨σ⟩ϕ ) , for some a ∈ R

(3.33)

i.e. if the two centered variables involved in the variances are proportional to each other. This can happen iff σ is supported on the two points that solve the above quadratic equation. It is not the case since σ is a continuous variable. Finally, since the diagonal elements of Hessδ,h Fϕ are positive with positive probability, by Sylvester’s criterion Hessδ,h Fϕ is positive definite with positive probability, and this grants joint strict convexity of Λϕ . (ii). Consider the function √

Fu (x, s) = log Eξ eu(x+ sξ) .

(3.34)

The argument of the log is a convolution of a log-concave function with a Gaussian measure, which is itself log-concave. Prékopa-Leindler theorem (see, e.g., [36]) indeed asserts that convolution preserves log concavity. As a consequence, F (x, s) is concave in x. Observe also that the very same object is the solution to a heat equation, being the convolution with a Gaussian kernel. We can thus show that its logarithm satisfies √ √ √ √ ∂x Fu = ⟨u′ (x + sξ)⟩u , ∂x2 Fu = ⟨u′′ (x + sξ) + (u′ (x + sξ))2 ⟩u − ⟨u′ (x + sξ)⟩2u √ √ 1 ∂s Fu = ⟨u′′ (x + sξ) + (u′ (x + sξ))2 ⟩u (3.35) 2

14

where √

Eξ eu(x+ sξ) (·) √ ⟨·⟩u = . Eξ eu(x+ sξ)

(3.36)

Combining derivatives together we get ∂x2 Fu + (∂x Fu )2 = 2∂s Fu ,

(3.37)

from which we also get 1 ∂x ∂s Fu = ∂x3 F + ∂x2 Fu ∂x Fu . 2 Then, using these definitions, we can recast √ Ψu (ρ, q, m) = Ey,v Fu (λym + qv, ρ − q) ,

(3.38)

(3.39)

and leverage the relations among derivatives to evaluate the Hessian of the above function in (q, m). To begin with, √ √ 2 Ψu = λ2 Ey,v ∂x2 Fu (λym + qv, ρ − q) ∂m Ψu = λEy,v y ∂x Fu (λym + qv, ρ − q) , ∂m (3.40) where we used y 2 = 1. Observe that, being u non-affine, it is not difficult to prove that 2 Fu cannot be affine either, and ∂x2 Fu < 0, entailing in turn ∂m Ψu < 0. Then, using integration by parts of v   v  1 1 1 ∂q Ψu = Ey,v √ ∂x Fu − ∂s Fu = Ey,v ∂x2 Fu − ∂x2 Fu − (∂x Fu )2 2 q 2 2 2 1 = − Ey,v (∂x Fu )2 ≤ 0 . (3.41) 2 A further derivation yields   v ∂q2 Ψu = −Ey,v ∂x Fu √ ∂x2 Fu − ∂s ∂x Fu 2 q  1 3 1 1 = −Ey,v ∂x F ∂x Fu + (∂x2 Fu )2 − ∂x Fu ∂x3 F − (∂x Fu )2 ∂x2 F 2 2 2 1 2  2 2 2 = −Ey,v (∂x Fu ) − (∂x Fu ) ∂x F < 0 (3.42) 2 where strict negativity follows again from strict concavity fo F in x. The mixed derivative finally reads ∂q ∂m Ψu = −Ey,v ∂x Fu λy∂x2 Fu .

(3.43)

The determinant of the Hessian thus reads  1 
2 det HessΨu = λ2 Ey,v − (∂x2 Fu )2 + (∂x Fu )2 ∂x2 Fu Ey,v ∂x2 Fu − λ2 Ey,v y∂x Fu ∂x2 F 2   1 ≥ λ2 Ey,v − (∂x2 Fu )2 + (∂x Fu )2 ∂x2 Fu Ey,v ∂x2 Fu − λ2 Ey,v (∂x Fu )2 ∂x2 F Ey,v ∂x2 F 2   2 1 2 (3.44) = − λ Ey,v (∂x Fu )2 Ey,v ∂x2 Fu > 0 2 through Cauchy-Schwartz inequality. Thanks to Sylvester’s criterion, HessΨu is negative definite. Hence Ψu is strictly concave.

15

We conclude with a consequence of log-concavity that will allow us to restrict the range of our interpolating functions: Lemma 2. The following inequality holds: ⟨S12 ⟩t ≥ ⟨S11 ⟩t .

(3.45)

Remark 6. The above inequality is the log-concave substitute for the positivity of the interpolation remainder. It is the only sign condition used in the construction of the lower bound, where it guarantees that both r(t) and δ(t) remain non-decreasing along the interpolation path, i.e. inside the physical region. In this sense it plays here the role that first-kind correlation inequalities play for the direct problem in mean-field spin glasses. Proof. The statement is again a consequence of Prékopa-Leindler theorem. In fact, one can define Z 1 t log dN σEξ e−βHN (σ,x) , (3.46) Y (x) = N where x = (xµ )µ≤M , and −HNt (σ, x) =

X

uyµ (Stµ + xµ ) + z · σ

µ≤M

X p κ + δ(t) r(t) − ϕ(σi ) − ∥σ∥2 + λh(t)θ · σ . 2 i≤N (3.47)

The joint Boltzmannfaktor for σ, ξ is straightforwardly log-concave. Hence Y (x) is concave by Prékopa-Leindler theorem, as it is the logarithm of the convolution of log-concave functions. This in particular entails Tr Hess Y (x = 0) = ⟨S11 ⟩t − ⟨S12 ⟩t ≤ 0 ,

(3.48)

for every fixed realization of the disorder.

3.3

Bounds for the quenched pressure

Proof of Theorem 1. First, note that we can replace αN by its limit α up to a vanishing remainder. In the following we shall thus use α only. Upper bound: Let us make the following initial choice: ṙ(t) = r > 0. Then, using Proposition 4, in particular the joint convexity in (δ, h) of the functional, we get β 2 r(ρ(1) − q(1)) pN (β, λ) ≤ αΨβu (ρ(1), q(1), m(1)) − 2 Z 1 h i β δ̇(t)ρ̇(t) + dt Λϕ (r, δ̇(t), ḣ(t)) + − βλḣ(t)ṁ(t) + ΩN , (3.49) 2 0 where we used Jensen’s inequality on Λϕ . Now we choose the pair δ̇, ḣ such that (δ̇(t), ḣ(t)) = arg inf Λϕ (r, δ, h) + δ,h∈Tr

βδ ρ̇(t) − βλhṁ(t) =: Fδ,h (Tr , r; ρ̇(t), ṁ(t)) 2

(3.50)

which is uniquely identified due to strict convexity. Tr ⊂ [0, ∞) × R here is a compact convex set to be chosen later. Over this set δ̇ ≥ 0 and then δ(t) ≥ 0. Therefore, when 16

inserted in the Gibbs measure, δ(t) corresponding to this choice preserves log-concavity and convergence simultaneously. Furthermore, since Tr is compact, Fδ,h (Tr , r; ρ̇(t), ṁ(t)) is jointly continuous in r, ρ̇(t), ṁ(t) and bounded in Tr by definition. Then β 2 r(ρ(1) − q(1)) pN (β, λ) ≤ αΨβu (ρ(1), q(1), m(1)) − 2 Z 1 h i βδ ρ̇(t) dt inf Λϕ (r, δ, h) + − βλhṁ(t) + ΩN + δ,h∈Tr 2 0 2 β r(ρ(1) − q(1)) ≤ αΨβu (ρ(1), q(1), m(1)) − 2 h i βδρ(1) + inf Λϕ (r, δ, h) + − βλhm(1) + ΩN δ,h∈Tr 2 ≤ sup sup inf Φu (ρ, q, m, r, δ, h) + ΩN , (3.51) ρ≥0 q,m∈Dρ δ,h∈Tr

√ √ where Dρ = [max(0, ρ − (βκ)−1 ), ρ] × [− ρ, ρ] is compact and convex. It is now time to choose the other interpolating functions m(t), ρ(t), q(t). We choose them according to the following system of coupled ODEs in t ∈ [0, 1]: ṁ(t) = E⟨M1 ⟩t = Fm (t; r; ρ(t), q(t), m(t), δ(t), h(t)) ρ̇(t) = E⟨Q11 ⟩t = Fρ (t; r; ρ(t), q(t), m(t), δ(t), h(t)) q̇(t) = E⟨Q12 ⟩t = Fq (t; r; ρ(t), q(t), m(t), δ(t), h(t))

(3.52)

(δ̇(t), ḣ(t)) = Fδ,h (Tr , r; ρ̇(t), ṁ(t)) . Observe that Fm2 , Fq ≤ Fρ by Cauchy-Schwartz, which is consistent with the constraints imposed by Dρ . Furthermore, using Brascamp-Lieb inequality [36], and that δ(t) ≥ 0, one readily gets N

1 X E⟨σi2 − ⟨σi ⟩2t ⟩t ≤ (βκ)−1 Fρ − F q = N i=1

⇒

ρ(t) − q(t) ≤ (βκ)−1 ∀ t ∈ [0, 1] ,

and hence, consistently with Dρ , q(t) ≥ ρ(t) − (βκ)−1 . The δ, h-components of the velocity field are bounded in Tr by definition. Thanks to Proposition 6 in Appendix, the velocity field grows at most as K(1+m2 (t)+ρ(t)+rt) with a proper constant K > 0. Therefore, by Lemma 8, we are guaranteed a global solution in t ∈ [0, 1] exists. By plugging the choices in (3.52) into the remainder, following Remark 5 and using Proposition 7 in the Appendix, the remainder vanishes once we take the limit. We thus take the lim sup on both sides of the bound, obtaining lim sup pN (β, λ) ≤ sup sup N →∞

inf Φu (ρ, q, m, r, δ, h)

ρ≥0 q,m∈Dρ δ,h∈Tr

(3.53)

for all r > 0. We then optimize the bound over r. Lower bound: Let us start by choosing ρ̇ = ρ > 0 constant. Then we can use the joint concavity of Ψβu and the sum rule to get βδ(1)ρ pN (β, λ) ≥ Λϕ (r(1), δ(1), h(1)) + 2 Z 1 h i β 2 ṙ(t)(ρ − q̇(t)) + dt αΨβu (ρ, q̇(t), ṁ(t)) − − βλḣ(t)ṁ(t) + ΩN , 2 0 17

(3.54)

where we used Jensen’s inequality for Ψβu . Now we choose (q̇(t), ṁ(t)) = arg sup αΨβu (ρ, q, m) − q,m∈Dρ

β 2 ṙ(t)(ρ − q) − βλḣ(t)m 2

=: Fq,m (Dρ , ρ; ṙ(t), ḣ(t)).

(3.55)

Thanks to compactness Fq,m (Dρ , ρ; ṙ(t), ḣ(t)) is jointly continuous in ṙ(t), ḣ(t) and bounded in Dρ by definition. By plugging this choice in the sum rule we get βδ(1)ρ pN (β, λ) ≥ Λϕ (r(1), δ(1), h(1)) + 2 Z 1 i h β 2 ṙ(t)(ρ − q) − βλḣ(t)m + ΩN + dt sup αΨβu (ρ, q, m) − 2 q,m∈Dρ 0 βδ(1)ρ ≥ Λϕ (r(1), δ(1), h(1)) + 2 h i β 2 r(1)(ρ − q) + sup αΨβu (ρ, q, m) − − βλh(1)m + ΩN 2 q,m∈Dρ h i βδρ β 2 r(ρ − q) ≥ inf inf sup Λϕ (r, δ, h) + + αΨβu (ρ, q, m) − − βλhm + ΩN r≥0 δ,h∈Tr q,m∈Dρ 2 2 ≥ inf sup

inf Φu (ρ, q, m, r, δ, h) + ΩN

r≥0 q,m∈Dρ δ,h∈Tr

(3.56)

Now we choose the remaining interpolating functions according to the following ODEs β 2 ṙ(t) = E⟨S12 ⟩t = Fr (t; ρ; r(t), q(t), m(t), δ(t), h(t)) β δ̇(t) = E⟨S12 ⟩t − E⟨S11 ⟩t = Fδ (t; ρ; r(t), q(t), m(t), δ(t), h(t)) ḣ(t) = E⟨H1 ⟩t = Fh (t; ρ; r(t), q(t), m(t), δ(t), h(t))

(3.57)

(q̇(t), ṁ(t)) = Fq,m (Dρ , ρ; ṙ(t), ḣ(t)) . Note that Fr ≥ 0 and Fδ ≥ 0 by Lemma 2. Furthermore, Fh2 ≤ αFr /β 2 by CauchyScwhartz inequality and β −1 Fδ ≤ −αE⟨u′′yµ (Stµ )⟩t ≤ αC by an application of Jensen’s √ √ inequality and H3. This allows us to take Tr = [0, αC] × [− αr, αr]. Finally, by Proposition 6 in Appendix, the velocity field of the above ODE system grows at most as K(1 + ρt + h2 (t) + r(t)). Therefore, following Lemma 8, there exists a global solution. Plugging these choices in the remainder, following Remark 5 and leveraging our Proposition 7, we make the remainder vanish by taking the lim inf on both sides of the main inequality: lim inf pN (β, λ) ≥ inf sup N →∞

inf Φu (ρ, q, m, r, δ, h)

r≥0 q,m∈Dρ δ,h∈Tr

(3.58)

which can then be optimized w.r.t. ρ. Remark 7. Recall the fixed point equations for stationary points in Proposition 1. By a simple application of Cauchy-Schwartz one immediately proves that ρ ≥ q, m2 and that h2 ≤ αr. Secondly, δ=−

 2α  α ∂ρ Ψβu (ρ, q, m) + ∂q Ψβu (ρ, q, m) = − 2 ∂x2 Ψβu (ρ, q, m + x)|x=0 ≥ 0 β βλ 18

(3.59)

by Prékopa-Leindler theorem (recall y 2 = 1). In addition, a simple use of Jensen’s inequality yields δ ≤ −αE⟨u′′y (ν)⟩βu ≤ αC

(3.60)

by H1. Furthermore, since ⟨·⟩ϕ is log-concave, we can use Brascamp-Lieb inequality to show:
−1 0 ≤ ⟨σ 2 ⟩ϕ − ⟨σ⟩2ϕ ≤ β −1 ⟨ ϕ′′ (σ) + κ + δ ⟩ϕ ≤ (βκ)−1 (3.61) where we used that ϕ is convex and δ ≥ 0. Therefore the constraints in Tr and Dρ are automatically satisfied by stationary points.

3.4

Ground state energy

To begin with, we prove the following: Lemma 3 (Reparameterization). The two bounds (2.15) and (2.16) can be recast as X , m, r, δ, h) , (3.62) N →∞ β X lim sup pN (β, λ) ≤ inf sup sup sup inf 2inf Φu (q + , q, m, r, δ, h). (3.63) r≥0 q≥0 X∈[0,κ−1 ] m2 ≤q+X/β δ∈[0,αC] h ≤αr β N →∞ lim inf pN (β, λ) ≥ sup inf sup

sup

inf

inf Φu (ρ, ρ −

ρ≥0 r≥0 m2 ≤ρ X∈[0,κ−1 ] δ∈[0,αC] h2 ≤αr

The fixed point equation for X then becomes
X = β E⟨σ 2 ⟩ϕ − E⟨σ⟩2ϕ .

(3.64)

In particular, at stationarity X ≤ κ−1 is verified. Proof. The rewriting of the two bounds and of the fixed point equation for X corresponds to a simple reparameterization X = β(ρ − q) in the original variational function. The bound X ≤ κ−1 is proved as in (3.61). Afterwards, we need the following scaling limits: Lemma 4. Let X ∈ [0, κ−1 ] and δ ∈ [0, αC]. The following scaling relations hold √  X ξ2  √ , q, m) = E sup uy (λym + qv + Xξ) − , β→∞ β 2 ξ √  X ξ2  √ lim β −1 Ψβu (ρ, ρ − , m) = E sup uy (λym + ρv + Xξ) − , β→∞ β 2 ξ   √ κ+δ 2 lim β −1 Λϕ (r, δ, h) = E sup − ϕ(σ) − σ + (z r + λhθ)σ . β→∞ 2 σ lim β −1 Ψβu (q +

(3.65) (3.66) (3.67)

The convergence is uniform over the variational parameters. √ Proof. By a change of variable ξ 7→ ξ β we get β −1 Ψβu (q +

X 1 , q, m) = EAβ + log β . β 2β 19

(3.68)

with 1 Aβ := log β

Z

dξ √ eβUq (ξ) , 2π

Uq (ξ) := u(λym +

√

√ qv +

Xξ) −

ξ2 . 2

(3.69)

Thanks to H1 we have −(1 + C/κ) ≤ −(1 + CX) ≤ Uq′′ (ξ) ≤ −1 and hence −

1 + C/κ 1 (ξ − ξ ∗ )2 ≤ Uq (ξ) − Uq (ξ ∗ ) ≤ − (ξ − ξ ∗ )2 . 2 2

with ξ ∗ = arg supξ Uq (ξ). This in turn entails Uq (ξ ∗ ) −

1 1 log β(1 + C/κ) ≤ Aβ ≤ Uq (ξ ∗ ) − log β 2β 2β

(3.70)

and thus β −1 Ψβu (q +

1 X , q, m) − E sup Uq (ξ) ≤ log β(1 + C/κ) . β β ξ

(3.71)

The r.h.s. is independent on q, X, m impliying also uniform convergence if X ≤ κ−1 . Note that this was possible only thanks to uniform bounds on the derivatives of u. The second part of the statement is analogous a part from minor fixes. One just needs to prove that Z   X 1 dξ β u(λym+√ρv+√Xξ)− ξ22 β→∞ −1 −−−→ 0 , (3.72) β Ψβu (ρ, ρ − , m) − E log √ e β β 2π uniformly over variational parameters. This follows from a uniform Lipschitz property of Ψβ . Define Z   q √ 2 1 dξ v+ Xξ)− ξ2 β u(λym+ ρ− tX β g(t) := E log p , (3.73) e β 2π/β whence it is evident that g(1) = β −1 Ψβu (ρ, ρ − Xβ , m), whereas g(0) is our target op to O(β −1 log β). Deriving w.r.t t we get ġ(t) = −

with ν(t) := λym +

X q E v ⟨u′ (ν(t))⟩βu tX 2β ρ − β

(3.74)

q p ρ − tX v + X/βξ. Note that in the square root in front of ξ, X β

has been divided by β to reuse the definition of ⟨·⟩βu . An integration by parts yields ġ(t) = −

2 X X E ⟨u′′ (ν(t))⟩βu − E u′ (ν(t)) − ⟨u′ (ν(t))⟩2βu βu . 2β 2

(3.75)

Thanks to H1 and that X ∈ [0, κ−1 ], the first term is uniformly bounded by an O(β −1 ). For the second term, since it is a variance, we can use Brascamp-Lieb inequality for ⟨·⟩βu : E u′ (ν(t))

2

− ⟨u′ (ν(t))⟩2βu βu ≤ 20

XC 2 C2 ≤ β κβ

(3.76)

where we used again |u′′ | ≤ C. (3.72) has thus been proved. The rest of the proof proceeds as for the first claim. For the third limit, call σ ∗ = arg inf σ Hϕ (σ). Observe that −K := −(κ + C(1 + α)) ≤ −(κ + δ + C) ≤ −Hϕ′′ (σ) ≤ −κ, where we used δ ∈ [0, αC]. Hence −

κ K (σ − σ ∗ )2 ≤ −Hϕ (σ) + Hϕ (σ ∗ ) ≤ − (σ − σ ∗ )2 2 2

(3.77)

which in turn yields βK 1 1 + sup −Hϕ (σ) ≤ log − log 2β 2π β σ

Z

dσe−βHϕ (σ) ≤ −

1 βκ log + sup −Hϕ (σ) . (3.78) 2β 2π σ

Finally β −1 Λϕ (r, δ, h) − E sup −Hϕ (σ) ≤ σ

1 βK log . 2β 2π

(3.79)

This is again uniform in the variational parameters r, δ, h, provided δ ∈ [0, αC]. Lemma 5. The following scalings hold X β→∞ , m, r, δ, h) −−−→ Φ̃u (X, ρ, m, r, δ, h) β

(3.80)

X β→∞ , q, m, r, δ, h) −−−→ Φ̃u (X, q, m, r, δ, h) β

(3.81)

β −1 Φu (ρ, ρ − β −1 Φu (q +

uniformly over the variational parameters if δ ∈ [0, αC], X ∈ [0, κ−1 ]. Furthermore, Φ̃ is jointly concave in (m, X). Proof. By direct inspection: X δρ rX α 1 1 Φu (ρ, ρ − , m, r, δ, h) = − + Ψβu (ρ, ρ − β −1 X, m) + Λϕ (r, δ, h) − λmh . β β 2 2 β β (3.82) Using then Lemma 4 we get the first scaling. The second is completely analogous. Note that Φu (ρ, ρ − Xβ , m, r, δ, h) is the composition of Φ with an affine transformation in the q-entry, w.r.t. which Φ is strictly concave, jointly with m. Hence, it is jointly concave in (m, X), and so it remains when divided by β. Therefore, being Φ̃ the limit of a jointly (m, X)-concave function, it remains jointly concave in (m, X). Remark 8. Since by Lemma 5 we have uniform convergence of the variational potential over the allowed optimization sets we also have that lim sup inf sup inf β −1 Φu (ρ, ρ −

β→∞

ρ

r m,X δ,h

lim inf sup sup inf β −1 Φu (q +

β→∞ r

q

m,X δ,h

X , m, r, δ, h) = sup inf sup inf Φ̃u (X, ρ, m, r, δ, h) r m,X δ,h β ρ (3.83)

X , q, m, r, δ, h) = inf sup sup inf Φ̃u (X, ρ, m, r, δ, h), r β ρ m,X δ,h (3.84)

21

where we omitted the optimization sets for brevity. However, keep in mind that the set m2 ≤ q + X/β → m2 ≤ q uniformly over the variational parameters other than q for β → ∞, thanks to the fact X ∈ [0, κ−1 ]. Note that in the last line we have re-baptized q as ρ, as it is a dummy variable, to have a more symmetric formula. Indeed, from a physical point of view, the self overlap ρ and the overlap q become equal in the β → ∞ limit. Lemma 6. The following estimate holds −eN −

βK pN 1 βκ 1 log ≤ ≤ −eN − log 2β 2π β 2β 2π

(3.85)

for a proper K > κ. As a consequence pN = lim inf −eN , N →∞ β→∞ N →∞ β

lim lim sup

lim lim inf

β→∞ N →∞

pN = lim sup −eN . β N →∞

(3.86)

Proof. We start with a simple expansion of the Hamiltonian with Lagrange remainder, around its minimum point σ ” : 1 −HN (σ) = −HN (σ ∗ ) − (σ − σ ∗ ) · Hess HN (σ̄)(σ − σ ∗ ) 2

(3.87)

where −HessHN (σ) =

X gµ gµ⊺ µ

N

u′′yµ (Sµ ) − diag(ϕ′′ (σi ))i≤N − 1κ .

(3.88)

Using H1, the convexity of ϕ and concavity of uy we get −1(κ + C + Cλmax (G⊺ G/N )) ⪯ −HessHN (σ) ⪯ −1κ .

(3.89)

From these bounds we readily get Z β(κ+C+Cλmax (G⊺ G/N )) 1 pN ∥σ−σ ∗ ∥2 2 ≥ −eN + E log dN σe− β Nβ Z βκ pN 1 ∗ 2 ≤ −eN + E log dN σe− 2 ∥σ−σ ∥ . β Nβ

(3.90) (3.91)

The result then follows by Gaussian integration. In particular, Z β(κ+C+Cλmax (G⊺ G/N )) 1 1 β(κ + C + Cλmax (G⊺ G/N )) ∥σ−σ ∗ ∥2 2 E log dN σe− = − E log Nβ 2β 2π 1 β ≥ − log (κ + C + CEλmax (G⊺ G/N )) . (3.92) 2β 2π Since Eλmax (G⊺ G/N ) is bounded by a constant depending on M/N , the lower bound is proved. The upper bound is proved with another simple Gaussian integration. The proof of Proposition 2 then follows.

22

3.5

Training and generalization errors

Proof of Corollary 1. Define the following auxiliary problem: 1 p̄N (β, λ) = E log N

Z

N g · σ  X κ∥σ∥2 i µ − ϕ(σi ) − d σ exp β uyµ √ β 2 N µ=1 i=1 M hX β

t

N

(3.93)

The above, after denoting ūy = uy βt /β, is again a perceptron model, which can be treated exactly as in Theorem 1. Moreover, the training error can be identified as −εN,t =

1 ∂ p̄N , α ∂βt βt =β

(3.94)

and p̄N is convex in βt . In addition, 0≤

2 D X
E2 E 1 D X ∂ 2 p̄N S S = E u − u µ0 µ0 yµ yµ ∂βt2 N µ µ

(3.95)

Using Brascamp-Lieb inequality one readily gets M M D 1 X D 1 X E GG⊺ ∂ 2 p̄N gµ · gν E ′ ′ ′ 2 ≤ E u (S )u (S ) ≤ E (u (S )) ∥ ∥op . µ0 ν0 µ0 yν ∂βt2 κN µ,ν=1 yµ N κN µ=1 yµ N

(3.96) ⊺

Recall that ∥ GG ∥op ≤ C(α) with probability exponentially close to 1. On the other N hand, thanks to H1, (u′y (s))2 ≤ 2C 2 (1 + s2 ). Using this, combined with Proposition 6 we conclude that ∂β2t p̄N is bounded. This implies directly that ∂βt p̄N converge [49, Theorem 1.1]. Therefore, the exchange of the N -limit and the βt derivative is allowed: −εt =

1 ∂ p̄N 1 ∂ p̄ lim = . N →∞ α ∂βt βt =β α ∂βt βt =β

(3.97)

Under the hypothesis of the corollary: p̄ = sup inf sup inf Φū (ρ, q, m, r, δ, h). ρ

r

q,m δ,h

(3.98)

Hence, ∗

Eξ uy (ν ∗ )eβuy (ν ) , −εt = α ∂βt p̄|βt =β = E Eξ eβuy (ν ∗ ) −1

ν ∗ = λym∗ +

√ ∗ √ q v + ρ∗ − q ∗ ξ

(3.99)

with ρ∗ , q ∗ , m∗ and r∗ , δ ∗ , h∗ solving the variational problem at βt = β. For the proof of Corollary 2 we first need to introduce an auxiliary model. Fix a positive number ϵ > 0 and consider a model with additional training data M +⌊ϵM ⌋ h N g · σ  g · σ   g · σ i X X κ∥σ∥2 µ µ µ −H̃N (σ) = u yµ √ +γ tWyµ √ + u yµ √ − ϕ(σi ) − . 2 N N N µ=1 i=1 µ=M +1 M X

(3.100)

23

γ ∈ [0, η) here is chosen as in the statement of Corollary 2 and t ∈ [−η, η]. The above can also be recast as ( M +⌊ϵM ⌋ N g · σ  X 2 X uy if µ ≤ M κ∥σ∥ µ ũµyµ √ , ũµy = −H̃N (σ) = − ϕ(σi ) − 2 γ(uy + tWy ) otherwise. N µ=1 i=1 (3.101) From the above we can define also a quenched pressure per particle Z 1 p̃N (t, γ) := E log dN σe−β H̃N (σ) . N From the above definitions, it is not difficult to verify that  g′ · σ  1 1 ′ √ εW := E W = lim+ ∂t p̃N (t = 0, γ) = ∂t ∂γ p̃N (t = 0, γ = 0) . (3.102) y g,N γ→0 βαϵγ βαϵ N We shall thus need to adapt Theorem 1 to p̃N . Lemma 7. Under the hypotheses of Corollary 2 one has N →∞

p̃N −−−→ p̃ := sup inf sup inf Φũ,γ (ρ, q, m, r, δ, h) ρ

r

q,m δ,h

(3.103)

with Φũ,γ defined in (2.22) and α′ = ϵα. Proof. The proof is identical to that of Theorem 1. For uniform convergence, it suffices to check that the remainder is uniformly bounded in t ∈ [0, 1]. We are now ready for the proof. Proof of Corollary 2. For reasons completely analogous to those discussed for p̄N , defined in (3.93), ∂γ2 p̃N is bounded by a constant. Furthermore, p̃N is convex in t. Therefore, for any s > 0 p̃N (t, γ) − p̃N (t − s, γ) ∂t p̃N (t, γ) p̃N (t + s, γ) − p̃N (t, γ) ≤ ≤ . γs γ γs

(3.104)

We can now expand p̃N up to second order, obtaining p̃N (t, γ) = p̃N (t, 0) + γ∂γ p̃N (t, 0) + O(γ 2 ) ,

(3.105)

thanks again to the fact that ∂γ2 p̃N ≤ C̄ uniformly in t ∈ [−η, η], and hence its derivative converges. Observe also that p̃N (t, 0) = p̃N (0, 0) is actually independent on t, leading to −C̄

γ ∂γ p̃N (t, 0) − ∂γ p̃N (t − s, 0) ∂t p̃N (t, γ) ∂γ p̃N (t + s, 0) − ∂γ p̃N (t, 0) γ + ≤ ≤ + C̄ . s s γ s s (3.106)

After sending γ → 0 we get ∂γ p̃N (t, 0) − ∂γ p̃N (t − s, 0) ∂γ p̃N (t + s, 0) − ∂γ p̃N (t, 0) ≤ ∂γ ∂t p̃N (t, 0) ≤ . s s 24

(3.107)

Now, recall that from the lower bound
lim inf ∂γ ∂t p̃N (t, 0) ≥ s−1 ∂γ p̃(t, 0) − ∂γ p̃(t − s, 0) N →∞

(3.108)

where we have again exchanged limit and derivative. Analogously
lim sup ∂γ ∂t p̃N (t, 0) ≤ s−1 ∂γ p̃(t + s, 0) − ∂γ p̃(t, 0) .

(3.109)

N →∞

By letting s → 0+ we then get two matching bounds, leading to lim lim γ −1 ∂t p̃N (0, γ) = ∂γ ∂t p̃ .

N →∞ γ→0+

(3.110)

The computations of the derivatives yield ′

Eξ Wy′ (ν ′ )eβγuy′ (ν ) ∂t p̃(0, γ) = βϵαγE , ′ Eξ eβγuy′ (ν )

∂γ ∂t p̃(0, 0) = βαϵEEξ Wy′ (ν ′ ) ,

(3.111)

√ ∗ ′ √ ∗ q v + ρ − q ∗ ξ).

(3.112)

which in turn finally entails W ′ ∗ lim εW g,N =: εg = EEξ Wy ′ (λy m +

N →∞

4

Numerical checks in notable learning settings

As already noted in Section 2, the two bounds in Theorem 1 differ only in the order of the optimizations over ρ and r, namely in the exchange of supρ and inf r . The optimization over these two variables is therefore the only possible source of a discrepancy between the two bounds. Since we have not yet established their equality under sufficiently general assumptions, we provide numerical evidence that the two optimization orders can be exchanged in several learning regimes of interest in optimization and machine learning. To this end, we optimize the variational potential over q, m, δ, and h at fixed values of ρ and r, thus obtaining the reduced variational potential (2.14) for different values of α, β, and κ, as well as for different regularization schemes, namely choices of ϕ. We chose these parameters to cover the main learning regimes previously investigated using statistical-physics methods [16, 17]. We then plot the resulting surface in the (ρ, r) plane and examine its critical-point structure. In particular, we investigate the existence of a saddle point, thereby providing numerical evidence that the supρ and inf r operations can be interchanged. Figure 1 shows Φ⋆u for the logistic, or cross-entropy, loss with L2 regularization. In this setting, the model is known to exhibit a learning transition at the threshold α = 2, above which the model cannot perfectly fit the training set [16]. Accordingly, the first and second rows display Φ⋆u below and above this threshold, for α = 1 and α = 3, respectively. The first two columns compare a finite-temperature regime, with β = 1, and a low-temperature regime, with β = 100. The latter approximates the zerotemperature limit, which captures the asymptotic behavior of optimization algorithms. Indeed, as β → ∞, the Gibbs measure concentrates on the ground-state configurations of the Hamiltonian. For the convex optimization problem considered here (logistic regression 25

−40

0

1

2

ρ

3

4

5

0.0

0.2

0.4

0.6

0.8

−50 −60

Φu⋆

0.3

−70

0.2

−90 0

0.0

1

2

ρ

r

(a) α = 1, β = 1, κ = 1.

3

4

5

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

0

1

2

ρ

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

Φu⋆

−0.8

−1.2 −1.4 −1.6

r

(d) α = 3, β = 1, κ = 1.

−70

20 0 −20 −40 −60

−80

0

−90

2

4

ρ

6

8

10

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

0

1

2

ρ

3

4

5

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

−160

−200 −220

(e) α = 3, β = 100, κ = 1.

20 0 −20 −40

−40

−25 −50 −75 −100 −125 −150 −175 −200

−140

−180

40

(c) α = 1, β = 100, κ = 0.01.

−120

−100 −120 −140 −160 −180 −200 −220

−0.6

−1.0

−60

(b) α = 1, β = 100, κ = 1.

−0.4

−0.25 −0.50 −0.75 −1.00 −1.25 −1.50 −1.75

40

−80

0.1

1.0

60

Φu⋆

−40

0.4

Φu⋆

0.6 0.5 0.4 0.3 0.2 0.1 0.0 −0.1

60

−50

0

1

2

ρ

3

4

5

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

−60 −80 −100

Φu⋆

0.5

Φu⋆

0.6

−120 −140 −160 −180

(f) α = 3, β = 100, κ = 0.01.

Figure 1: Numerical surfaces of the reduced variational potential Φ⋆u (ρ, r) for different choices of the model parameters, with logistic utility uy (s) = − log (1 + e−ys ), no additional separable regularization, ϕ(σi ) = 0, and Gaussian centroid distribution θ ∼ N (0, IN ). λ = 1 in all plots. The inner minimization over (δ, h) and the outer maximization over (q, m) are both performed using the L-BFGS-B method from scipy.optimize. In the degenerate case r = 0, where the constraint h2 ≤ αr forces h = 0, the remaining scalar minimization over δ is performed using the bounded method from scipy.optimize. For both optimization routines, the tolerance is set to 10−9 and the maximum number of iterations to 100. The surface is evaluated on a 50 × 50 grid in the (ρ, r) plane. with L2 regularization) these ground states coincide with the global minimizers to which gradient-based optimization methods converge. For weak regularization, the generalization error of this learning model is also known to exhibit a peak near the interpolation threshold, indicating overfitting [16]. This peak gradually disappears as the regularization strength κ increases. The last two columns of Figure 1 therefore compare a relatively strong, nearly optimal regularization regime, with κ = 1, and a weakly regularized regime, with κ = 0.01, in which the peak begins to emerge. Across all the parameter regimes considered, the surface Φ⋆u exhibits a saddle structure in the variables ρ and r. This supports that the two optimizations supρ and inf r can be interchanged and, consequently, that the lower and upper bounds in Theorem 1 coincide in these regimes. Figure 2 shows that the same saddle-point structure persists under smooth L1 regularization, across different values of α and at different temperatures. This provides further numerical evidence that the lower and upper bounds coincide in these regimes.

26

0.8

−20

0.7

1

2

ρ

3

4

5

0.6

0.5

0.4

0.4

0.2 0.0 0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

−30 −40

−40

−50

0.3

−60

−60

0.2

−80

−70

Φu⋆

−20

0.1

0

1

ρ

(a) α = 1, β = 1, κ = 1e − 4.

0

1

2

ρ

3

4

5

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

3

4

5

−80

(b) α = 1, β = 100, κ = 1e − 4.

0.2

−40

0.0

−60

−0.2 −0.4 −0.6

Φu⋆

0.25 0.00 −0.25 −0.50 −0.75 −1.00 −1.25

2

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

Φu⋆

0.6

−50

−80

−100

−100

−150

−0.8

−200

−1.0

0

−1.2

1

2

ρ

(c) α = 3, β = 1, κ = 1e − 4.

3

4

5

0.30 0.25 0.20 0.15 0.10 0.05 r 0.00

−120

Φu⋆

0

0.8

−140 −160 −180

(d) α = 3, β = 100, κ = 1e − 4.

Figure 2: Numerical surfaces of the reduced variational potential Φ⋆u (ρ, r) for different choices of the model parameters, with logistic utility uy (s) = − log (1 + e−ys ), no additional separable regularization, ϕ(σi ) = log cosh σi , and Gaussian centroid distribution θ ∼ N (0, IN ). λ = 1 in all plots. The parameters of the optimization in δ, h, q and m are the same as in Figure 1.

5

Conclusions and perspectives

In this work we developed a variational approach to a finite-temperature continuous-spin perceptron trained on Gaussian-mixture data. Under log-concavity assumptions, adaptive interpolation yields lower and upper bounds for the limiting quenched pressure that differ only in the order of two outer scalar optimizations. The remaining extrema are controlled by the concave–convex structure of a single potential, whose stationarity conditions recover the corresponding fixed-point equations. The same framework also gives access to the zero-temperature limit and, under additional regularity and uniqueness assumptions, to training and generalization observables. Beyond the specific model considered here, the main methodological contribution is therefore to organize the thermodynamics of the perceptron within a global variational structure. A natural question is whether the present strategy can be extended to multilayer architectures. The main obstruction is that the composition of several layers generally 27

destroys the log-concavity properties used here to prove concentration of the relevant overlaps and to control the interpolation remainder. The single-layer analysis nevertheless shows that the loss of positivity does not by itself rule out a variational description: in the present setting, the convex–concave structure of the potential provides an alternative organizing principle. A multilayer extension would therefore require either a different mechanism for identifying and concentrating the relevant overlaps, or a regime in which enough of this convex structure survives for the order parameters to be incorporated into a single variational potential. Whether this can occur is likely to depend on the architecture, scaling, and phase of the model. Low-rank multilayer perceptrons, in which the weight matrices have a rank assumed to be fixed and not scaling with the system’s size, provide one possible setting in which such a reduction may remain tractable. Preliminary, non-rigorous replica calculations in the proportional limit, where the network layers and the number of training data diverge proportionally (see [50, 41, 51, 52, 18, 53]), suggest that the problem may close in terms of a finite collection of overlaps. This perspective is also consistent with numerical observations on trained neural networks: the singular-value spectrum of the first-layer weight matrix exhibits a small number of isolated singular values above an approximately Gaussian-like bulk [54, 55]. Removing the bulk appears to leave classification performance nearly unchanged. These observations do not establish a general low-rank principle, but they suggest that an effective finite-dimensional structure may capture a substantial part of the learned representation and motivate a systematic study of this regime.

A

Lemmata

A.1

Existence criterion for ODEs

Lemma 8. Consider a system of ODEs ẋ = f (t; x) with x ∈ Rn with t ∈ [0, 1] and x(0) = x0 , with x20 ≥ 0. Suppose the velocity field is continuous in t and x and that 0 ≤ f1 (t; x)2 ≤ f2 (t; x)

(A.1)

Then, if ∥f (t, x)∥ ≤ K

1 + x21 +

n X

|xi |

(A.2)

i≥2

the system admits a global solution in t ∈ [0, 1]. Proof. Define the function W (t) = K(1 + x21 (t) + x2 (t) +

n X

|xi (t)|) .

(A.3)

i=3

The function x2 (t) is always nonnegative since its initial condition and the related component of the velocity field are both nonnegative. Denote the upper Dini derivative as D+ g(t) := lim sup h↓0

28

g(t + h) − g(t) . h

(A.4)

Then D+ |xi (t)| ≤ |ẋi (t)| = |fi |. Moreover +

D W ≤ K(2x1 ẋ1 + ẋ2 +

n X

|ẋi |) = K(2x1 f1 + f2 +

i=3

n X

|fi |) ≤ K(x21 + f12 + f2 +

n X

|fi |).

i=3

i=3

(A.5) By definition of W one has K −1 W ≥ x21 ,

W ≥ |fi | ∀3 ≤ i ≤ n,

W ≥ f2 ≥ f12 .

(A.6)

Hence D+ W (t) ≤ W (1 + nK)

(A.7)

which by Grönwall inequality yields W (t) ≤ e(1+nK)t W (0). Therefore, W (t) remains bounded in finite time, which is enough to guarantee existence of a global solution. Remark 9. The above Lemma generalizes to the presence of any number of constraints of the type (A.1).

A.2

Moment generating function estimates

Let Y be a random variable with a density of the form Z κ 2 1 −ϕ(y)− κ y2 2 e−ϕ(y)− 2 y dy pY (y) = e , cκ = cκ R

(A.8)

where κ > 0, and ϕ is a L-Lipschitz function. Notice that one can assume without loss that ϕ(0) = 0. Lemma 9. For any z ∈ R one has r r   z2 1 2π − L2 + 2(κ+L) 2 2π 2κ1 L2 +z2 (1+ Lκ2 ) zY e e ≤ EY e ≤ cκ κ + L cκ κ

(A.9)

Proof. Since ϕ is Lipschitz and ϕ(0) = 0 one has ϕ(y) ≥ −L|y| for every y ∈ R, hence r   1 2π EY exp(zY ) ≤ Eg∼N (0,κ−1 ) exp zg + L|g| cκ κ r h i 1 2π ≤ Eg∼N (0,κ−1 ) exp((z + L)g) + Eg∼N (0,κ−1 ) exp((z − L)g) cκ κ r   (z−L)2 1 2π (z+L)2 2κ 2κ = e +e cκ κ r  
2 2π 1 zL 2 2 = exp L +z cosh( ) cκ κ 2κ κ r    2 2
1 Lz 2 2π ≤ exp L2 + z 2 exp cκ κ 2κ 2κ2 On the other hand −ϕ(y) ≥ −L|y| and then 29

(A.10)

r 1 2π Eg∼N (0,κ−1 ) exp(zg − L|g|) EY exp(zY ) ≥ cκ κ r 1 2π L ≥ Eg∼N (0,κ−1 ) exp(zg − (g 2 + 1)) cκ κ 2 r 2 z 1 2π − L2 + 2(κ+L) = e cκ κ + L

A.3

(A.11)

Random matrices

Here some basic facts on random matrices. For the proof see for example [27, 56]. If A is an M × N matrix we denote by • Frobenius norm: ∥A∥F =

M X N X

!1/2 A2k,i

.

k=1 i=1

• Operator norm: ∥A∥op = sup ∥Ay∥. ∥y∥≤1

Recall also that ∥A∥op ≤ ∥A∥F . i.i.d.

Consider Z = √1N Z where Z = (Zµ,i )µ≤M i≤N and Zµ,i ∼ N (0, 1). Lemma 10. For every δ > 0, P ∥Z∥op ≥ 1 +

√

N δ2 α + δ ≤ 2 exp − 2



 ,

where α = M . Therefore, for every p ∈ N, N   E ∥Z∥pop < ∞.

A.4

(A.12)

(A.13)

Concentration inequalities

Lemma 11 (Tensorization). Let X1 ⊥ X2 be independent. Suppose that, for every fixed x2 and every smooth h,
  VarX1 h(X1 , x2 ) ≤ C1 EX1 ∥D1 h(X1 , x2 )∥2 , and similarly for block 2 with constant C2 . Then for every smooth F
Var F (X1 , X2 ) ≤ C1 E∥D1 F ∥2 + C2 E∥D2 F ∥2 .

30

Lemma 12 (Rademacher-Poincarè inequality). Let X = (X1 , . . . , Xn ) be i.i.d. Rademacher random variables and let f : {−1, 1}n → R. For x ∈ {−1, 1}n and i ∈ {1, . . . , n}, let x+i (resp. x−i ) denote x with its i-th coordinate set to +1 (resp. −1), and define the i-th discrete derivative f (x+i ) − f (x−i ) . (A.14) Di f (x) := 2 Then n X 
 Var f (X) ≤ E (Di f (X))2 . i=1

Proof. Since X1 , . . . , Xn are mutually independent, then for each coordinate (with the other coordinates frozen), i.e. for every fixed x−i and every h, 
 VarXi h(Xi , x−i ) ≤ EXi (Di h(Xi , x−i ))2 , which is exactly the Poincaré hypothesis for block i with constant Ci = 1 in the tensorization theorem proved previously. Applying that theorem with C1 = · · · = Cn = 1 gives directly n X   Var(f (X)) ≤ E (Di f (X))2 . i=1

B

Consequences of Brascamp-Lieb and Cramer-Rao inequalities

We start recalling the main definitions of the interpolating Hamiltonian using a more compact notation. Fix the values of a collection of real parameters ty , tθ , tm ∈ R and tZ , tz , tv , tξ ≥ 0, let G = (gµ )µ≤M = (gµ,i )i≤N µ≤M with 1 gµ = √ ty yµ θ + tZ Zµ N

(B.1)

where y = (yµ )µ≤M , Z = (Zµ )µ≤M and θ = (θi )i≤N are defined in (2.2). Consider also two families of independent standard Gaussian vectors z = (zi )i≤N ,

v = (vµ )µ≤M

(B.2)

From now on we will denote rescaled variables with an underline, as example G = N −1/2 G, z = N −1/2 z and v = N −1/2 v. Given a realization of D = (y, Z, z, v, θ) the (random) Hamiltonian we will consider is defined on the configurations space RN × RM ∋ x = (σ, ξ) = ((σi )i≤N , (ξµ )µ≤M ) as −H(x) =

X

u (Sµ (x)) −

µ≤M

κ 1 M log(2π) + (tz z + tθ θ) · σ (B.3) ϕ(σi ) − ∥σ∥2 − ∥ξ∥2 − 2 2 2 i≤N

X

where κ > 0 and 1 Sµ (x) = √ gµ · σ + yµ tm + tv vµ + tξ ξµ , N 31

µ≤M

(B.4)

Remark 10. In order to lighten the notation we dropped the dependency of the function u from the realization of yµ , namely in this section u ≡ uyµ . It’s clear that given t ∈ [0, 1] there is a choice of κ and all the t• such that (B.3) equals the interpolating Hamiltonian HNt in (3.1). p κ → κ + δ(t) tξ = ρ(t) − q(t) ty = (1 − t)λ tm = λm(t) p p (B.5) √ tθ = λh(t) tZ = 1 − t tz = r(t) tv = q(t). We will work with a more general Hamiltonian than (B.3), defined as follows. Consider two copies σ = (σ 1 , σ 2 ), ξ = (ξ 1 , ξ 2 ) and set x1 = (σ 1 , ξ 1 ) x2 = (σ 2 , ξ 2 ), and x = (x1 , x2 ) ∈  2 RN × RM . Consider an Hamiltonian function coupling the two x1 , x2 copies trough the overlap between σ 1 , σ 2 : Ha (x) = H(x1 ) + H(x2 ) + a (σ 1 · σ 2 )

(B.6)

where a ∈ R is a parameter and H(x) is defined in (B.3). Given a realization of G, v, z we denote by ⟨ ⟩(a) the average w.r.t. the induced (random) Gibbs measure νa (x)dx =

1 −Ha (x) e dx Za

(B.7)

Remark 11. Notice that by definition with the choice (B.5) one has ⟨·⟩(a) |a=0 ≡ ⟨·⟩t where ⟨·⟩t is defined in (B.7) and represents the average w.r.t. νt⊗2 , namely the (product)measure induced by the interpolating Hamiltonian. Clearly this correspondence holds for β = 1, however the general case cab easily recovered. Lemma 13 (Log-concavity). If κ − |a| > 0 then the function x 7→ Ha (x)

(B.8)

is strictly convex and then   νa is log-concave measure. Moreover for any x ∈ RN × RM

2

one has

xT HessHa x ≥ (κ − |a|)∥σ∥2 + ∥ξ∥2
xT HessHa x ≤ 2C∥G∥2op + κ + C + |a| ∥σ∥2 + (2Ct2ξ + 1)∥ξ∥2

(B.9)

Proof. By direct computation 2  ⃗g · σ ℓ X Xh µ + tξ ξµℓ u′′ (Sµ (xℓ )) √ N ℓ=1,2 µ≤M i + (σ ℓ )T Hessϕ σ ℓ + κ∥σ ℓ ∥2 + ∥ξ ℓ ∥2 + 2a(σ 1 · σ 2 )

xT HessHa x = −

(B.10)

Now since u is concave and ϕ is convex one has xT HessHa x ≥ (κ − |a|)∥σ∥2 + ∥ξ∥2

(B.11)

On the other hand H3 implies that u′′ , −ϕ′′ ≥ −C and then xT HessHa x ≤

2 X X h  gµ · σ ℓ + tξ ξµℓ + (κ + C + |a|)∥σ∥2 + ∥ξ∥2 C √ N ℓ=1,2 µ≤M

≤ (2C∥G∥2op + (κ + C + |a|)∥σ∥2 + (2Ct2ξ + 1)∥ξ∥2

32

(B.12)

We are now ready to prove the following: Proposition 5 (Control of the variance). Assume that the functions u, ϕ satisfies hypothesis H1-H4, then given t ∈ [0, 1] and the choice of Pt and for any realization of D = (y, Z, z, v, θ) one has 

2C∥G∥2op + κ + δ(t) + C

−1

D E ≤ Q11 − Q12 ≤ t

1 κ + δ(t)

(B.13)

where G = √1N G. Proof. Keeping in mind the choice (B.5) one D E D E Q11 − Q12 = Q11 − Q12 t

= (0)

1 X Var0 (σi ) N i≤N

(B.14)

where Vara denotes the variance w.r.t. the random Gibbs measure νa associated to the Hamiltonian Ha . We know from Lemma 13 that if κ − |a| > 0 then νa is log-concave the Hessian is bounded by (B.9). Therefore νa |a=0 is the a product measure with log-concave marginals with potential H. Hence from Brascamb-Lieb bound one has D E 1 (B.15) Var0 (σi ) ≤ [HessH ]−1 ≤ , ∀i ≤ N ii κ (0) On the other hand by Cramer-Rao bound one has D E −1  −1 Var0 (σi ) ≥ HessH ≥ 2C∥G∥2op + κ + C

∀i ≤ N

(B.16)

(0) ii

Clearly (B.15) and (B.16) implies (B.13).

C

Bounds on the order parameters

Let Ha the Hamiltonian in (B.6). Let fa be the free energy associated to Ha , its expectation is Z
1 1 Efa = E log Za = E E log dx exp − Ha (x) , (C.1) N N where the expectation E is taken over all sources of quenched randomness D = (y, Z, v, z, θ). Lemma 14 (Bounds for fa ). If the functions u, ϕ satisfies hypothesis H1,H2 and H3, and if |a| is small enough then for any realization of D one has   2 2 2 2 2 2 −c1 (1 + ∥G∥op + tm + tξ + tv ∥v∥ ) ≤ fa ≤ c2 1 + ∥tz z + tθ θ∥ (C.2) for some c1 , c2 > 0. Proof. We start noticing that since u ≤ 0 and 2|σ 1 · σ 2 | ≤ ∥σ∥2 one has −Ha (x) ≤ −

X ℓ=1,2

ϕ(σiℓ ) +

−κ + |a| ℓ 2 1 ℓ 2 ∥σ ∥ − ∥ξ ∥ + (tz z + tθ θ) · σ ℓ 2 2

33

(C.3)

and then Za ≤

YZ i≤N

 !2 −κ + |a| 2 dσi exp −ϕ(σi ) + σi + (tz zi + tθ θi )σi 2

Therefore if a is such that −κ + |a| < 0 , by Lemma 9 one has     √ 2N N L2 2 κ 2 2 exp L + (1 + )∥tz z + tθ θ∥ Za ≤ √ τ τ τ where τ = κ − |a|. Therefore (C.2) holds with   L2 1 L2 c2 = max log(4κ/τ ), , + 2 τ τ τ On the other hand the hypothesis H1 on the function u implies that  1 2  X X  u(Sµ (x)) ≥ − C 1 + √ gµ · σ + yµ tm + tv vµ + tξ ξµ N µ≤M µ≤M   2 2 2 2 2 2 2 ≥ −4CN α/4 + ∥G∥op ∥σ∥ + αtm + tξ ∥ξ∥ + tv ∥v∥

(C.4)

(C.5)

(C.6)

(C.7)

Hence   1 X −Ha (x) ≥ −8N C α/4 + αt2m + t2v ∥v∥2 − (1 + 8Ct2ξ )∥ξ ℓ ∥2 − M log(2π) 2 ℓ=1,2 " # X X − R ∥σ ℓ ∥2 − ϕ(σiℓ ) − (tz z + tθ θ) · σ ℓ ℓ=1,2

(C.8)

i≤N

and then where R = 4C∥G∥2op + κ+|a| 2 Z Z  2 dy − 1 (1+8Ct2ξ )y2 2M  Y −8CN (α/4+αt2m +t2v ∥v∥2 ) √ e 2 Za ≥ e dσi exp(−Rσi2 − ϕ(σi ) + (tz zi + tθ θi σi ) 2π i≤N      2π N −N 2Cα+2Cαt2m +4Ct2v ∥v∥2 +α log(1+8Ct2ξ ) ∥tz z + tθ θ∥2 exp −N (L − ) ≥e R+L (R + L) (C.9) where the last inequality is a consequence of Lemma 9. Therefore 1 log Za ≥ −2Cα−2Cαt2m −4Ct2v ∥v∥2 −α log(1+8Ct2ξ )+log(2π)−L−log(R+L) (C.10) N Now in order to obtain the explicit constant c1 in (C.2) one can use the inequality − log(x+ y) ≥ − log(x) − xy valid for x, y > 0 obtaining 1 R log Za ≥ (−2Cα + log 2π − L − log L) − 2Cαt2m − 4Ct2v ∥v∥2 − 8αCt2ξ − N L therefore

(C.11)

n κ + |a| 4C o , 2Cα, 8αC, (C.12) c1 = max 2Cα − log 2π + L + log L + 2L L Clearly the quantity (C.12) is meaningless for L = 0 that corresponds to the case ϕ ≡ 0. This latter case be treated directly replacing (9) with the exact moment generating function of a Gaussian r.v. 34

Now we apply the same strategy to gain control on the norm of the vector x w.r.t.
the Gibbs measure νa by means of its moment generating function s 7→ exp s∥x∥2 (a) . Lemma 15 (Moment generating function of the norms ). There exist a number ε > 0 and a costant Kx > 0 such that, for |s|, |a| ≤ ε,
exp s∥x∥2 (a) ≤ exp (N Kx BN ) (C.13) where BN ≡ BN (G, v, tz z + tθ θ, tm , tξ , tv ) = 1 + ∥G∥2op + t2m + t2ξ + t2v ∥v∥2 + ∥tz z + tθ θ∥2 (C.14) The constant Kx ≡ Kx (κ, α, C, L, a) is continuous at any κ > 0. Remark 12. The constant Kx in the previous statement is a key quantity for the rest of the proof so from now on Kx denotes always this constant. P Proof. Notice that by definition ∥x∥2 = ∥σ∥2 + ∥ξ∥2 = ℓ=1,2 ∥σ ℓ ∥2 + ∥ξ ℓ ∥2 and exp s∥x∥

2

1 = (a) Za

Z

2

dx e−Ha (x)+s∥x∥

(C.15)

For the denominator, using Lemma 14 one has 
 1 2 2 2 2 2 ≤ exp N c1 1 + ∥G∥op + tm + tξ + tv ∥v∥ . (C.16) Za An upper bound for the numerator can obtained trough the same arguments used in the proof (C.5), but with κ replaced by κs = κ − 2s and the ξ-quadratic coefficient shifted from 1 to 1 − 2s. Therefore i Xh −κs + |a| ℓ 2 1 − 2s ℓ 2 −Ha (x)+s∥x∥2 ≤ ϕ(σiℓ )+ ∥σ ∥ − ∥ξ ∥ +(tz z +tθ θ)·σ ℓ . (C.17) 2 2 ℓ=1,2 Provided 1 − 2s > 0, integrating out ξ ℓ ∈ RM for each of the two copies gives a factor (1 − 2s)−M , and provided τs := κs − |a| = κ − 2s − |a| > 0, Lemma 9 applies to the σ-integral as before, yielding  √ 2N  h i 2 κs N 2  L2  2 L + 1+ ∥tz z + tθ θ∥ ≤ (1 − 2s) exp dx e √ τs τs τs 
 ≤ exp N cs 1 + ∥tz z + tθ θ∥2 (C.18) where, using M = αN ,   L2 1 L2 cs = max −α log(1 − 2s), log(4κs /τs ), , + 2 . (C.19) τs τs τs Z

−Ha (x)+s∥x∥2

−M

The integrability conditions 1 − 2s > 0 and τs > 0 hold for all |s|, |a| ≤ ε provided ε < min

1 κ , . 2 3

35

(C.20)

Set c̄ = sup|s|≤ϵ cs then combining (C.18) with (C.16) one obtains E  D

 2 2 2 s∥x∥2 2 2 2 ≤ exp N c1 1 + ∥G∥op + tm + tξ + tv ∥v∥ + N c̄ 1 + ∥tz z + tθ θ∥ . (C.21) e (a)

Writing A := ∥G∥2op + t2m + t2ξ + t2v ∥v∥2 ≥ 0 and B := ∥tz z + tθ θ∥2 ≥ 0, so that BN = 1 + A + B, and using   c1 (1 + A) + c̄(1 + B) ≤ max(c1 , c̄) (1 + A) + (1 + B) (C.22) = max(c1 , c̄)(2 + A + B) ≤ 2 max(c1 , c̄)(1 + A + B), therefore

D

s∥x∥2

e

E (a)

≤ exp(N Kx BN )

(C.23)

with Kx = 2 max(c1 , c̄), proving (C.13). Notice since cs is continuous in (κ, s) then c̄ is continuous in κ and then same holds for Kx . A direct consequence of the previous result is the following Lemma 16 (Concentration of the norm). There exists ε > 0 such that if |a| ≤ ε one has: !+ * r 2Kx N BN ≤ exp(−Kx N ). (C.24) 1 ∥x∥ ≥ ε (a)

and given p ≥ 1 one has 
 2p 2p 2p 2p 2p + ∥t z + t θ∥ (C.25) ∥x∥2p (a) ≤ Cp,ε 1 + Kxp N p 1 + ∥G∥2p + t + t + t ∥v∥ z θ op m v ξ for some Cp,ε > 0. Proof. Fix s, δ > 0, by exponential Markov inequality, ⟨1(∥x∥ ≥ δ)⟩(a) ≤ e−sδ

2

exp s∥x∥2

(a)

Using Lemma 15 and choosing s = ε and r 2Kx N BN δ= ε

.

(C.26)

(C.27)

yields the first result. By Lemma 3.1.8 in [27] one has that if X ≥ 0 is a r.v. then   EX p ≤ 2p pp + (log EeX )p . (C.28) and then (C.13) and Jensen inequality implies there exists ε > 0 such that for any |s| ≤ ε one has ⟨∥x∥2p ⟩(a) ≤

 2 p  s

2p 2p 2p 2p 2p pp + 6p−1 Kxp N p 1 + ∥G∥2p op + tm + tξ + tv ∥v∥ + ∥tz z + tθ θ∥

 .

(C.29)

hence for s = ε (C.25) holds with Cp,ε = ( 2ε )p max(pp , 6p−1 ). We are now ready to prove the following: 36

Proposition 6. Assume that the functions u, ϕ satisfies hypothesis H1,H2,H3, then given t ∈ [0, 1] and the choice of Pt (i) There exists KQ = KQ (κ, δ(t), α, λ, C, L) > 0 independent on N and continuous w.r.t. the entry δ(t), such that
E⟨Q11 ⟩t ≤ KQ 1 + m2 (t) + h2 (t) + ρ(t) + r(t) . (C.30) (ii) Assuming ρ(t)−q(t) ≤ κ−1 for all t ∈ [0, 1] then there exists KS = KS (κ, δ(t), α, λ, C, L) independent of N and continuous w.r.t. the entry δ(t), such that
E⟨S12 ⟩t , |E⟨S11 ⟩t | ≤ KS 1 + m2 (t) + h2 (t) + ρ(t) + r(t) (C.31) (iii) All the order parameters have finite second moment, namely there exists K2 < ∞ such that 2 2 E S11 , E S12 , E Q211 t , E Q212 t , E H12 t , E M12 t , ≤ K2 t t

(C.32)

uniformly for t ∈ [0, 1] and N . Proof. (i) Let us start with a basic fact: for every p ≥ 1 there is Kp < ∞, independent of N (and obviously also from t and the choice of the path Pt )) , such that E∥θ∥2p , E∥z∥2p , E∥v∥2p , E∥y∥2p , E∥Z∥2p op ≤ Kp .

(C.33)

This follows from standard concentration/moment estimates for i.i.d. Gaussian vectors and matrices. Notice also that ⟨ ⟩t is the average w.r.t. the random Gibbs measure induced by the interpolating Hamiltonian, hence ⟨ ⟩t ≡ ⟨ ⟩(a) |a=0 and all the previous results (in particular Lemma 15 and 16) apply. By definition of Q11 and using (C.25) with p = 1 one gets

∥σ∥2 N

∥x∥2 N

E⟨Q11 ⟩t = E ≤E t t 
 2 2 2 ≤ C0 Kx 1 + E ∥G∥op + tm + tξ + t2v ∥v∥2 + ∥tz z + tθ θ∥2 + O(1/N )

(C.34)

for some absolute constant C0 independent from N and t, Pt and Kx ≡ Kx (κ+δ(t), α, C, L) is the same costant of Lemma 15. It is not difficult to check that ∥G∥2op ≤ 2αt2y ∥y∥2 ∥θ∥2 + 2t2Z ∥Z∥2op ,

∥tz z + tθ θ∥2 ≤ 2t2z ∥z∥2 + 2t2θ ∥θ∥2

(C.35)

and then (C.34) in combination with (C.33) proves (C.30). (ii). Consider the quantity * Σ := E

1 X ′ (u (Sµ ))2 N µ≤M

+ (C.36) t

We claim that from an upper bound for Σ follows directly an upper bound for S11 and S12 . Indeed for any µ ≤ M and replicas x1 , x2 one has 37

 1 ′ 1 2 (u (Sµ )) + (u′ (Sµ2 ))2 (C.37) 2 and hence, taking E⟨ ⟩, using that the two replicas are i.i.d. under the product measure, and summing over µ, one gets u′ (Sµ1 )u′ (Sµ2 ) ≤

E⟨S12 ⟩t ≤ Σ

(C.38)

On the other hand, using H3 (|u′′ | ≤ C) one gets |E⟨S11 ⟩t | ≤ Cα + Σ

(C.39)

Now by H1, (u′ (s))2 ≤ 2C 2 (1 + s2 ), thus entailing + * X 1 Σ ≤ 2C 2 α + 2C 2 E Sµ (x)2 (C.40) N µ≤M t √ Recall that Sµ (x) = gµ · σ/ N + yµ tm + tv vµ + tξ ξµ , therefore, by Cauchy–Schwarz,   1 X Sµ (x)2 ≤ 4 ∥G∥2op ∥σ∥2 + t2m ∥y∥2 + t2v ∥v∥2 + t2ξ ∥ξ∥2 , N µ≤M

(C.41)

√ where σ = σ/ N . We now take E⟨ · ⟩t of (C.41) term by term. The second and the third terms are handled directly     E t2m ∥y∥2 = αt2m , E t2v ⟨∥v∥2 ⟩t = αt2v (C.42) For the last term notice that ∥ξ∥2 ≤ ∥x∥2 ) and then one has the same bound of (C.34) and keeping in mind the hypothesis t2ξ = ρ(t) − q(t) ≤ κ−1 , one gets t2ξ E⟨∥ξ∥2 ⟩t ≤

KQ 1 + m2 (t) + h2 (t) + ρ(t) + r(t) , κ

(C.43)

For the remaining term, ∥G∥2op ∥σ∥2 = ∥G∥2op Q11 , by the Cauchy–Schwarz inequality applied to the expectation over the disorder, one has E ∥G∥2op Q11 t ≤

E∥G∥4op

1/2

E⟨Q211 ⟩t

1/2

.

(C.44)

The factor E∥G∥4op is bounded by a constant independent of N and t, Pt . The second factor is controlled by the p = 2 case of (C.25) which gives     ∥x∥4 ∥σ∥4 2 ≤E E⟨Q11 ⟩t = E N2 t N2 t (C.45) 
 ′ 2 4 4 4 4 4 4 ≤ C0 1 + Kx E 1 + ∥G∥op + tm + tξ + tv ∥v∥ + ∥tz z + tθ θ∥ hence E⟨Q211 ⟩t

1/2

≤

 p  C0′ 1 + Kx (1 + t2y + t2Z + t2m + t2ξ + t2v + t2z + t2θ )

Combining (C.42) (C.43) and (C.45) one gets (C.31). 38

(C.46)

(iii). Given a configuration x = (σ, ξ), set 1 X Σ(x) := Sµ (x)2 . N µ≤M By H3 we have |u′′ | ≤ C, and by H1, (u′ (s))2 ≤ 2C 2 (1 + s2 ). Hence  1 X  ′′ ′ 2 |S11 | ≤ |u (Sµ )| + (u (Sµ )) ≤ (C + 2C 2 ) α + 2C 2 Σ(x), N µ≤M

(C.47)

(C.48)

so that 2 S11 ≤ 2(C + 2C 2 )2 α2 + 8C 4 Σ(x)2 .

(C.49)

For S12 , the inequality 2|ab| ≤ a2 + b2 gives i   1 Xh ′ |S12 | ≤ (u (Sµ (x1 )))2 + (u′ (Sµ (x2 )))2 ≤ C 2 2α + Σ(x1 ) + Σ(x2 ) , (C.50) 2N µ≤M whence, using (a + b + c)2 ≤ 3(a2 + b2 + c2 ) and the fact that under νt⊗2 the two replicas are identically distributed, 2 S12 ≤ 12C 4 α2 + 6C 4 Σ(x)2 t . t

(C.51)

By (C.49)–(C.51) it suffices to prove sup sup E Σ(x)2 t < ∞

(C.52)

t∈[0,1] N

√ Since Sµ (x) = g µ · σ + yµ tm + tv vµ + tξ ξµ with g µ = gµ / N , and yµ2 = 1, Cauchy–   Schwarz yields Sµ2 ≤ 4 (g µ · σ)2 + t2m + t2v vµ2 + t2ξ ξµ2 . Summing over µ ≤ M and dividing P by N , and using µ (g µ · σ)2 = ∥Gσ∥2 ≤ ∥G∥2op ∥σ∥2 = N ∥G∥2op ∥σ∥2 , h i 2 2 2 2 2 2 2 Σ(x) ≤ 4 ∥G∥op ∥σ∥ + αtm + tv ∥v∥ + tξ ∥ξ∥ , (C.53) and therefore

i h Σ(x)2 ≤ 64 ∥G∥4op ∥σ∥4 + α2 t4m + t4v ∥v∥4 + t4ξ ∥ξ∥4 .

(C.54)

We need to bound E⟨·⟩t for all the terms in (C.54). For the second and the third term is enough to exploit the concentration of v, z and (D.1). For the last term note that ∥ξ∥4 ≤ ∥x∥4 /N 2 , and then applying (C.25) with p = 2: h i C2,ε 2 4 4 4 4 4 4 E ∥ξ∥4 t ≤ + C K E 1 + ∥G∥ + t + t + t ∥v∥ + ∥t z + t θ∥ (C.55) 2,ε z θ x op m ξ v N2 This implies t4ξ E⟨∥ξ∥4 ⟩t < ∞ uniformly in t ∈ [0, 1] and N . For the first term, Cauchy– Schwarz with respect to the disorder expectation and Jensen’s inequality ⟨∥σ∥4 ⟩2t ≤ ⟨∥σ∥8 ⟩t give h i  1/2  1/2 E ∥G∥4op ∥σ∥4 t ≤ E∥G∥8op E ∥σ∥8 t . (C.56) The first factor is finite uniformly in N, t. For the second, ∥σ∥8 ≤ ∥x∥8 /N 4 and (C.25) with p = 4 give, exactly as before, h i C4,ε 4 8 8 8 8 8 8 E ∥σ∥8 t ≤ + C K E 1 + ∥G∥ + t + t + t ∥v∥ + ∥t z + t θ∥ < ∞, (C.57) 4,ε x z θ op m ξ v N4 again uniformly in N, t. Bounds for the second moment of Q11 , Q12 , M1 , H1 can be easily obtained by the same methods.

39

D

Concentrations

The aim of this section is to establish concentration properties for the order parameters defined in (3.6) with respect to the (random) Boltzmann-Gibbs measure induced by the interpolating Hamiltonian (3.1). In particular we are interested in dependence of the above properties on the parameters of the interpolation, namely the real numbers κ, α, λ and the interpolation path which is determined by a collection of functions Pt = (ρ(t), q(t), m(t), r(t), δ(t), h(t)) obeying to the constraints ρ ≥ q ≥ 0 and r, δ ≥ 0. In this section we assume that these function are continuous which implies that sup max{ρ(t), r(t), δ(t), h(t), m(t)} < ∞

(D.1)

t∈[0,1]

D.1

Concentration of the free energy

We will prove that the perturbed free energy fa in (C.1) associated with the Hamiltonian Ha in (B.6) is a self-averaging quantity. Lemma 17 (Self-averaging of fa ). There exists ε > 0 such that if |a| ≤ ε then there exists Kf > 0 independent form t, N, a such that Var(fa ) ≤

Kf N

(D.2)

Proof. Recall that fa ≡ fa (y, Z, v, z, θ) :=

1 log Za (y, Z, v, z, θ) N

(D.3)

namely fa is a function of the random variables D = (y, Z, z, v, θ) that are mutually independent. By the law of total variance 
  
Var(fa ) = Ey,Z,v,z Varθ fa | y, Z, v, z + Vary,Z,v,z Eθ fa | y, Z, v, z . (D.4) The next lemma control the first term in the r.h.s. of (D.4). Lemma 18. Given a realization of (y, Z, v, z) consider the function fa (θ) :=

1 log Za (θ), N

|a| ≤ ε,

where ε > 0 is such that (C.25) holds. Assume that coordinates θ = (θi )i≤N are i.i.d. with a single component satisfies a Poincaré inequality with constant Cθ independent of N .Then there exists a constant Kθ < ∞, independent of N , t and |a| ≤ ε, such that
Kθ . Ey,Z,v,z Varθ fa | y, Z, v, z ≤ N

(D.5)

Proof. We first derive a bound for the gradient of fa with respect to θ. A direct computation gives 1 ∂θi fa = σ 1 A1 + σi2 A2 (a) . (D.6) N i where M ty X Aℓ := tθ + yµ u′yµ (Sµℓ ) ℓ = 1, 2, (D.7) N µ=1 40

By Jensen’s inequality, |∂θi fa |2 ≤

2 E 1 D 1 2 σ A + σ A . i 1 i 2 N2 (a)

(D.8)

Summing over i and using (x + y)2 ≤ 2x2 + 2y 2 , we obtain ∥∇θ fa ∥2 =

N X

|∂θi fa |2

i=1

1 ∥A1 σ 1 + A2 σ 2 ∥2 (a) N2 2 2 X 2 ℓ 2 A ∥σ ∥ (a) . ≤ 2 N ℓ=1 ℓ ≤

(D.9)

By Cauchy–Schwarz and yµ2 = 1, 2t2y 2 2 Aℓ ≤ 2tθ + 2 N

M X

!2 yµ u′yµ (Sµℓ )

µ=1

1 ≤ 2t2θ + 2αt2y

M X

N µ=1

2 u′yµ (Sµℓ ) ,

(D.10)

By hypothesis H1 one has M

A2ℓ ≤ C

1 X ℓ2 |S | 1+ N µ=1 µ

! .

Hence using (C.41) one gets "

# 2 ℓ 2 ℓ 4 ∥v∥ ∥ξ ∥ ∥σ ∥ A2ℓ ∥σ ℓ ∥2 ≤ Cℓ ∥σ ℓ ∥2 + ∥G∥2op + ∥σ ℓ ∥2 + ∥σ ℓ ∥2 . N N N

(D.11)

for some Cℓ < ∞ an then using the moment estimates in (C.25) one obtains uniformly in t ∈ [0, 1] and |a| ≤ ε, E A2ℓ ∥σ ℓ ∥2 (a) ≤ C̃ℓ N. (D.12) for some C̃ℓ > 0. Substituting (D.12) into (D.9) yields E∥∇θ fa ∥2 ≤

Cθ . N

(D.13)

for some Cθ > 0. Conditionally on (y, Z, v, z), since the product law Pθ⊗N satisfies the tensorized Poincaré inequality with the same constant Cθ ,
  Varθ fa | y, Z, v, z ≤ CP Eθ ∥∇θ fa ∥2 y, Z, v, z . Averaging over (y, Z, v, z) and using (D.13), we obtain
Kθ Ey,Z,v,z Varθ fa | y, Z, v, z ≤ . N for some Kθ > 0.

41

For the second term in (D.4) we notice that conditionally on θ, each of the remaining variables satisfies a Poincaré inequality. Indeed Z, z, v are independent standard Gaussian vectors, while y satisfies a Rademacher-Poincaré inequality (see Lemma 12 in the Appendix ) for a suitable difference operator. Hence by tensorization and Jensen inequality one gets  
X   Vary,Z,v,z Eθ fa | y, Z, v, z ≤ E (Dyµ fa )2 + E∥∇Z fa ∥2 + E∥∇z fa ∥2 + E∥∇v fa ∥2 µ

(D.14) where Dyµ denotes the difference operator defined in (A.14). Let’s start computing the gradient in the last three terms. * + 2 2 X X t u′ (Sµ (xℓ )σiℓ ∥∇Z fa ∥2 = (∂Zµ,i fa )2 = Z3 N µ,i µ,i ℓ=1,2 (a) * + * +  2 X 2 2 X  X X 2 t tZ u′ (Sµ (xℓ )σiℓ u′ (Sµ (xℓ ) (σiℓ )2 ≤ Z3 ≤ 3 N µ,i N ℓ=1,2 µ,ℓ i,ℓ (a) * + X  X 4C 2 t2Z  2 ℓ 2 ℓ 2 2 ℓ 2 ≤ M + ∥G∥ ∥σ ∥ + ∥ξ ∥ + ∥v∥ ∥σ ∥ op N3 ℓ ℓ (a) D  E  2 2  4C tZ M + ∥G∥2op ∥σ∥2 + ∥ξ∥2 + 2∥v∥2 ∥σ∥2 = N3 (a)  2 2  8C tZ 2 4 4 4 2 2 ≤ M + ∥G∥ ∥σ∥ + ∥ξ∥ + ∥σ∥ + 2∥v∥ ∥σ∥ op (a) N3 (D.15) where in the third line we used that |u′ (s)| ≤ C(1 + |s|) and subsequently the inequality (a + b)2 ≤ 2(a2 + b2 ). Taking the expectation and using the moment bounds (C.25) and (A.13) one obtains KZ (D.16) E∥∇Z fa ∥2 ≤ N for some KZ > 0. It’s also clear from (C.25) that supt∈[0,1] KZ < ∞ In the same manner one finds X

Kv 8C 2 t2v ∥G∥2op ∥σ∥2 + ∥ξ∥2 + ∥v∥2 (a) ≤ 2 N N 2 2t K z E ∥∇z fa ∥2 ≤ E z2 ∥σ∥2 (a) ≤ N N 2 ty X 4  Kθ E ∥∇θ fa ∥2 ≤ E 2 tθ + yµ u′ (Sµ ) ∥σ∥2 (a) ≤ N N µ≤M N E ∥∇v fa ∥2 ≤ E

(D.17)

for some Kv , Kz , Kθ > 0 with finite supremum norm for t ∈ [0, 1] . It remains to control the first term on the right-hand side of (D.14), namely the norm of the difference operator Dyµ fa . Fix µ ≤ M , and keep all the disorder variables other than yµ fixed. For η ∈ {−1, +1} set Hµ,η (x) := Ha (x)|yµ =η 42

(D.18)

and denote the related partition function by Zµ,η and its Gibbs expectations by (⟨·⟩µ,η . For x = (σ, ξ), write Sµη (x) for the effective field obtained after setting yµ = η, and define

∆µ (x) := u+1 Sµ+ (x) − u−1 Sµ− (x) . (D.19) namely the variation of the u contribution to energy under the µ-label replacement. Hence Hµ,− (x) − Hµ,+ (x) = Aµ (x)

(D.20)

P2 ℓ Aµ and then where Aµ (x) := ℓ=1 ∆µ (x ). Now observe that Zµ,+ = Zµ,− e µ,− Jensen’s inequality gives log Zµ,+ − log Zµ,− ≥ ⟨Aµ ⟩µ,− . (D.21) Interchanging the roles of + and − one gets log Zµ,+ − log Zµ,− ≤ ⟨Aµ ⟩µ,+ . Therefore, ⟨Aµ ⟩µ,− ≤ log Zµ,+ − log Zµ,− ≤ ⟨Aµ ⟩µ,+ . Since Dµ fa =

(D.22)

1 (log Zµ,+ − log Zµ,− ) , 2N

we obtain * 2 + X 1 1 |Dµ fa | ≤ max ⟨Aµ ⟩µ,η ≤ max ∆µ (xℓ ) 2N η∈{−1,+1} 2N η∈{−1,+1} ℓ=1

.

(D.23)

µ,η

Observe that the dependence of the effective field Sµ on yµ is affine, indeed Sµη (x) = A(0) µ (x) + ηB(x),

B(x) := ty

θ·σ + tm , N

(D.24)

(0)

where Aµ (x) is independent of η, hence Sµ+ (x) − Sµ− (x) = 2B(x).

(D.25)

From the definition (D.19) adding and subtracting u−1 (Sµ+ ) one obtains |∆µ (x)| ≤ u+1 (Sµ+ ) − u−1 (Sµ+ ) + u−1 (Sµ+ ) − u−1 (Sµ− ) .

(D.26)

The first term is bounded by (2.4) in H4 . For the second one, H1 and the fundamental theorem of calculus imply that, for all r, s ∈ R, Z 1
|uy (r) − uy (s)| ≤ |r − s| u′y s + λ(r − s) dλ 0
≤ C|r − s| 1 + |r| + |s| . (D.27) Therefore,

|∆µ (x)| ≤ C 1 + |Sµ+ (x)| + C|B(x)| 1 + |Sµ+ (x)| + |Sµ− (x)| . Since |Sµ− (x)| ≤ |Sµ+ (x)| + 2|B(x)|, 43

(D.28)

we obtain

|∆µ (x)| ≤ K∆ 1 + |Sµ+ (x)| 1 + |B(x)| + K∆ |B(x)|2 .

(D.29)

for some K∆ > 0. The same argument, adding and subtracting u+1 (Sµ− ), gives

|∆µ (x)| ≤ K∆ 1 + |Sµ− (x)| 1 + |B(x)| + K∆ |B(x)|2 .

(D.30)

Thus, for either η ∈ {−1, +1},

|∆µ (x)| ≤ K∆ 1 + |Sµη (x)| 1 + |B(x)| + K∆ |B(x)|2 .

(D.31)

and then from elementary inequalities that

|∆µ (x)|2 ≤ K˜∆ 1 + |Sµη (x)|2 1 + |B(x)|2 + K˜∆ |B(x)|4 .

(D.32)

for some K˜∆ > 0. Applying Jensen’s inequality to (D.23), and using max{a+ , a− }2 ≤ a2+ + a2− , we obtain K̃∆ E(Dµ fa )2 ≤ 2 N =

X

E

* 2 X

+

 1 + |Sµη (xℓ )|2 1 + |B(xℓ )|2 + |B(xℓ )|4

ℓ=1

η∈{−1,+1}

(D.33)

µ,η

2 X



 K̃∆ η ℓ 2 ℓ 2 ℓ 4 E 1 + |S (x )| 1 + |B(x )| + |B(x )| µ (a) N 2 ℓ=1

(D.34)

where we used the fact that for any measurable function F , one has 1 2

X

E ⟨F η ⟩µ,η = E⟨F ⟩(a) .

(D.35)

η∈{−1,+1}

Summing (D.34) over µ we obtain *M + M 2 X X X

K̃ ∆ E(Dµ fa )2 ≤ 2 E 1 + |Sµ (xℓ )|2 1 + |B(xℓ )|2 + M |B(xℓ )|4 N µ=1 µ=1 ℓ=1

. (D.36)

(a)

Consider the quantity M

1 X |Sµ (x)|2 . Σ(x) := N µ=1 then

M X

(D.37)

1 + |Sµ (x)|2 = M + N Σ(x),

µ=1

and hence M X µ=1

E(Dµ fa )2 ≤

2 K̃∆ X E N 2 ℓ=1

M + N Σ(xℓ ) 1 + |B(xℓ )|2 + M |B(xℓ )|4 (a) .

44

(D.38)

We now verify that every expectation on the right-hand side is O(N ). Since B(x) = + tm , and the interpolation coefficients ty , tm are uniformly bounded along the adty θ·σ N missible interpolation paths, there exists KB > 0 such that   |θ · σ| |B(x)| ≤ KB 1 + . N and then

(D.39)

    ∥θ∥4 ∥σ∥4 |θ · σ|4 ≤C 1+ . |B(x)| ≤ C 1 + N4 N4 4

(D.40)

By Cauchy–Schwarz,  E

∥θ∥4 ∥σ∥4 N4

 ≤ (a)

∥θ∥8 E 4 N

1/2

E

∥σ∥8 N4

!1/2 (D.41) (a)

The first factor is bounded because θ satisfy a Poincaré inequality. The second term is bounded by (C.25) with p = 4 for |a| ≤ ε. Thus E⟨|B(x)|4 ⟩(a) and E⟨|B(x)|2 ⟩(a) are uniformly bounded. Moreover using (C.54) one has that also E⟨Σ(x)2 ⟩(a) is uniformly bounded for |a| small enough and t ∈ [0, 1] and then one can easily conclude that M X

E(Dµ fa )2 ≤

µ=1

Ky . N

(D.42)

Finally, combining (D.17), (D.5) and (D.42), we conclude that Var(fa ) ≤

Kθ + K y + K Z + K v + K z Kf = . N N

The constants are uniform in N , t ∈ [0, 1], and |a| ≤ ε, which completes the proof.

D.2

Concentration of order parameters

Proposition 7. Assume that the functions u, ϕ satisfies hypothesis H1-H4 one has D E K 2 E Qab − E⟨Qab ⟩t ≤ √ , a, b = 1, 2 (D.43) t N D E K 2 E M1 − E⟨M1 ⟩t (D.44) ≤√ , t N for some constant K < ∞ uniformly in t ∈ [0, 1] and N . Proof. Let’s focus now on the concentration properties (D.43) and (D.44). We will prove only (D.43) for a ̸= b namely the concentration property of the overlap Q12 . The same argument (with only minor modifications) works also for Q11 and M1 . We start with the following We will now comlete the proof of (D.44), namely the concentration property for Q12 . Let’s start with the inequality D E

Q12 − E⟨Q12 ⟩

2 E

≤ 2E

Q12 − ⟨Q12 ⟩)2 + 2E ⟨Q12 ⟩ − E⟨Q12 ⟩)2

= 2 EVar0 (Q12 ) + 2 E(f0′ − Ef0′ )2 45

(D.45)

Recall that for any a ∈ R the quantity Vara denotes the variance w.r.t. the random Gibbs measure νa in (B.7). Fix some a ∈ R and consider 1 Vara (σ 1 · σ 2 ) (D.46) 2 N We already proved in Lemma 13 that for |a| small enough the νa is log-concave, therefore by Brascamb-Lieb inequality Z 1 2 Vara (σ · σ ) ≤ dxµa (x) ∇Tx (σ 1 · σ 2 )(Hess[Ha ])−1 ∇x (σ 1 · σ 2 ) (D.47) Vara (Q12 ) =

where Ha is defined in (B.6). By Lemma 13, in particular from (B.9) one gets ∇Tx (σ 1 · σ 2 )(Hess[Ha ])−1 ∇x (σ 1 · σ 2 ) ≤

1 ∥σ∥2 κ − |a|

(D.48)

Now combining (D.48) with (C.25) one gets that Vara (σ 1 · σ 2 ) ≤

1 ∥σ∥2 (a) κ − |a|


 N 2 2 2 2 2p 2 Ca 1 + Kx N 1 + ∥G∥op + tm + tξ + tv ∥v∥ + ∥tz z + tθ θ∥ ≤ κ − |a|

(D.49)

Hence E Var0 (Q12 ) ≤ KN1 for some K1 > 0 such that supt∈[0,1] K1 < ∞. We treat now the second term of the r.h.s. in (D.45). Since the function a 7→ fa is convex one has that for any a > 0 ′ f0′ − Ef0′ ≤ fa′ − f−a +

1 X fδ − Efδ a δ=±a,0

(D.50)

Now if |a| is small enough, by Proposition 17 one has E fδ − Efδ

2

≤

Kf N

(D.51)

for any δ = ±a, 0 and then 1 X 2 K2 E fδ − Efδ ≤ a δ=±a,0 N a2

(D.52)

′ for someRK2 > 0. In order to bound the quantity E(fa′ − f−a )2 we will use the identity a ′′ f (s)ds that implies a Z a
2   ′ ′ dsE (fs′′ )2 (D.53) E fa − f−a ≤ 2a

′ fa′ − f−a =

−a

Since fs′′ = N Vars (Q12 ) =

1 Vars (σ 1 · σ 2 ) N

(D.54)

from (D.49) one obtains   E (fs′′ )2 ≤ K1 46

(D.55)

uniformly for |s| ≤ ε with K1 > 0 such that supt∈[0,1] K1 < ∞. Finally one gets ′ E(fa′ − f−a )2 ≤ 4K1 a2

(D.56)

Combining (D.52) and (D.56) one gets that for |a| small enough E f0′ − Ef0′

2

≤ 4K1 a2 +

K2 N a2

(D.57)

Choosing a2 = √1N one gets the result.

References [1] Frank Rosenblatt. “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain”. In: Psychological Review 65.6 (1958), pp. 386–408. doi: 10.1037/h0042519. [2] Thomas M. Cover. “Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition”. In: IEEE Transactions on Electronic Computers EC-14.3 (1965), pp. 326–334. doi: 10 . 1109 / PGEC . 1965 . 264137. [3] Elizabeth Gardner. “The Space of Interactions in Neural Network Models”. In: Journal of Physics A: Mathematical and General 21.1 (1988), pp. 257–270. doi: 10.1088/0305-4470/21/1/030. [4] Elizabeth Gardner and Bernard Derrida. “Optimal Storage Properties of Neural Network Models”. In: Journal of Physics A: Mathematical and General 21.1 (1988), pp. 271–284. doi: 10.1088/0305-4470/21/1/031. [5] H. S. Seung, H. Sompolinsky, and N. Tishby. “Statistical Mechanics of Learning from Examples”. In: Physical Review A 45.8 (1992), pp. 6056–6091. doi: 10.1103/ PhysRevA.45.6056. [6] Timothy L. H. Watkin, Albrecht Rau, and Michael Biehl. “The Statistical Mechanics of Learning a Rule”. In: Reviews of Modern Physics 65.2 (1993), pp. 499–556. doi: 10.1103/RevModPhys.65.499. [7] Andreas Engel and Christian Van den Broeck. Statistical Mechanics of Learning. Cambridge University Press, 2001. doi: 10.1017/CBO9781139164542. [8] Pragya Sur and Emmanuel J. Candès. “A Modern Maximum-Likelihood Theory for High-Dimensional Logistic Regression”. In: Proceedings of the National Academy of Sciences 116.29 (2019), pp. 14516–14525. doi: 10.1073/pnas.1810420116. [9] Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. “Precise Error Analysis of Regularized M -Estimators in High Dimensions”. In: IEEE Transactions on Information Theory 64.8 (2018), pp. 5592–5628. doi: 10.1109/TIT.2018.2840720. [10] Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. “Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions”. In: Proceedings of the 24th International Conference on Artificial Intelligence and Statistics. Vol. 130. Proceedings of Machine Learning Research. PMLR, 2021, pp. 2773– 2781. 47

[11] Benjamin Aubin, Florent Krzakala, Yue M. Lu, and Lenka Zdeborová. “Generalization Error in High-Dimensional Perceptrons: Approaching Bayes Error with Convex Optimization”. In: Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc., 2020. url: https://proceedings.neurips.cc/paper/ 2020/hash/8f4576ad85410442a74ee3a7683757b3-Abstract.html. [12] Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. “Optimal errors and phase transitions in high-dimensional generalized linear models”. In: Proceedings of the National Academy of Sciences 116.12 (2019), pp. 5451– 5460. doi: 10.1073/pnas.1802705116. [13] Cédric Gerbelot, Alia Abbara, and Florent Krzakala. “Asymptotic Errors for Teacher– Student Convex Generalized Linear Models (or: How to Prove Kabashima’s Replica Formula)”. In: IEEE Transactions on Information Theory 69.3 (2023), pp. 1824– 1852. doi: 10.1109/TIT.2022.3222913. [14] Xiaoyi Mai and Zhenyu Liao. High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Loss. 2020. arXiv: 1905.13742 [stat.ML]. [15] Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. “Sharp Guarantees and Optimal Performance for Inference in Binary and Gaussian-Mixture Models”. In: Entropy 23.2 (2021), p. 178. doi: 10.3390/e23020178. [16] Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. “The Role of Regularization in Classification of High-dimensional Noisy Gaussian Mixture”. In: Proceedings of the 37th International Conference on Machine Learning. Ed. by Hal Daumé III and Aarti Singh. Vol. 119. Proceedings of Machine Learning Research. PMLR, 13–18 Jul 2020, pp. 6874–6883. url: https: //proceedings.mlr.press/v119/mignacco20a.html. [17] Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. “Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-Dimensions”. In: Advances in Neural Information Processing Systems. Ed. by Marc’Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. Vol. 34. Curran Associates, Inc., 2021, pp. 10144–10157. [18] Federica Gerace, Florent Krzakala, Bruno Loureiro, Ludovic Stephan, and Lenka Zdeborová. “Gaussian Universality of Perceptrons with Random Labels”. In: Physical Review E 109.3 (2024), p. 034305. doi: 10.1103/PhysRevE.109.034305. [19] Luca Pesce, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. “Are Gaussian Data All You Need? The Extents and Limits of Universality in High-Dimensional Generalized Linear Estimation”. In: Proceedings of the 40th International Conference on Machine Learning. Vol. 202. Proceedings of Machine Learning Research. PMLR, 2023, pp. 27680–27708. [20] Yatin Dandi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro, and Lenka Zdeborová. “Universality laws for Gaussian mixtures in generalized linear models”. In: Journal of Statistical Mechanics: Theory and Experiment 2024.10 (2024), p. 104015. doi: 10.1088/1742-5468/ad65e7. arXiv: 2302.08933.

48

[21] Marc Mézard. “The Space of Interactions in Neural Networks: Gardner’s Computation with the Cavity Method”. In: Journal of Physics A: Mathematical and General 22.12 (1989), pp. 2181–2190. doi: 10.1088/0305-4470/22/12/018. [22] Michel Talagrand. “Intersecting Random Half-Spaces: Toward the Gardner–Derrida Formula”. In: The Annals of Probability 28.2 (2000), pp. 725–758. doi: 10.1214/ aop/1019160259. [23] Michel Talagrand. “On the Gaussian Perceptron at High Temperature”. In: Mathematical Physics, Analysis and Geometry 5.1 (2002), pp. 77–99. doi: 10.1023/A: 1015840632110. [24] Mariya Shcherbina and Brunello Tirozzi. “Rigorous Solution of the Gardner Problem”. In: Communications in Mathematical Physics 234 (2003), pp. 383–422. doi: 10.1007/s00220-002-0783-3. url: https://doi.org/10.1007/s00220-0020783-3. [25] Erwin Bolthausen, Shuta Nakajima, Nike Sun, and Changji Xu. “Gardner Formula for Ising Perceptron Models at Small Densities”. In: Proceedings of the Thirty-Fifth Conference on Learning Theory. Vol. 178. Proceedings of Machine Learning Research. PMLR, 2022, pp. 1787–1911. [26] Manuel Sáenz and Pragya Sur. “Characterizing Finite-Dimensional Posterior Marginals in High-Dimensional GLMs via Leave-One-Out”. In: (2025). doi: 10.48550/arXiv. 2601.00091. arXiv: 2601.00091 [math.ST]. [27] Michel Talagrand. Mean Field Models for Spin Glasses: Volume I: Basic Examples. Springer, 2010. [28] Francesco Guerra and Fabio Lucio Toninelli. “The Thermodynamic Limit in Mean Field Spin Glass Models”. In: Communications in Mathematical Physics 230 (2002). [29] Francesco Guerra. “Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model”. In: Communications in Mathematical Physics 233 (2003). [30] Jean Barbier and Nicolas Macris. “The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference”. In: Probability Theory and Related Fields 174 (2019). [31] Jean Barbier and Nicolas Macris. “The adaptive interpolation method for proving replica formulas. Applications to the Curie–Weiss and Wigner spike models”. In: Journal of Physics A: Mathematical and Theoretical 52.29 (2019), p. 294002. doi: 10.1088/1751-8121/ab2735. [32] Hidetoshi Nishimori. Statistical Physics of Spin Glasses and Information Processing: an Introduction. Oxford; New York: Oxford University Press, 2001. [33] Satoshi Morita, Hidetoshi Nishimori, and Pierluigi Contucci. “Griffiths Inequalities in the Nishimori Line”. In: Progress of Theoretical Physics Supplement 157 (Jan. 2005), pp. 73–76. issn: 0375-9687. doi: 10.1143/PTPS.157.73. [34] Pierluigi Contucci, Satoshi Morita, and Hidetoshi Nishimori. “Surface Terms on the Nishimori Line of the Gaussian Edwards–Anderson Model”. In: Journal of Statistical Physics 122 (2006), pp. 303–312. doi: 10.1007/s10955-005-8020-z. [35] Jean Barbier and Dmitry Panchenko. “Strong Replica Symmetry in High-Dimensional Optimal Bayesian Inference”. In: Communications in Mathematical Physics 393 (2022), pp. 1199–1239. doi: 10.1007/s00220-022-04387-w. 49

[36] Herm Jan Brascamp and Elliott H. Lieb. “On Extensions of the Brunn–Minkowski and Prékopa–Leindler Theorems, Including Inequalities for Log Concave Functions, and with an Application to the Diffusion Equation”. In: Journal of Functional Analysis 22.4 (1976), pp. 366–389. doi: 10.1016/0022-1236(76)90004-5. [37] Jean Barbier, Dmitry Panchenko, and Manuel Sáenz. “Strong replica symmetry for high-dimensional disordered log-concave Gibbs measures”. In: Information and Inference: A Journal of the IMA 11.3 (2022), pp. 1079–1108. doi: 10.1093/imaiai/ iaab027. arXiv: 2009.12939. [38] Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. “An inference problem in a mismatched setting: a spin-glass model with Mattis interaction”. In: SciPost Phys. 12 (4 2022), p. 125. doi: 10.21468/SciPostPhys.12.4.125. [39] Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. “The Onset of Parisi’s Complexity in a Mismatched Inference Problem”. In: Entropy 26.1 (2024), p. 42. doi: 10.3390/e26010042. [40] Diego Alberici, Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. “The multi-species mean-field spin-glass on the Nishimori line”. In: Journal of Statistical Physics 182.1 (2021), pp. 1–20. [41] Francesco Camilli, Daria Tieplova, Eleonora Bergamin, and Jean Barbier. “InformationTheoretic Reduction of Deep Neural Networks to Linear Models in the Overparametrized Proportional Regime”. In: Proceedings of the Thirty-Eighth Conference on Learning Theory. Vol. 291. Proceedings of Machine Learning Research. PMLR, 2025, pp. 757– 798. [42] Jean Barbier, Francesco Camilli, Minh-Toan Nguyen, Mauro Pastore, and Rudy Skerk. “Statistical Physics of Deep Learning: Optimal Learning of a Multilayer Perceptron near Interpolation”. In: Physical Review X 16.3 (2026), p. 031014. doi: 10.1103/56sb-pdh6. [43] Diego Alberici, Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. “The solution of the deep Boltzmann machine on the Nishimori line”. In: Communications in Mathematical Physics (to appear) (July 2021). doi: 10 . 1007 / s00220 - 021 04165-0. [44] Hongbin Chen, Jean-Christophe Mourrat, and Jiaming Xia. “Statistical inference of finite-rank tensors”. en. In: Annales Henri Lebesgue 5 (2022), pp. 1161–1189. doi: 10.5802/ahl.146. url: https://www.numdam.org/articles/10.5802/ahl.146/. [45] Maurice Sion. “On General Minimax Theorems”. In: Pacific Journal of Mathematics 8.1 (1958), pp. 171–176. doi: 10.2140/pjm.1958.8.171. [46] Jean Barbier, Wei-Kuo Chen, Dmitry Panchenko, and Manuel Sáenz. “Performance of Bayesian linear regression in a model with mismatch”. In: Information and Inference: A Journal of the IMA 14.3 (Sept. 2025), iaaf019. issn: 2049-8772. doi: 10.1093/imaiai/iaaf019. [47] Yehoram Gordon. “Some Inequalities for Gaussian Processes and Applications”. In: Israel Journal of Mathematics 50.4 (1985), pp. 265–289. doi: 10.1007/BF02759761. [48] Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. “The Gaussian MinMax Theorem in the Presence of Convexity”. In: arXiv preprint arXiv:1408.4837 (2014). arXiv: 1408.4837 [math.OC]. 50

[49] Michael Aizenman and Pierluigi Contucci. “On the Stability of the Quenched State in Mean Field Spin Glass Models”. In: Journal of Statistical Physics 92.5–6 (1998), pp. 765–783. doi: 10.1023/A:1023080223894. [50] Francesco Camilli, Daria Tieplova, and Jean Barbier. “Fundamental limits of overparametrized shallow neural networks for supervised learning”. In: (2023). arXiv: 2307.05635 [cs.LG]. url: https://arxiv.org/abs/2307.05635. [51] S Ariosto, R Pacelli, M Pastore, F Ginelli, M Gherardi, and P Rotondo. “Statistical mechanics of deep learning beyond the infinite-width limit”. In: arXiv preprint arXiv:2209.04882 (2022). [52] Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. “Generalisation error in learning with random features and the hidden manifold model”. In: International Conference on Machine Learning. PMLR. 2020, pp. 3452–3462. [53] Hugo Cui, Florent Krzakala, and Lenka Zdeborov. “Bayes-optimal learning of deep random networks of extensive-width”. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23. Honolulu, Hawaii, USA: JMLR.org, 2023. [54] Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. “Spectral Dynamics of Learning in Restricted Boltzmann Machines”. In: EPL (Europhysics Letters) 119.6 (2017), p. 60001. doi: 10.1209/0295-5075/119/60001. [55] Charles H. Martin and Michael W. Mahoney. “Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning”. In: Journal of Machine Learning Research 22.165 (2021), pp. 1–73. doi: 10. 48550/arXiv.1810.01075. arXiv: 1810.01075 [cs.LG]. [56] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018. doi: 10.1017/9781108231596.

51

Related documents

• Similarity search generalisation in contrastive learning with InfoNCE loss#361453
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models#280221
• Feature-weighted maximum representative subsampling.#263160
• Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence#366246
• Level proximal subdifferential, variational convexity, and pointwise quadratic approximation#155486