Conceptio › Archive › arXiv CS

arXiv CSopen access

Too Sharp, Too Sure: When Calibration Follows Curvature

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Too Sharp, Too Sure: When Calibration Follows Curvature

Alessandro Morosini 1 Matea Gjika 1 Tomaso Poggio 1 Pierfrancesco Beneventano 1 Gradient Descent 0.20

102

0.15

Sharpness

ECE

Modern neural networks can achieve high accuracy while remaining poorly calibrated, producing confidence estimates that do not match empirical correctness. Yet calibration is often treated as a post-hoc attribute. We take a different perspective: we study calibration as a training-time phenomenon on small vision tasks, and ask whether calibrated solutions can be obtained reliably by intervening on the training procedure. We identify a tight coupling between calibration, curvature, and margins during training of deep networks under multiple gradient-based methods. Empirically, Expected Calibration Error (ECE) closely tracks curvature-based sharpness throughout optimization. Mathematically, we show that both ECE and Gauss–Newton curvature are controlled, up to problem-specific constants, by the same margindependent exponential tail functional along the trajectory. Guided by this mechanism, we introduce a margin-aware training objective that explicitly targets robust-margin tails and local smoothness, yielding improved out-of-sample calibration across optimizers without sacrificing accuracy.

0.10 0.05

101

0.00 0

2000 4000 6000 8000 10000

Step

2000 4000 6000 8000 10000

Stochastic Gradient Descent

Step

102

0.20 0.15

Sharpness

ECE

arXiv:2604.20614v1 [cs.LG] 22 Apr 2026

Abstract

0.10 0.05 0.00 0

2000 4000 6000 8000 10000

Step

lr = 2/50

lr = 2/100

101 100

2000 4000 6000 8000 10000

lr = 2/150

Step

lr = 2/200

Figure 1. Training dynamics for Gradient Descent and Stochastic Gradient Descent across learning rates on CIFAR-10. Expected Calibration Error closely tracks sharpness throughout training: both rise as the model enters the edge of stability regime, peak around the same time, and decay together as training progresses.

works is post-hoc calibration: models are trained for accuracy and their predicted probabilities are adjusted afterward. While effective in many regimes, this framing treats calibration as a post-training concern, rather than a property that emerges during training. Recent work, however, suggests that calibration may be influenced by training dynamics. For example, Sharpness-Aware Minimization (SAM), which biases optimization toward flatter regions of the loss landscape, has been observed to generally reduce overconfidence (Tan et al., 2026). Since sharpness is a geometric property shaped along the optimization trajectory, these results hint at a link between calibration and loss geometry during training. Yet this connection remains poorly understood, motivating the following question:

1. Introduction Neural networks are now routinely used in settings where a model’s stated uncertainty matters as much as its accuracy, for example, in risk-sensitive domains such as healthcare or autonomous driving. In these contexts, we would like predicted probabilities to reflect empirical correctness: among predictions made with confidence p, approximately a fraction p should be correct. However, modern deep networks are often miscalibrated, frequently exhibiting overconfidence even when wrong (Guo et al., 2017).

How does calibration evolve throughout optimization, and what aspects of the training govern it?

A widely adopted response to overconfidence in neural net1 Massachusetts Institute of Technology, Cambridge, MA, USA. Correspondence to: Alessandro Morosini <[email protected]>, Pierfrancesco Beneventano <[email protected]>.

Importantly, answering this question is non-trivial. Calibration is defined in terms of the model’s predictive confidence distribution, whereas most training-time analyses characterize optimization through loss-landscape geometry, using quantities such as curvature or sharpness to reason about

Code available at: https://github.com/AlessandroMorosini/2sharp2sure.

Too Sharp, Too Sure: When Calibration Follows Curvature

stability and generalization. Bridging these viewpoints is challenging, particularly early in training when predictions are still rapidly evolving. While several recent studies have examined relationships between sharpness, flat minima, and calibration at convergence, the resulting picture is mixed: curvature proxies do not reliably predict calibration across architectures, regularization schemes, or optimizers (MasonWilliams et al., 2024). Crucially, these analyses focus on converged solutions. By contrast, we study how calibration and loss geometry co-evolve during training—a perspective that not only clarifies their relationship but also reveals dynamics that can be exploited to improve calibration. After formalizing a training-time connection between calibration and loss geometry, we turn to a prescriptive goal:

This perspective also clarifies an often-observed phenomenon: training and test calibration can diverge even when accuracy improves (Carrell et al., 2022; Wu et al., 2025). Once most examples achieve large positive margins, a relatively small set of near-boundary or negative-margin points can dominate the margin functional, making calibration highly sensitive to how optimization shapes this tail. Based on the mathematical connection we establish between margins, curvature, and calibration, we design a new robustmargin-aware loss. This yields a principled training-time handle on calibration, which we confirm in Section 5: Contribution 4: We propose CalMO (CALibration with Margin Objective), a training objective that yields better-calibrated models without sacrificing accuracy.

Can we intervene on the training procedure to reliably obtain calibrated solutions?

Our margin-based view also yields concrete diagnostics for optimizer behavior. In particular, we observe that Muon induces unusually large training margins, leading to nearzero training ECE, but severe test-time overconfidence. Consistent with this finding, Muon is also the optimizer that benefits most from CalMO, as robust-margin control directly targets this failure mode.

Contributions. We study calibration during optimization by jointly tracking calibration metrics, such as Expected Calibration Error (ECE), and curvature-based sharpness proxies, such as Gauss–Newton (GN) sharpness, along the training trajectory (Section 3). We conduct this analysis throughout training rather than only at convergence. We observe across multiple gradient-based optimization methods that

2. Preliminaries and Related Work Contribution 1. Calibration error and curvature-based measures exhibit a strong and consistent temporal correlation throughout training.

Calibration. A model is calibrated if its predicted confidence values reflect empirical correctness frequencies: among all predictions made with confidence p, approximately a fraction p should be correct (Niculescu-Mizil & Caruana, 2005; DeGroot & Fienberg, 1983). Since calibration is a distributional property, it is typically assessed empirically from finite samples. The most widely used measure is ECE (Naeini et al., 2015), which compares accuracy and confidence after binning predictions by confidence. Predictions are grouped into M bins according to their confidence (i.e., the maximum predicted class probability); we denote by Bm the set of predictions whose confidence falls in bin m. For each bin, we compute the empirical accuracy acc(Bm ) and the average predicted confidence conf(Bm ). ECE is then defined as

Next, we probe whether the coupling between calibration and curvature is causal. We compare optimizers designed to minimize sharpness (i.e., favoring flatter minima) with methods that instead suppress steep descent directions along the trajectory. Despite both affecting curvature, we find that Contribution 2: Directional interventions yield consistently better in-sample calibration than flat-minima methods. This clarifies the distinction between “being flat” and “training in stable directions” and its relation to confidence. We then provide a unifying explanation through the lens of the margin in Section 4. Intuitively, both confidence and curvature are shaped by how strongly the model separates the correct class from its nearest competitor. We formalize this connection by showing mathematically that

ECE =

M X |Bm | m=1

acc(Bm ) − conf(Bm ) .

(1)

Calibration is also evaluated through other summary statistics such as the Maximum Calibration Error (Naeini et al., 2015), reliability diagrams (Guo et al., 2017), kernel-based metrics such as the Kernel Calibration Error (KCE) (Kumar et al., 2018), or similar related metrics. Further details, including the multiclass extension, are provided in Appendix B.1.

Contribution 3: A single margin-based functional controls both calibration error and Gauss–Newton sharpness, up to problem-dependent constants.

Too Sharp, Too Sure: When Calibration Follows Curvature

Mitigating miscalibration. Existing approaches to mitigating calibration error fall into two broad categories: posthoc methods and intrinsic training-time methods. Post-hoc methods modify model outputs without adjusting the parameters, and include techniques such as Platt scaling (Platt, 2000) and temperature scaling (Guo et al., 2017). Intrinsic methods incorporate calibration objectives directly into training by using regularization to penalize overconfident predictions (Pereyra et al., 2017), by incorporating differential proxies for calibration into the loss (Kumar et al., 2018; Bohdal et al., 2023), or by applying label smoothing (Müller et al., 2019). A more detailed overview of these methods is provided in Appendix A.

SGD

AdamW

Muon

SAM

Train

ECE .83±.08 .84±.07 .72±.07 KCE .83±.08 .84±.07 .70±.08

.63±.26 .61±.28

.92±.04 .91±.04

Test

ECE .96±.02 .97±.01 .15±.15 −.10±.28 .98±.01 KCE .97±.01 .97±.01 .21±.22 −.16±.31 .98±.01

Table 1. Pearson correlation between calibration metrics (ECE and KCE) and GN sharpness, mean ± std over 4 learning rates.

out training and by contrasting convergence-to-flatness with explicit suppression of unstable high-curvature directions.

3. The Coupling Between Calibration and Sharpness

An important perspective links calibration to adversarial robustness. Points with small robust margin have been shown to be more likely miscalibrated; motivated by this, RAdaLS (Qin et al., 2021) bins data points by robust margin and applies stronger label smoothing to low-margin samples. Moreover, state-of-the-art calibration losses can be unified as penalties on logit distances (Liu et al., 2022). These results suggest that calibration errors are tied to local margin geometry, rather than solely to global confidence statistics.

We track sharpness and ECE throughout training to study how loss landscape geometry relates to calibration. Following Cohen et al. (2021), we train an MLP (2 hidden layers, 200 units, tanh activation) on CIFAR-10 under crossentropy (CE) loss. This small-scale setup enables frequent computation of GN sharpness, which serves as a proxy for the top Hessian eigenvalue λmax . Models are trained using gradient descent (GD), stochastic gradient descent (SGD), AdamW (Kingma & Ba, 2015; Loshchilov & Hutter, 2019), Muon (Jordan et al., 2024), and SAM (Foret et al., 2021). We monitor GN sharpness and batch sharpness (Andreyev & Beneventano, 2024) as proxies for loss-landscape geometry, alongside ECE, KCE, loss, and accuracy.

Curvature along the trajectory. A complementary line of work treats curvature not as a static attribute of the final solution, but as a dynamical quantity that governs optimization stability throughout training. Early work established a link between “wide valleys” and generalization (Hochreiter & Schmidhuber, 1997), a picture later reinforced by the sharp-minima account of the large-batch generalization gap (Keskar et al., 2017), motivating the study of Hessian-based sharpness along the training trajectory. More recent analyses emphasize that curvature generally increases as gradient methods approach an edge-of-stability (EoS) regime, where the top curvature direction becomes commensurate with the inverse step size and the dynamics become oscillatory or unstable (Xing et al., 2018; Jastrz˛ebski et al., 2018; Jastrz˛ebski et al., 2019; Cohen et al., 2021; 2022; Andreyev & Beneventano, 2024), before often decreasing toward the end of training.

3.1. Calibration Temporally Correlates with Sharpness Figure 1 shows training dynamics for GD and SGD. Across all CE experiments1 , training ECE and GN sharpness follow the same trajectory: both quantities are small at initialization, increase as training enters an EoS regime, and decrease again later in training. This holds across optimizers and learning rates (see Figure 2 and Appendix C.1 for additional results). Similar observations extend to CIFAR-100 (Appendix C.1). Table 1 quantifies this effect, showing strong Pearson correlations between ECE and (batch) sharpness throughout training; KCE closely matches ECE across all settings, confirming the coupling is not a binning artifact.

In parallel, neighborhood-based objectives reshape the training trajectory by explicitly penalizing worst-case loss increases under small weight perturbations (e.g., SAM), thereby suppressing sensitivity to sharp directions (Foret et al., 2021; Zhou et al., 2025); such procedures have also been observed to improve confidence estimates under crossentropy (Tan et al., 2026). At the same time, evidence suggests that the sharpness–calibration relationship can be fragile across architectures and regularization schemes (MasonWilliams et al., 2024), pointing to the importance of how curvature directions are traversed, not only where optimization converges. Our trajectory-level study aligns with this viewpoint by jointly tracking calibration and curvature through-

As we show in Section 4, both calibration and sharpness converge to zero once all training points are correctly classified, so their coupling at the end of training is expected. What is surprising is the strong correlation during training, well before convergence, when the model is far from 1

We observe a similar temporal correlation on models trained with mean-squared error (MSE) loss. There, however, both GN sharpness and ECE increase and then plateau at high values. This behavior reflects the fact that MSE is not a proper scoring rule and induces systematic underconfidence; we therefore focus on CE in the main text and defer a detailed discussion of MSE to Appendix E.

Too Sharp, Too Sure: When Calibration Follows Curvature

SAM

interpolation and calibration is nontrivial. To our knowledge, this has not been observed or explained before. These results suggest that calibration does not depend on trainingmetrics at convergence, but on the trajectory itself: models that stay in lower-sharpness regions remain better calibrated throughout training, not just asymptotically. We formalize this connection in Section 4.

0.20

Sharpness

ECE

0.15 0.10 0.05

101 100

0.00 0

3.2. Converging to Flat Minima or Following Flat Directions?

2000 4000 6000 8000 10000

Step

2000 4000 6000 8000 10000

Step

Muon

The strong temporal correlation between sharpness and calibration established in the previous section raises a causal question: does calibration improve because optimization converges to flatter minima, or because training dynamics suppress movement along high-curvature directions? To disentangle these mechanisms, we formulate two competing hypotheses, and empirically find that suppressing directions of steep descent leads to improved in-sample calibration.

Sharpness

ECE

0.15 0.10 0.05

101

0.00 0

2000 4000 6000 8000 10000

Step

lr = 2/50

lr = 2/100

lr = 2/150

Step

lr = 2/200

Figure 2. Training dynamics for SAM and Muon across learning rates on CIFAR-10.

Hypothesis 1 (Flat Minima for Calibration). Training procedures that bias optimization toward flat minima lead to lower in-sample calibration error.

absolute flatness of the loss landscape. However, BulkSGD induces oscillatory dynamics and a sharpness divergence when too many dominant directions are projected out, making it impractical as a standalone optimizer. Together, these results support Hypothesis 2 over Hypothesis 1: suppressing updates along high-curvature directions during training leads to improved in-sample calibration, whereas convergence to flat minima alone does not.

Hypothesis 2 (Directional Flatness for Calibration). Training procedures that suppress updates along directions of steep curvature lead to lower in-sample calibration error, even if the final solution is not globally flat. We test Hypothesis 1 using SAM (Zhou et al., 2025), which explicitly penalizes worst-case loss perturbations within a local neighborhood, and is known to bias optimization toward flatter minima. To test Hypothesis 2, we use optimizers that directly suppress high-curvature directions during training. Muon (Jordan et al., 2024) rescales gradient components to equalize their magnitudes, effectively clamping updates along sharp directions while amplifying flatter ones. BulkSGD (Song et al., 2025) achieves a more extreme intervention by projecting gradients onto the subspace orthogonal to the top Hessian eigenvectors, thereby removing the steepest descent directions entirely. A more detailed analysis of the optimizers and their benefits in this experimental setting can be found in Appendix C.2.

3.3. Out-of-Sample Behavior

Figures 2 and 3 show the training dynamics. Although SAM consistently maintains lower sharpness than GD and SGD, its calibration trajectory closely mirrors that of standard training, with a comparable peak ECE and slower convergence. In contrast, both Muon and BulkSGD achieve substantially lower peak calibration error and faster ECE decay, despite exhibiting markedly different sharpness profiles.

The sharpness–calibration coupling is less consistent out of sample (Table 1). Across optimizers, test ECE does not consistently decrease alongside training ECE—in some cases it worsens as training progresses, even after sharpness and training calibration improve (Appendix C.1). Muon is an extreme example: training ECE drops to near zero while test ECE remains high, yielding a negative test correlation despite strong in-sample alignment. This reflects the calibration generalization gap in overparameterized models (Carrell et al., 2022; Berta et al., 2025; Wu et al., 2025): a model that fits training data well can become overconfident on misclassified test examples, causing test ECE to increase and decouple from sharpness. In Muon’s case, the near-zero training ECE is consistent with the large training margins it induces (Section 4): once these are extreme, the model becomes overconfident on test examples near the decision boundary, precisely where the margin functional is most sensitive.

Notably, Muon maintains low calibration error while operating in regimes that are not globally flat, and BulkSGD improves calibration even in the presence of pronounced instability. This suggests that calibration is sensitive to how optimization traverses sharp directions, rather than to the

Together with the findings from Section 3.2, these results point to an important distinction: on the one hand, directional interventions yield better in-sample calibration than flat-minima methods, suggesting that how optimization traverses curvature matters more than where it converges; on 4

Too Sharp, Too Sure: When Calibration Follows Curvature

2 (a) BulkSGD with learning rate 70

2 (b) BulkSGD with learning rate 30

Figure 3. Training dynamics for BulkSGD across different learning rates and number of projected-out gradients on CIFAR10.

the other hand, in-sample improvements do not automatically transfer to test data, pointing to a fundamental train– test gap. In the following section, we formalize this train– test gap and use it to design a training-time intervention that improves out-of-sample calibration.

Together, these results provide a mechanism-level explanation for the observed co-evolution of calibration and curvature during training, and clarify why training and test calibration can diverge even as accuracy improves. We conclude the section by connecting these theoretical regimes to the empirical training dynamics observed in Section 3.

4. Curvature and Calibration in the Separable and Non-separable Regimes

Setup and Notation. Let (X, Y ) ∼ π with Y ∈ {1, . . . , K}. A model θ ∈ Rd produces logits zθ (x) ∈ RK and probabilities pθ (x) = softmax(zθ (x)). Let ŷ(x) = arg maxk zθ (x)k (deterministic tie-break) and confidence P̂ (x) = maxk pθ (x)k . Define the true (logit) margin

In this section we explain the temporal alignment between calibration error and curvature observed in Section 3 through a common underlying quantity: the (robust) true logit margin. Our central claim is that, across training, both ECE and Gauss–Newton sharpness respond to the evolution of the same margin-dependent tail functional. This provides a concrete mechanism linking predictive confidence to losslandscape geometry.

mθ (x, y) := zθ (x)y − max zθ (x)j , j̸=y

and the robust true margin at radius ε > 0, mε,θ (x, y) := inf mθ (x + δ, y).

The analysis naturally separates into two regimes. Early in training, the data behave as overlap-dominated: a nontrivial fraction of examples exhibit small or negative true margins, and no uniform separability holds. The same regime persists at test time whenever accuracy is below 1, since any misclassified example has mθ (x, y) < 0 by definition—and, unlike on training, no cross-entropy mechanism pushes those margins to grow. In either case, neither calibration error nor curvature is forced to be small, and both can be dominated by a few hard or ambiguous points. This perspective aligns with observations that loss and curvature are controlled by a small set of strongly opposing examples (Rosenfeld & Risteski, 2024). Later in training on the training set, models trained with cross-entropy typically enter an interpolating regime in which all true margins become strictly positive. In this regime, calibration error and curvature become tightly coupled: once the margin tail contracts, both quantities are forced to decrease together.

∥δ∥≤ε

Let ECEM denote the population π/sample D binned ECE computed by binning P̂ (X) into M bins. Let Jθ (x) := ∂zθ (x)/∂θ ∈ RK×d and, for cross-entropy, Hz (p) := diag(p)−pp⊤ . Define the population Gauss–Newton matrix and its curvature proxy HGN (θ; π) := Eπ Jθ (X)⊤ Hz (pθ (X))Jθ (X) , λmax := λmax HGN (θ; π) . 4.1. Regime I: overlap-dominated (non-separable) behavior In this subsection all the quantities (ECE, GN matrix, robust margin, robust margin moment) are considered at a population level. See details in Appendix B. Definethe robust expo nential margin moment Q(θ) := E(X,Y )∼π e−mε,θ (X,Y ) . 5

Too Sharp, Too Sure: When Calibration Follows Curvature

Theorem 4.1 (Overlap regime: robust-margin upper bounds). For any θ and any distribution π,

4.3. Discussion: how the two-regime proxy matches the observed train/test split

ECEM ≤ (K − 1) Q(θ).

Section 3 shows strong co-evolution of training ECE and sharpness, well before convergence. Asymptotic interpolation alone is insufficient to explain this observation. Theorem 4.1–4.2 provide a mechanism-level lens: throughout training, both quantities respond to the evolution of the same margin-dependent tail functional, and in the interpolating regime this coupling becomes two-sided.

If additionally ∥Jθ (X)∥op ≤ CJ holds π-a.s., then λmax ≤ 2CJ2 (K − 1) Q(θ). Proof is in Appendix B.4. Interpretation (two bottlenecks). Theorem 4.1 exposes two multiplicative controls:

A remaining question is: why test ECE can increase while the logged (predicted) margin increases. Two caveats explain this:

• a probability bottleneck Q(θ), dominated by the tail of small/negative robust margins, • a geometry bottleneck CJ2 (how parameter perturbations move logits).

• The bounds depend on the true margin mθ (x, y), not the predicted margin zŷ − z(2) . A model can become more confidently wrong on a subset of test points: predicted margins increase, accuracy plateaus, and test ECE increases. • The sharpness control involves a geometry term (Jacobians). Even if Hz (p) contracts as predictions become more one-hot, large Jacobian norms (or failure of uniform Jacobian control) can keep curvature proxies large.

In overlap-dominated regimes (early training or test-time), a persistent set of small robust margins can keep Q(θ) bounded away from 0, so these bounds need not certify vanishing calibration error or curvature even if loss continues to decrease. 4.2. Regime II: Interpolating (separable) behavior on the training set

5. From Margin Theory to Calibrated Training

In this subsection all the quantities (ECE, GN matrix, robust margin, robust margin moment) are considered at a finitesample level. See details in Appendix B. Let γ(θ; D) := min1≤i≤n mθ (xi , yi ) P and the empirical exponential margin n moment QD (θ) := n1 i=1 e−mθt (xi ,yi ) . Theorem 4.2 (Interpolating regime: two-sided ECE–margin control and coupling to λmax ). Assume γ(θ; D) > 0 (all training points correctly classified with strictly positive true margin). Then

Section 4 establishes that calibration error is controlled by the margin functional Q(θ) = E[e−mε,θ (X,Y ) ], with the bound ECE ≤ (K − 1) Q(θ) holding on any distribution. This motivates directly targeting Q(θ) during training. We propose an objective that enforces robust margins. 5.1. Calibration with Margin Objective To minimize the ECE bound, we want large robust margins. For this, we combine two strategies: (i) directly raising the margin at adversarial points, and (ii) ensuring clean margins do not collapse under perturbation: h LCalMO (θ) = E(x,y) ℓCE (θ; x, y) + λr Rrob (θ; x, y) i (2) + λs Rsmooth (θ; x, y) .

1 QD (θ) ≤ ECEM ≤ (K − 1) QD (θ) K ≤ (K − 1) e−γ(θ;D) . If additionally maxi∈[n] ∥Jθ (xi )∥op ≤ CJ , then λmax ≤ 2CJ2 (K − 1) QD (θ) ≤ 2CJ2 K(K − 1) ECEM , equivalently ECEM ≥ λmax /(2CJ2 K(K − 1)).

where λr , λs ≥ 0 are hyperparameters and Rrob and Rsmooth are defined below.

Proof is in Appendix B.5.

Robustness regularizer. Following TRADES (Zhang et al., 2019), we encourage consistent predictions between clean and adversarial inputs: Rrob (θ; x, y) = DKL pθ (x) ∥ pθ (xadv ) ,

Interpretation. In the interpolating regime, ECEM is equivalent up to constants to the exponential margin moment QD (θ), and λmax is controlled by the same moment (under bounded Jacobians). This implies that once the training set is correctly classified, GN sharpness cannot be large without in-sample ECE also being large. Moreover, in this regime, empirical binning becomes immaterial: ECEM reduces to the mean misconfidence (formalized in Appendix B).

where xadv ∈ arg max∥x′ −x∥≤ε ℓCE (θ; x′ , y). Combined with cross-entropy at x, this raises the margin at the worst point in the ε-neighborhood, directly targeting mε,θ (x, y). 6

Too Sharp, Too Sure: When Calibration Follows Curvature

Smoothness regularizer. By Lemma B.5, if the margin has local Lipschitz constant Lm (x, y), then mε,θ (x, y) ≥ mθ (x, y) − εLm (x, y),

Optimizer Method

(3)

implying e−mε,θ ≤ e−mθ · eεLm . When Lm is large, this bound becomes vacuous even for large clean margins. To prevent this, we penalize:

Why naive margin maximization fails. As training progresses, clean margins mθ (xi , yi ) on training points grow, Qtrain → 0, and training ECE vanishes. Yet test ECE remains high. The issue is that large clean margins need not imply large robust margins: a model can achieve mθ (x, y) ≫ 0 while the margin collapses at x + δ for small perturbations. The robust margin mε,θ (x, y) = inf ∥δ∥≤ε mθ (x + δ, y) captures this distinction. If robust margins at training points are large, nearby test points— which lie within a ε-neighborhood under typical data distributions—inherit reasonable margins. Fragile margins, large only at training locations, provide no such transfer.

ECE ↓

SGD

CE 75.2 ± 1.2 0.081 ± 0.021 Flat. (λr =0) 80.8 ± 0.1 0.053 ± 0.003 Rob. (λs =0) 78.2 ± 0.9 0.062 ± 0.011 CalMO 80.1 ± 1.4 0.056 ± 0.001

AdamW

CE 80.7 ± 0.4 0.061 ± 0.007 Flat. (λr =0) 83.3 ± 0.9 0.047 ± 0.002 Rob. (λs =0) 82.4 ± 0.2 0.039 ± 0.009 CalMO 83.2 ± 1.1 0.045 ± 0.007

Muon

CE 80.3 ± 0.3 0.065 ± 0.016 Flat. (λr =0) 81.0 ± 0.2 0.046 ± 0.012 Rob. (λs =0) 81.9 ± 0.4 0.052 ± 0.010 CalMO 81.7 ± 0.9 0.019 ± 0.002

SAM

CE 85.0 ± 0.3 0.020 ± 0.005 Flat. (λr =0) 84.1 ± 0.3 0.016 ± 0.004 Rob. (λs =0) 85.5 ± 0.4 0.021 ± 0.006 CalMO 85.2 ± 0.4 0.017 ± 0.005

Rsmooth (θ; x, y) = ∥∇x mθ (x, y)∥22 . For neural networks with Lipschitz activations, mθ is locally Lipschitz with Lm (x, y) = ∥∇x mθ (x, y)∥ almost everywhere. This keeps mθ − εLm close to mθ , ensuring large clean margins translate to large robust margins.

Acc (%)

Table 2. CalMO vs CE. ResNet-20 on CIFAR-10, test set. Flat.: flatness only (λr =0, λs =0.01); Rob.: robustness only (λr =0.5, λs =0); CalMO: λr =0.5, λs =0.01. The relative importance of each term varies with optimizer, but CalMO strikes a balance across all settings.

(SAM: 0.020 → 0.017) to 0.046 (Muon: 0.065 → 0.019); test accuracy is preserved or improves in every case, by between +0.2 and +4.9 points, consistent with regularization helping under the fixed 10k-step budget. The Muon gap is the sharpest illustration of the mechanism of Section 4: directional optimization drives training margins to extreme values that collapse under input perturbation (fragile margins), keeping test ECE high, and CalMO’s robust-margin term directly targets this tail.

5.2. Empirical Results with CalMO Setup. We run a controlled fixed-budget experiment: within each optimizer, only the loss function varies; all other training choices are held fixed. Concretely, we train ResNet-20 (He et al., 2016) on CIFAR-10. We compare four gradient-based optimizers spanning different training dynamics: SGD, AdamW, Muon, and SAM. Learning rates are tuned per optimizer: η = 0.01 for SGD, Muon and SAM, η = 0.001 for AdamW. All runs use batch size 128 for 10,000 steps. We compare standard cross-entropy against CalMO with hyperparameters λr = 0.5 (robustness) and λs = 0.01 (flatness). The adversarial perturbation radius is ε = 8/255, computed via 3-step PGD. Adversarial examples are initialized at the clean input x (no random start) and updated with step size α = 2/255. We report test accuracy and ECE computed with 15 bins.

Computational cost. CalMO incurs additional cost from the robustness and flatness terms. The robustness term requires a 3-step PGD attack per iteration; however, these compute gradients with respect to the input rather than the model parameters, making each PGD step relatively cheap. The flatness term adds one forward pass and one inputgradient computation. This cost can be reduced by using only the flatness term (λr = 0). All methods are trained for 10k steps, and we report performance at the best validation step. Matching compute by training CE longer does not close the gap: CE reaches its validation optimum well before 10k steps, after which performance degrades.

These values are fixed without tuning; our aim is to validate that the robust-margin regularization motivated by Section 4 yields calibration gains over CE, and to demonstrate that training-time intervention on the loss landscape can improve calibration without relying on post-hoc corrections. We report test values in Table 2; training values are reported in Table A1 in the Appendix.

Ablation. To isolate the roles of robustness and flatness in shaping calibration, we ablate the CalMO objective into its components and evaluate their effects across optimizers. The results reveal a consistent but optimizer-dependent pattern. For SGD, enforcing flatness alone yields most of the calibration gains, while robustness provides limited benefit. For AdamW the two terms split roles: robustness yields the largest ECE reduction, while flatness yields the largest

Performance. CalMO lowers test ECE across all four optimizers (Table 2), with reductions ranging from 0.003 7

Too Sharp, Too Sure: When Calibration Follows Curvature

accuracy gain, suggesting that its adaptive preconditioning stabilizes parameter-space geometry along the directions relevant for accuracy, while leaving calibration sensitive to robust-margin control. Muon exhibits a different behavior: neither robustness nor flatness alone is sufficient to substantially reduce ECE, but their combination leads to the greatest calibration. This supports the theoretical mechanism developed in Section 4, where calibration is governed jointly by the robust-margin tail and geometric sensitivity. From this perspective, CalMO should not be viewed as a universally superior training objective, but rather as a targeted intervention that controls these two terms.

only; extending the trajectory-level analysis to languagemodel training, where calibration is a central open question, is a natural follow-up. More broadly, we see the trajectory perspective itself—tracking how calibration, curvature, and margins co-evolve rather than inspecting them only at convergence—as a lens applicable to other training-time phenomena in deep networks.

6. Conclusion We studied calibration as a training-time phenomenon rather than a static property of a converged model, and showed that calibration and sharpness are tightly coupled along the optimization trajectory across multiple optimizers. This coupling arises from a shared dependence on margin growth, explaining both the temporal co-evolution of Expected Calibration Error and curvature during training, as well as the frequent divergence between train and test calibration in overlap-dominated regimes. Building on this perspective, we distinguished between two competing mechanisms for improving calibration: convergence to flat minima and suppression of updates along high-curvature directions. Empirically, optimizers that implement directional control, such as Muon and BulkSGD, yielded consistently better in-sample calibration than methods targeting flat minima alone. Guided by the marginbased bounds, we proposed CalMO, a training objective that jointly regulates robust margins and local smoothness, and showed that their combined regulation can substantially improve out-of-sample calibration—most notably for directionally amplified optimizers such as Muon—while preserving predictive accuracy. Limitations. Our empirical study relies on explicit curvature diagnostics (Gauss–Newton / Hessian-based measurements), which are computationally expensive and constrain the scale of architectures and datasets we can probe. Our theoretical results identify a margin-tail mediator that upper-bounds both calibration error and curvature under a Jacobian-control assumption; we do not claim that these certificates are tight or that the Jacobian bounds hold uniformly in all deep networks. A natural next step is to develop scalable, distributionally robust proxies for the mediator (for example, low-rank spectral estimators, mini-batch surrogates, or input-space stability measurements) and to characterize when they preserve the qualitative regime predictions we derive. Our experiments also cover small-scale image classification 8

Too Sharp, Too Sure: When Calibration Follows Curvature

References

Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1019–1028. PMLR, 2017. URL https://proceedings.mlr. press/v70/dinh17b.html.

Andreyev, A. and Beneventano, P. Edge of stochastic stability: Revisiting the edge of stability for SGD. arXiv preprint arXiv:2412.20553, 2024. doi: 10.48550/ arXiv.2412.20553. URL https://arxiv.org/ abs/2412.20553.

Bartlett, P. L., Foster, D. J., and Telgarsky, M. Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Spectrally-normalized margin bounds for neural Sharpness-aware minimization for efficiently improving networks. In Advances in Neural Information generalization. In International Conference on Learning Processing Systems, volume 30, pp. 6240–6249, Representations, 2021. URL https://openreview. 2017. URL https://proceedings.neurips. net/forum?id=6Tm1mposlrM. ICLR 2021 Spotcc/paper_files/paper/2017/hash/ light. b22b257ad0519d4500539da3c8bcf4dd-Abstract. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On html. calibration of modern neural networks. In Proceedings Berta, E., Holzmüller, D., Jordan, M. I., and Bach, of the 34th International Conference on Machine LearnF. Rethinking early stopping: Refine, then caliing, volume 70 of Proceedings of Machine Learning Rebrate. arXiv preprint arXiv:2501.19195, 2025. doi: search, pp. 1321–1330. PMLR, 2017. URL https:// 10.48550/arXiv.2501.19195. URL https://arxiv. proceedings.mlr.press/v70/guo17a.html. org/abs/2501.19195. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learnBohdal, O., Yang, Y., and Hospedales, T. Meta-calibration: ing for image recognition. In Proceedings of the IEEE Learning of model calibration using differentiable exconference on computer vision and pattern recognition, pected calibration error. Transactions on Machine Learnpp. 770–778, 2016. ing Research, 2023. URL https://openreview. Hochreiter, S. and Schmidhuber, J. Flat minima. Neural net/forum?id=R2hUure38l. Accepted by TMLR. Computation, 9(1):1–42, 1997. doi: 10.1162/neco.1997. Carrell, A. M., Mallinar, N., Lucas, J., and Nakkiran, 9.1.1. P. The calibration generalization gap. arXiv preprint arXiv:2210.01964, 2022. URL https://arxiv. Hoffer, E., Hubara, I., and Soudry, D. Train longer, generalize better: Closing the generalization gap in large batch org/abs/2210.01964. training of neural networks. In Advances in Neural InforChaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Balmation Processing Systems, volume 30, pp. 1731–1741, dassi, C., Borgs, C., Chayes, J., Sagun, L., and Zecchina, 2017. URL https://proceedings.neurips. R. Entropy-SGD: Biasing gradient descent into wide valcc/paper_files/paper/2017/hash/ leys. In International Conference on Learning Represena5e0ff62be0b08456fc7f1e88812af3d-Abstract. tations, 2017. URL https://openreview.net/ html. forum?id=B1YfAfcgl. Jastrz˛ebski, S., Kenton, Z., Ballas, N., Fischer, A., Bengio, Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., and TalY., and Storkey, A. On the relation between the sharpest walkar, A. Gradient descent on neural networks directions of DNN loss and the SGD step length. In typically occurs at the edge of stability. In InInternational Conference on Learning Representations, ternational Conference on Learning Representations, 2019. URL https://openreview.net/forum? 2021. URL https://openreview.net/forum? id=SkgEaj05t7. id=jh-rTtvkGeM. ICLR 2021 Poster. Jastrz˛ebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, Cohen, J. M., Ghorbani, B., Krishnan, S., Agarwal, N., A., Bengio, Y., and Storkey, A. Three Factors InfluMedapati, S., Badura, M., Suo, D., Cardoze, D., Nado, Z., encing Minima in SGD. arXiv:1711.04623 [cs, stat], Dahl, G. E., and Gilmer, J. Adaptive gradient methods September 2018. URL http://arxiv.org/abs/ at the edge of stability, 2022. URL https://arxiv. 1711.04623. arXiv:1711.04623. org/abs/2207.14484. Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and DeGroot, M. H. and Fienberg, S. E. The comparison and Bengio, S. Fantastic generalization measures and where evaluation of forecasters. Journal of the Royal Statistical to find them. In International Conference on Learning Society: Series D (The Statistician), 32(1-2):12–22, 1983. Representations, 2020. URL https://openreview. doi: 10.2307/2987588. net/forum?id=SJgIPJBFvH. 9

Too Sharp, Too Sure: When Calibration Follows Curvature

Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: //kellerjordan.github.io/posts/muon/.

Liu, B., Ben Ayed, I., Galdran, A., and Dolz, J. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80–88, 2022.

Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum? id=H1oyRlYgg.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Bkg6RiCqY7. Maddox, W. J., Benton, G., and Wilson, A. G. Rethinking parameter counting in deep models: Effective dimensionality revisited. arXiv preprint arXiv:2003.02139, 2020. URL https://arxiv.org/abs/2003.02139.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https://arxiv.org/ abs/1412.6980.

Mason-Williams, I., Ekholm, F., and Huszár, F. Explicit Kull, M., Perello-Nieto, M., Kängsepp, M., Silva Filho, regularisation, sharpness and calibration. In NeurIPS T., Song, H., and Flach, P. Beyond temperature scaling: 2024 Workshop on Scientific Methods for Understanding Obtaining well-calibrated multi-class probabilities Deep Learning (SciForDL). OpenReview.net, October with Dirichlet calibration. In Advances in Neural 2024. URL https://openreview.net/forum? Information Processing Systems, volume 32, pp. 12316– id=ZQTiGcykl6. 12326, 2019. URL https://proceedings. neurips.cc/paper/2019/hash/ Möllenhoff, T. and Khan, M. E. SAM as an optimal relax8ca01ea920679a0fe3728441494041b9-Abstract. ation of Bayes. In International Conference on Learning html. Representations, 2023. URL https://openreview. net/forum?id=k4fevFqSQcX. Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embedMukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, dings. In Proceedings of the 35th International ConferP. H. S., and Dokania, P. K. Calibrating deep neural ence on Machine Learning, volume 80 of Proceedings networks using focal loss. In Advances in Neural of Machine Learning Research, pp. 2805–2814. PMLR, Information Processing Systems, volume 33, pp. 15288– 2018. URL https://proceedings.mlr.press/ 15299, 2020. URL https://proceedings. v80/kumar18a.html. neurips.cc/paper/2020/hash/ aeb7b30ef1d024a76f21a1d40e30c302-Abstract. Lengyel, D., Jennings, N., Parpas, P., and Kantas, N. On html. flat minima, large margins and generalizability. OpenReview (ICLR 2021 submission), 2021. URL https: //openreview.net/forum?id=Ki5Mv0iY8C. Li, Y. and Sur, P. Optimal and provable calibration in highdimensional binary classification: Angular calibration and Platt scaling. In Advances in Neural Information Processing Systems, 2025. URL https://openreview. net/forum?id=SgQAleMecy. NeurIPS 2025 Spotlight. Liang, T., Poggio, T., Rakhlin, A., and Stokes, J. FisherRao metric, geometry, and complexity of neural networks. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 888–896. PMLR, 2019.

Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? In Advances in Neural Information Processing Systems, volume 32, pp. 4696– 4705, 2019. URL https://proceedings. neurips.cc/paper/2019/hash/ f1748d6b0fd9d439f71450117eba2725-Abstract. html. Naeini, M. P., Cooper, G. F., and Hauskrecht, M. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2901–2907, 2015. doi: 10.1609/aaai.v29i1.9602. URL https://ojs.aaai. org/index.php/AAAI/article/view/9602. Nagarajan, V. and Kolter, J. Z. Deterministic PAC-bayesian generalization bounds for deep networks via generalizing noise-resilience. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Hygn2o0qKX.

Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017. 10

Too Sharp, Too Sure: When Calibration Follows Curvature

Neyshabur, B., Bhojanapalli, S., McAllester, D., and Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. Exploring generalization in deep Srebro, N. The implicit bias of gradient descent on separalearning. In Advances in Neural Information ble data. Journal of Machine Learning Research, 19(70): Processing Systems, volume 30, pp. 5947–5956, 1–57, 2018. URL https://jmlr.org/papers/ 2017. URL https://proceedings.neurips. v19/18-188.html. cc/paper_files/paper/2017/hash/ Stutz, D., Hein, M., and Schiele, B. Confidence10ce03a1ed01077e3e289f3e53c72813-Abstract. calibrated adversarial training: Generalizing to unhtml. seen attacks. In Daumé III, H. and Singh, A. (eds.), Niculescu-Mizil, A. and Caruana, R. Predicting good probProceedings of the 37th International Conference on abilities with supervised learning. In Proceedings of the Machine Learning, volume 119 of Proceedings of 22nd International Conference on Machine Learning, pp. Machine Learning Research, pp. 9155–9166. PMLR, 625–632, 2005. doi: 10.1145/1102351.1102430. 2020. URL https://proceedings.mlr.press/ v119/stutz20a.html. Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, Stutz, D., Hein, M., and Schiele, B. Relating adversarially B., and Snoek, J. Can you trust your model’s unrobust generalization to flat minima. In Proceedings of certainty? evaluating predictive uncertainty under the IEEE/CVF International Conference on Computer Vidataset shift. In Advances in Neural Information sion (ICCV), pp. 7807–7817, 2021. URL https: Processing Systems, volume 32, pp. 13991–14002, //openaccess.thecvf.com/content/ 2019. URL https://proceedings.neurips. ICCV2021/papers/Stutz_Relating_ cc/paper_files/paper/2019/hash/ Adversarially_Robust_Generalization_ 8558cb408c1d76621371888657d2eb1d-Abstract. to_Flat_Minima_ICCV_2021_paper.pdf. html. Tan, C., Zhou, Y., Ye, H., Dai, G., Liu, J., Song, Z., Zhang, Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and J., Zhao, Z., Hao, Y., and Xu, Y. Towards understanding Hinton, G. Regularizing neural networks by penalizing the calibration benefits of sharpness-aware minimization. confident output distributions. In ICLR 2017 Workshop In International Conference on Learning Representations, Track Proceedings. OpenReview.net, 2017. URL https: 2026. URL https://openreview.net/forum? //openreview.net/forum?id=HyhbYrGYe. id=c0ERcCz6lD. ICLR 2026 Poster. Platt, J. C. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Smola, A. J., Bartlett, P. L., Schölkopf, B., and Schuurmans, D. (eds.), Advances in Large Margin Classifiers, pp. 61–74. MIT Press, 2000.

Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., and Michalak, S. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips. cc/paper_files/paper/2019/hash/ 36ad8b5f42db492827016448975cc22d-Abstract. html.

Qin, Y., Wang, X., Beutel, A., and Chi, E. Improving calibration through the relationship with adversarial robustness. In Advances in Neural Information Processing Systems, volume 34, pp. 14358– 14369, 2021. URL https://proceedings. Tsuzuku, Y., Sato, I., and Sugiyama, M. Normalized neurips.cc/paper/2021/hash/ 78421a2e0e1168e5cd1b7a8d23773ce6-Abstract. flat minima: Exploring scale invariant definition of flat minima for neural networks using PAC-bayesian html. analysis. In Daumé III, H. and Singh, A. (eds.), Proceedings of the 37th International Conference on Rosenfeld, E. and Risteski, A. Outliers with opposing sigMachine Learning, volume 119 of Proceedings of nals have an outsized effect on neural network optimizaMachine Learning Research, pp. 9636–9647. PMLR, tion. In International Conference on Learning Represen2020. URL https://proceedings.mlr.press/ tations, 2024. URL https://openreview.net/ v119/tsuzuku20a.html. forum?id=kIZ3S3tel6. ICLR 2024 Poster. Song, M., Ahn, K., and Yun, C. Does SGD really happen in tiny subspaces? In International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=v6iLQBoIJw. ICLR 2025 Poster. 11

Wu, J., Bartlett, P., Telgarsky, M., and Yu, B. Benefits of early stopping in gradient descent for overparameterized logistic regression. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of

Too Sharp, Too Sure: When Calibration Follows Curvature

Proceedings of Machine Learning Research, pp. 67081– 67110. PMLR, 2025. URL https://proceedings. mlr.press/v267/wu25b.html. Xing, C., Arpit, D., Tsirigotis, C., and Bengio, Y. A walk with SGD. arXiv preprint arXiv:1802.08770, 2018. URL https://arxiv.org/abs/1802.08770. Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 694–699, 2002. doi: 10.1145/775047.775151. Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472–7482. PMLR, 2019. Zhang, J., Kailkhura, B., and Han, T. Y.-J. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. In Daumé III, H. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11117–11128. PMLR, 2020. URL https://proceedings.mlr.press/ v119/zhang20k.html. Zheng, Y., Zhang, R., and Mao, Y. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8152– 8161, 2021. doi: 10.1109/CVPR46437.2021.00806. URL https://openaccess.thecvf. com/content/CVPR2021/html/Zheng_ Regularizing_Neural_Networks_via_ Adversarial_Model_Perturbation_CVPR_ 2021_paper.html. Zhou, Z., Wang, M., Mao, Y., Li, B., and Yan, J. Sharpnessaware minimization efficiently selects flatter minima late in training. In International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=aD2uwhLbnA. ICLR 2025 Spotlight.

Too Sharp, Too Sure: When Calibration Follows Curvature

A. Further Related Work A.1. Mitigating Miscalibration: Extended Discussion Post-hoc calibration. Post-hoc methods learn a mapping from model scores to probabilities on a held-out set. Classical approaches include Platt scaling and isotonic/binning methods (Platt, 2000; Zadrozny & Elkan, 2002), with temperature scaling the de facto recipe for deep networks (Guo et al., 2017). More expressive calibrators—such as Dirichlet calibration and compositional strategies like Mix-n-Match—correct class- or confidence-dependent distortions while preserving accuracy (Kull et al., 2019; Zhang et al., 2020). Because post-hoc methods do not influence training dynamics, they provide limited mechanistic insight and can degrade under distribution shift (Ovadia et al., 2019). Intrinsic methods. Intrinsic methods incorporate calibration objectives directly into training. These include entropy-based regularization (Pereyra et al., 2017), label smoothing (Müller et al., 2019), augmentation schemes such as mixup that soften targets (Thulasidasan et al., 2019), and focal loss—originally introduced for class imbalance—which yields better calibrated classifiers even before post-hoc scaling (Lin et al., 2017; Mukhoti et al., 2020). Differentiable surrogates of calibration metrics further enable joint training for accuracy and calibration (Kumar et al., 2018; Bohdal et al., 2023). High-dimensional perspectives on miscalibration. Recent theoretical work highlights that miscalibration can arise intrinsically from high-dimensional statistical effects, even in well-specified problems. Li & Sur (2025) analyze confidence estimates in high-dimensional classification and show that predictive probabilities can systematically deviate from true correctness likelihoods due to margin concentration and estimation noise, suggesting that miscalibration need not stem from optimization failures alone. Our perspective is complementary: rather than asymptotic statistical limits, we study the finite-sample training-time evolution of margins and curvature. A.2. Robust Margins, Sharpness, and Calibration Robust margins and calibration. Cross-entropy training pushes predictions toward extreme softmax outputs. On linearly separable data, gradient descent drives margins to infinity, converging to a max-margin classifier (Soudry et al., 2018). While large margins aid classification, they also amplify overconfidence: Qin et al. (2021) showed that inputs with small robust margin are more likely to be miscalibrated, and proposed adaptive label smoothing on such points. Focal loss (Mukhoti et al., 2020) and label smoothing (Müller et al., 2019) can likewise curb overconfidence on hard examples. Foret et al. (2021)’s SAM, which biases optimization toward flatter minima, has been observed to lower calibration error (Zheng et al., 2021; Möllenhoff & Khan, 2023). These results share a common theme: controlling the growth or fragility of margins tends to improve calibration. Achieving both robustness and calibration is nevertheless non-trivial—standard adversarial training can degrade calibration without targeted interventions (Stutz et al., 2020). Flat minima and margins. Loss-landscape geometry has long been linked to generalization, with flat minima hypothesized to be preferable to sharp ones (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017). Dinh et al. (2017) complicated this picture: scaling symmetries in deep networks allow arbitrarily sharp solutions with identical outputs, motivating scale-invariant sharpness measures (Tsuzuku et al., 2020; Liang et al., 2019). Under such measures, flatter minima correlate with better generalization in CE-trained models (Jiang et al., 2020; Maddox et al., 2020). A structural correlate is the classification margin: flat basins of the CE loss align with large training margins (Lengyel et al., 2021; Jiang et al., 2020). Small-batch SGD, which implicitly enlarges margins (Hoffer et al., 2017), also finds flatter solutions than large-batch training (Keskar et al., 2017). Adversarially robust models—which have larger input margins—exhibit lower curvature in weight space (Stutz et al., 2021); conversely, weight-space flatness regularizers such as entropy-SGD (Chaudhari et al., 2017) improve adversarial robustness as a side effect (Stutz et al., 2021). Linear vs. non-linear caveats. In linear models trained with cross-entropy, the notion of “flat vs. sharp” is less meaningful: on separable data the weight norm grows without bound as margins maximize, driving the Hessian to zero while the classifier becomes arbitrarily confident. Meaningful cross-setting flatness comparisons therefore require correcting for reparameterization invariances (Dinh et al., 2017; Neyshabur et al., 2017; Tsuzuku et al., 2020). In deep non-linear networks with such corrections, large margins correspond to flatter minima (Lengyel et al., 2021). This distinction explains why margin-based analyses (Bartlett et al., 2017; Nagarajan & Kolter, 2019) are often preferred for theoretical guarantees in linear settings. 13

Too Sharp, Too Sure: When Calibration Follows Curvature

A.3. Positioning Relative to Prior Work The four literatures above—miscalibration under cross-entropy, sharpness/flatness and optimization stability, margin maximization via implicit bias, and robustness–calibration connections—together with recent work on calibration benefits of sharpness-aware optimizers (Tan et al., 2026), provide the backdrop for our contribution. Our work extends these literatures in three directions: • Trajectory-level analysis. Most prior work compares final solutions; we track calibration and curvature pathwise across training and show that they co-evolve, peaking together near the edge of stability and decaying together. • A shared margin-tail mediator. We prove that a single exponential margin moment and its robust variant simultaneously upper-bound ECE and Gauss–Newton sharpness, with a two-sided sandwich for ECE in the interpolating regime. The triangulation ECE ↔ GN sharpness ↔ robust margins is, to our knowledge, new. • Directional vs. flat-minima interventions. We distinguish optimizers that bias toward flat minima (SAM) from those that suppress steep descent directions along the trajectory (Muon, BulkSGD), and show empirically that the latter yield more reliable in-sample calibration gains. This motivates CalMO as a principled intervention on the mediator.

B. Proofs for Section 4 B.1. Definitions and basic reductions Fixed binning. Fix M ∈ N and deterministic bin edges 0 = a0 < a1 < · · · < aM = 1. Define bins Im := (am−1 , am ] for m = 1, . . . , M . Population ECE with fixed bins.

Let (X, Y ) ∼ π with Y ∈ {1, . . . , K}. For fixed θ, define the predicted label Yb := arg max zθ (X)k k

(using the deterministic tie-break rule from the main text), and the confidence Pb := max pθ (X)k = pθ (X)Yb . k

Let Bm := {Pb ∈ Im }. Define binwise accuracy and confidence by ( ( P(Yb = Y | Bm ), P(Bm ) > 0, E[Pb | Bm ], acc(Bm ) := conf(Bm ) := 0, P(Bm ) = 0, 0,

P(Bm ) > 0, P(Bm ) = 0.

The population binned calibration error is ECEM (θ; π) :=

M X

P(Bm ) |acc(Bm ) − conf(Bm )|.

m=1

Equivalently, with Z := 1{Yb = Y } − Pb, ECEM (θ; π) =

M X

P(Bm ) E[Z | Bm ] .

m=1

Empirical ECE.

Given a dataset D = {(xi , yi )}ni=1 , define Ybi := arg max zθ (xi )k k

(same deterministic tie-break),

Pbi := max pθ (xi )k ,

and bins Bm := {i : Pbi ∈ Im }. Let Zi := 1{Ybi = yi } − Pbi . Then ECEM (θ; D) :=

M X |Bm | m=1

with the convention that the inner average is 0 when |Bm | = 0. 14

1 X Zi , |Bm | i∈Bm

Too Sharp, Too Sure: When Calibration Follows Curvature

B.2. Core lemmas Lemma B.1 (ECE is bounded by the mean absolute correctness–confidence gap). (Population). Let Z := 1{Yb = Y } − Pb and let G := σ(B1 , . . . , BM ) be the σ-algebra generated by the bin events Bm := {Pb ∈ Im }. Note that Z ∈ [−1, 1], hence Z is integrable. Then h i E[Z | G]

ECEM (θ; π) = E

≤ E[|Z|].

(Empirical). For any dataset D, n

1X 1{Ybi = yi } − Pbi . n i=1

ECEM (θ; D) ≤

Proof. Population. For each bin Bm with P(Bm ) > 0, E[Z | Bm ] = P(Yb = Y | Bm ) − E[Pb | Bm ] = acc(Bm ) − conf(Bm ), and we set E[Z | Bm ] := 0 when P(Bm ) = 0. Therefore, ECEM (θ; π) =

M X

i h P(Bm ) E[Z | Bm ] = E E[Z | G] .

m=1

By Jensen’s inequality for the convex function u 7→ |u|, h i E E[Z | G] ≤ E E[|Z| | G] = E[|Z|]. Empirical. For each bin Bm with |Bm | > 0, let Zi := 1{Ybi = yi } − Pbi . Then acc(Bm ) − conf(Bm ) =

1 X 1 X Zi ≤ |Zi | |Bm | |Bm | i∈Bm

i∈Bm

by the triangle inequality. Multiplying by |Bm |/n and summing over m yields the claim. Lemma B.2 (Correctness–confidence gap is controlled by the true-class probability). For any (x, y) and θ, where yb(x) and Pb(x) are defined as above, 1{b y (x) = y} − Pb(x) ≤ 1 − pθ (x)y . Proof. Let yb = yb(x) and Pb = Pb(x) = maxk pθ (x)k = pθ (x)yb. If yb = y, then |1{b y = y} − Pb| = |1 − py | = 1 − py . P b b If y = y} − P | = P = pyb ≤ j̸=y pj = 1 − py , since pyb is one of the nonnegative summands in Pyb ̸= y, then |1{b j̸=y pj . Lemma B.3 (Softmax tail bound: 1 − py is exponentially controlled by the true margin). Let p = softmax(z) ∈ ∆K−1 and fix a label y. Define the true margin m := zy − maxj̸=y zj . Then 1 − py ≤

ezj −zy ≤ (K − 1)e−m .

(4)

j̸=y

Moreover, e−m ≤ 1 − py . 1 + (K − 1)e−m

(5)

1 −m e ≤ 1 − py ≤ (K − 1)e−m . K

(6)

In particular, if m ≥ 0 then

Too Sharp, Too Sure: When Calibration Follows Curvature

Proof. Write ezy

py = PK

k=1 e

1 1+

j̸=y e

P , zj −zy

1 − py =

j̸=y P

ezj −zy

j̸=y e

zj −zy

P zj −zy Let S := ≥ 0. Then 1 − py = S/(1 + S) ≤ S, proving the first inequality in (4). For each j ̸= y, j̸=y e zj − zy ≤ maxk̸=y zk − zy = −m, hence ezj −zy ≤ e−m and S ≤ (K − 1)e−m , proving the second inequality in (4). For (5), pick j ⋆ ∈ arg maxj̸=y zj so that zj ⋆ − zy = −m and hence S ≥ e−m . Therefore 1 − py =

S e−m e−m , ≥ ≥ 1+S 1+S 1 + (K − 1)e−m

where the last step uses S ≤ (K − 1)e−m . If m ≥ 0 then e−m ≤ 1 and thus 1 + (K − 1)e−m ≤ 1 + (K − 1) = K, giving (6). Lemma B.4 (Cross-entropy logit Hessian top eigenvalue is controlled by 1 − pmax ). Let p ∈ ∆K−1 and define Hz (p) := diag(p) − pp⊤ . Then λmax Hz (p) ≤ 2 1 − pmax , pmax := max pk . k

Proof. We use Gershgorin’s circle theorem for symmetric matrices. Write A := Hz (p), so that for each i, Aii = pi (1 − pi ), Let Ri :=

j̸=i |Aij | =

Aij = −pi pj

(i ̸= j).

j̸=i pi pj = pi (1 − pi ). Gershgorin implies every eigenvalue λ of A lies in at least one interval

λ ∈ [Aii − Ri , Aii + Ri ] = [0, 2pi (1 − pi )]

for some i.

Hence λmax (A) ≤ max 2pi (1 − pi ). i

Now fix k ⋆ ∈ arg maxk pk so that pk⋆ = pmax . If i = k ⋆ , then pi (1 − pi ) = pmax (1 − pmax ) ≤ 1 − pmax . If i ̸= k ⋆ , then pi ≤ 1 − pmax and pi (1 − pi ) ≤ pi ≤ 1 − pmax . Therefore maxi pi (1 − pi ) ≤ 1 − pmax and consequently λmax (Hz (p)) ≤ 2(1 − pmax ), as claimed. Lemma B.5 (Robust margin comparisons (trivial upper bound; Lipschitz lower bound)). For all (x, y), mε,θ (x, y) ≤ mθ (x, y)

=⇒

e−mθ (x,y) ≤ e−mε,θ (x,y) .

(7)

If moreover there exists Lm (x, y) ∈ [0, ∞) such that |mθ (x + δ, y) − mθ (x, y)| ≤ Lm (x, y) ∥δ∥

∀ ∥δ∥ ≤ ε,

then mε,θ (x, y) ≥ mθ (x, y) − εLm (x, y)

=⇒

e−mθ (x,y) ≥ e−εLm (x,y) e−mε,θ (x,y) .

Proof. Trivial robust-vs-clean comparison. By definition of the infimum and because δ = 0 is feasible, we have mε,θ (x, y) = inf mθ (x + δ, y) ≤ mθ (x, y). ∥δ∥≤ε

Since the map t 7→ e−t is decreasing, this implies e−mθ (x,y) ≤ e−mε,θ (x,y) , which is (7). 16

(8)

Too Sharp, Too Sure: When Calibration Follows Curvature

Lipschitz lower bound. Assume the stated local Lipschitz condition at (x, y). Then for any ∥δ∥ ≤ ε, mθ (x + δ, y) ≥ mθ (x, y) − Lm (x, y) ∥δ∥ ≥ mθ (x, y) − εLm (x, y). Taking the infimum over all ∥δ∥ ≤ ε yields mε,θ (x, y) ≥ mθ (x, y) − εLm (x, y), equivalently mθ (x, y) ≤ mε,θ (x, y) + εLm (x, y). Multiply by −1 (which flips the inequality) to get −mθ (x, y) ≥ −mε,θ (x, y) − εLm (x, y), and exponentiate to obtain e−mθ (x,y) ≥ e−mε,θ (x,y) e−εLm (x,y) , which is (8). Remark B.6 (Label-free GN bound via predicted margin). Because Hz (pθ (X)) depends only on X, one can avoid the label Y in the GN bound. Let yb(x) ∈ arg maxk zθ (x)k (with the deterministic tie-break rule) and define the predicted margin m b θ (x) := zθ (x)yb(x) − max zθ (x)j ≥ 0. j̸=y b(x)

Applying Lemma B.3 with y = yb(x) yields b θ (x) 1 − pmax (x) = 1 − pθ (x)yb(x) ≤ (K − 1)e−m .

Combining with Lemma B.4 gives b θ (x) λmax Hz (pθ (x)) ≤ 2 1 − pmax (x) ≤ 2(K − 1)e−m . cConsequently, under ∥Jθ (X)∥op ≤ CJ π-a.s., b (X) θ λmax HGN (θ; π) ≤ 2CJ2 (K − 1) E e−m . This can be substantially tighter than bounds routing through Y when the model is confidently incorrect. B.3. Rigorous restatement of the main theorems Notation alignment with Section 4.

Fix a robust radius ε > 0. To match the main-text notation, we use n

Q(θ; π) := E(X,Y )∼π e−mε,θ (X,Y )

and

1 X −mθ (xi ,yi ) QD (θ) := e . n i=1

When π (or D) is clear from context, we may drop it from the notation. For comparison with alternative functionals used in some intermediate lemmas, note that Q(θ; π) coincides with the quantity previously denoted Ψ0ε (θ; π), and QD (θ) coincides with the quantity previously denoted µ(θ; D). If a pointwise margin Lipschitz constant Lm (·, ·) is available, we also define the (generally looser) population functional Q+ (θ; π) := E(X,Y )∼π eεLm (X,Y ) e−mε,θ (X,Y ) , and the finite-sample robust moments n

Q0ε,D (θ) :=

1 X −mε,θ (xi ,yi ) e , n i=1

Q− ε,D (θ) :=

1 X −εLm (xi ,yi ) −mε,θ (xi ,yi ) e e . n i=1

The proofs appear in Subsections B.4 and B.5. Theorem 4.1 (Overlap regime: simultaneous robust-margin upper bounds). Let π be any distribution on X × {1, . . . , K} and let θ be any parameter vector. 17

Too Sharp, Too Sure: When Calibration Follows Curvature

For the population binned calibration error ECEM (θ; π), ECEM (θ; π) ≤ (K − 1) E e−mθ (X,Y ) ≤ (K − 1) Q(θ; π).

(i) Calibration upper bound.

If Lm is defined, then also ECEM (θ; π) ≤ (K − 1) Q+ (θ; π), but this is never tighter than the Q(θ; π) bound since Q+ (θ; π) ≥ Q(θ; π). (ii) Gauss–Newton curvature (top eigenvalue) upper bound. Assume additionally that the logit Jacobian is uniformly bounded in operator norm, ∥Jθ (X)∥op ≤ CJ π-a.s. Then the population Gauss–Newton matrix HGN (θ; π) := E(X,Y )∼π Jθ (X)⊤ Hz (pθ (X))Jθ (X) satisfies λmax HGN (θ; π)

≤ 2CJ2 (K − 1) E e−mθ (X,Y ) ≤ 2CJ2 (K − 1) Q(θ; π).

If Lm is defined, then also λmax (HGN (θ; π)) ≤ 2CJ2 (K − 1) Q+ (θ; π), again a looser bound than the one via Q(θ; π). (iii) What the bound can (and cannot) certify. If along a training trajectory {θt } the robust moment Q(θt ; π) fails to converge to 0, then the bounds in (i)–(ii) do not certify that ECEM (θt ; π) → 0 or λmax (HGN (θt ; π)) → 0. (iv) Remarks (label dependence and trivial clamping). • Label dependence vs. label-free curvature. λmax (HGN (θ; π)) depends only on the marginal law of X (since Hz (pθ (X)) is label-free), whereas the bound above routes through Y via mθ (X, Y ) (equivalently 1 − pθ (X)Y ). This is valid but can be loose, especially when the model is confidently wrong. A label-free alternative (via the predicted margin) is given in Remark B.6. • Bounds may exceed 1. Since ECEM (θ; π) ∈ [0, 1], any upper bound U can be trivially tightened to min{1, U }. Theorem 4.2 (Interpolating regime: two-sided ECE control and coupling to λmax ). Assume γ(θ; D) > 0, i.e. the training set D = {(xi , yi )}ni=1 is correctly classified with strictly positive true margin. (Strictness ensures Ybi = yi without tie-breaking subtleties.) (i) Two-sided control of in-sample ECE by the exponential margin moment. 1 QD (θ) ≤ ECEM (θ; D) ≤ (K − 1) QD (θ) ≤ (K − 1) e−γ(θ;D) . K (As always, one may clamp the upper bound by min{1, ·}.) (ii) In-sample GN curvature bound in terms of the same moment. Assume additionally that ∥Jθ (xi )∥op ≤ CJ for all i = 1, . . . , n. Then λmax HGN (θ; D) ≤ 2CJ2 (K − 1) QD (θ) ≤ 2CJ2 K(K − 1) ECEM (θ; D). (iii) Consequence: in the interpolating regime, curvature and ECE are forced to co-vary. Under the same assumptions, ECEM (θ; D) ≥

λmax (HGN (θ; D)) . 2CJ2 K(K − 1)

Thus, once the training set is correctly classified and Jacobians remain bounded, large GN curvature cannot occur without large in-sample ECE. 18

Too Sharp, Too Sure: When Calibration Follows Curvature

(iv) Robust-margin variant (optional; requires local Lipschitzness at (xi , yi )). Assume moreover that for each i there exists Lm (xi , yi ) ∈ [0, ∞) such that |mθ (xi + δ, yi ) − mθ (xi , yi )| ≤ Lm (xi , yi ) ∥δ∥

∀ ∥δ∥ ≤ ε.

Then, with the robust moments Q0ε,D (θ) and Q− ε,D (θ) defined above, 1 − Q (θ) ≤ ECEM (θ; D) ≤ (K − 1) Q0ε,D (θ), K ε,D and, under maxi ∥Jθ (xi )∥op ≤ CJ , λmax HGN (θ; D)

≤ 2CJ2 (K − 1) Q0ε,D (θ).

(v) Remark (binning irrelevance under perfect accuracy). Under γ(θ; D) > 0, every nonempty bin has empirical accuracy 1, so ECEM (θ; D) reduces to the mean misconfidence and becomes independent of the choice of bins. B.4. Proof of Theorem 4.1 Proof of Theorem 4.1. (i) Calibration bound. By Lemma B.1 (population version), h i ECEM (θ; π) ≤ E(X,Y )∼π 1{Yb = Y } − Pb . By Lemma B.2, 1{Yb = Y } − Pb ≤ 1 − pθ (X)Y , hence ECEM (θ; π) ≤ E(X,Y )∼π 1 − pθ (X)Y . Applying Lemma B.3 with z = zθ (X) and y = Y yields 1 − pθ (X)Y ≤ (K − 1)e−mθ (X,Y )

π-a.s.,

so ECEM (θ; π) ≤ (K − 1) E(X,Y )∼π e−mθ (X,Y ) . Finally, by Lemma B.5 (the trivial robust-vs-clean comparison), e−mθ (X,Y ) ≤ e−mε,θ (X,Y )

π-a.s.,

and therefore ECEM (θ; π) ≤ (K − 1) E(X,Y )∼π e−mε,θ (X,Y ) = (K − 1) Q(θ; π). If the local Lipschitz condition in Lemma B.5 holds so that Q+ (θ; π) is defined, then Q(θ; π) ≤ Q+ (θ; π) since eεLm (X,Y ) ≥ 1. (ii) GN curvature bound. Define the random PSD matrix A(X) := Jθ (X)⊤ Hz (pθ (X))Jθ (X) ⪰ 0. Then HGN (θ; π) = E(X,Y )∼π [A(X)]. Since λmax is convex on the PSD cone (equivalently, by the variational characterization), λmax HGN (θ; π) = λmax E[A(X)] ≤ E λmax (A(X)) . For each realization X, λmax (A(X)) ≤ ∥Jθ (X)∥2op λmax Hz (pθ (X)) . Under ∥Jθ (X)∥op ≤ CJ π-a.s., λmax (A(X)) ≤ CJ2 λmax Hz (pθ (X)) . 19

Too Sharp, Too Sure: When Calibration Follows Curvature

By Lemma B.4, λmax (Hz (pθ (X))) ≤ 2(1 − pmax (X)),

pmax (X) := max pθ (X)k . k

Since pmax (X) ≥ pθ (X)Y , we have 1 − pmax (X) ≤ 1 − pθ (X)Y , hence λmax (Hz (pθ (X))) ≤ 2 1 − pθ (X)Y ≤ 2(K − 1)e−mθ (X,Y )

π-a.s.

Therefore, λmax HGN (θ; π) ≤ 2CJ2 (K−1) E(X,Y )∼π e−mθ (X,Y ) ≤ 2CJ2 (K−1) E(X,Y )∼π e−mε,θ (X,Y ) = 2CJ2 (K−1) Q(θ; π), where the last inequality uses Lemma B.5. If the local Lipschitz condition in Lemma B.5 holds, then also Q(θ; π) ≤ Q+ (θ; π). (iii) Certification statement. Immediate from (i)–(ii): if Q(θt ; π) ̸→ 0 (or likewise Q+ (θt ; π) ̸→ 0), then the corresponding right-hand sides do not converge to 0 and therefore cannot certify ECEM (θt ; π) → 0 nor λmax (HGN (θt ; π)) → 0. B.5. Proof of Theorem 4.2 Proof of Theorem 4.2. Assume γ(θ; D) > 0, i.e. mθ (xi , yi ) > 0 for all i. Hence Ybi = yi for all i (no tie-breaking occurs). (i) Two-sided ECE–moment bounds. Because Ybi = yi , every nonempty bin Bm has empirical accuracy acc(Bm ) = 1. Therefore, for each nonempty bin, 1 − conf(Bm ) = 1 − conf(Bm ) Hence ECEM (θ; D) =

since conf(Bm ) ∈ [0, 1].

M n X |Bm | 1Xb Pi . 1 − conf(Bm ) = 1 − n n i=1 m=1

Since Pbi = maxk pθ (xi )k = pθ (xi )Ybi and Ybi = yi , we have Pbi = pθ (xi )yi , and thus n

ECEM (θ; D) = Recalling QD (θ) := n1

i=1 e

−mθ (xi ,yi )

1X 1 − pθ (xi )yi . n i=1

, Lemma B.3 implies (since mθ (xi , yi ) ≥ 0 for all i) that

1 −mθ (xi ,yi ) e ≤ 1 − pθ (xi )yi ≤ (K − 1) e−mθ (xi ,yi ) . K Averaging over i and using (9) yields 1 QD (θ) ≤ ECEM (θ; D) ≤ (K − 1) QD (θ). K Finally, mθ (xi , yi ) ≥ γ(θ; D) implies QD (θ) ≤ e−γ(θ;D) . (ii) GN curvature bound. For each i, define Hz,i := Hz (pθ (xi )) and Ji := Jθ (xi ). Then n

HGN (θ; D) =

1X ⊤ J Hz,i Ji ⪰ 0. n i=1 i

Since λmax is convex on the PSD cone, n n 1X 1X λmax HGN (θ; D) ≤ λmax (Ji⊤ Hz,i Ji ) ≤ ∥Ji ∥2op λmax (Hz,i ). n i=1 n i=1

Under ∥Ji ∥op ≤ CJ for all i, n 1X λmax HGN (θ; D) ≤ CJ2 · λmax (Hz,i ). n i=1

(9)

Too Sharp, Too Sure: When Calibration Follows Curvature

By Lemma B.4, λmax (Hz,i ) ≤ 2(1 − pmax,i ). Since Ybi = yi , we have pmax,i = pθ (xi )yi , hence by Lemma B.3, λmax (Hz,i ) ≤ 2 1 − pθ (xi )yi ≤ 2(K − 1)e−mθ (xi ,yi ) . Therefore

n 1 X −mθ (xi ,yi ) e = 2CJ2 (K − 1) QD (θ). λmax HGN (θ; D) ≤ 2CJ2 (K − 1) · n i=1

1 (iii) Coupling to ECEM (rearranged lower bound). Combining the bound in (ii) with ECEM (θ; D) ≥ K QD (θ) from (i) yields

λmax HGN (θ; D) ≤ 2CJ2 K(K − 1) ECEM (θ; D),

ECEM (θ; D) ≥

equivalently

λmax (HGN (θ; D)) . 2CJ2 K(K − 1)

(iv) Robust-margin variant. Assume that for each (xi , yi ) there exists Lm (xi , yi ) ∈ [0, ∞) such that |mθ (xi + δ, yi ) − mθ (xi , yi )| ≤ Lm (xi , yi )∥δ∥

∀∥δ∥ ≤ ε.

Define the robust moments (as in Appendix E.3) 1 Q0ε,D (θ) :=

n X

n i=1

−mε,θ (xi ,yi )

1 Q− ε,D (θ) :=

n X

n i=1

e−εLm (xi ,yi ) e−mε,θ (xi ,yi ) .

By Lemma B.5, e−mθ (xi ,yi ) ≥ e−εLm (xi ,yi ) e−mε,θ (xi ,yi )

and

e−mθ (xi ,yi ) ≤ e−mε,θ (xi ,yi ) .

1 QD (θ) ≤ ECEM (θ; D) ≤ (K − 1)QD (θ) to obtain Insert these bounds into K

1 − Q (θ) ≤ ECEM (θ; D) ≤ (K − 1)Q0ε,D (θ). K ε,D For λmax (HGN (θ; D)), repeat the argument in (ii) and apply e−mθ (xi ,yi ) ≤ e−mε,θ (xi ,yi ) in the final step to get λmax HGN (θ; D) ≤ 2CJ2 (K − 1) Q0ε,D (θ).

C. Additional Sharpness–Calibration Experiments C.1. Sharpness–Calibration Correlation Analysis We present detailed training dynamics for each optimizer on CIFAR-10 and CIFAR-100, showing the co-evolution of loss, accuracy, ECE, margin, and sharpness throughout training. Since computing the full Hessian eigenvalue is expensive, these experiments use an MLP with a 5K/5K train/validation split. For each dataset, a scatter summary visualizes the temporal coupling across optimizers in a single view; per-optimizer figures report training (left) and validation (right) metrics across learning rates. C.2. Optimizer Details: SAM, Muon, and BulkSGD There is literature to support the notion that SAM may lead to improved calibration metrics, specifically that SAM act as an implicit regularizer and therefore prevents overfitting during training (Tan et al., 2026). At every step, SAM solves min max L(w + ϵ) w

||ϵ||≤ρ

which explicitly penalizes the sharpness of the Hessian and leads to convergence to flatter minima (Zhou et al., 2025). We train networks using SAM to test the first hypothesis, looking to confirm that flat minima lead to lower calibration error. 21

Too Sharp, Too Sure: When Calibration Follows Curvature

GD 0.20

Training ECE

0.15 0.10

BulkSGD

=2/50 =2/100 =2/150 =2/200

0.175 0.150

0.15 0.10

0.05

0.00

=2/50 =2/100 =2/150 =2/200

0.200

Training ECE

0.20

Training ECE

SGD

=2/50 =2/100 =2/150 =2/200

0.125 0.100 0.075 0.050 0.025

100 max

150

0.000 0

Batch Sharpness

AdamW

200

0.175

Muon

Training ECE

0.04

0.125 0.100 0.075 0.050

0.02

=2/50 =2/100 =2/150 =2/200

0.08

0.150

0.06

400 max

=2/50 =2/100 =2/150 =2/200

0.200

Training ECE

SAM

=2/500 =2/1000 =2/1500 =2/2000

0.08

0.06 0.04 0.02

0.025 0.00

0.000 0

0.00 0

max

20 start

30 max

end

15 max

Figure A1. ECE vs. GN sharpness trajectories (CIFAR-10). Each curve traces the joint evolution of ECE and GN sharpness (λmax ) across training steps for one optimizer and learning rate, with a filled circle marking the first training step and a cross (×) the last; color encodes learning rate. Trajectories lie near the diagonal, visualizing the temporal coupling between the two quantities.

Loss & Accuracy

0.05

0.2 0

2000

4000

6000

Step

8000

0.0 10000

0.4 2 × 10

2000

4000

6000

Step

8000

2000

4000

6000

Step

0.2

0.1

0.0 8000 10000

0.0

4000

6000

Step

lr = 2/50

8000

2000

4000

6000

Step

8000 10000

Sharpness

102

1.0 1.5 2.0

2000

0.2

0.5

0 0

0.3

Margin

0.0

Margin

Sharpness

10000

Sharpness

0.6

0.00

Margin

3 × 100

ECE

0.10

0.4

0.8

Accuracy (solid)

0.15

Sharpness

0.4

10 2

Loss (dashed)

0.20

ECE

0.6

10 1

Expected Calibration Error 1.0

0.8

Accuracy (solid)

Loss (dashed)

Loss & Accuracy

Expected Calibration Error 1.0

10000

lr = 2/100

lr = 2/150

2000

4000

6000

Step

8000

10000

2000

4000

6000

Step

lr = 2/50

lr = 2/200

(a) Training metrics

8000 10000

lr = 2/100

101

100

2000

lr = 2/150

4000

6000

Step

8000 10000

lr = 2/200

(b) Validation metrics

Figure A2. Gradient Descent (GD). Training dynamics (loss, accuracy, ECE, margin, sharpness) across four learning rates on CIFAR-10; training (left) and validation (right).

Too Sharp, Too Sure: When Calibration Follows Curvature

Loss & Accuracy

2500

5000

Step

7500

0.4 0

0.0 10000

2500

5000

Step

7500

10000

Sharpness

102

2500

5000

Step

7500

0.2 0.1 0.0

0.0 10000

2500

Margin

0.0

0.3

0.2

0.00

Margin

5000

Step

7500

10000

7500

10000

Sharpness

101

Sharpness

Margin

0.5

Sharpness

Margin

0.6

ECE

3 × 100

2 × 10

0.4

0.8

Accuracy (solid)

0.10 0.05

0.2 0

Loss (dashed)

0.4 10 2

0.15

ECE

0.6

10 1

Expected Calibration Error 1.0

0.20

0.8

Accuracy (solid)

Loss (dashed)

Loss & Accuracy

Expected Calibration Error 1.0

1.0 1.5

101

2.0

0 0

2500

5000

Step

7500

lr = 2/50

100

10000

lr = 2/100

2500

lr = 2/150

5000

Step

7500

10000

2500

5000

Step

7500

lr = 2/50

lr = 2/200

(a) Training metrics

100

10000

lr = 2/100

2500

lr = 2/150

5000

Step

lr = 2/200

(b) Validation metrics

Figure A3. Stochastic Gradient Descent (SGD). Training dynamics (loss, accuracy, ECE, margin, sharpness) across four learning rates on CIFAR-10; training (left) and validation (right).

0.05

0.2 10 3 0

2500

5000

Step

7500

0.0 10000

0.6 0.4

2 × 10 0

2500

5000

Step

7500

0.4

0.8 3 × 100

0.00

Margin

10000

Sharpness

2500

5000

Step

7500

0.3 0.2

0.2

0.1

0.0 10000

0.0 0

2500

Margin

0.0

5000

Step

7500

10000

7500

10000

Sharpness 102

Margin

Sharpness

101

100

Sharpness

0.5

Margin

0.10

Expected Calibration Error 1.0

ECE

0.4

10 2

Loss (dashed)

0.6

0.15

ECE

Loss & Accuracy

4 × 100

0.20

0.8

Accuracy (solid)

Loss (dashed)

Expected Calibration Error 1.0

Accuracy (solid)

Loss & Accuracy 0

1.0 1.5

101

2.0

0 0

2500

5000

Step

lr = 2/50

7500

10000

lr = 2/100

lr = 2/150

2500

5000

Step

7500

10000

2500

5000

Step

lr = 2/50

lr = 2/200

(a) Training metrics

7500

10000

lr = 2/100

lr = 2/150

2500

5000

Step

lr = 2/200

(b) Validation metrics

Figure A4. Sharpness-Aware Minimization (SAM). Training dynamics (loss, accuracy, ECE, margin, sharpness) across four learning rates on CIFAR-10; training (left) and validation (right).

Too Sharp, Too Sure: When Calibration Follows Curvature

Loss & Accuracy

0.05

0.2 0

2500

5000

Step

7500

2500

Margin

5000

Step

7500

Sharpness

Margin

Sharpness

Margin

5.0

2500

101

5000

Step

7500

0.0

0.0 10000

5000

Step

7500

10000

7500

10000

Sharpness 102

2 4

101

2.5 0.0

2500

Margin

10.0

0.2

0.2 0

10000

12.5

7.5

0.4

3 × 100

0.4

ECE

0.6

4 × 10

2 × 100

0.00

0.0 10000

0.8

Accuracy (solid)

0.4

0.10

6 × 100

0.6

Sharpness

0.6

10 2 10 3

0.15

Loss (dashed)

ECE

Expected Calibration Error 1.0

0.8

Accuracy (solid)

Loss (dashed)

Loss & Accuracy

Expected Calibration Error 1.0

6 0

2500

5000

Step

7500

lr = 2/50

10000

lr = 2/100

lr = 2/150

2500

5000

Step

7500

10000

2500

5000

Step

7500

lr = 2/50

lr = 2/200

(a) Training metrics

10000

lr = 2/100

2500

lr = 2/150

5000

Step

lr = 2/200

(b) Validation metrics

Figure A5. Muon. Training dynamics (loss, accuracy, ECE, margin, sharpness) across four learning rates on CIFAR-10; training (left) and validation (right).

0.4

10 3

0.2

10 4 0

2500

5000

Step

7500

0.050 0.025 0

2500

0.6

3 × 100

0.4

Step

7500

10000

Sharpness

101

10.0

5000

0.5

0.8 4 × 100

2 × 100

0.000

0.0 10000

Margin

2500

5000

Step

7500

0.4

2.5

0.1

0.0 10000

0.0 0

10 1

2500

5000

Step

lr = 2/500

7500

10000

lr = 2/1000

lr = 2/1500

2500

5000

Step

7500

2 3

2500

5000

Step

lr = 2/500

lr = 2/2000

(a) Training metrics

Step

7500

10000

7500

10000

106 104 102

10000

5000

Sharpness

0.0

2500

108

Sharpness

Margin

Sharpness

5.0

0.2

Margin

100

0.3

0.2

7.5

Margin

0.075

Accuracy (solid)

Loss (dashed)

Accuracy (solid) ECE

Loss (dashed)

0.6

Expected Calibration Error 1.0

6 × 100

0.100

0.8

10 1

Loss & Accuracy

Expected Calibration Error 1.0

ECE

Loss & Accuracy 100

7500

10000

lr = 2/1000

lr = 2/1500

2500

5000

Step

lr = 2/2000

(b) Validation metrics

Figure A6. AdamW. Training dynamics (loss, accuracy, ECE, margin, sharpness) across four learning rates on CIFAR-10; training (left) and validation (right).

Too Sharp, Too Sure: When Calibration Follows Curvature

2500

5000

Step

7500

0.10 0.05

0.2 0

Loss (dashed)

0.4 10 2

ECE

0.6

10 1

0.15

0.6 0.4

2 × 10

0.0 10000

2500

5000

7500

Step

Sharpness

2500

7500

Step

0.1

0.0 10000

0.0 0

101

Sharpness

Margin

Sharpness

2500

5000

Step

7500

10000

7500

10000

Sharpness

0.5 102

0.2

Margin

0.0

5000

0.3

0.2

10000

0.4

0.8

3 × 100

0.00

Margin

Expected Calibration Error 1.0

ECE

0.8

Loss & Accuracy

4 × 100

0.20

Accuracy (solid)

Loss (dashed)

Expected Calibration Error 1.0

Accuracy (solid)

Loss & Accuracy 0

1.0 1.5

101

2.0 100

2500

5000

Step

7500

lr = 2/50

10000

2500

lr = 2/100

lr = 2/150

5000

Step

7500

10000

2500

5000

7500

Step

lr = 2/50

lr = 2/200

(a) Training metrics

10000

lr = 2/100

2500

lr = 2/150

5000

Step

lr = 2/200

(b) Validation metrics

Figure A7. BulkSGD. Training dynamics (loss, accuracy, ECE, margin, sharpness) across four learning rates on CIFAR-10; training (left) and validation (right).

SGD 0.40

=2/50 =2/100 =2/150 =2/200

0.35

0.35 0.30

Training ECE

0.25 0.20 0.15

0.30

0.25 0.20 0.15

0.10

0.05

0.00

0.00 0

100 max

0.30

0.15 0.10

Batch Sharpness

0.25 0.20 0.15

4 max

400

Muon

0.15 0.10 0.05

0.00 0

200 300 max

=2/50 =2/100 =2/150 =2/200

0.20

0.05

0.00

100

0.25

0.10

0.05

=2/50 =2/100 =2/150 =2/200

0.35

Training ECE

0.20

SAM

0.40

=2/500 =2/1000 =2/1500 =2/2000

=2/50 =2/100 =2/150 =2/200

0.00

150

AdamW

0.25

Training ECE

0.35

Training ECE

0.30

BulkSGD 0.40

=2/50 =2/100 =2/150 =2/200

Training ECE

0.40

0.00 0

20 start

30 max

40 end

2.5

5.0

7.5 10.0 max

Figure A8. ECE vs. GN sharpness trajectories (CIFAR-100). Same format as Figure A1.

12.5

15.0

Too Sharp, Too Sure: When Calibration Follows Curvature

0.1

0.2 2000

4000

6000

8000

Step

0.2

2000

Margin

4000

6000

Step

8000

2000

4000

0.1 0.0

0.0 8000 10000

6000

Step

4000

6000

Step

8000 10000

Sharpness

101

2000

Margin

Sharpness

0.2

0.2 0

10000

102

0.3

0.6 0.4

Sharpness

Margin

5 × 100

4 × 100

0.0

0.0 10000

0.8

Accuracy (solid)

10 1

0.3

101

Sharpness

0.4

Expected Calibration Error 1.0

6 × 100

Loss (dashed)

0.6

ECE

Accuracy (solid)

Loss (dashed)

0.4

0.8

100

Loss & Accuracy

Expected Calibration Error 1.0

ECE

Loss & Accuracy

100

5 0

2000

4000

6000

8000

Step

lr = 2/50

10000

lr = 2/100

lr = 2/150

2000

4000

6000

Step

8000

10 1 0

10000

2000

4000

6000

Step

8000 10000

lr = 2/100

lr = 2/150

lr = 2/50

lr = 2/200

(a) Training metrics

2000

4000

6000

Step

8000 10000

lr = 2/200

(b) Validation metrics

Figure A9. Gradient Descent (GD) — CIFAR-100. Training dynamics (loss, accuracy, ECE, margin, sharpness) across learning rates on CIFAR-100; training (left) and validation (right).

4000

6000

Step

8000

2000

Margin

4000

6000

Step

8000

Sharpness

Margin

Sharpness

Margin

100

6000

Step

lr = 2/50

8000

0.1

5000

Step

7500

0.0

0.0 10000

2500

Margin

5000

Step

7500

10000

7500

10000

Sharpness

2 3

101 100

4 10 1

10 1 4000

0.2

101

0 2000

2500

0.6

0.2 0

10000

102

0.3

0.4 4 × 100

0.0

0.0 10000

5 × 10

0.8

Accuracy (solid)

0.1

0.2 2000

0.2

Sharpness

10 1

Loss (dashed)

0.4

ECE

Accuracy (solid)

Loss (dashed)

0.6

0.3

Expected Calibration Error 1.0

6 × 100

0.4

0.8

100

Loss & Accuracy

Expected Calibration Error 1.0

ECE

Loss & Accuracy

10000

lr = 2/100

lr = 2/150

2000

4000

6000

Step

8000

10000

2500

5000

Step

lr = 2/50

lr = 2/200

(a) Training metrics

7500

10000

lr = 2/100

lr = 2/150

2500

5000

Step

lr = 2/200

(b) Validation metrics

Figure A10. Stochastic Gradient Descent (SGD) — CIFAR-100. Training dynamics (loss, accuracy, ECE, margin, sharpness) across learning rates on CIFAR-100; training (left) and validation (right).

Too Sharp, Too Sure: When Calibration Follows Curvature

0.3

0.6 0.4

10 1

0.1

0.2 0

2000

4000

6000

8000

Step

0.2

5 × 100

2000

Margin

4000

6000

Step

8000

0.3

0.6

0.2

Sharpness

0.1

0.2 0

10000

2500

5000

Step

7500

0.0

0.0 10000

2500

Margin

5000

Step

7500

10000

7500

10000

Sharpness

101

Sharpness

Margin

Sharpness

Margin

0.8

0.4 4 × 100

0.0

0.0 10000

Expected Calibration Error 1.0

ECE

0.8

Loss & Accuracy

6 × 100

Accuracy (solid)

100

Expected Calibration Error

Loss (dashed)

0.4

ECE

1.0

Accuracy (solid)

Loss (dashed)

Loss & Accuracy

2 3

101

5 0

2000

4000

6000

8000

Step

lr = 2/50

10000

lr = 2/100

lr = 2/150

2000

4000

6000

Step

8000

10000

2500

5000

Step

7500

lr = 2/50

lr = 2/200

(a) Training metrics

10000

lr = 2/100

2500

lr = 2/150

5000

Step

lr = 2/200

(b) Validation metrics

Figure A11. Sharpness-Aware Minimization (SAM) — CIFAR-100. Training dynamics (loss, accuracy, ECE, margin, sharpness) across learning rates on CIFAR-100; training (left) and validation (right).

0.6

10 1

0.4 10 2

0.1

0.2 2000

4000

6000

Step

8000

0.2

0.6

4 × 10 0

2000

Margin

4000

6000

Step

8000

10000

Sharpness

2500

0.4 0.2

5000

Step

7500

0.0

0.0 10000

2500

Margin

0.0

10.0

5000

Step

7500

10000

7500

10000

Sharpness

Margin

Sharpness

5.0

101

2.5

Sharpness

2.5

7.5

Margin

0.4

6 × 100

0.2

0.0

0.0 10000

0.6

0.8

101

ECE

0.3

Accuracy (solid)

0.8

Expected Calibration Error 1.0

Loss (dashed)

0.4

ECE

100

Loss & Accuracy

Expected Calibration Error 1.0

Accuracy (solid)

Loss (dashed)

Loss & Accuracy

5.0 7.5

101

10.0

0.0

12.5 0

2000

4000

6000

Step

lr = 2/50

8000

10000

lr = 2/100

lr = 2/150

2000

4000

6000

Step

8000

10000

2500

5000

Step

lr = 2/50

lr = 2/200

(a) Training metrics

7500

10000

lr = 2/100

lr = 2/150

2500

5000

Step

lr = 2/200

(b) Validation metrics

Figure A12. Muon — CIFAR-100. Training dynamics (loss, accuracy, ECE, margin, sharpness) across learning rates on CIFAR-100; training (left) and validation (right).

Too Sharp, Too Sure: When Calibration Follows Curvature

0.15

0.4

10 2 10 3 2500

5000

Step

7500

0.10

0.2

0.05

0.0 10000

0.00

0.8 6 × 10

0.4 4 × 100

2500

Margin

5000

Step

7500

10000

Sharpness

2500

0.2

5000

Step

7500

0.0

0.0 10000

0 2500

5000

Step

7500

lr = 2/500

4 6

10 1 0

2500

lr = 2/1000

lr = 2/1500

5000

Step

7500

10000

7500

10000

103 101

10000

7500

105

10000

Step

107

Sharpness

Margin

Sharpness

100

5000

Sharpness

2500

Margin

101

Margin

0.4

0.2 0

0.6

ECE

0.20

0.6

Accuracy (solid)

0.8

10 1

Expected Calibration Error

0.6

1.0

Loss (dashed)

100

ECE

0.25

Loss & Accuracy

Expected Calibration Error 1.0

Accuracy (solid)

Loss (dashed)

Loss & Accuracy

2500

5000

Step

7500

lr = 2/500

lr = 2/2000

(a) Training metrics

10000

lr = 2/1000

2500

lr = 2/1500

5000

Step

lr = 2/2000

(b) Validation metrics

Figure A13. AdamW — CIFAR-100. Training dynamics (loss, accuracy, ECE, margin, sharpness) across learning rates on CIFAR-100; training (left) and validation (right).

0.1

0.2 2000

4000

6000

Step

8000

2000

Margin

6000

Step

8000

2500

5000

Step

7500

4000

6000

Step

lr = 2/50

8000

101

lr = 2/100

lr = 2/150

2000

4000

6000

Step

8000

2500

5000

Step

lr = 2/50

lr = 2/200

(a) Training metrics

Step

7500

10000

7500

10000

101 100 10 1

10000

5000

102

10000

2500

Sharpness

5 2000

0.0

0.0 10000

102

0 0

0.1

Margin

Sharpness

0.2

0.2 0

10000

Sharpness

Margin

4000

0.3

0.6 0.4

4 × 100

0.0

0.0 10000

0.8

Accuracy (solid)

10 1

0.2

5 × 100

Sharpness

0.4

0.3

Expected Calibration Error 1.0

6 × 100

Loss (dashed)

0.6

ECE

Accuracy (solid)

Loss (dashed)

0.4

0.8

100

Loss & Accuracy

Expected Calibration Error 1.0

ECE

Loss & Accuracy

7500

10000

lr = 2/100

lr = 2/150

2500

5000

Step

lr = 2/200

(b) Validation metrics

Figure A14. BulkSGD — CIFAR-100. Training dynamics (loss, accuracy, ECE, margin, sharpness) across learning rates on CIFAR-100; training (left) and validation (right).

Too Sharp, Too Sure: When Calibration Follows Curvature

To test the second hypothesis, we apply optimizers that explicitly suppress the contribution of eigenvectors associated with directions of steep descent, through Muon and BulkSGD. Muon rescales the gradient components at each update, so all directions contribute with comparable magnitude. This means directions of steep descent are clamped, while the flatter directions are amplified (Jordan et al., 2024). Another method to suppress directions of steepest descent is using BulkSGD, which at each step projects the gradient to the space orthogonal to the subspace spanned by the top eigenvectors. We try projecting out the top eigenvector, as well as the top three and five eigenvectors (Song et al., 2025). We note that with BulkSGD, we entirely omit the directions of steepest descent, while with Muon we still allow small updates to be made in those directions. For BulkSGD, training suffers from high levels of instability depending on the number of dominant eigenvectors that are projected out. We observe that training loss is still minimized over 100,000 steps, however the trajectory features steep oscillations. Similarly, sharpness explodes to values in the thousands, which has not been previously observed with other optimizers. This could be due to the fact that, with the dominant eigenvectors projected out, the gradient continues to remain in areas of high curvature, without following the directions of steepest descent.

D. CalMO: Extended Results D.1. Per-Optimizer Training Dynamics To evaluate out-of-sample calibration, we train ResNet-20 on CIFAR-10 using the full dataset (45K training / 5K validation split). We compare standard cross-entropy loss against CalMO. Each plot shows accuracy, ECE, the margin functional Q(θ) = E[e−m ], and loss. For each optimizer, we report both training metrics (left) and validation metrics (right). Lines show mean over 3 seeds; shaded regions indicate ±1 standard deviation (multiplicative for log-scale plots). Accuracy

Accuracy

Expected Calibration Error

0.02

0.2 4000

6000

Step [e m]

8000

10000

100

4000

6000

Step

8000

10000

10 3 2000

4000

6000

Step

8000

10000

2000

104

4000

6000

Step [e m]

8000

10000

Cross-Entropy

2000

4000

6000

Step

8000

(a) Training metrics

8000

10000

8000

10000

100 6 × 10 1

2000

4000

6000

Step

8000

10000

Cross-Entropy

CalMO

6000

Step

100

10000

4000

Loss

102 10

2000

2 × 100

103

10 3 0

0.02

Q( )

10 1 10

2000

Loss

101

Loss

Q( )

2000

0.0

0.06 0.04

0.2

0.00 0

0.4

Loss

0.4

0.08

0.6

ECE

0.04

0.10

0.8

Accuracy

0.06

0.6

ECE

Accuracy

0.8

0.0

Expected Calibration Error

1.0

2000

4000

6000

Step

CalMO

(b) Validation metrics

Figure A15. SGD: CE vs. CalMO. Training dynamics on ResNet-20/CIFAR-10; training (left) and validation (right), mean ±1 std over 3 seeds.

D.2. Train–Test Calibration Gap Table A1 extends Table 2 with training accuracy and training ECE. All methods reach near-perfect training accuracy and near-zero training ECE after interpolation, confirming that the differences observed on the test set reflect generalization of calibration rather than training dynamics. D.3. Benchmark vs. Intrinsic Calibration Methods Table A2 compares CalMO to commonly used intrinsic calibration methods—label smoothing and focal loss—across optimizers on CIFAR-10 with ResNet-20 with a 90/10 train/validation split. For each model, we compute accuracy and 29

Too Sharp, Too Sure: When Calibration Follows Curvature Accuracy

Accuracy

Expected Calibration Error

1.0

0.02

0.6

ECE

0.4

0.04

Accuracy

ECE

0.4 0.2

0.00 2000

4000

6000

Step [e m]

8000

10000

Loss

100 10 1

10 3

10 3 6000

Step

8000

10000

8000

10000

0.06 0.04 0.02

2000

105

4000

6000

8000

Step [e m]

10000

2000

4000

6000

Step

8000

10000

8000

10000

Loss 2 × 100

104

10 2

4000

6000

Step

10 2

2000

4000

Loss

101

2000

Q( )

0.0

103

Loss

Accuracy

0.6

0.2

Q( )

0.08

0.8

0.0

Expected Calibration Error

1.0

0.06

102 101

100 6 × 10 1

100 0

Cross-Entropy

2000

4000

6000

Step

8000

10000

2000

4000

6000

8000

Step

10000

Cross-Entropy

CalMO

(a) Training metrics

2000

4000

6000

Step

CalMO

(b) Validation metrics

Figure A16. AdamW: CE vs. CalMO. Training dynamics on ResNet-20/CIFAR-10; training (left) and validation (right), mean ±1 std over 3 seeds. Accuracy

Expected Calibration Error

1.0

Accuracy

0.02

0.2

0.4

2000

4000

6000

Step [e m]

8000

10000

2000

4000

6000

Step

8000

0.0

10000

0.025 0

2000

Loss 10

100

Q( )

Loss

101 10 1

4000

6000

8000

Step [e m]

10000

2000

4000

6000

Step

8000

10000

8000

10000

Loss 2 × 100

108

Loss

0.00

0.075 0.050

0.2

102

Q( )

0.100

0.6

ECE

Accuracy

0.4

0.125

0.8

0.04

0.6

ECE

Accuracy

0.8

0.0

Expected Calibration Error

1.0

0.06

105

100

6 × 10 1

102

10 2

10 2 0

2000

4000

6000

Step

8000

10000

Cross-Entropy

2000

4000

6000

Step

8000

10000

2000

4000

6000

8000

Step

CalMO

10000

Cross-Entropy

(a) Training metrics

2000

4000

6000

Step

CalMO

(b) Validation metrics

Figure A17. Muon: CE vs. CalMO. Training dynamics on ResNet-20/CIFAR-10; training (left) and validation (right), mean ±1 std over 3 seeds. Train Acc (%)

Train ECE

Test Acc (%) ↑

Test ECE ↓

SGD

CE CalMO

100.0 ± 0.0 100.0 ± 0.0

0.000 ± 0.000 0.004 ± 0.000

75.2 ± 0.9 80.1 ± 1.1

0.081 ± 0.017 0.056 ± 0.001

AdamW

CE CalMO

100.0 ± 0.0 100.0 ± 0.0

0.000 ± 0.000 0.004 ± 0.000

80.7 ± 0.3 83.2 ± 0.9

0.061 ± 0.005 0.045 ± 0.005

SAM

CE CalMO

100.0 ± 0.0 100.0 ± 0.0

0.006 ± 0.002 0.014 ± 0.001

85.0 ± 0.2 85.2 ± 0.3

0.020 ± 0.004 0.017 ± 0.004

Muon

CE CalMO

99.5 ± 0.0 99.9 ± 0.0

0.003 ± 0.001 0.008 ± 0.000

80.3 ± 0.3 81.7 ± 0.7

0.065 ± 0.013 0.019 ± 0.002

Optimizer Loss

Table A1. Train–test calibration gap. CE vs. CalMO on ResNet-20/CIFAR-10; extends Table 2 with training metrics.

Too Sharp, Too Sure: When Calibration Follows Curvature

0.06

0.4

Accuracy

0.08

0.6

ECE

0.04

0.10

0.8

0.08

0.6

0.06

0.4

0.02

0.2

0.0

0.00

0.0

2000

4000

6000

Step [e m]

8000

10000

2000

4000

6000

Step

8000

10000

Expected Calibration Error

1.0

0.2

2000

4000

6000

Step [e m]

8000

10000

4000

6000

Step

8000

10000

8000

10000

Loss

10 1

101

Loss

2000

2 × 100

100

10 1

0.04 0.02

Loss

Q( )

Accuracy

0.8

100

Q( )

Accuracy

Expected Calibration Error 0.10

ECE

Accuracy 1.0

100 6 × 10 1

10 2

10 2 0

2000

4000

6000

Step

8000

10000

Cross-Entropy

2000

4000

6000

Step

8000

100

10000

4 × 10 1 0

2000

4000

6000

Step

8000

10000

Cross-Entropy

CalMO

(a) Training metrics

2000

4000

6000

Step

CalMO

(b) Validation metrics

Figure A18. SAM: CE vs. CalMO. Training dynamics on ResNet-20/CIFAR-10; training (left) and validation (right), mean ±1 std over 3 seeds.

Optimizer

Method

Acc (%)

ECE ↓

SGD

CE Label Smooth. Focal Loss CalMO

75.2 ± 1.2 81.0 ± 1.0 78.0 ± 2.9 80.1 ± 1.4

0.081 ± 0.021 0.082 ± 0.009 0.029 ± 0.018 0.056 ± 0.001

AdamW

CE Label Smooth. Focal Loss CalMO

80.7 ± 0.4 85.8 ± 0.1 80.5 ± 1.3 83.2 ± 1.1

0.061 ± 0.007 0.059 ± 0.002 0.042 ± 0.019 0.045 ± 0.007

Muon

CE Label Smooth. Focal Loss CalMO

80.3 ± 0.3 82.4 ± 0.7 78.7 ± 0.5 81.7 ± 0.9

0.065 ± 0.016 0.045 ± 0.003 0.025 ± 0.017 0.019 ± 0.002

SAM

CE Label Smooth. Focal Loss CalMO

85.0 ± 0.2 86.2 ± 0.5 84.0 ± 1.0 85.8 ± 0.3

0.020 ± 0.004 0.067 ± 0.004 0.089 ± 0.010 0.017 ± 0.005

Table A2. Accuracy and ECE at the best-validation step for CE, label smoothing, focal loss, and CalMO on ResNet-20/CIFAR-10 (90/10 train/validation split).

Too Sharp, Too Sure: When Calibration Follows Curvature

ECE at the training step that minimizes loss on the validation set. The results highlight that the effectiveness of calibration interventions is optimizer-dependent. For SGD and AdamW, focal loss or label smoothing often achieve the lowest ECE, consistent with prior observations that these methods implicitly regularize confidence. In contrast, for Muon, CalMO yields the largest reduction in ECE while maintaining competitive accuracy. Similarly for SAM, the flatness and robustness terms alone do not control ECE; their combination leads to the lowest miscalibration while maintaining a high accuracy. Across optimizers, CalMO tends to strike a favorable balance between predictive accuracy and calibration error, avoiding the larger accuracy–ECE trade-offs exhibited by some single-mechanism baselines.

E. Extension to Mean Squared Error E.1. GD and SGD Experiments We rerun the experimental setup of Section 3 with mean squared error (MSE) loss in place of cross-entropy, on 5000 CIFAR-10 training samples. Figures A19 and A20 report the training dynamics for gradient descent and stochastic gradient descent, respectively.

Figure A19. CIFAR-10 | Optimizer: Gradient Descent | Loss: Mean Squared Error

Figure A20. CIFAR-10 | Optimizer: Stochastic Gradient Descent | Loss: Mean Squared Error

The results show that MSE is extremely miscalibrated, resulting in severely underconfident models as evidenced by the reliability diagrams. This is a consequence of the fact that MSE is not a proper scoring rule. For LMSE , treating zθ (xi ) as free variables, the unique minimizer for a single example is ( 1, k = yi , ⋆ zk (xi ) = 0, k ̸= yi . 32

Too Sharp, Too Sure: When Calibration Follows Curvature

Thus the loss penalizes pushing zθ (xi )yi above 1 or the other logits below 0. However, when using the logits in the softmax before the ECE computation, this finite target pattern leads to underconfidence. In fact, at the optimum, the confidence is Pb(xi ) = pθ (xi )yi =

e e1 = ≈ 0.23 e1 + (K − 1)e0 e+K −1

for K = 10.

Consequently, in regimes where training accuracy is close to 1 but logits are near this finite pattern, the model is systematically underconfident on the training set (accuracy ≈ 1 vs. confidence ≈ 0.23 in the main bin), and the training ECE remains large instead of decaying towards zero as in the CE case. E.2. Asymptotics of MSE For MSE,

2 IK , K so the logit-level Hessian is constant and does not depend on the predicted probabilities pθ (xi ). Hence, unlike CE, the Gauss–Newton curvature does not attenuate as the model improves its fit: MSE Hz,i (θ) = ∇2zi LMSE (zi , yi ) =

MSE HGN (θ) =

1X 2 1X MSE · Ji (θ)⊤ Hz,i Ji (θ) = Ji (θ)⊤ Ji (θ), n i K n i

MSE so the eigenvalues of HGN (θ) are governed entirely by the Jacobians Ji (θ), with no probability-dependent factor diag(p) − ⊤ pp to drive curvature toward zero. Combined with the underconfidence analysis of Appendix E.1, this explains why neither sharpness nor training ECE collapse under MSE in the interpolation regime, in contrast with the CE case.

Record · ID 124064 · SHA-256 e8676733bf317f3b
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.