MOMENT-DP: Momentum Aware Evaluation of Correlated Noise for Differentially Private Training

MOMENT-DP: Momentum-Aware Evaluation of Correlated Noise for Differentially Private Training

arXiv:2611.11457v1 [cs.LG] 8 Nov 2026

Sunny Gupta Google Cloud sunnyguptaa@google.com

Raja Viswanadha Google Cloud rajaviswanadha@google.com

Abstract Temporally correlated noise can substantially improve the utility of differentially private optimization, yet candidate mechanisms are commonly designed or evaluated using only partial information about the optimization process, such as workload structure, learning-rate schedules, or local curvature. We study whether persistent optimizer state provides additional information that can change the relative utility of privacy-valid correlated-noise mechanisms. We introduce M OMENT-DP, an optimizer-state-aware framework that models how temporally correlated privacy noise propagates through momentum-based optimization. For locally quadratic objectives, M OMENT-DP expresses privacyinduced final loss through an optimizer transfer operator that jointly captures curvature, learning-rate scheduling, momentum, and temporal noise covariance. NoiseCurve’s curvature-aware quadratic criterion is recovered exactly as the zeromomentum, constant-learning-rate special case. Changing only momentum while holding curvature, schedule, privacy calibration, and candidate mechanisms fixed induces genuine mechanism-ranking reversals. The effect persists outside exact quadratic dynamics in per-example-clipped private logistic regression, while remaining unresolved in the neural regime we study. These results establish optimizer state as an under-modeled, decision-relevant axis in evaluating temporally correlated privacy noise.

1

Introduction

Differentially private stochastic gradient descent (DP-SGD) protects training examples by clipping individual gradients and adding calibrated Gaussian noise [1]. Rather than injecting independent perturbations at every iteration, DP-FTRL and matrix-factorization (MF) mechanisms correlate privacy noise over time and can substantially improve privacy–utility trade-offs [7, 4, 3]. This raises a basic question: how should temporal privacy-noise covariances be ranked for learning? Classical MF criteria operate largely at the workload level. More recent approaches incorporate additional optimization information: learning-rate-aware MF explicitly models the schedule [8], while NoiseCurve incorporates local curvature [6]. We investigate another information axis that remains under-modeled in this mechanism-evaluation lineage: persistent optimizer state. Momentum is not merely a learning-rate rescaling. Earlier gradients and privacy perturbations repeatedly influence later updates through the momentum buffer. Consequently, two privacy-valid covariances can exchange ordering as momentum changes even when curvature, learning-rate schedule, privacy calibration, and the candidate mechanisms remain fixed. We formalize this interaction using a state-space representation of private momentum SGD. For locally quadratic optimization, temporally correlated privacy noise with covariance K contributes expected final loss Privacy in the Era of Large Opaque Models (PriLOM) Workshop at NeurIPS 2026.

(a) State-Space Privacy Noise Transfer

Z ∼

N (0, σ 2 (K ⊗ Id ))

Momentum Buffer vt+1 =

(c) Private Logistic Regression (30 Seeds)

(b) Quadratic Damage vs. Momentum µ Parameter Error et+1 = et − ηt vt+1

µvt + Het + zt

State Memory (µvt )

Evaluator Information Frontier: • Static MF: Workload A (static prefix sums) • LR-Aware MF: Workload A, schedule {ηt } • NoiseCurve: Curvature H, constant η, µ = 0 • M OMENT-DP: Transfer G(H, {ηt }, µ) (State-Aware)

0.3

Excess Loss (Mean ± SE)

z0 , . . . , zT −1

Expected Damage Ddyn

Correlated Noise

Dense Optimal MF

(BandMF)

IID Gaussian (DP-SGD)

NoiseCurve

Rank 9 (Worst)

0.12

BandMF / Toeplitz

0.2

Rank 1

0.14

NoiseCurve (Native µ = 0)

0.25

Best Competitor

0.1

0.08

0.15

0.06

Rank Reversal Region Dense MF wins at µ ≥ 0.5

0.1

0.04 0.02

0.05 0

0.2

0.4 0.5

0.8

0.95

Rank 1

Rank 1

Rank 1

(BISR-last) (BISR-max)

(BandMF)

Rank 2

Rank 9

Rank 9

µ = 0.9 Cosine

µ = 0.9 W-Cos

0 µ=0 Const

Momentum Parameter µ

µ = 0.9 Const

Figure 1: M OMENT-DP overview. (a) Temporally correlated privacy perturbations are recursively filtered through persistent momentum state before affecting parameter error. (b) Holding curvature, learning rate, privacy calibration, and candidate covariances fixed, changing only momentum changes the oracle-best mechanism: NoiseCurve is optimal at µ = 0, while Dense MF is optimal at every tested µ ≥ 0.5. (c) In per-example-clipped private logistic regression (30 paired seeds), NoiseCurve changes from rank 2/9 at zero momentum to rank 9/9 in each of the three tested momentum conditions. Ddyn =

 σ2  tr HG(K ⊗ Id )G⊤ , 2

(1)

where the transfer operator G depends on curvature, the full learning-rate schedule, momentum, and the optimization horizon. A central result places Eq. (1) directly in the context of prior work: NoiseCurve’s exact quadratic criterion is precisely its zero-momentum, constant-learning-rate specialization. Thus, our question is not whether curvature matters—NoiseCurve already establishes that it does—but whether persistent optimizer state provides additional information needed to rank privacy-valid temporal covariances. Contributions. We make three contributions: (i) an exact optimizer-state-aware transfer formulation for temporally correlated privacy noise under quadratic momentum dynamics; (ii) an exact reduction to NoiseCurve’s quadratic objective at zero momentum and constant learning rate, numerically verified across 576 configurations; and (iii) controlled and nonlinear evidence that changing only optimizer state can reverse the relative utility of privacy-valid mechanisms, together with an explicit negative neural result.

2

MOMENT-DP

2.1

State-space formulation

Consider f (θ) = 12 (θ − θ⋆ )⊤ H(θ − θ⋆ ), H ⪰ 0, and let et = θt − θ⋆ . The stacked privacy perturbations satisfy Z = [z0⊤ , . . . , zT⊤−1 ]⊤ ∼ N (0, σ 2 (K ⊗ Id )),

where K ∈ RT ×T is the temporal covariance. Momentum SGD obeys vt+1 = Het + µvt + zt ,

et+1 = et − ηt vt+1 .

(2)

⊤ ⊤ With st = [e⊤ t , vt ] , the dynamics are st+1 = At st + Bt zt , where     I − ηt H −ηt µI −ηt I At = , Bt = . H µI I (z)

Let G map the stacked perturbations to final parameter error, eT = GZ. Then (z)

Cov(eT ) = σ 2 G(K ⊗ Id )G⊤ , and therefore

 σ2  tr HG(K ⊗ Id )G⊤ . (3) 2 Proposition 1 (Exact quadratic damage). Under fixed quadratic dynamics, predetermined learning rates, momentum µ, and temporal covariance K, Eq. (3) equals the exact expected final objective contribution caused solely by privacy noise. Ddyn =

2

The proof is given in Appendix A. Since M OMENT-DP is exact in this domain, quadratic ranking accuracy is calibration rather than independent predictive evidence. 2.2

NoiseCurve as an exact special case

For µ = 0 and constant η, Hessian eigenmode i satisfies (i)

(i)

(i)

et+1 = (1 − ηλi )et − ηzt .

(4)

Its temporal transfer vector therefore obeys gi = −ηVi,: , where Vij = (1 − ηλi )T −j−1 . Writing M = diag(λ1 , . . . , λd ), X = C ⊤ C, and K = X −1 gives
η2 σ2 tr X −1 V ⊤ M V , (5) 2 which is exactly NoiseCurve’s quadratic final-loss criterion [6]. Importantly, the identity is mode-wise: the full G ∈ Rd×dT is not identified with V ∈ Rd×T . Ddyn =

We verify this identity over 576 configurations; the worst relative numerical discrepancy is 9.7×10−16 . Full derivation and validation are in Appendices A and B. Momentum-dependent rankings. ∆(µ) =

For mechanisms a and b,  σ2  tr HGµ ((Ka − Kb ) ⊗ Id )G⊤ µ . 2

If ∆(µ1 )∆(µ2 ) < 0, their oracle ordering reverses. Any criterion independent of µ assigns the same ordering at both momentum values and therefore cannot be correct at both.

3

Experiments and Results

We evaluate IID Gaussian noise, dense MF, BandMF, Toeplitz MF, BISR, learning-rate-aware MF, and NoiseCurve under matched single-participation privacy semantics. Controlled experiments use (ε, δ) = (4, 10−5 ). Learning-rate-aware MF and NoiseCurve are independently reproduced and validated. To reduce outcome-informed selection, analytic crossover discovery is separated from downstream evaluation: cases are selected analytically, serialized and SHA-256 frozen, and then evaluated using an independent NumPy simulator. Full mechanism, privacy, solver, and protocol details appear in Appendices B–D. 3.1

Controlled momentum crossovers

All 12 frozen competitive quadratic crossover cases are independently confirmed. Because M OMENTDP is exact in this domain, the result establishes the existence and reproducibility of the crossovers rather than independent predictive superiority. Holding curvature and learning-rate schedule fixed while varying µ ∈ {0, 0.5, 0.9, 0.95}, the oraclebest mechanism changes in 7/9 tested curvature–schedule slices. The faithfully reproduced learningrate-aware ranking changes in 0/7 of those seven momentum-driven crossover slices. Across the broader exact grid, pairwise ranking accuracy is 58.0% for static MeanSE, 60.8% for learning-rate-aware MF, and 75.8% for a curvature-and-schedule score evaluated with momentum forced to zero. Full results are in Appendix E. 3.2

Direct NoiseCurve intervention

NoiseCurve is optimized once in its native condition (µ = 0, constant learning rate). We then hold H, η, privacy calibration, and the candidate pool fixed, and change only momentum. NoiseCurve correctly wins in its native zero-momentum regime. Its exact criterion contains no momentum parameter, so its score does not change under this intervention. The result should therefore not be interpreted as a defect of NoiseCurve; instead, it demonstrates that changing an optimizer-state variable absent from the zero-momentum criterion can change the preferred mechanism. 3

Table 1: Changing only momentum changes the oracle-best mechanism. µ

Oracle best

NC score

M OMENT-DP

0.00 0.50 0.80 0.90 0.95

NoiseCurve Dense MF Dense MF Dense MF Dense MF

correct wrong wrong wrong wrong

correct correct correct correct correct

Table 2: NoiseCurve under optimizer interventions in private logistic regression (mean ± SE over 30 paired seeds). Condition µ = 0, constant µ = .9, constant µ = .9, cosine µ = .9, warmup-cos.

3.3

NC rank

NC excess loss

2/9 9/9 9/9 9/9

0.00505 ± 0.00015 0.10590 ± 0.00487 0.01048 ± 0.00039 0.00999 ± 0.00043

Best excess loss 0.00497 ± 0.00017 0.08390 ± 0.00242 0.00840 ± 0.00023 0.00785 ± 0.00021

Nonlinear private logistic regression

We next test whether the distinction survives outside exact quadratic dynamics. The private logisticregression experiment uses d = 16, N = 4096, T = 128 disjoint single-participation shards, real per-example clipping, 30 paired seeds, and (ε, δ) = (4, 10−5 ). Unlike the quadratic setting, clipping, stochastic gradients, curvature drift, and mechanism-dependent trajectories are present. NoiseCurve is near-optimal in its native zero-momentum regime, but becomes the worst of nine candidate mechanisms in all three tested momentum conditions. The intervention was specified before observing the empirical outcomes. Across the broader logistic study, the local M OMENT-DP approximation improves statistically resolved pairwise ranking accuracy from 91.8% to 95.1% and reduces mean relative selection regret from 4.41% to 3.53%, while top-1 agreement remains unchanged at 44.4%. We therefore claim improved mechanism-ordering information rather than reliable universal top-1 selection.

4

Neural Boundary

A five-seed CIFAR-10 CNN pilot does not establish reliable neural mechanism ranking transfer. Across four optimizer conditions, best–worst mechanism spreads are only 0.29–0.48 percentage points and the local M OMENT-DP approximation does not consistently identify the empirical winner. We therefore report no confirmed neural ranking crossover. The experiment uses a short T = 100 single-participation trajectory and should be interpreted as a boundary test of the present local approximation, not as evidence that optimizer state is irrelevant to neural private training. Architecture, calibration, absolute results, and possible contributing factors are reported in Appendix G.

5

Conclusion

M OMENT-DP shows that the utility of temporally correlated privacy noise can depend on the optimizer state that consumes it. Its quadratic transfer formulation exactly recovers NoiseCurve at zero momentum and constant learning rate, while exposing mechanism-ranking reversals when momentum changes. Controlled experiments and per-example-clipped private logistic regression confirm that this distinction can be decision-relevant, whereas neural transfer remains unresolved. These results motivate evaluating privacy-mechanism selection and optimizer configuration jointly.

4

References [1] Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016. [2] Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning (ICML), 2018. [3] Christopher A. Choquette-Choo, Arun Ganesh, Ryan McKenna, H. Brendan McMahan, Keith Rush, Abhradeep Thakurta, and Zheng Xu. (amplified) banded matrix factorization: A unified approach to private training. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [4] Sergey Denisov, H. Brendan McMahan, Keith Rush, Adam Smith, and Abhradeep Guha Thakurta. Improved differential privacy for sgd via optimal private linear operators on adaptive streams. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [5] Duc Dm, Thao Do, Minh Son Hoang, Anh Le Duc Tran, Daeyoung Kim, and Huy Nguyen. FIBER: A differentially private optimizer with filter-aware innovation bias correction. arXiv preprint arXiv:2605.03425, 2026. [6] Xin Gu, Yingtai Xiao, Guanlin He, Jiamu Bai, Daniel Kifer, and Kiwan Maeng. Correlating cross-iteration noise for dp-sgd using model curvature. arXiv preprint arXiv:2510.05416, 2025. [7] Peter Kairouz, H. Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning (ICML), 2021. [8] Nikita P. Kalinin and Joel Daniel Andersson. Learning rate scheduling with matrix factorization for private training. In 7th Symposium on Foundations of Responsible Computing (FORC), volume 368 of Leibniz International Proceedings in Informatics (LIPIcs), pages 2:1–2:21, 2026. doi: 10.4230/LIPIcs.FORC.2026.2. [9] Nikita P. Kalinin, Ryan McKenna, Jalaj Upadhyay, and Christoph H. Lampert. Back to square roots: An optimal bound on the matrix factorization error for multi-epoch differentially private sgd. In International Conference on Learning Representations, 2026. [10] Xinwei Zhang, Zhiqi Bu, Borja Balle, Mingyi Hong, Meisam Razaviyayn, and Vahab Mirrokni. DiSK: Differentially private optimizer with simplified kalman filter for noise reduction. In International Conference on Learning Representations, 2025.

5

Appendix A A.1

Proofs and Additional Theory

Expanded proof of Proposition 1

For t < T , define ΦT,t = AT −1 · · · At , with empty-product convention ΦT,T = I2d . Let P = [I 0] extract the parameter-error block. Unrolling the state recursion gives the privacy-noise component (z)

sT =

T −1 X

ΦT,t+1 Bt zt .

t=0

Applying P , (z)

eT =

T −1 X

P ΦT,t+1 Bt zt .

t=0

Define Gt = P ΦT,t+1 Bt ,

G = [G0 , . . . , GT −1 ].

Then (z)

eT = GZ. Since
Z ∼ N 0, σ 2 (K ⊗ Id ) , we have h i (z) (z)⊤ E eT eT = σ 2 G(K ⊗ Id )G⊤ . Therefore, 

  1 (z)⊤ (z) 1  (z) (z)⊤ E eT HeT = tr HE[eT eT ] 2 2  σ2  = tr HG(K ⊗ Id )G⊤ . 2 This is Eq. (3). 6

A.2

Expanded derivation of Eq. (5)

For µ = 0 and constant η, Eq. (4) unrolls mode by mode as (i,z)

eT

= −η

T −1 X

(i)

(1 − ηλi )T −j−1 zj .

j=0

Define Vij = (1 − ηλi )T −j−1 and the temporal transfer vector gi = [gi0 , . . . , gi,T −1 ]. Then gi = −ηVi,: . The full operator G ∈ Rd×dT is obtained by combining these mode-wise temporal transfer vectors in the Hessian eigenbasis; the full block operator G is therefore not identified directly with the d × T matrix V . Let M = diag(λ1 , . . . , λd ),

X = C ⊤ C.

Since K = C −1 C −⊤ = X −1 , d

Ddyn =

σ2 X λi gi X −1 gi⊤ 2 i=1 d

=

η2 σ2 X ⊤ λi Vi,: X −1 Vi,: 2 i=1

=

η2 σ2 tr X −1 V ⊤ M V . 2

This is exactly NoiseCurve’s quadratic objective. A.3

Trajectory-weighted extension

If utility over the optimization trajectory is of interest rather than only final loss, let G(t) map the full perturbation sequence to et . A corresponding score is

Dtraj =

T X t=1

wt

 σ2  tr Ht G(t) (K ⊗ Id )G(t)⊤ . 2

We explored preregistered uniform weights, but final-state damage was more discriminative in the controlled ranking tasks and is therefore used for headline results. 7

Appendix B B.1

Prior-Work Reproduction and Numerical Validation

Learning-rate-aware matrix factorization

We implement the schedule-aware factorization of Kalinin and Andersson [8] directly from its published equations. For schedule multipliers χ, the prefix workload becomes Aχ = A1 diag(χ). The implementation is validated in its constant-learning-rate limit against the trusted Toeplitz strategy in JAX-Privacy, with machine-precision agreement. B.2

NoiseCurve Problem P1

NoiseCurve optimizes X ⋆ = arg min tr(QX −1 ),

Q = V ⊤ M V,

X

subject to diag(X) = 1,

X ≻ 0,

and a prescribed band structure. Our primary implementation parameterizes the free symmetric band entries of X and fixes its diagonal exactly. Optimization uses Adam with learning rate 0.02. The free entries are initialized as 0.05 N (0, 1) using random seed 0. Positive definiteness is enforced using a floor 10−3 . When λmin (X) < 10−3 , the objective receives the additional quadratic penalty 106 10−3 − λmin (X)

2

.

Default validation runs use 2000 optimizer steps. Dedicated solver-validation runs use 3000 steps, while the logistic-regression NoiseCurve mechanism uses 1500 steps. We independently cross-check the implementation using SciPy L-BFGS-B with maxiter = 2000,

ftol = 10−14 ,

gtol = 10−10 ,

for T ∈ {8, 16, 32},

b ∈ {2, 4}.

Four of six tested configurations agree within 0.01% objective value. The two T = 32 configurations differ by approximately 0.9% and 4.7%, respectively; both returned matrices remain positive definite. These T = 32 instances are used only for solver cross-validation and do not enter the headline mechanism rankings. Recovered strategies satisfy 8

∥C ⊤ C − X∥F ≈ 10−16 . ∥X∥F During development, an error was detected in the independent validation script: indefinite matrices were diagonally regularized without properly accounting for violation of the positive-definite constraint. The primary optimizer already contained the intended positive-definiteness penalty. The independent validation script was corrected before accepting scientific results based on the reproduction. B.3

NoiseCurve–M OMENT-DP identity grid

The 576-case identity grid uses T ∈ {8, 32, 128}, six curvature spectra, and ηλmax ∈ {0.05, 0.1, 0.3, 0.6}. We compute: 1. NoiseCurve’s published quadratic objective; 2. Ddyn using the implemented recurrence; and 3. direct covariance propagation. All 576 configurations agree to relative error below 10−10 . The worst observed discrepancy is 9.7 × 10−16 .

Appendix C

Privacy Validation and Mechanism Catalog

All headline controlled and logistic comparisons use the same declared privacy unit, singleparticipation semantics, and (ε, δ) target. Candidate strategy matrices are independently checked and, where necessary, normalized to sensitivity one before Gaussian calibration. For the controlled single-participation setting, we use the analytic Gaussian mechanism [2]. Because sensitivity and privacy target are common across the mechanisms in these comparisons, the scalar σ 2 factor is shared and therefore does not alter exact quadratic rankings. The competitive pool contains: • IID Gaussian; • dense optimal MF; • BandMF mean/max; • Toeplitz mean/max; • BISR variants; • learning-rate-aware MF; and • NoiseCurve. BISR denotes the Banded Inverse Square Root construction [9]. The strategies used in the controlled and logistic studies are exported from the JAX-Privacy implementation with min_sep = 128,

max_participations = 1, 9

num_bands = 16.

For the T = 128 experiments, banded constructions use semi-bandwidth b = 16. An exploratory last-only stress mechanism produced extreme MeanSE/MaxSE and was excluded from the competitive pool before the final comparative stage.

Appendix D

Blind Crossover Protocol

The blind protocol separates analytic discovery from independent validation to reduce outcomeinformed case selection. Discovery grid.

The analytic search spans

ηλmax ∈ {0.0025, 0.005, 0.01, 0.02, 0.05, 0.10, 0.20, 0.40, 0.60, 0.80}, momentum µ ∈ {0, 0.5, 0.8, 0.9, 0.95, 0.99}, and eight learning-rate schedule families: • constant; • warmup-constant; • cosine; • warmup-cosine; • step decay; • exponential decay; • front-loaded; and • back-loaded. The controlled objectives use six d = 16 curvature-spectrum families, including isotropic, condition numbers 10, 100, and 1000, and two additional stress spectra. Competitive-case selection. Pairwise mechanism disagreements qualify for the primary registry when their relative oracle margin exceeds 10%. A secondary control registry uses a 5% threshold. Cases are balanced across optimizer and curvature conditions before any downstream Monte Carlo outcome is examined. Freezing and hashing. Selected cases are serialized into a machine-readable registry and hashed using SHA-256 before independent simulation. Twelve cases form the primary frozen crossover registry. Independent validation. Validation uses a NumPy simulator implemented independently of the JAX/autodiff code used to compute Ddyn . The simulator does not import the analytic predictor or its state-transition implementation. Monte Carlo sampling is increased adaptively until the 95% confidence interval half-width is at most one quarter of the observed pairwise gap. All twelve primary frozen cases confirm the optimizer-aware ordering. The purpose of this protocol is not to make M OMENT-DP an independent predictor in the exact quadratic regime—Proposition 1 already establishes exactness—but to verify that the identified reversals correspond to genuine differences under independently implemented dynamics. 10

Appendix E E.1

Additional Quadratic Results

Controlled-grid ranking

Table 3: Controlled-grid mechanism ranking. M OMENT-DP is exact by construction in this domain. Score

Pairwise

Kendall τ

Rel. regret

Top-1

Static MeanSE LR-aware MF Curv.+sched., µ = 0 M OMENT-DP

58.0% 60.8% 75.8% 100%†

0.143 0.200 0.501 1.000†

12.7% 11.7% 20.1% 0.0%†

33.3% 33.3% 36.7% 100%†

† Exact under the modeled linear-quadratic dynamics.

E.2

Mechanism winners across momentum

Across a broader grid of momentum, schedule, and curvature spectra, the winning mechanism changes systematically. Representative exact damage values at moderate ill-conditioning and µ = 0.9 are: Schedule

Dense

BandMF

Toeplitz

Best

Constant Cosine Warmup-cosine

21.86 0.58 0.66

22.12 0.51 0.58

34.26 0.44 0.50

Dense Toeplitz Toeplitz

The result is therefore better described as an interaction among temporal covariance, curvature, momentum, and schedule than as a universal main effect of momentum. E.3

Structured held-outs

Representative pairwise accuracies for static, LR-aware, and curvature+schedule-at-zero-momentum scoring are: • held-out momentum: 55.0/53.2/60.8%; • held-out schedule: 58.5/62.8/68.6%; • held-out κ = 1000: 68.3/70.4/83.9%; • held-out BISR: 59.1/59.1/73.3%. M OMENT-DP is exact in these quadratic held-outs and is therefore not interpreted as an independent generalization baseline. E.4

Curvature perturbation and privacy strength

Quadratic pairwise rankings remain at 100% through 25% curvature perturbation in the tested regimes. At 50% perturbation: pairwise accuracy = 95.1%, top-1 match = 84.4%, and mean relative regret = 1.07%. This should not be interpreted as robustness to nonstationary neural curvature. 11

Under common sensitivity and Gaussian calibration, changing ε rescales exact covariance-induced damage by a common positive factor, so exact rankings are invariant in this controlled singleparticipation setting.

Appendix F F.1

Logistic-Regression Details

Data generation

The synthetic logistic-regression task uses d = 16,

N = 4096.

Let s = logspace(−2, −1, 16). Features are sampled independently as xi ∼ N (0, diag(s)) , and w⋆ ∼ N (0, I16 /16) . Labels are generated according to ⋆ pi = σ(x⊤ i w ),

yi ∼ Bernoulli(pi ).

The 4096 examples are divided into T = 128 disjoint shards of size 32. F.2

Clipping and privacy calibration

Before private optimization, the clipping threshold is calibrated as the median per-example gradient norm on the designated calibration split at initialization. The threshold is then fixed for the entire private training trajectory. Headline experiments use (ε, δ) = (4, 10−5 ) under single-participation semantics. F.3

Curvature estimation

NoiseCurve uses the normalized feature-only logistic upper bound Hcal =

1 X ⊤ Xcal , 4Ncal cal

Ncal = 500.

At θ = 0, the logistic weight satisfies p(1 − p) = 12

1 , 4

making this expression exact for the initialization curvature of the empirical objective on that calibration sample. F.4

Curvature-source robustness

As a post-hoc robustness diagnostic, we also compute Hfull =

1 ⊤ X X, 4N

N = 4096.

The largest relative eigenvalue discrepancy between Hcal and Hfull is 10.5%. The predicted top-1 mechanism is unchanged in all four optimizer conditions. The complete ranking is unchanged in three of four. Kendall correlations with empirical mechanism ordering are (0.389, 0.556, 0.444, 0.333) for Hcal , and (0.389, 0.556, 0.444, 0.222) for Hfull . Thus, replacing the calibration curvature by this full-data static estimate does not explain the nonlinear mismatch. This check rules out the choice between these two static curvature sources as the dominant explanation; it does not measure trajectory-wise curvature drift. The full-training-set estimate is used only as a post-hoc diagnostic and is not proposed as a privacyfree deployment procedure. F.5

Optimizer conditions

We evaluate: • L0: µ = 0, constant learning rate; • L1: µ = 0.9, constant learning rate; • L2: µ = 0.9, cosine schedule; • L3: µ = 0.9, warmup-cosine schedule. Banded mechanisms use semi-bandwidth b = 16. F.6

Broader ranking evaluation

Across 61 statistically resolved pairwise comparisons: Static pairwise accuracy = 91.8%, M OMENT-DP pairwise accuracy = 95.1%. Mean relative selection regret changes from 4.41%

→ 13

3.53%.

Top-1 agreement remains 44.4% for both methods. Four of the twelve frozen quadratic crossover cases remain statistically resolved in logistic regression, and M OMENT-DP predicts all four.

Appendix G G.1

CIFAR-10 CNN Details and Negative Result

Architecture and data

The CNN is implemented directly in JAX and contains 140,714 parameters: Conv(3 → 32, 3 × 3, pad = 1) + ReLU, Conv(32 → 32, 3 × 3, pad = 1) + ReLU, MaxPool(2 × 2), Conv(32 → 64, 3 × 3, pad = 1) + ReLU, Conv(64 → 64, 3 × 3, pad = 1) + ReLU, MaxPool(2 × 2), Conv(64 → 128, 3 × 3, pad = 1) + ReLU, GlobalAvgPool, Dense(128 → 10). No batch normalization or weight decay is used. CIFAR-10 is partitioned into: 40,000 private-training examples, 2,000 public-calibration examples, 3,000 validation examples, and 5,000 test examples. Training uses T = 100 disjoint private shards of size 400 and five paired seeds. G.2

Optimization and clipping

The per-example clipping threshold is calibrated as the median gradient norm over 200 public calibration examples and is then fixed during private training. Curvature is estimated exclusively on the public calibration split using: 14

• Hessian-vector products; • the top-30 Lanczos eigenvalues; and • a Hutchinson trace estimate. An initial configuration with η0 = 0.05 produced η0 λmax ≈ 6.5 and was rejected before scientific analysis. The corrected base learning rate is η0 =

0.3 ≈ 0.00229. λmax

The SGD condition uses µ = 0. The three momentum conditions use µ = 0.9. G.3

Mechanism pool

The CNN pilot evaluates five mechanism families: • IID Gaussian; • BandMF; • Toeplitz MF; • dense MF; and • learning-rate-aware MF. BandMF and Toeplitz constructions use semi-bandwidth b = 16. BISR is not included because the validated strategy available for that pilot was constructed at horizon T = 128, while the CNN uses T = 100. NoiseCurve is also absent because the CNN experiment was completed before the independent NoiseCurve reproduction and validation stage. The pilot was not retroactively rerun solely to enlarge the mechanism pool. G.4

Five-seed results Table 4: CIFAR-10 five-seed pilot after stability calibration. Condition SGD + constant Momentum + constant Momentum + cosine Momentum + warmup-cosine

G.5

Best

Worst

Spread

Combined SE

BandMF 12.3% IID/Toeplitz 22.7% IID 17.6% BandMF 17.1%

LR-aware 12.0% LR-aware 22.2% Dense 17.3% Toeplitz 16.8%

0.34 pp 0.48 pp 0.36 pp 0.29 pp

0.32 pp 0.34 pp 0.27 pp 0.15 pp

Interpretation of the negative result

The CNN pilot does not establish a reliable mechanism-ranking signal for the local M OMENT-DP approximation. Several properties distinguish this regime from the controlled and logistic studies: • single-participation semantics restrict the trajectory to only T = 100 updates; 15

• stochastic-gradient variability introduces an additional source of optimization noise; • per-example clipping changes gradient magnitude and direction; • curvature changes along the trajectory; and • different privacy-noise realizations induce different optimization trajectories. The parameter dimension is also much larger: d = 140,714 versus d = 16 in the controlled and logistic studies. For isotropic Gaussian perturbations, expected squared noise energy scales with dimension, which may make covariance-specific differences harder to resolve. We do not isolate this effect experimentally and therefore treat it only as a possible contributor. The observed best–worst differences are small and directionally inconsistent, so the experiment was not expanded to additional seeds solely to search for statistical significance. Accordingly, the CNN result is an empirical boundary of the current fixed-curvature local approximation rather than evidence that optimizer state is irrelevant to neural private training.

Appendix H H.1

Extended Discussion and Limitations

Evaluation versus covariance synthesis

The present work primarily studies mechanism evaluation and selection. Given a collection of privacy-valid temporal covariances, M OMENT-DP quantifies how each covariance is filtered by the specified optimizer dynamics. The transfer representation also suggests a constructive extension: one could directly optimize covariance structure against optimizer-aware privacy damage while imposing the sensitivity, positivedefiniteness, participation, and structural constraints required by the matrix mechanism. We deliberately separate evaluation from synthesis. Establishing that optimizer state changes mechanism selection does not require claiming that the current paper constructs the globally optimal momentum-specific covariance. H.2

Why neural transfer is harder

The strongest agreement with M OMENT-DP occurs where the transfer model is exact: fixed quadratic dynamics. Private logistic regression shows that the optimizer-state distinction can persist outside that domain, but the transfer is attenuated. In neural training, stochastic-gradient variability, clipping, curvature drift, and mechanism-dependent trajectory feedback become simultaneously important. A useful future direction is to explicitly decompose optimization variability into privacy-noise, minibatch-sampling, and clipping-induced components and identify regimes in which optimizerfiltered privacy noise is sufficiently large to influence mechanism selection. H.3

Trajectory feedback and nonstationary curvature

The transfer operator in the present paper is local. In nonlinear optimization, zt −→ θt+1 −→ ∇f (θt+1 ), H(θt+1 ) −→ future optimizer response. 16

Different privacy covariances can therefore induce different optimization trajectories and different sequences of effective curvature. A single fixed transfer operator cannot represent this feedback exactly. The logistic curvature-source diagnostic shows that replacing one static curvature estimate by a larger-sample static estimate does not resolve the observed disagreement. It does not directly measure temporal curvature drift. H.4

Adaptive optimizers

Persistent optimizer state under differential privacy has also been studied from other perspectives. DiSK uses Kalman filtering to denoise privatized gradients [10], while FIBER studies interactions between temporal filtering and AdamW’s adaptive second-moment state [5]. These approaches modify the private optimizer or its internal filtering procedure. M OMENT-DP addresses the complementary question of how a fixed privacy-valid temporal covariance is filtered by the optimizer and how that interaction changes mechanism ranking. Extending the present exact formulation to Adam or AdamW would require representing additional first- and second-moment states and nonlinear coordinate-wise normalization. H.5

Repeated participation and privacy amplification

The cleanest headline comparisons use single-participation semantics. In repeated-participation, subsampling, shuffling, or amplification regimes, candidate mechanisms may differ not only in covariance structure but also in effective sensitivity and privacy calibration. The observation that a common scalar Gaussian scale leaves exact quadratic rankings unchanged therefore does not automatically extend to these regimes. Optimizer-aware utility should be considered only after privacy validity and calibration have been established under the actual participation model. H.6

Practical implications

A practical implication of the results is that privacy mechanism and optimizer configuration need not be independent design choices. A covariance that performs well under one optimizer regime can become less favorable after introducing momentum even when its differential-privacy guarantee remains unchanged. The work does not establish accelerator-level speedups, lower hardware cost, or production-scale performance gains. M OMENT-DP operates at the level of optimization dynamics and temporal covariance and is therefore hardware agnostic. H.7

Curvature access and privacy

Curvature information can itself constitute a privacy boundary. If Hessians, Hessian-vector products, or related statistics are computed from protected examples, that access may require its own privacy treatment. The CNN experiment uses public calibration data for curvature estimation. For logistic regression, the full-training-data curvature evaluation described in Appendix F is used only as a post-hoc diagnostic rather than as part of the privacy-compatible training procedure. 17

Appendix I I.1

Ethics, AI Use, and Reproducibility

Ethics

This work studies mechanisms for differentially private training and does not involve human-subject experiments. Its intended impact is to improve the reliability and utility of privacy-preserving optimization. Utility comparisons are meaningful only after privacy semantics and sensitivity are matched. Any protected-data auxiliary computation, including curvature estimation, would need to be incorporated into the relevant privacy analysis. I.2

AI use

Generative AI tools were used for language editing and organization of manuscript drafts, assistance in formulating experimental checklists, and assistance with code drafting/debugging. All mathematical claims, literature relationships, experimental implementations, numerical results, and interpretations reported in the paper were reviewed and validated by the authors. The authors take responsibility for the final content. I.3

Reproducibility

The intended anonymized artifact includes: 1. environment and version manifests; 2. mechanism sensitivity checks; 3. learning-rate-aware and NoiseCurve reproduction tests; 4. the 576-case identity grid; 5. the frozen crossover registry and SHA-256 hash; 6. independent NumPy validation code; 7. logistic and CNN experiment manifests; and 8. scripts reproducing reported tables. Random sources for data generation, initialization, privacy noise, mechanism optimization, and replication are separated.

18

Related documents

• Privacy Accounting for Empirical Denoisers#1403506
• Privacy Accounting for Empirical Denoisers#1394155
• Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information#1447542
• Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information#1442296
• Towards Hyperparameter Transfer for Differentially Private Optimization#1387265