Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information

OPT2026: 18th Annual Workshop on Optimization for Machine Learning

Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information Xincheng Xu

XINCHENG . XU @ ANU . EDU . AU

Australian National University, Australia

Thilina Ranbaduge

THILINA . RANBADUGE @ CSIRO . AU

arXiv:2610.32175v1 [cs.LG] 23 Oct 2026

CSIRO, Australia

Thierry Rakotoarivelo

THIERRY. RAKOTOARIVELO @ CSIRO . AU

CSIRO, Australia

David Smith

DAVID . SMITH @ CSIRO . AU

CSIRO, Australia

Abstract Differentially Private Stochastic Gradient Descent (DPSGD) has become the de facto standard for training deep neural networks with rigorous privacy guarantees. However, the utility of DPSGD is constrained by gradient clipping bias and DP noise. While existing work has tried to address these issues, the geometric misalignment caused by clipping bias remains a critical challenge. To bridge this gap, we propose a novel framework, Curvature-Aware Differentially Private Stochastic Gradient Descent (CA-DPSGD), which leverages approximate second-order information to construct an optimal, curvature-aware clipping space. CA-DPSGD integrates a rectified estimator for the curvature matrix and is coupled with an anisotropic noise mechanism to ensure privacy guarantees. Extensive empirical experiments demonstrate that our approach outperforms state-of-the-art clipping methods.

1. Introduction Differentially Private Stochastic Gradient Descent (DPSGD) [1] is the most widely adopted optimization algorithm used for training deep models with Differential Privacy (DP). DPSGD enforces (ϵ,δ)-DP through per-sample gradient clipping and calibrated Gaussian noise. However, DPSGD results in performance degradation compared to standard non-private training. Different DPSGD variants have been proposed to mitigate these limitations, covering two main directions: (1) reducing DP noise variance from a temporal [36, 38] or geometric perspective [4, 34, 39], and (2) mitigating clipping bias through a temporal perspective [32, 37]. However, existing methods often suffer from complicated hyperparameter tuning, excessive computational and memory costs, and a lack of theoretical guarantees. To bridge this gap, we propose Curvature-Aware Differentially Private Stochastic Gradient Descent (CA-DPSGD), a novel method that reduces clipping bias from a geometric perspective. We first derive the optimal clipping space to minimize clipping bias while maintaining the same amount of DP noise compared to vanilla DPSGD by solving an optimization problem. We construct the clipping space in each training iteration via a private approximation of the second-order moments of the gradients. Crucially, to reduce the effect of DP noise, we design a DP variance-reduced estimator to correct the DP noise contribution in a privatized batch-gradient second-moment, coupled © X. Xu, T. Ranbaduge, T. Rakotoarivelo & D. Smith.

C URVATURE -AWARE C LIPPING FOR DPSGD

with an anisotropic noise injection mechanism to ensure DP guarantees. Our main contributions are summarized as follows: • We formulate an optimization problem to characterize the clipping geometry under a bias–noise trade-off and derive the corresponding curvature-aware clipping spaces. • We design a noise correction mechanism and an adaptive scaling factor to construct a scalable diagonal clipping geometry using only previously privatized gradients. • We establish privacy and convergence guarantees, evaluate CA-DPSGD on image and sentence classification benchmarks, and demonstrate its compatibility with existing DPSGD variants.

2. Related Work Existing literature on improving the utility of DPSGD [1] has explored adaptive clipping thresholds [3, 8, 30], privacy budget allocation [9, 22], geometric projection [4, 16, 34, 39], filtering [28, 36, 38], and clipping bias mitigation [7, 32, 37]. However, they may require additional tuning, historical states, structured covariance information, or specialized model architectures. In contrast, our method constructs a lightweight diagonal clipping geometry from previously privatized gradients and explicitly corrects the known contribution of DP noise in the second-moment estimate. In non-private optimization, second-order methods have demonstrated improved optimization by leveraging curvature information. Adam [19] utilizes diagonal second-order gradient statistics, while Shampoo [18], AdaHessian [33], and K-FAC [25] employ structured or block-diagonal curvature approximations. However, adapting these methods to the private setting is challenging, as the required per-sample statistics need to be protected, which degrades the accuracy of curvature estimation. Unlike approaches that use curvature information to modify the optimizer update, we leverage approximate second-order information to construct the clipping geometry prior to noise injection. More detailed discussions are provided in Appendix A.

3. Preliminaries In this work, we focus on the supervised deep learning setting. The goal is to train a model with parameters x ∈ Rd by minimizing an empirical loss function over a dataset D containing N data points. This P is formally expressed as the Empirical Risk Minimization (ERM) problem: 1 minx∈Rd f (x) = N ξ∈D f (x; ξ), where f (x; ξ) represents the loss function for a single data point ξ, and d represents the dimensionality of the model parameters. We use (·)t to denote the variables in training iteration t and denote the lower bound as f ∗ = inf f (x). The geometry of the parameter space is formally characterized by the Fisher Information Matrix (FIM) [2], defined as F (x) = Eξ [∇f (x; ξ)∇f (x; ξ)⊤ ]. At iteration t, we write Ft ≡ F (xt ). Under the negative log-likelihood setting, this quantity provides a local second-order characterization of the optimization landscape. DPSGD [1] samples a mini-batch Bt of size B, clips each per-sample gradient to n thresholdoC, 1 P C and adds Gaussian noise to the averaged clipped gradient: ḡt = B ξ∈Bt gt,ξ min 1, ∥gt,ξ ∥2 + wt , wt ∼ N (0, σ 2 Id ), where σ denotes the standard deviation of the Gaussian noise added to the

2

C URVATURE -AWARE C LIPPING FOR DPSGD

batch-averaged gradient. The model is then updated as xt+1 = xt − ηḡt . Formal definitions of differential privacy and the Gaussian mechanism are provided in Appendix B.

4. Curvature-Aware Clipping 4.1. Optimal Clipping Geometry Vanilla DPSGD utilizes isotropic clipping, projecting gradients onto a Euclidean ball, which does not account for the anisotropic structure of the gradient distribution and can lead to unnecessary clipping bias. To reduce this bias, we pgeneralize the clipping geometry using a bounded positive definite matrix Gt ≻ 0. Let ∥g∥ = go⊤ Gt g. The corresponding clipping operator with radius C n Gt C is clipGt (gt,ξ , C) = gt,ξ min 1, ∥gt,ξ ∥G . t Let bGt (C) = Eξ [gt,ξ − clipGt (gt,ξ , C)] denote the clipping bias. We obtain the following bound. Lemma 1 (Clipping Bias Bound) For a given clipping radius C, ∥bGt (C)∥Gt ≤

Tr(Gt Ft ) , C

(1)

where Ft is the gradient second-moment matrix defined in Section 3. The proof is provided in Appendix C. Directly minimizing Tr(Gt Ft ) yields the trivial solution Gt → 0, which reduces the clipping-bias bound at the cost of increasing the magnitude of the injected noise. In the transformed space, let wt′ ∼ N (0, σ 2 Id ). Mapping the noise back to the −1/2 ′ original parameter space gives Cov(Gt wt ) = σ 2 G−1 t . To keep the total noise variance equal to −1 that of isotropic DPSGD, we impose Tr(Gt ) = d and formulate min

Gt ≻0

Tr(Gt Ft ) C

s.t.

Tr(G−1 t ) = d.

(2)

Theorem 2 (Optimal Clipping Matrix) For Ft ≻ 0, the clipping matrix that minimizes the bias upper bound in Equation (2) is 1/2

Tr(Ft λt = d

−1/2 G⋆t = λt Ft ,

)

.

(3)

Theorem 2 assigns smaller clipping weights to directions with larger second moments, thereby adapting the clipping geometry to the anisotropic gradient distribution. The proof is provided in Appendix D. However, the calculation of G∗t could become computationally prohibitive in high-dimensional settings and may require additional privacy budget cost, which motivates the scalable diagonal approximation used below.

3

C URVATURE -AWARE C LIPPING FOR DPSGD

Algorithm 1 Curvature-Aware DPSGD (CA-DPSGD) Require: D, η, C, σ, T , β. for t = 0 to T − 1 do for each ξ ∈ Bt do g̃t,ξ = clipGt (gt,ξ , C) end for P g̃t = B1 ξ∈Bt g̃t,ξ + wt , wt ∼ N (0, σ 2 G−1 t ) xt+1 = xt − ηg̃t ṽt+1 = βṽt + (1 − β)g̃t2 √ 2 (σ 2 /λt )2 +4ṽt+1 −σ 2 /λt vt+1 = 2 1/2

λt+1 = Mean(vt+1 ) −1/2 Gt+1 = λt+1 diag(vt+1 ) end for return xT

4.2. Our Proposed Algorithm: CA-DPSGD Algorithm 1 summarizes how CA-DPSGD approximates the optimal clipping geometry during private training. Instead of directly estimating the per-sample second-moment matrix Ft , CA-DPSGD constructs a diagonal clipping matrix Gt from previously privatized batch gradients, which introduces no additional privacy cost by post-processing. Curvature-Aware Clipping and Anisotropic Noise. CA-DPSGD clips each per-sample gradient 1 according to the current Gt -norm: ∥gt,ξ ∥Gt = ∥(Gt ) 2 gt,ξ ∥2 . This is operationally equivalent to mapping the gradient to the transformed space, performing Euclidean clipping, and mapping it back. Accordingly, isotropic Gaussian noise in the transformed space induces the anisotropic covariance −1 σ 2 G−1 t in the original parameter space. The constraint Tr(Gt ) = d preserves the same total noise variance as isotropic DPSGD. Noise-Corrected Second-Moment Estimation. Directly using privatized batch gradients to estimate second-order information introduces the DP noise variance into the estimate. We therefore maintain an exponential moving average of the squared privatized gradients, as shown in Algorithm 1, and explicitly correct the known contribution of the anisotropic DP noise. Under the diag−1/2 onal geometry Gt = λt diag(vt ) and the temporal approximation vt ≈ vt+1 , we obtain ṽt+1 = vt+1 +

σ2 √ vt+1 . λt

(4)

Solving Equation (4) for its positive root gives the noise-corrected estimate vt+1 =

!2 p (σ 2 /λt )2 + 4ṽt+1 − σ 2 /λt . 2

(5)

The temporal approximation is supported by the stability result in Appendix E. Clipping-Geometry Update. Finally, enforcing Tr(G−1 t+1 ) = d gives 1/2

λt+1 = Mean(vt+1 ),

−1/2

Gt+1 = λt+1 diag(vt+1 ). 4

(6)

C URVATURE -AWARE C LIPPING FOR DPSGD

Table 1: Test accuracy (%) comparison on various datasets. Dataset

Model

(ϵ, δ)

DPSGD

Auto-S

DP-PSAC

CA-DPSGD

MNIST Fashion-MNIST CIFAR-10 ImageNette CelebA

4-layer CNN 4-layer CNN SimCLRv2 ResNet9 ResNet9

(3, 1e-5) (3, 1e-5) (2, 1e-5) (8, 1e-4) (8, 5e-6)

98.07 ± 0.10 86.45 ± 0.42 92.62 ± 0.03 60.91 ± 0.71 88.49 ± 0.13

98.13 ± 0.04 86.46 ± 0.07 92.57 ± 0.10 60.97 ± 0.57 88.61 ± 0.13

98.17 ± 0.10 86.56 ± 0.13 92.52 ± 0.12 61.39 ± 0.53 88.55 ± 0.12

98.32 ± 0.12 86.72 ± 0.18 92.82 ± 0.09 62.21 ± 0.47 88.67 ± 0.12

MNLI QQP QNLI SST-2

RoBERTa-base RoBERTa-base RoBERTa-base RoBERTa-base

(8, 1e-5) (8, 1e-5) (8, 1e-5) (8, 1e-5)

80.43 ± 0.03 85.58 ± 0.27 85.79 ± 0.42 89.38 ± 0.33

80.31 ± 0.39 85.40 ± 0.31 86.10 ± 0.56 89.42 ± 0.41

80.54 ± 0.16 85.63 ± 0.24 85.88 ± 0.51 89.51 ± 0.36

80.58 ± 0.27 85.64 ± 0.40 86.29 ± 0.65 89.54 ± 0.29

5. Theoretical Analysis Privacy Guarantee. Theorem 3 Let q = B/N be the sampling probability. There exist constants c1 and c2 such that, for any ϵ ≤ c1 q 2 T and δ > 0, Algorithm 1 satisfies (ϵ, δ)-DP if p C q T log(1/δ) σ ≥ c2 . (7) B ϵ 1/2

Under z = Gt g, the mechanism reduces to Euclidean clipping with isotropic Gaussian noise; adaptation of Gt is post-processing of previously privatized quantities. The proof is provided in Appendix F. Convergence Guarantee. We assume that f is L-smooth, the stochastic-gradient variance is bounded by r2 , and Gt ⪰ µId for some µ > 0. The formal assumptions are provided in Appendix G. B Theorem 4 Under the above assumptions, let η ≤ 2(2B+1)L . After T iterations,

min

0≤t<T

4(f (x0 ) − f E∥∇f (xt )∥22 ≤ ηT

∗)

  T −1 r2 4 X 2 + 2Lη dσ + + Tr(Gt Ft )2 . B µC 2 T

(8)

t=0

The bound separates the optimization error, stochastic and DP noise, and a geometry-dependent clipping-bias term. The last term shows how the choice of Gt affects clipping bias under anisotropic gradient second moments. The proof is provided in Appendix H.

6. Experiments Experimental Setting. We evaluate CA-DPSGD on five image classification datasets (MNIST [12], Fashion-MNIST [31], CIFAR-10 [20], ImageNette [11], and CelebA [24]) and four GLUE tasks [29] (MNLI, QQP, QNLI, and SST-2). We compare against vanilla DPSGD [1], Auto-S [8], and DPPSAC [30]. Full experimental settings are provided in Appendix I.

5

C URVATURE -AWARE C LIPPING FOR DPSGD

Main Results. Table 1 reports test accuracy across all nine datasets. CA-DPSGD achieves the highest mean accuracy across the evaluated methods, with larger improvements on more challenging settings, including gains of 1.30% over vanilla DPSGD on ImageNette and 0.50% on QNLI. These results indicate that adapting the clipping geometry can improve utility across both vision and language tasks. CA-DPSGD can also be incorporated into existing DPSGD variants, including IOM [32], KFSGD [38], and DPAdamBC [28]. Detailed results are provided in Appendix J. Additional ablations, including the adaptive λ mechanism, are provided in Appendix K.

7. Conclusion We proposed CA-DPSGD, a curvature-aware clipping framework that uses a noise-corrected secondmoment estimate to construct a scalable diagonal clipping geometry with anisotropic Gaussian noise. Theoretical analysis establishes privacy and convergence guarantees, while experiments across vision and language tasks show improvements over the evaluated clipping baselines. A limitation is that the diagonal geometry does not capture cross-parameter correlations, motivating more expressive clipping geometries in future work.

References [1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In SIGSAC Conference on Computer and Communications Security, CCS ’16, page 308–318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394. doi: 10.1145/2976749.2978318. URL https://doi.org/10.1145/2976749.2978318. [2] Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2): 251–276, 1998. [3] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems, 34:17455–17466, 2021. [4] Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. In International Conference on Machine Learning, pages 383–392, Virtual, 2021. PMLR, PMLR. [5] Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International conference on machine learning, pages 394–403. PMLR, 2018. [6] Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in neural information processing systems, 31, 2018. [7] Louis Béthune, Thomas Massena, Thibaut Boissin, Aurélien Bellet, Franck Mamalet, Yannick Prudent, Corentin Friedrich, Mathieu Serrurier, and David Vigouroux. DP-SGD without

6

C URVATURE -AWARE C LIPPING FOR DPSGD

clipping: The lipschitz neural network way. In International Conference on Learning Representations (ICLR), Vienna, Austria, 2024. ICLR. URL https://openreview.net/ forum?id=BEyEziZ4R6. [8] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. Advances in Neural Information Processing Systems, 36, 2024. [9] Lin Chen, Danyang Yue, Xiaofeng Ding, Zuan Wang, Kim-Kwang Raymond Choo, and Hai Jin. Differentially private deep learning with dynamic privacy budget allocation and adaptive optimization. IEEE Transactions on Information Forensics and Security, 18(1):4422–4435, 2023. [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR, 2020. [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [12] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012. [13] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In International Conference on the Theory and Applications of Cryptographic Techniques, pages 486–503, Lyon, France, 2006. Springer. [14] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014. [15] Arun Ganesh, Mahdi Haghifam, Thomas Steinke, and Abhradeep Guha Thakurta. Faster differentially private convex optimization via second-order methods. Advances in Neural Information Processing Systems, 36:79426–79438, 2023. [16] Atefeh Gilani, Naima Tasnim, Lalitha Sankar, and Oliver Kosut. Geoclip: Geometry-aware clipping for differentially private sgd. arXiv preprint arXiv:2506.06549, 2025. [17] Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems, 33:15042–15053, 2020. [18] Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR, 2018. [19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. 7

C URVATURE -AWARE C LIPPING FOR DPSGD

[20] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://www. cs.toronto.edu/˜kriz/learning-features-2009-TR.pdf. [21] Mounssif Krouka, Antti Koskela, and Tejas Kulkarni. Communication efficient differentially private federated learning using second order information. Proceedings on Privacy Enhancing Technologies, 2025. [22] Jaewoo Lee and Daniel Kifer. Concentrated differentially private gradient descent with adaptive per-iteration privacy budget. In SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1656–1665, London, UK, 2018. ACM. [23] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiv.org/abs/ 1907.11692. [24] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730– 3738, 2015. [25] James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417. PMLR, 2015. [26] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 2006. [27] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147. pmlr, 2013. [28] Qiaoyue Tang, Frederick Shpilevskiy, and Mathias Lécuyer. Dp-adambc: Your dp-adam is actually dp-sgd (unless you apply bias correction). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15276–15283, 2024. [29] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [30] Tianyu Xia, Shuheng Shen, Su Yao, Xinyi Fu, Ke Xu, Xiaolong Xu, and Xing Fu. Differentially private learning with per-sample adaptive clipping. In the AAAI Conference on Artificial Intelligence, volume 37, pages 10444–10452, Washington, DC, USA, 2023. AAAI. [31] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, pages 1–6, 2017. [32] Hanshen Xiao, Zihang Xiang, Di Wang, and Srinivas Devadas. A theory to instruct differentially-private learning via clipping bias reduction. In IEEE Symposium on Security and Privacy (SP), pages 2170–2189, San Francisco, CA, USA, 2023. IEEE, IEEE. 8

C URVATURE -AWARE C LIPPING FOR DPSGD

[33] Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10665–10673, 2021. [34] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In International Conference on Learning Representations (ICLR), Virtual, 2021. ICLR. URL https://openreview.net/ forum?id=7aogOj_VYO0. [35] Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. Advances in neural information processing systems, 31, 2018. [36] Xinwei Zhang, Zhiqi Bu, Mingyi Hong, and Meisam Razaviyayn. DOPPLER: differentially private optimizers with low-pass filter for privacy noise reduction. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Annual Conference on Neural Information Processing Systems (NeurIPS), pages 1–26, Vancouver, BC, Canada, 2024. Curran Associates. URL http://papers.nips.cc/paper_files/paper/2024/hash/ 49c466ccc038f39b08b1980a2b06673c-Abstract-Conference.html. [37] Xinwei Zhang, Zhiqi Bu, Steven Wu, and Mingyi Hong. Differentially private SGD without clipping bias: An error-feedback approach. In International Conference on Learning Representations (ICLR), pages 1–27, Vienna, Austria, 2024. ICLR. URL https: //openreview.net/forum?id=uFbWHyTlPn. [38] Xinwei Zhang, Zhiqi Bu, Borja Balle, Mingyi Hong, Meisam Razaviyayn, and Vahab Mirrokni. Disk: Differentially private optimizer with simplified kalman filter for noise reduction. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/ forum?id=Lfy9q7Icp9. [39] Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private SGD with gradient subspace identification. In International Conference on Learning Representations (ICLR), pages 1–28, Virtual, 2021. ICLR. URL https://openreview.net/ forum?id=7dpmlkBuJFC.

9

C URVATURE -AWARE C LIPPING FOR DPSGD

Appendix A. Additional Discussion of Related Work Clipping Bias Mitigation. To reduce the bias from gradient clipping, recent works have explored temporal-based methods. Xiao et al. [32] proposed to reduce the clipping bias using inner-outer momentum, enhanced network normalization, batch clipping with public data, and data pre-processing. Zhang et al. [37] introduced an error-feedback mechanism, DiceSGD, to accumulate the difference between clipped and unclipped gradients. However, maintaining historical buffers significantly increases computational and memory complexity, while error-feedback introduces additional variance as the accumulated error itself must be clipped and perturbed. Bethune et al. [7] derive analytical sensitivity bounds using Lipschitz-constrained networks. However, this approach relies on specialized model architectures, limiting its general applicability. Geometry-Aware and Private Second-Order Methods. Geometric information has also been explored in private optimization [15, 16, 21, 28]. However, these methods are limited by restrictive assumptions [15], hyperparameter instability [28], and are infeasible on large datasets [21]. GeoClip [16], the closest work to ours, also performs clipping and noise injection in an adaptive geometry estimated from privatized gradients. However, GeoClip minimizes DP noise, whereas we directly minimize an upper bound on clipping bias. We do not include GeoClip in our empirical comparison because its released implementation uses full-rank covariance estimation, which does not scale to the high-dimensional models considered in our experiments.

Appendix B. Differential Privacy Background Differential Privacy (DP) [14] is a rigorous mathematical technique to protect sensitive information within a dataset D, which is formally defined as follows: Definition 5 ((ϵ, δ)-Differential Privacy) A randomized algorithm M satisfies (ϵ, δ)-DP if for any two adjacent datasets D and D′ that differ by a single element, and for any subset of outputs S ⊆ Range(M), it holds that: Pr[M(D) ∈ S] ≤ eϵ Pr[M(D′ ) ∈ S] + δ, where ϵ and δ denote the privacy budget and the failure probability, respectively. To achieve DP, Gaussian noise [5] can be added based on the ℓ2 sensitivity of the function. Definition 6 (Gaussian Mechanism) Suppose an algorithm A : D → Rd has ℓ2 sensitivity ∆A . The Gaussian mechanism M(D) = A(D) + w, with w ∼ N (0, σ 2 Id ), is (ϵ, δ)-DP if the noise σ and privacy parameters satisfy:     ∆A ϵσ ∆A ϵσ ϵ Φ − −e Φ − − ≤ δ, 2σ ∆A 2σ ∆A where Φ(t) = Pr[N (0, 1) ≤ t] is the cumulative density function of the standard normal distribution and ∆A = supD,D′ ∥A(D) − A(D′ )∥2 , for any pair of adjacent datasets D and D′ .

10

C URVATURE -AWARE C LIPPING FOR DPSGD

Appendix C. Proof of Lemma 1 Proof We begin by define rGt (gt,ξ ) = gt,ξ − clipGt (gt,ξ , C) and we expand this as:      C C rGt (gt,ξ ) = 1 − min 1, gt,ξ = 1 − gt,ξ , ∥gt,ξ ∥Gt ∥gt,ξ ∥Gt +

(9)

where (x)+ = max(0, x). The magnitude of the bias is ∥bGt (C)∥Gt = ∥E[rGt (gt,ξ )]∥Gt . By Jensen’s inequality, ∥E[X]∥Gt ≤ E[∥X∥Gt ], we have: # "  C (10) ∥gt,ξ ∥Gt . ∥bGt (C)∥Gt ≤ E [∥rGt (gt,ξ )∥Gt ] = E 1 − ∥gt,ξ ∥Gt + Let x = ∥gt,ξ ∥Gt . We then prove for any x ≥ 0, C > 0:   C x2 1− x≤ . x + C

(11)

To verify the above inequality, we consider two cases: • Case 1 (x ≤ C): The LHS is 0. Since x2 /C ≥ 0, the inequality holds. • Case 2 (x > C): The LHS is x − C. The inequality x − C ≤ x2 /C can be rearrange to x2 − Cx + C 2 ≥ 0. Then, we observe: x2 − Cx + C 2 = (x −

C 2 3 2 ) + C > 0, 2 4

(12)

the inequality holds. Substituting Equation (11) into Equation (10) yields: " # ⊤ ]) Tr(Gt E[gt,ξ gt,ξ ∥gt,ξ ∥2Gt Tr(Gt Ft ) 1 ⊤ ∥bGt (C)∥Gt ≤ E = . = E[gt,ξ Gt gt,ξ ] = C C C C

(13)

This completes the proof.

Appendix D. Proof of Theorem 2 Proof We aim to minimize the following optimization problem: min

Gt ≻0

Tr(Gt Ft ) C

s.t.

Tr(G−1 t ) = d.

(14)

We introduce a Lagrange multiplier λ1 > 0 for the equality constraint. The Lagrangian function L(Gt , λ) is defined as: L(Gt , λ1 ) =

Tr(Gt Ft ) + λ1 Tr(G−1 t )−d . C 11

(15)

C URVATURE -AWARE C LIPPING FOR DPSGD

Then, the gradient is derived as follows: ∂Tr(G−1 1 ∂Tr(Gt Ft ) 1 ∂L t ) = + λ1 = Ft − λ1 G−2 t . ∂Gt C ∂Gt ∂Gt C

(16)

∂L = 0) yields: Setting the gradient to zero ( ∂G t

1 Ft = λ1 G−2 t . C

(17)

Rearranging the terms to solve for Gt , we have: G2t = λ1 CFt−1 .

(18)

Since Gt and Ft are symmetric positive definite matrices, we obtain: G∗t = Let scalar λt =

√

p −1 λ1 CFt 2 .

(19)

λ1 C. The optimal space is given by: −1

G∗t = λt Ft 2 . −1/2

Substituting G∗t = λt Ft

(20)

into the constraint Tr((G∗t )−1 ) = d gives 1 1/2 Tr(Ft ) = d, λt

and therefore

1/2

λt =

Tr(Ft d

)

.

(21)

(22)

which completes the proof.

Appendix E. Proof of Lemma 7 To theoretically justify the approximation ṽt ≈ ṽt+1 , we have the following lemma: Lemma 7 (Temporal Stability) Under the gradient clipping threshold C and DP noise variance σ 2 , the expected divergence between ṽt+1 and ṽt is bounded by:
E[|ṽt+1 − ṽt |1 ] ≤ O (1 − β)(C 2 + dσ 2 ) . (23) This lemma indicates that as β → 1, the difference between ṽt+1 and ṽt approaches 0. However, a large β could depend too heavily on on historical gradients and potentially slow the convergence. The detailed proof of this lemma is provided in Appendix E and we empirically verified how β would affect our performance in Appendix L. Proof Recall the update rule for the privatized second-moment estimator in CA-DPSGD: ṽt+1 = βṽt + (1 − β)g̃t2 , 12

(24)

C URVATURE -AWARE C LIPPING FOR DPSGD

We aim to bound the expected ℓ1 -norm of the difference ∆t = ṽt+1 − ṽt . Substituting the update rule into the difference term, yields: ṽt+1 − ṽt = (β − 1)ṽt + (1 − β)g̃t2 = (1 − β)(g̃t2 − ṽt ).

(25)

Taking the ℓ1 -norm and the expectation on both sides, and applying the triangle inequality, we have: E[∥ṽt+1 − ṽt ∥1 ] =(1 − β)E[∥g̃t2 − ṽt ∥1 ]
≤(1 − β) E[∥g̃t2 ∥1 ] + E[∥ṽt ∥1 ] .

(26)

Note that for any vector x, ∥x2 ∥1 = ∥x∥22 . By utilizing the Bounded Clipping Matrix Assumption and the Rayleigh quotient inequality, we have: E[∥g̃t+1 ∥22 ] ≤

C 2 + dσ 2 . µ

(27)

Since ṽt is an exponential moving average of previous noisy gradients, its expected norm is also bounded by: C 2 + dσ 2 E[∥ṽt ∥1 ] ≤ max E[∥g̃τ ∥22 ] ≤ . (28) τ ≤t µ Substituting these bounds back into the inequality, yields:  2  C + dσ 2 C 2 + dσ 2 2 E[∥ṽt+1 − ṽt ∥1 ] ≤ (1 − β) ( )+( ) = (1 − β)(C 2 + dσ 2 ). (29) µ µ µ This completes the proof.

Appendix F. Privacy Analysis 1

Proof We first define a transformed per-sample gradient as z̃t,ξ = (Gt ) 2 g̃t,ξ . According to Steps 3 - 4 in Algorithm 1, we obtain the l2 -norm of z̃t,ξ is bounded by: q 1 ⊤ G g̃ ∥z̃t,ξ ∥2 = ∥(Gt ) 2 g̃t,ξ ∥2 = g̃t,ξ (30) t t,ξ = ∥g̃t,ξ ∥Gt ≤ C. Then, we obtain that the covariance matrix of the injected DP noise in the transformed space wt′ = 1 (Gt ) 2 wt is: 1

1

Cov(wt′ ) = (Gt ) 2 (σ 2 (Gt )−1 )((Gt ) 2 )T = σ 2 Id .

(31)

Thus, by utilizingP the Gaussian mechanism in Definition 6 and privacy amplification by subsamqC δ 1 , T ))-DP. pling [6], z̃t = B ξ z̃t,ξ + wt′ in each training iteration is protected by (O( Bσ 1

Given Gt is derived from the history of previously privatized gradients and g̃t = (Gt )− 2 z̃t , by utilizing post-processing property [13], the noisy batch-averaged gradient g̃t is also protected by qC δ (O( Bσ , T ))-DP. We then apply the moments accountant method [1] over T iterations, and finally obtain that the overall privacy guarantee is (ϵ, δ)-DP, which completes the proof.

13

C URVATURE -AWARE C LIPPING FOR DPSGD

Appendix G. Assumptions for Convergence Analysis Assumption 1 (L-Smoothness) The loss function f (x) is a L-smooth function, which means that for any x, y ∈ Rd : ∥∇f (x) − ∇f (y)∥2 ≤ L∥x − y∥2 . Assumption 2 (Bounded Variance) The variance of the stochastic gradients is bounded. For any x ∈ Rd , we have: Eξ [∥∇f (x; ξ) − ∇f (x)∥22 ] ≤ r2 . Assumption 3 (Bounded Clipping Matrix) Gt is a positive definite matrix with bounded eigenvalues. There exists a constant µ > 0 such that: ∀t, Gt ⪰ µId . The Assumption 1 is widely adopted in non-convex optimization [35]. Assumption 2 is standard in the analysis of gradient clipping [17]. Assumption 3 is based on our definition of Gt , which ensures the matrix is non-singular, and is commonly used in non-private second-order optimizers [25, 26].

Appendix H. Proof of Theorem 4 We begin with the L-smoothness assumption. By substituting y = xt+1 and x = xt , we have: f (xt+1 ) ≤ f (xt ) + ⟨∇f (xt ), xt+1 − xt ⟩ +

L ∥xt+1 − xt ∥22 . 2

(32)

Lη 2 ∥g̃t ∥22 . 2

(33)

Using the update rule xt+1 = xt − ηg̃t , we get the following. f (xt+1 ) ≤ f (xt ) − η⟨∇f (xt ), g̃t ⟩ +

Applying the mathematics expectation and re-arranging terms, we obtain: E[f (xt+1 )] − f (xt ) ≤ −η⟨∇f (xt ), E[g̃t ]⟩ +

Lη 2 E[∥g̃t ∥22 ]. 2

(34)

We then analyze the first term −η⟨∇f (xt ), E[g̃t ]⟩. Using the definition of g̃t and the linearity of E[·], we have: E[g̃t ] = E[

1 X 1 X 1 X g̃t,ξ + wt ] = E[g̃t,ξ ] + E[wt ] = E[g̃t,ξ ] = ∇f (xt ) + bt B B B ξ

ξ

(35)

ξ

where the last equality we define bt as the difference between the expected per-sample clipped gradient and the true gradient: bt = E[g̃t,ξ ] − ∇f (xt ). (36)

14

C URVATURE -AWARE C LIPPING FOR DPSGD

Substituting Equation (35) into −η⟨∇f (xt ), E[g̃t ]⟩, we have: −η⟨∇f (xt ), E[g̃t ]⟩ = −η⟨∇f (xt ), ∇f (xt ) + bt ⟩ = −η∥∇f (xt )∥22 − η⟨∇f (xt ), bt ⟩ η η ≤ −η∥∇f (xt )∥22 + ∥∇f (xt )∥22 + ∥bt ∥22 2 2 η η 2 2 = − ∥∇f (xt )∥2 + ∥bt ∥2 , 2 2

(37)

where we apply −⟨a, b⟩ ≤ ∥a∥∥b∥ ≤ 21 ∥a∥2 + 21 ∥b∥2 to the inequality. 2 For the second term Lη2 E[∥g̃t ∥22 ], given the DP noise is independent, we have: E[∥g̃t ∥22 ] = E[∥

1 X g̃t,ξ ∥22 ] + E[∥wt ∥22 ]. B

(38)

ξ

For term E[∥ B1

2 2 2 ξ g̃t,ξ ∥2 ], By utilizing E[∥X∥2 ] = ∥E[X]∥2 + Tr(Var(X)), we have:

P

2

 E

1 X B

ξ∈Bt

g̃t,ξ

 2



 = E

1 X B

2

g̃t,ξ 

ξ∈Bt







+ Tr Var 

1 X B

2

g̃t,ξ  .

(39)

ξ∈Bt

h P i We analyze the two terms on the RHS separately. For E B1 ξ g̃t,ξ , since the expectation is a linear operator, we obtain:   X 1 X 1 g̃t,ξ  = E[g̃t,ξ ] = E[g̃t,ξ ]. (40) E B B ξ

ξ

Recalling the definition of bt in Equation (36), we have:  2 X 1 E g̃t,ξ  = ∥∇f (xt ) + bt ∥22 . B 

ξ







ξ

ξ

(41)

2

P For the term Tr Var B1 ξ∈Bt g̃t,ξ , since the samples ξ in the mini-batch Bt are sampled independently, the variance of their sum is the sum of their variances:   X 1 1 X 1 1 Var  g̃t,ξ  = 2 Var(g̃t,ξ ) = 2 · B · Var(g̃t,ξ ) = Var(g̃t,ξ ). (42) B B B B Taking the trace on both sides: 



Tr Var 

 1 X B

g̃t,ξ  =

ξ

15

1 Tr(Var(g̃t,ξ )). B

(43)

C URVATURE -AWARE C LIPPING FOR DPSGD

We then use the property Tr(Var(X)) ≤ E[∥X∥22 ], and obtain: Tr(Var(g̃t,ξ )) ≤ E[∥g̃t,ξ ∥22 ].

(44)

Next, we apply the contraction property of the clipping operation. Since ∥g̃t,ξ ∥2 = ∥clip(gt,ξ )∥2 ≤ ∥gt,ξ ∥2 , we have: E[∥g̃t,ξ ∥22 ] ≤ E[∥gt,ξ ∥22 ].

(45)

Utilizing the Bounded Variance Assumption, we obtain: Tr(Var(g̃t,ξ )) ≤ ∥∇f (xt )∥22 + r2 . Substituting this back into the variance term equation:    X 1 1 g̃t,ξ  ≤ (∥∇f (xt )∥22 + r2 ). Tr Var  B B

(46)

(47)

ξ

Combining the Expectation term and the Variance term, we obtain the final upper bound: 2



 ≤ ∥∇f (xt ) + bt ∥22 + 1 (∥∇f (xt )∥22 + r2 ). B

1 X E g̃t,ξ B ξ

(48)

2

For term E[∥wt ∥22 ], using the constraint Tr((Gt )−1 ) = d, yields: E[∥wt ∥22 ] = Tr(σ 2 (Gt )−1 ) = σ 2 Tr((Gt )−1 ) = dσ 2 .

(49)

Substituting Equation (48) and Equation (49) into Equation (38), we obtain: Lη 2 E[∥g̃t ∥22 ] 2   Lη 2 1 2 2 2 2 ≤ ∥∇f (xt ) + bt ∥2 + (∥∇f (xt )∥2 + r ) + dσ 2 B 2 Lη Lη 2 r2 ≤(Lη 2 + )∥∇f (xt )∥22 + Lη 2 ∥bt ∥22 + (dσ 2 + ), 2B 2 B

(50)

where the last inequality applies ∥a + b∥22 ≤ 2∥a∥22 + 2∥b∥22 . Then, substituting Equation (37) and Equation (50) into Equation (34), we obtain: η η E[f (xt+1 )] − f (xt ) ≤ − ∥∇f (xt )∥22 + ∥bt ∥22 2   2 2 Lη + Lη 2 + ∥∇f (xt )∥22 + Lη 2 ∥bt ∥22 2B   Lη 2 r2 2 + dσ + . 2 B

16

(51)

C URVATURE -AWARE C LIPPING FOR DPSGD

Rearranging the terms, we have:   1 η 2 ∥∇f (xt )∥22 E[f (xt+1 )] − f (xt ) ≤ − − Lη 1 + 2 2B     Lη 2 r2 η 2 + dσ + + + Lη 2 ∥bt ∥22 . 2 B 2 

(52)

Given Gt is defined as a bounded and positive definite matrix, there always exists a µ > 0, such that Gt ⪰ µI holds for all t. Therefore, by using the Rayleigh quotient inequality, we have:   1 1 Tr(Gt Ft ) 2 ∥bt ∥22 ≤ ∥bt ∥2Gt ≤ . (53) µ µ C Therefore, substituting Equation (53) into Equation (52), we have: E[f (xt+1 )] − f (xt )    1 η 2 ≤− − Lη 1 + ∥∇f (xt )∥22 2 2B      Tr(G F ) 2 r2 η Lη 2 t t 2 2 1 dσ + + + Lη + 2 B 2 µ C     2 2 η Lη r η Tr(Gt Ft ) 2 2 2 ≤ − ∥∇f (xt )∥2 + dσ + + , (54) 4 2 B µ C   1 B B where the last inequality applies η ≤ min 2L , 2(2B+1)L = 2(2B+1)L . Finally, averaging t from 0 to T − 1, we obtain:   T −1 T −1 Lη 2 T r2 η X ηX 2 2 ∗ ∥∇f (xt )∥2 + dσ + + (Tr(Gt Ft ))2 (55) f − f (x0 ) ≤ − 4 2 B µC 2 t=0

t=0

Rearranging terms and dividing both sides by T4η , we have: P −1   (Tr(Gt Ft ))2 4 Tt=0 4(f (x0 ) − f ∗ ) r2 2 2 min E[∥∇f (xt )∥2 ] ≤ + 2Lη dσ + + , t ηT B µC 2 T

(56)

which completes the proof.

Appendix I. Experimental Setting Datasets. We conduct experiments on five image classification datasets (MNIST [12], FashionMNIST [31], CIFAR-10 [20], ImageNette [11], and CelebA [24]) and four sentence classification datasets from GLUE benchmark [29] (MNLI, QQP, QNLI, and SST-2). For all datasets, we follow the same data splitting methods as established in prior works [8, 30]. Baselines. We compare CA-DPSGD against three baselines: vanilla DPSGD [1] as the de facto standard, and two state-of-the-art variants, Auto-S [8] and DP-PSAC [30], which enhance vanilla DPSGD by designing novel clipping functions. Optimizers. We utilize SGD with momentum [27] for MNIST, Fashion-MNIST, and CIFAR-10, and Adam [19] for all other datasets. 17

C URVATURE -AWARE C LIPPING FOR DPSGD

88.0 Test Accuracy (%)

Test Accuracy (%)

99.5 98.5 97.5 96.5 95.5

87.0 86.0 85.0 84.0

M M IO -IO CA

D D SG SG KF -KF CA

BC BC am am Ad Ad DP -DP CA

(a) MNIST

BC BC am am Ad Ad DP -DP CA

(b) Fashion-MNIST 71.0 Test Accuracy (%)

94.5 Test Accuracy (%)

D D SG SG KF -KF CA

M M IO -IO CA

93.5 92.5 91.5

66.0 61.0 56.0 51.0

90.5 M M IO -IO CA

D D SG SG KF -KF CA

BC BC am am Ad Ad DP -DP CA

D D SG SG KF -KF CA

M M IO -IO CA

(c) CIFAR-10

BC BC am am Ad Ad DP -DP CA

(d) Imagenette

Test Accuracy (%)

90.0 89.0 88.0 87.0 86.0 M M IO -IO CA

D D SG SG KF -KF A C

BC BC am am Ad PAd P D -D CA

(e) CelebA

Figure 1: Test accuracy (%) of IOM, KFSGD, and DP-AdamBC with/without CA-DPSGD on five image classification datasets. The prefix ”CA-” denotes the method is integrated with CA-DPSGD (e.g., CA-IOM represents IOM combined with CA-DPSGD).

Appendix J. Additional Experiments: Compatibility with Other DPSGD Variants Models and Hyperparameters. We use the model architectures and hyperparameter setting as in prior works [8, 30] to ensure a fair comparison. Specifically, we use a 4-layer CNN for MNIST and Fashion-MNIST, a pretrained SimCLRv2 model [10] for CIFAR-10, and a ResNet9 model for the ImageNette and CelebA dataset. For all datasets in the GLUE benchmark, we fine-tune a pre-trained RoBERTa-base model [23]. We set the hyperparameter β to 0.999.

18

C URVATURE -AWARE C LIPPING FOR DPSGD

Hardware and software information. All methods are implemented using Python3 and all experiments are run on an i7-7800X CPU with a NVIDIA RTX A6000 GPU (48 GB). All experiments are repeated five times, with the mean and standard deviation reported. We evaluate the effectiveness of CA-DPSGD as a clipping adapter by integrating it into three state-of-the-art DPSGD variants: (1) IOM [32], which mitigates clipping bias from a temporal perspective via per-sample momentum; (2) KFSGD [38], which reduces DP noise variance using a simplified Kalman filter; and (3) DP-AdamBC [28], which improves second-order optimisation using bias correction. Figure 1 illustrates the performance comparison of these methods with and without CA-DPSGD on five image classification datasets, where our method improves the utility of these methods on the image classification datasets in most cases. Specifically, on the Imagenette dataset, the results indicate that CA-DPSGD consistently enhances the utility of all three methods, improving the test accuracy of KFSGD and DP-AdamBC by approximately 2.0% and 1.8%, respectively. This implies that our method further mitigates the clipping bias, which is orthogonal to the improvements made by these variants. However, CA-IOM exhibits a marginal performance drop compared to vanilla IOM on FashionMNIST. This is because momentum smoothing in the IOM amplifies the contribution of lowcurvature dimensions in such a simple landscape. In contrast, CA-DPSGD prioritises high-curvature dimensions by relaxing their thresholds while enforcing stricter clipping on low-curvature dimensions. Consequently, this results in information loss in the low-curvature subspace, introducing an additional error that outweighs the benefits of geometric rectification.

Appendix K. Additional Experiments: Ablation Study As shown in the left column of Figure 2, the Noise Correction (NC) module improves the utility on MNIST and Fashion-MNIST in most cases. However, the performance gap between CA-DPSGD and its variant w/o NC is negligible on CIFAR-10. This is because we fine-tune a pre-trained model on CIFAR-10, a setting where the injected noise variance is small. Consequently, the error introduced by the DP noise is minimal, making the NC module less critical in this specific setting. In contrast, the right column of Figure 2 demonstrates the superiority of our adaptive λ mechanism over fixed values (λ2 ∈ {10−6 , 10−7 , 10−8 }). This performance difference stems from the fundamental bias-variance trade-off in DPSGD. Specifically, a large λ (e.g., 10−6 ) suppresses the variance of the injected DP noise, yielding relative good performance at low ϵ regimes. However, it introduces an excessive clipping bias that hinders convergence as ϵ increases. In contrast, a small λ reduces clipping bias but suffers from a high DP variance. Consequently, our adaptive approach achieves the best performance in most scenarios, reducing the need for manual hyperparameter tuning across diverse datasets and varying privacy budgets.

Appendix L. Additional Experiments: Parameter Study of β The divergence of vt is theoretically bounded by O(1 − β), implying that a larger β enforces stricter stationarity. This theoretical insight is empirically illustrated in Figure 3a(i), where increasing β significantly reduces the difference ∥vt+1 − vt ∥1 . Smaller β values (e.g. 0.1 and 0.5) induce excessive difference between vt+1 and vt ), thus degrading the convergence performance as shown in Figure 3a(ii).

19

Test Accuracy (%)

C URVATURE -AWARE C LIPPING FOR DPSGD

100.0

99.5

99.0

97.0

98.0

94.5

CA-DPSGD CA-DPSGD (λ 2 = 10−6 )

97.0 96.0

92.0

CA-DPSGD CA-DPSGD (w/o NC)

1

CA-DPSGD (λ 2 = 10−7 ) CA-DPSGD (λ 2 = 10−8 )

89.5

2

3 4 5 6 7 8 Privacy Budget (ε) (i) CA-DPSGD vs CA-DPSGD (w/o NC)

1

2

3 4 5 6 7 8 Privacy Budget (ε) (ii) CA-DPSGD with different λ values

1

2

1

2

Test Accuracy (%)

(a) MNIST 89.0

92.0

87.0

87.0

85.0

82.0

83.0

77.0

81.0

1

72.0

2

3 4 5 6 7 8 Privacy Budget (ε) (i) CA-DPSGD vs CA-DPSGD (w/o NC)

3 4 5 6 7 8 Privacy Budget (ε) (ii) CA-DPSGD with different λ values

Test Accuracy (%)

(b) Fashion-MNIST 96.0

94.5

94.0

92.5

92.0

90.5

90.0

88.5

88.0

1

86.5

2

3 4 5 6 7 8 Privacy Budget (ε) (i) CA-DPSGD vs CA-DPSGD (w/o NC)

3 4 5 6 7 8 Privacy Budget (ε) (ii) CA-DPSGD with different λ values

(c) CIFAR-10

Figure 2: Ablation study on test accuracy (%) of CA-DPSGD on three image classification datasets: MNIST, Fashion-MNIST, CIFAR-10. (a) Comparison of CA-DPSGD and CA-DPSGD without Noise Correction (w/o NC). (b) Comparison between the proposed adaptive λ mechanism and fixed λ values (λ2 ∈ {10−6 , 10−7 , 10−8 }). However, excessive stability could also hinder optimization. Figure 3a(ii) reveals that β = 0.9999 leads to slower convergence compared to β = 0.999. Therefore, we utilize β = 0.999 as an effective trade-off in our experiments, ensuring sufficient stability for the correction mechanism while maintaining efficient convergence.

20

10

−2

10

−3

10

−4

10

−5

2.2 Training Loss

log10 (‖vt + 1 − vt ‖1 )

C URVATURE -AWARE C LIPPING FOR DPSGD

0

10

20 30 Epoch

40

β = 0.9999 β = 0.999 β = 0.5 β = 0.1

2.0 1.8 1.6

50

0

(i) Stability of ‖vt + 1 − vt ‖1

10

20 30 Epoch

40

50

(ii) Convergence Curves

(a) Imagenette −6

10

−7

10

−8

10

−9

Training Loss

log10 (‖vt + 1 − vt ‖1 )

1.2 10

0

10

20 30 Epoch

40

β = 0.9999 β = 0.999 β = 0.5 β = 0.1

1.0

0.8

50

0

(i) Stability of ‖vt + 1 − vt ‖1

10

20 30 Epoch

40

50

(ii) Convergence Curves

(b) Fashion-MNIST

Figure 3: Hyperparameter analysis of β on different datasets. (a) Results on Imagenette and (b) Results on Fashion-MNIST. In both cases, the left panels show the stability of vt via log10 (∥vt+1 − vt ∥1 ), while the right panels show training loss curves. Larger β leads to more stable momentum updates but slower convergence. Figure 3b shows trends consistent with our previous analysis: larger β enhances stability, while smaller β accelerates convergence. For Fashion-Mnist, β = 0.9999 eventually achieves a performance comparable to β = 0.999, despite a slower convergence speed. This is because the simplicity of the Fashion-MNIST landscape allows effective convergence.

Appendix M. Notations We summarize the notation in Table 2.

21

C URVATURE -AWARE C LIPPING FOR DPSGD

Table 2: Table of Notations Notation ϵ δ xt d N B η C σ2 f (x) gt,ξ g̃t,ξ g̃t Ft G∗t λt vt ṽt β L r2

Description Privacy budget Failure factor Model parameters at iteration t Dimensionality of model parameters Total number of samples in dataset Batch size Learning rate Clipping threshold DP noise variance Loss function Per-sample gradient for sample ξ Clipped per-sample gradient Noisy batch-averaged gradient Fisher Information Matrix Optimal curvature-aware clipping matrix Adaptive scaling factor Rectified second-order moment estimate Noisy second-order moment (EMA) EMA factor Lipschitz smoothness constant Upper bound on gradient variance

22

Related documents

• Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information#1447542
• Optimizing Canaries for Privacy Auditing with Metagradient Descent#1061614
• Energy Time-Series Imputation with Differentially Private Diffusion Models via Clipping-Aware Objective Conditioning#1154832
• Deep Learning under Fractional-Order Differential Privacy#175148
• Towards Hyperparameter Transfer for Differentially Private Optimization#1387265