arXiv CSopen access

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum Naima Tasnim, Lalitha Sankar and Oliver Kosut

arXiv:2606.05435v1 [cs.LG] 3 Jun 2026

Arizona State University, Tempe, AZ, United States Email: {ntasnim2, lsankar, okosut} @ asu.edu

Abstract—Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacypreserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself. On the other hand, DP-Adam is an algorithm that exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum. We perform an analysis showing that DP-MacAdam estimates the gradient variances in a biasfree manner. In addition, we empirically evaluate the privacy and accuracy of DP-MacAdam, demonstrating that it achieves improved model utility compared to DP-SGD, AdaClip, and DPAdam baselines, without requiring manual tuning of the clipping threshold.

I. I NTRODUCTION The proliferation of machine learning models trained on sensitive data has made privacy-preserving optimization a central concern in modern deep learning. Differential privacy (DP) [1] has emerged as the gold standard for formal privacy guarantees, providing rigorous, mathematically provable protections against the leakage of individual training examples. In the context of deep learning, DP is most commonly realized through DP-SGD [2], which clips per-sample gradients to bound their sensitivity and injects calibrated Gaussian noise before each parameter update. While DP-SGD offers strong theoretical guarantees, its empirical performance often lags behind modern adaptive optimizers, motivating the development of differentially private variants of Adam [3], [4]. The Adam optimizer maintains exponential moving averages (EMA) of the first and second moments of stochastic gradients, yielding coordinate-wise adaptive learning rates. For a variety of ML tasks, including image and natural language processing, Adam consistently outperforms SGD and is now the de facto iterative optimization algorithm for training large models. It is therefore natural to apply Adam in the DP setting by feeding privatized gradients directly into its update rule, yielding the algorithm DP-Adam [5].

However, a crucial limitation of both DP-SGD and its Adam-based variants is the dependence on a fixed gradient clipping threshold C. In practice, the choice of C critically affects the privacy-utility tradeoff: a threshold that is too small discards signal through excessive clipping, while one that is too large increases sensitivity and forces larger noise additions. Pichapati et al. [6] address this limitation through AdaClip, a theoretically-motivated adaptive clipping strategy that maintains coordinate-wise exponential moving estimates of the gradient mean and variance, using them to center, scale, and clip gradients dynamically at each iteration. By adapting the clipping geometry to the local gradient distribution, AdaClip provably reduces the expected noise added per iteration compared to isotropic ℓ2 clipping, yielding improved model utility under the same privacy budget. The two approaches, DP-Adam and AdaClip, address key yet complementary limitations of DP-SGD. DP-Adam maintains running estimates of the gradient mean and second moment, and uses them to compute adaptive, coordinate-wise parameter updates. However, DP-Adam still relies on standard clipping, forgoing the noise reduction benefits of adaptive clipping. AdaClip, on the other hand, maintains coordinatewise estimates of the gradient mean and variance to perform adaptive clipping and reduce the additive noise, but discards these statistics after preprocessing and falls back to a plain SGD update, leaving the adaptive information unused. The combination of adaptive clipping and adaptive momentum updates under differential privacy therefore remains unexplored. We propose DP-MacAdam (Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum), which combines both adaptive clipping and adaptive momentum. Crucially, this combination comes at no additional privacy cost, since all moment estimates are derived entirely from the privatized gradients, and thus, satisfy (ϵ, δ)-DP by the post-processing property of DP. Main Contributions. Our main contributions are: • We propose DP-MacAdam, the first algorithm to combine AdaClip-style coordinate-wise adaptive clipping with Adam-style adaptive momentum under differential privacy guarantees. • We derive a novel bias correction factor that yields an unbiased estimate of the gradient variance from the EMA of noise-corrected instantaneous variance estimates, accounting for both DP noise inflation and EMA initialization bias.

We empirically demonstrate that DP-MacAdam outperforms state-of-the-art DP optimizers on MNIST and CIFAR-10 across a range of privacy budgets, without requiring manual tuning of the clipping threshold or dimensionality reduction as a preprocessing step. Related Work. Differentially private stochastic gradient descent was introduced by Abadi et al. [2], establishing the canonical recipe of per-sample gradient clipping followed by Gaussian noise addition. Convergence properties of the Adam optimizer were established by Kingma and Ba [3] and later refined by Bock et al. [4]. DP-Adam was first introduced in TensorFlow Privacy [7] as a straightforward extension of Adam to the DP setting. Tang et al. [5] identified the DP noise bias in Adam’s second moment estimator and proposed DPAdamBC as a correction. Choi et al. [8] extended this line of work by incorporating decoupled weight decay, yielding DP-AdamW and DP-AdamW-BC with improved empirical performance across image, text, and graph classification tasks. Adaptive clipping strategies for DP-SGD were proposed by Pichapati et al. [6] through AdaClip, which uses coordinatewise gradient statistics to minimize the expected noise norm added per iteration. Gilani et al. [9] generalized this idea through GeoClip, which clips and perturbs gradients in a transformed basis, provably reducing added noise compared to isotropic clipping. •

II. P ROBLEM S ETUP briefly review some relevant definitions. Let D =  We n (x(i) , y (i) ) i=1 denote a dataset of n i.i.d. samples, where x(i) ∈ X and y (i) ∈ Y. Given a parametric model θ ∈ Rd and a loss function ℓ : Rd × X × Y 7→ R, the goal is to find the model Pn parameters θ that minimize the empirical risk L(θ) = n1 i=1 ℓ(θ; x(i) , y (i) ). For notational simplicity, we omit the superscript (i) when referring to a generic sample x and its associated quantities. Two datasets D and D̃ are considered neighbors, denoted D ∼ D̃, if they differ by at most one entry. Differential privacy (DP) is defined with respect to all such neighboring datasets as follows. Definition 1 (Differential Privacy [1]): A randomized algorithm, or mechanism A : (X × Y)n → S is considered (ϵ, δ)-differentially private ((ϵ, δ)-DP) if, for every pair of neighboring datasets D ∼ D̃ ∈ (X × Y)n , and for all S ⊆ S, Pr{A(D) ∈ S} ≤ eϵ Pr{A(D̃) ∈ S} + δ.

(1)

A. Private Optimization and Gradient Clipping The empirical risk can be minimized using a variety of firstorder optimization methods, which differ in how they make use of the gradients g = ∇θ ℓ(θ; x, y) to update the model parameters. A standard approach to enforce DP in gradientbased optimization is to bound the sensitivity of each gradient update. Given a per-sample gradient gt at time step t, the ℓ2 sensitivity is controlled by clipping: gt ḡt = , (2) max(1, ∥gt ∥2 /C)

which ensures ∥ḡt ∥2 ≤ C for a fixed clipping threshold C > 0. Gaussian noise zt ∼ N (0, σ 2 C 2 Id ) is then added to the clipped gradient, yielding a privatized estimate g̃t = ḡt + zt that satisfies (ϵ, δ)-DP by the Gaussian mechanism [1]. Using the privatized gradient g̃t , DP-SGD [2] updates the model parameters along the negative gradient direction: θt = θt−1 − ηg̃t ,

(3)

where η is the learning rate. DP-Adam [7] extends the Adam optimizer [3], [4] to the differentially private setting by maintaining exponential moving averages of the privatized gradients and their second moments: mt = β1 mt−1 + (1 − β1 )g̃t ,

vt = β2 vt−1 + (1 − β2 )g̃t2 , (4)

where β1 , β2 are exponential decay rates. After computing bias-corrected estimates m̂t = mt /(1 − β1t ) and v̂t = vt /(1 − β2t ), DP-Adam updates the model parameters using θt = θt−1 − η √

m̂t , v̂t + γ

(5)

where γ is a stability constant. B. Adaptive Clipping Both DP-SGD and DP-Adam rely on a fixed, isotropic clipping threshold C, applied uniformly across all gradient coordinates to bound the ℓ2 -sensitivity. As discussed in the introduction, choosing C to avoid over- or under-clipping is a challenge. Pichapati et al. [6] address this through AdaClip, a coordinate-wise adaptive clipping strategy which first centers and scales the gradients using running estimates of the gradient mean mt ∈ Rd and scaling factor bt ∈ Rd : wt =

gt − mt , bt

w̄t =

wt , max(1, ∥wt ∥2 )

(6)

ensuring unit ℓ2 -norm sensitivity after the transform. Gaussian noise is added to the clipped transformed gradients, and the result is mapped back to the original gradient space: g̃t = bt ⊙ (w̄t + zt ) + mt .

(7)

The privatized gradient g̃t is then used in the gradient descent step. By centering and scaling gradients coordinate-wise before clipping, AdaClip concentrates the noise budget on dimensions with high variance—where it matters most—and adds less noise to dimensions that carry little information. This is shown to reduce the total noise added per iteration compared to fixed ℓ2 clipping, with the advantage growing with the parameter dimension d. III. D IFFERENTIALLY P RIVATE M ECHANISM FOR A DAPTIVE C LIPPING WITH A DAPTIVE M OMENTUM (DP-M AC A DAM ) We present DP-MacAdam in Algorithm 1, which integrates the coordinate-wise adaptive clipping strategy of AdaClip [6] with Adam’s adaptive momentum updates [3] under differential privacy. DP-MacAdam has several key properties. First,

instead of bounding the noise-corrected instantaneous variance estimate ut as done in AdaClip, we feed it into an exponential moving average st with the same decay rate β1 as the mean estimate mt . Second, to ensure that st an unbiased estimate of g̃t , we must account for the bias introduced by the weighted averaging of the past estimates. We derive this bias-correction factor κt = 2(β1 − β1t )/(1 + β1 ) in Appendix A. To ensure that ŝt is both lower and upper bounded, we clamp it between constants h1 and h2 . Finally, the bias-corrected estimate ŝt is used to update the adaptive scaling vector bt following the formula used in AdaClip [6]. We leave the derivation of the optimal choice of bt under our EMA-based variance estimate for future work. Algorithm 1 DP-MacAdam Input: Dataset D, learning rate η, noise multiplier σ, batch size B, hyperparameters β1 , β2 , h1 , h2 , stability constant γ, number of iterations T Output: θT 1: Initialize: θ0 ; m0 = 0, m̂0 = 0, v0 = 0, s20 = 0, b0 = (1/d) · 1 2: for t = 1 to T do n oB (i) (i) 3: Sample mini-batch Bt = (xt , yt ) i=1 4: for i = 1 to B  do  (i) (i) (i) 5: gt ← ∇θ ℓ θt−1 ; xt , yt (i)

(i)

6:

wt ←

7:

w̄t ←

(i)

gt − m̂t−1 bt−1 (i) wt (i)

max(1, ∥wt ∥2 ) 8: end for  σ2 9: zt ∼ N 0, 2 Id B 1 PB (i) 10: w̃t ← w̄ + zt B i=1 t 11: g̃t ← bt−1 ⊙ w̃t + m̂t−1 12: mt ← β1 mt−1 + (1 − β1 )g̃t 13: vt ← β2 vt−1 + (1 − β2 )g̃t2 vt mt 14: m̂t ← , v̂t ← 1 − β1t 1 − β2t m̂t 15: θt ← θt−1 − η √ v̂t + γ 16: ut ← (g̃t − m̂t ) ⊙ (g̃t − m̂t ) 17: st ← β1 st−1 + (1 − β1 )ut 2(β1 − β1t ) 18: κt ← 1 + β1    st σ2 19: ŝt ← min max − b2t−1 · 2 , h1 , h2 κt B P 1/2 1/4 1/2 d 20: bt ← ŝt · j=1 ŝt,j 21: end for 22: return θT A. Privacy Analysis The privacy of DP-MacAdam follows that of DP-SGD and DP-Adam. Since the only interaction with the private dataset D

occurs through the per-sample gradients, privacy is guaranteed at the point of noise injection in step 10 of Algorithm 1. All downstream computations—the moment estimates mt , vt , the variance estimate st , and the adaptive scaling vector bt — are functions solely of the privatized gradient g̃t and public hyperparameters. By the post-processing property of differential privacy [1], DP-MacAdam therefore inherits the same (ϵ, δ)-DP guarantee as DP-SGD, for any privacy accountant used to track the cumulative privacy loss across T iterations. We use the Connect-the-Dots [10] privacy accountant in our experiments. We state the privacy guarantee of Algorithm 1 formally in Proposition 1. The proof follows directly from the privacy analysis of DP-SGD [2]. Proposition 1 (Privacy guarantee of DP-MacAdam): For any privacy accountant Compose(T, θ1,...,T ) under which DP-SGD(θ, C, D, σ, B) [2] satisfies (ϵ, δ)-DP, DP-MacAdam(θ, D, σ, B) satisfies the same (ϵ, δ)-DP guarantee under the same accountant. B. Noise Scaling In DP-MacAdam, the adaptive parameters m̂t and bt naturally set the clipping threshold, so there is no need for a separate clipping hyperparameter C, as in DP-SGD and DPADAM. Thus, in step 6 of Algorithm 1, centering and scaling each entry of the gradient vector gt ensures that the ℓ2 norm of w̄t is at most 1. Gaussian noise is then added to the average of the clipped transformed gradients w̄t rather than to each individual gradient. For a batch size B, the average of B clipped gradients has ℓ2 -sensitivity 1/B (each (i) w̄t has at most unit norm, and changing one sample affects the average by at most 1/B), so the noise is scaled accordingly σ2 as zt ∼ N (0, B 2 Id ). This is equivalent to adding noise N (0, σ 2 Id ) to each individual gradient and averaging, but is more efficient in practice. After the map-back in step 11, the b2 σ 2 effective noise variance is t−1 B 2 ; this is why in step 19, this term is subtracted to calculate ŝt , so as to estimate the variance of the un-noised gradient.1 C. Bias-Corrected Variance Estimation In Algorithm 1, steps 5–11 follow the AdaClip approach to compute the centered and clipped gradient; steps 12–15 follow the Adam approach of computing the moving average and second moment of the gradients, and then taking the descent step. A key aspect of DP-MacAdam is in combining these statistics to update the center and scale factors for the next batch. This is captured in steps 16–19 where we form an instantaneous estimate of the gradient variance (step 16), compute its moving average (step 17), correct for the bias introduced by the moving average (step 18), bound the estimated variance (step 19), and use AdaClip’s update rule for center and scaling parameters for the next batch (step 20). 1 Clipping introduces a non-linear function to the gradient prior to noise, so even removing the noise does not give a completely unbiased estimate of the pure gradients. The adaptive clipping will mitigate this limitation over the course of the training process.

In Appendix A, we derive the bias correction factor κt used in step 19 of Algorithm 1. The goal is to obtain an unbiased estimate ŝt of the true gradient variance σg2 from the exponential moving average (EMA) st . The challenge here arises from the fact that st is a weighted average of past noise-corrected variance estimates ui , each of which is itself centered around a noisy mean m̂t —so the bias correction must account for both the EMA initialization bias and the correlation structure introduced by the weighted averaging. It is worth recalling that the empirical estimate of the variance from n samples (with equal weights) requires scaling by 1/(n − 1) rather than 1/n to ensure an unbiased estimate. This same phenomenon must be accounted for here for the exponential weighted average. D. Extension to DP-MacAdamBC A related line of work addresses a different but complementary bias in DP-Adam. Tang et al. [5] observe that the addition of DP noise introduces a constant upward shift Φ = (σC/B)2 in Adam’s second moment estimate v̂t , since the noise is independent of the gradient and its variance adds directly to the EMA of squared gradients. Under typical DP parameters, this bias dominates v̂t , effectively reducing DP-Adam to DP-SGD with momentum and a specific learning rate schedule. They propose DP-AdamBC, which corrects for this by subtracting the known bias Φ from the second moment estimate in the parameter update: p (8) θt = θt−1 − η m̂t / max(v̂t − Φ, γ ′ ). Since Φ is computable from public hyperparameters σ, C, and B, this correction comes at no additional privacy cost. The adaptive clipping strategy of DP-MacAdam is orthogonal to this correction and could be combined with DP-AdamBC by replacing the standard parameter update in Algorithm 1 with the bias-corrected update above, where in place of the constant Φ we use (σ/B)2 . We call this variant of the algorithm DPMacAdamBC. IV. E XPERIMENTAL R ESULTS We compare the performance of our proposed DPMacAdam with that of DP-SGD, AdaClip, DP-Adam, and DP-AdamBC. We focus on image classification tasks with MNIST [11] and CIFAR-10 [12] datasets. For MNIST, we train a two-layer fully connected neural network from scratch. The network takes flattened 28 × 28 pixel inputs and passes them through a hidden layer of 1000 units with ReLU activation, followed by a linear output layer of 10 units corresponding to the digit classes. The model has a total of d = 795,010 trainable parameters. For CIFAR-10 [12], we use a 5-layer CNN similar to that of [5], trained from scratch. The model takes 32 × 32 RGB images as input and has a total of d = 582,346 trainable parameters. Throughout our experiments, we use learning rate η = 0.1 for DP-SGD and η = 0.001 for the rest. We use the standard Adam momentum coefficients β1 = 0.9, β2 = 0.999, and γ = 10−8 . All models are trained for 5 epochs with a

batch size B = 256 for MNIST and B = 512 for CIFAR10, respectively. We adopt the same hyperparameter settings reported by [5] and [6] where applicable. The clipping norm for DP-SGD, DP-Adam and DP-AdamBC is set to C = 1.0. For various noise multiplier σ, the overall privacy budget ϵ is computed using the Connect-the-Dots accountant [10], with the privacy parameter δ = 10−5 . All results are reported as mean ± standard deviation over 5 random seeds. TABLE I: Test accuracy (%) on MNIST dataset across noise multipliers (σ); h1 = 10−9 and h2 = 10−6 . Algorithm DP-SGD AdaClip DP-Adam DP-AdamBC DP-MacAdam DP-MacAdam-BC

σ = 0.5 ε = 7.49 90.0 ± 0.1 86.5 ± 0.8 92.8 ± 0.1 87.9 ± 0.4 93.2 ± 0.1 92.2 ± 0.1

σ = 0.6 ε = 4.00 89.8 ± 0.1 84.9 ± 0.9 92.5 ± 0.1 86.8 ± 0.4 93.0 ± 0.1 92.3 ± 0.1

σ = 0.7 ε = 2.33 89.1 ± 0.1 83.4 ± 1.0 92.2 ± 0.1 85.8 ± 0.5 92.8 ± 0.2 92.2 ± 0.2

σ = 0.8 ε = 1.46 88.7 ± 0.1 81.5 ± 1.2 91.9 ± 0.2 84.8 ± 0.5 92.6 ± 0.2 92.1 ± 0.2

σ = 0.9 ε = 1.02 88.2 ± 0.1 79.8 ± 1.7 91.7 ± 0.1 84.0 ± 0.4 92.1 ± 0.1 92.0 ± 0.1

σ = 1.0 ε = 0.80 87.0 ± 0.1 79.8 ± 1.7 91.7 ± 0.1 84.0 ± 0.4 91.9 ± 0.1 91.8 ± 0.1

TABLE II: Test accuracy (%) on CIFAR-10 dataset across noise multipliers (σ); h1 = 5 × 10−5 and h2 = 1.0. Algorithm DP-SGD AdaClip DP-Adam DP-AdamBC DP-MacAdam DP-MacAdam-BC

σ = 0.5 ε = 10.40 43.76 ± 0.3 29.13 ± 0.6 58.90 ± 0.3 58.34 ± 0.2 59.96 ± 0.1 44.33 ± 1.4

σ = 0.6 ε = 5.88 43.75 ± 0.3 19.93 ± 0.4 56.96 ± 0.2 55.98 ± 0.3 58.56 ± 0.1 44.06 ± 2.1

σ = 0.8 ε = 2.45 43.67 ± 0.3 22.96 ± 0.5 54.59 ± 0.2 53.02 ± 0.2 55.83 ± 0.31 39.66 ± 1.0

σ = 1.1 ε = 1.11 43.76 ± 0.3 10.01 ± 0.1 52.04 ± 0.1 48.70 ± 0.6 52.53 ± 0.1 26.89 ± 10.5

σ = 1.5 ε = 0.66 43.60 ± 0.3 10.01 ± 0.1 49.92 ± 0.2 43.42 ± 0.6 48.49 ± 0.4 10.07 ± 0.1

We observe that DP-MacAdam outperforms the state-ofthe-art algorithms on both MNIST and CIFAR-10 across most evaluated privacy budgets, with the exception of the highest noise setting on CIFAR-10 (σ = 1.5). DP-MacAdamBC, however, underperforms relative to DP-MacAdam on MNIST, and performs poorly on CIFAR-10. Following [5], we hypothesize that the second moment bias correction yields larger gains on tasks where Adam and sign descent outperform SGD in the non-private case. It is worth noting that unlike [6], DP-MacAdam (i) operates directly on the raw inputs without spending privacy budget on dimensionality reduction via PCA, and (ii) applies clamping only after the bias-corrected variance estimate ŝt is computed rather than on the instantaneous estimate ut , yielding a more stable bt . V. C ONCLUSION We have proposed DP-MacAdam, a differentially private optimization algorithm that combines coordinate-wise adaptive clipping from AdaClip with Adam momentum updates. By maintaining running estimates of the gradient mean and variance from the privatized gradients, DP-MacAdam adapts the clipping threshold coordinate-wise at each iteration while simultaneously using the same statistics to drive adaptive parameter updates. We have derived a novel bias correction factor κt that accounts for both the DP noise inflation and the initialization bias introduced by the exponential moving average of the variance estimate. Empirical results on realworld datasets show that DP-MacAdam is consistently better than DP-Adam and DP-AdamBC for typical privacy values. Future work includes a formal convergence analysis of DPMacAdam, which can lead to determining the optimal choice

of the scaling vector bt as well as empirical evaluation on more complex learning tasks (e.g., NLP) over large datasets. A PPENDIX A B IAS C ORRECTION FACTOR κt In this analysis, we use β in place of β1 for simplicity. We also perform our analysis in an element-by-element basis, so we treat each vector as as a scalar. The EMA of the noisy gradients is mt = βµt−1 + (1 − β)g̃t =

t X

i ̸= j ̸= t:

E [(g̃t − g̃i )(g̃t − g̃j )] = E[g̃t2 ] − E[g̃t g̃j ] − E[g̃i g̃t ] + E[g̃i g̃j ] b2 σ 2 = σg2 + 2 . (19) B Combining everything, we have     t X X b2 σ 2 E[ut ] = 2 c2i + ci cj  σg2 + 2 . B i=1

(1 − β)β t−i g̃i ,

(9)

assuming m0 = 0. The bias-corrected mean estimate is thus: t X (1 − β)β t−i mt = g̃i . 1 − βt 1 − βt i=1

(10)

Now we take the weighted sum,     t t X X X X b2 σ 2 2 2   ck E[uk ] = ck 2 ci + ci cj σg + 2 B k=1

P (1 − β)β t−i . Note that i ci = 1. Now, the t 1−β instantaneous variance of g̃i is 2

ut = (g̃t − m̂t ) .

ci ui .

(12) =

Taking the expectation, we obtain t X

ci E[ui ].

(13)

The expectation at a given step t is   E[ut ] = E (g̃t − m̂t )2  !2  t t X X = E  g̃t ci − ci g̃i 

=

t X t X

(14) (15)

i=1

ci cj E [(g̃t − g̃i )(g̃t − g̃j )] .

(16)

i=1 j=1

Let µg and σg2 be the true variance of gt . Since g̃t has noise 2

b σ added, its mean and variance are µg and σg2 + t−1 B2

E [(g̃t − g̃i )(g̃t − g̃j )] = 0.

b σ B2



c2i +

X

,

(22)





ck 2

t X

=

=

t X

X i̸=k

ci cj 

(23)

i̸=j̸=k





2



X   X 2 X  ck 2 ci + ci − c2i  i̸=k

i̸=k

  X ck  c2i + (1 − ck )2 

k=1

i̸=k

t X

k X

ck

(24)

i̸=k

(25) !

c2i − c2k + 1 − 2ck + c2k

(26)

i=1 t t k X X X c2k ck − 2 ck c2i + = i=1 k=1 k=1 k=1 t X c2k =1− k=1 k=1 t X

(27) (28)

Therefore,

2

. We assume that the clipping parameter bt changes slowly enough that it is a constant, so bt = b. We also assume that across batches the gradients gt are i.i.d. Now, we consider different cases: • i = j = t, or i ̸= j, i = t, or i ̸= j, j = t:

•

t X

k=1

i=1

i=1

2 2

k=1

i=1

E[st ] = (1 − β )



i̸=j̸=k

where A=

Applying EMA, we have

t

i̸=k

= A σg2 +

(11)

t X

k=1

(21)

Let ci =

st = βst−1 + (1 − β)ut = (1 − β t )

(20)

i̸=j

i=1

m̂t =

•

(17)

i = j, i ̸= t, j ̸= t:   E (g̃t − g̃i )2 = E[g̃t2 ] − 2E[g̃t g̃i ] + E[g̃i2 ] = σg2 + m̂2t − 2m̂2t + σg2 + m̂2t   b2 σ 2 2 = 2 σg + 2 . (18) B

E[st ] = (1 − β t )

t X

ck E[uk ]

k=1 t

= (1 − β ) 1 −

t X

c2k

!

σg2 +

k=1

b2 σ 2 B2

 .

(29)

The bias correction factor is thus: t

κt = (1 − β ) 1 −

t X

! c2k

,

(30)

k=1

2(β − β t ) . We can also see 1+β 2 σ 2 that κstt − b2t−1 B 2 is an un-biased estimate of σg ; this explains the exact form of step 19 in Algorithm 1 prior to clamping. which can be simplified to κt =

ACKNOWLEDGMENT This work is supported in part by NSF grants CIF-2312666 and SCH-2205080. R EFERENCES [1] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating Noise to Sensitivity in Private Data Analysis,” in Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3. Springer, 2006, pp. 265–284. [2] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep Learning with Differential Privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308–318. [3] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, 2014. [4] S. Bock, J. Goppold, and M. Weiß, “An Improvement of the Convergence Proof of the ADAM-Optimizer,” arXiv preprint arXiv:1804.10587, 2018. [5] Q. Tang, F. Shpilevskiy, and M. Lécuyer, “DP-AdamBC: Your DPAdam Is Actually DP-SGD (Unless You Apply Bias Correction),” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 276–15 283.

[6] V. Pichapati, A. T. Suresh, F. X. Yu, S. J. Reddi, and S. Kumar, “AdaCliP: Adaptive Clipping for Private SGD,” arXiv preprint arXiv:1908.07643, 2019. [7] TensorFlow Privacy Authors, “TensorFlow Privacy,” https://github.com/ tensorflow/privacy/tree/master/tensorflow_privacy/privacy/optimizers, 2019, accessed: 2026, dp_optimizer.py, line 386. [8] J. Chooi, K. Cong, R. Li, and L. Sun, “DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning,” arXiv preprint arXiv:2511.07843, 2025. [9] A. Gilani, N. Tasnim, L. Sankar, and O. Kosut, “GeoClip: GeometryAware Clipping for Differentially Private SGD,” Advances in Neural Information Processing Systems, vol. 38, pp. 6233–6257, 2026. [10] V. Doroshenko, B. Ghazi, P. Kamath, R. Kumar, and P. Manurangsi, “Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions,” in Privacy Enhancing Technologies Symposium (PETS), 2022. [11] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. [12] A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” University of Toronto, Tech. Rep., 2009.