JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
1
Rényi Pufferfish Privacy with Gaussian-based Priors: From Single Gaussian to Mixture Model
arXiv:2604.23649v1 [cs.CR] 26 Apr 2026
Wenjin Yang, Ni Ding, Member, IEEE, Zijian Zhang∗ , Senior Member, IEEE, Zhen Li, Jing Sun, Jincheng An, Yong Liu∗ , Liehuang Zhu, Senior Member, IEEE
Abstract—Rényi Pufferfish Privacy (RPP) provides a Rényi divergence-based privacy framework for correlated data, but existing ∞-Wasserstein mechanisms are often conservative and sacrifice data utility. We study Gaussian mechanisms for RPP under Gaussian and Gaussian-mixture priors. For single Gaussian priors, we derive the exact Rényi divergence after Gaussian perturbation, obtain a relaxed closed-form sufficient condition for (α, ϵ)-RPP, and characterize the monotonicity of the calibrated noise with respect to the privacy budget ϵ and the Rényi order α. To handle more general non-Gaussian and multimodal priors, we approximate secret-conditioned outputs with Gaussian mixture models and introduce an optimal-transport-based sufficient condition for RPP. Experiments on three UCI datasets with statistical (RAW, MEAN) and model-output (BNN, GP) queries show that our prior-aware mechanisms consistently require less noise than a recent RPP additive-noise baseline, achieving an average noise reduction of 48.9%. These results show that our mechanisms can substantially improve the privacy-utility trade-off under RPP. Index Terms—Rényi divergence, Pufferfish privacy, Gaussian distribution, Gaussian mixture model
I. I NTRODUCTION
T
RUSTWORTHY and privacy-preserving data circulation is a critical challenge these days. When participating in data analysis, we often need to release admissible information from the data while keeping the privacy. For example, when training medical LLMs, we want the LLMs to capture the general medical knowledge (e.g., disease diagnosis, treatment recommendations, etc.) while preserving the privacy of the patients (e.g., race, gender, age, etc.). Existing cryptographic techniques, including zero-knowledge proofs [1], [2], homomorphic encryption [3], [4], secure multi-party computation [5], [6], and trusted execution environments [7], [8], provide strong protection under specific threat models, but they are often computationally expensive and rely on strict assumptions. This motivates the development of formal privacy notions that quantify privacy risk and enable mechanism design with provable privacy guarantees. To address this, data Wenjin Yang, Zijian Zhang, Zhen Li and Liehuang Zhu are with the School of Cyberspace Science and Technology, Beijing Institute of Technology, 100081 Beijing, China. (email: {wenjinyang, zhangzijian, zhen.li, liehuangz}@bit.edu.cn) Ni Ding, Jing Sun are with the School of Computer Science, University of Auckland, New Zealand. (email: [email protected], [email protected]) Jincheng An is with QAX Security Center, the Qi-AnXin Technology Group Inc., Beijing, China (email: [email protected]) Yong Liu is with the Qi An Xin Technology Group Inc. and Zhongguancun Laboratory, Beijing 100000, China (email: [email protected]) Corresponding authors: Zijian Zhang, Yong Liu. Manuscript received April 19, 2021; revised August 16, 2021.
privacy metrics have been proposed, and Differential Privacy (DP) [9], [10] has become the de facto standard. DP guarantees that the output distribution of a randomized mechanism changes only slightly when the data of any single individual is modified or removed. Nevertheless, such a guarantee may be insufficient when the sensitive attributes and released outputs are statistically correlated. Existing DP mechanisms often assume the query outputs attributes are independent of each other, which is not always true in real-world scenarios. In fact, the sensitive attributes and query outputs are often statistically dependent, which is called correlated data. In such a setting, privacy cannot always be characterized by neighboring-database notions 1 alone (the underlying setting of DP), because the adversary may exploit prior knowledge about the data distribution and perform powerful inference attacks [11], [12]. Pufferfish Privacy (PP), a flexible privacy framework formulated by [13], [14], addresses this issue by modeling the conditional distribution of the query outputs given the sensitive attributes and adversary’s prior knowledge. To protect the sensitive data, pufferfish privacy enforces that the statistical distinguishability between two output probability distributions conditioned on a pair of secrets is upper bounded by a given privacy budget ϵ. Its Rényidivergence-based variant, Rényi Pufferfish Privacy (RPP) [15], further replaces the measure with Rényi divergence [16], enabling a fine-grained privacy characterization. Despite the advantages of PP and RPP, mechanism design under this guarantee remains challenging. For PP, the first ∞-Wasserstein-based mechanism is proposed by [17]. As the ∞-Wasserstein distance is not computable due to the non-convexity of the underlying minimization problem [18], [19], [20] proposes a 1-Wasserstein (Kantorovich) mechanism, where the optimal transport plan (the minimizer) can be calculated directly by system parameters. [21] proposes a relaxed practical mechanism based on the W1 mechanism and theoretically shows the noise reduction in a low privacy budget regime. For RPP, [15] introduces the General Wasserstein Mechanism by relaxing the ∞-Wasserstein distance in [17], along with a δ-approximation to improve data utility. [22] defines Sliced Rényi Pufferfish Privacy and proposes a sliced Wasserstein mechanism. However, the first mechanism for RPP is often conservative and injects too much noise, which limits the utility of the mechanism, and the existing approaches 1 DP relies on the notion of neighboring-database D and D ′ , which is typically chosen to capture the contribution to the mechanism’s input by a single individual.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
improve data utility by relaxing the privacy guarantee, which is not always desirable. In our work, we start from the Rényi Pufferfish Privacy with a Gaussian prior. Gaussian prior is central to several active research directions, including machine learning, image/video reconstruction, generative modeling, etc. For example, in Bayesian neural networks, Gaussian priors are routinely imposed on network weights to regularize the model and quantify posterior uncertainty [23]. In image reconstruction, Gaussianprocess priors have been used to encode correlations in the image space or frequency space and support Bayesian reconstruction [24]. In generative modeling, structured Gaussian priors have also been used in GP-prior VAEs for modeling correlations in high-dimensional and temporally structured datasets [25]. This makes the Gaussian-prior setting a natural and practically meaningful starting point for mechanism design under Rényi Pufferfish Privacy. When a single Gaussian prior is too restrictive, the Gaussian mixture model provides a standard and flexible extension to capture general distributions. [26], [27] For single Gaussian prior, we first derive the Rényi divergence between two Gaussian mechanism outputs. To satisfy (ϵ, α)-RPP and preserve the data utility, we must determine the minimal variance of the Gaussian noise that satisfies the Rényi divergence is less than privacy budget ϵ. As the divergence is complex and cannot be solved directly, we analyze the monotonicity of the privacy loss with respect to the noise variance. The non-increasing property allows us to use binary search to find the minimal variance of the Gaussian noise that satisfies the Rényi divergence is less than privacy budget ϵ. To improve theoretically transparency and practically efficiency, we derive a closed-form sufficient condition by separatelly bound the two terms in the Rényi divergence. Then, we characterize the monotonicity of the calibrated noise with respect to the privacy budget ϵ and the Rényi order α, and show that the noise is non-increasing with respect to ϵ and the monotonicity with respect to α depends on the relative dominance between mean and variance of the Gaussian prior. To handle more general non-Gaussian and multimodal priors, we model secret-conditioned outputs with Gaussian mixture models and introduce an optimal-transport-based sufficient condition for (ϵ, α)-RPP. The main contributions of this work are listed below. 1) For a single Gaussian prior, we derive the exact Rényi divergence between the outputs of Gaussian mechanisms, along with a theoretically-convergent numerical method to determine noise variance. 2) By relaxing the transcendent condition of RPP, we propose a closed-form sufficient condition for (ϵ, α)-Rényi Pufferfish Privacy, which can be computed directly from the parameters of the Gaussian prior and the noise. We both theoretically and numerically show that the calibrated noise is decreasing with privacy budget, and the monotonicity with Rényi order is determined by the relative dominance between mean and variance of the Gaussian prior. 3) Extend to non-Gaussian priors by modeling outputs with Gaussian mixture models, we introduce an optimal-
2
transport-based sufficient condition for (ϵ, α)-RPP. 4) Experiments on three real-world datasets and both statistical and model-output queries show that the proposed mechanisms consistently require less noise than a recent additive-noise baseline, leading to improved utility under the same (ϵ, α)-RPP guarantee. Related Works. Pufferfish Privacy was introduced by [13], [14] as a general privacy framework that explicitly models correlated data and adversarial prior knowledge. Building on this framework, [17] proposed the first Wasserstein-based mechanism for correlated data, while [20] later introduced a Kantorovich (W1 )-based calibration method that makes standard PP substantially more tractable. More recently, [21] proposed a relaxed practical mechanism by relaxing the too strict W1 -based condition, and theoretically showed strict noise reduction, with especially pronounced reduction in the lowprivacy-budget regime. In parallel, [28] studied Gaussian-prior and GMM-based approximation under (ϵ, δ)-approximate PP, providing the first analysis of Gaussian/GMM priors in the PP setting. For Rényi Pufferfish Privacy, [15] generalized the Wasserstein mechanism to the Rényi-divergence-based setting. Subsequent work [22] relaxed the standard RPP to sliced RPP, along with a high-dimensional sliced mechanism. Other applications further develop this area, including Multi-user setting [29], composition [30], and quantum PP [31], [32]. Consequently, our work complements this line of research by filling the gap of mechanism design for Gaussian and GMM priors under the RPP framework. Notation. We use S to denote the set of secrets, Q ⊆ S ×S to denote the set of discriminative secret pairs, and P to denote the class of adversarial prior beliefs. For a query output X and a randomized mechanism M, the privatized release is denoted by Y = M(X) = X + Nθ , where Nθ ∼ N (0, θ2 ) is additive Gaussian noise. We write (ϵ, α)-RPP for (α, ϵ)Rényi Pufferfish Privacy, where α > 1 is the Rényi order and ϵ > 0 is the privacy budget. In the single-Gaussian setting, the secret-conditioned priors are written as X|si ∼ N (µi , σi2 ) and X|sj ∼ N (µj , σj2 ). In the GMM setting, we write PKs 2 P (x|s) = k=1 ws,k N (x; µs,k , σs,k ), where ws,k , µs,k , and 2 σs,k denote the mixture weight, mean, and variance of the k-th component, respectively. When comparing two Gaussian mixtures, π ∗ denotes the optimal transport coupling between their mixture components. Throughout the paper, a smaller calibrated noise θ (or equivalently θ2 ) indicates better utility under the same privacy guarantee. Organization. Section II reviews preliminaries on Pufferfish Privacy, Rényi Pufferfish Privacy, Gaussian mixture models, and optimal transport. Section III studies Gaussian mechanisms under single Gaussian priors, derives the exact Rényi divergence after Gaussian perturbation, and presents both numerical and closed-form calibration results. Section IV extends the framework to Gaussian mixture priors through GMM fitting and an optimal-transport-based sufficient condition for RPP. Section V evaluates the proposed mechanisms on real-world datasets using statistical and model-output queries. Finally, Section VI concludes the paper.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
3
II. P RELIMINARIES We review in this section the preliminaries and formulations needed to develop our work, including the Pufferfish Privacy and Rényi Pufferfish Privacy framework, the Gaussian mixture model, and its optimal transport plan. Pufferfish Privacy Framework. Unlike Differential Privacy (DP), which considers neighboring databases, Pufferfish Privacy explicitly models the correlation under potential secrets and the adversary’s prior knowledge about the data distribution. Let S denote the set of secrets, let Q ⊆ S × S denote the set of discriminative secret pairs, and let P denote the class of prior beliefs. A mechanism M satisfies ϵ-Pufferfish Privacy if for all (si , sj ) ∈ Q and all ρ ∈ P,
For another GMM denoted by P (x′ |sj , ρ) with mean µjl 2 and variance σjl , we introduce the optimal transport plan π ∗ between these two GMMs, which determines the optimal way to transform one GMM into another. π ∗ (x, x′ ) =
k=1 l=1
Tk→l (x) = µjl +
P (M(x)|si , ρ) ≤ e P (M(x)|sj , ρ).
Rényi Pufferfish Privacy (RPP) framework generalizes the original framework by replacing the max-divergence with Rényi divergence, allowing for a more accurate privacy analysis.
Definition II.1 (Rényi Pufferfish Privacy, RPP). For α > 1 and ϵ ≥ 0, a privacy mechanism M attains (α, ϵ)-Rényi Pufferfish Privacy if for all secret pairs (si , sj ) ∈ Q and all prior beliefs ρ, Dα (P (M(x)|si , ρ)∥P (M(x)|sj , ρ)) ≤ ϵ,
(1)
where Dα is the α-order Rényi divergence. As α → ∞, RPP recovers the standard ϵ-Pufferfish Privacy. In this work, we focus on the Gaussian mechanism M(x) = f (x) + Nθ , which gives a way to calibrate a zeromean Gaussian perturbation Nθ ∼ N (0, θ2 ) to the output of query f (·). Gaussian Mixture Model. The Gaussian mixture model (GMM) is a probabilistic model that can better characterize data distribution. For an arbitrary distributed X|si , we assume the adversary can train a Gaussian mixture model X|si ∼ GM(Di ) from the original database Di . Then, under the adversary’s prior belief ρ, the probability distribution of X given secret si is P (x|si , ρ) =
Ki X k=1
PKi
2 wik N (x; µik , σik ),
2 where k=1 wik = 1 and N (x; µik , σik ) is the Gaussian 2 component with mean µik and variance σik .
∗ 2 πk,l N (x; µik , σik ) · δ (x′ − Tk→l (x)) ,
∗ where πk,l represents the optimal mass transport from the k-th component of P (x|si , ρ) to the l-th component of P (x′ |sj , ρ). The term Tk→l (x) denotes the optimal Monge map between 2 the source component N (x; µik , σik ) and the target component 2 N (x′ ; µjl , σjl ), which admits the closed-form expression:
ϵ
While this definition is robust, it can be more accurate by introducing a Rényi divergence-based version, like the Rényi Differential Privacy [33], which quantifies privacy guarantees by bounding certain moments of the exponential of the privacy loss. To achieve tighter privacy accounting, Rényi divergence has been introduced into Pufferfish Privacy. For two probability measures P and Q on a common measurable space and positive order α ∈ (0, 1) ∪ (1, ∞), the Rényi divergence is defined as Z 1 ln p(x)α q(x)1−α dx. Dα (P ∥Q) = α−1
K X L X
σjl (x − µik ). σik
III. RPP G AUSSIAN P RIORS In this section, we propose the results for the Gaussian mechanism with Gaussian prior to obtain Rényi Pufferfish Privacy guarantees and the numerical method to determine the variance of the additive noise. To improve computational efficiency and analytical interpretability, we also present a closed-form solution for calibrated noise variance. We first introduce Rényi divergence between two Gaussian distributions [16] and the Gaussian mechanism with Gaussian prior. Definition III.1 (Gaussian Rényi divergence). For any simple order α, the Rényi divergence of a normal distribution N (µ0 , σ02 ) from another normal distribution N (µ1 , σ12 ) is α(µ − µ )2 0 1 Dα N (µ0 , σ02 ) ∥ N (µ1 , σ12 ) = 2σα2 σα 1 ln + . (2) 1 − α σ01−α σ1α
provided that σα2 = (1 − α)σ02 + ασ12 .
Due to the addition property of the Gaussian distribution [34], we have Proposition III.2 (Gaussian mechanism with Gaussian prior). For additive Gaussian noise N ∼ N (0, θ2 ) and Gaussian prior X ∼ N (µ, σ 2 ), the output Y = N + X still follows a Gaussian distribution, i.e., Y ∼ N (µ, σ 2 + θ2 ). A. Mechanism Design We now show that Rényi divergence between two Gaussians can be directly used as the condition to attain (ϵ, α)-Rényi Pufferfish Privacy for the Gaussian mechanism with Gaussian prior. Corollary III.3. For all secret pairs (si , sj ) ∈ Q and Gaussian priors x|si ∼ N (µi , σi2 ) and x′ |sj ∼ N (µj , σj2 ), let ϵ > 0 and α > 1, adding Gaussian noise N (0, θ2 ) with α(µi − µj )2 2(θ2 + (1 − α)σi2 + ασj2 ) +
θ2 + (1 − α)σi2 + ασj2 1 ln 2 ≤ϵ 2(1 − α) (θ + σi2 )1−α (θ2 + σj2 )α
(3)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
attains (ϵ, α)-Rényi Pufferfish Privacy. Proof. (Proof of Corollary III.3) For any secret pair (si , sj ) ∈ Q, the query outputs follows the Gaussian distribution, i.e., X|si ∼ N (x; µi , σi2 ) and X|sj ∼ N (x′ ; µj , σj2 ). The Gaussian mechanism calibrates a zero-mean Gaussian additive noise to query outputs as Y = X + Nθ where Nθ ∼ N (0, θ2 ). Then the results of Gaussian mechanism with Gaussian prior follows a new Gaussian mechanism, i.e., Y |si ∼ N (y; µi , σi2 + θ2 ) and Y |sj ∼ N (y; µj , σj2 +θ2 ). Thus, the Rényi divergence between P (y|si , ρ) and P (y|sj , ρ) is Dα (P (y|si , ρ)||P (y|sj , ρ)) Z 1 ln P (y|si , ρ)α P (y|sj , ρ)1−α dy, ≜ α−1 1 1 = ln α − 1 (2π)1/2 (θ2 + σi2 )α/2 (θ2 + σj2 )(1−α)/2 ! Z α − 1 (y − µj )2 α (y − µi )2 − 2 dy, exp θ2 + σj2 2 θ + σi2 2 s (σ 2 + θ2 )1−α (σ 2 + θ2 )α 1 j i ln = α−1 ασj 2 + (1 − α)σi2 + θ2 α(α − 1)(µi − µj )2 × exp , 2(ασj 2 + (1 − α)σi2 + θ2 ) α(µi − µj )2 = 2(θ2 + (1 − α)σi2 + ασj2 ) 2
θ + (1 − α)σi2 + ασj2 1 + ln 2 . 2(1 − α) (θ + σi2 )1−α (θ2 + σj2 )α
To attain (α, ϵ)-Rényi Pufferfish Privacy, the divergence should be bounded by privacy budget ϵ, α(µi − µj )2 2(θ2 + (1 − α)σi2 + ασj2 ) +
θ2 + (1 − α)σi2 + ασj2 1 ln 2 ≤ ϵ. 2(1 − α) (θ + σi2 )1−α (θ2 + σj2 )α
Corollary III.3 is proved. To satisfy the (ϵ, α)-RPP guarantee and preserve the utility of the mechanism, we must determine the minimal noise variance θ2 that satisfies the inequality (3) for all secret pairs (si , sj ) ∈ Q, rather than the feasible variance. To achieve this, we first establish the monotonicity of the privacy loss function (LHS of Eq. (3)) with respect to θ2 . Let g(θ2 ) denote the privacy loss function (the left-hand side of Eq. (3)) and let z = θ2 represent the noise variance. We have g(z) ≜
1 z + A(α) α∆ + ln , 2(z + A(α)) 2(1 − α) (z + σi2 )1−α (z + σj2 )α (4)
where A(α) = (1 − α)σi2 + ασj2 and ∆ = (µi − µj )2 . For the Rényi divergence to be well-defined, z + A > 0 should be satisfied, where θ2 + (1 − α)σi2 + ασj2 > 0. Proposition III.4 (Monotonicity of g(·)). For α > 1, the privacy loss function g(·) is monotonically non-increasing with respect to the Gaussian noise variance θ2 .
4
Proof. (Proof of Proposition III.4) Expanding g(z) using logarithmic properties and differentiating with respect to x, we obtain g ′ (z) = −
(σi2 − σj2 )2 α∆ . + 2(z + A(α))2 2(z + A(α))(z + σi2 )(z + σj2 )
Then, we have # " (σi2 − σj2 )2 α ∆ g (z) = − . + 2(z + A(α)) z + A(α) (z + σi2 )(z + σj2 ) ′
Given α > 1 and the domain constraint z + A(α) > 0, the coefficients and all terms within the brackets are non-negative. Consequently, g ′ (z) ≤ 0 for all valid z, ensuring that g(z) is monotonically non-increasing. The monotonicity guarantees that for any privacy budget ϵ > 0 and Rényi order α > 1, there exists a unique minimal noise variance θ2 that satisfies (ϵ, α)-Rényi Pufferfish Privacy requirement. Furthermore, it transforms the problem of finding the minimal θ2 to finding the root of g(θ2 ) = ϵ. Since Eq. (3) does not admit a closed-form solution of θ2 , we determine its root by numerical root-finding techniques (e.g., binary search) given that g(·) is strictly monotonically decreasing in θ2 . B. Determining Gaussian Variance Before addressing the general case, we consider the symmetric prior scenario where the variances are equal, i.e., σi2 = σj2 = σ 2 . In this setting, the logarithmic term in g(θ2 ) is simplified as (θ2 + σ 2 )1−α (θ2 + σ 2 )α equals θ2 + σ 2 . The privacy requirement simplifies to α(µi − µj )2 ≤ϵ 2(θ2 + σ 2 )
⇒
θ2 ≥
α(µi − µj )2 − σ2 . 2ϵ
(5)
This closed-form solution reveals that the required noise power scales linearly with the squared mean difference (µi − µj )2 and the order α, and is inversely proportional to the privacy budget ϵ, adjusted by the prior’s inherent variance σ 2 . For the general case where σi and σj are not equal, if the Rényi divergence naturely satisfies Dα (P (x|si , ρ)∥P (x|sj , ρ)) ≤ ϵ, the additive Gaussian noise is unnecessary. When this divergence is larger than privacy budget ϵ, due to the transcendental nature of privacy loss function g(·), there is no closed-form solution for θ2 . However, since the Rényi divergence Dα is monotonically decreasing with respect to the noise variance θ2 (Proposition III.4), we can find the numerical root of g(θ2 ) = ϵ using root-finding methods. Specifically, we employ a binary 2 2 search over a search interval [0, θmax ], where θmax can be safely initialized using the symmetric case estimate. The algorithm is summarized in Algorithm 1. C. Closed-form Gaussian Mechanism The main problems of determining Gaussian variance in Corollary III.3 and Alg. 1 lie in both computation and analysis, (a) Iterative root-finding algorithm leads to high computational complexity, which severely limits scalability in applications,
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Algorithm 1 Determination of Noise Variance θ2 Input: Secret parameters (µi , σi2 ), (µj , σj2 ), order α, budget ϵ, tolerance δtol Output: Minimal noise variance θ2 α(µi −µj )2 1: Initialize L ← 0 and R ← 2ϵ 2: while R − L > δtol do 3: M ← (L + R)/2 4: Compute val ← g(M ) using Eq. (3) 5: if val > ϵ then L ← M 6: else R ← M 7: end while 8: return R
e.g., training large machine learning models. (b) It’s unclear to analyze the propositions of θ due to the transcendental nature of g(·). Consequently, it is both theoretically and practically critical to derive a closed-form solution for θ2 to ensure O(1) computational efficiency, analytical transparency, and absolute privacy guarantees. The key difficulty in Corollary III.3 is that the exact Rényi privacy condition is transcendental in the noise variance θ2 , since θ2 appears both in the rational term and inside the logarithm. Therefore, instead of solving the exact equality, we derive a closed-form sufficient condition by constructing an upper bound on the exact divergence. Our relaxation keeps the mean-separation term exact, and relaxes only the logarithmic term induced by variance mismatch. This design preserves the main dependence on the prior means while converting the exact privacy condition into a quadratic inequality in the shifted noise variable. The results are shown as follows. Theorem III.5. (Closed-form Mechanism) Given privacy budget ϵ > 0 and Rényi divergence order α > 1, for Gaussian prior N (µi , σi2 ) and N (µj , σj2 ) under all secret pairs (si , sj ) ∈ Q, adding zero-mean Gaussian noise with variance q α 2 2 θ ≥ (6) Eq.A + (Eq.A) + Eq.B − σi2 4ϵ attains (ϵ, α)-Rényi Pufferfish Privacy, where Eq.A = (µi − 8ϵ(σj2 −σi2 )2 µj )2 − 2ϵ(σj2 − σi2 ) and Eq.B = . α−1 Proof. (Proof of Theorem III.5) Consider Dα (P (y|si , ρ)||P (y|sj , ρ)) α(µi − µj )2 = 2(θ2 + (1 − α)σi2 + ασj2 ) {z } | T1
θ2 + (1 − α)σi2 + ασj2 1 + ln 2 , 2(1 − α) (θ + σi2 )1−α (θ2 + σj2 )α | {z } T2
Denote ηi = θ2 +σi2 and ηj = θ2 +σj2 where θ2 is the variance of additive Gaussian noise, we have 1 ηi1−α ηiα T2 = ln , 2(α − 1) ηi + α(ηj − ηi )
5
η
( ηji )α 1 , = ln η +α(η i j −ηi ) 2(α − 1) ηi
1 ηj α(ηj − ηi ) = (α ln − ln(1 + )). 2(α − 1) ηi ηi Denote ν =
ηj −ηi ηi , we have
1 (α ln(1 + ν) − ln(1 + αν)), 2(α − 1) 1 αν ≤ (αν − ), 2(α − 1) 1 + αν α2 (ηj − ηi )2 . = 2(α − 1)ηi (ηi + α(ηj − ηi ))
T2 =
Thus, we have Dα P (y|si , ρ) ∥ P (y|sj , ρ) ≤ α2 (ηj − ηi )2 1 α(µi − µj )2 + . 2(ηi + α(ηj − ηi )) (α − 1)ηi To attain pufferfish privacy, we have 1 α2 (ηj − ηi )2 α(µi − µj )2 + ≤ ϵ, 2(ηi + α(ηj − ηi )) (α − 1)ηi Denote δ = σj2 − σi2 , ∆ = (µi − µj )2 and x = ηi , we have α2 δ 2 1 α∆ + ≤ ϵ, 2(x + αδ) (α − 1)x α2 δ 2 ⇒2ϵx2 + (α · 2ϵδ − α∆)x − ≥ 0. α−1 We have α θ2 ≥ (µi − µj )2 − 2ϵ(σj2 − σi2 ) 4ϵ s +
2 8ϵ(σj2 − σi2 )2 − σi2 (µi − µj )2 − 2ϵ(σj2 − σi2 ) + α−1
Consider the special case where the two Gaussian priors only differ in mean, i.e., σi = σj = σ, adding Gaussian noise N (0, θ2 ) with θ2 ≥
α(µi − µj )2 − σ2 , 2ϵ
which is the same as the result in Eq. (5). For the closed-form of noise variance, we analyze how it changes with the privacy budget ϵ. A straightforward insight is that the noise will be smaller with the privacy budget ϵ increases, which represents the low privacy guarantee. We formally prove this as follows. Proposition III.6. For Rényi divergence order α > 1, the variance of Gaussian noise θ2 is decreasing with the privacy budget ϵ.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Proof. (Proof of Proposition III.6) Denote ∆ = (µi − µj )2 and δ = σj2 − σi2 . The non-constant part of Eq. (6) can be rationalized as ! r 2 α 8ϵδ h(ϵ) = ∆ − 2ϵδ + (∆ − 2ϵδ)2 + 4ϵ α−1 =
2αδ 2 q . 8ϵδ 2 (α − 1) (∆ − 2ϵδ)2 + α−1 − (∆ − 2ϵδ)
Therefore, it suffices to show that r
8ϵδ 2 − (∆ − 2ϵδ) α−1 is increasing. Direct differentiation gives 2δ −(∆ − 2ϵδ) + α−1 . D′ (ϵ) = 2δ 1 + q 8ϵδ 2 2 (∆ − 2ϵδ) + α−1 D(ϵ) =
(∆ − 2ϵδ)2 +
If δ > 0, then D′ (ϵ) > 0 follows from r 8ϵδ 2 2δ (∆ − 2ϵδ)2 + > ∆ − 2ϵδ − , α−1 α−1 which is immediate when the right-hand side is negative; otherwise, after squaring, it reduces to 4∆δ 4δ 2 > , α−1 (α − 1)2 which holds under the same nonnegative-right-hand-side condition. If δ < 0, then D′ (ϵ) > 0 is equivalent to r 8ϵδ 2 2δ ∆ − 2ϵδ − > (∆ − 2ϵδ)2 + , α−1 α−1 and squaring gives −
4δ 2 4∆δ + > 0, α − 1 (α − 1)2
which is true since ∆ ≥ 0, δ < 0, and α > 1. Thus D(ϵ) is increasing for δ ̸= 0, so h(ϵ) is decreasing. When δ = 0, Eq. (6) reduces to α∆/(2ϵ), which is also decreasing in ϵ. Consider the Rényi divergence order, we also analyze the monotonicity of θ2 with respect to α. Proposition III.7. For any ϵ > 0, the variance of Gaussian noise θ2 is increasing with the Rényi divergence order α if 2 − α√ (µi − µj )2 > 2ϵ(σj2 − σi2 ) + 2ϵ|σj2 − σi2 |, (7) α−1 else θ2 is decreasing with α. Proof. (Proof of Proposition III.7) Taking the partial derivative of θ2 with respect to the Rényi divergence order α yields a fractional expression. For α > 1, the leading coefficient and the denominator of the derivative are strictly positive. Through algebraic rearrangement and factoring out these positive terms, 2 the sign of ∂θ ∂α is found to be solely determined by the following expression: α (µi − µj )2 − 2ϵ(σj2 − σi2 ) + s 2 8ϵ(σj2 − σi2 )2 (α − 2) (µi − µj )2 − 2ϵ(σj2 − σi2 ) + . α−1
6
To ensure that θ2 is strictly increasing with α, we require ∂θ 2 ∂α > 0. Rearranging the terms, this condition becomes: (µi − µj )2 − 2ϵ(σj2 − σi2 ) > s 2 8ϵ(σj2 − σi2 )2 2−α (µi − µj )2 − 2ϵ(σj2 − σi2 ) + . α α−1
To resolve the square root, we square both sides. We carefully account for the sign of the coefficient 2−α α , which differs depending on whether 1 < α ≤ 2 or α > 2. In both cases, after substituting the squared terms and canceling out the common factors, the inequality consistently simplifies to: 2 (2 − α)2 2ϵ(σj2 − σi2 )2 . (µi − µj )2 − 2ϵ(σj2 − σi2 ) > (α − 1)2
Taking the square root of both sides, and noting that the right side is strictly non-negative, we obtain: 2 − α√ (µi − µj )2 − 2ϵ(σj2 − σi2 ) > 2ϵ|σj2 − σi2 |. α−1 This directly recovers the condition in the proposition. Conversely, if the inequality is reversed, the core expression is 2 2 negative, yielding ∂θ ∂α < 0, which means θ is decreasing with α. This concludes the proof. Proposition III.7 shows that the effect of the Rényi order α on the required noise is not universally monotone. Instead, it depends on the relative dominance between mean and variance of the two secret-conditioned Gaussian priors. When mean dominates, increasing α makes the privacy requirement harder to satisfy and thus requires more noise. When variance dominates, the opposite trend may occur. Visualization. Figure 1 visualizes the calibrated noise scale for the Gaussian-prior setting and compares our closedform calibration with the baseline [15], which is an additive Gaussian mechanism based on ∞-Wasserstein distance. For both R AW and M EAN queries, the required noise parameter θ decreases as the privacy budget ϵ increases, which is consistent with Proposition III.6. Our mechanism also consistently requires smaller noise than the baseline, illustrating the utility gain of the closed-form calibration in Theorem III.5. Figure 2 illustrates the effect of the Rényi order α on the calibrated noise parameter θ for two Gaussian-prior examples. The case in Figure 2a corresponds to the α-decreasing regime in Proposition III.7, where larger α reduces the required noise, whereas the case in Figure 2b corresponds to the αincreasing regime, where larger α raises the required noise. Together, these examples confirm the two cases characterized by Proposition III.7. IV. RPP G AUSSIAN M IXTURE M ODELS Our proposed approaches in Section III established a foundational mechanism for a single Gaussian prior to achieve (ϵ, α)-Rényi Pufferfish Privacy. While the universal importance of the Gaussian measures is evident in various fields, there are still some data distributions that cannot be accurately represented by a single Gaussian distribution. For these data, our approaches with single Gaussian priors are not suitable, and it’s critical to propose a more general mechanism. To
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Noise vs. ϵ (α = 3.0, Gaussian, RAW)
Noise vs. ϵ (α = 3.0, Gaussian, MEAN)
Ours Pierquin et al.
Ours Pierquin et al. 3 Noise Parameter θ
50 Noise Parameter θ
7
40 30 20
2
Ks = arg 1
10
0
0 0
1
2 3 Privacy Budget ϵ
4
using the Bayesian information criterion (BIC), a widely adopted choice for GMM-based density estimation [35]. Concretely, for each secret s and a prescribed upper bound Kmax , we fit candidate models with K ∈ {1, . . . , Kmax } and choose
0
5
(a) R AW
1
2 3 Privacy Budget ϵ
4
5
(b) M EAN
Fig. 1. Noise parameter θ versus privacy budget ϵ for Gaussian-prior queries with α = 3.0. Gaussian Priors
min
1≤K≤Kmax
b K denotes the maximum-likelihood parameters for Kwhere Θ component GMM and qK is the number of free parameters. In our implementation, Kmax is additionally capped by the number of samples and the number of distinct observed values, preventing over-parameterization in small-sample or highly discretized regimes. We plot the original data and the fitted GMMs in three real-world datasets in Figure 3.
θ vs. ϵ α = 1.5 α=2 α=3 α=5
2
Density
Noise Parameter θ
N (0.0, 0.5 ) N (1.0, 4.02 )
x
µi = 0.0,
B. Mechanism Design We first introduce the optimal transport plan of GMMs, which is used as the foundational step to design the RPP mechanism for GMMs. Consider two conditional query distributions associated with secrets si and sj , defined as P (x|si ) =
Privacy σj2 = 16.0Budget (µi − µj )2 =ϵ1.00 RHS(α, ϵ) stays above this for shown α and ϵ ∈ [0.5, 5.0]
µj = 1.0 σi2 = 0.25,
P (x′ |sj ) =
θ vs. ϵ
Gaussian Priors
α = 1.5 α=2 α=3 α=5
Density
Noise Parameter θ
N (0.0, 1.02 ) N (3.5, 1.12 )
Fig. 2. Two Gaussian-prior examples illustrating how the required noise parameter θ changes with the Rényi order α across the privacy-budget sweep.
bridge this gap, we extend our framework to Gaussian Mixture Models (GMMs) in this section. A. GMM Fitting For query distributions that cannot be accurately represented by a single Gaussian, we approximate each conditional prior with a one-dimensional Gaussian mixture model (GMM). Given empirical query samples Ds = {xn }N n=1 associated with secret s, we fit P (x|s) =
Ks X k=1
2 ws,k N x; µs,k , σs,k ,
where k=1 ws,k = 1 and ws,k ≥ 0. We estimate the mixture parameters by maximum likelihood using the expectation-maximization (EM) algorithm, which is a standard and robust procedure for fitting finite Gaussian mixtures. Rather than fixing the number of mixture components, we select the model complexity in a data-dependent manner
l=1
2 wj,l N (x′ ; µj,l , σj,l ),
PL where k=1 wi,k = 1, l=1 wj,l = 1, and all mixture weights are nonnegative. Under the additive Gaussian mechanism with noise N (0, θ2 ), the privatized distributions remains mixtures of Gaussians, K X k=1
Privacy Budget ϵ
(b) α-Increasing Case
L X
2 wi,k N (x; µi,k , σi,k ),
PK
P (y|si ) = x
K X k=1
(a) α-Decreasing Case
PKs
b K ) + qK log N, −2 log p(Ds ; Θ
P (y|sj ) =
L X l=1
2 wi,k N (y; µi,k , σi,k + θ2 ), 2 wj,l N (y; µj,l , σj,l + θ2 ).
The optimal transport problem for GMMs is to find a global mapping that aligns multi-modal densities. We construct a hierarchical transport plan in two stages. First, we solve a discrete optimal transport problem over mixture weights to determine an optimal mass transfer between mixture components. Let ( ) L K X X K×L U(wi , wj ) = Π ∈ R+ πk,l = wi,k , πk,l = wj,l l=1
k=1
denote the transport polytope, we compute the coupling Π∗ = arg
min
Π∈U (wi ,wj )
K X L X
πk,l Ck,l ,
k=1 l=1
∗ where Π∗ = (πk,l ) and Ck,l is the component-level matching cost. In one dimension, we use the squared 2-Wasserstein distance between Gaussian components, as Ck,l = (µi,k − µj,l )2 + (σi,k − σj,l )2 , which characterizes the parameterspace distance between the k-th source component and the l-th target component. Second, for each source-target pair ∗ > 0, we define the Monge map Tk→l (x) = (k, l) with πk,l
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
8
·10−2 Data si
Data si
Data sj GMM si GMM sj
Data sj GMM si GMM sj
0.2
0.2
Data si Data sj GMM si GMM sj
Density
Density
Density
0.3
1
0.5
0.1
0.1
0
0 0
5
10
15
0 0
Value
(a) Adult
5
10
15
20
200
Value
400
600
Value
(b) Education
(c) Heart Disease
Fig. 3. The three real-world datasets are chosen to span demographic, education, and healthcare domains and to illustrate diverse non-Gaussian query distributions. The figure shows the fitted GMM densities on these datasets, where the solid lines represent the estimated mixture densities and the bars represent the empirical distributions.
σ
j,l (x − µi,k ). The construction yields a structured µj,l + σi,k transport plan that couples mixture components through Π∗ and transports mass via Gaussian-to-Gaussian maps. Based on this GMM-OT, we design the RPP mechanism for GMMs as follows.
Theorem IV.1. Given privacy budget ϵ > 0 and Rényi divergence Gaussian Mixture priors X|si ∼ PK order α > 1, for PModel L 2 2 k=1 wi,k N (µi,k , σi,k ) and X|sj ∼ l=1 wj,l N (µj,l , σj,l ), 2 adding Gaussian noise N (0, θ ) with ( α−1 ∗ Eπk,l exp α(µi,k − µj,l )2 2 2 2 2 2(σi,k + θ + α(σj,l − σi,k )) )! 2 2 2 α2 (σj,l − σi,k ) ≤ exp((α − 1)ϵ) + 2 + θ2 ) (α − 1)(σi,k (8) attains (ϵ, α)-Rényi Pufferfish Privacy for all (si , sj ) ∈ Q. Proof. (Proof of Theorem IV.1) We upper bound the Rényi divergence between the two privatized GMMs by exploiting plan π ∗ . P the∗ optimal transport 2 2 Let Pi (y) = and k,l πk,l N (y; µi,k , σi,k + θ ) P ∗ 2 2 Pj (y) = π N (y; µ , σ + θ ). Using the same j,l j,l k,l k,l coupling weights for the two mixtures allows us to compare the two distributions componentwise. We then have Z 1 Dα (Pi ||Pj ) = ln Pi (y)α Pj (y)1−α dy α−1 Z X 1 ∗ 2 ≤ ln πk,l N (y; µi,k , σi,k + θ2 )α α−1 k,l 2 × N (y; µj,l , σj,l + θ2 )1−α dy
=
X 1 ∗ ln πk,l ck,l (θ). α−1 k,l
Here s ck,l (θ) :=
2 + θ 2 )1−α (σ 2 + θ 2 )α (σi,k j,l 2 + θ2 ασj,l 2 + (1 − α)σi,k
× exp
α(α − 1)(µi,k − µj,l )2 2 + θ2 ) 2(ασj,l 2 + (1 − α)σi,k
! .
The second inequality follows from Jensen’s Inequality and the 1 positive parameter α−1 when α > 1, while the closed-form evaluation of the Gaussian integral follows the derivation of Corollary III.3. Define α−1 ψk,l (θ) := 2 + θ 2 + α(σ 2 − σ 2 )) 2(σi,k j,l i,k ! 2 2 2 2 α (σj,l − σi,k ) 2 . × α(µi,k − µj,l ) + 2 + θ2 ) (α − 1)(σi,k Then ck,l (θ) ≤ exp ψk,l (θ) , and hence Dα (Pi ||Pj ) ≤
X 1 ∗ ln πk,l exp ψk,l (θ) . α−1 k,l
Therefore, to attain pufferfish privacy it is sufficient to require X 1 ∗ ln πk,l exp ψk,l (θ) ≤ ϵ, α−1 k,l
which is equivalent to X ∗ πk,l exp ψk,l (θ) ≤ exp((α − 1)ϵ). k,l
This is exactly the condition stated in the theorem. Theorem IV.1 gives a sufficient condition to attain (ϵ, α)Rényi Pufferfish Privacy for Gaussian mixture priors, which can fit any data distributions. Figure 4 shows the corresponding privacy-budget sweep for GMM priors and compares our mixture-aware calibration with the additive baseline [15]. The required noise again decreases monotonically with ϵ for both R AW and M EAN queries, indicating that the GMM mechanism preserves the same privacy-budget trend as in the Gaussian case. Our method
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Noise vs. ϵ (α = 3.0, Gmm, RAW)
Noise vs. ϵ (α = 3.0, Gmm, MEAN) 1.5
Ours Pierquin et al. Noise Parameter θ
Noise Parameter θ
60
9
40
20
0
Ours Pierquin et al.
1
0.5
0
0
1
2 3 Privacy Budget ϵ
4
5
0
(a) R AW
1
2 3 Privacy Budget ϵ
4
5
Bayesian neural network separately on each secret-conditioned subset and using the predictive distribution as the query output. and GP corresponds to training a Gaussian process regressor on each secret-conditioned subset and using its predictive Gaussian as the query output. We organize the results along two standard privacy axes. First, we compare θ of our mechanism and baseline under an amount of privacy budget ϵ while fixing α = 3.0. Second, we plot θ of our mechanism under different Rényi order α.
(b) M EAN
Fig. 4. Noise parameter θ versus privacy budget ϵ for GMM-prior queries with α = 3.0.
also consistently attains a smaller θ than the baseline throughout the sweep, showing that Theorem IV.1 provides a tighter calibration when the secret-conditioned query distributions are better modeled by Gaussian mixtures. V. E XPERIMENTS We empirically evaluate our proposed Rényi Pufferfish Privacy mechanisms for Gaussian priors and GMM priors. For release Y , query output X, and zero-mean Gaussian noise Nθ , we have the mean square error (MSE) as E[(Y − X)2 ] = E[Nθ2 ] = θ2 . That is, a lower θ indicates that the mechanism attains the same (ϵ, α)-RPP requirement with less perturbation. In this work, we use the noise parameter θ to evaluate the utility of the mechanism. A. Experimental Setup Baseline. We compare our closed-form Gaussian-prior and GMM-prior calibration rules against the RPP additive-noise baseline of Pierquin et al. [15]. Datasets. The experiments cover three real-world datasets in the UCI machine learning repository, including Adult [36], Heart Disease [37], and Student Performance [38]. Across the three datasets, we consider a privacy-preserving release setting in which one attribute is designated as sensitive, and another is released. For the Adult dataset, race is treated as the sensitive attribute and education-num as the released attribute, with the secrets defined as “race = White” and “race = Other”. For the Heart Disease dataset, slope is treated as the sensitive attribute and oldpeak as the released attribute, with the secrets defined as “slope = 1” and “slope = 3”. For the Student Performance dataset, schoolsup is treated as the sensitive attribute and G3 as the released attribute, with the secrets defined as “schoolsup = no” and “schoolsup = yes”. In each case, the goal is to guarantee (ϵ, α)-statistical indistinguishability between the corresponding pair of secrets. Query. For each dataset, we evaluate two query families, Statistical Query (R AW, M EAN) and ML Query (BNN, GP). R AW corresponds to releasing the one-dimensional raw sample itself. M EAN corresponds to computing the mean on each secret-conditioned subset and approximating its output distribution by a univariate Gaussian using the sample mean and standard error. BNN corresponds to training a variational
B. Utility Figures 5, 6, and 7 compare our prior-aware calibration with the additive-noise baseline of Pierquin et al. [15] on the three real-world datasets. Across all datasets and all four query families, our mechanism consistently requires a smaller θ than the baseline for the same (ϵ, α) target, indicating better utility. This trend is particularly clear for the R AW and M EAN queries, where the query distributions are often visibly non-unimodal and thus benefit from the GMM-based calibration in Theorem IV.1. For the model-output queries, the same advantage remains under the Gaussian-prior calibration of Theorem III.5. Excluding regimes in which our mechanism requires no additional noise, the proposed method achieves an average noise reduction of 48.9% relative to the baseline. Overall, the improvement is not limited to a single dataset or query family; rather, it appears as a systematic downward shift of our curves relative to the baseline. When our method reaches zero noise earlier than the baseline, it indicates that prior-aware calibration can avoid unnecessary perturbation once the two secret-conditioned distributions already substantially overlap.
C. Monotonicity We next examine how the calibrated noise changes with the privacy parameters. For the budget sweep, the O URS curves in Figures 5, 6, and 7 are all monotonically decreasing in ϵ, which is consistent with Proposition III.6. This monotonicity holds for scalar queries as well as model-output queries. In some easier regimes, such as Adult GP and Adult BNN, the curves flatten at zero after a moderate privacy budget, which still agrees with the predicted non-increasing behavior. Figures 8 and 2 further show the monotonicity with respect to the Rényi order α. On all three real-world BNN queries, fixing ϵ and increasing α consistently shifts the curves upward, matching the increasing regime described in Proposition III.7. The ordering of the curves is stable across datasets: smaller α leads to uniformly lower noise, while larger α keeps the curves elevated over a wider range of ϵ. This pattern confirms that stronger tail-sensitive privacy requirements demand more perturbation, exactly as predicted by the theory. The model-output experiments further evaluate α-sensitivity for BNN and GP outputs. They follow the same qualitative pattern as the scalar query experiments: stronger Rényi orders require larger perturbations, while the proposed calibration remains below the baseline across datasets and model families.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Noise vs. ϵ (α = 3.0, Adult, MEAN)
Noise vs. ϵ (α = 3.0, Adult, RAW)
8
Ours Pierquin et al.
1
6
4
2
Noise Parameter θ
5
Noise vs. ϵ (α = 3.0, Adult, GP) ·10−4 1.2
Ours Pierquin et al. Noise Parameter θ
Noise Parameter θ
10
Noise vs. ϵ (α = 3.0, Adult, BNN)
Ours Pierquin et al.
Ours Pierquin et al. Noise Parameter θ
10
1
0.5
0.8 0.6 0.4 0.2
0
0 0
1
2 3 Privacy Budget ϵ
4
0
0
5
1
(a) R AW
2 3 Privacy Budget ϵ
4
0
5
0
1
(b) M EAN
2 3 Privacy Budget ϵ
4
5
0
1
(c) BNN
2 3 Privacy Budget ϵ
4
5
(d) GP
Fig. 5. Adult is chosen as a representative census-style demographic dataset, where the sensitive attribute is race, the released attribute is education-num, and the secrets are si =“race = White” and sj =“race = Other”. The figure shows the required noise parameter θ versus privacy budget ϵ with α = 3.0, where the four panels correspond to R AW, M EAN, BNN, and GP queries. Noise vs. ϵ (α = 3.0, Heart, RAW)
5
10
5
4
0
1
2 3 Privacy Budget ϵ
4
5
3 2 1
0
1
(a) R AW
2 3 Privacy Budget ϵ
4
Ours Pierquin et al.
6
0
0
0
Ours Pierquin et al. Noise Parameter θ
10
Ours Pierquin et al.
Noise Parameter θ
15
Noise Parameter θ
Noise Parameter θ
Ours Pierquin et al.
Noise vs. ϵ (α = 3.0, Heart, GP) ·10−3
Noise vs. ϵ (α = 3.0, Heart, BNN)
Noise vs. ϵ (α = 3.0, Heart, MEAN) 15
4
2
0 0
5
1
2 3 Privacy Budget ϵ
(b) M EAN
4
5
0
1
(c) BNN
2 3 Privacy Budget ϵ
4
5
(d) GP
Fig. 6. Heart Disease is chosen as a representative healthcare dataset, where the sensitive attribute is slope, the released attribute is oldpeak, and the secrets are si =“slope = 1” and sj =“slope = 3”. The figure shows the required noise parameter θ versus privacy budget ϵ with α = 3.0, where the four panels correspond to R AW, M EAN, BNN, and GP queries. Noise vs. ϵ (α = 3.0, Student, MEAN)
Ours Pierquin et al.
10 5 0
4
2
1
2 3 Privacy Budget ϵ
(a) R AW
4
5
1.5 1 0.5 0
0 0
0
1
2 3 Privacy Budget ϵ
4
5
(b) M EAN
Noise vs. ϵ (α = 3.0, Student, GP) ·10−3 Ours Pierquin et al.
Ours Pierquin et al.
2 Noise Parameter θ
15
3
Ours Pierquin et al.
6 Noise Parameter θ
Noise Parameter θ
20
Noise vs. ϵ (α = 3.0, Student, BNN)
Noise Parameter θ
Noise vs. ϵ (α = 3.0, Student, RAW)
2
1
0 0
1
2 3 Privacy Budget ϵ
4
5
0
(c) BNN
1
2 3 Privacy Budget ϵ
4
5
(d) GP
Fig. 7. Student Performance is chosen as a representative education-performance dataset, where the sensitive attribute is schoolsup, the released attribute is G3, and the secrets are si =“schoolsup = no” and sj =“schoolsup = yes”. The figure shows the required noise parameter θ versus privacy budget ϵ with α = 3.0, where the four panels correspond to R AW, M EAN, BNN, and GP queries.
D. Summary The experiments support three main conclusions. First, compared with the additive-noise baseline, our mechanisms consistently require less noise, which directly improves data utility under the same (ϵ, α)-Rényi Pufferfish Privacy guarantee. Second, the proposed framework is not limited to Gaussian priors, through the GMM-based construction, it also applies to general non-Gaussian and multimodal query distributions, extending the method beyond the unimodal setting. Third, the key properties established in the theoretical analysis are reflected by the empirical results, as the utility advantage and the monotonic trends with respect to ϵ and α consistently match the predictions of our theorems and propositions.
VI. C ONCLUSION This paper studies Gaussian mechanisms for Rényi Pufferfish Privacy (RPP) under single Gaussian and Gaussianmixture priors. For single Gaussian priors, we derived the exact Rényi divergence after Gaussian perturbation, established the monotonicity of the privacy loss with respect to the noise variance, and presented both a numerical calibration algorithm to determine the optimal noise variance. We also provide a closed-form sufficient condition along with its monotonicity with respect to privacy budget ϵ and Rényi order α. To move beyond the general setting, we further modeled secret-conditioned outputs with Gaussian mixture models and developed an optimal-transport-based sufficient
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
θ vs. ϵ (Heart Disease)
θ vs. ϵ (Student) α = 1.5 α=2 α=3 α=5
α = 1.5 α=2 α=3 α=5
Noise Parameter θ
α = 1.5 α=2 α=3 α=5
Noise Parameter θ
θ vs. ϵ (Adult)
Nosie Parameter θ
11
Privacy Budget ϵ
Privacy Budget ϵ
Privacy Budgetϵ
(a) Adult
(b) Heart Disease
(c) Student
Fig. 8. The three real-world datasets including Adult, Heart Disease, and Student Performance. Adult releases attribute education-num while preserving sensitive attribute race, where secrets si =“race = White” and sj =“race = Other”; Heart Disease releases attribute oldpeak while preserving sensitive attribute slope, and secrets si =“slope = 1” and sj =“slope = 3”; and Student Performance releases attribute G3 while preserving sensitive attribute schoolsup, where secrets si =“schoolsup = no” and sj =“schoolsup = yes”. The figure shows how the θ–ϵ curves of our BNN query calibration change under different fixed Rényi orders α ∈ {1.5, 2.0, 3.0, 5.0}.
condition for RPP under mixture priors. Experimental results on three UCI datasets with statistical and model-output queries showed that the proposed prior-aware mechanisms consistently require less noise than a recent additive-noise baseline, leading to improved utility under the same (ϵ, α)-RPP guarantee. R EFERENCES [1] S. Goldwasser, S. Micali, and C. Rackoff, “The knowledge complexity of interactive proof-systems,” in Providing sound foundations for cryptography: On the work of shafi goldwasser and silvio micali, 2019, pp. 203–225. [2] Z. Xing, Z. Zhang, Z. Zhang, Z. Li, M. Li, J. Liu, Z. Zhang, Y. Zhao, Q. Sun, L. Zhu et al., “Zero-knowledge proof-based verifiable decentralized machine learning in communication network: A comprehensive survey,” IEEE Communications Surveys & Tutorials, 2025. [3] C. Gentry, “Fully homomorphic encryption using ideal lattices,” in Proceedings of the forty-first annual ACM symposium on Theory of computing, 2009, pp. 169–178. [4] C. Hu and B. Li, “Maskcrypt: Federated learning with selective homomorphic encryption,” IEEE Transactions on Dependable and Secure Computing, vol. 22, no. 1, pp. 221–233, 2024. [5] A. C. Yao, “Protocols for secure computations,” in 23rd annual symposium on foundations of computer science (sfcs 1982). IEEE, 1982, pp. 160–164. [6] Z. Li, Z. Zhang, W. Yang, P. Wang, Z. Wang, M. Li, Y. Wu, X. Liu, J. Sun, and L. Zhu, “Ebyftves: Efficient byzantine fault tolerant-based verifiable secret-sharing in distributed privacy-preserving machine learning,” arXiv preprint arXiv:2509.12899, 2025. [7] V. Costan and S. Devadas, “Intel sgx explained,” Cryptology ePrint Archive, 2016. [8] Q. Burke, Y. Beugin, B. Hoak, R. King, E. Pauley, R. Sheatsley, M. Yu, T. He, T. F. La Porta, and P. McDaniel, “Securing cloud file systems with trusted execution,” IEEE Transactions on Dependable and Secure Computing, vol. 22, no. 3, pp. 1976–1992, 2024. [9] C. Dwork, “Differential privacy,” in International colloquium on automata, languages, and programming. Springer, 2006, pp. 1–12. [10] C. Dwork, F. McSherry, K. Nissim, and A. D. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings, vol. 3876, 2006, pp. 265–284. [11] Z. Chen, L. Yao, H. Hu, and G. Wu, “eta-inference: A data-aware and high-utility privacy model for relational data publishing,” IEEE Transactions on Dependable and Secure Computing, 2025. [12] L. Yao, X. Wang, H. Hu, and G. Wu, “A utility-aware anonymization model for multiple sensitive attributes based on association concealment,” IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 4, pp. 2045–2056, 2023.
[13] D. Kifer and A. Machanavajjhala, “A rigorous and customizable framework for privacy,” in Proceedings of the 31st ACM SIGMOD-SIGACTSIGART Symposium on Principles of Database Systems, PODS 2012, Scottsdale, AZ, USA, May 20-24, 2012, 2012, pp. 77–88. [14] ——, “Pufferfish: A framework for mathematical privacy definitions,” ACM Transactions on Database Systems (TODS), vol. 39, no. 1, pp. 1–36, 2014. [15] C. Pierquin, A. Bellet, M. Tommasi, and M. Boussard, “Rényi pufferfish privacy: General additive noise mechanisms and privacy amplification by iteration via shift reduction lemmas,” in Forty-first International Conference on Machine Learning, 2024. [16] T. Van Erven and P. Harremos, “Rényi divergence and kullback-leibler divergence,” IEEE Transactions on Information Theory, vol. 60, no. 7, pp. 3797–3820, 2014. [17] S. Song, Y. Wang, and K. Chaudhuri, “Pufferfish privacy mechanisms for correlated data,” in Proceedings of the 2017 ACM International Conference on Management of Data, 2017, pp. 1291–1306. [18] T. Champion, L. De Pascale, and P. Juutinen, “The ∞-wasserstein distance: Local solutions and existence of optimal transport maps,” SIAM Journal on Mathematical Analysis, vol. 40, no. 1, pp. 1–20, 2008. [19] L. De Pascale and J. Louet, “A study of the dual problem of the onedimensional l∞-optimal transport problem with applications,” Journal of Functional Analysis, vol. 276, no. 11, pp. 3304–3324, 2019. [20] N. Ding, “Kantorovich mechanism for pufferfish privacy,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2022, pp. 5084–5103. [21] W. Yang, N. Ding, Z. Zhang, J. Sun, Z. Li, Y. Wu, J. Sun, H. Lin, Y. Liu, J. An et al., “Noise reduction for pufferfish privacy: A practical noise calibration method,” arXiv preprint arXiv:2601.06385, 2026. [22] T. Zhang and Y. Vorobeychik, “Sliced r\’enyi pufferfish privacy: Directional additive noise mechanism and private learning with gradient clipping,” arXiv preprint arXiv:2512.01115, 2025. [23] S. W. Ober and L. Aitchison, “Global inducing point variational posteriors for bayesian neural networks and deep gaussian processes,” in International Conference on Machine Learning. PMLR, 2021, pp. 8248–8259. [24] Z. Qian, S. Wang, M. Mihajlovic, A. Geiger, and S. Tang, “3dgsavatar: Animatable avatars via deformable 3d gaussian splatting,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 5020–5030. [25] M. Y. Balık, M. Sinelnikov, P. Ong, and H. Lähdesmäki, “Bayesian basis function approximation for scalable gaussian process priors in deep generative models,” in Forty-second International Conference on Machine Learning, 2025. [26] M. Lucic, M. Faulkner, A. Krause, and D. Feldman, “Training gaussian mixture models at scale via coresets,” Journal of Machine Learning Research, vol. 18, no. 160, pp. 1–25, 2018. [27] M. A. Pathak and B. Raj, “Large margin gaussian mixture models
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
with differential privacy,” IEEE Transactions on dependable and secure computing, vol. 9, no. 4, pp. 463–469, 2012. [28] N. Ding, “Approximation of pufferfish privacy for gaussian priors,” IEEE Transactions on Information Forensics and Security, 2024. [29] N. Ding, S. Lu, W. Yang, and Z. Zhang, “Multi-user pufferfish privacy,” arXiv preprint arXiv:2512.18632, 2025. [30] J. Bai, G. He, X. Gu, D. Kifer, and K. Maeng, “Composition for pufferfish privacy,” arXiv preprint arXiv:2602.02718, 2026. [31] T. Nuradha, Z. Goldfeld, and M. M. Wilde, “Quantum pufferfish privacy: A flexible privacy framework for quantum systems,” IEEE Transactions on Information Theory, 2024. [32] T. Nuradha, V. Singh, and M. M. Wilde, “Measured hockey-stick divergence and its applications to quantum pufferfish privacy,” arXiv preprint arXiv:2501.12359, 2025. [33] I. Mironov, “Rényi differential privacy,” in 2017 IEEE 30th computer security foundations symposium (CSF). IEEE, 2017, pp. 263–275. [34] J. I. McCool, “Non-gaussian effects in microcontact,” International Journal of Machine Tools and Manufacture, vol. 32, no. 1-2, pp. 115– 123, 1992. [35] L. Scrucca, M. Fop, T. B. Murphy, and A. E. Raftery, “mclust 5: clustering, classification and density estimation using gaussian finite mixture models,” The R journal, vol. 8, no. 1, p. 289, 2016. [36] B. Becker and R. Kohavi, “Adult,” UCI Machine Learning Repository, 1996, DOI: https://doi.org/10.24432/C5XW20. [37] A. Janosi, W. Steinbrunn, M. Pfisterer, and R. Detrano, “Heart Disease,” UCI Machine Learning Repository, 1989, DOI: https://doi.org/10.24432/C52P4X. [38] P. Cortez, “Student Performance,” UCI Machine Learning Repository, 2008, DOI: https://doi.org/10.24432/C5TG7T.
12