arXiv CSopen access

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

arXiv:2606.08252v1 [cs.CR] 6 Jun 2026

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning SHENG WAN, HKUST & SUSTech, China DASHAN GAO, HKUST & SUSTech, China HANLIN GU, Webank, China LIXIN FAN, Webank, China DANING HU, SUSTech, China QIANG YANG, Webank, China Federated learning (FL) aims to protect data privacy by collaboratively learning a model without sharing private data among clients. Unlike traditional parameter-based FL methods that exchange model weights or gradients during training, emerging logit-based FL approaches share model outputs (logits) on public data. This strategy promotes model heterogeneity, reduces communication overhead, and enhances clients’ privacy. However, the potential privacy risks associated with these logit-based methods have been largely overlooked. This research presents the first theoretical and empirical analysis of a hidden privacy risk in logit-based FL methods – the risk that a semi-honest server (adversary) may learn clients’ private models from logits. To quantify and address this threat, we develop the Adaptive Model Stealing Attack (AdaMSA) by leveraging historical logits during training. Notably, we observe that this inherent privacy risk persists even when public data is unrelated to private data, emphasizing the urgency to address privacy vulnerabilities in logit-based FL methods. Moreover, our theoretical analysis establishes the bounds of this privacy risk. We then propose a simple but effective defense strategy that perturbs the transmitted logits in the direction that minimizes the privacy risk while maximally preserving the training performance. The experimental results validate our analysis and demonstrate the effectiveness of AdaMSA and our defense strategy. CCS Concepts: • Computing methodologies → Artificial intelligence; Distributed computing methodologies; • Security and privacy → Distributed systems security. Additional Key Words and Phrases: Logit-based Federated Learning, Privacy Risk, Perturbation Defense ACM Reference Format: Sheng Wan, Dashan Gao, Hanlin Gu, Lixin Fan, Daning Hu, and Qiang Yang. 2024. Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning. 1, 1 (June 2024), 20 pages. https://doi.org/XXXXXXX.XXXXXXX

1

INTRODUCTION

In recent years, data privacy regulations such as the General Data Protection Regulation have largely restricted the collection of annotated data on individuals for centralized training. Federated Learning (FL) [25] offers a promising approach that enables different clients to collaboratively train their models by sharing local model parameters or Authors’ addresses: Sheng Wan, [email protected], HKUST & SUSTech, Shenzhen, China; Dashan Gao, [email protected], HKUST & SUSTech, Shenzhen, China; Hanlin Gu, Webank, Shenzhen, China, ; Lixin Fan, Webank, Shenzhen, China, ; Daning Hu, SUSTech, Shenzhen, China, ; Qiang Yang, Webank, Shenzhen, China, . Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM

Manuscript submitted to ACM

1

2

Sheng et al.

Fig. 1. Illustration of logit-based FL and our attack setting. Private data and models are maintained locally, with only local predicted logits being transmitted to the server. The server is semi-honest and aims to infer client 𝑘 ′𝑠 private model via its logits 𝑝𝑘 on an unlabeled public dataset D𝑝𝑢𝑏 during training.

gradients without exchanging their respective raw data. However, recent studies [9, 38] have introduced gradient-based attacks, demonstrating the potential extraction of private training data from shared gradients or parameters. This revelation poses a substantial privacy leakage concern. Another line of FL studies [10, 11, 14, 22] adopt knowledge distillation [12] to exchange model outputs (i.e., logits) instead of model weights or gradients during training to reduce communication overhead and enable model heterogeneity. To differentiate this from the traditional federated learning approaches, which transmit parameters, we refer to this method as logit-based federated learning. To preserve clients’ privacy, these logit-based FL methods distill on public data to transfer knowledge and model parameters are stored locally during training, as depicted in Figure 1. Moreover, such public data can be unlabeled and insensitive that is sampled from other domains [11]. A natural question arises: Can the logit-sharing scheme ensure the privacy of each participant? Intuitively, restricting clients to share only the logits—i.e., the outputs of the final layer of the model—instead of sharing the entire model can be considered a form of data minimization. This approach appears to reduce the potential disclosure of private information within the shared updates sent to the adversary. However, we argue that the transmitted logits still pose a risk of privacy leakage for the private model. As the model remains hidden on local machines, it could still be exploited by adversaries as a prerequisite for gradient-based attacks [9, 38] or membership inference attacks [34]. Therefore, ensuring model privacy should be a fundamental requirement for logit-based federated learning. To our knowledge, we are the first to provide a theoretical and empirical analysis of a hidden privacy risk in logitbased FL that the semi-honest server intends to infer clients’ private models without knowing local model architecture or data distribution. To quantify the impacts of the privacy risk, we develop an effective attack dubbed as Adaptive Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

3

Table 1. Existing privacy attacks against FL.

Attack

CPA [28] mGAN [37] UFL [26] DLG [38] InvertGrad [9] AdaMSA (ours)

Adversary Semi-honest/ Malicious Client Malicious Client Semi-honest/ Malicious Client Semi-honest Server Semi-honest Server Semi-honest Server

Adversary’s Goal

Adversary’s Knowledge Model Gradient Logit

Applicable for Logit-based FL

Infer Membership

✓

✓

×

×

Infer Class Representative

✓

✓

×

×

Infer Property

✓

✓

×

×

Infer Training Data Infer Training Data Infer Model

✓ ✓ ×

✓ ✓ ×

✓ ✓ ✓

× × ✓

Model Stealing Attack (AdaMSA), which adaptively steals the private model by approximating its intermediate training states from previous iterations. In each iteration, the semi-honest server compels the attacking model to approximate the current state of the victim model by minimizing the distance between the output of the attacking model and a target logit. We propose combining the observed historical logits of the victim model to capture diverse aspects of private information. This approach enables us to obtain a more informative target logit, as empirically validated in Section 4.2. Moreover, we provide a theoretical analysis establishing the bounds of this privacy risk in logit-based FL in Section 3.3. To protect model privacy in logit-based FL, we propose a simple but effective perturbation-based defense strategy named Federated Logit Perturbation (FedLP). The key idea of our strategy is to perturb the logit in the direction that optimally obstructs the adversary while maximally preserving the model performance. As a result, our defense achieves a better trade-off compared to prior arts. We empirically evaluate our proposed attack and defense approach in three experimental settings, Close-world, Open-world-CF and Open-world-TI (see Section 5.1 for details). Our empirical findings validate our theoretical analysis and demonstrate the effectiveness of AdaMSA, revealing its success even in scenarios where public data lacks relevance to private data. Additionally, our proposed FedLP achieves a better utility and privacy trade-off compared to state-ofthe-art baselines. Our observation underscores the importance of this hidden privacy risk and highlights the urgency to address privacy vulnerabilities in logit-based FL methods. We hope our research can serve as a valuable tool, aiding researchers in evaluating privacy risks within logit-based FL and fostering the development of privacy-preserving FL methodologies. Our key contributions are summarized as follows: • To the best of our knowledge, we provide the first theoretical and empirical analysis of a hidden privacy risk in logit-based FL that the semi-honest server can infer clients’ private models according to logits. • To quantify the privacy risk, we develop an effective model stealing attack named AdaMSA, which steals private models by leveraging historical logits during training. Moreover, we provide a theoretical bound for the privacy risk in logit-based FL. • To prevent the privacy risk, we develop a simple but effective defense named FedLP by perturbing the transmitted logits in the direction that minimizes the privacy risk while maximally preserving the model performance. • We conduct empirical evaluations of our designed attack across three experimental settings: Close-world, Openworld-CF, and Open-world-TI. The results not only validate our analysis but also demonstrate that AdaMSA can Manuscript submitted to ACM

4

Sheng et al.

achieve up to a 3.69% improvement and our defense can achieve a better utility and privacy trade-off compared to the state-of-the-arts. 2

RELATED WORK

2.1

Privacy Risk in Federated Learning

Federated learning [15] allows multiple clients to collaboratively train a global model while keeping training data locally. Typical FL algorithms [16, 25] are parameter-based FL that shares local model parameters or gradients and aggregate local models in the server. Logit-based FL [10, 11, 14, 22] adopt knowledge distillation [12] to transmit model outputs (i.e., logits) instead of model weights or gradients during training to reduce communication overhead, enable model to be heterogeneous and preserve clients’ privacy. Previous studies have extensively investigated the privacy risks of sharing model parameters or gradients in FL, addressing concerns like class representatives leakage [37], membership leakage [28], property leakage [26] and training input leakage [9, 38]. However, these efforts primarily concentrate on gradient-based attacks under white-box assumptions, as detailed in Table 1. In other word, they have strong assumptions that the adversary knows the local model architecture and detailed training information such as gradients. In this work, we focus on logit-based FL, where model parameters or gradients are stored in clients’ local machines. A recent empirical study [35] delves into breaching the logit-based FL framework. While their emphasis lies in reconstructing class representations of private data and is contingent on the relevance of public data to private data, our objective is to steal the functionality of the private model. Importantly, our approach can succeed even in scenarios where public data lacks relevance to private data. To our knowledge, this research represents the first investigation providing both theoretical and empirical analyses on the privacy risk associated with logit-based FL. 2.2

Model Stealing Attack

Model stealing attacks [31, 32, 36] have demonstrated the ability to steal a deployed machine learning model in a black-box manner through limited query access and carefully calibrated proxy dataset. These attacks happen in the inference stage and aim to reduce the number of queries or eliminate the need of proxy dataset. However, in logit-based FL, the attack happens in the training stage, where the adversary can neither arbitrarily select the query dataset nor access to the private models or private dataset distribution. Instead, the adversary only observes the intermediate information (i.e., transmitted logits) from the victim during training. Based on this observation, we propose AdaMSA that leverages historical training information to obtain more informative target logits and therefore improve the attack performance. 2.3

Privacy Protection Strategy in Logit-based FL

Researchers have proposed some strategies [11, 22, 33] to prevent the potential privacy leakage in logit-based FL. Specifically, Li et al. [22] proposed to distill on a public dataset instead of private data to transfer predicted vectors. Gong et al. [11] further relaxed the public data to be unlabeled and insensitive data sampled from other domains to preserve data privacy. Moreover, Sattler et al. [33] and Gong et al. [11] adopted differential privacy (DP) to protect the transmitted logits. However, these paper fails to quantify the privacy risk inside logit-based FL and their defense strategies incur a significant loss in accuracy. In contrast, we first identify and quantify the privacy risk. Then we design a simple but effective perturbation strategy against our revealed privacy risk, which perturbs the logit in the direction Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

5

that maximally misleads the adversary while minimally persevering training performance. Therefore, it can achieve a better utility and privacy trade-off. 3

QUANTIFYING THE PRIVACY RISK

In this section, we begin by providing the problem setup and threat model. Then we propose an attack to quantify the privacy risk in logit-based FL and elaborate our proposed attack in details. Lastly, we give a theoretical analysis on the bound of this privacy risk. 3.1

Problem Setup and Threat Model

Problem Setup: As shown in Figure 1, there are 𝑛 clients and a central server. Each client has a private labeled dataset {D𝑖 }𝑛𝑖=1 and some unlabeled public data D𝑝𝑢𝑏 . The server coordinates the training process by aggregating the logits predicted locally by the clients. Specifically, the model 𝑓 (𝑥, 𝜃 ) takes input data 𝑥 ∈ D𝑝𝑢𝑏 and outputs logits 𝑝 = 𝑓 (𝑥, 𝜃 ), where logits represent the unnormalized prediction scores for each class. The locally predicted logits {𝑝𝑖 }𝑛𝑖=1 are aggregated by the server to compute an ensemble logit for D𝑝𝑢𝑏 . Then, clients train their local models {𝜃 𝑖 }𝑛𝑖=1 under the supervision of labels on {D𝑖 }𝑛𝑖=1 and the ensemble logits on D𝑝𝑢𝑏 . In the subsequent discussion, we use logits 𝑝 to represent the outputs of the model 𝑓 (𝑥, 𝜃 ) for simplicity. Threat Model: We assume that the server (i.e., adversary) is semi-honest, i.e., it completes the learning task as required but is curious about the clients’ local models. All clients are assumed honest. The adversary does not know the private data distribution or the victim model, including its parameters, hyperparameters, or architecture. Moreover, the adversary does not have the right to select public data. The adversary only knows that: 1) the unlabeled public dataset D𝑝𝑢𝑏 ; 2) the transmitted victim’s logits {𝑝 𝑡 }𝑇𝑡=1 on the public data during the training process with 𝑇 iterations in total. The adversary aims to approximate the functionality of the victim’s model 𝑓 (𝑥, 𝜃 ) by training an attacking model 𝑓 (𝑥, 𝜃˜) to achieve high accuracy. Specifically, the adversary seeks to maximize the classification accuracy (Acc) of the attacking model on the victim 𝑘’s private dataset D𝑘 :   max E𝑥∼D𝑘 Acc 𝑓 (𝑥, 𝜃˜) . 𝜃˜

We summarize the threat model in Table 2. In the following discussion, we assume that the adversary is interested in client 𝑘’s model, denoted as 𝜃 for simplicity. Table 2. Threat model.

Threat Model Semi-honest

Adversary Server

Attack Target Private model 𝜃

Adversary’s Knowledge D𝑝𝑢𝑏 and logits of 𝜃 on D𝑝𝑢𝑏 .

Notations: Table 3 provides a summary of the notations employed throughout this paper. 3.2

Adaptive Model Stealing Attack

To quantify the privacy risk inherent in logit-based FL, we propose an Adaptive Model Stealing Attack, denoted as AdaMSA. AdaMSA adaptively steals the victim’s private model by approximating its intermediate training states from prior iterations during training. Specifically, in each iteration, the server compels the attacking model to approximate the victim model by mimicking a designated target logit on public data. The crucial challenge here lies in the design of the target logit, as a more informative target logit can enhance the supervision for the attacking model. Manuscript submitted to ACM

6

Sheng et al. Table 3. Table of Notations

Symbol

Description

D𝑝𝑢𝑏 D𝑘 𝑥𝑖 𝑓 (𝑥, 𝜃 ) 𝑓 (𝑥, 𝜃˜) 𝑝𝑖𝑡 𝑝ˆ𝑇𝑖 𝑝˜𝑇𝑖 𝑇 𝑇0 𝑤𝑡 𝑍 𝜂 L𝐶𝐸

Public dataset used for training and evaluation. Private dataset of the victim 𝑘. A single data sample from the public dataset D𝑝𝑢𝑏 . Victim model with parameters 𝜃 . Attacking model with parameters 𝜃˜. Victim logits of 𝑥𝑖 predicted by the victim model at iteration 𝑡. Target logits for 𝑥𝑖 combining iterations 𝑇 − 𝑇0 to 𝑇 . Attacking logits of 𝑥𝑖 predicted by the attacking model at iteration 𝑇 . Maximum number of training iterations. Threshold controlling the range of past predictions. Weight assigned to logits from iteration 𝑡. Í Normalization factor ensuring 𝑇𝑡=𝑇 −𝑇0 𝑤𝑡 = 1. Learning rate used for gradient descent. Cross-entropy loss function.

Given the historical logits during training, we formulate the design of the target logit with two key considerations: 1) Historical predictions may contain valuable information, offering diverse perspectives of the data to enhance the supervision for the attacking model; 2) Predictions in the early rounds may not be sufficiently trained to provide informative supervision. Accordingly, we define the target logit 𝑝ˆ𝑇 as 𝑝ˆ𝑇 =

𝑇 ∑︁ 𝑡 · 𝑝𝑡 , 𝑍 𝑡 =𝑇 −𝑇 0

where 𝑇0 is the threshold governing the extent of past predictions to be considered, and 𝑍 =

Í𝑇

𝑘=𝑇 −𝑇0 𝑘 is a normalization 𝑡 factor that ensures the weights 𝑤𝑡 = 𝑍 sum to 1, effectively balancing the contributions of logits from different iterations. To emphasize predictions closer to the current iteration 𝑇 , we let 𝑍𝑡 increase linearly with the iteration number 𝑡,.

˜

To train the attacking model 𝜃 , we formulate the cross-entropy loss L𝐶𝐸 as the discrepancy between the target logit 𝑝ˆ𝑇 and the prediction of the attacking model (the attacking logit) 𝑝˜𝑇 on D𝑝𝑢𝑏 , defined as follows: min𝜃˜ E𝑥∼D𝑝𝑢𝑏 [L𝐶𝐸 (𝑝ˆ𝑇 , 𝑝˜𝑇 )]. Then we employ gradient descent to address this optimization problem: 𝜃˜ = 𝜃˜ − 𝜂∇𝜃˜ L̃𝐶𝐸 (𝑝ˆ𝑇 , 𝑝˜𝑇 ). The details of AdaMSA in one training iteration is given in Algorithm 1. By repeating this process, the semi-honest server is able to steal any desired intermediate private model throughout training, including the ultimately well-trained private model of the victim. Subsequently, we can quantify the privacy risk R in logit-based FL by evaluating the accuracy of the obtained attacking model 𝑓 (𝑥, 𝜃˜) on the victim’s private test dataset D𝑘 as follows:   R = E𝑥∼D𝑘 Acc 𝑓 (𝑥, 𝜃˜) . A higher accuracy indicates a greater degree of privacy leakage from the victim. Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

7

Algorithm 1: AdaMSA in one iteration 𝑁 , victim’s logits {𝑝 𝑡 } 𝑁 ,𝑇 Input: Public dataset D𝑝𝑢𝑏 = {𝑥𝑖 }𝑖=1 𝑖 𝑖=1,𝑡 =1 predicted on D𝑝𝑢𝑏 in the past 𝑇 iterations, attacking ˜ model with parameters 𝜃 , maximum number of iterations 𝑇 , and learning rate 𝜂. Output: Optimized attacking model parameters 𝜃˜. 1: while not converged do 2: Randomly shuffle the public dataset D𝑝𝑢𝑏 3: for 𝑥𝑖 ∈ D𝑝𝑢𝑏 do Í 4: Compute the ensemble of historical logits for 𝑥𝑖 : 𝑝ˆ𝑇𝑖 ← 𝑇𝑡=𝑇 −𝑇0 𝑍𝑡 · 𝑝𝑖𝑡 5: Compute the predicted logits from the attacking model: 𝑝˜𝑇𝑖 ← 𝑓 (𝑥𝑖 , 𝜃˜) 6: Update the attacking model 𝜃˜ via cross-entropy gradient: 𝜃˜ ← 𝜃˜ − 𝜂∇𝜃˜ L𝐶𝐸 (𝑝ˆ𝑇𝑖 , 𝑝˜𝑇𝑖 ) 7: end for 8: end while 9: return 𝜃˜

Here we provide a brief analysis of the computational complexity associated with our proposed attack. Assume the communication round is 𝑀. Then the computational complexity of AdaMSA, as described in Algorithm 1, is expressed as 𝑂 (𝑀 |D𝑝𝑢𝑏 |). In the context of logit-based Federated Learning, the communication cost, also proportionate to 𝑀 |D𝑝𝑢𝑏 |, is expected to be bounded due to the inherent communication bottleneck. Consequently, our attack will not be expected to impose a significant burden on the server. 3.3

Privacy Risk Analysis

Logit-based FL methods transfer knowledge through the transmitted logits on public data during training. As shown in Figure 2, we discern that the distance measured by the correlation between the private dataset and the public dataset plays a crucial role in determining the performance of logit-based FL methods. To have a deeper understanding of the inherent cause of privacy risk in this logit-sharing scheme, we start with quantifying the distance between the private dataset and the public dataset. Previous research has not extensively explored systematic methods for adjusting the distance between distinct datasets. To address this issue, we propose to construct a mixed dataset, which is then employed as the public dataset in the standard logit-based FL setting. This design equips us with the ability to systematically manipulate the distance between the mix and private datasets. Consider a simple case that we have 𝑛 private datasets sampled from same distribution D𝑝𝑟𝑖𝑣 and an unlabeled public dataset from another domain sampled from independent distribution D𝑝𝑢𝑏 . As shown in Figure 3, the mixed dataset is constructed through D𝑚𝑖𝑥 = (𝑆 1, 𝑆 2 ), where 𝑆 1 consists of 𝛼 |D𝑚𝑖𝑥 | instances sampled independently from D𝑝𝑟𝑖𝑣 and 𝑆 2 consists of (1 − 𝛼)|D𝑚𝑖𝑥 | instances sampled independently from D𝑝𝑢𝑏 . Through varying the weighting parameter 𝛼, we can thereby quantify and control the distance between private and mixed datasets. For instance, as 𝛼 tends to 1, the mixed dataset approaches to the private dataset, and vice versa. As previously illustrated in Section 3.2, the privacy risk is measured by the performance of AdaMSA on the victim’s private test dataset. In consequence, we can establish the bound of the privacy risk via the performance bound of AdaMSA on the victim’s private test dataset based on the prior art in domain adaptation [4]. Denote the empirical risk of model 𝜃 on mixed dataset as 𝜖 D𝑚𝑖𝑥 (𝜃, 𝑓𝑝 ) = E𝑥∼D𝑚𝑖𝑥 [|𝜃 (𝑥) − 𝑓𝑝 (𝑥)|],

(1) Manuscript submitted to ACM

8

Sheng et al.

(a) Open-world-CF

(b) Open-world-TI

Fig. 2. The impact of the distance between the private dataset and the public dataset on the utility of logit-based FL in Open-world-CF and Open-world-TI settings (See Section 5.1 for details). The distance is controlled by the weighting parameter 𝛼.

Fig. 3. Illustration of constructing the mixed dataset D𝑚𝑖𝑥 .

which measures the probability according the distribution D that 𝜃 disagrees with the ground truth label 𝑓𝑝 . For simplicity, we abbreviate 𝜖 (𝜃, 𝑓𝑝 ) as 𝜖 (𝜃 ). According to the definition provided in Equation 1, we can express the empirical risk of the mixed dataset in terms of the empirical risk of the public and private datasets, as demonstrated in Theorem 3.1. Theorem 3.1. Given a mixed dataset D𝑚𝑖𝑥 = (𝑆 1, 𝑆 2 ), where 𝑆 1 consists of 𝛼 |D𝑚𝑖𝑥 | instances sampled independently from D𝑝𝑟𝑖𝑣 , 𝑆 2 consists of (1 − 𝛼)|D𝑚𝑖𝑥 | instances sampled independently from D𝑝𝑢𝑏 , its empirical risk can be written as 𝜖 D𝑚𝑖𝑥 (𝜃 ) = 𝛼𝜖 D𝑝𝑟𝑖𝑣 (𝜃 ) + (1 − 𝛼)𝜖 D𝑝𝑢𝑏 (𝜃 ). Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

9

Proof. According to the definition given in Equation (1), we have 𝜖 D𝑚𝑖𝑥 (𝜃, 𝑓𝑝 ) =E𝑥∼D𝑚𝑖𝑥 [|𝜃 (𝑥) − 𝑓𝑝 (𝑥)|] ∑︁ 1 = |𝜃 (𝑥𝑖 ) − 𝑓𝑝 (𝑥𝑖 )| |D𝑚𝑖𝑥 | 𝑥 ∈ D 𝑖 𝑚𝑖𝑥 ∑︁ ∑︁ 1 = [ |𝜃 (𝑥𝑖 ) − 𝑓𝑝 (𝑥𝑖 )| + |D𝑚𝑖𝑥 | 𝑥 ∈ D 𝑥 ∈D 𝑖

=𝛼 ·

𝑝𝑟𝑖𝑣

𝑗

∑︁ 1 |𝛼 D𝑚𝑖𝑥 | 𝑥 ∈ D 𝑖

(1 − 𝛼) ·

|𝜃 (𝑥 𝑗 ) − 𝑓𝑝 (𝑥 𝑗 )|]

𝑝𝑢𝑏

|𝜃 (𝑥𝑖 ) − 𝑓𝑝 (𝑥𝑖 )|+

𝑝𝑟𝑖𝑣

∑︁ 1 |(1 − 𝛼)D𝑚𝑖𝑥 | 𝑥 ∈ D 𝑗

|𝜃 (𝑥 𝑗 ) − 𝑓𝑝 (𝑥 𝑗 )|

𝑝𝑢𝑏

=𝛼𝜖 D𝑝𝑟𝑖𝑣 (𝜃 ) + (1 − 𝛼)𝜖 D𝑝𝑢𝑏 (𝜃 ). Then we introduce some definitions for our latter analysis: Definition 3.2 (H-Divergence between Distributions). Given a domain X with D and D ′ probability distributions over X, let H be a hypothesis class on X and A H be the set of subsets of X that supports the hypothesis in H . The H-divergence between D and D ′ is defined as: 𝑑𝐻 (D, D ′ ) = 2𝑠𝑢𝑝𝐴∈𝐴𝐻 |𝑃𝑟 D (𝐴) − 𝑃𝑟 D ′ (𝐴)|. Definition 3.3 (Symmetric Difference Hypothesis Space). For a hypothesis space H , the symmetric difference hypothesis É ′ É ℎ (𝑥)|ℎ, ℎ ′ ∈ H }, where represents the XOR operation.

space H ΔH is defined as H ΔH = {ℎ(𝑥)

Subsequently, we can derive the bound of the difference between the empirical risk of 𝜃 on the mixed dataset D𝑚𝑖𝑥 and the private dataset D𝑝𝑟𝑖𝑣 and present our main theorem as follows: Theorem 3.4. Let 𝐻 be a hypothesis space of VC-dimension 𝑑 and ℎ be a hypothesis in class 𝐻 . Let D𝑚𝑖𝑥 be the mixed dataset as illustrated in Fig. 3 and D𝑝𝑟𝑖𝑣 be the private dataset. Let 𝑑 HΔH be the empirical distance induced by the symmetric difference hypothesis space. Then we can bound the difference between the empirical risks of the mixed dataset and the private dataset by 1 |𝜖 D𝑚𝑖𝑥 (ℎ) − 𝜖 D𝑝𝑟𝑖𝑣 (ℎ)| ≤ (1 − 𝛼)( 𝑑 HΔH (D𝑝𝑟𝑖𝑣 , D𝑝𝑢𝑏 ) + 𝜆), 2 where 𝜆 = 𝜖 D𝑝𝑟𝑖𝑣 (ℎ ∗ ) + 𝜖 D𝑝𝑢𝑏 (ℎ ∗ ) and ℎ ∗ is the ideal joint hypothesis minimizing the combined empirical risk: ℎ ∗ = 𝑎𝑟𝑔𝑚𝑖𝑛ℎ∈𝐻 𝜖 D𝑝𝑟𝑖𝑣 (ℎ) + 𝜖 D𝑝𝑢𝑏 (ℎ). Proof. The proof of Theorem 3.4 builds on the lemma 3 in [4], which demonstrates that for any hypotheses ℎ 1, ℎ 2 ∈ H , 1 |𝜖 D𝑝𝑟𝑖𝑣 (ℎ 1, ℎ 2 ) − 𝜖 D𝑝𝑢𝑏 (ℎ 1, ℎ 2 )| ≤ 𝑑 HΔH (D𝑝𝑟𝑖𝑣 , D𝑝𝑢𝑏 ), 2

(2)

and the triangle inequality for classification error [6], which demonstrates that for any hypothesis ℎ 1 ,ℎ 2 ,ℎ 3 ∈ H with respect to D, 𝜖 D (ℎ 1, ℎ 2 ) ≤ 𝜖 D (ℎ 1, ℎ 3 ) + 𝜖 D (ℎ 2, ℎ 3 ).

(3) Manuscript submitted to ACM

10

Sheng et al.

Then we can derive the bound of the difference between the empirical risk of the mixed dataset D𝑚𝑖𝑥 and the private dataset D𝑝𝑟𝑖𝑣 as |𝜖 D𝑚𝑖𝑥 (ℎ, 𝑓𝑝 ) − 𝜖 D𝑝𝑟𝑖𝑣 (ℎ, 𝑓𝑝 )| =(1 − 𝛼)|𝜖 D𝑝𝑟𝑖𝑣 (ℎ, 𝑓𝑝 ) − 𝜖 D𝑝𝑢𝑏 (ℎ, 𝑓𝑝 )| =(1−𝛼){|[𝜖 D𝑝𝑟𝑖𝑣 (ℎ, 𝑓𝑝 ) −𝜖 D𝑝𝑟𝑖𝑣 (ℎ, ℎ ∗ )] + [𝜖 D𝑝𝑢𝑏 (ℎ, ℎ ∗ )− 𝜖 D𝑝𝑢𝑏 (ℎ, 𝑓𝑝 )] + [𝜖 D𝑝𝑟𝑖𝑣 (ℎ, ℎ ∗ ) − 𝜖 D𝑝𝑢𝑏 (ℎ, ℎ ∗ )]|} ≤(1−𝛼) [|𝜖 D𝑝𝑟𝑖𝑣 (ℎ, 𝑓𝑝 ) − 𝜖 D𝑝𝑟𝑖𝑣 (ℎ, ℎ ∗ )|+|𝜖 D𝑝𝑢𝑏 (ℎ, 𝑓𝑝 )− 𝜖 D𝑝𝑢𝑏 (ℎ, ℎ ∗ )| + |𝜖 D𝑝𝑟𝑖𝑣 (ℎ, ℎ ∗ ) − 𝜖 D𝑝𝑢𝑏 (ℎ, ℎ ∗ )|] ∗

(4)

∗

≤(1 − 𝛼) [|𝜖 D𝑝𝑟𝑖𝑣 (ℎ , 𝑓𝑝 )| + |𝜖 D𝑝𝑢𝑏 (ℎ , 𝑓𝑝 )|+ |𝜖 D𝑝𝑟𝑖𝑣 (ℎ, ℎ ∗ ) − 𝜖 D𝑝𝑢𝑏 (ℎ, ℎ ∗ )|]

(5)

1 ≤(1 − 𝛼) ( 𝑑 HΔH (D𝑝𝑟𝑖𝑣 , D𝑝𝑢𝑏 ) + 𝜆). 2

(6)

In the proof, Equation (4) is derived from the absolute value inequality, Equation (5) is derived from the triangle inequality introduced in Equation (3) and the last step is derived by substituting Equation (2) and 𝜆 = 𝜖 D𝑝𝑟𝑖𝑣 (ℎ ∗ ) + 𝜖 D𝑝𝑢𝑏 (ℎ ∗ ). The attacking model 𝜃˜ is trained on mixed dataset and test on victim’s private dataset. According to Theorem 3.4, we establish that the bound of the privacy risk, quantified by the performance of 𝜃˜, is bounded by (1 − 𝛼) ( 12 𝑑 HΔH (D𝑝𝑟𝑖𝑣 , D𝑝𝑢𝑏 ) + 𝜆). When fixing D𝑝𝑟𝑖𝑣 and D𝑝𝑢𝑏 , this bound is exclusively related to weighting parameter 𝛼. Notably, the bound drops to 0 as 𝛼 increases to 1. This implies that, when 𝛼 increases, i.e. the mixed public data gets closer to the private data, the attacking model performs better on the private test dataset and more private information of the model, which is trained on its logits and the public dataset, is leaked. Here we briefly delve into the underlying cause of this privacy risk in logit-based FL. As depicted in Figure 2, we observe that local model training derives benefit from the knowledge contained in the ensemble logit, obtained through the aggregation of local predicted logits on the public data. While a more informative local logit contributes to a more informative ensemble logit, it concurrently exposes more privacy to the adversary, as illustrated in Theorem 3.4. Our observation aligns with empirical results, as demonstrated in Figure 4 in Section 5.3.

4

DEFENDING AGAINST THE PRIVACY RISK

Our observation in Section 3 shows that the privacy risk in logit-based FL mainly comes from the logit. In this section, we propose a defense strategy, named Federated Logit Perturbation (FedLP), that perturbs the transmitted logits of local models to defend against this privacy risk.

4.1

Design of FedLP

Defense Objective The defender (i.e. the clients) has two objectives. First, the defender aims to prevent an adversary from being able to replicate the functionality of its private model:   min𝜃˜ E𝑥∼D𝑝𝑟𝑖𝑣 𝐴𝑐𝑐 𝑓 (𝑥, 𝜃˜) , where 𝑓 (𝑥, 𝜃˜) denotes the functionality of the attacking model. Manuscript submitted to ACM

(7)

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

11

Second, the defender aims to preserve the training performance of the logit-based FL protocol so that the perturbation scale should be bounded by a non-negative constant 𝛾: ||𝑝 − 𝑝 ′ ||𝑏 ≤ 𝛾,

(8)

where 𝑝 is the original logit on the public data, 𝑝 ′ is the corresponding perturbed logit, 𝛾 > 0 is a pre-determined constant parameter and ||.||𝑏 denotes the 𝐿𝑏 norm. We note that the defender has no access to the adversary’s model and may be even unaware that it is under attack since the attack happens at the server side. Therefore, the defender has to prevent the privacy risk from the semi-honest server during the whole training process. Defense Problem Combining Equation (7) and (8), we can formulate a defense problem for the defender. However, this problem can not be directly solved since the attacking model parameters and its training details are unknown to the defender. Therefore, we need to approximate the first objective from the perspective of the defender. The first step is to estimate the attacking model 𝜃˜. As the goal of 𝜃˜ is to approximate the defender’s model 𝜃 in each training iteration, we estimate the attacking model obtained from the last iteration to be the same as the defender’s model 𝜃 in the current iteration 𝑇 : 𝜃˜𝑇′ −1 = 𝜃𝑇 −1, where 𝜃˜𝑇′ −1 and 𝜃𝑇 −1 are the estimated attacking model and the defender’s model in the last training iteration respectively. Without loss of generality, we assume that the adversary optimizes its attacking model through the gradient of an empirical loss on the public data, which is the most widely used optimization method in deep learning [30]. The gradient of an empirical loss with respect to parameter 𝜃 can be expressed as 𝐺 (𝜃, 𝑝) = ∇𝜃 L𝐶𝐸 (𝑓 (𝑥, 𝜃 ), 𝑝).

(9)

Based on the above assumptions, we restate the objectives of the defender as maximally changing the updated gradients of the estimated attacking model with minimum perturbation on the logit. That is, we can rewrite the first objective of the defender in iteration 𝑇 as maximizing the distance between the gradients of the estimated attacking model updated through the original logit and the perturbed logit, in terms of 𝐿𝑎 norm: max𝑝 ′ ||𝐺 (𝜃˜𝑇′ −1, 𝑝 ′ ) − 𝐺 (𝜃˜𝑇′ −1, 𝑝)||𝑎 =max𝑝 ′ ||𝐺 (𝜃𝑇 −1, 𝑝 ′ ) − 𝐺 (𝜃𝑇 −1, 𝑝)||𝑎

(10)

where 𝐺 (𝜃, 𝑝) is the gradient of the empirical loss with respect to the parameters 𝜃 . Combining Equation (10) and (8), we therefore reformulate the defense problem as a constrained optimization problem: max𝑝 ′ ||𝐺 (𝜃𝑇 −1, 𝑝 ′ ) − 𝐺 (𝜃𝑇 −1, 𝑝)||𝑎

(11)

s.t. ||𝑝 − 𝑝 ′ ||𝑏 ≤ 𝛾,

(12)

which allows the defender to trade off the utility and privacy in logit-based FL training. Worth mention that we can form multiple defense problems and corresponding defense strategies with different selections of (𝑎, 𝑏). In this paper, we set (𝑎, 𝑏) = (2, 1) as default and compare with other settings empirically in Section 5.4. Solution of FedLP Deep learning models usually involves millions of parameters and thus solving Equation (11) s.t. Equation (12) with respect to each sample in the public dataset requires a large computational cost for clients, which is Manuscript submitted to ACM

12

Sheng et al.

unaffordable for local devices in practice. Here, we give a simple heuristic solver to circumvent this computational issue. We perturb the logit 𝑝 on the public dataset in each dimension of itself by 𝑍 : 𝑝′ = 𝑝 + 𝑍 · 𝑒𝑗 , where 𝑒 𝑗 denotes a one-hot vector with 1 in the 𝑗-th dimension of 𝑝 and 0’s elsewhere. Then we select the one giving the largest perturbation in Equation (11). The local training with our defense in one iteration is given in algorithm 2. Note that Algorithm 2 is executed entirely on the client’s local machine except Step 12. In Step 12, only the perturbed output 𝑝 ′ is transmitted to the server. Moreover, 𝑝 ′ is carefully protected by the differential privacy mechanism before communication. We theoretically analyze the privacy of Algorithm 2 in Section 4.2 and empirically show that this simple solution is effective in Section 5.4. We provide a brief analysis of the computational complexity associated with FedLP. Since the dimension of 𝑝 is typically much smaller than |𝐷 𝑝𝑢𝑏 |, the computational complexity of FedLP, as outlined in Algorithm 2, is 𝑂 (|𝐷𝑝𝑢𝑏 |). Notably, our defense operates solely on the local client, meaning there is no additional burden on communication costs. Furthermore, we would like to clarify that increasing the security of FedLP (i.e., increasing 𝜎) does not result in higher computational or communication costs, as shown in Algorithm 2. Noise Selection and Privacy Guarantee The added noise 𝑍 has multiple choices, such as Laplace or Gaussian noise [1]. Following the prior works [8], we adopt 𝑍 as a Gaussian noise 𝑍 = N (0, 𝛾), where 𝛾 denotes the variance of √ the Gaussian noise. Let 𝜎 = 𝛾, which represents the corresponding standard deviation. A larger 𝛾 leads to stronger noise injection and therefore yields stronger privacy guarantees. We show that our proposed perturbation based defense strategy in Algorithm 2 preserves (𝛼, 𝜖)−Rényi differential privacy in section 4.2.

Algorithm 2: Local Training with FedLP in one iteration Input: Public dataset D𝑝𝑢𝑏 , local dataset D𝑝𝑟𝑖𝑣 : {𝑥, 𝑦}, local model in last iteration 𝜃𝑇 −1 , variance parameter 𝜎. Output: Updated local model 𝜃𝑇 . 1: Local Training: 2: Train local model with D𝑝𝑟𝑖𝑣 and update 𝜃 3: Logit Ensemble: 4: for 𝑥𝑖 in D𝑝𝑢𝑏 do 5: 𝑝𝑖 ← 𝜃𝑇 (𝑥𝑖 ) 6: for each dimension 𝑒 𝑗 of 𝑝𝑖 do 7: 𝑝𝑖′ ← 𝑝𝑖 + N (0, 𝜎 2 ) · 𝑒 𝑗 , for 𝑥𝑖 ∈ D𝑝𝑢𝑏 Í 8: Calculate ||𝐺𝑖 (𝜃𝑇 −1, 𝑝𝑖′ ) − 𝐺𝑖 (𝜃𝑇 −1, 𝑝𝑖 )|| 2 according to Equation (9) 𝑥𝑖 ∈ D𝑝𝑢𝑏

9: 10:

end for 𝑝 ′ ← argmax𝑝 ′

Í

||𝐺𝑖 (𝜃𝑇 −1, 𝑝𝑖 ) − 𝐺𝑖 (𝜃𝑇 −1, 𝑝𝑖′ )|| 2

𝑥𝑖 ∈ D𝑝𝑢𝑏

11: end for 12: Upload 𝑝 ′ to the server and then obtain the corresponding ensemble logits from the server 13: Distillation: 14: Train local model with ensemble logits on D𝑝𝑢𝑏 and update 𝜃 15: return 𝜃𝑇

Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning 4.2

13

Privacy Guarantee of FedLP

We first give definitions of differential privacy (DP) and 𝑙 2 -sensitivity [7, 8]. Then we show that our proposed perturbation based defense strategy in Algorithm 2 preserves (𝜖, 𝛿) − 𝐷𝑃. Definition 4.1. (Differential Privacy). A randomized mechanism 𝑓 : X → Y is (𝜖, 𝛿)-DP, if and only if for every pair of datasets 𝑋, 𝑋 ′ ∈ X that only differ in one sample and every possible output 𝐸 ⊆ 𝑟𝑎𝑛𝑔𝑒 (𝑓 ), the following inequality holds: P[𝑓 (𝑋 ) ∈ 𝐸] ≤ 𝑒 𝜀 P [𝑓 (𝑋 ′ ) ∈ 𝐸] + 𝛿. where 𝜖 > 0 represents the privacy budget, 𝛿 > 0 represents the probability that the maximum distance is not bounded by 𝜖 and 𝑟𝑎𝑛𝑔𝑒 (𝑓 ) denotes the set of all possible outputs of 𝑓 . Definition 4.2. (𝑙 2 −sensitivity). The 𝑙 2 -sensitivity of a function 𝑓 : X → R𝑑 is defined as ∥𝑓 (𝑋 ) − 𝑓 (𝑋 ′ )∥ 2 . Δ2 (𝑓 ) = max ′ 𝑋 ,𝑋 ∈ X

Theorem 4.3. For any 𝜖 > 0 and 𝛿 ∈ (0, 1), the mechanism described in Algorithm 2 with a sensitivity Δ2 preserves (𝜖, 𝛿) − 𝐷𝑃 if and only if 𝛾 ≥ 2𝑙𝑛1.25𝛿 · ( Δ𝜖2 ) 2 . Proof. The proof of Theorem 4.3 is based on the definition of Gaussian differential privacy and composition theorem of DP algorithms [8]. Theorem 4.4. (Gaussian Differential Privacy). Let 𝜖 ∈ (0, 1) be arbitrary. For 𝑐 2 > 2𝑙𝑛(1.25/𝛿), the Gaussian Mechanism with parameter 𝜎 ≥ 𝑐Δ2 (𝑓 ) is (𝜖, 𝛿)-differentially private. Theorem 4.5. (Composition of DP Algorithms). Suppose 𝑀 = (𝑀1, 𝑀2, ..., 𝑀𝑘 ) is a sequence of algorithms, where 𝑀𝑖 is Í𝑘 Í𝑘 𝑖=1 𝜖, 𝑖=1 𝛿)-DP.

(𝜖𝑖 , 𝛿𝑖 )-DP, and the 𝑀𝑖 ’s are potentially chosen sequentially and adaptively. Then 𝑀 is (

Theorem 4.6. (Privacy Guarantee of FedLP). If 𝑝 ′ = 𝑝 + 𝑍 · 𝑒 𝑗 , where 𝑍 is drawn from a Gaussian distribution and 𝑒 𝑗 is a unit vector indicating the direction of perturbation, then Our proposed FedLP in Algorithm 2 preserves (𝜖, 𝛿) − 𝐷𝑃. Proof. According to Theorem 4.4, when fixing privacy budget 𝜖𝑖 and 𝛿𝑖 , we can calibrate the added noise with proper 𝜎 according to the perturbation constraint scale 𝛾 for each client. The magnitude of 𝜎 remains unchanged and thus the perturbed output of each client still preserves (𝜖𝑖 , 𝛿𝑖 ) − 𝐷𝑃. Then, according to Theorem 4.5, the composition property Í Í of DP algorithms ensure that the ensemble result computed using the perturbed outputs is still ( 𝑘𝑖=1 𝜖, 𝑘𝑖=1 𝛿) private. Therefore, our proposed FedLP in Algorithm 2 preserves (𝜖, 𝛿) − 𝐷𝑃. Extension to Rényi Differential Privacy Analysis We extend our privacy analysis to include Rényi Differential Privacy (RDP) [27], which provides a more flexible and fine-grained framework for quantifying privacy guarantees compared to standard differential privacy. Unlike standard DP, RDP is defined independently and offers advantages in composition and subsampling analysis. Below, we present the definitions and theorems for RDP, as well as its application in our proposed method. Definition 4.7. (Rényi Differential Privacy). A randomized mechanism 𝑓 : X → Y satisfies (𝛼, 𝜖)-Rényi Differential Privacy (RDP) if for every pair of datasets 𝑋, 𝑋 ′ ∈ X differing in only one sample and for all 𝛼 > 1, the following inequality holds: 𝐷𝛼 (P[𝑓 (𝑋 ) ∈ ·] ∥ P[𝑓 (𝑋 ′ ) ∈ ·]) ≤ 𝜖, where 𝐷𝛼 denotes the Rényi divergence of order 𝛼. Manuscript submitted to ACM

14

Sheng et al. Theorem 4.8. For any 𝜖 > 0 and 𝛿 ∈ (0, 1), the mechanism described in Algorithm 2 with a sensitivity Δ2 preserves

2 (𝛼, 𝜖)-RDP if and only if 𝛾 ≥ 2𝛼 𝜖 [Δ2 (𝑓 )] .

Proof of Theorem 4.8. The proof of Theorem 4.8 is similar to the proof of Theorem 4.3, based on the definition of Gaussian differential privacy under RDP and the advanced composition theorem for RDP [27]. √︃ Theorem 4.9. (Gaussian Mechanism under RDP). For any 𝛼 > 1, the Gaussian Mechanism with parameter 𝜎 ≥ 2𝛼 𝜖 Δ2 (𝑓 ) ensures (𝛼, 𝜖)-Rényi Differential Privacy. Theorem 4.10. (Advanced Composition for RDP). Suppose 𝑀 = (𝑀1, 𝑀2, ..., 𝑀𝑘 ) is a sequence of algorithms where each Í𝑘 𝑖=1 𝜖𝑖 )-Rényi differentially private for any

𝑀𝑖 is (𝛼𝑖 , 𝜖𝑖 )-Rényi differentially private. Then the overall mechanism 𝑀 is (𝛼, 𝛼 ≥ max𝑖 𝛼𝑖 .

According to Theorem 4.9, fixing the privacy parameters 𝛼 and 𝜖, we can calibrate the added noise with proper 𝜎 2 according to the perturbation constraint scale 𝛾 for each client. Specifically, setting 𝛾 = 𝜎 2 ≥ 2𝛼 𝜖 [Δ2 (𝑓 )] ensures that

each individual query preserves (𝛼, 𝜖)-RDP. Then, by applying Theorem 4.10, the composition property of RDP algorithms guarantees that the ensemble result Í𝑘 𝑖=1 𝜖𝑖 )-Rényi differentially private. Therefore, our proposed FedLP

computed using the perturbed outputs remains (𝛼,

in Algorithm 2 preserves (𝛼, 𝜖) − 𝑅𝐷𝑃 for an appropriate choice of 𝛼. 5

EXPERIMENT

In this section, we empirically validate the effectiveness of AdaMSA against logit-based FL and FedLP against AdaMSA. Also, we assess various factors influencing the attack and defense performance. Specifically, we aim to answer two main questions through our experiments: • RQ1 Attack Evaluation: Is the proposed AdaMSA effective against logit-based FL? • RQ2 Defense Evaluation: Is the proposed FedLP effective against AdaMSA, i.e. can it achieve a better utility and privacy trade-off? 5.1

Experimental Settings

To evaluate our proposed attack and defense, we construct three experimental settings for the image classification task: • Close-world: Following the same experimental setting in Fedmd [22], we first construct a simple Close-world setting that public data and private data are similar. We use MNIST1 [21] without labels as the unlabeled public dataset and EMNIST2 [5] as the private dataset. We randomly select 300 images (30 for each class) in EMNIST as the private training dataset and then identically distributed to 10 clients. The rest of the private dataset (EMNIST) is used as the private test dataset. • Open-world-CF: To construct a cross-domain experimental setting, we first discard the labels in CIFAR103 [17] and utilize it as the unlabeled public dataset. We use SVHN4 [29] as the private dataset, which comprises house numbers from Google Street View images, and is unrelated to the images in CIFAR-10. We randomly select 1000 images in each class of SVHN as the private training dataset and identically distributed to 10 clients. The rest of the private dataset (SVHN) is used as the private test dataset. 1 https://yann.lecun.com/exdb/mnist/ 2 https://www.nist.gov/itl/products-and-services/emnist-dataset 3 https://www.cs.toronto.edu/ kriz/cifar.html 4 http://ufldl.stanford.edu/housenumbers/

Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

15

• Open-world-TI: Similar to Open-world-CF, we select the first 20 classes in TinyImagenet5 [19] with 500 images per class without labels as the unlabeled public dataset and SVHN [29] as the private dataset. Then we randomly select 1000 images in each class of SVHN as the private training dataset and identically distributed to 10 clients. The rest of the private dataset (SVHN) is used as the private test dataset. Implementation We adopt CNN [20] as backbones for all clients’ models. We use a 2-layer CNN with (128,256) parameters as the attacking model and a CNN with (128,256) parameters initialized from a different random seed as the victim model to report the main result in Table 4. All models are optimized by Adam with a 0.01 learning rate. For a fair comparison, we set the perturbation scale 𝛾 to be the same as 0.01 for DP-G, DP-L and our defense in Table 4 and 𝑇0, 𝑤 0 is set to be 3 and 0.5 respectively. For Close-world setting, the batch size of local training on private data is 10 and on public data is 128. The number of communication rounds is 10. For Open-world settings, the batch size of local training on the private dataset is 32 and on the public dataset is 256. The number of communication rounds is 30. We repeat each experiment three times and report the average accuracy as the results. Baselines We compare our proposed attack against MSA [36] and Naïve AdaMSA (𝑇0 = 0) baselines in three experimental settings. For MSA baseline [36], we let the attacking model learn from the victim’s logit in the final iteration. For Naïve AdaMSA baseline, we train an attacking model by approximating the victim’s current logit in each round and choose the attacking model with highest accuracy. For defense evaluation, we compare our proposed FedLP to five state-of-the-art baseline defenses: 1) Unprotected [22]: We follow the training process of the general logit-based FL approach [22]. For Unprotected defense, the public and private data are drawn from the same dataset, i.e., we set 𝛼 = 1 for the mixed dataset. Specifically, we randomly select a subset of the private test dataset without labels to serve as the public dataset, with the remaining portion being used for testing. No defense mechanisms are applied in this setting; 2) Cross-domain [24]: In Cross-domain defense, the public dataset is selected from another irrelevant and insensitive domain to prevent privacy leakage (i.e. we set 𝛼 = 0 for the mixed dataset); 3) One-shot [23]: In One-shot defense, we utilize one-shot distillation on unlabeled and domain-robust public data. Specifically, clients only communicate once with server. 4) Differential Privacy (DP): Recent works [11, 33] adopt Gaussian and Laplacian DP and add noise to the transmitted updates. We conduct their strategies on the unprotected baseline as DP-G [33] and DP-L defense [11], respectively. Evaluation Metric We gauge the attack performance by assessing the prediction accuracy (Acc) of the attacking model on the victim’s private test dataset. A greater Acc value indicates a more formidable attacking model. For defense evaluation, we evaluate all defenses on a utility loss vs. privacy loss curve at various points of the defenses. The utility loss Δ𝑈 of the defense is defined as Δ𝑈 = 𝑈𝑑 − 𝑈 0

(13)

where 𝑈 0 is the accuracy of local model in unprotected baseline and 𝑈𝑑 is the accuracy of local model under different defense. The privacy loss is defined as the prediction accuracy of the attacking model on the defender’s private test dataset. 5.2

Attack Performance Evaluation

Main Results Table 4 shows the performance of MSA, Naïve AdaMSA and AdaMSA in three settings. We observe that AdaMSA demonstrates significant improvements of up to 4.00%, 3.41%, and 3.11% in the respective settings, compared 5 https://www.kaggle.com/c/tiny-imagenet/data

Manuscript submitted to ACM

16

Sheng et al.

Table 4. Attack performance of AdaMSA and two attack baselines on the victim model with various defense baselines in Close-world, Open-world-CF and Open-world-TI settings. Victim denotes the performance of the victim model.

Setting

Close-world

Open-world-CF

Open-world-TI

Defense Unprotected [22] Cross-domain [24] One-shot [10] DP-G [33] DP-L [11] Unprotected [22] Cross-domain [24] One-shot [10] DP-G [33] DP-L [11] Unprotected [22] Cross-domain [24] One-shot [10] DP-G [33] DP-L [11]

Victim (%) 83.43 ± 1.07 82.97 ± 0.89 68.23 ± 1.21 74.68 ± 2.15 75.65 ± 1.97 71.11 ± 0.98 53.57 ± 1.13 63.41 ± 1.09 65.89 ± 1.77 65.22 ± 1.96 71.11 ± 0.98 51.14 ± 1.10 63.09 ± 1.14 65.76 ± 2.01 65.09 ± 1.98

MSA[36] (%) 80.10 ± 1.31 79.43 ± 1.07 65.03 ± 0.89 71.69 ± 1.14 72.01 ± 1.21 68.77 ± 1.13 52.21 ± 1.06 60.03 ± 0.98 62.20 ± 1.39 65.97 ± 1.28 69.34 ± 0.97 51.17 ± 1.05 62.36 ± 1.03 62.77 ± 1.20 66.11 ± 1.13

Naïve AdaMSA (%) 80.09 ± 1.35 79.12 ± 1.30 66.15 ± 0.73 72.07 ± 2.32 71.53 ± 1.44 68.99 ± 1.02 52.34 ± 1.47 59.66 ± 0.99 62.33 ± 1.56 66.07 ± 1.25 69.68 ± 1.01 50.43 ± 1.17 62.09 ± 0.86 63.01 ± 1.33 66.32 ± 1.51

AdaMSA(𝑇0 =3) (%) 83.79 ± 1.20 82.68 ± 1.01 67.18 ± 1.17 74.79 ± 1.20 75.53 ± 1.09 71.25 ± 0.71 53.78 ± 0.79 62.49 ± 0.91 65.61 ± 1.31 67.64 ± 1.22 72.13 ± 1.02 53.54 ± 0.80 64.41 ± 0.99 65.02 ± 1.18 68.41 ± 1.04

to Naïve AdaMSA and MSA baselines. This highlights the effectiveness of our attack design, utilizing historical logits to generate a more informative target logit, and thereby enhance the attack performance. Moreover, it is observed that AdaMSA consistently matches the performance of the victim model across various defense baselines in three settings. This reaffirms its capability to successfully steal the functionality of the victim’s private model. It is worth mentioning that we observed the performance of the attacking model surpassing that of the victim model in certain baselines. The surprising phenomenon can be attributed to the fact that the training process of the attacking model can be viewed as a form of self-distillation, which has been demonstrated to improve the model’s generalization ability on test data [3]. Under Limited Resource Assumption We would like to clarify that we follow the general assumption in FL that the server is powerful and has virtually unlimited computational resources. Under these conditions, AdaMSA(𝑇0 = 3) is the preferred choice. However, if the server’s computational resources are limited, the Naïve AdaMSA may offer a better balance between performance and efficiency, as AdaMSA(𝑇0 = 3) requires three times the computational resources. Table 5. The effect of combining historical logits on the attack performance of AdaMSA.

𝑇0 0 1 2 3 4

5.3

AdaMSA (%) Open-world-CF Open-world-TI 68.99 ± 1.02 69.68 ± 1.01 70.11 ± 0.94 70.51 ± 1.16 70.40 ± 1.05 71.88 ± 1.15 71.25 ± 0.71 72.13 ± 1.02 71.09 ± 1.46 71.88 ± 1.35

Ablation Studies

In this section, we conduct ablation analysis of the attack performance on the impact of the distance between private and public datasets, combining historical logits, data Heterogeneity and model architectures. Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

17

Table 6. The effect of data heterogeneity on the attack performance of AdaMSA in Open-world-CF setting.

Defense Unprotected [22] Cross-domain [24] One-shot [10] DP-G [33] DP-L [11]

𝛽 = 0.5 Victim (%) AdaMSA (%) 63.99 ± 0.85 64.02 ± 1.16 47.30 ± 0.98 47.57 ± 1.12 55.93 ± 1.34 55.97 ± 1.04 57.21 ± 0.49 56.88 ± 0.95 57.63 ± 1.40 57.54 ± 1.51

𝛽 =1 Victim (%) AdaMSA (%) 66.32 ± 0.79 66.26 ± 1.05 49.04 ±1.83 49.11 ± 1.64 57.30 ± 1.46 57.42 ± 1.02 59.48 ± 1.55 60.01 ± 0.87 59.38 ± 2.12 59.90 ± 1.52

𝛽 = 10 Victim (%) AdaMSA (%) 70.36 ± 1.25 69.79 ± 1.04 54.01 ± 0.99 54.03 ± 1.33 62.56 ± 1.23 62.66 ± 1.02 63.75 ± 1.50 64.11 ± 1.39 63.60 ± 1.82 63.59 ± 1.71

𝛽 = 100 Victim (%) AdaMSA (%) 71.09 ± 1.21 71.21 ± 0.99 54.24 ± 1.92 54.20 ± 1.65 63.06 ± 0.55 63.25 ± 0.97 65.88 ± 1.43 66.89 ± 1.03 65.59 ± 1.96 65.99 ± 1.97

Effect of the Distance Between Private and Public Datasets To evaluate the effect of this factor, we construct several mixed datasets (see section 3.3 for detailed illustration) in two Open-world settings as the public datasets through varying the weighting parameter 𝛼. The results are reported in Figure 4. It can be observed that increasing the value of 𝛼 (i.e. decrease the distance), will increase the utility at the cost of increasing privacy risk. Furthermore, this inherent privacy risk persists even when public data is unrelated to private data. This result indicates that the utility and privacy in logit-based FL are indeed two sides of a coin. That is, a more informative local logit results in a more informative ensemble logit to supervise the local model training, meanwhile it also exposes more privacy to the adversary. This result aligns with our analysis in Section 3.3.

Fig. 4. The relation between the utility/privacy risk and 𝛼 in Open-world-CF (left) and Open-world-TI (right) settings. Privacy risk is measured by the attack performance of AdaMSA.

Effect of Combining Historical Logits To evaluate the effect of combining historical logits, we vary 𝑇0 and test on the same victim model in two open-world settings. From the results in Table 5, we observe that, when 𝑇0 increases from 0 to 3, the attack performance gradually increases by 2.87% in Open-world-CF setting and 2.45% in Open-world-TI setting, demonstrating that combining more historical logits indeed improves the attack performance. The rationale behind this improvement is that the historical predictions close to the current round can be viewed as the different views of the victim model on the public data. Therefore, our designed target logit can be benefited from the ensemble of these muti-view predictions, improving model generalization ability on the test data [2]. However, when 𝑇0 increases to 4, the performance drops because the predictions in the early rounds may not be sufficiently trained and, instead of boosting, can act as a detriment. Effect of Data Heterogeneity To evaluate the effect of data heterogeneity, we split the training data according to a Dirichlet distribution following [13] in Openworld-CF setting. The non-iid level of data is controlled by the Dirichlet parameter 𝛽. From the result reported in Table 6, we observe that: 1) increasing the non-iid level (i.e. decrease 𝛽) will Manuscript submitted to ACM

18

Sheng et al.

decrease the utility of local model as well as the attacking model performance; 2) AdaMSA achieves similar performance compared to the victim model, indicating AdaMSA is still effective in the non-iid setting. Effect of Model Heterogeneity We vary the victim model parameters and architectures in the Open-world-CF setting to evaluate the effect of victim model heterogeneity on the attack performance. The results are reported in Table 7. We also find a similar result in the Open-world-TI setting, which is omitted due to the space limit. For a fair comparison, we perform the experiments in the unprotected baseline. The victim models vary from CNN to Alexnet [18] with different hyperparameters. The details are reported in Table 7. The results show that AdaMSA can achieve similar accuracy of the victim models under different model hyperparameters. This indicates that AdaMSA is robust to heterogeneous victim models in our setting. Table 7. The effect of model heterogeneity on the attack performance of AdaMSA in Open-world-CF setting.

Model CNN CNN CNN CNN CNN Alexnet [18]

5.4

Layer 2 2 2 3 3 8

Parameter 128,256 128,512 256,512 64,128,256 128,192,256 -

Victim Acc(%) 71.11 ± 1.21 70.15 ± 1.14 70.56 ± 1.30 71.11 ± 0.98 69.64 ± 1.28 75.10 ± 1.29

Attack Acc(%) 69.33 ± 0.99 70.28 ± 1.02 70.42 ± 0.79 71.25 ± 0.71 69.19 ± 1.13 75.33 ± 1.24

Defense Performance Evaluation

(a) Close-world

(b) Open-world-CF

(c) Open-world-TI

Fig. 5. Defense performance evaluation in Close-world, Open-world-CF and Open-world-TI settings. The ideal trade-off curve resides on the bottom left corner in the figure.

The results of the state-of-the-art defense strategies in logit-based FL and our proposed FedLP (denoted as Ours in the figure) on three settings are reported in Figure 5. The X-axis represents the privacy loss, i.e. the capability for adversarial to infer a client’s private model. Y-axis represents the utility loss brought by the defense methods. Comparing to the state-of-the-art baselines, our proposed FedLP defense is closest to the ideal trade-off, which should reside in the bottom left corner in Figure 5. For example, when privacy loss is 0.7, the utility loss of our defense is around 8% less than DP-G and DP-L in Close-world setting. This result indicates that our defense can provide a better utility and privacy trade-off compared to the state-of-the-art defense baselines. Manuscript submitted to ACM

Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

19

Table 8. The effect of hyperparameter (a,b) on the defense performance of AdaMSA in Open-world-CF and Open-world-TI setting.

Defense (2,1) (2,2)

Open-world-CF Privacy Loss Utility Loss(%) 0.61 12.11 ± 2.44 0.61 12.05 ± 2.50

Open-world-TI Privacy Loss Utility Loss(%) 0.61 11.88 ± 2.12 0.61 11.93 ± 2.49

Effect of Hyperparameter (a,b) We vary the values of the hyperparameters (𝑎, 𝑏) in the defense strategy under the Open-world-CF and Open-world-TI settings to evaluate their effect on defense performance. The results are presented in Table 8. For clearer comparison, we fix the privacy loss at approximately 0.61 and compare the corresponding utility loss. The results indicate that FedLP demonstrates robustness to the hyperparameters (𝑎, 𝑏) in our settings. 6

CONCLUSION

In this paper, we provide the first theoretical and empirical analysis of the privacy risk in logit-based FL that the semi-honest server can infer clients’ private models according to logits. To quantify the impacts of the privacy risk, we introduce AdaMSA, a method leveraging historical logits during training to enhance attack performance, and provide a theoretical analysis of the associated privacy risk bound. Moreover, we propose a perturbation-based defense named FedLP that perturbs the transmitted logit in the direction that minimizes the privacy risk while maximally preserving the training performance. Our experimental results showcase the effectiveness of AdaMSA and FedLP in various scenarios. REFERENCES [1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 308–318. [2] Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 (2020). [3] Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=Uuf2q9TfXGA [4] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. 2007. Learning bounds for domain adaptation. NeurIPS 20 (2007). [5] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In IJCNN. IEEE, 2921–2926. [6] Koby Crammer, Michael Kearns, and Jennifer Wortman. 2008. Learning from Multiple Sources. JMLR 9, 8 (2008). [7] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference. Springer, 265–284. [8] Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9, 3–4 (2014), 211–407. [9] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020. Inverting gradients-how easy is it to break privacy in federated learning? NeurIPS 33 (2020), 16937–16947. [10] Xuan Gong et al. 2021. Ensemble attention distillation for privacy-preserving federated learning. In ICCV. 15076–15086. [11] Xuan Gong et al. 2022. Preserving privacy in federated learning with ensemble cross-domain knowledge distillation. In AAAI, Vol. 36. 11891–11899. [12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015). [13] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019). [14] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2018. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479 (2018). [15] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, et al. 2019. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019). [16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. In ICML (PMLR, Vol. 119). PMLR, 5132–5143. https://proceedings.mlr.press/v119/karimireddy20a.html Manuscript submitted to ACM

20

Sheng et al.

[17] Alex Krizhevsky et al. 2009. Learning multiple layers of features from tiny images. (2009). [18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90. [19] Ya Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge. CS 231N 7, 7 (2015), 3. [20] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. 1989. Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems 2 (1989). [21] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. [22] Daliang Li and Junpu Wang. 2019. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019). [23] Qinbin Li, Bingsheng He, and Dawn Song. 2020. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXiv:2010.01017 1, 3 (2020). [24] Tao Lin et al. 2020. Ensemble distillation for robust model fusion in federated learning. NeurIPS 33 (2020), 2351–2363. [25] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282. [26] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. 2019. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP). IEEE, 691–706. [27] Ilya Mironov. 2017. Rényi Differential Privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF). 263–275. https://doi.org/10.1109/ CSF.2017.11 [28] Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP). IEEE, 739–753. [29] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. 2011. Reading digits in natural images with unsupervised feature learning. (2011). [30] Kyrylo Oliinyk. 2020. First-Order Optimization (Training) Algorithms in Deep Learning. (2020). [31] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. 2019. Knockoff nets: Stealing functionality of black-box models. In CVPR. 4954–4963. [32] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017. Practical black-box attacks against machine learning. In ACM ASIACCS. 506–519. [33] Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. 2021. Fedaux: Leveraging unlabeled auxiliary data in federated learning. IEEE TNNLS (2021). [34] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership Inference Attacks Against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy (SP). 3–18. https://doi.org/10.1109/SP.2017.41 [35] Hideaki Takahashi, Jingjing Liu, and Yang Liu. 2023. Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack. In CVPR. 12198–12207. [36] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. 2016. Stealing Machine Learning Models via Prediction APIs.. In USENIX security symposium, Vol. 16. 601–618. [37] Zhibo Wang et al. 2019. Beyond inferring class representatives: User-level privacy leakage from federated learning. In IEEE INFOCOM. IEEE, 2512–2520. [38] Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep leakage from gradients. NeurIPS 32 (2019).

Received 20 May 2024

Manuscript submitted to ACM