arXiv CSopen access

Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Yuhua Wang1 , Qinnan Zhang1* , Xiaodong Li2 , Huan Zhang1 , Yifan Sun2* , Wangjie Qiu1 , Hainan Zhang1 , Yongxin Tong3 , Zhiming Zheng1 1 School of Artificial Intelligence, Beihang University 2 School of Statistics, Renmin University of China 3 School of Computer Science and Engineering, Beihang University {yuhuawang, zhangqn}@buaa.edu.cn; [email protected]

1. Introduction Scaling laws for foundation models have been established on massive public datasets [1]. As general pre-training * Corresponding Authors. This work was supported by Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing.

Private data

Backbone

Adapter Global Protos Generation

Private data

Backbone

Adapter

Classifier

...

Prototype-based Personalized Federated Learning (ProtoPFL) enables efficient multi-domain adaptation by communicating compact class prototypes, but directly sharing them poses privacy risks. A common defense involves perexample ℓ2 clipping before prototype computation to bound sensitivity, followed by isotropic Gaussian noise to enforce Local Differential Privacy (LDP). However, Isotropic Gaussian Prototype Perturbation (IGPP) typically over-perturbs discriminative dimensions and struggles to balance the clipping threshold with representation fidelity. In this paper, we propose VPDR, a client-side privacy plug-in that seamlessly integrates into existing ProtoPFLs. Motivated by the observation that dimension-wise class variance reflects discriminability, we introduce Variance-adaptive Prototype Perturbation (VPP), which allocates less noise to discriminative subspaces, preserving semantic separability while ensuring privacy. We further develop Distillation-guided Clipping Regularization (DCR), which enables feature norms to adaptively concentrate near the predefined clipping threshold while maintaining prediction consistency. Theoretical analysis shows that our groupwise mechanism provides privacy guarantees no weaker than the isotropic baseline under the same privacy constraints. Extensive experiments on multi-domain benchmarks demonstrate that VPDR achieves a superior privacy-utility trade-off, outperforming IGPP in personalized federated fine-tuning without sacrificing robustness against realistic attacks.

Server

Client1

(a) ProtoPFL with IGPP. Isotropic Noise

Large Clipping Threshold

Adaptive Noise Magnitude

Abstract

Excessive noise !

Feature dimension

Small Clipping Threshold Shift !

Class 1

Class 2

Gaussian noise

(b) Noise vs. Discriminability.

Clipping distortion !

Magnitude

arXiv:2604.27833v1 [cs.CV] 30 Apr 2026

Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning

Feature dimension

Figure 1. Motivation illustration. In (b), the dashed circle shows uniform noise and the dashed ellipse our adaptive noise. Isotropic noise shifts the blue Class 1 prototype into the red Class 2 region. In (c), a generous clipping bound leaves feature norms almost unchanged but forces a large noise scale, while an aggressive bound shrinks many features, causing severe information loss.

has largely consumed available public data [39], leveraging private data becomes essential to overcome the emerging data-scarcity bottleneck [25]. Federated Learning (FL) [2, 19, 23, 31] offers a distributed paradigm for collaboratively fine-tuning pre-trained models without sharing raw data [50, 56]. Under domain skew, however, a single global model often underperforms or even fails to converge across clients [4, 40, 45]. To solve this, Personalized FL (PFL) [10, 21, 34, 47] adapts shared models with client-specific components [22, 34, 35, 49, 51]. In PFL, prototype-based methods [11, 15, 36, 42, 52, 53] exchange compact class statistics such as class means or centroids to align global and local semantics, enabling lightweight personalization while maintaining frozen backbones.

Despite their effectiveness, directly sharing prototypes poses significant privacy risks [43]. Since prototypes condense features into domain fingerprints, they can closely resemble individual instances under small sample sizes, opening the door to membership inference and reconstruction attacks [7, 27, 55]. A standard defense is to enforce Local Differential Privacy (LDP) [46]. In practice, clients first perform per-example ℓ2 clipping before prototype computation to bound sensitivity, then add calibrated isotropic Gaussian noise when transmitting prototypes, a recipe we call Isotropic Gaussian Prototype Perturbation (IGPP). However, as illustrated in Figure 1, IGPP suffers from two core drawbacks. 1 Noise and discriminability mismatch. Feature dimensions contribute unevenly to classification, with some carrying critical discriminability while others are largely redundant. Isotropic noise blindly over-perturbs the most informative dimensions and degrades class separability. 2 ℓ2 clipping-threshold dilemma. Prototype scales vary based on class size and domain characteristics. A larger threshold reduces clipping distortion but demands massive noise injection, whereas a smaller threshold forces excessive shrinkage, irrevocably erasing semantic content. Driven by these challenges, we introduce two modules that jointly account for prototype geometry and privacy. First, to address 1 , we derive dimension-wise discriminative scores from the intra- and inter-class variability of sample embeddings and use them to privately select a discriminative subspace. We then perform groupwise clipping on each embedding, construct prototypes from the clipped representations, and inject group-specific noise to reallocate the privacy budget toward task-relevant coordinates for better representation quality. Second, to tackle 2 , we append a differentiable soft-clipping layer to the local encoder and enforce prediction consistency between pre- and postclipped features via knowledge distillation. This guidance breaks the norm–weight compensation shortcut and drives feature norms to concentrate near the clipping threshold, reducing information loss under the same privacy budget. In this paper, we propose a client-side privacy plugin named VPDR, which comprises Variance-adaptive Prototype Perturbation (VPP) and Distillation-guided Clipping Regularization (DCR). We leverage VPP to steer massive noise away from discriminative dimensions, preserving task-relevant structure while avoiding over-perturbation. We also employ DCR, designing a consistency regularizer that mitigates clipping-induced artifacts while keeping the baseline task loss unchanged. This synergy enables ProtoPFL frameworks equipped with VPDR to achieve more robust personalization than IGPP-based baselines. Formal analysis and systematic evaluations under two privacy attacks further show that VPDR provides LDP guarantees no weaker than those of IGPP while achieving a superior privacy–utility trade-off. Our main contributions are:

• We fuse intra- and inter-class variance to quantify perdimension discriminability, yielding, to our knowledge, the first dimension-wise adaptive protection mechanism for prototype release with LDP guarantees. • We design a distillation-guided regularization that couples soft clipping with distillation consistency, suppressing clipping artifacts while enhancing both robustness and representation fidelity. • Integrated into existing ProtoPFLs, VPDR yields consistent gains on three multi-domain benchmarks, with ablations and privacy attacks supporting the effectiveness of both VPP and DCR. Code is available at https: //github.com/yuCoryx/ProtoPFL_VPDR.

2. Related Work Differential Privacy in FL. A large body of work studies DP for gradients or model updates in standard FL [3, 5, 12, 32, 48]. DP-FedAvg [12] applies server-side Gaussian noise to achieve client-level central DP, while BLUS+LUS [5] and DP-FedSAM [32] reduce the accuracy degradation via gradient sparsification and flatness-aware training. Extensions to PFL include PPSGD [3] and FedDPA [48] adaptively mix or regularize personalized parameters under joint DP guarantees. To guard against untrusted servers, LDP mechanisms have also been introduced [6, 14, 37, 41, 46]. UDP-FL [46] perturbs each user’s update with calibrated Gaussian noise, and later schemes such as ACS-FL [14], FedFR-ADP [41], and ALDP-FL [6] refine this idea with adaptive clipping, noise scaling, and budget control, while all providing LDP guarantees for each client. Prototype-based PFL. ProtoPFL uses class prototypes to align feature spaces across clients. FedProto [36] mitigates heterogeneity by exchanging prototypes, and FedPCL [11] improves semantic consistency through prototype-level contrastive learning. FPL [15], FedGMKD [54], and FedPLVM [42] integrate clustering and sparsity regularization to enhance robustness and generalization. More recent methods, including FedTGP [53], FedKTL [52], and MPFT [55], leverage server-side training and knowledge transfer to reinforce global semantics. Together, these advances shift ProtoPFL from static prototype sharing toward dynamic semantic alignment, improving personalization across clients. Despite growing interest in privacy for prototype release, existing defenses [11, 42, 55] often inject isotropic Gaussian noise into prototypes—typically without explicit per-example clipping or a full local DP accounting. As a result, they lack formal DP guarantees and overlook task-relevant prototype geometry, thereby degrading representation quality. We present the first LDP mechanism for ProtoPFL that combines groupwise clipping with adaptive noise allocation, offering privacy guarantees no weaker than isotropic perturbation and a better privacy–utility trade-off.

Client1

( x i , yi )  1

m

hh11

CTR Clip

t i

s i

DCR

G

KD

CE

④

1

DCR

KD

④ ③

Eq.(3)

① Private Prototype Calculation (inference)

1

VPP for ClientM

Server

⑤ Local Fine-Tuning

CE

t j

s j

⑤ Local Fine-Tuning

t M

VPP

 m} {

Private Partition z1, j ( j  [  B ]) z1, j ( j  [  A ])

Mc

z 2 ... z n

Groupwise Clipping Eq.(11)

M 



Clip

① Private Prototype Calculation (inference)

1 

②

( x j , y j )  M

CTR fM

ClientM

②

M

z 2 ... z n

Eq.(2)

M

Prototype Noise Addition p ( j  [  ]) B 1, j p ( j  [  ]) 1, j A

Eq.(13)

 p 1

m1

 ... p K mc

 cM 

Figure 2. Architecture illustration of the ProtoPFL with VPDR. ① Private Prototype Calculation: each client runs VPP (Section 4.2) to privately partition embeddings, apply groupwise clipping, and add adaptive noise to prototypes; ② Upload: clients send privatized prototypes; ③ Global Prototype Generation: the server aggregates or trains global prototypes; ④ Download: clients receive global prototypes; ⑤ Local Fine-Tuning: each client fine-tunes locally with DCR (Section 4.3) for feature-norm stabilization alongside the task loss.

3. Preliminaries

4. Methodology

3.1. Local Differential Privacy

4.1. Problem Statement and Motivation

We adopt client-side Local Differential Privacy (LDP) on each client’s dataset [37, 44, 46] in this work, which protects individual training examples from an untrusted server.

Personalized Federated Fine-Tuning. A federated system has M clients, each client m ∈ [M ] with dataset Dm of size nm drawn from a domain-shifted distribution Pm (x | y). PFL seeks a client-specific model Fm that minimizes: minFm E(x,y)∼Dm L Fm (x), y , (1)

Definition 3.1 ((ϵ, δ)–LDP [37, 46]). A randomized mechanism M : X → R satisfies (ϵ, δ)–LDP if for any adjacent D and D′ differing by one sample and measurable S ⊆ R, Pr[M(D) ∈ S] ≤ eϵ Pr[M(D′ ) ∈ S] + δ, where ϵ bounds the privacy loss and δ ∈ [0, 1) allows a failure probability. Definition 3.2 (Sensitivity [46]). Given a function f , the ℓp sensitivity of f is defined as ∆f = maxD∼D′ ∥f (D) − f (D′ )∥p , where ∥ · ∥p is the ℓ1 or ℓ2 norm. Definition 3.3 (Gaussian mechanism [46]). For ℓ2 sensitivity ∆f , the Gaussian mechanism releases M(D) = f (D) + N 0, (σ∆f )2 Id , which satisfies (ϵ, δ)–LDP for a suitable noise multiplier σ > 0. Theorem 3.1 (Noise calibration [46]). In T -round FL, sup pose each client m releases sem = sm + N 0, (σ∆s )2 Id , where sm = f (Dm ) has ℓ2 sensitivityp∆s . Then there exists a constant c1 > 0 such that σ ≥ c1 T ln(1/δ)/ϵ ensures (ϵ, δ)–LDP over T rounds for each client m.

3.2. Oneshot Laplace Mechanism The oneshot Laplace mechanism selects Top-k coordinates by adding Laplace noise once and ranking the perturbed scores, ensuring DP without iterative peeling [28]. Definition 3.4 (Oneshot Laplace for Top-k [28]). Given queries f1 , . . . , fd on the dataset D with common ℓ1 sensitivity ∆f , draw bi ∼ Lap(λ) and set yi = fi (D) + bi . The mechanism then outputs the indices of the k largest yi ’s. Theorem 3.2 (Oneshot Laplace Privacy [28]). If λ ≥ 2k ∆f /ϵ, then the oneshot Laplace mechanism is (ϵ, 0)–DP.

where L(·, ·) denotes the local loss. We write Fm = fm ◦ hm , with a frozen backbone g and a trainable adapter Am in feature extractor hm , followed by a classifier fm . Prototype-based Personalization. Each client maps private samples to d-dimensional embeddings via hm , yieldc c ing class-wise sets Zm = {zci = hm (xci ) | xci ∈ Dm }. For class c, client m calculates a class-wise prototype set: c c,k Km c c {pc,k (2) m } ← Calculate {zi } , Pm = pm k=1 , c is the number of local prototypes for class c on where Km c client m. Km = 1 recovers a single class mean [11, 15, 36, c 53], while Km > 1 yields multiple local centroids [42, 54, c M 55]. The server aggregates {Pm }m=1 and generates: Kgc c,k M c {pg } ← Generate ∪m=1 Pm , Pgc = pc,k , (3) g k=1

where Kgc is the number of global prototypes per class. Kgc = 1 denotes a global prototype obtained either by averaging [11, 36] or by training from client prototypes [53], and Kgc > 1 corresponds to clustering into multiple global centroids [15, 42]. The global set Pg = Pgc is then broadcast and used as global anchors in existing contrastive or classification objectives for multi-domain alignment. Isotropic Gaussian Prototype Perturbation (IGPP). To achieve client-side LDP, each client (i) applies per-example ℓ2 clipping before any averaging or clustering, zci = zci · min 1, R/∥zci ∥2 , (4)

(ii) compute prototypes from {zci }, and (iii) privatize each prototype release by adding isotropic Gaussian noise: e = p + ξ, ξ ∼ N 0, (σiso ∆)2 Id p (5) where ∆ equals 2R/ncm for class means and 2R/nc,k m for clustering of size nc,k within class c. The noise multiplier m is calibrated to (ϵ, δ)–LDP over T rounds via p (6) σiso ≥ c1 T ln(1/δ)/ϵ (Theorem 3.1). However, IGPP leaves two practical issues. First, applying isotropic noise perturbs all coordinates equally and can over-distort the most discriminative directions. Second, choosing a fixed ℓ2 clipping threshold is delicate. A large threshold inflates noise via ∆ while a small one severely distorts feature semantics. To address both, we propose VPDR, a client-side plug-in for ProtoPFL. As sketched in Figure 2, our Variance-adaptive Prototype Perturbation (VPP) redistributes noise across dimensions to minimize information loss while preserving the same LDP guarantee, and our Distillation-guided Clipping Regularization (DCR) on each local adaptation stabilizes the per-sample norm bound. A summary of notation is provided in Appendix B.

4.2. Variance-Adaptive Prototype Perturbation Isotropic noise overlooks the fact that feature dimensions matter differently for recognition: perturbing more informative dimensions harms utility more than redundant ones. Intuitively, dimensions with low intra-class variance often encode stable class-specific semantics, whereas those with high inter-class variance are more effective at separating classes. This motivates identifying an informative subspace and assigning it less perturbation. The main challenge, however, is that selecting such a subspace is itself datadependent and therefore privacy-sensitive. We then split the budget over T communication rounds via sequential composition: (ϵ, δ) = (ϵ1 , 0) + (ϵ2 , δ). Specifically, we allocate ϵ1 = rϵ with r ∈ (0, 1) to subspace selection and reserve ϵ2 = (1 − r)ϵ for prototype release (allocation diagram in Appendix C). This yields VPP, which adaptively reallocates perturbation while preserving the overall privacy guarantee. Discriminative Partition. For each j ∈ [d], we define X X Vjtra = (ncm − 1)s2c,j , Vjter = ncm (µc,j − µj )2 , (7) c

where the intra-class variance s2c,j , the class mean µc,j and the client-level mean µj are computed from encodings Zm . The ANOVA-normalized [30] discriminative score is: Vjter /(C − 1) , ζ > 0. (8) Sj = tra Vj /(nm − C) + ζ This normalization mitigates the effects of class-size imbalance and small-sample bias. Empirically, we find that Sj exhibits a strong linear correlation with the label mutual information I(zj ; y) (Figure 3a), with both Pear-

0.8

1.00

0.6

0.95

0.4 Photo Art Painting Cartoon Sketch

0.2 0.0

100

200

300

Discriminative Score Sj

(a) Sj vs. I(zj ; y) scatter.

Very Strong (0.9) Pearson r Spearman

0.90 0.85 P.

AP.

Ct.

Domain

Sk.

(b) Per-domain correlations.

Figure 3. Correlation between the discriminative score Sj and label mutual information I(zj ; y) on PACS.

son and Spearman coefficients consistently exceeding 0.90 across domains (Figure 3b). This makes Sj a reliable and lightweight proxy for dimension-wise discriminability. Although never uploaded, the selected index set determines the groupwise clipping bounds and noise covariance. Thus, the released prototypes remain a function of this private selection and must be included in privacy accounting. To privately isolate informative dimensions, we clip raw scores to bound the ℓ1 sensitivity, S j = clip(Sj , 0, H) := min{max(Sj , 0), H}, and apply the oneshot Laplace Topk mechanism [28]. Setting k = dA = ⌈ρd⌉ for ρ ∈ (0, 0.5] isolates the discriminative subspace IA , leaving the rest dB = d − dA as the non-discriminative subspace IB . Calibrating the per-round Laplace noise scale as λ ≥ 2dA HT /ϵ1

(9)

guarantees (ϵ1 , 0)-LDP for the partition step over T rounds as established by Theorem 4.1. Privatized Prototype Release. Given this partition, we decompose the feature vector as z = [zA ; zB ], where zA = z[IA ] and zB = z[IB ]. For a fixed global clipping threshold R, we allocate groupwise bounds: RA = RκA , RB = RκB , (10) p 2 2 where κA = dA /d, κB = dB /d, so that RA = + RB 2 R . We then independently apply groupwise ℓ2 clipping: zA = zA · min 1, RA /∥zA ∥2 , zB = zB · min 1, RB /∥zB ∥2 , (11) p

and form prototypes p = [pA ; pB ] by computing the mean or clustering on each group. This yields group sensitivities ∆A = 2RA /ncm = ∆κA and ∆B = 2RB /ncm = ∆κB . Let σref denote the noise multiplier of the reference isotropic Gaussian mechanism calibrated to (ϵ2 , δ). Theorem 4.2 establishes that groupwise reallocation can be performed without weakening the privacy guarantee, provided 2 2 2 that 1/σA + 1/σB ≤ 1/σref . We satisfy it by introducing weights wA , wB > 0 with wA + wB = 1 and parameterizing the groupwise multipliers as √ √ (12) σA = σref / wA , σB = σref / wB . For a parameter-free design, we set these weights solely

based on the group dimensions: wA = κB /(κA + κB ), wB = 1 − wA ,

(13)

The privatized prototype release is then formulated as: e = pA + ξ A , pB + ξ B , p (14) 2 2 ξ A ∼ N 0, (σA ∆A ) IdA , ξB ∼ N 0, (σB ∆B ) IdB . Under this construction, the release preserves the same (ϵ2 , δ)-LDP guarantee as the reference mechanism. Utility–privacy trade-off. Under the sequential composition (ϵ, δ) = (ϵ1 , 0) + (ϵ2 , δ), reserving budget for subspace selection implies ϵ2 < ϵ, which increases the reference multiplier σref . VPP offsets this by actively steering noise away from task-relevant coordinates. Specifically, ensuring the informative subspace IA is no noisier than the isotropic reference requires σA ∆A ≤ σref ∆. Substituting our weights (13) and sensitivities, we obtain: p κA (κA + κB )/κB ≤ 1 ⇔ κA ≤ κB ⇔ 0 < ρ ≤ 0.5. Consequently, restricting the selected discriminative dimensions to at most half of the feature space ensures that IA receives no more noise than the isotropic (ϵ2 , δ)-LDP mechanism, while the excess noise is shifted to the redundant subspace IB . As shown in Section 5, this targeted preservation of discriminative features delivers a net utility gain that outweighs the overhead induced by privacy budget splitting.

4.3. Distillation-Guided Clipping Regularization To ensure bounded sensitivity prior to adding calibrated Gaussian noise, clients typically apply per-example ℓ2 feature clipping before prototype construction. Hard clipping, however, is brittle: when a feature norm significantly exceeds the threshold (∥zi ∥2 ≫ R), hard clipping aggressively shrinks the vector and irrevocably erases structural information. Conversely, for undersized features (∥zi ∥2 ≪ R), the clipping operation is entirely ineffective, leaving small-magnitude vectors vulnerable to the injected noise. DCR addresses this dilemma by regularizing the feature space dynamically during local training. Soft Clipping. To mitigate sensitivity to R in the prototype perturbation phase, we append a differentiable soft-clipping layer at the end of the feature encoder during local training: R b zi = · zi , 0 < γ ≪ 1, (15) ∥zi ∥2 + γR where γ controls the strength. This expands small-norm features toward R and smoothly contracts large-norm features, avoiding the gradient discontinuities of hard clipping. Consistency Guidance. A potential failure mode is that the model counteracts soft clipping by inflating the classifier weights. As Figure 4a shows, pre-clipping feature norms tend to grow during mid-to-late training under soft clipping only. To decouple this shortcut, we introduce an Exponential Moving Average (EMA) teacher–student dis-

32 30 28 26 24 22 20

No DCR (baseline) Soft clipping only DCR (soft clipping + EMA KD) Clipping Threshold R

12 10 8 6 4 2

0 2 4 6 8 10 12 14 16 18 20 Training Round (a) Pre-clipping feature norm.

Soft clipping only DCR (soft clipping + EMA KD)

8 12 16 Training Round

(b) Teacher–student logit difference.

Figure 4. Evaluation of feature norm and logit difference of FedPLVM [42] with IGPP on Office–Caltech.

tillation that enforces prediction consistency between preand post-clipped features without parameter coupling. Concretely, we split the local classifier into a trainable student t head fm and a momentum teacher fm updated as t t θm = βθm + (1 − β)θm ,

(16)

t denote the student and teacher parameters, where θ and θm

respectively, and β ∈ [0, 1) is the EMA momentum. At each training step the teacher produces soft targets on the t original feature, y t = fm (z), while the student predicts on the soft-clipped feature, y s = fm (z). We minimize the KL divergence between their temperature-scaled distributions: y s yt softmax LKD = KL softmax , (17) τ τ where τ is the distillation temperature. This consistency prevents the model from bypassing clipping via weight rescaling, effectively breaking the norm–weight tug-of-war. Let LBASE denote the ProtoPFL objective (e.g., crossentropy for classification LCE or an InfoNCE-style contrastive alignment LCTR [11, 15, 42, 53]) as described in Figure 2. The overall local fine-tuning objective is L = LBASE + λ1 LKD .

(18)

Here λ1 is the KD weight. Empirically, DCR consistently reduces norm drift and concentrates pre-clipping feature norms near R during the mid-to-late training (Figure 4a). Also, it lowers the teacher–student logit discrepancy, providing evidence of stronger prediction consistency (Figure 4b). These behaviors make the framework less sensitive to the exact choice of R while preserving task semantics.

4.4. ProtoPFL with VPDR Algorithm 1 outlines ProtoPFL with VPDR. In each round, clients first privately select a discriminative subspace and release privatized prototypes for uplink transmission. The server aggregates or trains these representations into global prototypes and broadcasts them back. With the global prototypes fixed, clients conduct local personalized fine-tuning via DCR while optimizing the base ProtoPFL objective. Privacy Guarantees. VPDR confines all privacy-sensitive operations to the VPP during uplink transmission. Conversely, DCR is a purely local regularization that con-

Algorithm 1 ProtoPFL with VPDR Input: Communication rounds T , local epochs E, classes C , clients M , private dataset Dm = {xi , yi }inm , global clipping threshold R, global noise multiplier σ Output: Well-trained client models 1: for t = 1, 2, · · · , T do 2: // Local Private Prototype Calculation 3: for Client m = 1, 2, . . . , M do 4: Encode features zic = hm (xci ) 5: Compute dimension scores S via (7) and (8); 6: Privately partition to obtain IA and IB ; 7: Perform groupwise clipping {z ci } by (11); 8: for Class c = 1, 2, . . . , C do 9: Get local prototypes and add noise by (14). 10: end for em . 11: Upload the privatized prototype set P 12: end for 13: // Global Prototype Generation 14: Server aggregates or trains the global set Pg by (3). 15: // Local Personalized Fine-Tuning 16: for Client m = 1, 2, . . . , M do 17: for e = 1, 2, · · · , E do 18: Apply soft clipping to features via (15); 19: Update the EMA teacher via (16); 20: Update adapter and classifier by Eq. (18). 21: end for 22: end for 23: end for sumes no privacy budget. Under the budget split in Section 4.2, VPP is analyzed in two steps: Theorem 4.1 provides (ϵ1 , 0)–LDP for private subspace selection, and Theorem 4.2 provides (ϵ2 , δ)–LDP for prototype release. By sequential composition, ProtoPFL with VPDR satisfies overall (ϵ, δ)–LDP. Full proofs are deferred to Appendix D. Theorem 4.1 (Partitioning privacy). Let {Sj }dj=1 be the scores from (8) computed on client m, and define S j = clip(Sj , 0, H) under adjacency on Dm . In each round, execute the one-shot Laplace Top-k mechanism (k = dA ) by drawing bj ∼ Lap(λ) to form yj = S j + bj , outputting the indices of the top-k values. If λ ≥ 2dA HT /ϵ1 , the T -round partitioning process satisfies (ϵ1 , 0)-LDP for client m. Next, conditioned on the selected subspace, we calibrate noise for releasing the groupwise clipped prototypes. Theorem 4.2 (Release privacy). Let the isotropic Gaussian release with sensitivity ∆ achieve (ϵp 2 , δ)–LDP after T rounds using multiplier σref ≥ c2 T ln(1/δ)/ϵ2 . Under VPP, use groupwise covariance ΣVPP = diag((σA ∆A )2 IdA , (σB ∆B )2 IdB ) with ∆2 = ∆2A + ∆2B . If the group multipliers satisfy 1 1 1 (19) 2 + σ2 ≤ σ2 , σA B ref

then the T -round prototype release in VPP is also (ϵ2 , δ)–LDP for client m. Remark √ choice satisfying (19) is σA = √ 4.1. A convenient σref / w and σB = σref / 1 − w with w ∈ (0, 1). In VPP, we set w according to (13), so the noise allocation is fully determined by the discriminability-aware partition and automatically adapts to the relative dimensionalities dA and dB , without introducing any extra tunable hyperparameter. Complexity and Overheads. VPDR introduces only modest overhead to existing ProtoPFLs. (1) Computation: Per communication round, VPP incurs an extra O(nm d + d log d) time over baseline construction, while noise addition remains O(d). DCR adds O(nm dCE/B) training time for the teacher forward pass and KL divergence computation. (2) Memory: Overhead is minimal, requiring only O(d) to store the score vector and mask, alongside O(dC) for the momentum teacher head. (3) Communication: Complexity remains strictly unchanged at O(Cd), as clients solely transmit the final privatized prototypes.

5. Experiments 5.1. Experimental Setup Datasets. We conduct experiments on three domain-skew benchmarks: Digits [16, 18, 26, 29], which comprises MNIST (MNI.), USPS, SVHN and synthetic digits (SYN); Office–Caltech [13], consisting of Caltech (Cal.), Amazon (Amz.), DSLR (DSR.) and Webcam (Web.); PACS [20] covering Photo (P.), Art Painting (AP.), Cartoon (Ct.) and Sketch (Sk.). Details are provided in Appendix E.1. Model. All experiments use a ViT-small [8] backbone with a hidden size d of 512 and frozen pretrained weights. We insert lightweight adapters after each Transformer block, and train only the adapters and the classifier. Counterparts. We compare six ProtoPFL frameworks, all equipped with IGPP: FedProto [36], FedPCL [11], FPL [15], FedPLVM [42], FedTGP [53], and MPFT [55]. Personalization is achieved by exchanging only privatized prototypes. For MPFT, the server learns an adapter from the privatized prototypes and transmits it to clients, which is the post-processing of DP [9] and no extra privacy cost. Implementation Details. Each benchmark has four domain-specific clients [11, 42], training on 10% of Digits and 30% of Office–Caltech and PACS. We run communication rounds T = 20 with local epochs E = 2 and batch size 256. Local optimization is AdamW with learning rate 10−3 , weight decay 10−5 , and momentum 0.9. ProtoPFL hyperparameters are listed in Appendix E.2. Unless otherwise noted, we set (ϵ, δ)=(1, 10−5 ) and select the clipping radius R by grid search over {5, 10, 15, 20}. VPDR defaults are H = 0.1, r = 0.1, ρ = 0.2, γ = 0.05, β = 0.999,

Table 1. Comparison of Average Accuracy (AVG) and Standard Deviation (STD) under domain skew. The best is marked. Methods Framework +LDP

Digits Office–Caltech MNI. USPS SVHN SYN AVG↑ STD↓ Amz. Cal. DSR. Web. AVG↑ STD↓

+IGPP +VPDR +IGPP FedPCL [11] +VPDR +IGPP FPL [15] +VPDR +IGPP FedPLVM [42] +VPDR +IGPP FedTGP [53] +VPDR +IGPP MPFT [55] +VPDR

98.20 93.42 89.80 96.00 94.36 98.31 94.87 92.33 98.70 96.05 98.30 90.08 91.54 95.35 93.82 98.92 92.63 93.15 96.95 95.41 98.15 93.67 91.89 96.45 95.04 98.10 95.60 93.40 97.60 96.20 97.81 90.43 92.57 92.20 93.25 97.71 92.93 94.57 93.70 94.73 97.62 93.87 91.69 97.20 95.10 97.84 95.32 93.24 98.75 96.29 97.61 94.27 92.95 98.00 95.71 98.64 94.77 93.78 99.05 96.56

FedProto [36]

Test Accuracy (%)

100

3.45 2.78 3.41 2.63 2.44 2.35 2.88 1.79 2.44 2.14 2.16 2.32

PACS Ct. Sk. AVG↑ STD↓

AP.

95.32 91.07 81.38 95.51 90.82 6.62 97.70 92.71 91.52 80.37 90.58 7.31 96.71 93.55 84.85 98.32 93.36 6.01 98.80 95.60 91.88 84.59 92.71 6.11 95.29 92.41 64.52 94.92 86.79 13.53 97.31 85.33 80.34 55.92 79.73 16.08 96.33 92.86 74.97 96.61 90.19 8.90 98.50 91.93 87.18 68.03 86.91 11.84 94.76 92.36 82.15 96.31 91.40 6.37 98.10 94.13 91.16 82.08 91.37 6.28 96.34 93.72 85.02 99.65 93.68 6.36 98.86 96.78 93.15 83.93 93.18 5.86 95.61 91.26 82.05 96.61 91.38 6.64 97.90 92.15 84.11 74.20 87.09 9.34 97.30 92.23 84.76 99.21 93.38 6.07 98.80 93.71 86.04 77.63 89.05 8.38 96.34 92.38 83.97 96.55 92.31 5.88 98.80 96.62 92.09 84.29 92.95 5.80 97.34 96.61 86.21 100 95.04 6.19 100 97.32 94.83 87.01 94.79 5.64 95.76 92.52 84.03 96.61 92.23 5.74 99.63 96.33 93.16 84.50 93.41 5.89 97.81 94.15 86.58 98.31 94.71 5.83 99.40 97.70 95.65 86.88 94.91 5.04

100

AVG P. AP. 85 Ct. Sk. 80 0.01 0.05 0.1

0.2

0.3

0.4

Discriminative dimension ratio

AVG P. AP. 85 Ct. Sk. 80 0.05 0.1

0.2

0.3

0.4

Privacy Partition Ratio r

(a) The effect of ρ.

0.5

AVG P. AP. 85 Ct. Sk. 80 0.001 0.005 0.01 0.05 0.1 0.5 1.0

(b) The effect of r.

Soft Clipping Strength

85 80

AVG P. AP. Ct. Sk. 0.9 0.95 0.97 0.98 0.99 0.995 0.999

EMA Momentum

AVG P. AP. Ct. Sk. 1 2

85 80

KD Temperature

(d) The effect of β.

(e) The effect of τ .

Figure 5. Hyperparameter sensitivity of FedProto [36] framework with VPDR on PACS. Table 2. Ablation results of FedProto [36] framework. VPP

DCR

✓ ✓

VPP

DCR

✓ ✓

Amz.

Cal.

95.32 96.96 97.08 96.71

91.07 92.88 92.60 93.55

Office-Caltech DSR Web.

AP.

81.38 95.51 84.53 97.85 84.57 97.31 84.85 98.32 PACS Ct. Sk.

97.70 98.91 98.62 98.80

92.71 93.15 93.70 95.60

91.52 92.18 91.82 91.88

80.37 83.31 83.51 84.59

AVG↑

STD ↓

90.82 93.06 92.89 93.36

6.62 6.08 5.96 6.01

AVG↑

STD ↓

90.58 91.89 91.91 92.71

7.31 6.44 6.29 6.11

τ = 4 and λ1 = 0.05. We evaluate each personalized model on its own local data and report the average test accuracy.

5.2. Performance Comparison Table 1 shows that VPDR consistently improves average accuracy (AVG) compared with IGPP across all ProtoPFLs and benchmarks. Improvements are most pronounced on the more heterogeneous PACS and Office–Caltech. While a few per-domain scores fluctuate, the per-dataset averages for every framework increase, and dispersion (STD) typically shrinks on harder regimes, indicating that VPDR’s variance-adaptive noise and clipping stabilization primarily reduce multi-domain fluctuation where it matters most. Additional results and t-SNE plots are in Appendix E.3.

5.3. Diagnostic Analysis Hyperparameter Study. Figure 5 illustrates the impact of hyperparameters on PACS in FedProto [36]. The optimal

Table 3. Average accuracy (%) on CIFAR-10 under label skew. Methods

α = 0.1 α = 0.5 α = 1

+IGPP FedProto [36] +VPDR +IGPP FedPCL [11] +VPDR +IGPP FPL [15] +VPDR +IGPP FedPLVM [42] +VPDR +IGPP FedTGP [53] +VPDR +IGPP MPFT [55] +VPDR

42.76 44.14 18.22 20.92 28.42 37.18 37.33 40.17 40.71 43.03 46.79 47.33

82.31 85.00 73.63 73.98 69.14 80.14 76.95 77.42 79.92 82.46 79.06 80.17

90.23 89.99 87.78 88.29 88.19 88.73 87.21 82.76 87.68 89.61 91.23 91.26

α = 5 α = 10 96.15 96.34 95.55 95.63 95.39 96.04 95.06 95.47 95.28 96.32 96.64 96.53

96.30 96.55 95.67 95.61 95.58 96.29 95.32 95.34 95.47 96.42 96.71 96.64

range for the privacy partition ratio r lies between 0.1 and 0.2, while the discriminative subspace ratio ρ performs best around 0.2. The optimal soft clipping strength γ is 0.05, and the EMA momentum β achieves its best performance at 0.999. The KD temperature τ is most effective at 4, with little variation between 4 and 8. These trends confirm that VPDR delivers robust performance across a wide spectrum and that our selected defaults lie close to the optimum. Ablation Study. We ablate VPP and DCR within FedProto [36] as shown in Table 2. Starting from IGPP, adding either VPP or DCR alone improves performance on both Office–Caltech and PACS. Using both together yields the best average accuracy, confirming complementary gains. Scalability under Label Skew. We induce client-level label skew on CIFAR-10 [17] using a Dirichlet distribution with concentration α ∈ {0.1, 0.5, 1, 5, 10}, where smaller α means higher skew. Table 3 (M = 20, ϵ = 1) shows

Table 4. Privacy Attack Results of FedProto [36] on Office-Caltech under varying privacy budgets.

- NoLDP +IGPP 1 +VPP +VPDR +IGPP 2 +VPP +VPDR

Feature Reconstruction (FSH) Cos Sim↓ cFFD↑ Top-1 Hit(%)↓ 0.9999±0.0000 0.6437±0.0022 0.6456±0.0026 0.6456±0.0032 0.6710±0.0025 0.6706±0.0025 0.6718±0.0029

257.7±65.0 2214.6±21.0 2197.6±18.2 2207.1±27.7 2038.5±22.7 2040.7±26.6 2029.4±26.0

100.00±0.00 20.29±3.52 21.22±4.04 20.62±3.33 33.23±3.59 34.44±5.83 33.75±2.90

that VPDR yields the largest gains under extreme skew (α ≤ 0.5). While this advantage narrows toward the IGPP as data becomes balanced, it highlights that VPP’s discriminative focus and DCR’s norm stabilization thrive under high heterogeneity. Furthermore, we verify VPDR’s robustness under joint model–data heterogeneity and its generalization to text modality in Appendices E.4 and E.5.

ROC-AUC→ 0.5

Membership Inference (MIA) TPR@1%FPR↓ Advantage→ 0

0.6729±0.0124 0.5079±0.0340 0.4992±0.0293 0.5055±0.0275 0.5081±0.0336 0.5097±0.0228 0.5089±0.0299

0.0298±0.0200 0.0043±0.0045 0.0040±0.0050 0.0042±0.0044 0.0066±0.0036 0.0065±0.0016 0.0063±0.0031

Time (seconds)

ϵ LDP

IGPP (ProtoGen) VPDR+ (ProtoGen) VPDR (ProtoGen)

2 1 0

IGPP (FT) VPDR+ (FT) VPDR (FT) 2.26

2.13

0.99

1.99

1.02

2.06

2.22

0.90

FedProto

FedPCL

1.01

0.90

FPL

2.00

0.7881±0.0082 0.7317±0.0085 0.7282±0.0119 0.7321±0.0109 0.7399±0.0073 0.7406±0.0084 0.7404±0.0117

0.99

2.01

0.92

0.87

FedPLVM

FedTGP

Methods

2.63

2.44

2.29

2.25

0.93

0.87

F1 Score↓

0.4458±0.0170 0.2669±0.0300 0.2536±0.0279 0.2649±0.0299 0.2855±0.0332 0.2834±0.0244 0.2846±0.0309

1.11

2.21

0.87

MPFT

(a) Office-Caltech.

5.4. Defense Assessment Under Attacks

5.5. Computational Cost Figure 6 reports the per-round, per-client wall-clock time on Office-Caltech and PACS, decomposed into Prototype Generation (ProtoGen) and Local Fine-Tuning (FT). Across six ProtoPFL frameworks, VPDR increases total runtime by an average of only 0.28s (8.1%) compared to the IGPP

Time (seconds)

We consider a realistic deployment threat model in which an adversary can access client-uploaded prototypes, either as a semi-honest server or as an external eavesdropper. We evaluate two attacks against all clients per communication round: Feature-Space Hijacking (FSH) [55] and Membership Inference Attacks (MIA) [33]. FSH actively reconstructs raw client data by optimizing dummy inputs to match targeted prototypes. A successful defense yields low cosine similarity, high classifier feature fréchet distance (cFFD), and low Top-1 Hit rates. Conversely, MIA passively infers training set membership by thresholding sample-to-prototype distances, evaluated via ROCAUC (where values near 0.5 indicate random guessing), TPR@1%FPR, advantage, and F1 score. Detailed attack settings are deferred to Appendix E.6. Table 4 reports the attack outcomes on Office-Caltech under ϵ ∈ {1, 2}, with identical trends observed on Digits and PACS (Appendix E.6). Compared to NoLDP, all LDP mechanisms drastically degrade the attacker’s capabilities. Crucially, VPP and VPDR defense metrics match the IGPP baseline within a single standard deviation. At ϵ = 1, all three mechanisms drive MIA ROC-AUC to chance (∼ 0.50) and throttle the FSH Top-1 Hit rate to roughly 20%. These results confirm that VPDR’s varianceadaptive reallocation and soft clipping do not leak structural vulnerabilities, achieving superior utility under the same practical defense strength as isotropic Gaussian perturbation. More results varying the d and r are in Appendix E.7.

5 4

IGPP (ProtoGen) VPDR+ (ProtoGen) VPDR (ProtoGen)

IGPP (FT) VPDR+ (FT) VPDR (FT)

3.85

4.77 4.35

4.01

3.71

3.70

3 2 1 0

4.53 1.56

3.65

3.58

1.66

1.45

1.40

1.41

1.46

1.48

FedProto

FedPCL

FPL

3.65

1.56

3.79

1.51

1.45

FedPLVM

FedTGP

Methods

4.32 1.48 1.43

MPFT

(b) PACS.

Figure 6. Per-client average time (s) per communication round. Table 5. Runtime (ms) of FedProto with VPDR on Office-Caltech. Stage

Base

Extra VPDR

ProtoGen 1057 13.22 (Stats) + 9.78 (Top-k) = 23.00 FT 2055 1.58 (Clip) + 1.59 (Tch) + 1.38 (KL) = 4.55

Overhead 2.2% 0.2%

baseline, with phase-level increases of +0.09s for ProtoGen and +0.20s for FT. Table 5 details this cost for FedProto on Office-Caltech. VPDR operations incur marginal overheads: 2.2% in ProtoGen (variance statistics and Laplace Top-k) and 0.2% in FT (soft clipping, teacher-head forward, KL divergence term). Ultimately, VPDR delivers accuracy gains with acceptable or even negligible computational overhead and no additional server-side work.

6. Conclusion We introduce VPDR, a client-side plug-in for privacypreserving ProtoPFL that combines VPP, which reallocates noise across feature dimensions by discriminative strength, and DCR, which enforces soft-clipping and prediction consistency via knowledge distillation. Our analysis provides LDP guarantees no weaker than those of isotropic perturbation. Across diverse ProtoPFLs and multi-domain benchmarks, VPDR delivers higher utility at fixed privacy budgets and remains robust to label skew, while preserving communication cost and incurring only modest compute overhead. Evaluations against two attacks show near-chance performance, indicating strong protection.

Acknowledgements This work was supported by the Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, the National Natural Science Foundation of China (U25B2070, 62372493), the Beijing Natural Science Foundation (Z230001), the China Postdoctoral Fellowship Fund (2024M764092), and the Beihang Dare to Take Action Plan (JKF-20240773).

References [1] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Advances in Neural Information Processing Systems, pages 22300–22312. Curran Associates, Inc., 2022. 1 [2] Sikai Bai, Jie Zhang, Song Guo, Shuaicheng Li, Jingcai Guo, Jun Hou, Tao Han, and Xiaocheng Lu. Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27274–27283, 2024. 1 [3] Alberto Bietti, Chen-Yu Wei, Miroslav Dudik, John Langford, and Steven Wu. Personalization improves privacyaccuracy tradeoffs in federated learning. In International Conference on Machine Learning, pages 1945–1962. PMLR, 2022. 2, 1 [4] Yuhang Chen, Wenke Huang, and Mang Ye. Fair federated learning under domain skew with local consistency and domain diversity. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12077– 12086, 2024. 1 [5] Anda Cheng, Peisong Wang, Xi Sheryl Zhang, and Jian Cheng. Differentially private federated learning with local regularization and sparsification. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10112–10121, 2022. 2, 1 [6] Lixin Cui and Xu Wu. Aldp-fl for adaptive local differential privacy in federated learning. Scientific Reports, 15(1): 26679, 2025. 2, 1 [7] Rami Darwish, Mahmoud Abdelsalam, Sajad Khorsandroo, and Kaushik Roy. Fedp3e: Privacy-preserving prototype exchange for non-iid iot malware detection in cross-silo federated learning. arXiv preprint arXiv:2507.07258, 2025. 2 [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and Alexander Houlsby. An image is worth 16×16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 6, 4 [9] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006. 6 [10] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A

model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems, pages 3557–3568. Curran Associates, Inc., 2020. 1 [11] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems, pages 3557–3568. Curran Associates, Inc., 2020. 1, 2, 3, 5, 6, 7, 4 [12] Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557, 2017. 2, 1 [13] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066–2073, 2012. 6, 4 [14] Zaobo He, Lintao Wang, and Zhipeng Cai. Clustered federated learning with adaptive local differential privacy on heterogeneous iot data. IEEE Internet of Things Journal, 11(1): 137–146, 2024. 2, 1 [15] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16312–16322, 2023. 1, 2, 3, 5, 6, 7, 4 [16] Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):550–554, 1994. 6, 3 [17] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7, 4 [18] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 6, 3 [19] Gihun Lee, Minchan Jeong, Sangmook Kim, Jaehoon Oh, and Se-Young Yun. Fedsol: Stabilized orthogonal learning with proximal restrictions in federated learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12512–12522, 2024. 1 [20] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5543–5551, 2017. 6, 4 [21] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proceedings of the 38th International Conference on Machine Learning, pages 6357–6368. PMLR, 2021. 1 [22] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10082–10091, 2022. 1 [23] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. CommunicationEfficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017. 1

[24] Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263–275. IEEE, 2017. 3 [25] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling dataconstrained language models. In Advances in Neural Information Processing Systems, pages 50358–50376. Curran Associates, Inc., 2023. 1 [26] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, page 4. Granada, 2011. 6, 3 [27] Hyunsin Park and Sungrack Yun. Fedhide: Federated learning by hiding in the neighbors. In European Conference on Computer Vision, pages 405–422. Springer, 2024. 2 [28] Anqi Qiao, Weijie J. Su, and Michael I. Zhang. Oneshot differentially private top-k selection. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 2021. 3, 4 [29] P Roy, S Ghosh, S Bhattacharya, and U Pal. Effects of degradations on deep neural network architectures. arxiv 2018. arXiv preprint arXiv:1807.10108, 2018. 6, 3 [30] Henry Scheffe. The analysis of variance. John Wiley & Sons, 1999. 4 [31] Changlong Shi, He Zhao, Bingjie Zhang, Mingyuan Zhou, Dandan Guo, and Yi Chang. Fedawa: Adaptive optimization of aggregation weights in federated learning using client vectors. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 30651–30660, 2025. 1 [32] Yifan Shi, Yingqi Liu, Kang Wei, Li Shen, Xueqian Wang, and Dacheng Tao. Make landscape flatter in differentially private federated learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24552–24562, 2023. 2, 1 [33] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18, 2017. 8 [34] Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, pages 21394– 21405. Curran Associates, Inc., 2020. 1 [35] Rishub Tamirisa, Chulin Xie, Wenxuan Bao, Andy Zhou, Ron Arel, and Aviv Shamsian. Fedselect: Personalized federated learning with customized selection of parameters for fine-tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23985–23994, 2024. 1 [36] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI conference on artificial intelligence, pages 8432– 8440, 2022. 1, 2, 3, 6, 7, 8, 4, 5 [37] Linh Tran, Wei Sun, Stacy Patterson, and Ana Milanova. Privacy-preserving personalized federated prompt learning

for multimodal large language models. In The Thirteenth International Conference on Learning Representations, 2025. 2, 3 [38] Tim van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797–3820, 2014. 3 [39] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of llm scaling based on humangenerated data. In Forty-first International Conference on Machine Learning, 2024. 1 [40] Guancheng Wan, Wenke Huang, and Mang Ye. Federated graph learning under domain shift with generalizable prototypes. Proceedings of the AAAI Conference on Artificial Intelligence, 38(14):15429–15437, 2024. 1 [41] Debao Wang and Shaopeng Guan. Fedfr-adp: Adaptive differential privacy with feedback regulation for robust model performance in federated learning. Information Fusion, 116: 102796, 2025. 2, 1 [42] Lei Wang, Jieming Bian, Letian Zhang, Chen Chen, and Jie Xu. Taming cross-domain representation variance in federated prototype learning with heterogeneous data domains. In Advances in Neural Information Processing Systems, pages 88348–88372. Curran Associates, Inc., 2024. 1, 2, 3, 5, 6, 7, 4 [43] Li Wang, Qiang Wu, and Min Xu. Fedpcl-cdr: A federated prototype-based contrastive learning framework for privacy-preserving cross-domain recommendation. Neural Networks, 196:108380, 2026. 2 [44] Shaowei Wang, Liusheng Huang, Yiwen Nie, Xinyuan Zhang, Pengzhan Wang, Hongli Xu, and Wei Yang. Local differential private data aggregation for discrete distribution estimation. IEEE Transactions on Parallel and Distributed Systems, 30(9):2046–2059, 2019. 3 [45] Zheng Wang, Zihui Wang, Zheng Wang, Xiaoliang Fan, and Cheng Wang. Federated learning with domain shift eraser. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4978–4987, 2025. 1 [46] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Hang Su, Bo Zhang, and H. Vincent Poor. User-level privacy-preserving federated learning: Analysis and performance optimization. IEEE Transactions on Mobile Computing, 21(9):3388–3401, 2022. 2, 3, 1 [47] Chulin Xie, De-An Huang, Wenda Chu, Daguang Xu, Chaowei Xiao, Bo Li, and Anima Anandkumar. Perada: Parameter-efficient federated learning personalization with generalization guarantees. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23838–23848, 2024. 1 [48] Xiyuan Yang, Wenke Huang, and Mang Ye. Dynamic personalized federated learning with adaptive differential privacy. In Advances in Neural Information Processing Systems, pages 72181–72192. Curran Associates, Inc., 2023. 2, 1 [49] Xiyuan Yang, Wenke Huang, and Mang Ye. Fedas: Bridging inconsistency in personalized federated learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11986–11995, 2024. 1

[50] Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 6137–6147, New York, NY, USA, 2024. Association for Computing Machinery. 1 [51] Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37 (9):11237–11244, 2023. 1 [52] Jianqing Zhang, Yang Liu, Yang Hua, and Jian Cao. An upload-efficient scheme for transferring knowledge from a server-side pre-trained generator to clients in heterogeneous federated learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12109– 12119, 2024. 1, 2, 4 [53] Jianqing Zhang, Yang Liu, Yang Hua, and Jian Cao. Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15):16768–16776, 2024. 1, 2, 3, 5, 6, 7, 4 [54] Jianqiao Zhang, Caifeng Shan, and Jungong Han. Fedgmkd: An efficient prototype federated learning framework through knowledge distillation and discrepancy-aware aggregation. In Advances in Neural Information Processing Systems, pages 118326–118356. Curran Associates, Inc., 2024. 2, 3, 1 [55] Jingyuan Zhang, Yiyang Duan, Shuaicheng Niu, YANG CAO, and Wei Yang Bryan Lim. Enhancing federated domain adaptation with multi-domain prototype-based federated fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025. 2, 3, 6, 7, 8, 4 [56] Jia-Ying Zheng, Hainan Zhang, Lingxiang Wang, Wangjie Qiu, Hong-Wei Zheng, and Zhi-Ming Zheng. Safely learning with private data: A federated learning framework for large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5293–5306, Miami, Florida, USA, 2024. Association for Computational Linguistics. 1

Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning Supplementary Material

A. Related Work A.1. Differential Privacy in Federated Learning . A.2. Prototype-based Personalized FL . . . . . .

1 1 1

B. Notations

C. Privacy Budget Allocation and Calibration

D. Details for Privacy Analysis D.1. Proof of Theorem 4.1 . . . . . . . . . . . . . D.2. Proof of Theorem 4.2 . . . . . . . . . . . . .

2 2 3

E. Experimental Supplement E.1. Details of Datasets . . . . . . . . . . . . . . E.2. Hyperparameters of ProtoPFLs . . . . . . . E.3. More Results for Section 5.2 . . . . . . . . . E.4. Model with Data Heterogeneity . . . . . . . E.5. Generalization to Text Modality . . . . . . . E.6. More Details for Section 5.4 . . . . . . . . . E.7. Effect of d and r . . . . . . . . . . . . . . .

3 3 4 4 4 5 5 6

A. Related Work A.1. Differential Privacy in Federated Learning In privacy-preserving Federated Learning (FL), Differential Privacy (DP) has become a widely adopted approach to safeguard sensitive information during model updates. Early methods like DP-FedAvg [12] employed Gaussian mechanisms, using ℓ2 norm clipping and noise injection to ensure client-level DP prior to communication. This approach relies on the server aggregating updates from clients, with privacy guarantees applied at the server level, often referred to as server-side client-level DP. Subsequent advancements, such as BLUS+LUS [5] and DPFedSAM [32], incorporated local regularization and update sparsification techniques to mitigate sensitive gradient directions, while optimizing model flatness to minimize the performance degradation induced by DP noise. For Personalized FL (PFL), significant progress has been made in privacy-preserving methods. PPSGD [3] introduced a personalized parameter to balance local and global models while ensuring privacy through joint DP. Similarly, FedDPA [48] employs dynamic hierarchical Fisher information to select personalized parameters, and adaptively adjusts ℓ2 regularization to improve robustness against clipping and reduce noise interference.

However, these methods typically assume an honest server, whereas, in practice, servers are often honest-butcurious — they follow protocols but may attempt to infer sensitive information from client updates. To address this, Local Differential Privacy (LDP) ensures privacy even in the presence of semi-honest servers or external eavesdroppers. Representative methods like UDP-FL [46] add calibrated Gaussian noise to each client update, guaranteeing client-level LDP. More recent techniques, such as ACS-FL [14], enhance FL models under LDP by combining adaptive clipping, weight compression, and parameter shuffling. These methods consider layer-wise ranges and budget accumulation to reduce both noise and communication, improving the privacy-utility trade-off in heterogeneous IoT data. FedFR-ADP [41] introduces an adaptive LDP scheme that calibrates each client’s Gaussian noise based on heterogeneity measured via Earth Mover’s Distance, dynamically adjusting the privacy budget. ALDP-FL [6] fine-tunes per-layer clipping via moving-average norms and injects bounded layer-wise LDP noise, improving utility while mitigating reconstruction attacks. Despite these advancements, existing LDP methods primarily focus on privacy for gradient-based updates or model parameters, with limited attention to the privacy risks associated with prototype frameworks.

A.2. Prototype-based Personalized FL Prototype-based Personalized FL (ProtoPFL) leverages class prototypes to model multi-domain feature distribution shifts, enhancing personalization through shared and local prototypes. Early works like FedProto [36] introduced prototype exchange to mitigate gradient misalignment in heterogeneous data, while FedPCL [11] used prototype-level contrastive learning to align local prototypes without sharing model parameters. FPL [15] introduced cluster-based and unbiased prototypes to improve model stability and discriminability across domains, and FedGMKD [54] incorporated knowledge distillation and clustering to improve robustness to heterogeneous data without relying on public datasets. FedPLVM [42] proposed a dual-level clustering method to reduce communication costs while improving generalization with an α-sparsity loss function. More recent works have incorporated server-side training, such as FedTGP [53], which introduced trainable global prototypes and adaptive-margin contrastive learning to enhance inter-class distance and semantic consistency, and FedKTL [52], which uses a pre-trained server model to generate

Table 6. Summary of notation. Symbol

Clientm

Description

C, M ; c, m Number of classes and clients; class and client indices T, E, B Communication rounds, local epochs, and batch size Dm , Z m Client-m local dataset and embedding set nm , ncm , nc,k m Client-m total, per-class, and per-cluster sample counts d Embedding dimension g, hm , fm Backbone, client-m encoder, and classifier z, z, ẑ Original, ℓ2 -clipped, and soft-clipped features Prototype of class c on client m (cluster k) pc,k m Pm , P g Client-m local and global prototype set Sj Discriminability score of coordinate j Clipping cap applied to scores Sj H ρ Fraction of dimensions kept as discriminative IA , I B Discriminative / residual index sets dA , d B Dimensions of discriminative and residual subspaces R Global clipping threshold Groupwise clipping thresholds on IA and IB RA , R B ϵ, δ, ∆f LDP budget parameters and ℓ2 -sensitivity of f ϵ1 , ϵ 2 Privacy budgets for partition and prototype release r Privacy split ratio (ϵ1 = rϵ, ϵ2 = (1 − r)ϵ) ∆, ∆A , ∆B Isotropic and groupwise prototype sensitivities λ Noise scale for oneshot Laplace Top-k Noise multipliers of IGPP and reference mechanism σiso , σref σA , σ B Groupwise noise multipliers in VPP γ, β Soft-clipping strength and EMA momentum t , ys Client-m Teacher and student soft targets ym m τ, λ1 Distillation temperature and KD weight LBASE , LKD Base ProtoPFL loss, and distillation loss

class prototype pairs for multi-domain semantic alignment. MPFT [55] uploads a small number of client-generated prototypes for global model fine-tuning, enhancing multidomain adaptation. These advancements have shifted ProtoPFL from static class-center sharing to dynamic semantic alignment and knowledge fusion, improving feature space alignment and model generalization across clients. However, because class prototypes are directly derived from client-specific sensitive data, sharing them can inadvertently expose private semantic information, especially in highly heterogeneous and sensitive tasks. While some defenses have been proposed [11, 42, 55], such as injecting isotropic Gaussian noise into prototypes, these methods often lack formal privacy guarantees or fail to account for per-example clipping or local DP for each client. Our approach addresses this by providing formal LDP guarantees within ProtoPFL, ensuring privacy during prototype exchange while maintaining the utility of shared prototypes.

reference mechanism VPP

prototype release

private partition

T ln(1 /  ) 2

(2 ,  )   ref  c2

Theorem 4.1

Theorem 4.2

2 d HT (1 ,0)    A 1

(2 ,  )  ( A ,  B ),

Remark 4.1

 A2



 B2



1 2  ref

sequential composition

  r (, )  (1 ,0)  (2 , )  1 2  (1  r )

IGPP

(, )   iso  c1

T ln(1 /  ) 

Figure 7. Privacy allocation and calibration. Schematic view of how IGPP and VPP use the per-client budget (ϵ, δ) and how the reference isotropic mechanism is employed to calibrate VPP.

the local budget as (ϵ, δ) = (ϵ1 , 0) + (ϵ2 , δ) with ϵ1 = rϵ and ϵ2 = (1 − r)ϵ. The pure-DP share (ϵ1 , 0) is used for the private discriminative partition on scores Sj over T rounds (Theorem 4.1), while (ϵ2 , δ) is reserved for T rounds of prototype release. To analyze the release in VPP, we introduce a conceptual reference isotropic Gaussian mechanism from the same family as IGPP. IGPP corresponds to using this family under the full budget (ϵ, δ) with multiplier σiso , whereas the reference mechanism is calibrated only to (ϵ2 , δ) with multiplier σref , so that it achieves (ϵ2 , δ)–LDP over T rounds (Theorem 3.1). VPP then replaces this reference isotropic release with an anisotropic Gaussian mechanism with groupwise multipliers (σA , σB ). Under the calibration condition (19), Theorem 4.2 shows that its privacy loss is no larger than that of the reference mechanism, and hence the VPP release also satisfies (ϵ2 , δ)–LDP. By sequentially composing the partition step with budget (ϵ1 , 0) and the release step with budget (ϵ2 , δ), VPP attains an overall (ϵ, δ)–LDP guarantee. See details in Section D.2

D. Details for Privacy Analysis All mechanisms are analyzed as acting locally on a single client’s dataset under the per-example adjacency relation D ∼ D′ defined in Section 3.1. Consequently, the resulting (ϵ, δ) guarantees represent client-side Local DP guarantees.

B. Notations Table 6 presents the formal notation used in this paper.

C. Privacy Budget Allocation and Calibration Figure 7 compares how the per-client local DP budget (ϵ, δ) is allocated in IGPP and VPP. IGPP spends the entire budget on an isotropic Gaussian prototype mechanism, choosing σiso so that the resulting T -round mechanism is (ϵ, δ)–LDP for each client (Theorem 3.1). In contrast, VPP decomposes

D.1. Proof of Theorem 4.1 Theorem 4.1 (Partitioning privacy). Let {Sj }dj=1 be the scores from (8) computed on client m, and define S j = clip(Sj , 0, H) under adjacency on Dm . In each round, execute the one-shot Laplace Top-k mechanism (k = dA ) by drawing bj ∼ Lap(λ) to form yj = S j + bj , outputting the indices of the top-k values. If λ ≥ 2dA HT /ϵ1 , the T -round partitioning process satisfies (ϵ1 , 0)-LDP for client m.

Proof. Clipping yields per-coordinate ℓ1 sensitivity ′

∆f ≤ sup fj (D) − fj (D ) ≤ H. D∼D ′

By Theorem 3.2, the oneshot Laplace Top-k mechanism run on a client in each round is (ϵ1 /T, 0)–DP provided 2dA HT , λ ≥ ϵ1 and the index partition is obtained by post-processing. Composing over T rounds then yields (ϵ1 , 0)–LDP for client m in FL, as claimed.

D.2. Proof of Theorem 4.2 We adopt the Rényi (RDP) framework [24] and analyze each client’s local mechanism under the per-example adjacency D ∼ D′ defined in Section 3.1. Applying the RDPto-DP conversion then yields LDP guarantees [37, 46]. The relevant definitions are recalled below. Definition D.1 (Rényi Divergence [38]). Given two probability distributions P and Q, the Rényi divergence of order α > 1 between P and Q is defined as P (x) α 1 log Ex∼Q . Dα (P ∥Q) = α−1 Q(x) Definition D.2 (RDP [24]). A randomized mechanism M : X → R is said to satisfy (α, ϵ(α))-RDP if for all adjacent datasets D ∼ D′ ,

Proof. We view the isotropic Gaussian mechanism with sensitivity ∆ that achieves (ϵ2 , δ)–LDP as a reference mechanism. Let v = f (D) − f (D′ ) denote the difference vector for adjacent datasets D ∼ D′ . Based on our partition, we decompose this vector into v = [vA ; vB ]. The groupwise clipping bound the sensitivities such that ∥vA ∥2 ≤ ∆A and ∥vB ∥2 ≤ ∆B , satisfying ∆2 = ∆2A + ∆2B . Reference (isotropic) release. For covariance Σref = (σref ∆)2 Id , Corollary D.1 gives ϵ(α)ref =

α ∆2 α α ∥v∥22 ≤ = 2 . 2 2 2 (σref ∆) 2 (σref ∆) 2σref

(20)

VPP (groupwise) release. For covariance ΣVPP = diag (σA ∆A )2 IdA , (σB ∆B )2 IdB , Lemma D.2 yields α ∥vB ∥22 ∥vA ∥22 α + v ≤ ϵ(α)VPP = v ⊤ Σ−1 VPP 2 2 (σA ∆A )2 (σB ∆B )2 1 α 1 . (21) ≤ 2 + σ2 2 σA B Provided that the harmonic calibration condition in (19) holds, comparing (20) and (21) guarantees that ϵVPP (α) ≤ ϵref (α) for all α > 1. By Lemma D.1, the (ϵ2 , δ)–LDP guarantee proven for the reference mechanism seamlessly transfers to VPP. Thus, the VPP prototype release strictly satisfies (ϵ2 , δ)–LDP over T rounds for client m. Finally, as established in p Theorem 3.1, the reference multiplier must satisfy σref ≥ c2 T ln(1/δ)/ϵ2 , completing the proof.

Dα (M(D)∥M(D′ )) ≤ ϵ(α). Lemma D.1 ((α, ϵ(α))-RDP to (ϵ, δ)-LDP [24]). If a mechanism M satisfies (α, ϵ(α))-RDP under the per-example adjacency D ∼ D′ , then for any δ ∈ (0, 1) it also satisfies (ϵ, δ)-LDP with ϵ = ϵ(α) + log(1/δ)/(α − 1). Lemma D.2. Let α > 1 and let P = N (µ, Σ) and Q = N (µ′ , Σ) be two d-dimensional Gaussians with the same positive–definite covariance Σ ∈ Rd×d . Then α Dα (P ∥Q) = (µ − µ′ )⊤ Σ−1 (µ − µ′ ). 2 Corollary D.1. For isotropic covariance Σ = σ 2 Id , α Dα N (µ, σ 2 Id ) ∥ N (µ′ , σ 2 Id ) = ∥µ − µ′ ∥22 . 2σ 2 Theorem 4.2 (Release privacy). Let the isotropic Gaussian release with sensitivity ∆ achieve (ϵp 2 , δ)–LDP after T rounds using multiplier σref ≥ c2 T ln(1/δ)/ϵ2 . Under VPP, use groupwise covariance ΣVPP = diag((σA ∆A )2 IdA , (σB ∆B )2 IdB ) with ∆2 = ∆2A + ∆2B . If the group multipliers satisfy 1 1 1 + 2 ≤ 2 , (19) 2 σA σB σref then the T -round prototype release in VPP is also (ϵ2 , δ)–LDP for client m.

Remark on Weight Allocation. A straightforward choice √ satisfying the condition in (19) is σ = σ / w and σB = A ref √ 2 σref / 1 − w for any w ∈ (0, 1), which ensures 1/σA + 2 2 1/σB = 1/σref holds with exact equality. This parameterization calibrates VPP to the precise (ϵ2 , δ)–LDP budget of the isotropic reference mechanism while strategically redistributing noise across subspaces. By fixing w according to the group dimensions (as defined in (13)), the noise allocation becomes fully adaptive to the discriminability-aware partition, automatically assigning smaller multipliers (i.e., less noise) to the more informative subspace without requiring manual hyperparameter tuning.

E. Experimental Supplement E.1. Details of Datasets We evaluate our method on three multi-domain benchmarks: Digits, Office–Caltech, and PACS. Below we provide a detailed description of each benchmark. • Digits [16, 18, 26, 29] consists of several handwritten digit recognition datasets, including MNIST, USPS, SVHN, and synthetic digits. These datasets cover a variety of digit recognition tasks, providing a testbed for domain adaptation and skew studies.

Table 7. Comparison of Average Accuracy (AVG) and Standard Deviation (STD) under domain skew at ϵ = 0.5. The best is marked. Methods Framework +LDP

Digits Office–Caltech MNI. USPS SVHN SYN AVG↑ STD↓ Amz. Cal. DSR. Web. AVG↑ STD↓

+IGPP +VPDR +IGPP FedPCL [11] +VPDR +IGPP FPL [15] +VPDR +IGPP FedPLVM [42] +VPDR +IGPP FedTGP [53] +VPDR +IGPP MPFT [55] +VPDR

98.08 93.57 89.41 95.25 94.08 3.63 95.27 91.86 76.42 95.03 89.64 8.95 98.10 92.62 91.24 79.48 90.36 7.84 98.17 94.32 92.55 97.70 95.69 2.70 96.34 93.75 83.87 96.61 92.64 5.99 98.10 95.56 91.38 83.35 92.10 6.46 98.27 80.70 92.43 65.25 84.16 14.57 95.81 89.29 61.29 94.64 85.26 16.23 97.66 83.44 62.61 51.92 73.91 20.54 98.47 85.62 92.89 75.20 88.05 10.05 96.34 91.96 72.19 95.92 89.10 11.45 97.31 83.62 77.56 55.92 78.60 17.23 98.09 92.52 91.49 96.45 94.64 3.14 94.53 91.86 81.65 95.77 90.95 6.41 98.20 92.91 90.81 79.75 90.42 7.76 98.24 94.22 93.03 97.30 95.70 2.47 96.08 92.41 83.87 98.31 92.67 6.35 98.40 96.56 91.74 81.64 92.09 7.51 97.79 89.94 92.07 89.20 92.25 3.89 95.29 88.84 79.42 96.42 89.99 7.80 97.90 90.95 82.69 65.99 84.38 13.75 98.14 91.59 93.17 91.45 93.59 3.13 96.34 92.86 81.65 98.04 92.22 7.37 98.80 92.18 86.54 68.84 86.59 12.85 97.59 93.82 90.61 96.90 94.73 3.20 95.81 91.96 83.87 96.61 92.06 5.83 98.20 95.11 90.17 81.91 91.35 7.11 97.86 95.22 92.42 98.40 95.97 2.75 96.86 95.09 84.15 100 94.03 6.89 99.10 97.56 92.95 83.06 93.17 7.23 97.56 93.92 92.83 97.60 95.48 2.47 91.10 91.07 80.65 94.92 89.44 6.13 99.20 95.84 92.31 82.68 92.51 7.13 98.55 94.53 93.50 98.80 96.34 2.73 97.38 91.07 83.87 98.31 92.66 6.68 99.20 96.09 93.52 83.06 92.97 7.00

FedProto [36]

• Office-Caltech [13] is a benchmark for object recognition and consists of four domains: Caltech-256, Amazon, DSLR, and Webcam. The dataset is commonly used to evaluate domain adaptation and generalization across real-world and synthetic image domains. • PACS [20] is a cross-domain dataset for domain generalization. It includes four domains: Photo, Art Painting, Cartoon, and Sketch. PACS is widely used to assess model robustness and generalization in visual recognition across multiple domains. Additionally, we use CIFAR-10 in the single-domain setting to study the scalability under label skew. • CIFAR-10 [17] consists of 60,000 32x32 color images across 10 classes, with 50,000 training images and 10,000 test images. In this setting, we simulate label skew by partitioning the dataset among clients, where each client receives a non-uniform distribution of class labels. Specifically, we achieve label skew by applying a Dirichlet distribution Dir(α) with α > 0 to sample the class proportions for each client. The parameter α controls the degree of skew, with smaller values leading to more imbalanced local distributions.

E.2. Hyperparameters of ProtoPFLs Table 8 lists hyperparameters for ProtoPFL frameworks. Hyperparameters in different methodologies may share the same notation but represent distinct meanings.

E.3. More Results for Section 5.2 Table 7 demonstrates that VPDR consistently outperforms IGPP in terms of average accuracy (AVG) across all ProtoPFL frameworks and benchmarks at ϵ = 0.5. Furthermore, the qualitative t-SNE visualizations for the Digits dataset at ϵ = 1.0 (Figure 8) reveal that VPDR produces tighter and more distinct clusters as training progresses into the later rounds.

AP.

PACS Ct. Sk. AVG↑ STD↓

Table 8. Hyperparameters chosen for different methods. Methods

Hyperparameters

Value

FedProto FedPCL FPL

Proximal weight λ Contrastive temperature τ Contrastive temperature τ α-sparsity α Contrastive temperature τ Proto loss weight λ Prototype regularization weight λ Server training rounds margin learning threshold Distill temperature T Distill weight β Clustering sampling rate r Server learning rate ηs Server training rounds

0.1 0.07 0.07 0.25 0.07 0.5 1.0 100.0 100.0 4 0.3 0.1 0.001 20

FedPLVM

FedTGP

MPFT

MNIST: 95.52 @T=1

USPS: 82.96 @T=1

SVHN: 85.79 @T=1

SYN: 34.55 @T=1

MNIST: 98.31 @T=20

USPS: 94.87 @T=20

SVHN: 92.33 @T=20

SYN: 98.70 @T=20

Figure 8. T-SNE visualization of FedProto [36] framework with VPDR on Digits at T = 1 (top row) and T = 20 (bottom row). Test accuracy (%) is shown above each subplot.

E.4. Model with Data Heterogeneity To evaluate VPDR under highly challenging conditions, we couple architectural diversity with two forms of data skew: (i) domain shift across multi-domain benchmarks, and (ii) label imbalance on CIFAR-10. We instantiate two four-way heterogeneous model families [52, 53]: • Heterogeneous Feature Extractors (HtFE4 ). Each client uses a different backbone architecture: ResNet18/34, ViT-Tiny/Small [8]. All backbones feed into a shared 64-channel adapter and global- or patch-mean

Table 9. Model heterogeneity with domain skew. Methods FedProto [36] FedTGP [53]

+IGPP +VPDR +IGPP +VPDR

HtFE4 HtC4 Digits Office PACS Digits Office PACS 94.93 95.24 95.11 95.31

87.26 88.59 85.07 87.58

88.81 90.46 89.37 89.97

91.54 93.36 92.36 93.52

81.90 84.44 72.67 85.25

88.54 91.84 89.78 91.56

Table 10. Model heterogeneity with label skew. Methods +IGPP FedProto [36] +VPDR +IGPP FedTGP [53] +VPDR

HtFE4 α = 0.1 α = 0.5

HtC4 α = 0.1 α = 0.5

37.42 37.94 35.16 36.91

32.66 34.45 31.38 34.13

76.32 76.90 75.22 76.34

69.67 71.27 66.60 70.47

pooling to produce 512-dimensional features, so that prototypes lie in a common embedding space while encoder family, depth, and capacity vary across clients. • Heterogeneous Classifiers (HtC4 ). All clients share a ResNet-18 backbone and the same 512-dimensional representation, but attach classifier heads with increasing depth and width (from a single 512-d linear layer to 2–3 layer MLPs with hidden sizes up to 1024 → 512). Under these two model families, we evaluate two data regimes. For domain skew, we follow the main setup on Digits, Office–Caltech (Office), and PACS and report average test accuracy (%) at ϵ = 1.0 (Table 9). For label skew, we use Dirichlet splits of CIFAR-10 with α ∈ {0.1, 0.5} under the same privacy budget (Table 10). In all cases we run FedProto [36] and FedTGP [53] with IGPP and our VPDR to test whether the gains of VPDR persist under joint model–data heterogeneity. Across all configurations in Tables 9 and 10, VPDR consistently matches or improves upon IGPP for both FedProto and FedTGP. Under domain skew, gains on Digits are modest due to saturated accuracy, but VPDR yields clearly larger improvements on Office–Caltech and PACS, especially under HtC4 . For label skew, VPDR again brings steady gains under HtFE4 and more pronounced gains under HtC4 , and never hurts performance even in the highly skewed case with α = 0.1. Overall, VPDR remains robust and often more effective than IGPP when both model architectures and data distributions are heterogeneous.

E.5. Generalization to Text Modality To verify that our framework generalizes beyond computer vision tasks, we evaluate VPDR on the AG News dataset, a 4-class topic classification benchmark containing 120,000 training articles (World, Sports, Business, Sci/Tech). Using a RoBERTa-Base backbone (M = 20, ϵ = 1.0, R = 5.0), Table 11 shows that VPDR consistently outperforms the IGPP baseline across all Dirichlet skew levels. Notably, under extreme label skew (α = 0.1), VPDR improves average accuracy by nearly 3%. This confirms that VPDR’s noise

redistribution and norm regularization effectively preserve semantic representations in dense natural language embeddings just as robustly as they do in visual features. Table 11. Average test accuracy (%) of FedProto on AG News. Methods

α = 0.1

α = 0.5

α = 1.0

α = 5.0

α = 10.0

+ IGPP + VPDR

34.27 37.00

72.56 73.75

85.72 86.51

89.36 89.73

90.06 90.17

E.6. More Details for Section 5.4 We provide detailed descriptions of the attack protocols and metrics used in the privacy risk evaluation in Section 5.4. E.6.1. Feature-Space Hijacking (FSH) FSH aims to reconstruct an input whose encoder feature matches a target class prototype. Given a target prototype p∗ and an encoder path ϕ(·) that mirrors the client’s prototype pathway, we minimize 2

L = ϕ(x) − p∗ 2 + λTV TV(x), where the first term minimizes the feature distance between the reconstructed input and the prototype, and the second term penalizes total variation (TV) to promote smoothness in the reconstructed image. Metrics for FSH are as follows. • Cosine similarity: Measures the similarity between the reconstructed feature and the target prototype. Lower values in [−1, 1] indicate weaker alignment and hence stronger privacy. • Classifier Feature Fréchet Distance (cFFD): Fréchet distance between the distribution of reconstructed features and the real class feature distribution. Larger cFFD indicates greater mismatch and thus better privacy. • Top-1 Hit (%): Percentage of reconstructed images whose nearest class prototype matches the target class. Lower Top-1 hit means the attacker fails to recover the correct class, reducing privacy risk. Implementation. We optimize an unconstrained parameterization with Adam (learning rate 0.01, up to 10,000 steps), mapping to [0, 1] via tanh. We set λTV = 10−3 . For practical federated settings, the attack supports both singletarget and batched modes, with early stopping (patience 300, tolerance 10−6 ). Input shapes are inferred from a validation loader, and the feature path ϕ(·) reuses the clients’ prototype-extraction branch. We attack up to 10 classes with one prototype per class and a batch size of 16. E.6.2. Membership Inference Attack (MIA) MIA is a passive attack where the adversary aims to determine whether a given sample was part of the model’s training set, under the assumption that training samples lie closer to their class prototypes than non-members. Let ϕ(·) be the encoder path used to form prototypes. For a sample x and

Table 12. Privacy Attack Results of FedProto on Digits under varying privacy budgets. ϵ LDP - NoLDP +IGPP 1 +VPP +VPDR +IGPP 2 +VPP +VPDR

Feature Reconstruction (FSH) Cos Sim↓ cFFD↑ Top-1 Hit(%)↓ 0.9999±0.0000 0.8253±0.0018 0.8285±0.0026 0.8289±0.0020 0.8722±0.0025 0.8736±0.0027 0.8717±0.0029

304.8±36.5 989.2±28.0 933.8±30.2 940.9±27.3 764.7±24.0 765.7±25.8 777.1±15.2

100±0.00 83.33±1.54 83.00±2.05 83.23±2.37 97.12±0.92 97.50±0.00 97.50±0.00

ROC-AUC→ 0.5

Membership Inference (MIA) TPR@1%FPR↓ Advantage→ 0

0.6094±0.0224 0.5531±0.0188 0.5539±0.0166 0.5460±0.0282 0.5775±0.0194 0.5754±0.0165 0.5727±0.0320

0.0611±0.0723 0.0288±0.0381 0.0270±0.0552 0.0257±0.0177 0.0397±0.0573 0.0383±0.0549 0.0390±0.0557

0.2037±0.0358 0.1515±0.0263 0.1539±0.0246 0.1586±0.0213 0.1788±0.0275 0.1761±0.0240 0.1723±0.0234

F1 Score↓ 0.5844±0.0733 0.5444±0.0820 0.5440±0.0856 0.5447±0.0766 0.5586±0.0845 0.5578±0.0816 0.5534±0.0755

Table 13. Privacy Attack Results of FedProto on PACS under varying privacy budgets. ϵ LDP - NoLDP +IGPP 1 +VPP +VPDR +IGPP 2 +VPP +VPDR

Feature Reconstruction (FSH) Cos Sim↓ cFFD↑ Top-1 Hit(%)↓ 0.9999±0.0000 0.7271±0.0031 0.7230±0.0028 0.7219±0.0028 0.7811±0.0030 0.7874±0.0028 0.7862±0.0029

334.3±47.7 1620.7±14.4 1612.2±19.4 1626.2±0.0125 1237.8±11.4 1216.8±23.5 1242.1±17.9

100.00±0.00 54.21±2.81 56.43±1.87 55.71±2.14 65.89±2.95 65.57±1.87 65.93±2.04

privatized prototype p̃cm , we define the membership score s(x; p̃cm ) = −

2 ϕ(x) − p̃cm 2 .

We compute this score for each sample and compare it against a decision threshold to classify the sample as either a member or not. By sweeping the threshold over all possible values of s(·), we obtain an ROC curve, and summarize attack performance via the area under the curve (AUC). Metrics for MIA are as follows. • ROC-AUC: Measures the attacker’s ability to distinguish training-set members from non-members. Values closer to 0.5 indicate chance-level discrimination and thus stronger privacy. • TPR@1%FPR: The true positive rate (TPR) at a fixed false positive rate (FPR) of 1%. Lower TPR values indicate that the attacker is unable to reliably distinguish training samples from non-members at a low FPR. • Advantage: Defined as TPR − FPR at a chosen threshold, measuring the net gain over random guessing. Values near zero indicate near-chance performance. Unlike AUC, which aggregates performance across all thresholds, this metric focuses on a single threshold. • F1 score: Harmonic mean of precision and recall, computed at the optimal threshold. A lower F1 score indicates poor attack accuracy in distinguishing members from non-members, corresponding to stronger privacy. Implementation. The maximum number of samples per class in training and testing is set to 800, and the decision threshold is dynamically calculated by maximizing the F1 score on the precision-recall curve. E.6.3. More Results on Digits and PACS Tables 12 and 13 report additional privacy attack results on Digits and PACS under the same threat model and metrics

ROC-AUC→ 0.5

Membership Inference (MIA) TPR@1%FPR↓ Advantage→ 0

0.5642±0.0110 0.5009±0.0108 0.5029±0.0098 0.5004±0.0083 0.5038±0.0093 0.5075±0.0107 0.5026±0.0080

0.0187±0.0037 0.0089±0.0024 0.0094±0.0031 0.0102±0.0024 0.0087±0.0032 0.0092±0.0031 0.0101±0.0022

0.2570±0.0125 0.1097±0.0128 0.1101±0.0126 0.1066±0.0096 0.1135±0.0113 0.1156±0.0127 0.1076±0.0103

F1 Score↓ 0.7258±0.0043 0.7083±0.0028 0.7067±0.0014 0.7073±0.0014 0.7088±0.0027 0.7085±0.0030 0.7076±0.0015

as in Section 5.4. Across both datasets and ϵ ∈ {1, 2}, all three LDP mechanisms (IGPP, VPP, and VPDR) consistently reduce FSH success and drive MIA towards nearchance performance compared to the non-private baseline. The differences among IGPP, VPP, and VPDR are within one standard deviation and show no systematic trend, confirming that VPP and VPDR attain privacy protection comparable to IGPP under the same LDP budgets.

E.7. Effect of d and r Varying the embedding dimension d and the budget split ratio r changes how the total privacy budget is allocated between subspace selection and prototype release, which may affect utility and empirical attack performance. Nevertheless, after calibrating the noise according to our analysis, the overall (ϵ, δ)–LDP guarantee remains the same. On PACS (Tab. 14), the round-averaged MIA-AUC stays close to chance level (≈ 0.5) across all tested d and r, indicating no observable increase in membership leakage. In contrast, accuracy exhibits mild variation with a weak interaction between d and r: larger dimensions (d ≥ 512) slightly prefer a moderate split (r ≈ 0.1), whereas d = 256 is more tolerant to larger r (e.g., r = 0.2). Across all settings, VPDR consistently improves accuracy over IGPP while maintaining comparable (near-chance) AUC. Table 14. Average test accuracy and round-averaged MIA-AUC. d

IGPP VPDR (r=0.05) VPDR (r=0.1) VPDR (r=0.2) AVG AUC AVG AUC AVG AUC AVG AUC

256 90.80% 0.5072 92.08% 0.5041 92.59% 0.5051 92.82% 0.5029 512 90.58% 0.5009 92.26% 0.5011 92.71% 0.5004 92.67% 0.5015 1024 91.13% 0.5019 92.54% 0.4966 92.67% 0.5003 92.28% 0.5011

Record · ID 146954 · SHA-256 2a46c2e107dcb1f1
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.